Skip to contentSkip to conversation

virtiofsd fixes

PATCH Applied Liu Bo 6 participants 31 messages Apr 16, 2019 – May 20, 2019 Arm/actions semi architecture

lore
Share & tools
lore

Liu Bo

author
#0 lore raw

cover virtiofsd fixes

Hi,

This set contains a few fixes against virtiofsd.

Patch 1 fixes dio read hang.

Patch 2 adds ns support for file timestamp in stat(2).

Patch 3 fixes a problem found by generic/413 which shows vhost-user backend doesn't include all memory mappings in use.

Patch 4 uses fallocate(2) instead to support more modes.

thanks,
liubo

Eryu Guan (1):
virtiofsd: send reply correctly on read failure

Jiufei Xue (1):
virtiofsd: support nanosecond resolution for file timestamp

Xiaoguang Wang (2):
virtiofsd: use file-backend memory region for virtiofsd's cache area
virtiofsd: use fallocate(2) instead posix_fallocate(3)

 configure                          | 16 +++++++++++
 contrib/virtiofsd/fuse_lowlevel.c  |  4 +--
 contrib/virtiofsd/fuse_misc.h      |  1 +
 contrib/virtiofsd/fuse_virtio.c    | 12 ++++++--
 contrib/virtiofsd/passthrough_ll.c | 11 ++-----
 hw/virtio/vhost-user-fs.c          | 59 +++++++++++++++++++++++++++++++-------
 6 files changed, 80 insertions(+), 23 deletions(-)
Signature · 1 line
1.8.3.1
author
#1 lore raw

PATCH 1/4 virtiofsd: send reply correctly on read failure

From: Eryu Guan <[email protected]>

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

---
 contrib/virtiofsd/fuse_lowlevel.c |  4 ++--
 contrib/virtiofsd/fuse_virtio.c   | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +2 -2
index 111c6e1..aeb5fe2 100644
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv,
 	out.error = 0;
 
 	res = fuse_send_data_iov(req->se, req->ch, iov, 1, bufv, flags);
-	if (res <= 0) {
+	if (res >= 0) {
 		fuse_free_req(req);
 		return res;
 	} else {
-		return fuse_reply_err(req, res);
+		return fuse_reply_err(req, -res);
 	}
 }
 
contrib/virtiofsd/fuse_virtio.c +9 -3
index ca988aa..0b5736d 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = -errno;
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
 err:
-        ch->qi->reply_sent = true;
+        if (!ret)
+                ch->qi->reply_sent = true;
 
         return ret;
 }
Signature · 1 line
1.8.3.1
#2 lore raw

Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure

Quoted 10 lines · * Liu Bo ([email protected]) wrote:

From: Eryu Guan <[email protected]>

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

Thank you for spotting this.

Quoted 12 lines
---
 contrib/virtiofsd/fuse_lowlevel.c |  4 ++--
 contrib/virtiofsd/fuse_virtio.c   | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +0 -0
index 111c6e1..aeb5fe2 100644

contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527

Quoted 5 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv,
 	out.error = 0;
 
 	res = fuse_send_data_iov(req->se, req->ch, iov, 1, bufv, flags);
-	if (res <= 0) {
+	if (res >= 0) {

I need to go and ask the upstream fuse list about this; it's not clear to me what fuse_send_data_iov is supposed to return; let me clarify that and get back to you.

Dave

Quoted 45 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 		fuse_free_req(req);
 		return res;
 	} else {
-		return fuse_reply_err(req, res);
+		return fuse_reply_err(req, -res);
 	}
 }
 
contrib/virtiofsd/fuse_virtio.c +9 -3
index ca988aa..0b5736d 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = -errno;
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
 err:
-        ch->qi->reply_sent = true;
+        if (!ret)
+                ch->qi->reply_sent = true;
 
         return ret;
 }
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#3 lore raw

Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure

Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:

From: Eryu Guan <[email protected]>

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

Thank you for spotting this.

Quoted 14 lines
---
 contrib/virtiofsd/fuse_lowlevel.c |  4 ++--
 contrib/virtiofsd/fuse_virtio.c   | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +0 -0
index 111c6e1..aeb5fe2 100644
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv,
 	out.error = 0;
 
Quoted 7 lines

contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527

Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 	res = fuse_send_data_iov(req->se, req->ch, iov, 1, bufv, flags);
-	if (res <= 0) {
+	if (res >= 0) {

I need to go and ask the upstream fuse list about this; it's not clear to me what fuse_send_data_iov is supposed to return; let me clarify that and get back to you.

Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807

so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.

Quoted 17 lines

Dave

Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 		fuse_free_req(req);
 		return res;
 	} else {
-		return fuse_reply_err(req, res);
+		return fuse_reply_err(req, -res);
 	}
 }
 
contrib/virtiofsd/fuse_virtio.c +0 -0
index ca988aa..0b5736d 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
Quoted 7 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339

Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4 -2
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = -errno;

So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)

Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3 -0
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
Quoted 4 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388

Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2 -1
 err:
-        ch->qi->reply_sent = true;
+        if (!ret)
+                ch->qi->reply_sent = true;

Yes, I think that's OK.

Dave

Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0 -0
         return ret;
 }
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
Quoted 7 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#4 lore raw

Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure

Quoted 37 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:

From: Eryu Guan <[email protected]>

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

Thank you for spotting this.

Quoted 14 lines
---
 contrib/virtiofsd/fuse_lowlevel.c |  4 ++--
 contrib/virtiofsd/fuse_virtio.c   | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +0 -0
index 111c6e1..aeb5fe2 100644
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv,
 	out.error = 0;
 
Quoted 7 lines

contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527

Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 	res = fuse_send_data_iov(req->se, req->ch, iov, 1, bufv, flags);
-	if (res <= 0) {
+	if (res >= 0) {

I need to go and ask the upstream fuse list about this; it's not clear to me what fuse_send_data_iov is supposed to return; let me clarify that and get back to you.

Quoted 7 lines

Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807

so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.

I see, that makes sense.

Quoted 17 lines
Quoted 17 lines

Dave

Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 		fuse_free_req(req);
 		return res;
 	} else {
-		return fuse_reply_err(req, res);
+		return fuse_reply_err(req, -res);
 	}
 }
 
contrib/virtiofsd/fuse_virtio.c +0 -0
index ca988aa..0b5736d 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
Quoted 13 lines
Quoted 7 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339

Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4 -2
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = -errno;

So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)

OK, do you want me to fix both in the same patch?

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3 -0
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
Quoted 6 lines
Quoted 4 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388

Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2 -1
 err:
-        ch->qi->reply_sent = true;
+        if (!ret)
+                ch->qi->reply_sent = true;

Yes, I think that's OK.

Correct, this is the one causing fuse kernel side's hang.

thanks,
-liubo

Quoted 22 lines

Dave

Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0 -0
         return ret;
 }
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
Quoted 7 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#5 lore raw

Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure

Quoted 79 lines · * Liu Bo ([email protected]) wrote:
Quoted 37 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:

From: Eryu Guan <[email protected]>

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

Thank you for spotting this.

Quoted 14 lines
---
 contrib/virtiofsd/fuse_lowlevel.c |  4 ++--
 contrib/virtiofsd/fuse_virtio.c   | 12 +++++++++---
 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +0 -0
index 111c6e1..aeb5fe2 100644
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv,
 	out.error = 0;
 
Quoted 7 lines

contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527

Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 	res = fuse_send_data_iov(req->se, req->ch, iov, 1, bufv, flags);
-	if (res <= 0) {
+	if (res >= 0) {

I need to go and ask the upstream fuse list about this; it's not clear to me what fuse_send_data_iov is supposed to return; let me clarify that and get back to you.

Quoted 7 lines

Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807

so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.

I see, that makes sense.

Quoted 17 lines
Quoted 17 lines

Dave

Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1 -1
 		fuse_free_req(req);
 		return res;
 	} else {
-		return fuse_reply_err(req, res);
+		return fuse_reply_err(req, -res);
 	}
 }
 
contrib/virtiofsd/fuse_virtio.c +0 -0
index ca988aa..0b5736d 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
Quoted 13 lines
Quoted 7 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339

Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4 -2
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = -errno;

So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)

Quoted 1 line

OK, do you want me to fix both in the same patch?

Yes please.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3 -0
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388

Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2 -1
 err:
-        ch->qi->reply_sent = true;
+        if (!ret)
+                ch->qi->reply_sent = true;

Yes, I think that's OK.

Correct, this is the one causing fuse kernel side's hang.

Thanks!

Dave (Note, I'm out for a week)

Quoted 25 lines

thanks,
-liubo

Quoted 22 lines

Dave

Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0 -0
         return ret;
 }
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
Quoted 7 lines

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#6 lore raw

PATCH v2 1/1 virtiofsd: send reply correctly on read failure

In reply to Liu Bo

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

---
v2:
- return postive error number on read error
- also return postive EIO in short read case
- drop "-1" -> "-EOPNOTSUPP" update, which should be in a sperate patch

 contrib/virtiofsd/fuse_virtio.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/virtiofsd/fuse_virtio.c +10 -4
index ca988aac923f..7337f2a2c3ab 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = errno;
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -356,7 +361,7 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 if (ret != len) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret!=len\n", __func__);
-                        ret = -EIO;
+                        ret = EIO;
                         free(in_sg_cpy);
                         goto err;
                 }
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
 err:
-        ch->qi->reply_sent = true;
+        if (ret == 0)
+                ch->qi->reply_sent = true;
 
         return ret;
 }
Signature · 1 line
2.14.4.44.g2045bb6
Reviewed-by on v2 1/1
#7 lore raw

Re: [Virtio-fs] [PATCH v2] virtiofsd: send reply correctly on read failure

Quoted 11 lines · * Eryu Guan ([email protected]) wrote:

Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.

This is easily triggered by a direct read with non-aligned length.

Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.

Thank you, I've added that to my dev tree.

Quoted 55 lines

---
v2:
- return postive error number on read error
- also return postive EIO in short read case
- drop "-1" -> "-EOPNOTSUPP" update, which should be in a sperate patch

 contrib/virtiofsd/fuse_virtio.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/virtiofsd/fuse_virtio.c +10 -4
index ca988aac923f..7337f2a2c3ab 100644
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 ret = preadv(buf->buf[0].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0].pos);
 
                 if (se->debug)
-                        fprintf(stderr, "%s: preadv_res=%d len=%zd\n",
-                                __func__, ret, len);
+                        fprintf(stderr, "%s: preadv_res=%d(%s) len=%zd\n",
+                                __func__, ret, strerror(errno), len);
+                if (ret == -1) {
+                        ret = errno;
+                        free(in_sg_cpy);
+                        goto err;
+                }
                 if (ret < len && ret) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret < len\n", __func__);
@@ -356,7 +361,7 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
                 if (ret != len) {
                         if (se->debug)
                                 fprintf(stderr, "%s: ret!=len\n", __func__);
-                        ret = -EIO;
+                        ret = EIO;
                         free(in_sg_cpy);
                         goto err;
                 }
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,
         vu_queue_notify(&se->virtio_dev->dev, q);
 
 err:
-        ch->qi->reply_sent = true;
+        if (ret == 0)
+                ch->qi->reply_sent = true;
 
         return ret;
 }
Signature · 6 lines
2.14.4.44.g2045bb6

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#8 lore raw

PATCH 2/4 virtiofsd: support nanosecond resolution for file timestamp

In reply to Liu Bo

From: Jiufei Xue <[email protected]>

Define HAVE_STRUCT_STAT_ST_ATIM to 1 if st_atim' is member of struct stat' which means support nanosecond resolution for the file timestamp fields.

---
 configure                     | 16 ++++++++++++++++
 contrib/virtiofsd/fuse_misc.h |  1 +
 2 files changed, 17 insertions(+)
configure +16 -0
index 5c21a65..57336ec 100755
@@ -5147,6 +5147,19 @@ if compile_prog "" "" ; then
     strchrnul=yes
 fi
 
+#########################################
+# check if we have st_atim
+
+st_atim=no
+cat > $TMPC << EOF
+#include <sys/stat.h>
+#include <stddef.h>
+int main(void) { return offsetof(struct stat, st_atim); }
+EOF
+if compile_prog "" "" ; then
+    st_atim=yes
+fi
+
 ##########################################
 # check if trace backend exists
 
@@ -6757,6 +6770,9 @@ fi
 if test "$strchrnul" = "yes" ; then
   echo "HAVE_STRCHRNUL=y" >> $config_host_mak
 fi
+if test "$st_atim" = "yes" ; then
+  echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak
+fi
 if test "$byteswap_h" = "yes" ; then
   echo "CONFIG_BYTESWAP_H=y" >> $config_host_mak
 fi
contrib/virtiofsd/fuse_misc.h +1 -0
index 2f6663e..30a1d7f 100644
@@ -7,6 +7,7 @@
 */
 
 #include <pthread.h>
+#include "config-host.h"
 
 /*
   Versioned symbols cannot be used in some cases because it
Signature · 1 line
1.8.3.1
Reviewed-by on v1 2/4 Applied
#9 lore raw

Re: [Virtio-fs] [PATCH 2/4] virtiofsd: support nanosecond resolution for file timestamp

Quoted 8 lines · * Liu Bo ([email protected]) wrote:

From: Jiufei Xue <[email protected]>

Define HAVE_STRUCT_STAT_ST_ATIM to 1 if st_atim' is member of struct stat' which means support nanosecond resolution for the file timestamp fields.

Thanks, applied to my tree.

Quoted 58 lines
---
 configure                     | 16 ++++++++++++++++
 contrib/virtiofsd/fuse_misc.h |  1 +
 2 files changed, 17 insertions(+)
configure +16 -0
index 5c21a65..57336ec 100755
@@ -5147,6 +5147,19 @@ if compile_prog "" "" ; then
     strchrnul=yes
 fi
 
+#########################################
+# check if we have st_atim
+
+st_atim=no
+cat > $TMPC << EOF
+#include <sys/stat.h>
+#include <stddef.h>
+int main(void) { return offsetof(struct stat, st_atim); }
+EOF
+if compile_prog "" "" ; then
+    st_atim=yes
+fi
+
 ##########################################
 # check if trace backend exists
 
@@ -6757,6 +6770,9 @@ fi
 if test "$strchrnul" = "yes" ; then
   echo "HAVE_STRCHRNUL=y" >> $config_host_mak
 fi
+if test "$st_atim" = "yes" ; then
+  echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak
+fi
 if test "$byteswap_h" = "yes" ; then
   echo "CONFIG_BYTESWAP_H=y" >> $config_host_mak
 fi
contrib/virtiofsd/fuse_misc.h +1 -0
index 2f6663e..30a1d7f 100644
@@ -7,6 +7,7 @@
 */
 
 #include <pthread.h>
+#include "config-host.h"
 
 /*
   Versioned symbols cannot be used in some cases because it
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#10 lore raw

PATCH 3/4 virtiofsd: use file-backend memory region for virtiofsd's cache area

In reply to Liu Bo

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

To fix this issue, we create this vhost cache area with a file backend memory area.

---
 hw/virtio/vhost-user-fs.c | 59 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 49 insertions(+), 10 deletions(-)
hw/virtio/vhost-user-fs.c +49 -10
index a5110a9..fc2ed38 100644
@@ -300,6 +300,35 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev, int idx)
     return vhost_virtqueue_pending(&fs->vhost_dev, idx);
 }
 
+static int vuf_get_tmpfile(size_t size)
+{
+    const char *tmpdir = g_get_tmp_dir();
+    gchar *fname;
+    int fd;
+
+    fname = g_strdup_printf("%s/virtio-fs-cache-XXXXXX", tmpdir);
+    fd = mkstemp(fname);
+    if (fd < 0) {
+        fprintf(stderr, "%s: mkstemp(%s) failed\n", __func__, fname);
+        g_free(fname);
+        return -1;
+    }
+
+    /*
+     * Note: this temp file would be deleted until file is closed or
+     * process exits.
+     */
+    unlink(fname);
+    if (ftruncate(fd, size) == -1) {
+        fprintf(stderr, "%s: ftruncate(%s) failed\n", __func__, fname);
+        close(fd);
+        fd = -1;
+    }
+
+    g_free(fname);
+    return fd;
+}
+
 static void vuf_device_realize(DeviceState *dev, Error **errp)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(dev);
@@ -349,16 +378,26 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)
                          "no smaller than the page size");
         return;
     }
-    /* We need a region with some host memory, 'ram' is the easiest */
-    memory_region_init_ram_nomigrate(&fs->cache, OBJECT(vdev),
-                       "virtio-fs-cache",
-                       fs->conf.cache_size, NULL);
-    /*
-     * But we don't actually want anyone reading/writing the raw
-     * area with no cache data.
-     */
-    mprotect(memory_region_get_ram_ptr(&fs->cache), fs->conf.cache_size,
-             PROT_NONE);
+
+    if (fs->conf.cache_size) {
+            int fd;
+
+            fd = vuf_get_tmpfile(fs->conf.cache_size);
+            if (fd < 0) {
+                    error_setg(errp, "failed to initialize virtio-fs-cache");
+                    return;
+            }
+            memory_region_init_ram_from_fd(&fs->cache, OBJECT(vdev),
+                                           "virtio-fs-cache",
+                                           fs->conf.cache_size, true, fd, NULL);
+
+            /*
+             * But we don't actually want anyone reading/writing the raw
+             * area with no cache data.
+             */
+            mprotect(memory_region_get_ram_ptr(&fs->cache),
+                     fs->conf.cache_size, PROT_NONE);
+    }
 
     if (!vhost_user_init(&fs->vhost_user, &fs->conf.chardev, errp)) {
         return;
Signature · 1 line
1.8.3.1
#11 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).

Dave

Quoted 90 lines
---
 hw/virtio/vhost-user-fs.c | 59 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 49 insertions(+), 10 deletions(-)
hw/virtio/vhost-user-fs.c +49 -10
index a5110a9..fc2ed38 100644
@@ -300,6 +300,35 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev, int idx)
     return vhost_virtqueue_pending(&fs->vhost_dev, idx);
 }
 
+static int vuf_get_tmpfile(size_t size)
+{
+    const char *tmpdir = g_get_tmp_dir();
+    gchar *fname;
+    int fd;
+
+    fname = g_strdup_printf("%s/virtio-fs-cache-XXXXXX", tmpdir);
+    fd = mkstemp(fname);
+    if (fd < 0) {
+        fprintf(stderr, "%s: mkstemp(%s) failed\n", __func__, fname);
+        g_free(fname);
+        return -1;
+    }
+
+    /*
+     * Note: this temp file would be deleted until file is closed or
+     * process exits.
+     */
+    unlink(fname);
+    if (ftruncate(fd, size) == -1) {
+        fprintf(stderr, "%s: ftruncate(%s) failed\n", __func__, fname);
+        close(fd);
+        fd = -1;
+    }
+
+    g_free(fname);
+    return fd;
+}
+
 static void vuf_device_realize(DeviceState *dev, Error **errp)
 {
     VirtIODevice *vdev = VIRTIO_DEVICE(dev);
@@ -349,16 +378,26 @@ static void vuf_device_realize(DeviceState *dev, Error **errp)
                          "no smaller than the page size");
         return;
     }
-    /* We need a region with some host memory, 'ram' is the easiest */
-    memory_region_init_ram_nomigrate(&fs->cache, OBJECT(vdev),
-                       "virtio-fs-cache",
-                       fs->conf.cache_size, NULL);
-    /*
-     * But we don't actually want anyone reading/writing the raw
-     * area with no cache data.
-     */
-    mprotect(memory_region_get_ram_ptr(&fs->cache), fs->conf.cache_size,
-             PROT_NONE);
+
+    if (fs->conf.cache_size) {
+            int fd;
+
+            fd = vuf_get_tmpfile(fs->conf.cache_size);
+            if (fd < 0) {
+                    error_setg(errp, "failed to initialize virtio-fs-cache");
+                    return;
+            }
+            memory_region_init_ram_from_fd(&fs->cache, OBJECT(vdev),
+                                           "virtio-fs-cache",
+                                           fs->conf.cache_size, true, fd, NULL);
+
+            /*
+             * But we don't actually want anyone reading/writing the raw
+             * area with no cache data.
+             */
+            mprotect(memory_region_get_ram_ptr(&fs->cache),
+                     fs->conf.cache_size, PROT_NONE);
+    }
 
     if (!vhost_user_init(&fs->vhost_user, &fs->conf.chardev, errp)) {
         return;
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#12 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 38 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 7 lines
Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Quoted 4 lines

It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).

If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.

Stefan

author
#13 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 47 lines · Stefan Hajnoczi
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 7 lines
Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 5 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Can you please elaborate how the buffer is outside guest RAM? Is it also via direct IO?

Quoted 8 lines
Quoted 4 lines

It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).

If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.

In case of direct IO, kvm is able to make sure that guest's dax mapping is sync'd with the underlying host mmap region, isn't it?

thanks,
-liubo

#14 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 8 lines
Quoted 5 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Can you please elaborate how the buffer is outside guest RAM? Is it also via direct IO?

The DAX window is a PCI BAR on the virtio-fs PCI device. It is not guest RAM.

vhost-user only shares guest RAM with the vhost-user device backend process (virtiofsd). Therefore virtiofsd does not have access to the contents of the DAX window.

This only happens when the virtio-fs file system is mounted with the "dax" option.

Quoted 11 lines
Quoted 8 lines
Quoted 4 lines

It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).

If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.

In case of direct IO, kvm is able to make sure that guest's dax mapping is sync'd with the underlying host mmap region, isn't it?

See my explanation above. In theory a virtiofsd could keep track of the mapping region by performing the same mmap(2) system calls as the QEMU process based on the FUSE_SETUPMAPPING requests (currently only QEMU does the mmaps). What I'm saying here is that it gets even harder when multiple virtiofsd processes are involved because they do not have access to each other's DAX windows.

Stefan

#15 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Thanks
Vivek

#16 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Quoted 7 lines

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.

Stefan

#17 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 68 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Quoted 7 lines

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

Quoted 8 lines

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.

Yes, this doesn't look too bad; I need to tweak
vu_queue_pop->vu_queue_map_desc->virtqueue_map_desc
to give back a list of unmappable parts of the iovec (assuming that the first few elements of the iovec are mappable then the rest aren't and not allowing weird mixes).

One thing that worries me a bit is that if we do a read() or write() in the qemu code, it might block on the mmap'd backing file.

Dave

Quoted 1 line

Stefan

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#18 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 78 lines · Dr. David Alan Gilbert
Quoted 68 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Quoted 7 lines

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

Quoted 8 lines

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.

Quoted 8 lines

Yes, this doesn't look too bad; I need to tweak
vu_queue_pop->vu_queue_map_desc->virtqueue_map_desc
to give back a list of unmappable parts of the iovec (assuming that the first few elements of the iovec are mappable then the rest aren't and not allowing weird mixes).

One thing that worries me a bit is that if we do a read() or write() in the qemu code, it might block on the mmap'd backing file.

Even a memcpy (no read()/write()) could block if the page is not resident :(. We have to expect that this operation could block.

Stefan

author
#19 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

In reply to Stefan Hajnoczi
Quoted 63 lines · Stefan Hajnoczi
Quoted 62 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

Quoted 9 lines
Quoted 2 lines

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

It is probably not easy to do.

Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs

p = mmap(foo1, ...);
write(foo2, p, ...);

virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.

And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?

thanks,
-liubo

Quoted 6 lines

Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.

Stefan

#20 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:

[..]

Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

Quoted 9 lines
Quoted 2 lines

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

It is probably not easy to do.

Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs

p = mmap(foo1, ...);
write(foo2, p, ...);

Quoted 7 lines

virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.

And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?

Hi Liu Bo,

Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger FUSE_WRITE request to daemon with source address being in dax window of first virtiofs instance.

This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before FUSE_WRITE comes in, mapping for foo1 has already been establied.

IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in FUSE_WRITE(foo2) request. Did I miss something?

Vivek

author
#21 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 48 lines · Vivek Goyal

On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:

[..]

Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

Quoted 9 lines
Quoted 2 lines

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

It is probably not easy to do.

Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs

p = mmap(foo1, ...);
write(foo2, p, ...);

Quoted 7 lines

virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.

And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?

Hi Liu Bo,

Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger FUSE_WRITE request to daemon with source address being in dax window of first virtiofs instance.

Quoted 4 lines

This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before FUSE_WRITE comes in, mapping for foo1 has already been establied.

Bofore sending any message to qemu, fv_queue_thread() will fail at vu_queue_pop()->vu_queue_map_desc()->virtqueue_map_desc()->vu_gpa_to_va(), because virtiofsd (where foo2 is on) doesn't have the information of foo1's mapping within its address space.

It could be reproduced by fstests/generic/413.

thanks,
-liubo

Quoted 6 lines

IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in FUSE_WRITE(foo2) request. Did I miss something?

Vivek

#22 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

Quoted 54 lines · * Liu Bo ([email protected]) wrote:
Quoted 48 lines · Vivek Goyal

On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:

[..]

Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

Quoted 9 lines
Quoted 2 lines

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

It is probably not easy to do.

Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs

p = mmap(foo1, ...);
write(foo2, p, ...);

Quoted 7 lines

virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.

And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?

Hi Liu Bo,

Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger FUSE_WRITE request to daemon with source address being in dax window of first virtiofs instance.

Quoted 4 lines

This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before FUSE_WRITE comes in, mapping for foo1 has already been establied.

Quoted 4 lines

Bofore sending any message to qemu, fv_queue_thread() will fail at vu_queue_pop()->vu_queue_map_desc()->virtqueue_map_desc()->vu_gpa_to_va(), because virtiofsd (where foo2 is on) doesn't have the information of foo1's mapping within its address space.

The trick is to bounce the problem back to qemu; as soon as the mapping fails I add a flag to say the mapping failed; then that causes the request to be sent to qemu via a slave command - and qemu has all of the mappings so can read/write to all of the fs's.

Dave

Quoted 11 lines

It could be reproduced by fstests/generic/413.

thanks,
-liubo

Quoted 6 lines

IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in FUSE_WRITE(foo2) request. Did I miss something?

Vivek

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#23 lore raw

Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area

In reply to Liu Bo
Quoted 82 lines · * Liu Bo ([email protected]) wrote:
Quoted 63 lines · Stefan Hajnoczi
Quoted 62 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:

[  247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.167171] t_mmap_dio      D    0  2335   2102 0x00000000
[  247.168006] Call Trace:
[  247.169067]  ? __schedule+0x3d0/0x830
[  247.170219]  schedule+0x32/0x80
[  247.171328]  schedule_timeout+0x1e2/0x350
[  247.172416]  ? fuse_direct_io+0x2e5/0x6b0 [fuse]
[  247.173516]  wait_for_completion+0x123/0x190
[  247.174593]  ? wake_up_q+0x70/0x70
[  247.175640]  fuse_direct_IO+0x265/0x310 [fuse]
[  247.176724]  generic_file_read_iter+0xaa/0xd20
[  247.177824]  fuse_file_read_iter+0x81/0x130 [fuse]
[  247.178938]  ? fuse_simple_request+0x104/0x1b0 [fuse]
[  247.180041]  ? fuse_fsync_common+0xad/0x240 [fuse]
[  247.181136]  __vfs_read+0x108/0x190
[  247.181930]  vfs_read+0x91/0x130
[  247.182671]  ksys_read+0x52/0xc0
[  247.183454]  do_syscall_64+0x55/0x170
[  247.184200]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.

Quoted 2 lines

To fix this issue, we create this vhost cache area with a file backend memory area.

Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

That case happened with cache=none and the dax mount option.

The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.

Stefan,

Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?

Quoted 9 lines
Quoted 2 lines

This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.

Yes, interesting idea!

When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.

It is probably not easy to do.

Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs

p = mmap(foo1, ...);
write(foo2, p, ...);

Quoted 7 lines

virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.

And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?

I have a patchset I'm just tidying up that passes this case back to qemu to handle. I intend to post it by the end of the week.

What it does is that when the virtiofsd receives a read/write to an area of memory that it doesn't have a mapping for, it forms a new slave message back to qemu together with the fd asking qemu to read/write at the given GPA. Then it's upto QEMU to deal with it.

That should work even if there are two separate daemons.

It's not a pretty solution; but I think it should work.

Dave

Quoted 9 lines

thanks,
-liubo

Quoted 6 lines

Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.

Stefan

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#24 lore raw

PATCH 4/4 virtiofsd: use fallocate(2) instead posix_fallocate(3)

In reply to Liu Bo

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

---
 contrib/virtiofsd/passthrough_ll.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
contrib/virtiofsd/passthrough_ll.c +3 -8
index 10ea8aa..ccb5312 100644
@@ -1568,15 +1568,10 @@ static void lo_fallocate(fuse_req_t req, fuse_ino_t ino, int mode,
 			 off_t offset, off_t length, struct fuse_file_info *fi)
 {
 	int err;
-	(void) ino;
-
-	if (mode) {
-		fuse_reply_err(req, EOPNOTSUPP);
-		return;
-	}
 
-	err = posix_fallocate(lo_fi_fd(req, fi), offset,
-			      length);
+	err = fallocate(lo_fi_fd(req, fi), mode, offset, length);
+        if (err < 0)
+                err = errno;
 
 	fuse_reply_err(req, err);
 }
Signature · 1 line
1.8.3.1
#25 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 8 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

Miklos: Do you know?

Dave

Quoted 34 lines
---
 contrib/virtiofsd/passthrough_ll.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
contrib/virtiofsd/passthrough_ll.c +3 -8
index 10ea8aa..ccb5312 100644
@@ -1568,15 +1568,10 @@ static void lo_fallocate(fuse_req_t req, fuse_ino_t ino, int mode,
 			 off_t offset, off_t length, struct fuse_file_info *fi)
 {
 	int err;
-	(void) ino;
-
-	if (mode) {
-		fuse_reply_err(req, EOPNOTSUPP);
-		return;
-	}
 
-	err = posix_fallocate(lo_fi_fd(req, fi), offset,
-			      length);
+	err = fallocate(lo_fi_fd(req, fi), mode, offset, length);
+        if (err < 0)
+                err = errno;
 
 	fuse_reply_err(req, err);
 }
Signature · 6 lines
1.8.3.1

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#26 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

Quoted 4 lines
Quoted 1 line

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

The patch looks good to me.

Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.

Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.

Thanks,
Miklos

#27 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 16 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

Quoted 4 lines
Quoted 1 line

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

Quoted 8 lines

The patch looks good to me.

Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.

Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.

OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?

Dave

Quoted 2 lines

Thanks,
Miklos

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#28 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 23 lines · Dr. David Alan Gilbert
Quoted 21 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

Quoted 4 lines
Quoted 1 line

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

The patch looks good to me.

Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.

Quoted 6 lines
Quoted 3 lines

Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.

OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?

It would have to be wrapped in #ifdef HAVE_FALLOCATE to make it
portable. Otherwise I don't see why not.

Thanks,
Miklos

#29 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 29 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

Quoted 4 lines
Quoted 1 line

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

The patch looks good to me.

Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.

Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.

Quoted 5 lines
Quoted 2 lines

OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?

It would have to be wrapped in #ifdef HAVE_FALLOCATE to make it
portable. Otherwise I don't see why not.

OK, I'll take it for now.

Liu Bo: Can you make it portable please and post it upstream on the fuse list?

Dave

Quoted 2 lines

Thanks,
Miklos

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#30 lore raw

Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)

Quoted 33 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:

From: Xiaoguang Wang <[email protected]>

This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.

Quoted 4 lines
Quoted 1 line

We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?

Quoted 8 lines

The patch looks good to me.

Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.

Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.

OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?

Quoted 8 lines
Quoted 2 lines

It would have to be wrapped in #ifdef HAVE_FALLOCATE to make it
portable. Otherwise I don't see why not.

OK, I'll take it for now.

Liu Bo: Can you make it portable please and post it upstream on the fuse list?

No problem, will do it.

thanks,
-liubo