Skip to contentSkip to conversation

vhost: support cross page buf in async data path

PATCH In review [email protected] 3 participants 16 messages Jul 20, 2020 – Jul 29, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vhost: support cross page buf in async data path

From: Patrick Fu <[email protected]>

Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")
---
 lib/librte_vhost/virtio_net.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
lib/librte_vhost/virtio_net.c +3 -9
index 1d0be3dd4..44b22a8ad 100644
@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 		}
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
+		hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
+				buf_iova + buf_offset, cpy_len);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
-
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
-
+		if (unlikely(cpy_len >= cpy_threshold && hpa)) {
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
 						mbuf_offset), cpy_len);
Signature · 1 line
2.18.4
#1 lore raw

The title could be improved, it is not very clear IMHO.

Quoted 5 lines · On 7/20/20 4:52 AM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.

Wouldn't it be possible to have the buffer split into two iovecs?

Quoted 32 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")
---
 lib/librte_vhost/virtio_net.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
lib/librte_vhost/virtio_net.c +3 -9
index 1d0be3dd4..44b22a8ad 100644
@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 		}
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
+		hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
+				buf_iova + buf_offset, cpy_len);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
-
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
-
+		if (unlikely(cpy_len >= cpy_threshold && hpa)) {
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
 						mbuf_offset), cpy_len);
#2 lore raw

Hi Maxime,

Quoted 8 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data path

The title could be improved, it is not very clear IMHO.

How about:
vhost: fix async copy failure on buffers cross page boundary

Quoted 8 lines · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 5 lines

From: Patrick Fu <[email protected]>

Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.

Wouldn't it be possible to have the buffer split into two iovecs?

Technically we can do that, however, it will also introduce significant overhead:
- overhead from adding additional logic in vhost async data path to handle the case
- overhead from dma device to consume 2 iovecs
In average, I don't think dma copy can benefit too much for the buffer which are split into multiple pages. CPU copy shall be a more suitable method.

Quoted 33 lines

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev,

Quoted 12 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")
---
 lib/librte_vhost/virtio_net.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
(quoted diff) +0 -0

b/lib/librte_vhost/virtio_net.c index 1d0be3dd4..44b22a8ad 100644

lib/librte_vhost/virtio_net.c +0 -0
@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev,

struct vhost_virtqueue *vq,

Quoted 20 lines
lib/librte_vhost/virtio_net.c +3 -9
 		}
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
+		hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
+				buf_iova + buf_offset, cpy_len);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
-
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
-
+		if (unlikely(cpy_len >= cpy_threshold && hpa)) {
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
 						mbuf_offset), cpy_len);
#3 lore raw

Hi Patrick,

Quoted 20 lines · Patrick

Hi Maxime,

Quoted 8 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data path

The title could be improved, it is not very clear IMHO.

How about:
vhost: fix async copy failure on buffers cross page boundary

Quoted 7 lines · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 5 lines

From: Patrick Fu <[email protected]>

Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.

Quoted 6 lines
Quoted 1 line

Wouldn't it be possible to have the buffer split into two iovecs?

Technically we can do that, however, it will also introduce significant overhead:
- overhead from adding additional logic in vhost async data path to handle the case
- overhead from dma device to consume 2 iovecs
In average, I don't think dma copy can benefit too much for the buffer which are split into multiple pages. CPU copy shall be a more suitable method.

I think we should try, that would make a cleaner implementation. I don't think having to fallback to sync mode is a good idea because it adds an overhead on the CPU, which is what we try to avoid with this async mode.

Also, I am not convinced the overhead would be that significant, at least I hope so, otherwise it would mean this new path is just performing better because it takes a lot of shortcuts, like the vector path in Virtio PMD.

Regards,
Maxime

Quoted 35 lines
Quoted 34 lines

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev,

Quoted 13 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")
---
 lib/librte_vhost/virtio_net.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)
(quoted diff) +0 -0

b/lib/librte_vhost/virtio_net.c index 1d0be3dd4..44b22a8ad 100644

lib/librte_vhost/virtio_net.c +0 -0
@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev,

struct vhost_virtqueue *vq,

Quoted 20 lines
lib/librte_vhost/virtio_net.c +3 -9
 		}
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
+		hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
+				buf_iova + buf_offset, cpy_len);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
-
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
-
+		if (unlikely(cpy_len >= cpy_threshold && hpa)) {
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
 						mbuf_offset), cpy_len);
#4 lore raw

Hi Maxime,

Quoted 41 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 4:35 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data path

Hi Patrick,

Quoted 15 lines · Patrick

Hi Maxime,

Quoted 9 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data path

The title could be improved, it is not very clear IMHO.

How about:
vhost: fix async copy failure on buffers cross page boundary

Quoted 1 line · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

From: Patrick Fu <[email protected]>

Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.

Wouldn't it be possible to have the buffer split into two iovecs?

Technically we can do that, however, it will also introduce significant

overhead:

Quoted 4 lines

- overhead from adding additional logic in vhost async data path to
handle the case
- overhead from dma device to consume 2 iovecs In average, I don't
think dma copy can benefit too much for the buffer which are split into

multiple pages.

Quoted 1 line

CPU copy shall be a more suitable method.

Quoted 7 lines

I think we should try, that would make a cleaner implementation. I don't think having to fallback to sync mode is a good idea because it adds an overhead on the CPU, which is what we try to avoid with this async mode.

Also, I am not convinced the overhead would be that significant, at least I hope so, otherwise it would mean this new path is just performing better because it takes a lot of shortcuts, like the vector path in Virtio PMD.

I can make a trial patch and do some comparison. I will try to feedback the result by this weekend.

Thanks,

Patrick

#5 lore raw

PATCH v2 1/1 vhost: fix async copy fail on multi-page buffers

In reply to [email protected]

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
index 0f7212f88..05c202a57 100644
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size)
+{
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)
 {
lib/librte_vhost/virtio_net.c +25 -15
index 95a0bc19f..124a33a10 100644
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			}
 		}
 
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}
 	}
 
 out:
Signature · 1 line
2.18.4
#6 lore raw

PATCH v3 1/1 vhost: fix async copy fail on multi-page buffers

In reply to [email protected]

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
index 0f7212f88..05c202a57 100644
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size)
+{
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)
 {
lib/librte_vhost/virtio_net.c +25 -15
index 95a0bc19f..124a33a10 100644
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			}
 		}
 
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}
 	}
 
 out:
Signature · 1 line
2.18.4
#7 lore raw

Re: [dpdk-dev] [PATCH v3] vhost: fix async copy fail on multi-page buffers

Hi Patrick,

Quoted 63 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Monday, July 27, 2020 2:33 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v3] vhost: fix async copy fail on multi-page buffers

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the copy into
multiple async copy segments whenever possible

v3:
- added fixline

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h index renamed from lib/librte_vhost/vhost.h +0 -0

0f7212f88..05c202a57 100644

lib/librte_vhost/vhost.h +21 -0
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa,
 uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size) {
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;

lib/librte_vhost/vhost.h gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, · line 650

Quoted 10 lines
lib/librte_vhost/vhost.h +10 -0
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <

Should the '<' be '<=' here?

Quoted 59 lines
lib/librte_vhost/vhost.h +18 -0
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)  { diff --git

a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c index 95a0bc19f..124a33a10 100644

lib/librte_vhost/virtio_net.c +1 -0
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct
 vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
lib/librte_vhost/virtio_net.c +7 -9
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct
 vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct · line 1097

Quoted 10 lines
lib/librte_vhost/virtio_net.c +8 -1
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;

Will it be ok we just transform the uint64_t to uint32_t here? What if mapped_len > MAX uint32_t ?

Thanks!
Chenbo

Quoted 28 lines
lib/librte_vhost/virtio_net.c +3 -1
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
lib/librte_vhost/virtio_net.c +6 -4
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct
 vhost_virtqueue *vq,
 			}
 		}
 
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}
 	}
 

out:
--
2.18.4

#8 lore raw

Re: [dpdk-dev] [PATCH v3] vhost: fix async copy fail on multi-page buffers

Hi,

Quoted 21 lines

-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, July 27, 2020 9:14 PM
To: Fu, Patrick <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v3] vhost: fix async copy fail on multi-page buffers

Hi Patrick,

Quoted 12 lines

-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Monday, July 27, 2020 2:33 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v3] vhost: fix async copy fail on multi-page buffers

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
Quoted 9 lines

lib/librte_vhost/vhost.h gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size) · line 650

Quoted 6 lines
lib/librte_vhost/vhost.h +6 -0
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <

Should the '<' be '<=' here?

Yes, it should be. Will update in v4 patch

Quoted 45 lines
Quoted 45 lines

static __rte_always_inline uint64_t
hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len) {

(quoted diff) +0 -0

b/lib/librte_vhost/virtio_net.c index 95a0bc19f..124a33a10 100644

lib/librte_vhost/virtio_net.c +1 -0
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct
 vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
lib/librte_vhost/virtio_net.c +9 -9
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,
 struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
Quoted 11 lines

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1096

Quoted 7 lines
lib/librte_vhost/virtio_net.c +6 -1
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;

Will it be ok we just transform the uint64_t to uint32_t here? What if mapped_len > MAX uint32_t ?

"mapped_len" could not exceed "cpy_len" (according to the gpa_to_first_hpa() logic). As cpy_len is uint32, mapped_len is safe to down casting.

Thanks,

Patrick

#9 lore raw

PATCH v4 1/1 vhost: fix async copy fail on multi-page buffers

In reply to [email protected]

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
index 0f7212f88..05c202a57 100644
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size)
+{
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)
 {
lib/librte_vhost/virtio_net.c +25 -15
index 95a0bc19f..124a33a10 100644
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			}
 		}
 
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}
 	}
 
 out:
Signature · 1 line
2.18.4
#10 lore raw

Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers

Quoted 150 lines · On 7/28/20 5:28 AM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
index 0f7212f88..05c202a57 100644
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size)
+{
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)
 {
lib/librte_vhost/virtio_net.c +19 -11
index 95a0bc19f..124a33a10 100644
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 			}
 		}
 

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128

Quoted 10 lines
lib/librte_vhost/virtio_net.c +6 -4
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}

Is that really necessary to check if copy length is not 0?

Thanks,
Maxime

Quoted 4 lines
lib/librte_vhost/virtio_net.c +0 -0
 	}
 
 out:
#11 lore raw

Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers

Hi Maxime,

Quoted 103 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 28, 2020 9:55 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v4] vhost: fix async copy fail on multi-page buffers

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct

Quoted 35 lines · On 7/28/20 5:28 AM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
(quoted diff) +0 -0

b/lib/librte_vhost/virtio_net.c index 95a0bc19f..124a33a10 100644

lib/librte_vhost/virtio_net.c +0 -0
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct

vhost_virtqueue *vq,

Quoted 50 lines
lib/librte_vhost/virtio_net.c +19 -11
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,
 struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
Quoted 1 line

(void *)((uintptr_t)(buf_addr + buf_offset)),

@@ -1112,10

Quoted 1 line

+1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct

vhost_virtqueue *vq,

Quoted 3 lines
lib/librte_vhost/virtio_net.c +0 -0
 			}
 		}
Quoted 13 lines

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128

Quoted 10 lines
lib/librte_vhost/virtio_net.c +6 -4
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}

Is that really necessary to check if copy length is not 0?

The intension is to optimize for the case that ring buffers are NOT split (which should be the most common case). In that case, cpy_len will be zero and by this "if" statement we can save couple of cycles. With that said, the actual difference is minor. I'm open with either adding an "unlikely" to the "if", or removing this the "if". Would like to hear your option and submit modified patch.

Quoted 7 lines

Thanks,
Maxime

Quoted 4 lines

}

out:

#12 lore raw

Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers

Hi Maxime,

Quoted 129 lines

-----Original Message-----
From: Fu, Patrick
Sent: Wednesday, July 29, 2020 9:40 AM
To: 'Maxime Coquelin' <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: RE: [PATCH v4] vhost: fix async copy fail on multi-page buffers

Hi Maxime,

Quoted 10 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 28, 2020 9:55 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v4] vhost: fix async copy fail on multi-page buffers

Quoted 29 lines
Quoted 28 lines · On 7/28/20 5:28 AM, [email protected] wrote:

From: Patrick Fu <[email protected]>

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split

ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size

 lib/librte_vhost/vhost.h      | 50

+++++++++++++++++++++++++++++++++++

Quoted 67 lines
Quoted 9 lines
 lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++-----------
 2 files changed, 75 insertions(+), 15 deletions(-)
(quoted diff) +0 -0

b/lib/librte_vhost/virtio_net.c index 95a0bc19f..124a33a10 100644

lib/librte_vhost/virtio_net.c +0 -0
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev,

struct

vhost_virtqueue *vq,

Quoted 50 lines
lib/librte_vhost/virtio_net.c +19 -11
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,
 struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
Quoted 1 line

(void *)((uintptr_t)(buf_addr + buf_offset)),

@@ -1112,10

Quoted 1 line

+1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct

vhost_virtqueue *vq,

Quoted 3 lines
lib/librte_vhost/virtio_net.c +0 -0
 			}
 		}
Quoted 13 lines

lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128

Quoted 10 lines
lib/librte_vhost/virtio_net.c +6 -4
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
+		if (cpy_len) {
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
+		}

Is that really necessary to check if copy length is not 0?

Quoted 7 lines

The intension is to optimize for the case that ring buffers are NOT split (which should be the most common case). In that case, cpy_len will be zero and by this "if" statement we can save couple of cycles. With that said, the actual difference is minor. I'm open with either adding an "unlikely" to the "if", or removing this the "if". Would like to hear your option and submit modified patch.

I have a better way to handle the case (combine this "if" logic with the previous one). Please review my v5 patch for the code change.

Thanks,

Patrick

author
#13 lore raw

PATCH v5 1/1 vhost: fix async copy fail on multi-page buffers

In reply to [email protected]

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement in async_mbuf_to_desc()

 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++-----------
 2 files changed, 74 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
index 0f7212f88..070eb960f 100644
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)
 	return 0;
 }
 
+static __rte_always_inline rte_iova_t
+gpa_to_first_hpa(struct virtio_net *dev, uint64_t gpa,
+	uint64_t gpa_size, uint64_t *hpa_size)
+{
+	uint32_t i;
+	struct guest_page *page;
+	struct guest_page key;
+
+	*hpa_size = gpa_size;
+	if (dev->nr_guest_pages >= VHOST_BINARY_SEARCH_THRESH) {
+		key.guest_phys_addr = gpa & ~(dev->guest_pages[0].size - 1);
+		page = bsearch(&key, dev->guest_pages, dev->nr_guest_pages,
+			       sizeof(struct guest_page), guest_page_addrcmp);
+		if (page) {
+			if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			} else if (gpa < page->guest_phys_addr +
+						page->size) {
+				*hpa_size = page->guest_phys_addr +
+					page->size - gpa;
+				return gpa - page->guest_phys_addr +
+					page->host_phys_addr;
+			}
+		}
+	} else {
+		for (i = 0; i < dev->nr_guest_pages; i++) {
+			page = &dev->guest_pages[i];
+
+			if (gpa >= page->guest_phys_addr) {
+				if (gpa + gpa_size <=
+					page->guest_phys_addr + page->size) {
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				} else if (gpa < page->guest_phys_addr +
+							page->size) {
+					*hpa_size = page->guest_phys_addr +
+						page->size - gpa;
+					return gpa - page->guest_phys_addr +
+						page->host_phys_addr;
+				}
+			}
+		}
+	}
+
+	*hpa_size = 0;
+	return 0;
+}
+
 static __rte_always_inline uint64_t
 hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len)
 {
lib/librte_vhost/virtio_net.c +24 -15
index 95a0bc19f..bd9303c8a 100644
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
 	struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL;
 	int error = 0;
+	uint64_t mapped_len;
 
 	uint32_t tlen = 0;
 	int tvec_idx = 0;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 
 		cpy_len = RTE_MIN(buf_avail, mbuf_avail);
 
-		if (unlikely(cpy_len >= cpy_threshold)) {
-			hpa = (void *)(uintptr_t)gpa_to_hpa(dev,
-					buf_iova + buf_offset, cpy_len);
+		while (unlikely(cpy_len && cpy_len >= cpy_threshold)) {
+			hpa = (void *)(uintptr_t)gpa_to_first_hpa(dev,
+					buf_iova + buf_offset,
+					cpy_len, &mapped_len);
 
-			if (unlikely(!hpa)) {
-				error = -1;
-				goto out;
-			}
+			if (unlikely(!hpa || mapped_len < cpy_threshold))
+				break;
 
 			async_fill_vec(src_iovec + tvec_idx,
 				(void *)(uintptr_t)rte_pktmbuf_iova_offset(m,
-						mbuf_offset), cpy_len);
+				mbuf_offset), (size_t)mapped_len);
 
-			async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len);
+			async_fill_vec(dst_iovec + tvec_idx,
+					hpa, (size_t)mapped_len);
 
-			tlen += cpy_len;
+			tlen += (uint32_t)mapped_len;
+			cpy_len -= (uint32_t)mapped_len;
+			mbuf_avail  -= (uint32_t)mapped_len;
+			mbuf_offset += (uint32_t)mapped_len;
+			buf_avail  -= (uint32_t)mapped_len;
+			buf_offset += (uint32_t)mapped_len;
 			tvec_idx++;
-		} else {
+		}
+
+		if (likely(cpy_len)) {
 			if (unlikely(vq->batch_copy_nb_elems >= vq->size)) {
 				rte_memcpy(
 				(void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1110,12 +1118,13 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
 					cpy_len;
 				vq->batch_copy_nb_elems++;
 			}
+
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			buf_avail  -= cpy_len;
+			buf_offset += cpy_len;
 		}
 
-		mbuf_avail  -= cpy_len;
-		mbuf_offset += cpy_len;
-		buf_avail  -= cpy_len;
-		buf_offset += cpy_len;
 	}
 
 out:
Signature · 1 line
2.18.4
Reviewed-by on v5 1/1
#14 lore raw

Re: [dpdk-dev] [PATCH v5] vhost: fix async copy fail on multi-page buffers

Quoted 26 lines · Patrick Fu

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement in async_mbuf_to_desc()

Quoted 3 lines
 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++-----------
 2 files changed, 74 insertions(+), 15 deletions(-)

Thanks Patrick, it looks better to me now:

Maxime

#15 lore raw

Re: [dpdk-dev] [PATCH v5] vhost: fix async copy fail on multi-page buffers

In reply to Patrick Fu
Quoted 26 lines · Patrick Fu

Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.

Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")

---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible

v3:
- added fixline

v4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement in async_mbuf_to_desc()

Quoted 3 lines
 lib/librte_vhost/vhost.h      | 50 +++++++++++++++++++++++++++++++++++
 lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++-----------
 2 files changed, 74 insertions(+), 15 deletions(-)

Applied to dpdk-next-virtio/master.

Thanks,
Maxime