vhost: support cross page buf in async data path
PATCH 1/1 vhost: support cross page buf in async data path
From: Patrick Fu <[email protected]>
Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.
--- lib/librte_vhost/virtio_net.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)
lib/librte_vhost/virtio_net.c +3 -9
@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } cpy_len =RTE_MIN (buf_avail, mbuf_avail); + hpa = (void *)(uintptr_t )gpa_to_hpa (dev, + buf_iova + buf_offset, cpy_len); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); - -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } - +if (unlikely (cpy_len >= cpy_threshold && hpa)) {async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, mbuf_offset), cpy_len);
Signature · 1 line
2.18.4
The title could be improved, it is not very clear IMHO.
Quoted 5 lines · On 7/20/20 4:52 AM, [email protected] wrote:
From: Patrick Fu <[email protected]>
Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.
Wouldn't it be possible to have the buffer split into two iovecs?
Quoted 32 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>--- lib/librte_vhost/virtio_net.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)lib/librte_vhost/virtio_net.c +3
-9@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } cpy_len =RTE_MIN (buf_avail, mbuf_avail); + hpa = (void *)(uintptr_t )gpa_to_hpa (dev, + buf_iova + buf_offset, cpy_len); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); - -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } - +if (unlikely (cpy_len >= cpy_threshold && hpa)) {async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, mbuf_offset), cpy_len);
Hi Maxime,
Quoted 8 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data pathThe title could be improved, it is not very clear IMHO.
How about:
vhost: fix async copy failure on buffers cross page boundary
Quoted 8 lines · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 5 lines
From: Patrick Fu <[email protected]>
Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.
Wouldn't it be possible to have the buffer split into two iovecs?
Technically we can do that, however, it will also introduce significant overhead:
- overhead from adding additional logic in vhost async data path to handle the case
- overhead from dma device to consume 2 iovecs
In average, I don't think dma copy can benefit too much for the buffer which are split into multiple pages. CPU copy shall be a more suitable method.
Quoted 33 lines
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev,
Quoted 12 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>--- lib/librte_vhost/virtio_net.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)(quoted diff) +0
-0
b/lib/librte_vhost/virtio_net.cindex1d0be3dd4..44b22a8ad100644lib/librte_vhost/virtio_net.c +0
-0@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev,struct
vhost_virtqueue*vq,Quoted 20 lines
lib/librte_vhost/virtio_net.c +3
-9} cpy_len =RTE_MIN (buf_avail, mbuf_avail); + hpa = (void *)(uintptr_t )gpa_to_hpa (dev, + buf_iova + buf_offset, cpy_len); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); - -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } - +if (unlikely (cpy_len >= cpy_threshold && hpa)) {async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, mbuf_offset), cpy_len);
Hi Patrick,
Quoted 20 lines · Patrick
Hi Maxime,
Quoted 8 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data pathThe title could be improved, it is not very clear IMHO.
How about:
vhost: fix async copy failure on buffers cross page boundaryQuoted 7 lines · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 5 lines
From: Patrick Fu <[email protected]>
Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.
Quoted 6 lines
Quoted 1 line
Wouldn't it be possible to have the buffer split into two iovecs?
Technically we can do that, however, it will also introduce significant overhead:
- overhead from adding additional logic in vhost async data path to handle the case
- overhead from dma device to consume 2 iovecs
In average, I don't think dma copy can benefit too much for the buffer which are split into multiple pages. CPU copy shall be a more suitable method.
I think we should try, that would make a cleaner implementation. I don't think having to fallback to sync mode is a good idea because it adds an overhead on the CPU, which is what we try to avoid with this async mode.
Also, I am not convinced the overhead would be that significant, at least I hope so, otherwise it would mean this new path is just performing better because it takes a lot of shortcuts, like the vector path in Virtio PMD.
Regards,
Maxime
Quoted 35 lines
Quoted 34 lines
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev,
Quoted 13 lines
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>--- lib/librte_vhost/virtio_net.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-)(quoted diff) +0
-0
b/lib/librte_vhost/virtio_net.cindex1d0be3dd4..44b22a8ad100644lib/librte_vhost/virtio_net.c +0
-0@@ -1071,16 +1071,10 @@ async_mbuf_to_desc(struct virtio_net *dev,struct
vhost_virtqueue*vq,Quoted 20 lines
lib/librte_vhost/virtio_net.c +3
-9} cpy_len =RTE_MIN (buf_avail, mbuf_avail); + hpa = (void *)(uintptr_t )gpa_to_hpa (dev, + buf_iova + buf_offset, cpy_len); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); - -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } - +if (unlikely (cpy_len >= cpy_threshold && hpa)) {async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, mbuf_offset), cpy_len);
Hi Maxime,
Quoted 41 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 4:35 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data pathHi Patrick,
Quoted 15 lines · Patrick
Hi Maxime,
Quoted 9 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 21, 2020 12:40 AM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v1] vhost: support cross page buf in async data pathThe title could be improved, it is not very clear IMHO.
How about:
vhost: fix async copy failure on buffers cross page boundaryQuoted 1 line · On 7/20/20 4:52 AM, [email protected] wrote:
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
From: Patrick Fu <[email protected]>
Async copy fails when ring buffer cross two physical pages. This patch fix the failure by letting copies occur in sync mode if crossing page buffers are given.
Wouldn't it be possible to have the buffer split into two iovecs?
Technically we can do that, however, it will also introduce significant
overhead:
Quoted 4 lines
- overhead from adding additional logic in vhost async data path to
handle the case
- overhead from dma device to consume 2 iovecs In average, I don't
think dma copy can benefit too much for the buffer which are split intomultiple pages.
Quoted 1 line
CPU copy shall be a more suitable method.
Quoted 7 lines
I think we should try, that would make a cleaner implementation. I don't think having to fallback to sync mode is a good idea because it adds an overhead on the CPU, which is what we try to avoid with this async mode.
Also, I am not convinced the overhead would be that significant, at least I hope so, otherwise it would mean this new path is just performing better because it takes a lot of shortcuts, like the vector path in Virtio PMD.
I can make a trial patch and do some comparison. I will try to feedback the result by this weekend.
Thanks,
Patrick
PATCH v2 1/1 vhost: fix async copy fail on multi-page buffers
In reply to [email protected]From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) +{ +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { +if (gpa + gpa_size < + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_thva_to_gpa (struct virtio_net *dev,uint64_t vva,uint64_t len) {
lib/librte_vhost/virtio_net.c +25 -15
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy( (void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + } } out:
Signature · 1 line
2.18.4
PATCH v3 1/1 vhost: fix async copy fail on multi-page buffers
In reply to [email protected]From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible
v3:
- added fixline
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) +{ +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { +if (gpa + gpa_size < + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_thva_to_gpa (struct virtio_net *dev,uint64_t vva,uint64_t len) {
lib/librte_vhost/virtio_net.c +25 -15
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy( (void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + } } out:
Signature · 1 line
2.18.4
Re: [dpdk-dev] [PATCH v3] vhost: fix async copy fail on multi-page buffers
Hi Patrick,
Quoted 63 lines
-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Monday, July 27, 2020 2:33 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v3] vhost: fix async copy fail on multi-page buffersFrom: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the copy into
multiple async copy segments whenever possiblev3:
- added fixlinelib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)lib/librte_vhost/vhost.h index renamed from lib/librte_vhost/vhost.h +0
-0
0f7212f88..05c202a57100644lib/librte_vhost/vhost.h +21
-0@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) { +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa;
lib/librte_vhost/vhost.h gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, · line 650
Quoted 10 lines
lib/librte_vhost/vhost.h +10
-0+return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { + if (gpa + gpa_size <
Should the '<' be '<=' here?
Quoted 59 lines
lib/librte_vhost/vhost.h +18
-0+ page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_t hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len) { diff --git
a/lib/librte_vhost/virtio_net.cb/lib/librte_vhost/virtio_net.cindex95a0bc19f..124a33a10100644lib/librte_vhost/virtio_net.c +1
-0@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, structvhost_virtqueue *vq, struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;lib/librte_vhost/virtio_net.c +7
-9int tvec_idx =0 ;@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, structvhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len);
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct · line 1097
Quoted 10 lines
lib/librte_vhost/virtio_net.c +8
-1+async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len;
Will it be ok we just transform the uint64_t to uint32_t here? What if mapped_len > MAX uint32_t ?
Thanks!
Chenbo
Quoted 28 lines
lib/librte_vhost/virtio_net.c +3
-1tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy(lib/librte_vhost/virtio_net.c +6
-4(void *)((uintptr_t)(buf_addr + buf_offset)),@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + } }out:
--
2.18.4
Re: [dpdk-dev] [PATCH v3] vhost: fix async copy fail on multi-page buffers
Hi,
Quoted 21 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, July 27, 2020 9:14 PM
To: Fu, Patrick <[email protected]>; [email protected]; [email protected]
Subject: RE: [PATCH v3] vhost: fix async copy fail on multi-page buffersHi Patrick,
Quoted 12 lines
-----Original Message-----
From: Fu, Patrick <[email protected]>
Sent: Monday, July 27, 2020 2:33 PM
To: [email protected]; [email protected]; Xia, Chenbo <[email protected]>
Cc: Fu, Patrick <[email protected]>
Subject: [PATCH v3] vhost: fix async copy fail on multi-page bufferslib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)
Quoted 9 lines
lib/librte_vhost/vhost.h gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size) · line 650
Quoted 6 lines
lib/librte_vhost/vhost.h +6
-0+ }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { + if (gpa + gpa_size <Should the '<' be '<=' here?
Yes, it should be. Will update in v4 patch
Quoted 45 lines
Quoted 45 lines
static
__rte_always_inlineuint64_t
hva_to_gpa(struct virtio_net *dev, uint64_t vva, uint64_t len){(quoted diff) +0
-0
b/lib/librte_vhost/virtio_net.cindex95a0bc19f..124a33a10100644lib/librte_vhost/virtio_net.c +1
-0@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, structvhost_virtqueue *vq, struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;lib/librte_vhost/virtio_net.c +9
-9int tvec_idx =0 ;@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len);
Quoted 11 lines
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1096
Quoted 7 lines
lib/librte_vhost/virtio_net.c +6
-1- tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len;Will it be ok we just transform the
uint64_ttouint32_there? What ifmapped_len> MAXuint32_t?
"mapped_len" could not exceed "cpy_len" (according to the gpa_to_first_hpa() logic). As cpy_len is uint32, mapped_len is safe to down casting.
Thanks,
Patrick
PATCH v4 1/1 vhost: fix async copy fail on multi-page buffers
In reply to [email protected]From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible
v3:
- added fixline
v4:
- fix miss translation of the gpa which is the same length with host page size
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) +{ +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_thva_to_gpa (struct virtio_net *dev,uint64_t vva,uint64_t len) {
lib/librte_vhost/virtio_net.c +25 -15
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy( (void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } } - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + } } out:
Signature · 1 line
2.18.4
Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers
Quoted 150 lines · On 7/28/20 5:28 AM, [email protected] wrote:
From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possiblev3:
- added fixlinev4:
- fix miss translation of the gpa which is the same length with host page sizelib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)lib/librte_vhost/vhost.h +50
-0@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) +{ +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_thva_to_gpa (struct virtio_net *dev,uint64_t vva,uint64_t len) {lib/librte_vhost/virtio_net.c +19
-11@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy( (void *)((uintptr_t)(buf_addr + buf_offset)),@@ -1112,10 +1120,12 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, } }
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128
Quoted 10 lines
lib/librte_vhost/virtio_net.c +6
-4- mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + }
Is that really necessary to check if copy length is not 0?
Thanks,
Maxime
Quoted 4 lines
lib/librte_vhost/virtio_net.c +0
-0} out:
Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers
Hi Maxime,
Quoted 103 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 28, 2020 9:55 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v4] vhost: fix async copy fail on multi-page bufferslib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct
Quoted 35 lines · On 7/28/20 5:28 AM, [email protected] wrote:
From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possiblev3:
- added fixlinev4:
- fix miss translation of the gpa which is the same length with host page sizelib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)(quoted diff) +0
-0
b/lib/librte_vhost/virtio_net.cindex95a0bc19f..124a33a10100644lib/librte_vhost/virtio_net.c +0
-0@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct
vhost_virtqueue*vq,Quoted 50 lines
lib/librte_vhost/virtio_net.c +19
-11struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy(Quoted 1 line
(void *)((uintptr_t)(buf_addr +
buf_offset)),@@ -1112,10
Quoted 1 line
+1120,12 @@ async_mbuf_to_desc(struct
virtio_net*dev, struct
vhost_virtqueue*vq,Quoted 3 lines
lib/librte_vhost/virtio_net.c +0
-0} }
Quoted 13 lines
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128
Quoted 10 lines
lib/librte_vhost/virtio_net.c +6
-4- mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + }Is that really necessary to check if copy length is not 0?
The intension is to optimize for the case that ring buffers are NOT split (which should be the most common case). In that case, cpy_len will be zero and by this "if" statement we can save couple of cycles. With that said, the actual difference is minor. I'm open with either adding an "unlikely" to the "if", or removing this the "if". Would like to hear your option and submit modified patch.
Quoted 7 lines
Thanks,
MaximeQuoted 4 lines
}
out:
Re: [dpdk-dev] [PATCH v4] vhost: fix async copy fail on multi-page buffers
Hi Maxime,
Quoted 129 lines
-----Original Message-----
From: Fu, Patrick
Sent: Wednesday, July 29, 2020 9:40 AM
To: 'Maxime Coquelin' <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: RE: [PATCH v4] vhost: fix async copy fail on multi-page buffersHi Maxime,
Quoted 10 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Tuesday, July 28, 2020 9:55 PM
To: Fu, Patrick <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v4] vhost: fix async copy fail on multi-page buffersQuoted 29 lines
Quoted 28 lines · On 7/28/20 5:28 AM, [email protected] wrote:
From: Patrick Fu <[email protected]>
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for splitring")
Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possiblev3:
- added fixlinev4:
- fix miss translation of the gpa which is the same length with host page sizelib/librte_vhost/vhost.h | 50+++++++++++++++++++++++++++++++++++
Quoted 67 lines
Quoted 9 lines
lib/librte_vhost/virtio_net.c | 40 +++++++++++++++++----------- 2 files changed, 75 insertions(+), 15 deletions(-)(quoted diff) +0
-0
b/lib/librte_vhost/virtio_net.cindex95a0bc19f..124a33a10100644lib/librte_vhost/virtio_net.c +0
-0@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev,struct
vhost_virtqueue*vq,Quoted 50 lines
lib/librte_vhost/virtio_net.c +19
-11struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev,struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy(Quoted 1 line
(void *)((uintptr_t)(buf_addr +
buf_offset)),@@ -1112,10
Quoted 1 line
+1120,12 @@ async_mbuf_to_desc(struct
virtio_net*dev, struct
vhost_virtqueue*vq,Quoted 3 lines
lib/librte_vhost/virtio_net.c +0
-0} }Quoted 13 lines
lib/librte_vhost/virtio_net.c async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, · line 1128
Quoted 10 lines
lib/librte_vhost/virtio_net.c +6
-4- mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; +if (cpy_len) { + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + }Is that really necessary to check if copy length is not 0?
Quoted 7 lines
The intension is to optimize for the case that ring buffers are NOT split (which should be the most common case). In that case,
cpy_lenwill be zero and by this "if" statement we can save couple of cycles. With that said, the actual difference is minor. I'm open with either adding an "unlikely" to the "if", or removing this the "if". Would like to hear your option and submit modified patch.
I have a better way to handle the case (combine this "if" logic with the previous one). Please review my v5 patch for the code change.
Thanks,
Patrick
PATCH v5 1/1 vhost: fix async copy fail on multi-page buffers
In reply to [email protected]Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possible
v3:
- added fixline
v4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement in async_mbuf_to_desc()
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++----------- 2 files changed, 74 insertions(+), 15 deletions(-)
lib/librte_vhost/vhost.h +50 -0
@@ -616,6 +616,56 @@ gpa_to_hpa(struct virtio_net *dev, uint64_t gpa, uint64_t size)return 0 ; } +static __rte_always_inline rte_iova_t +gpa_to_first_hpa (struct virtio_net *dev,uint64_t gpa, +uint64_t gpa_size,uint64_t *hpa_size) +{ +uint32_t i; +struct guest_page *page; +struct guest_page key; + + *hpa_size = gpa_size; +if (dev->nr_guest_pages >=VHOST_BINARY_SEARCH_THRESH ) { + key.guest_phys_addr = gpa & ~(dev->guest_pages[0 ].size -1 ); + page =bsearch (&key, dev->guest_pages, dev->nr_guest_pages, +sizeof (struct guest_page ), guest_page_addrcmp); +if (page) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + }else { +for (i =0 ; i < dev->nr_guest_pages; i++) { + page = &dev->guest_pages[i]; + +if (gpa >= page->guest_phys_addr) { +if (gpa + gpa_size <= + page->guest_phys_addr + page->size) { +return gpa - page->guest_phys_addr + + page->host_phys_addr; + }else if (gpa < page->guest_phys_addr + + page->size) { + *hpa_size = page->guest_phys_addr + + page->size - gpa; +return gpa - page->guest_phys_addr + + page->host_phys_addr; + } + } + } + } + + *hpa_size =0 ; +return 0 ; +} + static __rte_always_inline uint64_thva_to_gpa (struct virtio_net *dev,uint64_t vva,uint64_t len) {
lib/librte_vhost/virtio_net.c +24 -15
@@ -980,6 +980,7 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,struct batch_copy_elem *batch_copy = vq->batch_copy_elems;struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr =NULL ;int error =0 ; +uint64_t mapped_len;uint32_t tlen =0 ;int tvec_idx =0 ;
@@ -1072,24 +1073,31 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len =RTE_MIN (buf_avail, mbuf_avail); -if (unlikely (cpy_len >= cpy_threshold)) { - hpa = (void *)(uintptr_t )gpa_to_hpa (dev, - buf_iova + buf_offset, cpy_len); +while (unlikely (cpy_len && cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t )gpa_to_first_hpa (dev, + buf_iova + buf_offset, + cpy_len, &mapped_len); -if (unlikely (!hpa)) { - error =-1 ; - goto out; - } +if (unlikely (!hpa || mapped_len < cpy_threshold)) +break ;async_fill_vec (src_iovec + tvec_idx, (void *)(uintptr_t )rte_pktmbuf_iova_offset (m, - mbuf_offset), cpy_len); + mbuf_offset), (size_t )mapped_len); -async_fill_vec (dst_iovec + tvec_idx, hpa, cpy_len); +async_fill_vec (dst_iovec + tvec_idx, + hpa, (size_t )mapped_len); - tlen += cpy_len; + tlen += (uint32_t )mapped_len; + cpy_len -= (uint32_t )mapped_len; + mbuf_avail -= (uint32_t )mapped_len; + mbuf_offset += (uint32_t )mapped_len; + buf_avail -= (uint32_t )mapped_len; + buf_offset += (uint32_t )mapped_len; tvec_idx++; - }else { + } + +if (likely (cpy_len)) {if (unlikely (vq->batch_copy_nb_elems >= vq->size)) { rte_memcpy( (void *)((uintptr_t)(buf_addr + buf_offset)),
@@ -1110,12 +1118,13 @@ async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, cpy_len; vq->batch_copy_nb_elems++; } + + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; } - mbuf_avail -= cpy_len; - mbuf_offset += cpy_len; - buf_avail -= cpy_len; - buf_offset += cpy_len; } out:
Signature · 1 line
2.18.4
Re: [dpdk-dev] [PATCH v5] vhost: fix async copy fail on multi-page buffers
Quoted 26 lines · Patrick Fu
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possiblev3:
- added fixlinev4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement inasync_mbuf_to_desc()
Quoted 3 lines
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++----------- 2 files changed, 74 insertions(+), 15 deletions(-)
Thanks Patrick, it looks better to me now:
Maxime
Maxime Coquelin
AppliedRe: [dpdk-dev] [PATCH v5] vhost: fix async copy fail on multi-page buffers
In reply to Patrick FuQuoted 26 lines · Patrick Fu
Async copy fails when single ring buffer vector is splited on multiple physical pages. This happens because current hpa address translation function doesn't handle multi-page buffers. A new gpa to hpa address conversion function, which returns the hpa on the first hitting host pages, is implemented in this patch. Async data path recursively calls this new function to construct a multi-segments async copy descriptor for ring buffers crossing physical page boundaries.
Fixes cd6760da1076 ("vhost: introduce async enqueue for split ring")Signed-off-by Patrick Fu <[email protected]>---
v2:
- change commit message and title
- v1 patch used CPU to copy multi-page buffers; v2 patch split the
copy into multiple async copy segments whenever possiblev3:
- added fixlinev4:
- fix miss translation of the gpa which is the same length with host page size
v5:
- combine redundant "if" statement inasync_mbuf_to_desc()
Quoted 3 lines
lib/librte_vhost/vhost.h | 50 +++++++++++++++++++++++++++++++++++ lib/librte_vhost/virtio_net.c | 39 ++++++++++++++++----------- 2 files changed, 74 insertions(+), 15 deletions(-)
Applied to dpdk-next-virtio/master.
Thanks,
Maxime