ixgbe: sync the first fragment unconditionally
Subsystems (3)
PATCH v2 1/1 ixgbe: sync the first fragment unconditionally
In Xen environment, if Xen-swiotlb is enabled, ixgbe driver could possibly allocate a page, DMA memory buffer, for the first fragment which is not suitable for Xen-swiotlb to do DMA operations. Xen-swiotlb have to internally allocate another page for doing DMA operations. It requires syncing between those two pages. However, since commit f3213d932173 ("ixgbe: Update driver to make use of DMA attributes in Rx path"), the unmap operation is performed with DMA_ATTR_SKIP_CPU_SYNC. As a result, the sync is not performed.
To fix this problem, always sync before possibly performing a page unmap operation.
attributes in Rx path")
---
Changes from v1:
* Imporved the patch description.
* Added Reviewed-by: and Fixes: as suggested by Alexander Duyck
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +9 -7
@@ -1825,13 +1825,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring,static void ixgbe_dma_sync_frag (struct ixgbe_ring *rx_ring,struct sk_buff *skb) { -/* if the page was released unmap it, else just sync our portion */ -if (unlikely (IXGBE_CB (skb)->page_released)) { -dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); - }else if (ring_uses_build_skb (rx_ring)) { +if (ring_uses_build_skb (rx_ring)) {unsigned long offset = (unsigned long )(skb->data) & ~PAGE_MASK ; dma_sync_single_range_for_cpu(rx_ring->dev,
@@ -1848,6 +1842,14 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } + +/* If the page was released, just unmap it. */ +if (unlikely (IXGBE_CB (skb)->page_released)) { +dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } }/**
Signature · 1 line
2.16.4
I think the description is not correct. Consider using something like below.
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
A NIC doesn't support directly write such buffer. So xen-swiotlb would use the pages, which are physically contiguous, from the swiotlb buffer for the NIC.
The unmap operation is used to copy the swiotlb buffer to the pages that are allocated by ixgbe.
Quoted 59 lines · Firo Yang
In Xen environment, if Xen-swiotlb is enabled, ixgbe driver could possibly allocate a page, DMA memory buffer, for the first fragment which is not suitable for Xen-swiotlb to do DMA operations. Xen-swiotlb have to internally allocate another page for doing DMA operations. It requires syncing between those two pages. However, since commit
f3213d932173("ixgbe: Update driver to make use of DMA attributes in Rx path"), the unmap operation is performed withDMA_ATTR_SKIP_CPU_SYNC. As a result, the sync is not performed.To fix this problem, always sync before possibly performing a page unmap operation.
Fixes f3213d932173 ("ixgbe: Update driver to make use of DMAattributes in Rx path")
Reviewed-by Alexander Duyck <[email protected]>Signed-off-by Firo Yang <[email protected]>---
Changes from v1:
* Imporved the patch description.
* Added Reviewed-by: and Fixes: as suggested by Alexander Duyckdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +9
-7@@ -1825,13 +1825,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring,static void ixgbe_dma_sync_frag (struct ixgbe_ring *rx_ring,struct sk_buff *skb) { -/* if the page was released unmap it, else just sync our portion */ -if (unlikely (IXGBE_CB (skb)->page_released)) { -dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); - }else if (ring_uses_build_skb (rx_ring)) { +if (ring_uses_build_skb (rx_ring)) {unsigned long offset = (unsigned long )(skb->data) & ~PAGE_MASK ; dma_sync_single_range_for_cpu(rx_ring->dev,@@ -1848,6 +1842,14 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } + +/* If the page was released, just unmap it. */ +if (unlikely (IXGBE_CB (skb)->page_released)) { +dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } }/**
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 3 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
But I don't think this issue relates to phsical memory contiguity because, in our case, one ixgbe_rx_buffer only associates at most one page.
If you take a look at the related code, you will find there are several reasons for mapping a DMA buffer to Xen-swiotlb area:
static dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
*/
if (dma_capable(dev, dev_addr, size) &&
!range_straddles_page_boundary(phys, size) &&
!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
swiotlb_force != SWIOTLB_FORCE)
goto done;// Firo
Quoted 67 lines
A NIC doesn't support directly write such buffer. So xen-swiotlb would use the pages, which are physically contiguous, from the swiotlb buffer for the NIC.
The unmap operation is used to copy the swiotlb buffer to the pages that are allocated by ixgbe.
Quoted 57 lines · Firo Yang
In Xen environment, if Xen-swiotlb is enabled, ixgbe driver could possibly allocate a page, DMA memory buffer, for the first fragment which is not suitable for Xen-swiotlb to do DMA operations. Xen-swiotlb have to internally allocate another page for doing DMA operations. It requires syncing between those two pages. However, since commit
f3213d932173("ixgbe: Update driver to make use of DMA attributes in Rx path"), the unmap operation is performed withDMA_ATTR_SKIP_CPU_SYNC. As a result, the sync is not performed.To fix this problem, always sync before possibly performing a page unmap operation.
Fixes f3213d932173 ("ixgbe: Update driver to make use of DMAattributes in Rx path")
Reviewed-by Alexander Duyck <[email protected]>Signed-off-by Firo Yang <[email protected]>---
Changes from v1:
* Imporved the patch description.
* Added Reviewed-by: and Fixes: as suggested by Alexander Duyckdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +9
-7@@ -1825,13 +1825,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring,static void ixgbe_dma_sync_frag (struct ixgbe_ring *rx_ring,struct sk_buff *skb) { -/* if the page was released unmap it, else just sync our portion */ -if (unlikely (IXGBE_CB (skb)->page_released)) { -dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); - }else if (ring_uses_build_skb (rx_ring)) { +if (ring_uses_build_skb (rx_ring)) {unsigned long offset = (unsigned long )(skb->data) & ~PAGE_MASK ; dma_sync_single_range_for_cpu(rx_ring->dev,@@ -1848,6 +1842,14 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } + +/* If the page was released, just unmap it. */ +if (unlikely (IXGBE_CB (skb)->page_released)) { +dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } }/**
PATCH v2 1/1 ixgbe: sync the first fragment unconditionally
Quoted 9 lines · On Wed, 7 Aug 2019 08:38:43 +0000 Firo Yang <[email protected]> wrote:
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 3 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
I think that neither of these descriptions are telling us what was truly broken. They lack what Alexander wrote on v1 thread of this patch.
ixgbe_dma_sync_frag is called only on case when the current descriptor has EOP bit set, skb was already allocated and you'll be adding a current buffer as a frag. DMA unmapping for the first frag was intentionally skipped and we will be unmapping it here, in ixgbe_dma_sync_frag. As Alex said, we're using the DMA_ATTR_SKIP_CPU_SYNC attribute which obliges us to perform a sync manually and it was missing.
So IMHO the commit description should include descriptions from both xen and ixgbe worlds, the v2 lacks info about ixgbe case.
BTW Alex, what was the initial reason for holding off with unmapping the first buffer? Asking because IIRC the i40e and ice behave a bit different here. We don't look there for EOP at all when building/constructing skb and not delaying the unmap of non-eop buffers.
Thanks,
Maciej
Quoted 82 lines
But I don't think this issue relates to phsical memory contiguity because, in our case, one
ixgbe_rx_bufferonly associates at most one page.If you take a look at the related code, you will find there are several reasons for mapping a DMA buffer to Xen-swiotlb area:
staticdma_addr_txen_swiotlb_map_page(struct device *dev, struct page *page,*/ if (dma_capable(dev, dev_addr, size) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && swiotlb_force != SWIOTLB_FORCE) goto done;// Firo
Quoted 67 lines
A NIC doesn't support directly write such buffer. So xen-swiotlb would use the pages, which are physically contiguous, from the swiotlb buffer for the NIC.
The unmap operation is used to copy the swiotlb buffer to the pages that are allocated by ixgbe.
Quoted 57 lines · Firo Yang
In Xen environment, if Xen-swiotlb is enabled, ixgbe driver could possibly allocate a page, DMA memory buffer, for the first fragment which is not suitable for Xen-swiotlb to do DMA operations. Xen-swiotlb have to internally allocate another page for doing DMA operations. It requires syncing between those two pages. However, since commit
f3213d932173("ixgbe: Update driver to make use of DMA attributes in Rx path"), the unmap operation is performed withDMA_ATTR_SKIP_CPU_SYNC. As a result, the sync is not performed.To fix this problem, always sync before possibly performing a page unmap operation.
Fixes f3213d932173 ("ixgbe: Update driver to make use of DMAattributes in Rx path")
Reviewed-by Alexander Duyck <[email protected]>Signed-off-by Firo Yang <[email protected]>---
Changes from v1:
* Imporved the patch description.
* Added Reviewed-by: and Fixes: as suggested by Alexander Duyckdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +9
-7@@ -1825,13 +1825,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring,static void ixgbe_dma_sync_frag (struct ixgbe_ring *rx_ring,struct sk_buff *skb) { -/* if the page was released unmap it, else just sync our portion */ -if (unlikely (IXGBE_CB (skb)->page_released)) { -dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); - }else if (ring_uses_build_skb (rx_ring)) { +if (ring_uses_build_skb (rx_ring)) {unsigned long offset = (unsigned long )(skb->data) & ~PAGE_MASK ; dma_sync_single_range_for_cpu(rx_ring->dev,@@ -1848,6 +1842,14 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } + +/* If the page was released, just unmap it. */ +if (unlikely (IXGBE_CB (skb)->page_released)) { +dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } }/**
PATCH v2 1/1 ixgbe: sync the first fragment unconditionally
Quoted 25 lines · Maciej Fijalkowski
Quoted 8 lines · On Wed, 7 Aug 2019 08:38:43 +0000 Firo Yang <[email protected]> wrote:
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 2 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
I think that neither of these descriptions are telling us what was truly broken. They lack what Alexander wrote on v1 thread of this patch.
ixgbe_dma_sync_fragis called only on case when the current descriptor has EOP bit set, skb was already allocated and you'll be adding a current buffer as a frag. DMA unmapping for the first frag was intentionally skipped and we will be unmapping it here, inixgbe_dma_sync_frag. As Alex said, we're using theDMA_ATTR_SKIP_CPU_SYNCattribute which obliges us to perform a sync manually and it was missing.So IMHO the commit description should include descriptions from both xen and ixgbe worlds, the v2 lacks info about ixgbe case.
Quoted 8 lines
BTW Alex, what was the initial reason for holding off with unmapping the first buffer? Asking because IIRC the i40e and ice behave a bit different here. We don't look there for EOP at all when building/constructing skb and not delaying the unmap of non-eop buffers.
Thanks,
Maciej
The reason why we have to hold off on unmapping the first buffer is because in the case of Receive Side Coalescing (RSC), also known as Large Receive Offload (LRO), the header of the packet is updated for each additional frame that is added. As such you can end up with the device writing data, header, data, header, data, header where each data write would update a new descriptor, but the header will only ever update the first descriptor in the chain. As such if we unmapped it earlier it would result in an IOMMU fault because the device would be rewriting the header after it had been unmapped.
The devices supported by the ixgbe driver are the only ones that have RSC/LRO support. As such this behavior is present for ixgbe, but not for other Intel NIC drivers including igb, igbvf, ixgbevf, i40e, etc.
- Alex
Quoted 27 lines · Maciej Fijalkowski
Quoted 9 lines · On Wed, 7 Aug 2019 08:38:43 +0000 Firo Yang <[email protected]> wrote:
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 3 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
I think that neither of these descriptions are telling us what was truly broken. They lack what Alexander wrote on v1 thread of this patch.
ixgbe_dma_sync_fragis called only on case when the current descriptor has EOP bit set, skb was already allocated and you'll be adding a current buffer as a frag. DMA unmapping for the first frag was intentionally skipped and we will be unmapping it here, inixgbe_dma_sync_frag. As Alex said, we're using theDMA_ATTR_SKIP_CPU_SYNCattribute which obliges us to perform a sync manually and it was missing.So IMHO the commit description should include descriptions from both xen and ixgbe worlds, the v2 lacks info about ixgbe case.
Quoted 7 lines
BTW Alex, what was the initial reason for holding off with unmapping the first buffer? Asking because IIRC the i40e and ice behave a bit different here. We don't look there for EOP at all when building/constructing skb and not delaying the unmap of non-eop buffers.
Thanks,
Maciej
The reason why we have to hold off on unmapping the first buffer is because in the case of Receive Side Coalescing (RSC), also known as Large Receive Offload (LRO), the header of the packet is updated for each additional frame that is added. As such you can end up with the device writing data, header, data, header, data, header where each data write would update a new descriptor, but the header will only ever update the first descriptor in the chain. As such if we unmapped it earlier it would result in an IOMMU fault because the device would be rewriting the header after it had been unmapped.
The devices supported by the ixgbe driver are the only ones that have RSC/LRO support. As such this behavior is present for ixgbe, but not for other Intel NIC drivers including igb, igbvf, ixgbevf, i40e, etc.
- Alex
Quoted 26 lines · Alexander Duyck
Quoted 24 lines · Maciej Fijalkowski
Quoted 9 lines · On Wed, 7 Aug 2019 08:38:43 +0000 Firo Yang <[email protected]> wrote:
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 3 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
I think that neither of these descriptions are telling us what was truly broken. They lack what Alexander wrote on v1 thread of this patch.
ixgbe_dma_sync_fragis called only on case when the current descriptor has EOP bit set, skb was already allocated and you'll be adding a current buffer as a frag. DMA unmapping for the first frag was intentionally skipped and we will be unmapping it here, inixgbe_dma_sync_frag. As Alex said, we're using theDMA_ATTR_SKIP_CPU_SYNCattribute which obliges us to perform a sync manually and it was missing.
Quoted 2 lines
Quoted 2 lines
So IMHO the commit description should include descriptions from both xen and ixgbe worlds, the v2 lacks info about ixgbe case.
Thank you. Will submit v3 with what Alexander worte on v1.
Thanks,
Firo
Quoted 25 lines
Quoted 8 lines
BTW Alex, what was the initial reason for holding off with unmapping the first buffer? Asking because IIRC the i40e and ice behave a bit different here. We don't look there for EOP at all when building/constructing skb and not delaying the unmap of non-eop buffers.
Thanks,
MaciejThe reason why we have to hold off on unmapping the first buffer is because in the case of Receive Side Coalescing (RSC), also known as Large Receive Offload (LRO), the header of the packet is updated for each additional frame that is added. As such you can end up with the device writing data, header, data, header, data, header where each data write would update a new descriptor, but the header will only ever update the first descriptor in the chain. As such if we unmapped it earlier it would result in an IOMMU fault because the device would be rewriting the header after it had been unmapped.
The devices supported by the ixgbe driver are the only ones that have RSC/LRO support. As such this behavior is present for ixgbe, but not for other Intel NIC drivers including igb, igbvf, ixgbevf, i40e, etc.
- Alex
Quoted 8 lines · Firo Yang
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 2 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
Yes. I was wrong. You don't need to tell the exact reason.
Quoted 3 lines
But I don't think this issue relates to phsical memory contiguity because, in our case, one
ixgbe_rx_bufferonly associates at most one page.
This is interesting.
I guess the performance of the NIC in your environment is not good due to the extra cost of bounce buffer.
Quoted 77 lines
If you take a look at the related code, you will find there are several reasons for mapping a DMA buffer to Xen-swiotlb area:
staticdma_addr_txen_swiotlb_map_page(struct device *dev, struct page *page,*/ if (dma_capable(dev, dev_addr, size) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && swiotlb_force != SWIOTLB_FORCE) goto done;// Firo
Quoted 65 lines
A NIC doesn't support directly write such buffer. So xen-swiotlb would use the pages, which are physically contiguous, from the swiotlb buffer for the NIC.
The unmap operation is used to copy the swiotlb buffer to the pages that are allocated by ixgbe.
Quoted 57 lines · Firo Yang
In Xen environment, if Xen-swiotlb is enabled, ixgbe driver could possibly allocate a page, DMA memory buffer, for the first fragment which is not suitable for Xen-swiotlb to do DMA operations. Xen-swiotlb have to internally allocate another page for doing DMA operations. It requires syncing between those two pages. However, since commit
f3213d932173("ixgbe: Update driver to make use of DMA attributes in Rx path"), the unmap operation is performed withDMA_ATTR_SKIP_CPU_SYNC. As a result, the sync is not performed.To fix this problem, always sync before possibly performing a page unmap operation.
Fixes f3213d932173 ("ixgbe: Update driver to make use of DMAattributes in Rx path")
Reviewed-by Alexander Duyck <[email protected]>Signed-off-by Firo Yang <[email protected]>---
Changes from v1:
* Imporved the patch description.
* Added Reviewed-by: and Fixes: as suggested by Alexander Duyckdrivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +9
-7@@ -1825,13 +1825,7 @@ static void ixgbe_pull_tail(struct ixgbe_ring *rx_ring,static void ixgbe_dma_sync_frag (struct ixgbe_ring *rx_ring,struct sk_buff *skb) { -/* if the page was released unmap it, else just sync our portion */ -if (unlikely (IXGBE_CB (skb)->page_released)) { -dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, -ixgbe_rx_pg_size (rx_ring), -DMA_FROM_DEVICE , -IXGBE_RX_DMA_ATTR ); - }else if (ring_uses_build_skb (rx_ring)) { +if (ring_uses_build_skb (rx_ring)) {unsigned long offset = (unsigned long )(skb->data) & ~PAGE_MASK ; dma_sync_single_range_for_cpu(rx_ring->dev,@@ -1848,6 +1842,14 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,skb_frag_size (frag), DMA_FROM_DEVICE); } + +/* If the page was released, just unmap it. */ +if (unlikely (IXGBE_CB (skb)->page_released)) { +dma_unmap_page_attrs (rx_ring->dev,IXGBE_CB (skb)->dma, +ixgbe_rx_pg_size (rx_ring), +DMA_FROM_DEVICE , +IXGBE_RX_DMA_ATTR ); + } }/**
Quoted 13 lines · Jacob Wen
Quoted 8 lines · Firo Yang
Quoted 1 line · Jacob Wen
I think the description is not correct. Consider using something like below.
Thank you for comments.
Quoted 2 lines
In Xen environment, due to memory fragmentation ixgbe may allocate a 'DMA' buffer with pages that are not physically contiguous.
Actually, I didn't look into the reason why ixgbe got a DMA buffer which was mapped to Xen-swiotlb area.
Yes. I was wrong. You don't need to tell the exact reason.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
But I don't think this issue relates to phsical memory contiguity because, in our case, one
ixgbe_rx_bufferonly associates at most one page.This is interesting.
I guess the performance of the NIC in your environment is not good due to the extra cost of bounce buffer.
If I recall correctly the Rx performance for ixgbe shouldn't be too bad even with a bounce buffer. The cost for map/unmap are expensive for a bounce buffer setup but the syncs are just copies so they are pretty cheap in comparison. The driver can take advantage of that on Rx since it leaves the pages mapped and just syncs the portion of the pages that are used.
Now if you are hitting the bounce buffer on the Tx side that is another matter as you have to allocate the buffer on demand and that is quite expensive.