Calling rte_eth_rx_burst() multiple times
Hi,
I'm pretty new to playing with DPDK. I was trying to see if I can always receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).
Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed rte_eth_rx_burst() would always receive some number of packets, eventually filling up MAX_BURST. But for multi-core case (4 CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).
I didn't see this problem for single-core case, in which it returned MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to call rte_eth_rx_burst() multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!
------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;
do {
ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
if (ret == 0) {
// don't break out but continue
} else if (ret > 0) {
off += ret;
cnt -= ret;
}
} while (cnt > 0);------------------------------------------------------------------------
Best,
Younghwan
Quoted 8 lines · Younghwan Go
Hi,
I'm pretty new to playing with DPDK. I was trying to see if I can always receive
MAX_BURSTpackets by callingrte_eth_rx_burst()multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).
Quoted 8 lines
Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed
rte_eth_rx_burst()would always receive some number of packets, eventually filling upMAX_BURST. But for multi-core case (4 CPUs, 4 ports),rte_eth_rx_burst()starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that insideixgbe_rx_scan_hw_ring()function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).
I think it is set by the hardware.
Quoted 7 lines
I didn't see this problem for single-core case, in which it returned
MAX_BURSTpackets at everyrte_eth_rx_burst()call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to callrte_eth_rx_burst()multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!
Although not mentioned in the documentation itself, as far as I know rte_eth_rx_burst() is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.
Quoted 17 lines
------------------------------------------------------------------------
int cnt =MAX_BURST; //MAX_BURST= 32
int off = 0;do { ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); if (ret == 0) { // don't break out but continue } else if (ret > 0) { off += ret; cnt -= ret; } } while (cnt > 0);------------------------------------------------------------------------
Best,
Younghwan
Hi Zoltan,
Thanks for the email.
2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
Quoted 23 lines · Younghwan Go
Quoted 8 lines
Hi,
I'm pretty new to playing with DPDK. I was trying to see if I can always receive
MAX_BURSTpackets by callingrte_eth_rx_burst()multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).Quoted 8 lines
Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed
rte_eth_rx_burst()would always receive some number of packets, eventually filling upMAX_BURST. But for multi-core case (4 CPUs, 4 ports),rte_eth_rx_burst()starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that insideixgbe_rx_scan_hw_ring()function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).I think it is set by the hardware.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
I didn't see this problem for single-core case, in which it returned
MAX_BURSTpackets at everyrte_eth_rx_burst()call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to callrte_eth_rx_burst()multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!Although not mentioned in the documentation itself, as far as I know
rte_eth_rx_burst()is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.
I'm calling rte_eth_rx_burst() on separate queue ids for each thread. I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each separate CPU core) as queue_id. I also made sure that this problem is not caused by threads conflicting by locking before calling rte_eth_rx_burst().
For RSS, I configured with ETH_RSS_IP for load balancing traffic to each port and queue. But even if RSS wasn't set, shouldn't at least one core be receiving packets? What I'm seeing is all threads getting stuck at rte_eth_rx_burst() with return value of 0s indefinitely.
Quoted 15 lines
Quoted 15 lines
------------------------------------------------------------------------
int cnt =MAX_BURST; //MAX_BURST= 32
int off = 0;do { ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); if (ret == 0) { // don't break out but continue } else if (ret > 0) { off += ret; cnt -= ret; } } while (cnt > 0);------------------------------------------------------------------------
Quoted 2 lines
Quoted 2 lines
Best,
Younghwan
Thanks,
Younghwan
Quoted 47 lines · Younghwan Go
Hi Zoltan,
Thanks for the email.
2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:
Quoted 30 lines · Younghwan Go
Quoted 8 lines
Hi,
I'm pretty new to playing with DPDK. I was trying to see if I can always receive
MAX_BURSTpackets by callingrte_eth_rx_burst()multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).Quoted 8 lines
Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed
rte_eth_rx_burst()would always receive some number of packets, eventually filling upMAX_BURST. But for multi-core case (4 CPUs, 4 ports),rte_eth_rx_burst()starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that insideixgbe_rx_scan_hw_ring()function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).I think it is set by the hardware.
Quoted 7 lines
I didn't see this problem for single-core case, in which it returned
MAX_BURSTpackets at everyrte_eth_rx_burst()call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to callrte_eth_rx_burst()multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!Quoted 5 lines
Although not mentioned in the documentation itself, as far as I know
rte_eth_rx_burst()is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.I'm calling
rte_eth_rx_burst()on separate queue ids for each thread. I'm actually usinglcore_id(= 0, 1, 2, 3 for 4 threads pinned to each separate CPU core) asqueue_id. I also made sure that this problem is not caused by threads conflicting by locking before callingrte_eth_rx_burst().
Quoted 11 lines
For RSS, I configured with
ETH_RSS_IPfor load balancing traffic to each port and queue. But even if RSS wasn't set, shouldn't at least one core be receiving packets? What I'm seeing is all threads getting stuck atrte_eth_rx_burst()with return value of 0s indefinitely.Quoted 6 lines
Quoted 6 lines
------------------------------------------------------------------------
int cnt =MAX_BURST; //MAX_BURST= 32
int off = 0;do { ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
Another thing which might cause your problem is that I don't see where do you release the buffers after received into m_table. You need to call rte_pktmbuf_free() on them at some point, otherwise your pool can get depleted, and the receive function can't refill the descriptor rings.
Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
if (ret == 0) { // don't break out but continue } else if (ret > 0) { off += ret; cnt -= ret; } } while (cnt > 0);------------------------------------------------------------------------
Best,
YounghwanThanks,
Younghwan