Skip to contentSkip to conversation

Calling rte_eth_rx_burst() multiple times

DISCUSSION Younghwan Go 2 participants 4 messages Oct 15, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

I'm pretty new to playing with DPDK. I was trying to see if I can always receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).

Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed rte_eth_rx_burst() would always receive some number of packets, eventually filling up MAX_BURST. But for multi-core case (4 CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).

I didn't see this problem for single-core case, in which it returned MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to call rte_eth_rx_burst() multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!

------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;

do {
    ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
    if (ret == 0) {
        // don't break out but continue
    } else if (ret > 0) {
        off += ret;
        cnt -= ret;
    }
} while (cnt > 0);

------------------------------------------------------------------------

Best,
Younghwan

#1 lore raw
Quoted 8 lines · Younghwan Go

Hi,

I'm pretty new to playing with DPDK. I was trying to see if I can always receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).

Quoted 8 lines

Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed rte_eth_rx_burst() would always receive some number of packets, eventually filling up MAX_BURST. But for multi-core case (4 CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).

I think it is set by the hardware.

Quoted 7 lines

I didn't see this problem for single-core case, in which it returned MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to call rte_eth_rx_burst() multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!

Although not mentioned in the documentation itself, as far as I know rte_eth_rx_burst() is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.

Quoted 17 lines

------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;

do {
     ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
     if (ret == 0) {
         // don't break out but continue
     } else if (ret > 0) {
         off += ret;
         cnt -= ret;
     }
} while (cnt > 0);

------------------------------------------------------------------------

Best,
Younghwan

#2 lore raw

Hi Zoltan,

Thanks for the email.

2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:

Quoted 23 lines · Younghwan Go
Quoted 8 lines

Hi,

I'm pretty new to playing with DPDK. I was trying to see if I can always receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).

Quoted 8 lines

Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed rte_eth_rx_burst() would always receive some number of packets, eventually filling up MAX_BURST. But for multi-core case (4 CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).

I think it is set by the hardware.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

I didn't see this problem for single-core case, in which it returned MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to call rte_eth_rx_burst() multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!

Although not mentioned in the documentation itself, as far as I know rte_eth_rx_burst() is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.

I'm calling rte_eth_rx_burst() on separate queue ids for each thread. I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each separate CPU core) as queue_id. I also made sure that this problem is not caused by threads conflicting by locking before calling rte_eth_rx_burst().

For RSS, I configured with ETH_RSS_IP for load balancing traffic to each port and queue. But even if RSS wasn't set, shouldn't at least one core be receiving packets? What I'm seeing is all threads getting stuck at rte_eth_rx_burst() with return value of 0s indefinitely.

Quoted 15 lines
Quoted 15 lines

------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;

do {
     ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);
     if (ret == 0) {
         // don't break out but continue
     } else if (ret > 0) {
         off += ret;
         cnt -= ret;
     }
} while (cnt > 0);

------------------------------------------------------------------------

Quoted 2 lines
Quoted 2 lines

Best,
Younghwan

Thanks,
Younghwan

#3 lore raw
Quoted 47 lines · Younghwan Go

Hi Zoltan,

Thanks for the email.

2015-10-15 오후 7:23에 Zoltan Kiss 이(가) 쓴 글:

Quoted 30 lines · Younghwan Go
Quoted 8 lines

Hi,

I'm pretty new to playing with DPDK. I was trying to see if I can always receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times on same <port, queue> pair (code shown below). I'm using DPDK-2.1.0 on 2 dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel 3.13.0-63-generic).

Quoted 8 lines

Since packet processing is slower (~10 Gbps) than pure RX speed (~40 Gbps), I assumed rte_eth_rx_burst() would always receive some number of packets, eventually filling up MAX_BURST. But for multi-core case (4 CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after some time, causing all cores to be blocked forever. Analyzing the DPDK code (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always returns 0 (where is this value set by the way?).

I think it is set by the hardware.

Quoted 7 lines

I didn't see this problem for single-core case, in which it returned MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break out of while loop when I receive 0, I keep receiving packets in next <port, queue> pairs. Does anyone know why this block might happen? Or am I not allowed to call rte_eth_rx_burst() multiple times on same <port, queue> pair if I get 0? Any help will be great! Thank you!

Quoted 5 lines

Although not mentioned in the documentation itself, as far as I know rte_eth_rx_burst() is not thread-safe. If you look in to receive functions, there are no locking anywhere. You should call it on separate queues from different threads, and configure e.g RSS to distribute the traffic by the hardware.

I'm calling rte_eth_rx_burst() on separate queue ids for each thread. I'm actually using lcore_id (= 0, 1, 2, 3 for 4 threads pinned to each separate CPU core) as queue_id. I also made sure that this problem is not caused by threads conflicting by locking before calling rte_eth_rx_burst().

Quoted 11 lines

For RSS, I configured with ETH_RSS_IP for load balancing traffic to each port and queue. But even if RSS wasn't set, shouldn't at least one core be receiving packets? What I'm seeing is all threads getting stuck at rte_eth_rx_burst() with return value of 0s indefinitely.

Quoted 6 lines
Quoted 6 lines

------------------------------------------------------------------------
int cnt = MAX_BURST; // MAX_BURST = 32
int off = 0;

do {
     ret = rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt);

Another thing which might cause your problem is that I don't see where do you release the buffers after received into m_table. You need to call rte_pktmbuf_free() on them at some point, otherwise your pool can get depleted, and the receive function can't refill the descriptor rings.

Quoted 14 lines
Quoted 11 lines
Quoted 11 lines
     if (ret == 0) {
         // don't break out but continue
     } else if (ret > 0) {
         off += ret;
         cnt -= ret;
     }
} while (cnt > 0);

------------------------------------------------------------------------

Best,
Younghwan

Thanks,
Younghwan