From: Younghwan Go Subject: Re: Calling rte_eth_rx_burst() multiple times Date: Thu, 15 Oct 2015 19:43:43 +0900 Message-ID: <561F835F.3020300@ndsl.kaist.edu> References: <561F64BA.2000502@ndsl.kaist.edu> <561F7E8C.1080508@linaro.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable To: Zoltan Kiss , dev@dpdk.org Return-path: Received: from mail.ndsl.kaist.edu (www.ndsl.kaist.ac.kr [143.248.57.3]) by dpdk.org (Postfix) with ESMTP id E21245693 for ; Thu, 15 Oct 2015 12:43:34 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.ndsl.kaist.edu (Postfix) with ESMTP id 4BE2AC09A4 for ; Thu, 15 Oct 2015 19:43:35 +0900 (KST) Received: from mail.ndsl.kaist.edu ([127.0.0.1]) by localhost (ndsl.kaist.ac.kr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qQpvbPqWxWBb for ; Thu, 15 Oct 2015 19:43:34 +0900 (KST) In-Reply-To: <561F7E8C.1080508@linaro.org> List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Zoltan, Thanks for the email. 2015-10-15 =EC=98=A4=ED=9B=84 7:23=EC=97=90 Zoltan Kiss =EC=9D=B4(=EA=B0=80= ) =EC=93=B4 =EA=B8=80: > > > On 15/10/15 09:32, Younghwan Go wrote: >> Hi, >> >> I'm pretty new to playing with DPDK. I was trying to see if I can alwa= ys >> receive MAX_BURST packets by calling rte_eth_rx_burst() multiple times >> on same pair (code shown below). I'm using DPDK-2.1.0 on= 2 >> dual-port Intel 82599ES 10Gbps NICs with Ubuntu 14.04.3 (kernel >> 3.13.0-63-generic). >> >> Since packet processing is slower (~10 Gbps) than pure RX speed (~40 >> Gbps), I assumed rte_eth_rx_burst() would always receive some number o= f >> packets, eventually filling up MAX_BURST. But for multi-core case (4 >> CPUs, 4 ports), rte_eth_rx_burst() starts to always return 0 after som= e >> time, causing all cores to be blocked forever. Analyzing the DPDK code >> (drivers/net/ixgbe/ixgbe_rxtx.c), I'm seeing that inside >> ixgbe_rx_scan_hw_ring() function, "rxdp->wb.upper.status.error" always >> returns 0 (where is this value set by the way?). > > I think it is set by the hardware. > >> >> I didn't see this problem for single-core case, in which it returned >> MAX_BURST packets at every rte_eth_rx_burst() call. Also, if I break o= ut >> of while loop when I receive 0, I keep receiving packets in next > queue> pairs. Does anyone know why this block might happen? Or am I no= t >> allowed to call rte_eth_rx_burst() multiple times on same >> pair if I get 0? Any help will be great! Thank you! > > Although not mentioned in the documentation itself, as far as I know=20 > rte_eth_rx_burst() is not thread-safe. If you look in to receive=20 > functions, there are no locking anywhere. You should call it on=20 > separate queues from different threads, and configure e.g RSS to=20 > distribute the traffic by the hardware. I'm calling rte_eth_rx_burst() on separate queue ids for each thread.=20 I'm actually using lcore_id (=3D 0, 1, 2, 3 for 4 threads pinned to each=20 separate CPU core) as queue_id. I also made sure that this problem is=20 not caused by threads conflicting by locking before calling=20 rte_eth_rx_burst(). For RSS, I configured with ETH_RSS_IP for load balancing traffic to each=20 port and queue. But even if RSS wasn't set, shouldn't at least one core=20 be receiving packets? What I'm seeing is all threads getting stuck at=20 rte_eth_rx_burst() with return value of 0s indefinitely. >> >> ----------------------------------------------------------------------= -- >> int cnt =3D MAX_BURST; // MAX_BURST =3D 32 >> int off =3D 0; >> do { >> ret =3D rte_eth_rx_burst(port_id, queue_id, &m_table[off], cnt); >> if (ret =3D=3D 0) { >> // don't break out but continue >> } else if (ret > 0) { >> off +=3D ret; >> cnt -=3D ret; >> } >> } while (cnt > 0); >> ----------------------------------------------------------------------= -- >> >> Best, >> Younghwan Thanks, Younghwan