Skip to contentSkip to conversation

Mbuf Structure Rework, part 2

PATCH Superseded Bruce Richardson 9 participants 63 messages Sep 3, 2014 – Sep 17, 2014 34 messages not in the archive

lore
Share & tools
lore
#0 lore raw

cover Mbuf Structure Rework, part 2

This patch set continues on from the changes in part 1, and depends upon that patch set.

This patch set reorders the fields in the mbuf structure and splits the structure across two cache lines, given lots of new space for new fields to be added. This set uses some of that space by expanding the ol_flags field and by adding in a packet_type field. A part 3 patchset is planned to introduce other new fields into the new mbuf structure.

With the splitting of the mbuf across multiple cache lines, performance degradations are seen inside the drivers, both fast-path and slow path. For the fast-path, this patchset reworks the way in which the pool pointer is used to free packets post-TX, which removes the perf regression. For the slow path, an alternative approach is taken - a new scattered packets RX function is introduced into the vector PMD. Using this function, throughput for the slow path RX-TX using testpmd is increased by up to 20% over the original baseline.

Bruce Richardson (12):
mbuf: reorder fields by time of use
mbuf: add packet_type field
mbuf: expand ol_flags field to 64-bits
mbuf: introduce a flag to indicate a control mbuf
mbuf: minor changes for readability
mbuf: use macros only to access the mbuf metadata
mbuf: add named points inside the mbuf structure
ixgbe: rework vector pmd following mbuf changes
mbuf: split mbuf across two cache lines.
mbuf: move l2_len and l3_len to second cache line
ixgbe: Fix perf regression due to moved pool ptr
ixgbe: Improve slow-path perf: vector scattered RX

Olivier Matz (1):
mbuf: replace data pointer by an offset

 app/test-pmd/csumonly.c                     |   2 +-
 app/test-pmd/flowgen.c                      |   2 +-
 app/test-pmd/icmpecho.c                     |   2 +-
 app/test-pmd/ieee1588fwd.c                  |   4 +-
 app/test-pmd/macfwd-retry.c                 |   2 +-
 app/test-pmd/macfwd.c                       |   2 +-
 app/test-pmd/macswap.c                      |   2 +-
 app/test-pmd/rxonly.c                       |   2 +-
 app/test-pmd/testpmd.c                      |   2 +-
 app/test-pmd/txonly.c                       |   7 +-
 app/test/packet_burst_generator.c           |   7 +-
 app/test/test_mbuf.c                        |   8 +-
 app/test/test_table_acl.c                   |   7 +-
 app/test/test_table_pipeline.c              |   8 +-
 examples/exception_path/main.c              |   3 +-
 examples/vhost/main.c                       |  37 +--
 examples/vhost_xen/main.c                   |  14 +-
 lib/librte_ip_frag/rte_ipv4_fragmentation.c |   6 +-
 lib/librte_ip_frag/rte_ipv6_fragmentation.c |   6 +-
 lib/librte_mbuf/rte_mbuf.c                  |  10 +-
 lib/librte_mbuf/rte_mbuf.h                  | 127 +++++----
 lib/librte_pmd_bond/rte_eth_bond_pmd.c      |   4 +-
 lib/librte_pmd_e1000/em_rxtx.c              |  12 +-
 lib/librte_pmd_e1000/igb_rxtx.c             |  13 +-
 lib/librte_pmd_i40e/i40e_rxtx.c             |  17 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c           |  56 ++--
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h           |  22 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c       | 390 +++++++++++++++++-----------
 lib/librte_pmd_pcap/rte_eth_pcap.c          |   9 +-
 lib/librte_pmd_virtio/virtio_rxtx.c         |  10 +-
 lib/librte_pmd_virtio/virtqueue.h           |   3 +-
 lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c       |   5 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c    |   2 +-
 33 files changed, 466 insertions(+), 337 deletions(-)
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-1-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#2 lore raw

PATCH 1/13 mbuf: replace data pointer by an offset

From: Olivier Matz <[email protected]>

Original patch:
The mbuf structure already contains a pointer to the beginning of the buffer (m->buf_addr). It is not needed to use 8 bytes again to store another pointer to the beginning of the data.

Using a 16 bits unsigned integer is enough as we know that a mbuf is never longer than 64KB. We gain 6 bytes in the structure thanks to this modification.

This version:
* Updated original patch to apply to latest on mainline.
* Disabled vector PMD in config as it relies heavily on the mbuf layout This will be re-enabled in a subsequent commit once vPMD has been reworked to take account of mbuf changes.

---
 app/test-pmd/csumonly.c                     |  2 +-
 app/test-pmd/flowgen.c                      |  2 +-
 app/test-pmd/icmpecho.c                     |  2 +-
 app/test-pmd/ieee1588fwd.c                  |  4 +--
 app/test-pmd/macfwd-retry.c                 |  2 +-
 app/test-pmd/macfwd.c                       |  2 +-
 app/test-pmd/macswap.c                      |  2 +-
 app/test-pmd/rxonly.c                       |  2 +-
 app/test-pmd/testpmd.c                      |  2 +-
 app/test-pmd/txonly.c                       |  7 ++---
 app/test/packet_burst_generator.c           |  7 ++---
 app/test/test_mbuf.c                        |  6 ++---
 app/test/test_table_acl.c                   |  7 ++---
 app/test/test_table_pipeline.c              |  8 ++----
 config/common_linuxapp                      |  2 +-
 examples/exception_path/main.c              |  3 ++-
 examples/vhost/main.c                       | 37 +++++++++++++------------
 examples/vhost_xen/main.c                   | 14 +++++-----
 lib/librte_ip_frag/rte_ipv4_fragmentation.c |  6 ++---
 lib/librte_ip_frag/rte_ipv6_fragmentation.c |  6 ++---
 lib/librte_mbuf/rte_mbuf.c                  |  6 ++---
 lib/librte_mbuf/rte_mbuf.h                  | 42 +++++++++++++----------------
 lib/librte_pmd_bond/rte_eth_bond_pmd.c      |  4 +--
 lib/librte_pmd_e1000/em_rxtx.c              | 12 ++++-----
 lib/librte_pmd_e1000/igb_rxtx.c             | 13 +++++----
 lib/librte_pmd_i40e/i40e_rxtx.c             | 17 ++++++------
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c           | 13 ++++-----
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h           |  3 +--
 lib/librte_pmd_pcap/rte_eth_pcap.c          |  9 ++++---
 lib/librte_pmd_virtio/virtio_rxtx.c         | 10 +++----
 lib/librte_pmd_virtio/virtqueue.h           |  3 +--
 lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c       |  5 ++--
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c    |  2 +-
 33 files changed, 130 insertions(+), 132 deletions(-)
app/test-pmd/csumonly.c +1 -1
index 28b66f5..2ce4c42 100644
@@ -263,7 +263,7 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 		pkt_ol_flags = mb->ol_flags;
 		ol_flags = (uint16_t) (pkt_ol_flags & (~PKT_TX_L4_MASK));
 
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
 		if (eth_type == ETHER_TYPE_VLAN) {
 			/* Only allow single VLAN label here */
app/test-pmd/flowgen.c +1 -1
index b091b6d..8b4ed9a 100644
@@ -175,7 +175,7 @@ pkt_burst_flow_gen(struct fwd_stream *fs)
 		pkt->next = NULL;
 
 		/* Initialize Ethernet header. */
-		eth_hdr = (struct ether_hdr *)pkt->data;
+		eth_hdr = rte_pktmbuf_mtod(pkt, struct ether_hdr *);
 		ether_addr_copy(&cfg_ether_dst, &eth_hdr->d_addr);
 		ether_addr_copy(&cfg_ether_src, &eth_hdr->s_addr);
 		eth_hdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
app/test-pmd/icmpecho.c +1 -1
index 4a277b8..7fd4b6d 100644
@@ -330,7 +330,7 @@ reply_to_icmp_echo_rqsts(struct fwd_stream *fs)
 	nb_replies = 0;
 	for (i = 0; i < nb_rx; i++) {
 		pkt = pkts_burst[i];
-		eth_h = (struct ether_hdr *) pkt->data;
+		eth_h = rte_pktmbuf_mtod(pkt, struct ether_hdr *);
 		eth_type = RTE_BE_TO_CPU_16(eth_h->ether_type);
 		l2_len = sizeof(struct ether_hdr);
 		if (verbose_level > 0) {
app/test-pmd/ieee1588fwd.c +2 -2
index ab5e06e..976aa28 100644
@@ -546,7 +546,7 @@ ieee1588_packet_fwd(struct fwd_stream *fs)
 	 * Check that the received packet is a PTP packet that was detected
 	 * by the hardware.
 	 */
-	eth_hdr = (struct ether_hdr *)mb->data;
+	eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 	eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
 	if (! (mb->ol_flags & PKT_RX_IEEE1588_PTP)) {
 		if (eth_type == ETHER_TYPE_1588) {
@@ -574,7 +574,7 @@ ieee1588_packet_fwd(struct fwd_stream *fs)
 	 * Check that the received PTP packet is a PTP V2 packet of type
 	 * PTP_SYNC_MESSAGE.
 	 */
-	ptp_hdr = (struct ptpv2_msg *) ((char *) mb->data +
+	ptp_hdr = (struct ptpv2_msg *) (rte_pktmbuf_mtod(mb, char *) +
 					sizeof(struct ether_hdr));
 	if (ptp_hdr->version != 0x02) {
 		printf("Port %u Received PTP V2 Ethernet frame with wrong PTP"
app/test-pmd/macfwd-retry.c +1 -1
index 5122983..83da26f 100644
@@ -119,7 +119,7 @@ pkt_burst_mac_retry_forward(struct fwd_stream *fs)
 	fs->rx_packets += nb_rx;
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		ether_addr_copy(&peer_eth_addrs[fs->peer_addr],
 				&eth_hdr->d_addr);
 		ether_addr_copy(&ports[fs->tx_port].eth_addr,
app/test-pmd/macfwd.c +1 -1
index 4b905cd..38bae23 100644
@@ -110,7 +110,7 @@ pkt_burst_mac_forward(struct fwd_stream *fs)
 	txp = &ports[fs->tx_port];
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		ether_addr_copy(&peer_eth_addrs[fs->peer_addr],
 				&eth_hdr->d_addr);
 		ether_addr_copy(&ports[fs->tx_port].eth_addr,
app/test-pmd/macswap.c +1 -1
index c5b3a0c..1786095 100644
@@ -110,7 +110,7 @@ pkt_burst_mac_swap(struct fwd_stream *fs)
 	txp = &ports[fs->tx_port];
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 
 		/* Swap dest and src mac addresses. */
 		ether_addr_copy(&eth_hdr->d_addr, &addr);
app/test-pmd/rxonly.c +1 -1
index 5bc74da..7ba36a1 100644
@@ -149,7 +149,7 @@ pkt_burst_receive(struct fwd_stream *fs)
 			rte_pktmbuf_free(mb);
 			continue;
 		}
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = RTE_BE_TO_CPU_16(eth_hdr->ether_type);
 		ol_flags = mb->ol_flags;
 		print_ether_addr("  src=", &eth_hdr->s_addr);
app/test-pmd/testpmd.c +1 -1
index b6596f8..32b705e 100644
@@ -404,7 +404,7 @@ testpmd_mbuf_ctor(struct rte_mempool *mp,
 			mb_ctor_arg->seg_buf_offset);
 	mb->buf_len      = mb_ctor_arg->seg_buf_size;
 	mb->ol_flags     = 0;
-	mb->data         = (char *) mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+	mb->data_off     = RTE_PKTMBUF_HEADROOM;
 	mb->nb_segs      = 1;
 	mb->l2_len       = 0;
 	mb->l3_len       = 0;
app/test-pmd/txonly.c +4 -3
index 8135264..2b3f0b9 100644
@@ -111,13 +111,13 @@ copy_buf_to_pkt_segs(void* buf, unsigned len, struct rte_mbuf *pkt,
 		seg = seg->next;
 	}
 	copy_len = seg->data_len - offset;
-	seg_buf = ((char *) seg->data + offset);
+	seg_buf = (rte_pktmbuf_mtod(seg, char *) + offset);
 	while (len > copy_len) {
 		rte_memcpy(seg_buf, buf, (size_t) copy_len);
 		len -= copy_len;
 		buf = ((char*) buf + copy_len);
 		seg = seg->next;
-		seg_buf = seg->data;
+		seg_buf = rte_pktmbuf_mtod(seg, char *);
 	}
 	rte_memcpy(seg_buf, buf, (size_t) len);
 }
@@ -126,7 +126,8 @@ static inline void
 copy_buf_to_pkt(void* buf, unsigned len, struct rte_mbuf *pkt, unsigned offset)
 {
 	if (offset + len <= pkt->data_len) {
-		rte_memcpy(((char *) pkt->data + offset), buf, (size_t) len);
+		rte_memcpy((rte_pktmbuf_mtod(pkt, char *) + offset),
+			buf, (size_t) len);
 		return;
 	}
 	copy_buf_to_pkt_segs(buf, len, pkt, offset);
app/test/packet_burst_generator.c +4 -3
index db7f023..9e747a4 100644
@@ -59,13 +59,13 @@ copy_buf_to_pkt_segs(void *buf, unsigned len, struct rte_mbuf *pkt,
 		seg = seg->next;
 	}
 	copy_len = seg->data_len - offset;
-	seg_buf = ((char *) seg->data + offset);
+	seg_buf = rte_pktmbuf_mtod(seg, char *) + offset;
 	while (len > copy_len) {
 		rte_memcpy(seg_buf, buf, (size_t) copy_len);
 		len -= copy_len;
 		buf = ((char *) buf + copy_len);
 		seg = seg->next;
-		seg_buf = seg->data;
+		seg_buf = rte_pktmbuf_mtod(seg, void *);
 	}
 	rte_memcpy(seg_buf, buf, (size_t) len);
 }
@@ -74,7 +74,8 @@ static inline void
 copy_buf_to_pkt(void *buf, unsigned len, struct rte_mbuf *pkt, unsigned offset)
 {
 	if (offset + len <= pkt->data_len) {
-		rte_memcpy(((char *) pkt->data + offset), buf, (size_t) len);
+		rte_memcpy(rte_pktmbuf_mtod(pkt, char *) + offset,
+				buf, (size_t) len);
 		return;
 	}
 	copy_buf_to_pkt_segs(buf, len, pkt, offset);
app/test/test_mbuf.c +3 -3
index b81e622..e3d896b 100644
@@ -432,7 +432,7 @@ test_pktmbuf_pool_ptr(void)
 			printf("rte_pktmbuf_alloc() failed (%u)\n", i);
 			ret = -1;
 		}
-		m[i]->data = RTE_PTR_ADD(m[i]->data, 64);
+		m[i]->data_off += 64;
 	}
 
 	/* free them */
@@ -451,8 +451,8 @@ test_pktmbuf_pool_ptr(void)
 			printf("rte_pktmbuf_alloc() failed (%u)\n", i);
 			ret = -1;
 		}
-		if (m[i]->data != RTE_PTR_ADD(m[i]->buf_addr, RTE_PKTMBUF_HEADROOM)) {
-			printf ("data pointer not set properly\n");
+		if (m[i]->data_off != RTE_PKTMBUF_HEADROOM) {
+			printf ("invalid data_off\n");
 			ret = -1;
 		}
 	}
app/test/test_table_acl.c +4 -3
index 4db680a..0f2b57e 100644
@@ -513,7 +513,7 @@ test_pipeline_single_filter(int expected_count)
 			struct rte_mbuf *mbuf;
 
 			mbuf = rte_pktmbuf_alloc(pool);
-			memset(mbuf->data, 0x00,
+			memset(rte_pktmbuf_mtod(mbuf, char *), 0x00,
 				sizeof(struct ipv4_5tuple));
 
 			five_tuple.proto = j;
@@ -522,7 +522,7 @@ test_pipeline_single_filter(int expected_count)
 			five_tuple.port_src = rte_bswap16(100 + j);
 			five_tuple.port_dst = rte_bswap16(200 + j);
 
-			memcpy(mbuf->data, &five_tuple,
+			memcpy(rte_pktmbuf_mtod(mbuf, char *), &five_tuple,
 				sizeof(struct ipv4_5tuple));
 			RTE_LOG(INFO, PIPELINE, "%s: Enqueue onto ring %d\n",
 				__func__, i);
@@ -549,7 +549,8 @@ test_pipeline_single_filter(int expected_count)
 			printf("Got %d object(s) from ring %d!\n", ret, i);
 			for (j = 0; j < ret; j++) {
 				mbuf = (struct rte_mbuf *)objs[j];
-				rte_hexdump(stdout, "mbuf", mbuf->data, 64);
+				rte_hexdump(stdout, "mbuf",
+					rte_pktmbuf_mtod(mbuf, char *), 64);
 				rte_pktmbuf_free(mbuf);
 			}
 			tx_count += ret;
app/test/test_table_pipeline.c +2 -6
index 15a038b..a0a9e04 100644
@@ -39,11 +39,6 @@
 #include "test_table.h"
 #include "test_table_pipeline.h"
 
-#define RTE_CBUF_UINT8_PTR(cbuf, offset)			\
-	(&cbuf->data[offset])
-#define RTE_CBUF_UINT32_PTR(cbuf, offset)			\
-	(&cbuf->data32[offset/sizeof(uint32_t)])
-
 #if 0
 
 static rte_pipeline_port_out_action_handler port_action_0x00
@@ -498,7 +493,8 @@ test_pipeline_single_filter(int test_type, int expected_count)
 			printf("Got %d object(s) from ring %d!\n", ret, i);
 			for (j = 0; j < ret; j++) {
 				mbuf = (struct rte_mbuf *)objs[j];
-				rte_hexdump(stdout, "Object:", mbuf->data,
+				rte_hexdump(stdout, "Object:",
+					rte_pktmbuf_mtod(mbuf, char *),
 					mbuf->data_len);
 				rte_pktmbuf_free(mbuf);
 			}
config/common_linuxapp +1 -1
index 5bee910..b140af7 100644
@@ -191,7 +191,7 @@ CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
 CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
 CONFIG_RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC=y
 CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=n
-CONFIG_RTE_IXGBE_INC_VECTOR=y
+CONFIG_RTE_IXGBE_INC_VECTOR=n
 CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
 
 #
examples/exception_path/main.c +2 -1
index 5045ef8..f286bf2 100644
@@ -302,7 +302,8 @@ main_loop(__attribute__((unused)) void *arg)
 			if (m == NULL)
 				continue;
 
-			ret = read(tap_fd, m->data, MAX_PACKET_SZ);
+			ret = read(tap_fd, rte_pktmbuf_mtod(m, void *),
+				MAX_PACKET_SZ);
 			lcore_stats[lcore_id].rx++;
 			if (unlikely(ret < 0)) {
 				FATAL_ERROR("Reading from %s interface failed",
examples/vhost/main.c +20 -17
index 4e1c103..85ee8b8 100644
@@ -1033,7 +1033,7 @@ virtio_dev_rx(struct virtio_net *dev, struct rte_mbuf **pkts, uint32_t count)
 
 		/* Copy mbuf data to buffer */
 		rte_memcpy((void *)(uintptr_t)buff_addr,
-			(const void *)buff->data,
+			rte_pktmbuf_mtod(buff, const void *),
 			rte_pktmbuf_data_len(buff));
 		PRINT_PACKET(dev, (uintptr_t)buff_addr,
 			rte_pktmbuf_data_len(buff), 0);
@@ -1438,7 +1438,7 @@ link_vmdq(struct virtio_net *dev, struct rte_mbuf *m)
 	int i, ret;
 
 	/* Learn MAC address of guest device from packet */
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	dev_ll = ll_root_used;
 
@@ -1525,7 +1525,7 @@ virtio_tx_local(struct virtio_net *dev, struct rte_mbuf *m)
 	struct ether_hdr *pkt_hdr;
 	uint64_t ret = 0;
 
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*get the used devices list*/
 	dev_ll = ll_root_used;
@@ -1593,7 +1593,7 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	unsigned len, ret, offset = 0;
 	const uint16_t lcore_id = rte_lcore_id();
 	struct virtio_net_data_ll *dev_ll = ll_root_used;
-	struct ether_hdr *pkt_hdr = (struct ether_hdr *)m->data;
+	struct ether_hdr *pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*check if destination is local VM*/
 	if ((vm2vm_mode == VM2VM_SOFTWARE) && (virtio_tx_local(dev, m) == 0))
@@ -1652,18 +1652,21 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	mbuf->nb_segs = m->nb_segs;
 
 	/* Copy ethernet header to mbuf. */
-	rte_memcpy((void*)mbuf->data, (const void*)m->data, ETH_HLEN);
+	rte_memcpy(rte_pktmbuf_mtod(mbuf, void *),
+		rte_pktmbuf_mtod(m, const void *),
+		ETH_HLEN);
 
 
 	/* Setup vlan header. Bytes need to be re-ordered for network with htons()*/
-	vlan_hdr = (struct vlan_ethhdr *) mbuf->data;
+	vlan_hdr = rte_pktmbuf_mtod(mbuf, struct vlan_ethhdr *);
 	vlan_hdr->h_vlan_encapsulated_proto = vlan_hdr->h_vlan_proto;
 	vlan_hdr->h_vlan_proto = htons(ETH_P_8021Q);
 	vlan_hdr->h_vlan_TCI = htons(vlan_tag);
 
 	/* Copy the remaining packet contents to the mbuf. */
-	rte_memcpy((void*) ((uint8_t*)mbuf->data + VLAN_ETH_HLEN),
-		(const void*) ((uint8_t*)m->data + ETH_HLEN), (m->data_len - ETH_HLEN));
+	rte_memcpy((void *)(rte_pktmbuf_mtod(mbuf, uint8_t *) + VLAN_ETH_HLEN),
+		(const void *)(rte_pktmbuf_mtod(m, uint8_t *) + ETH_HLEN),
+		(m->data_len - ETH_HLEN));
 
 	/* Copy the remaining segments for the whole packet. */
 	prev = mbuf;
@@ -1778,7 +1781,7 @@ virtio_dev_tx(struct virtio_net* dev, struct rte_mempool *mbuf_pool)
 		/* Setup dummy mbuf. This is copied to a real mbuf if transmitted out the physical port. */
 		m.data_len = desc->len;
 		m.pkt_len = desc->len;
-		m.data = (void*)(uintptr_t)buff_addr;
+		m.data_off = 0;
 
 		PRINT_PACKET(dev, (uintptr_t)buff_addr, desc->len, 0);
 
@@ -2333,7 +2336,7 @@ attach_rxmbuf_zcp(struct virtio_net *dev)
 	}
 
 	mbuf->buf_addr = (void *)(uintptr_t)(buff_addr - RTE_PKTMBUF_HEADROOM);
-	mbuf->data = (void *)(uintptr_t)(buff_addr);
+	mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 	mbuf->buf_physaddr = phys_addr - RTE_PKTMBUF_HEADROOM;
 	mbuf->data_len = desc->len;
 	MBUF_HEADROOM_UINT32(mbuf) = (uint32_t)desc_idx;
@@ -2370,7 +2373,7 @@ static inline void pktmbuf_detach_zcp(struct rte_mbuf *m)
 
 	buf_ofs = (RTE_PKTMBUF_HEADROOM <= m->buf_len) ?
 			RTE_PKTMBUF_HEADROOM : m->buf_len;
-	m->data = (char *) m->buf_addr + buf_ofs;
+	m->data_off = buf_ofs;
 
 	m->data_len = 0;
 }
@@ -2604,7 +2607,7 @@ virtio_tx_route_zcp(struct virtio_net *dev, struct rte_mbuf *m,
 	unsigned len, ret, offset = 0;
 	struct vpool *vpool;
 	struct virtio_net_data_ll *dev_ll = ll_root_used;
-	struct ether_hdr *pkt_hdr = (struct ether_hdr *)m->data;
+	struct ether_hdr *pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 	uint16_t vlan_tag = (uint16_t)vlan_tags[(uint16_t)dev->device_fh];
 
 	/*Add packet to the port tx queue*/
@@ -2681,11 +2684,11 @@ virtio_tx_route_zcp(struct virtio_net *dev, struct rte_mbuf *m,
 	mbuf->pkt_len = mbuf->data_len;
 	if (unlikely(need_copy)) {
 		/* Copy the packet contents to the mbuf. */
-		rte_memcpy((void *)((uint8_t *)mbuf->data),
-			(const void *) ((uint8_t *)m->data),
+		rte_memcpy(rte_pktmbuf_mtod(mbuf, void *),
+			rte_pktmbuf_mtod(m, void *),
 			m->data_len);
 	} else {
-		mbuf->data = m->data;
+		mbuf->data_off = m->data_off;
 		mbuf->buf_physaddr = m->buf_physaddr;
 		mbuf->buf_addr = m->buf_addr;
 	}
@@ -2819,8 +2822,8 @@ virtio_dev_tx_zcp(struct virtio_net *dev)
 		m.data_len = desc->len;
 		m.nb_segs = 1;
 		m.next = NULL;
-		m.data = (void *)(uintptr_t)buff_addr;
-		m.buf_addr = m.data;
+		m.data_off = 0;
+		m.buf_addr = (void *)(uintptr_t)buff_addr;
 		m.buf_physaddr = phys_addr;
 
 		/*
examples/vhost_xen/main.c +8 -6
index 8162cd8..56ffec8 100644
@@ -808,7 +808,7 @@ virtio_tx_local(struct virtio_net *dev, struct rte_mbuf *m)
 	struct ether_hdr *pkt_hdr;
 	uint64_t ret = 0;
 
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*get the used devices list*/
 	dev_ll = ll_root_used;
@@ -883,18 +883,20 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	mbuf->pkt_len = mbuf->data_len;
 
 	/* Copy ethernet header to mbuf. */
-	rte_memcpy((void*)mbuf->data, (const void*)m->data, ETH_HLEN);
+	rte_memcpy(rte_pktmbuf_mtod(mbuf, void*),
+			rte_pktmbuf_mtod(m, const void*), ETH_HLEN);
 
 
 	/* Setup vlan header. Bytes need to be re-ordered for network with htons()*/
-	vlan_hdr = (struct vlan_ethhdr *) mbuf->data;
+	vlan_hdr = rte_pktmbuf_mtod(mbuf, struct vlan_ethhdr *);
 	vlan_hdr->h_vlan_encapsulated_proto = vlan_hdr->h_vlan_proto;
 	vlan_hdr->h_vlan_proto = htons(ETH_P_8021Q);
 	vlan_hdr->h_vlan_TCI = htons(vlan_tag);
 
 	/* Copy the remaining packet contents to the mbuf. */
-	rte_memcpy((void*) ((uint8_t*)mbuf->data + VLAN_ETH_HLEN),
-		(const void*) ((uint8_t*)m->data + ETH_HLEN), (m->data_len - ETH_HLEN));
+	rte_memcpy((void *)(rte_pktmbuf_mtod(mbuf, uint8_t *) + VLAN_ETH_HLEN),
+		(const void *)(rte_pktmbuf_mtod(m, uint8_t *) + ETH_HLEN),
+		(m->data_len - ETH_HLEN));
 	tx_q->m_table[len] = mbuf;
 	len++;
 	if (enable_stats) {
@@ -981,7 +983,7 @@ virtio_dev_tx(struct virtio_net* dev, struct rte_mempool *mbuf_pool)
 
 		/* Setup dummy mbuf. This is copied to a real mbuf if transmitted out the physical port. */
 		m.data_len = desc->len;
-		m.data = (void*)(uintptr_t)buff_addr;
+		m.data_off = 0;
 		m.nb_segs = 1;
 
 		virtio_tx_route(dev, &m, mbuf_pool, 0);
lib/librte_ip_frag/rte_ipv4_fragmentation.c +3 -3
index 6b9f07d..a4ed923 100644
@@ -109,7 +109,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 	/* Fragment size should be a multiply of 8. */
 	IP_FRAG_ASSERT((frag_size & IPV4_HDR_FO_MASK) == 0);
 
-	in_hdr = (struct ipv4_hdr *) pkt_in->data;
+	in_hdr = rte_pktmbuf_mtod(pkt_in, struct ipv4_hdr *);
 	flag_offset = rte_cpu_to_be_16(in_hdr->fragment_offset);
 
 	/* If Don't Fragment flag is set */
@@ -165,7 +165,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 			if (len > (in_seg->data_len - in_seg_data_pos)) {
 				len = in_seg->data_len - in_seg_data_pos;
 			}
-			out_seg->data = (char*) in_seg->data + (uint16_t)in_seg_data_pos;
+			out_seg->data_off = in_seg->data_off + in_seg_data_pos;
 			out_seg->data_len = (uint16_t)len;
 			out_pkt->pkt_len = (uint16_t)(len +
 			    out_pkt->pkt_len);
@@ -188,7 +188,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 
 		/* Build the IP header */
 
-		out_hdr = (struct ipv4_hdr*) out_pkt->data;
+		out_hdr = rte_pktmbuf_mtod(out_pkt, struct ipv4_hdr *);
 
 		__fill_ipv4hdr_frag(out_hdr, in_hdr,
 		    (uint16_t)out_pkt->pkt_len,
lib/librte_ip_frag/rte_ipv6_fragmentation.c +3 -3
index e007662..4ffcc7c 100644
@@ -125,7 +125,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 	    (uint16_t)(pkt_in->pkt_len - sizeof (struct ipv6_hdr))))
 		return (-EINVAL);
 
-	in_hdr = (struct ipv6_hdr *) pkt_in->data;
+	in_hdr = rte_pktmbuf_mtod(pkt_in, struct ipv6_hdr *);
 
 	in_seg = pkt_in;
 	in_seg_data_pos = sizeof(struct ipv6_hdr);
@@ -171,7 +171,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 			if (len > (in_seg->data_len - in_seg_data_pos)) {
 				len = in_seg->data_len - in_seg_data_pos;
 			}
-			out_seg->data = (char *) in_seg->data + (uint16_t) in_seg_data_pos;
+			out_seg->data_off = in_seg->data_off + in_seg_data_pos;
 			out_seg->data_len = (uint16_t)len;
 			out_pkt->pkt_len = (uint16_t)(len +
 			    out_pkt->pkt_len);
@@ -196,7 +196,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 
 		/* Build the IP header */
 
-		out_hdr = (struct ipv6_hdr *) out_pkt->data;
+		out_hdr = rte_pktmbuf_mtod(out_pkt, struct ipv6_hdr *);
 
 		__fill_ipv6hdr_frag(out_hdr, in_hdr,
 		    (uint16_t) out_pkt->pkt_len - sizeof(struct ipv6_hdr),
lib/librte_mbuf/rte_mbuf.c +3 -3
index c1b2176..26e36eb 100644
@@ -117,7 +117,7 @@ rte_pktmbuf_init(struct rte_mempool *mp,
 	m->buf_len = (uint16_t)buf_len;
 
 	/* keep some headroom between start of buffer and data */
-	m->data = (char*) m->buf_addr + RTE_MIN(RTE_PKTMBUF_HEADROOM, m->buf_len);
+	m->data_off = RTE_MIN(RTE_PKTMBUF_HEADROOM, (uint16_t)m->buf_len);
 
 	/* init some constant fields */
 	m->pool = mp;
@@ -183,12 +183,12 @@ rte_pktmbuf_dump(FILE *f, const struct rte_mbuf *m, unsigned dump_len)
 		__rte_mbuf_sanity_check(m, 0);
 
 		fprintf(f, "  segment at 0x%p, data=0x%p, data_len=%u\n",
-		       m, m->data, (unsigned)m->data_len);
+			m, rte_pktmbuf_mtod(m, void *), (unsigned)m->data_len);
 		len = dump_len;
 		if (len > m->data_len)
 			len = m->data_len;
 		if (len != 0)
-			rte_hexdump(f, NULL, m->data, len);
+			rte_hexdump(f, NULL, rte_pktmbuf_mtod(m, void *), len);
 		dump_len -= len;
 		m = m->next;
 		nb_segs --;
lib/librte_mbuf/rte_mbuf.h +19 -23

135 lines not on this page · raw message

lib/librte_pmd_bond/rte_eth_bond_pmd.c +2 -2
index 5979ce5..506a448 100644
@@ -198,14 +198,14 @@ xmit_slave_hash(const struct rte_mbuf *buf, uint8_t slave_count, uint8_t policy)
 
 	switch (policy) {
 	case BALANCE_XMIT_POLICY_LAYER2:
-		eth_hdr = (struct ether_hdr *)buf->data;
+		eth_hdr = rte_pktmbuf_mtod(buf, struct ether_hdr *);
 
 		hash = ether_hash(eth_hdr);
 		hash ^= hash >> 8;
 		return hash % slave_count;
 
 	case BALANCE_XMIT_POLICY_LAYER23:
-		eth_hdr = (struct ether_hdr *)buf->data;
+		eth_hdr = rte_pktmbuf_mtod(buf, struct ether_hdr *);
 
 		if (buf->ol_flags & PKT_RX_VLAN_PKT)
 			eth_offset = sizeof(struct ether_hdr) + sizeof(struct vlan_hdr);
lib/librte_pmd_e1000/em_rxtx.c +6 -6
index 4f46bdf..67736d6 100644
@@ -90,8 +90,7 @@ rte_rxmbuf_alloc(struct rte_mempool *mp)
 }
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb)             \
-	(uint64_t) ((mb)->buf_physaddr +       \
-	(uint64_t) ((char *)((mb)->data) - (char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -793,8 +792,8 @@ eth_em_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.length) -
 				rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -963,7 +962,7 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1035,7 +1034,8 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
lib/librte_pmd_e1000/igb_rxtx.c +6 -7
index 4e9f42e..e364dda 100644
@@ -95,9 +95,7 @@ rte_rxmbuf_alloc(struct rte_mempool *mp)
 }
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr +		   \
-			(uint64_t) ((char *)((mb)->data) -     \
-				(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -776,8 +774,8 @@ eth_igb_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.wb.upper.length) -
 				      rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -952,7 +950,7 @@ eth_igb_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.wb.upper.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1033,7 +1031,8 @@ eth_igb_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		first_seg->ol_flags = pkt_flags;
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
lib/librte_pmd_i40e/i40e_rxtx.c +8 -9
index e41e8d0..25a5f6f 100644
@@ -78,9 +78,7 @@
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	((uint64_t)((mb)->buf_physaddr + \
-	(uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr)))
+	((uint64_t)((mb)->buf_physaddr + (mb)->data_off))
 
 static const struct rte_memzone *
 i40e_ring_dma_zone_reserve(struct rte_eth_dev *dev,
@@ -685,7 +683,7 @@ i40e_rx_alloc_bufs(struct i40e_rx_queue *rxq)
 		mb = rxep[i].mbuf;
 		rte_mbuf_refcnt_set(mb, 1);
 		mb->next = NULL;
-		mb->data = (char *)mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mb->data_off = RTE_PKTMBUF_HEADROOM;
 		mb->nb_segs = 1;
 		mb->port = rxq->port_id;
 		dma_addr = rte_cpu_to_le_64(\
@@ -842,8 +840,8 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		rx_packet_len = ((qword1 & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
 				I40E_RXD_QW1_LENGTH_PBUF_SHIFT) - rxq->crc_len;
 
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_prefetch0(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_prefetch0(RTE_PTR_ADD(rxm->buf_addr, RTE_PKTMBUF_HEADROOM));
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = rx_packet_len;
@@ -946,7 +944,7 @@ i40e_recv_scattered_pkts(void *rx_queue,
 		rx_packet_len = (qword1 & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
 					I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
 		rxm->data_len = rx_packet_len;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/**
 		 * If this is the first buffer of the received packet, set the
@@ -1015,7 +1013,8 @@ i40e_recv_scattered_pkts(void *rx_queue,
 				rte_le_to_cpu_32(rxd.wb.qword0.hi_dword.rss);
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_prefetch0(first_seg->data);
+		rte_prefetch0(RTE_PTR_ADD(first_seg->buf_addr,
+			first_seg->data_off));
 		rx_pkts[nb_rx++] = first_seg;
 		first_seg = NULL;
 	}
@@ -2131,7 +2130,7 @@ i40e_alloc_rx_queue_mbufs(struct i40e_rx_queue *rxq)
 
 		rte_mbuf_refcnt_set(mbuf, 1);
 		mbuf->next = NULL;
-		mbuf->data = (char *)mbuf->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 		mbuf->nb_segs = 1;
 		mbuf->port = rxq->port_id;
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +7 -6
index b897b1e..964ae06 100644
@@ -998,7 +998,7 @@ ixgbe_rx_alloc_bufs(struct igb_rx_queue *rxq)
 		mb = rxep[i].mbuf;
 		rte_mbuf_refcnt_set(mb, 1);
 		mb->next = NULL;
-		mb->data = (char *)mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mb->data_off = RTE_PKTMBUF_HEADROOM;
 		mb->nb_segs = 1;
 		mb->port = rxq->port_id;
 
@@ -1249,8 +1249,8 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.wb.upper.length) -
 				      rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -1432,7 +1432,7 @@ ixgbe_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.wb.upper.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1523,7 +1523,8 @@ ixgbe_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		}
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
@@ -3213,7 +3214,7 @@ ixgbe_alloc_rx_queue_mbufs(struct igb_rx_queue *rxq)
 
 		rte_mbuf_refcnt_set(mbuf, 1);
 		mbuf->next = NULL;
-		mbuf->data = (char *)mbuf->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 		mbuf->nb_segs = 1;
 		mbuf->port = rxq->port_id;
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -2
index 2f1f40c..38a3a03 100644
@@ -47,8 +47,7 @@
 #endif
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
lib/librte_pmd_pcap/rte_eth_pcap.c +6 -3
index 121de65..e60982e 100644
@@ -151,7 +151,8 @@ eth_pcap_rx(void *queue,
 
 		if (header.len <= buf_size) {
 			/* pcap packet will fit in the mbuf, go ahead and copy */
-			rte_memcpy(mbuf->data, packet, header.len);
+			rte_memcpy(rte_pktmbuf_mtod(mbuf, void *), packet,
+					header.len);
 			mbuf->data_len = (uint16_t)header.len;
 			mbuf->pkt_len = mbuf->data_len;
 			bufs[num_rx] = mbuf;
@@ -202,7 +203,8 @@ eth_pcap_tx_dumper(void *queue,
 		calculate_timestamp(&header.ts);
 		header.len = mbuf->data_len;
 		header.caplen = header.len;
-		pcap_dump((u_char*) dumper_q->dumper, &header, mbuf->data);
+		pcap_dump((u_char*) dumper_q->dumper, &header,
+				rte_pktmbuf_mtod(mbuf, void*));
 		rte_pktmbuf_free(mbuf);
 		num_tx++;
 	}
@@ -237,7 +239,8 @@ eth_pcap_tx(void *queue,
 
 	for (i = 0; i < nb_pkts; i++) {
 		mbuf = bufs[i];
-		ret = pcap_sendpacket(tx_queue->pcap, (u_char*) mbuf->data,
+		ret = pcap_sendpacket(tx_queue->pcap,
+				rte_pktmbuf_mtod(mbuf, u_char *),
 				mbuf->data_len);
 		if (unlikely(ret != 0))
 			break;
lib/librte_pmd_virtio/virtio_rxtx.c +4 -6
index 132ee45..29c9cea 100644
@@ -118,7 +118,7 @@ virtqueue_dequeue_burst_rx(struct virtqueue *vq, struct rte_mbuf **rx_pkts,
 		}
 
 		rte_prefetch0(cookie);
-		rte_packet_prefetch(cookie->data);
+		rte_packet_prefetch(rte_pktmbuf_mtod(cookie, void *));
 		rx_pkts[i]  = cookie;
 		vq->vq_used_cons_idx++;
 		vq_ring_free_chain(vq, desc_idx);
@@ -480,7 +480,7 @@ virtio_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		}
 
 		rxm->port = rxvq->port_id;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
@@ -584,7 +584,7 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 		if (seg_num == 0)
 			seg_num = 1;
 
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 		rxm->nb_segs = seg_num;
 		rxm->next = NULL;
 		rxm->pkt_len = (uint32_t)(len[0] - hdr_size);
@@ -622,9 +622,7 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 			while (extra_idx < rcv_cnt) {
 				rxm = rcv_pkts[extra_idx];
 
-				rxm->data =
-					(char *)rxm->buf_addr +
-					RTE_PKTMBUF_HEADROOM - hdr_size;
+				rxm->data_off = RTE_PKTMBUF_HEADROOM - hdr_size;
 				rxm->next = NULL;
 				rxm->pkt_len = (uint32_t)(len[extra_idx]);
 				rxm->data_len = (uint16_t)(len[extra_idx]);
lib/librte_pmd_virtio/virtqueue.h +1 -2
index d777feb..fdee054 100644
@@ -59,8 +59,7 @@
 #define VIRTQUEUE_MAX_NAME_SZ 32
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define VTNET_SQ_RQ_QUEUE_IDX 0
 #define VTNET_SQ_TQ_QUEUE_IDX 1
lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c +2 -3
index e74b6fd..263f9ce 100644
@@ -79,8 +79,7 @@
 
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -565,7 +564,7 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rxm->data_len = (uint16_t)rcd->len;
 			rxm->port = rxq->port_id;
 			rxm->vlan_tci = 0;
-			rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+			rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 			rx_pkts[nb_rx++] = rxm;
 
lib/librte_pmd_xenvirt/rte_eth_xenvirt.c +1 -1
index 22215ed..891cb58 100644
@@ -110,7 +110,7 @@ eth_xenvirt_rx(void *q, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		rxm = rx_pkts[i];
 		PMD_RX_LOG(DEBUG, "packet len:%d\n", len[i]);
 		rxm->next = NULL;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 		rxm->data_len = (uint16_t)(len[i] - sizeof(struct virtio_net_hdr));
 		rxm->nb_segs = 1;
 		rxm->port = pi->port_id;
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-2-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#4 lore raw

Re: [PATCH 01/13] mbuf: replace data pointer by an offset

Hi Bruce,

Quoted 28 lines · Bruce Richardson

From: Olivier Matz <[email protected]>

Original patch:
The mbuf structure already contains a pointer to the beginning of the buffer (m->buf_addr). It is not needed to use 8 bytes again to store another pointer to the beginning of the data.

Using a 16 bits unsigned integer is enough as we know that a mbuf is never longer than 64KB. We gain 6 bytes in the structure thanks to this modification.

This version:
* Updated original patch to apply to latest on mainline.
* Disabled vector PMD in config as it relies heavily on the mbuf layout This will be re-enabled in a subsequent commit once vPMD has been reworked to take account of mbuf changes.

[...]

lib/librte_mbuf/rte_mbuf.h +0 -0
index 32e8474..669e7f5 100644

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 128

Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +7 -0
@@ -119,6 +119,13 @@ struct rte_mbuf {
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
 	uint16_t buf_len;         /**< Length of segment buffer. */
+
+	/* valid for any segment */
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	uint16_t data_off;
+	uint16_t data_len;        /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
+

Here, the compiler will add some padding between "buf_len" and "next" (6 bytes in 64bits, 2 in 32bits). Shouldn't we add "reserved" fields or at least a comment ? Another idea would be to reorganize the fields to avoid padding, if possible.

Except this remarq, Acked-by: Olivier Matz <[email protected]>

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Re: [PATCH 01/13] mbuf: replace data pointer by an offset

Quoted 13 lines
Quoted 13 lines
lib/librte_mbuf/rte_mbuf.h +4 -0
index 32e8474..669e7f5 100644
@@ -119,6 +119,13 @@ struct rte_mbuf {
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
 	uint16_t buf_len;         /**< Length of segment buffer. */
+
+	/* valid for any segment */
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	uint16_t data_off;
Quoted 8 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 128

Quoted 3 lines
lib/librte_mbuf/rte_mbuf.h +3 -0
+	uint16_t data_len;        /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
+

Here, the compiler will add some padding between "buf_len" and "next" (6 bytes in 64bits, 2 in 32bits). Shouldn't we add "reserved" fields or at least a comment ? Another idea would be to reorganize the fields to avoid padding, if possible.

I've just seen that patch 02/13 reorders the mbuf fields. So maybe this remarq is useless.

Olivier

#7 lore raw

PATCH 2/13 mbuf: reorder fields by time of use

In reply to a message not in the archive

* Reorder the fields in the mbuf so that we have fields that are used$ together side-by-side in the structure. This means that we have a$ contiguous block of 8-bytes in the mbuf which are used to reset an mbuf$ of descriptor rearm, and a block of 16-bytes of data (excluding flags) which are set on RX from the received packet descriptor.
* Use dummy fields as appropraite to ensure alignment or to reserve gaps for later field additions.
* Place most items which are not used by fast-path RX separately at the end of the structure so they can later be moved to a separate cache line. [The l2/l3 length fields are not moved at this stage as doing so will cause overflow to the next cache line].

---
 lib/librte_mbuf/rte_mbuf.h | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)
lib/librte_mbuf/rte_mbuf.h +14 -11
index 669e7f5..f136d37 100644
@@ -115,16 +115,12 @@ extern "C" {
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
-	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
-	uint16_t buf_len;         /**< Length of segment buffer. */
 
-	/* valid for any segment */
-	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	/* next 8 bytes are initialised on RX descriptor rearm */
+	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
-	uint16_t data_len;        /**< Amount of data in segment buffer. */
-	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
 
 #ifdef RTE_MBUF_REFCNT
 	/**
@@ -142,14 +138,17 @@ struct rte_mbuf {
 #else
 	uint16_t refcnt_reserved;     /**< Do not use this field */
 #endif
-	uint16_t reserved;            /**< Unused field. Required for padding */
-	uint16_t ol_flags;            /**< Offload features. */
-
-	/* these fields are valid for first segment only */
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	/* offload features, valid for first segment only */
+	uint16_t ol_flags;      /**< Offload features. */
+	uint16_t reserved0;     /**< Unused field. Required for padding */
+	uint32_t reserved1;     /**< Unused field. Required for padding */
+
+	/* remaining bytes are set on RX when pulling packet from descriptor */
+	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t data_len;      /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
 	uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
 	uint16_t vlan_tci;      /**< VLAN Tag Control Identifier (CPU order). */
@@ -162,6 +161,10 @@ struct rte_mbuf {
 		uint32_t sched;     /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
+	/* fields only used in slow path or on TX */
+	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+
 	union {
 		uint8_t metadata[0];
 		uint16_t metadata16[0];
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-3-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

Acked-by
#9 lore raw

Re: [PATCH 02/13] mbuf: reorder fields by time of use

Hi Bruce,

Quoted 5 lines · Bruce Richardson

* Reorder the fields in the mbuf so that we have fields that are used$ together side-by-side in the structure. This means that we have a$ contiguous block of 8-bytes in the mbuf which are used to reset an mbuf$ of descriptor rearm, and a block of 16-bytes of data (excluding flags) which are set on RX from the received packet descriptor.

Some unwanted $ characters here.

Quoted 2 lines

* Use dummy fields as appropraite to ensure alignment or to reserve gaps for later field additions.

s/appropraite/appropriate

Quoted 6 lines

* Place most items which are not used by fast-path RX separately at the end of the structure so they can later be moved to a separate cache line. [The l2/l3 length fields are not moved at this stage as doing so will cause overflow to the next cache line].

#10 lore raw

PATCH 3/13 mbuf: add packet_type field

In reply to a message not in the archive

Replace a reserved slot with the new packet type field used to identify the type of the packet, i.e. what protocols are used.

---
 lib/librte_mbuf/rte_mbuf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_mbuf/rte_mbuf.h +1 -1
index f136d37..8d0c6fb 100644
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-4-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#12 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Hi Bruce,

Quoted 14 lines · Bruce Richardson

Replace a reserved slot with the new packet type field used to identify the type of the packet, i.e. what protocols are used.

---
 lib/librte_mbuf/rte_mbuf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_mbuf/rte_mbuf.h +0 -0
index f136d37..8d0c6fb 100644

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 8 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

I would use it :)
It's useful to store the IP protocol number (UDP, TCP etc) and version of IP (4, 6) and then relay packet to specific handler.

08.09.2014 16:17, Olivier MATZ пишет:

Quoted 26 lines

Hi Bruce,

Quoted 14 lines · Bruce Richardson

Replace a reserved slot with the new packet type field used to identify the type of the packet, i.e. what protocols are used.

---
 lib/librte_mbuf/rte_mbuf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_mbuf/rte_mbuf.h +0 -0
index f136d37..8d0c6fb 100644

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 8 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

Signature · 5 lines
Sincerely,

Yerden Zhumabekov
State Technical Service
Astana, KZ

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Hi Yerden,

Quoted 15 lines · Yerden Zhumabekov

08.09.2014 16:17, Olivier MATZ пишет:

Quoted 5 lines
Quoted 5 lines
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
Quoted 8 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 7 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

Quoted 3 lines

I would use it :)
It's useful to store the IP protocol number (UDP, TCP etc) and version of IP (4, 6) and then relay packet to specific handler.

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also need to define if this field must be filled by the application before transmitting a mbuf to a driver.

Regards,
Olivier

A message that is not in the archive (sent off-list or not yet received): CAD16F236028A64DBBC0158B1636EA4510F3E4F8@SHSMSX104.ccr.corp.intel.com

A message that is not in the archive (sent off-list or not yet received): CAD16F236028A64DBBC0158B1636EA4510F3E4F8-0J0gbvR4kTg/[email protected]

#19 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Hi Olivier,

Quoted 35 lines

-----Original Message-----
From: Zhu, Heqing
Sent: Tuesday, September 09, 2014 9:48 AM
To: Wu, Jingjing; Liu, Jijiang; Zhang, Helin
Cc: Zhu, Heqing
Subject: FW: [dpdk-dev] [PATCH 03/13] mbuf: add packet_type field

One of you need respond to this thread? Please make the answer generic and easy to understand/accept if possible.

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Olivier MATZ
Sent: Monday, September 08, 2014 7:17 PM
To: Yerden Zhumabekov; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 03/13] mbuf: add packet_type field

Hi Yerden,

Quoted 7 lines · Yerden Zhumabekov

08.09.2014 16:17, Olivier MATZ пишет:

Quoted 6 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 148

Quoted 6 lines
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor

*/

Quoted 8 lines
Quoted 7 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 6 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

Quoted 6 lines
Quoted 3 lines

I would use it :)
It's useful to store the IP protocol number (UDP, TCP etc) and version of IP (4, 6) and then relay packet to specific handler.

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

In some types of NIC, for an example, Intel Fortville, which supports various packet types, and packet type value is filled in a field of receive Descriptor by HW, Normally, application analyses easily what incoming packet format is if it know what packet type is. It is a common approach for analyzing packet format.

Quoted 4 lines

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata?

Yes, driver is in charge of filling it if this type of NIC has a packet type filed in receive Descriptor.

Quoted 3 lines

In the other direction (TX), we would also need
to define if this field must be filled by the application before transmitting a mbuf to a driver.

Normally, TX side don't care the packet type, instead, feature offload flag will be used in TX side.
In RX side, NIC HW analyses incoming packet and know what packet type is, and fill packet type value in Receive Descriptor.
Of course, I also don't object to add a packet type in TX side if mbuf TX space is enough, but from the present point of view, it is not necessary.

Quoted 2 lines

Regards,
Olivier

Regards,
Jijiang Liu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#21 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Quoted 11 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Olivier MATZ
Sent: Monday, September 8, 2014 7:17 PM
To: Yerden Zhumabekov; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 03/13] mbuf: add packet_type field

Hi Yerden,

On 09/08/2014 12:33 PM, Yerden Zhumabekov wrote:

Quoted 19 lines
Quoted 7 lines

08.09.2014 16:17, Olivier MATZ пишет:

Quoted 6 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 148

Quoted 6 lines
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor

*/

Quoted 11 lines
Quoted 7 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 6 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

I would use it :)
It's useful to store the IP protocol number (UDP, TCP etc) and version of IP (4, 6) and then relay packet to specific handler.

It is a common field which i40e PMD will use it to store the 'packet type ID'. i40e hardware can recognize more than a hundred of packet types of received packets, this is quite useful for upper layer stack or application. So this field is quite useful and will be filled by PMD.
In ixgbe/igb, it has less than 10 packet types which are marked in offload flags. From now on, it would be better to have new field here to put the hardware offloaded packet type in and it could be used for future NICs.

Quoted 10 lines

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also need to define if this field must be filled by the application before transmitting a mbuf to a driver.

Yes, PMD will fill it. I40e PMD will be the first one, ixgbe/igb can be kept as it is, or modified to be consistent. It is used for RX side only, and for TX side, it can be investigated to see if it can be used also. I think some new features in development can think of that.
Anyway, it is a quite useful field for i40e and future generation of NICs.

Quoted 3 lines

Regards,
Olivier

A message that is not in the archive (sent off-list or not yet received): F35DEAC7BCE34641BA9FAC6BCA4A12E70A784E49-0J0gbvR4kTg/[email protected]

#23 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Hello,

Quoted 18 lines · Helin

It is a common field which i40e PMD will use it to store the 'packet type ID'. i40e hardware can recognize more than a hundred of packet types of received packets, this is quite useful for upper layer stack or application. So this field is quite useful and will be filled by PMD.
In ixgbe/igb, it has less than 10 packet types which are marked in offload flags. From now on, it would be better to have new field here to put the hardware offloaded packet type in and it could be used for future NICs.

Quoted 10 lines

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also need to define if this field must be filled by the application before transmitting a mbuf to a driver.

Quoted 5 lines

Yes, PMD will fill it. I40e PMD will be the first one, ixgbe/igb can be kept as it is, or modified to be consistent. It is used for RX side only, and for TX side, it can be investigated to see if it can be used also. I think some new features in development can think of that.
Anyway, it is a quite useful field for i40e and future generation of NICs.

To me, having the support in a hardware for that feature is not a sufficient reason for adding this field. There are many hardware features that will never be integrated in dpdk.

This first version of the patch:

- just adds a field that is not used by any code, so it is useless. At least testpmd or an application example should show how to use it.

- does not describe what enhancement is provided by adding the field (performance? in this case, numbers + use case would help to convince people).

- does not describe what can be the content of the field. Is it a protocol number?

- does not explain if all drivers must fill this field. If yes, the patch has to update all drivers. If not, something must be done to mark the packet field as unknown by default.

Regards,
Olivier

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

Quoted 37 lines

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Tuesday, September 9, 2014 4:03 PM
To: Zhang, Helin; Yerden Zhumabekov; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 03/13] mbuf: add packet_type field

Hello,

Quoted 7 lines · Helin

It is a common field which i40e PMD will use it to store the 'packet type ID'. i40e hardware can recognize more than a hundred of packet types of received packets, this is quite useful for upper layer stack or application. So this field is quite useful and will be filled by PMD. In ixgbe/igb, it has less than 10 packet types which are marked in offload flags. From now on, it would be better to have new field here to put the hardware offloaded packet type in and it could be used for future

NICs.

Quoted 4 lines
Quoted 4 lines

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the

generic mbuf structure.

Quoted 7 lines
Quoted 7 lines

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also need to define if this field must be filled by the application before transmitting a mbuf to a driver.

Quoted 5 lines

Yes, PMD will fill it. I40e PMD will be the first one, ixgbe/igb can be kept as it is, or modified to be consistent. It is used for RX side only, and for TX side, it can be investigated to see if it can be used also. I think some new features in development can think of that. Anyway, it is a quite useful field for i40e and future generation of NICs.

Quoted 3 lines

To me, having the support in a hardware for that feature is not a sufficient reason for adding this field. There are many hardware features that will never be integrated in dpdk.

At least this field is quite important for i40e.
e.g. packet type=43 means that hardware recognize it as a VXLAN packet. To avoid checking what type of packet by software, hardware can offload that, and fill the packet type ID in that field. It cannot be put in ol_flags anymore, as it has more than 100 packet type can be recognized by hardware. Without it, vxlan feature cannot be implemented at all.

Quoted 6 lines

This first version of the patch:

- just adds a field that is not used by any code, so it is useless. At least testpmd or an application example should show how to use it.

It will be used at least in vxlan feature which is in development. Without it, vxlan cannot be completed. So this is a very important field i40e and future NICs.

Quoted 4 lines

- does not describe what enhancement is provided by adding the field (performance? in this case, numbers + use case would help to convince people).

I40e hardware can recognize received packets as different packet types, and there are about 256 packet types can be recognized by i40e hardware. It is not a enhancement, it is the key for at least i40e features.

Quoted 4 lines

- does not describe what can be the content of the field. Is it a protocol number?

The packet type is a offload feature of hardware, the value of it can mean one type of packet recognized by the i40e hardware. E.g.

| Packet type | Description            |
| 0         | Reserved              |
| 1         | MAC, PAY2            |
| 2         | MAC, TimeSync, PAY2    |

...
| 43 | MAC, IPV4, GRENAT, PAY3 |

Quoted 4 lines

- does not explain if all drivers must fill this field. If yes, the patch has to update all drivers. If not, something must be done to mark the packet field as unknown by default.

I40e needs it.
igb/ixgbe can be changed to support it, but not mandatory, as ol_flags can represent it.
Actually ixgbe and igb has packet type also, but the number of those types is less than 10, so it can be put in ol_flags. For i40e and future NICs, the number of that could be 256 or more, ol_flags does not have that many bits for it. The best idea is to fill the packet type ID directly into a field.

Quoted 2 lines

Regards,
Olivier

Regards,
Helin

#26 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

In reply to a message not in the archive
Quoted 62 lines

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Tuesday, September 09, 2014 9:03 AM
To: Zhang, Helin; Yerden Zhumabekov; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 03/13] mbuf: add packet_type field

Hello,

Quoted 1 line · Helin

It is a common field which i40e PMD will use it to store the 'packet type ID'.

i40e

Quoted 1 line

hardware can recognize more than a hundred of packet types of received

packets,

Quoted 1 line

this is quite useful for upper layer stack or application. So this field is quite

useful

Quoted 2 lines

and will be filled by PMD.
In ixgbe/igb, it has less than 10 packet types which are marked in offload flags.

From

Quoted 1 line

now on, it would be better to have new field here to put the hardware

offloaded

Quoted 8 lines

packet type in and it could be used for future NICs.

Quoted 6 lines

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or

is

Quoted 2 lines
Quoted 2 lines

it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also

need

Quoted 1 line
Quoted 1 line

to define if this field must be filled by the application before transmitting a

mbuf

Quoted 2 lines
Quoted 1 line

to a driver.

Yes, PMD will fill it. I40e PMD will be the first one, ixgbe/igb can be kept as it

is, or

Quoted 2 lines

modified to be consistent. It is used for RX side only, and for TX side, it can be investigated to see if it can be used also. I think some new features in

development

Quoted 2 lines

can think of that.
Anyway, it is a quite useful field for i40e and future generation of NICs.

To me, having the support in a hardware for that feature is not a sufficient reason for adding this field. There are many hardware features that will never be integrated in dpdk.

This first version of the patch:

- just adds a field that is not used by any code, so it is useless. At least testpmd or an application example should show how to use it.

- does not describe what enhancement is provided by adding the field (performance? in this case, numbers + use case would help to convince people).

- does not describe what can be the content of the field. Is it a protocol number?

Quoted 6 lines

- does not explain if all drivers must fill this field. If yes, the patch has to update all drivers. If not, something must be done to mark the packet field as unknown by default.

Regards,
Olivier

Hi,

Points taken. Really, this patch doesn't belong in this set as I had planned things and better belongs in patch set 3 (coming soon, I hope) which should propose the new field additions. I simply put it here to avoid having to start renumbering and renaming reserved fields in the structure, but that is possibly the lesser of the two evils.

However, with regards to adding new fields in, I would like to have some agreement that I can add fields in without actually pushing in the patch to use them - so long as sufficient rational is provided for using the field and there is a soon pending change to actually use it. This patch did not meet the criteria for explanation, but if updated to do so, I would like to have this patch accepted on the basis of that explanation so as to enable those working on the drivers to make us of it.

Regards,
/Bruce

#27 lore raw

Re: [PATCH 03/13] mbuf: add packet_type field

In reply to a message not in the archive
Quoted 29 lines · Olivier MATZ

Hi Yerden,

Quoted 15 lines · Yerden Zhumabekov

08.09.2014 16:17, Olivier MATZ пишет:

Quoted 5 lines
Quoted 5 lines
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -146,7 +146,7 @@ struct rte_mbuf {
 	uint32_t reserved1;     /**< Unused field. Required for padding */
 
Quoted 8 lines

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 152

Quoted 7 lines
lib/librte_mbuf/rte_mbuf.h +1 -1
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */

This patch adds a new fields that nobody uses. So why should we add it ?

Quoted 3 lines

I would use it :)
It's useful to store the IP protocol number (UDP, TCP etc) and version of IP (4, 6) and then relay packet to specific handler.

I'm not saying this field is useless. But even if it's useful for some applications like yours, it does not mean that it should go in the generic mbuf structure.

Quoted 6 lines

Also, for a new field, we should define who is in charge of filling it. Is is the driver? Does it mean that all drivers have to be modified to fill it? Or is it just a placeholder for applications? In this case, shouldn't we use application-specific metadata? In the other direction (TX), we would also need to define if this field must be filled by the application before transmitting a mbuf to a driver.

Funny, but these new fields (and extended mbuf) were prominent during the dpdk summit.

I think it’s going to be quite useful.

#28 lore raw

PATCH 4/13 mbuf: expand ol_flags field to 64-bits

In reply to a message not in the archive

The offload flags field (ol_flags) was 16-bits and had no further room for expansion. This patch increases the field size to 64-bits, using up the remaining reserved space in the single-cache-line mbuf.

NOTE: none of the values for existing flags have been changed, i.e. no new numbers have been explicitly reserved between existing flag definitions.

---
 app/test-pmd/config.c             |  8 ++---
 app/test-pmd/csumonly.c           |  8 ++---
 app/test-pmd/rxonly.c             |  2 +-
 app/test-pmd/testpmd.h            |  4 +--
 app/test-pmd/txonly.c             |  2 +-
 lib/librte_mbuf/rte_mbuf.c        |  2 +-
 lib/librte_mbuf/rte_mbuf.h        |  4 +--
 lib/librte_pmd_e1000/em_rxtx.c    | 34 ++++++++++-----------
 lib/librte_pmd_e1000/igb_rxtx.c   | 62 ++++++++++++++++++---------------------
 lib/librte_pmd_i40e/i40e_rxtx.c   | 31 ++++++++++----------
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 61 ++++++++++++++++++--------------------
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h |  2 +-
 12 files changed, 104 insertions(+), 116 deletions(-)
app/test-pmd/config.c +4 -4
index 606e34a..adfa9a8 100644
@@ -1714,14 +1714,14 @@ set_qmap(portid_t port_id, uint8_t is_rx, uint16_t queue_id, uint8_t map_value)
 }
 
 void
-tx_cksum_set(portid_t port_id, uint8_t cksum_mask)
+tx_cksum_set(portid_t port_id, uint64_t ol_flags)
 {
-	uint16_t tx_ol_flags;
+	uint64_t tx_ol_flags;
 	if (port_id_is_invalid(port_id))
 		return;
 	/* Clear last 4 bits and then set L3/4 checksum mask again */
-	tx_ol_flags = (uint16_t) (ports[port_id].tx_ol_flags & 0xFFF0);
-	ports[port_id].tx_ol_flags = (uint16_t) ((cksum_mask & 0xf) | tx_ol_flags);
+	tx_ol_flags = ports[port_id].tx_ol_flags & (~0x0Full);
+	ports[port_id].tx_ol_flags = ((ol_flags & 0xf) | tx_ol_flags);
 }
 
 void
app/test-pmd/csumonly.c +4 -4
index 2ce4c42..fcc4876 100644
@@ -217,9 +217,9 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 	uint16_t nb_rx;
 	uint16_t nb_tx;
 	uint16_t i;
-	uint16_t ol_flags;
-	uint16_t pkt_ol_flags;
-	uint16_t tx_ol_flags;
+	uint64_t ol_flags;
+	uint64_t pkt_ol_flags;
+	uint64_t tx_ol_flags;
 	uint16_t l4_proto;
 	uint16_t eth_type;
 	uint8_t  l2_len;
@@ -261,7 +261,7 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 		mb = pkts_burst[i];
 		l2_len  = sizeof(struct ether_hdr);
 		pkt_ol_flags = mb->ol_flags;
-		ol_flags = (uint16_t) (pkt_ol_flags & (~PKT_TX_L4_MASK));
+		ol_flags = (pkt_ol_flags & (~PKT_TX_L4_MASK));
 
 		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
app/test-pmd/rxonly.c +1 -1
index 7ba36a1..98c788b 100644
@@ -109,7 +109,7 @@ pkt_burst_receive(struct fwd_stream *fs)
 	struct rte_mbuf  *mb;
 	struct ether_hdr *eth_hdr;
 	uint16_t eth_type;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint16_t nb_rx;
 	uint16_t i;
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
app/test-pmd/testpmd.h +2 -2
index 09923a8..142091d 100644
@@ -141,7 +141,7 @@ struct rte_port {
 	struct fwd_stream       *rx_stream; /**< Port RX stream, if unique */
 	struct fwd_stream       *tx_stream; /**< Port TX stream, if unique */
 	unsigned int            socket_id;  /**< For NUMA support */
-	uint16_t                tx_ol_flags;/**< Offload Flags of TX packets. */
+	uint64_t                tx_ol_flags;/**< Offload Flags of TX packets. */
 	uint16_t                tx_vlan_id; /**< Tag Id. in TX VLAN packets. */
 	void                    *fwd_ctx;   /**< Forwarding mode context */
 	uint64_t                rx_bad_ip_csum; /**< rx pkts with bad ip checksum  */
@@ -494,7 +494,7 @@ void tx_vlan_pvid_set(portid_t port_id, uint16_t vlan_id, int on);
 
 void set_qmap(portid_t port_id, uint8_t is_rx, uint16_t queue_id, uint8_t map_value);
 
-void tx_cksum_set(portid_t port_id, uint8_t cksum_mask);
+void tx_cksum_set(portid_t port_id, uint64_t ol_flags);
 
 void set_verbose_level(uint16_t vb_level);
 void set_tx_pkt_segments(unsigned *seg_lengths, unsigned nb_segs);
app/test-pmd/txonly.c +1 -1
index 2b3f0b9..3d08005 100644
@@ -203,7 +203,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
 	uint16_t nb_tx;
 	uint16_t nb_pkt;
 	uint16_t vlan_tci;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint8_t  i;
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
 	uint64_t start_tsc;
lib/librte_mbuf/rte_mbuf.c +1 -1
index 26e36eb..9dfcac3 100644
@@ -174,7 +174,7 @@ rte_pktmbuf_dump(FILE *f, const struct rte_mbuf *m, unsigned dump_len)
 
 	fprintf(f, "dump mbuf at 0x%p, phys=%"PRIx64", buf_len=%u\n",
 	       m, (uint64_t)m->buf_physaddr, (unsigned)m->buf_len);
-	fprintf(f, "  pkt_len=%"PRIu32", ol_flags=%"PRIx16", nb_segs=%u, "
+	fprintf(f, "  pkt_len=%"PRIu32", ol_flags=%"PRIx64", nb_segs=%u, "
 	       "in_port=%u\n", m->pkt_len, m->ol_flags,
 	       (unsigned)m->nb_segs, (unsigned)m->port);
 	nb_segs = m->nb_segs;
lib/librte_mbuf/rte_mbuf.h +1 -3
index 8d0c6fb..6af5b2f 100644
@@ -141,9 +141,7 @@ struct rte_mbuf {
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	uint16_t ol_flags;      /**< Offload features. */
-	uint16_t reserved0;     /**< Unused field. Required for padding */
-	uint32_t reserved1;     /**< Unused field. Required for padding */
+	uint64_t ol_flags;      /**< Offload features. */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
 	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
lib/librte_pmd_e1000/em_rxtx.c +16 -18
index 67736d6..eb76660 100644
@@ -168,7 +168,7 @@ union em_vlan_macip {
  * Structure to check if new context need be built
  */
 struct em_ctx_info {
-	uint16_t flags;               /**< ol_flags related to context build. */
+	uint64_t flags;               /**< ol_flags related to context build. */
 	uint32_t cmp_mask;            /**< compare mask */
 	union em_vlan_macip hdrlen;  /**< L2 and L3 header lenghts */
 };
@@ -238,7 +238,7 @@ struct em_tx_queue {
 static inline void
 em_set_xmit_ctx(struct em_tx_queue* txq,
 		volatile struct e1000_context_desc *ctx_txd,
-		uint16_t flags,
+		uint64_t flags,
 		union em_vlan_macip hdrlen)
 {
 	uint32_t cmp_mask, cmd_len;
@@ -304,7 +304,7 @@ em_set_xmit_ctx(struct em_tx_queue* txq,
  * or create a new context descriptor.
  */
 static inline uint32_t
-what_ctx_update(struct em_tx_queue *txq, uint16_t flags,
+what_ctx_update(struct em_tx_queue *txq, uint64_t flags,
 		union em_vlan_macip hdrlen)
 {
 	/* If match with the current context */
@@ -377,7 +377,7 @@ em_xmit_cleanup(struct em_tx_queue *txq)
 }
 
 static inline uint32_t
-tx_desc_cksum_flags_to_upper(uint16_t ol_flags)
+tx_desc_cksum_flags_to_upper(uint64_t ol_flags)
 {
 	static const uint32_t l4_olinfo[2] = {0, E1000_TXD_POPTS_TXSM << 8};
 	static const uint32_t l3_olinfo[2] = {0, E1000_TXD_POPTS_IXSM << 8};
@@ -403,12 +403,12 @@ eth_em_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	uint32_t popts_spec;
 	uint32_t cmd_type_len;
 	uint16_t slen;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint16_t tx_id;
 	uint16_t tx_last;
 	uint16_t nb_tx;
 	uint16_t nb_used;
-	uint16_t tx_ol_req;
+	uint64_t tx_ol_req;
 	uint32_t ctx;
 	uint32_t new_ctx;
 	union em_vlan_macip hdrlen;
@@ -438,8 +438,7 @@ eth_em_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		ol_flags = tx_pkt->ol_flags;
 
 		/* If hardware offload required */
-		tx_ol_req = (uint16_t)(ol_flags & (PKT_TX_IP_CKSUM |
-							PKT_TX_L4_MASK));
+		tx_ol_req = (ol_flags & (PKT_TX_IP_CKSUM | PKT_TX_L4_MASK));
 		if (tx_ol_req) {
 			hdrlen.f.vlan_tci = tx_pkt->vlan_tci;
 			hdrlen.f.l2_len = tx_pkt->l2_len;
@@ -642,22 +641,21 @@ end_of_tx:
  *
  **********************************************************************/
 
-static inline uint16_t
+static inline uint64_t
 rx_desc_status_to_pkt_flags(uint32_t rx_status)
 {
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	/* Check if VLAN present */
-	pkt_flags = (uint16_t)((rx_status & E1000_RXD_STAT_VP) ?
-						PKT_RX_VLAN_PKT : 0);
+	pkt_flags = ((rx_status & E1000_RXD_STAT_VP) ?  PKT_RX_VLAN_PKT : 0);
 
 	return pkt_flags;
 }
 
-static inline uint16_t
+static inline uint64_t
 rx_desc_error_to_pkt_flags(uint32_t rx_error)
 {
-	uint16_t pkt_flags = 0;
+	uint64_t pkt_flags = 0;
 
 	if (rx_error & E1000_RXD_ERR_IPE)
 		pkt_flags |= PKT_RX_IP_CKSUM_BAD;
@@ -801,8 +799,8 @@ eth_em_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxm->port = rxq->port_id;
 
 		rxm->ol_flags = rx_desc_status_to_pkt_flags(status);
-		rxm->ol_flags = (uint16_t)(rxm->ol_flags |
-				rx_desc_error_to_pkt_flags(rxd.errors));
+		rxm->ol_flags = rxm->ol_flags |
+				rx_desc_error_to_pkt_flags(rxd.errors);
 
 		/* Only valid if PKT_RX_VLAN_PKT set in pkt_flags */
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
@@ -1027,8 +1025,8 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		first_seg->port = rxq->port_id;
 
 		first_seg->ol_flags = rx_desc_status_to_pkt_flags(status);
-		first_seg->ol_flags = (uint16_t)(first_seg->ol_flags |
-					rx_desc_error_to_pkt_flags(rxd.errors));
+		first_seg->ol_flags = first_seg->ol_flags |
+					rx_desc_error_to_pkt_flags(rxd.errors);
 
 		/* Only valid if PKT_RX_VLAN_PKT set in pkt_flags */
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
lib/librte_pmd_e1000/igb_rxtx.c +28 -34

169 lines not on this page · raw message

lib/librte_pmd_i40e/i40e_rxtx.c +16 -15
index 25a5f6f..cd05654 100644
@@ -91,27 +91,27 @@ static uint16_t i40e_xmit_pkts_simple(void *tx_queue,
 				      uint16_t nb_pkts);
 
 /* Translate the rx descriptor status to pkt flags */
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_status_to_pkt_flags(uint64_t qword)
 {
-	uint16_t flags;
+	uint64_t flags;
 
 	/* Check if VLAN packet */
-	flags = (uint16_t)(qword & (1 << I40E_RX_DESC_STATUS_L2TAG1P_SHIFT) ?
-							PKT_RX_VLAN_PKT : 0);
+	flags = qword & (1 << I40E_RX_DESC_STATUS_L2TAG1P_SHIFT) ?
+							PKT_RX_VLAN_PKT : 0;
 
 	/* Check if RSS_HASH */
-	flags |= (uint16_t)((((qword >> I40E_RX_DESC_STATUS_FLTSTAT_SHIFT) &
+	flags |= (((qword >> I40E_RX_DESC_STATUS_FLTSTAT_SHIFT) &
 					I40E_RX_DESC_FLTSTAT_RSS_HASH) ==
-			I40E_RX_DESC_FLTSTAT_RSS_HASH) ? PKT_RX_RSS_HASH : 0);
+			I40E_RX_DESC_FLTSTAT_RSS_HASH) ? PKT_RX_RSS_HASH : 0;
 
 	return flags;
 }
 
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_error_to_pkt_flags(uint64_t qword)
 {
-	uint16_t flags = 0;
+	uint64_t flags = 0;
 	uint64_t error_bits = (qword >> I40E_RXD_QW1_ERROR_SHIFT);
 
 #define I40E_RX_ERR_BITS 0x3f
@@ -143,12 +143,12 @@ i40e_rxd_error_to_pkt_flags(uint64_t qword)
 }
 
 /* Translate pkt types to pkt flags */
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_ptype_to_pkt_flags(uint64_t qword)
 {
 	uint8_t ptype = (uint8_t)((qword & I40E_RXD_QW1_PTYPE_MASK) >>
 					I40E_RXD_QW1_PTYPE_SHIFT);
-	static const uint16_t ip_ptype_map[I40E_MAX_PKT_TYPE] = {
+	static const uint64_t ip_ptype_map[I40E_MAX_PKT_TYPE] = {
 		0, /* PTYPE 0 */
 		0, /* PTYPE 1 */
 		0, /* PTYPE 2 */
@@ -567,7 +567,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq)
 	uint32_t rx_status;
 	int32_t s[I40E_LOOK_AHEAD], nb_dd;
 	int32_t i, j, nb_rx = 0;
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	rxdp = &rxq->rx_ring[rxq->rx_tail];
 	rxep = &rxq->sw_ring[rxq->rx_tail];
@@ -789,7 +789,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_packet_len;
 	uint16_t rx_id, nb_hold;
 	uint64_t dma_addr;
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	nb_rx = 0;
 	nb_hold = 0;
@@ -896,10 +896,11 @@ i40e_recv_scattered_pkts(void *rx_queue,
 	struct rte_mbuf *last_seg = rxq->pkt_last_seg;
 	struct rte_mbuf *nmb, *rxm;
 	uint16_t rx_id = rxq->rx_tail;
-	uint16_t nb_rx = 0, nb_hold = 0, rx_packet_len, pkt_flags;
+	uint16_t nb_rx = 0, nb_hold = 0, rx_packet_len;
 	uint32_t rx_status;
 	uint64_t qword1;
 	uint64_t dma_addr;
+	uint64_t pkt_flags;
 
 	while (nb_rx < nb_pkts) {
 		rxdp = &rx_ring[rx_id];
@@ -1046,7 +1047,7 @@ i40e_recv_scattered_pkts(void *rx_queue,
 
 /* Check if the context descriptor is needed for TX offloading */
 static inline uint16_t
-i40e_calc_context_desc(uint16_t flags)
+i40e_calc_context_desc(uint64_t flags)
 {
 	uint16_t mask = 0;
 
@@ -1075,7 +1076,7 @@ i40e_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	uint32_t td_offset;
 	uint32_t tx_flags;
 	uint32_t td_tag;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint8_t l2_len;
 	uint8_t l3_len;
 	uint16_t nb_used;
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +29 -32

167 lines not on this page · raw message

lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -1
index 38a3a03..70dbcb0 100644
@@ -178,7 +178,7 @@ union ixgbe_vlan_macip {
  */
 
 struct ixgbe_advctx_info {
-	uint16_t flags;           /**< ol_flags for context build. */
+	uint64_t flags;           /**< ol_flags for context build. */
 	uint32_t cmp_mask;        /**< compare mask for vlan_macip_lens */
 	union ixgbe_vlan_macip vlan_macip_lens; /**< vlan, mac ip length. */
 };
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-5-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#30 lore raw

Re: [PATCH 04/13] mbuf: expand ol_flags field to 64-bits

Hi Bruce,

Quoted 9 lines · Bruce Richardson

The offload flags field (ol_flags) was 16-bits and had no further room for expansion. This patch increases the field size to 64-bits, using up the remaining reserved space in the single-cache-line mbuf.

NOTE: none of the values for existing flags have been changed, i.e. no new numbers have been explicitly reserved between existing flag definitions.

The initial series I've proposed [1][2] had on more enhancement: the first patch [1] allowed to remove the definition of flag names in testpmd. Indeed, this is not really good because they must be kept synchronized with the flags in rte_mbuf. What do you think about this patch? Should it be integrated in your series? Or later? Or never? ;)

The second patch [2] changes the value of the flags. This is not needed now, but if we do it in the future, we should not forget to change app/test-pmd/cmdline.c accordingly. Maybe this could go in your patch directly as it does not hurt?

Olivier

[1] http://dpdk.org/ml/archives/dev/2014-May/002545.html
[2] http://dpdk.org/ml/archives/dev/2014-May/002546.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#32 lore raw

Re: [PATCH 04/13] mbuf: expand ol_flags field to 64-bits

Quoted 17 lines

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Monday, September 08, 2014 11:26 AM
To: Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 04/13] mbuf: expand ol_flags field to 64-bits

Hi Bruce,

Quoted 8 lines · Bruce Richardson

The offload flags field (ol_flags) was 16-bits and had no further room for expansion. This patch increases the field size to 64-bits, using up the remaining reserved space in the single-cache-line mbuf.

NOTE: none of the values for existing flags have been changed, i.e. no new numbers have been explicitly reserved between existing flag definitions.

Quoted 7 lines
Quoted 1 line

The initial series I've proposed [1][2] had on more enhancement: the first patch [1] allowed to remove the definition of flag names in testpmd. Indeed, this is not really good because they must be kept synchronized with the flags in rte_mbuf. What do you think about this patch? Should it be integrated in your series? Or later? Or never? ;)

No, it is a good change - I've just keep it out of my series for simplicity as I'm largely trying to keep the scope as small as possible. I would love to see that go in as a separate patch maybe once the mbuf rework is finished.

Quoted 5 lines

The second patch [2] changes the value of the flags. This is not needed now, but if we do it in the future, we should not forget to change app/test-pmd/cmdline.c accordingly. Maybe this could go in your patch directly as it does not hurt?

As above for now. Right now I'm just trying to get the structure worked out, and deal with any performance regressions that are found (such as what Pablo found last Friday :-( ).

/Bruce

Quoted 6 lines

Olivier

[1] http://dpdk.org/ml/archives/dev/2014-May/002545.html
[2] http://dpdk.org/ml/archives/dev/2014-May/002546.html

#33 lore raw

PATCH 5/13 mbuf: introduce a flag to indicate a control mbuf

In reply to a message not in the archive

Since the flags field is now 64-bits, we can allow one bit to be used to indicate a control i.e. non-packet mbuf. Dedicate the high bit (bit 63) for this purpose and add in a utility macro to test if a given mbuf has the bit set or not.

---
 lib/librte_mbuf/rte_mbuf.c |  2 ++
 lib/librte_mbuf/rte_mbuf.h | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+)
lib/librte_mbuf/rte_mbuf.c +2 -0
index 9dfcac3..52e7574 100644
@@ -69,7 +69,9 @@ rte_ctrlmbuf_init(struct rte_mempool *mp,
 		void *_m,
 		__attribute__((unused)) unsigned i)
 {
+	struct rte_mbuf *m = _m;
 	rte_pktmbuf_init(mp, opaque_arg, _m, i);
+	m->ol_flags |= CTRL_MBUF_FLAG;
 }
 
 /*
lib/librte_mbuf/rte_mbuf.h +19 -0
index 6af5b2f..7b0b4f2 100644
@@ -91,6 +91,7 @@ extern "C" {
 #define PKT_TX_IPV4_CSUM     0x1000 /**< Alias of PKT_TX_IP_CKSUM. */
 #define PKT_TX_IPV4          PKT_RX_IPV4_HDR /**< IPv4 with no IP checksum offload. */
 #define PKT_TX_IPV6          PKT_RX_IPV6_HDR /**< IPv6 packet */
+
 /*
  * Bit 14~13 used for L4 packet type with checksum enabled.
  *     00: Reserved
@@ -106,6 +107,9 @@ extern "C" {
 /* Bit 15 */
 #define PKT_TX_IEEE1588_TMST 0x8000 /**< TX IEEE1588 packet to timestamp. */
 
+/* Use final bit of flags to indicate a control mbuf */
+#define CTRL_MBUF_FLAG       (1ULL << 63)
+
 /**
  * Bit Mask to indicate what bits required for building TX context
  */
@@ -466,6 +470,21 @@ void rte_ctrlmbuf_init(struct rte_mempool *mp, void *opaque_arg,
  */
 #define rte_ctrlmbuf_len(m) rte_pktmbuf_data_len(m)
 
+/**
+ * Tests if an mbuf is a control mbuf
+ *
+ * @param m
+ *   The mbuf.to be tested
+ * @return
+ *   - True (1) if the mbuf is a control mbuf
+ *   - False(0) otherwise
+ */
+static inline int
+rte_is_ctrlmbuf(struct rte_mbuf *m)
+{
+	return (!!(m->ol_flags & CTRL_MBUF_FLAG));
+}
+
 /* Operations on pkt mbuf */
 
 /**
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-6-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

Acked-by
#35 lore raw

Re: [PATCH 05/13] mbuf: introduce a flag to indicate a control mbuf

Hi Bruce,

Quoted 9 lines · Bruce Richardson

Since the flags field is now 64-bits, we can allow one bit to be used to indicate a control i.e. non-packet mbuf. Dedicate the high bit (bit 63) for this purpose and add in a utility macro to test if a given mbuf has the bit set or not.

[...]

Quoted 9 lines

+/**
+ * Tests if an mbuf is a control mbuf
+ *
+ * @param m
+ * The mbuf.to be tested
+ * @return
+ * - True (1) if the mbuf is a control mbuf
+ * - False(0) otherwise
+ */

Typo s/mbuf.to/mbuf to/

#36 lore raw

PATCH 6/13 mbuf: minor changes for readability

In reply to a message not in the archive

* Ensure comments line up correctly
* Simplify the #ifdefs around the refcnt fields to make them clearer

---
 lib/librte_mbuf/rte_mbuf.h | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
lib/librte_mbuf/rte_mbuf.h +8 -9
index 7b0b4f2..5260001 100644
@@ -126,7 +126,6 @@ struct rte_mbuf {
 	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
 
-#ifdef RTE_MBUF_REFCNT
 	/**
 	 * 16-bit Reference counter.
 	 * It should only be accessed using the following functions:
@@ -136,12 +135,12 @@ struct rte_mbuf {
 	 * config option.
 	 */
 	union {
-		rte_atomic16_t refcnt_atomic;   /**< Atomically accessed refcnt */
-		uint16_t refcnt;                /**< Non-atomically accessed refcnt */
-	};
-#else
-	uint16_t refcnt_reserved;     /**< Do not use this field */
+#ifdef RTE_MBUF_REFCNT
+		rte_atomic16_t refcnt_atomic; /**< Atomically accessed refcnt */
+		uint16_t refcnt;              /**< Non-atomically accessed refcnt */
 #endif
+		uint16_t refcnt_reserved;     /**< Do not use this field */
+	};
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
@@ -155,12 +154,12 @@ struct rte_mbuf {
 	uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
 	uint16_t vlan_tci;      /**< VLAN Tag Control Identifier (CPU order). */
 	union {
-		uint32_t rss;       /**< RSS hash result if RSS enabled */
+		uint32_t rss;   /**< RSS hash result if RSS enabled */
 		struct {
 			uint16_t hash;
 			uint16_t id;
-		} fdir;             /**< Filter identifier if FDIR enabled */
-		uint32_t sched;     /**< Hierarchical scheduler */
+		} fdir;         /**< Filter identifier if FDIR enabled */
+		uint32_t sched; /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
 	/* fields only used in slow path or on TX */
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-7-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#39 lore raw

PATCH 7/13 mbuf: use macros only to access the mbuf metadata

In reply to a message not in the archive

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +8 -14
index 5260001..ca66d9a 100644
@@ -166,31 +166,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-8-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

Acked-by
#41 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Bruce,

Quoted 44 lines · Bruce Richardson

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +5 -12
index 5260001..ca66d9a 100644
@@ -166,31 +166,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 190

Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +3 -2
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.

I think it goes in the good direction. So:

Just one question: why not removing RTE_MBUF_METADATA*() macros? I'd just provide one macro that gives a (void*) to the first byte after the mbuf structure.

The format of the metadata is up to the application, that usually casts (m + 1) into a private structure, making the macros not very useful. I suggest to move these macros outside rte_mbuf.h, in the application-specific or library-specific header, what do you think?

Regards,
Olivier

A message that is not in the archive (sent off-list or not yet received): [email protected]

#43 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Quoted 73 lines

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Monday, September 08, 2014 1:06 PM
To: Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Bruce,

Quoted 14 lines · Bruce Richardson

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +0 -0
index 5260001..ca66d9a 100644

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 166

Quoted 1 line
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -166,31 +166,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.

*/

Quoted 29 lines
lib/librte_mbuf/rte_mbuf.h +5 -12
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 190

Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +3 -2
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.

I think it goes in the good direction. So:

Just one question: why not removing RTE_MBUF_METADATA*() macros? I'd just provide one macro that gives a (void*) to the first byte after the mbuf structure.

Quoted 8 lines

The format of the metadata is up to the application, that usually casts (m + 1) into a private structure, making the macros not very useful. I suggest to move these macros outside rte_mbuf.h, in the application-specific or library-specific header, what do you think?

Regards,
Olivier

Yes, I'll look into that.

/Bruce

A message that is not in the archive (sent off-list or not yet received): 59AF69C657FD0841A61C55336867B5B0343EFAA3-kPTMFJFq+rELt2AQoY/[email protected]

#45 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Bruce, Olivier,

What is the reason to remove this field? Please explain the rationale of removing this field.

We previously agreed we need to provide an easy and standard mechanism for applications to extend the mandatory per buffer metadata (mbuf) with optional application-dependent metadata. This field just provides a clean way for the apps to know where is the end of the mandatory metadata, i.e. the first location in the packet buffer where the app can add its own metadata (of course, the app has to manage the headroom space before the first byte of packet data). A zero-size field is the standard mechanism that DPDK uses extensively in pretty much every library to access memory immediately after a header structure.

The impact of removing this field is that there is no standard way to identify where the end of the mandatory metadata is, so each application will have to reinvent this. With no clear convention, we will end up with a lot of non-standard ways. Every time the format of the mbuf structure is going to be changed, this can potentially break applications that use custom metadata, while using this simple standard mechanism would prevent this. So why remove this?

Having applications define their optional meta-data is a real need. Please take a look at the Service Chaining IEFT emerging protocols (https://datatracker.ietf.org/wg/sfc/documents/), which provide standard mechanisms for applications to define their own packet meta-data and share it between the elements of the processing pipeline (for Service Chaining, these are typically virtual machines scattered amongst the data center).

And, in my opinion, there is no negative impact/cost associated with keeping this field.

Regards,
Cristian

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Richardson, Bruce
Sent: Tuesday, September 9, 2014 10:01 AM
To: Olivier MATZ; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Quoted 73 lines

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Monday, September 08, 2014 1:06 PM
To: Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Bruce,

Quoted 14 lines · Bruce Richardson

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +0 -0
index 5260001..ca66d9a 100644

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 166

Quoted 1 line
lib/librte_mbuf/rte_mbuf.h +0 -0
@@ -166,31 +166,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.

*/

Quoted 29 lines
lib/librte_mbuf/rte_mbuf.h +5 -12
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 190

Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +3 -2
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.

I think it goes in the good direction. So:

Just one question: why not removing RTE_MBUF_METADATA*() macros? I'd just provide one macro that gives a (void*) to the first byte after the mbuf structure.

Quoted 8 lines

The format of the metadata is up to the application, that usually casts (m + 1) into a private structure, making the macros not very useful. I suggest to move these macros outside rte_mbuf.h, in the application-specific or library-specific header, what do you think?

Regards,
Olivier

Yes, I'll look into that.

/Bruce
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

A message that is not in the archive (sent off-list or not yet received): 3EB4FA525960D640B5BDFFD6A3D891262E070D42-kPTMFJFq+rEMvF1YICWikbfspsVTdybXVpNB7YpNyf8@public.gmane.org

#47 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hello Cristian,

Quoted 2 lines

What is the reason to remove this field? Please explain the rationale of removing this field.

The rationale is explained in
http://dpdk.org/ml/archives/dev/2014-September/005232.html

"The format of the metadata is up to the application".

The type of data the application stores after the mbuf has not to be defined in the mbuf. These macros limits the types of metadata to uint8, uint16, uint32, uint64? What should I do if I need a void*, a struct foo ? Should we add a macro for each possible type?

Quoted 4 lines

We previously agreed we need to provide an easy and standard mechanism for applications to extend the mandatory per buffer metadata (mbuf) with optional application-dependent metadata.

Defining a structure in the application which does not pollute the rte_mbuf structure is "easy and standard(TM)" too.

Quoted 8 lines

This field just provides a clean way for the apps to know where is the end of the mandatory metadata, i.e. the first location in the packet buffer where the app can add its own metadata (of course, the app has to manage the headroom space before the first byte of packet data). A zero-size field is the standard mechanism that DPDK uses extensively in pretty much every library to access memory immediately after a header structure.

Having the following is clean too:

struct metadata {
     ...
};
struct app_mbuf {
     struct rte_mbuf mbuf;
     struct metadata metadata;
};

There is no need to define anything in the mbuf structure.

Quoted 9 lines

The impact of removing this field is that there is no standard way to identify where the end of the mandatory metadata is, so each application will have to reinvent this. With no clear convention, we will end up with a lot of non-standard ways. Every time the format of the mbuf structure is going to be changed, this can potentially break applications that use custom metadata, while using this simple standard mechanism would prevent this. So why remove this?

Waow. Five occurences of "standard" until now. Could you give a reference to the standard you're refering to? :)

Our application defines private metadata in mbufs in the way described above, we never changed that since we're supporting the dpdk. So I don't understand when you say that each time mbuf is reformatted it breaks the application.

Quoted 2 lines

Having applications define their optional meta-data is a real need.

Sure. This patch does not prevent this at all. You can continue to do exactly the same, but in the concerned application, not in the generic mbuf structure.

Quoted 3 lines

Please take a look at the Service Chaining IEFT emerging protocols (https://datatracker.ietf.org/wg/sfc/documents/), which provide standard mechanisms for applications to define their own

Six :)

I'm not sure these documents define the way to extend a packet structure with metadata in a C program. Again, Bruce's patch does not prevent to do what you need, it just moves it at the proper place.

Quoted 6 lines

packet meta-data and share it between the elements of the processing pipeline (for Service Chaining, these are typically virtual machines scattered amongst the data center).

And, in my opinion, there is no negative impact/cost associated with keeping this field.

To summarize what I think:

- this patch does not prevent to do what you want to do
- removing the macros help to have a shorter and more comprehensible mbuf structure
- the previous approach does not scale because it would require a macro per type

Quoted 8 lines

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

This is a public mailing list, this disclaimer is invalid.

Regards,
Olivier

A message that is not in the archive (sent off-list or not yet received): [email protected]

#49 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Olivier,

I agree that your suggested approach for application-dependent metadata makes sense, in fact the two approaches work in exactly the same way (packet metadata immediately after the regular mbuf), there is only a subtle difference, which is related to defining consistent DPDK usage guidelines.

1. Advertising the presence of application-dependent meta-data as supported mechanism
If we explicitly have a metadata zero-size field at the end of the mbuf, we basically tell people that adding their own application meta-data at the end of the mandatory meta-data (mbuf structure) is a mechanism that DPDK allows and supports, and will continue to do so for the foreseeable future. In other words, we guarantee that an application doing so will continue to build successfully with future releases of DPDK, and we will not introduce changes in DPDK that could potentially break this mechanism. It is also a hint to people of where to put their application dependent meta-data.

2. Defining a standard base address for the application-dependent metadata
- There are also libraries in DPDK that work with application dependent meta-data, currently these are the Packet Framework libraries: librte_port, librte_table, librte_pipeline. Of course, the library does not have the knowledge of the application dependent meta-data format, so they treat it as opaque array of bytes, with the offset and size of the array given as arguments. In my opinion, it is safer (and more elegant) if these libraries (and others) can rely on an mbuf API to access the application dependent meta-data (in an opaque way) rather than make an assumption about the mbuf (i.e. the location of custom metadata relative to the mbuf) that is not clearly supported/defined by the mbuf library.
- By having this API, we basically say: we define the custom meta-data base address (first location where custom metadata _could_ be placed) immediately after the mbuf, so libraries and apps accessing custom meta-data should do so by using a relative offset from this base rather than each application defining its own base: immediately after mbuf, or 128 bytes after mbuf, or 64 bytes before the end of the buffer, or other.

More (minor) comments inline below.

Thanks,
Cristian

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Friday, September 12, 2014 10:02 PM
To: Dumitrescu, Cristian; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hello Cristian,

Quoted 2 lines

What is the reason to remove this field? Please explain the rationale of removing this field.

The rationale is explained in
http://dpdk.org/ml/archives/dev/2014-September/005232.html

"The format of the metadata is up to the application".

The type of data the application stores after the mbuf has not to be defined in the mbuf. These macros limits the types of metadata to uint8, uint16, uint32, uint64? What should I do if I need a void*, a struct foo ? Should we add a macro for each possible type?

[Cristian] Actually, this is not correct, as macros to access metadata through pointers (to void or scalar types) are provided as well. This pointer can be converted by the application to the format is defines.

Quoted 4 lines

We previously agreed we need to provide an easy and standard mechanism for applications to extend the mandatory per buffer metadata (mbuf) with optional application-dependent metadata.

Defining a structure in the application which does not pollute the rte_mbuf structure is "easy and standard(TM)" too.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 8 lines

This field just provides a clean way for the apps to know where is the end of the mandatory metadata, i.e. the first location in the packet buffer where the app can add its own metadata (of course, the app has to manage the headroom space before the first byte of packet data). A zero-size field is the standard mechanism that DPDK uses extensively in pretty much every library to access memory immediately after a header structure.

Having the following is clean too:

struct metadata {
     ...
};
struct app_mbuf {
     struct rte_mbuf mbuf;
     struct metadata metadata;
};

There is no need to define anything in the mbuf structure.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 9 lines

The impact of removing this field is that there is no standard way to identify where the end of the mandatory metadata is, so each application will have to reinvent this. With no clear convention, we will end up with a lot of non-standard ways. Every time the format of the mbuf structure is going to be changed, this can potentially break applications that use custom metadata, while using this simple standard mechanism would prevent this. So why remove this?

Waow. Five occurences of "standard" until now.
[Cristian] I am sorry :)

Could you give a
reference to the standard you're refering to? :)

[Cristian] See the IEFT Service Function Chaining link below, the environment is different (data center pipeline vs. CPU core-level pipeline), but the concepts are very similar.

Our application defines private metadata in mbufs in the way described above, we never changed that since we're supporting the dpdk. So I don't understand when you say that each time mbuf is reformatted it breaks the application.

Quoted 2 lines

Having applications define their optional meta-data is a real need.

Sure. This patch does not prevent this at all. You can continue to do exactly the same, but in the concerned application, not in the generic mbuf structure.

Quoted 3 lines

Please take a look at the Service Chaining IEFT emerging protocols (https://datatracker.ietf.org/wg/sfc/documents/), which provide standard mechanisms for applications to define their own

Six :)

I'm not sure these documents define the way to extend a packet structure with metadata in a C program. Again, Bruce's patch does not prevent to do what you need, it just moves it at the proper place.

Quoted 6 lines

packet meta-data and share it between the elements of the processing pipeline (for Service Chaining, these are typically virtual machines scattered amongst the data center).

And, in my opinion, there is no negative impact/cost associated with keeping this field.

To summarize what I think:

- this patch does not prevent to do what you want to do
- removing the macros help to have a shorter and more comprehensible mbuf structure
- the previous approach does not scale because it would require a macro per type

Quoted 8 lines

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

This is a public mailing list, this disclaimer is invalid.

Regards,
Olivier

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

A message that is not in the archive (sent off-list or not yet received): 3EB4FA525960D640B5BDFFD6A3D891262E071FE6-kPTMFJFq+rEMvF1YICWikbfspsVTdybXVpNB7YpNyf8@public.gmane.org

#51 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

I completely agree with Cristian here, instead of leaving to applications where to place their meta data, we can provide a guidance by having this field about placement of application meta while maintaining transparency on the contents of application meta information.

Regards
Kannan Babu Ramia
Sr.System Architect
Communication Storage Infrastructure Group
DCG
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Dumitrescu, Cristian
Sent: Wednesday, September 17, 2014 1:37 AM
To: Olivier MATZ; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Olivier,

I agree that your suggested approach for application-dependent metadata makes sense, in fact the two approaches work in exactly the same way (packet metadata immediately after the regular mbuf), there is only a subtle difference, which is related to defining consistent DPDK usage guidelines.

1. Advertising the presence of application-dependent meta-data as supported mechanism If we explicitly have a metadata zero-size field at the end of the mbuf, we basically tell people that adding their own application meta-data at the end of the mandatory meta-data (mbuf structure) is a mechanism that DPDK allows and supports, and will continue to do so for the foreseeable future. In other words, we guarantee that an application doing so will continue to build successfully with future releases of DPDK, and we will not introduce changes in DPDK that could potentially break this mechanism. It is also a hint to people of where to put their application dependent meta-data.

2. Defining a standard base address for the application-dependent metadata
- There are also libraries in DPDK that work with application dependent meta-data, currently these are the Packet Framework libraries: librte_port, librte_table, librte_pipeline. Of course, the library does not have the knowledge of the application dependent meta-data format, so they treat it as opaque array of bytes, with the offset and size of the array given as arguments. In my opinion, it is safer (and more elegant) if these libraries (and others) can rely on an mbuf API to access the application dependent meta-data (in an opaque way) rather than make an assumption about the mbuf (i.e. the location of custom metadata relative to the mbuf) that is not clearly supported/defined by the mbuf library.
- By having this API, we basically say: we define the custom meta-data base address (first location where custom metadata _could_ be placed) immediately after the mbuf, so libraries and apps accessing custom meta-data should do so by using a relative offset from this base rather than each application defining its own base: immediately after mbuf, or 128 bytes after mbuf, or 64 bytes before the end of the buffer, or other.

More (minor) comments inline below.

Thanks,
Cristian

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Friday, September 12, 2014 10:02 PM
To: Dumitrescu, Cristian; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hello Cristian,

Quoted 2 lines

What is the reason to remove this field? Please explain the rationale of removing this field.

The rationale is explained in
http://dpdk.org/ml/archives/dev/2014-September/005232.html

"The format of the metadata is up to the application".

The type of data the application stores after the mbuf has not to be defined in the mbuf. These macros limits the types of metadata to uint8, uint16, uint32, uint64? What should I do if I need a void*, a struct foo ? Should we add a macro for each possible type?

[Cristian] Actually, this is not correct, as macros to access metadata through pointers (to void or scalar types) are provided as well. This pointer can be converted by the application to the format is defines.

Quoted 3 lines

We previously agreed we need to provide an easy and standard mechanism for applications to extend the mandatory per buffer metadata (mbuf) with optional application-dependent metadata.

Defining a structure in the application which does not pollute the rte_mbuf structure is "easy and standard(TM)" too.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 7 lines

This field just provides a clean way for the apps to know where is the end of the mandatory metadata, i.e. the first location in the packet buffer where the app can add its own metadata (of course, the app has to manage the headroom space before the first byte of packet data). A zero-size field is the standard mechanism that DPDK uses extensively in pretty much every library to access memory immediately after a header structure.

Having the following is clean too:

struct metadata {
     ...
};
struct app_mbuf {
     struct rte_mbuf mbuf;
     struct metadata metadata;
};

There is no need to define anything in the mbuf structure.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 8 lines

The impact of removing this field is that there is no standard way to identify where the end of the mandatory metadata is, so each application will have to reinvent this. With no clear convention, we will end up with a lot of non-standard ways. Every time the format of the mbuf structure is going to be changed, this can potentially break applications that use custom metadata, while using this simple standard mechanism would prevent this. So why remove this?

Waow. Five occurences of "standard" until now.
[Cristian] I am sorry :)

Could you give a
reference to the standard you're refering to? :)

[Cristian] See the IEFT Service Function Chaining link below, the environment is different (data center pipeline vs. CPU core-level pipeline), but the concepts are very similar.

Our application defines private metadata in mbufs in the way described above, we never changed that since we're supporting the dpdk. So I don't understand when you say that each time mbuf is reformatted it breaks the application.

Quoted 1 line

Having applications define their optional meta-data is a real need.

Sure. This patch does not prevent this at all. You can continue to do exactly the same, but in the concerned application, not in the generic mbuf structure.

Quoted 3 lines

Please take a look at the Service Chaining IEFT emerging protocols
(https://datatracker.ietf.org/wg/sfc/documents/), which provide standard mechanisms for applications to define their own

Six :)

I'm not sure these documents define the way to extend a packet structure with metadata in a C program. Again, Bruce's patch does not prevent to do what you need, it just moves it at the proper place.

Quoted 6 lines

packet meta-data and share it between the elements of the processing pipeline (for Service Chaining, these are typically virtual machines scattered amongst the data center).

And, in my opinion, there is no negative impact/cost associated with keeping this field.

To summarize what I think:

- this patch does not prevent to do what you want to do
- removing the macros help to have a shorter and more comprehensible mbuf structure
- the previous approach does not scale because it would require a macro per type

Quoted 8 lines

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

This is a public mailing list, this disclaimer is invalid.

Regards,
Olivier

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

A message that is not in the archive (sent off-list or not yet received): 682698A055A0F44AA47192B20141149711B1FFE6-yHIBzpp8AekFyVwBAnZdSLfspsVTdybXVpNB7YpNyf8@public.gmane.org

#53 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Quoted 7 lines

-----Original Message-----
From: Ramia, Kannan Babu
Sent: Tuesday, September 16, 2014 11:06 PM
To: Dumitrescu, Cristian; Olivier MATZ; Richardson, Bruce; [email protected]
Subject: RE: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Quoted 5 lines

I completely agree with Cristian here, instead of leaving to applications where to place their meta data, we can provide a guidance by having this field about placement of application meta while maintaining transparency on the contents of application meta information.

My opinion on this is that this is better served via documentation or a comment in the code. The reason is that this approach is not going to be suitable for all applications. The mbuf headroom being used by the metadata is actually designed to be used for any additional headers to be added to the packet - though other things can obviously be stored in it too. Therefore the amount of metadata that can be stored in it will depend from application to application, as any apps doing e.g. tunnelling will need the headroom for tunnelling headers and may only be able to store a small amount of metadata - potentially none. For larger amounts of metadata - I would feel that anything over 64-bytes or so - I have proposed adding in a separate userdata pointer in the mbuf structure so that apps have the option of storing the metadata externally e.g. pointing to a flow table entry or similar. [Please see mbuf rework patch set 3 proposal]. Because of this, I think it's better to put in a comment in the code indicating that metadata can go in the headroom, document this properly - including caveats and limitations - in the documentation, and provide an example of doing such - something we already have in the packet framework.

All that being said, and while I think this is a good patch, I don't feel too strongly about it. I'm happy enough if this particular patch does not get merged in for 1.8, as it's incidental to the overall mbuf changes.

Regards,
/Bruce

Quoted 197 lines

Regards
Kannan Babu Ramia
Sr.System Architect
Communication Storage Infrastructure Group
DCG
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Dumitrescu, Cristian
Sent: Wednesday, September 17, 2014 1:37 AM
To: Olivier MATZ; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the
mbuf metadata

Hi Olivier,

I agree that your suggested approach for application-dependent metadata makes sense, in fact the two approaches work in exactly the same way (packet metadata immediately after the regular mbuf), there is only a subtle difference, which is related to defining consistent DPDK usage guidelines.

1. Advertising the presence of application-dependent meta-data as supported mechanism If we explicitly have a metadata zero-size field at the end of the mbuf, we basically tell people that adding their own application meta-data at the end of the mandatory meta-data (mbuf structure) is a mechanism that DPDK allows and supports, and will continue to do so for the foreseeable future. In other words, we guarantee that an application doing so will continue to build successfully with future releases of DPDK, and we will not introduce changes in DPDK that could potentially break this mechanism. It is also a hint to people of where to put their application dependent meta-data.

2. Defining a standard base address for the application-dependent metadata
- There are also libraries in DPDK that work with application dependent meta- data, currently these are the Packet Framework libraries: librte_port, librte_table, librte_pipeline. Of course, the library does not have the knowledge of the application dependent meta-data format, so they treat it as opaque array of bytes, with the offset and size of the array given as arguments. In my opinion, it is safer (and more elegant) if these libraries (and others) can rely on an mbuf API to access the application dependent meta-data (in an opaque way) rather than make an assumption about the mbuf (i.e. the location of custom metadata relative to the mbuf) that is not clearly supported/defined by the mbuf library.
- By having this API, we basically say: we define the custom meta-data base address (first location where custom metadata _could_ be placed) immediately after the mbuf, so libraries and apps accessing custom meta-data should do so by using a relative offset from this base rather than each application defining its own base: immediately after mbuf, or 128 bytes after mbuf, or 64 bytes before the end of the buffer, or other.

More (minor) comments inline below.

Thanks,
Cristian

-----Original Message-----
From: Olivier MATZ [mailto:[email protected]]
Sent: Friday, September 12, 2014 10:02 PM
To: Dumitrescu, Cristian; Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hello Cristian,

Quoted 2 lines

What is the reason to remove this field? Please explain the rationale of removing this field.

The rationale is explained in
http://dpdk.org/ml/archives/dev/2014-September/005232.html

"The format of the metadata is up to the application".

The type of data the application stores after the mbuf has not to be defined in the mbuf. These macros limits the types of metadata to uint8, uint16, uint32, uint64? What should I do if I need a void*, a struct foo ? Should we add a macro for each possible type?

[Cristian] Actually, this is not correct, as macros to access metadata through pointers (to void or scalar types) are provided as well. This pointer can be converted by the application to the format is defines.

Quoted 3 lines

We previously agreed we need to provide an easy and standard mechanism for applications to extend the mandatory per buffer metadata (mbuf) with optional application-dependent metadata.

Defining a structure in the application which does not pollute the rte_mbuf structure is "easy and standard(TM)" too.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 7 lines

This field just provides a clean way for the apps to know where is the end of the mandatory metadata, i.e. the first location in the packet buffer where the app can add its own metadata (of course, the app has to manage the headroom space before the first byte of packet data). A zero-size field is the standard mechanism that DPDK uses extensively in pretty much every library to access memory immediately after a header structure.

Having the following is clean too:

struct metadata {
     ...
};
struct app_mbuf {
     struct rte_mbuf mbuf;
     struct metadata metadata;
};

There is no need to define anything in the mbuf structure.

[Cristian] I agree, both approaches work the same really, it is just the difference in advertising the presence of meta-data as supported mechanism and defining a standard base address for it.

Quoted 8 lines

The impact of removing this field is that there is no standard way to identify where the end of the mandatory metadata is, so each application will have to reinvent this. With no clear convention, we will end up with a lot of non-standard ways. Every time the format of the mbuf structure is going to be changed, this can potentially break applications that use custom metadata, while using this simple standard mechanism would prevent this. So why remove this?

Waow. Five occurences of "standard" until now.
[Cristian] I am sorry :)

Could you give a
reference to the standard you're refering to? :)

[Cristian] See the IEFT Service Function Chaining link below, the environment is different (data center pipeline vs. CPU core-level pipeline), but the concepts are very similar.

Our application defines private metadata in mbufs in the way described above, we never changed that since we're supporting the dpdk. So I don't understand when you say that each time mbuf is reformatted it breaks the application.

Quoted 1 line

Having applications define their optional meta-data is a real need.

Sure. This patch does not prevent this at all. You can continue to do exactly the same, but in the concerned application, not in the generic mbuf structure.

Quoted 3 lines

Please take a look at the Service Chaining IEFT emerging protocols
(https://datatracker.ietf.org/wg/sfc/documents/), which provide standard mechanisms for applications to define their own

Six :)

I'm not sure these documents define the way to extend a packet structure with metadata in a C program. Again, Bruce's patch does not prevent to do what you need, it just moves it at the proper place.

Quoted 6 lines

packet meta-data and share it between the elements of the processing pipeline (for Service Chaining, these are typically virtual machines scattered amongst the data center).

And, in my opinion, there is no negative impact/cost associated with keeping this field.

To summarize what I think:

- this patch does not prevent to do what you want to do
- removing the macros help to have a shorter and more comprehensible mbuf structure
- the previous approach does not scale because it would require a macro per type

Quoted 8 lines

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole

use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

This is a public mailing list, this disclaimer is invalid.

Regards,
Olivier

--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered
Number: 308263 Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

A message that is not in the archive (sent off-list or not yet received): 59AF69C657FD0841A61C55336867B5B0343F2BD2-kPTMFJFq+rELt2AQoY/[email protected]

#55 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

2014-09-17 10:31, Richardson, Bruce:

Quoted 16 lines
Quoted 6 lines

From: Ramia, Kannan Babu

I completely agree with Cristian here, instead of leaving to applications where to place their meta data, we can provide a guidance by having this field about placement of application meta while maintaining transparency on the contents of application meta information.

My opinion on this is that this is better served via documentation or a comment in the code. The reason is that this approach is not going to be suitable for all applications. The mbuf headroom being used by the metadata is actually designed to be used for any additional headers to be added to the packet - though other things can obviously be stored in it too. Therefore the amount of metadata that can be stored in it will depend from application to application, as any apps doing e.g. tunnelling will need the headroom for tunnelling headers and may only be able to store a small amount of metadata - potentially none. For larger amounts of metadata - I

Quoted 8 lines

would feel that anything over 64-bytes or so - I have proposed adding in a separate userdata pointer in the mbuf structure so that apps have the option of storing the metadata externally e.g. pointing to a flow table entry or similar. [Please see mbuf rework patch set 3 proposal]. Because of this, I think it's better to put in a comment in the code indicating that metadata can go in the headroom, document this properly - including caveats and limitations - in the documentation, and provide an example of doing such - something we already have in the packet framework.

I agree that replacing these markers by documentation give more accurate informations and (bonus) is simpler.
When documentation will be embedded in the git repository, I'd like to see a patch to document these mbuf usages.

Quoted 3 lines

All that being said, and while I think this is a good patch, I don't feel too strongly about it. I'm happy enough if this particular patch does not get merged in for 1.8, as it's incidental to the overall mbuf changes.

I think also it's a good patch so I keep it.

Signature · 1 line
Thomas
#56 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

In reply to a message not in the archive
Quoted 61 lines · Olivier MATZ

Hi Bruce,

Quoted 44 lines · Bruce Richardson

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +5 -12
index 5260001..ca66d9a 100644
@@ -166,31 +166,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 190

Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +3 -2
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.

I think it goes in the good direction. So:

Quoted 9 lines

Just one question: why not removing RTE_MBUF_METADATA*() macros? I'd just provide one macro that gives a (void*) to the first byte after the mbuf structure.

The format of the metadata is up to the application, that usually casts (m + 1) into a private structure, making the macros not very useful. I suggest to move these macros outside rte_mbuf.h, in the application-specific or library-specific header, what do you think?

Things look to work if I just move the definitions en-masse into rte_port.h. Is that the sort of change you would be thinking of? I was wondering about replacing the typed macros here with a generic one to access just beyond the definition of the mbuf structure, but on further thought, I believe that using the buf_addr pointer in the mbuf data structure is probably enough for most applications. [An alternative to moving the definitions into rte_port.h is to move them into rte_table.h and having port_frag.c use the buf_addr pointer instead of a macro to get at the metadata. All other references to the macros apart from two in that port file, are in the tables or in apps that use the tables lib.] What do you think?

/Bruce

#57 lore raw

Re: [PATCH 07/13] mbuf: use macros only to access the mbuf metadata

Hi Bruce,

Quoted 10 lines · Bruce Richardson
Quoted 9 lines

Just one question: why not removing RTE_MBUF_METADATA*() macros? I'd just provide one macro that gives a (void*) to the first byte after the mbuf structure.

The format of the metadata is up to the application, that usually casts (m + 1) into a private structure, making the macros not very useful. I suggest to move these macros outside rte_mbuf.h, in the application-specific or library-specific header, what do you think?

Quoted 13 lines

Things look to work if I just move the definitions en-masse into rte_port.h. Is that the sort of change you would be thinking of? I was wondering about replacing the typed macros here with a generic one to access just beyond the definition of the mbuf structure, but on further thought, I believe that using the buf_addr pointer in the mbuf data structure is probably enough for most applications. [An alternative to moving the definitions into rte_port.h is to move them into rte_table.h and having port_frag.c use the buf_addr pointer instead of a macro to get at the metadata. All other references to the macros apart from two in that port file, are in the tables or in apps that use the tables lib.]
What do you think?

Yes, moving the macros in rte_port.h looks good to me, since the libraries using rte_ports are the users of this specific metadata format.

Thanks!
Olivier

#58 lore raw

PATCH 8/13 mbuf: add named points inside the mbuf structure

In reply to a message not in the archive

Add markers or "labels" at given points inside the mbuf which can be used instead of individual fields to identify the start of logical sections inside the mbuf.

The use of typedefs and dummy fields was chosen over using unions because of a couple reasons:
* unions cause an extra level of indentation (more likely two levels as a union containing a struct for multiple fields would be needed). This makes the lines longer than they need to be and increases the need for wrapping. [This was the main reason]
* with markers, you can apply multiple markers at the same point if wanted.

---
 lib/librte_mbuf/rte_mbuf.h | 9 +++++++++
 1 file changed, 9 insertions(+)
lib/librte_mbuf/rte_mbuf.h +9 -0
index ca66d9a..591be95 100644
@@ -115,14 +115,22 @@ extern "C" {
  */
 #define PKT_TX_OFFLOAD_MASK (PKT_TX_VLAN_PKT | PKT_TX_IP_CKSUM | PKT_TX_L4_MASK)
 
+/* define a set of marker types that can be used to refer to set points in the
+ * mbuf */
+typedef void    *MARKER[0];   /**< generic marker for a point in a structure */
+typedef uint64_t MARKER64[0]; /**< marker that allows us to overwrite 8 bytes
+                               * with a single assignment */
 /**
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
+	MARKER cacheline0;
+
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
 
 	/* next 8 bytes are initialised on RX descriptor rearm */
+	MARKER64 rearm_data;
 	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
 
@@ -147,6 +155,7 @@ struct rte_mbuf {
 	uint64_t ol_flags;      /**< Offload features. */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
+	MARKER rx_descriptor_fields1;
 	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-9-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

Acked-by
#60 lore raw

Re: [PATCH 08/13] mbuf: add named points inside the mbuf structure

Quoted 13 lines · Bruce Richardson

Add markers or "labels" at given points inside the mbuf which can be used instead of individual fields to identify the start of logical sections inside the mbuf.

The use of typedefs and dummy fields was chosen over using unions because of a couple reasons:
* unions cause an extra level of indentation (more likely two levels as a union containing a struct for multiple fields would be needed). This makes the lines longer than they need to be and increases the need for wrapping. [This was the main reason]
* with markers, you can apply multiple markers at the same point if wanted.

Quoted 1 line
#61 lore raw

PATCH 9/13 ixgbe: rework vector pmd following mbuf changes

In reply to a message not in the archive

The vector PMD expects fields to be in a specific order so that it can do vector operations on multiple fields at a time. Following mbuf rework, adjust driver to take account of the new layout and re-enable it in the config.

---
 config/common_linuxapp                |   2 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |   2 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     |   4 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 136 +++++++++++++---------------------
 4 files changed, 54 insertions(+), 90 deletions(-)
config/common_linuxapp +1 -1
index b140af7..5bee910 100644
@@ -191,7 +191,7 @@ CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
 CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
 CONFIG_RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC=y
 CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=n
-CONFIG_RTE_IXGBE_INC_VECTOR=n
+CONFIG_RTE_IXGBE_INC_VECTOR=y
 CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
 
 #
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +1 -1
index c7f1642..f48c62a 100644
@@ -2167,7 +2167,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev,
 		if (!ixgbe_rx_vec_condition_check(dev)) {
 			PMD_INIT_LOG(INFO, "Vector rx enabled, please make "
 				     "sure RX burst size no less than 32.\n");
-			ixgbe_rxq_vec_setup(rxq, socket_id);
+			ixgbe_rxq_vec_setup(rxq);
 			dev->rx_pkt_burst = ixgbe_recv_pkts_vec;
 		}
 #endif
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +2 -2
index 70dbcb0..5e98b21 100644
@@ -115,6 +115,7 @@ struct igb_rx_queue {
 	struct igb_rx_entry *sw_ring; /**< address of RX software ring. */
 	struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */
 	struct rte_mbuf *pkt_last_seg; /**< Last segment of current packet. */
+	uint64_t            mbuf_initializer; /**< value to init mbufs */
 	uint16_t            nb_rx_desc; /**< number of RX descriptors. */
 	uint16_t            rx_tail;  /**< current value of RDT register. */
 	uint16_t            nb_rx_hold; /**< number of held free RX desc. */
@@ -126,7 +127,6 @@ struct igb_rx_queue {
 #ifdef RTE_IXGBE_INC_VECTOR
 	uint16_t            rxrearm_nb; /**< the idx we start the re-arming from */
 	uint16_t            rxrearm_start;  /**< number of remaining to be re-armed */
-	__m128i             misc_info;  /**< cache XMM combine port_id/crc/nb_segs */
 #endif
 	uint16_t            rx_free_thresh; /**< max free RX desc to hold. */
 	uint16_t            queue_id; /**< RX queue index. */
@@ -260,7 +260,7 @@ struct ixgbe_txq_ops {
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct igb_tx_queue *txq, unsigned int socket_id);
-int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq, unsigned int socket_id);
+int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
 #endif
 
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +50 -86

246 lines not on this page · raw message

Signature · 1 line
1.9.3
#62 lore raw

PATCH 10/13 mbuf: split mbuf across two cache lines.

In reply to a message not in the archive

This change splits the mbuf in two to move the pool and next pointers to the second cache line. This frees up 16 bytes in first cache line.

---
 app/test/test_mbuf.c       | 2 +-
 lib/librte_mbuf/rte_mbuf.h | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
app/test/test_mbuf.c +1 -1
index e3d896b..82136de 100644
@@ -782,7 +782,7 @@ test_failing_mbuf_sanity_check(void)
 static int
 test_mbuf(void)
 {
-	RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != 64);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != CACHE_LINE_SIZE * 2);
 
 	/* create pktmbuf pool if it does not exist */
 	if (pktmbuf_pool == NULL) {
lib/librte_mbuf/rte_mbuf.h +2 -1
index 591be95..db079ac 100644
@@ -171,7 +171,8 @@ struct rte_mbuf {
 		uint32_t sched; /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
-	/* fields only used in slow path or on TX */
+	/* second cache line - fields only used in slow path or on TX */
+	MARKER cacheline1 __rte_cache_aligned;
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-11-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#64 lore raw

Re: [PATCH 10/13] mbuf: split mbuf across two cache lines.

Hi Bruce,

Quoted 4 lines · Bruce Richardson

This change splits the mbuf in two to move the pool and next pointers to the second cache line. This frees up 16 bytes in first cache line.

I think you should provide more explanation in the commit log about why this change is needed. I remeber you already provided the arguments on the mailing list, but it could be interesting to have them in the log.

Regards,
Olivier

#65 lore raw

PATCH 11/13 mbuf: move l2_len and l3_len to second cache line

In reply to a message not in the archive

The l2_len and l3_len fields are used for TX offloads and so should be put on the second cache line, along with the other fields only used on TX.

---
 lib/librte_mbuf/rte_mbuf.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
lib/librte_mbuf/rte_mbuf.h +4 -2
index db079ac..d3c1745 100644
@@ -159,8 +159,7 @@ struct rte_mbuf {
 	uint16_t packet_type;   /**< Type of packet, e.g. protocols used */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
-	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
-	uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
+	uint16_t reserved;
 	uint16_t vlan_tci;      /**< VLAN Tag Control Identifier (CPU order). */
 	union {
 		uint32_t rss;   /**< RSS hash result if RSS enabled */
@@ -176,6 +175,9 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
+	/* fields to support TX offloads */
+	uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
+	uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1409759378-10113-12-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#67 lore raw

Re: [PATCH 11/13] mbuf: move l2_len and l3_len to second cache line

Hello Bruce,

I'm a little bit concerned about performance issues that would arise if these fields would go to the 2nd cache line.

For exampe, l2_len and l3_len fields are used by librte_ip_frag to find L3 and L4 headers position inside mbuf data. Thus, these values should be calculated by NIC offload, or by user on RX leg.

Secondly, (I wouldn't say on behalf of everyone, but) we use these fields in our libraries as well for needs of classification. For instance, in case you try to support other ethertypes which are not supported by NIC offload (MPLS, IPX etc), but you still need to point out L3 and L3 headers.

If my concerns are consistent, what would be possible suggestions?

03.09.2014 21:49, Bruce Richardson пишет:

Quoted 5 lines

The l2_len and l3_len fields are used for TX offloads and so should be put on the second cache line, along with the other fields only used on TX.

Signature · 5 lines
Sincerely,

Yerden Zhumabekov
STS, ACI
Astana, KZ

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 20140904102744.GA23231-IWE99D/oH1/[email protected]

#70 lore raw

Re: [PATCH 11/13] mbuf: move l2_len and l3_len to second cache line

I get your point. I've also read throught the code of various PMDs and have found no indication of setting l2_len/l3_len fields as well.

As for testing, we'd be happy to test the patchset but we are now in process of building our testing facilities so we are not ready to provide enough workload for the hardware/software. I was also wondering if anyone has run some test and can provide some numbers on that matter.

Personally, I don't think frag/reassemly app is a perfect example for evaluating 2nd cache line performance penalty. The offsets to L3 and L4 headers need to be calculated for all TCP/IP traffic and fragmented traffic is not representative in this case. Maybe it would be better to write an app which calculates these offsets for different set of mbufs and provides some stats. For example, l2fwd/l3fwd + additional l2_len and l3_len calculation.

And I'm also figuring out how to rewrite our app/libs (prefetch etc) to reflect the future changes in mbuf, hence my concerns :)

04.09.2014 16:27, Bruce Richardson пишет:

Quoted 40 lines

Hi Yerden,

I understand your concerns and it's good to have this discussion.

There are a number of reasons why I've moved these particular fields to the second cache line. Firstly, the main reason is that, obviously enough, not all fields will fit in cache line 0, and we need to prioritize what does get stored there. The guiding principle behind what fields get moved or not that I've chosen to use for this patch set is to move fields that are not used on the receive path (or the fastpath receive path, more specifically - so that we can move fields only used by jumbo frames that span mbufs) to the second cache line. From a search through the existing codebase, there are no drivers which set the l2/l3 length fields on RX, this is only used in reassembly libraries/apps and by the drivers on TX.

The other reason for moving it to the second cache line is that it logically belongs with all the other length fields that we need to add to enable tunneling support. [To get an idea of the extra fields that I propose adding to the mbuf, please see the RFC patchset I sent out previously as "[RFC PATCH 00/14] Extend the mbuf structure"]. While we probably can fit the 16-bits needed for l2/l3 length on the mbuf line 0, there is not enough room for all the lengths so we would end up splitting them with other fields in between.

So, in terms of what do to about this particular issue. I would hope that for applications that use these fields the impact should be small and/or possible to work around e.g. maybe prefetch second cache line on RX in driver. If not, then I'm happy to see about withdrawing this particular change and seeing if we can keep l2/l3 lengths on cache line zero, with other length fields being on cache line 1.

Question: would you consider the ip fragmentation and reassembly example apps in the Intel DPDK releases good examples to test to see the impacts of this change, or is there some other test you would prefer that I look to do? Can you perhaps test out the patch sets for the mbuf that I've upstreamed so far and let me know what regressions, if any, you see in your use-case scenarios?

Regards,
/Bruce

Signature · 5 lines
Sincerely,

Yerden Zhumabekov
STS, ACI
Astana, KZ

A message that is not in the archive (sent off-list or not yet received): [email protected]

#72 lore raw

Re: [PATCH 11/13] mbuf: move l2_len and l3_len to second cache line

Quoted 16 lines · Yerden Zhumabekov

I get your point. I've also read throught the code of various PMDs and have found no indication of setting l2_len/l3_len fields as well.

As for testing, we'd be happy to test the patchset but we are now in process of building our testing facilities so we are not ready to provide enough workload for the hardware/software. I was also wondering if anyone has run some test and can provide some numbers on that matter.

Personally, I don't think frag/reassemly app is a perfect example for evaluating 2nd cache line performance penalty. The offsets to L3 and L4 headers need to be calculated for all TCP/IP traffic and fragmented traffic is not representative in this case. Maybe it would be better to write an app which calculates these offsets for different set of mbufs and provides some stats. For example, l2fwd/l3fwd + additional l2_len and l3_len calculation.

Quoted 3 lines

And I'm also figuring out how to rewrite our app/libs (prefetch etc) to reflect the future changes in mbuf, hence my concerns :)

Just a final point on this. Note that the second cache line is always being read by the TX leg of the code to free back mbufs to their mbuf pool post- transmit. The overall fast-path RX+TX benchmarks show no performance degradation due to that access.

For sample apps, you make a good point indeed about the existing app not being very useful as they work on larger packets. I'll see what I can throw together here to make a more realistic test.

/Bruce

Quoted 49 lines

04.09.2014 16:27, Bruce Richardson ??????????:

Quoted 40 lines

Hi Yerden,

I understand your concerns and it's good to have this discussion.

There are a number of reasons why I've moved these particular fields to the second cache line. Firstly, the main reason is that, obviously enough, not all fields will fit in cache line 0, and we need to prioritize what does get stored there. The guiding principle behind what fields get moved or not that I've chosen to use for this patch set is to move fields that are not used on the receive path (or the fastpath receive path, more specifically - so that we can move fields only used by jumbo frames that span mbufs) to the second cache line. From a search through the existing codebase, there are no drivers which set the l2/l3 length fields on RX, this is only used in reassembly libraries/apps and by the drivers on TX.

The other reason for moving it to the second cache line is that it logically belongs with all the other length fields that we need to add to enable tunneling support. [To get an idea of the extra fields that I propose adding to the mbuf, please see the RFC patchset I sent out previously as "[RFC PATCH 00/14] Extend the mbuf structure"]. While we probably can fit the 16-bits needed for l2/l3 length on the mbuf line 0, there is not enough room for all the lengths so we would end up splitting them with other fields in between.

So, in terms of what do to about this particular issue. I would hope that for applications that use these fields the impact should be small and/or possible to work around e.g. maybe prefetch second cache line on RX in driver. If not, then I'm happy to see about withdrawing this particular change and seeing if we can keep l2/l3 lengths on cache line zero, with other length fields being on cache line 1.

Question: would you consider the ip fragmentation and reassembly example apps in the Intel DPDK releases good examples to test to see the impacts of this change, or is there some other test you would prefer that I look to do? Can you perhaps test out the patch sets for the mbuf that I've upstreamed so far and let me know what regressions, if any, you see in your use-case scenarios?

Regards,
/Bruce

Signature · 5 lines
Sincerely,

Yerden Zhumabekov
STS, ACI
Astana, KZ
#73 lore raw

PATCH 12/13 ixgbe: Fix perf regression due to moved pool ptr

In reply to a message not in the archive

Adjust the fast-path code to fix the regression caused by the pool pointer moving to the second cache line. This change adjusts the prefetching and also the way in which the mbufs are freed back to the mempool.
Note: slow-path e.g. path supporting jumbo frames, is still slower, but is dealt with by a later commit

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |  8 ++-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     | 14 +-----
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 93 +++++++++++++----------------------
 3 files changed, 39 insertions(+), 76 deletions(-)
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +3 -5
index f48c62a..ebbcee8 100644
@@ -142,10 +142,6 @@ ixgbe_tx_free_bufs(struct igb_tx_queue *txq)
 	 */
 	txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);
 
-	/* prefetch the mbufs that are about to be freed */
-	for (i = 0; i < txq->tx_rs_thresh; ++i)
-		rte_prefetch0((txep + i)->mbuf);
-
 	/* free buffers one at a time */
 	if ((txq->txq_flags & (uint32_t)ETH_TXQ_FLAGS_NOREFCOUNT) != 0) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
@@ -186,6 +182,7 @@ tx4(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 				((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 		txdp->read.olinfo_status =
 				(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+		rte_prefetch0(&(*pkts)->pool);
 	}
 }
 
@@ -205,6 +202,7 @@ tx1(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 			((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 	txdp->read.olinfo_status =
 			(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+	rte_prefetch0(&(*pkts)->pool);
 }
 
 /*
@@ -1876,7 +1874,7 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev,
 		PMD_INIT_LOG(INFO, "Using simple tx code path\n");
 #ifdef RTE_IXGBE_INC_VECTOR
 		if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ &&
-		    ixgbe_txq_vec_setup(txq, socket_id) == 0) {
+		    ixgbe_txq_vec_setup(txq) == 0) {
 			PMD_INIT_LOG(INFO, "Vector tx enabled.\n");
 			dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
 		}
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -13
index 5e98b21..dbb57af 100644
@@ -96,14 +96,6 @@ struct igb_tx_entry_v {
 };
 
 /**
- * continuous entry sequence, gather by the same mempool
- */
-struct igb_tx_entry_seq {
-	const struct rte_mempool* pool;
-	uint32_t same_pool;
-};
-
-/**
  * Structure associated with each RX queue.
  */
 struct igb_rx_queue {
@@ -191,10 +183,6 @@ struct igb_tx_queue {
 	volatile union ixgbe_adv_tx_desc *tx_ring;
 	uint64_t            tx_ring_phys_addr; /**< TX ring DMA address. */
 	struct igb_tx_entry *sw_ring;      /**< virtual address of SW ring. */
-#ifdef RTE_IXGBE_INC_VECTOR
-	/** continuous tx entry sequence within the same mempool */
-	struct igb_tx_entry_seq *sw_ring_seq;
-#endif
 	volatile uint32_t   *tdt_reg_addr; /**< Address of TDT register. */
 	uint16_t            nb_tx_desc;    /**< number of TX descriptors. */
 	uint16_t            tx_tail;       /**< current value of TDT reg. */
@@ -259,7 +247,7 @@ struct ixgbe_txq_ops {
 #ifdef RTE_IXGBE_INC_VECTOR
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
-int ixgbe_txq_vec_setup(struct igb_tx_queue *txq, unsigned int socket_id);
+int ixgbe_txq_vec_setup(struct igb_tx_queue *txq);
 int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
 #endif
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +35 -58

188 lines not on this page · raw message

Signature · 1 line
1.9.3
#74 lore raw

PATCH 13/13 ixgbe: Improve slow-path perf: vector scattered RX

In reply to a message not in the archive

Provide a wrapper routine to enable receive of scattered packets with a vector driver.

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |  16 ++++
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     |   1 +
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 173 ++++++++++++++++++++++++++++++++--
 3 files changed, 183 insertions(+), 7 deletions(-)
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +16 -0
index ebbcee8..8a2e0ee 100644
@@ -3477,12 +3477,20 @@ ixgbe_dev_rx_init(struct rte_eth_dev *dev)
 		if ((dev->data->dev_conf.rxmode.max_rx_pkt_len +
 				2 * IXGBE_VLAN_TAG_SIZE) > buf_size){
 			dev->data->scattered_rx = 1;
+#ifdef RTE_IXGBE_INC_VECTOR
+			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		}
 	}
 
 	if (dev->data->dev_conf.rxmode.enable_scatter) {
+#ifdef RTE_IXGBE_INC_VECTOR
+		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		dev->data->scattered_rx = 1;
 	}
 
@@ -3970,12 +3978,20 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
 		if ((dev->data->dev_conf.rxmode.max_rx_pkt_len +
 				2 * IXGBE_VLAN_TAG_SIZE) > buf_size) {
 			dev->data->scattered_rx = 1;
+#ifdef RTE_IXGBE_INC_VECTOR
+			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		}
 	}
 
 	if (dev->data->dev_conf.rxmode.enable_scatter) {
+#ifdef RTE_IXGBE_INC_VECTOR
+		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		dev->data->scattered_rx = 1;
 	}
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -0
index dbb57af..e2e037b 100644
@@ -246,6 +246,7 @@ struct ixgbe_txq_ops {
 
 #ifdef RTE_IXGBE_INC_VECTOR
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+uint16_t ixgbe_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct igb_tx_queue *txq);
 int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +166 -7

223 lines not on this page · raw message

Signature · 1 line
1.9.3
#75 lore raw

v2 cover Mbuf Structure Rework, part 2

In reply to a message not in the archive

This patch set continues on from the changes in part 1, and depends upon that patch set.

This patch set reorders the fields in the mbuf structure and splits the structure across two cache lines, given lots of new space for new fields to be added. This set uses some of that space by expanding the ol_flags field. A part 3 patchset is planned to introduce some other new fields into the new mbuf structure.

With the splitting of the mbuf across multiple cache lines, performance degradations are seen inside the drivers, both fast-path and slow path. For the fast-path, this patchset reworks the way in which the pool pointer is used to free packets post-TX, which removes the perf regression. For the slow path, an alternative approach is taken - a new scattered packets RX function is introduced into the vector PMD. Using this function, throughput for the slow path RX-TX using testpmd is increased by up to 20% over the original baseline.

Changes in V2:
* General minor updates follow comments on V1 set
* Updated a number of patches to include KNI mbuf changes where needed
* Deferred the patch to add the packet type field to a future patch set
* After removing meta-data element from mbuf structure also added in patch to move the macros to rte_port/rte_port.h

Bruce Richardson (11):
mbuf: reorder fields by time of use
mbuf: expand ol_flags field to 64-bits
mbuf: introduce a flag to indicate a control mbuf
mbuf: minor changes for readability
mbuf: use macros only to access the mbuf metadata
mbuf: add named points inside the mbuf structure
ixgbe: rework vector pmd following mbuf changes
mbuf: split mbuf across two cache lines.
mbuf: move l2_len and l3_len to second cache line
ixgbe: Fix perf regression due to moved pool ptr
ixgbe: Improve slow-path perf: vector scattered RX

Olivier Matz (1):
mbuf: replace data pointer by an offset

 app/test-pmd/config.c                              |   8 +-
 app/test-pmd/csumonly.c                            |  10 +-
 app/test-pmd/flowgen.c                             |   2 +-
 app/test-pmd/icmpecho.c                            |   2 +-
 app/test-pmd/ieee1588fwd.c                         |   4 +-
 app/test-pmd/macfwd-retry.c                        |   2 +-
 app/test-pmd/macfwd.c                              |   2 +-
 app/test-pmd/macswap.c                             |   2 +-
 app/test-pmd/rxonly.c                              |   4 +-
 app/test-pmd/testpmd.c                             |   2 +-
 app/test-pmd/testpmd.h                             |   4 +-
 app/test-pmd/txonly.c                              |   9 +-
 app/test/packet_burst_generator.c                  |   7 +-
 app/test/test_mbuf.c                               |   8 +-
 app/test/test_table_acl.c                          |   7 +-
 app/test/test_table_pipeline.c                     |   8 +-
 examples/exception_path/main.c                     |   3 +-
 examples/vhost/main.c                              |  37 +-
 examples/vhost_xen/main.c                          |  14 +-
 .../linuxapp/eal/include/exec-env/rte_kni_common.h |  20 +-
 lib/librte_eal/linuxapp/kni/kni_net.c              |  18 +-
 lib/librte_ip_frag/rte_ipv4_fragmentation.c        |   6 +-
 lib/librte_ip_frag/rte_ipv6_fragmentation.c        |   6 +-
 lib/librte_mbuf/rte_mbuf.c                         |  10 +-
 lib/librte_mbuf/rte_mbuf.h                         | 139 ++++----
 lib/librte_pmd_bond/rte_eth_bond_pmd.c             |   4 +-
 lib/librte_pmd_e1000/em_rxtx.c                     |  46 ++-
 lib/librte_pmd_e1000/igb_rxtx.c                    |  75 ++--
 lib/librte_pmd_i40e/i40e_rxtx.c                    |  48 +--
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c                  | 100 +++---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h                  |  24 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c              | 389 +++++++++++++--------
 lib/librte_pmd_pcap/rte_eth_pcap.c                 |   9 +-
 lib/librte_pmd_virtio/virtio_rxtx.c                |  10 +-
 lib/librte_pmd_virtio/virtqueue.h                  |   3 +-
 lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c              |   5 +-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c           |   2 +-
 37 files changed, 586 insertions(+), 463 deletions(-)
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1410441347-22840-1-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#77 lore raw

PATCH v2 1/13 mbuf: replace data pointer by an offset

From: Olivier Matz <[email protected]>

Original patch:
The mbuf structure already contains a pointer to the beginning of the buffer (m->buf_addr). It is not needed to use 8 bytes again to store another pointer to the beginning of the data.

Using a 16 bits unsigned integer is enough as we know that a mbuf is never longer than 64KB. We gain 6 bytes in the structure thanks to this modification.

This version:
* Updated original patch to apply to latest on mainline.
* Disabled vector PMD in config as it relies heavily on the mbuf layout This will be re-enabled in a subsequent commit once vPMD has been reworked to take account of mbuf changes.

Updated in V2:
* Updated the KNI buffer structure to match that of new mbuf
* fixed checkpatch errors.

---
 app/test-pmd/csumonly.c                            |  2 +-
 app/test-pmd/flowgen.c                             |  2 +-
 app/test-pmd/icmpecho.c                            |  2 +-
 app/test-pmd/ieee1588fwd.c                         |  4 +--
 app/test-pmd/macfwd-retry.c                        |  2 +-
 app/test-pmd/macfwd.c                              |  2 +-
 app/test-pmd/macswap.c                             |  2 +-
 app/test-pmd/rxonly.c                              |  2 +-
 app/test-pmd/testpmd.c                             |  2 +-
 app/test-pmd/txonly.c                              |  7 ++--
 app/test/packet_burst_generator.c                  |  7 ++--
 app/test/test_mbuf.c                               |  6 ++--
 app/test/test_table_acl.c                          |  7 ++--
 app/test/test_table_pipeline.c                     |  8 ++---
 config/common_linuxapp                             |  2 +-
 examples/exception_path/main.c                     |  3 +-
 examples/vhost/main.c                              | 37 ++++++++++---------
 examples/vhost_xen/main.c                          | 14 ++++----
 .../linuxapp/eal/include/exec-env/rte_kni_common.h | 10 +++---
 lib/librte_eal/linuxapp/kni/kni_net.c              | 18 ++++++----
 lib/librte_ip_frag/rte_ipv4_fragmentation.c        |  6 ++--
 lib/librte_ip_frag/rte_ipv6_fragmentation.c        |  6 ++--
 lib/librte_mbuf/rte_mbuf.c                         |  6 ++--
 lib/librte_mbuf/rte_mbuf.h                         | 42 ++++++++++------------
 lib/librte_pmd_bond/rte_eth_bond_pmd.c             |  4 +--
 lib/librte_pmd_e1000/em_rxtx.c                     | 12 +++----
 lib/librte_pmd_e1000/igb_rxtx.c                    | 13 ++++---
 lib/librte_pmd_i40e/i40e_rxtx.c                    | 17 +++++----
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c                  | 13 +++----
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h                  |  3 +-
 lib/librte_pmd_pcap/rte_eth_pcap.c                 |  9 +++--
 lib/librte_pmd_virtio/virtio_rxtx.c                | 10 +++---
 lib/librte_pmd_virtio/virtqueue.h                  |  3 +-
 lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c              |  5 ++-
 lib/librte_pmd_xenvirt/rte_eth_xenvirt.c           |  2 +-
 35 files changed, 146 insertions(+), 144 deletions(-)
app/test-pmd/csumonly.c +1 -1
index 28b66f5..2ce4c42 100644
@@ -263,7 +263,7 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 		pkt_ol_flags = mb->ol_flags;
 		ol_flags = (uint16_t) (pkt_ol_flags & (~PKT_TX_L4_MASK));
 
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
 		if (eth_type == ETHER_TYPE_VLAN) {
 			/* Only allow single VLAN label here */
app/test-pmd/flowgen.c +1 -1
index b091b6d..8b4ed9a 100644
@@ -175,7 +175,7 @@ pkt_burst_flow_gen(struct fwd_stream *fs)
 		pkt->next = NULL;
 
 		/* Initialize Ethernet header. */
-		eth_hdr = (struct ether_hdr *)pkt->data;
+		eth_hdr = rte_pktmbuf_mtod(pkt, struct ether_hdr *);
 		ether_addr_copy(&cfg_ether_dst, &eth_hdr->d_addr);
 		ether_addr_copy(&cfg_ether_src, &eth_hdr->s_addr);
 		eth_hdr->ether_type = rte_cpu_to_be_16(ETHER_TYPE_IPv4);
app/test-pmd/icmpecho.c +1 -1
index 4a277b8..7fd4b6d 100644
@@ -330,7 +330,7 @@ reply_to_icmp_echo_rqsts(struct fwd_stream *fs)
 	nb_replies = 0;
 	for (i = 0; i < nb_rx; i++) {
 		pkt = pkts_burst[i];
-		eth_h = (struct ether_hdr *) pkt->data;
+		eth_h = rte_pktmbuf_mtod(pkt, struct ether_hdr *);
 		eth_type = RTE_BE_TO_CPU_16(eth_h->ether_type);
 		l2_len = sizeof(struct ether_hdr);
 		if (verbose_level > 0) {
app/test-pmd/ieee1588fwd.c +2 -2
index ab5e06e..976aa28 100644
@@ -546,7 +546,7 @@ ieee1588_packet_fwd(struct fwd_stream *fs)
 	 * Check that the received packet is a PTP packet that was detected
 	 * by the hardware.
 	 */
-	eth_hdr = (struct ether_hdr *)mb->data;
+	eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 	eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
 	if (! (mb->ol_flags & PKT_RX_IEEE1588_PTP)) {
 		if (eth_type == ETHER_TYPE_1588) {
@@ -574,7 +574,7 @@ ieee1588_packet_fwd(struct fwd_stream *fs)
 	 * Check that the received PTP packet is a PTP V2 packet of type
 	 * PTP_SYNC_MESSAGE.
 	 */
-	ptp_hdr = (struct ptpv2_msg *) ((char *) mb->data +
+	ptp_hdr = (struct ptpv2_msg *) (rte_pktmbuf_mtod(mb, char *) +
 					sizeof(struct ether_hdr));
 	if (ptp_hdr->version != 0x02) {
 		printf("Port %u Received PTP V2 Ethernet frame with wrong PTP"
app/test-pmd/macfwd-retry.c +1 -1
index 5122983..83da26f 100644
@@ -119,7 +119,7 @@ pkt_burst_mac_retry_forward(struct fwd_stream *fs)
 	fs->rx_packets += nb_rx;
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		ether_addr_copy(&peer_eth_addrs[fs->peer_addr],
 				&eth_hdr->d_addr);
 		ether_addr_copy(&ports[fs->tx_port].eth_addr,
app/test-pmd/macfwd.c +1 -1
index 4b905cd..38bae23 100644
@@ -110,7 +110,7 @@ pkt_burst_mac_forward(struct fwd_stream *fs)
 	txp = &ports[fs->tx_port];
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		ether_addr_copy(&peer_eth_addrs[fs->peer_addr],
 				&eth_hdr->d_addr);
 		ether_addr_copy(&ports[fs->tx_port].eth_addr,
app/test-pmd/macswap.c +1 -1
index c5b3a0c..1786095 100644
@@ -110,7 +110,7 @@ pkt_burst_mac_swap(struct fwd_stream *fs)
 	txp = &ports[fs->tx_port];
 	for (i = 0; i < nb_rx; i++) {
 		mb = pkts_burst[i];
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 
 		/* Swap dest and src mac addresses. */
 		ether_addr_copy(&eth_hdr->d_addr, &addr);
app/test-pmd/rxonly.c +1 -1
index 5bc74da..7ba36a1 100644
@@ -149,7 +149,7 @@ pkt_burst_receive(struct fwd_stream *fs)
 			rte_pktmbuf_free(mb);
 			continue;
 		}
-		eth_hdr = (struct ether_hdr *) mb->data;
+		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = RTE_BE_TO_CPU_16(eth_hdr->ether_type);
 		ol_flags = mb->ol_flags;
 		print_ether_addr("  src=", &eth_hdr->s_addr);
app/test-pmd/testpmd.c +1 -1
index b426dfc..9f6cdc4 100644
@@ -404,7 +404,7 @@ testpmd_mbuf_ctor(struct rte_mempool *mp,
 			mb_ctor_arg->seg_buf_offset);
 	mb->buf_len      = mb_ctor_arg->seg_buf_size;
 	mb->ol_flags     = 0;
-	mb->data         = (char *) mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+	mb->data_off     = RTE_PKTMBUF_HEADROOM;
 	mb->nb_segs      = 1;
 	mb->l2_l3_len       = 0;
 	mb->vlan_tci     = 0;
app/test-pmd/txonly.c +4 -3
index 8135264..2b3f0b9 100644
@@ -111,13 +111,13 @@ copy_buf_to_pkt_segs(void* buf, unsigned len, struct rte_mbuf *pkt,
 		seg = seg->next;
 	}
 	copy_len = seg->data_len - offset;
-	seg_buf = ((char *) seg->data + offset);
+	seg_buf = (rte_pktmbuf_mtod(seg, char *) + offset);
 	while (len > copy_len) {
 		rte_memcpy(seg_buf, buf, (size_t) copy_len);
 		len -= copy_len;
 		buf = ((char*) buf + copy_len);
 		seg = seg->next;
-		seg_buf = seg->data;
+		seg_buf = rte_pktmbuf_mtod(seg, char *);
 	}
 	rte_memcpy(seg_buf, buf, (size_t) len);
 }
@@ -126,7 +126,8 @@ static inline void
 copy_buf_to_pkt(void* buf, unsigned len, struct rte_mbuf *pkt, unsigned offset)
 {
 	if (offset + len <= pkt->data_len) {
-		rte_memcpy(((char *) pkt->data + offset), buf, (size_t) len);
+		rte_memcpy((rte_pktmbuf_mtod(pkt, char *) + offset),
+			buf, (size_t) len);
 		return;
 	}
 	copy_buf_to_pkt_segs(buf, len, pkt, offset);
app/test/packet_burst_generator.c +4 -3
index db7f023..9e747a4 100644
@@ -59,13 +59,13 @@ copy_buf_to_pkt_segs(void *buf, unsigned len, struct rte_mbuf *pkt,
 		seg = seg->next;
 	}
 	copy_len = seg->data_len - offset;
-	seg_buf = ((char *) seg->data + offset);
+	seg_buf = rte_pktmbuf_mtod(seg, char *) + offset;
 	while (len > copy_len) {
 		rte_memcpy(seg_buf, buf, (size_t) copy_len);
 		len -= copy_len;
 		buf = ((char *) buf + copy_len);
 		seg = seg->next;
-		seg_buf = seg->data;
+		seg_buf = rte_pktmbuf_mtod(seg, void *);
 	}
 	rte_memcpy(seg_buf, buf, (size_t) len);
 }
@@ -74,7 +74,8 @@ static inline void
 copy_buf_to_pkt(void *buf, unsigned len, struct rte_mbuf *pkt, unsigned offset)
 {
 	if (offset + len <= pkt->data_len) {
-		rte_memcpy(((char *) pkt->data + offset), buf, (size_t) len);
+		rte_memcpy(rte_pktmbuf_mtod(pkt, char *) + offset,
+				buf, (size_t) len);
 		return;
 	}
 	copy_buf_to_pkt_segs(buf, len, pkt, offset);
app/test/test_mbuf.c +3 -3
index b81e622..1b25481 100644
@@ -432,7 +432,7 @@ test_pktmbuf_pool_ptr(void)
 			printf("rte_pktmbuf_alloc() failed (%u)\n", i);
 			ret = -1;
 		}
-		m[i]->data = RTE_PTR_ADD(m[i]->data, 64);
+		m[i]->data_off += 64;
 	}
 
 	/* free them */
@@ -451,8 +451,8 @@ test_pktmbuf_pool_ptr(void)
 			printf("rte_pktmbuf_alloc() failed (%u)\n", i);
 			ret = -1;
 		}
-		if (m[i]->data != RTE_PTR_ADD(m[i]->buf_addr, RTE_PKTMBUF_HEADROOM)) {
-			printf ("data pointer not set properly\n");
+		if (m[i]->data_off != RTE_PKTMBUF_HEADROOM) {
+			printf("invalid data_off\n");
 			ret = -1;
 		}
 	}
app/test/test_table_acl.c +4 -3
index 4db680a..0f2b57e 100644
@@ -513,7 +513,7 @@ test_pipeline_single_filter(int expected_count)
 			struct rte_mbuf *mbuf;
 
 			mbuf = rte_pktmbuf_alloc(pool);
-			memset(mbuf->data, 0x00,
+			memset(rte_pktmbuf_mtod(mbuf, char *), 0x00,
 				sizeof(struct ipv4_5tuple));
 
 			five_tuple.proto = j;
@@ -522,7 +522,7 @@ test_pipeline_single_filter(int expected_count)
 			five_tuple.port_src = rte_bswap16(100 + j);
 			five_tuple.port_dst = rte_bswap16(200 + j);
 
-			memcpy(mbuf->data, &five_tuple,
+			memcpy(rte_pktmbuf_mtod(mbuf, char *), &five_tuple,
 				sizeof(struct ipv4_5tuple));
 			RTE_LOG(INFO, PIPELINE, "%s: Enqueue onto ring %d\n",
 				__func__, i);
@@ -549,7 +549,8 @@ test_pipeline_single_filter(int expected_count)
 			printf("Got %d object(s) from ring %d!\n", ret, i);
 			for (j = 0; j < ret; j++) {
 				mbuf = (struct rte_mbuf *)objs[j];
-				rte_hexdump(stdout, "mbuf", mbuf->data, 64);
+				rte_hexdump(stdout, "mbuf",
+					rte_pktmbuf_mtod(mbuf, char *), 64);
 				rte_pktmbuf_free(mbuf);
 			}
 			tx_count += ret;
app/test/test_table_pipeline.c +2 -6
index 15a038b..a0a9e04 100644
@@ -39,11 +39,6 @@
 #include "test_table.h"
 #include "test_table_pipeline.h"
 
-#define RTE_CBUF_UINT8_PTR(cbuf, offset)			\
-	(&cbuf->data[offset])
-#define RTE_CBUF_UINT32_PTR(cbuf, offset)			\
-	(&cbuf->data32[offset/sizeof(uint32_t)])
-
 #if 0
 
 static rte_pipeline_port_out_action_handler port_action_0x00
@@ -498,7 +493,8 @@ test_pipeline_single_filter(int test_type, int expected_count)
 			printf("Got %d object(s) from ring %d!\n", ret, i);
 			for (j = 0; j < ret; j++) {
 				mbuf = (struct rte_mbuf *)objs[j];
-				rte_hexdump(stdout, "Object:", mbuf->data,
+				rte_hexdump(stdout, "Object:",
+					rte_pktmbuf_mtod(mbuf, char *),
 					mbuf->data_len);
 				rte_pktmbuf_free(mbuf);
 			}
config/common_linuxapp +1 -1
index 5bee910..b140af7 100644
@@ -191,7 +191,7 @@ CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
 CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
 CONFIG_RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC=y
 CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=n
-CONFIG_RTE_IXGBE_INC_VECTOR=y
+CONFIG_RTE_IXGBE_INC_VECTOR=n
 CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
 
 #
examples/exception_path/main.c +2 -1
index 5045ef8..f286bf2 100644
@@ -302,7 +302,8 @@ main_loop(__attribute__((unused)) void *arg)
 			if (m == NULL)
 				continue;
 
-			ret = read(tap_fd, m->data, MAX_PACKET_SZ);
+			ret = read(tap_fd, rte_pktmbuf_mtod(m, void *),
+				MAX_PACKET_SZ);
 			lcore_stats[lcore_id].rx++;
 			if (unlikely(ret < 0)) {
 				FATAL_ERROR("Reading from %s interface failed",
examples/vhost/main.c +20 -17
index 4e1c103..85ee8b8 100644
@@ -1033,7 +1033,7 @@ virtio_dev_rx(struct virtio_net *dev, struct rte_mbuf **pkts, uint32_t count)
 
 		/* Copy mbuf data to buffer */
 		rte_memcpy((void *)(uintptr_t)buff_addr,
-			(const void *)buff->data,
+			rte_pktmbuf_mtod(buff, const void *),
 			rte_pktmbuf_data_len(buff));
 		PRINT_PACKET(dev, (uintptr_t)buff_addr,
 			rte_pktmbuf_data_len(buff), 0);
@@ -1438,7 +1438,7 @@ link_vmdq(struct virtio_net *dev, struct rte_mbuf *m)
 	int i, ret;
 
 	/* Learn MAC address of guest device from packet */
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	dev_ll = ll_root_used;
 
@@ -1525,7 +1525,7 @@ virtio_tx_local(struct virtio_net *dev, struct rte_mbuf *m)
 	struct ether_hdr *pkt_hdr;
 	uint64_t ret = 0;
 
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*get the used devices list*/
 	dev_ll = ll_root_used;
@@ -1593,7 +1593,7 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	unsigned len, ret, offset = 0;
 	const uint16_t lcore_id = rte_lcore_id();
 	struct virtio_net_data_ll *dev_ll = ll_root_used;
-	struct ether_hdr *pkt_hdr = (struct ether_hdr *)m->data;
+	struct ether_hdr *pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*check if destination is local VM*/
 	if ((vm2vm_mode == VM2VM_SOFTWARE) && (virtio_tx_local(dev, m) == 0))
@@ -1652,18 +1652,21 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	mbuf->nb_segs = m->nb_segs;
 
 	/* Copy ethernet header to mbuf. */
-	rte_memcpy((void*)mbuf->data, (const void*)m->data, ETH_HLEN);
+	rte_memcpy(rte_pktmbuf_mtod(mbuf, void *),
+		rte_pktmbuf_mtod(m, const void *),
+		ETH_HLEN);
 
 
 	/* Setup vlan header. Bytes need to be re-ordered for network with htons()*/
-	vlan_hdr = (struct vlan_ethhdr *) mbuf->data;
+	vlan_hdr = rte_pktmbuf_mtod(mbuf, struct vlan_ethhdr *);
 	vlan_hdr->h_vlan_encapsulated_proto = vlan_hdr->h_vlan_proto;
 	vlan_hdr->h_vlan_proto = htons(ETH_P_8021Q);
 	vlan_hdr->h_vlan_TCI = htons(vlan_tag);
 
 	/* Copy the remaining packet contents to the mbuf. */
-	rte_memcpy((void*) ((uint8_t*)mbuf->data + VLAN_ETH_HLEN),
-		(const void*) ((uint8_t*)m->data + ETH_HLEN), (m->data_len - ETH_HLEN));
+	rte_memcpy((void *)(rte_pktmbuf_mtod(mbuf, uint8_t *) + VLAN_ETH_HLEN),
+		(const void *)(rte_pktmbuf_mtod(m, uint8_t *) + ETH_HLEN),
+		(m->data_len - ETH_HLEN));
 
 	/* Copy the remaining segments for the whole packet. */
 	prev = mbuf;
@@ -1778,7 +1781,7 @@ virtio_dev_tx(struct virtio_net* dev, struct rte_mempool *mbuf_pool)
 		/* Setup dummy mbuf. This is copied to a real mbuf if transmitted out the physical port. */
 		m.data_len = desc->len;
 		m.pkt_len = desc->len;
-		m.data = (void*)(uintptr_t)buff_addr;
+		m.data_off = 0;
 
 		PRINT_PACKET(dev, (uintptr_t)buff_addr, desc->len, 0);
 
@@ -2333,7 +2336,7 @@ attach_rxmbuf_zcp(struct virtio_net *dev)
 	}
 
 	mbuf->buf_addr = (void *)(uintptr_t)(buff_addr - RTE_PKTMBUF_HEADROOM);
-	mbuf->data = (void *)(uintptr_t)(buff_addr);
+	mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 	mbuf->buf_physaddr = phys_addr - RTE_PKTMBUF_HEADROOM;
 	mbuf->data_len = desc->len;
 	MBUF_HEADROOM_UINT32(mbuf) = (uint32_t)desc_idx;
@@ -2370,7 +2373,7 @@ static inline void pktmbuf_detach_zcp(struct rte_mbuf *m)
 
 	buf_ofs = (RTE_PKTMBUF_HEADROOM <= m->buf_len) ?
 			RTE_PKTMBUF_HEADROOM : m->buf_len;
-	m->data = (char *) m->buf_addr + buf_ofs;
+	m->data_off = buf_ofs;
 
 	m->data_len = 0;
 }
@@ -2604,7 +2607,7 @@ virtio_tx_route_zcp(struct virtio_net *dev, struct rte_mbuf *m,
 	unsigned len, ret, offset = 0;
 	struct vpool *vpool;
 	struct virtio_net_data_ll *dev_ll = ll_root_used;
-	struct ether_hdr *pkt_hdr = (struct ether_hdr *)m->data;
+	struct ether_hdr *pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 	uint16_t vlan_tag = (uint16_t)vlan_tags[(uint16_t)dev->device_fh];
 
 	/*Add packet to the port tx queue*/
@@ -2681,11 +2684,11 @@ virtio_tx_route_zcp(struct virtio_net *dev, struct rte_mbuf *m,
 	mbuf->pkt_len = mbuf->data_len;
 	if (unlikely(need_copy)) {
 		/* Copy the packet contents to the mbuf. */
-		rte_memcpy((void *)((uint8_t *)mbuf->data),
-			(const void *) ((uint8_t *)m->data),
+		rte_memcpy(rte_pktmbuf_mtod(mbuf, void *),
+			rte_pktmbuf_mtod(m, void *),
 			m->data_len);
 	} else {
-		mbuf->data = m->data;
+		mbuf->data_off = m->data_off;
 		mbuf->buf_physaddr = m->buf_physaddr;
 		mbuf->buf_addr = m->buf_addr;
 	}
@@ -2819,8 +2822,8 @@ virtio_dev_tx_zcp(struct virtio_net *dev)
 		m.data_len = desc->len;
 		m.nb_segs = 1;
 		m.next = NULL;
-		m.data = (void *)(uintptr_t)buff_addr;
-		m.buf_addr = m.data;
+		m.data_off = 0;
+		m.buf_addr = (void *)(uintptr_t)buff_addr;
 		m.buf_physaddr = phys_addr;
 
 		/*
examples/vhost_xen/main.c +8 -6
index 8162cd8..56ffec8 100644
@@ -808,7 +808,7 @@ virtio_tx_local(struct virtio_net *dev, struct rte_mbuf *m)
 	struct ether_hdr *pkt_hdr;
 	uint64_t ret = 0;
 
-	pkt_hdr = (struct ether_hdr *)m->data;
+	pkt_hdr = rte_pktmbuf_mtod(m, struct ether_hdr *);
 
 	/*get the used devices list*/
 	dev_ll = ll_root_used;
@@ -883,18 +883,20 @@ virtio_tx_route(struct virtio_net* dev, struct rte_mbuf *m, struct rte_mempool *
 	mbuf->pkt_len = mbuf->data_len;
 
 	/* Copy ethernet header to mbuf. */
-	rte_memcpy((void*)mbuf->data, (const void*)m->data, ETH_HLEN);
+	rte_memcpy(rte_pktmbuf_mtod(mbuf, void*),
+			rte_pktmbuf_mtod(m, const void*), ETH_HLEN);
 
 
 	/* Setup vlan header. Bytes need to be re-ordered for network with htons()*/
-	vlan_hdr = (struct vlan_ethhdr *) mbuf->data;
+	vlan_hdr = rte_pktmbuf_mtod(mbuf, struct vlan_ethhdr *);
 	vlan_hdr->h_vlan_encapsulated_proto = vlan_hdr->h_vlan_proto;
 	vlan_hdr->h_vlan_proto = htons(ETH_P_8021Q);
 	vlan_hdr->h_vlan_TCI = htons(vlan_tag);
 
 	/* Copy the remaining packet contents to the mbuf. */
-	rte_memcpy((void*) ((uint8_t*)mbuf->data + VLAN_ETH_HLEN),
-		(const void*) ((uint8_t*)m->data + ETH_HLEN), (m->data_len - ETH_HLEN));
+	rte_memcpy((void *)(rte_pktmbuf_mtod(mbuf, uint8_t *) + VLAN_ETH_HLEN),
+		(const void *)(rte_pktmbuf_mtod(m, uint8_t *) + ETH_HLEN),
+		(m->data_len - ETH_HLEN));
 	tx_q->m_table[len] = mbuf;
 	len++;
 	if (enable_stats) {
@@ -981,7 +983,7 @@ virtio_dev_tx(struct virtio_net* dev, struct rte_mempool *mbuf_pool)
 
 		/* Setup dummy mbuf. This is copied to a real mbuf if transmitted out the physical port. */
 		m.data_len = desc->len;
-		m.data = (void*)(uintptr_t)buff_addr;
+		m.data_off = 0;
 		m.nb_segs = 1;
 
 		virtio_tx_route(dev, &m, mbuf_pool, 0);
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +5 -5
index d0b82da..07908ac 100644
@@ -110,13 +110,13 @@ struct rte_kni_fifo {
 struct rte_kni_mbuf {
 	void *pool;
 	void *buf_addr;
-	char pad0[14];
-	uint16_t ol_flags;      /**< Offload features. */
+	char pad0[16];
 	void *next;
-	void *data;             /**< Start address of data in segment buffer. */
+	uint16_t data_off;      /**< Start address of data in segment buffer. */
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
-	char pad2[2];
-	uint16_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
+	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
+	char pad2[4];
+	uint16_t ol_flags;      /**< Offload features. */
 } __attribute__((__aligned__(64)));
 
 /*
lib/librte_eal/linuxapp/kni/kni_net.c +11 -7
index fa5fd3e..dd95db5 100644
@@ -162,7 +162,8 @@ kni_net_rx_normal(struct kni_dev *kni)
 	for (i = 0; i < num; i++) {
 		kva = (void *)va[i] - kni->mbuf_va + kni->mbuf_kva;
 		len = kva->data_len;
-		data_kva = kva->data - kni->mbuf_va + kni->mbuf_kva;
+		data_kva = kva->buf_addr + kva->data_off - kni->mbuf_va
+				+ kni->mbuf_kva;
 
 		skb = dev_alloc_skb(len + 2);
 		if (!skb) {
@@ -246,13 +247,14 @@ kni_net_rx_lo_fifo(struct kni_dev *kni)
 		for (i = 0; i < num; i++) {
 			kva = (void *)va[i] - kni->mbuf_va + kni->mbuf_kva;
 			len = kva->pkt_len;
-			data_kva = kva->data - kni->mbuf_va +
-						kni->mbuf_kva;
+			data_kva = kva->buf_addr + kva->data_off -
+					kni->mbuf_va + kni->mbuf_kva;
 
 			alloc_kva = (void *)alloc_va[i] - kni->mbuf_va +
 							kni->mbuf_kva;
-			alloc_data_kva = alloc_kva->data - kni->mbuf_va +
-								kni->mbuf_kva;
+			alloc_data_kva = alloc_kva->buf_addr +
+					alloc_kva->data_off - kni->mbuf_va +
+							kni->mbuf_kva;
 			memcpy(alloc_data_kva, data_kva, len);
 			alloc_kva->pkt_len = len;
 			alloc_kva->data_len = len;
@@ -321,7 +323,8 @@ kni_net_rx_lo_fifo_skb(struct kni_dev *kni)
 	for (i = 0; i < num; i++) {
 		kva = (void *)va[i] - kni->mbuf_va + kni->mbuf_kva;
 		len = kva->data_len;
-		data_kva = kva->data - kni->mbuf_va + kni->mbuf_kva;
+		data_kva = kva->buf_addr + kva->data_off - kni->mbuf_va +
+				kni->mbuf_kva;
 
 		skb = dev_alloc_skb(len + 2);
 		if (skb == NULL)
@@ -423,7 +426,8 @@ kni_net_tx(struct sk_buff *skb, struct net_device *dev)
 		void *data_kva;
 
 		pkt_kva = (void *)pkt_va - kni->mbuf_va + kni->mbuf_kva;
-		data_kva = pkt_kva->data - kni->mbuf_va + kni->mbuf_kva;
+		data_kva = pkt_kva->buf_addr + pkt_kva->data_off - kni->mbuf_va
+				+ kni->mbuf_kva;
 
 		len = skb->len;
 		memcpy(data_kva, skb->data, len);
lib/librte_ip_frag/rte_ipv4_fragmentation.c +3 -3
index 6b9f07d..a4ed923 100644
@@ -109,7 +109,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 	/* Fragment size should be a multiply of 8. */
 	IP_FRAG_ASSERT((frag_size & IPV4_HDR_FO_MASK) == 0);
 
-	in_hdr = (struct ipv4_hdr *) pkt_in->data;
+	in_hdr = rte_pktmbuf_mtod(pkt_in, struct ipv4_hdr *);
 	flag_offset = rte_cpu_to_be_16(in_hdr->fragment_offset);
 
 	/* If Don't Fragment flag is set */
@@ -165,7 +165,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 			if (len > (in_seg->data_len - in_seg_data_pos)) {
 				len = in_seg->data_len - in_seg_data_pos;
 			}
-			out_seg->data = (char*) in_seg->data + (uint16_t)in_seg_data_pos;
+			out_seg->data_off = in_seg->data_off + in_seg_data_pos;
 			out_seg->data_len = (uint16_t)len;
 			out_pkt->pkt_len = (uint16_t)(len +
 			    out_pkt->pkt_len);
@@ -188,7 +188,7 @@ rte_ipv4_fragment_packet(struct rte_mbuf *pkt_in,
 
 		/* Build the IP header */
 
-		out_hdr = (struct ipv4_hdr*) out_pkt->data;
+		out_hdr = rte_pktmbuf_mtod(out_pkt, struct ipv4_hdr *);
 
 		__fill_ipv4hdr_frag(out_hdr, in_hdr,
 		    (uint16_t)out_pkt->pkt_len,
lib/librte_ip_frag/rte_ipv6_fragmentation.c +3 -3
index e007662..4ffcc7c 100644
@@ -125,7 +125,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 	    (uint16_t)(pkt_in->pkt_len - sizeof (struct ipv6_hdr))))
 		return (-EINVAL);
 
-	in_hdr = (struct ipv6_hdr *) pkt_in->data;
+	in_hdr = rte_pktmbuf_mtod(pkt_in, struct ipv6_hdr *);
 
 	in_seg = pkt_in;
 	in_seg_data_pos = sizeof(struct ipv6_hdr);
@@ -171,7 +171,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 			if (len > (in_seg->data_len - in_seg_data_pos)) {
 				len = in_seg->data_len - in_seg_data_pos;
 			}
-			out_seg->data = (char *) in_seg->data + (uint16_t) in_seg_data_pos;
+			out_seg->data_off = in_seg->data_off + in_seg_data_pos;
 			out_seg->data_len = (uint16_t)len;
 			out_pkt->pkt_len = (uint16_t)(len +
 			    out_pkt->pkt_len);
@@ -196,7 +196,7 @@ rte_ipv6_fragment_packet(struct rte_mbuf *pkt_in,
 
 		/* Build the IP header */
 
-		out_hdr = (struct ipv6_hdr *) out_pkt->data;
+		out_hdr = rte_pktmbuf_mtod(out_pkt, struct ipv6_hdr *);
 
 		__fill_ipv6hdr_frag(out_hdr, in_hdr,
 		    (uint16_t) out_pkt->pkt_len - sizeof(struct ipv6_hdr),
lib/librte_mbuf/rte_mbuf.c +3 -3
index c1b2176..26e36eb 100644
@@ -117,7 +117,7 @@ rte_pktmbuf_init(struct rte_mempool *mp,
 	m->buf_len = (uint16_t)buf_len;
 
 	/* keep some headroom between start of buffer and data */
-	m->data = (char*) m->buf_addr + RTE_MIN(RTE_PKTMBUF_HEADROOM, m->buf_len);
+	m->data_off = RTE_MIN(RTE_PKTMBUF_HEADROOM, (uint16_t)m->buf_len);
 
 	/* init some constant fields */
 	m->pool = mp;
@@ -183,12 +183,12 @@ rte_pktmbuf_dump(FILE *f, const struct rte_mbuf *m, unsigned dump_len)
 		__rte_mbuf_sanity_check(m, 0);
 
 		fprintf(f, "  segment at 0x%p, data=0x%p, data_len=%u\n",
-		       m, m->data, (unsigned)m->data_len);
+			m, rte_pktmbuf_mtod(m, void *), (unsigned)m->data_len);
 		len = dump_len;
 		if (len > m->data_len)
 			len = m->data_len;
 		if (len != 0)
-			rte_hexdump(f, NULL, m->data, len);
+			rte_hexdump(f, NULL, rte_pktmbuf_mtod(m, void *), len);
 		dump_len -= len;
 		m = m->next;
 		nb_segs --;
lib/librte_mbuf/rte_mbuf.h +19 -23

135 lines not on this page · raw message

lib/librte_pmd_bond/rte_eth_bond_pmd.c +2 -2
index 5979ce5..506a448 100644
@@ -198,14 +198,14 @@ xmit_slave_hash(const struct rte_mbuf *buf, uint8_t slave_count, uint8_t policy)
 
 	switch (policy) {
 	case BALANCE_XMIT_POLICY_LAYER2:
-		eth_hdr = (struct ether_hdr *)buf->data;
+		eth_hdr = rte_pktmbuf_mtod(buf, struct ether_hdr *);
 
 		hash = ether_hash(eth_hdr);
 		hash ^= hash >> 8;
 		return hash % slave_count;
 
 	case BALANCE_XMIT_POLICY_LAYER23:
-		eth_hdr = (struct ether_hdr *)buf->data;
+		eth_hdr = rte_pktmbuf_mtod(buf, struct ether_hdr *);
 
 		if (buf->ol_flags & PKT_RX_VLAN_PKT)
 			eth_offset = sizeof(struct ether_hdr) + sizeof(struct vlan_hdr);
lib/librte_pmd_e1000/em_rxtx.c +6 -6
index ba7e3a9..ed3a6fc 100644
@@ -90,8 +90,7 @@ rte_rxmbuf_alloc(struct rte_mempool *mp)
 }
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb)             \
-	(uint64_t) ((mb)->buf_physaddr +       \
-	(uint64_t) ((char *)((mb)->data) - (char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -793,8 +792,8 @@ eth_em_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.length) -
 				rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -963,7 +962,7 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1035,7 +1034,8 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
lib/librte_pmd_e1000/igb_rxtx.c +6 -7
index d4a803e..41727e7 100644
@@ -95,9 +95,7 @@ rte_rxmbuf_alloc(struct rte_mempool *mp)
 }
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr +		   \
-			(uint64_t) ((char *)((mb)->data) -     \
-				(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -774,8 +772,8 @@ eth_igb_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.wb.upper.length) -
 				      rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -950,7 +948,7 @@ eth_igb_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.wb.upper.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1031,7 +1029,8 @@ eth_igb_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		first_seg->ol_flags = pkt_flags;
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
lib/librte_pmd_i40e/i40e_rxtx.c +8 -9
index e41e8d0..25a5f6f 100644
@@ -78,9 +78,7 @@
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	((uint64_t)((mb)->buf_physaddr + \
-	(uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr)))
+	((uint64_t)((mb)->buf_physaddr + (mb)->data_off))
 
 static const struct rte_memzone *
 i40e_ring_dma_zone_reserve(struct rte_eth_dev *dev,
@@ -685,7 +683,7 @@ i40e_rx_alloc_bufs(struct i40e_rx_queue *rxq)
 		mb = rxep[i].mbuf;
 		rte_mbuf_refcnt_set(mb, 1);
 		mb->next = NULL;
-		mb->data = (char *)mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mb->data_off = RTE_PKTMBUF_HEADROOM;
 		mb->nb_segs = 1;
 		mb->port = rxq->port_id;
 		dma_addr = rte_cpu_to_le_64(\
@@ -842,8 +840,8 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		rx_packet_len = ((qword1 & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
 				I40E_RXD_QW1_LENGTH_PBUF_SHIFT) - rxq->crc_len;
 
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_prefetch0(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_prefetch0(RTE_PTR_ADD(rxm->buf_addr, RTE_PKTMBUF_HEADROOM));
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = rx_packet_len;
@@ -946,7 +944,7 @@ i40e_recv_scattered_pkts(void *rx_queue,
 		rx_packet_len = (qword1 & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
 					I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
 		rxm->data_len = rx_packet_len;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/**
 		 * If this is the first buffer of the received packet, set the
@@ -1015,7 +1013,8 @@ i40e_recv_scattered_pkts(void *rx_queue,
 				rte_le_to_cpu_32(rxd.wb.qword0.hi_dword.rss);
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_prefetch0(first_seg->data);
+		rte_prefetch0(RTE_PTR_ADD(first_seg->buf_addr,
+			first_seg->data_off));
 		rx_pkts[nb_rx++] = first_seg;
 		first_seg = NULL;
 	}
@@ -2131,7 +2130,7 @@ i40e_alloc_rx_queue_mbufs(struct i40e_rx_queue *rxq)
 
 		rte_mbuf_refcnt_set(mbuf, 1);
 		mbuf->next = NULL;
-		mbuf->data = (char *)mbuf->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 		mbuf->nb_segs = 1;
 		mbuf->port = rxq->port_id;
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +7 -6
index 575a014..c661335 100644
@@ -997,7 +997,7 @@ ixgbe_rx_alloc_bufs(struct igb_rx_queue *rxq)
 		mb = rxep[i].mbuf;
 		rte_mbuf_refcnt_set(mb, 1);
 		mb->next = NULL;
-		mb->data = (char *)mb->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mb->data_off = RTE_PKTMBUF_HEADROOM;
 		mb->nb_segs = 1;
 		mb->port = rxq->port_id;
 
@@ -1248,8 +1248,8 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		pkt_len = (uint16_t) (rte_le_to_cpu_16(rxd.wb.upper.length) -
 				      rxq->crc_len);
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
-		rte_packet_prefetch(rxm->data);
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rte_packet_prefetch((char *)rxm->buf_addr + rxm->data_off);
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
 		rxm->pkt_len = pkt_len;
@@ -1431,7 +1431,7 @@ ixgbe_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		 */
 		data_len = rte_le_to_cpu_16(rxd.wb.upper.length);
 		rxm->data_len = data_len;
-		rxm->data = (char*) rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		/*
 		 * If this is the first buffer of the received packet,
@@ -1522,7 +1522,8 @@ ixgbe_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		}
 
 		/* Prefetch data of first segment, if configured to do so. */
-		rte_packet_prefetch(first_seg->data);
+		rte_packet_prefetch((char *)first_seg->buf_addr +
+			first_seg->data_off);
 
 		/*
 		 * Store the mbuf address into the next entry of the array
@@ -3212,7 +3213,7 @@ ixgbe_alloc_rx_queue_mbufs(struct igb_rx_queue *rxq)
 
 		rte_mbuf_refcnt_set(mbuf, 1);
 		mbuf->next = NULL;
-		mbuf->data = (char *)mbuf->buf_addr + RTE_PKTMBUF_HEADROOM;
+		mbuf->data_off = RTE_PKTMBUF_HEADROOM;
 		mbuf->nb_segs = 1;
 		mbuf->port = rxq->port_id;
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -2
index 41042ac..fd0023e 100644
@@ -47,8 +47,7 @@
 #endif
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
lib/librte_pmd_pcap/rte_eth_pcap.c +6 -3
index 121de65..2ff1e77 100644
@@ -151,7 +151,8 @@ eth_pcap_rx(void *queue,
 
 		if (header.len <= buf_size) {
 			/* pcap packet will fit in the mbuf, go ahead and copy */
-			rte_memcpy(mbuf->data, packet, header.len);
+			rte_memcpy(rte_pktmbuf_mtod(mbuf, void *), packet,
+					header.len);
 			mbuf->data_len = (uint16_t)header.len;
 			mbuf->pkt_len = mbuf->data_len;
 			bufs[num_rx] = mbuf;
@@ -202,7 +203,8 @@ eth_pcap_tx_dumper(void *queue,
 		calculate_timestamp(&header.ts);
 		header.len = mbuf->data_len;
 		header.caplen = header.len;
-		pcap_dump((u_char*) dumper_q->dumper, &header, mbuf->data);
+		pcap_dump((u_char *)dumper_q->dumper, &header,
+				rte_pktmbuf_mtod(mbuf, void*));
 		rte_pktmbuf_free(mbuf);
 		num_tx++;
 	}
@@ -237,7 +239,8 @@ eth_pcap_tx(void *queue,
 
 	for (i = 0; i < nb_pkts; i++) {
 		mbuf = bufs[i];
-		ret = pcap_sendpacket(tx_queue->pcap, (u_char*) mbuf->data,
+		ret = pcap_sendpacket(tx_queue->pcap,
+				rte_pktmbuf_mtod(mbuf, u_char *),
 				mbuf->data_len);
 		if (unlikely(ret != 0))
 			break;
lib/librte_pmd_virtio/virtio_rxtx.c +4 -6
index 132ee45..29c9cea 100644
@@ -118,7 +118,7 @@ virtqueue_dequeue_burst_rx(struct virtqueue *vq, struct rte_mbuf **rx_pkts,
 		}
 
 		rte_prefetch0(cookie);
-		rte_packet_prefetch(cookie->data);
+		rte_packet_prefetch(rte_pktmbuf_mtod(cookie, void *));
 		rx_pkts[i]  = cookie;
 		vq->vq_used_cons_idx++;
 		vq_ring_free_chain(vq, desc_idx);
@@ -480,7 +480,7 @@ virtio_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		}
 
 		rxm->port = rxvq->port_id;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 		rxm->nb_segs = 1;
 		rxm->next = NULL;
@@ -584,7 +584,7 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 		if (seg_num == 0)
 			seg_num = 1;
 
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 		rxm->nb_segs = seg_num;
 		rxm->next = NULL;
 		rxm->pkt_len = (uint32_t)(len[0] - hdr_size);
@@ -622,9 +622,7 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 			while (extra_idx < rcv_cnt) {
 				rxm = rcv_pkts[extra_idx];
 
-				rxm->data =
-					(char *)rxm->buf_addr +
-					RTE_PKTMBUF_HEADROOM - hdr_size;
+				rxm->data_off = RTE_PKTMBUF_HEADROOM - hdr_size;
 				rxm->next = NULL;
 				rxm->pkt_len = (uint32_t)(len[extra_idx]);
 				rxm->data_len = (uint16_t)(len[extra_idx]);
lib/librte_pmd_virtio/virtqueue.h +1 -2
index d777feb..fdee054 100644
@@ -59,8 +59,7 @@
 #define VIRTQUEUE_MAX_NAME_SZ 32
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define VTNET_SQ_RQ_QUEUE_IDX 0
 #define VTNET_SQ_TQ_QUEUE_IDX 1
lib/librte_pmd_vmxnet3/vmxnet3_rxtx.c +2 -3
index e74b6fd..263f9ce 100644
@@ -79,8 +79,7 @@
 
 
 #define RTE_MBUF_DATA_DMA_ADDR(mb) \
-	(uint64_t) ((mb)->buf_physaddr + (uint64_t)((char *)((mb)->data) - \
-	(char *)(mb)->buf_addr))
+	(uint64_t) ((mb)->buf_physaddr + (mb)->data_off)
 
 #define RTE_MBUF_DATA_DMA_ADDR_DEFAULT(mb) \
 	(uint64_t) ((mb)->buf_physaddr + RTE_PKTMBUF_HEADROOM)
@@ -565,7 +564,7 @@ vmxnet3_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 			rxm->data_len = (uint16_t)rcd->len;
 			rxm->port = rxq->port_id;
 			rxm->vlan_tci = 0;
-			rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+			rxm->data_off = RTE_PKTMBUF_HEADROOM;
 
 			rx_pkts[nb_rx++] = rxm;
 
lib/librte_pmd_xenvirt/rte_eth_xenvirt.c +1 -1
index 22215ed..891cb58 100644
@@ -110,7 +110,7 @@ eth_xenvirt_rx(void *q, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 		rxm = rx_pkts[i];
 		PMD_RX_LOG(DEBUG, "packet len:%d\n", len[i]);
 		rxm->next = NULL;
-		rxm->data = (char *)rxm->buf_addr + RTE_PKTMBUF_HEADROOM;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
 		rxm->data_len = (uint16_t)(len[i] - sizeof(struct virtio_net_hdr));
 		rxm->nb_segs = 1;
 		rxm->port = pi->port_id;
Signature · 1 line
1.9.3
#78 lore raw

PATCH v2 2/13 mbuf: reorder fields by time of use

In reply to a message not in the archive

* Reorder the fields in the mbuf so that we have fields that are used together side-by-side in the structure. This means that we have a contiguous block of 8-bytes in the mbuf which are used to reset an mbuf of descriptor rearm, and a block of 16-bytes of data (excluding flags) which are set on RX from the received packet descriptor.
* Use dummy fields as appropriate to ensure alignment or to reserve gaps for later field additions.
* Place most items which are not used by fast-path RX separately at the end of the structure so they can later be moved to a separate cache line. [The l2/l3 length fields are not moved at this stage as doing so will cause overflow to the next cache line].

Updated in V2:
* Updated the KNI buffer structure to match that of new mbuf
* Cleaned up commit message typos.

---
 .../linuxapp/eal/include/exec-env/rte_kni_common.h | 12 ++++++-----
 lib/librte_mbuf/rte_mbuf.h                         | 25 ++++++++++++----------
 2 files changed, 21 insertions(+), 16 deletions(-)
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +7 -5
index 07908ac..f2b502c 100644
@@ -108,15 +108,17 @@ struct rte_kni_fifo {
  * Padding is necessary to assure the offsets of these fields
  */
 struct rte_kni_mbuf {
-	void *pool;
 	void *buf_addr;
-	char pad0[16];
-	void *next;
+	char pad0[10];
 	uint16_t data_off;      /**< Start address of data in segment buffer. */
+	char pad1[4];
+	uint16_t ol_flags;      /**< Offload features. */
+	char pad2[8];
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
-	char pad2[4];
-	uint16_t ol_flags;      /**< Offload features. */
+	char pad3[8];
+	void *pool;
+	void *next;
 } __attribute__((__aligned__(64)));
 
 /*
lib/librte_mbuf/rte_mbuf.h +14 -11
index 71080e5..413a5a1 100644
@@ -115,16 +115,12 @@ extern "C" {
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
-	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
-	uint16_t buf_len;         /**< Length of segment buffer. */
 
-	/* valid for any segment */
-	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	/* next 8 bytes are initialised on RX descriptor rearm */
+	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
-	uint16_t data_len;        /**< Amount of data in segment buffer. */
-	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
 
 #ifdef RTE_MBUF_REFCNT
 	/**
@@ -142,14 +138,17 @@ struct rte_mbuf {
 #else
 	uint16_t refcnt_reserved;     /**< Do not use this field */
 #endif
-	uint16_t reserved;            /**< Unused field. Required for padding */
-	uint16_t ol_flags;            /**< Offload features. */
-
-	/* these fields are valid for first segment only */
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	/* offload features, valid for first segment only */
+	uint16_t ol_flags;      /**< Offload features. */
+	uint16_t reserved0;     /**< Unused field. Required for padding */
+	uint32_t reserved1;     /**< Unused field. Required for padding */
+
+	/* remaining bytes are set on RX when pulling packet from descriptor */
+	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t data_len;      /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	union {
 		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
 		struct {
@@ -167,6 +166,10 @@ struct rte_mbuf {
 		uint32_t sched;     /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
+	/* fields only used in slow path or on TX */
+	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+
 	union {
 		uint8_t metadata[0];
 		uint16_t metadata16[0];
Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1410441347-22840-3-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#80 lore raw

Re: [PATCH v2 02/13] mbuf: reorder fields by time of use

Hi Bruce,

Quoted 117 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Bruce Richardson
Sent: Thursday, September 11, 2014 9:16 PM
To: [email protected]
Subject: [dpdk-dev] [PATCH v2 02/13] mbuf: reorder fields by time of use

* Reorder the fields in the mbuf so that we have fields that are used together side-by-side in the structure. This means that we have a contiguous block of 8-bytes in the mbuf which are used to reset an mbuf of descriptor rearm, and a block of 16-bytes of data (excluding flags) which are set on RX from the received packet descriptor.
* Use dummy fields as appropriate to ensure alignment or to reserve gaps for later field additions.
* Place most items which are not used by fast-path RX separately at the end of the structure so they can later be moved to a separate cache line. [The l2/l3 length fields are not moved at this stage as doing so will cause overflow to the next cache line].

Updated in V2:
* Updated the KNI buffer structure to match that of new mbuf
* Cleaned up commit message typos.

---
 .../linuxapp/eal/include/exec-env/rte_kni_common.h | 12 ++++++-----
 lib/librte_mbuf/rte_mbuf.h                         | 25
++++++++++++----------
 2 files changed, 21 insertions(+), 16 deletions(-)
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +7 -5
index 07908ac..f2b502c 100644
@@ -108,15 +108,17 @@ struct rte_kni_fifo {
  * Padding is necessary to assure the offsets of these fields
  */
 struct rte_kni_mbuf {
-	void *pool;
 	void *buf_addr;
-	char pad0[16];
-	void *next;
+	char pad0[10];
 	uint16_t data_off;      /**< Start address of data in segment buffer. */
+	char pad1[4];
+	uint16_t ol_flags;      /**< Offload features. */
+	char pad2[8];
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len.
 */
-	char pad2[4];
-	uint16_t ol_flags;      /**< Offload features. */
+	char pad3[8];
+	void *pool;
+	void *next;
 } __attribute__((__aligned__(64)));
 

/*

lib/librte_mbuf/rte_mbuf.h index renamed from lib/librte_mbuf/rte_mbuf.h +0 -0

71080e5..413a5a1 100644

lib/librte_mbuf/rte_mbuf.h +10 -11
@@ -115,16 +115,12 @@ extern "C" {
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
-	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
-	uint16_t buf_len;         /**< Length of segment buffer. */
 
-	/* valid for any segment */
-	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	/* next 8 bytes are initialised on RX descriptor rearm */
+	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
-	uint16_t data_len;        /**< Amount of data in segment buffer. */
-	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
 
 #ifdef RTE_MBUF_REFCNT
 	/**
@@ -142,14 +138,17 @@ struct rte_mbuf {
 #else
 	uint16_t refcnt_reserved;     /**< Do not use this field */
 #endif
-	uint16_t reserved;            /**< Unused field. Required for padding
 */
-	uint16_t ol_flags;            /**< Offload features. */
-
-	/* these fields are valid for first segment only */
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	/* offload features, valid for first segment only */
+	uint16_t ol_flags;      /**< Offload features. */
+	uint16_t reserved0;     /**< Unused field. Required for padding */
+	uint32_t reserved1;     /**< Unused field. Required for padding */
+
+	/* remaining bytes are set on RX when pulling packet from descriptor */
+	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t data_len;      /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	union {
 		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
lib/librte_mbuf/rte_mbuf.h +4 -1
 		struct {
@@ -167,6 +166,10 @@ struct rte_mbuf {
 		uint32_t sched;     /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
+	/* fields only used in slow path or on TX */
+	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+
 	union {
 		uint8_t metadata[0];
 		uint16_t metadata16[0];
--
Quoted 1 line

1.9.3

It seems that you removed the inner_l3_len and inner_l2_len fields from the rte_mbuf.h structure, what is the reason?

/Jijiang Liu

A message that is not in the archive (sent off-list or not yet received): 1ED644BD7E0A5F4091CF203DAFB8E4CC01D701BA-0J0gbvR4kThpB2pF5aRoyrfspsVTdybXVpNB7YpNyf8@public.gmane.org

#82 lore raw

Re: [PATCH v2 02/13] mbuf: reorder fields by time of use

Quoted 132 lines

-----Original Message-----
From: Liu, Jijiang
Sent: Monday, September 15, 2014 8:12 AM
To: Richardson, Bruce; [email protected]
Subject: RE: [dpdk-dev] [PATCH v2 02/13] mbuf: reorder fields by time of use

Hi Bruce,

Quoted 63 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Bruce Richardson
Sent: Thursday, September 11, 2014 9:16 PM
To: [email protected]
Subject: [dpdk-dev] [PATCH v2 02/13] mbuf: reorder fields by time of use

* Reorder the fields in the mbuf so that we have fields that are used together side-by-side in the structure. This means that we have a contiguous block of 8-bytes in the mbuf which are used to reset an mbuf of descriptor rearm, and a block of 16-bytes of data (excluding flags) which are set on RX from the received packet descriptor.
* Use dummy fields as appropriate to ensure alignment or to reserve gaps for later field additions.
* Place most items which are not used by fast-path RX separately at the end of the structure so they can later be moved to a separate cache line. [The l2/l3 length fields are not moved at this stage as doing so will cause overflow to the next cache line].

Updated in V2:
* Updated the KNI buffer structure to match that of new mbuf
* Cleaned up commit message typos.

---
 .../linuxapp/eal/include/exec-env/rte_kni_common.h | 12 ++++++-----
 lib/librte_mbuf/rte_mbuf.h                         | 25
++++++++++++----------
 2 files changed, 21 insertions(+), 16 deletions(-)
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +7 -5
index 07908ac..f2b502c 100644
@@ -108,15 +108,17 @@ struct rte_kni_fifo {
  * Padding is necessary to assure the offsets of these fields
  */
 struct rte_kni_mbuf {
-	void *pool;
 	void *buf_addr;
-	char pad0[16];
-	void *next;
+	char pad0[10];
 	uint16_t data_off;      /**< Start address of data in segment buffer. */
+	char pad1[4];
+	uint16_t ol_flags;      /**< Offload features. */
+	char pad2[8];
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len.
 */
-	char pad2[4];
-	uint16_t ol_flags;      /**< Offload features. */
+	char pad3[8];
+	void *pool;
+	void *next;
 } __attribute__((__aligned__(64)));
 

/*

lib/librte_mbuf/rte_mbuf.h index renamed from lib/librte_mbuf/rte_mbuf.h +0 -0

71080e5..413a5a1 100644

lib/librte_mbuf/rte_mbuf.h extern "C" { · line 118

Quoted 4 lines
lib/librte_mbuf/rte_mbuf.h +0 -1
@@ -115,16 +115,12 @@ extern "C" {
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
-	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.

*/

Quoted 39 lines
lib/librte_mbuf/rte_mbuf.h +10 -10
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
-	uint16_t buf_len;         /**< Length of segment buffer. */
 
-	/* valid for any segment */
-	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+	/* next 8 bytes are initialised on RX descriptor rearm */
+	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
-	uint16_t data_len;        /**< Amount of data in segment buffer. */
-	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
 
 #ifdef RTE_MBUF_REFCNT
 	/**
@@ -142,14 +138,17 @@ struct rte_mbuf {
 #else
 	uint16_t refcnt_reserved;     /**< Do not use this field */
 #endif
-	uint16_t reserved;            /**< Unused field. Required for padding
 */
-	uint16_t ol_flags;            /**< Offload features. */
-
-	/* these fields are valid for first segment only */
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	/* offload features, valid for first segment only */
+	uint16_t ol_flags;      /**< Offload features. */
+	uint16_t reserved0;     /**< Unused field. Required for padding */
+	uint32_t reserved1;     /**< Unused field. Required for padding */
+
+	/* remaining bytes are set on RX when pulling packet from descriptor */
+	uint16_t reserved2;     /**< Unused field. Required for padding */
+	uint16_t data_len;      /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
 	union {
 		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
lib/librte_mbuf/rte_mbuf.h +0 -0
 		struct {

lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 170

Quoted 5 lines
lib/librte_mbuf/rte_mbuf.h +2 -0
@@ -167,6 +166,10 @@ struct rte_mbuf {
 		uint32_t sched;     /**< Hierarchical scheduler */
 	} hash;                 /**< hash information */
 
+	/* fields only used in slow path or on TX */
+	struct rte_mempool *pool; /**< Pool from which mbuf was allocated.

*/

Quoted 7 lines
lib/librte_mbuf/rte_mbuf.h +2 -1
+	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
+
 	union {
 		uint8_t metadata[0];
 		uint16_t metadata16[0];
--

1.9.3

It seems that you removed the inner_l3_len and inner_l2_len fields from the rte_mbuf.h structure, what is the reason?

Quoted 1 line

/Jijiang Liu

No, those fields were never added to the mbuf, they were simply proposed as a new field addition in an RFC patch set. The adding of new fields to the mbuf is beyond the scope of the two patch sets already submitted. Those patch sets merely reorder the existing l2 and l3 length fields without changing their size or name.

/Bruce

#83 lore raw

PATCH v2 3/13 mbuf: expand ol_flags field to 64-bits

In reply to a message not in the archive

The offload flags field (ol_flags) was 16-bits and had no further room for expansion. This patch increases the field size to 64-bits, using up the remaining reserved space in the single-cache-line mbuf.

NOTE: none of the values for existing flags have been changed, i.e. no new numbers have been explicitly reserved between existing flag definitions.

Updates in V2:
* Modified ol_flags field in kni mbuf structure

Updated kni data structure

---
 app/test-pmd/config.c                              |  8 +--
 app/test-pmd/csumonly.c                            |  8 +--
 app/test-pmd/rxonly.c                              |  2 +-
 app/test-pmd/testpmd.h                             |  4 +-
 app/test-pmd/txonly.c                              |  2 +-
 .../linuxapp/eal/include/exec-env/rte_kni_common.h |  4 +-
 lib/librte_mbuf/rte_mbuf.c                         |  2 +-
 lib/librte_mbuf/rte_mbuf.h                         |  4 +-
 lib/librte_pmd_e1000/em_rxtx.c                     | 34 ++++++------
 lib/librte_pmd_e1000/igb_rxtx.c                    | 62 ++++++++++------------
 lib/librte_pmd_i40e/i40e_rxtx.c                    | 31 +++++------
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c                  | 61 ++++++++++-----------
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h                  |  2 +-
 13 files changed, 106 insertions(+), 118 deletions(-)
app/test-pmd/config.c +4 -4
index 606e34a..adfa9a8 100644
@@ -1714,14 +1714,14 @@ set_qmap(portid_t port_id, uint8_t is_rx, uint16_t queue_id, uint8_t map_value)
 }
 
 void
-tx_cksum_set(portid_t port_id, uint8_t cksum_mask)
+tx_cksum_set(portid_t port_id, uint64_t ol_flags)
 {
-	uint16_t tx_ol_flags;
+	uint64_t tx_ol_flags;
 	if (port_id_is_invalid(port_id))
 		return;
 	/* Clear last 4 bits and then set L3/4 checksum mask again */
-	tx_ol_flags = (uint16_t) (ports[port_id].tx_ol_flags & 0xFFF0);
-	ports[port_id].tx_ol_flags = (uint16_t) ((cksum_mask & 0xf) | tx_ol_flags);
+	tx_ol_flags = ports[port_id].tx_ol_flags & (~0x0Full);
+	ports[port_id].tx_ol_flags = ((ol_flags & 0xf) | tx_ol_flags);
 }
 
 void
app/test-pmd/csumonly.c +4 -4
index 2ce4c42..fcc4876 100644
@@ -217,9 +217,9 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 	uint16_t nb_rx;
 	uint16_t nb_tx;
 	uint16_t i;
-	uint16_t ol_flags;
-	uint16_t pkt_ol_flags;
-	uint16_t tx_ol_flags;
+	uint64_t ol_flags;
+	uint64_t pkt_ol_flags;
+	uint64_t tx_ol_flags;
 	uint16_t l4_proto;
 	uint16_t eth_type;
 	uint8_t  l2_len;
@@ -261,7 +261,7 @@ pkt_burst_checksum_forward(struct fwd_stream *fs)
 		mb = pkts_burst[i];
 		l2_len  = sizeof(struct ether_hdr);
 		pkt_ol_flags = mb->ol_flags;
-		ol_flags = (uint16_t) (pkt_ol_flags & (~PKT_TX_L4_MASK));
+		ol_flags = (pkt_ol_flags & (~PKT_TX_L4_MASK));
 
 		eth_hdr = rte_pktmbuf_mtod(mb, struct ether_hdr *);
 		eth_type = rte_be_to_cpu_16(eth_hdr->ether_type);
app/test-pmd/rxonly.c +1 -1
index 7ba36a1..98c788b 100644
@@ -109,7 +109,7 @@ pkt_burst_receive(struct fwd_stream *fs)
 	struct rte_mbuf  *mb;
 	struct ether_hdr *eth_hdr;
 	uint16_t eth_type;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint16_t nb_rx;
 	uint16_t i;
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
app/test-pmd/testpmd.h +2 -2
index 09923a8..142091d 100644
@@ -141,7 +141,7 @@ struct rte_port {
 	struct fwd_stream       *rx_stream; /**< Port RX stream, if unique */
 	struct fwd_stream       *tx_stream; /**< Port TX stream, if unique */
 	unsigned int            socket_id;  /**< For NUMA support */
-	uint16_t                tx_ol_flags;/**< Offload Flags of TX packets. */
+	uint64_t                tx_ol_flags;/**< Offload Flags of TX packets. */
 	uint16_t                tx_vlan_id; /**< Tag Id. in TX VLAN packets. */
 	void                    *fwd_ctx;   /**< Forwarding mode context */
 	uint64_t                rx_bad_ip_csum; /**< rx pkts with bad ip checksum  */
@@ -494,7 +494,7 @@ void tx_vlan_pvid_set(portid_t port_id, uint16_t vlan_id, int on);
 
 void set_qmap(portid_t port_id, uint8_t is_rx, uint16_t queue_id, uint8_t map_value);
 
-void tx_cksum_set(portid_t port_id, uint8_t cksum_mask);
+void tx_cksum_set(portid_t port_id, uint64_t ol_flags);
 
 void set_verbose_level(uint16_t vb_level);
 void set_tx_pkt_segments(unsigned *seg_lengths, unsigned nb_segs);
app/test-pmd/txonly.c +1 -1
index 2b3f0b9..3d08005 100644
@@ -203,7 +203,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
 	uint16_t nb_tx;
 	uint16_t nb_pkt;
 	uint16_t vlan_tci;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint8_t  i;
 #ifdef RTE_TEST_PMD_RECORD_CORE_CYCLES
 	uint64_t start_tsc;
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +2 -2
index f2b502c..ab022bd 100644
@@ -112,8 +112,8 @@ struct rte_kni_mbuf {
 	char pad0[10];
 	uint16_t data_off;      /**< Start address of data in segment buffer. */
 	char pad1[4];
-	uint16_t ol_flags;      /**< Offload features. */
-	char pad2[8];
+	uint64_t ol_flags;      /**< Offload features. */
+	char pad2[2];
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
 	char pad3[8];
lib/librte_mbuf/rte_mbuf.c +1 -1
index 26e36eb..9dfcac3 100644
@@ -174,7 +174,7 @@ rte_pktmbuf_dump(FILE *f, const struct rte_mbuf *m, unsigned dump_len)
 
 	fprintf(f, "dump mbuf at 0x%p, phys=%"PRIx64", buf_len=%u\n",
 	       m, (uint64_t)m->buf_physaddr, (unsigned)m->buf_len);
-	fprintf(f, "  pkt_len=%"PRIu32", ol_flags=%"PRIx16", nb_segs=%u, "
+	fprintf(f, "  pkt_len=%"PRIu32", ol_flags=%"PRIx64", nb_segs=%u, "
 	       "in_port=%u\n", m->pkt_len, m->ol_flags,
 	       (unsigned)m->nb_segs, (unsigned)m->port);
 	nb_segs = m->nb_segs;
lib/librte_mbuf/rte_mbuf.h +1 -3
index 413a5a1..0e20e31 100644
@@ -141,9 +141,7 @@ struct rte_mbuf {
 	uint8_t nb_segs;        /**< Number of segments. */
 	uint8_t port;           /**< Input port. */
 
-	uint16_t ol_flags;      /**< Offload features. */
-	uint16_t reserved0;     /**< Unused field. Required for padding */
-	uint32_t reserved1;     /**< Unused field. Required for padding */
+	uint64_t ol_flags;      /**< Offload features. */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
 	uint16_t reserved2;     /**< Unused field. Required for padding */
lib/librte_pmd_e1000/em_rxtx.c +16 -18
index ed3a6fc..b8423b4 100644
@@ -168,7 +168,7 @@ union em_vlan_macip {
  * Structure to check if new context need be built
  */
 struct em_ctx_info {
-	uint16_t flags;              /**< ol_flags related to context build. */
+	uint64_t flags;              /**< ol_flags related to context build. */
 	uint32_t cmp_mask;           /**< compare mask */
 	union em_vlan_macip hdrlen;  /**< L2 and L3 header lenghts */
 };
@@ -238,7 +238,7 @@ struct em_tx_queue {
 static inline void
 em_set_xmit_ctx(struct em_tx_queue* txq,
 		volatile struct e1000_context_desc *ctx_txd,
-		uint16_t flags,
+		uint64_t flags,
 		union em_vlan_macip hdrlen)
 {
 	uint32_t cmp_mask, cmd_len;
@@ -304,7 +304,7 @@ em_set_xmit_ctx(struct em_tx_queue* txq,
  * or create a new context descriptor.
  */
 static inline uint32_t
-what_ctx_update(struct em_tx_queue *txq, uint16_t flags,
+what_ctx_update(struct em_tx_queue *txq, uint64_t flags,
 		union em_vlan_macip hdrlen)
 {
 	/* If match with the current context */
@@ -377,7 +377,7 @@ em_xmit_cleanup(struct em_tx_queue *txq)
 }
 
 static inline uint32_t
-tx_desc_cksum_flags_to_upper(uint16_t ol_flags)
+tx_desc_cksum_flags_to_upper(uint64_t ol_flags)
 {
 	static const uint32_t l4_olinfo[2] = {0, E1000_TXD_POPTS_TXSM << 8};
 	static const uint32_t l3_olinfo[2] = {0, E1000_TXD_POPTS_IXSM << 8};
@@ -403,12 +403,12 @@ eth_em_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 	uint32_t popts_spec;
 	uint32_t cmd_type_len;
 	uint16_t slen;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint16_t tx_id;
 	uint16_t tx_last;
 	uint16_t nb_tx;
 	uint16_t nb_used;
-	uint16_t tx_ol_req;
+	uint64_t tx_ol_req;
 	uint32_t ctx;
 	uint32_t new_ctx;
 	union em_vlan_macip hdrlen;
@@ -438,8 +438,7 @@ eth_em_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		ol_flags = tx_pkt->ol_flags;
 
 		/* If hardware offload required */
-		tx_ol_req = (uint16_t)(ol_flags & (PKT_TX_IP_CKSUM |
-							PKT_TX_L4_MASK));
+		tx_ol_req = (ol_flags & (PKT_TX_IP_CKSUM | PKT_TX_L4_MASK));
 		if (tx_ol_req) {
 			hdrlen.f.vlan_tci = tx_pkt->vlan_tci;
 			hdrlen.f.l2_len = tx_pkt->l2_len;
@@ -642,22 +641,21 @@ end_of_tx:
  *
  **********************************************************************/
 
-static inline uint16_t
+static inline uint64_t
 rx_desc_status_to_pkt_flags(uint32_t rx_status)
 {
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	/* Check if VLAN present */
-	pkt_flags = (uint16_t)((rx_status & E1000_RXD_STAT_VP) ?
-						PKT_RX_VLAN_PKT : 0);
+	pkt_flags = ((rx_status & E1000_RXD_STAT_VP) ?  PKT_RX_VLAN_PKT : 0);
 
 	return pkt_flags;
 }
 
-static inline uint16_t
+static inline uint64_t
 rx_desc_error_to_pkt_flags(uint32_t rx_error)
 {
-	uint16_t pkt_flags = 0;
+	uint64_t pkt_flags = 0;
 
 	if (rx_error & E1000_RXD_ERR_IPE)
 		pkt_flags |= PKT_RX_IP_CKSUM_BAD;
@@ -801,8 +799,8 @@ eth_em_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		rxm->port = rxq->port_id;
 
 		rxm->ol_flags = rx_desc_status_to_pkt_flags(status);
-		rxm->ol_flags = (uint16_t)(rxm->ol_flags |
-				rx_desc_error_to_pkt_flags(rxd.errors));
+		rxm->ol_flags = rxm->ol_flags |
+				rx_desc_error_to_pkt_flags(rxd.errors);
 
 		/* Only valid if PKT_RX_VLAN_PKT set in pkt_flags */
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
@@ -1027,8 +1025,8 @@ eth_em_recv_scattered_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		first_seg->port = rxq->port_id;
 
 		first_seg->ol_flags = rx_desc_status_to_pkt_flags(status);
-		first_seg->ol_flags = (uint16_t)(first_seg->ol_flags |
-					rx_desc_error_to_pkt_flags(rxd.errors));
+		first_seg->ol_flags = first_seg->ol_flags |
+					rx_desc_error_to_pkt_flags(rxd.errors);
 
 		/* Only valid if PKT_RX_VLAN_PKT set in pkt_flags */
 		rxm->vlan_tci = rte_le_to_cpu_16(rxd.special);
lib/librte_pmd_e1000/igb_rxtx.c +28 -34

169 lines not on this page · raw message

lib/librte_pmd_i40e/i40e_rxtx.c +16 -15
index 25a5f6f..cd05654 100644
@@ -91,27 +91,27 @@ static uint16_t i40e_xmit_pkts_simple(void *tx_queue,
 				      uint16_t nb_pkts);
 
 /* Translate the rx descriptor status to pkt flags */
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_status_to_pkt_flags(uint64_t qword)
 {
-	uint16_t flags;
+	uint64_t flags;
 
 	/* Check if VLAN packet */
-	flags = (uint16_t)(qword & (1 << I40E_RX_DESC_STATUS_L2TAG1P_SHIFT) ?
-							PKT_RX_VLAN_PKT : 0);
+	flags = qword & (1 << I40E_RX_DESC_STATUS_L2TAG1P_SHIFT) ?
+							PKT_RX_VLAN_PKT : 0;
 
 	/* Check if RSS_HASH */
-	flags |= (uint16_t)((((qword >> I40E_RX_DESC_STATUS_FLTSTAT_SHIFT) &
+	flags |= (((qword >> I40E_RX_DESC_STATUS_FLTSTAT_SHIFT) &
 					I40E_RX_DESC_FLTSTAT_RSS_HASH) ==
-			I40E_RX_DESC_FLTSTAT_RSS_HASH) ? PKT_RX_RSS_HASH : 0);
+			I40E_RX_DESC_FLTSTAT_RSS_HASH) ? PKT_RX_RSS_HASH : 0;
 
 	return flags;
 }
 
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_error_to_pkt_flags(uint64_t qword)
 {
-	uint16_t flags = 0;
+	uint64_t flags = 0;
 	uint64_t error_bits = (qword >> I40E_RXD_QW1_ERROR_SHIFT);
 
 #define I40E_RX_ERR_BITS 0x3f
@@ -143,12 +143,12 @@ i40e_rxd_error_to_pkt_flags(uint64_t qword)
 }
 
 /* Translate pkt types to pkt flags */
-static inline uint16_t
+static inline uint64_t
 i40e_rxd_ptype_to_pkt_flags(uint64_t qword)
 {
 	uint8_t ptype = (uint8_t)((qword & I40E_RXD_QW1_PTYPE_MASK) >>
 					I40E_RXD_QW1_PTYPE_SHIFT);
-	static const uint16_t ip_ptype_map[I40E_MAX_PKT_TYPE] = {
+	static const uint64_t ip_ptype_map[I40E_MAX_PKT_TYPE] = {
 		0, /* PTYPE 0 */
 		0, /* PTYPE 1 */
 		0, /* PTYPE 2 */
@@ -567,7 +567,7 @@ i40e_rx_scan_hw_ring(struct i40e_rx_queue *rxq)
 	uint32_t rx_status;
 	int32_t s[I40E_LOOK_AHEAD], nb_dd;
 	int32_t i, j, nb_rx = 0;
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	rxdp = &rxq->rx_ring[rxq->rx_tail];
 	rxep = &rxq->sw_ring[rxq->rx_tail];
@@ -789,7 +789,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
 	uint16_t rx_packet_len;
 	uint16_t rx_id, nb_hold;
 	uint64_t dma_addr;
-	uint16_t pkt_flags;
+	uint64_t pkt_flags;
 
 	nb_rx = 0;
 	nb_hold = 0;
@@ -896,10 +896,11 @@ i40e_recv_scattered_pkts(void *rx_queue,
 	struct rte_mbuf *last_seg = rxq->pkt_last_seg;
 	struct rte_mbuf *nmb, *rxm;
 	uint16_t rx_id = rxq->rx_tail;
-	uint16_t nb_rx = 0, nb_hold = 0, rx_packet_len, pkt_flags;
+	uint16_t nb_rx = 0, nb_hold = 0, rx_packet_len;
 	uint32_t rx_status;
 	uint64_t qword1;
 	uint64_t dma_addr;
+	uint64_t pkt_flags;
 
 	while (nb_rx < nb_pkts) {
 		rxdp = &rx_ring[rx_id];
@@ -1046,7 +1047,7 @@ i40e_recv_scattered_pkts(void *rx_queue,
 
 /* Check if the context descriptor is needed for TX offloading */
 static inline uint16_t
-i40e_calc_context_desc(uint16_t flags)
+i40e_calc_context_desc(uint64_t flags)
 {
 	uint16_t mask = 0;
 
@@ -1075,7 +1076,7 @@ i40e_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)
 	uint32_t td_offset;
 	uint32_t tx_flags;
 	uint32_t td_tag;
-	uint16_t ol_flags;
+	uint64_t ol_flags;
 	uint8_t l2_len;
 	uint8_t l3_len;
 	uint16_t nb_used;
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +29 -32

167 lines not on this page · raw message

lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -1
index fd0023e..0d633d6 100644
@@ -177,7 +177,7 @@ union ixgbe_vlan_macip {
  */
 
 struct ixgbe_advctx_info {
-	uint16_t flags;           /**< ol_flags for context build. */
+	uint64_t flags;           /**< ol_flags for context build. */
 	uint32_t cmp_mask;        /**< compare mask for vlan_macip_lens */
 	union ixgbe_vlan_macip vlan_macip_lens; /**< vlan, mac ip length. */
 };
Signature · 1 line
1.9.3
#84 lore raw

PATCH v2 4/13 mbuf: introduce a flag to indicate a control mbuf

In reply to a message not in the archive

Since the flags field is now 64-bits, we can allow one bit to be used to indicate a control i.e. non-packet mbuf. Dedicate the high bit (bit 63) for this purpose and add in a utility macro to test if a given mbuf has the bit set or not.

Updated in V2:
* Fix typo in comment in rte_mbuf.h

---
 lib/librte_mbuf/rte_mbuf.c |  2 ++
 lib/librte_mbuf/rte_mbuf.h | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+)
lib/librte_mbuf/rte_mbuf.c +2 -0
index 9dfcac3..52e7574 100644
@@ -69,7 +69,9 @@ rte_ctrlmbuf_init(struct rte_mempool *mp,
 		void *_m,
 		__attribute__((unused)) unsigned i)
 {
+	struct rte_mbuf *m = _m;
 	rte_pktmbuf_init(mp, opaque_arg, _m, i);
+	m->ol_flags |= CTRL_MBUF_FLAG;
 }
 
 /*
lib/librte_mbuf/rte_mbuf.h +19 -0
index 0e20e31..b9a1c66 100644
@@ -91,6 +91,7 @@ extern "C" {
 #define PKT_TX_IPV4_CSUM     0x1000 /**< Alias of PKT_TX_IP_CKSUM. */
 #define PKT_TX_IPV4          PKT_RX_IPV4_HDR /**< IPv4 with no IP checksum offload. */
 #define PKT_TX_IPV6          PKT_RX_IPV6_HDR /**< IPv6 packet */
+
 /*
  * Bit 14~13 used for L4 packet type with checksum enabled.
  *     00: Reserved
@@ -106,6 +107,9 @@ extern "C" {
 /* Bit 15 */
 #define PKT_TX_IEEE1588_TMST 0x8000 /**< TX IEEE1588 packet to timestamp. */
 
+/* Use final bit of flags to indicate a control mbuf */
+#define CTRL_MBUF_FLAG       (1ULL << 63)
+
 /**
  * Bit Mask to indicate what bits required for building TX context
  */
@@ -471,6 +475,21 @@ void rte_ctrlmbuf_init(struct rte_mempool *mp, void *opaque_arg,
  */
 #define rte_ctrlmbuf_len(m) rte_pktmbuf_data_len(m)
 
+/**
+ * Tests if an mbuf is a control mbuf
+ *
+ * @param m
+ *   The mbuf to be tested
+ * @return
+ *   - True (1) if the mbuf is a control mbuf
+ *   - False(0) otherwise
+ */
+static inline int
+rte_is_ctrlmbuf(struct rte_mbuf *m)
+{
+	return (!!(m->ol_flags & CTRL_MBUF_FLAG));
+}
+
 /* Operations on pkt mbuf */
 
 /**
Signature · 1 line
1.9.3
#85 lore raw

PATCH v2 5/13 mbuf: minor changes for readability

In reply to a message not in the archive

* Ensure comments line up correctly
* Simplify the #ifdefs around the refcnt fields to make them clearer

Updates in V2:
* None

---
 lib/librte_mbuf/rte_mbuf.h | 33 ++++++++++++++++-----------------
 1 file changed, 16 insertions(+), 17 deletions(-)
lib/librte_mbuf/rte_mbuf.h +16 -17
index b9a1c66..6ce852a 100644
@@ -126,7 +126,6 @@ struct rte_mbuf {
 	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
 
-#ifdef RTE_MBUF_REFCNT
 	/**
 	 * 16-bit Reference counter.
 	 * It should only be accessed using the following functions:
@@ -136,21 +135,21 @@ struct rte_mbuf {
 	 * config option.
 	 */
 	union {
-		rte_atomic16_t refcnt_atomic;   /**< Atomically accessed refcnt */
-		uint16_t refcnt;                /**< Non-atomically accessed refcnt */
-	};
-#else
-	uint16_t refcnt_reserved;     /**< Do not use this field */
+#ifdef RTE_MBUF_REFCNT
+		rte_atomic16_t refcnt_atomic; /**< Atomically accessed refcnt */
+		uint16_t refcnt;              /**< Non-atomically accessed refcnt */
 #endif
-	uint8_t nb_segs;        /**< Number of segments. */
-	uint8_t port;           /**< Input port. */
+		uint16_t refcnt_reserved;     /**< Do not use this field */
+	};
+	uint8_t nb_segs;          /**< Number of segments. */
+	uint8_t port;             /**< Input port. */
 
-	uint64_t ol_flags;      /**< Offload features. */
+	uint64_t ol_flags;        /**< Offload features. */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
-	uint16_t reserved2;     /**< Unused field. Required for padding */
-	uint16_t data_len;      /**< Amount of data in segment buffer. */
-	uint32_t pkt_len;       /**< Total pkt len: sum of all segments. */
+	uint16_t reserved2;       /**< Unused field. Required for padding */
+	uint16_t data_len;        /**< Amount of data in segment buffer. */
+	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
 	union {
 		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
 		struct {
@@ -158,15 +157,15 @@ struct rte_mbuf {
 			uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
 		};
 	};
-	uint16_t vlan_tci;      /**< VLAN Tag Control Identifier (CPU order). */
+	uint16_t vlan_tci;        /**< VLAN Tag Control Identifier (CPU order) */
 	union {
-		uint32_t rss;       /**< RSS hash result if RSS enabled */
+		uint32_t rss;     /**< RSS hash result if RSS enabled */
 		struct {
 			uint16_t hash;
 			uint16_t id;
-		} fdir;             /**< Filter identifier if FDIR enabled */
-		uint32_t sched;     /**< Hierarchical scheduler */
-	} hash;                 /**< hash information */
+		} fdir;           /**< Filter identifier if FDIR enabled */
+		uint32_t sched;   /**< Hierarchical scheduler */
+	} hash;                   /**< hash information */
 
 	/* fields only used in slow path or on TX */
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
Signature · 1 line
1.9.3
#86 lore raw

PATCH v2 6/13 mbuf: use macros only to access the mbuf metadata

In reply to a message not in the archive

Removed the explicit zero-sized metadata definition at the end of the mbuf data structure. Updated the metadata macros to take account of this change so that all existing code which uses those macros still works.

Updates in V2:
* None

---
 lib/librte_mbuf/rte_mbuf.h | 22 ++++++++--------------
 1 file changed, 8 insertions(+), 14 deletions(-)
lib/librte_mbuf/rte_mbuf.h +8 -14
index 6ce852a..f855e64 100644
@@ -171,31 +171,25 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
-	union {
-		uint8_t metadata[0];
-		uint16_t metadata16[0];
-		uint32_t metadata32[0];
-		uint64_t metadata64[0];
-	} __rte_cache_aligned;
 } __rte_cache_aligned;
 
 #define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(mbuf->metadata[offset])
+	(((uint8_t *)&(mbuf)[1])[offset])
 #define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(mbuf->metadata16[offset/sizeof(uint16_t)])
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
 #define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(mbuf->metadata32[offset/sizeof(uint32_t)])
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
 #define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(mbuf->metadata64[offset/sizeof(uint64_t)])
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
 
 #define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&mbuf->metadata[offset])
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&mbuf->metadata16[offset/sizeof(uint16_t)])
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&mbuf->metadata32[offset/sizeof(uint32_t)])
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
 #define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&mbuf->metadata64[offset/sizeof(uint64_t)])
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
 
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.
Signature · 1 line
1.9.3
#87 lore raw

PATCH v2 7/13 mbuf: move metadata macros to rte_port library

In reply to a message not in the archive

The metadata macros are only used by libs and apps using the rte_port packet framework library, so move them to a header file there.

---
 lib/librte_mbuf/rte_mbuf.h | 18 ------------------
 lib/librte_port/rte_port.h | 22 ++++++++++++++++++++++
 2 files changed, 22 insertions(+), 18 deletions(-)
lib/librte_mbuf/rte_mbuf.h +0 -18
index f855e64..2f1bd4a 100644
@@ -173,24 +173,6 @@ struct rte_mbuf {
 
 } __rte_cache_aligned;
 
-#define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
-	(((uint8_t *)&(mbuf)[1])[offset])
-#define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
-	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
-#define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
-	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
-#define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
-	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
-
-#define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
-	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
-#define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
-	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
-#define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
-	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
-#define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
-	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
-
 /**
  * Given the buf_addr returns the pointer to corresponding mbuf.
  */
lib/librte_port/rte_port.h +22 -0
index 1d763c2..6bd5039 100644
@@ -50,6 +50,28 @@ extern "C" {
 #include <stdint.h>
 #include <rte_mbuf.h>
 
+/**
+ * Macros to allow accessing metadata stored in the mbuf headroom
+ * just beyond the end of the mbuf data structure returned by a port
+ */
+#define RTE_MBUF_METADATA_UINT8(mbuf, offset)              \
+	(((uint8_t *)&(mbuf)[1])[offset])
+#define RTE_MBUF_METADATA_UINT16(mbuf, offset)             \
+	(((uint16_t *)&(mbuf)[1])[offset/sizeof(uint16_t)])
+#define RTE_MBUF_METADATA_UINT32(mbuf, offset)             \
+	(((uint32_t *)&(mbuf)[1])[offset/sizeof(uint32_t)])
+#define RTE_MBUF_METADATA_UINT64(mbuf, offset)             \
+	(((uint64_t *)&(mbuf)[1])[offset/sizeof(uint64_t)])
+
+#define RTE_MBUF_METADATA_UINT8_PTR(mbuf, offset)          \
+	(&RTE_MBUF_METADATA_UINT8(mbuf, offset))
+#define RTE_MBUF_METADATA_UINT16_PTR(mbuf, offset)         \
+	(&RTE_MBUF_METADATA_UINT16(mbuf, offset))
+#define RTE_MBUF_METADATA_UINT32_PTR(mbuf, offset)         \
+	(&RTE_MBUF_METADATA_UINT32(mbuf, offset))
+#define RTE_MBUF_METADATA_UINT64_PTR(mbuf, offset)         \
+	(&RTE_MBUF_METADATA_UINT64(mbuf, offset))
+
 /*
  * Port IN
  *
Signature · 1 line
1.9.3
#88 lore raw

PATCH v2 8/13 mbuf: add named points inside the mbuf structure

In reply to a message not in the archive

Add markers or "labels" at given points inside the mbuf which can be used instead of individual fields to identify the start of logical sections inside the mbuf.

The use of typedefs and dummy fields was chosen over using unions because of a couple reasons:
* unions cause an extra level of indentation (more likely two levels as a union containing a struct for multiple fields would be needed). This makes the lines longer than they need to be and increases the need for wrapping. [This was the main reason]
* with markers, you can apply multiple markers at the same point if wanted.

Updates in V2:
* None

---
 lib/librte_mbuf/rte_mbuf.h | 9 +++++++++
 1 file changed, 9 insertions(+)
lib/librte_mbuf/rte_mbuf.h +9 -0
index 2f1bd4a..34900d4 100644
@@ -115,14 +115,22 @@ extern "C" {
  */
 #define PKT_TX_OFFLOAD_MASK (PKT_TX_VLAN_PKT | PKT_TX_IP_CKSUM | PKT_TX_L4_MASK)
 
+/* define a set of marker types that can be used to refer to set points in the
+ * mbuf */
+typedef void    *MARKER[0];   /**< generic marker for a point in a structure */
+typedef uint64_t MARKER64[0]; /**< marker that allows us to overwrite 8 bytes
+                               * with a single assignment */
 /**
  * The generic rte_mbuf, containing a packet mbuf.
  */
 struct rte_mbuf {
+	MARKER cacheline0;
+
 	void *buf_addr;           /**< Virtual address of segment buffer. */
 	phys_addr_t buf_physaddr; /**< Physical address of segment buffer. */
 
 	/* next 8 bytes are initialised on RX descriptor rearm */
+	MARKER64 rearm_data;
 	uint16_t buf_len;         /**< Length of segment buffer. */
 	uint16_t data_off;
 
@@ -147,6 +155,7 @@ struct rte_mbuf {
 	uint64_t ol_flags;        /**< Offload features. */
 
 	/* remaining bytes are set on RX when pulling packet from descriptor */
+	MARKER rx_descriptor_fields1;
 	uint16_t reserved2;       /**< Unused field. Required for padding */
 	uint16_t data_len;        /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
Signature · 1 line
1.9.3
#89 lore raw

PATCH v2 9/13 ixgbe: rework vector pmd following mbuf changes

In reply to a message not in the archive

The vector PMD expects fields to be in a specific order so that it can do vector operations on multiple fields at a time. Following mbuf rework, adjust driver to take account of the new layout and re-enable it in the config.

Updates in V2:
* Remove extra line at end of file to eliminate git warning

---
 config/common_linuxapp                |   2 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |   2 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     |   4 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 135 ++++++++++++----------------------
 4 files changed, 53 insertions(+), 90 deletions(-)
config/common_linuxapp +1 -1
index b140af7..5bee910 100644
@@ -191,7 +191,7 @@ CONFIG_RTE_LIBRTE_IXGBE_DEBUG_DRIVER=n
 CONFIG_RTE_LIBRTE_IXGBE_PF_DISABLE_STRIP_CRC=n
 CONFIG_RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC=y
 CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=n
-CONFIG_RTE_IXGBE_INC_VECTOR=n
+CONFIG_RTE_IXGBE_INC_VECTOR=y
 CONFIG_RTE_IXGBE_RX_OLFLAGS_ENABLE=y
 
 #
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +1 -1
index 26cb176..1a46393 100644
@@ -2166,7 +2166,7 @@ ixgbe_dev_rx_queue_setup(struct rte_eth_dev *dev,
 		if (!ixgbe_rx_vec_condition_check(dev)) {
 			PMD_INIT_LOG(INFO, "Vector rx enabled, please make "
 				     "sure RX burst size no less than 32.\n");
-			ixgbe_rxq_vec_setup(rxq, socket_id);
+			ixgbe_rxq_vec_setup(rxq);
 			dev->rx_pkt_burst = ixgbe_recv_pkts_vec;
 		}
 #endif
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +2 -2
index 0d633d6..e92a864 100644
@@ -115,6 +115,7 @@ struct igb_rx_queue {
 	struct igb_rx_entry *sw_ring; /**< address of RX software ring. */
 	struct rte_mbuf *pkt_first_seg; /**< First segment of current packet. */
 	struct rte_mbuf *pkt_last_seg; /**< Last segment of current packet. */
+	uint64_t            mbuf_initializer; /**< value to init mbufs */
 	uint16_t            nb_rx_desc; /**< number of RX descriptors. */
 	uint16_t            rx_tail;  /**< current value of RDT register. */
 	uint16_t            nb_rx_hold; /**< number of held free RX desc. */
@@ -126,7 +127,6 @@ struct igb_rx_queue {
 #ifdef RTE_IXGBE_INC_VECTOR
 	uint16_t            rxrearm_nb; /**< the idx we start the re-arming from */
 	uint16_t            rxrearm_start;  /**< number of remaining to be re-armed */
-	__m128i             misc_info;  /**< cache XMM combine port_id/crc/nb_segs */
 #endif
 	uint16_t            rx_free_thresh; /**< max free RX desc to hold. */
 	uint16_t            queue_id; /**< RX queue index. */
@@ -259,7 +259,7 @@ struct ixgbe_txq_ops {
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct igb_tx_queue *txq, unsigned int socket_id);
-int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq, unsigned int socket_id);
+int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
 #endif
 
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +49 -86

242 lines not on this page · raw message

Signature · 1 line
1.9.3
#90 lore raw

PATCH v2 10/13 mbuf: split mbuf across two cache lines.

In reply to a message not in the archive

This change splits the mbuf in two to move the pool and next pointers to the second cache line. This frees up 16 bytes in first cache line.

The reason for this change is that we believe that there is no possible way that we can ever fit all the fields we need to fit into a 64-byte mbuf, and so we need to start looking at a 128-byte mbuf instead. Examples of new fields that need to fit in, include -
* 32-bits more for filter information for support for the new filters in the i40e driver (and possibly other future drivers)
* an additional 2-4 bytes for storing info on a second vlan tag to allow drivers to support double Vlan/QinQ
* 4-bytes for storing a sequence number to enable out of order packet processing and subsequent packet reordering
as well as potentially a number of other fields or splitting out fields that are superimposed over each other right now, e.g. for the qos scheduler. We also want to allow space for use by other non-Intel NIC drivers that may be open-sourced to dpdk.org in the future too, where they support fields and offloads that currently supported hardware doesn't.

If we accept the fact of a 2-cache-line mbuf, then the issue becomes how to rework things so that we spread our fields over the two cache lines while causing the lowest slow-down possible. The general approach that we are looking to take is to focus the first cache line on fields that are updated on RX , so that receive only deals with one cache line. The second cache line can be used for application data and information that will only be used on the TX leg. This would allow us to work on the first cache line in RX as now, and have the second cache line being prefetched in the background so that it is available when necessary. Hardware prefetches should help us out here. We also may move rarely used, or slow-path RX fields e.g. such as those for chained mbufs with jumbo frames, to the second cache line, depending upon the performance impact and bytes savings achieved.

Updated in V2:
* Expanded commit description to include contents of a previous mail describing some of the logic behind expanding mbuf to two cache lines
* Update kni mbuf structure

---
 app/test/test_mbuf.c                                          | 2 +-
 lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 6 +++---
 lib/librte_mbuf/rte_mbuf.h                                    | 3 ++-
 3 files changed, 6 insertions(+), 5 deletions(-)
app/test/test_mbuf.c +1 -1
index 1b25481..66bcbc5 100644
@@ -782,7 +782,7 @@ test_failing_mbuf_sanity_check(void)
 static int
 test_mbuf(void)
 {
-	RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != 64);
+	RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != CACHE_LINE_SIZE * 2);
 
 	/* create pktmbuf pool if it does not exist */
 	if (pktmbuf_pool == NULL) {
lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h +3 -3
index ab022bd..25ed672 100644
@@ -108,7 +108,7 @@ struct rte_kni_fifo {
  * Padding is necessary to assure the offsets of these fields
  */
 struct rte_kni_mbuf {
-	void *buf_addr;
+	void *buf_addr __attribute__((__aligned__(64)));
 	char pad0[10];
 	uint16_t data_off;      /**< Start address of data in segment buffer. */
 	char pad1[4];
@@ -117,9 +117,9 @@ struct rte_kni_mbuf {
 	uint16_t data_len;      /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;       /**< Total pkt len: sum of all segment data_len. */
 	char pad3[8];
-	void *pool;
+	void *pool __attribute__((__aligned__(64)));
 	void *next;
-} __attribute__((__aligned__(64)));
+};
 
 /*
  * Struct used to create a KNI device. Passed to the kernel in IOCTL call
lib/librte_mbuf/rte_mbuf.h +2 -1
index 34900d4..508021b 100644
@@ -176,7 +176,8 @@ struct rte_mbuf {
 		uint32_t sched;   /**< Hierarchical scheduler */
 	} hash;                   /**< hash information */
 
-	/* fields only used in slow path or on TX */
+	/* second cache line - fields only used in slow path or on TX */
+	MARKER cacheline1 __rte_cache_aligned;
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
Signature · 1 line
1.9.3
#91 lore raw

PATCH v2 11/13 mbuf: move l2_len and l3_len to second cache line

In reply to a message not in the archive

The l2_len and l3_len fields are used for TX offloads and so should be put on the second cache line, along with the other fields only used on TX.

Updates in V2:
* The l2 and l3 lengths can be accessed as a single uint16_t for performance, as well as individually.

---
 lib/librte_mbuf/rte_mbuf.h | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
lib/librte_mbuf/rte_mbuf.h +9 -7
index 508021b..1c6e115 100644
@@ -159,13 +159,7 @@ struct rte_mbuf {
 	uint16_t reserved2;       /**< Unused field. Required for padding */
 	uint16_t data_len;        /**< Amount of data in segment buffer. */
 	uint32_t pkt_len;         /**< Total pkt len: sum of all segments. */
-	union {
-		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
-		struct {
-			uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
-			uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
-		};
-	};
+	uint16_t reserved;
 	uint16_t vlan_tci;        /**< VLAN Tag Control Identifier (CPU order) */
 	union {
 		uint32_t rss;     /**< RSS hash result if RSS enabled */
@@ -181,6 +175,14 @@ struct rte_mbuf {
 	struct rte_mempool *pool; /**< Pool from which mbuf was allocated. */
 	struct rte_mbuf *next;    /**< Next segment of scattered packet. */
 
+	/* fields to support TX offloads */
+	union {
+		uint16_t l2_l3_len; /**< combined l2/l3 lengths as single var */
+		struct {
+			uint16_t l3_len:9;      /**< L3 (IP) Header Length. */
+			uint16_t l2_len:7;      /**< L2 (MAC) Header Length. */
+		};
+	};
 } __rte_cache_aligned;
 
 /**
Signature · 1 line
1.9.3
#92 lore raw

PATCH v2 12/13 ixgbe: Fix perf regression due to moved pool ptr

In reply to a message not in the archive

Adjust the fast-path code to fix the regression caused by the pool pointer moving to the second cache line. This change adjusts the prefetching and also the way in which the mbufs are freed back to the mempool.
Note: slow-path e.g. path supporting jumbo frames, is still slower, but is dealt with by a later commit

Updates in V2:
* fixup checkpatch issue

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |  8 ++-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     | 14 +-----
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 92 +++++++++++++----------------------
 3 files changed, 38 insertions(+), 76 deletions(-)
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +3 -5
index 1a46393..d6448a4 100644
@@ -142,10 +142,6 @@ ixgbe_tx_free_bufs(struct igb_tx_queue *txq)
 	 */
 	txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);
 
-	/* prefetch the mbufs that are about to be freed */
-	for (i = 0; i < txq->tx_rs_thresh; ++i)
-		rte_prefetch0((txep + i)->mbuf);
-
 	/* free buffers one at a time */
 	if ((txq->txq_flags & (uint32_t)ETH_TXQ_FLAGS_NOREFCOUNT) != 0) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
@@ -186,6 +182,7 @@ tx4(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 				((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 		txdp->read.olinfo_status =
 				(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+		rte_prefetch0(&(*pkts)->pool);
 	}
 }
 
@@ -205,6 +202,7 @@ tx1(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 			((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 	txdp->read.olinfo_status =
 			(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+	rte_prefetch0(&(*pkts)->pool);
 }
 
 /*
@@ -1875,7 +1873,7 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev,
 		PMD_INIT_LOG(INFO, "Using simple tx code path\n");
 #ifdef RTE_IXGBE_INC_VECTOR
 		if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ &&
-		    ixgbe_txq_vec_setup(txq, socket_id) == 0) {
+		    ixgbe_txq_vec_setup(txq) == 0) {
 			PMD_INIT_LOG(INFO, "Vector tx enabled.\n");
 			dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
 		}
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -13
index e92a864..a97fddb 100644
@@ -96,14 +96,6 @@ struct igb_tx_entry_v {
 };
 
 /**
- * continuous entry sequence, gather by the same mempool
- */
-struct igb_tx_entry_seq {
-	const struct rte_mempool* pool;
-	uint32_t same_pool;
-};
-
-/**
  * Structure associated with each RX queue.
  */
 struct igb_rx_queue {
@@ -190,10 +182,6 @@ struct igb_tx_queue {
 	volatile union ixgbe_adv_tx_desc *tx_ring;
 	uint64_t            tx_ring_phys_addr; /**< TX ring DMA address. */
 	struct igb_tx_entry *sw_ring;      /**< virtual address of SW ring. */
-#ifdef RTE_IXGBE_INC_VECTOR
-	/** continuous tx entry sequence within the same mempool */
-	struct igb_tx_entry_seq *sw_ring_seq;
-#endif
 	volatile uint32_t   *tdt_reg_addr; /**< Address of TDT register. */
 	uint16_t            nb_tx_desc;    /**< number of TX descriptors. */
 	uint16_t            tx_tail;       /**< current value of TDT reg. */
@@ -258,7 +246,7 @@ struct ixgbe_txq_ops {
 #ifdef RTE_IXGBE_INC_VECTOR
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
-int ixgbe_txq_vec_setup(struct igb_tx_queue *txq, unsigned int socket_id);
+int ixgbe_txq_vec_setup(struct igb_tx_queue *txq);
 int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
 #endif
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +34 -58

187 lines not on this page · raw message

Signature · 1 line
1.9.3

A message that is not in the archive (sent off-list or not yet received): 1410441347-22840-13-git-send-email-bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#94 lore raw

PATCH v3 12/13 ixgbe: Fix perf regression due to moved pool ptr

Adjust the fast-path code to fix the regression caused by the pool pointer moving to the second cache line. This change adjusts the prefetching and also the way in which the mbufs are freed back to the mempool.
Note: slow-path e.g. path supporting jumbo frames, is still slower, but is dealt with by a later commit

Updates in V2:
* fixup checkpatch issue

Updates in V3:
* The variable definitions for freeing mbufs now need to be included whether or not reference counting is enabled.

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |  8 ++-
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     | 14 +-----
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 94 +++++++++++++----------------------
 3 files changed, 38 insertions(+), 78 deletions(-)
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +3 -5
index 1a46393..d6448a4 100644
@@ -142,10 +142,6 @@ ixgbe_tx_free_bufs(struct igb_tx_queue *txq)
 	 */
 	txep = &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);
 
-	/* prefetch the mbufs that are about to be freed */
-	for (i = 0; i < txq->tx_rs_thresh; ++i)
-		rte_prefetch0((txep + i)->mbuf);
-
 	/* free buffers one at a time */
 	if ((txq->txq_flags & (uint32_t)ETH_TXQ_FLAGS_NOREFCOUNT) != 0) {
 		for (i = 0; i < txq->tx_rs_thresh; ++i, ++txep) {
@@ -186,6 +182,7 @@ tx4(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 				((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 		txdp->read.olinfo_status =
 				(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+		rte_prefetch0(&(*pkts)->pool);
 	}
 }
 
@@ -205,6 +202,7 @@ tx1(volatile union ixgbe_adv_tx_desc *txdp, struct rte_mbuf **pkts)
 			((uint32_t)DCMD_DTYP_FLAGS | pkt_len);
 	txdp->read.olinfo_status =
 			(pkt_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+	rte_prefetch0(&(*pkts)->pool);
 }
 
 /*
@@ -1875,7 +1873,7 @@ ixgbe_dev_tx_queue_setup(struct rte_eth_dev *dev,
 		PMD_INIT_LOG(INFO, "Using simple tx code path\n");
 #ifdef RTE_IXGBE_INC_VECTOR
 		if (txq->tx_rs_thresh <= RTE_IXGBE_TX_MAX_FREE_BUF_SZ &&
-		    ixgbe_txq_vec_setup(txq, socket_id) == 0) {
+		    ixgbe_txq_vec_setup(txq) == 0) {
 			PMD_INIT_LOG(INFO, "Vector tx enabled.\n");
 			dev->tx_pkt_burst = ixgbe_xmit_pkts_vec;
 		}
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +1 -13
index e92a864..a97fddb 100644
@@ -96,14 +96,6 @@ struct igb_tx_entry_v {
 };
 
 /**
- * continuous entry sequence, gather by the same mempool
- */
-struct igb_tx_entry_seq {
-	const struct rte_mempool* pool;
-	uint32_t same_pool;
-};
-
-/**
  * Structure associated with each RX queue.
  */
 struct igb_rx_queue {
@@ -190,10 +182,6 @@ struct igb_tx_queue {
 	volatile union ixgbe_adv_tx_desc *tx_ring;
 	uint64_t            tx_ring_phys_addr; /**< TX ring DMA address. */
 	struct igb_tx_entry *sw_ring;      /**< virtual address of SW ring. */
-#ifdef RTE_IXGBE_INC_VECTOR
-	/** continuous tx entry sequence within the same mempool */
-	struct igb_tx_entry_seq *sw_ring_seq;
-#endif
 	volatile uint32_t   *tdt_reg_addr; /**< Address of TDT register. */
 	uint16_t            nb_tx_desc;    /**< number of TX descriptors. */
 	uint16_t            tx_tail;       /**< current value of TDT reg. */
@@ -258,7 +246,7 @@ struct ixgbe_txq_ops {
 #ifdef RTE_IXGBE_INC_VECTOR
 uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
 uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
-int ixgbe_txq_vec_setup(struct igb_tx_queue *txq, unsigned int socket_id);
+int ixgbe_txq_vec_setup(struct igb_tx_queue *txq);
 int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
 #endif
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +34 -60

192 lines not on this page · raw message

Signature · 1 line
1.9.3
#95 lore raw

PATCH v2 13/13 ixgbe: Improve slow-path perf: vector scattered RX

In reply to a message not in the archive

Provide a wrapper routine to enable receive of scattered packets with a vector driver. This improves the performance of the slow-path RX.

Updates in V2:
* Fixed up some checkpatch errors and warnings

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx.c     |  16 ++++
 lib/librte_pmd_ixgbe/ixgbe_rxtx.h     |   8 +-
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 171 ++++++++++++++++++++++++++++++++--
 3 files changed, 186 insertions(+), 9 deletions(-)
lib/librte_pmd_ixgbe/ixgbe_rxtx.c +16 -0
index d6448a4..a80cade 100644
@@ -3476,12 +3476,20 @@ ixgbe_dev_rx_init(struct rte_eth_dev *dev)
 		if ((dev->data->dev_conf.rxmode.max_rx_pkt_len +
 				2 * IXGBE_VLAN_TAG_SIZE) > buf_size){
 			dev->data->scattered_rx = 1;
+#ifdef RTE_IXGBE_INC_VECTOR
+			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		}
 	}
 
 	if (dev->data->dev_conf.rxmode.enable_scatter) {
+#ifdef RTE_IXGBE_INC_VECTOR
+		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		dev->data->scattered_rx = 1;
 	}
 
@@ -3969,12 +3977,20 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
 		if ((dev->data->dev_conf.rxmode.max_rx_pkt_len +
 				2 * IXGBE_VLAN_TAG_SIZE) > buf_size) {
 			dev->data->scattered_rx = 1;
+#ifdef RTE_IXGBE_INC_VECTOR
+			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 			dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		}
 	}
 
 	if (dev->data->dev_conf.rxmode.enable_scatter) {
+#ifdef RTE_IXGBE_INC_VECTOR
+		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts_vec;
+#else
 		dev->rx_pkt_burst = ixgbe_recv_scattered_pkts;
+#endif
 		dev->data->scattered_rx = 1;
 	}
 
lib/librte_pmd_ixgbe/ixgbe_rxtx.h +6 -2
index a97fddb..7b5ac0e 100644
@@ -244,8 +244,12 @@ struct ixgbe_txq_ops {
 			 IXGBE_ADVTXD_DCMD_EOP)
 
 #ifdef RTE_IXGBE_INC_VECTOR
-uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
-uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
+uint16_t ixgbe_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts);
+uint16_t ixgbe_recv_scattered_pkts_vec(void *rx_queue,
+		struct rte_mbuf **rx_pkts, uint16_t nb_pkts);
+uint16_t ixgbe_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts);
 int ixgbe_txq_vec_setup(struct igb_tx_queue *txq);
 int ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq);
 int ixgbe_rx_vec_condition_check(struct rte_eth_dev *dev);
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c +164 -7

221 lines not on this page · raw message

Signature · 1 line
1.9.3
#96 lore raw
In reply to a message not in the archive

2014-09-11 14:15, Bruce Richardson:

Quoted 38 lines

This patch set continues on from the changes in part 1, and depends upon that patch set.

This patch set reorders the fields in the mbuf structure and splits the structure across two cache lines, given lots of new space for new fields to be added. This set uses some of that space by expanding the ol_flags field. A part 3 patchset is planned to introduce some other new fields into the new mbuf structure.

With the splitting of the mbuf across multiple cache lines, performance degradations are seen inside the drivers, both fast-path and slow path. For the fast-path, this patchset reworks the way in which the pool pointer is used to free packets post-TX, which removes the perf regression. For the slow path, an alternative approach is taken - a new scattered packets RX function is introduced into the vector PMD. Using this function, throughput for the slow path RX-TX using testpmd is increased by up to 20% over the original baseline.

Changes in V2:
* General minor updates follow comments on V1 set
* Updated a number of patches to include KNI mbuf changes where needed
* Deferred the patch to add the packet type field to a future patch set
* After removing meta-data element from mbuf structure also added in patch to move the macros to rte_port/rte_port.h

Bruce Richardson (11):
mbuf: reorder fields by time of use
mbuf: expand ol_flags field to 64-bits
mbuf: introduce a flag to indicate a control mbuf
mbuf: minor changes for readability
mbuf: use macros only to access the mbuf metadata
mbuf: add named points inside the mbuf structure
ixgbe: rework vector pmd following mbuf changes
mbuf: split mbuf across two cache lines.
mbuf: move l2_len and l3_len to second cache line
ixgbe: Fix perf regression due to moved pool ptr
ixgbe: Improve slow-path perf: vector scattered RX

Quoted 2 lines

Olivier Matz (1):
mbuf: replace data pointer by an offset

Applied for version 1.8.0.

Note that ixgbe patches are not reviewed but I make an exception here, because Intel is responsible of ixgbe performances and we have some time to fix it (if needed) before the release.

Signature · 1 line
Thomas