Skip to contentSkip to conversation

build: enable packet data prefetch

PATCH Superseded Marvin Liu 4 participants 11 messages Sep 22, 2020 – Nov 3, 2020

Subsystems (2)
lore
Share & tools
lore

Marvin Liu

author
#0 lore raw

PATCH 1/1 build: enable packet data prefetch

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

config/meson.build +3 -0
index 69f2aeb605..a0c828a437 100644
@@ -109,6 +109,9 @@ if not is_windows
 	add_project_link_arguments('-Wl,--no-as-needed', language: 'c')
 endif
 
+# do prefetch of packet data
+dpdk_conf.set('RTE_PMD_PACKET_PREFETCH', 1)
+
 # use pthreads if available for the platform
 if not is_windows
 	add_project_link_arguments('-pthread', language: 'c')
Signature · 1 line
2.17.1
#1 lore raw
Quoted 12 lines · On Tue, 22 Sep 2020 16:21:35 +0800 Marvin Liu <[email protected]> wrote:

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

config/meson.build +0 -0
index 69f2aeb605..a0c828a437 100644

config/meson.build if not is_windows · line 117

Quoted 9 lines
config/meson.build +3 -0
@@ -109,6 +109,9 @@ if not is_windows
 	add_project_link_arguments('-Wl,--no-as-needed', language: 'c')
 endif
 
+# do prefetch of packet data
+dpdk_conf.set('RTE_PMD_PACKET_PREFETCH', 1)
+
 # use pthreads if available for the platform
 if not is_windows
 	add_project_link_arguments('-pthread', language: 'c')

With meson, the project has been using rte_config.h for this.

author
#2 lore raw
Quoted 17 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Tuesday, September 22, 2020 10:12 PM
To: Liu, Yong <[email protected]>
Cc: Richardson, Bruce <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] build: enable packet data prefetch

Quoted 5 lines · On Tue, 22 Sep 2020 16:21:35 +0800 Marvin Liu <[email protected]> wrote:

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

Quoted 4 lines
Quoted 2 lines

With meson, the project has been using rte_config.h for this.

Thanks a lot, will send v2 for the change.

Regards,
Marvin

author
#3 lore raw

PATCH v2 1/1 config: enable packet data prefetch

In reply to Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

---
v2: move define from meson.build to rte_config.h

---
 config/rte_config.h | 1 +
 1 file changed, 1 insertion(+)
config/rte_config.h +1 -0
index 0bae630fd9..8b007c4c31 100644
@@ -101,6 +101,7 @@
 #define RTE_LIBRTE_GRAPH_STATS 1
 
 /****** driver defines ********/
+#define RTE_PMD_PACKET_PREFETCH 1
 
 /* QuickAssist device */
 /* Max. number of QuickAssist devices which can be attached */
Signature · 1 line
2.17.1
#4 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

23/09/2020 03:51, Marvin Liu:

Quoted 8 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

---
+#define RTE_PMD_PACKET_PREFETCH 1

We could also remove the related #ifdefs.

What can be the drawback of always enable those prefetches?

author
#5 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

Quoted 19 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, October 15, 2020 6:03 AM
To: Liu, Yong <[email protected]>
Cc: Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; Yigit, Ferruh <[email protected]>; [email protected]; [email protected]; David Christensen <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

23/09/2020 03:51, Marvin Liu:

Quoted 5 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

Quoted 8 lines
Quoted 3 lines

---
+#define RTE_PMD_PACKET_PREFETCH 1

We could also remove the related #ifdefs.

What can be the drawback of always enable those prefetches?

Hi Thomas,
I think the potential drawback is that current prefetch location cannot guarantee the best performance across different platforms. Each developer has tuned the performance by adding prefetch instruction and verified the result on himself platform. So prefetch location is based on certain platform, also it will be hard for developer to compare the results across platforms.

Thanks,
Marvin

#6 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

<snip>

Quoted 11 lines
Quoted 11 lines

23/09/2020 03:51, Marvin Liu:

Quoted 8 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as default.

---
+#define RTE_PMD_PACKET_PREFETCH 1

Quoted 8 lines
Quoted 4 lines

We could also remove the related #ifdefs.

What can be the drawback of always enable those prefetches?

Hi Thomas,
I think the potential drawback is that current prefetch location cannot guarantee the best performance across different platforms.

Then, does it make sense to enable this by default?

Quoted 7 lines

Each developer has tuned the performance by adding prefetch instruction and verified the result on himself platform.
So prefetch location is based on certain platform, also it will be hard for developer to compare the results across platforms.

Thanks,
Marvin

author
#7 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

Quoted 31 lines

-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Thursday, October 15, 2020 12:10 PM
To: Liu, Yong <[email protected]>; [email protected]
Cc: Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; Yigit, Ferruh <[email protected]>; [email protected]; David Christensen <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

<snip>

Quoted 6 lines
Quoted 6 lines

23/09/2020 03:51, Marvin Liu:

Quoted 4 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as

default.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

---
+#define RTE_PMD_PACKET_PREFETCH 1

We could also remove the related #ifdefs.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

What can be the drawback of always enable those prefetches?

Hi Thomas,
I think the potential drawback is that current prefetch location cannot guarantee the best performance across different platforms.

Then, does it make sense to enable this by default?

Now most of prefetch actions are placed after pointer of data is valid. I think this methodology can benefit all platforms. It's hard to say that it’s the best choice for all. But no more better solution in my mind.
At least, we need to allow user to enable packet data prefetch.

Regards,
Marvin

Quoted 7 lines
Quoted 7 lines

Each developer has tuned the performance by adding prefetch instruction and verified the result on himself platform.
So prefetch location is based on certain platform, also it will be hard for developer to compare the results across platforms.

Thanks,
Marvin

#8 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

15/10/2020 10:23, Liu, Yong:

Quoted 20 lines

From: Honnappa Nagarahalli <[email protected]>

Quoted 19 lines
Quoted 5 lines
Quoted 5 lines

23/09/2020 03:51, Marvin Liu:

Quoted 4 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as

default.

Quoted 13 lines
Quoted 9 lines
Quoted 4 lines

---
+#define RTE_PMD_PACKET_PREFETCH 1

We could also remove the related #ifdefs.

What can be the drawback of always enable those prefetches?

Hi Thomas,
I think the potential drawback is that current prefetch location cannot guarantee the best performance across different platforms.

Quoted 6 lines
Quoted 2 lines

Then, does it make sense to enable this by default?

Now most of prefetch actions are placed after pointer of data is valid. I think this methodology can benefit all platforms. It's hard to say that it’s the best choice for all. But no more better solution in my mind.
At least, we need to allow user to enable packet data prefetch.

In my opinion, it can be tested and measured.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Each developer has tuned the performance by adding prefetch instruction and verified the result on himself platform.
So prefetch location is based on certain platform, also it will be hard for developer to compare the results across platforms.

If it shows benefit on an architecture, then it should be enabled with #ifdef RTE_ARCH_XX

I am for removing the option RTE_PMD_PACKET_PREFETCH.

#9 lore raw

Re: [dpdk-dev] [PATCH v2] config: enable packet data prefetch

<snip>

Quoted 25 lines

15/10/2020 10:23, Liu, Yong:

Quoted 24 lines

From: Honnappa Nagarahalli <[email protected]>

Quoted 22 lines
Quoted 6 lines
Quoted 6 lines

23/09/2020 03:51, Marvin Liu:

Quoted 5 lines

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtualized data paths like virtio utilized this feature for acceleration. Since most modern cpus have support prefetch function, we can enable packet data prefetch as

default.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

---
+#define RTE_PMD_PACKET_PREFETCH 1

Quoted 8 lines
Quoted 4 lines

We could also remove the related #ifdefs.

What can be the drawback of always enable those prefetches?

Hi Thomas,
I think the potential drawback is that current prefetch location cannot guarantee the best performance across different platforms.

Then, does it make sense to enable this by default?

Quoted 7 lines
Quoted 1 line

Now most of prefetch actions are placed after pointer of data is valid. I

think this methodology can benefit all platforms.

Quoted 1 line

It's hard to say that it’s the best choice for all. But no more better solution

in my mind.

Quoted 1 line

At least, we need to allow user to enable packet data prefetch.

In my opinion, it can be tested and measured.

+ Joyce, to test this for VirtIO on Arm

Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Each developer has tuned the performance by adding prefetch instruction and verified the result on himself platform. So prefetch location is based on certain platform, also it will be hard for developer to compare the results across platforms.

If it shows benefit on an architecture, then it should be enabled with #ifdef RTE_ARCH_XX

I am for removing the option RTE_PMD_PACKET_PREFETCH.

author
#10 lore raw

PATCH v3 1/1 drivers/net: enable packet data prefetch on x86

In reply to Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Data paths like e1000 and virtio utilized this feature for packet data access acceleration. Enabled packet data prefetch on x86 architecture, as prefetch instruction has been supported from very early generation.

---
v3: replace build config with pre-defined architecture macro
v2: move define from meson.build to rte_config.h
---

drivers/net/e1000/em_rxtx.c +1 -1
index 19e3bffd46..5a9cd04c9c 100644
@@ -185,7 +185,7 @@ struct em_tx_queue {
 #define rte_em_prefetch(p)	do {} while(0)
 #endif
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p) rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)	do {} while(0)
drivers/net/e1000/igb_rxtx.c +1 -1
index dd520cd82c..45d4ee0b92 100644
@@ -198,7 +198,7 @@ struct igb_tx_queue {
 #define rte_igb_prefetch(p)	do {} while(0)
 #endif
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p) rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)	do {} while(0)
drivers/net/enic/enic_rxtx.c +1 -1
index 6a8718c086..671c32038b 100644
@@ -25,7 +25,7 @@
 #define rte_enic_prefetch(p) do {} while (0)
 #endif
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p) rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p) do {} while (0)
drivers/net/fm10k/fm10k_rxtx.c +1 -1
index 4accaa2cd6..4bc3afadd1 100644
@@ -10,7 +10,7 @@
 #include "fm10k.h"
 #include "base/fm10k_type.h"
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while (0)
drivers/net/igc/igc_txrx.c +1 -1
index 4654ec41f0..b8b504738e 100644
@@ -16,7 +16,7 @@
 #define rte_igc_prefetch(p)		do {} while (0)
 #endif
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)		rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)		do {} while (0)
drivers/net/ixgbe/ixgbe_rxtx.h +1 -1
index 6d2f7c9da3..4d39de9531 100644
@@ -41,7 +41,7 @@
 #define RX_RING_SZ ((IXGBE_MAX_RING_DESC + RTE_PMD_IXGBE_RX_MAX_BURST) * \
 		    sizeof(union ixgbe_adv_rx_desc))
 
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)
drivers/net/virtio/virtqueue.h +1 -1
index 42c4c9882f..0196290a5d 100644
@@ -106,7 +106,7 @@ virtqueue_store_flags_packed(struct vring_packed_desc *dp,
 		dp->flags = flags;
 	}
 }
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)
Signature · 1 line
2.17.1