Skip to contentSkip to conversation

net/virtio: enable packet data prefetch on x86

PATCH In review Marvin Liu 4 participants 9 messages Nov 11, 2020 – Nov 13, 2020 Network drivers

lore
Share & tools
lore

Marvin Liu

author
#0 lore raw

PATCH 1/1 net/virtio: enable packet data prefetch on x86

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

drivers/net/virtio/virtqueue.h +1 -1
index 42c4c9882..0196290a5 100644
@@ -106,7 +106,7 @@ virtqueue_store_flags_packed(struct vring_packed_desc *dp,
 		dp->flags = flags;
 	}
 }
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)
Signature · 1 line
2.17.1
#1 lore raw
Quoted 5 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

IIUC, this config item was set for all architectures under make compilation.

$ git grep RTE_PMD_PACKET_PREFETCH v20.08 config/
v20.08:config/common_base:CONFIG_RTE_PMD_PACKET_PREFETCH=y

Now that we switched to meson, it got lost and this patch only restores it for x86.
Can other architectures check this?

Thanks.

Signature · 1 line
David Marchand
#2 lore raw
Quoted 7 lines · David Marchand
Quoted 5 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

Quoted 9 lines

IIUC, this config item was set for all architectures under make compilation.

$ git grep RTE_PMD_PACKET_PREFETCH v20.08 config/
v20.08:config/common_base:CONFIG_RTE_PMD_PACKET_PREFETCH=y

Now that we switched to meson, it got lost and this patch only restores it for x86.
Can other architectures check this?

If it was globally enabled before, it probably should just be added to config/rte_config.h file.

/Bruce

#3 lore raw
Quoted 14 lines · Bruce Richardson
Quoted 12 lines · David Marchand
Quoted 5 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

IIUC, this config item was set for all architectures under make compilation.

$ git grep RTE_PMD_PACKET_PREFETCH v20.08 config/
v20.08:config/common_base:CONFIG_RTE_PMD_PACKET_PREFETCH=y
Quoted 7 lines
Quoted 4 lines

Now that we switched to meson, it got lost and this patch only restores it for x86.
Can other architectures check this?

If it was globally enabled before, it probably should just be added to config/rte_config.h file.

Restoring globally is the probably the best fix, yes.
I am still surprised nobody but x86 testers caught a perf regression.

Signature · 1 line
David marchand
#4 lore raw
In reply to Marvin Liu

Hi Marvin,

Quoted 13 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

drivers/net/virtio/virtqueue.h +0 -0
index 42c4c9882..0196290a5 100644

drivers/net/virtio/virtqueue.h virtqueue_store_flags_packed(struct vring_packed_desc *dp, · line 112

Quoted 8 lines
drivers/net/virtio/virtqueue.h +1 -1
@@ -106,7 +106,7 @@ virtqueue_store_flags_packed(struct vring_packed_desc *dp,
 		dp->flags = flags;
 	}
 }
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)

Thanks for catching this issue.
I agree it should be re-enabled by default, and not only on X86, not only on Virtio PMD.

AFAICS, prefetch was enabled for all platforms before the switch to Meson, so I see it as an involuntary change that needs to be reverted.

Then, I think having a possibility to disable it would be nice, so maybe we should add an option in our build system to do that.

Thanks,
Maxime

#5 lore raw
Quoted 27 lines · Maxime Coquelin
Quoted 13 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

drivers/net/virtio/virtqueue.h +0 -0
index 42c4c9882..0196290a5 100644

drivers/net/virtio/virtqueue.h virtqueue_store_flags_packed(struct vring_packed_desc *dp, · line 112

Quoted 8 lines
drivers/net/virtio/virtqueue.h +1 -1
@@ -106,7 +106,7 @@ virtqueue_store_flags_packed(struct vring_packed_desc *dp,
              dp->flags = flags;
      }
 }
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)

Thanks for catching this issue.
I agree it should be re-enabled by default, and not only on X86, not only on Virtio PMD.

Quoted 2 lines

AFAICS, prefetch was enabled for all platforms before the switch to Meson, so I see it as an involuntary change that needs to be reverted.

Yes and this will also solve https://patchwork.dpdk.org/patch/83468/.
Thanks.

Signature · 1 line
David Marchand
author
#6 lore raw
Quoted 43 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Thursday, November 12, 2020 4:58 PM
To: Liu, Yong <[email protected]>; Maxime Coquelin <[email protected]>
Cc: Xia, Chenbo <[email protected]>; dev <[email protected]>; Richardson, Bruce <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] net/virtio: enable packet data prefetch on x86

Quoted 9 lines · Maxime Coquelin

Review

Quoted 8 lines · Marvin Liu

Data prefetch instruction can preload data into cpu’s hierarchical cache before data access. Virtio datapath utilized this feature for data access acceleration. As config RTE_PMD_PACKET_PREFETCH was discarded, now packet data prefetch is enabled based on architecture.

(quoted diff) +0 -0

b/drivers/net/virtio/virtqueue.h

Quoted 4 lines

drivers/net/virtio/virtqueue.h virtqueue_store_flags_packed(struct

Quoted 4 lines

index 42c4c9882..0196290a5 100644

drivers/net/virtio/virtqueue.h +0 -0
@@ -106,7 +106,7 @@ virtqueue_store_flags_packed(struct

vring_packed_desc *dp,

Quoted 14 lines

drivers/net/virtio/virtqueue.h virtqueue_store_flags_packed(struct vring_packed_desc *dp, · line 112

Quoted 9 lines
drivers/net/virtio/virtqueue.h +1 -1
              dp->flags = flags;
      }
 }
-#ifdef RTE_PMD_PACKET_PREFETCH
+#if defined(RTE_ARCH_X86)
 #define rte_packet_prefetch(p)  rte_prefetch1(p)
 #else
 #define rte_packet_prefetch(p)  do {} while(0)

Thanks for catching this issue.
I agree it should be re-enabled by default, and not only on X86, not only on Virtio PMD.

Quoted 6 lines
Quoted 2 lines

AFAICS, prefetch was enabled for all platforms before the switch to Meson, so I see it as an involuntary change that needs to be reverted.

Yes and this will also solve https://patchwork.dpdk.org/patch/83468/.
Thanks.

Agreed, original patch was intended to recover prefetch configuration in meson build.
Please check http://patchwork.dpdk.org/patch/78451/.
And it leaded a discussion about how to utilize prefetch function optimally.
Due to no conclusion for current position is best for other platforms except x86, now only enable prefetch in virtio + x86.

Quoted 2 lines

--
David Marchand

#7 lore raw
Quoted 8 lines · Yong
Quoted 3 lines

Yes and this will also solve https://patchwork.dpdk.org/patch/83468/.
Thanks.

Agreed, original patch was intended to recover prefetch configuration in meson build.
Please check http://patchwork.dpdk.org/patch/78451/.
And it leaded a discussion about how to utilize prefetch function optimally.
Due to no conclusion for current position is best for other platforms except x86, now only enable prefetch in virtio + x86.

I disagree.
No conclusion means the best is to restore the previous state, i.e. enable this option for all platforms.

If later other architectures want to change this, this can revisit.

Signature · 1 line
David Marchand
author
#8 lore raw

+ more people into this conversation. IMHO, restore to previous state is the best choice by now.

Quoted 32 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, November 13, 2020 3:27 PM
To: Liu, Yong <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; dev <[email protected]>; Richardson, Bruce <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] net/virtio: enable packet data prefetch on x86

Quoted 5 lines · Yong
Quoted 3 lines

Yes and this will also solve https://patchwork.dpdk.org/patch/83468/.
Thanks.

Agreed, original patch was intended to recover prefetch configuration in

meson build.

Quoted 2 lines

Please check http://patchwork.dpdk.org/patch/78451/.
And it leaded a discussion about how to utilize prefetch function

optimally.

Quoted 1 line

Due to no conclusion for current position is best for other platforms

except x86, now only enable prefetch in virtio + x86.

I disagree.
No conclusion means the best is to restore the previous state, i.e. enable this option for all platforms.

If later other architectures want to change this, this can revisit.

--
David Marchand