Skip to contentSkip to conversation

ethdev: an API for cache stashing hints

RFC In review Wathsala Vithanage 13 participants 64 messages Jul 15, 2024 – Apr 14

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC v2 1/1 ethdev: an API for cache stashing hints

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions.  In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

---
 .mailmap                   |   1 +
 lib/ethdev/ethdev_driver.h |  67 +++++++++++
 lib/ethdev/rte_ethdev.c    | 153 +++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 225 +++++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |   6 +
 5 files changed, 452 insertions(+)
.mailmap +1 -0
index f1e64286a1..9c28b74655 100644
@@ -338,6 +338,7 @@ Dexia Li <[email protected]>
 Dexuan Cui <[email protected]>
 Dharmik Thakkar <[email protected]> <[email protected]>
 Dheemanth Mallikarjun <[email protected]>
+Dhruv Tripathi <[email protected]>
 Diana Wang <[email protected]>
 Didier Pallard <[email protected]>
 Dilshod Urazov <[email protected]>
lib/ethdev/ethdev_driver.h +67 -0

79 lines not on this page · raw message

lib/ethdev/rte_ethdev.c +153 -0

163 lines not on this page · raw message

lib/ethdev/rte_ethdev.h +225 -0

243 lines not on this page · raw message

lib/ethdev/version.map +6 -0
index 79f6f5293b..5eef0b4540 100644
@@ -325,6 +325,12 @@ EXPERIMENTAL {
 	rte_flow_template_table_resizable;
 	rte_flow_template_table_resize;
 	rte_flow_template_table_resize_complete;
+
+	# added in 24.07
+	rte_eth_dev_stashing_hints_rx;
+	rte_eth_dev_stashing_hints_tx;
+	rte_eth_dev_stashing_hints_discover;
+	rte_eth_dev_validate_stashing_hints;
 };
 
 INTERNAL {
Signature · 1 line
2.34.1
#1 lore raw
Quoted 76 lines · On Mon, 15 Jul 2024 22:11:41 +0000 Wathsala Vithanage…

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions.  In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

Quoted 2 lines

My initial reaction is negative on this. The DPDK does not need more nerd knobs for performance. If it is a performance win, it should be automatic and handled by the driver.

If you absolutely have to have another flag, then it should be in existing config (yes, extend the ABI) rather than adding more flags and calls in ethdev.

#2 lore raw
Quoted 7 lines

My initial reaction is negative on this. The DPDK does not need more nerd knobs for performance. If it is a performance win, it should be automatic and handled by the driver.

If you absolutely have to have another flag, then it should be in existing config (yes, extend the ABI) rather than adding more flags and calls in ethdev.

Thanks, Steve, for the feedback. My thesis is that in a DPDK-based packet processing system, the application is more knowledgeable of memory buffer (packets) usage than the generic underlying hardware or the PMD (I have provided some examples below with the hint they would map into). Recognizing such cases, PCI SIG introduced TLP Packet Processing Hints (TPH). Consequently, many interconnect designers enabled support for TPH in their interconnects so that based on steering tags provided by an application to a NIC, which sets them in the TLP header, memory buffers can be targeted toward a CPU at the desired level in the cache hierarchy. With this proposed API, applications provide cache-stashing hints to ethernet devices to improve memory access latencies from the CPU and the NIC to improve system performance.

Listed below are some use cases.

- A run-to-completion application may not need the next packet immediately in L1D. It may rather issue a prefetch and do other work with packet and application data already in L1D before it needs the next packet. A generic PMD will not know such subtleties in the application endpoint, and it would resolve to stash buffers into the L1D indiscriminately or not do it at all. But, with a hint from the application that buffers of the packets will be stashed at a cache level suitable for the application. (like UNIX MADV_DONOTNEED but for mbufs at cache line granularity)

- Similarly, a pipelined application may use a hint that advice the buffers are needed in L1D as soon as they arrive. (parallels MADV_WILLNEED)

- Let's call the time between a mbuf being allocated into an Rx queue, freed back into mempool in the Tx path, and once again reallocated back in the Same Rx queue the "buffer recycle window". The length of the buffer recycle window is a function of the application in question; the PMD or the NIC has no prior knowledge of this property of an application. A buffer may stay in the L1D of a CPU throughout the entire recycle window if the window is short enough for that application. An application with a short buffer recycle window may hint to the platform that the Tx buffer is not needed anytime soon in the CPU cache via a hint to avoid unnecessary cache invalidations when the buffer gets written by the Rx packet for the second time. (parallels MADV_DONOTNEED)

#3 lore raw
In reply to Stephen Hemminger
Quoted 83 lines · Stephen Hemminger
Quoted 76 lines · On Mon, 15 Jul 2024 22:11:41 +0000 Wathsala Vithanage…

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions. In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

Quoted 7 lines
Quoted 2 lines

My initial reaction is negative on this. The DPDK does not need more nerd knobs for performance. If it is a performance win, it should be automatic and handled by the driver.

IMO, DPDK provides low level APIs and they should provide flexibility for users to control what part of the data from NIC is stashed where. For ex: currently available systems across multiple architectures provide system wide configuration to control stashing data from the NIC to system cache. The configuration allows for all the data from NIC to be stated or none. Whereas some applications need access to just the headers and some others need access to all the packet data.

Quoted 2 lines

If you absolutely have to have another flag, then it should be in existing config (yes, extend the ABI) rather than adding more flags and calls in ethdev.

Agree. Extending the ABI would result in a better solution rather than another set of APIs.

#4 lore raw
In reply to Wathsala Vithanage
Quoted 69 lines

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions.  In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that

Quoted 8 lines

intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

Sounds like an interesting idea...
Do you plan to have a reference implementation in one (or few) actual PMDs?

#5 lore raw
In reply to Wathsala Vithanage
Quoted 76 lines · Wathsala Vithanage

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions.  In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

Quoted 3 lines

This is a fine grained config for performance improvement, it may help to see the performance impact and driver implementation complexity, before deciding how practical it is.
As ethdev API, as long as it is separate set of APIs, I don't see much concern to have them.

In existing FEC APIs, and now in the speed lane patchset [1], we are following similar design with three APIs:
rte_eth_X_set()
rte_eth_X_get()
rte_eth_X_get_capability()

Instead of adding RTE_ETH_DEV_CAPA_ macro and contaminating 'rte_eth_dev_info' with this edge use case, what do you think follow above design and have dedicated get capability API?

And I can see set() has two different APIs, 'rte_eth_dev_stashing_hints_rx' & 'rte_eth_dev_stashing_hints_tx', is there a reason to have two separate APIs instead of having one which gets RX & TX as argument, as done in internal device ops?

[1]
https://patches.dpdk.org/project/dpdk/list/?series=32450&state=*

#6 lore raw
Quoted 2 lines

rte_eth_X_get_capability()

rte_eth_dev_stashing_hints_discover is somewhat similar.

Quoted 3 lines

Instead of adding RTE_ETH_DEV_CAPA_ macro and contaminating 'rte_eth_dev_info' with this edge use case, what do you think follow above design and have dedicated get capability API?

I think it's better to have a dedicated API, given that we already have a fine grained capabilities discovery function. I will add this feedback to V3 of the RFC.

Quoted 5 lines

And I can see set() has two different APIs, 'rte_eth_dev_stashing_hints_rx' & 'rte_eth_dev_stashing_hints_tx', is there a reason to have two separate APIs instead of having one which gets RX & TX as argument, as done in internal device ops?

Some types/hints may only apply to a single queue direction, so I thought it would be better to separate them out into separate Rx and Tx APIs for ease of comprehension/use for the developer.
In fact, underneath, it uses one API for both Rx and Tx.

#7 lore raw
Quoted 19 lines · Wathsala Wathawana Vithanage
Quoted 2 lines

rte_eth_X_get_capability()

rte_eth_dev_stashing_hints_discover is somewhat similar.

Quoted 3 lines

Instead of adding RTE_ETH_DEV_CAPA_ macro and contaminating 'rte_eth_dev_info' with this edge use case, what do you think follow above design and have dedicated get capability API?

I think it's better to have a dedicated API, given that we already have a fine grained capabilities discovery function. I will add this feedback to V3 of the RFC.

Quoted 5 lines

And I can see set() has two different APIs, 'rte_eth_dev_stashing_hints_rx' & 'rte_eth_dev_stashing_hints_tx', is there a reason to have two separate APIs instead of having one which gets RX & TX as argument, as done in internal device ops?

Quoted 5 lines

Some types/hints may only apply to a single queue direction, so I thought it would be better to separate them out into separate Rx and Tx APIs for ease of comprehension/use for the developer.
In fact, underneath, it uses one API for both Rx and Tx.

Hi Wathsala,

Do you still pursue this RFC, should we expect a new version for this release?

Did you have any change to measure the impact of the changes in this patch?

Btw, do you think the LLC aware lcore selection patch [1] can be relevant or can it help for the cases this patch addresses?

[1]
https://patches.dpdk.org/project/dpdk/list/?series=32851

#8 lore raw
Quoted 14 lines · On Sun, 22 Sep 2024 22:43:55 +0100 Ferruh Yigit <[email protected]> wrote:
Quoted 11 lines
Quoted 5 lines

And I can see set() has two different APIs, 'rte_eth_dev_stashing_hints_rx' & 'rte_eth_dev_stashing_hints_tx', is there a reason to have two separate APIs instead of having one which gets RX & TX as argument, as done in internal device ops?

Some types/hints may only apply to a single queue direction, so I thought it would be better to separate them out into separate Rx and Tx APIs for ease of comprehension/use for the developer.
In fact, underneath, it uses one API for both Rx and Tx.

Hi Wathsala,

Quoted 8 lines

Do you still pursue this RFC, should we expect a new version for this release?

Did you have any change to measure the impact of the changes in this patch?

Btw, do you think the LLC aware lcore selection patch [1] can be relevant or can it help for the cases this patch addresses?

Don't think this is ready for 24.11 release. The patch fails multiple tests, has some doc issues and would need more exmaples and support.

Since it requires an ABI change, if you want to pursue it further send a new version and maybe it will be ready for next ABI breaking release 25.11.

#9 lore raw
Quoted 24 lines
Quoted 14 lines
Quoted 11 lines
Quoted 5 lines

And I can see set() has two different APIs, 'rte_eth_dev_stashing_hints_rx' & 'rte_eth_dev_stashing_hints_tx', is there a reason to have two separate APIs instead of having one which gets RX & TX as argument, as done in internal device ops?

Some types/hints may only apply to a single queue direction, so I thought it would be better to separate them out into separate Rx and Tx APIs for ease of comprehension/use for the developer. In fact, underneath, it uses one API for both Rx and Tx.

Hi Wathsala,

Quoted 8 lines

Do you still pursue this RFC, should we expect a new version for this release?

Did you have any change to measure the impact of the changes in this patch?

Btw, do you think the LLC aware lcore selection patch [1] can be relevant or can it help for the cases this patch addresses?

Quoted 5 lines

Don't think this is ready for 24.11 release. The patch fails multiple tests, has some doc issues and would need more exmaples and support.

Since it requires an ABI change, if you want to pursue it further send a new version and maybe it will be ready for next ABI breaking release 25.11.

I agree, I don't think it will be ready for 24.11 release. I will be sending out another version of this in the coming days.
As we discussed in the summit, this also requires some work in the kernel to make it functional.

#10 lore raw

v3 cover An API for Stashing Packets into CPU caches

In reply to Wathsala Vithanage

DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library, PCI library, and the PCI driver. In this design, the application via the ethdev stashing API provides hints to the PMD to indicate the underlying hardware at which processor and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination, it must extract the matching ST from the TPH ACPI _DSM of the PCIe root port to which the NIC is connected. To facilitate the extraction of STs, the PCI library and the PCI driver APIs are extended.

PMD's implementation of eth_dev_ops stashing_rx_hints_set and stashing_tx_hints_set function pointers are responsible for extracting the ST. The PCI bus driver provides the generic TPH ST extraction API that can be used by any PMD that drives a PCIe device. The extraction process begins by calling rte_pci_extract_tph_st() function in drivers/bus/pci/rte_bus_pci.h, which takes an initialized input object rte_tph_acpi__dsm_args and a pointer to rte_tph_acpi__dsm_return to store the ST returned by the TPH _DSM. rte_tph_acpi__dsm_arg and rte_tph_acpi__dsm_return objects are defined in lib/pci/rte_pci_tph.h as defined by the PCIe firmware specification and the associated ECN titled "Revised _DSM for Cache Locality TPH Features". The helper function rte_init_tph_acpi__dsm_args is used by the rte_pci_extract_tph_st() to convert lcore_id and cache_level provided by the PMD into well-formatted rte_tph_acpi__dsm_args. The processor or, in some cases, a container ID (which is synonymous with a core complex of a chiplet die) and the cache level in the rte_tph_acpi__dsm_args structure are not the same as the lcore_id and the cache_level provided by the application to the ethdev library, which PMD passes down to the rte_pci_extract_st() function. The rte_init_tph_acpi__dsm_args helper converts lcore_id to an APIC processor-id or a PPTT processor-container-id if the container of the lcore_id was requested as the target by the application. Similarly, it must convert cache_level to a PPTT cache-reference-id. These conversions are possible with the hwloc library or some other library DPDK may eventually provide. However, DPDK cannot execute the TPH _DSM directly, as it can only be done with kernel privileges. Therefore, appropriate mechanisms must be established in supported Operating Systems(Linux, FreeBSD, and Windows) to expose the _DSM return for a given argument. For instance, on Linux, this mechanism could be sysfs. Therefore, the implementation of rte_pci_extract_tph_st() is done in OS-specific files drivers/bus/pci/{bsd, linux, windows}/pci.c.

Once the ST is acquired from the OS-specific method described earlier, the stashing_rx_hints_set/stashing_tx_hints_set PMD implementations are ready to set the ST. As per PCIe specification, hints can be put on the MSI-X tables or using a device-specific method. Considering this, many NICs that support TPH allow setting steering tags and processing hints on the device's MSI-X table and queue contexts. For PMDs, setting the ST on queue contexts is the only viable method of using TPH. Therefore, the DPDK can only support setting ST in queue contexts. An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a processor cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP and the application should consider any values stored in the objects pointer invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_rx_config_set() and rte_eth_dev_stashing_tx_config_set() ethdev library function respectively. These functions execute the rte_pci_extract_tph_st() via eth_dev_ops pointers stashing_rx_hints_set and stashing_tx_hints_set. Both the functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue-id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. It also has the field 'container' to indicate if the target should be the container of the processor specified by the lcore_id in a chiplet-based SoC. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. If the objects field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via stashing_rx_hints_set and stashing_tx_hints_set, function pointers in eth_dev_ops, respectively.

Wathsala Vithanage (2):
pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API

 drivers/bus/pci/bsd/pci.c     |  12 +++
 drivers/bus/pci/linux/pci.c   |  12 +++
 drivers/bus/pci/rte_bus_pci.h |  22 +++++
 drivers/bus/pci/version.map   |   3 +
 drivers/bus/pci/windows/pci.c |  14 +++
 lib/ethdev/ethdev_driver.h    |  66 ++++++++++++++
 lib/ethdev/rte_ethdev.c       | 120 ++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h       | 156 ++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map        |   4 +
 lib/pci/meson.build           |   2 +
 lib/pci/rte_pci.h             |   2 +
 lib/pci/rte_pci_tph.c         |  20 +++++
 lib/pci/rte_pci_tph.h         | 111 ++++++++++++++++++++++++
 13 files changed, 544 insertions(+)
 create mode 100644 lib/pci/rte_pci_tph.c
 create mode 100644 lib/pci/rte_pci_tph.h
Signature · 1 line
2.34.1
#11 lore raw

RFC v3 1/2 pci: introduce the PCIe TLP Processing Hints API

Extend the PCI driver and the library to extract the Steering Tag (ST) for a given Processor/Processor Container and Cache ID pair and validate a Processing Hint from a TPH _DSM associated with a root port device. The rte_pci_device structure passed into the rte_pci_extract_tph_st() function could be a device or a root port. If it's a device, the function should trace it back to the root port and use its TPH _DSM to extract STs. The implementation of rte_pci_extract_tph_st() is dependent on the operating system.

rte_pci_extract_tph_st() should also be supplied with a rte_tph_acpi__dsm_args, and a rte_tph_acpi__dsm_return structures. These two structures are defined in the PCI library and comply with the TPH _DSM argument and return encoding specified in the PCI firmware ECN titled "Revised _DSM for Cache Locality TPH Features.". Use of rte_init_tph_acpi__dsm_args() is recommended for initializing the rte_tph_acpi__dsm_args struct which is capable of converting lcore ID, the cache level into values understood by the ACPI _DSM function. rte_tph_acpi__dsm_return struct will be initialized with the values returned by the TPH _DSM; it is up to the caller to use these values per the device's capabilities.

---
 drivers/bus/pci/bsd/pci.c     |  12 ++++
 drivers/bus/pci/linux/pci.c   |  12 ++++
 drivers/bus/pci/rte_bus_pci.h |  22 +++++++
 drivers/bus/pci/version.map   |   3 +
 drivers/bus/pci/windows/pci.c |  14 +++++
 lib/pci/meson.build           |   2 +
 lib/pci/rte_pci.h             |   2 +
 lib/pci/rte_pci_tph.c         |  21 +++++++
 lib/pci/rte_pci_tph.h         | 111 ++++++++++++++++++++++++++++++++++
 9 files changed, 199 insertions(+)
 create mode 100644 lib/pci/rte_pci_tph.c
 create mode 100644 lib/pci/rte_pci_tph.h
drivers/bus/pci/bsd/pci.c +12 -0
index 2f88252418..a143cecf45 100644
@@ -639,3 +639,15 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+int
+rte_pci_extract_tph_st(const struct rte_pci_device *dev,
+		       const struct rte_tph_acpi__dsm_args *args,
+		       struct rte_tph_acpi__dsm_return *ret)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(args);
+	RTE_SET_USED(ret);
+	/* BSD doesn't support this feature yet! */
+	return -1;
+}
drivers/bus/pci/linux/pci.c +12 -0
index 9056035b33..dffb945462 100644
@@ -803,3 +803,15 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+int
+rte_pci_extract_tph_st(const struct rte_pci_device *dev,
+		       const struct rte_tph_acpi__dsm_args *args,
+		       struct rte_tph_acpi__dsm_return *ret)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(args);
+	RTE_SET_USED(ret);
+	/* Linux doesn't support this feature yet! */
+	return -1;
+}
drivers/bus/pci/rte_bus_pci.h +22 -0
index 19a7b15b99..a8167e9b4b 100644
@@ -312,6 +312,28 @@ void rte_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Extract steering tag from the ACPI TPH _DSM of the root port
+ * of the device is connected to.
+ *
+ * @param device
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param args
+ *   An initialized args object for the _DSM.
+ * @param ret
+ *   A pointer to a _DSM return object to store the extracted steering tag.
+ * @return
+ *   0 on success, -1 on error extracting the steeting tag.
+ */
+__rte_experimental
+int rte_pci_extract_tph_st(const struct rte_pci_device *device,
+			   const struct rte_tph_acpi__dsm_args *args,
+			   struct rte_tph_acpi__dsm_return *ret);
+
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/version.map +3 -0
index cd653de5ac..5c89f80c8e 100644
@@ -31,6 +31,9 @@ EXPERIMENTAL {
 	rte_pci_find_capability;
 	rte_pci_find_next_capability;
 	rte_pci_has_capability_list;
+
+	# added in 24.11
+	rte_pci_extract_tph_st;
 };
 
 INTERNAL {
drivers/bus/pci/windows/pci.c +14 -0
index 36e6f89093..761f714a18 100644
@@ -500,3 +500,17 @@ rte_pci_scan(void)
 
 	return ret;
 }
+
+
+int
+rte_pci_extract_tph_st(const struct rte_pci_device *dev,
+		       const struct rte_tph_acpi__dsm_args *args,
+		       struct rte_tph_acpi__dsm_return *ret)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(args);
+	RTE_SET_USED(ret);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
lib/pci/meson.build +2 -0
index dd41cd5068..85e17c4257 100644
@@ -3,3 +3,5 @@
 
 sources = files('rte_pci.c')
 headers = files('rte_pci.h')
+headers = files('rte_pci_tph.h')
+headers = files('rte_pci_tph.c')
lib/pci/rte_pci.h +2 -0
index 9a50a12142..b7897640f1 100644
@@ -16,6 +16,8 @@
 #include <inttypes.h>
 #include <sys/types.h>
 
+#include <rte_pci_tph.h>
+
 #ifdef __cplusplus
 extern "C" {
 #endif
lib/pci/rte_pci_tph.c new file +21 -0
new file mode 100644
index 0000000000..3b0c7d4d97
@@ -0,0 +1,21 @@
+#include <errno.h>
+#include <rte_pci_tph.h>
+
+int
+rte_init_tph_acpi__dsm_args(uint16_t lcore_id, uint8_t type,
+			    uint8_t cache_level, uint8_t ph,
+			    struct rte_tph_acpi__dsm_args *args)
+{
+	RTE_SET_USED(lcore_id);
+	RTE_SET_USED(type);
+	RTE_SET_USED(cache_level);
+	RTE_SET_USED(ph);
+
+	if (!args)
+		return -EINVAL;
+	/* Use libhwloc or other mechanism provided by DPDK to
+	 * map lcore_id and cache_level to hardware IDs for
+	 * initializing args.
+	 */
+	return -ENOTSUP;
+}
lib/pci/rte_pci_tph.h new file +111 -0

111 lines not on this page · raw message

Signature · 1 line
2.34.1
#12 lore raw

Re: [RFC v3 1/2] pci: introduce the PCIe TLP Processing Hints API

Quoted 21 lines · On Mon, 21 Oct 2024 01:52:45 +0000 Wathsala Vithanage…

Extend the PCI driver and the library to extract the Steering Tag (ST) for a given Processor/Processor Container and Cache ID pair and validate a Processing Hint from a TPH _DSM associated with a root port device. The rte_pci_device structure passed into the rte_pci_extract_tph_st() function could be a device or a root port. If it's a device, the function should trace it back to the root port and use its TPH _DSM to extract STs. The implementation of rte_pci_extract_tph_st() is dependent on the operating system.

rte_pci_extract_tph_st() should also be supplied with a rte_tph_acpi__dsm_args, and a rte_tph_acpi__dsm_return structures. These two structures are defined in the PCI library and comply with the TPH _DSM argument and return encoding specified in the PCI firmware ECN titled "Revised _DSM for Cache Locality TPH Features.". Use of rte_init_tph_acpi__dsm_args() is recommended for initializing the rte_tph_acpi__dsm_args struct which is capable of converting lcore ID, the cache level into values understood by the ACPI _DSM function. rte_tph_acpi__dsm_return struct will be initialized with the values returned by the TPH _DSM; it is up to the caller to use these values per the device's capabilities.

Quoted 3 lines

While doing review, noticed that patch has minor whitespace issue.

/home/shemminger/DPDK/main/.git/worktrees/stash/rebase-apply/patch:123: new blank line at EOF. +
warning: 1 line adds whitespace errors.

#13 lore raw

RFC v3 2/2 ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 120 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 161 +++++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |   4 +
 4 files changed, 351 insertions(+)
lib/ethdev/ethdev_driver.h +66 -0
index 1fd4562b40..7caaea54a8 100644
@@ -1367,6 +1367,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -1393,6 +1455,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c +120 -0

130 lines not on this page · raw message

lib/ethdev/rte_ethdev.h +161 -0

179 lines not on this page · raw message

lib/ethdev/version.map +4 -0
index 12f48c70a0..49c8c46a00 100644
@@ -337,6 +337,10 @@ EXPERIMENTAL {
 	rte_eth_timesync_adjust_freq;
 	rte_flow_async_create_by_index_with_pattern;
 	rte_tm_node_query;
+	rte_eth_dev_stashing_rx_config_set;
+	rte_eth_dev_stashing_tx_config_set;
+	rte_eth_dev_stashing_capabilities_get;
+	rte_eth_dev_validate_stashing_config;
 };
 
 INTERNAL {
Signature · 1 line
2.34.1
#14 lore raw

RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 7 lines
lib/ethdev/rte_ethdev.h +7 -0
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET		0x0001
+
+/** Apply stashing hint to an rx descriptor. */

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6181

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+#define RTE_ETH_DEV_STASH_OBJECT_DESC		0x0002
+
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER		0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD	0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK		0x000f
+/**@}*/

Although I agree these would be sensible parameters, they seem orthogonal to the Ethdev RX port queue capabilities/configuration.

The RTE_ETH_DEV_STASH_OBJECT_DESC aligns well; no problem there.

How much of a packet is considered the "header"? L3, L4, Outer tunnel header, or both Outer and Inner headers? If it follows the configuration already established through the existing Ethdev APIs on the RX port queue, it aligns well too.

Now, it's the data part I'm wondering about:

If an RX port queue is configured for receiving into mbufs from one mempool containing 2 KB objects (i.e. segments of 2 KB contiguous RAM), how do the OFFSET, HEADER, and PAYLOAD hints work? And what is the corresponding Ethdev RX queue configuration?

Same questions, considering an RX port queue configured for receiving the first 128 B into an mbuf from one mempool and the remaining part of the packet into 2 KB mbufs from another mempool?

Please provide some explanatory examples using these APIs along with the existing APIs for setting up the RX port queue.

Packets may be jumbo frames, scattered into multiple 2 KB mbufs. That should not make any difference; i.e. I assume the OFFSET, HEADER and PAYLOAD hints are considered at a packet level, not segment level.

I also assume IP fragments are treated like any other IP packets; basically, the only difference is the size of the header (it only has the IP header, and no following headers).

#15 lore raw

Re: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage
Quoted 90 lines · Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

lib/ethdev/rte_ethdev.h +53 -0
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE           0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE           1
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice
+ *
+ * A structure used for configuring the cache stashing hints.
+ */
+struct rte_eth_stashing_config {
+       /** ID of the Processor/Container the stashing hints are
+        *  applied to
+        */
+       uint16_t        lcore_id;
+       /** Set if the target is a CPU containeri.lcore_id will be
+        * used to derive container ID
+        */
+       uint16_t        container : 1;
+       uint16_t        padding : 7;
+       /** Cache level of the CPU specified by the cpu_id the
+        *  stashing hints are applied to
+        */
+       uint16_t        cache_level : 8;
+       /** Object types the configuration is applied to
+        */
+       uint16_t        objects;
+       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+        *  in objects
+        */
+       off_t           offset;
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the ethernet device
+ *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
+
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)                             \
+       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+

I think, at one of point of time, we need to extend this to other device class like(cryptodev etc)
where the data needs to move over bus. In that context, all the above symbols better to be in
EAL and the device class subsystem(example ethdev) gives PMD callback.

#16 lore raw

RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 45 lines

From: Jerin Jacob [mailto:[email protected]]
Sent: Thursday, 24 October 2024 07.49

Quoted 5 lines · Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

Quoted 8 lines

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the

underlying

Quoted 5 lines

hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types,

etc,

Quoted 7 lines

via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with

the

Quoted 11 lines

following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

Quoted 13 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE           0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE           1
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed,

without prior notice

Quoted 2 lines

+ *
+ * A structure used for configuring the cache stashing hints.

This structure can only describe one stashing hint. Please use singular, not plural, in its description.

Quoted 6 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6140

Quoted 6 lines

+ */

lib/ethdev/rte_ethdev.h +5 -0
+struct rte_eth_stashing_config {
+       /** ID of the Processor/Container the stashing hints are
+        *  applied to
+        */
+       uint16_t        lcore_id;

The common type used for lcore_id is "unsigned int", ref. e.g. rte_lcore_id() return value. Alternatively uint32_t, ref. LCORE_ID_ANY.

Quoted 8 lines
Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+       /** Set if the target is a CPU containeri.lcore_id will be
+        * used to derive container ID
+        */
+       uint16_t        container : 1;
+       uint16_t        padding : 7;
+       /** Cache level of the CPU specified by the cpu_id the
+        *  stashing hints are applied to
+        */
Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6156

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+       uint16_t        cache_level : 8;
+       /** Object types the configuration is applied to
+        */
+       uint16_t        objects;
+       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+        *  in objects
+        */
+       off_t           offset;

off_t is for files, ptrdiff_t is for memory.

Quoted 33 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the

ethernet device

Quoted 14 lines

+ *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */

lib/ethdev/rte_ethdev.h +10 -0
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
+

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)

\

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+
Quoted 5 lines

I think, at one of point of time, we need to extend this to other device class like(cryptodev etc)
where the data needs to move over bus. In that context, all the above symbols better to be in
EAL and the device class subsystem(example ethdev) gives PMD callback.

+1

When generalizing this, perhaps "header" and "payload" should be renamed to "device-specific".

For ethdevs, the typical meaning of "device-specific" would be splitting at some header (as suggested by the "header" and "payload" enum values).

Furthermore, for ethdevs, using a "device-specific" would allow the device to split at some other point, controlled through other ethdev APIs.
E.g. the split point could be controlled by rte_flow; this would allow rte_flow to put entire packets in L2 cache for some packet types, and only the packet header in L2 cache for some other packet types. (Someone at the conference call suggested combining Steering Tags with rte_flow - this might be a way of doing it.)

#17 lore raw

RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 82 lines

-----Original Message-----
From: Morten Brørup <[email protected]>
Sent: Thursday, October 24, 2024 2:00 AM
To: Jerin Jacob <[email protected]>; Wathsala Wathawana Vithanage <[email protected]>
Cc: [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Dhruv Tripathi <[email protected]>
Subject: RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 45 lines

From: Jerin Jacob [mailto:[email protected]]
Sent: Thursday, 24 October 2024 07.49

Quoted 5 lines · Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

Quoted 8 lines

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the

underlying

Quoted 5 lines

hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types,

etc,

Quoted 7 lines

via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with

the

Quoted 11 lines

following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

Quoted 13 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE           0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE           1
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed,

without prior notice

Quoted 2 lines

+ *
+ * A structure used for configuring the cache stashing hints.

This structure can only describe one stashing hint. Please use singular, not plural, in its description.

Quoted 6 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6140

Quoted 6 lines

+ */

lib/ethdev/rte_ethdev.h +5 -0
+struct rte_eth_stashing_config {
+       /** ID of the Processor/Container the stashing hints are
+        *  applied to
+        */
+       uint16_t        lcore_id;
Quoted 4 lines

The common type used for lcore_id is "unsigned int", ref. e.g. rte_lcore_id() return value.
Alternatively uint32_t, ref. LCORE_ID_ANY.

+1

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+       /** Set if the target is a CPU containeri.lcore_id will be
+        * used to derive container ID
+        */
+       uint16_t        container : 1;
+       uint16_t        padding : 7;
+       /** Cache level of the CPU specified by the cpu_id the
+        *  stashing hints are applied to
+        */
Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6156

Quoted 8 lines
lib/ethdev/rte_ethdev.h +8 -0
+       uint16_t        cache_level : 8;
+       /** Object types the configuration is applied to
+        */
+       uint16_t        objects;
+       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+        *  in objects
+        */
+       off_t           offset;
Quoted 2 lines

off_t is for files, ptrdiff_t is for memory.

+1

Quoted 47 lines
Quoted 30 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 3 lines
lib/ethdev/rte_ethdev.h +3 -0
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the

ethernet device

Quoted 15 lines

+ *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */

lib/ethdev/rte_ethdev.h +11 -0
+
+/**
+ * Apply stashing hint to data at a given offset from the start of
+a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
+

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)

\

Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+

I think, at one of point of time, we need to extend this to other device class like(cryptodev etc) where the data needs to move over bus. In that context, all the above symbols better to be in EAL and the device class subsystem(example ethdev) gives PMD callback.

+1

When generalizing this, perhaps "header" and "payload" should be renamed to "device-specific".

For ethdevs, the typical meaning of "device-specific" would be splitting at some header (as suggested by the "header" and "payload" enum values).

Quoted 7 lines

Furthermore, for ethdevs, using a "device-specific" would allow the device to split at some other point, controlled through other ethdev APIs. E.g. the split point could be controlled by rte_flow; this would allow rte_flow to put entire packets in L2 cache for some packet types, and only the packet header in L2 cache for some other packet types. (Someone at the conference call suggested combining Steering Tags with rte_flow - this might be a way of doing it.)

+1

#18 lore raw

RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

In reply to Jerin Jacob
Quoted 113 lines

-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Thursday, October 24, 2024 12:49 AM
To: Wathsala Wathawana Vithanage <[email protected]>
Cc: [email protected]; Ferruh Yigit <[email protected]>; Andrew Rybchenko <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Dhruv Tripathi <[email protected]>
Subject: Re: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 99 lines · Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

lib/ethdev/rte_ethdev.h +62 -0
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE           0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE           1
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without
+prior notice
+ *
+ * A structure used for configuring the cache stashing hints.
+ */
+struct rte_eth_stashing_config {
+       /** ID of the Processor/Container the stashing hints are
+        *  applied to
+        */
+       uint16_t        lcore_id;
+       /** Set if the target is a CPU containeri.lcore_id will be
+        * used to derive container ID
+        */
+       uint16_t        container : 1;
+       uint16_t        padding : 7;
+       /** Cache level of the CPU specified by the cpu_id the
+        *  stashing hints are applied to
+        */
+       uint16_t        cache_level : 8;
+       /** Object types the configuration is applied to
+        */
+       uint16_t        objects;
+       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+        *  in objects
+        */
+       off_t           offset;
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the
+ethernet device  *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
+
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)                             \
Quoted 8 lines

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 2 lines
lib/ethdev/rte_ethdev.h +2 -0
+       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+

I think, at one of point of time, we need to extend this to other device class like(cryptodev etc) where the data needs to move over bus. In that context, all the above symbols better to be in EAL and the device class subsystem(example ethdev) gives PMD callback.

+1
I will make that change in the RFC v4.

#19 lore raw

Re: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage
Quoted 116 lines · On Mon, 21 Oct 2024 01:52:46 +0000 Wathsala Vithanage…

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 120 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 161 +++++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |   4 +
 4 files changed, 351 insertions(+)
lib/ethdev/ethdev_driver.h +62 -0
index 1fd4562b40..7caaea54a8 100644
@@ -1367,6 +1367,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1464

Quoted 10 lines
lib/ethdev/ethdev_driver.h +4 -0
@@ -1393,6 +1455,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */

Since eth_dev_ops is visible in application binary, it is part of the ABI. Therefore it can not be changed until 25.11 release.

#20 lore raw

Re: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Quoted 125 lines · Stephen Hemminger
Quoted 121 lines · On Mon, 21 Oct 2024 01:52:46 +0000 Wathsala Vithanage…

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 120 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 161 +++++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |   4 +
 4 files changed, 351 insertions(+)
lib/ethdev/ethdev_driver.h +64 -0
index 1fd4562b40..7caaea54a8 100644
@@ -1367,6 +1367,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
                                          enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+                                        struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+                                        struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+                                          uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -1393,6 +1455,10 @@ struct eth_dev_ops {
      eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
      eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
      eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+     eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+     eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
Quoted 8 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1464

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
+     /** Get supported stashing hints*/
+     eth_stashing_capabilities_get_t stashing_capabilities_get;
      /** Set list of multicast addresses */
      eth_set_mc_addr_list_t     set_mc_addr_list;
      mtu_set_t                  mtu_set;       /**< Set MTU */

Since eth_dev_ops is visible in application binary, it is part of the ABI. Therefore it can not be changed until 25.11 release.

The layout of eth_dev_ops is not exposed to applications as it is in a private header.
Could you clarify where you see a breakage for an application?

I see an ABI breakage for out of tree drivers though.
This could be avoided by moving those added ops at the end of the struct?

Signature · 1 line
David Marchand
#21 lore raw

Re: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

Your right my test was crude. Just do build and look at symbol table of static linked binary.
I was confused since pointer is exposed but not data structure

Quoted 169 lines · On Thu, Dec 5, 2024, 07:40 David Marchand <[email protected]> wrote:
Quoted 46 lines · Stephen Hemminger
Quoted 42 lines · On Mon, 21 Oct 2024 01:52:46 +0000 Wathsala Vithanage…

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache (e.g., Steering Tags int PCIe TLP Processing Hints).

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 +++++++++++++++
 lib/ethdev/rte_ethdev.c    | 120 +++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 161 +++++++++++++++++++++++++++++++++++++
 lib/ethdev/version.map     |   4 +
 4 files changed, 351 insertions(+)
Quoted 7 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1368

Quoted 7 lines
lib/ethdev/ethdev_driver.h +0 -0
index 1fd4562b40..7caaea54a8 100644
@@ -1367,6 +1367,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
                                          enum rte_eth_dev_operation

op);

Quoted 14 lines
Quoted 4 lines
lib/ethdev/ethdev_driver.h +3 -0
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache

stashing.

Quoted 1 line

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1382

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

Quoted 4 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 4 lines
lib/ethdev/ethdev_driver.h +4 -0
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev,

uint16_t queue_id,

Quoted 1 line

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1387

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+                                        struct

rte_eth_stashing_config *config);

Quoted 6 lines
Quoted 6 lines
lib/ethdev/ethdev_driver.h +6 -0
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
Quoted 8 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 8 lines
lib/ethdev/ethdev_driver.h +8 -0
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache

stashing.

Quoted 1 line

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1402

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

Quoted 4 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 4 lines
lib/ethdev/ethdev_driver.h +4 -0
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev,

uint16_t queue_id,

Quoted 1 line

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1407

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+                                        struct

rte_eth_stashing_config *config);

Quoted 6 lines
Quoted 6 lines
lib/ethdev/ethdev_driver.h +6 -0
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
Quoted 8 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 8 lines
lib/ethdev/ethdev_driver.h +8 -0
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache

stashing.

Quoted 1 line

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1422

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

Quoted 2 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1424

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for

supported

Quoted 4 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 4 lines
lib/ethdev/ethdev_driver.h +4 -0
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev

*dev,

Quoted 4 lines

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 4 lines
lib/ethdev/ethdev_driver.h +2 -0
+                                          uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an

Ethernet driver.

Quoted 3 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1455

Quoted 3 lines
lib/ethdev/ethdev_driver.h +0 -0
  */
@@ -1393,6 +1455,10 @@ struct eth_dev_ops {
      eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC

address */

Quoted 1 line

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1456

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
      eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address

*/

Quoted 1 line

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1457

Quoted 1 line
lib/ethdev/ethdev_driver.h +0 -0
      eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address

*/

Quoted 1 line

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1458

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+     eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx

cache stashing*/

Quoted 1 line

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1459

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+     eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx

cache stashing*/

Quoted 7 lines

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1464

Quoted 5 lines
lib/ethdev/ethdev_driver.h +2 -0
+     /** Get supported stashing hints*/
+     eth_stashing_capabilities_get_t stashing_capabilities_get;
      /** Set list of multicast addresses */
      eth_set_mc_addr_list_t     set_mc_addr_list;
      mtu_set_t                  mtu_set;       /**< Set MTU */

Since eth_dev_ops is visible in application binary, it is part of the

ABI.

Quoted 1 line

Therefore it can not be changed until 25.11 release.

The layout of eth_dev_ops is not exposed to applications as it is in a private header.
Could you clarify where you see a breakage for an application?

I see an ABI breakage for out of tree drivers though.
This could be avoided by moving those added ops at the end of the struct?

--
David Marchand

#22 lore raw

Re: [RFC v3 0/2] An API for Stashing Packets into CPU caches

In reply to Wathsala Vithanage

Hi,

Quoted 129 lines · Wathsala Vithanage

External email: Use caution opening links or attachments

DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library, PCI library, and the PCI driver. In this design, the application via the ethdev stashing API provides hints to the PMD to indicate the underlying hardware at which processor and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination, it must extract the matching ST from the TPH ACPI _DSM of the PCIe root port to which the NIC is connected. To facilitate the extraction of STs, the PCI library and the PCI driver APIs are extended.

PMD's implementation of eth_dev_ops stashing_rx_hints_set and stashing_tx_hints_set function pointers are responsible for extracting the ST. The PCI bus driver provides the generic TPH ST extraction API that can be used by any PMD that drives a PCIe device. The extraction process begins by calling rte_pci_extract_tph_st() function in drivers/bus/pci/rte_bus_pci.h, which takes an initialized input object rte_tph_acpi__dsm_args and a pointer to rte_tph_acpi__dsm_return to store the ST returned by the TPH _DSM. rte_tph_acpi__dsm_arg and rte_tph_acpi__dsm_return objects are defined in lib/pci/rte_pci_tph.h as defined by the PCIe firmware specification and the associated ECN titled "Revised _DSM for Cache Locality TPH Features". The helper function rte_init_tph_acpi__dsm_args is used by the rte_pci_extract_tph_st() to convert lcore_id and cache_level provided by the PMD into well-formatted rte_tph_acpi__dsm_args. The processor or, in some cases, a container ID (which is synonymous with a core complex of a chiplet die) and the cache level in the rte_tph_acpi__dsm_args structure are not the same as the lcore_id and the cache_level provided by the application to the ethdev library, which PMD passes down to the rte_pci_extract_st() function. The rte_init_tph_acpi__dsm_args helper converts lcore_id to an APIC processor-id or a PPTT processor-container-id if the container of the lcore_id was requested as the target by the application. Similarly, it must convert cache_level to a PPTT cache-reference-id. These conversions are possible with the hwloc library or some other library DPDK may eventually provide. However, DPDK cannot execute the TPH _DSM directly, as it can only be done with kernel privileges. Therefore, appropriate mechanisms must be established in supported Operating Systems(Linux, FreeBSD, and Windows) to expose the _DSM return for a given argument. For instance, on Linux, this mechanism could be sysfs. Therefore, the implementation of rte_pci_extract_tph_st() is done in OS-specific files drivers/bus/pci/{bsd, linux, windows}/pci.c.

Once the ST is acquired from the OS-specific method described earlier, the stashing_rx_hints_set/stashing_tx_hints_set PMD implementations are ready to set the ST. As per PCIe specification, hints can be put on the MSI-X tables or using a device-specific method. Considering this, many NICs that support TPH allow setting steering tags and processing hints on the device's MSI-X table and queue contexts. For PMDs, setting the ST on queue contexts is the only viable method of using TPH. Therefore, the DPDK can only support setting ST in queue contexts. An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a processor cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP and the application should consider any values stored in the objects pointer invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_rx_config_set() and rte_eth_dev_stashing_tx_config_set() ethdev library function respectively. These functions execute the rte_pci_extract_tph_st() via eth_dev_ops pointers stashing_rx_hints_set and stashing_tx_hints_set. Both the functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue-id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. It also has the field 'container' to indicate if the target should be the container of the processor specified by the lcore_id in a chiplet-based SoC. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. If the objects field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via stashing_rx_hints_set and stashing_tx_hints_set, function pointers in eth_dev_ops, respectively.

Wathsala Vithanage (2):
pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API

drivers/bus/pci/bsd/pci.c     |  12 +++
drivers/bus/pci/linux/pci.c   |  12 +++
drivers/bus/pci/rte_bus_pci.h |  22 +++++
drivers/bus/pci/version.map   |   3 +
drivers/bus/pci/windows/pci.c |  14 +++
lib/ethdev/ethdev_driver.h    |  66 ++++++++++++++
lib/ethdev/rte_ethdev.c       | 120 ++++++++++++++++++++++++++
lib/ethdev/rte_ethdev.h       | 156 ++++++++++++++++++++++++++++++++++
lib/ethdev/version.map        |   4 +
lib/pci/meson.build           |   2 +
lib/pci/rte_pci.h             |   2 +
lib/pci/rte_pci_tph.c         |  20 +++++
lib/pci/rte_pci_tph.h         | 111 ++++++++++++++++++++++++
13 files changed, 544 insertions(+)
Quoted 6 lines

create mode 100644 lib/pci/rte_pci_tph.c
create mode 100644 lib/pci/rte_pci_tph.h

—
2.34.1

Do you have some numbers about how much performance this feature can improve?

/Chenbo

#23 lore raw

RE: [RFC v3 0/2] An API for Stashing Packets into CPU caches

Quoted 150 lines

-----Original Message-----
From: Chenbo Xia <[email protected]>
Sent: Monday, October 21, 2024 2:35 AM
To: Wathsala Wathawana Vithanage <[email protected]>
Cc: [email protected]; nd <[email protected]>
Subject: Re: [RFC v3 0/2] An API for Stashing Packets into CPU caches

Hi,

Quoted 1 line

On Oct 21, 2024, at 09:52, Wathsala Vithanage

Quoted 56 lines · <[email protected]> wrote:

External email: Use caution opening links or attachments

DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library, PCI library, and the PCI driver. In this design, the application via the ethdev stashing API provides hints to the PMD to indicate the underlying hardware at which processor and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination, it must extract the matching ST from the TPH ACPI _DSM of the PCIe root port to which the NIC is connected. To facilitate the extraction of STs, the PCI library and the PCI driver APIs are extended.

PMD's implementation of eth_dev_ops stashing_rx_hints_set and stashing_tx_hints_set function pointers are responsible for extracting the ST. The PCI bus driver provides the generic TPH ST extraction API that can be used by any PMD that drives a PCIe device. The extraction process begins by calling rte_pci_extract_tph_st() function in drivers/bus/pci/rte_bus_pci.h, which takes an initialized input object rte_tph_acpi__dsm_args and a pointer to rte_tph_acpi__dsm_return to store the ST returned by the TPH _DSM. rte_tph_acpi__dsm_arg and rte_tph_acpi__dsm_return objects are defined in lib/pci/rte_pci_tph.h as defined by the PCIe firmware specification and the associated ECN titled "Revised _DSM for Cache Locality TPH Features". The helper function rte_init_tph_acpi__dsm_args is used by the rte_pci_extract_tph_st() to convert lcore_id and cache_level provided by the PMD into well-formatted rte_tph_acpi__dsm_args. The processor or, in some cases, a container ID (which is synonymous with a core complex of a chiplet die) and the cache level in the rte_tph_acpi__dsm_args structure are not the same as the lcore_id and

Quoted 8 lines

the cache_level provided by the application to the ethdev library, which PMD passes down to the rte_pci_extract_st() function. The rte_init_tph_acpi__dsm_args helper converts lcore_id to an APIC processor-id or a PPTT processor-container-id if the container of the lcore_id was requested as the target by the application. Similarly, it must convert cache_level to a PPTT cache-reference-id. These conversions are possible with the hwloc library or some other library DPDK may eventually provide. However, DPDK cannot execute the TPH

_DSM

Quoted 2 lines

directly, as it can only be done with kernel privileges. Therefore, appropriate mechanisms must be established in supported Operating

Systems(Linux, FreeBSD, and Windows) to expose the _DSM return for a given argument.

Quoted 50 lines

For instance, on Linux, this mechanism could be sysfs. Therefore, the implementation of rte_pci_extract_tph_st() is done in OS-specific files drivers/bus/pci/{bsd, linux, windows}/pci.c.

Once the ST is acquired from the OS-specific method described earlier, the stashing_rx_hints_set/stashing_tx_hints_set PMD implementations are ready to set the ST. As per PCIe specification, hints can be put on the MSI-X tables or using a device-specific method. Considering this, many NICs that support TPH allow setting steering tags and processing hints on the device's MSI-X table and queue contexts. For PMDs, setting the ST on queue contexts is the only viable method of using TPH. Therefore, the DPDK can only support setting ST in queue contexts. An application uses the cache stashing ethdev API by first calling the
rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a processor cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP and the application should consider any values stored in the objects pointer invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_rx_config_set() and rte_eth_dev_stashing_tx_config_set() ethdev library function respectively. These functions execute the rte_pci_extract_tph_st() via eth_dev_ops pointers stashing_rx_hints_set and stashing_tx_hints_set. Both the functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue-id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. It also has the field 'container' to indicate if the target should be the container of the processor specified by the lcore_id in a chiplet-based SoC. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. If the objects field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via stashing_rx_hints_set and stashing_tx_hints_set, function pointers in eth_dev_ops, respectively.

Wathsala Vithanage (2):
pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API

Quoted 8 lines
drivers/bus/pci/bsd/pci.c     |  12 +++
drivers/bus/pci/linux/pci.c   |  12 +++
drivers/bus/pci/rte_bus_pci.h |  22 +++++
drivers/bus/pci/version.map   |   3 +
drivers/bus/pci/windows/pci.c |  14 +++
lib/ethdev/ethdev_driver.h    |  66 ++++++++++++++
lib/ethdev/rte_ethdev.c       | 120 ++++++++++++++++++++++++++
lib/ethdev/rte_ethdev.h       | 156

++++++++++++++++++++++++++++++++++

Quoted 9 lines
lib/ethdev/version.map        |   4 +
lib/pci/meson.build           |   2 +
lib/pci/rte_pci.h             |   2 +
lib/pci/rte_pci_tph.c         |  20 +++++
lib/pci/rte_pci_tph.h         | 111 ++++++++++++++++++++++++
13 files changed, 544 insertions(+)

create mode 100644 lib/pci/rte_pci_tph.c create mode 100644 lib/pci/rte_pci_tph.h

Quoted 7 lines
Quoted 3 lines

—
2.34.1

Do you have some numbers about how much performance this feature can improve?

This patch requires some additional work done in the Linux kernel to get it working. I'm planning to test this on a supported HW platform soon by hardcoding some STs. The TPH enablement patch in the kernel reports a significant improvement.
https://patchew.org/linux/[email protected]/
I hope it will improve performance in DPDK too.

Please join the call scheduled for 10/23/24 to discuss what we need in the OS to support this feature.
https://inbox.dpdk.org/dev/PAWPR08MB890901574A3113840E7D7CCC9F472@PAWPR08MB8909.eurprd08.prod.outlook.com/

Thanks.

--wathsala

#24 lore raw

Re: [RFC v3 0/2] An API for Stashing Packets into CPU caches

In reply to Wathsala Vithanage
Quoted 24 lines · On Mon, 21 Oct 2024 01:52:44 +0000 Wathsala Vithanage…

DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

Quoted 9 lines

To enable TPH and fine-grained packet stashing, this API extends the ethdev library, PCI library, and the PCI driver. In this design, the application via the ethdev stashing API provides hints to the PMD to indicate the underlying hardware at which processor and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination, it must extract the matching ST from the TPH ACPI _DSM of the PCIe root port to which the NIC is connected. To facilitate the extraction of STs, the PCI library and the PCI driver APIs are extended.

There is a fundamental conflict with the increasing growth of "nerd knobs" like this in the DPDK. Users already have problems understanding DPDK and adding more complexity does not help.

So any new feature like this should be:
1. Just work right without any configuration. It can't suck by default.

2. The API's should be used in the drivers and core, not exposed up to the application. Most of the hot data structures are in the drivers now.

3. Fit into existing API models. Like rte_prefetch().

Is the goal of DPDK enabling high speed applications, or enabling vendor benchmarks?

#25 lore raw

RE: [RFC v3 0/2] An API for Stashing Packets into CPU caches

Hi Stephen,

Quoted 7 lines

There is a fundamental conflict with the increasing growth of "nerd knobs" like this in the DPDK. Users already have problems understanding DPDK and adding more complexity does not help.

So any new feature like this should be:
1. Just work right without any configuration. It can't suck by default.

By default, this feature is disabled. It can be only enabled by calling the following at the queue setup time.
rte_eth_dev_stashing_rx_config_set
rte_eth_dev_stashing_tx_config_set

It's unlikely for someone not familiar with TPH to call these functions. The performance for them should be as good as without this feature.

Quoted 4 lines

2. The API's should be used in the drivers and core, not exposed up to the application. Most of the hot data structures are in the drivers now.

PMDs don't know which CPU and cache level to use with TPH. That information needs to be conveyed to the PMD, for it to work. Please suggest alternatives.

Quoted 2 lines

3. Fit into existing API models. Like rte_prefetch().

PCIe TPH is a hint from a PCIe device to the system interconnect to push data into CPU caches. I cannot think of an existing API that matches the semantics of TPH.
rte_prefetch() is a hint to the CPU from the application, something totally different.

Quoted 2 lines

Is the goal of DPDK enabling high speed applications, or enabling vendor benchmarks?

This is a vendor agnostic feature from the PCI-SIG implemented by almost
every hardware vendor in their NICs and SoCs.
FYI: Kernel patch - https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

#26 lore raw

Re: [RFC v3 0/2] An API for Stashing Packets into CPU caches

Quoted 7 lines · On Tue, 22 Oct 2024 18:37:09 +0000 Wathsala Wathawana Vithanage…
Quoted 4 lines

2. The API's should be used in the drivers and core, not exposed up to the application. Most of the hot data structures are in the drivers now.

PMDs don't know which CPU and cache level to use with TPH. That information needs to be conveyed to the PMD, for it to work. Please suggest alternatives.

It would be better if EAL had a representation of CPU and cache hierarchy which it built. Then have the PMD (if it cared) be able to query the topology. That way the application would not have to care.

The DPDK already leaks too many PCI details in the API.

#27 lore raw

v4 cover An API for Stashing Packets into CPU caches

In reply to Wathsala Vithanage

ethdev: an API for cache stashing hints

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function. The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument. Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (3):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API

 drivers/bus/pci/bsd/pci.c          |  39 +++++++
 drivers/bus/pci/bus_pci_driver.h   |  43 ++++++++
 drivers/bus/pci/linux/pci.c        |  94 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h   |   9 ++
 drivers/bus/pci/linux/pci_vfio.c   | 166 +++++++++++++++++++++++++++++
 drivers/bus/pci/private.h          |   8 ++
 drivers/bus/pci/rte_bus_pci.h      |  67 ++++++++++++
 drivers/bus/pci/windows/pci.c      |  39 +++++++
 kernel/linux/uapi/linux/vfio_tph.h | 100 +++++++++++++++++
 lib/ethdev/ethdev_driver.h         |  66 ++++++++++++
 lib/ethdev/rte_ethdev.c            | 149 ++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h            | 158 +++++++++++++++++++++++++++
 12 files changed, 938 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h

--
2.43.0

#28 lore raw

RFC PATCH v4 1/3 pci: add non-merged Linux uAPI changes

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions. This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel. These additions have not yet been merged into the upstream kernel.

---
 drivers/bus/pci/linux/pci_init.h   |   1 +
 kernel/linux/uapi/linux/vfio_tph.h | 100 +++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
drivers/bus/pci/linux/pci_init.h +1 -0
index a4d37c0d0a..25b901f460 100644
@@ -6,6 +6,7 @@
 #define EAL_PCI_INIT_H_
 
 #include <rte_vfio.h>
+#include <uapi/linux/vfio_tph.h>
 
 #include "private.h"
 
kernel/linux/uapi/linux/vfio_tph.h new file +100 -0

100 lines not on this page · raw message

Signature · 1 line
2.43.0
#29 lore raw

Re: [RFC PATCH v4 1/3] pci: add non-merged Linux uAPI changes

Hi Wathsala,

Quoted 7 lines · Wathsala Vithanage

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions. This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel. These additions have not yet been merged into the upstream kernel.

Could you provide a link to the ongoing discussions (ml archive maybe)? Thanks.

Signature · 1 line
David Marchand
#30 lore raw

RE: [RFC PATCH v4 1/3] pci: add non-merged Linux uAPI changes

Linux uAPI changes

Quoted 13 lines

Hi Wathsala,

Quoted 7 lines · Wathsala Vithanage

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions. This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel. These additions have not yet been merged into the upstream kernel.

Quoted 2 lines

Could you provide a link to the ongoing discussions (ml archive maybe)? Thanks.

Hi David,

I initially introduced this feature as a VFIO feature IOCTL. Maintainers didn't like that idea and favored a new IOCTL for TPH.
I will be sending out the V2 with that change sometime this week (pending review internally). I sent out the DPDK patch before so that I can link this patch to the cover letter of the kernel patch to show them that there is a real user-space use case for this feature.

V1 is now largely outdated here is the link
https://lore.kernel.org/kvm/[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Thanks.

--wathsala

#31 lore raw

RFC PATCH v4 2/3 bus/pci: introduce the PCIe TLP Processing Hints API

In reply to Wathsala Vithanage

Extend the PCI bus driver to enable or disable TPH capability and set or get PCI Steering-Tags (STs) on an endpoint device. The functions rte_pci_tph_{enable, disable,st_set,st_get} provide the primary interface for DPDK device drivers. Implementation of the interface is OS dependent. For Linux, the kernel VFIO driver provides the implementation. rte_pci_tph_{enable, disable} functions enable and disable TPH capability, respectively. rte_pci_tph_enable enables TPH on the device in either of the device-specific, interrupt-vector, or no-steering-tag modes.

rte_pci_tph_st_{get, set} functions take an array of rte_tph_info objects with cpu-id, cache-level, flags (processing-hint, memory-type). The index in rte_tph_info is the MSI-X/MSI vector/ST-table index if TPH was enabled in the interrupt-vector mode; the rte_pci_tph_st_get function ignores it. Both rte_pci_tph_st_{set, get} functions return the steering-tag (st) and processing-hint-ignored (ph_ignore) fields via the same rte_tph_info object passed into them.

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected outcome. This should be resolved by resetting the endpoint device before further attempts to set steering tags.

---
 drivers/bus/pci/bsd/pci.c        |  39 ++++++++
 drivers/bus/pci/bus_pci_driver.h |  43 ++++++++
 drivers/bus/pci/linux/pci.c      |  94 +++++++++++++++++
 drivers/bus/pci/linux/pci_init.h |   8 ++
 drivers/bus/pci/linux/pci_vfio.c | 166 +++++++++++++++++++++++++++++++
 drivers/bus/pci/private.h        |   8 ++
 drivers/bus/pci/rte_bus_pci.h    |  67 +++++++++++++
 drivers/bus/pci/windows/pci.c    |  39 ++++++++
 8 files changed, 464 insertions(+)
drivers/bus/pci/bsd/pci.c +39 -0
index 5e2e09d5a4..257816ab8e 100644
@@ -650,3 +650,42 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+int
+rte_pci_tph_enable(const struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+int
+rte_pci_tph_disable(const struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
drivers/bus/pci/bus_pci_driver.h +43 -0

49 lines not on this page · raw message

drivers/bus/pci/linux/pci.c +94 -0

97 lines not on this page · raw message

drivers/bus/pci/linux/pci_init.h +8 -0
index 25b901f460..5b249c81b1 100644
@@ -76,6 +76,14 @@ int pci_vfio_ioport_unmap(struct rte_pci_ioport *p);
 int pci_vfio_map_resource(struct rte_pci_device *dev);
 int pci_vfio_unmap_resource(struct rte_pci_device *dev);
 
+/* TLP Processing Hints control functions */
+int pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode);
+int pci_vfio_tph_disable(const struct rte_pci_device *dev);
+int pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+int pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+
 int pci_vfio_is_enabled(void);
 
 #endif
drivers/bus/pci/linux/pci_vfio.c +166 -0

172 lines not on this page · raw message

drivers/bus/pci/private.h +8 -0
index 38109844b9..d2ec370320 100644
@@ -335,4 +335,12 @@ rte_pci_dev_iterate(const void *start,
 int
 rte_pci_devargs_parse(struct rte_devargs *da);
 
+/*
+ * TPH Steering-Tag operation types.
+ */
+enum rte_pci_st_op {
+	RTE_PCI_TPH_ST_SET, /* Set TPH Steering - Tags */
+	RTE_PCI_TPH_ST_GET  /* Get TPH Steering - Tags */
+};
+
 #endif /* _PCI_PRIVATE_H_ */
drivers/bus/pci/rte_bus_pci.h +67 -0

79 lines not on this page · raw message

drivers/bus/pci/windows/pci.c +39 -0
index e7e449306e..72c334e572 100644
@@ -511,3 +511,42 @@ rte_pci_scan(void)
 
 	return ret;
 }
+
+int
+rte_pci_tph_enable(const struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+int
+rte_pci_tph_disable(const struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
Signature · 1 line
2.43.0
#32 lore raw

Re: [RFC PATCH v4 2/3] bus/pci: introduce the PCIe TLP Processing Hints API

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 818

Quoted 10 lines · Wathsala Vithanage
drivers/bus/pci/linux/pci.c +2 -0
index c20d159218..463c06ad64 100644
@@ -814,3 +814,97 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
        return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.03)

The release currently prepared is 25.07.
(no need for a respin for just this comment, this series is not ready for merge in any case)

Signature · 1 line
David Marchand
#33 lore raw

RE: [RFC PATCH v4 2/3] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 10 lines

Hints API

Quoted 6 lines · Wathsala Vithanage
drivers/bus/pci/linux/pci.c +0 -0
index c20d159218..463c06ad64 100644
@@ -814,3 +814,97 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
Quoted 10 lines

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 818

Quoted 4 lines
drivers/bus/pci/linux/pci.c +2 -0
        return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.03)

The release currently prepared is 25.07.
(no need for a respin for just this comment, this series is not ready for merge in any case)

Yes, thanks for pointing this out. I only waw that after sending the patch out. I will be updating this with fixes to windows and BSD build failures.

#34 lore raw

RFC PATCH v4 3/3 ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 ++++++++++++++++
 lib/ethdev/rte_ethdev.c    | 149 ++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 158 +++++++++++++++++++++++++++++++++++++
 3 files changed, 373 insertions(+)
lib/ethdev/ethdev_driver.h +66 -0
index 2b4d2ae9c3..8a4012db08 100644
@@ -1376,6 +1376,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -1402,6 +1464,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c +149 -0

160 lines not on this page · raw message

lib/ethdev/rte_ethdev.h +158 -0

176 lines not on this page · raw message

Signature · 1 line
2.43.0
#35 lore raw

Re: [RFC PATCH v4 3/3] ethdev: introduce the cache stashing hints API

Quoted 26 lines · On Sat, 17 May 2025 15:17:35 +0000 Wathsala Vithanage…

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

Quoted 7 lines

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

Adding more control in DPDK is good but I have concerns.

The default must be that the application has all the caching enabled without calling this API. It would be bad if existing DPDK applications from network vendors had to be modified.

The DPDK should follow the "it just works" mantra and additional API's should be for special cases.

#36 lore raw

v5 cover An API for Cache Stashing with TPH

In reply to Wathsala Vithanage

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

 drivers/bus/pci/bsd/pci.c            |  43 +++++++
 drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
 drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h     |  14 +++
 drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
 drivers/bus/pci/private.h            |   8 ++
 drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
 drivers/bus/pci/windows/pci.c        |  43 +++++++
 drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
 kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
 lib/ethdev/ethdev_driver.h           |  66 +++++++++++
 lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
 lib/pci/rte_pci.h                    |  15 +++
 14 files changed, 1114 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
Signature · 1 line
2.43.0
#37 lore raw

PATCH v5 1/4 pci: add non-merged Linux uAPI changes

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions. This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel. These additions have not yet been merged into the upstream kernel.

---
 drivers/bus/pci/linux/pci_init.h   |   1 +
 kernel/linux/uapi/linux/vfio_tph.h | 102 +++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
drivers/bus/pci/linux/pci_init.h +1 -0
index a4d37c0d0a..25b901f460 100644
@@ -6,6 +6,7 @@
 #define EAL_PCI_INIT_H_
 
 #include <rte_vfio.h>
+#include <uapi/linux/vfio_tph.h>
 
 #include "private.h"
 
kernel/linux/uapi/linux/vfio_tph.h new file +102 -0

102 lines not on this page · raw message

Signature · 1 line
2.43.0
#38 lore raw

RE: [PATCH v5 1/4] pci: add non-merged Linux uAPI changes

Hi Maxim,

Here is the link to the kernel VFIO patch you have asked earlier (v4).
https://lore.kernel.org/kvm/[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Thanks

--wathsala

Quoted 158 lines

-----Original Message-----
From: Wathsala Vithanage <[email protected]>
Sent: Monday, June 2, 2025 5:38 PM
To: Chenbo Xia <[email protected]>; Nipun Gupta <[email protected]>; Maxime Coquelin <[email protected]>
Cc: [email protected]; nd <[email protected]>; Wathsala Wathawana Vithanage <[email protected]>; Dhruv Tripathi <[email protected]>
Subject: [PATCH v5 1/4] pci: add non-merged Linux uAPI changes

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions.
This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel.
These additions have not yet been merged into the upstream kernel.

---
 drivers/bus/pci/linux/pci_init.h   |   1 +
 kernel/linux/uapi/linux/vfio_tph.h | 102 +++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
drivers/bus/pci/linux/pci_init.h +1 -0
index a4d37c0d0a..25b901f460 100644
@@ -6,6 +6,7 @@
 #define EAL_PCI_INIT_H_
 
 #include <rte_vfio.h>
+#include <uapi/linux/vfio_tph.h>
 
 #include "private.h"
 
kernel/linux/uapi/linux/vfio_tph.h new file +6 -0
new file mode 100644
index 0000000000..9336c2e5af
@@ -0,0 +1,102 @@
+
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * VFIO API definition
+ *
+ * WARNING: CONTENTS OF THIS HEADER NEEDS TO BE MERGED INTO

KERNEL'S
+ * uapi/linux/vifo.h IN A FUTURE KERNEL RELEASE. UNTIL THEN IT'S TACKED
+ * ON TO DPDK'S kernel/linux/uapi DIRECTORY TO PREVENT BUILD FAILURES.
+ *
+ * Copyright (C) 2012 Red Hat, Inc. All rights reserved.
+ * Author: Alex Williamson <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */

kernel/linux/uapi/linux/vfio_tph.h new file +18 -0
+
+#ifndef _UAPIVFIO_TPH_H
+#define _UAPIVFIO_TPH_H
+
+/**
+ * VFIO_DEVICE_PCI_TPH	- _IO(VFIO_TYPE, VFIO_BASE + 22)
+ *
+ * This command is used to control PCIe TLP Processing Hints (TPH)
+ * capability in a PCIe device.
+ * It supports following operations on a PCIe device with respect to
+TPH
+ * capability.
+ *
+ * - Enabling/disabling TPH capability in a PCIe device.
+ *
+ *   Setting VFIO_DEVICE_TPH_ENABLE flag enables TPH in no-steering-tag,
+ *   interrupt-vector, or device-specific mode defined in the PCIe specficiation
+ *   when feature flags TPH_ST_NS_MODE, TPH_ST_IV_MODE, and

TPH_ST_DS_MODE are
+ * set respectively. TPH_ST_xx_MODE macros are defined in
+ * uapi/linux/pci_regs.h.
+ *
+ * VFIO_DEVICE_TPH_DISABLE disables PCIe TPH on the device.
+ *
+ * - Writing STs to MSI-X or ST table in a PCIe device.
+ *
+ * VFIO_DEVICE_TPH_SET_ST flag set steering tags on a device at an index in
+ * MSI-X or ST-table depending on the VFIO_TPH_ST_x_MODE flag used and
device

+ *   capabilities. The caller can set one or more steering tags by passing an
+ *   array of vfio_pci_tph_entry objects containing cpu_id, cache_level, and
+ *   MSI-X/ST-table index. The caller can also set the intended memory type and
+ *   the processing hint by setting VFIO_TPH_MEM_TYPE_x and

VFIO_TPH_HINT_x
+ * flags, respectively.
+ *
+ * - Reading Steering Tags (ST) from the host platform.
+ *

+ *   VFIO_DEVICE_TPH_GET_ST flags returns steering tags to the caller. Caller
+ *   can request one or more steering tags by passing an array of
+ *   vfio_pci_tph_entry objects. Steering Tag for each request is returned via
+ *   the st field in vfio_pci_tph_entry.

+ */

kernel/linux/uapi/linux/vfio_tph.h new file +8 -0
+struct vfio_pci_tph_entry {
+	/* in */
+	__u32 cpu_id;			/* CPU logical ID */
+	__u32 cache_level;		/* Cache level. L1 D= 0, L2D = 2, ...*/
+	__u8  flags;
+#define VFIO_TPH_MEM_TYPE_MASK		0x1
+#define VFIO_TPH_MEM_TYPE_SHIFT		0
+#define VFIO_TPH_MEM_TYPE_VMEM		0   /* Request volatile memory

ST */
+#define VFIO_TPH_MEM_TYPE_PMEM 1 /* Request persistent
memory ST */

kernel/linux/uapi/linux/vfio_tph.h new file +22 -0
+
+#define VFIO_TPH_HINT_MASK		0x3
+#define VFIO_TPH_HINT_SHIFT		1
+#define VFIO_TPH_HINT_BIDIR		0
+#define VFIO_TPH_HINT_REQSTR		(1 << VFIO_TPH_HINT_SHIFT)
+#define VFIO_TPH_HINT_TARGET		(2 << VFIO_TPH_HINT_SHIFT)
+#define VFIO_TPH_HINT_TARGET_PRIO	(3 << VFIO_TPH_HINT_SHIFT)
+	__u8  pad0;
+	__u16 index;			/* MSI-X/ST-table index to set ST */
+	/* out */
+	__u16 st;			/* Steering-Tag */
+	__u8  ph_ignore;		/* Platform ignored the Processing */
+	__u8  pad1;
+};
+
+struct vfio_pci_tph {
+	__u32 argsz;			/* Size of vfio_pci_tph and info[] */
+	__u32 flags;
+#define VFIO_TPH_ST_MODE_MASK		0x7
+
+#define VFIO_DEVICE_TPH_OP_SHIFT	3
+#define VFIO_DEVICE_TPH_OP_MASK		(0x7 <<

VFIO_DEVICE_TPH_OP_SHIFT)

kernel/linux/uapi/linux/vfio_tph.h new file +4 -0
+/* Enable TPH on device */
+#define VFIO_DEVICE_TPH_ENABLE		0
+/* Disable TPH on device */
+#define VFIO_DEVICE_TPH_DISABLE		(1 <<

VFIO_DEVICE_TPH_OP_SHIFT)

kernel/linux/uapi/linux/vfio_tph.h new file +2 -0
+/* Get steering-tags */
+#define VFIO_DEVICE_TPH_GET_ST		(2 <<

VFIO_DEVICE_TPH_OP_SHIFT)

kernel/linux/uapi/linux/vfio_tph.h new file +2 -0
+/* Set steering-tags */
+#define VFIO_DEVICE_TPH_SET_ST		(4 <<

VFIO_DEVICE_TPH_OP_SHIFT)

kernel/linux/uapi/linux/vfio_tph.h new file +8 -1
+	__u32 count;			/* Number of entries in ents[] */
+	struct vfio_pci_tph_entry ents[];
+#define VFIO_TPH_INFO_MAX	2048	/* Max entries in ents[] */
+};
+
+#define VFIO_DEVICE_PCI_TPH	_IO(VFIO_TYPE, VFIO_BASE + 22)
+
+#endif /* _UAPIVFIO_TPH_H */
--

2.43.0

#40 lore raw

Re: [PATCH v5 1/4] pci: add non-merged Linux uAPI changes

In reply to Wathsala Vithanage
Quoted 10 lines · On Mon, 2 Jun 2025 22:38:01 +0000 Wathsala Vithanage…

This commit is a hack to prevent build failures the next commit in this patch series causes due to missing vfio uapi definitions. This commit should NEVER BE MERGED.
Next commit in this patch series depends on additions to vfio uapi that enable TPH icotl in the vfio-pci driver in the Linux kernel. These additions have not yet been merged into the upstream kernel.

---

The series will not be reviewed or accepted until the kernel changes are merged upstream and in a kernel release from Linus.

#41 lore raw

PATCH v5 2/4 bus/pci: introduce the PCIe TLP Processing Hints API

In reply to Wathsala Vithanage

Extend the PCI bus driver to enable or disable TPH capability and set or get PCI Steering-Tags (STs) on an endpoint device. The functions rte_pci_tph_{enable, disable,st_set,st_get} provide the primary interface for DPDK device drivers. Implementation of the interface is OS dependent. For Linux, the kernel VFIO driver provides the implementation. rte_pci_tph_{enable, disable} functions enable and disable TPH capability, respectively. rte_pci_tph_enable enables TPH on the device in either of the device-specific, interrupt-vector, or no-steering-tag modes.

rte_pci_tph_st_{get, set} functions take an array of rte_tph_info objects with cpu-id, cache-level, flags (processing-hint, memory-type). The index in rte_tph_info is the MSI-X/MSI vector/ST-table index if TPH was enabled in the interrupt-vector mode; the rte_pci_tph_st_get function ignores it. Both rte_pci_tph_st_{set, get} functions return the steering-tag (st) and processing-hint-ignored (ph_ignore) fields via the same rte_tph_info object passed into them.

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected outcome. This should be resolved by resetting the endpoint device before further attempts to set steering tags.

---
 drivers/bus/pci/bsd/pci.c        |  43 ++++++++
 drivers/bus/pci/bus_pci_driver.h |  52 ++++++++++
 drivers/bus/pci/linux/pci.c      | 100 ++++++++++++++++++
 drivers/bus/pci/linux/pci_init.h |  13 +++
 drivers/bus/pci/linux/pci_vfio.c | 170 +++++++++++++++++++++++++++++++
 drivers/bus/pci/private.h        |   8 ++
 drivers/bus/pci/rte_bus_pci.h    |  67 ++++++++++++
 drivers/bus/pci/windows/pci.c    |  43 ++++++++
 lib/pci/rte_pci.h                |  15 +++
 9 files changed, 511 insertions(+)
drivers/bus/pci/bsd/pci.c +43 -0
index 5e2e09d5a4..dff750c4d6 100644
@@ -650,3 +650,46 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07)
+int
+rte_pci_tph_disable(struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07)
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07)
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
drivers/bus/pci/bus_pci_driver.h +52 -0

64 lines not on this page · raw message

drivers/bus/pci/linux/pci.c +100 -0

103 lines not on this page · raw message

drivers/bus/pci/linux/pci_init.h +13 -0
index 25b901f460..e71bfd2dce 100644
@@ -5,6 +5,7 @@
 #ifndef EAL_PCI_INIT_H_
 #define EAL_PCI_INIT_H_
 
+#include <rte_compat.h>
 #include <rte_vfio.h>
 #include <uapi/linux/vfio_tph.h>
 
@@ -76,6 +77,18 @@ int pci_vfio_ioport_unmap(struct rte_pci_ioport *p);
 int pci_vfio_map_resource(struct rte_pci_device *dev);
 int pci_vfio_unmap_resource(struct rte_pci_device *dev);
 
+/* TLP Processing Hints control functions */
+__rte_experimental
+int pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode);
+__rte_experimental
+int pci_vfio_tph_disable(const struct rte_pci_device *dev);
+__rte_experimental
+int pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+__rte_experimental
+int pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+
 int pci_vfio_is_enabled(void);
 
 #endif
drivers/bus/pci/linux/pci_vfio.c +170 -0

182 lines not on this page · raw message

drivers/bus/pci/private.h +8 -0
index 38109844b9..d2ec370320 100644
@@ -335,4 +335,12 @@ rte_pci_dev_iterate(const void *start,
 int
 rte_pci_devargs_parse(struct rte_devargs *da);
 
+/*
+ * TPH Steering-Tag operation types.
+ */
+enum rte_pci_st_op {
+	RTE_PCI_TPH_ST_SET, /* Set TPH Steering - Tags */
+	RTE_PCI_TPH_ST_GET  /* Get TPH Steering - Tags */
+};
+
 #endif /* _PCI_PRIVATE_H_ */
drivers/bus/pci/rte_bus_pci.h +67 -0

79 lines not on this page · raw message

drivers/bus/pci/windows/pci.c +43 -0

46 lines not on this page · raw message

lib/pci/rte_pci.h +15 -0
index 9a50a12142..da9cd666bf 100644
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register */
 
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode Supported */
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector Mode Supported */
+#define RTE_PCI_TPH_CAP_ST_DS      0x00000004 /* Device Specific Mode Supported */
+#define RTE_PCI_TPH_CAP_EXT_TPH    0x00000100 /* Ext TPH Requester Supported */
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location */
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+
 /** Formatting string for PCI device identifier: Ex: 0000:00:01.0 */
 #define PCI_PRI_FMT "%.4" PRIx32 ":%.2" PRIx8 ":%.2" PRIx8 ".%" PRIx8
 #define PCI_PRI_STR_SIZE sizeof("XXXXXXXX:XX:XX.X")
Signature · 1 line
2.43.0
#42 lore raw

RE: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 2 lines

From: Wathsala Vithanage [mailto:[email protected]]
Sent: Tuesday, 3 June 2025 00.38

Some nitpicking inline below.

[...]

Quoted 7 lines
drivers/bus/pci/bsd/pci.c +0 -0
index 5e2e09d5a4..dff750c4d6 100644
@@ -650,3 +650,46 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;

drivers/bus/pci/bsd/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 661

Quoted 10 lines
drivers/bus/pci/bsd/pci.c +9 -0
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for BSD */
+	return -1;

Either set rte_errno and return -1, or return -ENOTSUP.
If one of these two design patterns are already used in a module, please use the same design pattern for new functions. Applies to all dummy functions, BSD and Windows.

[...]

Quoted 39 lines
drivers/bus/pci/bus_pci_driver.h +1 -0
index 2cc1119072..b1c2829fc1 100644
@@ -46,6 +46,7 @@ struct rte_pci_device {
 	char *bus_info;                     /**< PCI bus specific info */
 	struct rte_intr_handle *vfio_req_intr_handle;
 				/**< Handler of VFIO request interrupt */
+	uint8_t tph_enabled;                /**< TPH enabled on this

device */
};

/**

drivers/bus/pci/bus_pci_driver.h +3 -0
@@ -194,6 +195,57 @@ struct rte_pci_ioport {
 	uint64_t len; /* only filled for memory mapped ports */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without

prior
+ * notice
+ *
+ * This structure is passed into the TPH Steering-Tag set or get function as an
+ * argument by the caller. Return values are set in the same structure in st and
+ * ph_ignore fields by the calee.
+ *
+ * Refer to PCI-SIG ECN "Revised _DSM for Cache Locality TPH Features" for
+ * details.
+ */

drivers/bus/pci/bus_pci_driver.h +4 -0
+struct rte_tph_info {
+	/* Input */
+	uint32_t cpu_id;	/*Logical CPU id*/
+	uint32_t cache_level;	/*Cache level relative to CPU.

l1d=0,l2d=1,...*/

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 217

Quoted 4 lines
drivers/bus/pci/bus_pci_driver.h +4 -0
+	uint8_t flags;		/*Memory type, procesisng hint etc.*/
+	uint16_t index;		/*Index in vector table to store the ST*/
+
+	/* Output */

I don't think these are Output fields when used in the set() function. Please update the documentation of this structure accordingly.

Quoted 45 lines
drivers/bus/pci/bus_pci_driver.h +2 -0
+	uint16_t st;		/*Steering tag returned by the platform*/
+	uint8_t ph_ignore;	/*Platform ignores PH for the returned

ST*/

drivers/bus/pci/bus_pci_driver.h +29 -0
+};
+
+#define RTE_PCI_TPH_MEM_TYPE_MASK		0x1
+#define RTE_PCI_TPH_MEM_TYPE_SHIFT		0
+/** Request volatile memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_VMEM		0
+/** Request persistent memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_PMEM		1
+
+/** TLP Processing Hints - PCIe 6.0 specification section 2.2.7.1.1 */
+#define RTE_PCI_TPH_HINT_MASK		0x3
+#define RTE_PCI_TPH_HINT_SHIFT		1
+/** Host and device access data equally */
+#define RTE_PCI_TPH_HINT_BIDIR		0
+/** Device accesses data more frequently */
+#define RTE_PCI_TPH_HINT_REQSTR		(1 << RTE_PCI_TPH_HINT_SHIFT)
+/** Host access data more frequently */
+#define RTE_PCI_TPH_HINT_TARGET		(2 << RTE_PCI_TPH_HINT_SHIFT)
+/** Host access data more frequently with a high temporal locality */
+#define RTE_PCI_TPH_HINT_TARGET_PRIO	(3 << RTE_PCI_TPH_HINT_SHIFT)
+
+#define RTE_PCI_TPH_ST_MODE_MASK   0x3
+/** TPH no ST mode */
+#define RTE_PCI_TPH_ST_NS_MODE	   0
+/** TPH interrupt vector mode */
+#define RTE_PCI_TPH_ST_IV_MODE	   1
+/** TPH device specific mode */
+#define RTE_PCI_TPH_ST_DS_MODE	   2
+
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/linux/pci.c +2 -0
index c20d159218..b5a8ba0a86 100644
@@ -814,3 +814,103 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 828

Quoted 10 lines
drivers/bus/pci/linux/pci.c +10 -0
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_enable(dev, mode);

Is it correct to return 0 when pci_vfio_is_enabled() returns false?

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 831

Quoted 3 lines
drivers/bus/pci/linux/pci.c +3 -0
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:

Please add "/* fall through */" or similar compiler hint.
Or even better, define a new __rte_fallthrough macro as __attribute__((fallthrough)) in rte_common.h. Refer to [1].

[1]: https://gcc.gnu.org/onlinedocs/gcc/Statement-Attributes.html

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 838

Quoted 7 lines
drivers/bus/pci/linux/pci.c +7 -0
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	if (!ret)

This looks wrong. "ret" is not Boolean, so please use "ret == 0" instead of "!ret". Applies elsewhere in this patch too, e.g. comparing empty "count" in pci_vfio_tph_st_op().

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 843

Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+		dev->tph_enabled = 1;
+
+	return ret;
+}
+

The above comments apply to all four functions.

[...]

Quoted 18 lines
drivers/bus/pci/linux/pci_vfio.c +1 -0
index 5317170231..bdbeb38658 100644
@@ -12,6 +12,7 @@
 #include <stdbool.h>
 
 #include <rte_log.h>
+#include <eal_export.h>
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_eal_paging.h>
@@ -1316,6 +1317,175 @@ pci_vfio_mmio_write(const struct rte_pci_device
 *dev, int bar,
 	return pwrite(fd, buf, len, offset + offs);
 }
 

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device · line 1330

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +9 -0
+static int
+pci_vfio_tph_ioctl(const struct rte_pci_device *dev, struct
 vfio_pci_tph *pci_tph)
+{
+	const struct rte_intr_handle *intr_handle = dev->intr_handle;
+	int vfio_dev_fd = 0, ret = 0;
+
+	vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+	if (vfio_dev_fd < 0) {
+		ret = -EINVAL;

This should be ret = rte_errno.
Or even better: Other code in /drivers/bus/pci/linux/pci_vfio.c follows the design pattern of setting rte_errno and returning -1 on error. Please use the same design pattern. Applies to all four functions.

Quoted 198 lines
drivers/bus/pci/linux/pci_vfio.c +160 -0
+		goto out;
+	}
+
+	ret = ioctl(vfio_dev_fd, VFIO_DEVICE_PCI_TPH, pci_tph);
+out:
+	return ret;
+}
+
+static int
+pci_vfio_tph_st_op(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count,
+		    enum rte_pci_st_op op)
+{
+	int ret = 0;
+	size_t argsz = 0, i;
+	struct vfio_pci_tph *pci_tph = NULL;
+	uint8_t mem_type = 0, hint = 0;
+
+	if (!count) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	argsz = sizeof(struct vfio_pci_tph) +
+		count * sizeof(struct vfio_pci_tph_entry);
+
+	pci_tph = rte_zmalloc(NULL, argsz, 0);
+	if (!pci_tph) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	pci_tph->argsz = argsz;
+	pci_tph->count = count;
+
+	switch (op) {
+	case RTE_PCI_TPH_ST_GET:
+		pci_tph->flags = VFIO_DEVICE_TPH_GET_ST;
+		break;
+	case RTE_PCI_TPH_ST_SET:
+		pci_tph->flags = VFIO_DEVICE_TPH_SET_ST;
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	for (i = 0; i < count; i++) {
+		pci_tph->ents[i].cpu_id = info[i].cpu_id;
+		pci_tph->ents[i].cache_level = info[i].cache_level;
+
+		mem_type = info[i].flags & RTE_PCI_TPH_MEM_TYPE_MASK;
+		switch (mem_type) {
+		case RTE_PCI_TPH_MEM_TYPE_VMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_VMEM;
+			break;
+		case RTE_PCI_TPH_MEM_TYPE_PMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_PMEM;
+			break;
+		default:
+			ret = -EINVAL;
+			goto out;
+		}
+
+		hint = info[i].flags & RTE_PCI_TPH_HINT_MASK;
+		switch (hint) {
+		case RTE_PCI_TPH_HINT_BIDIR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_BIDIR;
+			break;
+		case RTE_PCI_TPH_HINT_REQSTR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_REQSTR;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_TARGET;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET_PRIO:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_TARGET_PRIO;
+			break;
+		default:
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (op == RTE_PCI_TPH_ST_SET)
+			pci_tph->ents[i].index = info[i].index;
+	}
+
+	ret = pci_vfio_tph_ioctl(dev, pci_tph);
+	if (ret)
+		goto out;
+
+	/*
+	 * Kernel returns steering-tag and ph-ignore bits for
+	 * RTE_PCI_TPH_ST_SET too, therefore copy output for
+	 * both RTE_PCI_TPH_ST_SET and RTE_PCI_TPH_ST_GET
+	 * cases.
+	 */
+	for (i = 0; i < count; i++) {
+		info[i].st = pci_tph->ents[i].st;
+		info[i].ph_ignore = pci_tph->ents[i].ph_ignore;
+	}
+
+out:
+	if (pci_tph)
+		rte_free(pci_tph);
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_enable, 25.07)
+int
+pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode)
+{
+	int ret;
+
+	if (!(mode ^ (mode & VFIO_TPH_ST_MODE_MASK))) {
+		ret = -EINVAL;
+		goto out;
+	} else
+		mode &= VFIO_TPH_ST_MODE_MASK;
+
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_ENABLE | mode,
+		.count = 0
+	};
+
+	ret = pci_vfio_tph_ioctl(dev, &pci_tph);
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_disable, 25.07)
+int
+pci_vfio_tph_disable(const struct rte_pci_device *dev)
+{
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_DISABLE,
+		.count = 0
+	};
+
+	return pci_vfio_tph_ioctl(dev, &pci_tph);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_get, 25.07)
+int
+pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count)
+{
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_GET);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_set, 25.07)
+int
+pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count)
+{
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_SET);
+}
+
 int

pci_vfio_is_enabled(void)
{

drivers/bus/pci/private.h +8 -0
index 38109844b9..d2ec370320 100644
@@ -335,4 +335,12 @@ rte_pci_dev_iterate(const void *start,
 int
 rte_pci_devargs_parse(struct rte_devargs *da);
 
+/*
+ * TPH Steering-Tag operation types.
+ */
+enum rte_pci_st_op {
+	RTE_PCI_TPH_ST_SET, /* Set TPH Steering - Tags */
+	RTE_PCI_TPH_ST_GET  /* Get TPH Steering - Tags */
+};
+
 #endif /* _PCI_PRIVATE_H_ */
drivers/bus/pci/rte_bus_pci.h +2 -0
index 19a7b15b99..e4d4780f54 100644
@@ -31,6 +31,7 @@ extern "C" {
 struct rte_pci_device;
 struct rte_pci_driver;
 struct rte_pci_ioport;
+struct rte_tph_info;
 
 struct rte_devargs;
 
@@ -312,6 +313,72 @@ void rte_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**

drivers/bus/pci/rte_bus_pci.h void rte_pci_ioport_read(struct rte_pci_ioport *p, · line 326

Quoted 10 lines
drivers/bus/pci/rte_bus_pci.h +10 -0
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Enable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param mode
+ *   TPH mode the device must operate in.

The values of the "mode" parameter should be well defined in a DPDK public API. I cannot find them.

The return value is missing from the documentation. For all four functions here.

Quoted 143 lines
drivers/bus/pci/rte_bus_pci.h +55 -0
+ */
+__rte_experimental
+int rte_pci_tph_enable(struct rte_pci_device *dev, int mode);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Disable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ */
+__rte_experimental
+int rte_pci_tph_disable(struct rte_pci_device *dev);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/windows/pci.c +43 -0
index e7e449306e..218e667a5a 100644
@@ -511,3 +511,46 @@ rte_pci_scan(void)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07)
+int
+rte_pci_tph_disable(struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07)
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07)
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
lib/pci/rte_pci.h +4 -0
index 9a50a12142..da9cd666bf 100644
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register
 */
 
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode Supported
 */
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector Mode

Supported */
+#define RTE_PCI_TPH_CAP_ST_DS 0x00000004 /* Device Specific Mode
Supported */
+#define RTE_PCI_TPH_CAP_EXT_TPH 0x00000100 /* Ext TPH Requester
Supported */

lib/pci/rte_pci.h +9 -1
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location */
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+
 /** Formatting string for PCI device identifier: Ex: 0000:00:01.0 */
 #define PCI_PRI_FMT "%.4" PRIx32 ":%.2" PRIx8 ":%.2" PRIx8 ".%" PRIx8
 #define PCI_PRI_STR_SIZE sizeof("XXXXXXXX:XX:XX.X")
--

2.43.0

#43 lore raw

Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

In reply to Wathsala Vithanage
Quoted 18 lines · Wathsala Vithanage

Extend the PCI bus driver to enable or disable TPH capability and set or get PCI Steering-Tags (STs) on an endpoint device. The functions rte_pci_tph_{enable, disable,st_set,st_get} provide the primary interface for DPDK device drivers. Implementation of the interface is OS dependent. For Linux, the kernel VFIO driver provides the implementation. rte_pci_tph_{enable, disable} functions enable and disable TPH capability, respectively. rte_pci_tph_enable enables TPH on the device in either of the device-specific, interrupt-vector, or no-steering-tag modes.

rte_pci_tph_st_{get, set} functions take an array of rte_tph_info objects with cpu-id, cache-level, flags (processing-hint, memory-type). The index in rte_tph_info is the MSI-X/MSI vector/ST-table index if TPH was enabled in the interrupt-vector mode; the rte_pci_tph_st_get function ignores it. Both rte_pci_tph_st_{set, get} functions return the steering-tag (st) and processing-hint-ignored (ph_ignore) fields via the same rte_tph_info object passed into them.

Quoted 9 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected outcome. This should be resolved by resetting the endpoint device before further attempts to set steering tags.

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

Other comments inline below.

/Bruce

Quoted 20 lines
---
 drivers/bus/pci/bsd/pci.c        |  43 ++++++++
 drivers/bus/pci/bus_pci_driver.h |  52 ++++++++++
 drivers/bus/pci/linux/pci.c      | 100 ++++++++++++++++++
 drivers/bus/pci/linux/pci_init.h |  13 +++
 drivers/bus/pci/linux/pci_vfio.c | 170 +++++++++++++++++++++++++++++++
 drivers/bus/pci/private.h        |   8 ++
 drivers/bus/pci/rte_bus_pci.h    |  67 ++++++++++++
 drivers/bus/pci/windows/pci.c    |  43 ++++++++
 lib/pci/rte_pci.h                |  15 +++
 9 files changed, 511 insertions(+)
drivers/bus/pci/bsd/pci.c +0 -0
index 5e2e09d5a4..dff750c4d6 100644

<snip>

drivers/bus/pci/bus_pci_driver.h struct rte_pci_device { · line 49

Quoted 9 lines
drivers/bus/pci/bus_pci_driver.h +1 -0
index 2cc1119072..b1c2829fc1 100644
@@ -46,6 +46,7 @@ struct rte_pci_device {
 	char *bus_info;                     /**< PCI bus specific info */
 	struct rte_intr_handle *vfio_req_intr_handle;
 				/**< Handler of VFIO request interrupt */
+	uint8_t tph_enabled;                /**< TPH enabled on this device */

question: what would happen if we always enabled tph for each device. Does doing so disable the default handling for the device?

Quoted 16 lines
drivers/bus/pci/bus_pci_driver.h +9 -0
 };
 
 /**
@@ -194,6 +195,57 @@ struct rte_pci_ioport {
 	uint64_t len; /* only filled for memory mapped ports */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without prior
+ * notice
+ *
+ * This structure is passed into the TPH Steering-Tag set or get function as an
+ * argument by the caller. Return values are set in the same structure in st and
+ * ph_ignore fields by the calee.
+ *

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 216

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+ * Refer to PCI-SIG ECN "Revised _DSM for Cache Locality TPH Features" for
+ * details.
+ */
+struct rte_tph_info {
+	/* Input */
+	uint32_t cpu_id;	/*Logical CPU id*/
+	uint32_t cache_level;	/*Cache level relative to CPU. l1d=0,l2d=1,...*/
+	uint8_t flags;		/*Memory type, procesisng hint etc.*/
+	uint16_t index;		/*Index in vector table to store the ST*/
+

These fields should be reordered in order of size to avoid unnecessary gaps.

For the flags field, I dislike having different sets of flags all multiplexed into a single field. Can we instead of the flags field, and the set of #defines below split these out into separate enums, and then have separate fields for each one.

For example:

	struct rte_tph_info {
		uint32_t cpu_id;
		uint32_t cache_level;
		enum rte_tph_mem_type mem_type;
		enum rte_tph_hint hints;
		enum rte_tph_mode mode;
		...
	}

While the structure takes more space this way, this is not a datapath structure that we should be seeing large arrays of it, or that needs to be processed quickly, so usability should be prioritized over size/compactness.

Quoted 4 lines
drivers/bus/pci/bus_pci_driver.h +4 -0
+	/* Output */
+	uint16_t st;		/*Steering tag returned by the platform*/
+	uint8_t ph_ignore;	/*Platform ignores PH for the returned ST*/
+};

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 230

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+
+#define RTE_PCI_TPH_MEM_TYPE_MASK		0x1
+#define RTE_PCI_TPH_MEM_TYPE_SHIFT		0
+/** Request volatile memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_VMEM		0
+/** Request persistent memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_PMEM		1
+
+/** TLP Processing Hints - PCIe 6.0 specification section 2.2.7.1.1 */
+#define RTE_PCI_TPH_HINT_MASK		0x3

Looking at the mask usage below, does this mask not need to also be shifted by the TPH_HINT_SHIFT? Otherwise it overlaps with the type mask.

Quoted 8 lines
drivers/bus/pci/bus_pci_driver.h +8 -0
+#define RTE_PCI_TPH_HINT_SHIFT		1
+/** Host and device access data equally */
+#define RTE_PCI_TPH_HINT_BIDIR		0
+/** Device accesses data more frequently */
+#define RTE_PCI_TPH_HINT_REQSTR		(1 << RTE_PCI_TPH_HINT_SHIFT)
+/** Host access data more frequently */
+#define RTE_PCI_TPH_HINT_TARGET		(2 << RTE_PCI_TPH_HINT_SHIFT)
+/** Host access data more frequently with a high temporal locality */

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 248

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+#define RTE_PCI_TPH_HINT_TARGET_PRIO	(3 << RTE_PCI_TPH_HINT_SHIFT)
+
+#define RTE_PCI_TPH_ST_MODE_MASK   0x3
+/** TPH no ST mode */
+#define RTE_PCI_TPH_ST_NS_MODE	   0
+/** TPH interrupt vector mode */
+#define RTE_PCI_TPH_ST_IV_MODE	   1
+/** TPH device specific mode */
+#define RTE_PCI_TPH_ST_DS_MODE	   2
+

As above, I think these would be nicer defined in different enums, going to separate fields in the struct. That would also remove any ambiguity as to whether the masks include the shift or not.

Quoted 8 lines
drivers/bus/pci/bus_pci_driver.h +0 -0
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/linux/pci.c +0 -0
index c20d159218..b5a8ba0a86 100644

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 823

Quoted 10 lines
drivers/bus/pci/linux/pci.c +7 -0
@@ -814,3 +814,103 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	int ret = 0;
+

Should check here if dev->tph_enabled is already true.

Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_enable(dev, mode);

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 838

Quoted 10 lines
drivers/bus/pci/linux/pci.c +10 -0
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	if (!ret)

Prefer "ret == 0" for this comparison.

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 843

Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+		dev->tph_enabled = 1;
+
+	return ret;
+}
+

Function could probably be shortened to something like (including a check for already enabled, 2 lines shorter if we rely on checks in the vfio_tph_enable() call):

int
rte_pci_tph_enable(...)
{

#ifdef VFIO_PRESENT
	if (dev->kdrv == RTE_PCI_KDRV_VFIO && pci_vfio_is_enabled()) {
		if (dev->tph_enabled == 0) {
			int ret = pci_vfio_tph_enable(...);
			if (ret != 0)
				return ret;
			dev->tph_enabled = 1;
		}
		return 0;
	}

#endif
return -ENOTSUP
}

Quoted 15 lines
drivers/bus/pci/linux/pci.c +15 -0
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07)
+int
+rte_pci_tph_disable(struct rte_pci_device *dev)
+{
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_disable(dev);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 868

Quoted 10 lines
drivers/bus/pci/linux/pci.c +10 -0
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	if (!ret)
+		dev->tph_enabled = 0;
+
+	return ret;
+}

As above, we can shorten this function by replacing the switch with a straight check for kdrv == RTE_PCI_KDRV_VFIO. Same with functions below too.

Quoted 121 lines
drivers/bus/pci/linux/pci.c +48 -0
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07)
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_st_get(dev, info, count);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07)
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_st_set(dev, info, count);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	return ret;
+}
drivers/bus/pci/linux/pci_init.h +13 -0
index 25b901f460..e71bfd2dce 100644
@@ -5,6 +5,7 @@
 #ifndef EAL_PCI_INIT_H_
 #define EAL_PCI_INIT_H_
 
+#include <rte_compat.h>
 #include <rte_vfio.h>
 #include <uapi/linux/vfio_tph.h>
 
@@ -76,6 +77,18 @@ int pci_vfio_ioport_unmap(struct rte_pci_ioport *p);
 int pci_vfio_map_resource(struct rte_pci_device *dev);
 int pci_vfio_unmap_resource(struct rte_pci_device *dev);
 
+/* TLP Processing Hints control functions */
+__rte_experimental
+int pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode);
+__rte_experimental
+int pci_vfio_tph_disable(const struct rte_pci_device *dev);
+__rte_experimental
+int pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+__rte_experimental
+int pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+
 int pci_vfio_is_enabled(void);
 
 #endif
drivers/bus/pci/linux/pci_vfio.c +27 -0
index 5317170231..bdbeb38658 100644
@@ -12,6 +12,7 @@
 #include <stdbool.h>
 
 #include <rte_log.h>
+#include <eal_export.h>
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_eal_paging.h>
@@ -1316,6 +1317,175 @@ pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar,
 	return pwrite(fd, buf, len, offset + offs);
 }
 
+static int
+pci_vfio_tph_ioctl(const struct rte_pci_device *dev, struct vfio_pci_tph *pci_tph)
+{
+	const struct rte_intr_handle *intr_handle = dev->intr_handle;
+	int vfio_dev_fd = 0, ret = 0;
+
+	vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+	if (vfio_dev_fd < 0) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = ioctl(vfio_dev_fd, VFIO_DEVICE_PCI_TPH, pci_tph);
+out:
+	return ret;
+}
+
+static int
+pci_vfio_tph_st_op(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count,
+		    enum rte_pci_st_op op)
+{
+	int ret = 0;
+	size_t argsz = 0, i;
+	struct vfio_pci_tph *pci_tph = NULL;
+	uint8_t mem_type = 0, hint = 0;

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1355

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+
+	if (!count) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	argsz = sizeof(struct vfio_pci_tph) +
+		count * sizeof(struct vfio_pci_tph_entry);
+
+	pci_tph = rte_zmalloc(NULL, argsz, 0);

For ioctl we should not need pinned memory. Use regular malloc here.

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1356

Quoted 1 line
drivers/bus/pci/linux/pci_vfio.c +1 -0
+	if (!pci_tph) {

Coding style guidelines say to compare pointers explicitly to NULL.

Quoted 27 lines
drivers/bus/pci/linux/pci_vfio.c +27 -0
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	pci_tph->argsz = argsz;
+	pci_tph->count = count;
+
+	switch (op) {
+	case RTE_PCI_TPH_ST_GET:
+		pci_tph->flags = VFIO_DEVICE_TPH_GET_ST;
+		break;
+	case RTE_PCI_TPH_ST_SET:
+		pci_tph->flags = VFIO_DEVICE_TPH_SET_ST;
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	for (i = 0; i < count; i++) {
+		pci_tph->ents[i].cpu_id = info[i].cpu_id;
+		pci_tph->ents[i].cache_level = info[i].cache_level;
+
+		mem_type = info[i].flags & RTE_PCI_TPH_MEM_TYPE_MASK;
+		switch (mem_type) {
+		case RTE_PCI_TPH_MEM_TYPE_VMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_VMEM;

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1393

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+			break;
+		case RTE_PCI_TPH_MEM_TYPE_PMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_PMEM;
+			break;
+		default:
+			ret = -EINVAL;
+			goto out;
+		}
+
+		hint = info[i].flags & RTE_PCI_TPH_HINT_MASK;

As pointed out above, unshifted, this HINT_MASK overlaps with the TYPE_MASK.

Quoted 14 lines
drivers/bus/pci/linux/pci_vfio.c +14 -0
+		switch (hint) {
+		case RTE_PCI_TPH_HINT_BIDIR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_BIDIR;
+			break;
+		case RTE_PCI_TPH_HINT_REQSTR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_REQSTR;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_TARGET;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET_PRIO:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_TARGET_PRIO;
+			break;
+		default:

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1417

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (op == RTE_PCI_TPH_ST_SET)
+			pci_tph->ents[i].index = info[i].index;
+	}
+
+	ret = pci_vfio_tph_ioctl(dev, pci_tph);
+	if (ret)

Again, check explicitly for "== 0".

Quoted 6 lines
drivers/bus/pci/linux/pci_vfio.c +6 -0
+		goto out;
+
+	/*
+	 * Kernel returns steering-tag and ph-ignore bits for
+	 * RTE_PCI_TPH_ST_SET too, therefore copy output for
+	 * both RTE_PCI_TPH_ST_SET and RTE_PCI_TPH_ST_GET

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1433

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+	 * cases.
+	 */
+	for (i = 0; i < count; i++) {
+		info[i].st = pci_tph->ents[i].st;
+		info[i].ph_ignore = pci_tph->ents[i].ph_ignore;
+	}
+
+out:
+	if (pci_tph)
+		rte_free(pci_tph);

Free functions work fine with null pointers, so just call free without a null check.

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1443

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_enable, 25.07)
+int
+pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode)
+{
+	int ret;
+
+	if (!(mode ^ (mode & VFIO_TPH_ST_MODE_MASK))) {

So it's an error to twice set the mode to the same thing? should it not just be a no-op?

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+		ret = -EINVAL;
+		goto out;
+	} else
+		mode &= VFIO_TPH_ST_MODE_MASK;
+
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_ENABLE | mode,
+		.count = 0
+	};

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1463

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+
+	ret = pci_vfio_tph_ioctl(dev, &pci_tph);
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_disable, 25.07)
+int
+pci_vfio_tph_disable(const struct rte_pci_device *dev)
+{

Check here, or in caller to see if it's already enabled?

Quoted 194 lines
drivers/bus/pci/linux/pci_vfio.c +25 -0
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_DISABLE,
+		.count = 0
+	};
+
+	return pci_vfio_tph_ioctl(dev, &pci_tph);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_get, 25.07)
+int
+pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count)
+{
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_GET);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_set, 25.07)
+int
+pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count)
+{
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_SET);
+}
+
 int
 pci_vfio_is_enabled(void)
 {
drivers/bus/pci/private.h +8 -0
index 38109844b9..d2ec370320 100644
@@ -335,4 +335,12 @@ rte_pci_dev_iterate(const void *start,
 int
 rte_pci_devargs_parse(struct rte_devargs *da);
 
+/*
+ * TPH Steering-Tag operation types.
+ */
+enum rte_pci_st_op {
+	RTE_PCI_TPH_ST_SET, /* Set TPH Steering - Tags */
+	RTE_PCI_TPH_ST_GET  /* Get TPH Steering - Tags */
+};
+
 #endif /* _PCI_PRIVATE_H_ */
drivers/bus/pci/rte_bus_pci.h +67 -0
index 19a7b15b99..e4d4780f54 100644
@@ -31,6 +31,7 @@ extern "C" {
 struct rte_pci_device;
 struct rte_pci_driver;
 struct rte_pci_ioport;
+struct rte_tph_info;
 
 struct rte_devargs;
 
@@ -312,6 +313,72 @@ void rte_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Enable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param mode
+ *   TPH mode the device must operate in.
+ */
+__rte_experimental
+int rte_pci_tph_enable(struct rte_pci_device *dev, int mode);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Disable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ */
+__rte_experimental
+int rte_pci_tph_disable(struct rte_pci_device *dev);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/windows/pci.c +43 -0
index e7e449306e..218e667a5a 100644
@@ -511,3 +511,46 @@ rte_pci_scan(void)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07)
+int
+rte_pci_tph_disable(struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07)
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07)
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
lib/pci/rte_pci.h +5 -0
index 9a50a12142..da9cd666bf 100644
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register */
 
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode Supported */
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector Mode Supported */
+#define RTE_PCI_TPH_CAP_ST_DS      0x00000004 /* Device Specific Mode Supported */

lib/pci/rte_pci.h extern "C" { · line 154

Quoted 10 lines
lib/pci/rte_pci.h +10 -0
+#define RTE_PCI_TPH_CAP_EXT_TPH    0x00000100 /* Ext TPH Requester Supported */
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location */
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+

Where are all these values used? They don't seem to be needed by this patch. If needed in later patches, I'd suggest adding them there.

Quoted 6 lines
lib/pci/rte_pci.h +0 -0
 /** Formatting string for PCI device identifier: Ex: 0000:00:01.0 */
 #define PCI_PRI_FMT "%.4" PRIx32 ":%.2" PRIx8 ":%.2" PRIx8 ".%" PRIx8
 #define PCI_PRI_STR_SIZE sizeof("XXXXXXXX:XX:XX.X")
Signature · 1 line
2.43.0
#44 lore raw

RE: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 11 lines
Quoted 7 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected

outcome.

Quoted 2 lines

This should be resolved by resetting the endpoint device before further attempts to set steering tags.

Quoted 7 lines

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

The V1 of the kernel patch had that feature, but it was frowned upon, and I was asked to implement the IOCTL this way. Please find it here (V1)
https://lore.kernel.org/kvm/[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Quoted 2 lines

Other comments inline below.

I will address them in the next version.

Thanks.

--wathsala

Quoted 85 lines

/Bruce

Quoted 20 lines
---
 drivers/bus/pci/bsd/pci.c        |  43 ++++++++
 drivers/bus/pci/bus_pci_driver.h |  52 ++++++++++
 drivers/bus/pci/linux/pci.c      | 100 ++++++++++++++++++
 drivers/bus/pci/linux/pci_init.h |  13 +++
drivers/bus/pci/linux/pci_vfio.c | 170 +++++++++++++++++++++++++++++++
 drivers/bus/pci/private.h        |   8 ++
 drivers/bus/pci/rte_bus_pci.h    |  67 ++++++++++++
 drivers/bus/pci/windows/pci.c    |  43 ++++++++
 lib/pci/rte_pci.h                |  15 +++
 9 files changed, 511 insertions(+)
drivers/bus/pci/bsd/pci.c +0 -0
index 5e2e09d5a4..dff750c4d6 100644

<snip>

drivers/bus/pci/bus_pci_driver.h struct rte_pci_device { · line 49

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +1 -0
index 2cc1119072..b1c2829fc1 100644
@@ -46,6 +46,7 @@ struct rte_pci_device {
 	char *bus_info;                     /**< PCI bus specific info */
 	struct rte_intr_handle *vfio_req_intr_handle;
 				/**< Handler of VFIO request interrupt */
+	uint8_t tph_enabled;                /**< TPH enabled on this device */

question: what would happen if we always enabled tph for each device. Does doing so disable the default handling for the device?

Quoted 19 lines
drivers/bus/pci/bus_pci_driver.h +13 -0
 };
 
 /**
@@ -194,6 +195,57 @@ struct rte_pci_ioport {
 	uint64_t len; /* only filled for memory mapped ports */  };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without
+prior
+ * notice
+ *
+ * This structure is passed into the TPH Steering-Tag set or get
+function as an
+ * argument by the caller. Return values are set in the same
+structure in st and
+ * ph_ignore fields by the calee.
+ *
+ * Refer to PCI-SIG ECN "Revised _DSM for Cache Locality TPH

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 219

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+Features" for
+ * details.
+ */
+struct rte_tph_info {
+	/* Input */
+	uint32_t cpu_id;	/*Logical CPU id*/
+	uint32_t cache_level;	/*Cache level relative to CPU. l1d=0,l2d=1,...*/
+	uint8_t flags;		/*Memory type, procesisng hint etc.*/
+	uint16_t index;		/*Index in vector table to store the ST*/
+

These fields should be reordered in order of size to avoid unnecessary gaps.

For the flags field, I dislike having different sets of flags all multiplexed into a single field. Can we instead of the flags field, and the set of #defines below split these out into separate enums, and then have separate fields for each one.

For example:

	struct rte_tph_info {
		uint32_t cpu_id;
		uint32_t cache_level;
		enum rte_tph_mem_type mem_type;
		enum rte_tph_hint hints;
		enum rte_tph_mode mode;
		...
	}
Quoted 5 lines

While the structure takes more space this way, this is not a datapath structure that we should be seeing large arrays of it, or that needs to be processed quickly, so usability should be prioritized over size/compactness.

+1

Quoted 30 lines
Quoted 4 lines
drivers/bus/pci/bus_pci_driver.h +4 -0
+	/* Output */
+	uint16_t st;		/*Steering tag returned by the platform*/
+	uint8_t ph_ignore;	/*Platform ignores PH for the returned ST*/
+};

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 230

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+
+#define RTE_PCI_TPH_MEM_TYPE_MASK		0x1
+#define RTE_PCI_TPH_MEM_TYPE_SHIFT		0
+/** Request volatile memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_VMEM		0
+/** Request persistent memory ST */
+#define RTE_PCI_TPH_MEM_TYPE_PMEM		1
+
+/** TLP Processing Hints - PCIe 6.0 specification section 2.2.7.1.1 */
+#define RTE_PCI_TPH_HINT_MASK		0x3

Looking at the mask usage below, does this mask not need to also be shifted by the TPH_HINT_SHIFT? Otherwise it overlaps with the type mask.

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport {

Quoted 5 lines
drivers/bus/pci/bus_pci_driver.h +5 -0
+#define RTE_PCI_TPH_HINT_SHIFT		1
+/** Host and device access data equally */
+#define RTE_PCI_TPH_HINT_BIDIR		0
+/** Device accesses data more frequently */
+#define RTE_PCI_TPH_HINT_REQSTR		(1 <<

RTE_PCI_TPH_HINT_SHIFT)

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 240

Quoted 2 lines
drivers/bus/pci/bus_pci_driver.h +2 -0
+/** Host access data more frequently */
+#define RTE_PCI_TPH_HINT_TARGET		(2 <<

RTE_PCI_TPH_HINT_SHIFT)

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 242

Quoted 2 lines
drivers/bus/pci/bus_pci_driver.h +2 -0
+/** Host access data more frequently with a high temporal locality */
+#define RTE_PCI_TPH_HINT_TARGET_PRIO	(3 <<

RTE_PCI_TPH_HINT_SHIFT)

Quoted 14 lines

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 248

Quoted 9 lines
drivers/bus/pci/bus_pci_driver.h +9 -0
+
+#define RTE_PCI_TPH_ST_MODE_MASK   0x3
+/** TPH no ST mode */
+#define RTE_PCI_TPH_ST_NS_MODE	   0
+/** TPH interrupt vector mode */
+#define RTE_PCI_TPH_ST_IV_MODE	   1
+/** TPH device specific mode */
+#define RTE_PCI_TPH_ST_DS_MODE	   2
+

As above, I think these would be nicer defined in different enums, going to separate fields in the struct. That would also remove any ambiguity as to whether the masks include the shift or not.

+1

Quoted 9 lines
Quoted 9 lines
drivers/bus/pci/bus_pci_driver.h +0 -0
 #ifdef __cplusplus
 }
 #endif
drivers/bus/pci/linux/pci.c +0 -0
index c20d159218..b5a8ba0a86 100644
@@ -814,3 +814,103 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
Quoted 10 lines

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 823

Quoted 7 lines
drivers/bus/pci/linux/pci.c +5 -0
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07) int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode) {
+	int ret = 0;
+

Should check here if dev->tph_enabled is already true.

+1

Quoted 64 lines
Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_enable(dev, mode);

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 838

Quoted 10 lines
drivers/bus/pci/linux/pci.c +10 -0
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	if (!ret)

Prefer "ret == 0" for this comparison.

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 843

Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+		dev->tph_enabled = 1;
+
+	return ret;
+}
+

Function could probably be shortened to something like (including a check for already enabled, 2 lines shorter if we rely on checks in the vfio_tph_enable() call):

int
rte_pci_tph_enable(...)
{

#ifdef VFIO_PRESENT
	if (dev->kdrv == RTE_PCI_KDRV_VFIO && pci_vfio_is_enabled()) {
		if (dev->tph_enabled == 0) {
			int ret = pci_vfio_tph_enable(...);
			if (ret != 0)
				return ret;
			dev->tph_enabled = 1;
		}
		return 0;
	}

#endif
return -ENOTSUP
}

Quoted 18 lines
drivers/bus/pci/linux/pci.c +18 -0
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07) int
+rte_pci_tph_disable(struct rte_pci_device *dev) {
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_disable(dev);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
Quoted 9 lines

drivers/bus/pci/linux/pci.c rte_pci_ioport_unmap(struct rte_pci_ioport *p) · line 868

Quoted 5 lines
drivers/bus/pci/linux/pci.c +5 -0
+	if (!ret)
+		dev->tph_enabled = 0;
+
+	return ret;
+}

As above, we can shorten this function by replacing the switch with a straight check for kdrv == RTE_PCI_KDRV_VFIO. Same with functions below too.

+1

Quoted 121 lines
Quoted 58 lines
drivers/bus/pci/linux/pci.c +44 -0
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07) int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count) {
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_st_get(dev, info, count);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07) int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count) {
+	int ret = 0;
+
+	switch (dev->kdrv) {
+#ifdef VFIO_PRESENT
+	case RTE_PCI_KDRV_VFIO:
+		if (pci_vfio_is_enabled())
+			ret = pci_vfio_tph_st_set(dev, info, count);
+		break;
+#endif
+	case RTE_PCI_KDRV_IGB_UIO:
+	case RTE_PCI_KDRV_UIO_GENERIC:
+	default:
+		ret = -ENOTSUP;
+		break;
+	}
+
+	return ret;
+}
drivers/bus/pci/linux/pci_init.h +1 -0
index 25b901f460..e71bfd2dce 100644
@@ -5,6 +5,7 @@
 #ifndef EAL_PCI_INIT_H_
 #define EAL_PCI_INIT_H_
 
+#include <rte_compat.h>
 #include <rte_vfio.h>
 #include <uapi/linux/vfio_tph.h>
 

drivers/bus/pci/linux/pci_init.h int pci_vfio_ioport_unmap(struct rte_pci_ioport · line 85

Quoted 9 lines
drivers/bus/pci/linux/pci_init.h +6 -0
@@ -76,6 +77,18 @@ int pci_vfio_ioport_unmap(struct rte_pci_ioport
 *p);  int pci_vfio_map_resource(struct rte_pci_device *dev);  int
 pci_vfio_unmap_resource(struct rte_pci_device *dev);
 
+/* TLP Processing Hints control functions */ __rte_experimental int
+pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode);
+__rte_experimental int pci_vfio_tph_disable(const struct
+rte_pci_device *dev); __rte_experimental int
+pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);

__rte_experimental

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device

Quoted 20 lines
drivers/bus/pci/linux/pci_init.h +3 -0
+int pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+			struct rte_tph_info *info, size_t ent_count);
+
 int pci_vfio_is_enabled(void);
 
 #endif
drivers/bus/pci/linux/pci_vfio.c +1 -0
index 5317170231..bdbeb38658 100644
@@ -12,6 +12,7 @@
 #include <stdbool.h>
 
 #include <rte_log.h>
+#include <eal_export.h>
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_eal_paging.h>
@@ -1316,6 +1317,175 @@ pci_vfio_mmio_write(const struct rte_pci_device

*dev, int bar,

Quoted 32 lines
drivers/bus/pci/linux/pci_vfio.c +30 -0
 	return pwrite(fd, buf, len, offset + offs);  }
 
+static int
+pci_vfio_tph_ioctl(const struct rte_pci_device *dev, struct
+vfio_pci_tph *pci_tph) {
+	const struct rte_intr_handle *intr_handle = dev->intr_handle;
+	int vfio_dev_fd = 0, ret = 0;
+
+	vfio_dev_fd = rte_intr_dev_fd_get(intr_handle);
+	if (vfio_dev_fd < 0) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = ioctl(vfio_dev_fd, VFIO_DEVICE_PCI_TPH, pci_tph);
+out:
+	return ret;
+}
+
+static int
+pci_vfio_tph_st_op(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count,
+		    enum rte_pci_st_op op)
+{
+	int ret = 0;
+	size_t argsz = 0, i;
+	struct vfio_pci_tph *pci_tph = NULL;
+	uint8_t mem_type = 0, hint = 0;
+
+	if (!count) {
+		ret = -EINVAL;
+		goto out;
Quoted 9 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1355

Quoted 6 lines
drivers/bus/pci/linux/pci_vfio.c +6 -0
+	}
+
+	argsz = sizeof(struct vfio_pci_tph) +
+		count * sizeof(struct vfio_pci_tph_entry);
+
+	pci_tph = rte_zmalloc(NULL, argsz, 0);

For ioctl we should not need pinned memory. Use regular malloc here.

+1

Quoted 4 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1356

Quoted 1 line
drivers/bus/pci/linux/pci_vfio.c +1 -0
+	if (!pci_tph) {

Coding style guidelines say to compare pointers explicitly to NULL.

+1

Quoted 53 lines
Quoted 27 lines
drivers/bus/pci/linux/pci_vfio.c +27 -0
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	pci_tph->argsz = argsz;
+	pci_tph->count = count;
+
+	switch (op) {
+	case RTE_PCI_TPH_ST_GET:
+		pci_tph->flags = VFIO_DEVICE_TPH_GET_ST;
+		break;
+	case RTE_PCI_TPH_ST_SET:
+		pci_tph->flags = VFIO_DEVICE_TPH_SET_ST;
+		break;
+	default:
+		ret = -EINVAL;
+		goto out;
+	}
+
+	for (i = 0; i < count; i++) {
+		pci_tph->ents[i].cpu_id = info[i].cpu_id;
+		pci_tph->ents[i].cache_level = info[i].cache_level;
+
+		mem_type = info[i].flags & RTE_PCI_TPH_MEM_TYPE_MASK;
+		switch (mem_type) {
+		case RTE_PCI_TPH_MEM_TYPE_VMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_VMEM;

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1393

Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+			break;
+		case RTE_PCI_TPH_MEM_TYPE_PMEM:
+			pci_tph->ents[i].flags |= VFIO_TPH_MEM_TYPE_PMEM;
+			break;
+		default:
+			ret = -EINVAL;
+			goto out;
+		}
+
+		hint = info[i].flags & RTE_PCI_TPH_HINT_MASK;

As pointed out above, unshifted, this HINT_MASK overlaps with the TYPE_MASK.

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar,

Quoted 12 lines
drivers/bus/pci/linux/pci_vfio.c +12 -0
+		switch (hint) {
+		case RTE_PCI_TPH_HINT_BIDIR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_BIDIR;
+			break;
+		case RTE_PCI_TPH_HINT_REQSTR:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_REQSTR;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET:
+			pci_tph->ents[i].flags |= VFIO_TPH_HINT_TARGET;
+			break;
+		case RTE_PCI_TPH_HINT_TARGET_PRIO:
+			pci_tph->ents[i].flags |=

VFIO_TPH_HINT_TARGET_PRIO;

Quoted 15 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1417

Quoted 12 lines
drivers/bus/pci/linux/pci_vfio.c +12 -0
+			break;
+		default:
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (op == RTE_PCI_TPH_ST_SET)
+			pci_tph->ents[i].index = info[i].index;
+	}
+
+	ret = pci_vfio_tph_ioctl(dev, pci_tph);
+	if (ret)

Again, check explicitly for "== 0".

+1

Quoted 10 lines
Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +10 -0
+		goto out;
+
+	/*
+	 * Kernel returns steering-tag and ph-ignore bits for
+	 * RTE_PCI_TPH_ST_SET too, therefore copy output for
+	 * both RTE_PCI_TPH_ST_SET and RTE_PCI_TPH_ST_GET
+	 * cases.
+	 */
+	for (i = 0; i < count; i++) {
+		info[i].st = pci_tph->ents[i].st;
Quoted 9 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1433

Quoted 6 lines
drivers/bus/pci/linux/pci_vfio.c +6 -0
+		info[i].ph_ignore = pci_tph->ents[i].ph_ignore;
+	}
+
+out:
+	if (pci_tph)
+		rte_free(pci_tph);

Free functions work fine with null pointers, so just call free without a null check.

+1

Quoted 12 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, · line 1443

Quoted 8 lines
drivers/bus/pci/linux/pci_vfio.c +8 -0
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_enable, 25.07) int
+pci_vfio_tph_enable(const struct rte_pci_device *dev, int mode) {
+	int ret;
+
+	if (!(mode ^ (mode & VFIO_TPH_ST_MODE_MASK))) {

So it's an error to twice set the mode to the same thing? should it not just be a no-op?

This doesn't make sense; I will fix this in next version.

Quoted 12 lines
Quoted 12 lines
drivers/bus/pci/linux/pci_vfio.c +12 -0
+		ret = -EINVAL;
+		goto out;
+	} else
+		mode &= VFIO_TPH_ST_MODE_MASK;
+
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_ENABLE | mode,
+		.count = 0
+	};
+
+	ret = pci_vfio_tph_ioctl(dev, &pci_tph);
Quoted 9 lines

drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar,

Quoted 6 lines
drivers/bus/pci/linux/pci_vfio.c +6 -0
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_disable, 25.07) int
+pci_vfio_tph_disable(const struct rte_pci_device *dev) {

Check here, or in caller to see if it's already enabled?

This check should happen in rte_pci_tph_disable() rather than here which is specific for Linux. I will do the check where you pointed it out earlier.

Quoted 182 lines
Quoted 169 lines
drivers/bus/pci/linux/pci_vfio.c +18 -0
+	struct vfio_pci_tph pci_tph = {
+		.argsz = sizeof(struct vfio_pci_tph),
+		.flags = VFIO_DEVICE_TPH_DISABLE,
+		.count = 0
+	};
+
+	return pci_vfio_tph_ioctl(dev, &pci_tph); }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_get, 25.07) int
+pci_vfio_tph_st_get(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count) {
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_GET); }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(pci_vfio_tph_st_set, 25.07) int
+pci_vfio_tph_st_set(const struct rte_pci_device *dev,
+		    struct rte_tph_info *info, size_t count) {
+	return pci_vfio_tph_st_op(dev, info, count, RTE_PCI_TPH_ST_SET); }
+
 int
 pci_vfio_is_enabled(void)
 {
drivers/bus/pci/private.h +7 -0
index 38109844b9..d2ec370320 100644
@@ -335,4 +335,12 @@ rte_pci_dev_iterate(const void *start,  int
 rte_pci_devargs_parse(struct rte_devargs *da);
 
+/*
+ * TPH Steering-Tag operation types.
+ */
+enum rte_pci_st_op {
+	RTE_PCI_TPH_ST_SET, /* Set TPH Steering - Tags */
+	RTE_PCI_TPH_ST_GET  /* Get TPH Steering - Tags */ };
+
 #endif /* _PCI_PRIVATE_H_ */
(quoted diff) +0 -0

b/drivers/bus/pci/rte_bus_pci.h index 19a7b15b99..e4d4780f54 100644

drivers/bus/pci/rte_bus_pci.h +67 -0
@@ -31,6 +31,7 @@ extern "C" {
 struct rte_pci_device;
 struct rte_pci_driver;
 struct rte_pci_ioport;
+struct rte_tph_info;
 
 struct rte_devargs;
 
@@ -312,6 +313,72 @@ void rte_pci_ioport_read(struct rte_pci_ioport
 *p,  void rte_pci_ioport_write(struct rte_pci_ioport *p,
 		const void *data, size_t len, off_t offset);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Enable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ * @param mode
+ *   TPH mode the device must operate in.
+ */
+__rte_experimental
+int rte_pci_tph_enable(struct rte_pci_device *dev, int mode);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Disable TLP Processing Hints (TPH) in the endpoint device.
+ *
+ * @param dev
+ *   A pointer to a rte_pci_device structure describing the device
+ *   to use.
+ */
+__rte_experimental
+int rte_pci_tph_disable(struct rte_pci_device *dev);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Get PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Set PCI Steering-Tags (STs) for a list of stashing targets.
+ *
+ * @param mode
+ *   TPH mode the device must operate in.
+ * @param info
+ *   An array of rte_tph_info objects, each describing the target
+ *   cpu-id, cache-level, etc. Steering-tags for each target is
+ *   eturned via info array.
+ * @param count
+ *   The number of elements in the info array.
+ */
+__rte_experimental
+int rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		struct rte_tph_info *info, size_t count);
+
 #ifdef __cplusplus
 }
 #endif
(quoted diff) +0 -0

b/drivers/bus/pci/windows/pci.c index e7e449306e..218e667a5a 100644

drivers/bus/pci/windows/pci.c +35 -0
@@ -511,3 +511,46 @@ rte_pci_scan(void)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07) int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode) {
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07) int
+rte_pci_tph_disable(struct rte_pci_device *dev) {
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07) int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count) {
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07) int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count) {
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for windows */
+	return -1;
+}
lib/pci/rte_pci.h index renamed from lib/pci/rte_pci.h +0 -0

9a50a12142..da9cd666bf 100644

lib/pci/rte_pci.h extern "C" { · line 142

Quoted 6 lines
lib/pci/rte_pci.h +3 -0
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register */
 
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode Supported

*/

lib/pci/rte_pci.h extern "C" { · line 143

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector Mode

Supported */

lib/pci/rte_pci.h extern "C" { · line 144

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_DS      0x00000004 /* Device Specific Mode

Supported */

lib/pci/rte_pci.h extern "C" { · line 145

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_EXT_TPH    0x00000100 /* Ext TPH Requester

Supported */

Quoted 13 lines

lib/pci/rte_pci.h extern "C" { · line 154

Quoted 9 lines
lib/pci/rte_pci.h +9 -0
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location */
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+

Where are all these values used? They don't seem to be needed by this patch. If needed in later patches, I'd suggest adding them there.

RTE_PCI_TPH_CAP_ST_NS, RTE_PCI_TPH_CAP_ST_IV and RTE_PCI_TPH_CAP_ST_DS are used by drivers. I40e patch uses RTE_PCI_TPH_CAP_ST_DS. I will remove the rest, added here for completeness.

Quoted 6 lines
Quoted 6 lines
lib/pci/rte_pci.h +0 -1
 /** Formatting string for PCI device identifier: Ex: 0000:00:01.0 */
 #define PCI_PRI_FMT "%.4" PRIx32 ":%.2" PRIx8 ":%.2" PRIx8 ".%" PRIx8
 #define PCI_PRI_STR_SIZE sizeof("XXXXXXXX:XX:XX.X")
--

2.43.0

#45 lore raw

Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 34 lines · Wathsala Wathawana Vithanage
Quoted 11 lines
Quoted 7 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected

outcome.

Quoted 2 lines

This should be resolved by resetting the endpoint device before further attempts to set steering tags.

Quoted 7 lines

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

The V1 of the kernel patch had that feature, but it was frowned upon, and I was asked to implement the IOCTL this way. Please find it here (V1)
https://lore.kernel.org/kvm/[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Quoted 2 lines

Other comments inline below.

I will address them in the next version.

Thanks.

--wathsala

Quoted 2 lines

/Bruce

<snip>

Quoted 32 lines
Quoted 18 lines

lib/pci/rte_pci.h extern "C" { · line 142

Quoted 11 lines
lib/pci/rte_pci.h index renamed from lib/pci/rte_pci.h +0 -0

9a50a12142..da9cd666bf 100644

lib/pci/rte_pci.h +3 -0
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register */
 
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode Supported

*/

lib/pci/rte_pci.h extern "C" { · line 143

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector Mode

Supported */

lib/pci/rte_pci.h extern "C" { · line 144

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_DS      0x00000004 /* Device Specific Mode

Supported */

lib/pci/rte_pci.h extern "C" { · line 145

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_EXT_TPH    0x00000100 /* Ext TPH Requester

Supported */

Quoted 13 lines

lib/pci/rte_pci.h extern "C" { · line 154

Quoted 9 lines
lib/pci/rte_pci.h +9 -0
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location */
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+

Where are all these values used? They don't seem to be needed by this patch. If needed in later patches, I'd suggest adding them there.

Quoted 4 lines

RTE_PCI_TPH_CAP_ST_NS, RTE_PCI_TPH_CAP_ST_IV and RTE_PCI_TPH_CAP_ST_DS are used by drivers. I40e patch uses RTE_PCI_TPH_CAP_ST_DS. I will remove the rest, added here for completeness.

Having them all for completeness is fine. You can keep this as-is in next version then.

/Bruce

#46 lore raw

RE: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 91 lines

-----Original Message-----
From: Bruce Richardson <[email protected]>
Sent: Thursday, June 5, 2025 2:51 AM
To: Wathsala Wathawana Vithanage <[email protected]>
Cc: Chenbo Xia <[email protected]>; Nipun Gupta <[email protected]>; Anatoly Burakov <[email protected]>; Gaetan Rivet <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Dhruv Tripathi <[email protected]>
Subject: Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 37 lines · Wathsala Wathawana Vithanage
Quoted 12 lines
Quoted 8 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected

outcome.

Quoted 2 lines

This should be resolved by resetting the endpoint device before further attempts to set steering tags.

Quoted 7 lines

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

The V1 of the kernel patch had that feature, but it was frowned upon, and I was asked to implement the IOCTL this way. Please find it here (V1)
https://lore.kernel.org/kvm/20250221224638.1836909-1-wathsala.vithanag
[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Quoted 2 lines

Other comments inline below.

I will address them in the next version.

Thanks.

--wathsala

Quoted 2 lines

/Bruce

<snip>

Quoted 7 lines
Quoted 7 lines

lib/pci/rte_pci.h extern "C" { · line 138

Quoted 7 lines
lib/pci/rte_pci.h index renamed from lib/pci/rte_pci.h +0 -0

9a50a12142..da9cd666bf 100644

lib/pci/rte_pci.h +0 -0
@@ -137,6 +137,21 @@ extern "C" {
 /* Process Address Space ID (RTE_PCI_EXT_CAP_ID_PASID) */
 #define RTE_PCI_PASID_CTRL		0x06    /* PASID control register

*/

Quoted 4 lines
Quoted 4 lines

lib/pci/rte_pci.h extern "C" {

Quoted 4 lines
lib/pci/rte_pci.h +3 -0
+/* TPH Requester */
+#define RTE_PCI_TPH_CAP            4       /* capability register */
+#define RTE_PCI_TPH_CAP_ST_NS      0x00000001 /* No ST Mode

Supported

Quoted 2 lines
Quoted 2 lines

*/

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_IV      0x00000002 /* Interrupt Vector

Mode

Quoted 2 lines
Quoted 2 lines

Supported */

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_ST_DS      0x00000004 /* Device Specific

Mode

Quoted 4 lines
Quoted 4 lines

Supported */

lib/pci/rte_pci.h extern "C" { · line 144

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_EXT_TPH    0x00000100 /* Ext TPH Requester

Supported */

Quoted 1 line
lib/pci/rte_pci.h +1 -0
+#define RTE_PCI_TPH_CAP_LOC_MASK   0x00000600 /* ST Table Location

*/

Quoted 13 lines
Quoted 12 lines

lib/pci/rte_pci.h extern "C" { · line 154

Quoted 8 lines
lib/pci/rte_pci.h +8 -0
+#define RTE_PCI_TPH_LOC_NONE       0x00000000 /* Not present */
+#define RTE_PCI_TPH_LOC_CAP        0x00000200 /* In capability */
+#define RTE_PCI_TPH_LOC_MSIX       0x00000400 /* In MSI-X */
+#define RTE_PCI_TPH_CAP_ST_MASK    0x07FF0000 /* ST Table Size */
+#define RTE_PCI_TPH_CAP_ST_SHIFT   16      /* ST Table Size shift */
+#define RTE_PCI_TPH_BASE_SIZEOF    0xc     /* Size with no ST table */
+
+

Where are all these values used? They don't seem to be needed by this patch. If needed in later patches, I'd suggest adding them there.

Quoted 9 lines
Quoted 1 line

RTE_PCI_TPH_CAP_ST_NS, RTE_PCI_TPH_CAP_ST_IV and

RTE_PCI_TPH_CAP_ST_DS

Quoted 3 lines

are used by drivers. I40e patch uses RTE_PCI_TPH_CAP_ST_DS. I will remove the rest, added here for completeness.

Having them all for completeness is fine. You can keep this as-is in next version then.

+1

--wathsala

Quoted 1 line

/Bruce

#47 lore raw

Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

In reply to Wathsala Wathawana Vithanage
Quoted 15 lines · Wathsala Wathawana Vithanage
Quoted 15 lines
Quoted 8 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected

outcome.

Quoted 2 lines

This should be resolved by resetting the endpoint device before further attempts to set steering tags.

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

Quoted 9 lines
Quoted 4 lines

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

The V1 of the kernel patch had that feature, but it was frowned upon, and I was asked to implement the IOCTL this way. Please find it here (V1)
https://lore.kernel.org/kvm/[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Read the thread. However, from my reading, there is nothing in there that mandates having an interface where the user won't know the state on error. We need some method to have userspace know what tags were applied or not on failure. Resetting the whole device is not a good solution. Whatever API is provided, if it is going to take multiple ops in one go it needs to either return the number applied on failure, or if just returning success/failure, it should rollback the successful ones to give an all-or-nothing interface.

/Bruce

#48 lore raw

RE: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 48 lines

-----Original Message-----
From: Bruce Richardson <[email protected]>
Sent: Thursday, June 5, 2025 5:19 AM
To: Wathsala Wathawana Vithanage <[email protected]>
Cc: Chenbo Xia <[email protected]>; Nipun Gupta <[email protected]>; Anatoly Burakov <[email protected]>; Gaetan Rivet <[email protected]>; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Dhruv Tripathi <[email protected]>
Subject: Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

Quoted 20 lines · Wathsala Wathawana Vithanage
Quoted 12 lines
Quoted 8 lines

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected

outcome.

Quoted 2 lines

This should be resolved by resetting the endpoint device before further attempts to set steering tags.

Quoted 7 lines

This seems very clunky for the user. Is there a fundamental reason why we cannot report out what ones passed or failed?

If it's a limitation of the kernel IOCTL, how about just making one ioctl for each individual op requested, one at a time. That way we will know what failed to report it?

Quoted 6 lines

The V1 of the kernel patch had that feature, but it was frowned upon, and I was asked to implement the IOCTL this way. Please find it here (V1)
https://lore.kernel.org/kvm/20250221224638.1836909-1-wathsala.vithanag
[email protected]/T/#me73cf9b9c87da97d7d9461dfb97863b78ca1755b

Read the thread. However, from my reading, there is nothing in there that mandates having an interface where the user won't know the state on error. We need some method to have userspace know what tags were applied or not on failure. Resetting the whole device is not a good solution. Whatever API is provided, if it is going to take multiple ops in one go it needs to either return the number applied on failure, or if just returning success/failure, it should rollback the successful ones to give an all-or-nothing interface.

Quoted 1 line

/Bruce

I will bring this up in the V2 review. If there is a pushback we can alternatively change the API to do one tag at a time.

Thanks.

#49 lore raw

Re: [PATCH v5 2/4] bus/pci: introduce the PCIe TLP Processing Hints API

In reply to Wathsala Vithanage
Quoted 123 lines · Wathsala Vithanage

Extend the PCI bus driver to enable or disable TPH capability and set or get PCI Steering-Tags (STs) on an endpoint device. The functions rte_pci_tph_{enable, disable,st_set,st_get} provide the primary interface for DPDK device drivers. Implementation of the interface is OS dependent. For Linux, the kernel VFIO driver provides the implementation. rte_pci_tph_{enable, disable} functions enable and disable TPH capability, respectively. rte_pci_tph_enable enables TPH on the device in either of the device-specific, interrupt-vector, or no-steering-tag modes.

rte_pci_tph_st_{get, set} functions take an array of rte_tph_info objects with cpu-id, cache-level, flags (processing-hint, memory-type). The index in rte_tph_info is the MSI-X/MSI vector/ST-table index if TPH was enabled in the interrupt-vector mode; the rte_pci_tph_st_get function ignores it. Both rte_pci_tph_st_{set, get} functions return the steering-tag (st) and processing-hint-ignored (ph_ignore) fields via the same rte_tph_info object passed into them.

rte_pci_tph_st_{get, set} functions will return an error if processing any of the rte_tph_info objects fails. The API does not indicate which entry in the rte_tph_info array was executed successfully and which caused an error. Therefore, in case of an error, the caller should discard the output. If rte_pci_tph_set returns an error, it should be treated as a partial error. Hence, the steering-tag update on the device should be considered partial and inconsistent with the expected outcome. This should be resolved by resetting the endpoint device before further attempts to set steering tags.

---
 drivers/bus/pci/bsd/pci.c        |  43 ++++++++
 drivers/bus/pci/bus_pci_driver.h |  52 ++++++++++
 drivers/bus/pci/linux/pci.c      | 100 ++++++++++++++++++
 drivers/bus/pci/linux/pci_init.h |  13 +++
 drivers/bus/pci/linux/pci_vfio.c | 170 +++++++++++++++++++++++++++++++
 drivers/bus/pci/private.h        |   8 ++
 drivers/bus/pci/rte_bus_pci.h    |  67 ++++++++++++
 drivers/bus/pci/windows/pci.c    |  43 ++++++++
 lib/pci/rte_pci.h                |  15 +++
 9 files changed, 511 insertions(+)
drivers/bus/pci/bsd/pci.c +43 -0
index 5e2e09d5a4..dff750c4d6 100644
@@ -650,3 +650,46 @@ rte_pci_ioport_unmap(struct rte_pci_ioport *p)
 
 	return ret;
 }
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_enable, 25.07)
+int
+rte_pci_tph_enable(struct rte_pci_device *dev, int mode)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(mode);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_disable, 25.07)
+int
+rte_pci_tph_disable(struct rte_pci_device *dev)
+{
+	RTE_SET_USED(dev);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_get, 25.07)
+int
+rte_pci_tph_st_get(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_pci_tph_st_set, 25.07)
+int
+rte_pci_tph_st_set(const struct rte_pci_device *dev,
+		   struct rte_tph_info *info, size_t count)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info);
+	RTE_SET_USED(count);
+	/* This feature is not yet implemented for BSD */
+	return -1;
+}
drivers/bus/pci/bus_pci_driver.h +14 -0
index 2cc1119072..b1c2829fc1 100644
@@ -46,6 +46,7 @@ struct rte_pci_device {
 	char *bus_info;                     /**< PCI bus specific info */
 	struct rte_intr_handle *vfio_req_intr_handle;
 				/**< Handler of VFIO request interrupt */
+	uint8_t tph_enabled;                /**< TPH enabled on this device */
 };
 
 /**
@@ -194,6 +195,57 @@ struct rte_pci_ioport {
 	uint64_t len; /* only filled for memory mapped ports */
 };
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without prior
+ * notice
+ *
+ * This structure is passed into the TPH Steering-Tag set or get function as an
+ * argument by the caller. Return values are set in the same structure in st and
+ * ph_ignore fields by the calee.
+ *
+ * Refer to PCI-SIG ECN "Revised _DSM for Cache Locality TPH Features" for
+ * details.
+ */
+struct rte_tph_info {

drivers/bus/pci/bus_pci_driver.h struct rte_pci_ioport { · line 220

Quoted 10 lines
drivers/bus/pci/bus_pci_driver.h +10 -0
+	/* Input */
+	uint32_t cpu_id;	/*Logical CPU id*/
+	uint32_t cache_level;	/*Cache level relative to CPU. l1d=0,l2d=1,...*/
+	uint8_t flags;		/*Memory type, procesisng hint etc.*/
+	uint16_t index;		/*Index in vector table to store the ST*/
+
+	/* Output */
+	uint16_t st;		/*Steering tag returned by the platform*/
+	uint8_t ph_ignore;	/*Platform ignores PH for the returned ST*/
+};

Looking at the driver implementation in patch 4, I realised this use of the info struct for "get" API is very confusing. You partially populate the structure, then make an API call, passing that struct as input, and output is also filled into other fields of the same structure.

I dislike having the input and output fields of the structure mixed. Can we separate out the last two fields here into a separate output struct, or alternatively just drop them from the struct and have the get API take two additional arrays as output, one for the tags and another for the ph_ignore fields.

[Note: documentation also needs to cover what PH is]

/Bruce

#50 lore raw

PATCH v5 3/4 ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---
 lib/ethdev/ethdev_driver.h |  66 ++++++++++++++++
 lib/ethdev/rte_ethdev.c    | 149 ++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 158 +++++++++++++++++++++++++++++++++++++
 3 files changed, 373 insertions(+)
lib/ethdev/ethdev_driver.h +66 -0
index 2b4d2ae9c3..8a4012db08 100644
@@ -1376,6 +1376,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -1402,6 +1464,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c +149 -0

160 lines not on this page · raw message

lib/ethdev/rte_ethdev.h +158 -0

176 lines not on this page · raw message

Signature · 1 line
2.43.0
#51 lore raw

RE: [PATCH v5 3/4] ethdev: introduce the cache stashing hints API

Quoted 29 lines

From: Wathsala Vithanage [mailto:[email protected]]
Sent: Tuesday, 3 June 2025 00.38

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

Quoted 7 lines

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

Nitpicking:

In DPDK, "objects" usually refer to mbufs, mempool elements or similar.
So please use the term "flags" instead of "objects". Also in the macros, definitions and comments.

And DPDK usually uses 64 bit for flags, so they should be uint64_t. Unless there is a fast path performance related reason to make them shorter.

If you use the same flags for both RX and TX, instead of providing separate definitions for TX flags, the descriptions of the flags should reflect this.

The capabilities APIs should output the list/range of valid "offset" values for RX and TX.

And if the offset cannot be negative, please use an unsigned type for it.

The lib/ethdev/rte_ethdev.c follows the design pattern of setting rte_errno and returning -1 on error. Please follow the same design pattern in the new functions added to the rte_ethdev library.

#52 lore raw

Re: [PATCH v5 3/4] ethdev: introduce the cache stashing hints API

In reply to Wathsala Vithanage
Quoted 26 lines · Wathsala Vithanage

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

Quoted 8 lines

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

---

Few small comments inline below

/Bruce

Quoted 39 lines
 lib/ethdev/ethdev_driver.h |  66 ++++++++++++++++
 lib/ethdev/rte_ethdev.c    | 149 ++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 158 +++++++++++++++++++++++++++++++++++++
 3 files changed, 373 insertions(+)
lib/ethdev/ethdev_driver.h +26 -0
index 2b4d2ae9c3..8a4012db08 100644
@@ -1376,6 +1376,68 @@ enum rte_eth_dev_operation {
 typedef uint64_t (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *
+ * @param dev
+ *   Port (ethdev) handle.

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { · line 1414

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.

What about on failure of the underlying ioctl call?

Quoted 92 lines
lib/ethdev/ethdev_driver.h +30 -0
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t queue_id,
+					   struct rte_eth_stashing_config *config);
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet driver.
  */
@@ -1402,6 +1464,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache stashing*/
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache stashing*/
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c +37 -0
index d4197322a0..ae666c370b 100644
@@ -158,6 +158,7 @@ static const struct {
 	{RTE_ETH_DEV_CAPA_RXQ_SHARE, "RXQ_SHARE"},
 	{RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP, "FLOW_RULE_KEEP"},
 	{RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP, "FLOW_SHARED_OBJECT_KEEP"},
+	{RTE_ETH_DEV_CAPA_CACHE_STASHING, "CACHE_STASHING"},
 };
 
 enum {
@@ -7419,5 +7420,153 @@ int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,
 	return ret;
 }
 
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_validate_stashing_config, 25.07)
+int
+rte_eth_dev_validate_stashing_config(uint16_t port_id, uint16_t queue_id,
+				     uint8_t queue_direction,
+				     struct rte_eth_stashing_config *config)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+	int ret = 0;
+	uint16_t nb_queues;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (!config) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid stashing configuration");
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/*
+	 * Check for invalid objects
+	 */
+	if (!RTE_ETH_DEV_STASH_OBJECTS_VALID(config->objects)) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid stashing objects");
+		ret = -EINVAL;
+		goto out;
+	}
+
+	dev = &rte_eth_devices[port_id];
+
+	nb_queues = (queue_direction == RTE_ETH_DEV_RX_QUEUE) ?
+				      dev->data->nb_rx_queues :
+				      dev->data->nb_tx_queues;
+
+	if (queue_id >= nb_queues) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid Rx queue_id=%u", queue_id);

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id, · line 7468

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret < 0)
+		goto out;
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING) {

Nit: check if all this can fit on one line under 100 chars. I think it can.

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id, · line 7476

Quoted 8 lines
lib/ethdev/rte_ethdev.c +8 -0
+		ret = -ENOTSUP;
+		goto out;
+	}
+
+	if (*dev->dev_ops->stashing_rx_hints_set == NULL ||
+	    *dev->dev_ops->stashing_tx_hints_set == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not implemented "
+				    "in %s for %s", dev_info.driver_name,

Don't split error messages across lines. Text strings are allowed to go over the 100 char limit if necessary to avoid splitting.

Quoted 119 lines
lib/ethdev/rte_ethdev.c +94 -0
+				    dev_info.device->name);
+		ret = -ENOSYS;
+	}
+
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_rx_config_set, 25.07)
+int
+rte_eth_dev_stashing_rx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config)
+{
+	struct rte_eth_dev *dev;
+	int ret = 0;
+
+	ret = rte_eth_dev_validate_stashing_config(port_id, queue_id,
+						   RTE_ETH_DEV_RX_QUEUE,
+						   config);
+	if (ret < 0)
+		goto out;
+
+	dev = &rte_eth_devices[port_id];
+
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_rx_hints_set)(dev, queue_id,
+		      config));
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_tx_config_set, 25.07)
+int
+rte_eth_dev_stashing_tx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config)
+{
+	struct rte_eth_dev *dev;
+	int ret = 0;
+
+	ret = rte_eth_dev_validate_stashing_config(port_id, queue_id,
+						   RTE_ETH_DEV_TX_QUEUE,
+						   config);
+	if (ret < 0)
+		goto out;
+
+	dev = &rte_eth_devices[port_id];
+
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_rx_hints_set) (dev, queue_id,
+		       config));
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_capabilities_get, 25.07)
+int
+rte_eth_dev_stashing_capabilities_get(uint16_t port_id, uint16_t *objects)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+	int ret = 0;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (!objects) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	dev = &rte_eth_devices[port_id];
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret < 0)
+		goto out;
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING) {
+		ret = -ENOTSUP;
+		goto out;
+	}
+
+	if (*dev->dev_ops->stashing_capabilities_get == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not implemented "
+				    "in %s for %s", dev_info.driver_name,
+				    dev_info.device->name);
+		ret = -ENOSYS;
+		goto out;
+	}
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_capabilities_get)(dev, objects));
+out:
+	return ret;
+}
+
 RTE_EXPORT_SYMBOL(rte_eth_dev_logtype)
 RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO);
lib/ethdev/rte_ethdev.h +4 -0
index ea7f8c4a1a..1398f8c837 100644
@@ -1667,6 +1667,9 @@ struct rte_eth_conf {
 #define RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP RTE_BIT64(4)
 /**@}*/
 
+/** Device supports stashing to CPU/system caches. */
+#define RTE_ETH_DEV_CAPA_CACHE_STASHING RTE_BIT64(5)
+
 /*
  * Fallback default preferred Rx/Tx port parameters.
  * These are used if an application requests default parameters
@@ -1838,6 +1841,7 @@ struct rte_eth_dev_info {
 	struct rte_eth_dev_portconf default_txportconf;
 	/** Generic device capabilities (RTE_ETH_DEV_CAPA_). */
 	uint64_t dev_capa;
+	uint16_t stashing_capa;
 	/**
 	 * Switching information for ports on a device with a
 	 * embedded managed interconnect/switch.

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 9 lines
lib/ethdev/rte_ethdev.h +6 -0
@@ -6173,6 +6177,160 @@ int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *
 __rte_experimental
 int rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config);
 
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE		0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE		1
+

I'd prefer an enum for these.
Why are the necessary since we have separate rx and tx functions for the caching hints.

Quoted 11 lines
lib/ethdev/rte_ethdev.h +11 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice
+ *
+ * A structure used for configuring the cache stashing hints.
+ */
+struct rte_eth_stashing_config {
+	/**
+	 * lcore_id of the processor the stashing hints are applied to.
+	 */

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6206

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+	uint32_t	lcore_id;
+	/**
+	 * Zero based cache level relative to the CPU.
+	 * E.g. l1d = 0, l2d = 1,...
+	 */
+	uint32_t	cache_level;
+	/**
+	 * Object types the configuration is applied to
+	 */
+	uint16_t	objects;

What are the objects? That needs to be covered by the docs, or make this an enum type so that it's clear from the typesystem what it applies to [though that would require an array of objects and count, it may be clearer for user].

Quoted 133 lines
lib/ethdev/rte_ethdev.h +127 -0
+	/**
+	 * The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+	 *  in objects
+	 */
+	int		offset;
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the ethernet device
+ *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET		0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC		0x0002
+
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER		0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD	0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK		0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)				\
+	((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * @internal
+ * Helper function to validate stashing hints configuration.
+ */
+__rte_experimental
+int rte_eth_dev_validate_stashing_config(uint16_t port_id, uint16_t queue_id,
+					 uint8_t queue_direction,
+					 struct rte_eth_stashing_config *config);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Provide cache stashing hints for improved memory access latencies for
+ * packets received by the NIC.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param queue_id
+ *  The index of the receive queue to which hints are applied.
+ * @param config
+ *  Stashing configuration.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX object types used in conjuection in objects
+ *  parameter.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is unavailable.
+ *  - (-ENOSYS) if PMD does not implement cache stashing hints.
+ *  - (0) on Success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_rx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Configure cache stashing for improved memory access latencies for Tx
+ * queue completion descriptors being sent to host system by the NIC.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param queue_id
+ *  The index of the receive queue to which hints are applied.
+ * @param config
+ *  Stashing configuration.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX object types are used in conjuection in objects
+ *  parameter.
+ *  - (-EINVAL) if hints are incompatible with TX queues.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is unavailable.
+ *  - (-ENOSYS) if PMD does not implement cache stashing hints.
+ *  - (0) on Success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_tx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Discover cache stashing objects supported in the ethernet device.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param objects
+ *  Supported objects vector set by the ethernet device.
+ * @return
+ *  On return types and hints parameters will have bits set for supported
+ *  object types.
+ *  - (-ENOTSUP) if the device or the platform does not support cache stashing.
+ *  - (-ENOSYS)  if the underlying PMD hasn't implemented cache stashing
+ *  feature.
+ *  - (-EINVAL)  on NULL values for types or hints parameters.
+ *  - (0) on success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_capabilities_get(uint16_t port_id, uint16_t *objects);
+
 #ifdef __cplusplus
 }
 #endif
Signature · 1 line
2.43.0
#53 lore raw

RE: [PATCH v5 3/4] ethdev: introduce the cache stashing hints API

Quoted 31 lines · Wathsala Vithanage
Quoted 30 lines

Extend the ethdev library to enable the stashing of different data objects, such as the ones listed below, into CPU caches directly from the NIC.

- Rx/Tx queue descriptors
- Rx packets
- Packet headers
- packet payloads
- Data of a packet at an offset from the start of the packet

The APIs are designed in a hardware/vendor agnostic manner such that supporting PMDs could use any capabilities available in the underlying hardware for fine-grained stashing of data objects into a CPU cache

The API provides an interface to query the availability of stashing capabilities, i.e., platform/NIC support, stashable object types, etc, via the rte_eth_dev_stashing_capabilities_get interface.

The function pair rte_eth_dev_stashing_rx_config_set and rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU, cache level, and data object types) on the Rx and Tx queues.

PMDs that support stashing must register their implementations with the following eth_dev_ops callbacks, which are invoked by the ethdev functions listed above.

- stashing_capabilities_get
- stashing_rx_hints_set
- stashing_tx_hints_set

Quoted 7 lines
Quoted 4 lines

---

Few small comments inline below

Thank you, Bruce. I will address them in the next version. As asked by @stephen@networkplumber I will only send it only if kernel piece makes it into Linus branch.

--wathsala

Quoted 493 lines

/Bruce

Quoted 17 lines
 lib/ethdev/ethdev_driver.h |  66 ++++++++++++++++
 lib/ethdev/rte_ethdev.c    | 149 ++++++++++++++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h    | 158 +++++++++++++++++++++++++++++++++++++
 3 files changed, 373 insertions(+)
lib/ethdev/ethdev_driver.h +4 -0
index 2b4d2ae9c3..8a4012db08 100644
@@ -1376,6 +1376,68 @@ enum rte_eth_dev_operation {  typedef uint64_t
 (*eth_get_restore_flags_t)(struct rte_eth_dev *dev,
 					    enum rte_eth_dev_operation op);
 
+/**
+ * @internal
+ * Set cache stashing hints in Rx queue.
+ *

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { typedef uint64_t · line 1392

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Rx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 4 lines
lib/ethdev/ethdev_driver.h +4 -0
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_rx_hints_set_t)(struct rte_eth_dev *dev, uint16_t

queue_id,

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { typedef uint64_t · line 1397

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+					   struct rte_eth_stashing_config

*config);

Quoted 5 lines
lib/ethdev/ethdev_driver.h +5 -0
+
+/**
+ * @internal
+ * Set cache stashing hints in Tx queue.
+ *

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param queue_id
+ *   Tx queue.
+ * @param config
+ *   Stashing hints configuration for the queue.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { typedef uint64_t · line 1414

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.

What about on failure of the underlying ioctl call?

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { typedef uint64_t · line 1416

Quoted 2 lines
lib/ethdev/ethdev_driver.h +2 -0
+ */
+typedef int (*eth_stashing_tx_hints_set_t)(struct rte_eth_dev *dev, uint16_t

queue_id,

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation { typedef uint64_t · line 1417

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+					   struct rte_eth_stashing_config

*config);

Quoted 5 lines
lib/ethdev/ethdev_driver.h +5 -0
+
+/**
+ * @internal
+ * Get cache stashing object types supported in the ethernet device.
+ * The return value indicates availability of stashing hints support

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+ * in the hardware and the PMD.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param objects
+ *   PMD sets supported bits on return.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing

feature.

lib/ethdev/ethdev_driver.h enum rte_eth_dev_operation {

Quoted 10 lines
lib/ethdev/ethdev_driver.h +8 -0
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_stashing_capabilities_get_t)(struct rte_eth_dev *dev,
+					     uint16_t *objects);
+
 /**
  * @internal A structure containing the functions exported by an Ethernet

driver.

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1464

Quoted 3 lines
lib/ethdev/ethdev_driver.h +0 -0
  */
@@ -1402,6 +1464,10 @@ struct eth_dev_ops {
 	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC

address */

lib/ethdev/ethdev_driver.h struct eth_dev_ops {

Quoted 3 lines
lib/ethdev/ethdev_driver.h +1 -0
 	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
 	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_stashing_rx_hints_set_t   stashing_rx_hints_set; /**< Set Rx cache

stashing*/

lib/ethdev/ethdev_driver.h struct eth_dev_ops { · line 1468

Quoted 1 line
lib/ethdev/ethdev_driver.h +1 -0
+	eth_stashing_tx_hints_set_t   stashing_tx_hints_set; /**< Set Tx cache

stashing*/

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t

Quoted 19 lines
lib/ethdev/ethdev_driver.h +2 -0
+	/** Get supported stashing hints*/
+	eth_stashing_capabilities_get_t stashing_capabilities_get;
 	/** Set list of multicast addresses */
 	eth_set_mc_addr_list_t     set_mc_addr_list;
 	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c index renamed from lib/ethdev/rte_ethdev.c +0 -0

d4197322a0..ae666c370b 100644

lib/ethdev/rte_ethdev.c +1 -0
@@ -158,6 +158,7 @@ static const struct {
 	{RTE_ETH_DEV_CAPA_RXQ_SHARE, "RXQ_SHARE"},
 	{RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP, "FLOW_RULE_KEEP"},
 	{RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP,
 "FLOW_SHARED_OBJECT_KEEP"},
+	{RTE_ETH_DEV_CAPA_CACHE_STASHING, "CACHE_STASHING"},
 };
 

enum {

lib/ethdev/rte_ethdev.c +0 -0
@@ -7419,5 +7420,153 @@ int rte_eth_dev_map_aggr_tx_affinity(uint16_t

port_id, uint16_t tx_queue_id,

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,

Quoted 4 lines
lib/ethdev/rte_ethdev.c +0 -0
 	return ret;
 }
 
 

+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_validate_stashing_config,

Quoted 24 lines
lib/ethdev/rte_ethdev.c +24 -0
+25.07) int rte_eth_dev_validate_stashing_config(uint16_t port_id,
+uint16_t queue_id,
+				     uint8_t queue_direction,
+				     struct rte_eth_stashing_config *config) {
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+	int ret = 0;
+	uint16_t nb_queues;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (!config) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid stashing configuration");
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/*
+	 * Check for invalid objects
+	 */
+	if (!RTE_ETH_DEV_STASH_OBJECTS_VALID(config->objects)) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid stashing objects");
+		ret = -EINVAL;
+		goto out;

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+	}
+
+	dev = &rte_eth_devices[port_id];
+
+	nb_queues = (queue_direction == RTE_ETH_DEV_RX_QUEUE) ?
+				      dev->data->nb_rx_queues :
+				      dev->data->nb_tx_queues;
+
+	if (queue_id >= nb_queues) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid Rx queue_id=%u",

queue_id);

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id, · line 7468

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+		ret = -EINVAL;
+		goto out;
+	}
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret < 0)
+		goto out;
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING) {

Nit: check if all this can fit on one line under 100 chars. I think it can.

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,

Quoted 7 lines
lib/ethdev/rte_ethdev.c +7 -0
+		ret = -ENOTSUP;
+		goto out;
+	}
+
+	if (*dev->dev_ops->stashing_rx_hints_set == NULL ||
+	    *dev->dev_ops->stashing_tx_hints_set == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not

implemented "

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t · line 7475

Quoted 1 line
lib/ethdev/rte_ethdev.c +1 -0
+				    "in %s for %s", dev_info.driver_name,

Don't split error messages across lines. Text strings are allowed to go over the 100 char limit if necessary to avoid splitting.

Quoted 43 lines
lib/ethdev/rte_ethdev.c +43 -0
+				    dev_info.device->name);
+		ret = -ENOSYS;
+	}
+
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_rx_config_set,
+25.07) int rte_eth_dev_stashing_rx_config_set(uint16_t port_id,
+uint16_t queue_id,
+				   struct rte_eth_stashing_config *config) {
+	struct rte_eth_dev *dev;
+	int ret = 0;
+
+	ret = rte_eth_dev_validate_stashing_config(port_id, queue_id,
+						   RTE_ETH_DEV_RX_QUEUE,
+						   config);
+	if (ret < 0)
+		goto out;
+
+	dev = &rte_eth_devices[port_id];
+
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_rx_hints_set)(dev, queue_id,
+		      config));
+out:
+	return ret;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_tx_config_set,
+25.07) int rte_eth_dev_stashing_tx_config_set(uint16_t port_id,
+uint16_t queue_id,
+				   struct rte_eth_stashing_config *config) {
+	struct rte_eth_dev *dev;
+	int ret = 0;
+
+	ret = rte_eth_dev_validate_stashing_config(port_id, queue_id,
+						   RTE_ETH_DEV_TX_QUEUE,
+						   config);
+	if (ret < 0)
+		goto out;
+

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +9 -0
+	dev = &rte_eth_devices[port_id];
+
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_rx_hints_set) (dev, queue_id,
+		       config));
+out:
+	return ret;
+}
+
 

+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_eth_dev_stashing_capabilities_get,

Quoted 17 lines
lib/ethdev/rte_ethdev.c +17 -0
+25.07) int rte_eth_dev_stashing_capabilities_get(uint16_t port_id,
+uint16_t *objects) {
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+	int ret = 0;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (!objects) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	dev = &rte_eth_devices[port_id];
+
+	ret = rte_eth_dev_info_get(port_id, &dev_info);
+	if (ret < 0)

lib/ethdev/rte_ethdev.c int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,

Quoted 10 lines
lib/ethdev/rte_ethdev.c +10 -0
+		goto out;
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING) {
+		ret = -ENOTSUP;
+		goto out;
+	}
+
+	if (*dev->dev_ops->stashing_capabilities_get == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not

implemented "

Quoted 39 lines
lib/ethdev/rte_ethdev.c +11 -0
+				    "in %s for %s", dev_info.driver_name,
+				    dev_info.device->name);
+		ret = -ENOSYS;
+		goto out;
+	}
+	ret = eth_err(port_id,
+		      (*dev->dev_ops->stashing_capabilities_get)(dev, objects));
+out:
+	return ret;
+}
+
 RTE_EXPORT_SYMBOL(rte_eth_dev_logtype)
 RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO); diff --git

a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index ea7f8c4a1a..1398f8c837 100644

lib/ethdev/rte_ethdev.h +0 -0
@@ -1667,6 +1667,9 @@ struct rte_eth_conf {  #define

RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP RTE_BIT64(4) /**@}*/

lib/ethdev/rte_ethdev.h +4 -1
+/** Device supports stashing to CPU/system caches. */ #define
+RTE_ETH_DEV_CAPA_CACHE_STASHING RTE_BIT64(5)
+
 /*
  * Fallback default preferred Rx/Tx port parameters.
  * These are used if an application requests default parameters @@
-1838,6 +1841,7 @@ struct rte_eth_dev_info {
 	struct rte_eth_dev_portconf default_txportconf;
 	/** Generic device capabilities (RTE_ETH_DEV_CAPA_). */
 	uint64_t dev_capa;
+	uint16_t stashing_capa;
 	/**
 	 * Switching information for ports on a device with a
 	 * embedded managed interconnect/switch.
@@ -6173,6 +6177,160 @@ int rte_eth_cman_config_set(uint16_t port_id,

const struct rte_eth_cman_config * __rte_experimental int rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config);

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6185

Quoted 6 lines
lib/ethdev/rte_ethdev.h +6 -0
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_RX_QUEUE		0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_TX_QUEUE		1
+

I'd prefer an enum for these.
Why are the necessary since we have separate rx and tx functions for the caching hints.

Quoted 12 lines
lib/ethdev/rte_ethdev.h +12 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this structure may change, or be removed, without
+prior notice
+ *
+ * A structure used for configuring the cache stashing hints.
+ */
+struct rte_eth_stashing_config {
+	/**
+	 * lcore_id of the processor the stashing hints are applied to.
+	 */

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6206

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+	uint32_t	lcore_id;
+	/**
+	 * Zero based cache level relative to the CPU.
+	 * E.g. l1d = 0, l2d = 1,...
+	 */
+	uint32_t	cache_level;
+	/**
+	 * Object types the configuration is applied to
+	 */
+	uint16_t	objects;

What are the objects? That needs to be covered by the docs, or make this an enum type so that it's clear from the typesystem what it applies to [though that would require an array of objects and count, it may be clearer for user].

Quoted 22 lines
lib/ethdev/rte_ethdev.h +22 -0
+	/**
+	 * The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
+	 *  in objects
+	 */
+	int		offset;
+};
+
+/**@{@name Stashable Rx/Tx queue object types supported by the
+ethernet device  *@see rte_eth_dev_stashing_capabilities_get
+ *@see rte_eth_dev_stashing_rx_config_set
+ *@see rte_eth_dev_stashing_tx_config_set
+ */
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_OBJECT_OFFSET		0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_OBJECT_DESC		0x0002
+

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_HEADER		0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD	0x0008
+
+#define __RTE_ETH_DEV_STASH_OBJECT_MASK		0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)
	\
Quoted 5 lines
lib/ethdev/rte_ethdev.h +5 -0
+	((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
+
+/**
+ *
+ * @warning

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * @internal
+ * Helper function to validate stashing hints configuration.
+ */
+__rte_experimental
+int rte_eth_dev_validate_stashing_config(uint16_t port_id, uint16_t queue_id,
+					 uint8_t queue_direction,
+					 struct rte_eth_stashing_config

*config);

Quoted 15 lines
lib/ethdev/rte_ethdev.h +15 -0
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Provide cache stashing hints for improved memory access latencies
+for
+ * packets received by the NIC.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param queue_id

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ *  The index of the receive queue to which hints are applied.
+ * @param config
+ *  Stashing configuration.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX object types used in conjuection in
+objects
+ *  parameter.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is

unavailable.

Quoted 22 lines

+ * - (-ENOSYS) if PMD does not implement cache stashing hints.
+ * - (0) on Success.
+ */

lib/ethdev/rte_ethdev.h +19 -0
+__rte_experimental
+int rte_eth_dev_stashing_rx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Configure cache stashing for improved memory access latencies for
+Tx
+ * queue completion descriptors being sent to host system by the NIC.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param queue_id
+ *  The index of the receive queue to which hints are applied.

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+ * @param config
+ *  Stashing configuration.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX object types are used in
+conjuection in objects
+ *  parameter.
+ *  - (-EINVAL) if hints are incompatible with TX queues.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is

unavailable.

Quoted 40 lines

+ * - (-ENOSYS) if PMD does not implement cache stashing hints.
+ * - (0) on Success.
+ */

lib/ethdev/rte_ethdev.h +31 -1
+__rte_experimental
+int rte_eth_dev_stashing_tx_config_set(uint16_t port_id, uint16_t queue_id,
+				   struct rte_eth_stashing_config *config);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior
+notice
+ *
+ * Discover cache stashing objects supported in the ethernet device.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param objects
+ *  Supported objects vector set by the ethernet device.
+ * @return
+ *  On return types and hints parameters will have bits set for
+supported
+ *  object types.
+ *  - (-ENOTSUP) if the device or the platform does not support cache stashing.
+ *  - (-ENOSYS)  if the underlying PMD hasn't implemented cache
+stashing
+ *  feature.
+ *  - (-EINVAL)  on NULL values for types or hints parameters.
+ *  - (0) on success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_capabilities_get(uint16_t port_id, uint16_t
+*objects);
+
 #ifdef __cplusplus
 }
 #endif
--

2.43.0

#54 lore raw

PATCH v5 4/4 net/i40e: enable TPH in i40e

In reply to Wathsala Vithanage

Adds stashing_capabilities_get, stashing_{rx,tx}_hinst_set functions to eth_dev_ops of the i40e driver. Enables TPH in device-specific mode, so that steering-tags are set in LAN queue contexts.

---
 drivers/net/intel/i40e/i40e_ethdev.c | 128 +++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)
drivers/net/intel/i40e/i40e_ethdev.c +128 -0
index 90eba3419f..326e323dfb 100644
@@ -18,6 +18,7 @@
 #include <rte_pci.h>
 #include <bus_pci_driver.h>
 #include <rte_ether.h>
+#include <rte_ethdev.h>
 #include <ethdev_driver.h>
 #include <ethdev_pci.h>
 #include <rte_memzone.h>
@@ -410,6 +411,11 @@ static int i40e_fec_get_capability(struct rte_eth_dev *dev,
 	struct rte_eth_fec_capa *speed_fec_capa, unsigned int num);
 static int i40e_fec_get(struct rte_eth_dev *dev, uint32_t *fec_capa);
 static int i40e_fec_set(struct rte_eth_dev *dev, uint32_t fec_capa);
+static int i40e_stashing_cap_get(struct rte_eth_dev *dev, uint16_t *objects);
+static int i40e_stashing_rx_hints_set(struct rte_eth_dev *dev, uint16_t queue_id,
+				     struct rte_eth_stashing_config *config);
+static int i40e_stashing_tx_hints_set(struct rte_eth_dev *dev, uint16_t queue_id,
+				     struct rte_eth_stashing_config *config);
 
 static const char *const valid_keys[] = {
 	ETH_I40E_FLOATING_VEB_ARG,
@@ -527,6 +533,9 @@ static const struct eth_dev_ops i40e_eth_dev_ops = {
 	.fec_get_capability           = i40e_fec_get_capability,
 	.fec_get                      = i40e_fec_get,
 	.fec_set                      = i40e_fec_set,
+	.stashing_capabilities_get    = i40e_stashing_cap_get,
+	.stashing_rx_hints_set        = i40e_stashing_rx_hints_set,
+	.stashing_tx_hints_set        = i40e_stashing_tx_hints_set,
 };
 
 /* store statistics names and its offset in stats structure */
@@ -3878,6 +3887,7 @@ i40e_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info)
 		RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP |
 		RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP;
 	dev_info->dev_capa &= ~RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP;
+	dev_info->dev_capa |= RTE_ETH_DEV_CAPA_CACHE_STASHING;
 
 	dev_info->hash_key_size = (I40E_PFQF_HKEY_MAX_INDEX + 1) *
 						sizeof(uint32_t);
@@ -12544,6 +12554,124 @@ i40e_fec_set(struct rte_eth_dev *dev, uint32_t fec_capa)
 	return 0;
 }
 
+static int
+i40e_stashing_cap_get(struct rte_eth_dev *dev, uint16_t *objects)
+{
+	RTE_SET_USED(dev);
+
+	*objects = RTE_ETH_DEV_STASH_OBJECT_DESC |
+		   RTE_ETH_DEV_STASH_OBJECT_HEADER |
+		   RTE_ETH_DEV_STASH_OBJECT_PAYLOAD;
+
+	return 0;
+}
+
+static int
+i40e_stashing_hints_set(struct rte_eth_dev *dev, uint16_t queue_id,
+			struct rte_eth_stashing_config *config,
+			enum i40e_hmc_lan_rsrc_type hmc_type)
+{
+	struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev);
+	int err = I40E_SUCCESS;
+	struct i40e_hmc_obj_rxq rxq;
+	struct i40e_hmc_obj_txq txq;
+
+	if (!config) {
+		err = -EINVAL;
+		goto out;
+	}
+
+	struct rte_tph_info tph = {
+		.cpu_id = config->lcore_id,
+		.cache_level = config->cache_level,
+		.flags = RTE_PCI_TPH_MEM_TYPE_VMEM | RTE_PCI_TPH_HINT_BIDIR,
+		.index = 0,
+	};
+
+	if (!pci_dev->tph_enabled)
+		err = rte_pci_tph_enable(pci_dev, RTE_PCI_TPH_CAP_ST_DS);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed enabling TPH");
+		goto out;
+	}
+
+	err = rte_pci_tph_st_get(pci_dev, &tph, 1);
+	if (err) {
+		PMD_DRV_LOG(ERR, "Failed ST read for lcore: %u and cache-level: %u",
+			    tph.cpu_id, tph.cache_level);
+		goto out;
+	}
+
+	switch (hmc_type) {
+	case I40E_HMC_LAN_RX:
+		err = i40e_get_lan_rx_queue_context(hw, queue_id, &rxq);
+		if (err != I40E_SUCCESS) {
+			PMD_DRV_LOG(ERR, "Failed to get LAN RX queue context");
+			goto out;
+		}
+
+		rxq.cpuid = tph.st;
+
+		if (config->objects & RTE_ETH_DEV_STASH_OBJECT_DESC) {
+			rxq.tphrdesc_ena = 1;
+			rxq.tphwdesc_ena = 1;
+		}
+
+		if (config->objects & RTE_ETH_DEV_STASH_OBJECT_PAYLOAD)
+			rxq.tphdata_ena = 1;
+
+		if (config->objects & RTE_ETH_DEV_STASH_OBJECT_HEADER)
+			rxq.tphhead_ena = 1;
+
+		err = i40e_set_lan_rx_queue_context(hw, queue_id, &rxq);
+		if (err != I40E_SUCCESS)
+			PMD_DRV_LOG(ERR, "Failed to set LAN RX queue context");
+		break;
+	case I40E_HMC_LAN_TX:
+		err = i40e_get_lan_tx_queue_context(hw, queue_id, &txq);
+		if (err != I40E_SUCCESS) {
+			PMD_DRV_LOG(ERR, "Failed to get LAN TX queue context");
+			goto out;
+		}
+
+		txq.cpuid = tph.st;
+
+		if (config->objects & RTE_ETH_DEV_STASH_OBJECT_DESC) {
+			txq.tphrdesc_ena = 1;
+			txq.tphwdesc_ena = 1;
+		}
+
+		if (config->objects & (RTE_ETH_DEV_STASH_OBJECT_PAYLOAD |
+			       RTE_ETH_DEV_STASH_OBJECT_HEADER))
+			txq.tphrpacket_ena = 1;
+
+		err = i40e_set_lan_tx_queue_context(hw, queue_id, &txq);
+		if (err != I40E_SUCCESS)
+			PMD_DRV_LOG(ERR, "Failed to set LAN TX queue context");
+		break;
+	default:
+		err = -EINVAL;
+	}
+
+out:
+	return err;
+}
+
+static int
+i40e_stashing_rx_hints_set(struct rte_eth_dev *dev, uint16_t queue_id,
+			  struct rte_eth_stashing_config *config)
+{
+	return i40e_stashing_hints_set(dev, queue_id, config, I40E_HMC_LAN_RX);
+}
+
+static int
+i40e_stashing_tx_hints_set(struct rte_eth_dev *dev, uint16_t queue_id,
+			  struct rte_eth_stashing_config *config)
+{
+	return i40e_stashing_hints_set(dev, queue_id, config, I40E_HMC_LAN_TX);
+}
+
 RTE_LOG_REGISTER_SUFFIX(i40e_logtype_init, init, NOTICE);
 RTE_LOG_REGISTER_SUFFIX(i40e_logtype_driver, driver, NOTICE);
 #ifdef RTE_ETHDEV_DEBUG_RX
Signature · 1 line
2.43.0
#55 lore raw

Re: [PATCH v5 0/4] An API for Cache Stashing with TPH

In reply to Wathsala Vithanage
Quoted 94 lines · On Mon, 2 Jun 2025 22:38:00 +0000 Wathsala Vithanage…

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

Quoted 17 lines
 drivers/bus/pci/bsd/pci.c            |  43 +++++++
 drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
 drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h     |  14 +++
 drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
 drivers/bus/pci/private.h            |   8 ++
 drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
 drivers/bus/pci/windows/pci.c        |  43 +++++++
 drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
 kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
 lib/ethdev/ethdev_driver.h           |  66 +++++++++++
 lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
 lib/pci/rte_pci.h                    |  15 +++
 14 files changed, 1114 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h

How will this impact existing applications that never use the API? It is crucial that existing 3rd party applications, just work without modifications. We don't want to hear from Network Virtual Appliance vendors that there is a performance regression in DPDK. They are already reluctant to keep up with DPDK versions.

I.e if the application does nothing caching must be enabled.

#56 lore raw

RE: [PATCH v5 0/4] An API for Cache Stashing with TPH

Quoted 115 lines
Quoted 62 lines

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

Quoted 15 lines

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH

in device-specific mode.

Quoted 19 lines

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus
driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

Quoted 17 lines
 drivers/bus/pci/bsd/pci.c            |  43 +++++++
 drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
 drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h     |  14 +++
 drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
 drivers/bus/pci/private.h            |   8 ++
 drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
 drivers/bus/pci/windows/pci.c        |  43 +++++++
 drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
 kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
 lib/ethdev/ethdev_driver.h           |  66 +++++++++++
 lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
 lib/pci/rte_pci.h                    |  15 +++
 14 files changed, 1114 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
Quoted 7 lines

How will this impact existing applications that never use the API?
It is crucial that existing 3rd party applications, just work without modifications. We don't want to hear from Network Virtual Appliance vendors that there is a performance regression in DPDK. They are already reluctant to keep up with DPDK versions.

I.e if the application does nothing caching must be enabled.

It won't affect such applications.

--wathsala

#57 lore raw

Re: [PATCH v5 0/4] An API for Cache Stashing with TPH

In reply to Wathsala Vithanage

Hi Wathsala,

Sorry to ask if this patchset is under development or stopped?

PCIe Steer-tag provides a mechanism for precise data stash, which delivers a positive performance gain and is therefore a valuable feature I think.

This patchset concludes with the statement: "the PMDs should only enable TPH in device-specific mode", I don't think such restraints should be made, the framework should be compatible with various device capabilities:
1. The PCIe protocol defines two modes: one is the interrupt-vector mode, and the other is the device-specific mode. A device may choose to support either one or both.
2. If device support device-specific mode, it has a large degree of freedom to implement, such as locate ST table in self-defined place (just like '[PATCH v5 4/4] net/i40e: enable TPH in i40e'), and also support only stash part of data (e.g. only desc or header or even an offset data).
3. If device only support interrupt-vector mode (which each TLP will use ST from an ST table entry), we could also support it, in this framework, it could only report basic stash capability.

Thanks

Quoted 111 lines · Wathsala Vithanage

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

 drivers/bus/pci/bsd/pci.c            |  43 +++++++
 drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
 drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h     |  14 +++
 drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
 drivers/bus/pci/private.h            |   8 ++
 drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
 drivers/bus/pci/windows/pci.c        |  43 +++++++
 drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
 kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
 lib/ethdev/ethdev_driver.h           |  66 +++++++++++
 lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
 lib/pci/rte_pci.h                    |  15 +++
 14 files changed, 1114 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
#58 lore raw

Re: [PATCH v5 0/4] An API for Cache Stashing with TPH

Hi Wathsala,

Looking forward to your reply.

Thanks

Quoted 139 lines · fengchengwen

Hi Wathsala,

Sorry to ask if this patchset is under development or stopped?

PCIe Steer-tag provides a mechanism for precise data stash, which delivers a positive performance gain and is therefore a valuable feature I think.

This patchset concludes with the statement: "the PMDs should only enable TPH in device-specific mode", I don't think such restraints should be made, the framework should be compatible with various device capabilities:
1. The PCIe protocol defines two modes: one is the interrupt-vector mode, and the other is the device-specific mode. A device may choose to support either one or both.
2. If device support device-specific mode, it has a large degree of freedom to implement, such as locate ST table in self-defined place (just like '[PATCH v5 4/4] net/i40e: enable TPH in i40e'), and also support only stash part of data (e.g. only desc or header or even an offset data).
3. If device only support interrupt-vector mode (which each TLP will use ST from an ST table entry), we could also support it, in this framework, it could only report basic stash capability.

Thanks

Quoted 111 lines · Wathsala Vithanage

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

 drivers/bus/pci/bsd/pci.c            |  43 +++++++
 drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
 drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
 drivers/bus/pci/linux/pci_init.h     |  14 +++
 drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
 drivers/bus/pci/private.h            |   8 ++
 drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
 drivers/bus/pci/windows/pci.c        |  43 +++++++
 drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
 kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
 lib/ethdev/ethdev_driver.h           |  66 +++++++++++
 lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
 lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
 lib/pci/rte_pci.h                    |  15 +++
 14 files changed, 1114 insertions(+)
 create mode 100644 kernel/linux/uapi/linux/vfio_tph.h
#59 lore raw

Re: [PATCH v5 0/4] An API for Cache Stashing with TPH

Hi,

Thank you for your enthusiasm :)
This patch is barred until kernel VFIO-TPH patch gets merged.

--wathsala

Quoted 145 lines · fengchengwen

Hi Wathsala,

Looking forward to your reply.

Thanks

Quoted 138 lines · fengchengwen

Hi Wathsala,

Sorry to ask if this patchset is under development or stopped?

PCIe Steer-tag provides a mechanism for precise data stash, which delivers a positive performance gain and is therefore a valuable feature I think.

This patchset concludes with the statement: "the PMDs should only enable TPH in device-specific mode", I don't think such restraints should be made, the framework should be compatible with various device capabilities:
1. The PCIe protocol defines two modes: one is the interrupt-vector mode, and the other is the device-specific mode. A device may choose to support either one or both.
2. If device support device-specific mode, it has a large degree of freedom to implement, such as locate ST table in self-defined place (just like '[PATCH v5 4/4] net/i40e: enable TPH in i40e'), and also support only stash part of data (e.g. only desc or header or even an offset data).
3. If device only support interrupt-vector mode (which each TLP will use ST from an ST table entry), we could also support it, in this framework, it could only report basic stash capability.

Thanks

Quoted 111 lines · Wathsala Vithanage

Today, DPDK applications benefit from Direct Cache Access (DCA) features like Intel DDIO and Arm's write-allocate-to-SLC. However, those features do not allow fine-grained control of direct cache access, such as stashing packets into upper-level caches (L2 caches) of a processor or the shared cache of a chiplet. PCIe TLP Processing Hints (TPH) addresses this need in a vendor-agnostic manner. TPH capability has existed since PCI Express Base Specification revision 3.0; today, numerous Network Interface Cards and interconnects from different vendors support TPH capability. TPH comprises a steering tag (ST) and a processing hint (PH). ST specifies the cache level of a CPU at which the data should be written to (or DCAed into), while PH is a hint provided by the PCIe requester to the completer on an upcoming traffic pattern. Some NIC vendors bundle TPH capability with fine-grained control over the type of objects that can be stashed into CPU caches, such as

- Rx/Tx queue descriptors
- Packet-headers
- Packet-payloads
- Data from a given offset from the start of a packet

Note that stashable object types are outside the scope of the PCIe standard; therefore, vendors could support any combination of the above items as they see fit.

To enable TPH and fine-grained packet stashing, this API extends the ethdev library and the PCI bus driver. In this design, the application provides hints to the PMD via the ethdev stashing API to indicate the underlying hardware at which CPU and cache level it prefers a packet to end up. Once the PMD receives a CPU and a cache-level combination (or a list of such combinations), it must extract the matching ST from the PCI bus driver for such combinations. The PCI bus driver implements the TPH functions in an OS specific way; for Linux, it depends on the TPH capabilities of the VFIO kernel driver.

An application uses the cache stashing ethdev API by first calling the rte_eth_dev_stashing_capabilities_get() function to find out what object types can be stashed into a CPU cache by the NIC out of the object types in the bulleted list above. This function takes a port_id and a pointer to a uint16_t to report back the object type flags. PMD implements the stashing_capabilities_get function pointer in eth_dev_ops. If the underlying platform or the NIC does not support TPH, this function returns -ENOTSUP, and the application should consider any values stored in the object invalid.

Once the application knows the supported object types that can be stashed, the next step is to set the steering tags for the packets associated with Rx and Tx queues via rte_eth_dev_stashing_{rx,tx}_config_set() ethdev library functions. Both functions have an identical signature, a port_id, a queue_id, and a config object. The port_id and the queue_id are used to locate the device and the queue. The config object is of type struct rte_eth_stashing_config, which specifies the lcore_id and the cache_level, indicating where objects from this queue should be stashed. The 'objects' field in the config sets the types of objects the application wishes to stash based on the capabilities found earlier. Note that if the 'objects' field includes the flag RTE_ETH_DEV_STASH_OBJECT_OFFSET, the 'offset' field must be used to set the desired offset. These functions invoke PMD implementations of the stashing functionality via the stashing_{rx,tx}_hints_set function callbacks in the eth_dev_ops, respectively.

The PMD's implementation of the stashing_rx_hints_set() and stashing_tx_hints_set() functions is ultimately responsible for extracting the ST via the API provided by the PCI bus driver. Before extracting STs, the PMD should enable the TPH capability in the endpoint device by calling the rte_pci_tph_enable() function.  The application begins the ST extraction process by calling the rte_pci_tph_st_get() function in drivers/bus/pci/rte_bus_pci.h, which returns STs via the same rte_tph_info objects array passed into it as an argument.  Once PMD acquires ST, the stashing_{rx,tx}_hints_set callbacks implemented in the PMD are ready to set the ST as per the rte_eth_stashing_config object passed to them by the higher-level ethdev functions ret_eth_dev_stashing_{rx,tx}_hints(). As per the PCIe specification, STs can be placed on the MSI-X tables or in a device-specific location. For PMDs, setting the STs on queue contexts is the only viable way of using TPH. Therefore, the PMDs should only enable TPH in device-specific mode.

V4->V5:
* Enable stashing-hints (TPH) in Intel i40e driver.
* Update exported symbol version from 25.03 to 25.07.
* Add TPH mode macros.

V3->V4:
* Add VFIO IOCTL based ST extraction mechanism to Linux PCI bus driver
* Remove ST extraction via direct access to ACPI _DSM
* Replace rte_pci_extract_tph_st() with rte_pci_tph_st_get() in PCI bus driver.

Wathsala Vithanage (4):
pci: add non-merged Linux uAPI changes bus/pci: introduce the PCIe TLP Processing Hints API
ethdev: introduce the cache stashing hints API net/i40e: enable TPH in i40e

  drivers/bus/pci/bsd/pci.c            |  43 +++++++
  drivers/bus/pci/bus_pci_driver.h     |  52 ++++++++
  drivers/bus/pci/linux/pci.c          | 100 ++++++++++++++++
  drivers/bus/pci/linux/pci_init.h     |  14 +++
  drivers/bus/pci/linux/pci_vfio.c     | 170 +++++++++++++++++++++++++++
  drivers/bus/pci/private.h            |   8 ++
  drivers/bus/pci/rte_bus_pci.h        |  67 +++++++++++
  drivers/bus/pci/windows/pci.c        |  43 +++++++
  drivers/net/intel/i40e/i40e_ethdev.c | 127 ++++++++++++++++++++
  kernel/linux/uapi/linux/vfio_tph.h   | 102 ++++++++++++++++
  lib/ethdev/ethdev_driver.h           |  66 +++++++++++
  lib/ethdev/rte_ethdev.c              | 149 +++++++++++++++++++++++
  lib/ethdev/rte_ethdev.h              | 158 +++++++++++++++++++++++++
  lib/pci/rte_pci.h                    |  15 +++
  14 files changed, 1114 insertions(+)

create mode 100644 kernel/linux/uapi/linux/vfio_tph.h

#60 lore raw
In reply to Wathsala Vithanage
Quoted 106 lines · Wathsala Vithanage

An application provides cache stashing hints to the ethernet devices to improve memory access latencies from the CPU and the NIC. This patch introduces three distinct hints for this purpose.

The RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED hint indicates that the host (CPU) requires the data written by the NIC immediately. This implies that the CPU expects to read data from its local cache rather than LLC or main memory if possible. This would improve memory access latency in the Rx path. For PCI devices with TPH capability, these hints translate into DWHR (Device Writes Host Reads) access pattern. This hint is only valid for receive queues.

The RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA hint indicates that the host and the device access the data structure equally. Rx/Tx queue descriptors fit the description of such data. This hint applies to both Rx and Tx directions.  In the PCI TPH context, this hint translates into a Bi-Directional access pattern.

RTE_ETH_DEV_STASH_HINT_DEV_ONLY hint indicates that the CPU is not involved in a given device's receive or transmit paths. This implies that only devices are involved in the IO path. Depending on the implementation, this hint may result in data getting placed in a cache close to the device or not cached at all. For PCI devices with TPH capability, this hint translates into D*D* (DWDR, DRDW, DWDW, DRDR) access patterns. This is a bidirectional hint, and it can be applied to both Rx and Tx queues.

The RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED hint indicates that the device reads data written by the host (CPU) that may still be in the host's local cache but is not required by the host anytime soon. This hint is intended to prevent unnecessary cache invalidations that cause interconnect latencies when a device writes to a buffer already in host cache memory. In DPDK, this could happen with the recycling of mbufs where a mbuf is placed in the Tx queue that then gets back into mempool and gets recycled back into the Rx queue, all while a copy is being held in the CPU's local cache unnecessarily. By using this hint on supported platforms, the mbuf will be invalidated after the device completes the buffer reading, but it will be well before the buffer gets recycled and updated in the Rx path. This hint is only valid for transmit queues.

Applications use three main interfaces in the ethdev library to discover and set cache stashing hints. rte_eth_dev_stashing_hints_tx interface is used to set hints on a Tx queue. rte_eth_dev_stashing_hints_rx interface is used to set hints on an Rx queue. Both of these functions take the following parameters as inputs: a port_id (the id of the ethernet device), a cpu_id (the target CPU), a cache_level (the level of the cache hierarchy the data should be stashed into), a queue_id (the queue the hints are applied to). In addition to the above list of parameters, a type parameter indicates the type of the object the application expects to be stashed by the hardware. Depending on the hardware, these may vary. Intel E810 NICs support the stashing of Rx/Tx descriptors, packet headers, and packet payloads. These are indicated by the macros RTE_ETH_DEV_STASH_TYPE_DESC, RTE_ETH_DEV_STASH_TYPE_HEADER, RTE_ETH_DEV_STASH_TYPE_PAYLOAD. Hardware capable of stashing data at any given offset into a packet can use the RTE_ETH_DEV_STASH_TYPE_OFFSET type. When an offset is used, the offset parameter in the above two functions should be set appropriately.

rte_eth_dev_stashing_hints_discover is used to discover the object types and hints supported in the platform and the device. The function takes types and hints pointers used as a bit vector to indicate hints and types supported by the NIC. An application that intends to use stashing hints should first discover supported hints and types and then use the functions rte_eth_dev_stashing_hints_tx and rte_eth_dev_stashing_hints_rx as required to set stashing hints accordingly. eth_dev_ops structure has been updated with two new ops that a PMD should implement to support cache stashing hints. A PMD that intends to support cache stashing hints should initialize the set_stashing_hints function pointer to a function that issues hints to the underlying hardware in compliance with platform capabilities. The same PMD should also implement a function that can return two-bit fields indicating supported types and hints and then initialize the discover_stashing_hints function pointer with it. If the NIC supports cache stashing hints, the NIC should always set the RTE_ETH_DEV_CAPA_CACHE_STASHING device capability.

---
  .mailmap                   |   1 +
  lib/ethdev/ethdev_driver.h |  67 +++++++++++
  lib/ethdev/rte_ethdev.c    | 153 +++++++++++++++++++++++++
  lib/ethdev/rte_ethdev.h    | 225 +++++++++++++++++++++++++++++++++++++
  lib/ethdev/version.map     |   6 +
  5 files changed, 452 insertions(+)
.mailmap +1 -0
index f1e64286a1..9c28b74655 100644
@@ -338,6 +338,7 @@ Dexia Li <[email protected]>
  Dexuan Cui <[email protected]>
  Dharmik Thakkar <[email protected]> <[email protected]>
  Dheemanth Mallikarjun <[email protected]>
+Dhruv Tripathi <[email protected]>
  Diana Wang <[email protected]>
  Didier Pallard <[email protected]>
  Dilshod Urazov <[email protected]>
lib/ethdev/ethdev_driver.h +0 -0
index 883e59a927..b90dc8793b 100644
@@ -1235,6 +1235,70 @@ typedef int (*eth_count_aggr_ports_t)(struct rte_eth_dev *dev);
  typedef int (*eth_map_aggr_tx_affinity_t)(struct rte_eth_dev *dev, uint16_t tx_queue_id,
  					  uint8_t affinity);
 

lib/ethdev/ethdev_driver.h typedef int (*eth_count_aggr_ports_t)(struct rte_eth_dev *dev); · line 1247

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+/**
+ * @internal
+ * Set cache stashing hint in the ethernet device.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param cpuid
+ *   ID of the targeted CPU.
+ * @param cache_level
+ *   Level of the cache to stash data.

If we had a hwtopo API in DPDK, we could just use a node id in such a graph (of CPUs and caches) to describe were the data ideally would land. In such a case, you could have a node id for DDR as well, and thus you could drop the notion of "stashing". Just a "drop off the data here, please, if you can" API.

I don't think this API and its documentation should talk about what the "CPU" needs, since it's somewhat misleading.

For example, you can imagine you want the packet payload to land in the LLC, even though it's not for any CPU to consume, in case you know with some certaintly that the packet will soon be transmitted (and thus consumed by the NIC).

The same scenario can happen, the consumer is an accelerator (e.g., a crypto engine).

Likewise, you may know that the whole packet will be read by some CPU core, but you also know the system tends to buffer packets before they are being processed. In such a case, it's better to go to DRAM right away, to avoid trashing the LLC (or some other cache).

Also, why do you need to use the word "host"? Seems like a PCI thing. This may be implemented in PCI, but surely can be done (and has been done) without PCI.

Quoted 454 lines
lib/ethdev/ethdev_driver.h +57 -0
+ * @param queue_id
+ *   List of receive queue ids used in rte_eth_rx_burst().
+ * @param queue_direction
+ *   RTE_ETH_DEV_QUEUE_TYPE_RX if queue that corresponds to queue_id is an
+ *   rx queue.
+ *   RTE_ETH_DEV_QUEUE_TYPE_TX if queue that corresponds to queue_id is a
+ *   tx queue.
+ * @param types
+ *   A vector of stashing types to apply hints on a given queue direction.
+ *   hints are applied on the types specified in types vector.
+ *   types can include queue descriptors (RTE_ETH_DEV_STASH_TYPE_DESC),
+ *   packet headers (RTE_ETH_DEV_STASH_TYPE_HEADER),
+ *   packet payloads (RTE_ETH_DEV_STASH_TYPE_PAYLOAD) or
+ *   to an offset (RTE_ETH_DEV_STASH_TYPE_OFFSET) in to a packet.
+ *   types have to be compatible with the queue_direction or an -EINVAL will
+ *   be returned.
+ * @param hints
+ *   Cache stashing hints
+ * @param offset
+ *   Offset into the packet if RTE_ETH_DEV_STASH_TYPE_OFFSET is set in hints.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on invalid arguments.
+ *   0 on success.
+ */
+typedef int (*eth_set_stashing_hints_t)(struct rte_eth_dev *dev, uint16_t cpuid,
+					uint8_t cache_level,
+					uint16_t queue_id, uint8_t queue_direction,
+					uint16_t types, uint8_t hints, off_t offset);
+
+/**
+ * @internal
+ * Discover cache stashing hints and object types supported in the ethernet device.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param types
+ *   Set bits for supported object types.
+ * @param hints
+ *   Set bits for supported stashing hints.
+ *
+ * @return
+ *   -ENOTSUP if the device or the platform does not support cache stashing.
+ *   -ENOSYS  if the underlying PMD hasn't implemented cache stashing feature.
+ *   -EINVAL  on NULL values for types or hints parameters.
+ *   On return, types and hints parameters will have bits set for supported
+ *   object types and hints.
+ *   0 on success.
+ */
+typedef int (*eth_discover_stashing_hints_t)(struct rte_eth_dev *dev,
+					     uint16_t *types, uint16_t *hints);
+
  /**
   * @internal A structure containing the functions exported by an Ethernet driver.
   */
@@ -1257,6 +1321,9 @@ struct eth_dev_ops {
  	eth_mac_addr_remove_t      mac_addr_remove; /**< Remove MAC address */
  	eth_mac_addr_add_t         mac_addr_add;  /**< Add a MAC address */
  	eth_mac_addr_set_t         mac_addr_set;  /**< Set a MAC address */
+	eth_set_stashing_hints_t   set_stashing_hints; /**< Set cache stashing*/
+	/**Discover supported stashing hints*/
+	eth_discover_stashing_hints_t discover_stashing_hints;
  	/** Set list of multicast addresses */
  	eth_set_mc_addr_list_t     set_mc_addr_list;
  	mtu_set_t                  mtu_set;       /**< Set MTU */
lib/ethdev/rte_ethdev.c +153 -0
index f1c658f49e..fafc94223e 100644
@@ -153,6 +153,7 @@ static const struct {
  	{RTE_ETH_DEV_CAPA_RXQ_SHARE, "RXQ_SHARE"},
  	{RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP, "FLOW_RULE_KEEP"},
  	{RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP, "FLOW_SHARED_OBJECT_KEEP"},
+	{RTE_ETH_DEV_CAPA_CACHE_STASHING, "CACHE_STASHING"},
  };
 
  enum {
@@ -7008,4 +7009,156 @@ int rte_eth_dev_map_aggr_tx_affinity(uint16_t port_id, uint16_t tx_queue_id,
  	return ret;
  }
 
+int
+rte_eth_dev_validate_stashing_hints(uint16_t port_id, uint16_t queue_id,
+				    uint8_t queue_direction, uint16_t types,
+				    uint16_t hints)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+	uint16_t nb_queues;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	/*
+	 * Check for invalid types
+	 */
+	if (!RTE_ETH_DEV_STASH_TYPE_VALID(types)) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid stashing type");
+		return -EINVAL;
+	}
+
+	/*
+	 * Ensure that hints (HOST_DONOTNEED, HOST_WILLNEED, BI_DIR_DATA, and
+	 * DEV_ONLY etc.) are not mixed incorrectly in the hint argument.
+	 * Only hints of one queue direction (Rx or Tx) can be combined in the
+	 * hint argument. If the hint argument contains hint types compatible
+	 * with both Rx and Tx directions it can be applied to any queue of the
+	 * two queue types.
+	 */
+	if (!RTE_ETH_DEV_STASH_HINT_IS_RXTX(hints)) {
+		/*
+		 * This is not a Rx and a Tx hint.
+		 * Therefore it can only be applied to single queue direction.
+		 */
+		if (RTE_ETH_DEV_STASH_HINT_IS_TX(hints) ==
+		    RTE_ETH_DEV_STASH_HINT_IS_RX(hints)) {
+			RTE_ETHDEV_LOG_LINE(ERR, "This hint is not compatible "
+					    "with both Rx and Tx paths");
+			return -EINVAL;
+		}
+		/*
+		 * Ensure that hint is compatible with the specified queue
+		 * direction in the queue_direction argument.
+		 */
+		if (((queue_direction == RTE_ETH_DEV_QUEUE_TYPE_TX) &&
+		    RTE_ETH_DEV_STASH_HINT_IS_RX(hints)) ||
+		    ((queue_direction == RTE_ETH_DEV_QUEUE_TYPE_RX) &&
+		    RTE_ETH_DEV_STASH_HINT_IS_TX(hints))) {
+			RTE_ETHDEV_LOG_LINE(ERR, "Hints are not applicable to "
+					    "this queue type");
+			return -EINVAL;
+		}
+	}
+
+	dev = &rte_eth_devices[port_id];
+
+	nb_queues = (queue_direction == RTE_ETH_DEV_QUEUE_TYPE_RX) ?
+				      dev->data->nb_rx_queues :
+				      dev->data->nb_tx_queues;
+
+	if (queue_id >= nb_queues) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Invalid Rx queue_id=%u", queue_id);
+		return -EINVAL;
+	}
+
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING)
+		return -ENOTSUP;
+
+	if (*dev->dev_ops->set_stashing_hints == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not implemented "
+				    "in %s for %s", dev_info.driver_name,
+				    dev_info.device->name);
+		return -ENOSYS;
+	}
+
+	return 0;
+}
+
+int
+rte_eth_dev_stashing_hints_rx(uint16_t port_id, uint16_t cpuid,
+			      uint8_t cache_level, uint16_t queue_id,
+			      uint16_t types, off_t offset,
+			      uint16_t hints)
+{
+	struct rte_eth_dev *dev;
+
+	int ret = rte_eth_dev_validate_stashing_hints(port_id, queue_id,
+						      RTE_ETH_DEV_QUEUE_TYPE_RX,
+						      types, hints);
+	if (ret < 0)
+		return ret;
+
+	dev = &rte_eth_devices[port_id];
+
+	return eth_err(port_id, (*dev->dev_ops->set_stashing_hints)(dev, cpuid,
+		       cache_level, queue_id, RTE_ETH_DEV_QUEUE_TYPE_RX,
+		       types, hints, offset));
+}
+
+int
+rte_eth_dev_stashing_hints_tx(uint16_t port_id, uint16_t cpuid,
+			      uint8_t cache_level, uint16_t queue_id,
+			      uint16_t types, off_t offset,
+			      uint16_t hints)
+{
+	struct rte_eth_dev *dev;
+
+	int ret = rte_eth_dev_validate_stashing_hints(port_id, queue_id,
+						      RTE_ETH_DEV_QUEUE_TYPE_TX,
+						      types, hints);
+	if (ret < 0)
+		return ret;
+
+	dev = &rte_eth_devices[port_id];
+
+	return eth_err(port_id,
+		       (*dev->dev_ops->set_stashing_hints) (dev, cpuid,
+		       cache_level, queue_id, RTE_ETH_DEV_QUEUE_TYPE_TX, types,
+		       hints, offset));
+}
+
+int
+rte_eth_dev_stashing_hints_discover(uint16_t port_id, uint16_t *types,
+				    uint16_t *hints)
+{
+	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (!types || !hints)
+		return -EINVAL;
+
+	dev = &rte_eth_devices[port_id];
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	if ((dev_info.dev_capa & RTE_ETH_DEV_CAPA_CACHE_STASHING) !=
+	    RTE_ETH_DEV_CAPA_CACHE_STASHING)
+		return -ENOTSUP;
+
+	if (*dev->dev_ops->discover_stashing_hints == NULL) {
+		RTE_ETHDEV_LOG_LINE(ERR, "Stashing hints are not implemented "
+				    "in %s for %s", dev_info.driver_name,
+				    dev_info.device->name);
+		return -ENOSYS;
+	}
+	return eth_err(port_id,
+		       (*dev->dev_ops->discover_stashing_hints)
+		       (dev, types, hints));
+}
+
  RTE_LOG_REGISTER_DEFAULT(rte_eth_dev_logtype, INFO);
lib/ethdev/rte_ethdev.h +196 -0
index 548fada1c7..a42f272885 100644
@@ -1648,6 +1648,9 @@ struct rte_eth_conf {
  #define RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP RTE_BIT64(4)
  /**@}*/
 
+/** Device supports stashing to CPU/system caches. */
+#define RTE_ETH_DEV_CAPA_CACHE_STASHING RTE_BIT64(5)
+
  /*
   * Fallback default preferred Rx/Tx port parameters.
   * These are used if an application requests default parameters
@@ -1819,6 +1822,8 @@ struct rte_eth_dev_info {
  	struct rte_eth_dev_portconf default_txportconf;
  	/** Generic device capabilities (RTE_ETH_DEV_CAPA_). */
  	uint64_t dev_capa;
+	uint16_t stashing_hints_capa;
+	uint16_t stashing_types_capa;
  	/**
  	 * Switching information for ports on a device with a
  	 * embedded managed interconnect/switch.
@@ -5964,6 +5969,226 @@ int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *
  __rte_experimental
  int rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config);
 
+
+
+/** Queue type is RX. */
+#define RTE_ETH_DEV_QUEUE_TYPE_RX		0
+/** Queue type is TX. */
+#define RTE_ETH_DEV_QUEUE_TYPE_TX		1
+
+/**@{@name Ethernet device cache stashing hints
+ *@see rte_eth_dev_stashing_hints_discover
+ *@see rte_eth_dev_stashing_hints_rx
+ *@see rte_eth_dev_stashing_hints_tx
+ */
+/**
+ * Data read by the device could still be in a CPU local cache memory but
+ * not required by the CPU before ethernet device is done with Tx.
+ * In other words CPU does not mind evicting the relevant cache line(s)
+ * from it's local cache.
+ */
+#define RTE_ETH_DEV_STASH_HINT_HOST_DONTNEED	0x001
+
+/**
+ * Data is read and written equally by the CPU and the NIC.
+ */
+#define RTE_ETH_DEV_STASH_HINT_BI_DIR_DATA	0x100
+
+/**
+ * Data written by the device is read by a CPU immediately. CPU prefers
+ * availability of the data in it's local cache memory by the time read
+ * takes place.
+ */
+#define RTE_ETH_DEV_STASH_HINT_HOST_WILLNEED	0x010
+
+/**
+ * Data written by the device is only read by device.
+ * Host CPUs do not read this data or write to the location of the data.
+ */
+#define RTE_ETH_DEV_STASH_HINT_DEV_ONLY		0x200
+
+
+#define __RTE_ETH_DEV_STASH_HINT_TX_MASK	0x00f
+
+#define __RTE_ETH_DEV_STASH_HINT_RX_MASK	0x0f0
+
+#define __RTE_ETH_DEV_STASH_HINT_RXTX_MASK	0xf00
+
+
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_HINT_IS_TX(h)				\
+	((!((h) & ~(__RTE_ETH_DEV_STASH_HINT_TX_MASK))) && (h))
+
+#define RTE_ETH_DEV_STASH_HINT_IS_RX(h)				\
+	((!((h) & ~(__RTE_ETH_DEV_STASH_HINT_RX_MASK))) && (h))
+
+#define RTE_ETH_DEV_STASH_HINT_IS_RXTX(h)		\
+	((!((h) & ~(__RTE_ETH_DEV_STASH_HINT_RXTX_MASK))) && (h))
+
+/**@{@name Stashable Rx/Tx queue object types supported by the ethernet device
+ *@see rte_eth_dev_stashing_hints_discover
+ *@see rte_eth_dev_stashing_hints_rx
+ *@see rte_eth_dev_stashing_hints_tx
+ */
+
+/**
+ * Apply stashing hint to data at a given offset from the start of a
+ * received packet.
+ */
+#define RTE_ETH_DEV_STASH_TYPE_OFFSET	0x0001
+
+/** Apply stashing hint to an rx descriptor. */
+#define RTE_ETH_DEV_STASH_TYPE_DESC	0x0002
+
+/** Apply stashing hint to a header of a received packet. */
+#define RTE_ETH_DEV_STASH_TYPE_HEADER	0x0004
+
+/** Apply stashing hint to a payload of a received packet. */
+#define RTE_ETH_DEV_STASH_TYPE_PAYLOAD	0x0008
+#define __RTE_ETH_DEV_STASH_TYPE_MASK	0x000f
+/**@}*/
+
+#define RTE_ETH_DEV_STASH_TYPE_VALID(t)				\
+	((!((t) & (~__RTE_ETH_DEV_STASH_TYPE_MASK))) && (t))
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * @internal
+ * Helper function to validate stashing hints.
+ */
+__rte_experimental
+int rte_eth_dev_validate_stashing_hints(uint16_t port_id, uint16_t queue_id,
+					uint8_t queue_direction, uint16_t type,
+					uint16_t hint);
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Provide cache stashing hints for improved memory access latencies for
+ * packets received by the NIC. Hints the underlying hardware that CPU indicated
+ * in cpuid parameter prefers to have the data specified in the type parameter
+ * at a level in the memory hierarchy specified in cache_level parameter for
+ * access pattern(s) specified in hints parameter.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param cpuid
+ *  ID of the targeted CPU for the hint.
+ * @param cache_level
+ *  The preferred level of the cache the packets are expected at the time of
+ *  retrieval.
+ * @param queue_id
+ *  The index of the receive queue to which hints are applied.
+ * @param types
+ *  A vector of stashing types to apply hints on receive queue.
+ *  Hints are applied on the types specified in types vector.
+ *  types can include receive queue descriptors (RTE_ETH_DEV_STASH_TYPE_DESC),
+ *  packet headers (RTE_ETH_DEV_STASH_TYPE_HEADER),
+ *  packet payloads (RTE_ETH_DEV_STASH_TYPE_PAYLOAD) or
+ *  to an offset (RTE_ETH_DEV_STASH_TYPE_OFFSET) in to packet.
+ *  Types used should be compatible with RX queues, if not -EINVAL will be
+ *  returned.
+ * @param offset
+ *  Offset into the packet if RTE_ETH_DEV_STASH_TYPE_RX_OFFSET is set in hints.
+ * @param hints
+ *  A vector of stashing hints to the device and the platform.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX types are used in conjuection in type
+ *  parameter.
+ *  - (-EINVAL) if hints are incompatible with RX queues.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is unavailable.
+ *  - (-ENOSYS) if PMD does not implement cache stashing hints.
+ *  - (0) on Success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_hints_rx(uint16_t port_id, uint16_t cpuid,
+				 uint8_t cache_level, uint16_t queue_id,
+				 uint16_t types, off_t offset, uint16_t hints);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Provide cache stashing hints for improved memory access latencies for
+ * packets being transmitted by the NIC. Hints the underlying hardware that CPU
+ * prefers to have the data specified in the type parameter at a level in the
+ * memory hierarchy specified in cache_level parameter for an access pattern
+ * specified in hints parameter.
+ * This feature is available only in supported NICs and platforms.
+ *
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param cpuid
+ *  ID of the targeted CPU for the hint.
+ * @param cache_level
+ *  The preferred level of the cache the packets are expected at the time of
+ *  transmission.
+ * @param queue_id
+ *  The index of the transmit queue which hints are applied to.
+ * @param types
+ *  A vector of stashing types to apply hints on transmit queue.
+ *  hints are applied on types specified in types vector.
+ *  types can innclude transmit queue descriptors (RTE_ETH_DEV_STASH_TYPE_DESC),
+ *  packet headers (RTE_ETH_DEV_STASH_TYPE_HEADER),
+ *  packet payloads (RTE_ETH_DEV_STASH_TYPE_PAYLOAD) or
+ *  to an offset (RTE_ETH_DEV_STASH_TYPE_OFFSET) in to packet.
+ *  Types used should be compatible with TX queues, if not -EINVAL will be
+ *  returned.
+ * @param offset
+ *  Offset into the packet if RTE_ETH_DEV_STASH_TYPE_RX_OFFSET is set in hints.
+ * @param hints
+ *  A vector of stashing hints to the device and the platform.
+ * @return
+ *  - (-ENODEV) on incorrect port_ids.
+ *  - (-EINVAL) if both RX and TX types are used in conjuection in type
+ *  parameter.
+ *  - (-EINVAL) if hints are incompatible with TX queues.
+ *  - (-EINVAL) on invalid queue_id.
+ *  - (-ENOTSUP) if RTE_ETH_DEV_CAPA_CACHE_STASHING capability is unavailable.
+ *  - (-ENOSYS) if PMD does not implement cache stashing hints.
+ *  - (0) on Success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_hints_tx(uint16_t port_id, uint16_t cpuid,
+				 uint8_t cache_level, uint16_t queue_id,

lib/ethdev/rte_ethdev.h int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config * · line 6172

Quoted 10 lines
lib/ethdev/rte_ethdev.h +10 -0
+				 uint16_t types, off_t offset, uint16_t hints);
+
+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Discover cache stashing hints and object types supported in the ethernet
+ * device.
+ *

Why is this needed?

It seems to me the application should just give hints what it thinks is best, and then the PMD should do the best it can with that information.

No validating, no discovering.

Now you have a lot of API verbiage for a fairly esoteric function.

Just to be clear: I think an API like this is a great idea. Controlling how the NIC and other devices loads and stores data can be crucial.

Quoted 39 lines
lib/ethdev/rte_ethdev.h +19 -0
+ * @param port_id
+ *  The port identifier of the Ethernet device.
+ * @param types
+ *  Supported types vector set by the ethernet device.
+ * @param hints
+ *  Supported hints vector set by the ethernet device.
+ * @return
+ *  On return types and hints parameters will have bits set for supported
+ *  object types.
+ *  - (-ENOTSUP) if the device or the platform does not support cache stashing.
+ *  - (-ENOSYS)  if the underlying PMD hasn't implemented cache stashing
+ *  feature.
+ *  - (-EINVAL)  on NULL values for types or hints parameters.
+ *  - (0) on success.
+ */
+__rte_experimental
+int rte_eth_dev_stashing_hints_discover(uint16_t port_id, uint16_t *types,
+					uint16_t *hints);
+
  #include <rte_ethdev_core.h>
 
  /**
lib/ethdev/version.map +6 -0
index 79f6f5293b..5eef0b4540 100644
@@ -325,6 +325,12 @@ EXPERIMENTAL {
  	rte_flow_template_table_resizable;
  	rte_flow_template_table_resize;
  	rte_flow_template_table_resize_complete;
+
+	# added in 24.07
+	rte_eth_dev_stashing_hints_rx;
+	rte_eth_dev_stashing_hints_tx;
+	rte_eth_dev_stashing_hints_discover;
+	rte_eth_dev_validate_stashing_hints;
  };
 
  INTERNAL {
#61 lore raw
Quoted 36 lines · On Wed, 23 Oct 2024 19:59:35 +0200 Mattias Rönnblom <[email protected]>…
Quoted 8 lines
lib/ethdev/ethdev_driver.h +0 -0
index 883e59a927..b90dc8793b 100644
@@ -1235,6 +1235,70 @@ typedef int (*eth_count_aggr_ports_t)(struct rte_eth_dev *dev);
  typedef int (*eth_map_aggr_tx_affinity_t)(struct rte_eth_dev *dev, uint16_t tx_queue_id,
  					  uint8_t affinity);
 

lib/ethdev/ethdev_driver.h typedef int (*eth_count_aggr_ports_t)(struct rte_eth_dev *dev); · line 1247

Quoted 10 lines
lib/ethdev/ethdev_driver.h +10 -0
+/**
+ * @internal
+ * Set cache stashing hint in the ethernet device.
+ *
+ * @param dev
+ *   Port (ethdev) handle.
+ * @param cpuid
+ *   ID of the targeted CPU.
+ * @param cache_level
+ *   Level of the cache to stash data.  

If we had a hwtopo API in DPDK, we could just use a node id in such a graph (of CPUs and caches) to describe were the data ideally would land. In such a case, you could have a node id for DDR as well, and thus you could drop the notion of "stashing". Just a "drop off the data here, please, if you can" API.

I don't think this API and its documentation should talk about what the "CPU" needs, since it's somewhat misleading.

For example, you can imagine you want the packet payload to land in the LLC, even though it's not for any CPU to consume, in case you know with some certaintly that the packet will soon be transmitted (and thus consumed by the NIC).

The same scenario can happen, the consumer is an accelerator (e.g., a crypto engine).

Quoted 8 lines

Likewise, you may know that the whole packet will be read by some CPU core, but you also know the system tends to buffer packets before they are being processed. In such a case, it's better to go to DRAM right away, to avoid trashing the LLC (or some other cache).

Also, why do you need to use the word "host"? Seems like a PCI thing. This may be implemented in PCI, but surely can be done (and has been done) without PCI.

+1 for the concept of having a CPU and PCI topology map that can be queried by drivers and application. Dumpster diving into sysfs is hard to get right and keeps growing. I wonder if there exists an open source library that is a good enough starting point for this already.

#62 lore raw
In reply to Mattias Rönnblom
Quoted 17 lines

If we had a hwtopo API in DPDK, we could just use a node id in such a graph (of CPUs and caches) to describe were the data ideally would land. In such a case, you could have a node id for DDR as well, and thus you could drop the notion of "stashing". Just a "drop off the data here, please, if you can" API.

I don't think this API and its documentation should talk about what the "CPU" needs, since it's somewhat misleading.

For example, you can imagine you want the packet payload to land in the LLC, even though it's not for any CPU to consume, in case you know with some certaintly that the packet will soon be transmitted (and thus consumed by the NIC).

The same scenario can happen, the consumer is an accelerator (e.g., a crypto engine).

Quoted 9 lines

Likewise, you may know that the whole packet will be read by some CPU core, but you also know the system tends to buffer packets before they are being processed. In such a case, it's better to go to DRAM right away, to avoid trashing the LLC (or some other cache).

Also, why do you need to use the word "host"? Seems like a PCI thing. This may be implemented in PCI, but surely can be done (and has been done) without PCI.

Thanks, Mattias. V2 is outdated, please provide feedback on V3.

#63 lore raw
In reply to Mattias Rönnblom
Quoted 1 line · Mattias Rönnblom

On 2024-07-16 00:11, Wathsala Vithanage wrote:

...

Quoted 11 lines
Quoted 10 lines

+/**
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change, or be removed, without prior notice
+ *
+ * Discover cache stashing hints and object types supported in the ethernet
+ * device.
+ *

Quoted 3 lines

Why is this needed?

It seems to me the application should just give hints what it thinks is best, and then the PMD should do the best it can with that information.

+1, I think it is a very good idea

Quoted 6 lines

No validating, no discovering.

Now you have a lot of API verbiage for a fairly esoteric function.

Just to be clear: I think an API like this is a great idea. Controlling how the NIC and other devices loads and stores data can be crucial.

+1