raw/ioat: enhancements and new hardware support
PATCH v5 7/25 raw/ioat: rename functions to be operation-agnostic
In reply to Bruce RichardsonSince the hardware supported by the ioat driver is capable of operations other than just copies, we can rename the doorbell and completion-return functions to not have "copies" in their names. These functions are not copy-specific, and so would apply for other operations which may be added later to the driver.
Also add a suitable warning using deprecation attribute for any code using the old functions names.
--- doc/guides/rawdevs/ioat.rst | 16 ++++++++-------- doc/guides/rel_notes/release_20_11.rst | 9 +++++++++ doc/guides/sample_app_ug/ioat.rst | 8 ++++---- drivers/raw/ioat/ioat_rawdev_test.c | 12 ++++++------ drivers/raw/ioat/rte_ioat_rawdev.h | 14 +++++++------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 20 ++++++++++++++++---- examples/ioat/ioatfwd.c | 4 ++-- lib/librte_eal/include/rte_common.h | 1 + 8 files changed, 53 insertions(+), 31 deletions(-)
doc/guides/rawdevs/ioat.rst +8 -8
@@ -157,9 +157,9 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ To perform data copies using IOAT rawdev devices, the functions -``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used. +``rte_ioat_enqueue_copy()`` and ``rte_ioat_perform_ops()`` should be used. Once copies have been completed, the completion will be reported back when -the application calls ``rte_ioat_completed_copies()``. +the application calls ``rte_ioat_completed_ops()``. The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to that function
@@ -172,11 +172,11 @@ pointers if packet data is being copied. While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on the device ring, the copy will not actually be performed until after the -application calls the ``rte_ioat_do_copies()`` function. This function +application calls the ``rte_ioat_perform_ops()`` function. This function informs the device hardware of the elements enqueued on the ring, and the device will begin to process them. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple -enqueue calls between calls to the ``rte_ioat_do_copies()`` function. +enqueue calls between calls to the ``rte_ioat_perform_ops()`` function. The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue a burst of copies to the device and start the hardware processing of them:
@@ -210,10 +210,10 @@ a burst of copies to the device and start the hardware processing of them: return -1; } } - rte_ioat_do_copies(dev_id); + rte_ioat_perform_ops(dev_id); To retrieve information about completed copies, the API -``rte_ioat_completed_copies()`` should be used. This API will return to the +``rte_ioat_completed_ops()`` should be used. This API will return to the application a set of completion handles passed in when the relevant copies were enqueued.
@@ -223,9 +223,9 @@ is correct before freeing the data buffers using the returned handles: .. code-block:: C - if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src, + if (rte_ioat_completed_ops(dev_id, 64, (void *)completed_src, (void *)completed_dst) != RTE_DIM(srcs)) { - printf("Error with rte_ioat_completed_copies\n"); + printf("Error with rte_ioat_completed_ops\n"); return -1; } for (i = 0; i < RTE_DIM(srcs); i++) {
doc/guides/rel_notes/release_20_11.rst +9 -0
@@ -121,6 +121,11 @@ New Features The ioat rawdev driver has been updated and enhanced. Changes include: * Added a per-device configuration flag to disable management of user-provided completion handles + * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, + and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()`` + to better reflect the APIs' purposes, and remove the implication that + they are limited to copy operations only. + [Note: The old API is still provided but marked as deprecated in the code] Removed Items
@@ -234,6 +239,10 @@ API Changes * bpf: ``RTE_BPF_XTYPE_NUM`` has been dropped from ``rte_bpf_xtype``. +* raw/ioat: As noted above, the ``rte_ioat_do_copies()`` and + ``rte_ioat_completed_copies()`` functions have been renamed to + ``rte_ioat_perform_ops()`` and ``rte_ioat_completed_ops()`` respectively. + ABI Changes -----------
doc/guides/sample_app_ug/ioat.rst +4 -4
@@ -394,7 +394,7 @@ packet using ``pktmbuf_sw_copy()`` function and enqueue them to an rte_ring: nb_enq = ioat_enqueue_packets(pkts_burst, nb_rx, rx_config->ioat_ids[i]); if (nb_enq > 0) - rte_ioat_do_copies(rx_config->ioat_ids[i]); + rte_ioat_perform_ops(rx_config->ioat_ids[i]); } else { /* Perform packet software copy, free source packets */ int ret;
@@ -433,7 +433,7 @@ The packets are received in burst mode using ``rte_eth_rx_burst()`` function. When using hardware copy mode the packets are enqueued in copying device's buffer using ``ioat_enqueue_packets()`` which calls ``rte_ioat_enqueue_copy()``. When all received packets are in the -buffer the copy operations are started by calling ``rte_ioat_do_copies()``. +buffer the copy operations are started by calling ``rte_ioat_perform_ops()``. Function ``rte_ioat_enqueue_copy()`` operates on physical address of the packet. Structure ``rte_mbuf`` contains only physical address to start of the data buffer (``buf_iova``). Thus the address is adjusted
@@ -490,7 +490,7 @@ or indirect mbufs, then multiple copy operations must be used. All completed copies are processed by ``ioat_tx_port()`` function. When using -hardware copy mode the function invokes ``rte_ioat_completed_copies()`` +hardware copy mode the function invokes ``rte_ioat_completed_ops()`` on each assigned IOAT channel to gather copied packets. If software copy mode is used the function dequeues copied packets from the rte_ring. Then each packet MAC address is changed if it was enabled. After that copies are sent
@@ -510,7 +510,7 @@ in burst mode using `` rte_eth_tx_burst()``. for (i = 0; i < tx_config->nb_queues; i++) { if (copy_mode == COPY_MODE_IOAT_NUM) { /* Deque the mbufs from IOAT device. */ - nb_dq = rte_ioat_completed_copies( + nb_dq = rte_ioat_completed_ops( tx_config->ioat_ids[i], MAX_PKT_BURST, (void *)mbufs_src, (void *)mbufs_dst); } else {
drivers/raw/ioat/ioat_rawdev_test.c +6 -6
@@ -65,12 +65,12 @@ test_enqueue_copies(int dev_id)PRINT_ERR ("Error with rte_ioat_enqueue_copy\n" );return -1 ; } -rte_ioat_do_copies (dev_id); +rte_ioat_perform_ops (dev_id);usleep (10 ); -if (rte_ioat_completed_copies (dev_id,1 , (void *)&completed[0 ], +if (rte_ioat_completed_ops (dev_id,1 , (void *)&completed[0 ], (void *)&completed[1 ]) !=1 ) { -PRINT_ERR ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_ops\n" );return -1 ; }if (completed[0 ] != src || completed[1 ] != dst) {
@@ -119,12 +119,12 @@ test_enqueue_copies(int dev_id)return -1 ; } } -rte_ioat_do_copies (dev_id); +rte_ioat_perform_ops (dev_id);usleep (100 ); -if (rte_ioat_completed_copies (dev_id,64 , (void *)completed_src, +if (rte_ioat_completed_ops (dev_id,64 , (void *)completed_src, (void *)completed_dst) !=RTE_DIM (srcs)) { -PRINT_ERR ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_ops\n" );return -1 ; }for (i =0 ; i <RTE_DIM (srcs); i++) {
drivers/raw/ioat/rte_ioat_rawdev.h +7 -7
@@ -74,19 +74,19 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, int fence);/** -* Trigger hardware to begin performing enqueued copy operations +* Trigger hardware to begin performing enqueued operations * * This API is used to write the "doorbell" to the hardware to trigger it -* to begin the copy operations previously enqueued by rte_ioat_enqueue_copy() +* to begin the operations previously enqueued by rte_ioat_enqueue_copy() * * @param dev_id * The rawdev device id of the ioat instance */ static inline void -rte_ioat_do_copies (int dev_id); +rte_ioat_perform_ops (int dev_id);/** -* Returns details of copy operations that have been completed +* Returns details of operations that have been completed * * If the hdls_disable option was not set when the device was configured, * the function will return to the caller the user-provided "handles" for
@@ -104,11 +104,11 @@ rte_ioat_do_copies(int dev_id);* NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @param src_hdls -* Array to hold the source handle parameters of the completed copies. +* Array to hold the source handle parameters of the completed ops. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @param dst_hdls -* Array to hold the destination handle parameters of the completed copies. +* Array to hold the destination handle parameters of the completed ops. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @return
@@ -117,7 +117,7 @@ rte_ioat_do_copies(int dev_id);* to the src_hdls and dst_hdls array parameters. */ static inline int -rte_ioat_completed_copies (int dev_id,uint8_t max_copies, +rte_ioat_completed_ops (int dev_id,uint8_t max_copies,uintptr_t *src_hdls,uintptr_t *dst_hdls);/* include the implementation details from a separate file */
drivers/raw/ioat/rte_ioat_rawdev_fns.h +16 -4
@@ -83,10 +83,10 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, }/* -* Trigger hardware to begin performing enqueued copy operations +* Trigger hardware to begin performing enqueued operations */ static inline void -rte_ioat_do_copies (int dev_id) +rte_ioat_perform_ops (int dev_id) {struct rte_ioat_rawdev *ioat = (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;
@@ -114,10 +114,10 @@ rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error) }/* -* Returns details of copy operations that have been completed +* Returns details of operations that have been completed */ static inline int -rte_ioat_completed_copies (int dev_id,uint8_t max_copies, +rte_ioat_completed_ops (int dev_id,uint8_t max_copies,uintptr_t *src_hdls,uintptr_t *dst_hdls) { struct rte_ioat_rawdev *ioat =
@@ -165,4 +165,16 @@ rte_ioat_completed_copies(int dev_id, uint8_t max_copies,return count; } +static inline void +__rte_deprecated_msg("use rte_ioat_perform_ops() instead") +rte_ioat_do_copies(int dev_id) { rte_ioat_perform_ops(dev_id); } + +static inline int +__rte_deprecated_msg("use rte_ioat_completed_ops() instead") +rte_ioat_completed_copies(int dev_id, uint8_t max_copies, + uintptr_t *src_hdls, uintptr_t *dst_hdls) +{ +return rte_ioat_completed_ops (dev_id, max_copies, src_hdls, dst_hdls); +} +#endif /* _RTE_IOAT_RAWDEV_FNS_H_ */
examples/ioat/ioatfwd.c +2 -2
@@ -406,7 +406,7 @@ ioat_rx_port(struct rxtx_port_config *rx_config) nb_enq =ioat_enqueue_packets (pkts_burst, nb_rx, rx_config->ioat_ids[i]);if (nb_enq >0 ) -rte_ioat_do_copies (rx_config->ioat_ids[i]); +rte_ioat_perform_ops (rx_config->ioat_ids[i]); }else {/* Perform packet software copy, free source packets */ int ret;
@@ -452,7 +452,7 @@ ioat_tx_port(struct rxtx_port_config *tx_config)for (i =0 ; i < tx_config->nb_queues; i++) {if (copy_mode ==COPY_MODE_IOAT_NUM ) {/* Deque the mbufs from IOAT device. */ - nb_dq =rte_ioat_completed_copies ( + nb_dq =rte_ioat_completed_ops ( tx_config->ioat_ids[i],MAX_PKT_BURST , (void *)mbufs_src, (void *)mbufs_dst); }else {
lib/librte_eal/include/rte_common.h +1 -0
@@ -85,6 +85,7 @@ typedef uint16_t unaligned_uint16_t;/******* Macro to mark functions and fields scheduled for removal *****/ #define __rte_deprecated __attribute__((__deprecated__)) +#define __rte_deprecated_msg (msg) __attribute__((__deprecated__(msg)))/** * Mark a function or variable to a weak reference.
Signature · 1 line
2.25.1
PATCH v5 8/25 raw/ioat: add separate API for fence call
In reply to Bruce RichardsonRather than having the fence signalled via a flag on a descriptor - which requires reading the docs to find out whether the flag needs to go on the last descriptor before, or the first descriptor after the fence - we can instead add a separate fence API call. This becomes unambiguous to use, since the fence call explicitly comes between two other enqueue calls. It also allows more freedom of implementation in the driver code.
--- doc/guides/rawdevs/ioat.rst | 3 +-- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/raw/ioat/ioat_rawdev_test.c | 6 ++---- drivers/raw/ioat/rte_ioat_rawdev.h | 26 ++++++++++++++++++++------ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 22 +++++++++++++++++++--- examples/ioat/ioatfwd.c | 12 ++++-------- 6 files changed, 50 insertions(+), 23 deletions(-)
doc/guides/rawdevs/ioat.rst +1 -2
@@ -203,8 +203,7 @@ a burst of copies to the device and start the hardware processing of them: dsts[i]->buf_iova + dsts[i]->data_off, length, (uintptr_t)srcs[i], - (uintptr_t)dsts[i], - 0 /* nofence */) != 1) { + (uintptr_t)dsts[i]) != 1) { printf("Error with rte_ioat_enqueue_copy for buffer %u\n", i); return -1;
doc/guides/rel_notes/release_20_11.rst +4 -0
@@ -126,6 +126,10 @@ New Features to better reflect the APIs' purposes, and remove the implication that they are limited to copy operations only. [Note: The old API is still provided but marked as deprecated in the code] + * Added a new API ``rte_ioat_fence()`` to add a fence between operations. + This API replaces the ``fence`` flag parameter in the ``rte_ioat_enqueue_copies()`` function, + and is clearer as there is no ambiguity as to whether the flag should be + set on the last operation before the fence or the first operation after it. Removed Items
drivers/raw/ioat/ioat_rawdev_test.c +2 -4
@@ -60,8 +60,7 @@ test_enqueue_copies(int dev_id) dst->buf_iova + dst->data_off, length, (uintptr_t)src, - (uintptr_t)dst, - 0 /* no fence */) != 1) { + (uintptr_t)dst) != 1) {PRINT_ERR ("Error with rte_ioat_enqueue_copy\n" );return -1 ; }
@@ -112,8 +111,7 @@ test_enqueue_copies(int dev_id) dsts[i]->buf_iova + dsts[i]->data_off, length, (uintptr_t)srcs[i], - (uintptr_t)dsts[i], - 0 /* nofence */) != 1) { + (uintptr_t)dsts[i]) != 1) {PRINT_ERR ("Error with rte_ioat_enqueue_copy for buffer %u\n" , i);return -1 ;
drivers/raw/ioat/rte_ioat_rawdev.h +20 -6
@@ -61,17 +61,31 @@ struct rte_ioat_rawdev_config {* operation has been completed and the user polls for the completion details. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. -* @param fence -* A flag parameter indicating that hardware should not begin to perform any -* subsequently enqueued copy operations until after this operation has -* completed * @return * Number of operations enqueued, either 0 or 1 */ static inline int rte_ioat_enqueue_copy (int dev_id,phys_addr_t src,phys_addr_t dst, -unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl, -int fence); +unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl); + +/** +* Add a fence to force ordering between operations +* +* This adds a fence to a sequence of operations to enforce ordering, such that +* all operations enqueued before the fence must be completed before operations +* after the fence. +* NOTE: Since this fence may be added as a flag to the last operation enqueued, +* this API may not function correctly when called immediately after an +* "rte_ioat_perform_ops" call i.e. before any new operations are enqueued. +* +* @param dev_id +* The rawdev device id of the ioat instance +* @return +* Number of fences enqueued, either 0 or 1 +*/ +static inline int +rte_ioat_fence (int dev_id); +/** * Trigger hardware to begin performing enqueued operations
drivers/raw/ioat/rte_ioat_rawdev_fns.h +19 -3
@@ -47,8 +47,7 @@ struct rte_ioat_rawdev {*/ static inline int rte_ioat_enqueue_copy (int dev_id,phys_addr_t src,phys_addr_t dst, -unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl, -int fence) +unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl) {struct rte_ioat_rawdev *ioat = (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;
@@ -69,7 +68,7 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, desc = &ioat->desc_ring[write]; desc->size = length;/* set descriptor write-back every 16th descriptor */ - desc->u.control_raw = (uint32_t )((!!fence <<4 ) | (!(write &0xF )) <<3 ); + desc->u.control_raw = (uint32_t )((!(write &0xF )) <<3 ); desc->src_addr = src; desc->dest_addr = dst;
@@ -82,6 +81,23 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,return 1 ; } +/* add fence to last written descriptor */ +static inline int +rte_ioat_fence (int dev_id) +{ +struct rte_ioat_rawdev *ioat = + (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; +unsigned short write = ioat->next_write; +unsigned short mask = ioat->ring_size -1 ; +struct rte_ioat_generic_hw_desc *desc; + + write = (write -1 ) & mask; + desc = &ioat->desc_ring[write]; + + desc->u.control.fence =1 ; +return 0 ; +} +/* * Trigger hardware to begin performing enqueued operations */
examples/ioat/ioatfwd.c +4 -8
@@ -361,15 +361,11 @@ ioat_enqueue_packets(struct rte_mbuf **pkts,for (i =0 ; i < nb_rx; i++) {/* Perform data copy */ ret =rte_ioat_enqueue_copy (dev_id, - pkts[i]->buf_iova - - addr_offset, - pkts_copy[i]->buf_iova - - addr_offset, -rte_pktmbuf_data_len (pkts[i]) - + addr_offset, + pkts[i]->buf_iova - addr_offset, + pkts_copy[i]->buf_iova - addr_offset, +rte_pktmbuf_data_len (pkts[i]) + addr_offset, (uintptr_t )pkts[i], - (uintptr_t )pkts_copy[i], -0 /* nofence */ ); + (uintptr_t )pkts_copy[i]);if (ret !=1 )break ;
Signature · 1 line
2.25.1
PATCH v5 9/25 raw/ioat: make the HW register spec private
In reply to Bruce RichardsonOnly a few definitions from the hardware spec are actually used in the driver runtime, so we can copy over those few and make the rest of the spec a private header in the driver.
---
drivers/raw/ioat/ioat_rawdev.c | 3 ++
.../raw/ioat/{rte_ioat_spec.h => ioat_spec.h} | 26 -----------
drivers/raw/ioat/meson.build | 3 +-
drivers/raw/ioat/rte_ioat_rawdev_fns.h | 43 +++++++++++++++++--
4 files changed, 44 insertions(+), 31 deletions(-)
rename drivers/raw/ioat/{rte_ioat_spec.h => ioat_spec.h} (91%)drivers/raw/ioat/ioat_rawdev.c +3 -0
@@ -4,10 +4,12 @@#include <rte_cycles.h> #include <rte_bus_pci.h> +#include <rte_memzone.h> #include <rte_string_fns.h> #include <rte_rawdev_pmd.h> #include "rte_ioat_rawdev.h" +#include "ioat_spec.h" static struct rte_pci_driver ioat_pmd_drv;
@@ -268,6 +270,7 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) ioat->rawdev = rawdev; ioat->mz = mz; ioat->regs = dev->mem_resource[0 ].addr; + ioat->doorbell = &ioat->regs->dmacount; ioat->ring_size =0 ; ioat->desc_ring =NULL ; ioat->status_addr = ioat->mz->iova +
drivers/raw/ioat/ioat_spec.h renamed from drivers/raw/ioat/rte_ioat_spec.h +0 -26
@@ -86,32 +86,6 @@ struct rte_ioat_registers {#define RTE_IOAT_CHANCMP_ALIGN 8/* CHANCMP address must be 64-bit aligned */ -struct rte_ioat_generic_hw_desc { -uint32_t size; -union { -uint32_t control_raw; -struct { -uint32_t int_enable:1 ; -uint32_t src_snoop_disable:1 ; -uint32_t dest_snoop_disable:1 ; -uint32_t completion_update:1 ; -uint32_t fence:1 ; -uint32_t reserved2:1 ; -uint32_t src_page_break:1 ; -uint32_t dest_page_break:1 ; -uint32_t bundle:1 ; -uint32_t dest_dca:1 ; -uint32_t hint:1 ; -uint32_t reserved:13 ; -uint32_t op:8 ; - } control; - } u; -uint64_t src_addr; -uint64_t dest_addr; -uint64_t next; -uint64_t op_specific[4 ]; -}; -struct rte_ioat_dma_hw_desc {uint32_t size;union {
drivers/raw/ioat/meson.build +1 -2
@@ -8,5 +8,4 @@ sources = files('ioat_rawdev.c', deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h', - 'rte_ioat_rawdev_fns.h', - 'rte_ioat_spec.h') + 'rte_ioat_rawdev_fns.h')
drivers/raw/ioat/rte_ioat_rawdev_fns.h +40 -3
69 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v5 10/25 usertools/dpdk-devbind.py: add support for DSA HW
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
Intel Data Streaming Accelerator (Intel DSA) is a high-performance data copy and transformation accelerator which will be integrated in future Intel processors [1].
Add DSA device support to dpdk-devbind.py script.
[1] https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
--- doc/guides/rel_notes/release_20_11.rst | 2 ++ usertools/dpdk-devbind.py | 6 +++++- 2 files changed, 7 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_20_11.rst +2 -0
@@ -120,6 +120,8 @@ New Features The ioat rawdev driver has been updated and enhanced. Changes include: + * Added support for Intel\ |reg| Data Streaming Accelerator hardware. + For more information, see https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator * Added a per-device configuration flag to disable management of user-provided completion handles * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()``
usertools/dpdk-devbind.py +5 -1
@@ -48,6 +48,8 @@ 'SVendor': None, 'SDevice': None} intel_ioat_icx = {'Class': '08', 'Vendor': '8086', 'Device': '0b00', 'SVendor': None, 'SDevice': None} +intel_idxd_spr = {'Class': '08', 'Vendor': '8086', 'Device': '0b25', + 'SVendor': None, 'SDevice': None} intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', 'SVendor': None, 'SDevice': None} intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
@@ -59,7 +61,9 @@ eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] -misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, octeontx2_dma] +misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, intel_idxd_spr, + intel_ntb_skx, intel_ntb_icx, + octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
2.25.1
PATCH v5 11/25 raw/ioat: add skeleton for VFIO/UIO based DSA device
In reply to Bruce RichardsonAdd in the basic probe/remove skeleton code for DSA devices which are bound directly to vfio or uio driver. The kernel module for supporting these uses the "idxd" name, so that name is used as function and file prefix to avoid conflict with existing "ioat" prefixed functions.
Since we are adding new files to the driver and there will be common definitions shared between the various files, we create a new internal header file ioat_private.h to hold common macros and function prototypes.
--- doc/guides/rawdevs/ioat.rst | 69 ++++++++++----------------------- drivers/raw/ioat/idxd_pci.c | 56 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 27 +++++++++++++ drivers/raw/ioat/ioat_rawdev.c | 9 +---- drivers/raw/ioat/meson.build | 6 ++- 5 files changed, 108 insertions(+), 59 deletions(-) create mode 100644 drivers/raw/ioat/idxd_pci.c create mode 100644 drivers/raw/ioat/ioat_private.h
doc/guides/rawdevs/ioat.rst +20 -49
@@ -3,10 +3,12 @@ .. include:: <isonum.txt> -IOAT Rawdev Driver for Intel\ |reg| QuickData Technology -====================================================================== +IOAT Rawdev Driver +=================== The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| +Data Streaming Accelerator `(Intel DSA) +<https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology `(Intel I/OAT) <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
@@ -17,61 +19,30 @@ be done by software, freeing up CPU cycles for other tasks. Hardware Requirements ---------------------- -On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can -be detected by checking the output of the ``lspci`` command, where the -hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For -example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz, -lspci shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) - 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) - 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) - 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) - 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) - 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) - 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) - 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) - -On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci -shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - +The ``dpdk-devbind.py`` script, included with DPDK, +can be used to show the presence of supported hardware. +Running ``dpdk-devbind.py --status-dev misc`` will show all the miscellaneous, +or rawdev-based devices on the system. +For Intel\ |reg| QuickData Technology devices, the hardware will be often listed as "Crystal Beach DMA", +or "CBDMA". +For Intel\ |reg| DSA devices, they are currently (at time of writing) appearing as devices with type "0b25", +due to the absence of pci-id database entries for them at this point. Compilation ------------ -For builds done with ``make``, the driver compilation is enabled by the -``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is -enabled by default in builds for x86 platforms, and disabled in other -configurations. - -For builds using ``meson`` and ``ninja``, the driver will be built when the -target platform is x86-based. +For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. +No additional compilation steps are necessary. Device Setup ------------- -The Intel\ |reg| QuickData Technology HW devices will need to be bound to a -user-space IO driver for use. The script ``dpdk-devbind.py`` script -included with DPDK can be used to view the state of the devices and to bind -them to a suitable DPDK-supported kernel driver. When querying the status -of the devices, they will appear under the category of "Misc (rawdev) -devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be -used to see the state of those devices alone. +The HW devices to be used will need to be bound to a user-space IO driver for use. +The ``dpdk-devbind.py`` script can be used to view the state of the devices +and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1 Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/raw/ioat/idxd_pci.c new file +56 -0
56 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2020 Intel Corporation +*/ + +#ifndef _IOAT_PRIVATE_H_ +#define _IOAT_PRIVATE_H_ + +/** +* @file idxd_private.h +* +* Private data structures for the idxd/DSA part of ioat device driver +* +* @warning +* @b EXPERIMENTAL: these structures and APIs may change without prior notice +*/ + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev.c +1 -8
@@ -10,6 +10,7 @@#include "rte_ioat_rawdev.h" #include "ioat_spec.h" +#include "ioat_private.h" static struct rte_pci_driver ioat_pmd_drv;
@@ -29,14 +30,6 @@ static struct rte_pci_driver ioat_pmd_drv;RTE_LOG_REGISTER (ioat_pmd_logtype, rawdev.ioat,INFO ); -#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ - ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) - -#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) -#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) -#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) -#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) -#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)#define COMPLETION_SZ sizeof(__m128i)
drivers/raw/ioat/meson.build +4 -2
@@ -3,8 +3,10 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' -sources = files('ioat_rawdev.c', - 'ioat_rawdev_test.c') +sources = files( + 'idxd_pci.c', + 'ioat_rawdev.c', + 'ioat_rawdev_test.c') deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h',
Signature · 1 line
2.25.1
PATCH v5 12/25 raw/ioat: add vdev probe for DSA/idxd devices
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
The Intel DSA devices can be exposed to userspace via kernel driver, so can be used without having to bind them to vfio/uio. Therefore we add support for using those kernel-configured devices as vdevs, taking as parameter the individual HW work queue to be used by the vdev.
--- doc/guides/rawdevs/ioat.rst | 68 +++++++++++++++++-- drivers/raw/ioat/idxd_vdev.c | 123 +++++++++++++++++++++++++++++++++++ drivers/raw/ioat/meson.build | 6 +- 3 files changed, 192 insertions(+), 5 deletions(-) create mode 100644 drivers/raw/ioat/idxd_vdev.c
doc/guides/rawdevs/ioat.rst +64 -4
@@ -37,9 +37,62 @@ No additional compilation steps are necessary. Device Setup ------------- +Depending on support provided by the PMD, HW devices can either use the kernel configured driver +or be bound to a user-space IO driver for use. +For example, Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers, +such as ``vfio-pci``. + +Intel\ |reg| DSA devices using idxd kernel driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To use a Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured. +The `accel-config <https://github.com/intel/idxd-config>`_ utility library can be used for configuration. + +.. note:: + The device configuration can also be done by directly interacting with the sysfs nodes. + +There are some mandatory configuration steps before being able to use a device with an application. +The internal engines, which do the copies or other operations, +and the work-queues, which are used by applications to assign work to the device, +need to be assigned to groups, and the various other configuration options, +such as priority or queue depth, need to be set for each queue. + +To assign an engine to a group:: + + $ accel-config config-engine dsa0/engine0.0 --group-id=0 + $ accel-config config-engine dsa0/engine0.1 --group-id=1 + +To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used. +However, the work queues also need to be configured depending on the use-case. +Some configuration options include: + +* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously. +* priority: WQ priority between 1 and 15. Larger value means higher priority. +* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device. +* type: WQ type (kernel/mdev/user). Determines how the device is presented. +* name: identifier given to the WQ. + +Example configuration for a work queue:: + + $ accel-config config-wq dsa0/wq0.0 --group-id=0 \ + --mode=dedicated --priority=10 --wq-size=8 \ + --type=user --name=app1 + +Once the devices have been configured, they need to be enabled:: + + $ accel-config enable-device dsa0 + $ accel-config enable-wq dsa0/wq0.0 + +Check the device configuration:: + + $ accel-config list + +Devices using VFIO/UIO drivers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The HW devices to be used will need to be bound to a user-space IO driver for use. The ``dpdk-devbind.py`` script can be used to view the state of the devices -and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``. For example:: $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1
@@ -47,9 +100,16 @@ For example:: Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Once bound to a suitable kernel device driver, the HW devices will be found -as part of the PCI scan done at application initialization time. No vdev -parameters need to be passed to create or initialize the device. +For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will +be found as part of the device scan done at application initialization time without +the need to pass parameters to the application. + +If the device is bound to the IDXD kernel driver (and previously configured with sysfs), +then a specific work queue needs to be passed to the application via a vdev parameter. +This vdev parameter take the driver name and work queue name as parameters. +For example, to use work queue 0 on Intel\ |reg| DSA instance 0:: + + $ dpdk-test --no-pci --vdev=rawdev_idxd,wq=0.0 Once probed successfully, the device will appear as a ``rawdev``, that is a "raw device type" inside DPDK, and can be accessed using APIs from the
drivers/raw/ioat/idxd_vdev.c new file +123 -0
123 lines not on this page · raw message
drivers/raw/ioat/meson.build +5 -1
@@ -5,9 +5,13 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' sources = files( 'idxd_pci.c', + 'idxd_vdev.c', 'ioat_rawdev.c', 'ioat_rawdev_test.c') -deps += ['rawdev', 'bus_pci', 'mbuf'] +deps += ['bus_pci', + 'bus_vdev', + 'mbuf', + 'rawdev'] install_headers('rte_ioat_rawdev.h', 'rte_ioat_rawdev_fns.h')
Signature · 1 line
2.25.1
PATCH v5 13/25 raw/ioat: include example configuration script
In reply to Bruce RichardsonDevices managed by the idxd kernel driver must be configured for DPDK use before it can be used by the ioat driver. This example script serves both as a quick way to get the driver set up with a simple configuration, and as the basis for users to modify it and create their own configuration scripts.
--- doc/guides/rawdevs/ioat.rst | 2 + drivers/raw/ioat/dpdk_idxd_cfg.py | 79 +++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) create mode 100755 drivers/raw/ioat/dpdk_idxd_cfg.py
doc/guides/rawdevs/ioat.rst +2 -0
@@ -50,6 +50,8 @@ The `accel-config <https://github.com/intel/idxd-config>`_ utility library can b .. note:: The device configuration can also be done by directly interacting with the sysfs nodes. + An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py`` + included in the driver source directory. There are some mandatory configuration steps before being able to use a device with an application. The internal engines, which do the copies or other operations,
drivers/raw/ioat/dpdk_idxd_cfg.py new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v5 14/25 raw/ioat: create rawdev instances on idxd PCI probe
In reply to Bruce RichardsonWhen a matching device is found via PCI probe create a rawdev instance for each queue on the hardware. Use empty self-test function for these devices so that the overall rawdev_autotest does not report failures.
--- drivers/raw/ioat/idxd_pci.c | 237 ++++++++++++++++++++++++- drivers/raw/ioat/ioat_common.c | 68 +++++++ drivers/raw/ioat/ioat_private.h | 33 ++++ drivers/raw/ioat/ioat_rawdev_test.c | 7 + drivers/raw/ioat/ioat_spec.h | 64 +++++++ drivers/raw/ioat/meson.build | 1 + drivers/raw/ioat/rte_ioat_rawdev_fns.h | 35 +++- 7 files changed, 442 insertions(+), 3 deletions(-) create mode 100644 drivers/raw/ioat/ioat_common.c
drivers/raw/ioat/idxd_pci.c +235 -2
@@ -3,8 +3,10 @@*/ #include <rte_bus_pci.h> +#include <rte_memzone.h> #include "ioat_private.h" +#include "ioat_spec.h" #define IDXD_VENDOR_ID 0x8086#define IDXD_DEVICE_ID_SPR 0x0B25
@@ -16,17 +18,246 @@ const struct rte_pci_id pci_id_idxd_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static inline int +idxd_pci_dev_command (struct idxd_rawdev *idxd,enum rte_idxd_cmds command) +{ +uint8_t err_code; +uint16_t qid = idxd->qid; +int i =0 ; + +if (command >= idxd_disable_wq && command <= idxd_reset_wq) + qid = (1 << qid); +rte_spinlock_lock (&idxd->u.pci->lk); + idxd->u.pci->regs->cmd = (command <<IDXD_CMD_SHIFT ) | qid; + +do { +rte_pause (); + err_code = idxd->u.pci->regs->cmdstatus; +if (++i >=1000 ) { +IOAT_PMD_ERR ("Timeout waiting for command response from HW" ); +rte_spinlock_unlock (&idxd->u.pci->lk); +return err_code; + } + }while (idxd->u.pci->regs->cmdstatus &CMDSTATUS_ACTIVE_MASK ); +rte_spinlock_unlock (&idxd->u.pci->lk); + +return err_code &CMDSTATUS_ERR_MASK ; +} + +static int +idxd_is_wq_enabled (struct idxd_rawdev *idxd) +{ +uint32_t state = idxd->u.pci->wq_regs[idxd->qid].wqcfg[WQ_STATE_IDX ]; +return ((state >>WQ_STATE_SHIFT ) &WQ_STATE_MASK ) ==0x1 ; +} + +static const struct rte_rawdev_ops idxd_pci_ops = { + .dev_close = idxd_rawdev_close, + .dev_selftest = idxd_rawdev_test, +}; + +/* each portal uses 4 x 4k pages */ +#define IDXD_PORTAL_SIZE (4096 * 4) + +static int +init_pci_device (struct rte_pci_device *dev,struct idxd_rawdev *idxd) +{ +struct idxd_pci_common *pci; +uint8_t nb_groups, nb_engines, nb_wqs; +uint16_t grp_offset, wq_offset;/* how far into bar0 the regs are */ +uint16_t wq_size, total_wq_size; +uint8_t lg2_max_batch, lg2_max_copy_size; +unsigned int i, err_code; + + pci =malloc (sizeof (*pci)); +if (pci ==NULL ) { +IOAT_PMD_ERR ("%s: Can't allocate memory" , __func__); + goto err; + } +rte_spinlock_init (&pci->lk); + +/* assign the bar registers, and then configure device */ + pci->regs = dev->mem_resource[0 ].addr; + grp_offset = (uint16_t )pci->regs->offsets[0 ]; + pci->grp_regs =RTE_PTR_ADD (pci->regs, grp_offset *0x100 ); + wq_offset = (uint16_t )(pci->regs->offsets[0 ] >>16 ); + pci->wq_regs =RTE_PTR_ADD (pci->regs, wq_offset *0x100 ); + pci->portals = dev->mem_resource[2 ].addr; + +/* sanity check device status */ +if (pci->regs->gensts &GENSTS_DEV_STATE_MASK ) { +/* need function-level-reset (FLR) or is enabled */ +IOAT_PMD_ERR ("Device status is not disabled, cannot init" ); + goto err; + } +if (pci->regs->cmdstatus &CMDSTATUS_ACTIVE_MASK ) { +/* command in progress */ +IOAT_PMD_ERR ("Device has a command in progress, cannot init" ); + goto err; + } + +/* read basic info about the hardware for use when configuring */ + nb_groups = (uint8_t )pci->regs->grpcap; + nb_engines = (uint8_t )pci->regs->engcap; + nb_wqs = (uint8_t )(pci->regs->wqcap >>16 ); + total_wq_size = (uint16_t )pci->regs->wqcap; + lg2_max_copy_size = (uint8_t )(pci->regs->gencap >>16 ) &0x1F ; + lg2_max_batch = (uint8_t )(pci->regs->gencap >>21 ) &0x0F ; + +IOAT_PMD_DEBUG ("nb_groups = %u, nb_engines = %u, nb_wqs = %u" , + nb_groups, nb_engines, nb_wqs); + +/* zero out any old config */ +for (i =0 ; i < nb_groups; i++) { + pci->grp_regs[i].grpengcfg =0 ; + pci->grp_regs[i].grpwqcfg[0 ] =0 ; + } +for (i =0 ; i < nb_wqs; i++) + pci->wq_regs[i].wqcfg[0 ] =0 ; + +/* put each engine into a separate group to avoid reordering */ +if (nb_groups > nb_engines) + nb_groups = nb_engines; +if (nb_groups < nb_engines) + nb_engines = nb_groups; + +/* assign engines to groups, round-robin style */ +for (i =0 ; i < nb_engines; i++) { +IOAT_PMD_DEBUG ("Assigning engine %u to group %u" , + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpengcfg |= (1ULL << i); + } + +/* now do the same for queues and give work slots to each queue */ + wq_size = total_wq_size / nb_wqs; +IOAT_PMD_DEBUG ("Work queue size = %u, max batch = 2^%u, max copy = 2^%u" , + wq_size, lg2_max_batch, lg2_max_copy_size); +for (i =0 ; i < nb_wqs; i++) { +/* add engine "i" to a group */ +IOAT_PMD_DEBUG ("Assigning work queue %u to group %u" , + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpwqcfg[0 ] |= (1ULL << i); +/* now configure it, in terms of size, max batch, mode */ + pci->wq_regs[i].wqcfg[WQ_SIZE_IDX ] = wq_size; + pci->wq_regs[i].wqcfg[WQ_MODE_IDX ] = (1 <<WQ_PRIORITY_SHIFT ) | +WQ_MODE_DEDICATED ; + pci->wq_regs[i].wqcfg[WQ_SIZES_IDX ] = lg2_max_copy_size | + (lg2_max_batch <<WQ_BATCH_SZ_SHIFT ); + } + +/* dump the group configuration to output */ +for (i =0 ; i < nb_groups; i++) { +IOAT_PMD_DEBUG ("## Group %d" , i); +IOAT_PMD_DEBUG (" GRPWQCFG: %" PRIx64, pci->grp_regs[i].grpwqcfg[0 ]); +IOAT_PMD_DEBUG (" GRPENGCFG: %" PRIx64, pci->grp_regs[i].grpengcfg); +IOAT_PMD_DEBUG (" GRPFLAGS: %" PRIx32, pci->grp_regs[i].grpflags); + } + + idxd->u.pci = pci; + idxd->max_batches = wq_size; + +/* enable the device itself */ + err_code =idxd_pci_dev_command (idxd, idxd_enable_dev); +if (err_code) { +IOAT_PMD_ERR ("Error enabling device: code %#x" , err_code); +return err_code; + } +IOAT_PMD_DEBUG ("IDXD Device enabled OK" ); + +return nb_wqs; + +err: +free (pci); +return -1 ; +} +static int idxd_rawdev_probe_pci (struct rte_pci_driver *drv,struct rte_pci_device *dev) { -int ret =0 ; +struct idxd_rawdev idxd = {{0 }};/* Double {} to avoid error on BSD12 */ +uint8_t nb_wqs; +int qid, ret =0 ;char name[PCI_PRI_STR_SIZE ];rte_pci_device_name (&dev->addr, name,sizeof (name));IOAT_PMD_INFO ("Init %s on NUMA node %d" , name, dev->device.numa_node); dev->device.driver = &drv->driver; -return ret; + ret =init_pci_device (dev, &idxd); +if (ret <0 ) { +IOAT_PMD_ERR ("Error initializing PCI hardware" ); +return ret; + } + nb_wqs = (uint8_t )ret; + +/* set up one device for each queue */ +for (qid =0 ; qid < nb_wqs; qid++) { +char qname[32 ]; + +/* add the queue number to each device name */ +snprintf (qname,sizeof (qname),"%s-q%d" , name, qid); + idxd.qid = qid; + idxd.public.portal =RTE_PTR_ADD (idxd.u.pci->portals, + qid *IDXD_PORTAL_SIZE ); +if (idxd_is_wq_enabled (&idxd)) +IOAT_PMD_ERR ("Error, WQ %u seems enabled" , qid); + ret =idxd_rawdev_create (qname, &dev->device, + &idxd, &idxd_pci_ops); +if (ret !=0 ) { +IOAT_PMD_ERR ("Failed to create rawdev %s" , name); +if (qid ==0 )/* if no devices using this, free pci */ +free (idxd.u.pci); +return ret; + } + } + +return 0 ; +} + +static int +idxd_rawdev_destroy (const char *name) +{ +int ret; +uint8_t err_code; +struct rte_rawdev *rdev; +struct idxd_rawdev *idxd; + +if (!name) { +IOAT_PMD_ERR ("Invalid device name" ); +return -EINVAL ; + } + + rdev =rte_rawdev_pmd_get_named_dev (name); +if (!rdev) { +IOAT_PMD_ERR ("Invalid device name (%s)" , name); +return -EINVAL ; + } + + idxd = rdev->dev_private; + +/* disable the device */ + err_code =idxd_pci_dev_command (idxd, idxd_disable_dev); +if (err_code) { +IOAT_PMD_ERR ("Error disabling device: code %#x" , err_code); +return err_code; + } +IOAT_PMD_DEBUG ("IDXD Device disabled OK" ); + +/* free device memory */ +if (rdev->dev_private !=NULL ) { +IOAT_PMD_DEBUG ("Freeing device driver memory" ); + rdev->dev_private =NULL ; +rte_free (idxd->public.batch_ring); +rte_free (idxd->public.hdl_ring); +rte_memzone_free (idxd->mz); + } + +/* rte_rawdev_close is called by pmd_release */ + ret =rte_rawdev_pmd_release (rdev); +if (ret) +IOAT_PMD_DEBUG ("Device cleanup failed" ); + +return 0 ; } static int
@@ -40,6 +271,8 @@ idxd_rawdev_remove_pci(struct rte_pci_device *dev)IOAT_PMD_INFO ("Closing %s on NUMA node %d" , name, dev->device.numa_node); + ret =idxd_rawdev_destroy (name); +return ret; }
drivers/raw/ioat/ioat_common.c new file +68 -0
68 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h +33 -0
@@ -14,6 +14,10 @@* @b EXPERIMENTAL: these structures and APIs may change without prior notice */ +#include <rte_spinlock.h> +#include <rte_rawdev_pmd.h> +#include "rte_ioat_rawdev.h" +extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
@@ -24,4 +28,33 @@ extern int ioat_pmd_logtype; #define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) #define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) +struct idxd_pci_common { +rte_spinlock_t lk; +volatile struct rte_idxd_bar0 *regs; +volatile struct rte_idxd_wqcfg *wq_regs; +volatile struct rte_idxd_grpcfg *grp_regs; +volatile void *portals; +}; + +struct idxd_rawdev { +struct rte_idxd_rawdev public;/* the public members, must be first */ + +struct rte_rawdev *rawdev; +const struct rte_memzone *mz; +uint8_t qid; +uint16_t max_batches; + +union { +struct idxd_pci_common *pci; + } u; +}; + +extern int idxd_rawdev_create (const char *name,struct rte_device *dev, +const struct idxd_rawdev *idxd, +const struct rte_rawdev_ops *ops); + +extern int idxd_rawdev_close (struct rte_rawdev *dev); + +extern int idxd_rawdev_test (uint16_t dev_id); +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev_test.c +7 -0
@@ -7,6 +7,7 @@#include <rte_mbuf.h> #include "rte_rawdev.h" #include "rte_ioat_rawdev.h" +#include "ioat_private.h" #define MAX_SUPPORTED_RAWDEVS 64#define TEST_SKIPPED 77
@@ -267,3 +268,9 @@ ioat_rawdev_test(uint16_t dev_id)free (ids);return -1 ; } + +int +idxd_rawdev_test(uint16_t dev_id __rte_unused) +{ +return 0 ; +}
drivers/raw/ioat/ioat_spec.h +64 -0
70 lines not on this page · raw message
drivers/raw/ioat/meson.build +1 -0
@@ -6,6 +6,7 @@ reason = 'only supported on x86' sources = files( 'idxd_pci.c', 'idxd_vdev.c', + 'ioat_common.c', 'ioat_rawdev.c', 'ioat_rawdev_test.c') deps += ['bus_pci',
drivers/raw/ioat/rte_ioat_rawdev_fns.h +34 -1
@@ -41,9 +41,20 @@ struct rte_ioat_generic_hw_desc {/** * @internal -* Structure representing a device instance +* Identify the data path to use. +* Must be first field of rte_ioat_rawdev and rte_idxd_rawdev structs +*/ +enum rte_ioat_dev_type { +RTE_IOAT_DEV , +RTE_IDXD_DEV , +}; + +/** +* @internal +* Structure representing an IOAT device instance */ struct rte_ioat_rawdev { +enum rte_ioat_dev_type type;struct rte_rawdev *rawdev;const struct rte_memzone *mz;const struct rte_memzone *desc_mz;
@@ -79,6 +90,28 @@ struct rte_ioat_rawdev {#define RTE_IOAT_CHANSTS_HALTED 0x3#define RTE_IOAT_CHANSTS_ARMED 0x4 +/** +* @internal +* Structure representing an IDXD device instance +*/ +struct rte_idxd_rawdev { +enum rte_ioat_dev_type type; +void *portal;/* address to write the batch descriptor */ + +/* counters to track the batches and the individual op handles */ +uint16_t batch_ring_sz;/* size of batch ring */ +uint16_t hdl_ring_sz;/* size of the user hdl ring */ + +uint16_t next_batch;/* where we write descriptor ops */ +uint16_t next_completed;/* batch where we read completions */ +uint16_t next_ret_hdl;/* the next user hdl to return */ +uint16_t last_completed_hdl;/* the last user hdl that has completed */ +uint16_t next_free_hdl;/* where the handle for next op will go */ + +struct rte_idxd_user_hdl *hdl_ring; +struct rte_idxd_desc_batch *batch_ring; +}; +/* * Enqueue a copy operation onto the ioat device */
Signature · 1 line
2.25.1
PATCH v5 15/25 raw/ioat: create rawdev instances for idxd vdevs
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
For each vdev (DSA work queue) instance, create a rawdev instance.
--- drivers/raw/ioat/idxd_vdev.c | 106 +++++++++++++++++++++++++++++++- drivers/raw/ioat/ioat_private.h | 4 ++ 2 files changed, 109 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_vdev.c +105 -1
@@ -2,6 +2,12 @@* Copyright(c) 2020 Intel Corporation */ +#include <fcntl.h> +#include <unistd.h> +#include <limits.h> +#include <sys/mman.h> + +#include <rte_memzone.h> #include <rte_bus_vdev.h> #include <rte_kvargs.h> #include <rte_string_fns.h>
@@ -24,6 +30,36 @@ struct idxd_vdev_args {uint8_t wq_id; }; +static const struct rte_rawdev_ops idxd_vdev_ops = { + .dev_close = idxd_rawdev_close, + .dev_selftest = idxd_rawdev_test, +}; + +static void * +idxd_vdev_mmap_wq (struct idxd_vdev_args *args) +{ +void *addr; +char path[PATH_MAX ]; +int fd; + +snprintf (path,sizeof (path),"/dev/dsa/wq%u.%u" , + args->device_id, args->wq_id); + fd =open (path,O_RDWR ); +if (fd <0 ) { +IOAT_PMD_ERR ("Failed to open device path" ); +return NULL ; + } + + addr =mmap (NULL ,0x1000 ,PROT_WRITE ,MAP_SHARED , fd,0 ); +close (fd); +if (addr ==MAP_FAILED ) { +IOAT_PMD_ERR ("Failed to mmap device" ); +return NULL ; + } + +return addr; +} +static int idxd_rawdev_parse_wq(const char *key __rte_unused, const char *value,void *extra_args)
@@ -70,10 +106,32 @@ idxd_vdev_parse_params(struct rte_kvargs *kvlist, struct idxd_vdev_args *args)return -EINVAL ; } +static int +idxd_vdev_get_max_batches (struct idxd_vdev_args *args) +{ +char sysfs_path[PATH_MAX ]; +FILE *f; +int ret; + +snprintf (sysfs_path,sizeof (sysfs_path), +"/sys/bus/dsa/devices/wq%u.%u/size" , + args->device_id, args->wq_id); + f =fopen (sysfs_path,"r" ); +if (f ==NULL ) +return -1 ; + +if (fscanf (f,"%d" , &ret) !=1 ) + ret =-1 ; + +fclose (f); +return ret; +} +static int idxd_rawdev_probe_vdev (struct rte_vdev_device *vdev) {struct rte_kvargs *kvlist; +struct idxd_rawdev idxd = {{0 }};/* double {} to avoid error on BSD12 */ struct idxd_vdev_args vdev_args;const char *name;int ret =0 ;
@@ -96,13 +154,32 @@ idxd_rawdev_probe_vdev(struct rte_vdev_device *vdev)return -EINVAL ; } + idxd.qid = vdev_args.wq_id; + idxd.u.vdev.dsa_id = vdev_args.device_id; + idxd.max_batches =idxd_vdev_get_max_batches (&vdev_args); + + idxd.public.portal =idxd_vdev_mmap_wq (&vdev_args); +if (idxd.public.portal ==NULL ) { +IOAT_PMD_ERR ("WQ mmap failed" ); +return -ENOENT ; + } + + ret =idxd_rawdev_create (name, &vdev->device, &idxd, &idxd_vdev_ops); +if (ret) { +IOAT_PMD_ERR ("Failed to create rawdev %s" , name); +return ret; + } +return 0 ; }static int idxd_rawdev_remove_vdev (struct rte_vdev_device *vdev) { +struct idxd_rawdev *idxd;const char *name; +struct rte_rawdev *rdev; +int ret =0 ; name =rte_vdev_device_name (vdev);if (name ==NULL )
@@ -110,7 +187,34 @@ idxd_rawdev_remove_vdev(struct rte_vdev_device *vdev)IOAT_PMD_INFO ("Remove DSA vdev %p" , name); -return 0 ; + rdev =rte_rawdev_pmd_get_named_dev (name); +if (!rdev) { +IOAT_PMD_ERR ("Invalid device name (%s)" , name); +return -EINVAL ; + } + + idxd = rdev->dev_private; + +/* free context and memory */ +if (rdev->dev_private !=NULL ) { +IOAT_PMD_DEBUG ("Freeing device driver memory" ); + rdev->dev_private =NULL ; + +if (munmap (idxd->public.portal,0x1000 ) <0 ) { +IOAT_PMD_ERR ("Error unmapping portal" ); + ret = -errno; + } + +rte_free (idxd->public.batch_ring); +rte_free (idxd->public.hdl_ring); + +rte_memzone_free (idxd->mz); + } + +if (rte_rawdev_pmd_release (rdev)) +IOAT_PMD_ERR ("Device cleanup failed" ); + +return ret; }struct rte_vdev_driver idxd_rawdev_drv_vdev = {
drivers/raw/ioat/ioat_private.h +4 -0
@@ -45,6 +45,10 @@ struct idxd_rawdev {uint16_t max_batches;union { +struct { +unsigned int dsa_id; + } vdev; +struct idxd_pci_common *pci; } u; };
Signature · 1 line
2.25.1
PATCH v5 16/25 raw/ioat: add datapath data structures for idxd devices
In reply to Bruce RichardsonAdd in the relevant data structures for the data path for DSA devices. Also include a device dump function to output the status of each device.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 34 +++++++++++ drivers/raw/ioat/ioat_private.h | 2 + drivers/raw/ioat/ioat_rawdev_test.c | 3 +- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 80 ++++++++++++++++++++++++++ 6 files changed, 120 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -54,6 +54,7 @@ idxd_is_wq_enabled(struct idxd_rawdev *idxd)static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, + .dump = idxd_dev_dump, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -33,6 +33,7 @@ struct idxd_vdev_args {static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, + .dump = idxd_dev_dump, }; static void *
drivers/raw/ioat/ioat_common.c +34 -0
@@ -14,6 +14,36 @@ idxd_rawdev_close(struct rte_rawdev *dev __rte_unused)return 0 ; } +int +idxd_dev_dump (struct rte_rawdev *dev,FILE *f) +{ +struct idxd_rawdev *idxd = dev->dev_private; +struct rte_idxd_rawdev *rte_idxd = &idxd->public; +int i; + +fprintf (f,"Raw Device #%d\n" , dev->dev_id); +fprintf (f,"Driver: %s\n\n" , dev->driver_name); + +fprintf (f,"Portal: %p\n" , rte_idxd->portal); +fprintf (f,"Batch Ring size: %u\n" , rte_idxd->batch_ring_sz); +fprintf (f,"Comp Handle Ring size: %u\n\n" , rte_idxd->hdl_ring_sz); + +fprintf (f,"Next batch: %u\n" , rte_idxd->next_batch); +fprintf (f,"Next batch to be completed: %u\n" , rte_idxd->next_completed); +for (i =0 ; i < rte_idxd->batch_ring_sz; i++) { +struct rte_idxd_desc_batch *b = &rte_idxd->batch_ring[i]; +fprintf (f,"Batch %u @%p: submitted=%u, op_count=%u, hdl_end=%u\n" , + i, b, b->submitted, b->op_count, b->hdl_end); + } + +fprintf (f,"\n" ); +fprintf (f,"Next free hdl: %u\n" , rte_idxd->next_free_hdl); +fprintf (f,"Last completed hdl: %u\n" , rte_idxd->last_completed_hdl); +fprintf (f,"Next returned hdl: %u\n" , rte_idxd->next_ret_hdl); + +return 0 ; +} + int idxd_rawdev_create(const char *name, struct rte_device *dev, const struct idxd_rawdev *base_idxd,
@@ -25,6 +55,10 @@ idxd_rawdev_create(const char *name, struct rte_device *dev,char mz_name[RTE_MEMZONE_NAMESIZE ];int ret =0 ; +RTE_BUILD_BUG_ON (sizeof (struct rte_idxd_hw_desc ) !=64 ); +RTE_BUILD_BUG_ON (offsetof(struct rte_idxd_hw_desc , size) !=32 ); +RTE_BUILD_BUG_ON (sizeof (struct rte_idxd_completion ) !=32 ); +if (!name) {IOAT_PMD_ERR ("Invalid name of the device!" ); ret = -EINVAL ;
drivers/raw/ioat/ioat_private.h +2 -0
@@ -61,4 +61,6 @@ extern int idxd_rawdev_close(struct rte_rawdev *dev);extern int idxd_rawdev_test (uint16_t dev_id); +extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f); +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev_test.c +2 -1
@@ -270,7 +270,8 @@ ioat_rawdev_test(uint16_t dev_id) }int -idxd_rawdev_test(uint16_t dev_id __rte_unused) +idxd_rawdev_test (uint16_t dev_id) { +rte_rawdev_dump (dev_id, stdout);return 0 ; }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +80 -0
86 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v5 17/25 raw/ioat: add configure function for idxd devices
In reply to Bruce RichardsonAdd configure function for idxd devices, taking the same parameters as the existing configure function for ioat. The ring_size parameter is used to compute the maximum number of bursts to be supported by the driver, given that the hardware works on individual bursts of descriptors at a time.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 64 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 3 ++ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 1 + 5 files changed, 70 insertions(+)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -55,6 +55,7 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, + .dev_configure = idxd_dev_configure, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -34,6 +34,7 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, + .dev_configure = idxd_dev_configure, }; static void *
drivers/raw/ioat/ioat_common.c +64 -0
70 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h +3 -0
@@ -59,6 +59,9 @@ extern int idxd_rawdev_create(const char *name, struct rte_device *dev,extern int idxd_rawdev_close (struct rte_rawdev *dev); +extern int idxd_dev_configure (const struct rte_rawdev *dev, +rte_rawdev_obj_t config,size_t config_size); +extern int idxd_rawdev_test (uint16_t dev_id);extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);
drivers/raw/ioat/rte_ioat_rawdev_fns.h +1 -0
@@ -187,6 +187,7 @@ struct rte_idxd_rawdev {uint16_t next_ret_hdl;/* the next user hdl to return */ uint16_t last_completed_hdl;/* the last user hdl that has completed */ uint16_t next_free_hdl;/* where the handle for next op will go */ +uint16_t hdls_disable;/* disable tracking completion handles */ struct rte_idxd_user_hdl *hdl_ring;struct rte_idxd_desc_batch *batch_ring;
Signature · 1 line
2.25.1
PATCH v5 18/25 raw/ioat: add start and stop functions for idxd devices
In reply to Bruce RichardsonAdd the start and stop functions for DSA hardware devices using the vfio/uio kernel drivers. For vdevs using the idxd kernel driver, the device must be started using sysfs before the device node appears for vdev use - making start/stop functions in the driver unnecessary.
--- drivers/raw/ioat/idxd_pci.c | 50 +++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
drivers/raw/ioat/idxd_pci.c +50 -0
@@ -51,11 +51,61 @@ idxd_is_wq_enabled(struct idxd_rawdev *idxd)return ((state >>WQ_STATE_SHIFT ) &WQ_STATE_MASK ) ==0x1 ; } +static void +idxd_pci_dev_stop (struct rte_rawdev *dev) +{ +struct idxd_rawdev *idxd = dev->dev_private; +uint8_t err_code; + +if (!idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Work queue %d already disabled" , idxd->qid); +return ; + } + + err_code =idxd_pci_dev_command (idxd, idxd_disable_wq); +if (err_code ||idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Failed disabling work queue %d, error code: %#x" , + idxd->qid, err_code); +return ; + } +IOAT_PMD_DEBUG ("Work queue %d disabled OK" , idxd->qid); +} + +static int +idxd_pci_dev_start (struct rte_rawdev *dev) +{ +struct idxd_rawdev *idxd = dev->dev_private; +uint8_t err_code; + +if (idxd_is_wq_enabled (idxd)) { +IOAT_PMD_WARN ("WQ %d already enabled" , idxd->qid); +return 0 ; + } + +if (idxd->public.batch_ring ==NULL ) { +IOAT_PMD_ERR ("WQ %d has not been fully configured" , idxd->qid); +return -EINVAL ; + } + + err_code =idxd_pci_dev_command (idxd, idxd_enable_wq); +if (err_code || !idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Failed enabling work queue %d, error code: %#x" , + idxd->qid, err_code); +return err_code ==0 ?-1 : err_code; + } + +IOAT_PMD_DEBUG ("Work queue %d enabled OK" , idxd->qid); + +return 0 ; +} +static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, + .dev_start = idxd_pci_dev_start, + .dev_stop = idxd_pci_dev_stop, };/* each portal uses 4 x 4k pages */
Signature · 1 line
2.25.1
PATCH v5 19/25 raw/ioat: add data path for idxd devices
In reply to Bruce RichardsonAdd support for doing copies using DSA hardware. This is implemented by just switching on the device type field at the start of the inline functions. Since there is no hardware which will have both device types present this branch will always be predictable after the first call, meaning it has little to no perf penalty.
--- drivers/raw/ioat/ioat_common.c | 1 + drivers/raw/ioat/ioat_rawdev.c | 1 + drivers/raw/ioat/rte_ioat_rawdev_fns.h | 201 +++++++++++++++++++++++-- 3 files changed, 192 insertions(+), 11 deletions(-)
drivers/raw/ioat/ioat_common.c +1 -0
@@ -153,6 +153,7 @@ idxd_rawdev_create(const char *name, struct rte_device *dev, idxd = rawdev->dev_private; *idxd = *base_idxd;/* copy over the main fields already passed in */ + idxd->public.type =RTE_IDXD_DEV ; idxd->rawdev = rawdev; idxd->mz = mz;
drivers/raw/ioat/ioat_rawdev.c +1 -0
@@ -260,6 +260,7 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) rawdev->driver_name = dev->device.driver->name; ioat = rawdev->dev_private; + ioat->type =RTE_IOAT_DEV ; ioat->rawdev = rawdev; ioat->mz = mz; ioat->regs = dev->mem_resource[0 ].addr;
drivers/raw/ioat/rte_ioat_rawdev_fns.h +190 -11
243 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v5 20/25 raw/ioat: add info function for idxd devices
In reply to Bruce RichardsonAdd the info get function for DSA devices, returning just the ring size info about the device, same as is returned for existing IOAT/CBDMA devices.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 18 ++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 3 +++ 4 files changed, 23 insertions(+)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -106,6 +106,7 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_configure = idxd_dev_configure, .dev_start = idxd_pci_dev_start, .dev_stop = idxd_pci_dev_stop, + .dev_info_get = idxd_dev_info_get, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -35,6 +35,7 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, + .dev_info_get = idxd_dev_info_get, }; static void *
drivers/raw/ioat/ioat_common.c +18 -0
@@ -44,6 +44,24 @@ idxd_dev_dump(struct rte_rawdev *dev, FILE *f)return 0 ; } +int +idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info, +size_t info_size) +{ +struct rte_ioat_rawdev_config *cfg = dev_info; +struct idxd_rawdev *idxd = dev->dev_private; +struct rte_idxd_rawdev *rte_idxd = &idxd->public; + +if (info_size !=sizeof (*cfg)) +return -EINVAL ; + +if (cfg !=NULL ) { + cfg->ring_size = rte_idxd->hdl_ring_sz; + cfg->hdls_disable = rte_idxd->hdls_disable; + } +return 0 ; +} +int idxd_dev_configure (const struct rte_rawdev *dev,rte_rawdev_obj_t config,size_t config_size)
drivers/raw/ioat/ioat_private.h +3 -0
@@ -62,6 +62,9 @@ extern int idxd_rawdev_close(struct rte_rawdev *dev);extern int idxd_dev_configure (const struct rte_rawdev *dev,rte_rawdev_obj_t config,size_t config_size); +extern int idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info, +size_t info_size); +extern int idxd_rawdev_test (uint16_t dev_id);extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);
Signature · 1 line
2.25.1
PATCH v5 21/25 raw/ioat: create separate statistics structure
In reply to Bruce RichardsonRather than having the xstats as fields inside the main driver structure, create a separate structure type for them.
As part of the change, when updating the stats functions referring to the stats by the old path, we can simplify them to use the id to directly index into the stats structure, making the code shorter and simpler.
--- drivers/raw/ioat/ioat_rawdev.c | 40 +++++++------------------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 30 ++++++++++++------- 2 files changed, 29 insertions(+), 41 deletions(-)
drivers/raw/ioat/ioat_rawdev.c +10 -30
@@ -132,16 +132,14 @@ ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[], uint64_t values[], unsigned int n) {const struct rte_ioat_rawdev *ioat = dev->dev_private; +const uint64_t *stats = (const void *)&ioat->xstats;unsigned int i;for (i =0 ; i < n; i++) { -switch (ids[i]) { -case 0 : values[i] = ioat->enqueue_failed;break ; -case 1 : values[i] = ioat->enqueued;break ; -case 2 : values[i] = ioat->started;break ; -case 3 : values[i] = ioat->completed;break ; -default : values[i] =0 ;break ; - } +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + values[i] = stats[ids[i]]; +else + values[i] =0 ; }return n; }
@@ -167,35 +165,17 @@ static int ioat_xstats_reset(struct rte_rawdev *dev, const uint32_t *ids, uint32_t nb_ids) { struct rte_ioat_rawdev *ioat = dev->dev_private; +uint64_t *stats = (void *)&ioat->xstats;unsigned int i;if (!ids) { - ioat->enqueue_failed =0 ; - ioat->enqueued =0 ; - ioat->started =0 ; - ioat->completed =0 ; +memset (&ioat->xstats,0 ,sizeof (ioat->xstats));return 0 ; } -for (i =0 ; i < nb_ids; i++) { -switch (ids[i]) { -case 0 : - ioat->enqueue_failed =0 ; -break ; -case 1 : - ioat->enqueued =0 ; -break ; -case 2 : - ioat->started =0 ; -break ; -case 3 : - ioat->completed =0 ; -break ; -default : -IOAT_PMD_WARN ("Invalid xstat id - cannot reset value" ); -break ; - } - } +for (i =0 ; i < nb_ids; i++) +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + stats[ids[i]] =0 ;return 0 ; }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +19 -11
@@ -49,17 +49,31 @@ enum rte_ioat_dev_type { RTE_IDXD_DEV, }; +/** +* @internal +* some statistics for tracking, if added/changed update xstats fns +*/ +struct rte_ioat_xstats { +uint64_t enqueue_failed; +uint64_t enqueued; +uint64_t started; +uint64_t completed; +}; +/** * @internal * Structure representing an IOAT device instance */ struct rte_ioat_rawdev { +/* common fields at the top - match those in rte_idxd_rawdev */ enum rte_ioat_dev_type type; +struct rte_ioat_xstats xstats; +struct rte_rawdev *rawdev;const struct rte_memzone *mz;const struct rte_memzone *desc_mz; -volatile uint16_t *doorbell; + volatile uint16_t *doorbell __rte_cache_aligned;phys_addr_t status_addr;phys_addr_t ring_addr;
@@ -72,12 +86,6 @@ struct rte_ioat_rawdev {unsigned short next_read;unsigned short next_write; -/* some statistics for tracking, if added/changed update xstats fns*/ - uint64_t enqueue_failed __rte_cache_aligned; -uint64_t enqueued; -uint64_t started; -uint64_t completed; -/* to report completions, the device will write status back here */ volatile uint64_t status __rte_cache_aligned;
@@ -209,7 +217,7 @@ __ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,struct rte_ioat_generic_hw_desc *desc;if (space ==0 ) { - ioat->enqueue_failed++; + ioat->xstats.enqueue_failed++;return 0 ; }
@@ -228,7 +236,7 @@ __ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, (int64_t)src_hdl);rte_prefetch0 (&ioat->desc_ring[ioat->next_write & mask]); - ioat->enqueued++; + ioat->xstats.enqueued++;return 1 ; }
@@ -261,7 +269,7 @@ __ioat_perform_ops(int dev_id) .control.completion_update = 1;rte_compiler_barrier (); *ioat->doorbell = ioat->next_write; - ioat->started = ioat->enqueued; + ioat->xstats.started = ioat->xstats.enqueued; }/**
@@ -328,7 +336,7 @@ __ioat_completed_ops(int dev_id, uint8_t max_copies, end: ioat->next_read = read; - ioat->completed += count; + ioat->xstats.completed += count;return count; }
Signature · 1 line
2.25.1
PATCH v5 22/25 raw/ioat: move xstats functions to common file
In reply to Bruce RichardsonThe xstats functions can be used by all ioat devices so move them from the ioat_rawdev.c file to ioat_common.c, and add the function prototypes to the internal header file.
--- drivers/raw/ioat/ioat_common.c | 59 +++++++++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 10 ++++++ drivers/raw/ioat/ioat_rawdev.c | 58 -------------------------------- 3 files changed, 69 insertions(+), 58 deletions(-)
drivers/raw/ioat/ioat_common.c +59 -0
@@ -5,9 +5,68 @@#include <rte_rawdev_pmd.h> #include <rte_memzone.h> #include <rte_common.h> +#include <rte_string_fns.h> #include "ioat_private.h" +static const char *const xstat_names[] = { +"failed_enqueues" ,"successful_enqueues" , +"copies_started" ,"copies_completed" +}; + +int +ioat_xstats_get (const struct rte_rawdev *dev,const unsigned int ids[], +uint64_t values[],unsigned int n) +{ +const struct rte_ioat_rawdev *ioat = dev->dev_private; +const uint64_t *stats = (const void *)&ioat->xstats; +unsigned int i; + +for (i =0 ; i < n; i++) { +if (ids[i] >sizeof (ioat->xstats)/sizeof (*stats)) + values[i] =0 ; +else + values[i] = stats[ids[i]]; + } +return n; +} + +int +ioat_xstats_get_names (const struct rte_rawdev *dev, +struct rte_rawdev_xstats_name *names, +unsigned int size) +{ +unsigned int i; + +RTE_SET_USED (dev); +if (size <RTE_DIM (xstat_names)) +return RTE_DIM (xstat_names); + +for (i =0 ; i <RTE_DIM (xstat_names); i++) +strlcpy (names[i].name, xstat_names[i],sizeof (names[i])); + +return RTE_DIM (xstat_names); +} + +int +ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids,uint32_t nb_ids) +{ +struct rte_ioat_rawdev *ioat = dev->dev_private; +uint64_t *stats = (void *)&ioat->xstats; +unsigned int i; + +if (!ids) { +memset (&ioat->xstats,0 ,sizeof (ioat->xstats)); +return 0 ; + } + +for (i =0 ; i < nb_ids; i++) +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + stats[ids[i]] =0 ; + +return 0 ; +} +int idxd_rawdev_close(struct rte_rawdev *dev __rte_unused) {
drivers/raw/ioat/ioat_private.h +10 -0
@@ -53,6 +53,16 @@ struct idxd_rawdev { } u; }; +int ioat_xstats_get (const struct rte_rawdev *dev,const unsigned int ids[], +uint64_t values[],unsigned int n); + +int ioat_xstats_get_names (const struct rte_rawdev *dev, +struct rte_rawdev_xstats_name *names, +unsigned int size); + +int ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids, +uint32_t nb_ids); +extern int idxd_rawdev_create (const char *name,struct rte_device *dev,const struct idxd_rawdev *idxd,const struct rte_rawdev_ops *ops);
drivers/raw/ioat/ioat_rawdev.c +0 -58
64 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v5 23/25 raw/ioat: add xstats tracking for idxd devices
In reply to Bruce RichardsonAdd update of the relevant stats for the data path functions and point the overall device struct xstats function pointers to the existing ioat functions.
At this point, all necessary hooks for supporting the existing unit tests are in place so call them for each device.
--- drivers/raw/ioat/idxd_pci.c | 3 +++ drivers/raw/ioat/idxd_vdev.c | 3 +++ drivers/raw/ioat/ioat_rawdev_test.c | 2 +- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 6 ++++++ 4 files changed, 13 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_pci.c +3 -0
@@ -107,6 +107,9 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_start = idxd_pci_dev_start, .dev_stop = idxd_pci_dev_stop, .dev_info_get = idxd_dev_info_get, + .xstats_get = ioat_xstats_get, + .xstats_get_names = ioat_xstats_get_names, + .xstats_reset = ioat_xstats_reset, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +3 -0
@@ -36,6 +36,9 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_info_get = idxd_dev_info_get, + .xstats_get = ioat_xstats_get, + .xstats_get_names = ioat_xstats_get_names, + .xstats_reset = ioat_xstats_reset, }; static void *
drivers/raw/ioat/ioat_rawdev_test.c +1 -1
@@ -273,5 +273,5 @@ int idxd_rawdev_test(uint16_t dev_id) {rte_rawdev_dump (dev_id, stdout); -return 0 ; +return ioat_rawdev_test (dev_id); }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +6 -0
@@ -184,6 +184,8 @@ struct rte_idxd_user_hdl {*/ struct rte_idxd_rawdev {enum rte_ioat_dev_type type; +struct rte_ioat_xstats xstats; +void *portal;/* address to write the batch descriptor */ /* counters to track the batches and the individual op handles */
@@ -369,9 +371,11 @@ __idxd_write_desc(int dev_id, const struct rte_idxd_hw_desc *desc,if (++idxd->next_free_hdl == idxd->hdl_ring_sz) idxd->next_free_hdl =0 ; + idxd->xstats.enqueued++;return 1 ; failed: + idxd->xstats.enqueue_failed++; rte_errno =ENOSPC ;return 0 ; }
@@ -429,6 +433,7 @@ __idxd_perform_ops(int dev_id)if (++idxd->next_batch == idxd->batch_ring_sz) idxd->next_batch =0 ; + idxd->xstats.started = idxd->xstats.enqueued; }static __rte_always_inline int
@@ -466,6 +471,7 @@ __idxd_completed_ops(int dev_id, uint8_t max_ops, idxd->next_ret_hdl = h_idx; + idxd->xstats.completed += n;return n; }
Signature · 1 line
2.25.1
PATCH v5 24/25 raw/ioat: clean up use of common test function
In reply to Bruce RichardsonNow that all devices can pass the same set of unit tests, eliminate the temporary idxd_rawdev_test function and move the prototype for ioat_rawdev_test to the proper internal header file, to be used by all device instances.
--- drivers/raw/ioat/idxd_pci.c | 2 +- drivers/raw/ioat/idxd_vdev.c | 2 +- drivers/raw/ioat/ioat_private.h | 4 ++-- drivers/raw/ioat/ioat_rawdev.c | 2 -- drivers/raw/ioat/ioat_rawdev_test.c | 7 ------- 5 files changed, 4 insertions(+), 13 deletions(-)
drivers/raw/ioat/idxd_pci.c +1 -1
@@ -101,7 +101,7 @@ idxd_pci_dev_start(struct rte_rawdev *dev)static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, - .dev_selftest = idxd_rawdev_test, + .dev_selftest = ioat_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_start = idxd_pci_dev_start,
drivers/raw/ioat/idxd_vdev.c +1 -1
@@ -32,7 +32,7 @@ struct idxd_vdev_args {static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, - .dev_selftest = idxd_rawdev_test, + .dev_selftest = ioat_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_info_get = idxd_dev_info_get,
drivers/raw/ioat/ioat_private.h +2 -2
@@ -63,6 +63,8 @@ int ioat_xstats_get_names(const struct rte_rawdev *dev,int ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids,uint32_t nb_ids); +extern int ioat_rawdev_test (uint16_t dev_id); +extern int idxd_rawdev_create (const char *name,struct rte_device *dev,const struct idxd_rawdev *idxd,const struct rte_rawdev_ops *ops);
@@ -75,8 +77,6 @@ extern int idxd_dev_configure(const struct rte_rawdev *dev,extern int idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info,size_t info_size); -extern int idxd_rawdev_test (uint16_t dev_id); -extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev.c +0 -2
@@ -128,8 +128,6 @@ ioat_dev_close(struct rte_rawdev *dev __rte_unused)return 0 ; } -extern int ioat_rawdev_test (uint16_t dev_id); -static int ioat_rawdev_create (const char *name,struct rte_pci_device *dev) {
drivers/raw/ioat/ioat_rawdev_test.c +0 -7
@@ -268,10 +268,3 @@ ioat_rawdev_test(uint16_t dev_id)free (ids);return -1 ; } - -int -idxd_rawdev_test (uint16_t dev_id) -{ -rte_rawdev_dump (dev_id, stdout); -return ioat_rawdev_test (dev_id); -}
Signature · 1 line
2.25.1
PATCH v5 25/25 raw/ioat: add fill operation
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
Add fill operation enqueue support for IOAT and IDXD. The fill enqueue is similar to the copy enqueue, but takes a 'pattern' rather than a source address to transfer to the destination address. This patch also includes an additional test case for the new operation type.
--- doc/guides/rawdevs/ioat.rst | 10 ++++ doc/guides/rel_notes/release_20_11.rst | 2 + drivers/raw/ioat/ioat_rawdev_test.c | 62 ++++++++++++++++++++++++ drivers/raw/ioat/rte_ioat_rawdev.h | 26 +++++++++++ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 65 ++++++++++++++++++++++++-- 5 files changed, 160 insertions(+), 5 deletions(-)
doc/guides/rawdevs/ioat.rst +10 -0
@@ -285,6 +285,16 @@ is correct before freeing the data buffers using the returned handles: } +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The IOAT driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_ioat_enqueue_fill()`` function rather +than the ``rte_ioat_enqueue_copy()`` function. + + Querying Device Statistics ~~~~~~~~~~~~~~~~~~~~~~~~~~~
doc/guides/rel_notes/release_20_11.rst +2 -0
@@ -122,6 +122,8 @@ New Features * Added support for Intel\ |reg| Data Streaming Accelerator hardware. For more information, see https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator + * Added support for the fill operation via the API ``rte_ioat_enqueue_fill()``, + where the hardware fills an area of memory with a repeating pattern. * Added a per-device configuration flag to disable management of user-provided completion handles * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()``
drivers/raw/ioat/ioat_rawdev_test.c +62 -0
80 lines not on this page · raw message
drivers/raw/ioat/rte_ioat_rawdev.h +26 -0
@@ -37,6 +37,32 @@ struct rte_ioat_rawdev_config {bool hdls_disable;/**< if set, ignore user-supplied handle params */ }; +/** +* Enqueue a fill operation onto the ioat device +* +* This queues up a fill operation to be performed by hardware, but does not +* trigger hardware to begin that operation. +* +* @param dev_id +* The rawdev device id of the ioat instance +* @param pattern +* The pattern to populate the destination buffer with +* @param dst +* The physical address of the destination buffer +* @param length +* The length of the destination buffer +* @param dst_hdl +* An opaque handle for the destination data, to be returned when this +* operation has been completed and the user polls for the completion details. +* NOTE: If hdls_disable configuration option for the device is set, this +* parameter is ignored. +* @return +* Number of operations enqueued, either 0 or 1 +*/ +static inline int +rte_ioat_enqueue_fill (int dev_id,uint64_t pattern,phys_addr_t dst, +unsigned int length,uintptr_t dst_hdl); +/** * Enqueue a copy operation onto the ioat device *
drivers/raw/ioat/rte_ioat_rawdev_fns.h +60 -5
102 lines not on this page · raw message
Signature · 1 line
2.25.1
v6 cover raw/ioat: enhancements and new hardware support
In reply to Bruce RichardsonThis patchset adds some small enhancements, some rework and also support for new hardware to the ioat rawdev driver. Most rework and enhancements are largely self-explanatory from the individual patches.
The new hardware support is for the Intel(R) DSA accelerator which will be present in future Intel processors. A description of this new hardware is covered in [1]. Functions specific to the new hardware use the "idxd" prefix, for consistency with the kernel driver.
[1] https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
---
V6:
* Add explicit __rte_experimental tag on all functions. [Previously header just had note saying all contents are experimental]
V5:
* Rebased to latest main branch.
V4:
* Fixed compile with FreeBSD clang
* Improved autotests for fill operation
V3:
* More doc updates including release note updates throughout the set
* Added in fill operation
* Added in fix for missing close operation
* Added in fix for doc building to ensure ioat is in in the index
V2:
* Included documentation additions in the set
* Split off the rawdev unit test changes to a separate patchset for easier review
* General code improvements and cleanups
Bruce Richardson (19):
doc/api: add ioat driver to index
raw/ioat: enable use from C++ code
raw/ioat: include extra info in error messages raw/ioat: split header for readability raw/ioat: rename functions to be operation-agnostic raw/ioat: add separate API for fence call raw/ioat: make the HW register spec private raw/ioat: add skeleton for VFIO/UIO based DSA device raw/ioat: include example configuration script raw/ioat: create rawdev instances on idxd PCI probe raw/ioat: add datapath data structures for idxd devices raw/ioat: add configure function for idxd devices raw/ioat: add start and stop functions for idxd devices raw/ioat: add data path for idxd devices raw/ioat: add info function for idxd devices raw/ioat: create separate statistics structure raw/ioat: move xstats functions to common file raw/ioat: add xstats tracking for idxd devices raw/ioat: clean up use of common test function
Cheng Jiang (1):
raw/ioat: add a flag to control copying handle parameters
Kevin Laatz (5):
raw/ioat: fix missing close function usertools/dpdk-devbind.py: add support for DSA HW raw/ioat: add vdev probe for DSA/idxd devices raw/ioat: create rawdev instances for idxd vdevs raw/ioat: add fill operation
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
doc/guides/rawdevs/ioat.rst | 163 +++--
doc/guides/rel_notes/release_20_11.rst | 23 +
doc/guides/sample_app_ug/ioat.rst | 8 +-
drivers/raw/ioat/dpdk_idxd_cfg.py | 79 +++
drivers/raw/ioat/idxd_pci.c | 345 ++++++++++
drivers/raw/ioat/idxd_vdev.c | 233 +++++++
drivers/raw/ioat/ioat_common.c | 244 +++++++
drivers/raw/ioat/ioat_private.h | 82 +++
drivers/raw/ioat/ioat_rawdev.c | 92 +--
drivers/raw/ioat/ioat_rawdev_test.c | 130 +++-
.../raw/ioat/{rte_ioat_spec.h => ioat_spec.h} | 90 ++-
drivers/raw/ioat/meson.build | 15 +-
drivers/raw/ioat/rte_ioat_rawdev.h | 226 +++----
drivers/raw/ioat/rte_ioat_rawdev_fns.h | 595 ++++++++++++++++++
examples/ioat/ioatfwd.c | 16 +-
lib/librte_eal/include/rte_common.h | 1 +
usertools/dpdk-devbind.py | 6 +-
19 files changed, 1996 insertions(+), 354 deletions(-)
create mode 100755 drivers/raw/ioat/dpdk_idxd_cfg.py
create mode 100644 drivers/raw/ioat/idxd_pci.c
create mode 100644 drivers/raw/ioat/idxd_vdev.c
create mode 100644 drivers/raw/ioat/ioat_common.c
create mode 100644 drivers/raw/ioat/ioat_private.h
rename drivers/raw/ioat/{rte_ioat_spec.h => ioat_spec.h} (74%)
create mode 100644 drivers/raw/ioat/rte_ioat_rawdev_fns.hSignature · 1 line
2.25.1
PATCH v6 1/25 doc/api: add ioat driver to index
Add the ioat driver to the doxygen documentation.
--- doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + 2 files changed, 2 insertions(+)
doc/api/doxy-api-index.md +1 -0
@@ -44,6 +44,7 @@ The public API headers are grouped by topics: [ixgbe] (@ref rte_pmd_ixgbe.h), [i40e] (@ref rte_pmd_i40e.h), [ice] (@ref rte_pmd_ice.h), + [ioat] (@ref rte_ioat_rawdev.h), [bnxt] (@ref rte_pmd_bnxt.h), [dpaa] (@ref rte_pmd_dpaa.h), [dpaa2] (@ref rte_pmd_dpaa2.h),
doc/api/doxy-api.conf.in +1 -0
@@ -18,6 +18,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/drivers/net/softnic \ @TOPDIR@/drivers/raw/dpaa2_cmdif \ @TOPDIR@/drivers/raw/dpaa2_qdma \ + @TOPDIR@/drivers/raw/ioat \ @TOPDIR@/lib/librte_eal/include \ @TOPDIR@/lib/librte_eal/include/generic \ @TOPDIR@/lib/librte_acl \
Signature · 1 line
2.25.1
PATCH v6 2/25 raw/ioat: fix missing close function
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
When rte_rawdev_pmd_release() is called, rte_rawdev_close() looks for a dev_close function for the device causing a segmentation fault when no close() function is implemented for a driver.
This patch resolves the issue by adding a stub function ioat_dev_close().
--- drivers/raw/ioat/ioat_rawdev.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/raw/ioat/ioat_rawdev.c +7 -0
@@ -203,6 +203,12 @@ ioat_xstats_reset(struct rte_rawdev *dev, const uint32_t *ids, uint32_t nb_ids)return 0 ; } +static int +ioat_dev_close(struct rte_rawdev *dev __rte_unused) +{ +return 0 ; +} +extern int ioat_rawdev_test (uint16_t dev_id); static int
@@ -212,6 +218,7 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) .dev_configure = ioat_dev_configure, .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, + .dev_close = ioat_dev_close, .dev_info_get = ioat_dev_info_get, .xstats_get = ioat_xstats_get, .xstats_get_names = ioat_xstats_get_names,
Signature · 1 line
2.25.1
PATCH v6 3/25 raw/ioat: enable use from C++ code
In reply to Bruce RichardsonTo allow the header file to be used from C++ code we need to ensure all typecasts are explicit, and include an 'extern "C"' guard.
--- drivers/raw/ioat/rte_ioat_rawdev.h | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
drivers/raw/ioat/rte_ioat_rawdev.h +17 -6
@@ -5,6 +5,10 @@ #ifndef _RTE_IOAT_RAWDEV_H_ #define _RTE_IOAT_RAWDEV_H_ +#ifdef __cplusplus +extern "C" { +#endif + /** * @file rte_ioat_rawdev.h *
@@ -100,7 +104,8 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl, int fence) { - struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; + struct rte_ioat_rawdev *ioat = + (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;unsigned short read = ioat->next_read;unsigned short write = ioat->next_write;unsigned short mask = ioat->ring_size -1 ;
@@ -141,7 +146,8 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,static inline void rte_ioat_do_copies (int dev_id) { -struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; +struct rte_ioat_rawdev *ioat = + (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; ioat->desc_ring[(ioat->next_write -1 ) & (ioat->ring_size -1 )].u .control.completion_update =1 ;rte_compiler_barrier ();
@@ -190,7 +196,8 @@ static inline int rte_ioat_completed_copies(int dev_id, uint8_t max_copies, uintptr_t *src_hdls, uintptr_t *dst_hdls) { - struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; + struct rte_ioat_rawdev *ioat = + (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;unsigned short mask = (ioat->ring_size -1 );unsigned short read = ioat->next_read;unsigned short end_read, count;
@@ -212,13 +219,13 @@ rte_ioat_completed_copies(int dev_id, uint8_t max_copies,__m128i hdls0 =_mm_load_si128 (&ioat->hdls[read & mask]);__m128i hdls1 =_mm_load_si128 (&ioat->hdls[(read +1 ) & mask]); -_mm_storeu_si128 ((void *)&src_hdls[i], +_mm_storeu_si128 ((__m128i *)&src_hdls[i],_mm_unpacklo_epi64 (hdls0, hdls1)); -_mm_storeu_si128 ((void *)&dst_hdls[i], +_mm_storeu_si128 ((__m128i *)&dst_hdls[i],_mm_unpackhi_epi64 (hdls0, hdls1)); }for (; i < count; i++, read++) { -uintptr_t *hdls = (void *)&ioat->hdls[read & mask]; +uintptr_t *hdls = (uintptr_t *)&ioat->hdls[read & mask]; src_hdls[i] = hdls[0 ]; dst_hdls[i] = hdls[1 ]; }
@@ -228,4 +235,8 @@ rte_ioat_completed_copies(int dev_id, uint8_t max_copies,return count; } +#ifdef __cplusplus +} +#endif +#endif /* _RTE_IOAT_RAWDEV_H_ */
Signature · 1 line
2.25.1
PATCH v6 4/25 raw/ioat: include extra info in error messages
In reply to Bruce RichardsonIn case of any failures, include the function name and the line number of the error message in the message, to make tracking down the failure easier.
--- drivers/raw/ioat/ioat_rawdev_test.c | 53 +++++++++++++++++++---------- 1 file changed, 35 insertions(+), 18 deletions(-)
drivers/raw/ioat/ioat_rawdev_test.c +35 -18
@@ -16,6 +16,23 @@ int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */static struct rte_mempool *pool;static unsigned short expected_ring_size[MAX_SUPPORTED_RAWDEVS ]; +#define PRINT_ERR (...) print_err(__func__, __LINE__, __VA_ARGS__) + +static inline int +__rte_format_printf(3, 4) +print_err(const char *func, int lineno, const char *format, ...) +{ + va_list ap; +int ret; + + ret = fprintf(stderr, "In %s:%d - ", func, lineno); + va_start(ap, format); + ret += vfprintf(stderr, format, ap); +va_end (ap); + + return ret; +} + static int test_enqueue_copies(int dev_id) {
@@ -45,7 +62,7 @@ test_enqueue_copies(int dev_id) (uintptr_t)src, (uintptr_t)dst, 0 /* no fence */) != 1) { -printf ("Error with rte_ioat_enqueue_copy\n" ); +PRINT_ERR ("Error with rte_ioat_enqueue_copy\n" );return -1 ; }rte_ioat_do_copies (dev_id);
@@ -53,18 +70,18 @@ test_enqueue_copies(int dev_id)if (rte_ioat_completed_copies (dev_id,1 , (void *)&completed[0 ], (void *)&completed[1 ]) !=1 ) { -printf ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_copies\n" );return -1 ; }if (completed[0 ] != src || completed[1 ] != dst) { -printf ("Error with completions: got (%p, %p), not (%p,%p)\n" , +PRINT_ERR ("Error with completions: got (%p, %p), not (%p,%p)\n" , completed[0 ], completed[1 ], src, dst);return -1 ; }for (i =0 ; i < length; i++)if (dst_data[i] != src_data[i]) { -printf ("Data mismatch at char %u\n" , i); +PRINT_ERR ("Data mismatch at char %u\n" , i);return -1 ; }rte_pktmbuf_free (src);
@@ -97,7 +114,7 @@ test_enqueue_copies(int dev_id) (uintptr_t)srcs[i], (uintptr_t)dsts[i], 0 /* nofence */) != 1) { -printf ("Error with rte_ioat_enqueue_copy for buffer %u\n" , +PRINT_ERR ("Error with rte_ioat_enqueue_copy for buffer %u\n" , i);return -1 ; }
@@ -107,18 +124,18 @@ test_enqueue_copies(int dev_id)if (rte_ioat_completed_copies (dev_id,64 , (void *)completed_src, (void *)completed_dst) !=RTE_DIM (srcs)) { -printf ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_copies\n" );return -1 ; }for (i =0 ; i <RTE_DIM (srcs); i++) {char *src_data, *dst_data;if (completed_src[i] != srcs[i]) { -printf ("Error with source pointer %u\n" , i); +PRINT_ERR ("Error with source pointer %u\n" , i);return -1 ; }if (completed_dst[i] != dsts[i]) { -printf ("Error with dest pointer %u\n" , i); +PRINT_ERR ("Error with dest pointer %u\n" , i);return -1 ; }
@@ -126,7 +143,7 @@ test_enqueue_copies(int dev_id) dst_data = rte_pktmbuf_mtod(dsts[i], char *);for (j =0 ; j < length; j++)if (src_data[j] != dst_data[j]) { -printf ("Error with copy of packet %u, byte %u\n" , +PRINT_ERR ("Error with copy of packet %u, byte %u\n" , i, j);return -1 ; }
@@ -159,26 +176,26 @@ ioat_rawdev_test(uint16_t dev_id)rte_rawdev_info_get (dev_id, &info,sizeof (p));if (p.ring_size != expected_ring_size[dev_id]) { -printf ("Error, initial ring size is not as expected (Actual: %d, Expected: %d)\n" , +PRINT_ERR ("Error, initial ring size is not as expected (Actual: %d, Expected: %d)\n" , (int )p.ring_size, expected_ring_size[dev_id]);return -1 ; } p.ring_size =IOAT_TEST_RINGSIZE ;if (rte_rawdev_configure (dev_id, &info,sizeof (p)) !=0 ) { -printf ("Error with rte_rawdev_configure()\n" ); +PRINT_ERR ("Error with rte_rawdev_configure()\n" );return -1 ; }rte_rawdev_info_get (dev_id, &info,sizeof (p));if (p.ring_size !=IOAT_TEST_RINGSIZE ) { -printf ("Error, ring size is not %d (%d)\n" , +PRINT_ERR ("Error, ring size is not %d (%d)\n" ,IOAT_TEST_RINGSIZE , (int )p.ring_size);return -1 ; } expected_ring_size[dev_id] = p.ring_size;if (rte_rawdev_start (dev_id) !=0 ) { -printf ("Error with rte_rawdev_start()\n" ); +PRINT_ERR ("Error with rte_rawdev_start()\n" );return -1 ; }
@@ -189,7 +206,7 @@ ioat_rawdev_test(uint16_t dev_id)2048 ,/* data room size */ info.socket_id);if (pool ==NULL ) { -printf ("Error with mempool creation\n" ); +PRINT_ERR ("Error with mempool creation\n" );return -1 ; }
@@ -198,14 +215,14 @@ ioat_rawdev_test(uint16_t dev_id) snames =malloc (sizeof (*snames) * nb_xstats);if (snames ==NULL ) { -printf ("Error allocating xstat names memory\n" ); +PRINT_ERR ("Error allocating xstat names memory\n" ); goto err; }rte_rawdev_xstats_names_get (dev_id, snames, nb_xstats); ids =malloc (sizeof (*ids) * nb_xstats);if (ids ==NULL ) { -printf ("Error allocating xstat ids memory\n" ); +PRINT_ERR ("Error allocating xstat ids memory\n" ); goto err; }for (i =0 ; i < nb_xstats; i++)
@@ -213,7 +230,7 @@ ioat_rawdev_test(uint16_t dev_id) stats =malloc (sizeof (*stats) * nb_xstats);if (stats ==NULL ) { -printf ("Error allocating xstat memory\n" ); +PRINT_ERR ("Error allocating xstat memory\n" ); goto err; }
@@ -233,7 +250,7 @@ ioat_rawdev_test(uint16_t dev_id)rte_rawdev_stop (dev_id);if (rte_rawdev_xstats_reset (dev_id,NULL ,0 ) !=0 ) { -printf ("Error resetting xstat values\n" ); +PRINT_ERR ("Error resetting xstat values\n" ); goto err; }
Signature · 1 line
2.25.1
PATCH v6 5/25 raw/ioat: add a flag to control copying handle parameters
In reply to Bruce RichardsonFrom: Cheng Jiang <[email protected]>
Add a flag which controls whether rte_ioat_enqueue_copy and rte_ioat_completed_copies function should process handle parameters. Not doing so can improve the performance when handle parameters are not necessary.
--- doc/guides/rawdevs/ioat.rst | 3 ++ doc/guides/rel_notes/release_20_11.rst | 6 ++++ drivers/raw/ioat/ioat_rawdev.c | 2 ++ drivers/raw/ioat/rte_ioat_rawdev.h | 45 +++++++++++++++++++------- 4 files changed, 45 insertions(+), 11 deletions(-)
doc/guides/rawdevs/ioat.rst +3 -0
@@ -129,6 +129,9 @@ output, the ``dev_private`` structure element cannot be NULL, and must point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring size to be used by the device. The ring size must be a power of two, between 64 and 4096. +If it is not needed, the tracking by the driver of user-provided completion +handles may be disabled by setting the ``hdls_disable`` flag in +the configuration structure also. The following code shows how the device is configured in ``test_ioat_rawdev.c``:
doc/guides/rel_notes/release_20_11.rst +6 -0
@@ -116,6 +116,12 @@ New Features * Extern objects and functions can be plugged into the pipeline. * Transaction-oriented table updates. +* **Updated ioat rawdev driver** + + The ioat rawdev driver has been updated and enhanced. Changes include: + + * Added a per-device configuration flag to disable management of user-provided completion handles + Removed Items -------------
drivers/raw/ioat/ioat_rawdev.c +2 -0
@@ -58,6 +58,7 @@ ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config,return -EINVAL ; ioat->ring_size = params->ring_size; + ioat->hdls_disable = params->hdls_disable;if (ioat->desc_ring !=NULL ) {rte_memzone_free (ioat->desc_mz); ioat->desc_ring =NULL ;
@@ -122,6 +123,7 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info,return -EINVAL ; cfg->ring_size = ioat->ring_size; + cfg->hdls_disable = ioat->hdls_disable;return 0 ; }
drivers/raw/ioat/rte_ioat_rawdev.h +34 -11
97 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 6/25 raw/ioat: split header for readability
In reply to Bruce RichardsonRather than having a single long complicated header file for general use we can split things so that there is one header with all the publicly needed information - data structs and function prototypes - while the rest of the internal details are put separately. This makes it easier to read, understand and use the APIs.
--- drivers/raw/ioat/meson.build | 1 + drivers/raw/ioat/rte_ioat_rawdev.h | 147 +--------------------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 168 +++++++++++++++++++++++++ 3 files changed, 175 insertions(+), 141 deletions(-) create mode 100644 drivers/raw/ioat/rte_ioat_rawdev_fns.h
drivers/raw/ioat/meson.build +1 -0
@@ -8,4 +8,5 @@ sources = files('ioat_rawdev.c', deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h', + 'rte_ioat_rawdev_fns.h', 'rte_ioat_spec.h')
drivers/raw/ioat/rte_ioat_rawdev.h +6 -141
178 lines not on this page · raw message
drivers/raw/ioat/rte_ioat_rawdev_fns.h new file +168 -0
168 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 7/25 raw/ioat: rename functions to be operation-agnostic
In reply to Bruce RichardsonSince the hardware supported by the ioat driver is capable of operations other than just copies, we can rename the doorbell and completion-return functions to not have "copies" in their names. These functions are not copy-specific, and so would apply for other operations which may be added later to the driver.
Also add a suitable warning using deprecation attribute for any code using the old functions names.
--- doc/guides/rawdevs/ioat.rst | 16 ++++++++-------- doc/guides/rel_notes/release_20_11.rst | 9 +++++++++ doc/guides/sample_app_ug/ioat.rst | 8 ++++---- drivers/raw/ioat/ioat_rawdev_test.c | 12 ++++++------ drivers/raw/ioat/rte_ioat_rawdev.h | 17 ++++++++++------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 20 ++++++++++++++++---- examples/ioat/ioatfwd.c | 4 ++-- lib/librte_eal/include/rte_common.h | 1 + 8 files changed, 56 insertions(+), 31 deletions(-)
doc/guides/rawdevs/ioat.rst +8 -8
@@ -157,9 +157,9 @@ Performing Data Copies ~~~~~~~~~~~~~~~~~~~~~~~ To perform data copies using IOAT rawdev devices, the functions -``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used. +``rte_ioat_enqueue_copy()`` and ``rte_ioat_perform_ops()`` should be used. Once copies have been completed, the completion will be reported back when -the application calls ``rte_ioat_completed_copies()``. +the application calls ``rte_ioat_completed_ops()``. The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the device ring for copying at a later point. The parameters to that function
@@ -172,11 +172,11 @@ pointers if packet data is being copied. While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on the device ring, the copy will not actually be performed until after the -application calls the ``rte_ioat_do_copies()`` function. This function +application calls the ``rte_ioat_perform_ops()`` function. This function informs the device hardware of the elements enqueued on the ring, and the device will begin to process them. It is expected that, for efficiency reasons, a burst of operations will be enqueued to the device via multiple -enqueue calls between calls to the ``rte_ioat_do_copies()`` function. +enqueue calls between calls to the ``rte_ioat_perform_ops()`` function. The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue a burst of copies to the device and start the hardware processing of them:
@@ -210,10 +210,10 @@ a burst of copies to the device and start the hardware processing of them: return -1; } } - rte_ioat_do_copies(dev_id); + rte_ioat_perform_ops(dev_id); To retrieve information about completed copies, the API -``rte_ioat_completed_copies()`` should be used. This API will return to the +``rte_ioat_completed_ops()`` should be used. This API will return to the application a set of completion handles passed in when the relevant copies were enqueued.
@@ -223,9 +223,9 @@ is correct before freeing the data buffers using the returned handles: .. code-block:: C - if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src, + if (rte_ioat_completed_ops(dev_id, 64, (void *)completed_src, (void *)completed_dst) != RTE_DIM(srcs)) { - printf("Error with rte_ioat_completed_copies\n"); + printf("Error with rte_ioat_completed_ops\n"); return -1; } for (i = 0; i < RTE_DIM(srcs); i++) {
doc/guides/rel_notes/release_20_11.rst +9 -0
@@ -121,6 +121,11 @@ New Features The ioat rawdev driver has been updated and enhanced. Changes include: * Added a per-device configuration flag to disable management of user-provided completion handles + * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, + and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()`` + to better reflect the APIs' purposes, and remove the implication that + they are limited to copy operations only. + [Note: The old API is still provided but marked as deprecated in the code] Removed Items
@@ -234,6 +239,10 @@ API Changes * bpf: ``RTE_BPF_XTYPE_NUM`` has been dropped from ``rte_bpf_xtype``. +* raw/ioat: As noted above, the ``rte_ioat_do_copies()`` and + ``rte_ioat_completed_copies()`` functions have been renamed to + ``rte_ioat_perform_ops()`` and ``rte_ioat_completed_ops()`` respectively. + ABI Changes -----------
doc/guides/sample_app_ug/ioat.rst +4 -4
@@ -394,7 +394,7 @@ packet using ``pktmbuf_sw_copy()`` function and enqueue them to an rte_ring: nb_enq = ioat_enqueue_packets(pkts_burst, nb_rx, rx_config->ioat_ids[i]); if (nb_enq > 0) - rte_ioat_do_copies(rx_config->ioat_ids[i]); + rte_ioat_perform_ops(rx_config->ioat_ids[i]); } else { /* Perform packet software copy, free source packets */ int ret;
@@ -433,7 +433,7 @@ The packets are received in burst mode using ``rte_eth_rx_burst()`` function. When using hardware copy mode the packets are enqueued in copying device's buffer using ``ioat_enqueue_packets()`` which calls ``rte_ioat_enqueue_copy()``. When all received packets are in the -buffer the copy operations are started by calling ``rte_ioat_do_copies()``. +buffer the copy operations are started by calling ``rte_ioat_perform_ops()``. Function ``rte_ioat_enqueue_copy()`` operates on physical address of the packet. Structure ``rte_mbuf`` contains only physical address to start of the data buffer (``buf_iova``). Thus the address is adjusted
@@ -490,7 +490,7 @@ or indirect mbufs, then multiple copy operations must be used. All completed copies are processed by ``ioat_tx_port()`` function. When using -hardware copy mode the function invokes ``rte_ioat_completed_copies()`` +hardware copy mode the function invokes ``rte_ioat_completed_ops()`` on each assigned IOAT channel to gather copied packets. If software copy mode is used the function dequeues copied packets from the rte_ring. Then each packet MAC address is changed if it was enabled. After that copies are sent
@@ -510,7 +510,7 @@ in burst mode using `` rte_eth_tx_burst()``. for (i = 0; i < tx_config->nb_queues; i++) { if (copy_mode == COPY_MODE_IOAT_NUM) { /* Deque the mbufs from IOAT device. */ - nb_dq = rte_ioat_completed_copies( + nb_dq = rte_ioat_completed_ops( tx_config->ioat_ids[i], MAX_PKT_BURST, (void *)mbufs_src, (void *)mbufs_dst); } else {
drivers/raw/ioat/ioat_rawdev_test.c +6 -6
@@ -65,12 +65,12 @@ test_enqueue_copies(int dev_id)PRINT_ERR ("Error with rte_ioat_enqueue_copy\n" );return -1 ; } -rte_ioat_do_copies (dev_id); +rte_ioat_perform_ops (dev_id);usleep (10 ); -if (rte_ioat_completed_copies (dev_id,1 , (void *)&completed[0 ], +if (rte_ioat_completed_ops (dev_id,1 , (void *)&completed[0 ], (void *)&completed[1 ]) !=1 ) { -PRINT_ERR ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_ops\n" );return -1 ; }if (completed[0 ] != src || completed[1 ] != dst) {
@@ -119,12 +119,12 @@ test_enqueue_copies(int dev_id)return -1 ; } } -rte_ioat_do_copies (dev_id); +rte_ioat_perform_ops (dev_id);usleep (100 ); -if (rte_ioat_completed_copies (dev_id,64 , (void *)completed_src, +if (rte_ioat_completed_ops (dev_id,64 , (void *)completed_src, (void *)completed_dst) !=RTE_DIM (srcs)) { -PRINT_ERR ("Error with rte_ioat_completed_copies\n" ); +PRINT_ERR ("Error with rte_ioat_completed_ops\n" );return -1 ; }for (i =0 ; i <RTE_DIM (srcs); i++) {
drivers/raw/ioat/rte_ioat_rawdev.h +10 -7
@@ -69,24 +69,26 @@ struct rte_ioat_rawdev_config {* Number of operations enqueued, either 0 or 1 */ static inline int +__rte_experimentalrte_ioat_enqueue_copy (int dev_id,phys_addr_t src,phys_addr_t dst,unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl,int fence);/** -* Trigger hardware to begin performing enqueued copy operations +* Trigger hardware to begin performing enqueued operations * * This API is used to write the "doorbell" to the hardware to trigger it -* to begin the copy operations previously enqueued by rte_ioat_enqueue_copy() +* to begin the operations previously enqueued by rte_ioat_enqueue_copy() * * @param dev_id * The rawdev device id of the ioat instance */ static inline void -rte_ioat_do_copies (int dev_id); +__rte_experimental +rte_ioat_perform_ops (int dev_id);/** -* Returns details of copy operations that have been completed +* Returns details of operations that have been completed * * If the hdls_disable option was not set when the device was configured, * the function will return to the caller the user-provided "handles" for
@@ -104,11 +106,11 @@ rte_ioat_do_copies(int dev_id);* NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @param src_hdls -* Array to hold the source handle parameters of the completed copies. +* Array to hold the source handle parameters of the completed ops. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @param dst_hdls -* Array to hold the destination handle parameters of the completed copies. +* Array to hold the destination handle parameters of the completed ops. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. * @return
@@ -117,7 +119,8 @@ rte_ioat_do_copies(int dev_id);* to the src_hdls and dst_hdls array parameters. */ static inline int -rte_ioat_completed_copies (int dev_id,uint8_t max_copies, +__rte_experimental +rte_ioat_completed_ops (int dev_id,uint8_t max_copies,uintptr_t *src_hdls,uintptr_t *dst_hdls);/* include the implementation details from a separate file */
drivers/raw/ioat/rte_ioat_rawdev_fns.h +16 -4
@@ -83,10 +83,10 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, }/* -* Trigger hardware to begin performing enqueued copy operations +* Trigger hardware to begin performing enqueued operations */ static inline void -rte_ioat_do_copies (int dev_id) +rte_ioat_perform_ops (int dev_id) {struct rte_ioat_rawdev *ioat = (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;
@@ -114,10 +114,10 @@ rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error) }/* -* Returns details of copy operations that have been completed +* Returns details of operations that have been completed */ static inline int -rte_ioat_completed_copies (int dev_id,uint8_t max_copies, +rte_ioat_completed_ops (int dev_id,uint8_t max_copies,uintptr_t *src_hdls,uintptr_t *dst_hdls) { struct rte_ioat_rawdev *ioat =
@@ -165,4 +165,16 @@ rte_ioat_completed_copies(int dev_id, uint8_t max_copies,return count; } +static inline void +__rte_deprecated_msg("use rte_ioat_perform_ops() instead") +rte_ioat_do_copies(int dev_id) { rte_ioat_perform_ops(dev_id); } + +static inline int +__rte_deprecated_msg("use rte_ioat_completed_ops() instead") +rte_ioat_completed_copies(int dev_id, uint8_t max_copies, + uintptr_t *src_hdls, uintptr_t *dst_hdls) +{ +return rte_ioat_completed_ops (dev_id, max_copies, src_hdls, dst_hdls); +} +#endif /* _RTE_IOAT_RAWDEV_FNS_H_ */
examples/ioat/ioatfwd.c +2 -2
@@ -406,7 +406,7 @@ ioat_rx_port(struct rxtx_port_config *rx_config) nb_enq =ioat_enqueue_packets (pkts_burst, nb_rx, rx_config->ioat_ids[i]);if (nb_enq >0 ) -rte_ioat_do_copies (rx_config->ioat_ids[i]); +rte_ioat_perform_ops (rx_config->ioat_ids[i]); }else {/* Perform packet software copy, free source packets */ int ret;
@@ -452,7 +452,7 @@ ioat_tx_port(struct rxtx_port_config *tx_config)for (i =0 ; i < tx_config->nb_queues; i++) {if (copy_mode ==COPY_MODE_IOAT_NUM ) {/* Deque the mbufs from IOAT device. */ - nb_dq =rte_ioat_completed_copies ( + nb_dq =rte_ioat_completed_ops ( tx_config->ioat_ids[i],MAX_PKT_BURST , (void *)mbufs_src, (void *)mbufs_dst); }else {
lib/librte_eal/include/rte_common.h +1 -0
@@ -85,6 +85,7 @@ typedef uint16_t unaligned_uint16_t;/******* Macro to mark functions and fields scheduled for removal *****/ #define __rte_deprecated __attribute__((__deprecated__)) +#define __rte_deprecated_msg (msg) __attribute__((__deprecated__(msg)))/** * Mark a function or variable to a weak reference.
Signature · 1 line
2.25.1
PATCH v6 8/25 raw/ioat: add separate API for fence call
In reply to Bruce RichardsonRather than having the fence signalled via a flag on a descriptor - which requires reading the docs to find out whether the flag needs to go on the last descriptor before, or the first descriptor after the fence - we can instead add a separate fence API call. This becomes unambiguous to use, since the fence call explicitly comes between two other enqueue calls. It also allows more freedom of implementation in the driver code.
--- doc/guides/rawdevs/ioat.rst | 3 +-- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/raw/ioat/ioat_rawdev_test.c | 6 ++---- drivers/raw/ioat/rte_ioat_rawdev.h | 27 ++++++++++++++++++++------ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 22 ++++++++++++++++++--- examples/ioat/ioatfwd.c | 12 ++++-------- 6 files changed, 51 insertions(+), 23 deletions(-)
doc/guides/rawdevs/ioat.rst +1 -2
@@ -203,8 +203,7 @@ a burst of copies to the device and start the hardware processing of them: dsts[i]->buf_iova + dsts[i]->data_off, length, (uintptr_t)srcs[i], - (uintptr_t)dsts[i], - 0 /* nofence */) != 1) { + (uintptr_t)dsts[i]) != 1) { printf("Error with rte_ioat_enqueue_copy for buffer %u\n", i); return -1;
doc/guides/rel_notes/release_20_11.rst +4 -0
@@ -126,6 +126,10 @@ New Features to better reflect the APIs' purposes, and remove the implication that they are limited to copy operations only. [Note: The old API is still provided but marked as deprecated in the code] + * Added a new API ``rte_ioat_fence()`` to add a fence between operations. + This API replaces the ``fence`` flag parameter in the ``rte_ioat_enqueue_copies()`` function, + and is clearer as there is no ambiguity as to whether the flag should be + set on the last operation before the fence or the first operation after it. Removed Items
drivers/raw/ioat/ioat_rawdev_test.c +2 -4
@@ -60,8 +60,7 @@ test_enqueue_copies(int dev_id) dst->buf_iova + dst->data_off, length, (uintptr_t)src, - (uintptr_t)dst, - 0 /* no fence */) != 1) { + (uintptr_t)dst) != 1) {PRINT_ERR ("Error with rte_ioat_enqueue_copy\n" );return -1 ; }
@@ -112,8 +111,7 @@ test_enqueue_copies(int dev_id) dsts[i]->buf_iova + dsts[i]->data_off, length, (uintptr_t)srcs[i], - (uintptr_t)dsts[i], - 0 /* nofence */) != 1) { + (uintptr_t)dsts[i]) != 1) {PRINT_ERR ("Error with rte_ioat_enqueue_copy for buffer %u\n" , i);return -1 ;
drivers/raw/ioat/rte_ioat_rawdev.h +21 -6
@@ -61,18 +61,33 @@ struct rte_ioat_rawdev_config {* operation has been completed and the user polls for the completion details. * NOTE: If hdls_disable configuration option for the device is set, this * parameter is ignored. -* @param fence -* A flag parameter indicating that hardware should not begin to perform any -* subsequently enqueued copy operations until after this operation has -* completed * @return * Number of operations enqueued, either 0 or 1 */ static inline int __rte_experimentalrte_ioat_enqueue_copy (int dev_id,phys_addr_t src,phys_addr_t dst, -unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl, -int fence); +unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl); + +/** +* Add a fence to force ordering between operations +* +* This adds a fence to a sequence of operations to enforce ordering, such that +* all operations enqueued before the fence must be completed before operations +* after the fence. +* NOTE: Since this fence may be added as a flag to the last operation enqueued, +* this API may not function correctly when called immediately after an +* "rte_ioat_perform_ops" call i.e. before any new operations are enqueued. +* +* @param dev_id +* The rawdev device id of the ioat instance +* @return +* Number of fences enqueued, either 0 or 1 +*/ +static inline int +__rte_experimental +rte_ioat_fence (int dev_id); +/** * Trigger hardware to begin performing enqueued operations
drivers/raw/ioat/rte_ioat_rawdev_fns.h +19 -3
@@ -47,8 +47,7 @@ struct rte_ioat_rawdev {*/ static inline int rte_ioat_enqueue_copy (int dev_id,phys_addr_t src,phys_addr_t dst, -unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl, -int fence) +unsigned int length,uintptr_t src_hdl,uintptr_t dst_hdl) {struct rte_ioat_rawdev *ioat = (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private;
@@ -69,7 +68,7 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, desc = &ioat->desc_ring[write]; desc->size = length;/* set descriptor write-back every 16th descriptor */ - desc->u.control_raw = (uint32_t )((!!fence <<4 ) | (!(write &0xF )) <<3 ); + desc->u.control_raw = (uint32_t )((!(write &0xF )) <<3 ); desc->src_addr = src; desc->dest_addr = dst;
@@ -82,6 +81,23 @@ rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,return 1 ; } +/* add fence to last written descriptor */ +static inline int +rte_ioat_fence (int dev_id) +{ +struct rte_ioat_rawdev *ioat = + (struct rte_ioat_rawdev *)rte_rawdevs[dev_id].dev_private; +unsigned short write = ioat->next_write; +unsigned short mask = ioat->ring_size -1 ; +struct rte_ioat_generic_hw_desc *desc; + + write = (write -1 ) & mask; + desc = &ioat->desc_ring[write]; + + desc->u.control.fence =1 ; +return 0 ; +} +/* * Trigger hardware to begin performing enqueued operations */
examples/ioat/ioatfwd.c +4 -8
@@ -361,15 +361,11 @@ ioat_enqueue_packets(struct rte_mbuf **pkts,for (i =0 ; i < nb_rx; i++) {/* Perform data copy */ ret =rte_ioat_enqueue_copy (dev_id, - pkts[i]->buf_iova - - addr_offset, - pkts_copy[i]->buf_iova - - addr_offset, -rte_pktmbuf_data_len (pkts[i]) - + addr_offset, + pkts[i]->buf_iova - addr_offset, + pkts_copy[i]->buf_iova - addr_offset, +rte_pktmbuf_data_len (pkts[i]) + addr_offset, (uintptr_t )pkts[i], - (uintptr_t )pkts_copy[i], -0 /* nofence */ ); + (uintptr_t )pkts_copy[i]);if (ret !=1 )break ;
Signature · 1 line
2.25.1
PATCH v6 9/25 raw/ioat: make the HW register spec private
In reply to Bruce RichardsonOnly a few definitions from the hardware spec are actually used in the driver runtime, so we can copy over those few and make the rest of the spec a private header in the driver.
---
drivers/raw/ioat/ioat_rawdev.c | 3 ++
.../raw/ioat/{rte_ioat_spec.h => ioat_spec.h} | 26 -----------
drivers/raw/ioat/meson.build | 3 +-
drivers/raw/ioat/rte_ioat_rawdev_fns.h | 43 +++++++++++++++++--
4 files changed, 44 insertions(+), 31 deletions(-)
rename drivers/raw/ioat/{rte_ioat_spec.h => ioat_spec.h} (91%)drivers/raw/ioat/ioat_rawdev.c +3 -0
@@ -4,10 +4,12 @@#include <rte_cycles.h> #include <rte_bus_pci.h> +#include <rte_memzone.h> #include <rte_string_fns.h> #include <rte_rawdev_pmd.h> #include "rte_ioat_rawdev.h" +#include "ioat_spec.h" static struct rte_pci_driver ioat_pmd_drv;
@@ -268,6 +270,7 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) ioat->rawdev = rawdev; ioat->mz = mz; ioat->regs = dev->mem_resource[0 ].addr; + ioat->doorbell = &ioat->regs->dmacount; ioat->ring_size =0 ; ioat->desc_ring =NULL ; ioat->status_addr = ioat->mz->iova +
drivers/raw/ioat/ioat_spec.h renamed from drivers/raw/ioat/rte_ioat_spec.h +0 -26
@@ -86,32 +86,6 @@ struct rte_ioat_registers {#define RTE_IOAT_CHANCMP_ALIGN 8/* CHANCMP address must be 64-bit aligned */ -struct rte_ioat_generic_hw_desc { -uint32_t size; -union { -uint32_t control_raw; -struct { -uint32_t int_enable:1 ; -uint32_t src_snoop_disable:1 ; -uint32_t dest_snoop_disable:1 ; -uint32_t completion_update:1 ; -uint32_t fence:1 ; -uint32_t reserved2:1 ; -uint32_t src_page_break:1 ; -uint32_t dest_page_break:1 ; -uint32_t bundle:1 ; -uint32_t dest_dca:1 ; -uint32_t hint:1 ; -uint32_t reserved:13 ; -uint32_t op:8 ; - } control; - } u; -uint64_t src_addr; -uint64_t dest_addr; -uint64_t next; -uint64_t op_specific[4 ]; -}; -struct rte_ioat_dma_hw_desc {uint32_t size;union {
drivers/raw/ioat/meson.build +1 -2
@@ -8,5 +8,4 @@ sources = files('ioat_rawdev.c', deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h', - 'rte_ioat_rawdev_fns.h', - 'rte_ioat_spec.h') + 'rte_ioat_rawdev_fns.h')
drivers/raw/ioat/rte_ioat_rawdev_fns.h +40 -3
69 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 10/25 usertools/dpdk-devbind.py: add support for DSA HW
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
Intel Data Streaming Accelerator (Intel DSA) is a high-performance data copy and transformation accelerator which will be integrated in future Intel processors [1].
Add DSA device support to dpdk-devbind.py script.
[1] https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
--- doc/guides/rel_notes/release_20_11.rst | 2 ++ usertools/dpdk-devbind.py | 6 +++++- 2 files changed, 7 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_20_11.rst +2 -0
@@ -120,6 +120,8 @@ New Features The ioat rawdev driver has been updated and enhanced. Changes include: + * Added support for Intel\ |reg| Data Streaming Accelerator hardware. + For more information, see https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator * Added a per-device configuration flag to disable management of user-provided completion handles * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()``
usertools/dpdk-devbind.py +5 -1
@@ -48,6 +48,8 @@ 'SVendor': None, 'SDevice': None} intel_ioat_icx = {'Class': '08', 'Vendor': '8086', 'Device': '0b00', 'SVendor': None, 'SDevice': None} +intel_idxd_spr = {'Class': '08', 'Vendor': '8086', 'Device': '0b25', + 'SVendor': None, 'SDevice': None} intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', 'SVendor': None, 'SDevice': None} intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
@@ -59,7 +61,9 @@ eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] -misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, intel_ntb_skx, intel_ntb_icx, octeontx2_dma] +misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, intel_idxd_spr, + intel_ntb_skx, intel_ntb_icx, + octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties
Signature · 1 line
2.25.1
PATCH v6 11/25 raw/ioat: add skeleton for VFIO/UIO based DSA device
In reply to Bruce RichardsonAdd in the basic probe/remove skeleton code for DSA devices which are bound directly to vfio or uio driver. The kernel module for supporting these uses the "idxd" name, so that name is used as function and file prefix to avoid conflict with existing "ioat" prefixed functions.
Since we are adding new files to the driver and there will be common definitions shared between the various files, we create a new internal header file ioat_private.h to hold common macros and function prototypes.
--- doc/guides/rawdevs/ioat.rst | 69 ++++++++++----------------------- drivers/raw/ioat/idxd_pci.c | 56 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 27 +++++++++++++ drivers/raw/ioat/ioat_rawdev.c | 9 +---- drivers/raw/ioat/meson.build | 6 ++- 5 files changed, 108 insertions(+), 59 deletions(-) create mode 100644 drivers/raw/ioat/idxd_pci.c create mode 100644 drivers/raw/ioat/ioat_private.h
doc/guides/rawdevs/ioat.rst +20 -49
@@ -3,10 +3,12 @@ .. include:: <isonum.txt> -IOAT Rawdev Driver for Intel\ |reg| QuickData Technology -====================================================================== +IOAT Rawdev Driver +=================== The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| +Data Streaming Accelerator `(Intel DSA) +<https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator>`_ and for Intel\ |reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology `(Intel I/OAT) <https://www.intel.com/content/www/us/en/wireless-network/accel-technology.html>`_.
@@ -17,61 +19,30 @@ be done by software, freeing up CPU cycles for other tasks. Hardware Requirements ---------------------- -On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can -be detected by checking the output of the ``lspci`` command, where the -hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For -example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz, -lspci shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) - 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) - 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) - 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) - 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) - 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) - 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) - 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) - -On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci -shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - +The ``dpdk-devbind.py`` script, included with DPDK, +can be used to show the presence of supported hardware. +Running ``dpdk-devbind.py --status-dev misc`` will show all the miscellaneous, +or rawdev-based devices on the system. +For Intel\ |reg| QuickData Technology devices, the hardware will be often listed as "Crystal Beach DMA", +or "CBDMA". +For Intel\ |reg| DSA devices, they are currently (at time of writing) appearing as devices with type "0b25", +due to the absence of pci-id database entries for them at this point. Compilation ------------ -For builds done with ``make``, the driver compilation is enabled by the -``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is -enabled by default in builds for x86 platforms, and disabled in other -configurations. - -For builds using ``meson`` and ``ninja``, the driver will be built when the -target platform is x86-based. +For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. +No additional compilation steps are necessary. Device Setup ------------- -The Intel\ |reg| QuickData Technology HW devices will need to be bound to a -user-space IO driver for use. The script ``dpdk-devbind.py`` script -included with DPDK can be used to view the state of the devices and to bind -them to a suitable DPDK-supported kernel driver. When querying the status -of the devices, they will appear under the category of "Misc (rawdev) -devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be -used to see the state of those devices alone. +The HW devices to be used will need to be bound to a user-space IO driver for use. +The ``dpdk-devbind.py`` script can be used to view the state of the devices +and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1 Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/raw/ioat/idxd_pci.c new file +56 -0
56 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h new file +27 -0
@@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2020 Intel Corporation +*/ + +#ifndef _IOAT_PRIVATE_H_ +#define _IOAT_PRIVATE_H_ + +/** +* @file idxd_private.h +* +* Private data structures for the idxd/DSA part of ioat device driver +* +* @warning +* @b EXPERIMENTAL: these structures and APIs may change without prior notice +*/ + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev.c +1 -8
@@ -10,6 +10,7 @@#include "rte_ioat_rawdev.h" #include "ioat_spec.h" +#include "ioat_private.h" static struct rte_pci_driver ioat_pmd_drv;
@@ -29,14 +30,6 @@ static struct rte_pci_driver ioat_pmd_drv;RTE_LOG_REGISTER (ioat_pmd_logtype, rawdev.ioat,INFO ); -#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ - ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) - -#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) -#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) -#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) -#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) -#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc)#define COMPLETION_SZ sizeof(__m128i)
drivers/raw/ioat/meson.build +4 -2
@@ -3,8 +3,10 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' -sources = files('ioat_rawdev.c', - 'ioat_rawdev_test.c') +sources = files( + 'idxd_pci.c', + 'ioat_rawdev.c', + 'ioat_rawdev_test.c') deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h',
Signature · 1 line
2.25.1
PATCH v6 12/25 raw/ioat: add vdev probe for DSA/idxd devices
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
The Intel DSA devices can be exposed to userspace via kernel driver, so can be used without having to bind them to vfio/uio. Therefore we add support for using those kernel-configured devices as vdevs, taking as parameter the individual HW work queue to be used by the vdev.
--- doc/guides/rawdevs/ioat.rst | 68 +++++++++++++++++-- drivers/raw/ioat/idxd_vdev.c | 123 +++++++++++++++++++++++++++++++++++ drivers/raw/ioat/meson.build | 6 +- 3 files changed, 192 insertions(+), 5 deletions(-) create mode 100644 drivers/raw/ioat/idxd_vdev.c
doc/guides/rawdevs/ioat.rst +64 -4
@@ -37,9 +37,62 @@ No additional compilation steps are necessary. Device Setup ------------- +Depending on support provided by the PMD, HW devices can either use the kernel configured driver +or be bound to a user-space IO driver for use. +For example, Intel\ |reg| DSA devices can use the IDXD kernel driver or DPDK-supported drivers, +such as ``vfio-pci``. + +Intel\ |reg| DSA devices using idxd kernel driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To use a Intel\ |reg| DSA device bound to the IDXD kernel driver, the device must first be configured. +The `accel-config <https://github.com/intel/idxd-config>`_ utility library can be used for configuration. + +.. note:: + The device configuration can also be done by directly interacting with the sysfs nodes. + +There are some mandatory configuration steps before being able to use a device with an application. +The internal engines, which do the copies or other operations, +and the work-queues, which are used by applications to assign work to the device, +need to be assigned to groups, and the various other configuration options, +such as priority or queue depth, need to be set for each queue. + +To assign an engine to a group:: + + $ accel-config config-engine dsa0/engine0.0 --group-id=0 + $ accel-config config-engine dsa0/engine0.1 --group-id=1 + +To assign work queues to groups for passing descriptors to the engines a similar accel-config command can be used. +However, the work queues also need to be configured depending on the use-case. +Some configuration options include: + +* mode (Dedicated/Shared): Indicates whether a WQ may accept jobs from multiple queues simultaneously. +* priority: WQ priority between 1 and 15. Larger value means higher priority. +* wq-size: the size of the WQ. Sum of all WQ sizes must be less that the total-size defined by the device. +* type: WQ type (kernel/mdev/user). Determines how the device is presented. +* name: identifier given to the WQ. + +Example configuration for a work queue:: + + $ accel-config config-wq dsa0/wq0.0 --group-id=0 \ + --mode=dedicated --priority=10 --wq-size=8 \ + --type=user --name=app1 + +Once the devices have been configured, they need to be enabled:: + + $ accel-config enable-device dsa0 + $ accel-config enable-wq dsa0/wq0.0 + +Check the device configuration:: + + $ accel-config list + +Devices using VFIO/UIO drivers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The HW devices to be used will need to be bound to a user-space IO driver for use. The ``dpdk-devbind.py`` script can be used to view the state of the devices -and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +and to bind them to a suitable DPDK-supported driver, such as ``vfio-pci``. For example:: $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1
@@ -47,9 +100,16 @@ For example:: Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Once bound to a suitable kernel device driver, the HW devices will be found -as part of the PCI scan done at application initialization time. No vdev -parameters need to be passed to create or initialize the device. +For devices bound to a suitable DPDK-supported VFIO/UIO driver, the HW devices will +be found as part of the device scan done at application initialization time without +the need to pass parameters to the application. + +If the device is bound to the IDXD kernel driver (and previously configured with sysfs), +then a specific work queue needs to be passed to the application via a vdev parameter. +This vdev parameter take the driver name and work queue name as parameters. +For example, to use work queue 0 on Intel\ |reg| DSA instance 0:: + + $ dpdk-test --no-pci --vdev=rawdev_idxd,wq=0.0 Once probed successfully, the device will appear as a ``rawdev``, that is a "raw device type" inside DPDK, and can be accessed using APIs from the
drivers/raw/ioat/idxd_vdev.c new file +123 -0
123 lines not on this page · raw message
drivers/raw/ioat/meson.build +5 -1
@@ -5,9 +5,13 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' sources = files( 'idxd_pci.c', + 'idxd_vdev.c', 'ioat_rawdev.c', 'ioat_rawdev_test.c') -deps += ['rawdev', 'bus_pci', 'mbuf'] +deps += ['bus_pci', + 'bus_vdev', + 'mbuf', + 'rawdev'] install_headers('rte_ioat_rawdev.h', 'rte_ioat_rawdev_fns.h')
Signature · 1 line
2.25.1
PATCH v6 13/25 raw/ioat: include example configuration script
In reply to Bruce RichardsonDevices managed by the idxd kernel driver must be configured for DPDK use before it can be used by the ioat driver. This example script serves both as a quick way to get the driver set up with a simple configuration, and as the basis for users to modify it and create their own configuration scripts.
--- doc/guides/rawdevs/ioat.rst | 2 + drivers/raw/ioat/dpdk_idxd_cfg.py | 79 +++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) create mode 100755 drivers/raw/ioat/dpdk_idxd_cfg.py
doc/guides/rawdevs/ioat.rst +2 -0
@@ -50,6 +50,8 @@ The `accel-config <https://github.com/intel/idxd-config>`_ utility library can b .. note:: The device configuration can also be done by directly interacting with the sysfs nodes. + An example of how this may be done can be seen in the script ``dpdk_idxd_cfg.py`` + included in the driver source directory. There are some mandatory configuration steps before being able to use a device with an application. The internal engines, which do the copies or other operations,
drivers/raw/ioat/dpdk_idxd_cfg.py new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 14/25 raw/ioat: create rawdev instances on idxd PCI probe
In reply to Bruce RichardsonWhen a matching device is found via PCI probe create a rawdev instance for each queue on the hardware. Use empty self-test function for these devices so that the overall rawdev_autotest does not report failures.
--- drivers/raw/ioat/idxd_pci.c | 237 ++++++++++++++++++++++++- drivers/raw/ioat/ioat_common.c | 68 +++++++ drivers/raw/ioat/ioat_private.h | 33 ++++ drivers/raw/ioat/ioat_rawdev_test.c | 7 + drivers/raw/ioat/ioat_spec.h | 64 +++++++ drivers/raw/ioat/meson.build | 1 + drivers/raw/ioat/rte_ioat_rawdev_fns.h | 35 +++- 7 files changed, 442 insertions(+), 3 deletions(-) create mode 100644 drivers/raw/ioat/ioat_common.c
drivers/raw/ioat/idxd_pci.c +235 -2
@@ -3,8 +3,10 @@*/ #include <rte_bus_pci.h> +#include <rte_memzone.h> #include "ioat_private.h" +#include "ioat_spec.h" #define IDXD_VENDOR_ID 0x8086#define IDXD_DEVICE_ID_SPR 0x0B25
@@ -16,17 +18,246 @@ const struct rte_pci_id pci_id_idxd_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static inline int +idxd_pci_dev_command (struct idxd_rawdev *idxd,enum rte_idxd_cmds command) +{ +uint8_t err_code; +uint16_t qid = idxd->qid; +int i =0 ; + +if (command >= idxd_disable_wq && command <= idxd_reset_wq) + qid = (1 << qid); +rte_spinlock_lock (&idxd->u.pci->lk); + idxd->u.pci->regs->cmd = (command <<IDXD_CMD_SHIFT ) | qid; + +do { +rte_pause (); + err_code = idxd->u.pci->regs->cmdstatus; +if (++i >=1000 ) { +IOAT_PMD_ERR ("Timeout waiting for command response from HW" ); +rte_spinlock_unlock (&idxd->u.pci->lk); +return err_code; + } + }while (idxd->u.pci->regs->cmdstatus &CMDSTATUS_ACTIVE_MASK ); +rte_spinlock_unlock (&idxd->u.pci->lk); + +return err_code &CMDSTATUS_ERR_MASK ; +} + +static int +idxd_is_wq_enabled (struct idxd_rawdev *idxd) +{ +uint32_t state = idxd->u.pci->wq_regs[idxd->qid].wqcfg[WQ_STATE_IDX ]; +return ((state >>WQ_STATE_SHIFT ) &WQ_STATE_MASK ) ==0x1 ; +} + +static const struct rte_rawdev_ops idxd_pci_ops = { + .dev_close = idxd_rawdev_close, + .dev_selftest = idxd_rawdev_test, +}; + +/* each portal uses 4 x 4k pages */ +#define IDXD_PORTAL_SIZE (4096 * 4) + +static int +init_pci_device (struct rte_pci_device *dev,struct idxd_rawdev *idxd) +{ +struct idxd_pci_common *pci; +uint8_t nb_groups, nb_engines, nb_wqs; +uint16_t grp_offset, wq_offset;/* how far into bar0 the regs are */ +uint16_t wq_size, total_wq_size; +uint8_t lg2_max_batch, lg2_max_copy_size; +unsigned int i, err_code; + + pci =malloc (sizeof (*pci)); +if (pci ==NULL ) { +IOAT_PMD_ERR ("%s: Can't allocate memory" , __func__); + goto err; + } +rte_spinlock_init (&pci->lk); + +/* assign the bar registers, and then configure device */ + pci->regs = dev->mem_resource[0 ].addr; + grp_offset = (uint16_t )pci->regs->offsets[0 ]; + pci->grp_regs =RTE_PTR_ADD (pci->regs, grp_offset *0x100 ); + wq_offset = (uint16_t )(pci->regs->offsets[0 ] >>16 ); + pci->wq_regs =RTE_PTR_ADD (pci->regs, wq_offset *0x100 ); + pci->portals = dev->mem_resource[2 ].addr; + +/* sanity check device status */ +if (pci->regs->gensts &GENSTS_DEV_STATE_MASK ) { +/* need function-level-reset (FLR) or is enabled */ +IOAT_PMD_ERR ("Device status is not disabled, cannot init" ); + goto err; + } +if (pci->regs->cmdstatus &CMDSTATUS_ACTIVE_MASK ) { +/* command in progress */ +IOAT_PMD_ERR ("Device has a command in progress, cannot init" ); + goto err; + } + +/* read basic info about the hardware for use when configuring */ + nb_groups = (uint8_t )pci->regs->grpcap; + nb_engines = (uint8_t )pci->regs->engcap; + nb_wqs = (uint8_t )(pci->regs->wqcap >>16 ); + total_wq_size = (uint16_t )pci->regs->wqcap; + lg2_max_copy_size = (uint8_t )(pci->regs->gencap >>16 ) &0x1F ; + lg2_max_batch = (uint8_t )(pci->regs->gencap >>21 ) &0x0F ; + +IOAT_PMD_DEBUG ("nb_groups = %u, nb_engines = %u, nb_wqs = %u" , + nb_groups, nb_engines, nb_wqs); + +/* zero out any old config */ +for (i =0 ; i < nb_groups; i++) { + pci->grp_regs[i].grpengcfg =0 ; + pci->grp_regs[i].grpwqcfg[0 ] =0 ; + } +for (i =0 ; i < nb_wqs; i++) + pci->wq_regs[i].wqcfg[0 ] =0 ; + +/* put each engine into a separate group to avoid reordering */ +if (nb_groups > nb_engines) + nb_groups = nb_engines; +if (nb_groups < nb_engines) + nb_engines = nb_groups; + +/* assign engines to groups, round-robin style */ +for (i =0 ; i < nb_engines; i++) { +IOAT_PMD_DEBUG ("Assigning engine %u to group %u" , + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpengcfg |= (1ULL << i); + } + +/* now do the same for queues and give work slots to each queue */ + wq_size = total_wq_size / nb_wqs; +IOAT_PMD_DEBUG ("Work queue size = %u, max batch = 2^%u, max copy = 2^%u" , + wq_size, lg2_max_batch, lg2_max_copy_size); +for (i =0 ; i < nb_wqs; i++) { +/* add engine "i" to a group */ +IOAT_PMD_DEBUG ("Assigning work queue %u to group %u" , + i, i % nb_groups); + pci->grp_regs[i % nb_groups].grpwqcfg[0 ] |= (1ULL << i); +/* now configure it, in terms of size, max batch, mode */ + pci->wq_regs[i].wqcfg[WQ_SIZE_IDX ] = wq_size; + pci->wq_regs[i].wqcfg[WQ_MODE_IDX ] = (1 <<WQ_PRIORITY_SHIFT ) | +WQ_MODE_DEDICATED ; + pci->wq_regs[i].wqcfg[WQ_SIZES_IDX ] = lg2_max_copy_size | + (lg2_max_batch <<WQ_BATCH_SZ_SHIFT ); + } + +/* dump the group configuration to output */ +for (i =0 ; i < nb_groups; i++) { +IOAT_PMD_DEBUG ("## Group %d" , i); +IOAT_PMD_DEBUG (" GRPWQCFG: %" PRIx64, pci->grp_regs[i].grpwqcfg[0 ]); +IOAT_PMD_DEBUG (" GRPENGCFG: %" PRIx64, pci->grp_regs[i].grpengcfg); +IOAT_PMD_DEBUG (" GRPFLAGS: %" PRIx32, pci->grp_regs[i].grpflags); + } + + idxd->u.pci = pci; + idxd->max_batches = wq_size; + +/* enable the device itself */ + err_code =idxd_pci_dev_command (idxd, idxd_enable_dev); +if (err_code) { +IOAT_PMD_ERR ("Error enabling device: code %#x" , err_code); +return err_code; + } +IOAT_PMD_DEBUG ("IDXD Device enabled OK" ); + +return nb_wqs; + +err: +free (pci); +return -1 ; +} +static int idxd_rawdev_probe_pci (struct rte_pci_driver *drv,struct rte_pci_device *dev) { -int ret =0 ; +struct idxd_rawdev idxd = {{0 }};/* Double {} to avoid error on BSD12 */ +uint8_t nb_wqs; +int qid, ret =0 ;char name[PCI_PRI_STR_SIZE ];rte_pci_device_name (&dev->addr, name,sizeof (name));IOAT_PMD_INFO ("Init %s on NUMA node %d" , name, dev->device.numa_node); dev->device.driver = &drv->driver; -return ret; + ret =init_pci_device (dev, &idxd); +if (ret <0 ) { +IOAT_PMD_ERR ("Error initializing PCI hardware" ); +return ret; + } + nb_wqs = (uint8_t )ret; + +/* set up one device for each queue */ +for (qid =0 ; qid < nb_wqs; qid++) { +char qname[32 ]; + +/* add the queue number to each device name */ +snprintf (qname,sizeof (qname),"%s-q%d" , name, qid); + idxd.qid = qid; + idxd.public.portal =RTE_PTR_ADD (idxd.u.pci->portals, + qid *IDXD_PORTAL_SIZE ); +if (idxd_is_wq_enabled (&idxd)) +IOAT_PMD_ERR ("Error, WQ %u seems enabled" , qid); + ret =idxd_rawdev_create (qname, &dev->device, + &idxd, &idxd_pci_ops); +if (ret !=0 ) { +IOAT_PMD_ERR ("Failed to create rawdev %s" , name); +if (qid ==0 )/* if no devices using this, free pci */ +free (idxd.u.pci); +return ret; + } + } + +return 0 ; +} + +static int +idxd_rawdev_destroy (const char *name) +{ +int ret; +uint8_t err_code; +struct rte_rawdev *rdev; +struct idxd_rawdev *idxd; + +if (!name) { +IOAT_PMD_ERR ("Invalid device name" ); +return -EINVAL ; + } + + rdev =rte_rawdev_pmd_get_named_dev (name); +if (!rdev) { +IOAT_PMD_ERR ("Invalid device name (%s)" , name); +return -EINVAL ; + } + + idxd = rdev->dev_private; + +/* disable the device */ + err_code =idxd_pci_dev_command (idxd, idxd_disable_dev); +if (err_code) { +IOAT_PMD_ERR ("Error disabling device: code %#x" , err_code); +return err_code; + } +IOAT_PMD_DEBUG ("IDXD Device disabled OK" ); + +/* free device memory */ +if (rdev->dev_private !=NULL ) { +IOAT_PMD_DEBUG ("Freeing device driver memory" ); + rdev->dev_private =NULL ; +rte_free (idxd->public.batch_ring); +rte_free (idxd->public.hdl_ring); +rte_memzone_free (idxd->mz); + } + +/* rte_rawdev_close is called by pmd_release */ + ret =rte_rawdev_pmd_release (rdev); +if (ret) +IOAT_PMD_DEBUG ("Device cleanup failed" ); + +return 0 ; } static int
@@ -40,6 +271,8 @@ idxd_rawdev_remove_pci(struct rte_pci_device *dev)IOAT_PMD_INFO ("Closing %s on NUMA node %d" , name, dev->device.numa_node); + ret =idxd_rawdev_destroy (name); +return ret; }
drivers/raw/ioat/ioat_common.c new file +68 -0
68 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h +33 -0
@@ -14,6 +14,10 @@* @b EXPERIMENTAL: these structures and APIs may change without prior notice */ +#include <rte_spinlock.h> +#include <rte_rawdev_pmd.h> +#include "rte_ioat_rawdev.h" +extern int ioat_pmd_logtype; #define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \
@@ -24,4 +28,33 @@ extern int ioat_pmd_logtype; #define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) #define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) +struct idxd_pci_common { +rte_spinlock_t lk; +volatile struct rte_idxd_bar0 *regs; +volatile struct rte_idxd_wqcfg *wq_regs; +volatile struct rte_idxd_grpcfg *grp_regs; +volatile void *portals; +}; + +struct idxd_rawdev { +struct rte_idxd_rawdev public;/* the public members, must be first */ + +struct rte_rawdev *rawdev; +const struct rte_memzone *mz; +uint8_t qid; +uint16_t max_batches; + +union { +struct idxd_pci_common *pci; + } u; +}; + +extern int idxd_rawdev_create (const char *name,struct rte_device *dev, +const struct idxd_rawdev *idxd, +const struct rte_rawdev_ops *ops); + +extern int idxd_rawdev_close (struct rte_rawdev *dev); + +extern int idxd_rawdev_test (uint16_t dev_id); +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev_test.c +7 -0
@@ -7,6 +7,7 @@#include <rte_mbuf.h> #include "rte_rawdev.h" #include "rte_ioat_rawdev.h" +#include "ioat_private.h" #define MAX_SUPPORTED_RAWDEVS 64#define TEST_SKIPPED 77
@@ -267,3 +268,9 @@ ioat_rawdev_test(uint16_t dev_id)free (ids);return -1 ; } + +int +idxd_rawdev_test(uint16_t dev_id __rte_unused) +{ +return 0 ; +}
drivers/raw/ioat/ioat_spec.h +64 -0
70 lines not on this page · raw message
drivers/raw/ioat/meson.build +1 -0
@@ -6,6 +6,7 @@ reason = 'only supported on x86' sources = files( 'idxd_pci.c', 'idxd_vdev.c', + 'ioat_common.c', 'ioat_rawdev.c', 'ioat_rawdev_test.c') deps += ['bus_pci',
drivers/raw/ioat/rte_ioat_rawdev_fns.h +34 -1
@@ -41,9 +41,20 @@ struct rte_ioat_generic_hw_desc {/** * @internal -* Structure representing a device instance +* Identify the data path to use. +* Must be first field of rte_ioat_rawdev and rte_idxd_rawdev structs +*/ +enum rte_ioat_dev_type { +RTE_IOAT_DEV , +RTE_IDXD_DEV , +}; + +/** +* @internal +* Structure representing an IOAT device instance */ struct rte_ioat_rawdev { +enum rte_ioat_dev_type type;struct rte_rawdev *rawdev;const struct rte_memzone *mz;const struct rte_memzone *desc_mz;
@@ -79,6 +90,28 @@ struct rte_ioat_rawdev {#define RTE_IOAT_CHANSTS_HALTED 0x3#define RTE_IOAT_CHANSTS_ARMED 0x4 +/** +* @internal +* Structure representing an IDXD device instance +*/ +struct rte_idxd_rawdev { +enum rte_ioat_dev_type type; +void *portal;/* address to write the batch descriptor */ + +/* counters to track the batches and the individual op handles */ +uint16_t batch_ring_sz;/* size of batch ring */ +uint16_t hdl_ring_sz;/* size of the user hdl ring */ + +uint16_t next_batch;/* where we write descriptor ops */ +uint16_t next_completed;/* batch where we read completions */ +uint16_t next_ret_hdl;/* the next user hdl to return */ +uint16_t last_completed_hdl;/* the last user hdl that has completed */ +uint16_t next_free_hdl;/* where the handle for next op will go */ + +struct rte_idxd_user_hdl *hdl_ring; +struct rte_idxd_desc_batch *batch_ring; +}; +/* * Enqueue a copy operation onto the ioat device */
Signature · 1 line
2.25.1
PATCH v6 15/25 raw/ioat: create rawdev instances for idxd vdevs
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
For each vdev (DSA work queue) instance, create a rawdev instance.
--- drivers/raw/ioat/idxd_vdev.c | 106 +++++++++++++++++++++++++++++++- drivers/raw/ioat/ioat_private.h | 4 ++ 2 files changed, 109 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_vdev.c +105 -1
@@ -2,6 +2,12 @@* Copyright(c) 2020 Intel Corporation */ +#include <fcntl.h> +#include <unistd.h> +#include <limits.h> +#include <sys/mman.h> + +#include <rte_memzone.h> #include <rte_bus_vdev.h> #include <rte_kvargs.h> #include <rte_string_fns.h>
@@ -24,6 +30,36 @@ struct idxd_vdev_args {uint8_t wq_id; }; +static const struct rte_rawdev_ops idxd_vdev_ops = { + .dev_close = idxd_rawdev_close, + .dev_selftest = idxd_rawdev_test, +}; + +static void * +idxd_vdev_mmap_wq (struct idxd_vdev_args *args) +{ +void *addr; +char path[PATH_MAX ]; +int fd; + +snprintf (path,sizeof (path),"/dev/dsa/wq%u.%u" , + args->device_id, args->wq_id); + fd =open (path,O_RDWR ); +if (fd <0 ) { +IOAT_PMD_ERR ("Failed to open device path" ); +return NULL ; + } + + addr =mmap (NULL ,0x1000 ,PROT_WRITE ,MAP_SHARED , fd,0 ); +close (fd); +if (addr ==MAP_FAILED ) { +IOAT_PMD_ERR ("Failed to mmap device" ); +return NULL ; + } + +return addr; +} +static int idxd_rawdev_parse_wq(const char *key __rte_unused, const char *value,void *extra_args)
@@ -70,10 +106,32 @@ idxd_vdev_parse_params(struct rte_kvargs *kvlist, struct idxd_vdev_args *args)return -EINVAL ; } +static int +idxd_vdev_get_max_batches (struct idxd_vdev_args *args) +{ +char sysfs_path[PATH_MAX ]; +FILE *f; +int ret; + +snprintf (sysfs_path,sizeof (sysfs_path), +"/sys/bus/dsa/devices/wq%u.%u/size" , + args->device_id, args->wq_id); + f =fopen (sysfs_path,"r" ); +if (f ==NULL ) +return -1 ; + +if (fscanf (f,"%d" , &ret) !=1 ) + ret =-1 ; + +fclose (f); +return ret; +} +static int idxd_rawdev_probe_vdev (struct rte_vdev_device *vdev) {struct rte_kvargs *kvlist; +struct idxd_rawdev idxd = {{0 }};/* double {} to avoid error on BSD12 */ struct idxd_vdev_args vdev_args;const char *name;int ret =0 ;
@@ -96,13 +154,32 @@ idxd_rawdev_probe_vdev(struct rte_vdev_device *vdev)return -EINVAL ; } + idxd.qid = vdev_args.wq_id; + idxd.u.vdev.dsa_id = vdev_args.device_id; + idxd.max_batches =idxd_vdev_get_max_batches (&vdev_args); + + idxd.public.portal =idxd_vdev_mmap_wq (&vdev_args); +if (idxd.public.portal ==NULL ) { +IOAT_PMD_ERR ("WQ mmap failed" ); +return -ENOENT ; + } + + ret =idxd_rawdev_create (name, &vdev->device, &idxd, &idxd_vdev_ops); +if (ret) { +IOAT_PMD_ERR ("Failed to create rawdev %s" , name); +return ret; + } +return 0 ; }static int idxd_rawdev_remove_vdev (struct rte_vdev_device *vdev) { +struct idxd_rawdev *idxd;const char *name; +struct rte_rawdev *rdev; +int ret =0 ; name =rte_vdev_device_name (vdev);if (name ==NULL )
@@ -110,7 +187,34 @@ idxd_rawdev_remove_vdev(struct rte_vdev_device *vdev)IOAT_PMD_INFO ("Remove DSA vdev %p" , name); -return 0 ; + rdev =rte_rawdev_pmd_get_named_dev (name); +if (!rdev) { +IOAT_PMD_ERR ("Invalid device name (%s)" , name); +return -EINVAL ; + } + + idxd = rdev->dev_private; + +/* free context and memory */ +if (rdev->dev_private !=NULL ) { +IOAT_PMD_DEBUG ("Freeing device driver memory" ); + rdev->dev_private =NULL ; + +if (munmap (idxd->public.portal,0x1000 ) <0 ) { +IOAT_PMD_ERR ("Error unmapping portal" ); + ret = -errno; + } + +rte_free (idxd->public.batch_ring); +rte_free (idxd->public.hdl_ring); + +rte_memzone_free (idxd->mz); + } + +if (rte_rawdev_pmd_release (rdev)) +IOAT_PMD_ERR ("Device cleanup failed" ); + +return ret; }struct rte_vdev_driver idxd_rawdev_drv_vdev = {
drivers/raw/ioat/ioat_private.h +4 -0
@@ -45,6 +45,10 @@ struct idxd_rawdev {uint16_t max_batches;union { +struct { +unsigned int dsa_id; + } vdev; +struct idxd_pci_common *pci; } u; };
Signature · 1 line
2.25.1
PATCH v6 16/25 raw/ioat: add datapath data structures for idxd devices
In reply to Bruce RichardsonAdd in the relevant data structures for the data path for DSA devices. Also include a device dump function to output the status of each device.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 34 +++++++++++ drivers/raw/ioat/ioat_private.h | 2 + drivers/raw/ioat/ioat_rawdev_test.c | 3 +- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 80 ++++++++++++++++++++++++++ 6 files changed, 120 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -54,6 +54,7 @@ idxd_is_wq_enabled(struct idxd_rawdev *idxd)static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, + .dump = idxd_dev_dump, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -33,6 +33,7 @@ struct idxd_vdev_args {static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, + .dump = idxd_dev_dump, }; static void *
drivers/raw/ioat/ioat_common.c +34 -0
@@ -14,6 +14,36 @@ idxd_rawdev_close(struct rte_rawdev *dev __rte_unused)return 0 ; } +int +idxd_dev_dump (struct rte_rawdev *dev,FILE *f) +{ +struct idxd_rawdev *idxd = dev->dev_private; +struct rte_idxd_rawdev *rte_idxd = &idxd->public; +int i; + +fprintf (f,"Raw Device #%d\n" , dev->dev_id); +fprintf (f,"Driver: %s\n\n" , dev->driver_name); + +fprintf (f,"Portal: %p\n" , rte_idxd->portal); +fprintf (f,"Batch Ring size: %u\n" , rte_idxd->batch_ring_sz); +fprintf (f,"Comp Handle Ring size: %u\n\n" , rte_idxd->hdl_ring_sz); + +fprintf (f,"Next batch: %u\n" , rte_idxd->next_batch); +fprintf (f,"Next batch to be completed: %u\n" , rte_idxd->next_completed); +for (i =0 ; i < rte_idxd->batch_ring_sz; i++) { +struct rte_idxd_desc_batch *b = &rte_idxd->batch_ring[i]; +fprintf (f,"Batch %u @%p: submitted=%u, op_count=%u, hdl_end=%u\n" , + i, b, b->submitted, b->op_count, b->hdl_end); + } + +fprintf (f,"\n" ); +fprintf (f,"Next free hdl: %u\n" , rte_idxd->next_free_hdl); +fprintf (f,"Last completed hdl: %u\n" , rte_idxd->last_completed_hdl); +fprintf (f,"Next returned hdl: %u\n" , rte_idxd->next_ret_hdl); + +return 0 ; +} + int idxd_rawdev_create(const char *name, struct rte_device *dev, const struct idxd_rawdev *base_idxd,
@@ -25,6 +55,10 @@ idxd_rawdev_create(const char *name, struct rte_device *dev,char mz_name[RTE_MEMZONE_NAMESIZE ];int ret =0 ; +RTE_BUILD_BUG_ON (sizeof (struct rte_idxd_hw_desc ) !=64 ); +RTE_BUILD_BUG_ON (offsetof(struct rte_idxd_hw_desc , size) !=32 ); +RTE_BUILD_BUG_ON (sizeof (struct rte_idxd_completion ) !=32 ); +if (!name) {IOAT_PMD_ERR ("Invalid name of the device!" ); ret = -EINVAL ;
drivers/raw/ioat/ioat_private.h +2 -0
@@ -61,4 +61,6 @@ extern int idxd_rawdev_close(struct rte_rawdev *dev);extern int idxd_rawdev_test (uint16_t dev_id); +extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f); +#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev_test.c +2 -1
@@ -270,7 +270,8 @@ ioat_rawdev_test(uint16_t dev_id) }int -idxd_rawdev_test(uint16_t dev_id __rte_unused) +idxd_rawdev_test (uint16_t dev_id) { +rte_rawdev_dump (dev_id, stdout);return 0 ; }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +80 -0
86 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 17/25 raw/ioat: add configure function for idxd devices
In reply to Bruce RichardsonAdd configure function for idxd devices, taking the same parameters as the existing configure function for ioat. The ring_size parameter is used to compute the maximum number of bursts to be supported by the driver, given that the hardware works on individual bursts of descriptors at a time.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 64 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 3 ++ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 1 + 5 files changed, 70 insertions(+)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -55,6 +55,7 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, + .dev_configure = idxd_dev_configure, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -34,6 +34,7 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, + .dev_configure = idxd_dev_configure, }; static void *
drivers/raw/ioat/ioat_common.c +64 -0
70 lines not on this page · raw message
drivers/raw/ioat/ioat_private.h +3 -0
@@ -59,6 +59,9 @@ extern int idxd_rawdev_create(const char *name, struct rte_device *dev,extern int idxd_rawdev_close (struct rte_rawdev *dev); +extern int idxd_dev_configure (const struct rte_rawdev *dev, +rte_rawdev_obj_t config,size_t config_size); +extern int idxd_rawdev_test (uint16_t dev_id);extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);
drivers/raw/ioat/rte_ioat_rawdev_fns.h +1 -0
@@ -187,6 +187,7 @@ struct rte_idxd_rawdev {uint16_t next_ret_hdl;/* the next user hdl to return */ uint16_t last_completed_hdl;/* the last user hdl that has completed */ uint16_t next_free_hdl;/* where the handle for next op will go */ +uint16_t hdls_disable;/* disable tracking completion handles */ struct rte_idxd_user_hdl *hdl_ring;struct rte_idxd_desc_batch *batch_ring;
Signature · 1 line
2.25.1
PATCH v6 18/25 raw/ioat: add start and stop functions for idxd devices
In reply to Bruce RichardsonAdd the start and stop functions for DSA hardware devices using the vfio/uio kernel drivers. For vdevs using the idxd kernel driver, the device must be started using sysfs before the device node appears for vdev use - making start/stop functions in the driver unnecessary.
--- drivers/raw/ioat/idxd_pci.c | 50 +++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
drivers/raw/ioat/idxd_pci.c +50 -0
@@ -51,11 +51,61 @@ idxd_is_wq_enabled(struct idxd_rawdev *idxd)return ((state >>WQ_STATE_SHIFT ) &WQ_STATE_MASK ) ==0x1 ; } +static void +idxd_pci_dev_stop (struct rte_rawdev *dev) +{ +struct idxd_rawdev *idxd = dev->dev_private; +uint8_t err_code; + +if (!idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Work queue %d already disabled" , idxd->qid); +return ; + } + + err_code =idxd_pci_dev_command (idxd, idxd_disable_wq); +if (err_code ||idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Failed disabling work queue %d, error code: %#x" , + idxd->qid, err_code); +return ; + } +IOAT_PMD_DEBUG ("Work queue %d disabled OK" , idxd->qid); +} + +static int +idxd_pci_dev_start (struct rte_rawdev *dev) +{ +struct idxd_rawdev *idxd = dev->dev_private; +uint8_t err_code; + +if (idxd_is_wq_enabled (idxd)) { +IOAT_PMD_WARN ("WQ %d already enabled" , idxd->qid); +return 0 ; + } + +if (idxd->public.batch_ring ==NULL ) { +IOAT_PMD_ERR ("WQ %d has not been fully configured" , idxd->qid); +return -EINVAL ; + } + + err_code =idxd_pci_dev_command (idxd, idxd_enable_wq); +if (err_code || !idxd_is_wq_enabled (idxd)) { +IOAT_PMD_ERR ("Failed enabling work queue %d, error code: %#x" , + idxd->qid, err_code); +return err_code ==0 ?-1 : err_code; + } + +IOAT_PMD_DEBUG ("Work queue %d enabled OK" , idxd->qid); + +return 0 ; +} +static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, + .dev_start = idxd_pci_dev_start, + .dev_stop = idxd_pci_dev_stop, };/* each portal uses 4 x 4k pages */
Signature · 1 line
2.25.1
PATCH v6 19/25 raw/ioat: add data path for idxd devices
In reply to Bruce RichardsonAdd support for doing copies using DSA hardware. This is implemented by just switching on the device type field at the start of the inline functions. Since there is no hardware which will have both device types present this branch will always be predictable after the first call, meaning it has little to no perf penalty.
--- drivers/raw/ioat/ioat_common.c | 1 + drivers/raw/ioat/ioat_rawdev.c | 1 + drivers/raw/ioat/rte_ioat_rawdev_fns.h | 201 +++++++++++++++++++++++-- 3 files changed, 192 insertions(+), 11 deletions(-)
drivers/raw/ioat/ioat_common.c +1 -0
@@ -153,6 +153,7 @@ idxd_rawdev_create(const char *name, struct rte_device *dev, idxd = rawdev->dev_private; *idxd = *base_idxd;/* copy over the main fields already passed in */ + idxd->public.type =RTE_IDXD_DEV ; idxd->rawdev = rawdev; idxd->mz = mz;
drivers/raw/ioat/ioat_rawdev.c +1 -0
@@ -260,6 +260,7 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) rawdev->driver_name = dev->device.driver->name; ioat = rawdev->dev_private; + ioat->type =RTE_IOAT_DEV ; ioat->rawdev = rawdev; ioat->mz = mz; ioat->regs = dev->mem_resource[0 ].addr;
drivers/raw/ioat/rte_ioat_rawdev_fns.h +190 -11
243 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 20/25 raw/ioat: add info function for idxd devices
In reply to Bruce RichardsonAdd the info get function for DSA devices, returning just the ring size info about the device, same as is returned for existing IOAT/CBDMA devices.
--- drivers/raw/ioat/idxd_pci.c | 1 + drivers/raw/ioat/idxd_vdev.c | 1 + drivers/raw/ioat/ioat_common.c | 18 ++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 3 +++ 4 files changed, 23 insertions(+)
drivers/raw/ioat/idxd_pci.c +1 -0
@@ -106,6 +106,7 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_configure = idxd_dev_configure, .dev_start = idxd_pci_dev_start, .dev_stop = idxd_pci_dev_stop, + .dev_info_get = idxd_dev_info_get, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +1 -0
@@ -35,6 +35,7 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_selftest = idxd_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, + .dev_info_get = idxd_dev_info_get, }; static void *
drivers/raw/ioat/ioat_common.c +18 -0
@@ -44,6 +44,24 @@ idxd_dev_dump(struct rte_rawdev *dev, FILE *f)return 0 ; } +int +idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info, +size_t info_size) +{ +struct rte_ioat_rawdev_config *cfg = dev_info; +struct idxd_rawdev *idxd = dev->dev_private; +struct rte_idxd_rawdev *rte_idxd = &idxd->public; + +if (info_size !=sizeof (*cfg)) +return -EINVAL ; + +if (cfg !=NULL ) { + cfg->ring_size = rte_idxd->hdl_ring_sz; + cfg->hdls_disable = rte_idxd->hdls_disable; + } +return 0 ; +} +int idxd_dev_configure (const struct rte_rawdev *dev,rte_rawdev_obj_t config,size_t config_size)
drivers/raw/ioat/ioat_private.h +3 -0
@@ -62,6 +62,9 @@ extern int idxd_rawdev_close(struct rte_rawdev *dev);extern int idxd_dev_configure (const struct rte_rawdev *dev,rte_rawdev_obj_t config,size_t config_size); +extern int idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info, +size_t info_size); +extern int idxd_rawdev_test (uint16_t dev_id);extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);
Signature · 1 line
2.25.1
PATCH v6 21/25 raw/ioat: create separate statistics structure
In reply to Bruce RichardsonRather than having the xstats as fields inside the main driver structure, create a separate structure type for them.
As part of the change, when updating the stats functions referring to the stats by the old path, we can simplify them to use the id to directly index into the stats structure, making the code shorter and simpler.
--- drivers/raw/ioat/ioat_rawdev.c | 40 +++++++------------------- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 30 ++++++++++++------- 2 files changed, 29 insertions(+), 41 deletions(-)
drivers/raw/ioat/ioat_rawdev.c +10 -30
@@ -132,16 +132,14 @@ ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[], uint64_t values[], unsigned int n) {const struct rte_ioat_rawdev *ioat = dev->dev_private; +const uint64_t *stats = (const void *)&ioat->xstats;unsigned int i;for (i =0 ; i < n; i++) { -switch (ids[i]) { -case 0 : values[i] = ioat->enqueue_failed;break ; -case 1 : values[i] = ioat->enqueued;break ; -case 2 : values[i] = ioat->started;break ; -case 3 : values[i] = ioat->completed;break ; -default : values[i] =0 ;break ; - } +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + values[i] = stats[ids[i]]; +else + values[i] =0 ; }return n; }
@@ -167,35 +165,17 @@ static int ioat_xstats_reset(struct rte_rawdev *dev, const uint32_t *ids, uint32_t nb_ids) { struct rte_ioat_rawdev *ioat = dev->dev_private; +uint64_t *stats = (void *)&ioat->xstats;unsigned int i;if (!ids) { - ioat->enqueue_failed =0 ; - ioat->enqueued =0 ; - ioat->started =0 ; - ioat->completed =0 ; +memset (&ioat->xstats,0 ,sizeof (ioat->xstats));return 0 ; } -for (i =0 ; i < nb_ids; i++) { -switch (ids[i]) { -case 0 : - ioat->enqueue_failed =0 ; -break ; -case 1 : - ioat->enqueued =0 ; -break ; -case 2 : - ioat->started =0 ; -break ; -case 3 : - ioat->completed =0 ; -break ; -default : -IOAT_PMD_WARN ("Invalid xstat id - cannot reset value" ); -break ; - } - } +for (i =0 ; i < nb_ids; i++) +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + stats[ids[i]] =0 ;return 0 ; }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +19 -11
@@ -49,17 +49,31 @@ enum rte_ioat_dev_type { RTE_IDXD_DEV, }; +/** +* @internal +* some statistics for tracking, if added/changed update xstats fns +*/ +struct rte_ioat_xstats { +uint64_t enqueue_failed; +uint64_t enqueued; +uint64_t started; +uint64_t completed; +}; +/** * @internal * Structure representing an IOAT device instance */ struct rte_ioat_rawdev { +/* common fields at the top - match those in rte_idxd_rawdev */ enum rte_ioat_dev_type type; +struct rte_ioat_xstats xstats; +struct rte_rawdev *rawdev;const struct rte_memzone *mz;const struct rte_memzone *desc_mz; -volatile uint16_t *doorbell; + volatile uint16_t *doorbell __rte_cache_aligned;phys_addr_t status_addr;phys_addr_t ring_addr;
@@ -72,12 +86,6 @@ struct rte_ioat_rawdev {unsigned short next_read;unsigned short next_write; -/* some statistics for tracking, if added/changed update xstats fns*/ - uint64_t enqueue_failed __rte_cache_aligned; -uint64_t enqueued; -uint64_t started; -uint64_t completed; -/* to report completions, the device will write status back here */ volatile uint64_t status __rte_cache_aligned;
@@ -209,7 +217,7 @@ __ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst,struct rte_ioat_generic_hw_desc *desc;if (space ==0 ) { - ioat->enqueue_failed++; + ioat->xstats.enqueue_failed++;return 0 ; }
@@ -228,7 +236,7 @@ __ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, (int64_t)src_hdl);rte_prefetch0 (&ioat->desc_ring[ioat->next_write & mask]); - ioat->enqueued++; + ioat->xstats.enqueued++;return 1 ; }
@@ -261,7 +269,7 @@ __ioat_perform_ops(int dev_id) .control.completion_update = 1;rte_compiler_barrier (); *ioat->doorbell = ioat->next_write; - ioat->started = ioat->enqueued; + ioat->xstats.started = ioat->xstats.enqueued; }/**
@@ -328,7 +336,7 @@ __ioat_completed_ops(int dev_id, uint8_t max_copies, end: ioat->next_read = read; - ioat->completed += count; + ioat->xstats.completed += count;return count; }
Signature · 1 line
2.25.1
PATCH v6 22/25 raw/ioat: move xstats functions to common file
In reply to Bruce RichardsonThe xstats functions can be used by all ioat devices so move them from the ioat_rawdev.c file to ioat_common.c, and add the function prototypes to the internal header file.
--- drivers/raw/ioat/ioat_common.c | 59 +++++++++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 10 ++++++ drivers/raw/ioat/ioat_rawdev.c | 58 -------------------------------- 3 files changed, 69 insertions(+), 58 deletions(-)
drivers/raw/ioat/ioat_common.c +59 -0
@@ -5,9 +5,68 @@#include <rte_rawdev_pmd.h> #include <rte_memzone.h> #include <rte_common.h> +#include <rte_string_fns.h> #include "ioat_private.h" +static const char *const xstat_names[] = { +"failed_enqueues" ,"successful_enqueues" , +"copies_started" ,"copies_completed" +}; + +int +ioat_xstats_get (const struct rte_rawdev *dev,const unsigned int ids[], +uint64_t values[],unsigned int n) +{ +const struct rte_ioat_rawdev *ioat = dev->dev_private; +const uint64_t *stats = (const void *)&ioat->xstats; +unsigned int i; + +for (i =0 ; i < n; i++) { +if (ids[i] >sizeof (ioat->xstats)/sizeof (*stats)) + values[i] =0 ; +else + values[i] = stats[ids[i]]; + } +return n; +} + +int +ioat_xstats_get_names (const struct rte_rawdev *dev, +struct rte_rawdev_xstats_name *names, +unsigned int size) +{ +unsigned int i; + +RTE_SET_USED (dev); +if (size <RTE_DIM (xstat_names)) +return RTE_DIM (xstat_names); + +for (i =0 ; i <RTE_DIM (xstat_names); i++) +strlcpy (names[i].name, xstat_names[i],sizeof (names[i])); + +return RTE_DIM (xstat_names); +} + +int +ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids,uint32_t nb_ids) +{ +struct rte_ioat_rawdev *ioat = dev->dev_private; +uint64_t *stats = (void *)&ioat->xstats; +unsigned int i; + +if (!ids) { +memset (&ioat->xstats,0 ,sizeof (ioat->xstats)); +return 0 ; + } + +for (i =0 ; i < nb_ids; i++) +if (ids[i] <sizeof (ioat->xstats)/sizeof (*stats)) + stats[ids[i]] =0 ; + +return 0 ; +} +int idxd_rawdev_close(struct rte_rawdev *dev __rte_unused) {
drivers/raw/ioat/ioat_private.h +10 -0
@@ -53,6 +53,16 @@ struct idxd_rawdev { } u; }; +int ioat_xstats_get (const struct rte_rawdev *dev,const unsigned int ids[], +uint64_t values[],unsigned int n); + +int ioat_xstats_get_names (const struct rte_rawdev *dev, +struct rte_rawdev_xstats_name *names, +unsigned int size); + +int ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids, +uint32_t nb_ids); +extern int idxd_rawdev_create (const char *name,struct rte_device *dev,const struct idxd_rawdev *idxd,const struct rte_rawdev_ops *ops);
drivers/raw/ioat/ioat_rawdev.c +0 -58
64 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v6 23/25 raw/ioat: add xstats tracking for idxd devices
In reply to Bruce RichardsonAdd update of the relevant stats for the data path functions and point the overall device struct xstats function pointers to the existing ioat functions.
At this point, all necessary hooks for supporting the existing unit tests are in place so call them for each device.
--- drivers/raw/ioat/idxd_pci.c | 3 +++ drivers/raw/ioat/idxd_vdev.c | 3 +++ drivers/raw/ioat/ioat_rawdev_test.c | 2 +- drivers/raw/ioat/rte_ioat_rawdev_fns.h | 6 ++++++ 4 files changed, 13 insertions(+), 1 deletion(-)
drivers/raw/ioat/idxd_pci.c +3 -0
@@ -107,6 +107,9 @@ static const struct rte_rawdev_ops idxd_pci_ops = { .dev_start = idxd_pci_dev_start, .dev_stop = idxd_pci_dev_stop, .dev_info_get = idxd_dev_info_get, + .xstats_get = ioat_xstats_get, + .xstats_get_names = ioat_xstats_get_names, + .xstats_reset = ioat_xstats_reset, };/* each portal uses 4 x 4k pages */
drivers/raw/ioat/idxd_vdev.c +3 -0
@@ -36,6 +36,9 @@ static const struct rte_rawdev_ops idxd_vdev_ops = { .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_info_get = idxd_dev_info_get, + .xstats_get = ioat_xstats_get, + .xstats_get_names = ioat_xstats_get_names, + .xstats_reset = ioat_xstats_reset, }; static void *
drivers/raw/ioat/ioat_rawdev_test.c +1 -1
@@ -273,5 +273,5 @@ int idxd_rawdev_test(uint16_t dev_id) {rte_rawdev_dump (dev_id, stdout); -return 0 ; +return ioat_rawdev_test (dev_id); }
drivers/raw/ioat/rte_ioat_rawdev_fns.h +6 -0
@@ -184,6 +184,8 @@ struct rte_idxd_user_hdl {*/ struct rte_idxd_rawdev {enum rte_ioat_dev_type type; +struct rte_ioat_xstats xstats; +void *portal;/* address to write the batch descriptor */ /* counters to track the batches and the individual op handles */
@@ -369,9 +371,11 @@ __idxd_write_desc(int dev_id, const struct rte_idxd_hw_desc *desc,if (++idxd->next_free_hdl == idxd->hdl_ring_sz) idxd->next_free_hdl =0 ; + idxd->xstats.enqueued++;return 1 ; failed: + idxd->xstats.enqueue_failed++; rte_errno =ENOSPC ;return 0 ; }
@@ -429,6 +433,7 @@ __idxd_perform_ops(int dev_id)if (++idxd->next_batch == idxd->batch_ring_sz) idxd->next_batch =0 ; + idxd->xstats.started = idxd->xstats.enqueued; }static __rte_always_inline int
@@ -466,6 +471,7 @@ __idxd_completed_ops(int dev_id, uint8_t max_ops, idxd->next_ret_hdl = h_idx; + idxd->xstats.completed += n;return n; }
Signature · 1 line
2.25.1
PATCH v6 24/25 raw/ioat: clean up use of common test function
In reply to Bruce RichardsonNow that all devices can pass the same set of unit tests, eliminate the temporary idxd_rawdev_test function and move the prototype for ioat_rawdev_test to the proper internal header file, to be used by all device instances.
--- drivers/raw/ioat/idxd_pci.c | 2 +- drivers/raw/ioat/idxd_vdev.c | 2 +- drivers/raw/ioat/ioat_private.h | 4 ++-- drivers/raw/ioat/ioat_rawdev.c | 2 -- drivers/raw/ioat/ioat_rawdev_test.c | 7 ------- 5 files changed, 4 insertions(+), 13 deletions(-)
drivers/raw/ioat/idxd_pci.c +1 -1
@@ -101,7 +101,7 @@ idxd_pci_dev_start(struct rte_rawdev *dev)static const struct rte_rawdev_ops idxd_pci_ops = { .dev_close = idxd_rawdev_close, - .dev_selftest = idxd_rawdev_test, + .dev_selftest = ioat_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_start = idxd_pci_dev_start,
drivers/raw/ioat/idxd_vdev.c +1 -1
@@ -32,7 +32,7 @@ struct idxd_vdev_args {static const struct rte_rawdev_ops idxd_vdev_ops = { .dev_close = idxd_rawdev_close, - .dev_selftest = idxd_rawdev_test, + .dev_selftest = ioat_rawdev_test, .dump = idxd_dev_dump, .dev_configure = idxd_dev_configure, .dev_info_get = idxd_dev_info_get,
drivers/raw/ioat/ioat_private.h +2 -2
@@ -63,6 +63,8 @@ int ioat_xstats_get_names(const struct rte_rawdev *dev,int ioat_xstats_reset (struct rte_rawdev *dev,const uint32_t *ids,uint32_t nb_ids); +extern int ioat_rawdev_test (uint16_t dev_id); +extern int idxd_rawdev_create (const char *name,struct rte_device *dev,const struct idxd_rawdev *idxd,const struct rte_rawdev_ops *ops);
@@ -75,8 +77,6 @@ extern int idxd_dev_configure(const struct rte_rawdev *dev,extern int idxd_dev_info_get (struct rte_rawdev *dev,rte_rawdev_obj_t dev_info,size_t info_size); -extern int idxd_rawdev_test (uint16_t dev_id); -extern int idxd_dev_dump (struct rte_rawdev *dev,FILE *f);#endif /* _IOAT_PRIVATE_H_ */
drivers/raw/ioat/ioat_rawdev.c +0 -2
@@ -128,8 +128,6 @@ ioat_dev_close(struct rte_rawdev *dev __rte_unused)return 0 ; } -extern int ioat_rawdev_test (uint16_t dev_id); -static int ioat_rawdev_create (const char *name,struct rte_pci_device *dev) {
drivers/raw/ioat/ioat_rawdev_test.c +0 -7
@@ -268,10 +268,3 @@ ioat_rawdev_test(uint16_t dev_id)free (ids);return -1 ; } - -int -idxd_rawdev_test (uint16_t dev_id) -{ -rte_rawdev_dump (dev_id, stdout); -return ioat_rawdev_test (dev_id); -}
Signature · 1 line
2.25.1
PATCH v6 25/25 raw/ioat: add fill operation
In reply to Bruce RichardsonFrom: Kevin Laatz <[email protected]>
Add fill operation enqueue support for IOAT and IDXD. The fill enqueue is similar to the copy enqueue, but takes a 'pattern' rather than a source address to transfer to the destination address. This patch also includes an additional test case for the new operation type.
--- doc/guides/rawdevs/ioat.rst | 10 ++++ doc/guides/rel_notes/release_20_11.rst | 2 + drivers/raw/ioat/ioat_rawdev_test.c | 62 ++++++++++++++++++++++++ drivers/raw/ioat/rte_ioat_rawdev.h | 27 +++++++++++ drivers/raw/ioat/rte_ioat_rawdev_fns.h | 65 ++++++++++++++++++++++++-- 5 files changed, 161 insertions(+), 5 deletions(-)
doc/guides/rawdevs/ioat.rst +10 -0
@@ -285,6 +285,16 @@ is correct before freeing the data buffers using the returned handles: } +Filling an Area of Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The IOAT driver also has support for the ``fill`` operation, where an area +of memory is overwritten, or filled, with a short pattern of data. +Fill operations can be performed in much the same was as copy operations +described above, just using the ``rte_ioat_enqueue_fill()`` function rather +than the ``rte_ioat_enqueue_copy()`` function. + + Querying Device Statistics ~~~~~~~~~~~~~~~~~~~~~~~~~~~
doc/guides/rel_notes/release_20_11.rst +2 -0
@@ -122,6 +122,8 @@ New Features * Added support for Intel\ |reg| Data Streaming Accelerator hardware. For more information, see https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator + * Added support for the fill operation via the API ``rte_ioat_enqueue_fill()``, + where the hardware fills an area of memory with a repeating pattern. * Added a per-device configuration flag to disable management of user-provided completion handles * Renamed the ``rte_ioat_do_copies()`` API to ``rte_ioat_perform_ops()``, and renamed the ``rte_ioat_completed_copies()`` API to ``rte_ioat_completed_ops()``
drivers/raw/ioat/ioat_rawdev_test.c +62 -0
80 lines not on this page · raw message
drivers/raw/ioat/rte_ioat_rawdev.h +27 -0
@@ -37,6 +37,33 @@ struct rte_ioat_rawdev_config {bool hdls_disable;/**< if set, ignore user-supplied handle params */ }; +/** +* Enqueue a fill operation onto the ioat device +* +* This queues up a fill operation to be performed by hardware, but does not +* trigger hardware to begin that operation. +* +* @param dev_id +* The rawdev device id of the ioat instance +* @param pattern +* The pattern to populate the destination buffer with +* @param dst +* The physical address of the destination buffer +* @param length +* The length of the destination buffer +* @param dst_hdl +* An opaque handle for the destination data, to be returned when this +* operation has been completed and the user polls for the completion details. +* NOTE: If hdls_disable configuration option for the device is set, this +* parameter is ignored. +* @return +* Number of operations enqueued, either 0 or 1 +*/ +static inline int +__rte_experimental +rte_ioat_enqueue_fill (int dev_id,uint64_t pattern,phys_addr_t dst, +unsigned int length,uintptr_t dst_hdl); +/** * Enqueue a copy operation onto the ioat device *
drivers/raw/ioat/rte_ioat_rawdev_fns.h +60 -5
102 lines not on this page · raw message
Signature · 1 line
2.25.1
Thomas Monjalon
AppliedQuoted 10 lines
This patchset adds some small enhancements, some rework and also support for new hardware to the ioat rawdev driver. Most rework and enhancements are largely self-explanatory from the individual patches.
The new hardware support is for the Intel(R) DSA accelerator which will be present in future Intel processors. A description of this new hardware is covered in [1]. Functions specific to the new hardware use the "idxd" prefix, for consistency with the kernel driver.
[1] https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
Applied, thanks