V1 event/dlb add Intel DLB PMD
cover V1 event/dlb add Intel DLB PMD
Hello Jerin and the DPDK community.
The following patchset adds support for a new eventdev PMD. The DLB PMD adds support for the Intel Dynamic Load Balancer (DLB) hardware. The DLB is a PCIe device that provides load-balanced, prioritized scheduling of core-to-core communication. The device consists of queues and arbiters that connect producer and consumer cores, and implements load-balanced queueing features including:
- Lock-free multi-producer/multi-consumer operation.
- Multiple priority levels for varying traffic types.
- 'Direct' traffic (i.e. multi-producer/single-consumer)
- Simple unordered load-balanced distribution.
- Atomic lock-free load balancing across multiple consumers.
- Queue element reordering feature allowing ordered load-balanced distribution.
The DLB hardware supports both load balanced and directed ports and queues. Unlike other eventdev devices already in the repo, not all DLB ports and queues are equally capable. In particular, directed ports are limited to a single link, and must be connected to a directed queue.
Additionally, even though LDB ports may link multiple queues, the number of queues that may be linked is limited by hardware. Another difference is that DLB does not have a straightforward way of carrying the flow_id in the queue elements (QE) that the hardware operates on.
Due to the above restrictions, we would like to extend the eventdev interface in manner that allows our PMD to take full advantage of the DLB hardware, while also adding useful functionality for non-Intel PMDs. Our proposed changes are contained in the first two patches in this patchset.
While reviewing the code, please be aware that this PMD has full control over the DLB hardware. Intel will be extending the DLB PMD in the future (not as part of this first series) with a mode that we refer to as the bifurcated PMD. The bifurcated PMD communicates with a kernel driver to configure the device, ports, and queues, and memory maps device MMIO so datapath operations occur purely in user-space.
The framework to support both the PF PMD and bifurcated PMD exists in this patchset, and is why the iface.[ch] layer is present.
Thank You and Best Regards,
Tim McDaniel
McDaniel, Timothy (27):
eventdev: dlb upstream prerequisites
eventdev: do not pass disable_implicit_release bit to trace macro event/dlb: add shared code version 10.7.9
event/dlb: add make and meson build infrastructure event/dlb: add DLB documentation
event/dlb: add dynamic logging
event/dlb: add private data structures and constants event/dlb: add definitions shared with LKM or shared code event/dlb: inline functions used in multiple files event/dlb: add PFPMD-specific interface layer to shared code event/dlb: add flexible PMD to device interfaces event/dlb: add the PMD's public interfaces
event/dlb: add xstats support
event/dlb: add PMD self-tests
event/dlb: add probe
event/dlb: add infos_get and configure
event/dlb: add queue_def_conf and port_def_conf event/dlb: add queue setup
event/dlb: add port_setup
event/dlb: add port_link
event/dlb: add queue_release and port_release event/dlb: add port_unlink and port_unlinks_in_progress event/dlb: add eventdev_start
event/dlb: add timout_ticks, dump, xstats, and selftest event/dlb: add enqueue and its burst variants event/dlb: add dequeue, dequeue_burst, and variants event/dlb: add eventdev_stop and eventdev_close
app/test-eventdev/evt_common.h | 1 + app/test-eventdev/test_order_atq.c | 4 + app/test-eventdev/test_order_common.c | 6 +- app/test-eventdev/test_order_queue.c | 4 + app/test-eventdev/test_perf_atq.c | 1 + app/test-eventdev/test_perf_queue.c | 1 + app/test-eventdev/test_pipeline_atq.c | 1 + app/test-eventdev/test_pipeline_queue.c | 1 + app/test/test_eventdev.c | 4 +- config/common_base | 17 + config/rte_config.h | 6 + doc/guides/eventdevs/dlb.rst | 497 + drivers/event/Makefile | 5 + drivers/event/dlb/Makefile | 36 + drivers/event/dlb/dlb.c | 4306 +++++++++ drivers/event/dlb/dlb_iface.c | 105 + drivers/event/dlb/dlb_iface.h | 92 + drivers/event/dlb/dlb_inline_fns.h | 80 + drivers/event/dlb/dlb_log.h | 24 + drivers/event/dlb/dlb_priv.h | 595 ++ drivers/event/dlb/dlb_selftest.c | 1628 ++++ drivers/event/dlb/dlb_user.h | 1351 +++ drivers/event/dlb/dlb_xstats.c | 1251 +++ drivers/event/dlb/meson.build | 16 + drivers/event/dlb/pf/base/dlb_hw_types.h | 360 + drivers/event/dlb/pf/base/dlb_mbox.h | 645 ++ drivers/event/dlb/pf/base/dlb_osdep.h | 350 + drivers/event/dlb/pf/base/dlb_osdep_bitmap.h | 449 + drivers/event/dlb/pf/base/dlb_osdep_list.h | 131 + drivers/event/dlb/pf/base/dlb_osdep_types.h | 31 + drivers/event/dlb/pf/base/dlb_regs.h | 2646 ++++++ drivers/event/dlb/pf/base/dlb_resource.c | 9699 ++++++++++++++++++++ drivers/event/dlb/pf/base/dlb_resource.h | 1625 ++++ drivers/event/dlb/pf/base/dlb_user.h | 1084 +++ drivers/event/dlb/pf/dlb_main.c | 670 ++ drivers/event/dlb/pf/dlb_main.h | 90 + drivers/event/dlb/pf/dlb_pf.c | 839 ++ drivers/event/dlb/rte_pmd_dlb.c | 39 + drivers/event/dlb/rte_pmd_dlb.h | 69 + drivers/event/dlb/rte_pmd_dlb_event_version.map | 6 + drivers/event/dpaa2/dpaa2_eventdev.c | 2 +- drivers/event/meson.build | 3 + drivers/event/octeontx/ssovf_evdev.c | 2 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/event/sw/sw_evdev.c | 5 +- drivers/event/sw/sw_evdev_selftest.c | 9 +- .../eventdev_pipeline/pipeline_worker_generic.c | 8 +- examples/eventdev_pipeline/pipeline_worker_tx.c | 3 + examples/l2fwd-event/l2fwd_event_generic.c | 5 +- examples/l2fwd-event/l2fwd_event_internal_port.c | 5 +- examples/l3fwd/l3fwd_event_generic.c | 5 +- examples/l3fwd/l3fwd_event_internal_port.c | 5 +- lib/librte_eal/x86/include/rte_cpuflags.h | 1 + lib/librte_eal/x86/rte_cpuflags.c | 1 + lib/librte_eventdev/rte_event_crypto_adapter.c | 7 +- lib/librte_eventdev/rte_event_eth_tx_adapter.c | 16 +- lib/librte_eventdev/rte_eventdev.c | 72 +- lib/librte_eventdev/rte_eventdev.h | 51 +- lib/librte_eventdev/rte_eventdev_pmd_pci.h | 54 + lib/librte_eventdev/rte_eventdev_trace.h | 15 +- 60 files changed, 28987 insertions(+), 49 deletions(-) create mode 100644 doc/guides/eventdevs/dlb.rst create mode 100644 drivers/event/dlb/Makefile create mode 100644 drivers/event/dlb/dlb.c create mode 100644 drivers/event/dlb/dlb_iface.c create mode 100644 drivers/event/dlb/dlb_iface.h create mode 100644 drivers/event/dlb/dlb_inline_fns.h create mode 100644 drivers/event/dlb/dlb_log.h create mode 100644 drivers/event/dlb/dlb_priv.h create mode 100644 drivers/event/dlb/dlb_selftest.c create mode 100644 drivers/event/dlb/dlb_user.h create mode 100644 drivers/event/dlb/dlb_xstats.c create mode 100644 drivers/event/dlb/meson.build create mode 100644 drivers/event/dlb/pf/base/dlb_hw_types.h create mode 100644 drivers/event/dlb/pf/base/dlb_mbox.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_bitmap.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_list.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_types.h create mode 100644 drivers/event/dlb/pf/base/dlb_regs.h create mode 100644 drivers/event/dlb/pf/base/dlb_resource.c create mode 100644 drivers/event/dlb/pf/base/dlb_resource.h create mode 100644 drivers/event/dlb/pf/base/dlb_user.h create mode 100644 drivers/event/dlb/pf/dlb_main.c create mode 100644 drivers/event/dlb/pf/dlb_main.h create mode 100644 drivers/event/dlb/pf/dlb_pf.c create mode 100644 drivers/event/dlb/rte_pmd_dlb.c create mode 100644 drivers/event/dlb/rte_pmd_dlb.h create mode 100644 drivers/event/dlb/rte_pmd_dlb_event_version.map
Signature · 1 line
2.13.6
PATCH 2/27 eventdev: do not pass disable_implicit_release bit to trace macro
The disable implicit_release indicator is now a bitfield, so it must be converted to uint8_t when calling the tracepoint macro.
--- lib/librte_eventdev/rte_event_crypto_adapter.c | 7 ++++++- lib/librte_eventdev/rte_event_eth_tx_adapter.c | 14 ++++++++++++-- lib/librte_eventdev/rte_eventdev.c | 10 +++++++--- lib/librte_eventdev/rte_eventdev_trace.h | 15 +++++++++------ 4 files changed, 34 insertions(+), 12 deletions(-)
lib/librte_eventdev/rte_event_crypto_adapter.c +6 -1
@@ -205,8 +205,10 @@ rte_event_crypto_adapter_create_ext(uint8_t id, uint8_t dev_id,struct rte_event_crypto_adapter *adapter;char mem_name[CRYPTO_ADAPTER_NAME_LEN ];struct rte_event_dev_info dev_info; +struct rte_event_port_conf *port_conf = conf_arg;int socket_id;uint8_t i; +uint8_t disable_impl_rel;int ret;EVENT_CRYPTO_ADAPTER_ID_VALID_OR_ERR_RET (id, -EINVAL );
@@ -268,8 +270,11 @@ rte_event_crypto_adapter_create_ext(uint8_t id, uint8_t dev_id, event_crypto_adapter[id] = adapter; + disable_impl_rel = !!(port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); +rte_eventdev_trace_crypto_adapter_create (id, dev_id, adapter, conf_arg, - mode); + disable_impl_rel, mode);return 0 ; }
lib/librte_eventdev/rte_event_eth_tx_adapter.c +12 -2
@@ -910,6 +910,7 @@ rte_event_eth_tx_adapter_create(uint8_t id, uint8_t dev_id, struct rte_event_port_conf *port_conf) {struct rte_eventdev *dev; +uint8_t disable_impl_rel;int ret;if (port_conf ==NULL )
@@ -943,8 +944,12 @@ rte_event_eth_tx_adapter_create(uint8_t id, uint8_t dev_id, txa_dev_id_array[id] =TXA_INVALID_DEV_ID ;return ret; } + + disable_impl_rel = !!(port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); +rte_eventdev_trace_eth_tx_adapter_create (id, dev_id,NULL , port_conf, - ret); + disable_impl_rel, ret); txa_dev_id_array[id] = dev_id;return 0 ; }
@@ -955,6 +960,8 @@ rte_event_eth_tx_adapter_create_ext(uint8_t id, uint8_t dev_id, void *conf_arg) {struct rte_eventdev *dev; +struct rte_event_port_conf *port_conf = conf_arg; +uint8_t disable_impl_rel;int ret;RTE_EVENT_ETH_TX_ADAPTER_ID_VALID_OR_ERR_RET (id, -EINVAL );
@@ -986,8 +993,11 @@ rte_event_eth_tx_adapter_create_ext(uint8_t id, uint8_t dev_id,return ret; } + disable_impl_rel = !!(port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); +rte_eventdev_trace_eth_tx_adapter_create (id, dev_id, conf_cb, conf_arg, - ret); + disable_impl_rel, ret); txa_dev_id_array[id] = dev_id;return 0 ; }
lib/librte_eventdev/rte_eventdev.c +7 -3
@@ -736,6 +736,7 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id, {struct rte_eventdev *dev;struct rte_event_port_conf def_conf; +uint8_t disable_impl_rel;int diag;RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET (dev_id, -EINVAL );
@@ -812,7 +813,12 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id,if (!diag) diag =rte_event_port_unlink (dev_id, port_id,NULL ,0 ); -rte_eventdev_trace_port_setup (dev_id, port_id, port_conf, diag); + disable_impl_rel = !!(port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); + +rte_eventdev_trace_port_setup (dev_id, port_id, port_conf, + disable_impl_rel, diag); +if (diag <0 )return diag;
@@ -1422,10 +1428,8 @@ rte_event_pmd_allocate(const char *name, int socket_id) eventdev->data = eventdev_data;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { -strlcpy (eventdev->data->name, name,RTE_EVENTDEV_NAME_MAX_LEN ); - eventdev->data->dev_id = dev_id; eventdev->data->socket_id = socket_id; eventdev->data->dev_started =0 ;
lib/librte_eventdev/rte_eventdev_trace.h +9 -6
@@ -53,13 +53,14 @@ RTE_TRACE_POINT(RTE_TRACE_POINT ( rte_eventdev_trace_port_setup, RTE_TRACE_POINT_ARGS(uint8_t dev_id, uint8_t port_id, - const struct rte_event_port_conf *port_conf, int rc), + const struct rte_event_port_conf *port_conf, + uint8_t disable_impl_rel, int rc), rte_trace_point_emit_u8(dev_id);rte_trace_point_emit_u8 (port_id);rte_trace_point_emit_i32 (port_conf->new_event_threshold);rte_trace_point_emit_u16 (port_conf->dequeue_depth);rte_trace_point_emit_u16 (port_conf->enqueue_depth); -rte_trace_point_emit_u8 (port_conf->disable_implicit_release); +rte_trace_point_emit_u8 (disable_impl_rel);rte_trace_point_emit_int (rc); )
@@ -159,13 +160,14 @@ RTE_TRACE_POINT(RTE_TRACE_POINT ( rte_eventdev_trace_eth_tx_adapter_create, RTE_TRACE_POINT_ARGS(uint8_t adptr_id, uint8_t dev_id, void *conf_cb, - struct rte_event_port_conf *port_conf, int rc), + struct rte_event_port_conf *port_conf, uint8_t disable_impl_rel, + int rc), rte_trace_point_emit_u8(adptr_id);rte_trace_point_emit_u8 (dev_id);rte_trace_point_emit_i32 (port_conf->new_event_threshold);rte_trace_point_emit_u16 (port_conf->dequeue_depth);rte_trace_point_emit_u16 (port_conf->enqueue_depth); -rte_trace_point_emit_u8 (port_conf->disable_implicit_release); +rte_trace_point_emit_u8 (disable_impl_rel);rte_trace_point_emit_ptr (conf_cb);rte_trace_point_emit_int (rc); )
@@ -249,7 +251,8 @@ RTE_TRACE_POINT(RTE_TRACE_POINT ( rte_eventdev_trace_crypto_adapter_create, RTE_TRACE_POINT_ARGS(uint8_t adptr_id, uint8_t dev_id, void *adapter, - struct rte_event_port_conf *port_conf, uint8_t mode), + struct rte_event_port_conf *port_conf, uint8_t disable_impl_rel, + uint8_t mode), rte_trace_point_emit_u8(adptr_id);rte_trace_point_emit_u8 (dev_id);rte_trace_point_emit_ptr (adapter);
@@ -257,7 +260,7 @@ RTE_TRACE_POINT(rte_trace_point_emit_i32 (port_conf->new_event_threshold);rte_trace_point_emit_u16 (port_conf->dequeue_depth);rte_trace_point_emit_u16 (port_conf->enqueue_depth); -rte_trace_point_emit_u8 (port_conf->disable_implicit_release); +rte_trace_point_emit_u8 (disable_impl_rel); ) RTE_TRACE_POINT(
Signature · 1 line
2.13.6
PATCH 1/27 eventdev: dlb upstream prerequisites
In reply to McDaniel, TimothyThe DLB hardware does not conform exactly to the eventdev interface.
1) It has a limit on the number of queues that may be linked to a port.
2) Some ports a further restricted to a maximum of 1 linked queue.
3) It does not (currently) have the ability to carry the flow_id as part of the event (QE) payload.
Due to the above, we would like to propose the following enhancements.
1) Add new fields to the rte_event_dev_info struct. These fields allow the device to advertize its capabilities so that applications can take the appropriate actions based on those capabilities.
struct rte_event_dev_info {
uint32_t max_event_port_links;
/**< Maximum number of queues that can be linked to a single event
* port by this device.
*/
uint8_t max_single_link_event_port_queue_pairs;
/**< Maximum number of event ports and queues that are optimized for
* (and only capable of) single-link configurations supported by this
* device. These ports and queues are not accounted for in
* max_event_ports or max_event_queues.
*/
}2) Add a new field to the rte_event_dev_config struct. This field allows the application to specify how many of its ports are limited to a single link, or will be used in single link mode.
/** Event device configuration structure */
struct rte_event_dev_config {
uint8_t nb_single_link_event_port_queues;
/**< Number of event ports and queues that will be singly-linked to
* each other. These are a subset of the overall event ports and
* queues; this value cannot exceed *nb_event_ports* or
* *nb_event_queues*. If the device has ports and queues that are
* optimized for single-link usage, this field is a hint for how many
* to allocate; otherwise, regular event ports and queues can be used.
*/
}3) Replace the dedicated implicit_release_disabled field with a bit field of explicit port capabilities. The implicit_release_disable functionality is assiged to one bit, and a port-is-single-link-only attribute is assigned to other, with the remaining bits available for future assignment.
* Event port configuration bitmap flags */
#define RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL (1ULL << 0)
/**< Configure the port not to release outstanding events in
* rte_event_dev_dequeue_burst(). If set, all events received through
* the port must be explicitly released with RTE_EVENT_OP_RELEASE or
* RTE_EVENT_OP_FORWARD. Must be unset if the device is not
* RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable.
*/
#define RTE_EVENT_PORT_CFG_SINGLE_LINK (1ULL << 1)
/**< This event port links only to a single event queue.
*
* @see rte_event_port_setup(), rte_event_port_link()
*/
#define RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE 3
/**
* The implicit release disable attribute of the port
*/
struct rte_event_port_conf {
uint32_t event_port_cfg; /**< Port cfg flags(EVENT_PORT_CFG_) */
}4) Add UMWAIT/UMONITOR bit to rte_cpuflags
5) Added a new API that is useful for probing PCI devices.
/**
* @internal
* Wrapper for use by pci drivers as a .probe function to attach to a event
* interface. Same as rte_event_pmd_pci_probe, except caller can specify
* the name.
*/
static inline int
rte_event_pmd_pci_probe_named(struct rte_pci_driver *pci_drv,
struct rte_pci_device *pci_dev, size_t private_data_size, eventdev_pmd_pci_callback_t devinit, const char *name);
--- app/test-eventdev/evt_common.h | 1 + app/test-eventdev/test_order_atq.c | 4 ++ app/test-eventdev/test_order_common.c | 6 ++- app/test-eventdev/test_order_queue.c | 4 ++ app/test-eventdev/test_perf_atq.c | 1 + app/test-eventdev/test_perf_queue.c | 1 + app/test-eventdev/test_pipeline_atq.c | 1 + app/test-eventdev/test_pipeline_queue.c | 1 + app/test/test_eventdev.c | 4 +- drivers/event/dpaa2/dpaa2_eventdev.c | 2 +- drivers/event/octeontx/ssovf_evdev.c | 2 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/event/sw/sw_evdev.c | 5 +- drivers/event/sw/sw_evdev_selftest.c | 9 ++-- .../eventdev_pipeline/pipeline_worker_generic.c | 8 ++- examples/eventdev_pipeline/pipeline_worker_tx.c | 3 ++ examples/l2fwd-event/l2fwd_event_generic.c | 5 +- examples/l2fwd-event/l2fwd_event_internal_port.c | 5 +- examples/l3fwd/l3fwd_event_generic.c | 5 +- examples/l3fwd/l3fwd_event_internal_port.c | 5 +- lib/librte_eal/x86/include/rte_cpuflags.h | 1 + lib/librte_eal/x86/rte_cpuflags.c | 1 + lib/librte_eventdev/rte_event_eth_tx_adapter.c | 2 +- lib/librte_eventdev/rte_eventdev.c | 62 +++++++++++++++++++--- lib/librte_eventdev/rte_eventdev.h | 51 +++++++++++++++--- lib/librte_eventdev/rte_eventdev_pmd_pci.h | 54 +++++++++++++++++++ 26 files changed, 208 insertions(+), 37 deletions(-)
app/test-eventdev/evt_common.h +1 -0
@@ -169,6 +169,7 @@ evt_configure_eventdev(struct evt_options *opt, uint8_t nb_queues, .dequeue_timeout_ns = opt->deq_tmo_nsec, .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues = 0, .nb_events_limit = info.max_num_events, .nb_event_queue_flows = opt->nb_flows, .nb_event_port_dequeue_depth =
app/test-eventdev/test_order_atq.c +4 -0
@@ -34,6 +34,8 @@ order_atq_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.sub_event_type ==0 ) {/* stage 0 from producer */ order_atq_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)
@@ -68,6 +70,8 @@ order_atq_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].sub_event_type ==0 ) {/*stage 0 */ order_atq_process_stage_0 (&ev[i]); }else if (ev[i].sub_event_type ==1 ) {/* stage 1 */
app/test-eventdev/test_order_common.c +5 -1
@@ -49,6 +49,7 @@ order_producer(void *arg)const uint32_t flow = (uintptr_t )m % nb_flows;/* Maintain seq number per flow */ m->seqn = producer_flow_seq[flow]++; + m->udata64 = flow; ev.flow_id = flow; ev.mbuf = m;
@@ -318,10 +319,11 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, opt->wkr_deq_dep = dev_info.max_event_port_dequeue_depth;/* port configuration */ -const struct rte_event_port_conf p_conf = { +struct rte_event_port_conf p_conf = { .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg =0 , };/* setup one port per worker, linking to all queues */
@@ -351,6 +353,8 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, p->queue_id =0 ; p->t = t; + p_conf.new_event_threshold /=2 ; + ret =rte_event_port_setup (opt->dev_id, port, &p_conf);if (ret) {evt_err ("failed to setup producer port %d" , port);
app/test-eventdev/test_order_queue.c +4 -0
@@ -34,6 +34,8 @@ order_queue_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)
@@ -68,6 +70,8 @@ order_queue_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev[i]); }else if (ev[i].queue_id ==1 ) {/* from atomic queue */
app/test-eventdev/test_perf_atq.c +1 -0
@@ -204,6 +204,7 @@ perf_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, 1 /* stride */, nb_queues,
app/test-eventdev/test_perf_queue.c +1 -0
@@ -219,6 +219,7 @@ perf_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, nb_stages /* stride */,
app/test-eventdev/test_pipeline_atq.c +1 -0
@@ -356,6 +356,7 @@ pipeline_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port)
app/test-eventdev/test_pipeline_queue.c +1 -0
@@ -379,6 +379,7 @@ pipeline_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port) {
app/test/test_eventdev.c +2 -2
@@ -559,10 +559,10 @@ test_eventdev_port_setup(void)if (!(info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE )) { pconf.enqueue_depth = info.max_event_port_enqueue_depth; - pconf.disable_implicit_release =1 ; + pconf.event_port_cfg =RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; ret =rte_event_port_setup (TEST_DEV_ID ,0 , &pconf);TEST_ASSERT (ret == -EINVAL ,"Expected -EINVAL, %d" , ret); - pconf.disable_implicit_release =0 ; + pconf.event_port_cfg =0 ; } ret = rte_event_port_setup(TEST_DEV_ID, info.max_event_ports,
drivers/event/dpaa2/dpaa2_eventdev.c +1 -1
@@ -537,7 +537,7 @@ dpaa2_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id,DPAA2_EVENT_MAX_PORT_DEQUEUE_DEPTH ; port_conf->enqueue_depth =DPAA2_EVENT_MAX_PORT_ENQUEUE_DEPTH ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static int
drivers/event/octeontx/ssovf_evdev.c +1 -1
@@ -224,7 +224,7 @@ ssovf_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold = edev->max_num_events; port_conf->dequeue_depth =1 ; port_conf->enqueue_depth =1 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static void
drivers/event/skeleton/skeleton_eventdev.c +1 -1
@@ -209,7 +209,7 @@ skeleton_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =32 *1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static void
drivers/event/sw/sw_evdev.c +3 -2
@@ -175,7 +175,8 @@ sw_port_setup(struct rte_eventdev *dev, uint8_t port_id, } p->inflight_max = conf->new_event_threshold; - p->implicit_release = !conf->disable_implicit_release; + p->implicit_release = !(conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL );/* check if ring exists, same as rx_worker above */ snprintf(buf, sizeof(buf), "sw%d_p%u, %s", dev->data->dev_id,
@@ -508,7 +509,7 @@ sw_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static int
drivers/event/sw/sw_evdev_selftest.c +5 -4
@@ -172,7 +172,7 @@ create_ports(struct test *t, int num_ports) .new_event_threshold = 1024, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (num_ports >MAX_PORTS )return -1 ;
@@ -1227,7 +1227,7 @@ port_reconfig_credits(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);
@@ -1317,7 +1317,7 @@ port_single_lb_reconfig(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);
@@ -3079,7 +3079,8 @@ worker_loopback(struct test *t, uint8_t disable_implicit_release)* only be initialized once - and this needs to be set for multiple runs */ conf.new_event_threshold =512 ; - conf.disable_implicit_release = disable_implicit_release; + conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (rte_event_port_setup (evdev,0 , &conf) <0 ) {printf ("Error setting up RX port\n" );
examples/eventdev_pipeline/pipeline_worker_generic.c +6 -2
@@ -129,6 +129,7 @@ setup_eventdev_generic(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =1 , .nb_events_limit =4096 , .nb_event_queue_flows =1024 , .nb_event_port_dequeue_depth =128 ,
@@ -138,12 +139,13 @@ setup_eventdev_generic(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type, .priority =RTE_EVENT_DEV_PRIORITY_NORMAL , .nb_atomic_flows =1024 , - .nb_atomic_order_sequences =1024 , + .nb_atomic_order_sequences =1024 , };struct rte_event_queue_conf tx_q_conf = { .priority =RTE_EVENT_DEV_PRIORITY_HIGHEST ,
@@ -167,7 +169,8 @@ setup_eventdev_generic(struct worker_data *worker_data) disable_implicit_release = (dev_info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE ); - wkr_p_conf.disable_implicit_release = disable_implicit_release; + wkr_p_conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (dev_info.max_num_events < config.nb_events_limit) config.nb_events_limit = dev_info.max_num_events;
@@ -417,6 +420,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };if (adptr_p_conf.new_event_threshold > dev_info.max_num_events)
examples/eventdev_pipeline/pipeline_worker_tx.c +3 -0
@@ -436,6 +436,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =0 , .nb_events_limit =4096 , .nb_event_queue_flows =1024 , .nb_event_port_dequeue_depth =128 ,
@@ -445,6 +446,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type,
@@ -746,6 +748,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };init_ports (nb_ports);
examples/l2fwd-event/l2fwd_event_generic.c +3 -2
@@ -126,8 +126,9 @@ l2fwd_event_port_setup_generic(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;
examples/l2fwd-event/l2fwd_event_internal_port.c +3 -2
@@ -123,8 +123,9 @@ l2fwd_event_port_setup_internal_port(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {
examples/l3fwd/l3fwd_event_generic.c +3 -2
@@ -115,8 +115,9 @@ l3fwd_event_port_setup_generic(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;
examples/l3fwd/l3fwd_event_internal_port.c +3 -2
@@ -113,8 +113,9 @@ l3fwd_event_port_setup_internal_port(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {
lib/librte_eal/x86/include/rte_cpuflags.h +1 -0
@@ -130,6 +130,7 @@ enum rte_cpu_flag_t {RTE_CPUFLAG_CLDEMOTE ,/**< Cache Line Demote */ RTE_CPUFLAG_MOVDIRI ,/**< Direct Store Instructions */ RTE_CPUFLAG_MOVDIR64B ,/**< Direct Store Instructions 64B */ +RTE_CPUFLAG_UMWAIT ,/**< UMONITOR/UMWAIT */ RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection *//* The last item */
lib/librte_eal/x86/rte_cpuflags.c +1 -0
@@ -137,6 +137,7 @@ const struct feature_entry rte_cpu_feature_table[] = { FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25) FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27) FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28) + FEAT_DEF(UMWAIT, 0x00000007, 0, RTE_REG_ECX, 5) FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8) };
lib/librte_eventdev/rte_event_eth_tx_adapter.c +1 -1
@@ -286,7 +286,7 @@ txa_service_conf_cb(uint8_t __rte_unused id, uint8_t dev_id,return ret; } - pc->disable_implicit_release =0 ; + pc->event_port_cfg =0 ; ret =rte_event_port_setup (dev_id, port_id, pc);if (ret) { RTE_EDEV_LOG_ERR("failed to setup event port %u\n",
lib/librte_eventdev/rte_eventdev.c +55 -7
92 lines not on this page · raw message
lib/librte_eventdev/rte_eventdev.h +44 -7
93 lines not on this page · raw message
lib/librte_eventdev/rte_eventdev_pmd_pci.h +54 -0
60 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH 4/27 event/dlb: add make and meson build infrastructure
In reply to McDaniel, Timothy--- config/common_base | 17 +++++++++++++++++ config/rte_config.h | 6 ++++++ drivers/event/Makefile | 5 +++++ drivers/event/dlb/Makefile | 36 ++++++++++++++++++++++++++++++++++++ drivers/event/dlb/meson.build | 16 ++++++++++++++++ drivers/event/meson.build | 3 +++ 6 files changed, 83 insertions(+) create mode 100644 drivers/event/dlb/Makefile create mode 100644 drivers/event/dlb/meson.build
config/common_base +17 -0
@@ -789,6 +789,23 @@ CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=n CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=n # +# Compile PMD for dlb event device +# +CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV=y +# Specify an upper bound on cycles between each CQ access when hard polling +CONFIG_RTE_LIBRTE_PMD_DLB_POLL_INTERVAL=1000 +# Specify control state to use when executing UMWAIT instruction. +# If 0, then use CO.2, which has a slower wakeup time, but greater +# power savings. +# If 1, then use C0.1, which has a faster wakeup time, but smaller +# power savings. +CONFIG_RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE=0 +# Suppress statistics calculation +CONFIG_RTE_LIBRTE_PMD_DLB_QUELL_STATS=n +# Specify software credit quanta +CONFIG_RTE_LIBRTE_PMD_DLB_SW_CREDIT_QUANTA=32 + +# # Compile raw device support # EXPERIMENTAL: API may change without prior notice #
config/rte_config.h +6 -0
@@ -133,4 +133,10 @@/* QEDE PMD defines */ #define RTE_LIBRTE_QEDE_FW "" +/* DLB PMD defines */ +#define RTE_LIBRTE_PMD_DLB_POLL_INTERVAL 1000 +#define RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE 0 +#undef RTE_LIBRTE_PMD_DLB_QUELL_STATS +#define RTE_LIBRTE_PMD_DLB_SW_CREDIT_QUANTA 32 +#endif /* _RTE_CONFIG_H_ */
drivers/event/Makefile +5 -0
@@ -16,5 +16,10 @@ ifeq ($(CONFIG_RTE_EAL_VFIO)$(CONFIG_RTE_LIBRTE_FSLMC_BUS),yy) DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV) += dpaa2 endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_OPDL_EVENTDEV) += opdl +# DLB limited to x86 architectures +ifneq ($(filter y,$(CONFIG_RTE_ARCH_X86_64) \ + $(CONFIG_RTE_ARCH_X86)),) +DIRS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += dlb +endif include $(RTE_SDK)/mk/rte.subdir.mk
drivers/event/dlb/Makefile new file +36 -0
@@ -0,0 +1,36 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2016-2020 Intel Corporation. + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_dlb_eventdev.a + +# build flags +CFLAGS += -O3 -pthread +CFLAGS += $(WERROR_FLAGS) +# rte_mp_ APIs are still marked experimental +CFLAGS += -DALLOW_EXPERIMENTAL_API +LDLIBS += -lrte_eal -lrte_eventdev -lrte_mbuf -lrte_kvargs -lrte_ring +LDLIBS += -lrte_mempool -lpthread -lrte_pci -lrte_bus_pci + +# library version +LIBABIVER := 1 + +# versioning export map +EXPORT_MAP := rte_pmd_dlb_event_version.map + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += dlb.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += dlb_iface.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += dlb_xstats.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += rte_pmd_dlb.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += dlb_selftest.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += pf/dlb_pf.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += pf/dlb_main.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_DLB_EVENTDEV) += pf/base/dlb_resource.c + +# export include files +SYMLINK-y-include += rte_pmd_dlb.h + +include $(RTE_SDK)/mk/rte.lib.mk
drivers/event/dlb/meson.build new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019-2020 Intel Corporation + +sources = files('dlb.c', + 'dlb_iface.c', + 'dlb_xstats.c', + 'rte_pmd_dlb.c', + 'dlb_selftest.c', + 'pf/dlb_pf.c', + 'pf/dlb_main.c', + 'pf/base/dlb_resource.c' +) + +allow_experimental_apis = true +deps += ['mbuf', 'mempool', 'ring', 'pci', 'bus_pci'] +install_headers('rte_pmd_dlb.h')
drivers/event/meson.build +3 -0
@@ -6,6 +6,9 @@ if not (toolchain == 'gcc' and cc.version().version_compare('<4.8.6') and dpdk_conf.has('RTE_ARCH_ARM64')) drivers += 'octeontx' endif +if dpdk_conf.has('RTE_ARCH_X86_64') or dpdk_conf.has('RTE_ARCH_X86') + drivers += 'dlb' +endif std_deps = ['eventdev', 'kvargs'] config_flag_fmt = 'RTE_LIBRTE_@0@_EVENTDEV_PMD' driver_name_fmt = 'rte_pmd_@0@_event'
Signature · 1 line
2.13.6
PATCH 3/27 event/dlb: add shared code version 10.7.9
In reply to McDaniel, TimothyThe DLB shared code is auto generated by Intel, and is being committed here so that it can be built in the DPDK environment. The shared code should not be modified. The shared code must be present in order to successfully build the DLB PMD.
--- drivers/event/dlb/pf/base/dlb_hw_types.h | 360 + drivers/event/dlb/pf/base/dlb_mbox.h | 645 ++ drivers/event/dlb/pf/base/dlb_osdep.h | 350 + drivers/event/dlb/pf/base/dlb_osdep_bitmap.h | 449 ++ drivers/event/dlb/pf/base/dlb_osdep_list.h | 131 + drivers/event/dlb/pf/base/dlb_osdep_types.h | 31 + drivers/event/dlb/pf/base/dlb_regs.h | 2646 +++++++ drivers/event/dlb/pf/base/dlb_resource.c | 9699 ++++++++++++++++++++++++++ drivers/event/dlb/pf/base/dlb_resource.h | 1625 +++++ drivers/event/dlb/pf/base/dlb_user.h | 1084 +++ 10 files changed, 17020 insertions(+) create mode 100644 drivers/event/dlb/pf/base/dlb_hw_types.h create mode 100644 drivers/event/dlb/pf/base/dlb_mbox.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_bitmap.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_list.h create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_types.h create mode 100644 drivers/event/dlb/pf/base/dlb_regs.h create mode 100644 drivers/event/dlb/pf/base/dlb_resource.c create mode 100644 drivers/event/dlb/pf/base/dlb_resource.h create mode 100644 drivers/event/dlb/pf/base/dlb_user.h
drivers/event/dlb/pf/base/dlb_hw_types.h new file +360 -0
@@ -0,0 +1,360 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#ifndef __DLB_HW_TYPES_H +#define __DLB_HW_TYPES_H + +#include "dlb_user.h" +#include "dlb_osdep_types.h" +#include "dlb_osdep_list.h" + +#define DLB_MAX_NUM_VFS 16 +#define DLB_MAX_NUM_DOMAINS 32 +#define DLB_MAX_NUM_LDB_QUEUES 128 +#define DLB_MAX_NUM_LDB_PORTS 64 +#define DLB_MAX_NUM_DIR_PORTS 128 +#define DLB_MAX_NUM_LDB_CREDITS 16384 +#define DLB_MAX_NUM_DIR_CREDITS 4096 +#define DLB_MAX_NUM_LDB_CREDIT_POOLS 64 +#define DLB_MAX_NUM_DIR_CREDIT_POOLS 64 +#define DLB_MAX_NUM_HIST_LIST_ENTRIES 5120 +#define DLB_MAX_NUM_AQOS_ENTRIES 2048 +#define DLB_MAX_NUM_TOTAL_OUTSTANDING_COMPLETIONS 4096 +#define DLB_MAX_NUM_QIDS_PER_LDB_CQ 8 +#define DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS 4 +#define DLB_MAX_NUM_SEQUENCE_NUMBER_MODES 6 +#define DLB_QID_PRIORITIES 8 +#define DLB_NUM_ARB_WEIGHTS 8 +#define DLB_MAX_WEIGHT 255 +#define DLB_MAX_PORT_CREDIT_QUANTUM 1023 +#define DLB_MAX_CQ_COMP_CHECK_LOOPS 409600 +#define DLB_MAX_QID_EMPTY_CHECK_LOOPS (32 * 64 * 1024 * (800 / 30)) +#define DLB_HZ 800000000 + +/* Used for DLB A-stepping workaround for hardware write buffer lock up issue */ +#define DLB_A_STEP_MAX_PORTS 128 + +#define DLB_PF_DEV_ID 0x270B +#define DLB_VF_DEV_ID 0x270C + +/* Interrupt related macros */ +#define DLB_PF_NUM_NON_CQ_INTERRUPT_VECTORS 8 +#define DLB_PF_NUM_CQ_INTERRUPT_VECTORS 64 +#define DLB_PF_TOTAL_NUM_INTERRUPT_VECTORS \ + (DLB_PF_NUM_NON_CQ_INTERRUPT_VECTORS + \ + DLB_PF_NUM_CQ_INTERRUPT_VECTORS) +#define DLB_PF_NUM_COMPRESSED_MODE_VECTORS \ + (DLB_PF_NUM_NON_CQ_INTERRUPT_VECTORS + 1) +#define DLB_PF_NUM_PACKED_MODE_VECTORS DLB_PF_TOTAL_NUM_INTERRUPT_VECTORS +#define DLB_PF_COMPRESSED_MODE_CQ_VECTOR_ID DLB_PF_NUM_NON_CQ_INTERRUPT_VECTORS + +#define DLB_VF_NUM_NON_CQ_INTERRUPT_VECTORS 1 +#define DLB_VF_NUM_CQ_INTERRUPT_VECTORS 31 +#define DLB_VF_BASE_CQ_VECTOR_ID 0 +#define DLB_VF_LAST_CQ_VECTOR_ID 30 +#define DLB_VF_MBOX_VECTOR_ID 31 +#define DLB_VF_TOTAL_NUM_INTERRUPT_VECTORS \ + (DLB_VF_NUM_NON_CQ_INTERRUPT_VECTORS + \ + DLB_VF_NUM_CQ_INTERRUPT_VECTORS) + +#define DLB_PF_NUM_ALARM_INTERRUPT_VECTORS 4 +/* DLB ALARM interrupts */ +#define DLB_INT_ALARM 0 +/* VF to PF Mailbox Service Request */ +#define DLB_INT_VF_TO_PF_MBOX 1 +/* HCW Ingress Errors */ +#define DLB_INT_INGRESS_ERROR 3 + +#define DLB_ALARM_HW_SOURCE_SYS 0 +#define DLB_ALARM_HW_SOURCE_DLB 1 + +#define DLB_ALARM_HW_UNIT_CHP 1 +#define DLB_ALARM_HW_UNIT_LSP 3 + +#define DLB_ALARM_HW_CHP_AID_OUT_OF_CREDITS 6 +#define DLB_ALARM_HW_CHP_AID_ILLEGAL_ENQ 7 +#define DLB_ALARM_HW_LSP_AID_EXCESS_TOKEN_POPS 15 +#define DLB_ALARM_SYS_AID_ILLEGAL_HCW 0 +#define DLB_ALARM_SYS_AID_ILLEGAL_QID 3 +#define DLB_ALARM_SYS_AID_DISABLED_QID 4 +#define DLB_ALARM_SYS_AID_ILLEGAL_CQID 6 + +/* Hardware-defined base addresses */ +#define DLB_LDB_PP_BASE 0x2100000 +#define DLB_LDB_PP_STRIDE 0x1000 +#define DLB_LDB_PP_BOUND \ + (DLB_LDB_PP_BASE + DLB_LDB_PP_STRIDE * DLB_MAX_NUM_LDB_PORTS) +#define DLB_DIR_PP_BASE 0x2000000 +#define DLB_DIR_PP_STRIDE 0x1000 +#define DLB_DIR_PP_BOUND \ + (DLB_DIR_PP_BASE + DLB_DIR_PP_STRIDE * DLB_MAX_NUM_DIR_PORTS) + +struct dlb_resource_id { +u32 phys_id; +u32 virt_id; +u8 vf_owned; +u8 vf_id; +}; + +struct dlb_freelist { +u32 base; +u32 bound; +u32 offset; +}; + +static inline u32 dlb_freelist_count (struct dlb_freelist *list) +{ +return (list->bound - list->base) - list->offset; +} + +struct dlb_hcw { +u64 data; +/* Word 3 */ +u16 opaque; +u8 qid; +u8 sched_type:2 ; +u8 priority:3 ; +u8 msg_type:3 ; +/* Word 4 */ +u16 lock_id; +u8 meas_lat:1 ; +u8 rsvd1:2 ; +u8 no_dec:1 ; +u8 cmp_id:4 ; +u8 cq_token:1 ; +u8 qe_comp:1 ; +u8 qe_frag:1 ; +u8 qe_valid:1 ; +u8 int_arm:1 ; +u8 error:1 ; +u8 rsvd:2 ; +}; + +struct dlb_ldb_queue { +struct dlb_list_entry domain_list; +struct dlb_list_entry func_list; +struct dlb_resource_id id; +struct dlb_resource_id domain_id; +u32 num_qid_inflights; +struct dlb_freelist aqed_freelist; +u8 sn_cfg_valid; +u32 sn_group; +u32 sn_slot; +u32 num_mappings; +u8 num_pending_additions; +u8 owned; +u8 configured; +}; + +/* Directed ports and queues are paired by nature, so the driver tracks them +* with a single data structure. +*/ +struct dlb_dir_pq_pair { +struct dlb_list_entry domain_list; +struct dlb_list_entry func_list; +struct dlb_resource_id id; +struct dlb_resource_id domain_id; +u8 ldb_pool_used; +u8 dir_pool_used; +u8 queue_configured; +u8 port_configured; +u8 owned; +u8 enabled; +u32 ref_cnt; +}; + +enum dlb_qid_map_state { +/* The slot doesn't contain a valid queue mapping */ +DLB_QUEUE_UNMAPPED , +/* The slot contains a valid queue mapping */ +DLB_QUEUE_MAPPED , +/* The driver is mapping a queue into this slot */ +DLB_QUEUE_MAP_IN_PROGRESS , +/* The driver is unmapping a queue from this slot */ +DLB_QUEUE_UNMAP_IN_PROGRESS , +/* The driver is unmapping a queue from this slot, and once complete +* will replace it with another mapping. +*/ +DLB_QUEUE_UNMAP_IN_PROGRESS_PENDING_MAP , +}; + +struct dlb_ldb_port_qid_map { +u16 qid; +u8 priority; +u16 pending_qid; +u8 pending_priority; +enum dlb_qid_map_state state; +}; + +struct dlb_ldb_port { +struct dlb_list_entry domain_list; +struct dlb_list_entry func_list; +struct dlb_resource_id id; +struct dlb_resource_id domain_id; +u8 ldb_pool_used; +u8 dir_pool_used; +u8 init_tkn_cnt; +u32 hist_list_entry_base; +u32 hist_list_entry_limit; +/* The qid_map represents the hardware QID mapping state. */ +struct dlb_ldb_port_qid_map qid_map[DLB_MAX_NUM_QIDS_PER_LDB_CQ ]; +u32 ref_cnt; +u8 num_pending_removals; +u8 num_mappings; +u8 owned; +u8 enabled; +u8 configured; +}; + +struct dlb_credit_pool { +struct dlb_list_entry domain_list; +struct dlb_list_entry func_list; +struct dlb_resource_id id; +struct dlb_resource_id domain_id; +u32 total_credits; +u32 avail_credits; +u8 owned; +u8 configured; +}; + +struct dlb_sn_group { +u32 mode; +u32 sequence_numbers_per_queue; +u32 slot_use_bitmap; +u32 id; +}; + +static inline bool dlb_sn_group_full (struct dlb_sn_group *group) +{ +u32 mask[6 ] = { +0xffffffff ,/* 32 SNs per queue */ +0x0000ffff ,/* 64 SNs per queue */ +0x000000ff ,/* 128 SNs per queue */ +0x0000000f ,/* 256 SNs per queue */ +0x00000003 ,/* 512 SNs per queue */ +0x00000001 };/* 1024 SNs per queue */ + +return group->slot_use_bitmap == mask[group->mode]; +} + +static inline int dlb_sn_group_alloc_slot (struct dlb_sn_group *group) +{ +int bound[6 ] = {32 ,16 ,8 ,4 ,2 ,1 }; +int i; + +for (i =0 ; i < bound[group->mode]; i++) { +if (!(group->slot_use_bitmap & (1 << i))) { + group->slot_use_bitmap |=1 << i; +return i; + } + } + +return -1 ; +} + +static inline void dlb_sn_group_free_slot (struct dlb_sn_group *group,int slot) +{ + group->slot_use_bitmap &= ~(1 << slot); +} + +static inline int dlb_sn_group_used_slots (struct dlb_sn_group *group) +{ +int i, cnt =0 ; + +for (i =0 ; i <32 ; i++) + cnt += !!(group->slot_use_bitmap & (1 << i)); + +return cnt; +} + +struct dlb_domain { +struct dlb_function_resources *parent_func; +struct dlb_list_entry func_list; +struct dlb_list_head used_ldb_queues; +struct dlb_list_head used_ldb_ports; +struct dlb_list_head used_dir_pq_pairs; +struct dlb_list_head used_ldb_credit_pools; +struct dlb_list_head used_dir_credit_pools; +struct dlb_list_head avail_ldb_queues; +struct dlb_list_head avail_ldb_ports; +struct dlb_list_head avail_dir_pq_pairs; +struct dlb_list_head avail_ldb_credit_pools; +struct dlb_list_head avail_dir_credit_pools; +u32 total_hist_list_entries; +u32 avail_hist_list_entries; +u32 hist_list_entry_base; +u32 hist_list_entry_offset; +struct dlb_freelist qed_freelist; +struct dlb_freelist dqed_freelist; +struct dlb_freelist aqed_freelist; +struct dlb_resource_id id; +int num_pending_removals; +int num_pending_additions; +u8 configured; +u8 started; +}; + +struct dlb_bitmap ; + +struct dlb_function_resources { +u32 num_avail_domains; +struct dlb_list_head avail_domains; +struct dlb_list_head used_domains; +u32 num_avail_ldb_queues; +struct dlb_list_head avail_ldb_queues; +u32 num_avail_ldb_ports; +struct dlb_list_head avail_ldb_ports; +u32 num_avail_dir_pq_pairs; +struct dlb_list_head avail_dir_pq_pairs; +struct dlb_bitmap *avail_hist_list_entries; +struct dlb_bitmap *avail_qed_freelist_entries; +struct dlb_bitmap *avail_dqed_freelist_entries; +struct dlb_bitmap *avail_aqed_freelist_entries; +u32 num_avail_ldb_credit_pools; +struct dlb_list_head avail_ldb_credit_pools; +u32 num_avail_dir_credit_pools; +struct dlb_list_head avail_dir_credit_pools; +u32 num_enabled_ldb_ports;/* (PF only) */ +u8 locked;/* (VF only) */ +}; + +/* After initialization, each resource in dlb_hw_resources is located in one of +* the following lists: +* -- The PF's available resources list. These are unconfigured resources owned +* by the PF and not allocated to a DLB scheduling domain. +* -- A VF's available resources list. These are VF-owned unconfigured +* resources not allocated to a DLB scheduling domain. +* -- A domain's available resources list. These are domain-owned unconfigured +* resources. +* -- A domain's used resources list. These are are domain-owned configured +* resources. +* +* A resource moves to a new list when a VF or domain is created or destroyed, +* or when the resource is configured. +*/ +struct dlb_hw_resources { +struct dlb_ldb_queue ldb_queues[DLB_MAX_NUM_LDB_QUEUES ]; +struct dlb_ldb_port ldb_ports[DLB_MAX_NUM_LDB_PORTS ]; +struct dlb_dir_pq_pair dir_pq_pairs[DLB_MAX_NUM_DIR_PORTS ]; +struct dlb_credit_pool ldb_credit_pools[DLB_MAX_NUM_LDB_CREDIT_POOLS ]; +struct dlb_credit_pool dir_credit_pools[DLB_MAX_NUM_DIR_CREDIT_POOLS ]; +struct dlb_sn_group sn_groups[DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS ]; +}; + +struct dlb_hw { +/* BAR 0 address */ +void *csr_kva; +unsigned long csr_phys_addr; +/* BAR 2 address */ +void *func_kva; +unsigned long func_phys_addr; + +/* Resource tracking */ +struct dlb_hw_resources rsrcs; +struct dlb_function_resources pf; +struct dlb_function_resources vf[DLB_MAX_NUM_VFS ]; +struct dlb_domain domains[DLB_MAX_NUM_DOMAINS ]; +}; + +#endif /* __DLB_HW_TYPES_H */
drivers/event/dlb/pf/base/dlb_mbox.h new file +645 -0
645 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_osdep.h new file +350 -0
350 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_osdep_bitmap.h new file +449 -0
449 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_osdep_list.h new file +131 -0
131 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_osdep_types.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#ifndef __DLB_OSDEP_TYPES_H +#define __DLB_OSDEP_TYPES_H + +#include <linux/types.h> + +#include <inttypes.h> +#include <ctype.h> +#include <stdint.h> +#include <stdbool.h> +#include <string.h> +#include <unistd.h> +#include <errno.h> + +/* Types for user mode PF PMD */ +typedef uint8_t u8 ; +typedef int8_t s8 ; +typedef uint16_t u16 ; +typedef int16_t s16 ; +typedef uint32_t u32 ; +typedef int32_t s32 ; +typedef uint64_t u64 ; + +#define __iomem + +/* END types for user mode PF PMD */ + +#endif /* __DLB_OSDEP_TYPES_H */
drivers/event/dlb/pf/base/dlb_regs.h new file +2646 -0
1,034 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_resource.c new file +9699 -0
9,699 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_resource.h new file +1625 -0
1,625 lines not on this page · raw message
drivers/event/dlb/pf/base/dlb_user.h new file +1084 -0
1,084 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.13.6
Part of the diff is not shown here. View the raw message.
PATCH 6/27 event/dlb: add dynamic logging
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_log.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 drivers/event/dlb/dlb_log.h
drivers/event/dlb/dlb_log.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2020 Intel Corporation +*/ + +#ifndef _DLB_EVDEV_LOG_H_ +#define _DLB_EVDEV_LOG_H_ + +extern int eventdev_dlb_log_level; + +/* Dynamic logging */ +#define DLB_LOG_IMPL(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, eventdev_dlb_log_level, "%s" fmt "\n", \ + __func__, ##args) + +#define DLB_LOG_INFO(fmt, args...) \ + DLB_LOG_IMPL(INFO, fmt, ## args) + +#define DLB_LOG_DBG(fmt, args...) \ + DLB_LOG_IMPL(DEBUG, fmt, ## args) + +#define DLB_LOG_ERR(fmt, args...) \ + DLB_LOG_IMPL(ERR, fmt, ## args) + +#endif /* _DLB_EVDEV_LOG_H_ */
Signature · 1 line
2.13.6
PATCH 5/27 event/dlb: add DLB documentation
In reply to McDaniel, Timothy--- doc/guides/eventdevs/dlb.rst | 497 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 497 insertions(+) create mode 100644 doc/guides/eventdevs/dlb.rst
doc/guides/eventdevs/dlb.rst new file +497 -0
@@ -0,0 +1,497 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2020 Intel Corporation. + +Driver for the Intel® Dynamic Load Balancer (DLB) +================================================== + +The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer. + +.. note:: + + This PMD is disabled by default in the build configuration files, owing to + an external dependency on the `Netlink Protocol Library Suite + <http://www.infradead.org/~tgr/libnl/>`_ (libnl-3 and libnl-genl-3) which + must be installed on the board. Once the Netlink libraries are installed, + the PMD can be enabled by setting CONFIG_RTE_LIBRTE_PMD_DLB_QM=y and + recompiling the DPDK. + +Prerequisites +------------- + +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup + the basic DPDK environment. + +- Learn about the DLB device and its capabilities at `Intel Support + <http://www.intel.com/support>`_. FIXME: Add real link when documentation + becomes available. + +- The DLB kernel module. If it is not included in the machine's OS + distribution, download it from <FIXME: Add 01.org link when available> and + follow the build instructions. + +Configuration +------------- + +The DLB eventdev supports two modes of operation: + +* Bifurcated mode: the PMD is created as a vdev device and depends on the Linux + DLB kernel driver for device access. The bifurcated PMD's configuration + accesses are performed through the kernel driver, and (performance-critical) + datapath functions execute entirely in user-space. + + This mode supports both PF and VF devices, but is supported on Linux only. + +* PF PMD mode: the PF PMD is a user-space PMD that uses VFIO to gain direct + device access. To use this operation mode, the PCIe PF device must be bound + to a DPDK-compatible VFIO driver, such as vfio-pci. The PF PMD does not work + with PCIe VFs, but is portable to all environments (Linux, FreeBSD, etc.) + that DPDK supports. (Note: PF PMD testing has been limited to Linux at this + time.) + +The vdev device can be created from the application code or from the EAL +command line like so: + +* Call ``rte_vdev_init("dlb1_event")`` from the application. + +* Use ``--vdev="dlb1_event"`` in the EAL options, which will call + rte_vdev_init() internally. + +Example: + +.. code-block:: console + + ./your_eventdev_application --vdev="dlb1_event" + +Note: The dlb vdev can be instatiated with the name "event_dlb" as well. + +Eventdev API Notes +------------------ + +The DLB provides the functions of a DPDK event device; specifically, it +supports atomic, ordered, and parallel scheduling events from queues to ports. +However, the DLB hardware is not a perfect match to the eventdev API. Some DLB +features are abstracted by the PMD (e.g. directed ports), some are only +accessible as vdev command-line parameters, and certain eventdev features are +not supported (e.g. the event flow ID is not maintained during scheduling). + +In general the dlb PMD is designed for ease-of-use and doesn't require a +detailed understanding of the hardware, but these details are important when +writing high-performance code. This section describes the places where the +eventdev API and DLB misalign. + +Wait (timeout_ticks) Parameter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The eventdev API rte_event_dequeue_burst(..) can wait for an event to +arrive. Three different forms of waiting are supported by the dlb PMD: +polling, blocking on a hardware interrupt, and waiting using umonitor/umwait. +Which form of wait to use can be specified using the hybrid timeout data +structure below.The application should use the appropriate bybrid timeout +struct below and cast it to uint32_t or uint64_t, as appropriate. + +If RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT is set, then the timeout_ticks +parameter supplied to rte_event_dequeue_burst(..) is used to control if and how +to wait, and dequeue_timeout_ns is ignored. + +If RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT is not set, then dequeue_timeout_ns +supplied to the rte_event_dev_configure API is used to control if and how to +wait, and the timeout_ticks value is ignored. + +The application should use the appropriate bybrid timeout struct below and cast +it to uint32_t (for rte_event_dev_configure) or uint64_t (for +rte_event_dequeue_burst), as appropriate. + +Hybrid timeout_ticks +^^^^^^^^^^^^^^^^^^^^ +#. If poll_ticks is not 0 and neither interrupt_wait or umonitor_wait are set, + then we will busy poll for up to poll_ticks time. +#. If the interrupt_wait bit is set and the CQ is empty, then enter kernel + to wait for an interrupt after busy polling for poll_ticks time. There + is no guarantee how much time we spend in the API when using interrupt_wait. +#. If umonitor_wait is set, then repeatedly issue a umwait instruction + until the requested number of events have been dequeued, or until + poll_ticks has expired. + +Note: It is invalid to set both interrupt_wait and umonitor_wait. + +The hybrid timeout data structures are currently located in +drivers/event/dlb/dlb_timeout.h: + +.. code-block:: c + + struct rte_hybrid_timeout_ticks_64 { + RTE_STD_C11 + union { + uint64_t val64; + struct { + uint64_t poll_ticks:62; + uint64_t umonitor_wait:1; + uint64_t interrupt_wait:1; + }; + }; + }; + struct rte_hybrid_timeout_ns_32 { + RTE_STD_C11 + union { + uint32_t val32; + struct { + uint32_t poll_ns:30; + uint32_t umonitor_wait:1; + uint32_t interrupt_wait:1; + }; + }; + }; + +VAS Configuration +~~~~~~~~~~~~~~~~~ + +A VAS is a scheduling domain, of which there are 32 in the DLB. (Producer +ports in one VAS cannot enqueue events to a different VAS, except through the +`Data Mover`_.) When a VAS is configured, it allocates load-balanced and +directed queues, ports, credits, and other hardware resources. Some VAS +resource allocations are user-controlled -- the number of queues, for example +-- and others, like credit pools (one directed and one load-balanced pool per +VAS), are not. + +The dlb PMD creates a single VAS per DLB device. Supporting multiple VASes +per DLB device is a planned feature, where each VAS will be represented as a +separate event device. + +The DLB is a closed system eventdev, and as such the ``nb_events_limit`` device +setup argument and the per-port ``new_event_threshold`` argument apply as +defined in the eventdev header file. The limit is applied to all enqueues, +regardless of whether it will consume a directed or load-balanced credit. + +Load-balanced and Directed Ports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DLB ports come in two flavors: load-balanced and directed. The eventdev API +does not have the same concept, but it has a similar one: ports and queues that +are singly-linked (i.e. linked to a single queue or port, respectively). + +The ``rte_event_dev_info_get()`` function reports the number of available +event ports and queues (among other things). For the DLB PMD, max_event_ports +and max_event_queues report the number of available load-balanced ports and +queues, and max_single_link_event_port_queue_pairs reports the number of +available directed ports and queues. + +When a VAS is created in ``rte_event_dev_configure()``, the user specifies +``nb_event_ports`` and ``nb_single_link_event_port_queues``, which control the +total number of ports (load-balanced and directed) and the number of directed +ports. Hence, the number of requested load-balanced ports is ``nb_event_ports +- nb_single_link_event_ports``. The ``nb_event_queues`` field specifies the +total number of queues (load-balanced and directed). The number of directed +queues comes from ``nb_single_link_event_port_queues``, since directed ports +and queues come in pairs. + +When a port is setup, the ``RTE_EVENT_PORT_CFG_SINGLE_LINK`` flag determines +whether it should be configured as a directed (the flag is set) or a +load-balanced (the flag is unset) port. Similarly, the +``RTE_EVENT_QUEUE_CFG_SINGLE_LINK`` queue configuration flag controls +whether it is a directed or load-balanced queue. + +Load-balanced ports can only be linked to load-balanced queues, and directed +ports can only be linked to directed queues. Furthermore, directed ports can +only be linked to a single directed queue (and vice versa), and that link +cannot change after the eventdev is started. + +The eventdev API doesn't have a directed scheduling type. To support directed +traffic, the dlb PMD detects when an event is being sent to a directed queue +and overrides its scheduling type. Note that the originally selected scheduling +type (atomic, ordered, or parallel) is not preserved, and an event's sched_type +will be set to ``RTE_SCHED_TYPE_ATOMIC`` when it is dequeued from a directed +port. + +Flow ID +~~~~~~~ + +The flow ID field is not preserved in the event when it is scheduled in the +DLB, because the DLB hardware control word format does not have sufficient +space to preserve every event field. As a result, the flow ID specified with +the enqueued event will not be in the dequeued event. If this field is +required, the application should pass it through an out-of-band path (for +example in the mbuf's udata64 field, if the event points to an mbuf) or +reconstruct the flow ID after receiving the event. + +Also, the DLB hardware control word supports a 16-bit flow ID. Since struct +rte_event's flow_id field is 20 bits, the DLB PMD drops the most significant +four bits from the event's flow ID. + +Hardware Credits +~~~~~~~~~~~~~~~~ + +DLB uses a hardware credit scheme to prevent software from overflowing hardware +event storage, with each unit of storage represented by a credit. A port spends +a credit to enqueue an event, and hardware refills the ports with credits as the +events are scheduled to ports. Refills come from credit pools, and each port is +a member of a load-balanced credit pool and a directed credit pool. The +load-balanced credits are used to enqueue to load-balanced queues, and directed +credits are used for directed queues. + +An dlb eventdev contains one load-balanced and one directed credit pool. These +pools' sizes are controlled by the nb_events_limit field in struct +rte_event_dev_config. The load-balanced pool is sized to contain +nb_events_limit credits, and the directed pool is sized to contain +nb_events_limit/4 credits. The directed pool size can be overriden with the +num_dir_credits vdev argument, like so: + + .. code-block:: console + + --vdev=dlb1_event,num_dir_credits=<value> + +This can be used if the default allocation is too low or too high for the +specific application needs. The PMD also supports a vdev arg that limits the +max_num_events reported by rte_event_dev_info_get(): + + .. code-block:: console + + --vdev=dlb1_event,max_num_events=<value> + +By default, max_num_events is reported as the total available load-balanced +credits. If multiple DLB-based applications are being used, it may be desirable +to control how many load-balanced credits each application uses, particularly +when application(s) are written to configure nb_events_limit equal to the +reported max_num_events. + +Each port is a member of both credit pools. A port's credit allocation is +defined by its low watermark, high watermark, and refill quanta. These three +parameters are calculated by the dlb PMD like so: + +- The load-balanced high watermark is set to the port's enqueue_depth. + The directed high watermark is set to the minimum of the enqueue_depth and + the directed pool size divided by the total number of ports. +- The refill quanta is set to half the high watermark. +- The low watermark is set to the minimum of 8 and the refill quanta. + +When the eventdev is started, each port is pre-allocated a high watermark's +worth of credits. For example, if an eventdev contains four ports with enqueue +depths of 32 and a load-balanced credit pool size of 4096, each port will start +with 32 load-balanced credits, and there will be 3968 credits available to +replenish the ports. Thus, a single port is not capable of enqueueing up to the +nb_events_limit (without any events being dequeued), since the other ports are +retaining their initial credit allocation; in short, all ports must enqueue in +order to reach the limit. + +If a port attempts to enqueue and has no credits available, the enqueue +operation will fail and the application must retry the enqueue. Credits are +replenished asynchronously by the DLB hardware. + +Software Credits +~~~~~~~~~~~~~~~~ + +The DLB is a "closed system" event dev, and the DLB PMD layers a software +credit scheme on top of the hardware credit scheme in order to comply with +the per-port backpressure described in the eventdev API. + +The DLB's hardware scheme is local to a queue/pipeline stage: a port spends a +credit when it enqueues to a queue, and credits are later replenished after the +events are dequeued and released. + +In the software credit scheme, a credit is consumed when a new (.op = +RTE_EVENT_OP_NEW) event is injected into the system, and the credit is +replenished when the event is released from the system (either explicitly with +RTE_EVENT_OP_RELEASE or implicitly in dequeue_burst()). + +In this model, an event is "in the system" from its first enqueue into eventdev +until it is last dequeued. If the event goes through multiple event queues, it +is still considered "in the system" while a worker thread is processing it. + +A port will fail to enqueue if the number of events in the system exceeds its +``new_event_threshold`` (specified at port setup time). A port will also fail +to enqueue if it lacks enough hardware credits to enqueue; load-balanced +credits are used to enqueue to a load-balanced queue, and directed credits are +used to enqueue to a directed queue. + +The out-of-credit situations are typically transient, and an eventdev +application using the DLB ought to retry its enqueues if they fail. +If enqueue fails, DLB PMD sets rte_errno as follows: + +- -ENOSPC: Credit exhaustion (either hardware or software) +- -EINVAL: Invalid argument, such as port ID, queue ID, or sched_type. + +Depending on the pipeline the application has constructed, it's possible to +enter a credit deadlock scenario wherein the worker thread lacks the credit +to enqueue an event, and it must dequeue an event before it can recover the +credit. If the worker thread retries its enqueue indefinitely, it will not +make forward progress. Such deadlock is possible if the application has event +"loops", in which an event in dequeued from queue A and later enqueued back to +queue A. + +Due to this, workers should stop retrying after a time, release the events it +is attempting to enqueue, and dequeue more events. It is important that the +worker release the events and don't simply set them aside to retry the enqueue +again later, because the port has limited history list size (by default, twice +the port's dequeue_depth). + +Priority +~~~~~~~~ + +The DLB supports event priority and per-port queue service priority, as +described in the eventdev header file. The DLB does not support 'global' event +queue priority established at queue creation time. + +DLB supports 8 event and queue service priority levels. For both priority +types, the PMD uses the upper three bits of the priority field to determine the +DLB priority, discarding the 5 least significant bits. The 5 least significant +event priority bits are not preserved when an event is enqueued. + +Load-Balanced Queues +~~~~~~~~~~~~~~~~~~~~ + +A load-balanced queue can support atomic and ordered scheduling, or atomic and +unordered scheduling, but not atomic and unordered and ordered scheduling. A +queue's scheduling types are controlled by the event queue configuration. + +If the user sets the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag, the +``nb_atomic_order_sequences`` determines the supported scheduling types. +With non-zero ``nb_atomic_order_sequences``, the queue is configured for atomic +and ordered scheduling. In this case, ``RTE_SCHED_TYPE_PARALLEL`` scheduling is +supported by scheduling those events as ordered events. Note that when the +event is dequeued, its sched_type will be ``RTE_SCHED_TYPE_ORDERED``. Else if +``nb_atomic_order_sequences`` is zero, the queue is configured for atomic and +unordered scheduling. In this case, ``RTE_SCHED_TYPE_ORDERED`` is unsupported. + +If the ``RTE_EVENT_QUEUE_CFG_ALL_TYPES`` flag is not set, schedule_type +dictates the queue's scheduling type. + +The ``nb_atomic_order_sequences`` queue configuration field sets the ordered +queue's reorder buffer size. DLB has 4 groups of ordered queues, where each +group is configured to contain either 1 queue with 1024 reorder entries, 2 +queues with 512 reorder entries, and so on down to 32 queues with 32 entries. + +When a load-balanced queue is created, the PMD will configure a new sequence +number group on-demand if num_sequence_numbers does not match a pre-existing +group with available reorder buffer entries. If all sequence number groups are +in use, no new group will be created and queue configuration will fail. (Note +that when the PMD is used with a virtual DLB device, it cannot change the +sequence number configuration.) + +The queue's ``nb_atomic_flows`` parameter is ignored by the DLB PMD, because +the DLB doesn't limit the number of flows a queue can track. In the DLB, all +load-balanced queues can use the full 16-bit flow ID range. + +Reconfiguration +~~~~~~~~~~~~~~~ + +The Eventdev API allows one to reconfigure a device, its ports, and its queues +by first stopping the device, calling the configuration function(s), then +restarting the device. The DLB doesn't support configuring an individual queue +or port without first reconfiguring the entire device, however, so there are +certain reconfiguration sequences that are valid in the eventdev API but not +supported by the PMD. + +Specifically, the PMD supports the following configuration sequence: +1. Configure and start the device +2. Stop the device +3. (Optional) Reconfigure the device +4. (Optional) If step 3 is run: + + a. Setup queue(s). The reconfigured queue(s) lose their previous port links. + b. The reconfigured port(s) lose their previous queue links. + +5. (Optional, only if steps 4a and 4b are run) Link port(s) to queue(s) +6. Restart the device. If the device is reconfigured in step 3 but one or more + of its ports or queues are not, the PMD will apply their previous + configuration (including port->queue links) at this time. + +The PMD does not support the following configuration sequences: +1. Configure and start the device +2. Stop the device +3. Setup queue or setup port +4. Start the device + +This sequence is not supported because the event device must be reconfigured +before its ports or queues can be. + +Ordered Fragments +~~~~~~~~~~~~~~~~~ + +The DLB has a fourth enqueue type: partial enqueue. When a thread is processing +an ordered event, it can perform up to 16 "partial" enqueues, which allows a +single received ordered event to result in multiple reordered events. + +For example, consider the case where three events (A, then B, then C) are +enqueued with ordered scheduling and are received by three different ports. +The ports that receive A and C forward events A' and C', while the port that +receives B generates three partial enqueues -- B1', B2', and B3' -- followed by +a release operation. The DLB will reorder the events in the following order: + +A', B1', B2', B3', C' + +This functionality is not available explicitly through the eventdev API, but +the dlb PMD provides it through an additional (DLB-specific) event operation, +RTE_EVENT_DLB_OP_FRAG. + +Deferred Scheduling +~~~~~~~~~~~~~~~~~~~ + +The DLB PMD's default behavior for managing a CQ is to "pop" the CQ once per +dequeued event before returning from rte_event_dequeue_burst(). This frees the +corresponding entries in the CQ, which enables the DLB to schedule more events +to it. + +To support applications seeking finer-grained scheduling control -- for example +deferring scheduling to get the best possible priority scheduling and +load-balancing -- the PMD supports a deferred scheduling mode. In this mode, +the CQ entry is not popped until the *subsequent* rte_event_dequeue_burst() +call. This mode only applies to load-balanced event ports with dequeue depth of +1. + +To enable deferred scheduling, use the defer_sched vdev argument like so: + + .. code-block:: console + + --vdev=dlb1_event,defer_sched=on + +Atomic Inflights Allocation +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the last stage prior to scheduling an atomic event to a CQ, DLB holds the +inflight event in a temporary buffer that is divided among load-balanced +queues. If a queue's atomic buffer storage fills up, this can result in +head-of-line-blocking. For example: +- An LDB queue allocated N atomic buffer entries +- All N entries are filled with events from flow X, which is pinned to CQ 0. + +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be +scheduled. The likelihood of this case depends on the eventdev configuration, +traffic behavior, event processing latency, potential for a worker to be +interrupted or otherwise delayed, etc. + +By default, the PMD allocates 16 buffer entries for each load-balanced queue, +which provides an even division across all 128 queues but potentially wastes +buffer space (e.g. if not all queues are used, or aren't used for atomic +scheduling). + +The PMD provides a dev arg to override the default per-queue allocation. To +increase a vdev's per-queue atomic-inflight allocation to (for example) 64: + + .. code-block:: console + + --vdev=dlb1_event,atm_inflights=64 + +Atomic Inflights Allocation +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the last stage prior to scheduling an atomic event to a CQ, DLB holds the +inflight event in a temporary buffer that is divided among load-balanced +queues. If a queue's atomic buffer storage fills up, this can result in +head-of-line-blocking. For example: +- An LDB queue allocated N atomic buffer entries +- All N entries are filled with events from flow X, which is pinned to CQ 0. + +Until CQ 0 releases 1+ events, no other atomic flows for that LDB queue can be +scheduled. The likelihood of this case depends on the eventdev configuration, +traffic behavior, event processing latency, potential for a worker to be +interrupted or otherwise delayed, etc. + +By default, the PMD allocates 16 buffer entries for each load-balanced queue, +which provides an even division across all 128 queues but potentially wastes +buffer space (e.g. if not all queues are used, or aren't used for atomic +scheduling). + +The PMD provides a dev arg to override the default per-queue allocation. To +increase a vdev's per-queue atomic-inflight allocation to (for example) 64: + + .. code-block:: console + + --vdev=dlb1_event,atm_inflights=64
Signature · 1 line
2.13.6
PATCH 7/27 event/dlb: add private data structures and constants
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_priv.h | 595 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 595 insertions(+) create mode 100644 drivers/event/dlb/dlb_priv.h
drivers/event/dlb/dlb_priv.h new file +595 -0
@@ -0,0 +1,595 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#ifndef _DLB_PRIV_H_ +#define _DLB_PRIV_H_ + +#include <emmintrin.h> +#include <rte_eventdev.h> +#include <stdbool.h> +#include "rte_config.h" +#include "dlb_user.h" +#include "dlb_log.h" +#include "rte_pmd_dlb.h" + +#ifndef RTE_LIBRTE_PMD_DLB_QUELL_STATS +#define DLB_INC_STAT (_stat, _incr_val) ((_stat) += _incr_val) +#else +#define DLB_INC_STAT (_stat, _incr_val) +#endif + +#define EVDEV_DLB_NAME_PMD_STR "dlb_event" + +/* command line arg strings */ +#define NUMA_NODE_ARG "numa_node" +#define DLB_MAX_NUM_EVENTS "max_num_events" +#define DLB_NUM_DIR_CREDITS "num_dir_credits" +#define DEV_ID_ARG "dev_id" +#define DLB_DEFER_SCHED_ARG "defer_sched" +#define DLB_NUM_ATM_INFLIGHTS_ARG "atm_inflights" + +/* Begin HW related defines and structs */ + +#define DLB_MAX_NUM_DOMAINS 32 +#define DLB_MAX_NUM_VFS 16 +#define DLB_MAX_NUM_LDB_QUEUES 128 +#define DLB_MAX_NUM_LDB_PORTS 64 +#define DLB_MAX_NUM_DIR_PORTS 128 +#define DLB_MAX_NUM_DIR_QUEUES 128 +#define DLB_MAX_NUM_FLOWS (64 * 1024) +#define DLB_MAX_NUM_LDB_CREDITS 16384 +#define DLB_MAX_NUM_DIR_CREDITS 4096 +#define DLB_MAX_NUM_LDB_CREDIT_POOLS 64 +#define DLB_MAX_NUM_DIR_CREDIT_POOLS 64 +#define DLB_MAX_NUM_HIST_LIST_ENTRIES 5120 +#define DLB_MAX_NUM_ATM_INFLIGHTS 2048 +#define DLB_MAX_NUM_QIDS_PER_LDB_CQ 8 +#define DLB_QID_PRIORITIES 8 +#define DLB_MAX_DEVICE_PATH 32 +#define DLB_MIN_DEQUEUE_TIMEOUT_NS 1 +#define DLB_NOTIFY_ENTRY_OFFSET 2048 +#define DLB_NUM_SN_GROUPS 4 +#define DLB_MAX_LDB_SN_ALLOC 1024 +/* Note: "- 1" here to support the timeout range check in eventdev_autotest */ +#define DLB_MAX_DEQUEUE_TIMEOUT_NS (UINT32_MAX - 1) + +/* 5120 total hist list entries and 64 total ldb ports, which +* makes for 5120/64 == 80 hist list entries per port. However, CQ +* depth must be a power of 2 and must also be >= HIST LIST entries. +* As a result we just limit the maximum dequeue depth to 64. +*/ +#define DLB_MIN_LDB_CQ_DEPTH 1 +#define DLB_MIN_DIR_CQ_DEPTH 8 +#define DLB_MAX_CQ_DEPTH 64 +#define DLB_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT \ + DLB_MAX_CQ_DEPTH + +/* +* Static per queue/port provisioning values +*/ +#define DLB_NUM_ATOMIC_INFLIGHTS_PER_QUEUE 16 + +#define CQ_BASE (is_dir) ((is_dir) ? DLB_DIR_CQ_BASE : DLB_LDB_CQ_BASE) +#define CQ_SIZE (is_dir) ((is_dir) ? DLB_DIR_CQ_MAX_SIZE : DLB_LDB_CQ_MAX_SIZE) +#define PP_BASE (is_dir) ((is_dir) ? DLB_DIR_PP_BASE : DLB_LDB_PP_BASE) +#define PC_BASE (is_dir) ((is_dir) ? DLB_DIR_PC_BASE : DLB_LDB_PC_BASE) + +#define PAGE_SIZE (sysconf(_SC_PAGESIZE)) + +#define DLB_NUM_QES_PER_CACHE_LINE 4 + +#define DLB_MAX_ENQUEUE_DEPTH 64 +#define DLB_MIN_ENQUEUE_DEPTH 4 + +#define DLB_NAME_SIZE 64 + +#define DLB_1K 1024 +#define DLB_2K (2 * DLB_1K) +#define DLB_4K (4 * DLB_1K) +#define DLB_16K (16 * DLB_1K) +#define DLB_32K (32 * DLB_1K) +#define DLB_1MB (DLB_1K * DLB_1K) +#define DLB_16MB (16 * DLB_1MB) + +/* Use the upper 3 bits of the event priority to select the DLB priority */ +#define EV_TO_DLB_PRIO (x) ((x) >> 5) +#define DLB_TO_EV_PRIO (x) ((x) << 5) + +enum dlb_hw_port_type { +DLB_LDB , +DLB_DIR , + +/* NUM_DLB_PORT_TYPES must be last */ +NUM_DLB_PORT_TYPES +}; + +#define PORT_TYPE (p) ((p)->is_directed ? DLB_DIR : DLB_LDB) + +/* Do not change - must match hardware! */ +enum dlb_hw_sched_type { +DLB_SCHED_ATOMIC =0 , +DLB_SCHED_UNORDERED , +DLB_SCHED_ORDERED , +DLB_SCHED_DIRECTED , + +/* DLB_NUM_HW_SCHED_TYPES must be last */ +DLB_NUM_HW_SCHED_TYPES +}; + +/* vdev args */ +struct dlb_devargs { +int socket_id; +int max_num_events; +int num_dir_credits_override; +int dev_id; +int defer_sched; +int num_atm_inflights; +}; + +struct dlb_hw_rsrcs { +int32_t nb_events_limit; +uint32_t num_queues;/**> Total queues (lb + dir) */ +uint32_t num_ldb_queues;/**> Number of available ldb queues */ +uint32_t num_ldb_ports;/**< Number of load balanced ports */ +uint32_t num_dir_ports;/**< Number of directed ports */ +uint32_t num_ldb_credits;/**< Number of load balanced credits */ +uint32_t num_dir_credits;/**< Number of directed credits */ +uint32_t reorder_window_size;/**< Size of reorder window */ +}; + +struct dlb_hw_resource_info { +/**> Max resources that can be provided */ +struct dlb_hw_rsrcs hw_rsrc_max; +int num_sched_domains; +uint32_t socket_id; +/**> EAL flags passed to this QM instance, allowing the application to +* identify the pmd backend indicating hardware or software. +*/ +const char *eal_flags; +}; + +struct dlb_hw_queue_params { +/**> qid type: atomic, ordered, etc. */ +enum dlb_hw_sched_type type; +/**> size of the reorder window */ +uint32_t reorder_window_size; +/**> priority of this QID: +* 0 = highest +* 128 = default +* 255 = lowest +*/ +uint8_t priority; +}; + +/* hw-specific format - do not change */ + +struct dlb_event_type { +uint8_t major:4 ; +uint8_t unused:4 ; +uint8_t sub; +}; + +union dlb_opaque_data { +uint16_t opaque_data; +struct dlb_event_type event_type; +}; + +struct dlb_msg_info { +uint8_t qid; +uint8_t sched_type:2 ; +uint8_t priority:3 ; +uint8_t msg_type:3 ; +}; + +#define DLB_NEW_CMD_BYTE 0x08 +#define DLB_FWD_CMD_BYTE 0x0A +#define DLB_COMP_CMD_BYTE 0x02 +#define DLB_NOOP_CMD_BYTE 0x00 +#define DLB_POP_CMD_BYTE 0x01 + +/* hw-specific format - do not change */ +struct dlb_enqueue_qe { +uint64_t data; +/* Word 3 */ +union dlb_opaque_data u; +uint8_t qid; +uint8_t sched_type:2 ; +uint8_t priority:3 ; +uint8_t msg_type:3 ; +/* Word 4 */ +uint16_t lock_id; +uint8_t meas_lat:1 ; +uint8_t rsvd1:2 ; +uint8_t no_dec:1 ; +uint8_t cmp_id:4 ; +union { +uint8_t cmd_byte; +struct { +uint8_t cq_token:1 ; +uint8_t qe_comp:1 ; +uint8_t qe_frag:1 ; +uint8_t qe_valid:1 ; +uint8_t int_arm:1 ; +uint8_t error:1 ; +uint8_t rsvd:2 ; + }; + }; +}; + +/* hw-specific format - do not change */ +struct dlb_cq_pop_qe { +uint64_t data; +union dlb_opaque_data u; +uint8_t qid; +uint8_t sched_type:2 ; +uint8_t priority:3 ; +uint8_t msg_type:3 ; +uint16_t tokens:10 ; +uint16_t rsvd2:6 ; +uint8_t meas_lat:1 ; +uint8_t rsvd1:2 ; +uint8_t no_dec:1 ; +uint8_t cmp_id:4 ; +union { +uint8_t cmd_byte; +struct { +uint8_t cq_token:1 ; +uint8_t qe_comp:1 ; +uint8_t qe_frag:1 ; +uint8_t qe_valid:1 ; +uint8_t int_arm:1 ; +uint8_t error:1 ; +uint8_t rsvd:2 ; + }; + }; +}; + +/* hw-specific format - do not change */ +struct dlb_dequeue_qe { +uint64_t data; +union dlb_opaque_data u; +uint8_t qid; +uint8_t sched_type:2 ; +uint8_t priority:3 ; +uint8_t msg_type:3 ; +uint16_t pp_id:10 ; +uint16_t rsvd0:6 ; +uint8_t debug; +uint8_t cq_gen:1 ; +uint8_t qid_depth:1 ; +uint8_t rsvd1:3 ; +uint8_t error:1 ; +uint8_t rsvd2:2 ; +}; + +union dlb_port_config { +struct dlb_create_ldb_port_args ldb; +struct dlb_create_dir_port_args dir; +}; + +enum DLB_PORT_STATE { +PORT_CLOSED , +PORT_STARTED , +PORT_STOPPED +}; + +enum dlb_configuration_state { +/* The resource has not been configured */ +DLB_NOT_CONFIGURED , +/* The resource was configured, but the device was stopped */ +DLB_PREV_CONFIGURED , +/* The resource is currently configured */ +DLB_CONFIGURED +}; + +struct dlb_port { +uint32_t id; +bool is_directed; +bool gen_bit; +uint16_t dir_credits; +uint32_t dequeue_depth; +enum dlb_token_pop_mode token_pop_mode; +union dlb_port_config cfg; +int pp_mmio_base; +uint16_t cached_ldb_credits; +uint16_t ldb_pushcount_at_credit_expiry; +uint16_t ldb_credits; +uint16_t cached_dir_credits; +uint16_t dir_pushcount_at_credit_expiry; +bool int_armed; +bool use_rsvd_token_scheme; +uint8_t cq_rsvd_token_deficit; +uint16_t owed_tokens; +int16_t issued_releases; +int16_t token_pop_thresh; +int cq_depth; +uint16_t cq_idx; +uint16_t cq_idx_unmasked; +uint16_t cq_depth_mask; +uint16_t gen_bit_shift; +enum DLB_PORT_STATE state; +enum dlb_configuration_state config_state; +int num_mapped_qids; +uint8_t *qid_mappings; +struct dlb_enqueue_qe *qe4;/* Cache line's worth of QEs (4) */ +struct dlb_cq_pop_qe *consume_qe; +struct dlb_eventdev *dlb;/* back ptr */ +struct dlb_eventdev_port *ev_port;/* back ptr */ +}; + +/* Per-process per-port mmio and memory pointers */ +struct process_local_port_data { +uint64_t *pp_addr; +uint16_t *ldb_popcount; +uint16_t *dir_popcount; +struct dlb_dequeue_qe *cq_base; +bool mmaped; +}; + +struct dlb_eventdev ; + +struct dlb_port_low_level_io_functions { +void (*pp_enqueue_four)(struct dlb_enqueue_qe *qe4,uint64_t *pp_addr); +}; + +struct dlb_config { +int configured; +int reserved; +uint32_t ldb_credit_pool_id; +uint32_t dir_credit_pool_id; +uint32_t num_ldb_credits; +uint32_t num_dir_credits; +struct dlb_create_sched_domain_args resources; +}; + +struct dlb_hw_dev { +char device_name[DLB_NAME_SIZE ]; +char device_path[DLB_MAX_DEVICE_PATH ]; +int device_path_id; +char domain_device_path[DLB_MAX_DEVICE_PATH ]; +struct dlb_config cfg; +struct dlb_hw_resource_info info; +void *pf_dev;/* opaque pointer to PF PMD dev (struct dlb_dev) */ +int device_id; +uint32_t domain_id; +int domain_id_valid; +rte_spinlock_t resource_lock;/* for MP support */ +}; __rte_cache_aligned + +/* End HW related defines and structs */ + +/* Begin DLB PMD Eventdev related defines and structs */ + +#define DLB_MAX_NUM_QUEUES \ + (DLB_MAX_NUM_DIR_QUEUES + DLB_MAX_NUM_LDB_QUEUES) + +#define DLB_MAX_NUM_PORTS (DLB_MAX_NUM_DIR_PORTS + DLB_MAX_NUM_LDB_PORTS) +#define DLB_MAX_INPUT_QUEUE_DEPTH 256 + +/* Used for parsing dir ports/queues. */ + +/* Note: eventdev currently is limited to 64 ports, but DLB hardware has 128 +* directed ports/queues. +*/ +struct dlb_dir_resource_list { +int entries; +uint32_t id[DLB_MAX_NUM_DIR_PORTS ]; +}; + +/** Structure to hold the queue to port link establishment attributes */ + +struct dlb_event_queue_link { +uint8_t queue_id; +uint8_t priority; +bool mapped; +bool valid; +}; + +struct dlb_traffic_stats { +uint64_t rx_ok; +uint64_t rx_drop; +uint64_t rx_interrupt_wait; +uint64_t rx_umonitor_umwait; +uint64_t tx_ok; +uint64_t total_polls; +uint64_t zero_polls; +uint64_t tx_nospc_ldb_hw_credits; +uint64_t tx_nospc_dir_hw_credits; +uint64_t tx_nospc_inflight_max; +uint64_t tx_nospc_new_event_limit; +uint64_t tx_nospc_inflight_credits; +}; + +struct dlb_port_stats { +struct dlb_traffic_stats traffic; +uint64_t tx_op_cnt[4 ];/* indexed by rte_event.op */ +uint64_t tx_implicit_rel; +uint64_t tx_sched_cnt[DLB_NUM_HW_SCHED_TYPES ]; +uint64_t tx_invalid; +uint64_t rx_sched_cnt[DLB_NUM_HW_SCHED_TYPES ]; +uint64_t rx_sched_invalid; +uint64_t enq_ok[DLB_MAX_NUM_QUEUES ];/* per-queue enq_ok */ +}; + +struct dlb_eventdev_port { +struct dlb_port qm_port;/* hw specific data structure */ +struct rte_event_port_conf conf;/* user-supplied configuration */ +uint16_t inflight_credits;/* num credits this port has right now */ +uint16_t credit_update_quanta; +struct dlb_eventdev *dlb;/* backlink optimization */ + struct dlb_port_stats stats __rte_cache_aligned; +struct dlb_event_queue_link link[DLB_MAX_NUM_QIDS_PER_LDB_CQ ]; +int num_links; +uint32_t id; +/* num releases yet to be completed on this port. +* Only applies to load-balanced ports. +*/ +uint16_t outstanding_releases; +uint16_t inflight_max;/* app requested max inflights for this port */ +/* setup_done is set when the event port is setup */ +bool setup_done; +/* enq_configured is set when the qm port is created */ +bool enq_configured; +uint8_t implicit_release;/* release events before dequeueing */ +} __rte_cache_aligned; + +struct dlb_queue { +uint32_t num_qid_inflights;/* User config */ +uint32_t num_atm_inflights;/* User config */ +enum dlb_configuration_state config_state; +int sched_type;/* LB queue only */ +uint32_t id; +bool is_directed; +}; + +struct dlb_eventdev_queue { +struct dlb_queue qm_queue; +struct rte_event_queue_conf conf;/* User config */ +uint64_t enq_ok; +uint32_t id; +bool setup_done; +uint8_t num_links; +}; + +struct dlb_device_stats { +/* Device specific */ +}; + +enum dlb_run_state { +DLB_RUN_STATE_STOPPED =0 , +DLB_RUN_STATE_STOPPING , +DLB_RUN_STATE_STARTING , +DLB_RUN_STATE_STARTED +}; + +#define DLB_IS_VDEV true +#define DLB_NOT_VDEV false + +struct dlb_eventdev { +struct dlb_eventdev_port ev_ports[DLB_MAX_NUM_PORTS ]; +struct dlb_eventdev_queue ev_queues[DLB_MAX_NUM_QUEUES ]; +uint8_t qm_ldb_to_ev_queue_id[DLB_MAX_NUM_QUEUES ]; +uint8_t qm_dir_to_ev_queue_id[DLB_MAX_NUM_QUEUES ]; + +/* store num stats and offset of the stats for each queue */ +uint16_t xstats_count_per_qid[DLB_MAX_NUM_QUEUES ]; +uint16_t xstats_offset_for_qid[DLB_MAX_NUM_QUEUES ]; + +/* store num stats and offset of the stats for each port */ +uint16_t xstats_count_per_port[DLB_MAX_NUM_PORTS ]; +uint16_t xstats_offset_for_port[DLB_MAX_NUM_PORTS ]; +struct dlb_get_num_resources_args hw_rsrc_query_results; +uint32_t xstats_count_mode_queue; +struct dlb_hw_dev qm_instance;/* strictly hw related */ +uint64_t global_dequeue_wait_ticks; +struct dlb_xstats_entry *xstats; +struct rte_eventdev *event_dev;/* backlink to dev */ +uint32_t xstats_count_mode_port; +uint32_t xstats_count_mode_dev; +uint32_t xstats_count; +rte_atomic32_t inflights; +uint32_t new_event_limit; +int max_num_events_override; +int num_dir_credits_override; +volatile enum dlb_run_state run_state; +uint16_t num_dir_queues;/* total num of evdev dir queues requested */ +uint16_t num_dir_credits; +uint16_t num_ldb_credits; +uint16_t num_queues;/* total queues */ +uint16_t num_ldb_queues;/* total num of evdev ldb queues requested */ +uint16_t num_ports;/* total num of evdev ports requested */ +uint16_t num_ldb_ports;/* total num of ldb ports requested */ +uint16_t num_dir_ports;/* total num of dir ports requested */ +bool is_vdev; +bool umwait_allowed; +bool global_dequeue_wait;/* Not using per dequeue wait if true */ +bool defer_sched; +unsigned int num_atm_inflights_per_queue; +enum dlb_cq_poll_modes poll_mode; +uint8_t revision; +bool configured; +/* Device stats */ + struct dlb_device_stats stats __rte_cache_aligned; +}; + +/* End Eventdev related defines and structs */ + +/* Forwards for non-inlined functions */ + +void dlb_free_qe_mem (struct dlb_port *port); + +int dlb_init_qe_mem (struct dlb_port *port,char *mz_name); + +int dlb_init_send_qe (struct dlb_port *port,char *mz_name); + +int dlb_init_partial_qe (struct dlb_port *port,char *mz_name); + +int dlb_init_fwd_qe (struct dlb_port *port,char *mz_name); + +int dlb_init_consume_qe (struct dlb_port *port,char *mz_name); + +int dlb_init_complete_qe (struct dlb_port *port,char *mz_name); + +int dlb_init_noop_qe (struct dlb_port *port,char *mz_name); + +int dlb_uninit (const char *name); + +void dlb_eventdev_dump (struct rte_eventdev *dev,FILE *f); + +int dlb_xstats_init (struct dlb_eventdev *dlb); + +void dlb_xstats_uninit (struct dlb_eventdev *dlb); + +int dlb_eventdev_xstats_get (const struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode,uint8_t queue_port_id, +const unsigned int ids[],uint64_t values[],unsigned int n); + +int dlb_eventdev_xstats_get_names (const struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode,uint8_t queue_port_id, +struct rte_event_dev_xstats_name *xstat_names, +unsigned int *ids,unsigned int size); + +uint64_t dlb_eventdev_xstats_get_by_name (const struct rte_eventdev *dev, +const char *name,unsigned int *id); + +int dlb_eventdev_xstats_reset (struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode, +int16_t queue_port_id, +const uint32_t ids[], +uint32_t nb_ids); + +int test_dlb_eventdev (void ); + +int dlb_primary_eventdev_probe (struct rte_eventdev *dev, +const char *name, +struct dlb_devargs *dlb_args, +bool is_vdev); +int dlb_secondary_eventdev_probe (struct rte_eventdev *dev, +const char *name, +bool is_vdev); +uint32_t +dlb_get_queue_depth (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *queue); + +/* arg parsing helper functions */ +int set_numa_node(const char *key __rte_unused, const char *value, +void *opaque); +int set_dir_ports(const char *key __rte_unused, + const char *value __rte_unused, + void *opaque __rte_unused); +int set_dir_queues(const char *key __rte_unused, + const char *value __rte_unused, + void *opaque __rte_unused); +int set_max_num_events(const char *key __rte_unused, const char *value, +void *opaque); +int set_num_dir_credits(const char *key __rte_unused, const char *value, +void *opaque); +void dlb_drain (struct rte_eventdev *dev); +void dlb_entry_points_init (struct rte_eventdev *dev); +int dlb_parse_params (const char *params, +const char *name, +struct dlb_devargs *dlb_args); +int dlb_string_to_int (int *result,const char *str); + +#endif /* _DLB_PRIV_H_ */
Signature · 1 line
2.13.6
PATCH 9/27 event/dlb: inline functions used in multiple files
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_inline_fns.h | 80 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 drivers/event/dlb/dlb_inline_fns.h
drivers/event/dlb/dlb_inline_fns.h new file +80 -0
@@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include "rte_memcpy.h" +#include "rte_io.h" + +/* Inline functions required in more than one source file. +*/ + +static inline struct dlb_eventdev * +dlb_pmd_priv (const struct rte_eventdev *eventdev) +{ +return eventdev->data->dev_private; +} + +static inline void +dlb_umonitor (volatile void *addr) +{ + asmvolatile (".byte 0xf3, 0x0f, 0xae, 0xf7\t\n" + : + :"D" (addr)); +} + +static inline void +dlb_umwait (int state,uint64_t timeout) +{ +uint32_t eax = timeout &UINT32_MAX ; +uint32_t edx = timeout >>32 ; + + asmvolatile (".byte 0xf2, 0x0f, 0xae, 0xf7\t\n" + : + :"D" (state),"a" (eax),"d" (edx)); +} + +static inline void +dlb_movntdq (struct dlb_enqueue_qe *qe4,uint64_t *pp_addr) +{ +/* Move entire 64B cache line of QEs, 128 bits (16B) at a time. */ +long long *_qe = (long long *)qe4; +__v2di src_data0 = (__v2di ){_qe[0 ], _qe[1 ]}; +__v2di src_data1 = (__v2di ){_qe[2 ], _qe[3 ]}; +__v2di src_data2 = (__v2di ){_qe[4 ], _qe[5 ]}; +__v2di src_data3 = (__v2di ){_qe[6 ], _qe[7 ]}; + +__builtin_ia32_movntdq ((__v2di *)pp_addr +0 , (__v2di )src_data0); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +1 , (__v2di )src_data1); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +2 , (__v2di )src_data2); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +3 , (__v2di )src_data3); +rte_wmb (); +} + +static inline void +dlb_movntdq_single (void *qe4,void *pp_addr) +{ +long long *_qe = (long long *)qe4; +__v2di src_data0 = (__v2di ){_qe[0 ], _qe[1 ]}; + +__builtin_ia32_movntdq ((__v2di *)pp_addr, (__v2di )src_data0); +} + +static inline void +dlb_cldemote (void *addr) +{ +/* Load addr into RSI, then demote the cache line of the address +* contained in that register. +*/ + asmvolatile (".byte 0x0f, 0x1c, 0x06" ::"S" (addr)); +} + +static inline void +dlb_movdir64b (struct dlb_enqueue_qe *qe4,uint64_t *pp_addr) +{ + asmvolatile (".byte 0x66, 0x0f, 0x38, 0xf8, 0x02" + : + :"a" (pp_addr),"d" (qe4)); +}
Signature · 1 line
2.13.6
PATCH 8/27 event/dlb: add definitions shared with LKM or shared code
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_user.h | 1351 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1351 insertions(+) create mode 100644 drivers/event/dlb/dlb_user.h
drivers/event/dlb/dlb_user.h new file +1351 -0
@@ -0,0 +1,1351 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#ifndef __DLB_USER_H +#define __DLB_USER_H + +#define DLB_MAX_NAME_LEN 64 + +#include <linux/types.h> + +enum dlb_error { +DLB_ST_SUCCESS =0 , +DLB_ST_NAME_EXISTS , +DLB_ST_DOMAIN_UNAVAILABLE , +DLB_ST_LDB_PORTS_UNAVAILABLE , +DLB_ST_DIR_PORTS_UNAVAILABLE , +DLB_ST_LDB_QUEUES_UNAVAILABLE , +DLB_ST_LDB_CREDITS_UNAVAILABLE , +DLB_ST_DIR_CREDITS_UNAVAILABLE , +DLB_ST_LDB_CREDIT_POOLS_UNAVAILABLE , +DLB_ST_DIR_CREDIT_POOLS_UNAVAILABLE , +DLB_ST_SEQUENCE_NUMBERS_UNAVAILABLE , +DLB_ST_INVALID_DOMAIN_ID , +DLB_ST_INVALID_QID_INFLIGHT_ALLOCATION , +DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE , +DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE , +DLB_ST_INVALID_LDB_CREDIT_POOL_ID , +DLB_ST_INVALID_DIR_CREDIT_POOL_ID , +DLB_ST_INVALID_POP_COUNT_VIRT_ADDR , +DLB_ST_INVALID_LDB_QUEUE_ID , +DLB_ST_INVALID_CQ_DEPTH , +DLB_ST_INVALID_CQ_VIRT_ADDR , +DLB_ST_INVALID_PORT_ID , +DLB_ST_INVALID_QID , +DLB_ST_INVALID_PRIORITY , +DLB_ST_NO_QID_SLOTS_AVAILABLE , +DLB_ST_QED_FREELIST_ENTRIES_UNAVAILABLE , +DLB_ST_DQED_FREELIST_ENTRIES_UNAVAILABLE , +DLB_ST_INVALID_DIR_QUEUE_ID , +DLB_ST_DIR_QUEUES_UNAVAILABLE , +DLB_ST_INVALID_LDB_CREDIT_LOW_WATERMARK , +DLB_ST_INVALID_LDB_CREDIT_QUANTUM , +DLB_ST_INVALID_DIR_CREDIT_LOW_WATERMARK , +DLB_ST_INVALID_DIR_CREDIT_QUANTUM , +DLB_ST_DOMAIN_NOT_CONFIGURED , +DLB_ST_PID_ALREADY_ATTACHED , +DLB_ST_PID_NOT_ATTACHED , +DLB_ST_INTERNAL_ERROR , +DLB_ST_DOMAIN_IN_USE , +DLB_ST_IOMMU_MAPPING_ERROR , +DLB_ST_FAIL_TO_PIN_MEMORY_PAGE , +DLB_ST_UNABLE_TO_PIN_POPCOUNT_PAGES , +DLB_ST_UNABLE_TO_PIN_CQ_PAGES , +DLB_ST_DISCONTIGUOUS_CQ_MEMORY , +DLB_ST_DISCONTIGUOUS_POP_COUNT_MEMORY , +DLB_ST_DOMAIN_STARTED , +DLB_ST_LARGE_POOL_NOT_SPECIFIED , +DLB_ST_SMALL_POOL_NOT_SPECIFIED , +DLB_ST_NEITHER_POOL_SPECIFIED , +DLB_ST_DOMAIN_NOT_STARTED , +DLB_ST_INVALID_MEASUREMENT_DURATION , +DLB_ST_INVALID_PERF_METRIC_GROUP_ID , +DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES , +DLB_ST_DOMAIN_RESET_FAILED , +DLB_ST_MBOX_ERROR , +DLB_ST_INVALID_HIST_LIST_DEPTH , +DLB_ST_NO_MEMORY , +}; + +static const char dlb_error_strings[][128 ] = { +"DLB_ST_SUCCESS" , +"DLB_ST_NAME_EXISTS" , +"DLB_ST_DOMAIN_UNAVAILABLE" , +"DLB_ST_LDB_PORTS_UNAVAILABLE" , +"DLB_ST_DIR_PORTS_UNAVAILABLE" , +"DLB_ST_LDB_QUEUES_UNAVAILABLE" , +"DLB_ST_LDB_CREDITS_UNAVAILABLE" , +"DLB_ST_DIR_CREDITS_UNAVAILABLE" , +"DLB_ST_LDB_CREDIT_POOLS_UNAVAILABLE" , +"DLB_ST_DIR_CREDIT_POOLS_UNAVAILABLE" , +"DLB_ST_SEQUENCE_NUMBERS_UNAVAILABLE" , +"DLB_ST_INVALID_DOMAIN_ID" , +"DLB_ST_INVALID_QID_INFLIGHT_ALLOCATION" , +"DLB_ST_ATOMIC_INFLIGHTS_UNAVAILABLE" , +"DLB_ST_HIST_LIST_ENTRIES_UNAVAILABLE" , +"DLB_ST_INVALID_LDB_CREDIT_POOL_ID" , +"DLB_ST_INVALID_DIR_CREDIT_POOL_ID" , +"DLB_ST_INVALID_POP_COUNT_VIRT_ADDR" , +"DLB_ST_INVALID_LDB_QUEUE_ID" , +"DLB_ST_INVALID_CQ_DEPTH" , +"DLB_ST_INVALID_CQ_VIRT_ADDR" , +"DLB_ST_INVALID_PORT_ID" , +"DLB_ST_INVALID_QID" , +"DLB_ST_INVALID_PRIORITY" , +"DLB_ST_NO_QID_SLOTS_AVAILABLE" , +"DLB_ST_QED_FREELIST_ENTRIES_UNAVAILABLE" , +"DLB_ST_DQED_FREELIST_ENTRIES_UNAVAILABLE" , +"DLB_ST_INVALID_DIR_QUEUE_ID" , +"DLB_ST_DIR_QUEUES_UNAVAILABLE" , +"DLB_ST_INVALID_LDB_CREDIT_LOW_WATERMARK" , +"DLB_ST_INVALID_LDB_CREDIT_QUANTUM" , +"DLB_ST_INVALID_DIR_CREDIT_LOW_WATERMARK" , +"DLB_ST_INVALID_DIR_CREDIT_QUANTUM" , +"DLB_ST_DOMAIN_NOT_CONFIGURED" , +"DLB_ST_PID_ALREADY_ATTACHED" , +"DLB_ST_PID_NOT_ATTACHED" , +"DLB_ST_INTERNAL_ERROR" , +"DLB_ST_DOMAIN_IN_USE" , +"DLB_ST_IOMMU_MAPPING_ERROR" , +"DLB_ST_FAIL_TO_PIN_MEMORY_PAGE" , +"DLB_ST_UNABLE_TO_PIN_POPCOUNT_PAGES" , +"DLB_ST_UNABLE_TO_PIN_CQ_PAGES" , +"DLB_ST_DISCONTIGUOUS_CQ_MEMORY" , +"DLB_ST_DISCONTIGUOUS_POP_COUNT_MEMORY" , +"DLB_ST_DOMAIN_STARTED" , +"DLB_ST_LARGE_POOL_NOT_SPECIFIED" , +"DLB_ST_SMALL_POOL_NOT_SPECIFIED" , +"DLB_ST_NEITHER_POOL_SPECIFIED" , +"DLB_ST_DOMAIN_NOT_STARTED" , +"DLB_ST_INVALID_MEASUREMENT_DURATION" , +"DLB_ST_INVALID_PERF_METRIC_GROUP_ID" , +"DLB_ST_LDB_PORT_REQUIRED_FOR_LDB_QUEUES" , +"DLB_ST_DOMAIN_RESET_FAILED" , +"DLB_ST_MBOX_ERROR" , +"DLB_ST_INVALID_HIST_LIST_DEPTH" , +"DLB_ST_NO_MEMORY" , +}; + +struct dlb_cmd_response { +__u32 status;/* Interpret using enum dlb_error */ +__u32 id; +}; + +/******************************/ +/* 'dlb' device file commands */ +/******************************/ + +#define DLB_DEVICE_VERSION (x) (((x) >> 8) & 0xFF) +#define DLB_DEVICE_REVISION (x) ((x) & 0xFF) + +enum dlb_revisions { +DLB_REV_A0 =0 , +DLB_REV_A1 =1 , +DLB_REV_A2 =2 , +DLB_REV_A3 =3 , +DLB_REV_B0 =4 , +}; + +/* +* DLB_CMD_GET_DEVICE_VERSION: Query the DLB device version. +* +* This ioctl interface is the same in all driver versions and is always +* the first ioctl. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id[7:0]: Device revision. +* response.id[15:8]: Device version. +*/ + +struct dlb_get_device_version_args { +/* Output parameters */ +__u64 response; +}; + +#define DLB_VERSION_MAJOR_NUMBER 10 +#define DLB_VERSION_MINOR_NUMBER 7 +#define DLB_VERSION_REVISION_NUMBER 9 +#define DLB_VERSION (DLB_VERSION_MAJOR_NUMBER << 24 | \ + DLB_VERSION_MINOR_NUMBER << 16 | \ + DLB_VERSION_REVISION_NUMBER) + +#define DLB_VERSION_GET_MAJOR_NUMBER (x) (((x) >> 24) & 0xFF) +#define DLB_VERSION_GET_MINOR_NUMBER (x) (((x) >> 16) & 0xFF) +#define DLB_VERSION_GET_REVISION_NUMBER (x) ((x) & 0xFFFF) + +static inline __u8 dlb_version_incompatible (__u32 version) +{ +__u8 inc; + + inc =DLB_VERSION_GET_MAJOR_NUMBER (version) !=DLB_VERSION_MAJOR_NUMBER ; + inc |= (int )DLB_VERSION_GET_MINOR_NUMBER (version) < +DLB_VERSION_MINOR_NUMBER ; + +return inc; +} + +/* +* DLB_CMD_GET_DRIVER_VERSION: Query the DLB driver version. The major number +* is changed when there is an ABI-breaking change, the minor number is +* changed if the API is changed in a backwards-compatible way, and the +* revision number is changed for fixes that don't affect the API. +* +* If the kernel driver's API version major number and the header's +* DLB_VERSION_MAJOR_NUMBER differ, the two are incompatible, or if the +* major numbers match but the kernel driver's minor number is less than +* the header file's, they are incompatible. The DLB_VERSION_INCOMPATIBLE +* macro should be used to check for compatibility. +* +* This ioctl interface is the same in all driver versions. Applications +* should check the driver version before performing any other ioctl +* operations. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Driver API version. Use the DLB_VERSION_GET_MAJOR_NUMBER, +* DLB_VERSION_GET_MINOR_NUMBER, and +* DLB_VERSION_GET_REVISION_NUMBER macros to interpret the field. +*/ + +struct dlb_get_driver_version_args { +/* Output parameters */ +__u64 response; +}; + +/* +* DLB_CMD_CREATE_SCHED_DOMAIN: Create a DLB scheduling domain and reserve the +* resources (queues, ports, etc.) that it contains. +* +* Input parameters: +* - num_ldb_queues: Number of load-balanced queues. +* - num_ldb_ports: Number of load-balanced ports. +* - num_dir_ports: Number of directed ports. A directed port has one directed +* queue, so no num_dir_queues argument is necessary. +* - num_atomic_inflights: This specifies the amount of temporary atomic QE +* storage for the domain. This storage is divided among the domain's +* load-balanced queues that are configured for atomic scheduling. +* - num_hist_list_entries: Amount of history list storage. This is divided +* among the domain's CQs. +* - num_ldb_credits: Amount of load-balanced QE storage (QED). QEs occupy this +* space until they are scheduled to a load-balanced CQ. One credit +* represents the storage for one QE. +* - num_dir_credits: Amount of directed QE storage (DQED). QEs occupy this +* space until they are scheduled to a directed CQ. One credit represents +* the storage for one QE. +* - num_ldb_credit_pools: Number of pools into which the load-balanced credits +* are placed. +* - num_dir_credit_pools: Number of pools into which the directed credits are +* placed. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: domain ID. +*/ +struct dlb_create_sched_domain_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 num_ldb_queues; +__u32 num_ldb_ports; +__u32 num_dir_ports; +__u32 num_atomic_inflights; +__u32 num_hist_list_entries; +__u32 num_ldb_credits; +__u32 num_dir_credits; +__u32 num_ldb_credit_pools; +__u32 num_dir_credit_pools; +}; + +/* +* DLB_CMD_GET_NUM_RESOURCES: Return the number of available resources +* (queues, ports, etc.) that this device owns. +* +* Output parameters: +* - num_domains: Number of available scheduling domains. +* - num_ldb_queues: Number of available load-balanced queues. +* - num_ldb_ports: Number of available load-balanced ports. +* - num_dir_ports: Number of available directed ports. There is one directed +* queue for every directed port. +* - num_atomic_inflights: Amount of available temporary atomic QE storage. +* - max_contiguous_atomic_inflights: When a domain is created, the temporary +* atomic QE storage is allocated in a contiguous chunk. This return value +* is the longest available contiguous range of atomic QE storage. +* - num_hist_list_entries: Amount of history list storage. +* - max_contiguous_hist_list_entries: History list storage is allocated in +* a contiguous chunk, and this return value is the longest available +* contiguous range of history list entries. +* - num_ldb_credits: Amount of available load-balanced QE storage. +* - max_contiguous_ldb_credits: QED storage is allocated in a contiguous +* chunk, and this return value is the longest available contiguous range +* of load-balanced credit storage. +* - num_dir_credits: Amount of available directed QE storage. +* - max_contiguous_dir_credits: DQED storage is allocated in a contiguous +* chunk, and this return value is the longest available contiguous range +* of directed credit storage. +* - num_ldb_credit_pools: Number of available load-balanced credit pools. +* - num_dir_credit_pools: Number of available directed credit pools. +* - padding0: Reserved for future use. +*/ +struct dlb_get_num_resources_args { +/* Output parameters */ +__u32 num_sched_domains; +__u32 num_ldb_queues; +__u32 num_ldb_ports; +__u32 num_dir_ports; +__u32 num_atomic_inflights; +__u32 max_contiguous_atomic_inflights; +__u32 num_hist_list_entries; +__u32 max_contiguous_hist_list_entries; +__u32 num_ldb_credits; +__u32 max_contiguous_ldb_credits; +__u32 num_dir_credits; +__u32 max_contiguous_dir_credits; +__u32 num_ldb_credit_pools; +__u32 num_dir_credit_pools; +__u32 padding0; +}; + +/* +* DLB_CMD_SAMPLE_PERF_COUNTERS: Gather a set of DLB performance data by +* enabling performance counters for a user-specified measurement duration. +* This ioctl is blocking; the calling thread sleeps in the kernel driver +* for the duration of the measurement, then writes the data to user +* memory before returning. +* +* Certain metrics cannot be measured simultaneously, so multiple +* invocations of this command are necessary to gather all metrics. +* Metrics that can be collected simultaneously are grouped together in +* struct dlb_perf_metric_group_X. +* +* The driver allows only one active measurement at a time. If a thread +* calls this command while a measurement is ongoing, the thread will +* block until the original measurement completes. +* +* This ioctl is not supported for VF devices. +* +* Input parameters: +* - measurement_duration_us: Duration, in microseconds, of the +* measurement period. The duration must be between 1us and 60s, +* inclusive. +* - perf_metric_group_id: ID of the metric group to measure. +* - perf_metric_group_data: Pointer to union dlb_perf_metric_group_data +* structure. The driver will interpret the union according to +* perf_metric_group_ID. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_perf_metric_group_0 { +__u32 dlb_iosf_to_sys_enq_count; +__u32 dlb_sys_to_iosf_deq_count; +__u32 dlb_sys_to_dlb_enq_count; +__u32 dlb_dlb_to_sys_deq_count; +}; + +struct dlb_perf_metric_group_1 { +__u32 dlb_push_ptr_update_count; +}; + +struct dlb_perf_metric_group_2 { +__u32 dlb_avg_hist_list_depth; +}; + +struct dlb_perf_metric_group_3 { +__u32 dlb_avg_qed_depth; +}; + +struct dlb_perf_metric_group_4 { +__u32 dlb_avg_dqed_depth; +}; + +struct dlb_perf_metric_group_5 { +__u32 dlb_noop_hcw_count; +__u32 dlb_bat_t_hcw_count; +}; + +struct dlb_perf_metric_group_6 { +__u32 dlb_comp_hcw_count; +__u32 dlb_comp_t_hcw_count; +}; + +struct dlb_perf_metric_group_7 { +__u32 dlb_enq_hcw_count; +__u32 dlb_enq_t_hcw_count; +}; + +struct dlb_perf_metric_group_8 { +__u32 dlb_renq_hcw_count; +__u32 dlb_renq_t_hcw_count; +}; + +struct dlb_perf_metric_group_9 { +__u32 dlb_rel_hcw_count; +}; + +struct dlb_perf_metric_group_10 { +__u32 dlb_frag_hcw_count; +__u32 dlb_frag_t_hcw_count; +}; + +union dlb_perf_metric_group_data { +struct dlb_perf_metric_group_0 group_0; +struct dlb_perf_metric_group_1 group_1; +struct dlb_perf_metric_group_2 group_2; +struct dlb_perf_metric_group_3 group_3; +struct dlb_perf_metric_group_4 group_4; +struct dlb_perf_metric_group_5 group_5; +struct dlb_perf_metric_group_6 group_6; +struct dlb_perf_metric_group_7 group_7; +struct dlb_perf_metric_group_8 group_8; +struct dlb_perf_metric_group_9 group_9; +struct dlb_perf_metric_group_10 group_10; +}; + +struct dlb_sample_perf_counters_args { +/* Output parameters */ +__u64 elapsed_time_us; +__u64 response; +/* Input parameters */ +__u32 measurement_duration_us; +__u32 perf_metric_group_id; +__u64 perf_metric_group_data; +}; + +/* +* DLB_CMD_MEASURE_SCHED_COUNTS: Measure the DLB scheduling activity for a +* user-specified measurement duration. This ioctl is blocking; the +* calling thread sleeps in the kernel driver for the duration of the +* measurement, then writes the result to user memory before returning. +* +* Unlike the DLB_CMD_SAMPLE_PERF_COUNTERS ioctl, multiple threads can +* measure scheduling counts simultaneously. +* +* Note: VF devices can only measure the scheduling counts of their CQs; +* all other counts will be set to 0. +* +* Input parameters: +* - measurement_duration_us: Duration, in microseconds, of the +* measurement period. The duration must be between 1us and 60s, +* inclusive. +* - padding0: Reserved for future use. +* - sched_count_data: Pointer to a struct dlb_sched_count data structure. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_sched_counts { +__u64 ldb_sched_count; +__u64 dir_sched_count; +__u64 ldb_cq_sched_count[64 ]; +__u64 dir_cq_sched_count[128 ]; +}; + +struct dlb_measure_sched_count_args { +/* Output parameters */ +__u64 elapsed_time_us; +__u64 response; +/* Input parameters */ +__u32 measurement_duration_us; +__u32 padding0; +__u64 sched_count_data; +}; + +/* +* DLB_CMD_SET_SN_ALLOCATION: Configure a sequence number group +* +* Input parameters: +* - group: Sequence number group ID. +* - num: Number of sequence numbers per queue. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_set_sn_allocation_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 group; +__u32 num; +}; + +/* +* DLB_CMD_GET_SN_ALLOCATION: Get a sequence number group's configuration +* +* Input parameters: +* - group: Sequence number group ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Specified group's number of sequence numbers per queue. +*/ +struct dlb_get_sn_allocation_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 group; +__u32 padding0; +}; + +enum dlb_cq_poll_modes { +DLB_CQ_POLL_MODE_STD , +DLB_CQ_POLL_MODE_SPARSE , + +/* NUM_DLB_CQ_POLL_MODE must be last */ +NUM_DLB_CQ_POLL_MODE , +}; + +/* +* DLB_CMD_QUERY_CQ_POLL_MODE: Query the CQ poll mode the kernel driver is using +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: CQ poll mode (see enum dlb_cq_poll_modes). +*/ +struct dlb_query_cq_poll_mode_args { +/* Output parameters */ +__u64 response; +}; + +/* +* DLB_CMD_GET_SN_OCCUPANCY: Get a sequence number group's occupancy +* +* Each sequence number group has one or more slots, depending on its +* configuration. I.e.: +* - If configured for 1024 sequence numbers per queue, the group has 1 slot +* - If configured for 512 sequence numbers per queue, the group has 2 slots +* ... +* - If configured for 32 sequence numbers per queue, the group has 32 slots +* +* This ioctl returns the group's number of in-use slots. If its occupancy is +* 0, the group's sequence number allocation can be reconfigured. +* +* Input parameters: +* - group: Sequence number group ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Specified group's number of used slots. +*/ +struct dlb_get_sn_occupancy_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 group; +__u32 padding0; +}; + +enum dlb_user_interface_commands { +DLB_CMD_GET_DEVICE_VERSION , +DLB_CMD_CREATE_SCHED_DOMAIN , +DLB_CMD_GET_NUM_RESOURCES , +DLB_CMD_GET_DRIVER_VERSION , +DLB_CMD_SAMPLE_PERF_COUNTERS , +DLB_CMD_SET_SN_ALLOCATION , +DLB_CMD_GET_SN_ALLOCATION , +DLB_CMD_MEASURE_SCHED_COUNTS , +DLB_CMD_QUERY_CQ_POLL_MODE , +DLB_CMD_GET_SN_OCCUPANCY , + +/* NUM_DLB_CMD must be last */ +NUM_DLB_CMD , +}; + +/*******************************/ +/* 'domain' device file alerts */ +/*******************************/ + +/* Scheduling domain device files can be read to receive domain-specific +* notifications, for alerts such as hardware errors. +* +* Each alert is encoded in a 16B message. The first 8B contains the alert ID, +* and the second 8B is optional and contains additional information. +* Applications should cast read data to a struct dlb_domain_alert, and +* interpret the struct's alert_id according to dlb_domain_alert_id. The read +* length must be 16B, or the function will return -EINVAL. +* +* Reads are destructive, and in the case of multiple file descriptors for the +* same domain device file, an alert will be read by only one of the file +* descriptors. +* +* The driver stores alerts in a fixed-size alert ring until they are read. If +* the alert ring fills completely, subsequent alerts will be dropped. It is +* recommended that DLB applications dedicate a thread to perform blocking +* reads on the device file. +*/ +enum dlb_domain_alert_id { +/* A destination domain queue that this domain connected to has +* unregistered, and can no longer be sent to. The aux alert data +* contains the queue ID. +*/ +DLB_DOMAIN_ALERT_REMOTE_QUEUE_UNREGISTER , +/* A producer port in this domain attempted to send a QE without a +* credit. aux_alert_data[7:0] contains the port ID, and +* aux_alert_data[15:8] contains a flag indicating whether the port is +* load-balanced (1) or directed (0). +*/ +DLB_DOMAIN_ALERT_PP_OUT_OF_CREDITS , +/* Software issued an illegal enqueue for a port in this domain. An +* illegal enqueue could be: +* - Illegal (excess) completion +* - Illegal fragment +* - Illegal enqueue command +* aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8] +* contains a flag indicating whether the port is load-balanced (1) or +* directed (0). +*/ +DLB_DOMAIN_ALERT_PP_ILLEGAL_ENQ , +/* Software issued excess CQ token pops for a port in this domain. +* aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8] +* contains a flag indicating whether the port is load-balanced (1) or +* directed (0). +*/ +DLB_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS , +/* A enqueue contained either an invalid command encoding or a REL, +* REL_T, RLS, FWD, FWD_T, FRAG, or FRAG_T from a directed port. +* +* aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8] +* contains a flag indicating whether the port is load-balanced (1) or +* directed (0). +*/ +DLB_DOMAIN_ALERT_ILLEGAL_HCW , +/* The QID must be valid and less than 128. +* +* aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8] +* contains a flag indicating whether the port is load-balanced (1) or +* directed (0). +*/ +DLB_DOMAIN_ALERT_ILLEGAL_QID , +/* An enqueue went to a disabled QID. +* +* aux_alert_data[7:0] contains the port ID, and aux_alert_data[15:8] +* contains a flag indicating whether the port is load-balanced (1) or +* directed (0). +*/ +DLB_DOMAIN_ALERT_DISABLED_QID , +/* The device containing this domain was reset. All applications using +* the device need to exit for the driver to complete the reset +* procedure. +* +* aux_alert_data doesn't contain any information for this alert. +*/ +DLB_DOMAIN_ALERT_DEVICE_RESET , +/* User-space has enqueued an alert. +* +* aux_alert_data contains user-provided data. +*/ +DLB_DOMAIN_ALERT_USER , + +/* Number of DLB domain alerts */ +NUM_DLB_DOMAIN_ALERTS +}; + +static const char dlb_domain_alert_strings[][128 ] = { +"DLB_DOMAIN_ALERT_REMOTE_QUEUE_UNREGISTER" , +"DLB_DOMAIN_ALERT_PP_OUT_OF_CREDITS" , +"DLB_DOMAIN_ALERT_PP_ILLEGAL_ENQ" , +"DLB_DOMAIN_ALERT_PP_EXCESS_TOKEN_POPS" , +"DLB_DOMAIN_ALERT_ILLEGAL_HCW" , +"DLB_DOMAIN_ALERT_ILLEGAL_QID" , +"DLB_DOMAIN_ALERT_DISABLED_QID" , +"DLB_DOMAIN_ALERT_DEVICE_RESET" , +"DLB_DOMAIN_ALERT_USER" , +}; + +struct dlb_domain_alert { +__u64 alert_id; +__u64 aux_alert_data; +}; + +/*********************************/ +/* 'domain' device file commands */ +/*********************************/ + +/* +* DLB_DOMAIN_CMD_CREATE_LDB_POOL: Configure a load-balanced credit pool. +* Input parameters: +* - num_ldb_credits: Number of load-balanced credits (QED space) for this +* pool. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: pool ID. +*/ +struct dlb_create_ldb_pool_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 num_ldb_credits; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_CREATE_DIR_POOL: Configure a directed credit pool. +* Input parameters: +* - num_dir_credits: Number of directed credits (DQED space) for this pool. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Pool ID. +*/ +struct dlb_create_dir_pool_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 num_dir_credits; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_CREATE_LDB_QUEUE: Configure a load-balanced queue. +* Input parameters: +* - num_atomic_inflights: This specifies the amount of temporary atomic QE +* storage for this queue. If zero, the queue will not support atomic +* scheduling. +* - num_sequence_numbers: This specifies the number of sequence numbers used +* by this queue. If zero, the queue will not support ordered scheduling. +* If non-zero, the queue will not support unordered scheduling. +* - num_qid_inflights: The maximum number of QEs that can be inflight +* (scheduled to a CQ but not completed) at any time. If +* num_sequence_numbers is non-zero, num_qid_inflights must be set equal +* to num_sequence_numbers. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Queue ID. +*/ +struct dlb_create_ldb_queue_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 num_sequence_numbers; +__u32 num_qid_inflights; +__u32 num_atomic_inflights; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_CREATE_DIR_QUEUE: Configure a directed queue. +* Input parameters: +* - port_id: Port ID. If the corresponding directed port is already created, +* specify its ID here. Else this argument must be 0xFFFFFFFF to indicate +* that the queue is being created before the port. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Queue ID. +*/ +struct dlb_create_dir_queue_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__s32 port_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_CREATE_LDB_PORT: Configure a load-balanced port. +* Input parameters: +* - ldb_credit_pool_id: Load-balanced credit pool this port will belong to. +* - dir_credit_pool_id: Directed credit pool this port will belong to. +* - ldb_credit_high_watermark: Number of load-balanced credits from the pool +* that this port will own. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_high_watermark: Number of directed credits from the pool that +* this port will own. +* +* If this port's scheduling domain doesn't have any directed queues, +* this argument is ignored and the port is given no directed credits. +* - ldb_credit_low_watermark: Load-balanced credit low watermark. When the +* port's credits reach this watermark, they become eligible to be +* refilled by the DLB as credits until the high watermark +* (num_ldb_credits) is reached. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_low_watermark: Directed credit low watermark. When the port's +* credits reach this watermark, they become eligible to be refilled by +* the DLB as credits until the high watermark (num_dir_credits) is +* reached. +* +* If this port's scheduling domain doesn't have any directed queues, +* this argument is ignored and the port is given no directed credits. +* - ldb_credit_quantum: Number of load-balanced credits for the DLB to refill +* per refill operation. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_quantum: Number of directed credits for the DLB to refill per +* refill operation. +* +* If this port's scheduling domain doesn't have any directed queues, +* this argument is ignored and the port is given no directed credits. +* - padding0: Reserved for future use. +* - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and +* 1024, inclusive. +* - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that +* the CQ interrupt won't fire until there are N or more outstanding CQ +* tokens. +* - cq_history_list_size: Number of history list entries. This must be greater +* than or equal to cq_depth. +* - padding1: Reserved for future use. +* - padding2: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: port ID. +*/ +struct dlb_create_ldb_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 ldb_credit_pool_id; +__u32 dir_credit_pool_id; +__u16 ldb_credit_high_watermark; +__u16 ldb_credit_low_watermark; +__u16 ldb_credit_quantum; +__u16 dir_credit_high_watermark; +__u16 dir_credit_low_watermark; +__u16 dir_credit_quantum; +__u16 padding0; +__u16 cq_depth; +__u16 cq_depth_threshold; +__u16 cq_history_list_size; +__u32 padding1; +}; + +/* +* DLB_DOMAIN_CMD_CREATE_DIR_PORT: Configure a directed port. +* Input parameters: +* - ldb_credit_pool_id: Load-balanced credit pool this port will belong to. +* - dir_credit_pool_id: Directed credit pool this port will belong to. +* - ldb_credit_high_watermark: Number of load-balanced credits from the pool +* that this port will own. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_high_watermark: Number of directed credits from the pool that +* this port will own. +* - ldb_credit_low_watermark: Load-balanced credit low watermark. When the +* port's credits reach this watermark, they become eligible to be +* refilled by the DLB as credits until the high watermark +* (num_ldb_credits) is reached. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_low_watermark: Directed credit low watermark. When the port's +* credits reach this watermark, they become eligible to be refilled by +* the DLB as credits until the high watermark (num_dir_credits) is +* reached. +* - ldb_credit_quantum: Number of load-balanced credits for the DLB to refill +* per refill operation. +* +* If this port's scheduling domain doesn't have any load-balanced queues, +* this argument is ignored and the port is given no load-balanced +* credits. +* - dir_credit_quantum: Number of directed credits for the DLB to refill per +* refill operation. +* - cq_depth: Depth of the port's CQ. Must be a power-of-two between 8 and +* 1024, inclusive. +* - cq_depth_threshold: CQ depth interrupt threshold. A value of N means that +* the CQ interrupt won't fire until there are N or more outstanding CQ +* tokens. +* - qid: Queue ID. If the corresponding directed queue is already created, +* specify its ID here. Else this argument must be 0xFFFFFFFF to indicate +* that the port is being created before the queue. +* - padding1: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: Port ID. +*/ +struct dlb_create_dir_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 ldb_credit_pool_id; +__u32 dir_credit_pool_id; +__u16 ldb_credit_high_watermark; +__u16 ldb_credit_low_watermark; +__u16 ldb_credit_quantum; +__u16 dir_credit_high_watermark; +__u16 dir_credit_low_watermark; +__u16 dir_credit_quantum; +__u16 cq_depth; +__u16 cq_depth_threshold; +__s32 queue_id; +__u32 padding1; +}; + +/* +* DLB_DOMAIN_CMD_START_DOMAIN: Mark the end of the domain configuration. This +* must be called before passing QEs into the device, and no configuration +* ioctls can be issued once the domain has started. Sending QEs into the +* device before calling this ioctl will result in undefined behavior. +* Input parameters: +* - (None) +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_start_domain_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +}; + +/* +* DLB_DOMAIN_CMD_MAP_QID: Map a load-balanced queue to a load-balanced port. +* Input parameters: +* - port_id: Load-balanced port ID. +* - qid: Load-balanced queue ID. +* - priority: Queue->port service priority. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_map_qid_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 qid; +__u32 priority; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_UNMAP_QID: Unmap a load-balanced queue to a load-balanced +* port. +* Input parameters: +* - port_id: Load-balanced port ID. +* - qid: Load-balanced queue ID. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_unmap_qid_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 qid; +}; + +/* +* DLB_DOMAIN_CMD_ENABLE_LDB_PORT: Enable scheduling to a load-balanced port. +* Input parameters: +* - port_id: Load-balanced port ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_enable_ldb_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_ENABLE_DIR_PORT: Enable scheduling to a directed port. +* Input parameters: +* - port_id: Directed port ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_enable_dir_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +}; + +/* +* DLB_DOMAIN_CMD_DISABLE_LDB_PORT: Disable scheduling to a load-balanced port. +* Input parameters: +* - port_id: Load-balanced port ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_disable_ldb_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_DISABLE_DIR_PORT: Disable scheduling to a directed port. +* Input parameters: +* - port_id: Directed port ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_disable_dir_port_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT: Block on a CQ interrupt until a QE +* arrives for the specified port. If a QE is already present, the ioctl +* will immediately return. +* +* Note: Only one thread can block on a CQ's interrupt at a time. Doing +* otherwise can result in hung threads. +* +* Input parameters: +* - port_id: Port ID. +* - is_ldb: True if the port is load-balanced, false otherwise. +* - arm: Tell the driver to arm the interrupt. +* - cq_gen: Current CQ generation bit. +* - padding0: Reserved for future use. +* - cq_va: VA of the CQ entry where the next QE will be placed. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_block_on_cq_interrupt_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u8 is_ldb; +__u8 arm; +__u8 cq_gen; +__u8 padding0; +__u64 cq_va; +}; + +/* +* DLB_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT: Enqueue a domain alert that will be +* read by one reader thread. +* +* Input parameters: +* - aux_alert_data: user-defined auxiliary data. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +*/ +struct dlb_enqueue_domain_alert_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u64 aux_alert_data; +}; + +/* +* DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH: Get a load-balanced queue's depth. +* Input parameters: +* - queue_id: The load-balanced queue ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: queue depth. +*/ +struct dlb_get_ldb_queue_depth_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 queue_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH: Get a directed queue's depth. +* Input parameters: +* - queue_id: The directed queue ID. +* - padding0: Reserved for future use. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: queue depth. +*/ +struct dlb_get_dir_queue_depth_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 queue_id; +__u32 padding0; +}; + +/* +* DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS: Get number of queue unmap operations in +* progress for a load-balanced port. +* +* Note: This is a snapshot; the number of unmap operations in progress +* is subject to change at any time. +* +* Input parameters: +* - port_id: Load-balanced port ID. +* +* Output parameters: +* - response: pointer to a struct dlb_cmd_response. +* response.status: Detailed error code. In certain cases, such as if the +* response pointer is invalid, the driver won't set status. +* response.id: number of unmaps in progress. +*/ +struct dlb_pending_port_unmaps_args { +/* Output parameters */ +__u64 response; +/* Input parameters */ +__u32 port_id; +__u32 padding0; +}; + +enum dlb_domain_user_interface_commands { +DLB_DOMAIN_CMD_CREATE_LDB_POOL , +DLB_DOMAIN_CMD_CREATE_DIR_POOL , +DLB_DOMAIN_CMD_CREATE_LDB_QUEUE , +DLB_DOMAIN_CMD_CREATE_DIR_QUEUE , +DLB_DOMAIN_CMD_CREATE_LDB_PORT , +DLB_DOMAIN_CMD_CREATE_DIR_PORT , +DLB_DOMAIN_CMD_START_DOMAIN , +DLB_DOMAIN_CMD_MAP_QID , +DLB_DOMAIN_CMD_UNMAP_QID , +DLB_DOMAIN_CMD_ENABLE_LDB_PORT , +DLB_DOMAIN_CMD_ENABLE_DIR_PORT , +DLB_DOMAIN_CMD_DISABLE_LDB_PORT , +DLB_DOMAIN_CMD_DISABLE_DIR_PORT , +DLB_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT , +DLB_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT , +DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH , +DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH , +DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS , + +/* NUM_DLB_DOMAIN_CMD must be last */ +NUM_DLB_DOMAIN_CMD , +}; + +/* +* Base addresses for memory mapping the consumer queue (CQ) and popcount (PC) +* memory space, and producer port (PP) MMIO space. The CQ, PC, and PP +* addresses are per-port. Every address is page-separated (e.g. LDB PP 0 is at +* 0x2100000 and LDB PP 1 is at 0x2101000). +*/ +#define DLB_LDB_CQ_BASE 0x3000000 +#define DLB_LDB_CQ_MAX_SIZE 65536 +#define DLB_LDB_CQ_OFFS (id) (DLB_LDB_CQ_BASE + (id) * DLB_LDB_CQ_MAX_SIZE) + +#define DLB_DIR_CQ_BASE 0x3800000 +#define DLB_DIR_CQ_MAX_SIZE 65536 +#define DLB_DIR_CQ_OFFS (id) (DLB_DIR_CQ_BASE + (id) * DLB_DIR_CQ_MAX_SIZE) + +#define DLB_LDB_PC_BASE 0x2300000 +#define DLB_LDB_PC_MAX_SIZE 4096 +#define DLB_LDB_PC_OFFS (id) (DLB_LDB_PC_BASE + (id) * DLB_LDB_PC_MAX_SIZE) + +#define DLB_DIR_PC_BASE 0x2200000 +#define DLB_DIR_PC_MAX_SIZE 4096 +#define DLB_DIR_PC_OFFS (id) (DLB_DIR_PC_BASE + (id) * DLB_DIR_PC_MAX_SIZE) + +#define DLB_LDB_PP_BASE 0x2100000 +#define DLB_LDB_PP_MAX_SIZE 4096 +#define DLB_LDB_PP_OFFS (id) (DLB_LDB_PP_BASE + (id) * DLB_LDB_PP_MAX_SIZE) + +#define DLB_DIR_PP_BASE 0x2000000 +#define DLB_DIR_PP_MAX_SIZE 4096 +#define DLB_DIR_PP_OFFS (id) (DLB_DIR_PP_BASE + (id) * DLB_DIR_PP_MAX_SIZE) + +/*******************/ +/* dlb ioctl codes */ +/*******************/ + +#define DLB_IOC_MAGIC 'h' + +#define DLB_IOC_GET_DEVICE_VERSION \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_GET_DEVICE_VERSION, \ + struct dlb_get_driver_version_args) +#define DLB_IOC_CREATE_SCHED_DOMAIN \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_CREATE_SCHED_DOMAIN, \ + struct dlb_create_sched_domain_args) +#define DLB_IOC_GET_NUM_RESOURCES \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_GET_NUM_RESOURCES, \ + struct dlb_get_num_resources_args) +#define DLB_IOC_GET_DRIVER_VERSION \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_GET_DRIVER_VERSION, \ + struct dlb_get_driver_version_args) +#define DLB_IOC_SAMPLE_PERF_COUNTERS \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_SAMPLE_PERF_COUNTERS, \ + struct dlb_sample_perf_counters_args) +#define DLB_IOC_SET_SN_ALLOCATION \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_SET_SN_ALLOCATION, \ + struct dlb_set_sn_allocation_args) +#define DLB_IOC_GET_SN_ALLOCATION \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_GET_SN_ALLOCATION, \ + struct dlb_get_sn_allocation_args) +#define DLB_IOC_MEASURE_SCHED_COUNTS \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_MEASURE_SCHED_COUNTS, \ + struct dlb_measure_sched_count_args) +#define DLB_IOC_QUERY_CQ_POLL_MODE \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_QUERY_CQ_POLL_MODE, \ + struct dlb_query_cq_poll_mode_args) +#define DLB_IOC_GET_SN_OCCUPANCY \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_CMD_GET_SN_OCCUPANCY, \ + struct dlb_get_sn_occupancy_args) +#define DLB_IOC_CREATE_LDB_POOL \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_LDB_POOL, \ + struct dlb_create_ldb_pool_args) +#define DLB_IOC_CREATE_DIR_POOL \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_DIR_POOL, \ + struct dlb_create_dir_pool_args) +#define DLB_IOC_CREATE_LDB_QUEUE \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_LDB_QUEUE, \ + struct dlb_create_ldb_queue_args) +#define DLB_IOC_CREATE_DIR_QUEUE \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_DIR_QUEUE, \ + struct dlb_create_dir_queue_args) +#define DLB_IOC_CREATE_LDB_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_LDB_PORT, \ + struct dlb_create_ldb_port_args) +#define DLB_IOC_CREATE_DIR_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_CREATE_DIR_PORT, \ + struct dlb_create_dir_port_args) +#define DLB_IOC_START_DOMAIN \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_START_DOMAIN, \ + struct dlb_start_domain_args) +#define DLB_IOC_MAP_QID \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_MAP_QID, \ + struct dlb_map_qid_args) +#define DLB_IOC_UNMAP_QID \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_UNMAP_QID, \ + struct dlb_unmap_qid_args) +#define DLB_IOC_ENABLE_LDB_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_ENABLE_LDB_PORT, \ + struct dlb_enable_ldb_port_args) +#define DLB_IOC_ENABLE_DIR_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_ENABLE_DIR_PORT, \ + struct dlb_enable_dir_port_args) +#define DLB_IOC_DISABLE_LDB_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_DISABLE_LDB_PORT, \ + struct dlb_disable_ldb_port_args) +#define DLB_IOC_DISABLE_DIR_PORT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_DISABLE_DIR_PORT, \ + struct dlb_disable_dir_port_args) +#define DLB_IOC_BLOCK_ON_CQ_INTERRUPT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_BLOCK_ON_CQ_INTERRUPT, \ + struct dlb_block_on_cq_interrupt_args) +#define DLB_IOC_ENQUEUE_DOMAIN_ALERT \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_ENQUEUE_DOMAIN_ALERT, \ + struct dlb_enqueue_domain_alert_args) +#define DLB_IOC_GET_LDB_QUEUE_DEPTH \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_GET_LDB_QUEUE_DEPTH, \ + struct dlb_get_ldb_queue_depth_args) +#define DLB_IOC_GET_DIR_QUEUE_DEPTH \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_GET_DIR_QUEUE_DEPTH, \ + struct dlb_get_dir_queue_depth_args) +#define DLB_IOC_PENDING_PORT_UNMAPS \ + _IOWR(DLB_IOC_MAGIC, \ + DLB_DOMAIN_CMD_PENDING_PORT_UNMAPS, \ + struct dlb_pending_port_unmaps_args) + +#endif /* __DLB_USER_H */
Signature · 1 line
2.13.6
PATCH 10/27 event/dlb: add PFPMD-specific interface layer to shared code
In reply to McDaniel, Timothy--- drivers/event/dlb/pf/dlb_main.c | 670 ++++++++++++++++++++++++++++++++ drivers/event/dlb/pf/dlb_main.h | 90 +++++ drivers/event/dlb/pf/dlb_pf.c | 839 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1599 insertions(+) create mode 100644 drivers/event/dlb/pf/dlb_main.c create mode 100644 drivers/event/dlb/pf/dlb_main.h create mode 100644 drivers/event/dlb/pf/dlb_pf.c
drivers/event/dlb/pf/dlb_main.c new file +670 -0
@@ -0,0 +1,670 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include <stdint.h> +#include <stdbool.h> +#include <stdio.h> +#include <errno.h> +#include <assert.h> +#include <unistd.h> +#include <string.h> + +#include <rte_malloc.h> +#include <rte_errno.h> + +#include "base/dlb_resource.h" +#include "base/dlb_osdep.h" +#include "base/dlb_regs.h" +#include "dlb_main.h" +#include "../dlb_user.h" + + +#define PF_ID_ZERO 0/* PF ONLY! */ +#define NO_OWNER_VF 0/* PF ONLY! */ +#define NOT_VF_REQ false/* PF ONLY! */ + +unsigned int dlb_unregister_timeout_s =DLB_DEFAULT_UNREGISTER_TIMEOUT_S ; + +#define DLB_PCI_CFG_SPACE_SIZE 256 +#define DLB_PCI_CAP_POINTER 0x34 +#define DLB_PCI_CAP_NEXT (hdr) (((hdr) >> 8) & 0xFC) +#define DLB_PCI_CAP_ID (hdr) ((hdr) & 0xFF) +#define DLB_PCI_EXT_CAP_NEXT (hdr) (((hdr) >> 20) & 0xFFC) +#define DLB_PCI_EXT_CAP_ID (hdr) ((hdr) & 0xFFFF) +#define DLB_PCI_EXT_CAP_ID_ERR 1 +#define DLB_PCI_ERR_UNCOR_MASK 8 +#define DLB_PCI_ERR_UNC_UNSUP 0x00100000 + +#define DLB_PCI_EXP_DEVCTL 8 +#define DLB_PCI_LNKCTL 16 +#define DLB_PCI_SLTCTL 24 +#define DLB_PCI_RTCTL 28 +#define DLB_PCI_EXP_DEVCTL2 40 +#define DLB_PCI_LNKCTL2 48 +#define DLB_PCI_SLTCTL2 56 +#define DLB_PCI_CMD 4 +#define DLB_PCI_X_CMD 2 +#define DLB_PCI_EXP_DEVSTA 10 +#define DLB_PCI_EXP_DEVSTA_TRPND 0x20 +#define DLB_PCI_EXP_DEVCTL_BCR_FLR 0x8000 +#define DLB_PCI_PASID_CTRL 6 +#define DLB_PCI_PASID_CAP 4 + +#define DLB_PCI_CAP_ID_EXP 0x10 +#define DLB_PCI_CAP_ID_MSIX 0x11 +#define DLB_PCI_EXT_CAP_ID_PAS 0x1B +#define DLB_PCI_EXT_CAP_ID_PRI 0x13 +#define DLB_PCI_EXT_CAP_ID_ACS 0xD + +#define DLB_PCI_PASID_CAP_EXEC 0x2 +#define DLB_PCI_PASID_CAP_PRIV 0x4 +#define DLB_PCI_PASID_CTRL_ENABLE 0x1 +#define DLB_PCI_PRI_CTRL_ENABLE 0x1 +#define DLB_PCI_PRI_ALLOC_REQ 0xC +#define DLB_PCI_PRI_CTRL 0x4 +#define DLB_PCI_MSIX_FLAGS 0x2 +#define DLB_PCI_MSIX_FLAGS_ENABLE 0x8000 +#define DLB_PCI_MSIX_FLAGS_MASKALL 0x4000 +#define DLB_PCI_ERR_ROOT_STATUS 0x30 +#define DLB_PCI_ERR_COR_STATUS 0x10 +#define DLB_PCI_ERR_UNCOR_STATUS 0x4 +#define DLB_PCI_COMMAND_INTX_DISABLE 0x400 +#define DLB_PCI_ACS_CAP 0x4 +#define DLB_PCI_ACS_CTRL 0x6 +#define DLB_PCI_ACS_SV 0x1 +#define DLB_PCI_ACS_RR 0x4 +#define DLB_PCI_ACS_CR 0x8 +#define DLB_PCI_ACS_UF 0x10 +#define DLB_PCI_ACS_EC 0x20 + +static inline void +dlb_movntdq (void *qe4,void *pp_addr) +{ +/* Move entire 64B cache line of QEs, 128 bits (16B) at a time. */ +long long *_qe = (long long *)qe4; +__v2di src_data0 = (__v2di ){_qe[0 ], _qe[1 ]}; +__v2di src_data1 = (__v2di ){_qe[2 ], _qe[3 ]}; +__v2di src_data2 = (__v2di ){_qe[4 ], _qe[5 ]}; +__v2di src_data3 = (__v2di ){_qe[6 ], _qe[7 ]}; +__builtin_ia32_movntdq ((__v2di *)pp_addr +0 , (__v2di )src_data0); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +1 , (__v2di )src_data1); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +2 , (__v2di )src_data2); +rte_wmb (); +__builtin_ia32_movntdq ((__v2di *)pp_addr +3 , (__v2di )src_data3); +rte_wmb (); +} + +static inline void +dlb_movdir64b (void *qe4,void *pp_addr) +{ + asmvolatile (".byte 0x66, 0x0f, 0x38, 0xf8, 0x02" + : + :"a" (pp_addr),"d" (qe4)); +} + + +static int dlb_pci_find_ext_capability (struct rte_pci_device *pdev,uint32_t id) +{ +uint32_t hdr; +size_t sz; +int pos; + + pos =DLB_PCI_CFG_SPACE_SIZE ; + sz =sizeof (hdr); + +while (pos >0xFF ) { +if (rte_pci_read_config (pdev, &hdr, sz, pos) != (int )sz) +return -1 ; + +if (DLB_PCI_EXT_CAP_ID (hdr) == id) +return pos; + + pos =DLB_PCI_EXT_CAP_NEXT (hdr); + } + +return -1 ; +} + +static int dlb_pci_find_capability (struct rte_pci_device *pdev,uint32_t id) +{ +uint8_t pos; +int ret; +uint16_t hdr; + + ret =rte_pci_read_config (pdev, &pos,1 ,DLB_PCI_CAP_POINTER ); + pos &=0xFC ; + +if (ret !=1 ) +return -1 ; + +while (pos >0x3F ) { + ret =rte_pci_read_config (pdev, &hdr,2 , pos); +if (ret !=2 ) +return -1 ; + +if (DLB_PCI_CAP_ID (hdr) == id) +return pos; + +if (DLB_PCI_CAP_ID (hdr) ==0xFF ) +return -1 ; + + pos =DLB_PCI_CAP_NEXT (hdr); + } + +return -1 ; +} + +static int dlb_mask_ur_err (struct rte_pci_device *pdev) +{ +uint32_t mask; +size_t sz =sizeof (mask); +int pos =dlb_pci_find_ext_capability (pdev,DLB_PCI_EXT_CAP_ID_ERR ); + +if (pos <0 ) { +printf ("[%s()] failed to find the aer capability\n" , + __func__); +return pos; + } + + pos +=DLB_PCI_ERR_UNCOR_MASK ; + +if (rte_pci_read_config (pdev, &mask, sz, pos) != (int )sz) { +printf ("[%s()] Failed to read uncorrectable error mask reg\n" , + __func__); +return -1 ; + } + +/* Mask Unsupported Request errors */ + mask |=DLB_PCI_ERR_UNC_UNSUP ; + +if (rte_pci_write_config (pdev, &mask, sz, pos) != (int )sz) { +printf ("[%s()] Failed to write uncorrectable error mask reg at offset %d\n" , + __func__, pos); +return -1 ; + } + +return 0 ; +} + +struct dlb_dev * +dlb_probe (struct rte_pci_device *pdev) +{ +struct dlb_dev *dlb_dev; +int ret =0 ; + +DLB_INFO (dlb_dev,"probe\n" ); + + dlb_dev =rte_malloc ("DLB_PF" ,sizeof (struct dlb_dev ), +RTE_CACHE_LINE_SIZE ); + +if (!dlb_dev) { + ret = -ENOMEM ; + goto dlb_dev_malloc_fail; + } + +/* PCI Bus driver has already mapped bar space into process. +* Save off our IO register and FUNC addresses. +*/ + +/* BAR 0 */ +if (pdev->mem_resource[0 ].addr ==NULL ) { +DLB_ERR (dlb_dev,"probe: BAR 0 addr (csr_kva) is NULL\n" ); + ret = -EINVAL ; + goto pci_mmap_bad_addr; + } + dlb_dev->hw.func_kva = (void *)(uintptr_t )pdev->mem_resource[0 ].addr; + dlb_dev->hw.func_phys_addr = pdev->mem_resource[0 ].phys_addr; + +DLB_INFO (dlb_dev,"DLB FUNC VA=%p, PA=%p, len=%" PRIu64"\n" , + (void *)dlb_dev->hw.func_kva, + (void *)dlb_dev->hw.func_phys_addr, + pdev->mem_resource[0 ].len); + +/* BAR 2 */ +if (pdev->mem_resource[2 ].addr ==NULL ) { +DLB_ERR (dlb_dev,"probe: BAR 2 addr (func_kva) is NULL\n" ); + ret = -EINVAL ; + goto pci_mmap_bad_addr; + } + dlb_dev->hw.csr_kva = (void *)(uintptr_t )pdev->mem_resource[2 ].addr; + dlb_dev->hw.csr_phys_addr = pdev->mem_resource[2 ].phys_addr; + +DLB_INFO (dlb_dev,"DLB CSR VA=%p, PA=%p, len=%" PRIu64"\n" , + (void *)dlb_dev->hw.csr_kva, + (void *)dlb_dev->hw.csr_phys_addr, + pdev->mem_resource[2 ].len); + + dlb_dev->pdev = pdev; + + ret =dlb_pf_reset (dlb_dev); +if (ret) + goto dlb_reset_fail; + +/* DLB incorrectly sends URs in response to certain messages. Mask UR +* errors to prevent these from being propagated to the MCA. +*/ + ret =dlb_mask_ur_err (pdev); +if (ret) + goto mask_ur_err_fail; + + ret =dlb_pf_init_interrupts (dlb_dev); +if (ret) + goto init_interrupts_fail; + + ret =dlb_pf_init_driver_state (dlb_dev); +if (ret) + goto init_driver_state_fail; + + ret =dlb_resource_init (&dlb_dev->hw); +if (ret) + goto resource_init_fail; + + dlb_dev->revision =os_get_dev_revision (&dlb_dev->hw); + +dlb_pf_init_hardware (dlb_dev); + +return dlb_dev; + +resource_init_fail: +dlb_resource_free (&dlb_dev->hw); +init_driver_state_fail: +dlb_pf_free_interrupts (dlb_dev); +init_interrupts_fail: +mask_ur_err_fail: +dlb_reset_fail: +pci_mmap_bad_addr: +rte_free (dlb_dev); +dlb_dev_malloc_fail: + rte_errno = ret; +return NULL ; +} + +int +dlb_pf_reset (struct dlb_dev *dlb_dev) +{ +int ret =0 ; +int i =0 ; +uint32_t dword[16 ]; +uint16_t cmd; +off_t off; + +uint16_t dev_ctl_word; +uint16_t dev_ctl2_word; +uint16_t lnk_word; +uint16_t lnk_word2; +uint16_t slt_word; +uint16_t slt_word2; +uint16_t rt_ctl_word; +uint32_t pri_reqs_dword; +uint16_t pri_ctrl_word; + +int pcie_cap_offset; +int pasid_cap_offset; +int pri_cap_offset; +int msix_cap_offset; +int err_cap_offset; +int acs_cap_offset; +int wait_count; + +uint16_t devsta_busy_word; +uint16_t devctl_word; +uint16_t pasid_ctrl_word; +uint16_t pasid_features; + +struct rte_pci_device *pdev = dlb_dev->pdev; + +/* Save PCI config state */ + +for (i =0 ; i <16 ; i++) { +if (rte_pci_read_config (pdev, &dword[i],4 , i *4 ) !=4 ) +return ret; + } + + pcie_cap_offset =dlb_pci_find_capability (pdev,DLB_PCI_CAP_ID_EXP ); + +if (pcie_cap_offset <0 ) { +printf ("[%s()] failed to find the pcie capability\n" , + __func__); +return pcie_cap_offset; + } + + off = pcie_cap_offset +DLB_PCI_EXP_DEVCTL ; +if (rte_pci_read_config (pdev, &dev_ctl_word,2 , off) !=2 ) + dev_ctl_word =0 ; + + off = pcie_cap_offset +DLB_PCI_LNKCTL ; +if (rte_pci_read_config (pdev, &lnk_word,2 , off) !=2 ) + lnk_word =0 ; + + off = pcie_cap_offset +DLB_PCI_SLTCTL ; +if (rte_pci_read_config (pdev, &slt_word,2 , off) !=2 ) + slt_word =0 ; + + off = pcie_cap_offset +DLB_PCI_RTCTL ; +if (rte_pci_read_config (pdev, &rt_ctl_word,2 , off) !=2 ) + rt_ctl_word =0 ; + + off = pcie_cap_offset +DLB_PCI_EXP_DEVCTL2 ; +if (rte_pci_read_config (pdev, &dev_ctl2_word,2 , off) !=2 ) + dev_ctl2_word =0 ; + + off = pcie_cap_offset +DLB_PCI_LNKCTL2 ; +if (rte_pci_read_config (pdev, &lnk_word2,2 , off) !=2 ) + lnk_word2 =0 ; + + off = pcie_cap_offset +DLB_PCI_SLTCTL2 ; +if (rte_pci_read_config (pdev, &slt_word2,2 , off) !=2 ) + slt_word2 =0 ; + + pri_cap_offset =dlb_pci_find_ext_capability (pdev, +DLB_PCI_EXT_CAP_ID_PRI ); +if (pri_cap_offset >=0 ) { + off = pri_cap_offset +DLB_PCI_PRI_ALLOC_REQ ; +if (rte_pci_read_config (pdev, &pri_reqs_dword,4 , off) !=4 ) + pri_reqs_dword =0 ; + } + +/* clear the PCI command register before issuing the FLR */ + + off =DLB_PCI_CMD ; + cmd =0 ; +if (rte_pci_write_config (pdev, &cmd,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + +/* issue the FLR */ +for (wait_count =0 ; wait_count <4 ; wait_count++) { +int sleep_time; + + off = pcie_cap_offset +DLB_PCI_EXP_DEVSTA ; + ret =rte_pci_read_config (pdev, &devsta_busy_word,2 , off); +if (ret !=2 ) { +printf ("[%s()] failed to read the pci device status\n" , + __func__); +return ret; + } + +if (!(devsta_busy_word &DLB_PCI_EXP_DEVSTA_TRPND )) +break ; + + sleep_time = (1 << (wait_count)) *100 ; +rte_delay_ms (sleep_time); + } + +if (wait_count ==4 ) { +printf ("[%s()] wait for pci pending transactions timed out\n" , + __func__); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_EXP_DEVCTL ; + ret =rte_pci_read_config (pdev, &devctl_word,2 , off); +if (ret !=2 ) { +printf ("[%s()] failed to read the pcie device control\n" , + __func__); +return ret; + } + + devctl_word |=DLB_PCI_EXP_DEVCTL_BCR_FLR ; + +if (rte_pci_write_config (pdev, &devctl_word,2 , off) !=2 ) { +printf ("[%s()] failed to write the pcie device control at offset %d\n" , + __func__, (int )off); +return -1 ; + } + +rte_delay_ms (100 ); + +/* Restore PCI config state */ + +if (pcie_cap_offset >=0 ) { + off = pcie_cap_offset +DLB_PCI_EXP_DEVCTL ; +if (rte_pci_write_config (pdev, &dev_ctl_word,2 , off) !=2 ) { +printf ("[%s()] failed to write the pcie device control at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_LNKCTL ; +if (rte_pci_write_config (pdev, &lnk_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_SLTCTL ; +if (rte_pci_write_config (pdev, &slt_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_RTCTL ; +if (rte_pci_write_config (pdev, &rt_ctl_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_EXP_DEVCTL2 ; +if (rte_pci_write_config (pdev, &dev_ctl2_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_LNKCTL2 ; +if (rte_pci_write_config (pdev, &lnk_word2,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pcie_cap_offset +DLB_PCI_SLTCTL2 ; +if (rte_pci_write_config (pdev, &slt_word2,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + + pasid_cap_offset =dlb_pci_find_ext_capability (pdev, +DLB_PCI_EXT_CAP_ID_PAS ); +if (pasid_cap_offset >=0 ) { + off = pasid_cap_offset +DLB_PCI_PASID_CAP ; +if (rte_pci_read_config (pdev, &pasid_features,2 , off) !=2 ) + pasid_features =0 ; + + pasid_features &=DLB_PCI_PASID_CAP_EXEC ; + pasid_features &=DLB_PCI_PASID_CAP_PRIV ; + pasid_ctrl_word =DLB_PCI_PASID_CTRL_ENABLE | pasid_features; + + off = pasid_cap_offset +DLB_PCI_PASID_CTRL ; +if (rte_pci_write_config (pdev, &pasid_ctrl_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + +if (pri_cap_offset >=0 ) { + pri_ctrl_word =DLB_PCI_PRI_CTRL_ENABLE ; + + off = pri_cap_offset +DLB_PCI_PRI_ALLOC_REQ ; +if (rte_pci_write_config (pdev, &pri_reqs_dword,4 , off) !=4 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = pri_cap_offset +DLB_PCI_PRI_CTRL ; +if (rte_pci_write_config (pdev, &pri_ctrl_word,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + + err_cap_offset =dlb_pci_find_ext_capability (pdev, +DLB_PCI_EXT_CAP_ID_ERR ); +if (err_cap_offset >=0 ) { +uint32_t tmp; + + off = err_cap_offset +DLB_PCI_ERR_ROOT_STATUS ; +if (rte_pci_read_config (pdev, &tmp,4 , off) !=4 ) + tmp =0 ; + +if (rte_pci_write_config (pdev, &tmp,4 , off) !=4 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = err_cap_offset +DLB_PCI_ERR_COR_STATUS ; +if (rte_pci_read_config (pdev, &tmp,4 , off) !=4 ) + tmp =0 ; + +if (rte_pci_write_config (pdev, &tmp,4 , off) !=4 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = err_cap_offset +DLB_PCI_ERR_UNCOR_STATUS ; +if (rte_pci_read_config (pdev, &tmp,4 , off) !=4 ) + tmp =0 ; + +if (rte_pci_write_config (pdev, &tmp,4 , off) !=4 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + +for (i =16 ; i >0 ; i--) { + off = (i -1 ) *4 ; +if (rte_pci_write_config (pdev, &dword[i -1 ],4 , off) !=4 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + + off =DLB_PCI_CMD ; +if (rte_pci_read_config (pdev, &cmd,2 , off) ==2 ) { + cmd &= ~DLB_PCI_COMMAND_INTX_DISABLE ; +if (rte_pci_write_config (pdev, &cmd,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space\n" , + __func__); +return -1 ; + } + } + + msix_cap_offset =dlb_pci_find_capability (pdev,DLB_PCI_CAP_ID_MSIX ); +if (msix_cap_offset >=0 ) { + off = msix_cap_offset +DLB_PCI_MSIX_FLAGS ; +if (rte_pci_read_config (pdev, &cmd,2 , off) ==2 ) { + cmd |=DLB_PCI_MSIX_FLAGS_ENABLE ; + cmd |=DLB_PCI_MSIX_FLAGS_MASKALL ; +if (rte_pci_write_config (pdev, &cmd,2 , off) !=2 ) { +printf ("[%s()] failed to write msix flags\n" , + __func__); +return -1 ; + } + } + + off = msix_cap_offset +DLB_PCI_MSIX_FLAGS ; +if (rte_pci_read_config (pdev, &cmd,2 , off) ==2 ) { + cmd &= ~DLB_PCI_MSIX_FLAGS_MASKALL ; +if (rte_pci_write_config (pdev, &cmd,2 , off) !=2 ) { +printf ("[%s()] failed to write msix flags\n" , + __func__); +return -1 ; + } + } + } + + acs_cap_offset =dlb_pci_find_ext_capability (pdev, +DLB_PCI_EXT_CAP_ID_ACS ); +if (acs_cap_offset >=0 ) { +uint16_t acs_cap, acs_ctrl, acs_mask; + off = acs_cap_offset +DLB_PCI_ACS_CAP ; +if (rte_pci_read_config (pdev, &acs_cap,2 , off) !=2 ) + acs_cap =0 ; + + off = acs_cap_offset +DLB_PCI_ACS_CTRL ; +if (rte_pci_read_config (pdev, &acs_ctrl,2 , off) !=2 ) + acs_ctrl =0 ; + + acs_mask =DLB_PCI_ACS_SV |DLB_PCI_ACS_RR ; + acs_mask |= (DLB_PCI_ACS_CR |DLB_PCI_ACS_UF ); + acs_ctrl |= (acs_cap & acs_mask); + +if (rte_pci_write_config (pdev, &acs_ctrl,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + + off = acs_cap_offset +DLB_PCI_ACS_CTRL ; +if (rte_pci_read_config (pdev, &acs_ctrl,2 , off) !=2 ) + acs_ctrl =0 ; + + acs_mask =DLB_PCI_ACS_RR |DLB_PCI_ACS_CR |DLB_PCI_ACS_EC ; + acs_ctrl &= ~acs_mask; + + off = acs_cap_offset +DLB_PCI_ACS_CTRL ; +if (rte_pci_write_config (pdev, &acs_ctrl,2 , off) !=2 ) { +printf ("[%s()] failed to write pci config space at offset %d\n" , + __func__, (int )off); +return -1 ; + } + } + +return 0 ; +} + +/*******************************/ +/****** Driver management ******/ +/*******************************/ + +int +dlb_pf_init_driver_state (struct dlb_dev *dlb_dev) +{ +int i; + +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_MOVDIR64B )) + dlb_dev->enqueue_four = dlb_movdir64b; +else + dlb_dev->enqueue_four = dlb_movntdq; + +/* Initialize software state */ +for (i =0 ; i <DLB_MAX_NUM_LDB_PORTS ; i++) +dlb_list_init_head (&dlb_dev->ldb_port_pages[i].list); + +for (i =0 ; i <DLB_MAX_NUM_DIR_PORTS ; i++) +dlb_list_init_head (&dlb_dev->dir_port_pages[i].list); + +rte_spinlock_init (&dlb_dev->resource_mutex); +rte_spinlock_init (&dlb_dev->measurement_lock); + +return 0 ; +} + +void +dlb_pf_init_hardware (struct dlb_dev *dlb_dev) +{ +dlb_disable_dp_vasr_feature (&dlb_dev->hw); + +dlb_enable_excess_tokens_alarm (&dlb_dev->hw); + +if (dlb_dev->revision >=DLB_REV_B0 ) { +dlb_hw_enable_sparse_ldb_cq_mode (&dlb_dev->hw); +dlb_hw_enable_sparse_dir_cq_mode (&dlb_dev->hw); + } +}
drivers/event/dlb/pf/dlb_main.h new file +90 -0
90 lines not on this page · raw message
drivers/event/dlb/pf/dlb_pf.c new file +839 -0
839 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH 11/27 event/dlb: add flexible PMD to device interfaces
In reply to McDaniel, TimothyThe PMD uses function pointers so that it can be executed as a bifurcated PMD accessing hardware via a linux kernel mode driver, or as a PF PMD where it has complete control of the hardware device.
Note that this interface is not used in the data path.
--- drivers/event/dlb/dlb_iface.c | 105 ++++++++++++++++++++++++++++++++++++++++++ drivers/event/dlb/dlb_iface.h | 92 ++++++++++++++++++++++++++++++++++++ 2 files changed, 197 insertions(+) create mode 100644 drivers/event/dlb/dlb_iface.c create mode 100644 drivers/event/dlb/dlb_iface.h
drivers/event/dlb/dlb_iface.c new file +105 -0
@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include <stdbool.h> +#include <stdint.h> +#include <rte_debug.h> +#include <rte_bus_pci.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_mbuf.h> +#include <rte_ring.h> +#include <rte_errno.h> +#include <rte_kvargs.h> +#include <rte_malloc.h> +#include <rte_cycles.h> +#include <rte_io.h> +#include <rte_eventdev.h> +#include <rte_eventdev_pmd.h> + +#include "dlb_priv.h" + +/* DLB PMD Internal interface function pointers. +* If VDEV (bifurcated PMD), these will resolve to functions that issue ioctls +* serviced by DLB kernel module. +* If PCI (PF PMD), these will be implemented locally in user mode. +*/ + +void (*dlb_iface_low_level_io_init)(struct dlb_eventdev *dlb); + +int (*dlb_iface_open)(struct dlb_hw_dev *handle,const char *name); + +void (*dlb_iface_domain_close)(struct dlb_eventdev *dlb); + +int (*dlb_iface_get_driver_version)(struct dlb_hw_dev *handle, +struct dlb_cmd_response *response); + +int (*dlb_iface_get_device_version)(struct dlb_hw_dev *handle, +uint8_t *revision); + +int (*dlb_iface_get_num_resources)(struct dlb_hw_dev *handle, +struct dlb_get_num_resources_args *rsrcs); + +int (*dlb_iface_sched_domain_create)(struct dlb_hw_dev *handle, +struct dlb_create_sched_domain_args *args); + +int (*dlb_iface_ldb_credit_pool_create)(struct dlb_hw_dev *handle, +struct dlb_create_ldb_pool_args *cfg); + +int (*dlb_iface_dir_credit_pool_create)(struct dlb_hw_dev *handle, +struct dlb_create_dir_pool_args *cfg); + +int (*dlb_iface_ldb_queue_create)(struct dlb_hw_dev *handle, +struct dlb_create_ldb_queue_args *cfg); + +int (*dlb_iface_dir_queue_create)(struct dlb_hw_dev *handle, +struct dlb_create_dir_queue_args *cfg); + +int (*dlb_iface_ldb_port_create)(struct dlb_hw_dev *handle, +struct dlb_create_ldb_port_args *cfg, +enum dlb_cq_poll_modes poll_mode); + +int (*dlb_iface_dir_port_create)(struct dlb_hw_dev *handle, +struct dlb_create_dir_port_args *cfg, +enum dlb_cq_poll_modes poll_mode); + +int (*dlb_iface_map_qid)(struct dlb_hw_dev *handle, +struct dlb_map_qid_args *cfg); + +int (*dlb_iface_unmap_qid)(struct dlb_hw_dev *handle, +struct dlb_unmap_qid_args *cfg); + +int (*dlb_iface_sched_domain_start)(struct dlb_hw_dev *handle, +struct dlb_start_domain_args *cfg); + +int (*dlb_iface_block_on_cq_interrupt)(struct dlb_hw_dev *handle, +int port_id,bool is_ldb, +volatile void *cq_va,uint8_t cq_gen, +bool arm); + +int (*dlb_iface_pending_port_unmaps)(struct dlb_hw_dev *handle, +struct dlb_pending_port_unmaps_args *args); + +int (*dlb_iface_get_ldb_queue_depth)(struct dlb_hw_dev *handle, +struct dlb_get_ldb_queue_depth_args *args); + +int (*dlb_iface_get_dir_queue_depth)(struct dlb_hw_dev *handle, +struct dlb_get_dir_queue_depth_args *args); + +int (*dlb_iface_enqueue_domain_alert)(struct dlb_hw_dev *handle, +uint64_t alert_data); + +int (*dlb_iface_get_cq_poll_mode)(struct dlb_hw_dev *handle, +enum dlb_cq_poll_modes *mode); + +int (*dlb_iface_get_sn_allocation)(struct dlb_hw_dev *handle, +struct dlb_get_sn_allocation_args *args); + +int (*dlb_iface_set_sn_allocation)(struct dlb_hw_dev *handle, +struct dlb_set_sn_allocation_args *args); + +int (*dlb_iface_get_sn_occupancy)(struct dlb_hw_dev *handle, +struct dlb_get_sn_occupancy_args *args); + +void (*dlb_iface_port_mmap)(struct dlb_port *qm_port);
drivers/event/dlb/dlb_iface.h new file +92 -0
92 lines not on this page · raw message
Signature · 1 line
2.13.6
PATCH 12/27 event/dlb: add the PMD's public interfaces
In reply to McDaniel, Timothy--- drivers/event/dlb/rte_pmd_dlb.c | 39 ++++++++++++++ drivers/event/dlb/rte_pmd_dlb.h | 69 +++++++++++++++++++++++++ drivers/event/dlb/rte_pmd_dlb_event_version.map | 6 +++ 3 files changed, 114 insertions(+) create mode 100644 drivers/event/dlb/rte_pmd_dlb.c create mode 100644 drivers/event/dlb/rte_pmd_dlb.h create mode 100644 drivers/event/dlb/rte_pmd_dlb_event_version.map
drivers/event/dlb/rte_pmd_dlb.c new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2020 Intel Corporation +*/ + +#include "rte_eventdev.h" +#include "rte_eventdev_pmd.h" + +#include "rte_pmd_dlb.h" +#include "dlb_priv.h" +#include "dlb_inline_fns.h" + +int +rte_pmd_dlb_set_token_pop_mode (uint8_t dev_id, +uint8_t port_id, +enum dlb_token_pop_mode mode) +{ +struct dlb_eventdev *dlb; +struct rte_eventdev *dev; + +RTE_EVENTDEV_VALID_DEVID_OR_ERR_RET (dev_id, -EINVAL ); + dev = &rte_eventdevs[dev_id]; + + dlb =dlb_pmd_priv (dev); + +if (mode >=NUM_TOKEN_POP_MODES ) +return -EINVAL ; + +/* The event device must be configured, but not yet started */ +if (!dlb->configured || dlb->run_state !=DLB_RUN_STATE_STOPPED ) +return -EINVAL ; + +/* The token pop mode must be set before configuring the port */ +if (port_id >= dlb->num_ports || dlb->ev_ports[port_id].setup_done) +return -EINVAL ; + + dlb->ev_ports[port_id].qm_port.token_pop_mode = mode; + +return 0 ; +}
drivers/event/dlb/rte_pmd_dlb.h new file +69 -0
69 lines not on this page · raw message
drivers/event/dlb/rte_pmd_dlb_event_version.map new file +6 -0
@@ -0,0 +1,6 @@ +DPDK_20.08 { + global: + + rte_pmd_dlb_set_token_pop_mode; + local: *; +};
Signature · 1 line
2.13.6
PATCH 13/27 event/dlb: add xstats support
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_xstats.c | 1251 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1251 insertions(+) create mode 100644 drivers/event/dlb/dlb_xstats.c
drivers/event/dlb/dlb_xstats.c new file +1251 -0
@@ -0,0 +1,1251 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include <stdint.h> +#include <stdbool.h> +#include <stdio.h> +#include <sys/mman.h> +#include <sys/fcntl.h> +#include <errno.h> +#include <assert.h> +#include <unistd.h> +#include <string.h> + +#include <rte_debug.h> +#include <rte_bus_vdev.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_mbuf.h> +#include <rte_ring.h> +#include <rte_errno.h> +#include <rte_kvargs.h> +#include <rte_malloc.h> +#include <rte_cycles.h> +#include <rte_io.h> +#include <rte_eventdev.h> +#include <rte_eventdev_pmd.h> +#include <rte_eventdev_pmd_vdev.h> + +#include "dlb_priv.h" +#include "dlb_inline_fns.h" + +enum dlb_xstats_type { +/* common to device and port */ + rx_ok,/**< Receive an event */ + rx_drop,/**< Error bit set in received QE */ + rx_interrupt_wait,/**< Wait on an interrupt */ + rx_umonitor_umwait,/**< Block using umwait */ + tx_ok,/**< Transmit an event */ + total_polls,/**< Call dequeue_burst */ + zero_polls,/**< Call dequeue burst and return 0 */ + tx_nospc_ldb_hw_credits,/**< Insufficient LDB h/w credits */ + tx_nospc_dir_hw_credits,/**< Insufficient DIR h/w credits */ + tx_nospc_inflight_max,/**< Reach the new_event_threshold */ + tx_nospc_new_event_limit,/**< Insufficient s/w credits */ + tx_nospc_inflight_credits,/**< Port has too few s/w credits */ +/* device specific */ + nb_events_limit, + inflight_events, + ldb_pool_size, + dir_pool_size, +/* port specific */ + tx_new,/**< Send an OP_NEW event */ + tx_fwd,/**< Send an OP_FORWARD event */ + tx_rel,/**< Send an OP_RELEASE event */ + tx_implicit_rel,/**< Issue an implicit event release */ + tx_sched_ordered,/**< Send a SCHED_TYPE_ORDERED event */ + tx_sched_unordered,/**< Send a SCHED_TYPE_PARALLEL event */ + tx_sched_atomic,/**< Send a SCHED_TYPE_ATOMIC event */ + tx_sched_directed,/**< Send a directed event */ + tx_invalid,/**< Send an event with an invalid op */ + outstanding_releases,/**< # of releases a port owes */ + max_outstanding_releases,/**< max # of releases a port can owe */ + rx_sched_ordered,/**< Dequeue an ordered event */ + rx_sched_unordered,/**< Dequeue an unordered event */ + rx_sched_atomic,/**< Dequeue an atomic event */ + rx_sched_directed,/**< Dequeue an directed event */ + rx_sched_invalid,/**< Dequeue event sched type invalid */ +/* common to port and queue */ + is_configured,/**< Port is configured */ + is_load_balanced,/**< Port is LDB */ + hw_id,/**< Hardware ID */ +/* queue specific */ + num_links,/**< Number of ports linked */ + sched_type,/**< Queue sched type */ + enq_ok,/**< # events enqueued to the queue */ + current_depth/**< Current queue depth */ +}; + +typedef uint64_t (*dlb_xstats_fn )(struct dlb_eventdev *dlb, +uint16_t obj_idx,/* port or queue id */ +enum dlb_xstats_type stat,int extra_arg); + +enum dlb_xstats_fn_type { +DLB_XSTATS_FN_DEV , +DLB_XSTATS_FN_PORT , +DLB_XSTATS_FN_QUEUE +}; + +struct dlb_xstats_entry { +struct rte_event_dev_xstats_name name; +uint64_t reset_value;/* an offset to be taken away to emulate resets */ +enum dlb_xstats_fn_type fn_id; +enum dlb_xstats_type stat; +enum rte_event_dev_xstats_mode mode; +int extra_arg; +uint16_t obj_idx; +uint8_t reset_allowed;/* when set, this value can be reset */ +}; + +/* Some device stats are simply a summation of the corresponding port values */ +static uint64_t +dlb_device_traffic_stat_get (struct dlb_eventdev *dlb,int which_stat) +{ +int i; +uint64_t val =0 ; + +for (i =0 ; i <DLB_MAX_NUM_PORTS ; i++) { +struct dlb_eventdev_port *port = &dlb->ev_ports[i]; + +if (!port->setup_done) +continue ; + +switch (which_stat) { +case rx_ok: + val += port->stats.traffic.rx_ok; +break ; +case rx_drop: + val += port->stats.traffic.rx_drop; +break ; +case rx_interrupt_wait: + val += port->stats.traffic.rx_interrupt_wait; +break ; +case rx_umonitor_umwait: + val += port->stats.traffic.rx_umonitor_umwait; +break ; +case tx_ok: + val += port->stats.traffic.tx_ok; +break ; +case total_polls: + val += port->stats.traffic.total_polls; +break ; +case zero_polls: + val += port->stats.traffic.zero_polls; +break ; +case tx_nospc_ldb_hw_credits: + val += port->stats.traffic.tx_nospc_ldb_hw_credits; +break ; +case tx_nospc_dir_hw_credits: + val += port->stats.traffic.tx_nospc_dir_hw_credits; +break ; +case tx_nospc_inflight_max: + val += port->stats.traffic.tx_nospc_inflight_max; +break ; +case tx_nospc_new_event_limit: + val += port->stats.traffic.tx_nospc_new_event_limit; +break ; +case tx_nospc_inflight_credits: + val += port->stats.traffic.tx_nospc_inflight_credits; +break ; +default : +return -1 ; + } + } +return val; +} + +static uint64_t +get_dev_stat(struct dlb_eventdev *dlb, uint16_t obj_idx __rte_unused, + enum dlb_xstats_type type, int extra_arg __rte_unused) +{ +switch (type) { +case rx_ok: +case rx_drop: +case rx_interrupt_wait: +case rx_umonitor_umwait: +case tx_ok: +case total_polls: +case zero_polls: +case tx_nospc_ldb_hw_credits: +case tx_nospc_dir_hw_credits: +case tx_nospc_inflight_max: +case tx_nospc_new_event_limit: +case tx_nospc_inflight_credits: +return dlb_device_traffic_stat_get (dlb, type); +case nb_events_limit: +return dlb->new_event_limit; +case inflight_events: +return rte_atomic32_read (&dlb->inflights); +case ldb_pool_size: +return dlb->num_ldb_credits; +case dir_pool_size: +return dlb->num_dir_credits; +default :return -1 ; + } +} + +static uint64_t +get_port_stat (struct dlb_eventdev *dlb,uint16_t obj_idx, + enum dlb_xstats_type type, int extra_arg __rte_unused) +{ +struct dlb_eventdev_port *ev_port = &dlb->ev_ports[obj_idx]; + +switch (type) { +case rx_ok:return ev_port->stats.traffic.rx_ok; + +case rx_drop:return ev_port->stats.traffic.rx_drop; + +case rx_interrupt_wait:return ev_port->stats.traffic.rx_interrupt_wait; + +case rx_umonitor_umwait: +return ev_port->stats.traffic.rx_umonitor_umwait; + +case tx_ok:return ev_port->stats.traffic.tx_ok; + +case total_polls:return ev_port->stats.traffic.total_polls; + +case zero_polls:return ev_port->stats.traffic.zero_polls; + +case tx_nospc_ldb_hw_credits: +return ev_port->stats.traffic.tx_nospc_ldb_hw_credits; + +case tx_nospc_dir_hw_credits: +return ev_port->stats.traffic.tx_nospc_dir_hw_credits; + +case tx_nospc_inflight_max: +return ev_port->stats.traffic.tx_nospc_inflight_max; + +case tx_nospc_new_event_limit: +return ev_port->stats.traffic.tx_nospc_new_event_limit; + +case tx_nospc_inflight_credits: +return ev_port->stats.traffic.tx_nospc_inflight_credits; + +case is_configured:return ev_port->setup_done; + +case is_load_balanced:return !ev_port->qm_port.is_directed; + +case hw_id:return ev_port->qm_port.id; + +case tx_new:return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_NEW ]; + +case tx_fwd:return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD ]; + +case tx_rel:return ev_port->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE ]; + +case tx_implicit_rel:return ev_port->stats.tx_implicit_rel; + +case tx_sched_ordered: +return ev_port->stats.tx_sched_cnt[DLB_SCHED_ORDERED ]; + +case tx_sched_unordered: +return ev_port->stats.tx_sched_cnt[DLB_SCHED_UNORDERED ]; + +case tx_sched_atomic: +return ev_port->stats.tx_sched_cnt[DLB_SCHED_ATOMIC ]; + +case tx_sched_directed: +return ev_port->stats.tx_sched_cnt[DLB_SCHED_DIRECTED ]; + +case tx_invalid:return ev_port->stats.tx_invalid; + +case outstanding_releases:return ev_port->outstanding_releases; + +case max_outstanding_releases: +return DLB_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT ; + +case rx_sched_ordered: +return ev_port->stats.rx_sched_cnt[DLB_SCHED_ORDERED ]; + +case rx_sched_unordered: +return ev_port->stats.rx_sched_cnt[DLB_SCHED_UNORDERED ]; + +case rx_sched_atomic: +return ev_port->stats.rx_sched_cnt[DLB_SCHED_ATOMIC ]; + +case rx_sched_directed: +return ev_port->stats.rx_sched_cnt[DLB_SCHED_DIRECTED ]; + +case rx_sched_invalid:return ev_port->stats.rx_sched_invalid; + +default :return -1 ; + } +} + +static uint64_t +get_queue_stat (struct dlb_eventdev *dlb,uint16_t obj_idx, + enum dlb_xstats_type type, int extra_arg __rte_unused) +{ +struct dlb_eventdev_queue *ev_queue = + &dlb->ev_queues[obj_idx]; + +switch (type) { +case is_configured:return ev_queue->setup_done; + +case is_load_balanced:return !ev_queue->qm_queue.is_directed; + +case hw_id:return ev_queue->qm_queue.id; + +case num_links:return ev_queue->num_links; + +case sched_type:return ev_queue->qm_queue.sched_type; + +case enq_ok: + { +int port_count =0 ; +uint64_t enq_ok_tally =0 ; + + ev_queue->enq_ok =0 ; +for (port_count =0 ; port_count <DLB_MAX_NUM_PORTS ; + port_count++) { +struct dlb_eventdev_port *ev_port = + &dlb->ev_ports[port_count]; + enq_ok_tally += ev_port->stats.enq_ok[ev_queue->id]; + } + ev_queue->enq_ok = enq_ok_tally; +return ev_queue->enq_ok; + } + +case current_depth:return dlb_get_queue_depth (dlb, ev_queue); + +default :return -1 ; + } +} + +int +dlb_xstats_init (struct dlb_eventdev *dlb) +{ +/* +* define the stats names and types. Used to build up the device +* xstats array +* There are multiple set of stats: +* - device-level, +* - per-port, +* - per-qid, +* +* For each of these sets, we have three parallel arrays, one for the +* names, the other for the stat type parameter to be passed in the fn +* call to get that stat. The third array allows resetting or not. +* All these arrays must be kept in sync +*/ +static const char *const dev_stats[] = { +"rx_ok" , +"rx_drop" , +"rx_interrupt_wait" , +"rx_umonitor_umwait" , +"tx_ok" , +"total_polls" , +"zero_polls" , +"tx_nospc_ldb_hw_credits" , +"tx_nospc_dir_hw_credits" , +"tx_nospc_inflight_max" , +"tx_nospc_new_event_limit" , +"tx_nospc_inflight_credits" , +"nb_events_limit" , +"inflight_events" , +"ldb_pool_size" , +"dir_pool_size" , + }; +static const enum dlb_xstats_type dev_types[] = { + rx_ok, + rx_drop, + rx_interrupt_wait, + rx_umonitor_umwait, + tx_ok, + total_polls, + zero_polls, + tx_nospc_ldb_hw_credits, + tx_nospc_dir_hw_credits, + tx_nospc_inflight_max, + tx_nospc_new_event_limit, + tx_nospc_inflight_credits, + nb_events_limit, + inflight_events, + ldb_pool_size, + dir_pool_size, + }; +/* Note: generated device stats are not allowed to be reset. */ +static const uint8_t dev_reset_allowed[] = { +0 ,/* rx_ok */ +0 ,/* rx_drop */ +0 ,/* rx_interrupt_wait */ +0 ,/* rx_umonitor_umwait */ +0 ,/* tx_ok */ +0 ,/* total_polls */ +0 ,/* zero_polls */ +0 ,/* tx_nospc_ldb_hw_credits */ +0 ,/* tx_nospc_dir_hw_credits */ +0 ,/* tx_nospc_inflight_max */ +0 ,/* tx_nospc_new_event_limit */ +0 ,/* tx_nospc_inflight_credits */ +0 ,/* nb_events_limit */ +0 ,/* inflight_events */ +0 ,/* ldb_pool_size */ +0 ,/* dir_pool_size */ + }; +static const char *const port_stats[] = { +"is_configured" , +"is_load_balanced" , +"hw_id" , +"rx_ok" , +"rx_drop" , +"rx_interrupt_wait" , +"rx_umonitor_umwait" , +"tx_ok" , +"total_polls" , +"zero_polls" , +"tx_nospc_ldb_hw_credits" , +"tx_nospc_dir_hw_credits" , +"tx_nospc_inflight_max" , +"tx_nospc_new_event_limit" , +"tx_nospc_inflight_credits" , +"tx_new" , +"tx_fwd" , +"tx_rel" , +"tx_implicit_rel" , +"tx_sched_ordered" , +"tx_sched_unordered" , +"tx_sched_atomic" , +"tx_sched_directed" , +"tx_invalid" , +"outstanding_releases" , +"max_outstanding_releases" , +"rx_sched_ordered" , +"rx_sched_unordered" , +"rx_sched_atomic" , +"rx_sched_directed" , +"rx_sched_invalid" + }; +static const enum dlb_xstats_type port_types[] = { + is_configured, + is_load_balanced, + hw_id, + rx_ok, + rx_drop, + rx_interrupt_wait, + rx_umonitor_umwait, + tx_ok, + total_polls, + zero_polls, + tx_nospc_ldb_hw_credits, + tx_nospc_dir_hw_credits, + tx_nospc_inflight_max, + tx_nospc_new_event_limit, + tx_nospc_inflight_credits, + tx_new, + tx_fwd, + tx_rel, + tx_implicit_rel, + tx_sched_ordered, + tx_sched_unordered, + tx_sched_atomic, + tx_sched_directed, + tx_invalid, + outstanding_releases, + max_outstanding_releases, + rx_sched_ordered, + rx_sched_unordered, + rx_sched_atomic, + rx_sched_directed, + rx_sched_invalid + }; +static const uint8_t port_reset_allowed[] = { +0 ,/* is_configured */ +0 ,/* is_load_balanced */ +0 ,/* hw_id */ +1 ,/* rx_ok */ +1 ,/* rx_drop */ +1 ,/* rx_interrupt_wait */ +1 ,/* rx_umonitor_umwait */ +1 ,/* tx_ok */ +1 ,/* total_polls */ +1 ,/* zero_polls */ +1 ,/* tx_nospc_ldb_hw_credits */ +1 ,/* tx_nospc_dir_hw_credits */ +1 ,/* tx_nospc_inflight_max */ +1 ,/* tx_nospc_new_event_limit */ +1 ,/* tx_nospc_inflight_credits */ +1 ,/* tx_new */ +1 ,/* tx_fwd */ +1 ,/* tx_rel */ +1 ,/* tx_implicit_rel */ +1 ,/* tx_sched_ordered */ +1 ,/* tx_sched_unordered */ +1 ,/* tx_sched_atomic */ +1 ,/* tx_sched_directed */ +1 ,/* tx_invalid */ +0 ,/* outstanding_releases */ +0 ,/* max_outstanding_releases */ +1 ,/* rx_sched_ordered */ +1 ,/* rx_sched_unordered */ +1 ,/* rx_sched_atomic */ +1 ,/* rx_sched_directed */ +1 /* rx_sched_invalid */ + }; + +/* QID specific stats */ +static const char *const qid_stats[] = { +"is_configured" , +"is_load_balanced" , +"hw_id" , +"num_links" , +"sched_type" , +"enq_ok" , +"current_depth" , + }; +static const enum dlb_xstats_type qid_types[] = { + is_configured, + is_load_balanced, + hw_id, + num_links, + sched_type, + enq_ok, + current_depth, + }; +static const uint8_t qid_reset_allowed[] = { +0 ,/* is_configured */ +0 ,/* is_load_balanced */ +0 ,/* hw_id */ +0 ,/* num_links */ +0 ,/* sched_type */ +1 ,/* enq_ok */ +0 ,/* current_depth */ + }; + +/* ---- end of stat definitions ---- */ + +/* check sizes, since a missed comma can lead to strings being +* joined by the compiler. +*/ +RTE_BUILD_BUG_ON (RTE_DIM (dev_stats) !=RTE_DIM (dev_types)); +RTE_BUILD_BUG_ON (RTE_DIM (port_stats) !=RTE_DIM (port_types)); +RTE_BUILD_BUG_ON (RTE_DIM (qid_stats) !=RTE_DIM (qid_types)); + +RTE_BUILD_BUG_ON (RTE_DIM (dev_stats) !=RTE_DIM (dev_reset_allowed)); +RTE_BUILD_BUG_ON (RTE_DIM (port_stats) !=RTE_DIM (port_reset_allowed)); +RTE_BUILD_BUG_ON (RTE_DIM (qid_stats) !=RTE_DIM (qid_reset_allowed)); + +/* other vars */ +const unsigned int count =RTE_DIM (dev_stats) + +DLB_MAX_NUM_PORTS *RTE_DIM (port_stats) + +DLB_MAX_NUM_QUEUES *RTE_DIM (qid_stats); +unsigned int i, port, qid, stat_id =0 ; + + dlb->xstats =rte_zmalloc_socket (NULL , +sizeof (dlb->xstats[0 ]) * count,0 , + dlb->qm_instance.info.socket_id); +if (dlb->xstats ==NULL ) +return -ENOMEM ; + +#define sname dlb->xstats[stat_id].name.name +for (i =0 ; i <RTE_DIM (dev_stats); i++, stat_id++) { + dlb->xstats[stat_id] = (struct dlb_xstats_entry ) { + .fn_id =DLB_XSTATS_FN_DEV , + .stat = dev_types[i], + .mode =RTE_EVENT_DEV_XSTATS_DEVICE , + .reset_allowed = dev_reset_allowed[i], + }; +snprintf (sname,sizeof (sname),"dev_%s" , dev_stats[i]); + } + dlb->xstats_count_mode_dev = stat_id; + +for (port =0 ; port <DLB_MAX_NUM_PORTS ; port++) { + dlb->xstats_offset_for_port[port] = stat_id; + +uint32_t count_offset = stat_id; + +for (i =0 ; i <RTE_DIM (port_stats); i++, stat_id++) { + dlb->xstats[stat_id] = (struct dlb_xstats_entry ){ + .fn_id =DLB_XSTATS_FN_PORT , + .obj_idx = port, + .stat = port_types[i], + .mode =RTE_EVENT_DEV_XSTATS_PORT , + .reset_allowed = port_reset_allowed[i], + }; +snprintf (sname,sizeof (sname),"port_%u_%s" , + port, port_stats[i]); + } + + dlb->xstats_count_per_port[port] = stat_id - count_offset; + } + + dlb->xstats_count_mode_port = stat_id - dlb->xstats_count_mode_dev; + +for (qid =0 ; qid <DLB_MAX_NUM_QUEUES ; qid++) { +uint32_t count_offset = stat_id; + + dlb->xstats_offset_for_qid[qid] = stat_id; + +for (i =0 ; i <RTE_DIM (qid_stats); i++, stat_id++) { + dlb->xstats[stat_id] = (struct dlb_xstats_entry ){ + .fn_id =DLB_XSTATS_FN_QUEUE , + .obj_idx = qid, + .stat = qid_types[i], + .mode =RTE_EVENT_DEV_XSTATS_QUEUE , + .reset_allowed = qid_reset_allowed[i], + }; +snprintf (sname,sizeof (sname),"qid_%u_%s" , + qid, qid_stats[i]); + } + + dlb->xstats_count_per_qid[qid] = stat_id - count_offset; + } + + dlb->xstats_count_mode_queue = stat_id - + (dlb->xstats_count_mode_dev + dlb->xstats_count_mode_port); +#undef sname + + dlb->xstats_count = stat_id; + +return 0 ; +} + +void +dlb_xstats_uninit (struct dlb_eventdev *dlb) +{ +rte_free (dlb->xstats); + dlb->xstats_count =0 ; +} + +int +dlb_eventdev_xstats_get_names (const struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode,uint8_t queue_port_id, +struct rte_event_dev_xstats_name *xstats_names, +unsigned int *ids,unsigned int size) +{ +const struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +unsigned int i; +unsigned int xidx =0 ; + +RTE_SET_USED (mode); +RTE_SET_USED (queue_port_id); + +uint32_t xstats_mode_count =0 ; +uint32_t start_offset =0 ; + +switch (mode) { +case RTE_EVENT_DEV_XSTATS_DEVICE : + xstats_mode_count = dlb->xstats_count_mode_dev; +break ; +case RTE_EVENT_DEV_XSTATS_PORT : +if (queue_port_id >=DLB_MAX_NUM_PORTS ) +break ; + xstats_mode_count = dlb->xstats_count_per_port[queue_port_id]; + start_offset = dlb->xstats_offset_for_port[queue_port_id]; +break ; +case RTE_EVENT_DEV_XSTATS_QUEUE : +#if (DLB_MAX_NUM_QUEUES <=255 )/* max 8 bit value */ +if (queue_port_id >=DLB_MAX_NUM_QUEUES ) +break ; +#endif + xstats_mode_count = dlb->xstats_count_per_qid[queue_port_id]; + start_offset = dlb->xstats_offset_for_qid[queue_port_id]; +break ; +default : +return -EINVAL ; + }; + +if (xstats_mode_count > size || !ids || !xstats_names) +return xstats_mode_count; + +for (i =0 ; i < dlb->xstats_count && xidx < size; i++) { +if (dlb->xstats[i].mode != mode) +continue ; + +if (mode !=RTE_EVENT_DEV_XSTATS_DEVICE && + queue_port_id != dlb->xstats[i].obj_idx) +continue ; + + xstats_names[xidx] = dlb->xstats[i].name; +if (ids) + ids[xidx] = start_offset + xidx; + xidx++; + } +return xidx; +} + +static int +dlb_xstats_update (struct dlb_eventdev *dlb, +enum rte_event_dev_xstats_mode mode, +uint8_t queue_port_id,const unsigned int ids[], +uint64_t values[],unsigned int n,const uint32_t reset) +{ +unsigned int i; +unsigned int xidx =0 ; + +RTE_SET_USED (mode); +RTE_SET_USED (queue_port_id); + +uint32_t xstats_mode_count =0 ; + +switch (mode) { +case RTE_EVENT_DEV_XSTATS_DEVICE : + xstats_mode_count = dlb->xstats_count_mode_dev; +break ; +case RTE_EVENT_DEV_XSTATS_PORT : +if (queue_port_id >=DLB_MAX_NUM_PORTS ) + goto invalid_value; + xstats_mode_count = dlb->xstats_count_per_port[queue_port_id]; +break ; +case RTE_EVENT_DEV_XSTATS_QUEUE : +#if (DLB_MAX_NUM_QUEUES <=255 )/* max 8 bit value */ +if (queue_port_id >=DLB_MAX_NUM_QUEUES ) + goto invalid_value; +#endif + xstats_mode_count = dlb->xstats_count_per_qid[queue_port_id]; +break ; +default : + goto invalid_value; + }; + +for (i =0 ; i < n && xidx < xstats_mode_count; i++) { +struct dlb_xstats_entry *xs = &dlb->xstats[ids[i]]; +dlb_xstats_fn fn; + +if (ids[i] > dlb->xstats_count || xs->mode != mode) +continue ; + +if (mode !=RTE_EVENT_DEV_XSTATS_DEVICE && + queue_port_id != xs->obj_idx) +continue ; + +switch (xs->fn_id) { +case DLB_XSTATS_FN_DEV : + fn = get_dev_stat; +break ; +case DLB_XSTATS_FN_PORT : + fn = get_port_stat; +break ; +case DLB_XSTATS_FN_QUEUE : + fn = get_queue_stat; +break ; +default : +rte_panic ("Unexpected xstat fn_id %d\n" , xs->fn_id); + } + +uint64_t val =fn (dlb, xs->obj_idx, xs->stat, + xs->extra_arg) - xs->reset_value; + +if (values) + values[xidx] = val; + +if (xs->reset_allowed && reset) + xs->reset_value += val; + + xidx++; + } + +return xidx; + +invalid_value: +return -EINVAL ; +} + +int +dlb_eventdev_xstats_get (const struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode,uint8_t queue_port_id, +const unsigned int ids[],uint64_t values[],unsigned int n) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +const uint32_t reset =0 ; + +return dlb_xstats_update (dlb, mode, queue_port_id, ids, values, n, + reset); +} + +uint64_t +dlb_eventdev_xstats_get_by_name (const struct rte_eventdev *dev, +const char *name,unsigned int *id) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +unsigned int i; +dlb_xstats_fn fn; + +for (i =0 ; i < dlb->xstats_count; i++) { +struct dlb_xstats_entry *xs = &dlb->xstats[i]; + +if (strncmp (xs->name.name, name, +RTE_EVENT_DEV_XSTATS_NAME_SIZE ) ==0 ){ +if (id !=NULL ) + *id = i; + +switch (xs->fn_id) { +case DLB_XSTATS_FN_DEV : + fn = get_dev_stat; +break ; +case DLB_XSTATS_FN_PORT : + fn = get_port_stat; +break ; +case DLB_XSTATS_FN_QUEUE : + fn = get_queue_stat; +break ; +default : +rte_panic ("Unexpected xstat fn_id %d\n" , + xs->fn_id); + } + +return fn (dlb, xs->obj_idx, xs->stat, + xs->extra_arg) - xs->reset_value; + } + } +if (id !=NULL ) + *id = (uint32_t )-1 ; +return (uint64_t )-1 ; +} + +static void +dlb_xstats_reset_range (struct dlb_eventdev *dlb,uint32_t start, +uint32_t num) +{ +uint32_t i; +dlb_xstats_fn fn; + +for (i = start; i < start + num; i++) { +struct dlb_xstats_entry *xs = &dlb->xstats[i]; + +if (!xs->reset_allowed) +continue ; + +switch (xs->fn_id) { +case DLB_XSTATS_FN_DEV : + fn = get_dev_stat; +break ; +case DLB_XSTATS_FN_PORT : + fn = get_port_stat; +break ; +case DLB_XSTATS_FN_QUEUE : + fn = get_queue_stat; +break ; +default : +rte_panic ("Unexpected xstat fn_id %d\n" , xs->fn_id); + } + +uint64_t val =fn (dlb, xs->obj_idx, xs->stat, xs->extra_arg); + xs->reset_value = val; + } +} + +static int +dlb_xstats_reset_queue (struct dlb_eventdev *dlb,uint8_t queue_id, +const uint32_t ids[],uint32_t nb_ids) +{ +const uint32_t reset =1 ; + +if (ids) { +uint32_t nb_reset =dlb_xstats_update (dlb, +RTE_EVENT_DEV_XSTATS_QUEUE , + queue_id, ids,NULL , nb_ids, + reset); +return nb_reset == nb_ids ?0 : -EINVAL ; + } + +if (ids ==NULL ) +dlb_xstats_reset_range (dlb, + dlb->xstats_offset_for_qid[queue_id], + dlb->xstats_count_per_qid[queue_id]); + +return 0 ; +} + +static int +dlb_xstats_reset_port (struct dlb_eventdev *dlb,uint8_t port_id, +const uint32_t ids[],uint32_t nb_ids) +{ +const uint32_t reset =1 ; +int offset = dlb->xstats_offset_for_port[port_id]; +int nb_stat = dlb->xstats_count_per_port[port_id]; + +if (ids) { +uint32_t nb_reset =dlb_xstats_update (dlb, +RTE_EVENT_DEV_XSTATS_PORT , port_id, + ids,NULL , nb_ids, + reset); +return nb_reset == nb_ids ?0 : -EINVAL ; + } + +dlb_xstats_reset_range (dlb, offset, nb_stat); +return 0 ; +} + +static int +dlb_xstats_reset_dev (struct dlb_eventdev *dlb,const uint32_t ids[], +uint32_t nb_ids) +{ +uint32_t i; + +if (ids) { +for (i =0 ; i < nb_ids; i++) { +uint32_t id = ids[i]; + +if (id >= dlb->xstats_count_mode_dev) +return -EINVAL ; +dlb_xstats_reset_range (dlb, id,1 ); + } + }else { +for (i =0 ; i < dlb->xstats_count_mode_dev; i++) +dlb_xstats_reset_range (dlb, i,1 ); + } + +return 0 ; +} + +int +dlb_eventdev_xstats_reset (struct rte_eventdev *dev, +enum rte_event_dev_xstats_mode mode, +int16_t queue_port_id, +const uint32_t ids[], +uint32_t nb_ids) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +uint32_t i; + +/* handle -1 for queue_port_id here, looping over all ports/queues */ +switch (mode) { +case RTE_EVENT_DEV_XSTATS_DEVICE : +if (dlb_xstats_reset_dev (dlb, ids, nb_ids)) +return -EINVAL ; +break ; +case RTE_EVENT_DEV_XSTATS_PORT : +if (queue_port_id ==-1 ) { +for (i =0 ; i <DLB_MAX_NUM_PORTS ; i++) { +if (dlb_xstats_reset_port (dlb, i, ids, + nb_ids)) +return -EINVAL ; + } + }else if (queue_port_id <DLB_MAX_NUM_PORTS ) { +if (dlb_xstats_reset_port (dlb, queue_port_id, ids, + nb_ids)) +return -EINVAL ; + }else { +return -EINVAL ; + } +break ; +case RTE_EVENT_DEV_XSTATS_QUEUE : +if (queue_port_id ==-1 ) { +for (i =0 ; i <DLB_MAX_NUM_QUEUES ; i++) { +if (dlb_xstats_reset_queue (dlb, i, ids, + nb_ids)) +return -EINVAL ; + } + }else if (queue_port_id <DLB_MAX_NUM_QUEUES ) { +if (dlb_xstats_reset_queue (dlb, queue_port_id, ids, + nb_ids)) +return -EINVAL ; + }else { +return -EINVAL ; + } +break ; + }; + +return 0 ; +} + +void +dlb_eventdev_dump (struct rte_eventdev *dev,FILE *f) +{ +struct dlb_eventdev *dlb; +struct dlb_hw_dev *handle; +int i; + +if (!f) { +printf ("Invalid file pointer\n" ); +return ; + } + +if (!dev) { +fprintf (f,"Invalid event device\n" ); +return ; + } + + dlb =dlb_pmd_priv (dev); + +if (!dlb) { +fprintf (f,"DLB Event device cannot be dumped!\n" ); +return ; + } + +if (!dlb->configured) +fprintf (f,"DLB Event device is not configured\n" ); + + handle = &dlb->qm_instance; + +fprintf (f,"================\n" ); +fprintf (f,"DLB Device Dump\n" ); +fprintf (f,"================\n" ); + +fprintf (f,"Processor supports umonitor/umwait instructions = %s\n" , + dlb->umwait_allowed ?"yes" :"no" ); + +/* Generic top level device information */ + +fprintf (f,"device is configured and run state =" ); +if (dlb->run_state ==DLB_RUN_STATE_STOPPED ) +fprintf (f,"STOPPED\n" ); +else if (dlb->run_state ==DLB_RUN_STATE_STOPPING ) +fprintf (f,"STOPPING\n" ); +else if (dlb->run_state ==DLB_RUN_STATE_STARTING ) +fprintf (f,"STARTING\n" ); +else if (dlb->run_state ==DLB_RUN_STATE_STARTED ) +fprintf (f,"STARTED\n" ); +else +fprintf (f,"UNEXPECTED\n" ); + +fprintf (f, +"dev ID=%d, dom ID=%u, name=%s, path=%s, sock=%u, evdev=%p\n" , + handle->device_id, handle->domain_id, handle->device_name, + handle->device_path, handle->info.socket_id, dlb->event_dev); + +fprintf (f,"domain device path=%s\n" , handle->domain_device_path); + +fprintf (f,"num dir ports=%u, num dir queues=%u\n" , + dlb->num_dir_ports, dlb->num_dir_queues); + +fprintf (f,"num ldb ports=%u, num ldb queues=%u\n" , + dlb->num_ldb_ports, dlb->num_ldb_queues); + +fprintf (f,"dir_credit_pool_id=%u, num_credits=%u\n" , + handle->cfg.dir_credit_pool_id, handle->cfg.num_dir_credits); + +fprintf (f,"ldb_credit_pool_id=%u, num_credits=%u\n" , + handle->cfg.ldb_credit_pool_id, handle->cfg.num_ldb_credits); + +fprintf (f,"num atomic inflights=%u, hist list entries=%u\n" , + handle->cfg.resources.num_atomic_inflights, + handle->cfg.resources.num_hist_list_entries); + +fprintf (f,"results from most recent hw resource query:\n" ); + +fprintf (f,"\tnum_sched_domains = %u\n" , + dlb->hw_rsrc_query_results.num_sched_domains); + +fprintf (f,"\tnum_ldb_queues = %u\n" , + dlb->hw_rsrc_query_results.num_ldb_queues); + +fprintf (f,"\tnum_ldb_ports = %u\n" , + dlb->hw_rsrc_query_results.num_ldb_ports); + +fprintf (f,"\tnum_dir_ports = %u\n" , + dlb->hw_rsrc_query_results.num_dir_ports); + +fprintf (f,"\tnum_atomic_inflights = %u\n" , + dlb->hw_rsrc_query_results.num_atomic_inflights); + +fprintf (f,"\tmax_contiguous_atomic_inflights = %u\n" , + dlb->hw_rsrc_query_results.max_contiguous_atomic_inflights); + +fprintf (f,"\tnum_hist_list_entries = %u\n" , + dlb->hw_rsrc_query_results.num_hist_list_entries); + +fprintf (f,"\tmax_contiguous_hist_list_entries = %u\n" , + dlb->hw_rsrc_query_results.max_contiguous_hist_list_entries); + +fprintf (f,"\tnum_ldb_credits = %u\n" , + dlb->hw_rsrc_query_results.num_ldb_credits); + +fprintf (f,"\tmax_contiguous_ldb_credits = %u\n" , + dlb->hw_rsrc_query_results.max_contiguous_ldb_credits); + +fprintf (f,"\tnum_dir_credits = %u\n" , + dlb->hw_rsrc_query_results.num_dir_credits); + +fprintf (f,"\tmax_contiguous_dir_credits = %u\n" , + dlb->hw_rsrc_query_results.max_contiguous_dir_credits); + +fprintf (f,"\tnum_ldb_credit_pools = %u\n" , + dlb->hw_rsrc_query_results.num_ldb_credit_pools); + +fprintf (f,"\tnum_dir_credit_pools = %u\n" , + dlb->hw_rsrc_query_results.num_dir_credit_pools); + +/* Port level information */ + +for (i =0 ; i < dlb->num_ports; i++) { +struct dlb_eventdev_port *p = &dlb->ev_ports[i]; +int j; + +if (!p->enq_configured) +fprintf (f,"Port_%d is not configured\n" , i); + +fprintf (f,"Port_%d\n" , i); +fprintf (f,"=======\n" ); + +fprintf (f,"\tevport_%u is configured, setup done=%d\n" , + p->id, p->setup_done); + +fprintf (f,"\tconfig state=%d, port state=%d\n" , + p->qm_port.config_state, p->qm_port.state); + +fprintf (f,"\tport is %s\n" , + p->qm_port.is_directed ?"directed" :"load balanced" ); + +fprintf (f,"\toutstanding releases=%u\n" , + p->outstanding_releases); + +fprintf (f,"\tinflight max=%u, inflight credits=%u\n" , + p->inflight_max, p->inflight_credits); + +fprintf (f,"\tcredit update quanta=%u, implicit release =%u\n" , + p->credit_update_quanta, p->implicit_release); + +fprintf (f,"\tnum_links=%d, queues -> " , p->num_links); + +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) { +if (p->link[j].valid) +fprintf (f,"id=%u prio=%u " , + p->link[j].queue_id, + p->link[j].priority); + } +fprintf (f,"\n" ); + +fprintf (f,"\thardware port id=%u\n" , p->qm_port.id); + +fprintf (f,"\tcached_ldb_credits=%u\n" , + p->qm_port.cached_ldb_credits); + +fprintf (f,"\tldb_pushcount_at_credit_expiry = %u\n" , + p->qm_port.ldb_pushcount_at_credit_expiry); + +fprintf (f,"\tldb_credits = %u\n" , + p->qm_port.ldb_credits); + +fprintf (f,"\tcached_dir_credits = %u\n" , + p->qm_port.cached_dir_credits); + +fprintf (f,"\tdir_pushcount_at_credit_expiry=%u\n" , + p->qm_port.dir_pushcount_at_credit_expiry); + +fprintf (f,"\tdir_credits = %u\n" , + p->qm_port.dir_credits); + +fprintf (f,"\tgenbit=%d, cq_idx=%d, cq_depth=%d\n" , + p->qm_port.gen_bit, + p->qm_port.cq_idx, + p->qm_port.cq_depth); + +fprintf (f,"\tuse reserved token scheme=%d, cq_rsvd_token_deficit=%u\n" , + p->qm_port.use_rsvd_token_scheme, + p->qm_port.cq_rsvd_token_deficit); + +fprintf (f,"\tinterrupt armed=%d\n" , + p->qm_port.int_armed); + +fprintf (f,"\tPort statistics\n" ); + +fprintf (f,"\t\trx_ok %" PRIu64"\n" , + p->stats.traffic.rx_ok); + +fprintf (f,"\t\trx_drop %" PRIu64"\n" , + p->stats.traffic.rx_drop); + +fprintf (f,"\t\trx_interrupt_wait %" PRIu64"\n" , + p->stats.traffic.rx_interrupt_wait); + +fprintf (f,"\t\trx_umonitor_umwait %" PRIu64"\n" , + p->stats.traffic.rx_umonitor_umwait); + +fprintf (f,"\t\ttx_ok %" PRIu64"\n" , + p->stats.traffic.tx_ok); + +fprintf (f,"\t\ttotal_polls %" PRIu64"\n" , + p->stats.traffic.total_polls); + +fprintf (f,"\t\tzero_polls %" PRIu64"\n" , + p->stats.traffic.zero_polls); + +fprintf (f,"\t\ttx_nospc_ldb_hw_credits %" PRIu64"\n" , + p->stats.traffic.tx_nospc_ldb_hw_credits); + +fprintf (f,"\t\ttx_nospc_dir_hw_credits %" PRIu64"\n" , + p->stats.traffic.tx_nospc_dir_hw_credits); + +fprintf (f,"\t\ttx_nospc_inflight_max %" PRIu64"\n" , + p->stats.traffic.tx_nospc_inflight_max); + +fprintf (f,"\t\ttx_nospc_new_event_limit %" PRIu64"\n" , + p->stats.traffic.tx_nospc_new_event_limit); + +fprintf (f,"\t\ttx_nospc_inflight_credits %" PRIu64"\n" , + p->stats.traffic.tx_nospc_inflight_credits); + +fprintf (f,"\t\ttx_new %" PRIu64"\n" , + p->stats.tx_op_cnt[RTE_EVENT_OP_NEW ]); + +fprintf (f,"\t\ttx_fwd %" PRIu64"\n" , + p->stats.tx_op_cnt[RTE_EVENT_OP_FORWARD ]); + +fprintf (f,"\t\ttx_rel %" PRIu64"\n" , + p->stats.tx_op_cnt[RTE_EVENT_OP_RELEASE ]); + +fprintf (f,"\t\ttx_implicit_rel %" PRIu64"\n" , + p->stats.tx_implicit_rel); + +fprintf (f,"\t\ttx_sched_ordered %" PRIu64"\n" , + p->stats.tx_sched_cnt[DLB_SCHED_ORDERED ]); + +fprintf (f,"\t\ttx_sched_unordered %" PRIu64"\n" , + p->stats.tx_sched_cnt[DLB_SCHED_UNORDERED ]); + +fprintf (f,"\t\ttx_sched_atomic %" PRIu64"\n" , + p->stats.tx_sched_cnt[DLB_SCHED_ATOMIC ]); + +fprintf (f,"\t\ttx_sched_directed %" PRIu64"\n" , + p->stats.tx_sched_cnt[DLB_SCHED_DIRECTED ]); + +fprintf (f,"\t\ttx_invalid %" PRIu64"\n" , + p->stats.tx_invalid); + +fprintf (f,"\t\trx_sched_ordered %" PRIu64"\n" , + p->stats.rx_sched_cnt[DLB_SCHED_ORDERED ]); + +fprintf (f,"\t\trx_sched_unordered %" PRIu64"\n" , + p->stats.rx_sched_cnt[DLB_SCHED_UNORDERED ]); + +fprintf (f,"\t\trx_sched_atomic %" PRIu64"\n" , + p->stats.rx_sched_cnt[DLB_SCHED_ATOMIC ]); + +fprintf (f,"\t\trx_sched_directed %" PRIu64"\n" , + p->stats.rx_sched_cnt[DLB_SCHED_DIRECTED ]); + +fprintf (f,"\t\trx_sched_invalid %" PRIu64"\n" , + p->stats.rx_sched_invalid); + } + +/* Queue level information */ + +for (i =0 ; i < dlb->num_queues; i++) { +struct dlb_eventdev_queue *q = &dlb->ev_queues[i]; +int j, k; + +if (!q->setup_done) +fprintf (f,"Queue_%d is not configured\n" , i); + +fprintf (f,"Queue_%d\n" , i); +fprintf (f,"========\n" ); + +fprintf (f,"\tevqueue_%u is set up\n" , q->id); + +fprintf (f,"\tqueue is %s\n" , + q->qm_queue.is_directed ?"directed" :"load balanced" ); + +fprintf (f,"\tnum_links=%d, ports -> " , q->num_links); + +for (j =0 ; j < dlb->num_ports; j++) { +struct dlb_eventdev_port *p = &dlb->ev_ports[j]; + +for (k =0 ; k <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; k++) { +if (p->link[k].valid && + p->link[k].queue_id == q->id) +fprintf (f,"id=%u prio=%u " , + p->id, p->link[k].priority); + } + } +fprintf (f,"\n" ); + +fprintf (f,"\tcurrent depth: %u events\n" , +dlb_get_queue_depth (dlb, q)); + +fprintf (f,"\tnum qid inflights=%u, sched_type=%d\n" , + q->qm_queue.num_qid_inflights, q->qm_queue.sched_type); + } +}
Signature · 1 line
2.13.6
PATCH 14/27 event/dlb: add PMD self-tests
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb_selftest.c | 1628 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 1628 insertions(+) create mode 100644 drivers/event/dlb/dlb_selftest.c
drivers/event/dlb/dlb_selftest.c new file +1628 -0
@@ -0,0 +1,1628 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include <stdio.h> +#include <string.h> +#include <stdint.h> +#include <errno.h> +#include <unistd.h> +#include <sys/queue.h> + +#include <rte_memory.h> +#include <rte_memzone.h> +#include <rte_launch.h> +#include <rte_eal.h> +#include <rte_per_lcore.h> +#include <rte_lcore.h> +#include <rte_debug.h> +#include <rte_ethdev.h> +#include <rte_cycles.h> +#include <rte_eventdev.h> +#include <rte_pause.h> +#include <rte_bus_vdev.h> + +#include "dlb_priv.h" +#include "rte_pmd_dlb.h" + +#define MAX_PORTS 32 +#define MAX_QIDS 32 +#define DEFAULT_NUM_SEQ_NUMS 32 + +static struct rte_mempool *eventdev_func_mempool; +static int evdev; + +struct test { +struct rte_mempool *mbuf_pool; +int nb_qids; +}; + +/* initialization and config */ +static inline int +init (struct test *t,int nb_queues,int nb_ports) +{ +struct rte_event_dev_config config = {0 }; +struct rte_event_dev_info info; +int ret; + +memset (t,0 ,sizeof (*t)); + + t->mbuf_pool = eventdev_func_mempool; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues = nb_queues; + config.nb_event_ports = nb_ports; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) +printf ("%d: Error configuring device\n" , __LINE__); + +return ret; +} + +static inline int +create_ports (int num_ports) +{ +int i; + +if (num_ports >MAX_PORTS ) +return -1 ; + +for (i =0 ; i < num_ports; i++) { +struct rte_event_port_conf conf; + +if (rte_event_port_default_conf_get (evdev, i, &conf)) { +printf ("%d: Error querying default port conf\n" , + __LINE__); +return -1 ; + } + +if (rte_event_port_setup (evdev, i, &conf) <0 ) { +printf ("%d: Error setting up port %d\n" , i, __LINE__); +return -1 ; + } + } + +return 0 ; +} + +static inline int +create_lb_qids (struct test *t,int num_qids,uint32_t flags) +{ +int i; + +for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) { +struct rte_event_queue_conf conf; + +if (rte_event_queue_default_conf_get (evdev, i, &conf)) { +printf ("%d: Error querying default queue conf\n" , + __LINE__); +return -1 ; + } + + conf.schedule_type = flags; + +if (conf.schedule_type ==RTE_SCHED_TYPE_PARALLEL ) + conf.nb_atomic_order_sequences =0 ; +else + conf.nb_atomic_order_sequences =DEFAULT_NUM_SEQ_NUMS ; + +if (rte_event_queue_setup (evdev, i, &conf) <0 ) { +printf ("%d: error creating qid %d\n" , __LINE__, i); +return -1 ; + } + } + + t->nb_qids += num_qids; +if (t->nb_qids >MAX_QIDS ) +return -1 ; + +return 0 ; +} + +static inline int +create_atomic_qids (struct test *t,int num_qids) +{ +return create_lb_qids (t, num_qids,RTE_SCHED_TYPE_ATOMIC ); +} + +static inline int +create_ordered_qids (struct test *t,int num_qids) +{ +return create_lb_qids (t, num_qids,RTE_SCHED_TYPE_ORDERED ); +} + +static inline int +create_unordered_qids (struct test *t,int num_qids) +{ +return create_lb_qids (t, num_qids,RTE_SCHED_TYPE_PARALLEL ); +} + +static inline int +create_directed_qids (struct test *t,int num_qids,const uint8_t ports[]) +{ +static struct rte_event_queue_conf conf; +int i; + + conf.priority =RTE_EVENT_DEV_PRIORITY_NORMAL ; + conf.event_queue_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +for (i = t->nb_qids; i < t->nb_qids + num_qids; i++) { +uint8_t queue_id; + +if (rte_event_queue_setup (evdev, i, &conf) <0 ) { +printf ("%d: error creating qid %d\n" , __LINE__, i); +return -1 ; + } + + queue_id = i; + +if (rte_event_port_link (evdev, ports[i - t->nb_qids], + &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: error creating link for qid %d\n" , + __LINE__, i); +return -1 ; + } + } + + t->nb_qids += num_qids; +if (t->nb_qids >MAX_QIDS ) +return -1 ; + +return 0 ; +} + +/* destruction */ +static inline int +cleanup(struct test *t __rte_unused) +{ +int ret; + +rte_event_dev_stop (evdev); + ret =rte_event_dev_close (evdev); +if (ret) +return -1 ; + +return 0 ; +}; + +static inline int +enqueue_timeout (uint8_t port_id,struct rte_event *ev,uint64_t tmo_us) +{ +const uint64_t start =rte_get_timer_cycles (); +const uint64_t ticks = (tmo_us *rte_get_timer_hz ()) /1E6 ; + +while ((rte_get_timer_cycles () - start) < ticks) { +if (rte_event_enqueue_burst (evdev, port_id, ev,1 ) ==1 ) +return 0 ; + +if (rte_errno != -ENOSPC ) +return -1 ; + } + +return -1 ; +} + +static void +flush(uint8_t id __rte_unused, struct rte_event event, void *arg __rte_unused) +{ +rte_pktmbuf_free (event.mbuf); +} + +static int +test_stop_flush (struct test *t)/* test to check we can properly flush events */ +{ +struct rte_event ev; +uint32_t dequeue_depth; +unsigned int i, count; +uint8_t queue_id; + + ev.op =RTE_EVENT_OP_NEW ; + +if (init (t,2 ,1 ) <0 || +create_ports (1 ) <0 || +create_atomic_qids (t,2 ) <0 ) { +printf ("%d: Error initializing device\n" , __LINE__); +return -1 ; + } + +if (rte_event_port_link (evdev,0 ,NULL ,NULL ,0 ) !=2 ) { +printf ("%d: Error linking queues to the port\n" , __LINE__); + goto err; + } + +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: Error with start call\n" , __LINE__); + goto err; + } + +/* Unlink queue 1 so the PMD's stop callback has to cleanup an unlinked +* queue. +*/ + queue_id =1 ; + +if (rte_event_port_unlink (evdev,0 , &queue_id,1 ) !=1 ) { +printf ("%d: Error unlinking queue 1 from port\n" , __LINE__); + goto err; + } + + count =rte_mempool_avail_count (t->mbuf_pool); + +if (rte_event_port_attr_get (evdev, +0 , +RTE_EVENT_PORT_ATTR_DEQ_DEPTH , + &dequeue_depth)) { +printf ("%d: Error retrieveing dequeue depth\n" , __LINE__); + goto err; + } + +/* Send QEs to queue 0 */ +for (i =0 ; i < dequeue_depth +1 ; i++) { + ev.mbuf =rte_pktmbuf_alloc (t->mbuf_pool); + ev.queue_id =0 ; + ev.sched_type =RTE_SCHED_TYPE_ATOMIC ; + +if (enqueue_timeout (0 , &ev,1000 )) { +printf ("%d: Error enqueuing events\n" , __LINE__); + goto err; + } + } + +/* Send QEs to queue 1 */ +for (i =0 ; i < dequeue_depth +1 ; i++) { + ev.mbuf =rte_pktmbuf_alloc (t->mbuf_pool); + ev.queue_id =1 ; + ev.sched_type =RTE_SCHED_TYPE_ATOMIC ; + +if (enqueue_timeout (0 , &ev,1000 )) { +printf ("%d: Error enqueuing events\n" , __LINE__); + goto err; + } + } + +/* Now the DLB is scheduling events from the port to the IQ, and at +* least one event should be remaining in each queue. +*/ + +if (rte_event_dev_stop_flush_callback_register (evdev, flush,NULL )) { +printf ("%d: Error installing the flush callback\n" , __LINE__); + goto err; + } + +cleanup (t); + +if (count !=rte_mempool_avail_count (t->mbuf_pool)) { +printf ("%d: Error executing the flush callback\n" , __LINE__); + goto err; + } + +if (rte_event_dev_stop_flush_callback_register (evdev,NULL ,NULL )) { +printf ("%d: Error uninstalling the flush callback\n" , __LINE__); + goto err; + } + +return 0 ; +err: +cleanup (t); +return -1 ; +} + +static int +test_single_link (void ) +{ +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +uint8_t queue_id; +int ret; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =2 ; + config.nb_event_ports =2 ; + config.nb_single_link_event_port_queues =1 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +/* Create a directed port */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + + port_conf.event_port_cfg =RTE_EVENT_PORT_CFG_SINGLE_LINK ; + +if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) { +printf ("%d: port 0 setup expected to succeed\n" , __LINE__); + goto err; + } + +/* Attempt to create another directed port */ +if (rte_event_port_setup (evdev,1 , &port_conf) ==0 ) { +printf ("%d: port 1 setup expected to fail\n" , __LINE__); + goto err; + } + + port_conf.event_port_cfg =0 ; + +/* Create a load-balanced port */ +if (rte_event_port_setup (evdev,1 , &port_conf) <0 ) { +printf ("%d: port 1 setup expected to succeed\n" , __LINE__); + goto err; + } + +/* Create a directed queue */ +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + + queue_conf.event_queue_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 0 setup expected to succeed\n" , __LINE__); + goto err; + } + +/* Attempt to create another directed queue */ +if (rte_event_queue_setup (evdev,1 , &queue_conf) ==0 ) { +printf ("%d: queue 1 setup expected to fail\n" , __LINE__); + goto err; + } + +/* Create a load-balanced queue */ + queue_conf.event_queue_cfg =0 ; + +if (rte_event_queue_setup (evdev,1 , &queue_conf) <0 ) { +printf ("%d: queue 1 setup expected to succeed\n" , __LINE__); + goto err; + } + +/* Attempt to link directed and load-balanced resources */ + queue_id =1 ; +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) ==1 ) { +printf ("%d: port 0 link expected to fail\n" , __LINE__); + goto err; + } + + queue_id =0 ; +if (rte_event_port_link (evdev,1 , &queue_id,NULL ,1 ) ==1 ) { +printf ("%d: port 1 link expected to fail\n" , __LINE__); + goto err; + } + +/* Link ports to queues */ + queue_id =0 ; +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 0 link expected to succeed\n" , __LINE__); + goto err; + } + + queue_id =1 ; +if (rte_event_port_link (evdev,1 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 1 link expected to succeed\n" , __LINE__); + goto err; + } + + ret =rte_event_dev_close (evdev); +if (ret) { +printf ("rte_event_dev_close err %d\n" , ret); + goto err; + } + +return 0 ; + +err: +rte_event_dev_close (evdev); +return -1 ; +} + +#define NUM_LDB_PORTS 64 +#define NUM_LDB_QUEUES 128 + +static int +test_info_get (void ) +{ +struct rte_event_dev_config config = {0 }; +struct rte_event_dev_info info; +int ret; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + +if (info.max_event_ports !=NUM_LDB_PORTS ) { +printf ("%d: Got %u ports, expected %u\n" , + __LINE__, info.max_event_ports,NUM_LDB_PORTS ); + goto err; + } + +if (info.max_event_queues !=NUM_LDB_QUEUES ) { +printf ("%d: Got %u queues, expected %u\n" , + __LINE__, info.max_event_queues,NUM_LDB_QUEUES ); + goto err; + } + + config.nb_event_ports = info.max_event_ports; + config.nb_event_queues =NUM_LDB_QUEUES + info.max_event_ports /2 ; + config.nb_single_link_event_port_queues = info.max_event_ports /2 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); + goto err; + } + +/* The DLB PMD only reports load-balanced ports and queues in its +* info_get function. Confirm that these values don't include the +* directed port or queue counts. +*/ + +if (info.max_event_ports !=NUM_LDB_PORTS ) { +printf ("%d: Got %u ports, expected %u\n" , + __LINE__, info.max_event_ports,NUM_LDB_PORTS ); + goto err; + } + +if (info.max_event_queues !=NUM_LDB_QUEUES ) { +printf ("%d: Got %u queues, expected %u\n" , + __LINE__, info.max_event_queues,NUM_LDB_QUEUES ); + goto err; + } + + ret =rte_event_dev_close (evdev); +if (ret) { +printf ("rte_event_dev_close err %d\n" , ret); + goto err; + } + +return 0 ; + +err: +rte_event_dev_close (evdev); +return -1 ; +} + +static int +test_reconfiguration_link (struct test *t) +{ +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +uint8_t queue_id; +int ret, i; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =2 ; + config.nb_event_ports =2 ; + config.nb_single_link_event_port_queues =0 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + +/* Configure the device with 2 LDB ports and 2 LDB queues */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure the ports and queues */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + +for (i =0 ; i <2 ; i++) { +if (rte_event_port_setup (evdev, i, &port_conf) <0 ) { +printf ("%d: port %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + +for (i =0 ; i <2 ; i++) { +if (rte_event_queue_setup (evdev, i, &queue_conf) <0 ) { +printf ("%d: queue %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Link P0->Q0 and P1->Q1 */ +for (i =0 ; i <2 ; i++) { + queue_id = i; + +if (rte_event_port_link (evdev, i, &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port %d link expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Stop the device */ +rte_event_dev_stop (evdev); + +/* Reconfigure device */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error re-configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure P1 and Q1, leave P0 and Q0 to be configured by the PMD. */ +if (rte_event_port_setup (evdev,1 , &port_conf) <0 ) { +printf ("%d: port 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_setup (evdev,1 , &queue_conf) <0 ) { +printf ("%d: queue 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q0 and Q1 */ +for (i =0 ; i <2 ; i++) { + queue_id = i; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P0->Q%d link expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Link P1->Q0 and Q1 */ +for (i =0 ; i <2 ; i++) { + queue_id = i; + +if (rte_event_port_link (evdev,1 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P1->Q%d link expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Stop the device */ +rte_event_dev_stop (evdev); + +/* Configure device with 2 DIR ports and 2 DIR queues */ + config.nb_single_link_event_port_queues =2 ; + + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure the ports and queues */ + port_conf.event_port_cfg =RTE_EVENT_PORT_CFG_SINGLE_LINK ; + +for (i =0 ; i <2 ; i++) { +if (rte_event_port_setup (evdev, i, &port_conf) <0 ) { +printf ("%d: port %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + } + + queue_conf.event_queue_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +for (i =0 ; i <2 ; i++) { +if (rte_event_queue_setup (evdev, i, &queue_conf) <0 ) { +printf ("%d: queue %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Link P0->Q0 and P1->Q1 */ +for (i =0 ; i <2 ; i++) { + queue_id = i; + +if (rte_event_port_link (evdev, i, &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port %d link expected to succeed\n" , + __LINE__, i); + goto err; + } + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Stop the device */ +rte_event_dev_stop (evdev); + +/* Reconfigure device */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error re-configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure P1 and Q0, leave P0 and Q1 to be configured by the PMD. */ +if (rte_event_port_setup (evdev,1 , &port_conf) <0 ) { +printf ("%d: port 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q1 */ + queue_id =1 ; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P0->Q%d link expected to succeed\n" , + __LINE__, i); + goto err; + } + +/* Link P1->Q0 */ + queue_id =0 ; + +if (rte_event_port_link (evdev,1 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P1->Q%d link expected to succeed\n" , + __LINE__, i); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +rte_event_dev_stop (evdev); + + config.nb_event_queues =5 ; + config.nb_event_ports =5 ; + config.nb_single_link_event_port_queues =1 ; + + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error re-configuring device\n" , __LINE__); +return -1 ; + } + +for (i =0 ; i < config.nb_event_queues -1 ; i++) { + port_conf.event_port_cfg =0 ; + queue_conf.event_queue_cfg =0 ; + +if (rte_event_port_setup (evdev, i, &port_conf) <0 ) { +printf ("%d: port %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + +if (rte_event_queue_setup (evdev, i, &queue_conf) <0 ) { +printf ("%d: queue %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + + queue_id = i; + +if (rte_event_port_link (evdev, i, &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P%d->Q%d link expected to succeed\n" , + __LINE__, i, i); + goto err; + } + } + + port_conf.event_port_cfg =RTE_EVENT_PORT_CFG_SINGLE_LINK ; + queue_conf.event_queue_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +if (rte_event_port_setup (evdev, i, &port_conf) <0 ) { +printf ("%d: port %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + +if (rte_event_queue_setup (evdev, i, &queue_conf) <0 ) { +printf ("%d: queue %d setup expected to succeed\n" , + __LINE__, i); + goto err; + } + + queue_id = i; + +if (rte_event_port_link (evdev, i, &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: P%d->Q%d link expected to succeed\n" , + __LINE__, i, i); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Stop the device */ +rte_event_dev_stop (evdev); + + config.nb_event_ports +=1 ; + +/* Reconfigure device with 1 more load-balanced port */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error re-configuring device\n" , __LINE__); +return -1 ; + } + + port_conf.event_port_cfg =0 ; + +/* Configure the new port */ +if (rte_event_port_setup (evdev, config.nb_event_ports -1 , + &port_conf) <0 ) { +printf ("%d: port 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +cleanup (t); +return 0 ; + +err: +cleanup (t); +return -1 ; +} + +static int +test_load_balanced_traffic (struct test *t) +{ +uint64_t timeout; +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +struct rte_event ev; +uint8_t queue_id; +int ret; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =1 ; + config.nb_event_ports =1 ; + config.nb_single_link_event_port_queues =0 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + +/* Configure the device with 1 LDB port and queue */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure the ports and queues */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + +if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) { +printf ("%d: port 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q0 */ + queue_id =0 ; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 0 link expected to succeed\n" , + __LINE__); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Enqueue 1 NEW event */ + ev.op =RTE_EVENT_OP_NEW ; + ev.sched_type =RTE_SCHED_TYPE_ATOMIC ; + ev.queue_id =0 ; + ev.priority =0 ; + ev.u64 =0 ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +/* Dequeue and enqueue 1 FORWARD event */ + timeout =0xFFFFFFFFF ; +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + + ev.op =RTE_EVENT_OP_FORWARD ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +/* Dequeue and enqueue 1 RELEASE operation */ +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + + ev.op =RTE_EVENT_OP_RELEASE ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +cleanup (t); +return 0 ; + +err: +cleanup (t); +return -1 ; +} + +static int +test_directed_traffic (struct test *t) +{ +uint64_t timeout; +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +struct rte_event ev; +uint8_t queue_id; +int ret; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =1 ; + config.nb_event_ports =1 ; + config.nb_single_link_event_port_queues =1 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + +/* Configure the device with 1 DIR port and queue */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + +/* Configure the ports and queues */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + + port_conf.event_port_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) { +printf ("%d: port 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + + queue_conf.event_queue_cfg =RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q0 */ + queue_id =0 ; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 0 link expected to succeed\n" , + __LINE__); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Enqueue 1 NEW event */ + ev.op =RTE_EVENT_OP_NEW ; + ev.queue_id =0 ; + ev.priority =0 ; + ev.u64 =0 ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +/* Dequeue and enqueue 1 FORWARD event */ + timeout =0xFFFFFFFFF ; +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + +if (ev.queue_id !=0 ) { +printf ("%d: invalid dequeued event queue ID (%d)\n" , + __LINE__, ev.queue_id); + goto err; + } + + ev.op =RTE_EVENT_OP_FORWARD ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +/* Dequeue and enqueue 1 RELEASE operation */ +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + + ev.op =RTE_EVENT_OP_RELEASE ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + +cleanup (t); +return 0 ; + +err: +cleanup (t); +return -1 ; +} + +static int +test_deferred_sched (struct test *t) +{ +uint64_t timeout; +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +const int num_events =128 ; +struct rte_event ev; +uint8_t queue_id; +int ret, i; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =1 ; + config.nb_event_ports =2 ; + config.nb_single_link_event_port_queues =0 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + +/* Configure the device with 2 LDB ports and 1 queue */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + + ret =rte_pmd_dlb_set_token_pop_mode (evdev,0 ,DEFERRED_POP ); +if (ret <0 ) { +printf ("%d: Error setting deferred scheduling\n" , __LINE__); + goto err; + } + + ret =rte_pmd_dlb_set_token_pop_mode (evdev,1 ,DEFERRED_POP ); +if (ret <0 ) { +printf ("%d: Error setting deferred scheduling\n" , __LINE__); + goto err; + } + +/* Configure the ports and queues */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + + port_conf.dequeue_depth =1 ; + +if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) { +printf ("%d: port 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_port_setup (evdev,1 , &port_conf) <0 ) { +printf ("%d: port 1 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + + queue_conf.schedule_type =RTE_SCHED_TYPE_PARALLEL ; + queue_conf.nb_atomic_order_sequences =0 ; + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q0 and P1->Q0 */ + queue_id =0 ; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 0 link expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_port_link (evdev,1 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 1 link expected to succeed\n" , + __LINE__); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + +/* Enqueue 128 NEW events */ + ev.op =RTE_EVENT_OP_NEW ; + ev.sched_type =RTE_SCHED_TYPE_PARALLEL ; + ev.queue_id =0 ; + ev.priority =0 ; + ev.u64 =0 ; + +for (i =0 ; i < num_events; i++) { +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + +/* Dequeue two events from port 0 (dequeue_depth * 2 due to the +* reserved token scheme) +*/ + timeout =0xFFFFFFFFF ; +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + +/* Dequeue (and release) all other events from port 1. Deferred +* scheduling ensures no other events are scheduled to port 0 without a +* subsequent rte_event_dequeue_burst() call. +*/ +for (i =0 ; i < num_events -2 ; i++) { +if (rte_event_dequeue_burst (evdev,1 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + + ev.op =RTE_EVENT_OP_RELEASE ; + +if (rte_event_enqueue_burst (evdev,1 , &ev,1 ) !=1 ) { +printf ("%d: RELEASE enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + +cleanup (t); +return 0 ; + +err: +cleanup (t); +return -1 ; +} + +static int +test_delayed_pop (struct test *t) +{ +uint64_t timeout; +struct rte_event_dev_config config = {0 }; +struct rte_event_queue_conf queue_conf; +struct rte_event_port_conf port_conf; +struct rte_event_dev_info info; +int ret, i, num_events; +struct rte_event ev; +uint8_t queue_id; + +if (rte_event_dev_info_get (evdev, &info)) { +printf ("%d: Error querying device info\n" , __LINE__); +return -1 ; + } + + config.nb_event_queues =1 ; + config.nb_event_ports =1 ; + config.nb_single_link_event_port_queues =0 ; + config.nb_event_queue_flows = info.max_event_queue_flows; + config.nb_events_limit = info.max_num_events; + config.nb_event_port_dequeue_depth = info.max_event_port_dequeue_depth; + config.nb_event_port_enqueue_depth = info.max_event_port_enqueue_depth; + config.dequeue_timeout_ns = info.max_dequeue_timeout_ns; + config.event_dev_cfg =RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ; + +/* Configure the device with 1 LDB port and queue */ + ret =rte_event_dev_configure (evdev, &config); +if (ret <0 ) { +printf ("%d: Error configuring device\n" , __LINE__); +return -1 ; + } + + ret =rte_pmd_dlb_set_token_pop_mode (evdev,0 ,DELAYED_POP ); +if (ret <0 ) { +printf ("%d: Error setting deferred scheduling\n" , __LINE__); + goto err; + } + +/* Configure the ports and queues */ +if (rte_event_port_default_conf_get (evdev,0 , &port_conf)) { +printf ("%d: Error querying default port conf\n" , __LINE__); + goto err; + } + + port_conf.dequeue_depth =16 ; + port_conf.event_port_cfg =RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; + +if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) { +printf ("%d: port 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +if (rte_event_queue_default_conf_get (evdev,0 , &queue_conf)) { +printf ("%d: Error querying default queue conf\n" , __LINE__); + goto err; + } + +if (rte_event_queue_setup (evdev,0 , &queue_conf) <0 ) { +printf ("%d: queue 0 setup expected to succeed\n" , + __LINE__); + goto err; + } + +/* Link P0->Q0 */ + queue_id =0 ; + +if (rte_event_port_link (evdev,0 , &queue_id,NULL ,1 ) !=1 ) { +printf ("%d: port 0 link expected to succeed\n" , + __LINE__); + goto err; + } + +/* Start the device */ +if (rte_event_dev_start (evdev) <0 ) { +printf ("%d: device start failed\n" , __LINE__); + goto err; + } + + num_events =2 * port_conf.dequeue_depth; + +/* Enqueue 2 * dequeue_depth NEW events. Due to the PMD's reserved +* token scheme, the port will initially behave as though its +* dequeue_depth is twice the requested size. +*/ + ev.op =RTE_EVENT_OP_NEW ; + ev.sched_type =RTE_SCHED_TYPE_PARALLEL ; + ev.queue_id =0 ; + ev.priority =0 ; + ev.u64 =0 ; + +for (i =0 ; i < num_events; i++) { +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + +/* Flush these events out of the CQ */ + timeout =0xFFFFFFFFF ; + +for (i =0 ; i < num_events; i++) { +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + } + + ev.op =RTE_EVENT_OP_RELEASE ; + +for (i =0 ; i < num_events; i++) { +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: RELEASE enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + +/* Enqueue 2 * dequeue_depth NEW events again */ + ev.op =RTE_EVENT_OP_NEW ; + ev.sched_type =RTE_SCHED_TYPE_ATOMIC ; + ev.queue_id =0 ; + ev.priority =0 ; + ev.u64 =0 ; + +for (i =0 ; i < num_events; i++) { +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: NEW enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + +/* Dequeue dequeue_depth events but only release dequeue_depth - 2. +* Delayed pop won't perform the pop and no more events will be +* scheduled. +*/ +for (i =0 ; i < port_conf.dequeue_depth; i++) { +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + } + + ev.op =RTE_EVENT_OP_RELEASE ; + +for (i =0 ; i < port_conf.dequeue_depth -2 ; i++) { +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: RELEASE enqueue expected to succeed\n" , + __LINE__); + goto err; + } + } + + timeout =0x10000 ; + + ret =rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout); +if (ret !=0 ) { +printf ("%d: event dequeue expected to fail (ret = %d)\n" , + __LINE__, ret); + goto err; + } + +/* Release one more event. This will trigger the token pop, and +* another batch of events will be scheduled to the device. +*/ + ev.op =RTE_EVENT_OP_RELEASE ; + +if (rte_event_enqueue_burst (evdev,0 , &ev,1 ) !=1 ) { +printf ("%d: RELEASE enqueue expected to succeed\n" , + __LINE__); + goto err; + } + + timeout =0xFFFFFFFFF ; + +for (i =0 ; i < port_conf.dequeue_depth -1 ; i++) { +if (rte_event_dequeue_burst (evdev,0 , &ev,1 , timeout) !=1 ) { +printf ("%d: event dequeue expected to succeed\n" , + __LINE__); + goto err; + } + } + +cleanup (t); +return 0 ; + +err: +cleanup (t); +return -1 ; +} + +static int +do_selftest (void ) +{ +struct test t; +int ret; + +/* Only create mbuf pool once, reuse for each test run */ +if (!eventdev_func_mempool) { + eventdev_func_mempool = +rte_pktmbuf_pool_create ("EVENTDEV_DLB_SA_MBUF_POOL" , + (1 <<12 ),/* 4k buffers */ +32 /*MBUF_CACHE_SIZE*/ , +0 , +512 ,/* use very small mbufs */ +rte_socket_id ()); +if (!eventdev_func_mempool) { +printf ("ERROR creating mempool\n" ); + goto test_fail; + } + } + t.mbuf_pool = eventdev_func_mempool; + +printf ("*** Running Stop Flush test...\n" ); + ret =test_stop_flush (&t); +if (ret !=0 ) { +printf ("ERROR - Stop Flush test FAILED.\n" ); +return ret; + } + +printf ("*** Running Single Link test...\n" ); + ret =test_single_link (); +if (ret !=0 ) { +printf ("ERROR - Single Link test FAILED.\n" ); + + goto test_fail; + } + +printf ("*** Running Info Get test...\n" ); + ret =test_info_get (); +if (ret !=0 ) { +printf ("ERROR - Stop Flush test FAILED.\n" ); +return ret; + } + +printf ("*** Running Reconfiguration Link test...\n" ); + ret =test_reconfiguration_link (&t); +if (ret !=0 ) { +printf ("ERROR - Reconfiguration Link test FAILED.\n" ); + + goto test_fail; + } + +printf ("*** Running Load-Balanced Traffic test...\n" ); + ret =test_load_balanced_traffic (&t); +if (ret !=0 ) { +printf ("ERROR - Load-Balanced Traffic test FAILED.\n" ); + + goto test_fail; + } + +printf ("*** Running Directed Traffic test...\n" ); + ret =test_directed_traffic (&t); +if (ret !=0 ) { +printf ("ERROR - Directed Traffic test FAILED.\n" ); + + goto test_fail; + } + +printf ("*** Running Deferred Scheduling test...\n" ); + ret =test_deferred_sched (&t); +if (ret !=0 ) { +printf ("ERROR - Deferred Scheduling test FAILED.\n" ); + + goto test_fail; + } + +printf ("*** Running Delayed Pop test...\n" ); + ret =test_delayed_pop (&t); +if (ret !=0 ) { +printf ("ERROR - Delayed Pop test FAILED.\n" ); + + goto test_fail; + } + +return 0 ; + +test_fail: +return -1 ; +} + +int +test_dlb_eventdev (void ) +{ +const char *dlb_eventdev_name ="event_dlb" ; +uint8_t num_evdevs =rte_event_dev_count (); +int i, ret =0 ; +int found =0 , skipped =0 , passed =0 , failed =0 ; +struct rte_event_dev_info info; + +for (i =0 ; found + skipped < num_evdevs && i <RTE_EVENT_MAX_DEVS ; + i++) { + ret =rte_event_dev_info_get (i, &info); +if (ret <0 ) +continue ; + +/* skip non-dlb event devices */ +if (strncmp (info.driver_name, dlb_eventdev_name, +sizeof (*info.driver_name)) !=0 ) { + skipped++; +continue ; + } + + evdev =rte_event_dev_get_dev_id (info.driver_name); +if (evdev <0 ) { +printf ("Could not get dev_id for eventdev with name %s, i=%d\n" , + info.driver_name, i); + skipped++; +continue ; + } + found++; +printf ("Running selftest on eventdev %s\n" , info.driver_name); + ret =do_selftest (); +if (ret ==0 ) { + passed++; +printf ("Selftest passed for eventdev %s\n" , + info.driver_name); + }else { + failed++; +printf ("Selftest failed for eventdev %s, err=%d\n" , + info.driver_name, ret); + } + } + +if (found ==0 ) { +printf ("No dlb eventdevs found - attempt to create %s\n" , + dlb_eventdev_name); +if (rte_vdev_init (dlb_eventdev_name,NULL ) <0 ) { +printf ("Error creating eventdev %s\n" , + dlb_eventdev_name); + goto test_fail; + } + evdev =rte_event_dev_get_dev_id (dlb_eventdev_name); +if (evdev <0 ) { +printf ("Error finding newly created eventdev %s\n" , + dlb_eventdev_name); + goto test_fail; + } + found++; +printf ("Running selftest on eventdev %s\n" , dlb_eventdev_name); + ret =do_selftest (); +if (ret ==0 ) { + passed++; +printf ("Selftest passed for eventdev %s\n" , + dlb_eventdev_name); + }else { + failed++; +printf ("Selftest failed for eventdev %s, err=%d\n" , + dlb_eventdev_name, ret); + } + } + +test_fail: +printf ("Ran selftest on %d eventdevs, %d skipped, %d passed, %d failed\n" , + found, skipped, passed, failed); +return ret; +}
Signature · 1 line
2.13.6
PATCH 15/27 event/dlb: add probe
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 519 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 519 insertions(+) create mode 100644 drivers/event/dlb/dlb.c
drivers/event/dlb/dlb.c new file +519 -0
@@ -0,0 +1,519 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright(c) 2016-2020 Intel Corporation +*/ + +#include <stdint.h> +#include <stdbool.h> +#include <stdio.h> +#include <sys/mman.h> +#include <sys/fcntl.h> +#include <errno.h> +#include <assert.h> +#include <unistd.h> +#include <string.h> +#include <pthread.h> +#include <nmmintrin.h> + +#include <rte_common.h> +#include <rte_config.h> +#include <rte_debug.h> +#include <rte_log.h> +#include <rte_dev.h> +#include <rte_mbuf.h> +#include <rte_ring.h> +#include <rte_errno.h> +#include <rte_kvargs.h> +#include <rte_malloc.h> +#include <rte_cycles.h> +#include <rte_io.h> +#include <rte_eventdev.h> +#include <rte_eventdev_pmd.h> +#include <rte_string_fns.h> +#include <rte_prefetch.h> + +#include "dlb_priv.h" +#include "dlb_iface.h" +#include "dlb_inline_fns.h" + +/* +* Resources exposed to eventdev. Some values overridden at runtime using +* values returned by the DLB kernel driver. +*/ +#if (RTE_EVENT_MAX_QUEUES_PER_DEV >UINT8_MAX ) +#error "RTE_EVENT_MAX_QUEUES_PER_DEV cannot fit in member max_event_queues" +#endif +static struct rte_event_dev_info evdev_dlb_default_info = { + .driver_name ="" ,/* probe will set */ + .min_dequeue_timeout_ns =DLB_MIN_DEQUEUE_TIMEOUT_NS , + .max_dequeue_timeout_ns =DLB_MAX_DEQUEUE_TIMEOUT_NS , +#if (RTE_EVENT_MAX_QUEUES_PER_DEV < DLB_MAX_NUM_LDB_QUEUES) + .max_event_queues =RTE_EVENT_MAX_QUEUES_PER_DEV , +#else + .max_event_queues =DLB_MAX_NUM_LDB_QUEUES , +#endif + .max_event_queue_flows =DLB_MAX_NUM_FLOWS , + .max_event_queue_priority_levels =DLB_QID_PRIORITIES , + .max_event_priority_levels =DLB_QID_PRIORITIES , + .max_event_ports =DLB_MAX_NUM_LDB_PORTS , + .max_event_port_dequeue_depth =DLB_MAX_CQ_DEPTH , + .max_event_port_enqueue_depth =DLB_MAX_ENQUEUE_DEPTH , + .max_event_port_links =DLB_MAX_NUM_QIDS_PER_LDB_CQ , + .max_num_events =DLB_MAX_NUM_LDB_CREDITS , + .max_single_link_event_port_queue_pairs =DLB_MAX_NUM_DIR_PORTS , + .event_dev_cap = (RTE_EVENT_DEV_CAP_QUEUE_QOS | +RTE_EVENT_DEV_CAP_EVENT_QOS | +RTE_EVENT_DEV_CAP_BURST_MODE | +RTE_EVENT_DEV_CAP_DISTRIBUTED_SCHED | +RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE | +RTE_EVENT_DEV_CAP_QUEUE_ALL_TYPES ), +}; + +/* These functions will vary based on processor capabilities */ +static struct dlb_port_low_level_io_functions qm_mmio_fns; + +static int +dlb_hw_query_resources (struct dlb_eventdev *dlb) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_hw_resource_info *dlb_info = &handle->info; +int ret; + +/* Query driver resources provisioned for this VF */ + + ret =dlb_iface_get_num_resources (handle, + &dlb->hw_rsrc_query_results); +if (ret) { +DLB_LOG_ERR ("ioctl get dlb num resources, err=%d\n" , + ret); +return ret; + } + +/* Complete filling in device resource info returned to evdev app, +* overriding any default values. +* The capabilities (CAPs) were set at compile time. +*/ + + evdev_dlb_default_info.max_event_queues = + dlb->hw_rsrc_query_results.num_ldb_queues; + + evdev_dlb_default_info.max_event_ports = + dlb->hw_rsrc_query_results.num_ldb_ports; + + evdev_dlb_default_info.max_num_events = + dlb->hw_rsrc_query_results.max_contiguous_ldb_credits; + +/* Save off values used when creating the scheduling domain. */ + + handle->info.num_sched_domains = + dlb->hw_rsrc_query_results.num_sched_domains; + + handle->info.hw_rsrc_max.nb_events_limit = + dlb->hw_rsrc_query_results.max_contiguous_ldb_credits; + + handle->info.hw_rsrc_max.num_queues = + dlb->hw_rsrc_query_results.num_ldb_queues + + dlb->hw_rsrc_query_results.num_dir_ports; + + handle->info.hw_rsrc_max.num_ldb_queues = + dlb->hw_rsrc_query_results.num_ldb_queues; + + handle->info.hw_rsrc_max.num_ldb_ports = + dlb->hw_rsrc_query_results.num_ldb_ports; + + handle->info.hw_rsrc_max.num_dir_ports = + dlb->hw_rsrc_query_results.num_dir_ports; + + handle->info.hw_rsrc_max.reorder_window_size = + dlb->hw_rsrc_query_results.num_hist_list_entries; + +rte_memcpy (dlb_info, &handle->info.hw_rsrc_max,sizeof (*dlb_info)); + +return 0 ; +} + +/* Wrapper for string to int conversion. Substituted for atoi(...), which is +* unsafe. +*/ +#define RTE_BASE_10 10 +int dlb_string_to_int (int *result,const char *str) +{ +long ret; + +if (str ==NULL || result ==NULL ) +return -EINVAL ; + + errno =0 ; + ret =strtol (str,NULL ,RTE_BASE_10 ); +if (errno) +return -errno; + +/* long int and int may be different width for some architectures */ +if (ret <INT_MIN || ret >INT_MAX ) +return -EINVAL ; + + *result = ret; +return 0 ; +} + +int +set_numa_node(const char *key __rte_unused, const char *value, void *opaque) +{ +int *socket_id = opaque; +int ret; + + ret =dlb_string_to_int (socket_id, value); +if (ret <0 ) +return ret; + +if (*socket_id >RTE_MAX_NUMA_NODES ) +return -EINVAL ; + +return 0 ; +} + +int +set_max_num_events(const char *key __rte_unused, +const char *value, +void *opaque) +{ +int *max_num_events = opaque; +int ret; + +if (value ==NULL || opaque ==NULL ) { +DLB_LOG_ERR ("NULL pointer\n" ); +return -EINVAL ; + } + + ret =dlb_string_to_int (max_num_events, value); +if (ret <0 ) +return ret; + +if (*max_num_events <0 || *max_num_events >DLB_MAX_NUM_LDB_CREDITS ) { +DLB_LOG_ERR ("dlb: max_num_events must be between 0 and %d\n" , +DLB_MAX_NUM_LDB_CREDITS ); +return -EINVAL ; + } + +return 0 ; +} + +int +set_num_dir_credits(const char *key __rte_unused, +const char *value, +void *opaque) +{ +int *num_dir_credits = opaque; +int ret; + +if (value ==NULL || opaque ==NULL ) { +DLB_LOG_ERR ("NULL pointer\n" ); +return -EINVAL ; + } + + ret =dlb_string_to_int (num_dir_credits, value); +if (ret <0 ) +return ret; + +if (*num_dir_credits <0 || + *num_dir_credits >DLB_MAX_NUM_DIR_CREDITS ) { +DLB_LOG_ERR ("dlb: num_dir_credits must be between 0 and %d\n" , +DLB_MAX_NUM_DIR_CREDITS ); +return -EINVAL ; + } + +return 0 ; +} + +static int +set_dev_id(const char *key __rte_unused, +const char *value, +void *opaque) +{ +int *dev_id = opaque; +int ret; + +if (value ==NULL || opaque ==NULL ) { +DLB_LOG_ERR ("NULL pointer\n" ); +return -EINVAL ; + } + + ret =dlb_string_to_int (dev_id, value); +if (ret <0 ) +return ret; + +return 0 ; +} + +static int +set_defer_sched(const char *key __rte_unused, +const char *value, +void *opaque) +{ +int *defer_sched = opaque; + +if (value ==NULL || opaque ==NULL ) { +DLB_LOG_ERR ("NULL pointer\n" ); +return -EINVAL ; + } + +if (strncmp (value,"on" ,2 ) !=0 ) { +DLB_LOG_ERR ("Invalid defer_sched argument \"%s\" (expected \"on\")\n" , + value); +return -EINVAL ; + } + + *defer_sched =1 ; + +return 0 ; +} + +static int +set_num_atm_inflights(const char *key __rte_unused, +const char *value, +void *opaque) +{ +int *num_atm_inflights = opaque; +int ret; + +if (value ==NULL || opaque ==NULL ) { +DLB_LOG_ERR ("NULL pointer\n" ); +return -EINVAL ; + } + + ret =dlb_string_to_int (num_atm_inflights, value); +if (ret <0 ) +return ret; + +if (*num_atm_inflights <0 || + *num_atm_inflights >DLB_MAX_NUM_ATM_INFLIGHTS ) { +DLB_LOG_ERR ("dlb: atm_inflights must be between 0 and %d\n" , +DLB_MAX_NUM_ATM_INFLIGHTS ); +return -EINVAL ; + } + +return 0 ; +} + + +static void +dlb_qm_mmio_fn_init (void ) +{ +/* Process-local function pointers for performing low level port i/o */ + +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_MOVDIR64B )) + qm_mmio_fns.pp_enqueue_four = dlb_movdir64b; +else + qm_mmio_fns.pp_enqueue_four = dlb_movntdq; +} + +int +dlb_primary_eventdev_probe (struct rte_eventdev *dev, +const char *name, +struct dlb_devargs *dlb_args, +bool is_vdev) +{ +struct dlb_eventdev *dlb; +int err, i; + + dlb = dev->data->dev_private; + + dlb->event_dev = dev;/* backlink */ + dlb->is_vdev = is_vdev;/* vdev or pf */ + + evdev_dlb_default_info.driver_name = name; + + dlb->max_num_events_override = dlb_args->max_num_events; + dlb->num_dir_credits_override = dlb_args->num_dir_credits_override; + dlb->qm_instance.device_path_id = dlb_args->dev_id; + dlb->defer_sched = dlb_args->defer_sched; + dlb->num_atm_inflights_per_queue = dlb_args->num_atm_inflights; + +/* Open the interface. +* For vdev mode, this means open the dlb kernel module. +*/ + err =dlb_iface_open (&dlb->qm_instance, name); +if (err <0 ) { +DLB_LOG_ERR ("could not open event hardware device, err=%d\n" , + err); +return err; + } + + err =dlb_iface_get_device_version (&dlb->qm_instance, &dlb->revision); +if (err <0 ) { +DLB_LOG_ERR ("dlb: failed to get the device version, err=%d\n" , + err); +return err; + } + + err =dlb_hw_query_resources (dlb); +if (err) { +DLB_LOG_ERR ("get resources err=%d for %s\n" , + err, name); +return err; + } + + err =dlb_iface_get_cq_poll_mode (&dlb->qm_instance, &dlb->poll_mode); +if (err <0 ) { +DLB_LOG_ERR ("dlb: failed to get the poll mode, err=%d\n" , + err); +return err; + } + +/* Complete xtstats runtime initialization */ + err =dlb_xstats_init (dlb); +if (err) { +DLB_LOG_ERR ("dlb: failed to init xstats, err=%d\n" , err); +return err; + } + +/* Initialize each port's token pop mode */ +for (i =0 ; i <DLB_MAX_NUM_PORTS ; i++) + dlb->ev_ports[i].qm_port.token_pop_mode =AUTO_POP ; + +rte_spinlock_init (&dlb->qm_instance.resource_lock); + +dlb_qm_mmio_fn_init (); + +dlb_iface_low_level_io_init (dlb); + +dlb_entry_points_init (dev); + +return 0 ; +} + +int +dlb_secondary_eventdev_probe (struct rte_eventdev *dev, +const char *name, +bool is_vdev) +{ +struct dlb_eventdev *dlb; +int err; + +RTE_SET_USED (is_vdev); + + dlb = dev->data->dev_private; + + evdev_dlb_default_info.driver_name = name; + + err =dlb_iface_open (&dlb->qm_instance, name); +if (err <0 ) { +DLB_LOG_ERR ("could not open event hardware device, err=%d\n" , + err); +return err; + } + + err =dlb_hw_query_resources (dlb); +if (err) { +DLB_LOG_ERR ("get resources err=%d for %s\n" , + err, name); +return err; + } + +dlb_qm_mmio_fn_init (); + +dlb_iface_low_level_io_init (dlb); + +dlb_entry_points_init (dev); + +return 0 ; +} + +int +dlb_parse_params (const char *params, +const char *name, +struct dlb_devargs *dlb_args) +{ +int ret =0 ; +static const char *const args[] = {NUMA_NODE_ARG , +DLB_MAX_NUM_EVENTS , +DLB_NUM_DIR_CREDITS , +DEV_ID_ARG , +DLB_DEFER_SCHED_ARG , +DLB_NUM_ATM_INFLIGHTS_ARG , +NULL }; + +if (params !=NULL && params[0 ] !='\0' ) { +struct rte_kvargs *kvlist =rte_kvargs_parse (params, args); + +if (!kvlist) { +RTE_LOG (INFO ,PMD , +"Ignoring unsupported parameters when creating device '%s'\n" , + name); + }else { +int ret =rte_kvargs_process (kvlist,NUMA_NODE_ARG , + set_numa_node, + &dlb_args->socket_id); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing numa node parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + + ret =rte_kvargs_process (kvlist,DLB_MAX_NUM_EVENTS , + set_max_num_events, + &dlb_args->max_num_events); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing max_num_events parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + + ret =rte_kvargs_process (kvlist, +DLB_NUM_DIR_CREDITS , + set_num_dir_credits, + &dlb_args->num_dir_credits_override); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing num_dir_credits parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + + ret =rte_kvargs_process (kvlist,DEV_ID_ARG , + set_dev_id, + &dlb_args->dev_id); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing dev_id parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + + ret =rte_kvargs_process (kvlist,DLB_DEFER_SCHED_ARG , + set_defer_sched, + &dlb_args->defer_sched); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing defer_sched parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + + ret =rte_kvargs_process (kvlist, +DLB_NUM_ATM_INFLIGHTS_ARG , + set_num_atm_inflights, + &dlb_args->num_atm_inflights); +if (ret !=0 ) { +DLB_LOG_ERR ("%s: Error parsing atm_inflights parameter" , + name); +rte_kvargs_free (kvlist); +return ret; + } + +rte_kvargs_free (kvlist); + } + } +return ret; +} + +/* declared extern in header, for access from other .c files */ +int eventdev_dlb_log_level; + +RTE_INIT (evdev_dlb_init_log) +{ + eventdev_dlb_log_level =rte_log_register ("pmd.event.dlb" ); +if (eventdev_dlb_log_level >=0 ) +rte_log_set_level (eventdev_dlb_log_level,RTE_LOG_NOTICE ); +}
Signature · 1 line
2.13.6
PATCH 16/27 event/dlb: add infos_get and configure
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 401 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 401 insertions(+)
drivers/event/dlb/dlb.c +401 -0
@@ -220,6 +220,394 @@ set_num_dir_credits(const char *key __rte_unused, DLB_MAX_NUM_DIR_CREDITS);return -EINVAL ; } +return 0 ; +} + +/* VDEV-only notes: +* This function first unmaps all memory mappings and closes the +* domain's file descriptor, which causes the driver to reset the +* scheduling domain. Once that completes (when close() returns), we +* can safely free the dynamically allocated memory used by the +* scheduling domain. +* +* PF-only notes: +* We will maintain a use count and use that to determine when +* a reset is required. In PF mode, we never mmap, or munmap +* device memory, and we own the entire physical PCI device. +*/ + +static void +dlb_hw_reset_sched_domain (const struct rte_eventdev *dev,bool reconfig) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +enum dlb_configuration_state config_state; +int i, j; + +/* Close and reset the domain */ +dlb_iface_domain_close (dlb); + +/* Free all dynamically allocated port memory */ +for (i =0 ; i < dlb->num_ports; i++) +dlb_free_qe_mem (&dlb->ev_ports[i].qm_port); + +/* If reconfiguring, mark the device's queues and ports as "previously +* configured." If the user doesn't reconfigure them, the PMD will +* reapply their previous configuration when the device is started. +*/ + config_state = (reconfig) ?DLB_PREV_CONFIGURED :DLB_NOT_CONFIGURED ; + +for (i =0 ; i < dlb->num_ports; i++) { + dlb->ev_ports[i].qm_port.config_state = config_state; +/* Reset setup_done so ports can be reconfigured */ + dlb->ev_ports[i].setup_done = false; +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) + dlb->ev_ports[i].link[j].mapped = false; + } + +for (i =0 ; i < dlb->num_queues; i++) + dlb->ev_queues[i].qm_queue.config_state = config_state; + +for (i =0 ; i <DLB_MAX_NUM_QUEUES ; i++) + dlb->ev_queues[i].setup_done = false; + + dlb->num_ports =0 ; + dlb->num_ldb_ports =0 ; + dlb->num_dir_ports =0 ; + dlb->num_queues =0 ; + dlb->num_ldb_queues =0 ; + dlb->num_dir_queues =0 ; + dlb->configured = false; +} + +static int +dlb_ldb_credit_pool_create (struct dlb_hw_dev *handle) +{ +struct dlb_create_ldb_pool_args cfg; +struct dlb_cmd_response response; +int ret; + +if (handle ==NULL ) +return -EINVAL ; + +if (!handle->cfg.resources.num_ldb_credits) { + handle->cfg.ldb_credit_pool_id =0 ; + handle->cfg.num_ldb_credits =0 ; +return 0 ; + } + + cfg.response = (uintptr_t )&response; + cfg.num_ldb_credits = handle->cfg.resources.num_ldb_credits; + + ret =dlb_iface_ldb_credit_pool_create (handle, + &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: ldb_credit_pool_create ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); + } + + handle->cfg.ldb_credit_pool_id = response.id; + handle->cfg.num_ldb_credits = cfg.num_ldb_credits; + +return ret; +} + +static int +dlb_dir_credit_pool_create (struct dlb_hw_dev *handle) +{ +struct dlb_create_dir_pool_args cfg; +struct dlb_cmd_response response; +int ret; + +if (handle ==NULL ) +return -EINVAL ; + +if (!handle->cfg.resources.num_dir_credits) { + handle->cfg.dir_credit_pool_id =0 ; + handle->cfg.num_dir_credits =0 ; +return 0 ; + } + + cfg.response = (uintptr_t )&response; + cfg.num_dir_credits = handle->cfg.resources.num_dir_credits; + + ret =dlb_iface_dir_credit_pool_create (handle, + &cfg); +if (ret <0 ) +DLB_LOG_ERR ("dlb: dir_credit_pool_create ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); + + handle->cfg.dir_credit_pool_id = response.id; + handle->cfg.num_dir_credits = cfg.num_dir_credits; + +return ret; +} + +static int +dlb_hw_create_sched_domain (struct dlb_hw_dev *handle, +struct dlb_eventdev *dlb, +const struct dlb_hw_rsrcs *resources_asked) +{ +int ret =0 ; +struct dlb_create_sched_domain_args *config_params; +struct dlb_cmd_response response; + +if (resources_asked ==NULL ) { +DLB_LOG_ERR ("dlb: dlb_create NULL parameter\n" ); + ret =EINVAL ; + goto error_exit; + } + +/* Map generic qm resources to dlb resources */ + config_params = &handle->cfg.resources; + + config_params->response = (uintptr_t )&response; + +/* DIR ports and queues */ + + config_params->num_dir_ports = + resources_asked->num_dir_ports; + + config_params->num_dir_credits = + resources_asked->num_dir_credits; + +/* LDB ports and queues */ + + config_params->num_ldb_queues = + resources_asked->num_ldb_queues; + + config_params->num_ldb_ports = + resources_asked->num_ldb_ports; + + config_params->num_ldb_credits = + resources_asked->num_ldb_credits; + + config_params->num_atomic_inflights = + dlb->num_atm_inflights_per_queue * + config_params->num_ldb_queues; + + config_params->num_hist_list_entries = config_params->num_ldb_ports * +DLB_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT ; + +/* dlb limited to 1 credit pool per queue type */ + config_params->num_ldb_credit_pools =1 ; + config_params->num_dir_credit_pools =1 ; + +DLB_LOG_DBG ("sched domain create - ldb_qs=%d, ldb_ports=%d, dir_ports=%d, atomic_inflights=%d, hist_list_entries=%d, ldb_credits=%d, dir_credits=%d, ldb_cred_pools=%d, dir-credit_pools=%d\n" , + config_params->num_ldb_queues, + config_params->num_ldb_ports, + config_params->num_dir_ports, + config_params->num_atomic_inflights, + config_params->num_hist_list_entries, + config_params->num_ldb_credits, + config_params->num_dir_credits, + config_params->num_ldb_credit_pools, + config_params->num_dir_credit_pools); + +/* Configure the QM */ + + ret =dlb_iface_sched_domain_create (handle, config_params); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: domain create failed, device_id = %d, (driver ret = %d, extra status: %s)\n" , + handle->device_id, + ret, + dlb_error_strings[response.status]); + goto error_exit; + } + + handle->domain_id = response.id; + handle->domain_id_valid =1 ; + + config_params->response =0 ; + + ret =dlb_ldb_credit_pool_create (handle); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: create ldb credit pool failed\n" ); + goto error_exit2; + } + + ret =dlb_dir_credit_pool_create (handle); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: create dir credit pool failed\n" ); + goto error_exit2; + } + + handle->cfg.configured = true; + +return 0 ; + +error_exit2: +dlb_iface_domain_close (dlb); + +error_exit: + +return ret; +} + +/* End HW specific */ +static void +dlb_eventdev_info_get (struct rte_eventdev *dev, +struct rte_event_dev_info *dev_info) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +int ret; + + ret =dlb_hw_query_resources (dlb); +if (ret) { +const struct rte_eventdev_data *data = dev->data; + +DLB_LOG_ERR ("get resources err=%d, devid=%d\n" , + ret, data->dev_id); +/* fn is void, so fall through and return values set up in +* probe +*/ + } + +/* Add num resources currently owned by this domain. +* These would become available if the scheduling domain were reset due +* to the application recalling eventdev_configure to *reconfigure* the +* domain. +*/ + evdev_dlb_default_info.max_event_ports += dlb->num_ldb_ports; + evdev_dlb_default_info.max_event_queues += dlb->num_ldb_queues; + evdev_dlb_default_info.max_num_events += dlb->num_ldb_credits; + +/* In DLB A-stepping hardware, applications are limited to 128 +* configured ports (load-balanced or directed). The reported number of +* available ports must reflect this. +*/ +if (dlb->revision <DLB_REV_B0 ) { +int used_ports; + + used_ports =DLB_MAX_NUM_LDB_PORTS +DLB_MAX_NUM_DIR_PORTS - + dlb->hw_rsrc_query_results.num_ldb_ports - + dlb->hw_rsrc_query_results.num_dir_ports; + + evdev_dlb_default_info.max_event_ports = +RTE_MIN (evdev_dlb_default_info.max_event_ports, +128 - used_ports); + } + + evdev_dlb_default_info.max_event_queues = +RTE_MIN (evdev_dlb_default_info.max_event_queues, +RTE_EVENT_MAX_QUEUES_PER_DEV ); + + evdev_dlb_default_info.max_num_events = +RTE_MIN (evdev_dlb_default_info.max_num_events, + dlb->max_num_events_override); + + *dev_info = evdev_dlb_default_info; +} + +/* Note: 1 QM instance per QM device, QM instance/device == event device */ +static int +dlb_eventdev_configure (const struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_hw_rsrcs *rsrcs = &handle->info.hw_rsrc_max; +const struct rte_eventdev_data *data = dev->data; +const struct rte_event_dev_config *config = &data->dev_conf; +int ret; + +/* If this eventdev is already configured, we must release the current +* scheduling domain before attempting to configure a new one. +*/ +if (dlb->configured) { +dlb_hw_reset_sched_domain (dev, true); + + ret =dlb_hw_query_resources (dlb); +if (ret) { +DLB_LOG_ERR ("get resources err=%d, devid=%d\n" , + ret, data->dev_id); +return ret; + } + } + +if (config->nb_event_queues > rsrcs->num_queues) { +DLB_LOG_ERR ("nb_event_queues parameter (%d) exceeds the QM device's capabilities (%d).\n" , + config->nb_event_queues, + rsrcs->num_queues); +return -EINVAL ; + } +if (config->nb_event_ports > (rsrcs->num_ldb_ports + + rsrcs->num_dir_ports)) { +DLB_LOG_ERR ("nb_event_ports parameter (%d) exceeds the QM device's capabilities (%d).\n" , + config->nb_event_ports, + (rsrcs->num_ldb_ports + rsrcs->num_dir_ports)); +return -EINVAL ; + } +if (config->nb_events_limit > rsrcs->nb_events_limit) { +DLB_LOG_ERR ("nb_events_limit parameter (%d) exceeds the QM device's capabilities (%d).\n" , + config->nb_events_limit, + rsrcs->nb_events_limit); +return -EINVAL ; + } + +if (config->event_dev_cfg &RTE_EVENT_DEV_CFG_PER_DEQUEUE_TIMEOUT ) { + dlb->global_dequeue_wait = false; + }else { +uint32_t timeout32; + + dlb->global_dequeue_wait = true; + + timeout32 = config->dequeue_timeout_ns; + +/* PF PMD does not support interrupts, and +* UMONITOR wait is temporarily disabled. +*/ + + dlb->global_dequeue_wait_ticks = + timeout32 * (rte_get_timer_hz () /1E9 ); + } + +/* Does this platform support umonitor/umwait? */ +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_UMWAIT )) { +if (RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE !=0 && +RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE !=1 ) { +DLB_LOG_ERR ("invalid value (%d) for RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE must be 0 or 1.\n" , +RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE ); +return -EINVAL ; + } + dlb->umwait_allowed = true; + } + + rsrcs->num_dir_ports = config->nb_single_link_event_port_queues; + rsrcs->num_ldb_ports = config->nb_event_ports - rsrcs->num_dir_ports; +/* 1 dir queue per dir port */ + rsrcs->num_ldb_queues = config->nb_event_queues - rsrcs->num_dir_ports; + +/* Scale down nb_events_limit by 4 for directed credits, since there +* are 4x as many load-balanced credits. +*/ + rsrcs->num_ldb_credits =0 ; + rsrcs->num_dir_credits =0 ; + +if (rsrcs->num_ldb_queues) + rsrcs->num_ldb_credits = config->nb_events_limit; +if (rsrcs->num_dir_ports) + rsrcs->num_dir_credits = config->nb_events_limit /4 ; +if (dlb->num_dir_credits_override !=-1 ) + rsrcs->num_dir_credits = dlb->num_dir_credits_override; + +if (dlb_hw_create_sched_domain (handle, dlb, rsrcs) <0 ) { +DLB_LOG_ERR ("dlb_hw_create_sched_domain failed\n" ); +return -ENODEV ; + } + + dlb->new_event_limit = config->nb_events_limit; +rte_atomic32_set (&dlb->inflights,0 ); + +/* Save number of ports/queues for this event dev */ + dlb->num_ports = config->nb_event_ports; + dlb->num_queues = config->nb_event_queues; + dlb->num_dir_ports = rsrcs->num_dir_ports; + dlb->num_ldb_ports = dlb->num_ports - dlb->num_dir_ports; + dlb->num_ldb_queues = dlb->num_queues - dlb->num_dir_ports; + dlb->num_dir_queues = dlb->num_dir_ports; + dlb->num_ldb_credits = rsrcs->num_ldb_credits; + dlb->num_dir_credits = rsrcs->num_dir_credits; + + dlb->configured = true;return 0 ; }
@@ -294,6 +682,19 @@ set_num_atm_inflights(const char *key __rte_unused,return 0 ; } +void +dlb_entry_points_init (struct rte_eventdev *dev) +{ +static struct rte_eventdev_ops dlb_eventdev_entry_ops = { + .dev_infos_get = dlb_eventdev_info_get, + .dev_configure = dlb_eventdev_configure, + }; + +/* Expose PMD's eventdev interface */ + + dev->dev_ops = &dlb_eventdev_entry_ops; + +}static void dlb_qm_mmio_fn_init (void )
Signature · 1 line
2.13.6
PATCH 17/27 event/dlb: add queue_def_conf and port_def_conf
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
drivers/event/dlb/dlb.c +31 -0
@@ -612,6 +612,35 @@ dlb_eventdev_configure(const struct rte_eventdev *dev)return 0 ; } +static void +dlb_eventdev_port_default_conf_get (struct rte_eventdev *dev, +uint8_t port_id, +struct rte_event_port_conf *port_conf) +{ +RTE_SET_USED (port_id); +struct dlb_eventdev *dlb; + + dlb =dlb_pmd_priv (dev); + + port_conf->new_event_threshold = dlb->new_event_limit; + port_conf->dequeue_depth =32 ; + port_conf->enqueue_depth =DLB_MAX_ENQUEUE_DEPTH ; + port_conf->event_port_cfg =0 ; +} + +static void +dlb_eventdev_queue_default_conf_get (struct rte_eventdev *dev, +uint8_t queue_id, +struct rte_event_queue_conf *queue_conf) +{ +RTE_SET_USED (dev); +RTE_SET_USED (queue_id); + queue_conf->nb_atomic_flows =1024 ; + queue_conf->nb_atomic_order_sequences =32 ; + queue_conf->event_queue_cfg =0 ; + queue_conf->priority =0 ; +} + static int set_dev_id(const char *key __rte_unused, const char *value,
@@ -688,6 +717,8 @@ dlb_entry_points_init(struct rte_eventdev *dev)static struct rte_eventdev_ops dlb_eventdev_entry_ops = { .dev_infos_get = dlb_eventdev_info_get, .dev_configure = dlb_eventdev_configure, + .queue_def_conf = dlb_eventdev_queue_default_conf_get, + .port_def_conf = dlb_eventdev_port_default_conf_get, };/* Expose PMD's eventdev interface */
Signature · 1 line
2.13.6
PATCH 18/27 event/dlb: add queue setup
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 295 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 295 insertions(+)
drivers/event/dlb/dlb.c +295 -0
@@ -223,6 +223,65 @@ set_num_dir_credits(const char *key __rte_unused,return 0 ; } +static int32_t +dlb_hw_create_ldb_queue (struct dlb_eventdev *dlb, +struct dlb_queue *queue, +const struct rte_event_queue_conf *evq_conf) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_create_ldb_queue_args cfg; +struct dlb_cmd_response response; +int32_t ret; +uint32_t qm_qid; +int sched_type =-1 ; + +if (evq_conf ==NULL ) +return -EINVAL ; + +if (evq_conf->event_queue_cfg &RTE_EVENT_QUEUE_CFG_ALL_TYPES ) { +if (evq_conf->nb_atomic_order_sequences !=0 ) + sched_type =RTE_SCHED_TYPE_ORDERED ; +else + sched_type =RTE_SCHED_TYPE_PARALLEL ; + }else { + sched_type = evq_conf->schedule_type; + } + + cfg.response = (uintptr_t )&response; + cfg.num_atomic_inflights = dlb->num_atm_inflights_per_queue; + cfg.num_sequence_numbers = evq_conf->nb_atomic_order_sequences; + cfg.num_qid_inflights = evq_conf->nb_atomic_order_sequences; + +if (sched_type !=RTE_SCHED_TYPE_ORDERED ) { + cfg.num_sequence_numbers =0 ; + cfg.num_qid_inflights =2048 ; + } + + ret =dlb_iface_ldb_queue_create (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: create LB event queue error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return -EINVAL ; + } + + qm_qid = response.id; + +/* Save off queue config for debug, resource lookups, and reconfig */ + queue->num_qid_inflights = cfg.num_qid_inflights; + queue->num_atm_inflights = cfg.num_atomic_inflights; + + queue->sched_type = sched_type; + queue->config_state =DLB_CONFIGURED ; + +DLB_LOG_DBG ("Created LB event queue %d, nb_inflights=%d, nb_seq=%d, qid inflights=%d\n" , + qm_qid, + cfg.num_atomic_inflights, + cfg.num_sequence_numbers, + cfg.num_qid_inflights); + +return qm_qid; +} +/* VDEV-only notes: * This function first unmaps all memory mappings and closes the * domain's file descriptor, which causes the driver to reset the
@@ -444,6 +503,7 @@ dlb_hw_create_sched_domain(struct dlb_hw_dev *handle, }/* End HW specific */ +static void dlb_eventdev_info_get (struct rte_eventdev *dev,struct rte_event_dev_info *dev_info)
@@ -641,6 +701,240 @@ dlb_eventdev_queue_default_conf_get(struct rte_eventdev *dev, queue_conf->priority =0 ; } +static int32_t +dlb_get_sn_allocation (struct dlb_eventdev *dlb,int group) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_get_sn_allocation_args cfg; +struct dlb_cmd_response response; +int ret; + + cfg.group = group; + cfg.response = (uintptr_t )&response; + + ret =dlb_iface_get_sn_allocation (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: get_sn_allocation ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return response.id; +} + +static int +dlb_set_sn_allocation (struct dlb_eventdev *dlb,int group,int num) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_set_sn_allocation_args cfg; +struct dlb_cmd_response response; +int ret; + + cfg.num = num; + cfg.group = group; + cfg.response = (uintptr_t )&response; + + ret =dlb_iface_set_sn_allocation (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: set_sn_allocation ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return ret; +} + +static int32_t +dlb_get_sn_occupancy (struct dlb_eventdev *dlb,int group) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_get_sn_occupancy_args cfg; +struct dlb_cmd_response response; +int ret; + + cfg.group = group; + cfg.response = (uintptr_t )&response; + + ret =dlb_iface_get_sn_occupancy (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: get_sn_occupancy ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return response.id; +} + +/* Query the current sequence number allocations and, if they conflict with the +* requested LDB queue configuration, attempt to re-allocate sequence numbers. +* This is best-effort; if it fails, the PMD will attempt to configure the +* load-balanced queue and return an error. +*/ +static void +dlb_program_sn_allocation (struct dlb_eventdev *dlb, +const struct rte_event_queue_conf *queue_conf) +{ +int grp_occupancy[DLB_NUM_SN_GROUPS ]; +int grp_alloc[DLB_NUM_SN_GROUPS ]; +int i, sequence_numbers; + + sequence_numbers = (int )queue_conf->nb_atomic_order_sequences; + +for (i =0 ; i <DLB_NUM_SN_GROUPS ; i++) { +int total_slots; + + grp_alloc[i] =dlb_get_sn_allocation (dlb, i); +if (grp_alloc[i] <0 ) +return ; + + total_slots =DLB_MAX_LDB_SN_ALLOC / grp_alloc[i]; + + grp_occupancy[i] =dlb_get_sn_occupancy (dlb, i); +if (grp_occupancy[i] <0 ) +return ; + +/* DLB has at least one available slot for the requested +* sequence numbers, so no further configuration required. +*/ +if (grp_alloc[i] == sequence_numbers && + grp_occupancy[i] < total_slots) +return ; + } + +/* None of the sequence number groups are configured for the requested +* sequence numbers, so we have to reconfigure one of them. This is +* only possible if a group is not in use. +*/ +for (i =0 ; i <DLB_NUM_SN_GROUPS ; i++) { +if (grp_occupancy[i] ==0 ) +break ; + } + +if (i ==DLB_NUM_SN_GROUPS ) { +printf ("[%s()] No groups with %d sequence_numbers are available or have free slots\n" , + __func__, sequence_numbers); +return ; + } + +/* Attempt to configure slot i with the requested number of sequence +* numbers. Ignore the return value -- if this fails, the error will be +* caught during subsequent queue configuration. +*/ +dlb_set_sn_allocation (dlb, i, sequence_numbers); +} + +static int +dlb_eventdev_ldb_queue_setup (struct rte_eventdev *dev, +struct dlb_eventdev_queue *ev_queue, +const struct rte_event_queue_conf *queue_conf) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +int32_t qm_qid; + +if (queue_conf->nb_atomic_order_sequences) +dlb_program_sn_allocation (dlb, queue_conf); + + qm_qid =dlb_hw_create_ldb_queue (dlb, + &ev_queue->qm_queue, + queue_conf); +if (qm_qid <0 ) { +DLB_LOG_ERR ("Failed to create the load-balanced queue\n" ); + +return qm_qid; + } + + dlb->qm_ldb_to_ev_queue_id[qm_qid] = ev_queue->id; + + ev_queue->qm_queue.id = qm_qid; + +return 0 ; +} + +static int dlb_num_dir_queues_setup (struct dlb_eventdev *dlb) +{ +int i, num =0 ; + +for (i =0 ; i < dlb->num_queues; i++) { +if (dlb->ev_queues[i].setup_done && + dlb->ev_queues[i].qm_queue.is_directed) + num++; + } + +return num; +} + +static void +dlb_queue_link_teardown (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *ev_queue) +{ +struct dlb_eventdev_port *ev_port; +int i, j; + +for (i =0 ; i < dlb->num_ports; i++) { + ev_port = &dlb->ev_ports[i]; + +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) { +if (!ev_port->link[j].valid || + ev_port->link[j].queue_id != ev_queue->id) +continue ; + + ev_port->link[j].valid = false; + ev_port->num_links--; + } + } + + ev_queue->num_links =0 ; +} + +static int +dlb_eventdev_queue_setup (struct rte_eventdev *dev, +uint8_t ev_qid, +const struct rte_event_queue_conf *queue_conf) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +struct dlb_eventdev_queue *ev_queue; +int ret; + +if (!queue_conf) +return -EINVAL ; + +if (ev_qid >= dlb->num_queues) +return -EINVAL ; + + ev_queue = &dlb->ev_queues[ev_qid]; + + ev_queue->qm_queue.is_directed = queue_conf->event_queue_cfg & +RTE_EVENT_QUEUE_CFG_SINGLE_LINK ; + ev_queue->id = ev_qid; + ev_queue->conf = *queue_conf; + +if (!ev_queue->qm_queue.is_directed) { + ret =dlb_eventdev_ldb_queue_setup (dev, ev_queue, queue_conf); + }else { +/* The directed queue isn't setup until link time, at which +* point we know its directed port ID. Directed queue setup +* will only fail if this queue is already setup or there are +* no directed queues left to configure. +*/ + ret =0 ; + + ev_queue->qm_queue.config_state =DLB_NOT_CONFIGURED ; + +if (ev_queue->setup_done || +dlb_num_dir_queues_setup (dlb) == dlb->num_dir_queues) + ret = -EINVAL ; + } + +/* Tear down pre-existing port->queue links */ +if (!ret && dlb->run_state ==DLB_RUN_STATE_STOPPED ) +dlb_queue_link_teardown (dlb, ev_queue); + +if (!ret) + ev_queue->setup_done = true; + +return ret; +} + static int set_dev_id(const char *key __rte_unused, const char *value,
@@ -718,6 +1012,7 @@ dlb_entry_points_init(struct rte_eventdev *dev) .dev_infos_get = dlb_eventdev_info_get, .dev_configure = dlb_eventdev_configure, .queue_def_conf = dlb_eventdev_queue_default_conf_get, + .queue_setup = dlb_eventdev_queue_setup, .port_def_conf = dlb_eventdev_port_default_conf_get, };
Signature · 1 line
2.13.6
PATCH 19/27 event/dlb: add port_setup
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 1241 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1241 insertions(+)
drivers/event/dlb/dlb.c +1241 -0
@@ -71,6 +71,29 @@ static struct rte_event_dev_info evdev_dlb_default_info = {/* These functions will vary based on processor capabilities */ static struct dlb_port_low_level_io_functions qm_mmio_fns; +struct process_local_port_data +dlb_port[DLB_MAX_NUM_PORTS ][NUM_DLB_PORT_TYPES ]; + +static __rte_always_inline uint16_t +dlb_read_pc (struct process_local_port_data *port_data,bool ldb) +{ +volatile uint16_t *popcount; + +if (ldb) + popcount = port_data->ldb_popcount; +else + popcount = port_data->dir_popcount; + +return *popcount; +} + +static __rte_always_inline void +dlb_pp_write (struct dlb_enqueue_qe *qe4, +struct process_local_port_data *port_data) +{ + qm_mmio_fns.pp_enqueue_four (qe4, port_data->pp_addr); +} +static int dlb_hw_query_resources (struct dlb_eventdev *dlb) {
@@ -171,6 +194,46 @@ set_numa_node(const char *key __rte_unused, const char *value, void *opaque)return 0 ; } +static inline int +dlb_consume_qe_immediate (struct dlb_port *qm_port,int num) +{ +struct process_local_port_data *port_data; +struct dlb_cq_pop_qe *qe; + +RTE_ASSERT (qm_port->config_state ==DLB_CONFIGURED ); + +if (qm_port->use_rsvd_token_scheme) { +/* Check if there's a deficit of reserved tokens, and return +* early if there are no (unreserved) tokens to consume. +*/ +if (num <= qm_port->cq_rsvd_token_deficit) { + qm_port->cq_rsvd_token_deficit -= num; + qm_port->owed_tokens =0 ; +return 0 ; + } + num -= qm_port->cq_rsvd_token_deficit; + qm_port->cq_rsvd_token_deficit =0 ; + } + + qe = qm_port->consume_qe; + + qe->tokens = num -1 ; + qe->int_arm =0 ; + +/* No store fence needed since no pointer is being sent, and CQ token +* pops can be safely reordered with other HCWs. +*/ + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +dlb_movntdq_single (qe, port_data->pp_addr); + +DLB_LOG_DBG ("dlb: consume immediate - %d QEs\n" , num); + + qm_port->owed_tokens =0 ; + +return 0 ; +} + int set_max_num_events(const char *key __rte_unused, const char *value,
@@ -223,6 +286,455 @@ set_num_dir_credits(const char *key __rte_unused,return 0 ; } +static inline uint16_t +dlb_event_enqueue_delayed (void *event_port, +const struct rte_event events[]); + +static inline uint16_t +dlb_event_enqueue_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num); + +static inline uint16_t +dlb_event_enqueue_new_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num); + +static inline uint16_t +dlb_event_enqueue_forward_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num); + +int +dlb_init_qe_mem (struct dlb_port *qm_port,char *mz_name) +{ +int ret, sz; + + sz =DLB_NUM_QES_PER_CACHE_LINE *sizeof (struct dlb_enqueue_qe ); + + qm_port->qe4 =rte_malloc (mz_name, sz,RTE_CACHE_LINE_SIZE ); + +if (qm_port->qe4 ==NULL ) { +DLB_LOG_ERR ("dlb: no qe4 memory\n" ); + ret = -ENOMEM ; + goto error_exit; + } + +memset (qm_port->qe4,0 , sz); + + ret =dlb_init_consume_qe (qm_port, mz_name); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: dlb_init_consume_qe ret=%d\n" , + ret); + goto error_exit; + } + +return 0 ; + +error_exit: + +dlb_free_qe_mem (qm_port); + +return ret; +} + +int +dlb_init_consume_qe (struct dlb_port *qm_port,char *mz_name) +{ +struct dlb_cq_pop_qe *qe; + + qe =rte_malloc (mz_name, +DLB_NUM_QES_PER_CACHE_LINE * +sizeof (struct dlb_cq_pop_qe ), +RTE_CACHE_LINE_SIZE ); + +if (qe ==NULL ) { +DLB_LOG_ERR ("dlb: no memory for consume_qe\n" ); +return -ENOMEM ; + } + + qm_port->consume_qe = qe; + +memset (qe,0 ,DLB_NUM_QES_PER_CACHE_LINE * +sizeof (struct dlb_cq_pop_qe )); + + qe->qe_valid =0 ; + qe->qe_frag =0 ; + qe->qe_comp =0 ; + qe->cq_token =1 ; +/* Tokens value is 0-based; i.e. '0' returns 1 token, '1' returns 2, +* and so on. +*/ + qe->tokens =0 ;/* set at run time */ + qe->meas_lat =0 ; + qe->no_dec =0 ; +/* Completion IDs are disabled */ + qe->cmp_id =0 ; + +return 0 ; +} + +static int +dlb_hw_create_ldb_port (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +uint32_t dequeue_depth, +uint32_t cq_depth, +uint32_t enqueue_depth, +uint16_t rsvd_tokens, +bool use_rsvd_token_scheme) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_create_ldb_port_args cfg = {0 }; +struct dlb_cmd_response response = {0 }; +int ret; +struct dlb_port *qm_port =NULL ; +char mz_name[RTE_MEMZONE_NAMESIZE ]; +uint32_t qm_port_id; + +if (handle ==NULL ) +return -EINVAL ; + +if (cq_depth <DLB_MIN_LDB_CQ_DEPTH || + cq_depth >DLB_MAX_INPUT_QUEUE_DEPTH ) { +DLB_LOG_ERR ("dlb: invalid cq_depth, must be %d-%d\n" , +DLB_MIN_LDB_CQ_DEPTH ,DLB_MAX_INPUT_QUEUE_DEPTH ); +return -EINVAL ; + } + +if (enqueue_depth <DLB_MIN_ENQUEUE_DEPTH ) { +DLB_LOG_ERR ("dlb: invalid enqueue_depth, must be at least %d\n" , +DLB_MIN_ENQUEUE_DEPTH ); +return -EINVAL ; + } + +rte_spinlock_lock (&handle->resource_lock); + + cfg.response = (uintptr_t )&response; + +/* We round up to the next power of 2 if necessary */ + cfg.cq_depth =rte_align32pow2 (cq_depth); + cfg.cq_depth_threshold = rsvd_tokens; + + cfg.cq_history_list_size =DLB_NUM_HIST_LIST_ENTRIES_PER_LDB_PORT ; + +/* User controls the LDB high watermark via enqueue depth. The DIR high +* watermark is equal, unless the directed credit pool is too small. +*/ + cfg.ldb_credit_high_watermark = enqueue_depth; + +/* If there are no directed ports, the kernel driver will ignore this +* port's directed credit settings. Don't use enqueue_depth if it would +* require more directed credits than are available. +*/ + cfg.dir_credit_high_watermark = +RTE_MIN (enqueue_depth, + handle->cfg.num_dir_credits / dlb->num_ports); + + cfg.ldb_credit_quantum = cfg.ldb_credit_high_watermark /2 ; + cfg.ldb_credit_low_watermark =RTE_MIN (16 , cfg.ldb_credit_quantum); + + cfg.dir_credit_quantum = cfg.dir_credit_high_watermark /2 ; + cfg.dir_credit_low_watermark =RTE_MIN (16 , cfg.dir_credit_quantum); + +/* Per QM values */ + +/* DEBUG +* DLB_LOG_ERR("create ldb port - grp=%d, devId=%d\n", +* handle->cfg.domain_id, handle->device_id); +*/ + + cfg.ldb_credit_pool_id = handle->cfg.ldb_credit_pool_id; + cfg.dir_credit_pool_id = handle->cfg.dir_credit_pool_id; + + ret =dlb_iface_ldb_port_create (handle, &cfg, dlb->poll_mode); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: dlb_ldb_port_create error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); + goto error_exit; + } + + qm_port_id = response.id; + +DLB_LOG_DBG ("dlb: ev_port %d uses qm LB port %d <<<<<\n" , + ev_port->id, qm_port_id); + + qm_port = &ev_port->qm_port; + qm_port->ev_port = ev_port;/* back ptr */ + qm_port->dlb = dlb;/* back ptr */ +/* +* Allocate and init local qe struct(s). +* Note: MOVDIR64 requires the enqueue QE (qe4) to be aligned. +*/ + +snprintf (mz_name,sizeof (mz_name),"%s_ldb_port%d" , + handle->device_name, + ev_port->id); + + ret =dlb_init_qe_mem (qm_port, mz_name); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: init_qe_mem failed, ret=%d\n" , ret); + goto error_exit; + } + + qm_port->pp_mmio_base =DLB_LDB_PP_BASE +PAGE_SIZE * qm_port_id; + qm_port->id = qm_port_id; + +/* The credit window is one high water mark of QEs */ + qm_port->ldb_pushcount_at_credit_expiry =0 ; + qm_port->cached_ldb_credits = cfg.ldb_credit_high_watermark; +/* The credit window is one high water mark of QEs */ + qm_port->dir_pushcount_at_credit_expiry =0 ; + qm_port->cached_dir_credits = cfg.dir_credit_high_watermark; + qm_port->cq_depth = cfg.cq_depth; +/* CQs with depth < 8 use an 8-entry queue, but withhold credits so +* the effective depth is smaller. +*/ + qm_port->cq_depth = cfg.cq_depth <=8 ?8 : cfg.cq_depth; + qm_port->cq_idx =0 ; + qm_port->cq_idx_unmasked =0 ; +if (dlb->poll_mode ==DLB_CQ_POLL_MODE_SPARSE ) + qm_port->cq_depth_mask = (qm_port->cq_depth *4 ) -1 ; +else + qm_port->cq_depth_mask = qm_port->cq_depth -1 ; + + qm_port->gen_bit_shift =__builtin_popcount (qm_port->cq_depth_mask); +/* starting value of gen bit - it toggles at wrap time */ + qm_port->gen_bit =1 ; + + qm_port->use_rsvd_token_scheme = use_rsvd_token_scheme; + qm_port->cq_rsvd_token_deficit = rsvd_tokens; + qm_port->int_armed = false; + +/* Save off for later use in info and lookup APIs. */ + qm_port->qid_mappings = &dlb->qm_ldb_to_ev_queue_id[0 ]; + +/* When using the reserved token scheme, token_pop_thresh is +* initially 2 * dequeue_depth. Once the tokens are reserved, +* the enqueue code re-assigns it to dequeue_depth. +*/ + qm_port->dequeue_depth = dequeue_depth; + qm_port->token_pop_thresh = cq_depth; + +/* When the deferred scheduling vdev arg is selected, use deferred pop +* for all single-entry CQs. +*/ +if (cfg.cq_depth ==1 || (cfg.cq_depth ==2 && use_rsvd_token_scheme)) { +if (dlb->defer_sched) + qm_port->token_pop_mode =DEFERRED_POP ; + } + +/* The default enqueue functions do not include delayed-pop support for +* performance reasons. +*/ +if (qm_port->token_pop_mode ==DELAYED_POP ) { + dlb->event_dev->enqueue = dlb_event_enqueue_delayed; + dlb->event_dev->enqueue_burst = + dlb_event_enqueue_burst_delayed; + dlb->event_dev->enqueue_new_burst = + dlb_event_enqueue_new_burst_delayed; + dlb->event_dev->enqueue_forward_burst = + dlb_event_enqueue_forward_burst_delayed; + } + + qm_port->owed_tokens =0 ; + qm_port->issued_releases =0 ; + +/* Save config message too. */ +rte_memcpy (&qm_port->cfg.ldb, &cfg,sizeof (cfg)); + +/* update state */ + qm_port->state =PORT_STARTED ;/* enabled at create time */ + qm_port->config_state =DLB_CONFIGURED ; + + qm_port->dir_credits = cfg.dir_credit_high_watermark; + qm_port->ldb_credits = cfg.ldb_credit_high_watermark; + +DLB_LOG_DBG ("dlb: created ldb port %d, depth = %d, ldb credits=%d, dir credits=%d\n" , + qm_port_id, + cq_depth, + qm_port->ldb_credits, + qm_port->dir_credits); + +rte_spinlock_unlock (&handle->resource_lock); + +return 0 ; + +error_exit: + +if (qm_port) { +dlb_free_qe_mem (qm_port); + qm_port->pp_mmio_base =0 ; + } + +rte_spinlock_unlock (&handle->resource_lock); + +DLB_LOG_ERR ("dlb: create ldb port failed!\n" ); + +return ret; +} + +static int +dlb_hw_create_dir_port (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +uint32_t dequeue_depth, +uint32_t cq_depth, +uint32_t enqueue_depth, +uint16_t rsvd_tokens, +bool use_rsvd_token_scheme) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_create_dir_port_args cfg = {0 }; +struct dlb_cmd_response response = {0 }; +int ret; +struct dlb_port *qm_port =NULL ; +char mz_name[RTE_MEMZONE_NAMESIZE ]; +uint32_t qm_port_id; + +if (dlb ==NULL || handle ==NULL ) +return -EINVAL ; + +if (cq_depth <DLB_MIN_DIR_CQ_DEPTH || + cq_depth >DLB_MAX_INPUT_QUEUE_DEPTH ) { +DLB_LOG_ERR ("dlb: invalid cq_depth, must be %d-%d\n" , +DLB_MIN_DIR_CQ_DEPTH ,DLB_MAX_INPUT_QUEUE_DEPTH ); +return -EINVAL ; + } + +rte_spinlock_lock (&handle->resource_lock); + +/* Directed queues are configured at link time. */ + cfg.queue_id =-1 ; + + cfg.response = (uintptr_t )&response; + +/* We round up to the next power of 2 if necessary */ + cfg.cq_depth =rte_align32pow2 (cq_depth); + cfg.cq_depth_threshold = rsvd_tokens; + +/* User controls the LDB high watermark via enqueue depth. The DIR high +* watermark is equal, unless the directed credit pool is too small. +*/ + cfg.ldb_credit_high_watermark = enqueue_depth; + +/* Don't use enqueue_depth if it would require more directed credits +* than are available. +*/ + cfg.dir_credit_high_watermark = +RTE_MIN (enqueue_depth, + handle->cfg.num_dir_credits / dlb->num_ports); + + cfg.ldb_credit_quantum = cfg.ldb_credit_high_watermark /2 ; + cfg.ldb_credit_low_watermark =RTE_MIN (16 , cfg.ldb_credit_quantum); + + cfg.dir_credit_quantum = cfg.dir_credit_high_watermark /2 ; + cfg.dir_credit_low_watermark =RTE_MIN (16 , cfg.dir_credit_quantum); + +/* Per QM values */ + + cfg.ldb_credit_pool_id = handle->cfg.ldb_credit_pool_id; + cfg.dir_credit_pool_id = handle->cfg.dir_credit_pool_id; + + ret =dlb_iface_dir_port_create (handle, &cfg, dlb->poll_mode); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: dlb_dir_port_create error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); + goto error_exit; + } + + qm_port_id = response.id; + +DLB_LOG_DBG ("dlb: ev_port %d uses qm DIR port %d <<<<<\n" , + ev_port->id, qm_port_id); + + qm_port = &ev_port->qm_port; + qm_port->ev_port = ev_port;/* back ptr */ + qm_port->dlb = dlb;/* back ptr */ + +/* +* Init local qe struct(s). +* Note: MOVDIR64 requires the enqueue QE to be aligned +*/ + +snprintf (mz_name,sizeof (mz_name),"%s_dir_port%d" , + handle->device_name, + ev_port->id); + + ret =dlb_init_qe_mem (qm_port, mz_name); + +if (ret <0 ) { +DLB_LOG_ERR ("dlb: init_qe_mem failed, ret=%d\n" , ret); + goto error_exit; + } + + qm_port->pp_mmio_base =DLB_DIR_PP_BASE +PAGE_SIZE * qm_port_id; + qm_port->id = qm_port_id; + +/* The credit window is one high water mark of QEs */ + qm_port->ldb_pushcount_at_credit_expiry =0 ; + qm_port->cached_ldb_credits = cfg.ldb_credit_high_watermark; +/* The credit window is one high water mark of QEs */ + qm_port->dir_pushcount_at_credit_expiry =0 ; + qm_port->cached_dir_credits = cfg.dir_credit_high_watermark; + qm_port->cq_depth = cfg.cq_depth; + qm_port->cq_idx =0 ; + qm_port->cq_idx_unmasked =0 ; +if (dlb->poll_mode ==DLB_CQ_POLL_MODE_SPARSE ) + qm_port->cq_depth_mask = (cfg.cq_depth *4 ) -1 ; +else + qm_port->cq_depth_mask = cfg.cq_depth -1 ; + + qm_port->gen_bit_shift =__builtin_popcount (qm_port->cq_depth_mask); +/* starting value of gen bit - it toggles at wrap time */ + qm_port->gen_bit =1 ; + + qm_port->use_rsvd_token_scheme = use_rsvd_token_scheme; + qm_port->cq_rsvd_token_deficit = rsvd_tokens; + qm_port->int_armed = false; + +/* Save off for later use in info and lookup APIs. */ + qm_port->qid_mappings = &dlb->qm_dir_to_ev_queue_id[0 ]; + + qm_port->dequeue_depth = dequeue_depth; + +/* Directed ports are auto-pop, by default. */ + qm_port->token_pop_mode =AUTO_POP ; + qm_port->owed_tokens =0 ; + qm_port->issued_releases =0 ; + +/* Save config message too. */ +rte_memcpy (&qm_port->cfg.dir, &cfg,sizeof (cfg)); + +/* update state */ + qm_port->state =PORT_STARTED ;/* enabled at create time */ + qm_port->config_state =DLB_CONFIGURED ; + + qm_port->dir_credits = cfg.dir_credit_high_watermark; + qm_port->ldb_credits = cfg.ldb_credit_high_watermark; + +DLB_LOG_DBG ("dlb: created dir port %d, depth = %d cr=%d,%d\n" , + qm_port_id, + cq_depth, + cfg.dir_credit_high_watermark, + cfg.ldb_credit_high_watermark); + +rte_spinlock_unlock (&handle->resource_lock); + +return 0 ; + +error_exit: + +if (qm_port) { + qm_port->pp_mmio_base =0 ; +dlb_free_qe_mem (qm_port); + } + +rte_spinlock_unlock (&handle->resource_lock); + +DLB_LOG_ERR ("dlb: create dir port failed!\n" ); + +return ret; +} + static int32_t dlb_hw_create_ldb_queue(struct dlb_eventdev *dlb, struct dlb_queue *queue,
@@ -282,6 +794,15 @@ dlb_hw_create_ldb_queue(struct dlb_eventdev *dlb,return qm_qid; } +static inline void +dlb_hw_do_enqueue (struct dlb_port *qm_port, +struct process_local_port_data *port_data) +{ +DLB_LOG_DBG ("dlb: Flushing QE(s) to DLB\n" ); + +dlb_pp_write (qm_port->qe4, port_data); +} +/* VDEV-only notes: * This function first unmaps all memory mappings and closes the * domain's file descriptor, which causes the driver to reset the
@@ -558,6 +1079,598 @@ dlb_eventdev_info_get(struct rte_eventdev *dev, *dev_info = evdev_dlb_default_info; } +static inline int +dlb_check_enqueue_sw_credits (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port) +{ +uint32_t sw_inflights =rte_atomic32_read (&dlb->inflights); +const int num =1 ; + +if (unlikely (ev_port->inflight_max < sw_inflights)) { +DLB_INC_STAT (ev_port->stats.traffic.tx_nospc_inflight_max,1 ); + rte_errno = -ENOSPC ; +return 1 ; + } + +if (ev_port->inflight_credits < num) { +/* check if event enqueue brings ev_port over max threshold */ +uint32_t credit_update_quanta = ev_port->credit_update_quanta; + +if (sw_inflights + credit_update_quanta > + dlb->new_event_limit) { +DLB_INC_STAT ( + ev_port->stats.traffic.tx_nospc_new_event_limit, +1 ); + rte_errno = -ENOSPC ; +return 1 ; + } + +rte_atomic32_add (&dlb->inflights, credit_update_quanta); + ev_port->inflight_credits += (credit_update_quanta); + +if (ev_port->inflight_credits < num) { +DLB_INC_STAT ( + ev_port->stats.traffic.tx_nospc_inflight_credits, +1 ); + rte_errno = -ENOSPC ; +return 1 ; + } + } + +return 0 ; +} + +static inline void +dlb_replenish_sw_credits (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port) +{ +uint16_t quanta = ev_port->credit_update_quanta; + +if (ev_port->inflight_credits >= quanta *2 ) { +rte_atomic32_sub (&dlb->inflights, quanta); + ev_port->inflight_credits -= quanta; + } +} + +static inline int +dlb_check_enqueue_hw_ldb_credits (struct dlb_port *qm_port, +struct process_local_port_data *port_data) +{ +if (unlikely (qm_port->cached_ldb_credits ==0 )) { +uint16_t pc; + + pc =dlb_read_pc (port_data, true); + + qm_port->cached_ldb_credits = pc - + qm_port->ldb_pushcount_at_credit_expiry; +if (unlikely (qm_port->cached_ldb_credits ==0 )) { +DLB_INC_STAT ( + qm_port->ev_port->stats.traffic.tx_nospc_ldb_hw_credits, +1 ); + +DLB_LOG_DBG ("ldb credits exhausted\n" ); +return 1 ; + } + qm_port->ldb_pushcount_at_credit_expiry += + qm_port->cached_ldb_credits; + } + +return 0 ; +} + +static inline int +dlb_check_enqueue_hw_dir_credits (struct dlb_port *qm_port, +struct process_local_port_data *port_data) +{ +if (unlikely (qm_port->cached_dir_credits ==0 )) { +uint16_t pc; + + pc =dlb_read_pc (port_data, false); + + qm_port->cached_dir_credits = pc - + qm_port->dir_pushcount_at_credit_expiry; + +if (unlikely (qm_port->cached_dir_credits ==0 )) { +DLB_INC_STAT ( + qm_port->ev_port->stats.traffic.tx_nospc_dir_hw_credits, +1 ); + +DLB_LOG_DBG ("dir credits exhausted\n" ); +return 1 ; + } + qm_port->dir_pushcount_at_credit_expiry += + qm_port->cached_dir_credits; + } + +return 0 ; +} + +static inline int +dlb_event_enqueue_prep (struct dlb_eventdev_port *ev_port, +struct dlb_port *qm_port, +const struct rte_event ev[], +struct process_local_port_data *port_data, +uint8_t *sched_type, +uint8_t *queue_id) +{ +struct dlb_eventdev *dlb = ev_port->dlb; +struct dlb_eventdev_queue *ev_queue; +uint16_t *cached_credits =NULL ; +struct dlb_queue *qm_queue; + + ev_queue = &dlb->ev_queues[ev->queue_id]; + qm_queue = &ev_queue->qm_queue; + *queue_id = qm_queue->id; + +/* Ignore sched_type and hardware credits on release events */ +if (ev->op ==RTE_EVENT_OP_RELEASE ) + goto op_check; + +if (!qm_queue->is_directed) { +/* Load balanced destination queue */ + +if (dlb_check_enqueue_hw_ldb_credits (qm_port, port_data)) { + rte_errno = -ENOSPC ; +return 1 ; + } + cached_credits = &qm_port->cached_ldb_credits; + +switch (ev->sched_type) { +case RTE_SCHED_TYPE_ORDERED : +DLB_LOG_DBG ("dlb: put_qe: RTE_SCHED_TYPE_ORDERED\n" ); +if (qm_queue->sched_type !=RTE_SCHED_TYPE_ORDERED ) { +DLB_LOG_ERR ("dlb: tried to send ordered event to unordered queue %d\n" , + *queue_id); + rte_errno = -EINVAL ; +return 1 ; + } + *sched_type =DLB_SCHED_ORDERED ; +break ; +case RTE_SCHED_TYPE_ATOMIC : +DLB_LOG_DBG ("dlb: put_qe: RTE_SCHED_TYPE_ATOMIC\n" ); + *sched_type =DLB_SCHED_ATOMIC ; +break ; +case RTE_SCHED_TYPE_PARALLEL : +DLB_LOG_DBG ("dlb: put_qe: RTE_SCHED_TYPE_PARALLEL\n" ); +if (qm_queue->sched_type ==RTE_SCHED_TYPE_ORDERED ) + *sched_type =DLB_SCHED_ORDERED ; +else + *sched_type =DLB_SCHED_UNORDERED ; +break ; +default : +DLB_LOG_ERR ("Unsupported LDB sched type in put_qe\n" ); +DLB_INC_STAT (ev_port->stats.tx_invalid,1 ); + rte_errno = -EINVAL ; +return 1 ; + } + }else { +/* Directed destination queue */ + +if (dlb_check_enqueue_hw_dir_credits (qm_port, port_data)) { + rte_errno = -ENOSPC ; +return 1 ; + } + cached_credits = &qm_port->cached_dir_credits; + +DLB_LOG_DBG ("dlb: put_qe: RTE_SCHED_TYPE_DIRECTED\n" ); + + *sched_type =DLB_SCHED_DIRECTED ; + } + +op_check: +switch (ev->op) { +case RTE_EVENT_OP_NEW : +/* Check that a sw credit is available */ +if (dlb_check_enqueue_sw_credits (dlb, ev_port)) { + rte_errno = -ENOSPC ; +return 1 ; + } + ev_port->inflight_credits--; + (*cached_credits)--; +break ; +case RTE_EVENT_OP_FORWARD : +/* Check for outstanding_releases underflow. If this occurs, +* the application is not using the EVENT_OPs correctly; for +* example, forwarding or releasing events that were not +* dequeued. +*/ +RTE_ASSERT (ev_port->outstanding_releases >0 ); + ev_port->outstanding_releases--; + qm_port->issued_releases++; + (*cached_credits)--; +break ; +case RTE_EVENT_OP_RELEASE : + ev_port->inflight_credits++; +/* Check for outstanding_releases underflow. If this occurs, +* the application is not using the EVENT_OPs correctly; for +* example, forwarding or releasing events that were not +* dequeued. +*/ +RTE_ASSERT (ev_port->outstanding_releases >0 ); + ev_port->outstanding_releases--; + qm_port->issued_releases++; +/* Replenish s/w credits if enough are cached */ +dlb_replenish_sw_credits (dlb, ev_port); +break ; + } + +DLB_INC_STAT (ev_port->stats.tx_op_cnt[ev->op],1 ); +DLB_INC_STAT (ev_port->stats.traffic.tx_ok,1 ); + +#ifndef RTE_LIBRTE_PMD_DLB_QUELL_STATS +if (ev->op !=RTE_EVENT_OP_RELEASE ) { +DLB_INC_STAT (ev_port->stats.enq_ok[ev->queue_id],1 ); +DLB_INC_STAT (ev_port->stats.tx_sched_cnt[*sched_type],1 ); + } +#endif + +return 0 ; +} + +static uint8_t cmd_byte_map[NUM_DLB_PORT_TYPES ][DLB_NUM_HW_SCHED_TYPES ] = { + { +/* Load-balanced cmd bytes */ + [RTE_EVENT_OP_NEW ] =DLB_NEW_CMD_BYTE , + [RTE_EVENT_OP_FORWARD ] =DLB_FWD_CMD_BYTE , + [RTE_EVENT_OP_RELEASE ] =DLB_COMP_CMD_BYTE , + }, + { +/* Directed cmd bytes */ + [RTE_EVENT_OP_NEW ] =DLB_NEW_CMD_BYTE , + [RTE_EVENT_OP_FORWARD ] =DLB_NEW_CMD_BYTE , + [RTE_EVENT_OP_RELEASE ] =DLB_NOOP_CMD_BYTE , + }, +}; + +static inline void +dlb_event_build_hcws (struct dlb_port *qm_port, +const struct rte_event ev[], +int num, +uint8_t *sched_type, +uint8_t *queue_id) +{ +struct dlb_enqueue_qe *qe; +uint16_t sched_word[4 ]; +__m128i sse_qe[2 ]; +int i; + + qe = qm_port->qe4; + + sse_qe[0 ] =_mm_setzero_si128 (); + sse_qe[1 ] =_mm_setzero_si128 (); + +switch (num) { +case 4 : +/* Construct the metadata portion of two HCWs in one 128b SSE +* register. HCW metadata is constructed in the SSE registers +* like so: +* sse_qe[0][63:0]: qe[0]'s metadata +* sse_qe[0][127:64]: qe[1]'s metadata +* sse_qe[1][63:0]: qe[2]'s metadata +* sse_qe[1][127:64]: qe[3]'s metadata +*/ + +/* Convert the event operation into a command byte and store it +* in the metadata: +* sse_qe[0][63:56] = cmd_byte_map[is_directed][ev[0].op] +* sse_qe[0][127:120] = cmd_byte_map[is_directed][ev[1].op] +* sse_qe[1][63:56] = cmd_byte_map[is_directed][ev[2].op] +* sse_qe[1][127:120] = cmd_byte_map[is_directed][ev[3].op] +*/ +#define DLB_QE_CMD_BYTE 7 + sse_qe[0 ] =_mm_insert_epi8 (sse_qe[0 ], + cmd_byte_map[qm_port->is_directed][ev[0 ].op], +DLB_QE_CMD_BYTE ); + sse_qe[0 ] =_mm_insert_epi8 (sse_qe[0 ], + cmd_byte_map[qm_port->is_directed][ev[1 ].op], +DLB_QE_CMD_BYTE +8 ); + sse_qe[1 ] =_mm_insert_epi8 (sse_qe[1 ], + cmd_byte_map[qm_port->is_directed][ev[2 ].op], +DLB_QE_CMD_BYTE ); + sse_qe[1 ] =_mm_insert_epi8 (sse_qe[1 ], + cmd_byte_map[qm_port->is_directed][ev[3 ].op], +DLB_QE_CMD_BYTE +8 ); + +/* Store priority, scheduling type, and queue ID in the sched +* word array because these values are re-used when the +* destination is a directed queue. +*/ + sched_word[0 ] =EV_TO_DLB_PRIO (ev[0 ].priority) <<10 | + sched_type[0 ] <<8 | + queue_id[0 ]; + sched_word[1 ] =EV_TO_DLB_PRIO (ev[1 ].priority) <<10 | + sched_type[1 ] <<8 | + queue_id[1 ]; + sched_word[2 ] =EV_TO_DLB_PRIO (ev[2 ].priority) <<10 | + sched_type[2 ] <<8 | + queue_id[2 ]; + sched_word[3 ] =EV_TO_DLB_PRIO (ev[3 ].priority) <<10 | + sched_type[3 ] <<8 | + queue_id[3 ]; + +/* Store the event priority, scheduling type, and queue ID in +* the metadata: +* sse_qe[0][31:16] = sched_word[0] +* sse_qe[0][95:80] = sched_word[1] +* sse_qe[1][31:16] = sched_word[2] +* sse_qe[1][95:80] = sched_word[3] +*/ +#define DLB_QE_QID_SCHED_WORD 1 + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + sched_word[0 ], +DLB_QE_QID_SCHED_WORD ); + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + sched_word[1 ], +DLB_QE_QID_SCHED_WORD +4 ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + sched_word[2 ], +DLB_QE_QID_SCHED_WORD ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + sched_word[3 ], +DLB_QE_QID_SCHED_WORD +4 ); + +/* If the destination is a load-balanced queue, store the lock +* ID. If it is a directed queue, DLB places this field in +* bytes 10-11 of the received QE, so we format it accordingly: +* sse_qe[0][47:32] = dir queue ? sched_word[0] : flow_id[0] +* sse_qe[0][111:96] = dir queue ? sched_word[1] : flow_id[1] +* sse_qe[1][47:32] = dir queue ? sched_word[2] : flow_id[2] +* sse_qe[1][111:96] = dir queue ? sched_word[3] : flow_id[3] +*/ +#define DLB_QE_LOCK_ID_WORD 2 + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + (sched_type[0 ] ==DLB_SCHED_DIRECTED ) ? + sched_word[0 ] : ev[0 ].flow_id, +DLB_QE_LOCK_ID_WORD ); + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + (sched_type[1 ] ==DLB_SCHED_DIRECTED ) ? + sched_word[1 ] : ev[1 ].flow_id, +DLB_QE_LOCK_ID_WORD +4 ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + (sched_type[2 ] ==DLB_SCHED_DIRECTED ) ? + sched_word[2 ] : ev[2 ].flow_id, +DLB_QE_LOCK_ID_WORD ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + (sched_type[3 ] ==DLB_SCHED_DIRECTED ) ? + sched_word[3 ] : ev[3 ].flow_id, +DLB_QE_LOCK_ID_WORD +4 ); + +/* Store the event type and sub event type in the metadata: +* sse_qe[0][15:0] = flow_id[0] +* sse_qe[0][79:64] = flow_id[1] +* sse_qe[1][15:0] = flow_id[2] +* sse_qe[1][79:64] = flow_id[3] +*/ +#define DLB_QE_EV_TYPE_WORD 0 + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + ev[0 ].sub_event_type <<8 | + ev[0 ].event_type, +DLB_QE_EV_TYPE_WORD ); + sse_qe[0 ] =_mm_insert_epi16 (sse_qe[0 ], + ev[1 ].sub_event_type <<8 | + ev[1 ].event_type, +DLB_QE_EV_TYPE_WORD +4 ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + ev[2 ].sub_event_type <<8 | + ev[2 ].event_type, +DLB_QE_EV_TYPE_WORD ); + sse_qe[1 ] =_mm_insert_epi16 (sse_qe[1 ], + ev[3 ].sub_event_type <<8 | + ev[3 ].event_type, +DLB_QE_EV_TYPE_WORD +4 ); + +/* Store the metadata to memory (use the double-precision +* _mm_storeh_pd because there is no integer function for +* storing the upper 64b): +* qe[0] metadata = sse_qe[0][63:0] +* qe[1] metadata = sse_qe[0][127:64] +* qe[2] metadata = sse_qe[1][63:0] +* qe[3] metadata = sse_qe[1][127:64] +*/ +_mm_storel_epi64 ((__m128i *)&qe[0 ].u.opaque_data, sse_qe[0 ]); +_mm_storeh_pd ((double *)&qe[1 ].u.opaque_data, + (__m128d ) sse_qe[0 ]); +_mm_storel_epi64 ((__m128i *)&qe[2 ].u.opaque_data, sse_qe[1 ]); +_mm_storeh_pd ((double *)&qe[3 ].u.opaque_data, + (__m128d ) sse_qe[1 ]); + + qe[0 ].data = ev[0 ].u64; + qe[1 ].data = ev[1 ].u64; + qe[2 ].data = ev[2 ].u64; + qe[3 ].data = ev[3 ].u64; + +break ; +case 3 : +case 2 : +case 1 : +/* At least one QE will be valid, so only zero out three */ + qe[1 ].cmd_byte =0 ; + qe[2 ].cmd_byte =0 ; + qe[3 ].cmd_byte =0 ; + +for (i =0 ; i < num; i++) { + qe[i].cmd_byte = + cmd_byte_map[qm_port->is_directed][ev[i].op]; + qe[i].sched_type = sched_type[i]; + qe[i].data = ev[i].u64; + qe[i].qid = queue_id[i]; + qe[i].priority =EV_TO_DLB_PRIO (ev[i].priority); + qe[i].lock_id = ev[i].flow_id; +if (sched_type[i] ==DLB_SCHED_DIRECTED ) { +struct dlb_msg_info *info = + (struct dlb_msg_info *)&qe[i].lock_id; + + info->qid = queue_id[i]; + info->sched_type =DLB_SCHED_DIRECTED ; + info->priority = qe[i].priority; + } + qe[i].u.event_type.major = ev[i].event_type; + qe[i].u.event_type.sub = ev[i].sub_event_type; + } +break ; + } +} + +static inline void +dlb_construct_token_pop_qe (struct dlb_port *qm_port,int idx) +{ +struct dlb_cq_pop_qe *qe = (void *)qm_port->qe4; +int num = qm_port->owed_tokens; + +if (qm_port->use_rsvd_token_scheme) { +/* Check if there's a deficit of reserved tokens, and return +* early if there are no (unreserved) tokens to consume. +*/ +if (num <= qm_port->cq_rsvd_token_deficit) { + qm_port->cq_rsvd_token_deficit -= num; + qm_port->owed_tokens =0 ; +return ; + } + num -= qm_port->cq_rsvd_token_deficit; + qm_port->cq_rsvd_token_deficit =0 ; + } + + qe[idx].cmd_byte =DLB_POP_CMD_BYTE ; + qe[idx].tokens = num -1 ; + + qm_port->owed_tokens =0 ; +} + +static inline uint16_t +__dlb_event_enqueue_burst (void *event_port, +const struct rte_event events[], +uint16_t num, +bool use_delayed) +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_port *qm_port = &ev_port->qm_port; +struct process_local_port_data *port_data; +int i, cnt; + +RTE_ASSERT (ev_port->enq_configured); +RTE_ASSERT (events !=NULL ); + + rte_errno =0 ; + cnt =0 ; + + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +if (!port_data->mmaped) +dlb_iface_port_mmap (qm_port); + +/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that +* application writes complete before enqueueing the release HCW. +*/ +rte_wmb (); + +for (i =0 ; i < num; i +=DLB_NUM_QES_PER_CACHE_LINE ) { +uint8_t sched_types[DLB_NUM_QES_PER_CACHE_LINE ]; +uint8_t queue_ids[DLB_NUM_QES_PER_CACHE_LINE ]; +int j =0 ; + +for (; j <DLB_NUM_QES_PER_CACHE_LINE && (i + j) < num; j++) { +const struct rte_event *ev = &events[i + j]; + +if (dlb_event_enqueue_prep (ev_port, qm_port, ev, + port_data, &sched_types[j], + &queue_ids[j])) +break ; + } + +if (j ==0 ) +break ; + +dlb_event_build_hcws (qm_port, &events[i], j, + sched_types, queue_ids); + +/* The delayed-pop code causes an unnecessary performance +* penalty when it is not in use. The use_delayed argument +* allows the compiler to create a version of this function +* with these checks factored out that the PMD can call +* when delayed-pop is not in use. +*/ +if (use_delayed && + qm_port->token_pop_mode ==DELAYED_POP && j <4 && + qm_port->issued_releases >= qm_port->token_pop_thresh -1 ) { + +dlb_construct_token_pop_qe (qm_port, j); + +/* Reset the releases counter for the next QE batch */ + qm_port->issued_releases -= qm_port->token_pop_thresh; + +/* When using delayed token pop mode, the initial token +* threshold is the full CQ depth. After the first +* token pop, we need to reset it to the dequeue_depth. +*/ + qm_port->token_pop_thresh = qm_port->dequeue_depth; + } + +dlb_hw_do_enqueue (qm_port, port_data); + + cnt += j; + +if (j <DLB_NUM_QES_PER_CACHE_LINE ) +break ; + } + +if (use_delayed && qm_port->token_pop_mode ==DELAYED_POP && + qm_port->issued_releases >= qm_port->token_pop_thresh -1 ) { +dlb_consume_qe_immediate (qm_port, qm_port->owed_tokens); + qm_port->issued_releases -= qm_port->token_pop_thresh; + qm_port->token_pop_thresh = qm_port->dequeue_depth; + } + +RTE_ASSERT (!((cnt ==0 && rte_errno != -ENOSPC ))); + +return cnt; +} + +static inline uint16_t +dlb_event_enqueue_burst (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, false); +} + +static inline uint16_t +dlb_event_enqueue_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, true); +} + +static inline uint16_t +dlb_event_enqueue (void *event_port, +const struct rte_event events[]) +{ +return __dlb_event_enqueue_burst (event_port, events,1 , false); +} + +static inline uint16_t +dlb_event_enqueue_delayed (void *event_port, +const struct rte_event events[]) +{ +return __dlb_event_enqueue_burst (event_port, events,1 , true); +} + +static uint16_t +dlb_event_enqueue_new_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, true); +} + +static uint16_t +dlb_event_enqueue_forward_burst_delayed (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, true); +} +/* Note: 1 QM instance per QM device, QM instance/device == event device */ static int dlb_eventdev_configure (const struct rte_eventdev *dev)
@@ -935,6 +2048,133 @@ dlb_eventdev_queue_setup(struct rte_eventdev *dev,return ret; } +static void +dlb_port_link_teardown (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port) +{ +struct dlb_eventdev_queue *ev_queue; +int i; + +for (i =0 ; i <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; i++) { +if (!ev_port->link[i].valid) +continue ; + + ev_queue = &dlb->ev_queues[ev_port->link[i].queue_id]; + + ev_port->link[i].valid = false; + ev_port->num_links--; + ev_queue->num_links--; + } +} + +static int +dlb_eventdev_port_setup (struct rte_eventdev *dev, +uint8_t ev_port_id, +const struct rte_event_port_conf *port_conf) +{ +struct dlb_eventdev *dlb; +struct dlb_eventdev_port *ev_port; +bool use_rsvd_token_scheme; +uint32_t adj_cq_depth; +uint16_t rsvd_tokens; +int ret; + +if (dev ==NULL || port_conf ==NULL ) { +DLB_LOG_ERR ("Null parameter\n" ); +return -EINVAL ; + } + + dlb =dlb_pmd_priv (dev); + +if (ev_port_id >=DLB_MAX_NUM_PORTS ) +return -EINVAL ; + +if (port_conf->dequeue_depth > + evdev_dlb_default_info.max_event_port_dequeue_depth || + port_conf->enqueue_depth > + evdev_dlb_default_info.max_event_port_enqueue_depth) +return -EINVAL ; + + ev_port = &dlb->ev_ports[ev_port_id]; +/* configured? */ +if (ev_port->setup_done) { +DLB_LOG_ERR ("evport %d is already configured\n" , ev_port_id); +return -EINVAL ; + } + +/* The reserved token interrupt arming scheme requires that one or more +* CQ tokens be reserved by the PMD. This limits the amount of CQ space +* usable by the DLB, so in order to give an *effective* CQ depth equal +* to the user-requested value, we double CQ depth and reserve half of +* its tokens. If the user requests the max CQ depth (256) then we +* cannot double it, so we reserve one token and give an effective +* depth of 255 entries. +*/ + use_rsvd_token_scheme = true; + rsvd_tokens =1 ; + adj_cq_depth = port_conf->dequeue_depth; + +if (use_rsvd_token_scheme && adj_cq_depth <256 ) { + rsvd_tokens = adj_cq_depth; + adj_cq_depth *=2 ; + } + + ev_port->qm_port.is_directed = port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_SINGLE_LINK ; + +if (!ev_port->qm_port.is_directed) { + ret =dlb_hw_create_ldb_port (dlb, + ev_port, + port_conf->dequeue_depth, + adj_cq_depth, + port_conf->enqueue_depth, + rsvd_tokens, + use_rsvd_token_scheme); +if (ret <0 ) { +DLB_LOG_ERR ("Failed to create the lB port ve portId=%d\n" , + ev_port_id); +return ret; + } + }else { + ret =dlb_hw_create_dir_port (dlb, + ev_port, + port_conf->dequeue_depth, + adj_cq_depth, + port_conf->enqueue_depth, + rsvd_tokens, + use_rsvd_token_scheme); +if (ret <0 ) { +DLB_LOG_ERR ("Failed to create the DIR port\n" ); +return ret; + } + } + +/* Save off port config for reconfig */ + dlb->ev_ports[ev_port_id].conf = *port_conf; + + dlb->ev_ports[ev_port_id].id = ev_port_id; + dlb->ev_ports[ev_port_id].enq_configured = true; + dlb->ev_ports[ev_port_id].setup_done = true; + dlb->ev_ports[ev_port_id].inflight_max = + port_conf->new_event_threshold; + dlb->ev_ports[ev_port_id].implicit_release = + !(port_conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); + dlb->ev_ports[ev_port_id].outstanding_releases =0 ; + dlb->ev_ports[ev_port_id].inflight_credits =0 ; + dlb->ev_ports[ev_port_id].credit_update_quanta = +RTE_LIBRTE_PMD_DLB_SW_CREDIT_QUANTA ; + dlb->ev_ports[ev_port_id].dlb = dlb;/* reverse link */ + +/* Tear down pre-existing port->queue links */ +if (dlb->run_state ==DLB_RUN_STATE_STOPPED ) +dlb_port_link_teardown (dlb, &dlb->ev_ports[ev_port_id]); + + dev->data->ports[ev_port_id] = &dlb->ev_ports[ev_port_id]; + +return 0 ; +} + static int set_dev_id(const char *key __rte_unused, const char *value,
@@ -1014,6 +2254,7 @@ dlb_entry_points_init(struct rte_eventdev *dev) .queue_def_conf = dlb_eventdev_queue_default_conf_get, .queue_setup = dlb_eventdev_queue_setup, .port_def_conf = dlb_eventdev_port_default_conf_get, + .port_setup = dlb_eventdev_port_setup, };/* Expose PMD's eventdev interface */
Signature · 1 line
2.13.6
PATCH 20/27 event/dlb: add port_link
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 303 insertions(+)
drivers/event/dlb/dlb.c +303 -0
@@ -794,6 +794,29 @@ dlb_hw_create_ldb_queue(struct dlb_eventdev *dlb,return qm_qid; } +static int32_t +dlb_hw_create_dir_queue (struct dlb_eventdev *dlb,int32_t qm_port_id) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_create_dir_queue_args cfg; +struct dlb_cmd_response response; +int32_t ret; + + cfg.response = (uintptr_t )&response; + +/* The directed port is always configured before its queue */ + cfg.port_id = qm_port_id; + + ret =dlb_iface_dir_queue_create (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: create DIR event queue error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return -EINVAL ; + } + +return response.id; +} +static inline void dlb_hw_do_enqueue (struct dlb_port *qm_port,struct process_local_port_data *port_data)
@@ -803,6 +826,42 @@ dlb_hw_do_enqueue(struct dlb_port *qm_port,dlb_pp_write (qm_port->qe4, port_data); } +static int16_t +dlb_hw_map_ldb_qid_to_port (struct dlb_hw_dev *handle, +uint32_t qm_port_id, +uint16_t qm_qid, +uint8_t priority) +{ +struct dlb_map_qid_args cfg; +struct dlb_cmd_response response; +int32_t ret; + +if (handle ==NULL ) +return -EINVAL ; + +/* Build message */ + cfg.response = (uintptr_t )&response; + cfg.port_id = qm_port_id; + cfg.qid = qm_qid; + cfg.priority =EV_TO_DLB_PRIO (priority); + + ret =dlb_iface_map_qid (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: map qid error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +DLB_LOG_ERR ("dlb: device_id=%d grp=%d, qm_port=%d, qm_qid=%d prio=%d\n" , + handle->device_id, + handle->domain_id, cfg.port_id, + cfg.qid, + cfg.priority); + }else { +DLB_LOG_DBG ("dlb: mapped queue %d to qm_port %d\n" , + qm_qid, qm_port_id); + } + +return ret; +} +/* VDEV-only notes: * This function first unmaps all memory mappings and closes the * domain's file descriptor, which causes the driver to reset the
@@ -1785,6 +1844,249 @@ dlb_eventdev_configure(const struct rte_eventdev *dev)return 0 ; } +static int +dlb_event_queue_join_ldb (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +struct dlb_eventdev_queue *ev_queue, +uint8_t priority) +{ +int first_avail =-1 ; +int ret, i; + +for (i =0 ; i <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; i++) { +if (ev_port->link[i].valid) { +if (ev_port->link[i].queue_id == ev_queue->id && + ev_port->link[i].priority == priority) { +if (ev_port->link[i].mapped) +return 0 ;/* already mapped */ + first_avail = i; + } + }else { +if (first_avail ==-1 ) + first_avail = i; + } + } +if (first_avail ==-1 ) { +DLB_LOG_ERR ("dlb: qm_port %d has no available QID slots.\n" , + ev_port->qm_port.id); +return -EINVAL ; + } + + ret =dlb_hw_map_ldb_qid_to_port (&dlb->qm_instance, + ev_port->qm_port.id, + ev_queue->qm_queue.id, + priority); + +if (!ret) + ev_port->link[first_avail].mapped = true; + +return ret; +} + +static int +dlb_eventdev_dir_queue_setup (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *ev_queue, +struct dlb_eventdev_port *ev_port) +{ +int32_t qm_qid; + + qm_qid =dlb_hw_create_dir_queue (dlb, ev_port->qm_port.id); + +if (qm_qid <0 ) { +DLB_LOG_ERR ("Failed to create the DIR queue\n" ); +return qm_qid; + } + + dlb->qm_dir_to_ev_queue_id[qm_qid] = ev_queue->id; + + ev_queue->qm_queue.id = qm_qid; + +return 0 ; +} + +static int +dlb_do_port_link (struct rte_eventdev *dev, +struct dlb_eventdev_queue *ev_queue, +struct dlb_eventdev_port *ev_port, +uint8_t prio) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +int err; + +/* Don't link until start time. */ +if (dlb->run_state ==DLB_RUN_STATE_STOPPED ) +return 0 ; + +if (ev_queue->qm_queue.is_directed) + err =dlb_eventdev_dir_queue_setup (dlb, ev_queue, ev_port); +else + err =dlb_event_queue_join_ldb (dlb, ev_port, ev_queue, prio); + +if (err) { +DLB_LOG_ERR ("port link failure for %s ev_q %d, ev_port %d\n" , + ev_queue->qm_queue.is_directed ?"DIR" :"LDB" , + ev_queue->id, ev_port->id); + + rte_errno = err; +return -1 ; + } + +return 0 ; +} + +static int +dlb_validate_port_link (struct dlb_eventdev_port *ev_port, +uint8_t queue_id, +bool link_exists, +int index) +{ +struct dlb_eventdev *dlb = ev_port->dlb; +struct dlb_eventdev_queue *ev_queue; +bool port_is_dir, queue_is_dir; + +if (queue_id > dlb->num_queues) { + rte_errno = -EINVAL ; +return -1 ; + } + + ev_queue = &dlb->ev_queues[queue_id]; + +if (!ev_queue->setup_done && + ev_queue->qm_queue.config_state !=DLB_PREV_CONFIGURED ) { + rte_errno = -EINVAL ; +return -1 ; + } + + port_is_dir = ev_port->qm_port.is_directed; + queue_is_dir = ev_queue->qm_queue.is_directed; + +if (port_is_dir != queue_is_dir) { +DLB_LOG_ERR ("%s queue %u can't link to %s port %u\n" , + queue_is_dir ?"DIR" :"LDB" , ev_queue->id, + port_is_dir ?"DIR" :"LDB" , ev_port->id); + + rte_errno = -EINVAL ; +return -1 ; + } + +/* Check if there is space for the requested link */ +if (!link_exists && index ==-1 ) { +DLB_LOG_ERR ("no space for new link\n" ); + rte_errno = -ENOSPC ; +return -1 ; + } + +/* Check if the directed port is already linked */ +if (ev_port->qm_port.is_directed && ev_port->num_links >0 && + !link_exists) { +DLB_LOG_ERR ("Can't link DIR port %d to >1 queues\n" , + ev_port->id); + rte_errno = -EINVAL ; +return -1 ; + } + +/* Check if the directed queue is already linked */ +if (ev_queue->qm_queue.is_directed && ev_queue->num_links >0 && + !link_exists) { +DLB_LOG_ERR ("Can't link DIR queue %d to >1 ports\n" , + ev_queue->id); + rte_errno = -EINVAL ; +return -1 ; + } + +return 0 ; +} + +static int +dlb_eventdev_port_link (struct rte_eventdev *dev,void *event_port, +const uint8_t queues[],const uint8_t priorities[], +uint16_t nb_links) + +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_eventdev *dlb; +int i, j; + +RTE_SET_USED (dev); + +if (!ev_port) { +DLB_LOG_ERR ("dlb: evport not setup\n" ); + rte_errno = -EINVAL ; +return 0 ; + } + +if (!ev_port->setup_done && + ev_port->qm_port.config_state !=DLB_PREV_CONFIGURED ) { +DLB_LOG_ERR ("dlb: evport not setup\n" ); + rte_errno = -EINVAL ; +return 0 ; + } + +/* Note: rte_event_port_link() ensures the PMD won't receive a NULL +* queues pointer. +*/ +if (nb_links ==0 ) { +DLB_LOG_DBG ("dlb: nb_links is 0\n" ); +return 0 ;/* Ignore and return success */ + } + + dlb = ev_port->dlb; + +DLB_LOG_DBG ("Linking %u queues to %s port %d\n" , + nb_links, + ev_port->qm_port.is_directed ?"DIR" :"LDB" , + ev_port->id); + +for (i =0 ; i < nb_links; i++) { +struct dlb_eventdev_queue *ev_queue; +uint8_t queue_id, prio; +bool found = false; +int index =-1 ; + + queue_id = queues[i]; + prio = priorities[i]; + +/* Check if the link already exists. */ +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) +if (ev_port->link[j].valid) { +if (ev_port->link[j].queue_id == queue_id) { + found = true; + index = j; +break ; + } + }else { +if (index ==-1 ) + index = j; + } + +/* could not link */ +if (index ==-1 ) +break ; + +/* Check if already linked at the requested priority */ +if (found && ev_port->link[j].priority == prio) +continue ; + +if (dlb_validate_port_link (ev_port, queue_id, found, index)) +break ;/* return index of offending queue */ + + ev_queue = &dlb->ev_queues[queue_id]; + +if (dlb_do_port_link (dev, ev_queue, ev_port, prio)) +break ;/* return index of offending queue */ + + ev_queue->num_links++; + + ev_port->link[index].queue_id = queue_id; + ev_port->link[index].priority = prio; + ev_port->link[index].valid = true; +/* Entry already exists? If so, then must be prio change */ +if (!found) + ev_port->num_links++; + } +return i; +} + static void dlb_eventdev_port_default_conf_get(struct rte_eventdev *dev, uint8_t port_id,
@@ -2255,6 +2557,7 @@ dlb_entry_points_init(struct rte_eventdev *dev) .queue_setup = dlb_eventdev_queue_setup, .port_def_conf = dlb_eventdev_port_default_conf_get, .port_setup = dlb_eventdev_port_setup, + .port_link = dlb_eventdev_port_link, };/* Expose PMD's eventdev interface */
Signature · 1 line
2.13.6
PATCH 21/27 event/dlb: add queue_release and port_release
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
drivers/event/dlb/dlb.c +25 -0
@@ -2477,6 +2477,29 @@ dlb_eventdev_port_setup(struct rte_eventdev *dev,return 0 ; } +static void +dlb_eventdev_port_release (void *port) +{ +RTE_SET_USED (port); + +/* This function intentionally left blank. dlb does not support +* reconfiguring individual queues or ports -- the entire device +* must be reconfigured. +*/ +} + +static void +dlb_eventdev_queue_release (struct rte_eventdev *dev,uint8_t id) +{ +RTE_SET_USED (dev); +RTE_SET_USED (id); + +/* This function intentionally left blank. dlb does not support +* reconfiguring individual queues or ports -- the entire device +* must be reconfigured. +*/ +} + static int set_dev_id(const char *key __rte_unused, const char *value,
@@ -2555,8 +2578,10 @@ dlb_entry_points_init(struct rte_eventdev *dev) .dev_configure = dlb_eventdev_configure, .queue_def_conf = dlb_eventdev_queue_default_conf_get, .queue_setup = dlb_eventdev_queue_setup, + .queue_release = dlb_eventdev_queue_release, .port_def_conf = dlb_eventdev_port_default_conf_get, .port_setup = dlb_eventdev_port_setup, + .port_release = dlb_eventdev_port_release, .port_link = dlb_eventdev_port_link, };
Signature · 1 line
2.13.6
PATCH 22/27 event/dlb: add port_unlink and port_unlinks_in_progress
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 166 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+)
drivers/event/dlb/dlb.c +166 -0
@@ -862,6 +862,30 @@ dlb_hw_map_ldb_qid_to_port(struct dlb_hw_dev *handle,return ret; } +static int16_t +dlb_hw_unmap_ldb_qid_from_port (struct dlb_hw_dev *handle, +uint32_t qm_port_id, +uint16_t qm_qid) +{ +struct dlb_unmap_qid_args cfg; +struct dlb_cmd_response response; +int32_t ret; + +if (handle ==NULL ) +return -EINVAL ; + + cfg.response = (uintptr_t )&response; + cfg.port_id = qm_port_id; + cfg.qid = qm_qid; + + ret =dlb_iface_unmap_qid (handle, &cfg); +if (ret <0 ) +DLB_LOG_ERR ("dlb: unmap qid error, ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); + +return ret; +} +/* VDEV-only notes: * This function first unmaps all memory mappings and closes the * domain's file descriptor, which causes the driver to reset the
@@ -1905,6 +1929,42 @@ dlb_eventdev_dir_queue_setup(struct dlb_eventdev *dlb, }static int +dlb_event_queue_detach_ldb (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +struct dlb_eventdev_queue *ev_queue) +{ +int ret, i; + +/* Don't unlink until start time. */ +if (dlb->run_state ==DLB_RUN_STATE_STOPPED ) +return 0 ; + +for (i =0 ; i <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; i++) { +if (ev_port->link[i].valid && + ev_port->link[i].queue_id == ev_queue->id) +break ;/* found */ + } + +/* This is expected with eventdev API! +* It blindly attemmpts to unmap all queues. +*/ +if (i ==DLB_MAX_NUM_QIDS_PER_LDB_CQ ) { +DLB_LOG_DBG ("dlb: ignoring LB QID %d not mapped for qm_port %d.\n" , + ev_queue->qm_queue.id, + ev_port->qm_port.id); +return 0 ; + } + + ret =dlb_hw_unmap_ldb_qid_from_port (&dlb->qm_instance, + ev_port->qm_port.id, + ev_queue->qm_queue.id); +if (!ret) + ev_port->link[i].mapped = false; + +return ret; +} + +static int dlb_do_port_link(struct rte_eventdev *dev, struct dlb_eventdev_queue *ev_queue, struct dlb_eventdev_port *ev_port,
@@ -2477,6 +2537,109 @@ dlb_eventdev_port_setup(struct rte_eventdev *dev,return 0 ; } +static int +dlb_eventdev_port_unlink (struct rte_eventdev *dev,void *event_port, +uint8_t queues[],uint16_t nb_unlinks) +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_eventdev *dlb; +int i; + +RTE_SET_USED (dev); + +if (!ev_port->setup_done) { +DLB_LOG_ERR ("dlb: evport %d is not configured\n" , + ev_port->id); + rte_errno = -EINVAL ; +return 0 ; + } + +if (!queues || nb_unlinks ==0 ) { +DLB_LOG_DBG ("dlb: queues is NULL or nb_unlinks is 0\n" ); +return 0 ;/* Ignore and return success */ + } + +if (ev_port->qm_port.is_directed) { +DLB_LOG_DBG ("dlb: ignore unlink from dir port %d\n" , + ev_port->id); + rte_errno =0 ; +return nb_unlinks;/* as if success */ + } + + dlb = ev_port->dlb; + +for (i =0 ; i < nb_unlinks; i++) { +struct dlb_eventdev_queue *ev_queue; +int ret, j; + +if (queues[i] >= dlb->num_queues) { +DLB_LOG_ERR ("dlb: invalid queue id %d\n" , queues[i]); + rte_errno = -EINVAL ; +return i;/* return index of offending queue */ + } + + ev_queue = &dlb->ev_queues[queues[i]]; + +/* Does a link exist? */ +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) +if (ev_port->link[j].queue_id == queues[i] && + ev_port->link[j].valid) +break ; + +if (j ==DLB_MAX_NUM_QIDS_PER_LDB_CQ ) +continue ; + + ret =dlb_event_queue_detach_ldb (dlb, ev_port, ev_queue); +if (ret) { +DLB_LOG_ERR ("unlink err=%d for port %d queue %d\n" , + ret, ev_port->id, queues[i]); + rte_errno = -ENOENT ; +return i;/* return index of offending queue */ + } + + ev_port->link[j].valid = false; + ev_port->num_links--; + ev_queue->num_links--; + } + +return nb_unlinks; +} + +static int +dlb_eventdev_port_unlinks_in_progress (struct rte_eventdev *dev, +void *event_port) +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_eventdev *dlb; +struct dlb_hw_dev *handle; +struct dlb_pending_port_unmaps_args cfg; +struct dlb_cmd_response response; +int ret; + +RTE_SET_USED (dev); + +if (!ev_port->setup_done) { +DLB_LOG_ERR ("dlb: evport %d is not configured\n" , + ev_port->id); + rte_errno = -EINVAL ; +return 0 ; + } + + cfg.port_id = ev_port->qm_port.id; + cfg.response = (uintptr_t )&response; + dlb = ev_port->dlb; + handle = &dlb->qm_instance; + ret =dlb_iface_pending_port_unmaps (handle, &cfg); + +if (ret <0 ) { +DLB_LOG_ERR ("dlb: num_unlinks_in_progress ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return response.id; +} +static void dlb_eventdev_port_release (void *port) {
@@ -2583,6 +2746,9 @@ dlb_entry_points_init(struct rte_eventdev *dev) .port_setup = dlb_eventdev_port_setup, .port_release = dlb_eventdev_port_release, .port_link = dlb_eventdev_port_link, + .port_unlink = dlb_eventdev_port_unlink, + .port_unlinks_in_progress = + dlb_eventdev_port_unlinks_in_progress, };/* Expose PMD's eventdev interface */
Signature · 1 line
2.13.6
PATCH 24/27 event/dlb: add timout_ticks, dump, xstats, and selftest
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/event/dlb/dlb.c +21 -0
@@ -1820,6 +1820,20 @@ dlb_event_enqueue_forward_burst_delayed(void *event_port,return __dlb_event_enqueue_burst (event_port, events, num, true); } + +static int +dlb_eventdev_timeout_ticks (struct rte_eventdev *dev,uint64_t ns, +uint64_t *timeout_ticks) +{ +RTE_SET_USED (dev); +uint64_t cycles_per_ns; + + cycles_per_ns =rte_get_timer_hz () /1E9 ; + *timeout_ticks = ns * cycles_per_ns; + +return 0 ; +} +/* Note: 1 QM instance per QM device, QM instance/device == event device */ static int dlb_eventdev_configure (const struct rte_eventdev *dev)
@@ -2891,6 +2905,13 @@ dlb_entry_points_init(struct rte_eventdev *dev) .port_unlink = dlb_eventdev_port_unlink, .port_unlinks_in_progress = dlb_eventdev_port_unlinks_in_progress, + .timeout_ticks = dlb_eventdev_timeout_ticks, + .dump = dlb_eventdev_dump, + .xstats_get = dlb_eventdev_xstats_get, + .xstats_get_names = dlb_eventdev_xstats_get_names, + .xstats_get_by_name = dlb_eventdev_xstats_get_by_name, + .xstats_reset = dlb_eventdev_xstats_reset, + .dev_selftest = test_dlb_eventdev, };/* Expose PMD's eventdev interface */
Signature · 1 line
2.13.6
PATCH 23/27 event/dlb: add eventdev_start
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+)
drivers/event/dlb/dlb.c +142 -0
@@ -1162,6 +1162,72 @@ dlb_eventdev_info_get(struct rte_eventdev *dev, *dev_info = evdev_dlb_default_info; } +static int +dlb_eventdev_reapply_configuration (struct rte_eventdev *dev); + +static int +dlb_eventdev_apply_port_links (struct rte_eventdev *dev); + +static int +dlb_eventdev_start (struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_start_domain_args cfg; +struct dlb_cmd_response response; +int ret, i; + +rte_spinlock_lock (&dlb->qm_instance.resource_lock); +if (dlb->run_state !=DLB_RUN_STATE_STOPPED ) { +DLB_LOG_ERR ("bad state %d for dev_start\n" , + (int )dlb->run_state); +rte_spinlock_unlock (&dlb->qm_instance.resource_lock); +return -EINVAL ; + } + dlb->run_state =DLB_RUN_STATE_STARTING ; +rte_spinlock_unlock (&dlb->qm_instance.resource_lock); + +/* If the device was configured more than once, some event ports and/or +* queues may need to be reconfigured. +*/ + ret =dlb_eventdev_reapply_configuration (dev); +if (ret) +return ret; + +/* The DLB PMD delays port links until the device is started. */ + ret =dlb_eventdev_apply_port_links (dev); +if (ret) +return ret; + + cfg.response = (uintptr_t )&response; + +for (i =0 ; i < dlb->num_ports; i++) { +if (!dlb->ev_ports[i].setup_done) { +DLB_LOG_ERR ("dlb: port %d not setup" , i); +return -ESTALE ; + } + } + +for (i =0 ; i < dlb->num_queues; i++) { +if (dlb->ev_queues[i].num_links ==0 ) { +DLB_LOG_ERR ("dlb: queue %d is not linked" , i); +return -ENOLINK ; + } + } + + ret =dlb_iface_sched_domain_start (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: sched_domain_start ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + + dlb->run_state =DLB_RUN_STATE_STARTED ; +DLB_LOG_DBG ("dlb: sched_domain_start completed OK\n" ); + +return 0 ; +} +static inline int dlb_check_enqueue_sw_credits (struct dlb_eventdev *dlb,struct dlb_eventdev_port *ev_port)
@@ -2058,6 +2124,40 @@ dlb_validate_port_link(struct dlb_eventdev_port *ev_port, }static int +dlb_eventdev_apply_port_links (struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +int i; + +/* Perform requested port->queue links */ +for (i =0 ; i < dlb->num_ports; i++) { +struct dlb_eventdev_port *ev_port = &dlb->ev_ports[i]; +int j; + +for (j =0 ; j <DLB_MAX_NUM_QIDS_PER_LDB_CQ ; j++) { +struct dlb_eventdev_queue *ev_queue; +uint8_t prio, queue_id; + +if (!ev_port->link[j].valid) +continue ; + + prio = ev_port->link[j].priority; + queue_id = ev_port->link[j].queue_id; + +if (dlb_validate_port_link (ev_port, queue_id, true, j)) +return -EINVAL ; + + ev_queue = &dlb->ev_queues[queue_id]; + +if (dlb_do_port_link (dev, ev_queue, ev_port, prio)) +return -EINVAL ; + } + } + +return 0 ; +} + +static int dlb_eventdev_port_link (struct rte_eventdev *dev,void *event_port,const uint8_t queues[],const uint8_t priorities[],uint16_t nb_links)
@@ -2538,6 +2638,47 @@ dlb_eventdev_port_setup(struct rte_eventdev *dev, }static int +dlb_eventdev_reapply_configuration (struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +int ret, i; + +/* If an event queue or port was previously configured, but hasn't been +* reconfigured, reapply its original configuration. +*/ +for (i =0 ; i < dlb->num_queues; i++) { +struct dlb_eventdev_queue *ev_queue; + + ev_queue = &dlb->ev_queues[i]; + +if (ev_queue->qm_queue.config_state !=DLB_PREV_CONFIGURED ) +continue ; + + ret =dlb_eventdev_queue_setup (dev, i, &ev_queue->conf); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: failed to reconfigure queue %d" , i); +return ret; + } + } + +for (i =0 ; i < dlb->num_ports; i++) { +struct dlb_eventdev_port *ev_port = &dlb->ev_ports[i]; + +if (ev_port->qm_port.config_state !=DLB_PREV_CONFIGURED ) +continue ; + + ret =dlb_eventdev_port_setup (dev, i, &ev_port->conf); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: failed to reconfigure ev_port %d" , + i); +return ret; + } + } + +return 0 ; +} + +static int dlb_eventdev_port_unlink (struct rte_eventdev *dev,void *event_port,uint8_t queues[],uint16_t nb_unlinks) {
@@ -2739,6 +2880,7 @@ dlb_entry_points_init(struct rte_eventdev *dev)static struct rte_eventdev_ops dlb_eventdev_entry_ops = { .dev_infos_get = dlb_eventdev_info_get, .dev_configure = dlb_eventdev_configure, + .dev_start = dlb_eventdev_start, .queue_def_conf = dlb_eventdev_queue_default_conf_get, .queue_setup = dlb_eventdev_queue_setup, .queue_release = dlb_eventdev_queue_release,
Signature · 1 line
2.13.6
PATCH 25/27 event/dlb: add enqueue and its burst variants
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/event/dlb/dlb.c +20 -0
@@ -1805,6 +1805,14 @@ dlb_event_enqueue_delayed(void *event_port, }static uint16_t +dlb_event_enqueue_new_burst (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, false); +} + +static uint16_t dlb_event_enqueue_new_burst_delayed (void *event_port,const struct rte_event events[],uint16_t num)
@@ -1813,6 +1821,14 @@ dlb_event_enqueue_new_burst_delayed(void *event_port, }static uint16_t +dlb_event_enqueue_forward_burst (void *event_port, +const struct rte_event events[], +uint16_t num) +{ +return __dlb_event_enqueue_burst (event_port, events, num, false); +} + +static uint16_t dlb_event_enqueue_forward_burst_delayed (void *event_port,const struct rte_event events[],uint16_t num)
@@ -2917,6 +2933,10 @@ dlb_entry_points_init(struct rte_eventdev *dev)/* Expose PMD's eventdev interface */ dev->dev_ops = &dlb_eventdev_entry_ops; + dev->enqueue = dlb_event_enqueue; + dev->enqueue_burst = dlb_event_enqueue_burst; + dev->enqueue_new_burst = dlb_event_enqueue_new_burst; + dev->enqueue_forward_burst = dlb_event_enqueue_forward_burst; }
Signature · 1 line
2.13.6
PATCH 27/27 event/dlb: add eventdev_stop and eventdev_close
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 263 insertions(+)
drivers/event/dlb/dlb.c +263 -0
@@ -305,6 +305,23 @@ dlb_event_enqueue_forward_burst_delayed(void *event_port,const struct rte_event events[], uint16_t num); +void +dlb_free_qe_mem (struct dlb_port *qm_port) +{ +if (qm_port ==NULL ) +return ; + +if (qm_port->qe4) { +rte_free (qm_port->qe4); + qm_port->qe4 =NULL ; + } + +if (qm_port->consume_qe) { +rte_free (qm_port->consume_qe); + qm_port->consume_qe =NULL ; + } +} +int dlb_init_qe_mem (struct dlb_port *qm_port,char *mz_name) {
@@ -1922,6 +1939,250 @@ dlb_eventdev_start(struct rte_eventdev *dev)return 0 ; } +static void +dlb_flush_port (struct rte_eventdev *dev,int port_id) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +eventdev_stop_flush_t flush; +struct rte_event ev; +uint8_t dev_id; +void *arg; +int i; + + flush = dev->dev_ops->dev_stop_flush; + dev_id = dev->data->dev_id; + arg = dev->data->dev_stop_flush_arg; + +while (rte_event_dequeue_burst (dev_id, port_id, &ev,1 ,0 )) { +if (flush) +flush (dev_id, ev, arg); + +if (dlb->ev_ports[port_id].qm_port.is_directed) +continue ; + + ev.op =RTE_EVENT_OP_RELEASE ; + +rte_event_enqueue_burst (dev_id, port_id, &ev,1 ); + } + +/* Enqueue any additional outstanding releases */ + ev.op =RTE_EVENT_OP_RELEASE ; + +for (i = dlb->ev_ports[port_id].outstanding_releases; i >0 ; i--) +rte_event_enqueue_burst (dev_id, port_id, &ev,1 ); +} + +static uint32_t +dlb_get_ldb_queue_depth (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *queue) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_get_ldb_queue_depth_args cfg; +struct dlb_cmd_response response; +int ret; + + cfg.queue_id = queue->qm_queue.id; + cfg.response = (uintptr_t )&response; + + ret =dlb_iface_get_ldb_queue_depth (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: get_ldb_queue_depth ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return response.id; +} + +static uint32_t +dlb_get_dir_queue_depth (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *queue) +{ +struct dlb_hw_dev *handle = &dlb->qm_instance; +struct dlb_get_dir_queue_depth_args cfg; +struct dlb_cmd_response response; +int ret; + + cfg.queue_id = queue->qm_queue.id; + cfg.response = (uintptr_t )&response; + + ret =dlb_iface_get_dir_queue_depth (handle, &cfg); +if (ret <0 ) { +DLB_LOG_ERR ("dlb: get_dir_queue_depth ret=%d (driver status: %s)\n" , + ret, dlb_error_strings[response.status]); +return ret; + } + +return response.id; +} + +uint32_t +dlb_get_queue_depth (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *queue) +{ +if (queue->qm_queue.is_directed) +return dlb_get_dir_queue_depth (dlb, queue); +else +return dlb_get_ldb_queue_depth (dlb, queue); +} + +static bool +dlb_queue_is_empty (struct dlb_eventdev *dlb, +struct dlb_eventdev_queue *queue) +{ +return dlb_get_queue_depth (dlb, queue) ==0 ; +} + +static bool +dlb_linked_queues_empty (struct dlb_eventdev *dlb) +{ +int i; + +for (i =0 ; i < dlb->num_queues; i++) { +if (dlb->ev_queues[i].num_links ==0 ) +continue ; +if (!dlb_queue_is_empty (dlb, &dlb->ev_queues[i])) +return false; + } + +return true; +} + +static bool +dlb_queues_empty (struct dlb_eventdev *dlb) +{ +int i; + +for (i =0 ; i < dlb->num_queues; i++) { +if (!dlb_queue_is_empty (dlb, &dlb->ev_queues[i])) +return false; + } + +return true; +} + +void +dlb_drain (struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); +struct dlb_eventdev_port *ev_port =NULL ; +uint8_t dev_id; +int i; + + dev_id = dev->data->dev_id; + +while (!dlb_linked_queues_empty (dlb)) { +/* Flush all the ev_ports, which will drain all their connected +* queues. +*/ +for (i =0 ; i < dlb->num_ports; i++) +dlb_flush_port (dev, i); + } + +/* The queues are empty, but there may be events left in the ports. */ +for (i =0 ; i < dlb->num_ports; i++) +dlb_flush_port (dev, i); + +/* If the domain's queues are empty, we're done. */ +if (dlb_queues_empty (dlb)) +return ; + +/* Else, there must be at least one unlinked load-balanced queue. +* Select a load-balanced port with which to drain the unlinked +* queue(s). +*/ +for (i =0 ; i < dlb->num_ports; i++) { + ev_port = &dlb->ev_ports[i]; + +if (!ev_port->qm_port.is_directed) +break ; + } + +if (i == dlb->num_ports) { +DLB_LOG_ERR ("internal error: no LDB ev_ports\n" ); +return ; + } + + rte_errno =0 ; +rte_event_port_unlink (dev_id, ev_port->id,NULL ,0 ); + +if (rte_errno) { +DLB_LOG_ERR ("internal error: failed to unlink ev_port %d\n" , + ev_port->id); +return ; + } + +for (i =0 ; i < dlb->num_queues; i++) { +uint8_t qid, prio; +int ret; + +if (dlb_queue_is_empty (dlb, &dlb->ev_queues[i])) +continue ; + + qid = i; + prio =0 ; + +/* Link the ev_port to the queue */ + ret =rte_event_port_link (dev_id, ev_port->id, &qid, &prio,1 ); +if (ret !=1 ) { +DLB_LOG_ERR ("internal error: failed to link ev_port %d to queue %d\n" , + ev_port->id, qid); +return ; + } + +/* Flush the queue */ +while (!dlb_queue_is_empty (dlb, &dlb->ev_queues[i])) +dlb_flush_port (dev, ev_port->id); + +/* Drain any extant events in the ev_port. */ +dlb_flush_port (dev, ev_port->id); + +/* Unlink the ev_port from the queue */ + ret =rte_event_port_unlink (dev_id, ev_port->id, &qid,1 ); +if (ret !=1 ) { +DLB_LOG_ERR ("internal error: failed to unlink ev_port %d to queue %d\n" , + ev_port->id, qid); +return ; + } + } +} + +static void +dlb_eventdev_stop (struct rte_eventdev *dev) +{ +struct dlb_eventdev *dlb =dlb_pmd_priv (dev); + +rte_spinlock_lock (&dlb->qm_instance.resource_lock); + +if (dlb->run_state ==DLB_RUN_STATE_STOPPED ) { +DLB_LOG_DBG ("Internal error: already stopped\n" ); +rte_spinlock_unlock (&dlb->qm_instance.resource_lock); +return ; + }else if (dlb->run_state !=DLB_RUN_STATE_STARTED ) { +DLB_LOG_ERR ("Internal error: bad state %d for dev_stop\n" , + (int )dlb->run_state); +rte_spinlock_unlock (&dlb->qm_instance.resource_lock); +return ; + } + + dlb->run_state =DLB_RUN_STATE_STOPPING ; + +rte_spinlock_unlock (&dlb->qm_instance.resource_lock); + +dlb_drain (dev); + + dlb->run_state =DLB_RUN_STATE_STOPPED ; +} + +static int +dlb_eventdev_close (struct rte_eventdev *dev) +{ +dlb_hw_reset_sched_domain (dev, false); + +return 0 ; +} + +static inline int dlb_check_enqueue_sw_credits (struct dlb_eventdev *dlb,struct dlb_eventdev_port *ev_port)
@@ -3782,6 +4043,8 @@ dlb_entry_points_init(struct rte_eventdev *dev) .dev_infos_get = dlb_eventdev_info_get, .dev_configure = dlb_eventdev_configure, .dev_start = dlb_eventdev_start, + .dev_stop = dlb_eventdev_stop, + .dev_close = dlb_eventdev_close, .queue_def_conf = dlb_eventdev_queue_default_conf_get, .queue_setup = dlb_eventdev_queue_setup, .queue_release = dlb_eventdev_queue_release,
Signature · 1 line
2.13.6
PATCH 26/27 event/dlb: add dequeue, dequeue_burst, and variants
In reply to McDaniel, Timothy--- drivers/event/dlb/dlb.c | 879 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 879 insertions(+)
drivers/event/dlb/dlb.c +879 -0
@@ -826,6 +826,700 @@ dlb_hw_do_enqueue(struct dlb_port *qm_port,dlb_pp_write (qm_port->qe4, port_data); } +static __rte_always_inline int +dlb_recv_qe (struct dlb_port *qm_port,struct dlb_dequeue_qe *qe, +uint8_t *offset) +{ +uint8_t xor_mask[2 ][4 ] = { {0x0F ,0x0E ,0x0C ,0x08 }, + {0x00 ,0x01 ,0x03 ,0x07 } }; +uint8_t and_mask[4 ] = {0x0F ,0x0E ,0x0C ,0x08 }; +volatile struct dlb_dequeue_qe *cq_addr; +__m128i *qes = (__m128i *)qe; +uint64_t *cache_line_base; +uint8_t gen_bits; + + cq_addr = dlb_port[qm_port->id][PORT_TYPE (qm_port)].cq_base; + cq_addr = &cq_addr[qm_port->cq_idx]; + + cache_line_base = (void *)(((uintptr_t )cq_addr) & ~0x3F ); + *offset = ((uintptr_t )cq_addr &0x30 ) >>4 ; + +/* Load the next CQ cache line from memory. Pack these reads as tight +* as possible to reduce the chance that DLB invalidates the line while +* the CPU is reading it. Read the cache line backwards to ensure that +* if QE[N] (N > 0) is valid, then QEs[0:N-1] are too. +* +* (Valid QEs start at &qe[offset]) +*/ + qes[3 ] =_mm_load_si128 ((__m128i *)&cache_line_base[6 ]); + qes[2 ] =_mm_load_si128 ((__m128i *)&cache_line_base[4 ]); + qes[1 ] =_mm_load_si128 ((__m128i *)&cache_line_base[2 ]); + qes[0 ] =_mm_load_si128 ((__m128i *)&cache_line_base[0 ]); + +/* Evict the cache line ASAP */ +dlb_cldemote (cache_line_base); + +/* Extract and combine the gen bits */ + gen_bits = ((_mm_extract_epi8 (qes[0 ],15 ) &0x1 ) <<0 ) | + ((_mm_extract_epi8 (qes[1 ],15 ) &0x1 ) <<1 ) | + ((_mm_extract_epi8 (qes[2 ],15 ) &0x1 ) <<2 ) | + ((_mm_extract_epi8 (qes[3 ],15 ) &0x1 ) <<3 ); + +/* XOR the combined bits such that a 1 represents a valid QE */ + gen_bits ^= xor_mask[qm_port->gen_bit][*offset]; + +/* Mask off gen bits we don't care about */ + gen_bits &= and_mask[*offset]; + +return __builtin_popcount (gen_bits); +} + +static __rte_always_inline int +dlb_recv_qe_sparse (struct dlb_port *qm_port,struct dlb_dequeue_qe *qe) +{ +volatile struct dlb_dequeue_qe *cq_addr; +uint8_t xor_mask[2 ] = {0x0F ,0x00 }; +const uint8_t and_mask =0x0F ; +__m128i *qes = (__m128i *)qe; +uint8_t gen_bits, gen_bit; +uintptr_t addr[4 ]; +uint16_t idx; + + cq_addr = dlb_port[qm_port->id][PORT_TYPE (qm_port)].cq_base; + + idx = qm_port->cq_idx; + +/* Load the next 4 QEs */ + addr[0 ] = (uintptr_t )&cq_addr[idx]; + addr[1 ] = (uintptr_t )&cq_addr[(idx +4 ) & qm_port->cq_depth_mask]; + addr[2 ] = (uintptr_t )&cq_addr[(idx +8 ) & qm_port->cq_depth_mask]; + addr[3 ] = (uintptr_t )&cq_addr[(idx +12 ) & qm_port->cq_depth_mask]; + +/* Prefetch next batch of QEs (all CQs occupy minimum 8 cache lines) */ +rte_prefetch0 (&cq_addr[(idx +16 ) & qm_port->cq_depth_mask]); +rte_prefetch0 (&cq_addr[(idx +20 ) & qm_port->cq_depth_mask]); +rte_prefetch0 (&cq_addr[(idx +24 ) & qm_port->cq_depth_mask]); +rte_prefetch0 (&cq_addr[(idx +28 ) & qm_port->cq_depth_mask]); + +/* Correct the xor_mask for wrap-around QEs */ + gen_bit = qm_port->gen_bit; + xor_mask[gen_bit] ^= !!((idx +4 ) > qm_port->cq_depth_mask) <<1 ; + xor_mask[gen_bit] ^= !!((idx +8 ) > qm_port->cq_depth_mask) <<2 ; + xor_mask[gen_bit] ^= !!((idx +12 ) > qm_port->cq_depth_mask) <<3 ; + +/* Read the cache lines backwards to ensure that if QE[N] (N > 0) is +* valid, then QEs[0:N-1] are too. +*/ + qes[3 ] =_mm_load_si128 ((__m128i *)(void *)addr[3 ]); +rte_compiler_barrier (); + qes[2 ] =_mm_load_si128 ((__m128i *)(void *)addr[2 ]); +rte_compiler_barrier (); + qes[1 ] =_mm_load_si128 ((__m128i *)(void *)addr[1 ]); +rte_compiler_barrier (); + qes[0 ] =_mm_load_si128 ((__m128i *)(void *)addr[0 ]); + +/* Extract and combine the gen bits */ + gen_bits = ((_mm_extract_epi8 (qes[0 ],15 ) &0x1 ) <<0 ) | + ((_mm_extract_epi8 (qes[1 ],15 ) &0x1 ) <<1 ) | + ((_mm_extract_epi8 (qes[2 ],15 ) &0x1 ) <<2 ) | + ((_mm_extract_epi8 (qes[3 ],15 ) &0x1 ) <<3 ); + +/* XOR the combined bits such that a 1 represents a valid QE */ + gen_bits ^= xor_mask[gen_bit]; + +/* Mask off gen bits we don't care about */ + gen_bits &= and_mask; + +return __builtin_popcount (gen_bits); +} + +static inline void +dlb_inc_cq_idx (struct dlb_port *qm_port,int cnt) +{ +uint16_t idx = qm_port->cq_idx_unmasked + cnt; + + qm_port->cq_idx_unmasked = idx; + qm_port->cq_idx = idx & qm_port->cq_depth_mask; + qm_port->gen_bit = (~(idx >> qm_port->gen_bit_shift)) &0x1 ; +} + +static inline bool +dlb_cq_is_empty (struct dlb_port *qm_port) +{ +volatile struct dlb_dequeue_qe *qe_ptr; +struct dlb_dequeue_qe qe; + + qe_ptr = dlb_port[qm_port->id][PORT_TYPE (qm_port)].cq_base; + qe = qe_ptr[qm_port->cq_idx]; + +return (qe.cq_gen != qm_port->gen_bit); +} + +static inline void dlb_send_rsvd_token_int_arm (struct dlb_port *qm_port) +{ +struct dlb_cq_pop_qe *qe = qm_port->consume_qe; +struct process_local_port_data *port_data; + + qe->tokens =0 ; + +rte_memcpy (&qm_port->qe4[0 ], qe,sizeof (struct dlb_cq_pop_qe )); + +/* Set int_arm to true in qm_port->qe4[0] so consume_qe isn't +* affected. +*/ + qm_port->qe4[0 ].int_arm =1 ; + +/* Fill remaining 3 entries (1,2, and 3) with noop QEs */ + +memset (&qm_port->qe4[1 ], +0 , + (DLB_NUM_QES_PER_CACHE_LINE -1 ) * +sizeof (struct dlb_cq_pop_qe )); + +/* No store fence needed since no pointer is being sent, and CQ token +* pops can be safely reordered with other HCWs. +*/ + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +dlb_pp_write (qm_port->qe4, port_data); + +DLB_LOG_DBG ("dlb: send rsvd token int arm\n" ); +} + +static inline void dlb_issue_int_arm_hcw (struct dlb_port *qm_port) +{ +dlb_send_rsvd_token_int_arm (qm_port); + + qm_port->cq_rsvd_token_deficit =1 ; + qm_port->int_armed = true; + +/* The delayed token return needs to account for this token pop HCW */ + qm_port->issued_releases--; +} + +static inline int +dlb_block_on_cq_interrupt (struct dlb_hw_dev *handle, +struct dlb_port *qm_port, +struct dlb_eventdev_port *ev_port) +{ +struct process_local_port_data *port_data; +int ret; + +/* If the interrupt is not armed, either sleep-poll (see comment below) +* or arm the interrupt. +*/ +if (!qm_port->int_armed) { +/* The reserved token scheme requires setting the interrupt +* depth threshold equal to the number of reserved tokens. +* Until the port receives its set of reserved tokens it cannot +* block on the interrupt, since it will not fire until the CQ +* contains at least num-reserved-tokens QEs. In lieu of that, +* we sleep-poll the CQ, which gives a similar behavior (other +* threads can run) albeit with a potentially slower response +* time. +*/ +if (qm_port->use_rsvd_token_scheme && + qm_port->cq_rsvd_token_deficit) { +sched_yield (); +return 0 ; + } + +dlb_issue_int_arm_hcw (qm_port); + } + + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +/* Note: it's safe to access the per-process cq_base address here, +* since the PMD won't block on the CQ until after attempting at least +* one CQ dequeue. +*/ + ret =dlb_iface_block_on_cq_interrupt ( + handle, + qm_port->id, + !ev_port->qm_port.is_directed, + &port_data->cq_base[qm_port->cq_idx], + qm_port->gen_bit, + false); + +/* If the CQ int ioctl was unsuccessful, the interrupt remains armed */ + qm_port->int_armed = (ret !=0 ); + +return ret; +} + +static inline int +dlb_process_dequeue_qes (struct dlb_eventdev_port *ev_port, +struct dlb_port *qm_port, +struct rte_event *events, +struct dlb_dequeue_qe *qes, +int cnt) +{ +uint8_t *qid_mappings = qm_port->qid_mappings; +int i, num; + +RTE_SET_USED (ev_port);/* avoids unused variable error */ + +for (i =0 , num =0 ; i < cnt; i++) { +struct dlb_dequeue_qe *qe = &qes[i]; +int sched_type_map[4 ] = { + [DLB_SCHED_ATOMIC ] =RTE_SCHED_TYPE_ATOMIC , + [DLB_SCHED_UNORDERED ] =RTE_SCHED_TYPE_PARALLEL , + [DLB_SCHED_ORDERED ] =RTE_SCHED_TYPE_ORDERED , + [DLB_SCHED_DIRECTED ] =RTE_SCHED_TYPE_ATOMIC , + }; + +DLB_LOG_DBG ("dequeue success, data = 0x%llx, qid=%d, event_type=%d, subevent=%d\npp_id = %d, sched_type = %d, qid = %d, err=%d\n" , + (long long )qe->data, qe->qid, + qe->u.event_type.major, + qe->u.event_type.sub, + qe->pp_id, qe->sched_type, qe->qid, qe->error); + +/* Fill in event information. +* Note that flow_id must be embedded in the data by +* the app, such as the mbuf RSS hash field if the data +* buffer is a mbuf. +*/ +if (unlikely (qe->error)) { +DLB_LOG_ERR ("QE error bit ON\n" ); +DLB_INC_STAT (ev_port->stats.traffic.rx_drop,1 ); +dlb_consume_qe_immediate (qm_port,1 ); +continue ;/* Ignore */ + } + + events[num].u64 = qe->data; + events[num].queue_id = qid_mappings[qe->qid]; + events[num].priority =DLB_TO_EV_PRIO ((uint8_t )qe->priority); + events[num].event_type = qe->u.event_type.major; + events[num].sub_event_type = qe->u.event_type.sub; + events[num].sched_type = sched_type_map[qe->sched_type]; +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qe->sched_type],1 ); + +DLB_INC_STAT (ev_port->stats.traffic.rx_ok,1 ); + + num++; + } + +return num; +} + +static inline int +dlb_process_dequeue_four_qes (struct dlb_eventdev_port *ev_port, +struct dlb_port *qm_port, +struct rte_event *events, +struct dlb_dequeue_qe *qes) +{ +int sched_type_map[] = { + [DLB_SCHED_ATOMIC ] =RTE_SCHED_TYPE_ATOMIC , + [DLB_SCHED_UNORDERED ] =RTE_SCHED_TYPE_PARALLEL , + [DLB_SCHED_ORDERED ] =RTE_SCHED_TYPE_ORDERED , + [DLB_SCHED_DIRECTED ] =RTE_SCHED_TYPE_ATOMIC , + }; +const int num_events =DLB_NUM_QES_PER_CACHE_LINE ; +uint8_t *qid_mappings = qm_port->qid_mappings; +__m128i sse_evt[2 ]; +#ifdef DLB_QUELL_STATS + (void )ev_port;/* avoids unused variable error */ +#endif +int i; + +/* In the unlikely case that any of the QE error bits are set, process +* them one at a time. +*/ +if (unlikely (qes[0 ].error || qes[1 ].error || + qes[2 ].error || qes[3 ].error)) +return dlb_process_dequeue_qes (ev_port, qm_port, events, + qes, num_events); + +for (i =0 ; i <DLB_NUM_QES_PER_CACHE_LINE ; i++) { +DLB_LOG_DBG ("dequeue success, data = 0x%llx, qid=%d, event_type=%d, subevent=%d\npp_id = %d, sched_type = %d, qid = %d, err=%d\n" , + (long long )qes[i].data, qes[i].qid, + qes[i].u.event_type.major, + qes[i].u.event_type.sub, + qes[i].pp_id, qes[i].sched_type, qes[i].qid, + qes[i].error); + } + + events[0 ].u64 = qes[0 ].data; + events[1 ].u64 = qes[1 ].data; + events[2 ].u64 = qes[2 ].data; + events[3 ].u64 = qes[3 ].data; + +/* Construct the metadata portion of two struct rte_events +* in one 128b SSE register. Event metadata is constructed in the SSE +* registers like so: +* sse_evt[0][63:0]: event[0]'s metadata +* sse_evt[0][127:64]: event[1]'s metadata +* sse_evt[1][63:0]: event[2]'s metadata +* sse_evt[1][127:64]: event[3]'s metadata +*/ + sse_evt[0 ] =_mm_setzero_si128 (); + sse_evt[1 ] =_mm_setzero_si128 (); + +/* Convert the hardware queue ID to an event queue ID and store it in +* the metadata: +* sse_evt[0][47:40] = qid_mappings[qes[0].qid] +* sse_evt[0][111:104] = qid_mappings[qes[1].qid] +* sse_evt[1][47:40] = qid_mappings[qes[2].qid] +* sse_evt[1][111:104] = qid_mappings[qes[3].qid] +*/ +#define RTE_EVENT_QUEUE_ID_BYTE 5 + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], + qid_mappings[qes[0 ].qid], +RTE_EVENT_QUEUE_ID_BYTE ); + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], + qid_mappings[qes[1 ].qid], +RTE_EVENT_QUEUE_ID_BYTE +8 ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], + qid_mappings[qes[2 ].qid], +RTE_EVENT_QUEUE_ID_BYTE ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], + qid_mappings[qes[3 ].qid], +RTE_EVENT_QUEUE_ID_BYTE +8 ); + +/* Convert the hardware priority to an event priority and store it in +* the metadata: +* sse_evt[0][55:48] = DLB_TO_EV_PRIO(qes[0].priority) +* sse_evt[0][119:112] = DLB_TO_EV_PRIO(qes[1].priority) +* sse_evt[1][55:48] = DLB_TO_EV_PRIO(qes[2].priority) +* sse_evt[1][119:112] = DLB_TO_EV_PRIO(qes[3].priority) +*/ +#define RTE_EVENT_PRIO_BYTE 6 + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], +DLB_TO_EV_PRIO ((uint8_t )qes[0 ].priority), +RTE_EVENT_PRIO_BYTE ); + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], +DLB_TO_EV_PRIO ((uint8_t )qes[1 ].priority), +RTE_EVENT_PRIO_BYTE +8 ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], +DLB_TO_EV_PRIO ((uint8_t )qes[2 ].priority), +RTE_EVENT_PRIO_BYTE ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], +DLB_TO_EV_PRIO ((uint8_t )qes[3 ].priority), +RTE_EVENT_PRIO_BYTE +8 ); + +/* Write the event type and sub event type to the event metadata. Leave +* flow ID unspecified, since the hardware does not maintain it during +* scheduling: +* sse_evt[0][31:0] = qes[0].u.event_type.major << 28 | +* qes[0].u.event_type.sub << 20; +* sse_evt[0][95:64] = qes[1].u.event_type.major << 28 | +* qes[1].u.event_type.sub << 20; +* sse_evt[1][31:0] = qes[2].u.event_type.major << 28 | +* qes[2].u.event_type.sub << 20; +* sse_evt[1][95:64] = qes[3].u.event_type.major << 28 | +* qes[3].u.event_type.sub << 20; +*/ +#define RTE_EVENT_EV_TYPE_DW 0 +#define RTE_EVENT_EV_TYPE_SHIFT 28 +#define RTE_EVENT_SUB_EV_TYPE_SHIFT 20 + sse_evt[0 ] =_mm_insert_epi32 (sse_evt[0 ], + qes[0 ].u.event_type.major <<RTE_EVENT_EV_TYPE_SHIFT | + qes[0 ].u.event_type.sub <<RTE_EVENT_SUB_EV_TYPE_SHIFT , +RTE_EVENT_EV_TYPE_DW ); + sse_evt[0 ] =_mm_insert_epi32 (sse_evt[0 ], + qes[1 ].u.event_type.major <<RTE_EVENT_EV_TYPE_SHIFT | + qes[1 ].u.event_type.sub <<RTE_EVENT_SUB_EV_TYPE_SHIFT , +RTE_EVENT_EV_TYPE_DW +2 ); + sse_evt[1 ] =_mm_insert_epi32 (sse_evt[1 ], + qes[2 ].u.event_type.major <<RTE_EVENT_EV_TYPE_SHIFT | + qes[2 ].u.event_type.sub <<RTE_EVENT_SUB_EV_TYPE_SHIFT , +RTE_EVENT_EV_TYPE_DW ); + sse_evt[1 ] =_mm_insert_epi32 (sse_evt[1 ], + qes[3 ].u.event_type.major <<RTE_EVENT_EV_TYPE_SHIFT | + qes[3 ].u.event_type.sub <<RTE_EVENT_SUB_EV_TYPE_SHIFT , +RTE_EVENT_EV_TYPE_DW +2 ); + +/* Write the sched type to the event metadata. 'op' and 'rsvd' are not +* set: +* sse_evt[0][39:32] = sched_type_map[qes[0].sched_type] << 6 +* sse_evt[0][103:96] = sched_type_map[qes[1].sched_type] << 6 +* sse_evt[1][39:32] = sched_type_map[qes[2].sched_type] << 6 +* sse_evt[1][103:96] = sched_type_map[qes[3].sched_type] << 6 +*/ +#define RTE_EVENT_SCHED_TYPE_BYTE 4 +#define RTE_EVENT_SCHED_TYPE_SHIFT 6 + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], + sched_type_map[qes[0 ].sched_type] <<RTE_EVENT_SCHED_TYPE_SHIFT , +RTE_EVENT_SCHED_TYPE_BYTE ); + sse_evt[0 ] =_mm_insert_epi8 (sse_evt[0 ], + sched_type_map[qes[1 ].sched_type] <<RTE_EVENT_SCHED_TYPE_SHIFT , +RTE_EVENT_SCHED_TYPE_BYTE +8 ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], + sched_type_map[qes[2 ].sched_type] <<RTE_EVENT_SCHED_TYPE_SHIFT , +RTE_EVENT_SCHED_TYPE_BYTE ); + sse_evt[1 ] =_mm_insert_epi8 (sse_evt[1 ], + sched_type_map[qes[3 ].sched_type] <<RTE_EVENT_SCHED_TYPE_SHIFT , +RTE_EVENT_SCHED_TYPE_BYTE +8 ); + +/* Store the metadata to the event (use the double-precision +* _mm_storeh_pd because there is no integer function for storing the +* upper 64b): +* events[0].event = sse_evt[0][63:0] +* events[1].event = sse_evt[0][127:64] +* events[2].event = sse_evt[1][63:0] +* events[3].event = sse_evt[1][127:64] +*/ +_mm_storel_epi64 ((__m128i *)&events[0 ].event, sse_evt[0 ]); +_mm_storeh_pd ((double *)&events[1 ].event, (__m128d ) sse_evt[0 ]); +_mm_storel_epi64 ((__m128i *)&events[2 ].event, sse_evt[1 ]); +_mm_storeh_pd ((double *)&events[3 ].event, (__m128d ) sse_evt[1 ]); + +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qes[0 ].sched_type],1 ); +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qes[1 ].sched_type],1 ); +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qes[2 ].sched_type],1 ); +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qes[3 ].sched_type],1 ); + +DLB_INC_STAT (ev_port->stats.traffic.rx_ok, num_events); + +return num_events; +} + +static inline int +dlb_dequeue_wait (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +struct dlb_port *qm_port, +uint64_t timeout, +uint64_t start_ticks) +{ +struct process_local_port_data *port_data; +uint64_t elapsed_ticks; + + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + + elapsed_ticks =rte_get_timer_cycles () - start_ticks; + +/* Wait/poll time expired */ +if (elapsed_ticks >= timeout) { +/* Interrupts not supported by PF PMD */ +return 1 ; + }else if (dlb->umwait_allowed) { +volatile struct dlb_dequeue_qe *cq_base; + + cq_base = port_data->cq_base; + +/* Block on cache line write to CQ. Note: it's +* safe to access the per-process cq_base +* address here, since the PMD has already +* attempted at least one CQ dequeue. +*/ +dlb_umonitor (&cq_base[qm_port->cq_idx]); + +/* Avoid race condition. Check if still empty */ +if (dlb_cq_is_empty (qm_port)) { +dlb_umwait (RTE_LIBRTE_PMD_DLB_UMWAIT_CTL_STATE , + timeout + start_ticks); +DLB_INC_STAT (ev_port->stats.traffic.rx_umonitor_umwait, +1 ); + } + }else { +uint64_t poll_interval =RTE_LIBRTE_PMD_DLB_POLL_INTERVAL ; +uint64_t curr_ticks =rte_get_timer_cycles (); +uint64_t init_ticks = curr_ticks; + +while ((curr_ticks - start_ticks < timeout) && + (curr_ticks - init_ticks < poll_interval)) + curr_ticks =rte_get_timer_cycles (); + } + +return 0 ; +} + +static inline int16_t +dlb_hw_dequeue (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +struct rte_event *events, +uint16_t max_num, +uint64_t dequeue_timeout_ticks) +{ +uint64_t timeout; +uint64_t start_ticks =0ULL ; +struct dlb_port *qm_port; +int num =0 ; + + qm_port = &ev_port->qm_port; + +/* We have a special implementation for waiting. Wait can be: +* 1) no waiting at all +* 2) busy poll only +* 3) wait for interrupt. If wakeup and poll time +* has expired, then return to caller +* 4) umonitor/umwait repeatedly up to poll time +*/ + +/* If configured for per dequeue wait, then use wait value provided +* to this API. Otherwise we must use the global +* value from eventdev config time. +*/ +if (!dlb->global_dequeue_wait) + timeout = dequeue_timeout_ticks; +else + timeout = dlb->global_dequeue_wait_ticks; + +if (timeout) + start_ticks =rte_get_timer_cycles (); + +while (num < max_num) { +struct dlb_dequeue_qe qes[DLB_NUM_QES_PER_CACHE_LINE ]; +uint8_t offset; +int num_avail; + +/* Copy up to 4 QEs from the current cache line into qes */ + num_avail =dlb_recv_qe (qm_port, qes, &offset); + +/* But don't process more than the user requested */ + num_avail =RTE_MIN (num_avail, max_num - num); + +dlb_inc_cq_idx (qm_port, num_avail); + +if (num_avail ==DLB_NUM_QES_PER_CACHE_LINE ) + num +=dlb_process_dequeue_four_qes (ev_port, + qm_port, + &events[num], + &qes[offset]); +else if (num_avail) + num +=dlb_process_dequeue_qes (ev_port, + qm_port, + &events[num], + &qes[offset], + num_avail); +else if ((timeout ==0 ) || (num >0 )) +/* Not waiting in any form, or 1+ events received? */ +break ; +else if (dlb_dequeue_wait (dlb, ev_port, qm_port, + timeout, start_ticks)) +break ; + } + + qm_port->owed_tokens += num; + +if (num && qm_port->token_pop_mode ==AUTO_POP ) +dlb_consume_qe_immediate (qm_port, num); + + ev_port->outstanding_releases += num; + +return num; +} + +static inline int +dlb_process_dequeue_qe(struct dlb_eventdev_port *ev_port __rte_unused, +struct dlb_port *qm_port, +struct rte_event *event, +struct dlb_dequeue_qe *qe) +{ +int sched_type_map[4 ] = { + [DLB_SCHED_ATOMIC ] =RTE_SCHED_TYPE_ATOMIC , + [DLB_SCHED_UNORDERED ] =RTE_SCHED_TYPE_PARALLEL , + [DLB_SCHED_ORDERED ] =RTE_SCHED_TYPE_ORDERED , + [DLB_SCHED_DIRECTED ] =RTE_SCHED_TYPE_ATOMIC , + }; +uint8_t *qid_mappings = qm_port->qid_mappings; + +DLB_LOG_DBG ("dequeue success, data = 0x%llx, qid=%d, event_type=%d, subevent=%d\npp_id = %d, sched_type = %d, qid = %d, err=%d\n" , + (long long )qe->data, qe->qid, + qe->u.event_type.major, + qe->u.event_type.sub, + qe->pp_id, qe->sched_type, qe->qid, qe->error); + +/* Fill in event information. +* Note that flow_id must be embedded in the data by +* the app, such as the mbuf RSS hash field if the data +* buffer is a mbuf. +*/ +if (unlikely (qe->error)) { +DLB_LOG_ERR ("QE error bit ON\n" ); +DLB_INC_STAT (ev_port->stats.traffic.rx_drop,1 ); +dlb_consume_qe_immediate (qm_port,1 ); +return 0 ; + } + + event->u64 = qe->data; + event->queue_id = qid_mappings[qe->qid]; + event->priority =DLB_TO_EV_PRIO ((uint8_t )qe->priority); + event->event_type = qe->u.event_type.major; + event->sub_event_type = qe->u.event_type.sub; + event->sched_type = sched_type_map[qe->sched_type]; +DLB_INC_STAT (ev_port->stats.rx_sched_cnt[qe->sched_type],1 ); + +DLB_INC_STAT (ev_port->stats.traffic.rx_ok,1 ); + +return 1 ; +} + +static inline int16_t +dlb_hw_dequeue_sparse (struct dlb_eventdev *dlb, +struct dlb_eventdev_port *ev_port, +struct rte_event *events, +uint16_t max_num, +uint64_t dequeue_timeout_ticks) +{ +uint64_t timeout; +uint64_t start_ticks =0ULL ; +struct dlb_port *qm_port; +int num =0 ; + + qm_port = &ev_port->qm_port; + +/* We have a special implementation for waiting. Wait can be: +* 1) no waiting at all +* 2) busy poll only +* 3) wait for interrupt. If wakeup and poll time +* has expired, then return to caller +* 4) umonitor/umwait repeatedly up to poll time +*/ + +/* If configured for per dequeue wait, then use wait value provided +* to this API. Otherwise we must use the global +* value from eventdev config time. +*/ +if (!dlb->global_dequeue_wait) + timeout = dequeue_timeout_ticks; +else + timeout = dlb->global_dequeue_wait_ticks; + +if (timeout) + start_ticks =rte_get_timer_cycles (); + +while (num < max_num) { +struct dlb_dequeue_qe qes[DLB_NUM_QES_PER_CACHE_LINE ]; +int num_avail; + +/* Copy up to 4 QEs from the current cache line into qes */ + num_avail =dlb_recv_qe_sparse (qm_port, qes); + +/* But don't process more than the user requested */ + num_avail =RTE_MIN (num_avail, max_num - num); + +dlb_inc_cq_idx (qm_port, num_avail <<2 ); + +if (num_avail ==DLB_NUM_QES_PER_CACHE_LINE ) + num +=dlb_process_dequeue_four_qes (ev_port, + qm_port, + &events[num], + &qes[0 ]); +else if (num_avail) + num +=dlb_process_dequeue_qes (ev_port, + qm_port, + &events[num], + &qes[0 ], + num_avail); +else if ((timeout ==0 ) || (num >0 )) +/* Not waiting in any form, or 1+ events received? */ +break ; +else if (dlb_dequeue_wait (dlb, ev_port, qm_port, + timeout, start_ticks)) +break ; + } + + qm_port->owed_tokens += num; + +if (num && qm_port->token_pop_mode ==AUTO_POP ) +dlb_consume_qe_immediate (qm_port, num); + + ev_port->outstanding_releases += num; + +return num; +} + static int16_t dlb_hw_map_ldb_qid_to_port(struct dlb_hw_dev *handle, uint32_t qm_port_id,
@@ -1964,6 +2658,181 @@ dlb_eventdev_configure(const struct rte_eventdev *dev)return 0 ; } +static void +dlb_event_release (struct dlb_eventdev *dlb, +uint8_t port_id, +int n) +{ +struct process_local_port_data *port_data; +struct dlb_eventdev_port *ev_port; +struct dlb_port *qm_port; +int i, cnt; + +if (port_id > dlb->num_ports) { +DLB_LOG_ERR ("Invalid port id %d in dlb-event_release\n" , + port_id); + rte_errno = -EINVAL ; +return ; + } + + ev_port = &dlb->ev_ports[port_id]; + qm_port = &ev_port->qm_port; + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + + cnt =0 ; + +if (qm_port->is_directed) { + cnt = n; + goto sw_credit_update; + } + +/* Since MOVDIR64B is weakly-ordered, use an SFENCE to ensure that +* application writes complete before enqueueing the release HCW. +*/ +rte_wmb (); + +for (i =0 ; i < n; i +=DLB_NUM_QES_PER_CACHE_LINE ) { +int j; + +/* Zero-out QEs */ + qm_port->qe4[0 ].cmd_byte =0 ; + qm_port->qe4[1 ].cmd_byte =0 ; + qm_port->qe4[2 ].cmd_byte =0 ; + qm_port->qe4[3 ].cmd_byte =0 ; + +for (j =0 ; j <DLB_NUM_QES_PER_CACHE_LINE && (i + j) < n; j++) + qm_port->qe4[j].cmd_byte =DLB_COMP_CMD_BYTE ; + + qm_port->issued_releases += j; + +if (j ==0 ) +break ; + +if (qm_port->token_pop_mode ==DELAYED_POP && j <4 && + qm_port->issued_releases >= qm_port->token_pop_thresh -1 ) { +dlb_construct_token_pop_qe (qm_port, j); + +/* Reset the releases counter for the next QE batch */ + qm_port->issued_releases -= qm_port->token_pop_thresh; + +/* When using delayed token pop mode, the initial token +* threshold is the full CQ depth. After the first +* token pop, we need to reset it to the dequeue_depth. +*/ + qm_port->token_pop_thresh = qm_port->dequeue_depth; + } + +dlb_hw_do_enqueue (qm_port, port_data); + + cnt += j; + } + +if (qm_port->token_pop_mode ==DELAYED_POP && + qm_port->issued_releases >= qm_port->token_pop_thresh -1 ) { +dlb_consume_qe_immediate (qm_port, qm_port->owed_tokens); + qm_port->issued_releases -= qm_port->token_pop_thresh; + qm_port->token_pop_thresh = qm_port->dequeue_depth; + } + +sw_credit_update: +/* each release returns one credit */ +if (!ev_port->outstanding_releases) +rte_panic ("%s: Outstanding releases underflowed.\n" , __func__); + ev_port->outstanding_releases -= cnt; + ev_port->inflight_credits += cnt; + +/* Replenish s/w credits if enough releases are performed */ +dlb_replenish_sw_credits (dlb, ev_port); +} + +static uint16_t +dlb_event_dequeue_burst (void *event_port,struct rte_event *ev,uint16_t num, +uint64_t wait) +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_port *qm_port = &ev_port->qm_port; +struct dlb_eventdev *dlb = ev_port->dlb; +struct process_local_port_data *port_data; +uint16_t cnt; + + rte_errno =0 ; + +RTE_ASSERT (ev_port->setup_done); +RTE_ASSERT (ev !=NULL ); + + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +if (!port_data->mmaped) +dlb_iface_port_mmap (qm_port); + +if (ev_port->implicit_release && ev_port->outstanding_releases >0 ) { +uint16_t out_rels = ev_port->outstanding_releases; + +dlb_event_release (dlb, ev_port->id, out_rels); + +DLB_INC_STAT (ev_port->stats.tx_implicit_rel, out_rels); + } + +if (qm_port->token_pop_mode ==DEFERRED_POP && + qm_port->owed_tokens) +dlb_consume_qe_immediate (qm_port, qm_port->owed_tokens); + + cnt =dlb_hw_dequeue (dlb, ev_port, ev, num, wait); + +DLB_INC_STAT (ev_port->stats.traffic.total_polls,1 ); +DLB_INC_STAT (ev_port->stats.traffic.zero_polls, ((cnt ==0 ) ?1 :0 )); +return cnt; +} + +static uint16_t +dlb_event_dequeue (void *event_port,struct rte_event *ev,uint64_t wait) +{ +return dlb_event_dequeue_burst (event_port, ev,1 , wait); +} + +static uint16_t +dlb_event_dequeue_burst_sparse (void *event_port,struct rte_event *ev, +uint16_t num,uint64_t wait) +{ +struct dlb_eventdev_port *ev_port = event_port; +struct dlb_port *qm_port = &ev_port->qm_port; +struct dlb_eventdev *dlb = ev_port->dlb; +struct process_local_port_data *port_data; +uint16_t cnt; + + rte_errno =0 ; + +RTE_ASSERT (ev_port->setup_done); +RTE_ASSERT (ev !=NULL ); + + port_data = &dlb_port[qm_port->id][PORT_TYPE (qm_port)]; + +if (!port_data->mmaped) +dlb_iface_port_mmap (qm_port); + +if (ev_port->implicit_release && ev_port->outstanding_releases >0 ) { +uint16_t out_rels = ev_port->outstanding_releases; + +dlb_event_release (dlb, ev_port->id, out_rels); + +DLB_INC_STAT (ev_port->stats.tx_implicit_rel, out_rels); + } + +if (qm_port->token_pop_mode ==DEFERRED_POP && + qm_port->owed_tokens) + cnt =dlb_hw_dequeue_sparse (dlb, ev_port, ev, num, wait); + +DLB_INC_STAT (ev_port->stats.traffic.total_polls,1 ); +DLB_INC_STAT (ev_port->stats.traffic.zero_polls, ((cnt ==0 ) ?1 :0 )); +return cnt; +} + +static uint16_t +dlb_event_dequeue_sparse (void *event_port,struct rte_event *ev,uint64_t wait) +{ +return dlb_event_dequeue_burst_sparse (event_port, ev,1 , wait); +} + static int dlb_event_queue_join_ldb(struct dlb_eventdev *dlb, struct dlb_eventdev_port *ev_port,
@@ -2907,6 +3776,8 @@ set_num_atm_inflights(const char *key __rte_unused,void dlb_entry_points_init (struct rte_eventdev *dev) { +struct dlb_eventdev *dlb; +static struct rte_eventdev_ops dlb_eventdev_entry_ops = { .dev_infos_get = dlb_eventdev_info_get, .dev_configure = dlb_eventdev_configure,
@@ -2937,7 +3808,15 @@ dlb_entry_points_init(struct rte_eventdev *dev) dev->enqueue_burst = dlb_event_enqueue_burst; dev->enqueue_new_burst = dlb_event_enqueue_new_burst; dev->enqueue_forward_burst = dlb_event_enqueue_forward_burst; + dev->dequeue = dlb_event_dequeue; + dev->dequeue_burst = dlb_event_dequeue_burst; + + dlb = dev->data->dev_private; +if (dlb->poll_mode ==DLB_CQ_POLL_MODE_SPARSE ) { + dev->dequeue = dlb_event_dequeue_sparse; + dev->dequeue_burst = dlb_event_dequeue_burst_sparse; + } } static void
Signature · 1 line
2.13.6
Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites
In reply to McDaniel, TimothyQuoted 8 lines · Timothy
The DLB hardware does not conform exactly to the eventdev interface.
1) It has a limit on the number of queues that may be linked to a port.
2) Some ports a further restricted to a maximum of 1 linked queue.
3) It does not (currently) have the ability to carry theflow_idas part of the event (QE) payload.Due to the above, we would like to propose the following enhancements.
Thanks, McDaniel, Good to see new HW PMD for eventdev.
+ Ray and Neil.
Hello McDaniel,
I assume this patchset is for v20.08. It is adding new elements in pubic structures. Have you checked the ABI breakage?
I will review the rest of the series if there is NO ABI breakage as we can not have the ABI breakage 20.08 version.
ABI validator
~~~~~~~~~~~~~~
1. meson build
2. Compile and install known stable abi libs i.e ToT. DESTDIR=$PWD/install-meson-stable ninja -C build install Compile and install with patches to be verified. DESTDIR=$PWD/install-meson-new ninja -C build install
3. Gen ABI for both
devtools/gen-abi.sh install-meson-stable devtools/gen-abi.sh install-meson-new
4. Run abi checker
devtools/check-abi.sh install-meson-stable install-meson-new
DPDK_ABI_REF_DIR=/build/dpdk/reference/ DPDK_ABI_REF_VERSION=v20.02 ./devtools/test-meson-builds.shDPDK_ABI_REF_DIR - needs an absolute path, for reasons that are still unclear to me.DPDK_ABI_REF_VERSION - you need to use the last DPDK release.
Quoted 817 lines
1) Add new fields to the
rte_event_dev_infostruct. These fields allow the device to advertize its capabilities so that applications can take the appropriate actions based on those capabilities.struct rte_event_dev_info { uint32_t max_event_port_links; /**< Maximum number of queues that can be linked to a single event * port by this device. */ uint8_t max_single_link_event_port_queue_pairs; /**< Maximum number of event ports and queues that are optimized for * (and only capable of) single-link configurations supported by this * device. These ports and queues are not accounted for in * max_event_ports or max_event_queues. */ }2) Add a new field to the
rte_event_dev_configstruct. This field allows the application to specify how many of its ports are limited to a single link, or will be used in single link mode./** Event device configuration structure */ struct rte_event_dev_config { uint8_t nb_single_link_event_port_queues; /**< Number of event ports and queues that will be singly-linked to * each other. These are a subset of the overall event ports and * queues; this value cannot exceed *nb_event_ports* or * *nb_event_queues*. If the device has ports and queues that are * optimized for single-link usage, this field is a hint for how many * to allocate; otherwise, regular event ports and queues can be used. */ }3) Replace the dedicated
implicit_release_disabledfield with a bit field of explicit port capabilities. Theimplicit_release_disablefunctionality is assiged to one bit, and a port-is-single-link-only attribute is assigned to other, with the remaining bits available for future assignment.* Event port configuration bitmap flags */ #define RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL (1ULL << 0) /**< Configure the port not to release outstanding events in * rte_event_dev_dequeue_burst(). If set, all events received through * the port must be explicitly released with RTE_EVENT_OP_RELEASE or * RTE_EVENT_OP_FORWARD. Must be unset if the device is not * RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. */ #define RTE_EVENT_PORT_CFG_SINGLE_LINK (1ULL << 1) /**< This event port links only to a single event queue. * * @see rte_event_port_setup(), rte_event_port_link() */ #define RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE 3 /** * The implicit release disable attribute of the port */ struct rte_event_port_conf { uint32_t event_port_cfg; /**< Port cfg flags(EVENT_PORT_CFG_) */ }4) Add UMWAIT/UMONITOR bit to
rte_cpuflags5) Added a new API that is useful for probing PCI devices.
/**
* @internal
* Wrapper for use by pci drivers as a .probe function to attach to a event
* interface. Same asrte_event_pmd_pci_probe, except caller can specify
* the name.
*/
static inline int
rte_event_pmd_pci_probe_named(structrte_pci_driver*pci_drv,
structrte_pci_device*pci_dev,size_tprivate_data_size,eventdev_pmd_pci_callback_tdevinit, const char *name);Change-Id I4cf00015296e2b3feca9886895765554730594beSigned-off-by McDaniel, Timothy <[email protected]>--- app/test-eventdev/evt_common.h | 1 + app/test-eventdev/test_order_atq.c | 4 ++ app/test-eventdev/test_order_common.c | 6 ++- app/test-eventdev/test_order_queue.c | 4 ++ app/test-eventdev/test_perf_atq.c | 1 + app/test-eventdev/test_perf_queue.c | 1 + app/test-eventdev/test_pipeline_atq.c | 1 + app/test-eventdev/test_pipeline_queue.c | 1 + app/test/test_eventdev.c | 4 +- drivers/event/dpaa2/dpaa2_eventdev.c | 2 +- drivers/event/octeontx/ssovf_evdev.c | 2 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/event/sw/sw_evdev.c | 5 +- drivers/event/sw/sw_evdev_selftest.c | 9 ++-- .../eventdev_pipeline/pipeline_worker_generic.c | 8 ++- examples/eventdev_pipeline/pipeline_worker_tx.c | 3 ++ examples/l2fwd-event/l2fwd_event_generic.c | 5 +- examples/l2fwd-event/l2fwd_event_internal_port.c | 5 +- examples/l3fwd/l3fwd_event_generic.c | 5 +- examples/l3fwd/l3fwd_event_internal_port.c | 5 +- lib/librte_eal/x86/include/rte_cpuflags.h | 1 + lib/librte_eal/x86/rte_cpuflags.c | 1 + lib/librte_eventdev/rte_event_eth_tx_adapter.c | 2 +- lib/librte_eventdev/rte_eventdev.c | 62 +++++++++++++++++++--- lib/librte_eventdev/rte_eventdev.h | 51 +++++++++++++++--- lib/librte_eventdev/rte_eventdev_pmd_pci.h | 54 +++++++++++++++++++ 26 files changed, 208 insertions(+), 37 deletions(-)app/test-eventdev/evt_common.h +1
-0@@ -169,6 +169,7 @@ evt_configure_eventdev(struct evt_options *opt, uint8_t nb_queues, .dequeue_timeout_ns = opt->deq_tmo_nsec, .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues = 0, .nb_events_limit = info.max_num_events, .nb_event_queue_flows = opt->nb_flows, .nb_event_port_dequeue_depth =app/test-eventdev/test_order_atq.c +4
-0@@ -34,6 +34,8 @@ order_atq_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.sub_event_type ==0 ) {/* stage 0 from producer */ order_atq_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)@@ -68,6 +70,8 @@ order_atq_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].sub_event_type ==0 ) {/*stage 0 */ order_atq_process_stage_0 (&ev[i]); }else if (ev[i].sub_event_type ==1 ) {/* stage 1 */ app/test-eventdev/test_order_common.c +5
-1@@ -49,6 +49,7 @@ order_producer(void *arg)const uint32_t flow = (uintptr_t )m % nb_flows;/* Maintain seq number per flow */ m->seqn = producer_flow_seq[flow]++; + m->udata64 = flow; ev.flow_id = flow; ev.mbuf = m;@@ -318,10 +319,11 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, opt->wkr_deq_dep = dev_info.max_event_port_dequeue_depth;/* port configuration */ -const struct rte_event_port_conf p_conf = { +struct rte_event_port_conf p_conf = { .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg =0 , };/* setup one port per worker, linking to all queues */ @@ -351,6 +353,8 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, p->queue_id =0 ; p->t = t; + p_conf.new_event_threshold /=2 ; + ret =rte_event_port_setup (opt->dev_id, port, &p_conf);if (ret) {evt_err ("failed to setup producer port %d" , port);app/test-eventdev/test_order_queue.c +4
-0@@ -34,6 +34,8 @@ order_queue_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)@@ -68,6 +70,8 @@ order_queue_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev[i]); }else if (ev[i].queue_id ==1 ) {/* from atomic queue */ app/test-eventdev/test_perf_atq.c +1
-0@@ -204,6 +204,7 @@ perf_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, 1 /* stride */, nb_queues,app/test-eventdev/test_perf_queue.c +1
-0@@ -219,6 +219,7 @@ perf_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, nb_stages /* stride */,app/test-eventdev/test_pipeline_atq.c +1
-0@@ -356,6 +356,7 @@ pipeline_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port)app/test-eventdev/test_pipeline_queue.c +1
-0@@ -379,6 +379,7 @@ pipeline_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port) {app/test/test_eventdev.c +2
-2@@ -559,10 +559,10 @@ test_eventdev_port_setup(void)if (!(info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE )) { pconf.enqueue_depth = info.max_event_port_enqueue_depth; - pconf.disable_implicit_release =1 ; + pconf.event_port_cfg =RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; ret =rte_event_port_setup (TEST_DEV_ID ,0 , &pconf);TEST_ASSERT (ret == -EINVAL ,"Expected -EINVAL, %d" , ret); - pconf.disable_implicit_release =0 ; + pconf.event_port_cfg =0 ; } ret = rte_event_port_setup(TEST_DEV_ID, info.max_event_ports,drivers/event/dpaa2/dpaa2_eventdev.c +1
-1@@ -537,7 +537,7 @@ dpaa2_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id,DPAA2_EVENT_MAX_PORT_DEQUEUE_DEPTH ; port_conf->enqueue_depth =DPAA2_EVENT_MAX_PORT_ENQUEUE_DEPTH ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static intdrivers/event/octeontx/ssovf_evdev.c +1
-1@@ -224,7 +224,7 @@ ssovf_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold = edev->max_num_events; port_conf->dequeue_depth =1 ; port_conf->enqueue_depth =1 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static voiddrivers/event/skeleton/skeleton_eventdev.c +1
-1@@ -209,7 +209,7 @@ skeleton_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =32 *1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static voiddrivers/event/sw/sw_evdev.c +3
-2@@ -175,7 +175,8 @@ sw_port_setup(struct rte_eventdev *dev, uint8_t port_id, } p->inflight_max = conf->new_event_threshold; - p->implicit_release = !conf->disable_implicit_release; + p->implicit_release = !(conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL );/* check if ring exists, same as rx_worker above */ snprintf(buf, sizeof(buf), "sw%d_p%u, %s", dev->data->dev_id,@@ -508,7 +509,7 @@ sw_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static intdrivers/event/sw/sw_evdev_selftest.c +5
-4@@ -172,7 +172,7 @@ create_ports(struct test *t, int num_ports) .new_event_threshold = 1024, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (num_ports >MAX_PORTS )return -1 ;@@ -1227,7 +1227,7 @@ port_reconfig_credits(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);@@ -1317,7 +1317,7 @@ port_single_lb_reconfig(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);@@ -3079,7 +3079,8 @@ worker_loopback(struct test *t, uint8_t disable_implicit_release)* only be initialized once - and this needs to be set for multiple runs */ conf.new_event_threshold =512 ; - conf.disable_implicit_release = disable_implicit_release; + conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (rte_event_port_setup (evdev,0 , &conf) <0 ) {printf ("Error setting up RX port\n" );examples/eventdev_pipeline/pipeline_worker_generic.c +6
-2@@ -129,6 +129,7 @@ setup_eventdev_generic(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =1 , .nb_events_limit =4096 , .nb_event_queue_flows =1024 , .nb_event_port_dequeue_depth =128 ,@@ -138,12 +139,13 @@ setup_eventdev_generic(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type, .priority =RTE_EVENT_DEV_PRIORITY_NORMAL , .nb_atomic_flows =1024 , - .nb_atomic_order_sequences =1024 , + .nb_atomic_order_sequences =1024 , };struct rte_event_queue_conf tx_q_conf = { .priority =RTE_EVENT_DEV_PRIORITY_HIGHEST ,@@ -167,7 +169,8 @@ setup_eventdev_generic(struct worker_data *worker_data) disable_implicit_release = (dev_info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE ); - wkr_p_conf.disable_implicit_release = disable_implicit_release; + wkr_p_conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (dev_info.max_num_events < config.nb_events_limit) config.nb_events_limit = dev_info.max_num_events;@@ -417,6 +420,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };if (adptr_p_conf.new_event_threshold > dev_info.max_num_events)examples/eventdev_pipeline/pipeline_worker_tx.c +3
-0@@ -436,6 +436,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =0 , .nb_events_limit =4096 , .nb_event_queue_flows =1024 , .nb_event_port_dequeue_depth =128 ,@@ -445,6 +446,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type,@@ -746,6 +748,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };init_ports (nb_ports);examples/l2fwd-event/l2fwd_event_generic.c +3
-2@@ -126,8 +126,9 @@ l2fwd_event_port_setup_generic(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;examples/l2fwd-event/l2fwd_event_internal_port.c +3
-2@@ -123,8 +123,9 @@ l2fwd_event_port_setup_internal_port(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {examples/l3fwd/l3fwd_event_generic.c +3
-2@@ -115,8 +115,9 @@ l3fwd_event_port_setup_generic(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;examples/l3fwd/l3fwd_event_internal_port.c +3
-2@@ -113,8 +113,9 @@ l3fwd_event_port_setup_internal_port(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {lib/librte_eal/x86/include/rte_cpuflags.h +1
-0@@ -130,6 +130,7 @@ enum rte_cpu_flag_t {RTE_CPUFLAG_CLDEMOTE ,/**< Cache Line Demote */ RTE_CPUFLAG_MOVDIRI ,/**< Direct Store Instructions */ RTE_CPUFLAG_MOVDIR64B ,/**< Direct Store Instructions 64B */ +RTE_CPUFLAG_UMWAIT ,/**< UMONITOR/UMWAIT */ RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection *//* The last item */ lib/librte_eal/x86/rte_cpuflags.c +1
-0@@ -137,6 +137,7 @@ const struct feature_entry rte_cpu_feature_table[] = { FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25) FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27) FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28) + FEAT_DEF(UMWAIT, 0x00000007, 0, RTE_REG_ECX, 5) FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8) };lib/librte_eventdev/rte_event_eth_tx_adapter.c +1
-1@@ -286,7 +286,7 @@ txa_service_conf_cb(uint8_t __rte_unused id, uint8_t dev_id,return ret; } - pc->disable_implicit_release =0 ; + pc->event_port_cfg =0 ; ret =rte_event_port_setup (dev_id, port_id, pc);if (ret) { RTE_EDEV_LOG_ERR("failed to setup event port %u\n",lib/librte_eventdev/rte_eventdev.c +55
-7@@ -437,9 +437,29 @@ rte_event_dev_configure(uint8_t dev_id, dev_id);return -EINVAL ; } -if (dev_conf->nb_event_queues > info.max_event_queues) { -RTE_EDEV_LOG_ERR ("%d nb_event_queues=%d > max_event_queues=%d" , - dev_id, dev_conf->nb_event_queues, info.max_event_queues); +if (dev_conf->nb_event_queues > info.max_event_queues + + info.max_single_link_event_port_queue_pairs) { +RTE_EDEV_LOG_ERR ("%d nb_event_queues=%d > max_event_queues=%d + max_single_link_event_port_queue_pairs=%d" , + dev_id, dev_conf->nb_event_queues, + info.max_event_queues, + info.max_single_link_event_port_queue_pairs); +return -EINVAL ; + } +if (dev_conf->nb_event_queues - + dev_conf->nb_single_link_event_port_queues > + info.max_event_queues) { +RTE_EDEV_LOG_ERR ("id%d nb_event_queues=%d - nb_single_link_event_port_queues=%d > max_event_queues=%d" , + dev_id, dev_conf->nb_event_queues, + dev_conf->nb_single_link_event_port_queues, + info.max_event_queues); +return -EINVAL ; + } +if (dev_conf->nb_single_link_event_port_queues > + dev_conf->nb_event_queues) { +RTE_EDEV_LOG_ERR ("dev%d nb_single_link_event_port_queues=%d > nb_event_queues=%d" , + dev_id, + dev_conf->nb_single_link_event_port_queues, + dev_conf->nb_event_queues);return -EINVAL ; }@@ -448,9 +468,31 @@ rte_event_dev_configure(uint8_t dev_id,RTE_EDEV_LOG_ERR ("dev%d nb_event_ports cannot be zero" , dev_id);return -EINVAL ; } -if (dev_conf->nb_event_ports > info.max_event_ports) { -RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d > max_event_ports= %d" , - dev_id, dev_conf->nb_event_ports, info.max_event_ports); +if (dev_conf->nb_event_ports > info.max_event_ports + + info.max_single_link_event_port_queue_pairs) { +RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d > max_event_ports=%d + max_single_link_event_port_queue_pairs=%d" , + dev_id, dev_conf->nb_event_ports, + info.max_event_ports, + info.max_single_link_event_port_queue_pairs); +return -EINVAL ; + } +if (dev_conf->nb_event_ports - + dev_conf->nb_single_link_event_port_queues + > info.max_event_ports) { +RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d - nb_single_link_event_port_queues=%d > max_event_ports=%d" , + dev_id, dev_conf->nb_event_ports, + dev_conf->nb_single_link_event_port_queues, + info.max_event_ports); +return -EINVAL ; + } + +if (dev_conf->nb_single_link_event_port_queues > + dev_conf->nb_event_ports) { +RTE_EDEV_LOG_ERR ( +"dev%d nb_single_link_event_port_queues=%d > nb_event_ports=%d" , + dev_id, + dev_conf->nb_single_link_event_port_queues, + dev_conf->nb_event_ports);return -EINVAL ; }@@ -737,7 +779,8 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id,return -EINVAL ; } -if (port_conf && port_conf->disable_implicit_release && +if (port_conf && + (port_conf->event_port_cfg &RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ) && !(dev->data->event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE )) { RTE_EDEV_LOG_ERR(@@ -809,6 +852,7 @@ rte_event_port_attr_get(uint8_t dev_id, uint8_t port_id, uint32_t attr_id, uint32_t *attr_value) {struct rte_eventdev *dev; +uint32_t config;if (!attr_value)return -EINVAL ;@@ -830,6 +874,10 @@ rte_event_port_attr_get(uint8_t dev_id, uint8_t port_id, uint32_t attr_id,case RTE_EVENT_PORT_ATTR_NEW_EVENT_THRESHOLD : *attr_value = dev->data->ports_cfg[port_id].new_event_threshold;break ; +case RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE : + config = dev->data->ports_cfg[port_id].event_port_cfg; + *attr_value = !!(config &RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); +break ;default :return -EINVAL ; };lib/librte_eventdev/rte_eventdev.h +44
-7@@ -291,6 +291,13 @@ struct rte_event;* single queue to each port or map a single queue to many port. */ +#define RTE_EVENT_DEV_CAP_CARRY_FLOW_ID (1ULL << 9) +/**< Event device is capable of carrying the flow ID from the enqueued +* event to the dequeued event. If the flag is set, the dequeued event's flow +* ID matches the corresponding enqueued event's flow ID. If the flag is not +* set, the dequeued event's flow ID field is uninitialized. +*/ +/* Event device priority levels */ #define RTE_EVENT_DEV_PRIORITY_HIGHEST 0/**< Highest priority expressed across eventdev subsystem @@ -380,6 +387,10 @@ struct rte_event_dev_info {* event port by this device. * A device that does not support bulk enqueue will set this as 1. */ +uint32_t max_event_port_links; +/**< Maximum number of queues that can be linked to a single event +* port by this device. +*/ int32_t max_num_events;/**< A *closed system* event dev has a limit on the number of events it * can manage at a time. An *open system* event dev does not have a @@ -387,6 +398,12 @@ struct rte_event_dev_info {*/ uint32_t event_dev_cap;/**< Event device capabilities(RTE_EVENT_DEV_CAP_)*/ +uint8_t max_single_link_event_port_queue_pairs; +/**< Maximum number of event ports and queues that are optimized for +* (and only capable of) single-link configurations supported by this +* device. These ports and queues are not accounted for in +* max_event_ports or max_event_queues. +*/ };/** @@ -494,6 +511,14 @@ struct rte_event_dev_config {*/ uint32_t event_dev_cfg;/**< Event device config flags(RTE_EVENT_DEV_CFG_)*/ +uint8_t nb_single_link_event_port_queues; +/**< Number of event ports and queues that will be singly-linked to +* each other. These are a subset of the overall event ports and +* queues; this value cannot exceed *nb_event_ports* or +* *nb_event_queues*. If the device has ports and queues that are +* optimized for single-link usage, this field is a hint for how many +* to allocate; otherwise, regular event ports and queues can be used. +*/ };/** @@ -671,6 +696,20 @@ rte_event_queue_attr_get(uint8_t dev_id, uint8_t queue_id, uint32_t attr_id,/* Event port specific APIs */ +/* Event port configuration bitmap flags */ +#define RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL (1ULL << 0) +/**< Configure the port not to release outstanding events in +* rte_event_dev_dequeue_burst(). If set, all events received through +* the port must be explicitly released with RTE_EVENT_OP_RELEASE or +* RTE_EVENT_OP_FORWARD. Must be unset if the device is not +* RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. +*/ +#define RTE_EVENT_PORT_CFG_SINGLE_LINK (1ULL << 1) +/**< This event port links only to a single event queue. +* +* @see rte_event_port_setup(), rte_event_port_link() +*/ +/** Event port configuration structure */ struct rte_event_port_conf {int32_t new_event_threshold;@@ -698,13 +737,7 @@ struct rte_event_port_conf {* which previously supplied to rte_event_dev_configure(). * Ignored when device is not RTE_EVENT_DEV_CAP_BURST_MODE capable. */ -uint8_t disable_implicit_release; -/**< Configure the port not to release outstanding events in -* rte_event_dev_dequeue_burst(). If true, all events received through -* the port must be explicitly released with RTE_EVENT_OP_RELEASE or -* RTE_EVENT_OP_FORWARD. Must be false when the device is not -* RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. -*/ +uint32_t event_port_cfg;/**< Port cfg flags(EVENT_PORT_CFG_) */ };/** @@ -769,6 +802,10 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id,* The new event threshold of the port */ #define RTE_EVENT_PORT_ATTR_NEW_EVENT_THRESHOLD 2 +/** +* The implicit release disable attribute of the port +*/ +#define RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE 3/** * Get an attribute from a port. lib/librte_eventdev/rte_eventdev_pmd_pci.h +54
-1@@ -88,6 +88,60 @@ rte_event_pmd_pci_probe(struct rte_pci_driver *pci_drv,return -ENXIO ; } +/** +* @internal +* Wrapper for use by pci drivers as a .probe function to attach to a event +* interface. Same as rte_event_pmd_pci_probe, except caller can specify +* the name. +*/ +static inline int +rte_event_pmd_pci_probe_named (struct rte_pci_driver *pci_drv, +struct rte_pci_device *pci_dev, +size_t private_data_size, +eventdev_pmd_pci_callback_t devinit, +const char *name) +{ +struct rte_eventdev *eventdev; + +int retval; + +if (devinit ==NULL ) +return -EINVAL ; + + eventdev =rte_event_pmd_allocate (name, + pci_dev->device.numa_node); +if (eventdev ==NULL ) +return -ENOMEM ; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { + eventdev->data->dev_private = +rte_zmalloc_socket ( +"eventdev private structure" , + private_data_size, +RTE_CACHE_LINE_SIZE , +rte_socket_id ()); + +if (eventdev->data->dev_private ==NULL ) +rte_panic ("Cannot allocate memzone for private " +"device data" ); + } + + eventdev->dev = &pci_dev->device; + +/* Invoke PMD device initialization function */ + retval =devinit (eventdev); +if (retval ==0 ) +return 0 ; + +RTE_EDEV_LOG_ERR ("driver %s: (vendor_id=0x%x device_id=0x%x)" +" failed" , pci_drv->driver.name, + (unsigned int ) pci_dev->id.vendor_id, + (unsigned int ) pci_dev->id.device_id); + +rte_event_pmd_release (eventdev); + +return -ENXIO ; +}/** * @internal -- 2.13.6
Re: [dpdk-dev] [PATCH 01/27] eventdev: dlb upstream prerequisites
Quoted 420 lines · Jerin Jacob
Quoted 7 lines · Timothy
The DLB hardware does not conform exactly to the eventdev interface.
1) It has a limit on the number of queues that may be linked to a port.
2) Some ports a further restricted to a maximum of 1 linked queue.
3) It does not (currently) have the ability to carry theflow_idas part of the event (QE) payload.Due to the above, we would like to propose the following enhancements.
Thanks, McDaniel, Good to see new HW PMD for eventdev.
+ Ray and Neil.
Hello McDaniel,
I assume this patchset is for v20.08. It is adding new elements in pubic structures. Have you checked the ABI breakage?I will review the rest of the series if there is NO ABI breakage as we can not have the ABI breakage 20.08 version.
ABI validator
~~~~~~~~~~~~~~
1. meson build
2. Compile and install known stable abi libs i.e ToT. DESTDIR=$PWD/install-meson-stable ninja -C build install Compile and install with patches to be verified. DESTDIR=$PWD/install-meson-new ninja -C build install
3. Gen ABI for both
devtools/gen-abi.shinstall-meson-stabledevtools/gen-abi.shinstall-meson-new
4. Run abi checker
devtools/check-abi.shinstall-meson-stable install-meson-newDPDK_ABI_REF_DIR=/build/dpdk/reference/ DPDK_ABI_REF_VERSION=v20.02
./devtools/test-meson-builds.shDPDK_ABI_REF_DIR- needs an absolute path, for reasons that are still unclear to me.DPDK_ABI_REF_VERSION- you need to use the last DPDK release.Quoted 378 lines
1) Add new fields to the
rte_event_dev_infostruct. These fields allow the device to advertize its capabilities so that applications can take the appropriate actions based on those capabilities.struct rte_event_dev_info { uint32_t max_event_port_links; /**< Maximum number of queues that can be linked to a single event * port by this device. */ uint8_t max_single_link_event_port_queue_pairs; /**< Maximum number of event ports and queues that are optimized for * (and only capable of) single-link configurations supported by this * device. These ports and queues are not accounted for in * max_event_ports or max_event_queues. */ }2) Add a new field to the
rte_event_dev_configstruct. This field allows the application to specify how many of its ports are limited to a single link, or will be used in single link mode./** Event device configuration structure */ struct rte_event_dev_config { uint8_t nb_single_link_event_port_queues; /**< Number of event ports and queues that will be singly-linked to * each other. These are a subset of the overall event ports and * queues; this value cannot exceed *nb_event_ports* or * *nb_event_queues*. If the device has ports and queues that are * optimized for single-link usage, this field is a hint for how many * to allocate; otherwise, regular event ports and queues can be used. */ }3) Replace the dedicated
implicit_release_disabledfield with a bit field of explicit port capabilities. Theimplicit_release_disablefunctionality is assiged to one bit, and a port-is-single-link-only attribute is assigned to other, with the remaining bits available for future assignment.* Event port configuration bitmap flags */ #define RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL (1ULL << 0) /**< Configure the port not to release outstanding events in * rte_event_dev_dequeue_burst(). If set, all events received through * the port must be explicitly released with RTE_EVENT_OP_RELEASE or * RTE_EVENT_OP_FORWARD. Must be unset if the device is not * RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. */ #define RTE_EVENT_PORT_CFG_SINGLE_LINK (1ULL << 1) /**< This event port links only to a single event queue. * * @see rte_event_port_setup(), rte_event_port_link() */ #define RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE 3 /** * The implicit release disable attribute of the port */ struct rte_event_port_conf { uint32_t event_port_cfg; /**< Port cfg flags(EVENT_PORT_CFG_) */ }4) Add UMWAIT/UMONITOR bit to
rte_cpuflags5) Added a new API that is useful for probing PCI devices.
/**
* @internal
* Wrapper for use by pci drivers as a .probe function to attach to a event
* interface. Same asrte_event_pmd_pci_probe, except caller can specify
* the name.
*/
static inline int
rte_event_pmd_pci_probe_named(structrte_pci_driver*pci_drv,
structrte_pci_device*pci_dev,size_tprivate_data_size,eventdev_pmd_pci_callback_tdevinit, const char *name);Change-Id I4cf00015296e2b3feca9886895765554730594beSigned-off-by McDaniel, Timothy <[email protected]>--- app/test-eventdev/evt_common.h | 1 + app/test-eventdev/test_order_atq.c | 4 ++ app/test-eventdev/test_order_common.c | 6 ++- app/test-eventdev/test_order_queue.c | 4 ++ app/test-eventdev/test_perf_atq.c | 1 + app/test-eventdev/test_perf_queue.c | 1 + app/test-eventdev/test_pipeline_atq.c | 1 + app/test-eventdev/test_pipeline_queue.c | 1 + app/test/test_eventdev.c | 4 +- drivers/event/dpaa2/dpaa2_eventdev.c | 2 +- drivers/event/octeontx/ssovf_evdev.c | 2 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/event/sw/sw_evdev.c | 5 +- drivers/event/sw/sw_evdev_selftest.c | 9 ++-- .../eventdev_pipeline/pipeline_worker_generic.c | 8 ++- examples/eventdev_pipeline/pipeline_worker_tx.c | 3 ++ examples/l2fwd-event/l2fwd_event_generic.c | 5 +- examples/l2fwd-event/l2fwd_event_internal_port.c | 5 +- examples/l3fwd/l3fwd_event_generic.c | 5 +- examples/l3fwd/l3fwd_event_internal_port.c | 5 +- lib/librte_eal/x86/include/rte_cpuflags.h | 1 + lib/librte_eal/x86/rte_cpuflags.c | 1 + lib/librte_eventdev/rte_event_eth_tx_adapter.c | 2 +- lib/librte_eventdev/rte_eventdev.c | 62 +++++++++++++++++++--- lib/librte_eventdev/rte_eventdev.h | 51 +++++++++++++++--- lib/librte_eventdev/rte_eventdev_pmd_pci.h | 54 +++++++++++++++++++ 26 files changed, 208 insertions(+), 37 deletions(-)app/test-eventdev/evt_common.h +1
-0@@ -169,6 +169,7 @@ evt_configure_eventdev(struct evt_options *opt, uint8_t nb_queues, .dequeue_timeout_ns = opt->deq_tmo_nsec, .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues = 0, .nb_events_limit = info.max_num_events, .nb_event_queue_flows = opt->nb_flows, .nb_event_port_dequeue_depth =app/test-eventdev/test_order_atq.c +4
-0@@ -34,6 +34,8 @@ order_atq_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.sub_event_type ==0 ) {/* stage 0 from producer */ order_atq_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)@@ -68,6 +70,8 @@ order_atq_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].sub_event_type ==0 ) {/*stage 0 */ order_atq_process_stage_0 (&ev[i]); }else if (ev[i].sub_event_type ==1 ) {/* stage 1 */ app/test-eventdev/test_order_common.c +5
-1@@ -49,6 +49,7 @@ order_producer(void *arg)const uint32_t flow = (uintptr_t )m % nb_flows;/* Maintain seq number per flow */ m->seqn = producer_flow_seq[flow]++; + m->udata64 = flow; ev.flow_id = flow; ev.mbuf = m;@@ -318,10 +319,11 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, opt->wkr_deq_dep = dev_info.max_event_port_dequeue_depth;/* port configuration */ -const struct rte_event_port_conf p_conf = { +struct rte_event_port_conf p_conf = { .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg =0 , };/* setup one port per worker, linking to all queues */ @@ -351,6 +353,8 @@ order_event_dev_port_setup(struct evt_test *test, struct evt_options *opt, p->queue_id =0 ; p->t = t; + p_conf.new_event_threshold /=2 ; + ret =rte_event_port_setup (opt->dev_id, port, &p_conf);if (ret) {evt_err ("failed to setup producer port %d" , port);app/test-eventdev/test_order_queue.c +4
-0@@ -34,6 +34,8 @@ order_queue_worker(void *arg)continue ; } + ev.flow_id = ev.mbuf->udata64; +if (ev.queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev); while (rte_event_enqueue_burst(dev_id, port, &ev, 1)@@ -68,6 +70,8 @@ order_queue_worker_burst(void *arg) }for (i =0 ; i < nb_rx; i++) { + ev[i].flow_id = ev[i].mbuf->udata64; +if (ev[i].queue_id ==0 ) {/* from ordered queue */ order_queue_process_stage_0 (&ev[i]); }else if (ev[i].queue_id ==1 ) {/* from atomic queue */ app/test-eventdev/test_perf_atq.c +1
-0@@ -204,6 +204,7 @@ perf_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, 1 /* stride */, nb_queues,app/test-eventdev/test_perf_queue.c +1
-0@@ -219,6 +219,7 @@ perf_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = dev_info.max_event_port_dequeue_depth, .new_event_threshold = dev_info.max_num_events, + .event_port_cfg = 0, }; ret = perf_event_dev_port_setup(test, opt, nb_stages /* stride */,app/test-eventdev/test_pipeline_atq.c +1
-0@@ -356,6 +356,7 @@ pipeline_atq_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port)app/test-eventdev/test_pipeline_queue.c +1
-0@@ -379,6 +379,7 @@ pipeline_queue_eventdev_setup(struct evt_test *test, struct evt_options *opt) .dequeue_depth = opt->wkr_deq_dep, .enqueue_depth = info.max_event_port_dequeue_depth, .new_event_threshold = info.max_num_events, + .event_port_cfg = 0, };if (!t->internal_port) {app/test/test_eventdev.c +2
-2@@ -559,10 +559,10 @@ test_eventdev_port_setup(void)if (!(info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE )) { pconf.enqueue_depth = info.max_event_port_enqueue_depth; - pconf.disable_implicit_release =1 ; + pconf.event_port_cfg =RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; ret =rte_event_port_setup (TEST_DEV_ID ,0 , &pconf);TEST_ASSERT (ret == -EINVAL ,"Expected -EINVAL, %d" , ret); - pconf.disable_implicit_release =0 ; + pconf.event_port_cfg =0 ; } ret = rte_event_port_setup(TEST_DEV_ID, info.max_event_ports,drivers/event/dpaa2/dpaa2_eventdev.c +1
-1@@ -537,7 +537,7 @@ dpaa2_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id,DPAA2_EVENT_MAX_PORT_DEQUEUE_DEPTH ; port_conf->enqueue_depth =DPAA2_EVENT_MAX_PORT_ENQUEUE_DEPTH ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static intdrivers/event/octeontx/ssovf_evdev.c +1
-1@@ -224,7 +224,7 @@ ssovf_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold = edev->max_num_events; port_conf->dequeue_depth =1 ; port_conf->enqueue_depth =1 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static voiddrivers/event/skeleton/skeleton_eventdev.c +1
-1@@ -209,7 +209,7 @@ skeleton_eventdev_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =32 *1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static voiddrivers/event/sw/sw_evdev.c +3
-2@@ -175,7 +175,8 @@ sw_port_setup(struct rte_eventdev *dev, uint8_t port_id, } p->inflight_max = conf->new_event_threshold; - p->implicit_release = !conf->disable_implicit_release; + p->implicit_release = !(conf->event_port_cfg & +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL );/* check if ring exists, same as rx_worker above */ snprintf(buf, sizeof(buf), "sw%d_p%u, %s", dev->data->dev_id,@@ -508,7 +509,7 @@ sw_port_def_conf(struct rte_eventdev *dev, uint8_t port_id, port_conf->new_event_threshold =1024 ; port_conf->dequeue_depth =16 ; port_conf->enqueue_depth =16 ; - port_conf->disable_implicit_release =0 ; + port_conf->event_port_cfg =0 ; } static intdrivers/event/sw/sw_evdev_selftest.c +5
-4@@ -172,7 +172,7 @@ create_ports(struct test *t, int num_ports) .new_event_threshold = 1024, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (num_ports >MAX_PORTS )return -1 ;@@ -1227,7 +1227,7 @@ port_reconfig_credits(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);@@ -1317,7 +1317,7 @@ port_single_lb_reconfig(struct test *t) .new_event_threshold = 128, .dequeue_depth = 32, .enqueue_depth = 64, - .disable_implicit_release = 0, + .event_port_cfg = 0, };if (rte_event_port_setup (evdev,0 , &port_conf) <0 ) {printf ("%d Error setting up port\n" , __LINE__);@@ -3079,7 +3079,8 @@ worker_loopback(struct test *t, uint8_t disable_implicit_release)* only be initialized once - and this needs to be set for multiple runs */ conf.new_event_threshold =512 ; - conf.disable_implicit_release = disable_implicit_release; + conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (rte_event_port_setup (evdev,0 , &conf) <0 ) {printf ("Error setting up RX port\n" );examples/eventdev_pipeline/pipeline_worker_generic.c +1
-0@@ -129,6 +129,7 @@ setup_eventdev_generic(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =1 ,
Quoted 8 lines
examples/eventdev_pipeline/pipeline_worker_generic.c setup_eventdev_generic(struct worker_data *worker_data) · line 142
Quoted 8 lines
examples/eventdev_pipeline/pipeline_worker_generic.c +1
-0.nb_events_limit = 4096, .nb_event_queue_flows = 1024, .nb_event_port_dequeue_depth = 128,@@ -138,12 +139,13 @@ setup_eventdev_generic(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0,
No need to set this value; it's guaranteed to be 0 anyways. You might argue you do it for readability, but two other fields of that struct is already implicitly initialized.
This would apply to other of your changes as well.
Quoted 266 lines
Quoted 266 lines
examples/eventdev_pipeline/pipeline_worker_generic.c +4
-2};struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type, .priority =RTE_EVENT_DEV_PRIORITY_NORMAL , .nb_atomic_flows =1024 , - .nb_atomic_order_sequences =1024 , + .nb_atomic_order_sequences =1024 , };struct rte_event_queue_conf tx_q_conf = { .priority =RTE_EVENT_DEV_PRIORITY_HIGHEST ,@@ -167,7 +169,8 @@ setup_eventdev_generic(struct worker_data *worker_data) disable_implicit_release = (dev_info.event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE ); - wkr_p_conf.disable_implicit_release = disable_implicit_release; + wkr_p_conf.event_port_cfg = disable_implicit_release ? +RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL :0 ;if (dev_info.max_num_events < config.nb_events_limit) config.nb_events_limit = dev_info.max_num_events;@@ -417,6 +420,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };if (adptr_p_conf.new_event_threshold > dev_info.max_num_events)examples/eventdev_pipeline/pipeline_worker_tx.c +3
-0@@ -436,6 +436,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data)struct rte_event_dev_config config = { .nb_event_queues = nb_queues, .nb_event_ports = nb_ports, + .nb_single_link_event_port_queues =0 , .nb_events_limit =4096 , .nb_event_queue_flows =1024 , .nb_event_port_dequeue_depth =128 ,@@ -445,6 +446,7 @@ setup_eventdev_worker_tx_enq(struct worker_data *worker_data) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };struct rte_event_queue_conf wkr_q_conf = { .schedule_type = cdata.queue_type,@@ -746,6 +748,7 @@ init_adapters(uint16_t nb_ports) .dequeue_depth = cdata.worker_cq_depth, .enqueue_depth = 64, .new_event_threshold = 4096, + .event_port_cfg = 0, };init_ports (nb_ports);examples/l2fwd-event/l2fwd_event_generic.c +3
-2@@ -126,8 +126,9 @@ l2fwd_event_port_setup_generic(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;examples/l2fwd-event/l2fwd_event_internal_port.c +3
-2@@ -123,8 +123,9 @@ l2fwd_event_port_setup_internal_port(struct l2fwd_resources *rsrc)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {examples/l3fwd/l3fwd_event_generic.c +3
-2@@ -115,8 +115,9 @@ l3fwd_event_port_setup_generic(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ; evt_rsrc->deq_depth = def_p_conf.dequeue_depth; for (event_p_id = 0; event_p_id < evt_rsrc->evp.nb_ports;examples/l3fwd/l3fwd_event_internal_port.c +3
-2@@ -113,8 +113,9 @@ l3fwd_event_port_setup_internal_port(void)if (def_p_conf.enqueue_depth < event_p_conf.enqueue_depth) event_p_conf.enqueue_depth = def_p_conf.enqueue_depth; - event_p_conf.disable_implicit_release = - evt_rsrc->disable_implicit_release; + event_p_conf.event_port_cfg =0 ; +if (evt_rsrc->disable_implicit_release) + event_p_conf.event_port_cfg |=RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ;for (event_p_id =0 ; event_p_id < evt_rsrc->evp.nb_ports; event_p_id++) {lib/librte_eal/x86/include/rte_cpuflags.h +1
-0@@ -130,6 +130,7 @@ enum rte_cpu_flag_t {RTE_CPUFLAG_CLDEMOTE ,/**< Cache Line Demote */ RTE_CPUFLAG_MOVDIRI ,/**< Direct Store Instructions */ RTE_CPUFLAG_MOVDIR64B ,/**< Direct Store Instructions 64B */ +RTE_CPUFLAG_UMWAIT ,/**< UMONITOR/UMWAIT */ RTE_CPUFLAG_AVX512VP2INTERSECT, /**< AVX512 Two Register Intersection *//* The last item */ lib/librte_eal/x86/rte_cpuflags.c +1
-0@@ -137,6 +137,7 @@ const struct feature_entry rte_cpu_feature_table[] = { FEAT_DEF(CLDEMOTE, 0x00000007, 0, RTE_REG_ECX, 25) FEAT_DEF(MOVDIRI, 0x00000007, 0, RTE_REG_ECX, 27) FEAT_DEF(MOVDIR64B, 0x00000007, 0, RTE_REG_ECX, 28) + FEAT_DEF(UMWAIT, 0x00000007, 0, RTE_REG_ECX, 5) FEAT_DEF(AVX512VP2INTERSECT, 0x00000007, 0, RTE_REG_EDX, 8) };lib/librte_eventdev/rte_event_eth_tx_adapter.c +1
-1@@ -286,7 +286,7 @@ txa_service_conf_cb(uint8_t __rte_unused id, uint8_t dev_id,return ret; } - pc->disable_implicit_release =0 ; + pc->event_port_cfg =0 ; ret =rte_event_port_setup (dev_id, port_id, pc);if (ret) { RTE_EDEV_LOG_ERR("failed to setup event port %u\n",lib/librte_eventdev/rte_eventdev.c +55
-7@@ -437,9 +437,29 @@ rte_event_dev_configure(uint8_t dev_id, dev_id);return -EINVAL ; } -if (dev_conf->nb_event_queues > info.max_event_queues) { -RTE_EDEV_LOG_ERR ("%d nb_event_queues=%d > max_event_queues=%d" , - dev_id, dev_conf->nb_event_queues, info.max_event_queues); +if (dev_conf->nb_event_queues > info.max_event_queues + + info.max_single_link_event_port_queue_pairs) { +RTE_EDEV_LOG_ERR ("%d nb_event_queues=%d > max_event_queues=%d + max_single_link_event_port_queue_pairs=%d" , + dev_id, dev_conf->nb_event_queues, + info.max_event_queues, + info.max_single_link_event_port_queue_pairs); +return -EINVAL ; + } +if (dev_conf->nb_event_queues - + dev_conf->nb_single_link_event_port_queues > + info.max_event_queues) { +RTE_EDEV_LOG_ERR ("id%d nb_event_queues=%d - nb_single_link_event_port_queues=%d > max_event_queues=%d" , + dev_id, dev_conf->nb_event_queues, + dev_conf->nb_single_link_event_port_queues, + info.max_event_queues); +return -EINVAL ; + } +if (dev_conf->nb_single_link_event_port_queues > + dev_conf->nb_event_queues) { +RTE_EDEV_LOG_ERR ("dev%d nb_single_link_event_port_queues=%d > nb_event_queues=%d" , + dev_id, + dev_conf->nb_single_link_event_port_queues, + dev_conf->nb_event_queues);return -EINVAL ; }@@ -448,9 +468,31 @@ rte_event_dev_configure(uint8_t dev_id,RTE_EDEV_LOG_ERR ("dev%d nb_event_ports cannot be zero" , dev_id);return -EINVAL ; } -if (dev_conf->nb_event_ports > info.max_event_ports) { -RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d > max_event_ports= %d" , - dev_id, dev_conf->nb_event_ports, info.max_event_ports); +if (dev_conf->nb_event_ports > info.max_event_ports + + info.max_single_link_event_port_queue_pairs) { +RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d > max_event_ports=%d + max_single_link_event_port_queue_pairs=%d" , + dev_id, dev_conf->nb_event_ports, + info.max_event_ports, + info.max_single_link_event_port_queue_pairs); +return -EINVAL ; + } +if (dev_conf->nb_event_ports - + dev_conf->nb_single_link_event_port_queues + > info.max_event_ports) { +RTE_EDEV_LOG_ERR ("id%d nb_event_ports=%d - nb_single_link_event_port_queues=%d > max_event_ports=%d" , + dev_id, dev_conf->nb_event_ports, + dev_conf->nb_single_link_event_port_queues, + info.max_event_ports); +return -EINVAL ; + } + +if (dev_conf->nb_single_link_event_port_queues > + dev_conf->nb_event_ports) { +RTE_EDEV_LOG_ERR ( +"dev%d nb_single_link_event_port_queues=%d > nb_event_ports=%d" , + dev_id, + dev_conf->nb_single_link_event_port_queues, + dev_conf->nb_event_ports);return -EINVAL ; }@@ -737,7 +779,8 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id,return -EINVAL ; } -if (port_conf && port_conf->disable_implicit_release && +if (port_conf && + (port_conf->event_port_cfg &RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ) && !(dev->data->event_dev_cap &RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE )) { RTE_EDEV_LOG_ERR(@@ -809,6 +852,7 @@ rte_event_port_attr_get(uint8_t dev_id, uint8_t port_id, uint32_t attr_id, uint32_t *attr_value) {struct rte_eventdev *dev; +uint32_t config;if (!attr_value)return -EINVAL ;@@ -830,6 +874,10 @@ rte_event_port_attr_get(uint8_t dev_id, uint8_t port_id, uint32_t attr_id,case RTE_EVENT_PORT_ATTR_NEW_EVENT_THRESHOLD : *attr_value = dev->data->ports_cfg[port_id].new_event_threshold;break ; +case RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE : + config = dev->data->ports_cfg[port_id].event_port_cfg; + *attr_value = !!(config &RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL ); +break ;default :return -EINVAL ; };lib/librte_eventdev/rte_eventdev.h +0
-0@@ -291,6 +291,13 @@ struct rte_event;* single queue to each port or map a single queue to many port. */
Quoted 7 lines
lib/librte_eventdev/rte_eventdev.h struct rte_event; · line 300
Quoted 7 lines
lib/librte_eventdev/rte_eventdev.h +7
-0+#define RTE_EVENT_DEV_CAP_CARRY_FLOW_ID (1ULL << 9) +/**< Event device is capable of carrying the flow ID from the enqueued +* event to the dequeued event. If the flag is set, the dequeued event's flow +* ID matches the corresponding enqueued event's flow ID. If the flag is not +* set, the dequeued event's flow ID field is uninitialized. +*/ +
The dequeued event's value should be undefined, to let an implementation overwrite an existing value. Replace "is capable of carrying" with "carries".
Is "maintain" better than "carry"? Or "preserve". I don't know.
Quoted 11 lines
lib/librte_eventdev/rte_eventdev.h struct rte_event_dev_info { · line 393
Quoted 11 lines
lib/librte_eventdev/rte_eventdev.h +4
-0/* Event device priority levels */ #define RTE_EVENT_DEV_PRIORITY_HIGHEST 0/**< Highest priority expressed across eventdev subsystem @@ -380,6 +387,10 @@ struct rte_event_dev_info {* event port by this device. * A device that does not support bulk enqueue will set this as 1. */ +uint32_t max_event_port_links; +/**< Maximum number of queues that can be linked to a single event +* port by this device. +*/
Eventdev API supports 255 queues, so you should use an uint8_t here.
Quoted 146 lines
Quoted 146 lines
lib/librte_eventdev/rte_eventdev.h +33
-7int32_t max_num_events;/**< A *closed system* event dev has a limit on the number of events it * can manage at a time. An *open system* event dev does not have a @@ -387,6 +398,12 @@ struct rte_event_dev_info {*/ uint32_t event_dev_cap;/**< Event device capabilities(RTE_EVENT_DEV_CAP_)*/ +uint8_t max_single_link_event_port_queue_pairs; +/**< Maximum number of event ports and queues that are optimized for +* (and only capable of) single-link configurations supported by this +* device. These ports and queues are not accounted for in +* max_event_ports or max_event_queues. +*/ };/** @@ -494,6 +511,14 @@ struct rte_event_dev_config {*/ uint32_t event_dev_cfg;/**< Event device config flags(RTE_EVENT_DEV_CFG_)*/ +uint8_t nb_single_link_event_port_queues; +/**< Number of event ports and queues that will be singly-linked to +* each other. These are a subset of the overall event ports and +* queues; this value cannot exceed *nb_event_ports* or +* *nb_event_queues*. If the device has ports and queues that are +* optimized for single-link usage, this field is a hint for how many +* to allocate; otherwise, regular event ports and queues can be used. +*/ };/** @@ -671,6 +696,20 @@ rte_event_queue_attr_get(uint8_t dev_id, uint8_t queue_id, uint32_t attr_id,/* Event port specific APIs */ +/* Event port configuration bitmap flags */ +#define RTE_EVENT_PORT_CFG_DISABLE_IMPL_REL (1ULL << 0) +/**< Configure the port not to release outstanding events in +* rte_event_dev_dequeue_burst(). If set, all events received through +* the port must be explicitly released with RTE_EVENT_OP_RELEASE or +* RTE_EVENT_OP_FORWARD. Must be unset if the device is not +* RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. +*/ +#define RTE_EVENT_PORT_CFG_SINGLE_LINK (1ULL << 1) +/**< This event port links only to a single event queue. +* +* @see rte_event_port_setup(), rte_event_port_link() +*/ +/** Event port configuration structure */ struct rte_event_port_conf {int32_t new_event_threshold;@@ -698,13 +737,7 @@ struct rte_event_port_conf {* which previously supplied to rte_event_dev_configure(). * Ignored when device is not RTE_EVENT_DEV_CAP_BURST_MODE capable. */ -uint8_t disable_implicit_release; -/**< Configure the port not to release outstanding events in -* rte_event_dev_dequeue_burst(). If true, all events received through -* the port must be explicitly released with RTE_EVENT_OP_RELEASE or -* RTE_EVENT_OP_FORWARD. Must be false when the device is not -* RTE_EVENT_DEV_CAP_IMPLICIT_RELEASE_DISABLE capable. -*/ +uint32_t event_port_cfg;/**< Port cfg flags(EVENT_PORT_CFG_) */ };/** @@ -769,6 +802,10 @@ rte_event_port_setup(uint8_t dev_id, uint8_t port_id,* The new event threshold of the port */ #define RTE_EVENT_PORT_ATTR_NEW_EVENT_THRESHOLD 2 +/** +* The implicit release disable attribute of the port +*/ +#define RTE_EVENT_PORT_ATTR_IMPLICIT_RELEASE_DISABLE 3/** * Get an attribute from a port. lib/librte_eventdev/rte_eventdev_pmd_pci.h +54
-1@@ -88,6 +88,60 @@ rte_event_pmd_pci_probe(struct rte_pci_driver *pci_drv,return -ENXIO ; } +/** +* @internal +* Wrapper for use by pci drivers as a .probe function to attach to a event +* interface. Same as rte_event_pmd_pci_probe, except caller can specify +* the name. +*/ +static inline int +rte_event_pmd_pci_probe_named (struct rte_pci_driver *pci_drv, +struct rte_pci_device *pci_dev, +size_t private_data_size, +eventdev_pmd_pci_callback_t devinit, +const char *name) +{ +struct rte_eventdev *eventdev; + +int retval; + +if (devinit ==NULL ) +return -EINVAL ; + + eventdev =rte_event_pmd_allocate (name, + pci_dev->device.numa_node); +if (eventdev ==NULL ) +return -ENOMEM ; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { + eventdev->data->dev_private = +rte_zmalloc_socket ( +"eventdev private structure" , + private_data_size, +RTE_CACHE_LINE_SIZE , +rte_socket_id ()); + +if (eventdev->data->dev_private ==NULL ) +rte_panic ("Cannot allocate memzone for private " +"device data" ); + } + + eventdev->dev = &pci_dev->device; + +/* Invoke PMD device initialization function */ + retval =devinit (eventdev); +if (retval ==0 ) +return 0 ; + +RTE_EDEV_LOG_ERR ("driver %s: (vendor_id=0x%x device_id=0x%x)" +" failed" , pci_drv->driver.name, + (unsigned int ) pci_dev->id.vendor_id, + (unsigned int ) pci_dev->id.device_id); + +rte_event_pmd_release (eventdev); + +return -ENXIO ; +}/** * @internal -- 2.13.6