Skip to contentSkip to conversation

graph: introduce graph subsystem

PATCH Superseded [email protected] 11 participants 219 messages Mar 18, 2020 – May 12, 2020 Page 1 of 5

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover graph: introduce graph subsystem

From: Jerin Jacob <[email protected]>

It is the v1 version of the DPDK graph support based on the following RFC http://patches.dpdk.org/cover/65432/

This patch set contains an implementation of graph architecture for packet processing using DPDK primitives.

Using graph traversal for packet processing is a proven architecture that has been implemented in various open source libraries.

Graph architecture for packet processing enables abstracting the data processing functions as “nodes” and “links” them together to create a complex “graph” to create reusable/modular data processing functions.

The patchset further includes performance enhancements and modularity to the DPDK as discussed in more detail below.

RFC..v1:
--------

1) Split the patch to more logical ones for review.
2) Added doxygen comments for the API
3) Code cleanup
4) Additional performance improvements.
Delta between l3fwd and l3fwd-graph is negligible now. (~1%) on octeontx2.
5) Added SIMD routines for x86 in additional to arm64.

Pending items (Will be addressed in v2)
-------------------------------------------------
1) Add documentation as a patch for programming guide and l3fwd-graph user guide.

Addional nodes planned for v20.08
----------------------------------
1) Packet classification node
2) Support for IPV6 LPM node

This patchset contains
-----------------------------
1) The API definition to "create" nodes and "link" together to create a "graph" for packet processing. See, lib/librte_graph/rte_graph.h

2) The Fast path API definition for the graph walker and enqueue function used by the workers. See, lib/librte_graph/rte_graph_worker.h

3) Optimized SW implementation for (1) and (2). See, lib/librte_graph/

4) Test case to verify the graph infrastructure functionality See, app/test/test_graph.c

5) Performance test cases to evaluate the cost of graph walker and nodes enqueue fast-path function for various combinations.

See app/test/test_graph_perf.c

6) Packet processing nodes(Null, Rx, Tx, Pkt drop, IPV4 rewrite, IPv4 lookup)
using graph infrastructure. See lib/librte_node/*

7) An example application to showcase l3fwd
(functionality same as existing examples/l3fwd) using graph infrastructure and use packets processing nodes (item (6)). See examples/l3fwd-graph/.

Performance
-----------
1) Graph walk and node enqueue overhead can be tested with performance test case application [1]
# If all packets go from a node to another node (we call it as # "homerun") then it will be just a pointer swap for a burst of packets. # In the worst case, a couple of handful cycles to move an object from a node to another node.

2) Performance comparison with existing l3fwd (The complete static code with out any nodes) vs modular l3fwd-graph with 5 nodes (ip4_lookup, ip4_rewrite, ethdev_tx, ethdev_rx, pkt_drop). Here is graphical representation of the l3fwd-graph as Graphviz dot file:
http://bit.ly/39UPPGm

# l3fwd-graph performance is -1.2% wrt static l3fwd.

# We have simulated the similar test with existing librte_pipeline

# application [4].
ip_pipline application is -48.62% wrt static l3fwd.

The above results are on octeontx2. It may vary on other platforms. The platforms with higher L1 and L2 caches will have further better performance.

Tested architectures:
--------------------
1) AArch64
2) X86

Graph library Features
----------------------
1) Nodes as plugins
2) Support for out of tree nodes
3) Multi-process support.
4) Low overhead graph walk and node enqueue
5) Low overhead statistics collection infrastructure
6) Support to export the graph as a Graphviz dot file. See rte_graph_export()
Example of exported graph: http://bit.ly/2PqbqOy
7) Allow having another graph walk implementation
in the future by segregating the fast path and slow path code.

Advantages of Graph architecture:
---------------------------------

1) Memory latency is the enemy for high-speed packet processing, moving the similar packet processing code to a node will reduce the I cache and D caches misses.
2) Exploits the probability that most packets will follow the same nodes in the graph.
3) Allow SIMD instructions for packet processing of the node.
4) The modular scheme allows having reusable nodes for the consumers.
5) The modular scheme allows us to abstract the vendor HW specific optimizations as a node.

Why Graph architecture
-----------------------
1) We believe, Graph architecture provides the best performance for reusable/modular packet processing framework.
Since DPDK does not have it, it is good to have it in DPDK.

2) Based on our experience, NPU HW accelerates are so different than one vendor
to another vendor. Going forward, We believe, API abstraction may not be enough
abstract the difference in HW. The Vendor-specific nodes can abstract the HW
differences and reuse generic the nodes as needed.
This would help both the silicon vendors and DPDK end users.

3) The framework enables the protocol stack as use native mbuf for graph processing to avoid any conversion between the formats for better performance.

4) DPDK becomes the "goto library" for userspace HW acceleration. It is good to have native Graph packet processing library in DPDK.

5) Obviously, Our customers are interested in Graph library in DPDK :-)

Identified tweaking for better performance on different targets
---------------------------------------------------------------
1) Test with various burst size values (256, 128, 64, 32) using CONFIG_RTE_GRAPH_BURST_SIZE config option.
Based on our testing, on x86 and arm64 servers, The sweet spot is 256 burst size.
While on arm64 embedded SoCs, it is either 64 or 128.

2) Disable node statistics (use CONFIG_RTE_LIBRTE_GRAPH_STATS config option)
if not needed.

3) Use arm64 optimized memory copy for arm64 architecture by selecting CONFIG_RTE_ARCH_ARM64_MEMCPY.

Commands to run tests
---------------------

[1]
perf test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

[2]
functionality test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

[3]
l3fwd-graph:
./l3fwd-graph -c 0x100 -- -p 0x3 --config="(0, 0, 8)" -P

[4]

# ./ip_pipeline --c 0xff0000 -- -s route.cli

Route.cli: (Copy paste to the shell to avoid dos format issues)

https://pastebin.com/raw/B4Ktx7TT

Programming guide and API walk-through
--------------------------------------
# Anatomy of Node:
~~~~~~~~~~~~~~~~~
See the
https://github.com/jerinjacobk/share/blob/master/Anatomy_of_a_node.svg

The above diagram depicts the anatomy of a node. The node is the basic building block of the graph framework.

A node consists of:
a) process():

The callback function will be invoked by worker thread using rte_graph_walk() function when there is data to be processed by the node.
A graph node process the function using process() and enqueue to next downstream node using rte_node_enqueue*() function.

b) Context memory:

It is memory allocated by the library to store the node-specific context information. which will be used by process(), init(), fini() callbacks.

c) init():

The callback function which will be invoked by rte_graph_create() on when a node
gets attached to a graph.

d) fini():

The callback function which will be invoked by rte_graph_destroy() on when a node
gets detached to a graph.

e) Node name:

It is the name of the node. When a node registers to graph library, the library
gives the ID as rte_node_t type. Both ID or Name shall be used lookup the node.
rte_node_from_name(), rte_node_id_to_name() are the node lookup functions.

f) nb_edges:

Number of downstream nodes connected to this node. The next_nodes[] stores the
downstream nodes objects. rte_node_edge_update() and rte_node_edge_shrink()
functions shall be used to update the next_node[] objects. Consumers of the node
APIs are free to update the next_node[] objects till rte_graph_create() invoked.

g) next_node[]:

The dynamic array to store the downstream nodes connected to this node.

# Node creation and registration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) Node implementer creates the node by implementing ops and attributes of
'struct rte_node_register'
b) The library registers the node by invoking RTE_NODE_REGISTER on library load
using the constructor scheme.
The constructor scheme used here to support multi-process.

# Link the Nodes to create the graph topology
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See the
https://github.com/jerinjacobk/share/blob/master/Link_the_nodes.svg

The above diagram shows a graph topology after linking the N nodes.

Once nodes are available to the program, Application or node public API functions
can links them together to create a complex packet processing graph.

There are multiple different types of strategies to link the nodes.

Method a) Provide the next_nodes[] at the node registration time. See 'struct rte_node_register::nb_edges'. This is a use case to address the static
node scheme where one knows upfront the next_nodes[] of the node.

Method b) Use rte_node_edge_get(), rte_node_edge_update(), rte_node_edge_shrink() to
Update the next_nodes[] links for the node dynamically.

Method c) Use rte_node_clone() to clone a already existing node. When rte_node_clone() invoked, The library, would clone all the attributes
of the node and creates a new one. The name for cloned node shall be "parent_node_name-user_provided_name". This method enables the use case of Rx and Tx
nodes where multiple of those nodes need to be cloned based on the number of CPU
available in the system. The cloned nodes will be identical, except the "context memory".
Context memory will have information of port, queue pair incase of Rx and Tx ethdev nodes.

# Create the graph object
~~~~~~~~~~~~~~~~~~~~~~~~~
Now that the nodes are linked, Its time to create a graph by including the required nodes. The application can provide a set of node patterns to
form a graph object.
The fnmatch() API used underneath for the pattern matching to include the required nodes.

The rte_graph_create() API shall be used to create the graph.

Example of a graph object creation:

{"ethdev_rx_0_0", ipv4-*, ethdev_tx_0_*"}

In the above example, A graph object will be created with ethdev Rx node of port 0 and queue 0, all ipv4* nodes in the system, and ethdev tx node of port 0 with all queues.

# Multi core graph processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the current graph library implementation, specifically, rte_graph_walk() and rte_node_enqueue* fast path API functions are designed to work on single-core to have better performance. The fast path API works on graph object, So the multi-core graph processing strategy would be to create graph object PER WORKER.

# In fast path:
~~~~~~~~~~~~~~~

Typical fast-path code looks like below, where the application gets the fast-path graph object through rte_graph_lookup() on the worker thread and run the rte_graph_walk() in a tight loop.

struct rte_graph *graph = rte_graph_lookup("worker0");

while (!done) {
    rte_graph_walk(graph);
}

# Context update when graph walk in action
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The fast-path object for the node is struct rte_node.

It may be possible that in slow-path or after the graph walk-in action, the user needs to update the context of the node hence access to struct rte_node * memory.

rte_graph_foreach_node(), rte_graph_node_get(), rte_graph_node_get_by_name()
APIs can be used to to get the struct rte_node*. rte_graph_foreach_node() iterator
function works on struct rte_graph * fast-path graph object while others works on graph ID or name.

# Get the node statistics using graph cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The user may need to know the aggregate stats of the node across multiple graph objects. Especially the situation where each graph object bound to a worker thread.

Introduced a graph cluster object for statistics. rte_graph_cluster_stats_create()
shall be used for creating a graph cluster with multiple graph objects and
rte_graph_cluster_stats_get() to get the aggregate node statistics.

An example statistics output from rte_graph_cluster_stats_get()

+-----------+------------+-------------+---------------+------------+---------------+-----------+
|Node       |calls       |objs         |realloc_count  |objs/call
|objs/sec(10E6) |cycles/call|
+------------------------+-------------+---------------+------------+---------------+-----------+
|node0      |12977424    |3322220544   |5              |256.000
|3047.151872    |20.0000    |
|node1      |12977653    |3322279168   |0              |256.000
|3047.210496    |17.0000    |
|node2      |12977696    |3322290176   |0              |256.000
|3047.221504    |17.0000    |
|node3      |12977734    |3322299904   |0              |256.000
|3047.231232    |17.0000    |
|node4      |12977784    |3322312704   |1              |256.000
|3047.243776    |17.0000    |
|node5      |12977825    |3322323200   |0              |256.000
|3047.254528    |17.0000    |
+-----------+------------+-------------+---------------+------------+---------------+-----------+

# Node writing guide lines
~~~~~~~~~~~~~~~~~~~~~~~~~~

The process() function of a node is fast-path function and that needs to be written
carefully to achieve max performance.

Broadly speaking, there are two different types of nodes.

1) First kind of nodes are those that have a fixed next_nodes[] for the complete burst (like ethdev_rx, ethdev_tx) and it is simple to write. Process() function can move the obj burst to the next node either using rte_node_next_stream_move() or using rte_node_next_stream_get() and rte_node_next_stream_put().

2) The second kind of such node is intermediate nodes that decide what is the next_node[]
to send to on a per-packet basis. In these nodes,

a) Firstly, there has to be the best possible packet processing logic.

b) Secondly, each packet needs to be queued to its next node. This can be done using rte_node_enqueue_[x1|x2|x4]() api's if they are to single next or
rte_node_enqueue_next() that takes array of nexts.

In scenario where multiple intermediate nodes are present but most of the time each node using same next node for all its packets, cost of moving every pointer from current node's stream to next node's stream could be avoided. This is called home run and rte_node_next_stream_move() could be used to just move stream from current node to next node with least number of cycles.
Since this can be avoided only in the case where all the packets are destined to the same next node, node implementation should be also having worst case handling where every packet could be going to different next node.

Example of intermediate node implementation with home run:
a) Start with speculation that next_node = ctx->next_node. This could be the next_node application used in the previous function call of this node.
b) Get the next_node stream array with required space using rte_node_next_stream_get(next_node, space)
c) while n_left_from > 0 // Pkts left to be sent
prefetch next pkt_set and process current pkt_set to find their next node
d) if all the next nodes of the current pkt_set match speculated next node, just count them as successfully speculated('last_spec') till now and continue the loop without actually moving them to the next node. else if there is a mismatch,
copy all the pkt_set pointers that were 'last_spec' and move
the current pkt_set to their respective next's nodes using
rte_enqueue_next_x1(). Also one of the next_node can be updated
as
speculated next_node if it is more probable.
Finally reset 'last_spec' to zero.
e) if n_left_from != 0
goto c) to process remaining pkts.
f) if last_spec == nb_objs,
All the objects passed were successfully speculated to single next
node.
So, the current stream can be moved to next node using rte_node_next_stream_move(node, next_node). This is home run where memcpy of buffer pointers to next node is avoided.
g) Update the ctx->next_node with more probable next node.

# In-tree node documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) librte_node/ethdev_rx.c:
This node does rte_eth_rx_burst() into stream buffer passed to it
(src node stream) and does rte_node_next_stream_move() only when there are packets received. Each rte_node works on only on
one rx port and queue that it gets from node->context.
For each (port X, rx_queue Y), a rte_node is cloned from
ethdev_rx_base_node as "ethdev_rx-X-Y" in rte_node_eth_config() along with updating node->context. Each graph needs to be associated with a unique rte_node for a (port, rx_queue).

b) librte_node/ethdev_tx.c:
This node does rte_eth_tx_burst() for a burst of objs received by
it.
It sends the burst to a fixed Tx Port and Queue information from node->context. For each (port X), this rte_node is cloned from ethdev_tx_node_base as "ethdev_tx-X" in rte_node_eth_config() along with updating node->context.
Since each graph doesn't need more than one Txq, per port, a Txq is assigned based on graph id to each rte_node instance. Each graph needs to be associated with a rte_node for each
(port).

c) librte_node/pkt_drop.c:
This node frees all the objects passed to it considering them as
rte_mbufs that need to be freed.

d) librte_node/ip4_lookup.c:
This node is an intermediate node that does lpm lookup for the receive ipv4 packets and the result determines each packets next node.
a) On successful lpm lookup, the result contains the nex_node id and next-hop id with which the packet needs to be further processed.
b) On lpm lookup failure, objects are redirected to pkt_drop node. rte_node_ip4_route_add() is control path API to add ipv4 routes. To achieve home run, we use rte_node_stream_move() as mentioned in above sections.

e) librte_node/ip4_rewrite.c:
This node gets packets from ip4_lookup node with next-hop id for each packet is embedded in rte_node_mbuf_priv1(mbuf)->nh. This id is used to determine the L2 header to be written to the pkt before sending the pkt out to a particular ethdev_tx node. rte_node_ip4_rewrite_add() is control path API to add next-hop info.

f) librte_node/null.c:
This is null node that just ignores the set of objects passed to
it and reports that all are processed.

Jerin Jacob (12):
graph: define the public API for graph support
graph: implement node registration
graph: implement node operations
graph: implement node debug routines
graph: implement internal graph operation helpers
graph: populate fastpath memory for graph reel
graph: implement create and destroy APIs
graph: implement graph operation APIs
graph: implement Graphviz export
graph: implement debug routines
graph: implement stats support
graph: implement fastpath API routines

Kiran Kumar K (2):
graph: add unit test case
node: add ipv4 rewrite node

Nithin Dabilpuram (10):
node: add log infra and null node
node: add ethdev Rx node
node: add ethdev Tx node
node: add ethdev Rx and Tx node ctrl API
node: ipv4 lookup for arm64
node: add ipv4 rewrite and lookup ctrl API
node: add pkt drop node
l3fwd-graph: add graph based l3fwd skeleton
l3fwd-graph: add ethdev configuration changes
l3fwd-graph: add graph config and main loop

Pavan Nikhilesh (2):
graph: add performance testcase
node: ipv4 lookup for x86

 MAINTAINERS                            |   13 +
 app/test/Makefile                      |    5 +
 app/test/meson.build                   |   11 +-
 app/test/test_graph.c                  |  820 +++++++++++++++++
 app/test/test_graph_perf.c             | 1057 ++++++++++++++++++++++
 config/common_base                     |   12 +
 config/rte_config.h                    |    4 +
 doc/api/doxy-api-index.md              |    5 +
 doc/api/doxy-api.conf.in               |    2 +
 examples/Makefile                      |    3 +
 examples/l3fwd-graph/Makefile          |   58 ++
 examples/l3fwd-graph/main.c            | 1113 ++++++++++++++++++++++++
 examples/l3fwd-graph/meson.build       |   13 +
 examples/meson.build                   |    6 +-
 lib/Makefile                           |    6 +
 lib/librte_graph/Makefile              |   28 +
 lib/librte_graph/graph.c               |  590 +++++++++++++
 lib/librte_graph/graph_debug.c         |   84 ++
 lib/librte_graph/graph_ops.c           |  169 ++++
 lib/librte_graph/graph_populate.c      |  234 +++++
 lib/librte_graph/graph_private.h       |  346 ++++++++
 lib/librte_graph/graph_stats.c         |  406 +++++++++
 lib/librte_graph/meson.build           |   11 +
 lib/librte_graph/node.c                |  421 +++++++++
 lib/librte_graph/rte_graph.h           |  786 +++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   47 +
 lib/librte_graph/rte_graph_worker.h    |  541 ++++++++++++
 lib/librte_node/Makefile               |   30 +
 lib/librte_node/ethdev_ctrl.c          |  115 +++
 lib/librte_node/ethdev_rx.c            |  221 +++++
 lib/librte_node/ethdev_rx_priv.h       |   81 ++
 lib/librte_node/ethdev_tx.c            |   86 ++
 lib/librte_node/ethdev_tx_priv.h       |   62 ++
 lib/librte_node/ip4_lookup.c           |  631 ++++++++++++++
 lib/librte_node/ip4_rewrite.c          |  325 +++++++
 lib/librte_node/ip4_rewrite_priv.h     |   77 ++
 lib/librte_node/log.c                  |   14 +
 lib/librte_node/meson.build            |    8 +
 lib/librte_node/node_private.h         |   96 ++
 lib/librte_node/null.c                 |   23 +
 lib/librte_node/pkt_drop.c             |   26 +
 lib/librte_node/rte_node_eth_api.h     |   70 ++
 lib/librte_node/rte_node_ip4_api.h     |   87 ++
 lib/librte_node/rte_node_version.map   |    9 +
 lib/meson.build                        |    5 +-
 meson.build                            |    1 +
 mk/rte.app.mk                          |    2 +
 47 files changed, 8755 insertions(+), 5 deletions(-)
 create mode 100644 app/test/test_graph.c
 create mode 100644 app/test/test_graph_perf.c
 create mode 100644 examples/l3fwd-graph/Makefile
 create mode 100644 examples/l3fwd-graph/main.c
 create mode 100644 examples/l3fwd-graph/meson.build
 create mode 100644 lib/librte_graph/Makefile
 create mode 100644 lib/librte_graph/graph.c
 create mode 100644 lib/librte_graph/graph_debug.c
 create mode 100644 lib/librte_graph/graph_ops.c
 create mode 100644 lib/librte_graph/graph_populate.c
 create mode 100644 lib/librte_graph/graph_private.h
 create mode 100644 lib/librte_graph/graph_stats.c
 create mode 100644 lib/librte_graph/meson.build
 create mode 100644 lib/librte_graph/node.c
 create mode 100644 lib/librte_graph/rte_graph.h
 create mode 100644 lib/librte_graph/rte_graph_version.map
 create mode 100644 lib/librte_graph/rte_graph_worker.h
 create mode 100644 lib/librte_node/Makefile
 create mode 100644 lib/librte_node/ethdev_ctrl.c
 create mode 100644 lib/librte_node/ethdev_rx.c
 create mode 100644 lib/librte_node/ethdev_rx_priv.h
 create mode 100644 lib/librte_node/ethdev_tx.c
 create mode 100644 lib/librte_node/ethdev_tx_priv.h
 create mode 100644 lib/librte_node/ip4_lookup.c
 create mode 100644 lib/librte_node/ip4_rewrite.c
 create mode 100644 lib/librte_node/ip4_rewrite_priv.h
 create mode 100644 lib/librte_node/log.c
 create mode 100644 lib/librte_node/meson.build
 create mode 100644 lib/librte_node/node_private.h
 create mode 100644 lib/librte_node/null.c
 create mode 100644 lib/librte_node/pkt_drop.c
 create mode 100644 lib/librte_node/rte_node_eth_api.h
 create mode 100644 lib/librte_node/rte_node_ip4_api.h
 create mode 100644 lib/librte_node/rte_node_version.map
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/26 graph: define the public API for graph support

From: Jerin Jacob <[email protected]>

Graph architecture abstracts the data processing functions as "node" and "link" them together to create a complex "graph" to enable reusable/modular data processing functions.

These APIs enables graph framework operations such as create, lookup, dump and destroy on graph and node operations such as clone, edge update, and edge shrink, etc. The API also allows creating the stats cluster to monitor per graph and per node stats.

This patch defines the public API for graph support. This patch also adds support for the build infrastructure and update the MAINTAINERS file for the graph subsystem.

---
 MAINTAINERS                            |   5 +
 config/common_base                     |   7 +
 config/rte_config.h                    |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 lib/Makefile                           |   3 +
 lib/librte_graph/Makefile              |  22 +
 lib/librte_graph/graph.c               |   5 +
 lib/librte_graph/meson.build           |  11 +
 lib/librte_graph/rte_graph.h           | 786 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   3 +
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 13 files changed, 850 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/Makefile
 create mode 100644 lib/librte_graph/graph.c
 create mode 100644 lib/librte_graph/meson.build
 create mode 100644 lib/librte_graph/rte_graph.h
 create mode 100644 lib/librte_graph/rte_graph_version.map
MAINTAINERS +5 -0
index c3785554f..32d0ea032 100644
@@ -1468,6 +1468,11 @@ F: examples/bpf/
 F: app/test/test_bpf.c
 F: doc/guides/prog_guide/bpf_lib.rst
 
+Graph - EXPERIMENTAL
+M: Jerin Jacob <[email protected]>
+M: Kiran Kumar K <[email protected]>
+F: lib/librte_graph/
+
 
 Test Applications
 -----------------
config/common_base +7 -0
index 7ca2f28b1..04a96aef5 100644
@@ -1075,6 +1075,13 @@ CONFIG_RTE_LIBRTE_BPF_ELF=n
 #
 CONFIG_RTE_LIBRTE_IPSEC=y
 
+#
+# Compile librte_graph
+#
+CONFIG_RTE_LIBRTE_GRAPH=y
+CONFIG_RTE_GRAPH_BURST_SIZE=256
+CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+
 #
 # Compile the test application
 #
config/rte_config.h +4 -0
index d30786bc0..e9201fd46 100644
@@ -98,6 +98,10 @@
 /* KNI defines */
 #define RTE_KNI_PREEMPT_DEFAULT 1
 
+/* rte_graph defines */
+#define RTE_GRAPH_BURST_SIZE 256
+#define RTE_LIBRTE_GRAPH_STATS 1
+
 /****** driver defines ********/
 
 /* QuickAssist device */
doc/api/doxy-api-index.md +1 -0
index dff496be0..5cc50f750 100644
@@ -159,6 +159,7 @@ The public API headers are grouped by topics:
   * [pipeline]         (@ref rte_pipeline.h)
     [port_in_action]   (@ref rte_port_in_action.h)
     [table_action]     (@ref rte_table_action.h)
+  * [graph]            (@ref rte_graph.h):
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
doc/api/doxy-api.conf.in +1 -0
index 1c4392eec..759a7213e 100644
@@ -33,6 +33,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_eventdev \
                           @TOPDIR@/lib/librte_fib \
                           @TOPDIR@/lib/librte_flow_classify \
+                          @TOPDIR@/lib/librte_graph \
                           @TOPDIR@/lib/librte_gro \
                           @TOPDIR@/lib/librte_gso \
                           @TOPDIR@/lib/librte_hash \
lib/Makefile +3 -0
index 46b91ae1a..1f572b659 100644
@@ -119,6 +119,9 @@ DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
 DIRS-$(CONFIG_RTE_LIBRTE_RCU) += librte_rcu
 DEPDIRS-librte_rcu := librte_eal
 
+DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
+DEPDIRS-librte_graph := librte_eal
+
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
 endif
lib/librte_graph/Makefile new file +22 -0
new file mode 100644
index 000000000..26fe514f3
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_graph.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_graph_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_graph/graph.c new file +5 -0
new file mode 100644
index 000000000..a55bf443a
@@ -0,0 +1,5 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "rte_graph.h"
lib/librte_graph/meson.build new file +11 -0
new file mode 100644
index 000000000..455cf2ba5
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+name = 'graph'
+
+sources = files('graph.c')
+headers = files('rte_graph.h')
+allow_experimental_apis = true
+
+deps += ['eal']
lib/librte_graph/rte_graph.h new file +786 -0

786 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map new file +3 -0
new file mode 100644
index 000000000..55ef35df5
@@ -0,0 +1,3 @@
+EXPERIMENTAL {
+
+};
lib/meson.build +1 -1
index 9c3cc55d5..c43d86bb9 100644
@@ -30,7 +30,7 @@ libraries = [
 	# add pkt framework libs which use other libs from above
 	'port', 'table', 'pipeline',
 	# flow_classify lib depends on pkt framework table lib
-	'flow_classify', 'bpf', 'telemetry']
+	'flow_classify', 'bpf', 'graph', 'telemetry']
 
 if is_windows
 	libraries = ['kvargs','eal'] # only supported libraries for windows
mk/rte.app.mk +1 -0
index d295ca0a5..b1195f09a 100644
@@ -98,6 +98,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RCU)            += -lrte_rcu
+_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH)          += -lrte_graph
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KNI)            += -lrte_kni
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/26 graph: implement node registration

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding rte_node_register() API implementation includes allocating memory for node object, check for duplicate node name and add the allocated node to STAILQ node_list for future use.

---
 lib/librte_graph/Makefile              |   1 +
 lib/librte_graph/graph.c               |  18 +++-
 lib/librte_graph/graph_private.h       |  74 ++++++++++++++++
 lib/librte_graph/meson.build           |   2 +-
 lib/librte_graph/node.c                | 115 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   4 +
 6 files changed, 212 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_private.h
 create mode 100644 lib/librte_graph/node.c
lib/librte_graph/Makefile +1 -0
index 26fe514f3..933d0ee49 100644
@@ -14,6 +14,7 @@ LDLIBS += -lrte_eal
 EXPORT_MAP := rte_graph_version.map
 
 # all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 
 # install header files
lib/librte_graph/graph.c +17 -1
index a55bf443a..a9c124896 100644
@@ -2,4 +2,20 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
-#include "rte_graph.h"
+#include <rte_spinlock.h>
+
+#include "graph_private.h"
+
+static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+
+void
+graph_spinlock_lock(void)
+{
+	rte_spinlock_lock(&graph_lock);
+}
+
+void
+graph_spinlock_unlock(void)
+{
+	rte_spinlock_unlock(&graph_lock);
+}
lib/librte_graph/graph_private.h new file +74 -0

74 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index 455cf2ba5..5754ac23b 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('graph.c')
+sources = files('node.c', 'graph.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
lib/librte_graph/node.c new file +115 -0

115 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map +4 -0
index 55ef35df5..0884c09f1 100644
@@ -1,3 +1,7 @@
 EXPERIMENTAL {
+	global:
 
+	__rte_node_register;
+
+	local: *;
 };
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/26 graph: implement node operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding node-specific API implementation like cloning node, updating edges for the node, shrinking edges of a node, retrieving edges of a node.

---
 lib/librte_graph/graph_private.h       |  10 +
 lib/librte_graph/node.c                | 269 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  10 +
 3 files changed, 289 insertions(+)
lib/librte_graph/graph_private.h +10 -0
index ba56927fa..28405ddb7 100644
@@ -12,6 +12,16 @@
 #include <rte_eal.h>
 #include <rte_graph.h>
 
+
+#define ID_CHECK(id, id_max)                                                   \
+	do {                                                                   \
+		if ((id) >= (id_max)) {                                        \
+			rte_errno = EINVAL;                                    \
+			goto fail;                                             \
+		}                                                              \
+	} while (0)
+
+
 /**
  * @internal
  *
lib/librte_graph/node.c +269 -0

272 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map +10 -0
index 0884c09f1..412386356 100644
@@ -3,5 +3,15 @@ EXPERIMENTAL {
 
 	__rte_node_register;
 
+	rte_node_clone;
+	rte_node_edge_count;
+	rte_node_edge_get;
+	rte_node_edge_shrink;
+	rte_node_edge_update;
+	rte_node_from_name;
+	rte_node_id_to_name;
+	rte_node_list_dump;
+	rte_node_max_count;
+
 	local: *;
 };
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/26 graph: implement node debug routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding node debug API implementation support to dump single or all the node objects to the given file.

---
 lib/librte_graph/Makefile              |  1 +
 lib/librte_graph/graph_debug.c         | 25 ++++++++++++++++++++
 lib/librte_graph/graph_private.h       | 12 ++++++++++
 lib/librte_graph/meson.build           |  2 +-
 lib/librte_graph/node.c                | 32 ++++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  1 +
 6 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/graph_debug.c
lib/librte_graph/Makefile +1 -0
index 933d0ee49..2a6d86933 100644
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
lib/librte_graph/graph_debug.c new file +25 -0
new file mode 100644
index 000000000..75238e7ca
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_common.h>
+#include <rte_debug.h>
+
+#include "graph_private.h"
+
+void
+node_dump(FILE *f, struct node *n)
+{
+	rte_edge_t i;
+
+	fprintf(f, "node <%s>\n", n->name);
+	fprintf(f, "  id=%" PRIu32 "\n", n->id);
+	fprintf(f, "  flags=0x%" PRIx64 "\n", n->flags);
+	fprintf(f, "  addr=%p\n", n);
+	fprintf(f, "  process=%p\n", n->process);
+	fprintf(f, "  nb_edges=%d\n", n->nb_edges);
+
+	for (i = 0; i < n->nb_edges; i++)
+		fprintf(f, "     edge[%d] <%s>\n", i, n->next_nodes[i]);
+}
+
lib/librte_graph/graph_private.h +12 -0
index 28405ddb7..a8efee7c8 100644
@@ -81,4 +81,16 @@ void graph_spinlock_lock(void);
  */
 void graph_spinlock_unlock(void);
 
+/**
+ * @internal
+ *
+ * Dump internal node object data.
+ *
+ * @param f
+ *   FILE pointer to dump the info.
+ * @param g
+ *   Pointer to the internal node object.
+ */
+void node_dump(FILE *f, struct node *n);
+
 #endif /* _RTE_GRAPH_PRIVATE_H_ */
lib/librte_graph/meson.build +1 -1
index 5754ac23b..01512182f 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c')
+sources = files('node.c', 'graph.c', 'graph_debug.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
lib/librte_graph/node.c +32 -0
index 8de857889..2f9c2ea4c 100644
@@ -377,6 +377,38 @@ rte_node_edge_get(rte_node_t id, char *next_nodes[])
 	return rc;
 }
 
+static void
+node_scan_dump(FILE *f, rte_node_t id, bool all)
+{
+	struct node *node;
+
+	RTE_ASSERT(f != NULL);
+	NODE_ID_CHECK(id);
+
+	STAILQ_FOREACH(node, &node_list, next) {
+		if (all == true) {
+			node_dump(f, node);
+		} else if (node->id == id) {
+			node_dump(f, node);
+			return;
+		}
+	}
+fail:
+	return;
+}
+
+void
+rte_node_dump(FILE *f, rte_node_t id)
+{
+	node_scan_dump(f, id, false);
+}
+
+void
+rte_node_list_dump(FILE *f)
+{
+	node_scan_dump(f, 0, true);
+}
+
 rte_node_t
 rte_node_max_count(void)
 {
lib/librte_graph/rte_graph_version.map +1 -0
index 412386356..f2c2139c5 100644
@@ -4,6 +4,7 @@ EXPERIMENTAL {
 	__rte_node_register;
 
 	rte_node_clone;
+	rte_node_dump;
 	rte_node_edge_count;
 	rte_node_edge_get;
 	rte_node_edge_shrink;
Signature · 1 line
2.25.1
#5 lore raw

PATCH 5/26 graph: implement internal graph operation helpers

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding internal graph API helpers support to check whether a graph has isolated nodes and any node have a loop to itself and BFS algorithm implementation etc.

---
 lib/librte_graph/Makefile        |   1 +
 lib/librte_graph/graph.c         |   2 +-
 lib/librte_graph/graph_ops.c     | 169 ++++++++++++++++++++++++++++++
 lib/librte_graph/graph_private.h | 173 +++++++++++++++++++++++++++++++
 lib/librte_graph/meson.build     |   2 +-
 5 files changed, 345 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_ops.c
lib/librte_graph/Makefile +1 -0
index 2a6d86933..39ecb2652 100644
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
 
 # install header files
lib/librte_graph/graph.c +1 -1
index a9c124896..4c3f2fe7b 100644
@@ -7,7 +7,7 @@
 #include "graph_private.h"
 
 static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
-
+int rte_graph_logtype;
 void
 graph_spinlock_lock(void)
 {
lib/librte_graph/graph_ops.c new file +169 -0

169 lines not on this page · raw message

lib/librte_graph/graph_private.h +173 -0

203 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index 01512182f..16e0625c1 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_debug.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
Signature · 1 line
2.25.1
#6 lore raw

PATCH 6/26 graph: populate fastpath memory for graph reel

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding support to create and populate the memory for graph reel. This includes reserving the memory in the memzone, populating the nodes, Allocating memory for node-specific streams to hold objects.

Once it is populated the reel memory contains the following sections.

+---------------------+
|   Graph Header      |
+---------------------+
|   Fence             |
+---------------------+
|   Circular buffer   |
+---------------------+
|   Fence             |
+---------------------+
|   Node Object 0     |
+------------------- -+
|   Node Object 1     |
+------------------- -+
|   Node Object 2     |
+------------------- -+
|   Node Object n     |
+------------------- -+
---
 lib/librte_graph/Makefile              |   2 +
 lib/librte_graph/graph.c               |  16 ++
 lib/librte_graph/graph_populate.c      | 234 +++++++++++++++++++++++++
 lib/librte_graph/graph_private.h       |  64 +++++++
 lib/librte_graph/meson.build           |   4 +-
 lib/librte_graph/node.c                |   5 +
 lib/librte_graph/rte_graph_version.map |   1 +
 lib/librte_graph/rte_graph_worker.h    | 107 +++++++++++
 8 files changed, 431 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_populate.c
 create mode 100644 lib/librte_graph/rte_graph_worker.h
lib/librte_graph/Makefile +2 -0
index 39ecb2652..7bfd7d51f 100644
@@ -18,8 +18,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph_worker.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_graph/graph.c +16 -0
index 4c3f2fe7b..e1930b7d2 100644
@@ -2,6 +2,7 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
+#include <rte_malloc.h>
 #include <rte_spinlock.h>
 
 #include "graph_private.h"
@@ -19,3 +20,18 @@ graph_spinlock_unlock(void)
 {
 	rte_spinlock_unlock(&graph_lock);
 }
+
+void __rte_noinline
+__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
+{
+	uint16_t size = node->size;
+
+	RTE_VERIFY(size != UINT16_MAX);
+	/* Allocate double amount of size to avoid immediate realloc */
+	size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, size * 2));
+	node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+					RTE_CACHE_LINE_SIZE, graph->socket);
+	RTE_VERIFY(node->objs);
+	node->size = size;
+	node->realloc_count++;
+}
lib/librte_graph/graph_populate.c new file +234 -0

234 lines not on this page · raw message

lib/librte_graph/graph_private.h +64 -0

76 lines not on this page · raw message

lib/librte_graph/meson.build +2 -2
index 16e0625c1..fb203a5e2 100644
@@ -4,8 +4,8 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
-headers = files('rte_graph.h')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+headers = files('rte_graph.h', 'rte_graph_worker.h')
 allow_experimental_apis = true
 
 deps += ['eal']
lib/librte_graph/node.c +5 -0
index 2f9c2ea4c..639269870 100644
@@ -61,6 +61,11 @@ __rte_node_register(const struct rte_node_register *reg)
 	rte_edge_t i;
 	size_t sz;
 
+	/* Limit Node specific metadata to one cacheline on 64B CL machine */
+	RTE_BUILD_BUG_ON((offsetof(struct rte_node, nodes) -
+			  offsetof(struct rte_node, ctx)) !=
+			 RTE_CACHE_LINE_MIN_SIZE);
+
 	graph_spinlock_lock();
 
 	/* Check sanity */
lib/librte_graph/rte_graph_version.map +1 -0
index f2c2139c5..a9fe1b610 100644
@@ -2,6 +2,7 @@ EXPERIMENTAL {
 	global:
 
 	__rte_node_register;
+	__rte_node_stream_alloc;
 
 	rte_node_clone;
 	rte_node_dump;
lib/librte_graph/rte_graph_worker.h new file +107 -0

107 lines not on this page · raw message

Signature · 1 line
2.25.1
#7 lore raw

PATCH 7/26 graph: implement create and destroy APIs

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding graph specific API implementations like graph create and graph destroy. This detect loops in the graph, check for isolated nodes and operation to verify the validity of graph.

---
 lib/librte_graph/graph.c               | 321 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   2 +
 2 files changed, 323 insertions(+)
lib/librte_graph/graph.c +321 -0
index e1930b7d2..d060ffe70 100644
@@ -2,13 +2,34 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
+#include <rte_graph.h>
 #include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_spinlock.h>
+#include <rte_string_fns.h>
 
 #include "graph_private.h"
 
+static struct graph_head graph_list = STAILQ_HEAD_INITIALIZER(graph_list);
 static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+static rte_graph_t graph_id;
 int rte_graph_logtype;
+
+#define GRAPH_ID_CHECK(id) ID_CHECK(id, graph_id)
+
+/* Private functions */
+struct graph_head *
+graph_list_head_get(void)
+{
+	return &graph_list;
+}
+
 void
 graph_spinlock_lock(void)
 {
@@ -21,6 +42,306 @@ graph_spinlock_unlock(void)
 	rte_spinlock_unlock(&graph_lock);
 }
 
+static int
+graph_node_add(struct graph *graph, struct node *node)
+{
+	struct graph_node *graph_node;
+	size_t sz;
+
+	/* Skip the duplicate nodes */
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (strncmp(node->name, graph_node->node->name,
+			    RTE_NODE_NAMESIZE) == 0)
+			return 0;
+
+	/* Allocate new graph node object */
+	sz = sizeof(*graph_node) + node->nb_edges * sizeof(struct node *);
+	graph_node = calloc(1, sz);
+
+	if (graph_node == NULL)
+		SET_ERR_JMP(ENOMEM, free, "Failed to calloc %s object",
+			    node->name);
+
+	/* Initialize the graph node */
+	graph_node->node = node;
+
+	/* Add to graph node list */
+	STAILQ_INSERT_TAIL(&graph->node_list, graph_node, next);
+	return 0;
+
+free:
+	free(graph_node);
+	return -rte_errno;
+}
+
+static struct graph_node *
+node_to_graph_node(struct graph *graph, struct node *node)
+{
+	struct graph_node *graph_node;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (graph_node->node == node)
+			return graph_node;
+
+	SET_ERR_JMP(ENODEV, fail, "Found isolated node %s", node->name);
+fail:
+	return NULL;
+}
+
+static int
+graph_node_edges_add(struct graph *graph)
+{
+	struct graph_node *graph_node;
+	struct node *adjacency;
+	const char *next;
+	rte_edge_t i;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			next = graph_node->node->next_nodes[i];
+			adjacency = node_from_name(next);
+			if (adjacency == NULL)
+				SET_ERR_JMP(EINVAL, fail,
+					    "Node %s not registered", next);
+			if (graph_node_add(graph, adjacency))
+				goto fail;
+		}
+	}
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static int
+graph_adjacency_list_update(struct graph *graph)
+{
+	struct graph_node *graph_node, *tmp;
+	struct node *adjacency;
+	const char *next;
+	rte_edge_t i;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			next = graph_node->node->next_nodes[i];
+			adjacency = node_from_name(next);
+			if (adjacency == NULL)
+				SET_ERR_JMP(EINVAL, fail,
+					    "Node %s not registered", next);
+			tmp = node_to_graph_node(graph, adjacency);
+			if (tmp == NULL)
+				goto fail;
+			graph_node->adjacency_list[i] = tmp;
+		}
+	}
+
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static int
+expand_pattern_to_node(struct graph *graph, const char *pattern)
+{
+	struct node_head *node_head = node_list_head_get();
+	bool found = false;
+	struct node *node;
+
+	/* Check for pattern match */
+	STAILQ_FOREACH(node, node_head, next) {
+		if (fnmatch(pattern, node->name, 0) == 0) {
+			if (graph_node_add(graph, node))
+				goto fail;
+			found = true;
+		}
+	}
+	if (found == false)
+		SET_ERR_JMP(EFAULT, fail, "Pattern %s node not found", pattern);
+
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static void
+graph_cleanup(struct graph *graph)
+{
+	struct graph_node *graph_node;
+
+	while (!STAILQ_EMPTY(&graph->node_list)) {
+		graph_node = STAILQ_FIRST(&graph->node_list);
+		STAILQ_REMOVE_HEAD(&graph->node_list, next);
+		free(graph_node);
+	}
+}
+
+static int
+graph_node_init(struct graph *graph)
+{
+	struct graph_node *graph_node;
+	const char *name;
+	int rc;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		if (graph_node->node->init) {
+			name = graph_node->node->name;
+			rc = graph_node->node->init(
+				graph->graph,
+				graph_node_name_to_ptr(graph->graph, name));
+			if (rc)
+				SET_ERR_JMP(rc, err, "Node %s init() failed",
+					    name);
+		}
+	}
+
+	return 0;
+err:
+	return -rte_errno;
+}
+
+static void
+graph_node_fini(struct graph *graph)
+{
+	struct graph_node *graph_node;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (graph_node->node->fini)
+			graph_node->node->fini(
+				graph->graph,
+				graph_node_name_to_ptr(graph->graph,
+						       graph_node->node->name));
+}
+
+rte_graph_t
+rte_graph_create(const char *name, struct rte_graph_param *prm)
+{
+	struct graph *graph;
+	const char *pattern;
+	uint16_t i;
+
+	graph_spinlock_lock();
+
+	/* Check arguments sanity */
+	if (prm == NULL)
+		SET_ERR_JMP(EINVAL, fail, "Param should not be NULL");
+
+	if (name == NULL)
+		SET_ERR_JMP(EINVAL, fail, "Graph name should not be NULL");
+
+	/* Check for existence of duplicate graph */
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (strncmp(name, graph->name, RTE_GRAPH_NAMESIZE) == 0)
+			SET_ERR_JMP(EEXIST, fail, "Found duplicate graph %s",
+				    name);
+
+	/* Create graph object */
+	graph = calloc(1, sizeof(*graph));
+	if (graph == NULL)
+		SET_ERR_JMP(ENOMEM, fail, "Failed to calloc graph object");
+
+	/* Initialize the graph object */
+	STAILQ_INIT(&graph->node_list);
+	if (rte_strscpy(graph->name, name, RTE_GRAPH_NAMESIZE) < 0)
+		SET_ERR_JMP(E2BIG, free, "Too big name=%s", name);
+
+	/* Expand node pattern and add the nodes to the graph */
+	for (i = 0; i < prm->nb_node_patterns; i++) {
+		pattern = prm->node_patterns[i];
+		if (expand_pattern_to_node(graph, pattern))
+			goto graph_cleanup;
+	}
+
+	/* Go over all the nodes edges and add them to the graph */
+	if (graph_node_edges_add(graph))
+		goto graph_cleanup;
+
+	/* Update adjacency list of all nodes in the graph */
+	if (graph_adjacency_list_update(graph))
+		goto graph_cleanup;
+
+	/* Make sure at least a source node present in the graph */
+	if (!graph_src_nodes_count(graph))
+		goto graph_cleanup;
+
+	/* Make sure no node is pointing to source node */
+	if (graph_node_has_edge_to_src_node(graph))
+		goto graph_cleanup;
+
+	/* Don't allow node has loop to self */
+	if (graph_node_has_loop_edge(graph))
+		goto graph_cleanup;
+
+	/* Do BFS from src nodes on the graph to find isolated nodes */
+	if (graph_has_isolated_node(graph))
+		goto graph_cleanup;
+
+	/* Initialize graph object */
+	graph->socket = prm->socket_id;
+	graph->src_node_count = graph_src_nodes_count(graph);
+	graph->node_count = graph_nodes_count(graph);
+	graph->id = graph_id;
+
+	/* Allocate the Graph fast path memory and populate the data */
+	if (graph_fp_mem_create(graph))
+		goto graph_cleanup;
+
+	/* Call init() of the all the nodes in the graph */
+	if (graph_node_init(graph))
+		goto graph_mem_destroy;
+
+	/* All good, Lets add the graph to the list */
+	graph_id++;
+	STAILQ_INSERT_TAIL(&graph_list, graph, next);
+
+	graph_spinlock_unlock();
+	return graph->id;
+
+graph_mem_destroy:
+	graph_fp_mem_destroy(graph);
+graph_cleanup:
+	graph_cleanup(graph);
+free:
+	free(graph);
+fail:
+	graph_spinlock_unlock();
+	return RTE_GRAPH_ID_INVALID;
+}
+
+rte_graph_t
+rte_graph_destroy(const char *graph_name)
+{
+	rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+	struct graph *graph, *tmp;
+	const char *name;
+
+	graph_spinlock_lock();
+
+	graph = STAILQ_FIRST(&graph_list);
+	while (graph != NULL) {
+		tmp = STAILQ_NEXT(graph, next);
+		name = graph->name;
+		if (strncmp(name, graph_name, RTE_GRAPH_NAMESIZE) == 0) {
+			/* Call fini() of the all the nodes in the graph */
+			graph_node_fini(graph);
+			/* Destroy graph fast path memory */
+			rc = graph_fp_mem_destroy(graph);
+			if (rc)
+				SET_ERR_JMP(rc, done, "MZ %s free failed",
+					    name);
+
+			graph_cleanup(graph);
+			STAILQ_REMOVE(&graph_list, graph, graph, next);
+			rc = graph->id;
+			free(graph);
+			graph_id--;
+			goto done;
+		}
+		graph = tmp;
+	}
+done:
+	graph_spinlock_unlock();
+	return rc;
+}
+
 void __rte_noinline
 __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 {
lib/librte_graph/rte_graph_version.map +2 -0
index a9fe1b610..dcbd78c02 100644
@@ -4,6 +4,8 @@ EXPERIMENTAL {
 	__rte_node_register;
 	__rte_node_stream_alloc;
 
+	rte_graph_create;
+	rte_graph_destroy;
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#8 lore raw

PATCH 8/26 graph: implement graph operation APIs

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding support for graph specific API implementation like Graph lookup to get graph object, retrieving graph ID From name and graph name from ID.

---
 lib/librte_graph/graph.c               | 131 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   8 ++
 2 files changed, 139 insertions(+)
lib/librte_graph/graph.c +131 -0
index d060ffe70..72a82c2a8 100644
@@ -211,6 +211,54 @@ graph_node_fini(struct graph *graph)
 						       graph_node->node->name));
 }
 
+static struct rte_graph *
+graph_mem_fixup_node_ctx(struct rte_graph *graph)
+{
+	struct rte_node *node;
+	struct node *node_db;
+	rte_graph_off_t off;
+	rte_node_t count;
+	const char *name;
+
+	rte_graph_foreach_node(count, off, graph, node) {
+		if (node->parent_id == RTE_NODE_ID_INVALID) /* Static node */
+			name = node->name;
+		else /* Cloned node */
+			name = node->parent;
+
+		node_db = node_from_name(name);
+		if (node_db == NULL)
+			SET_ERR_JMP(ENOLINK, fail, "Node %s not found", name);
+		node->process = node_db->process;
+	}
+
+	return graph;
+fail:
+	return NULL;
+}
+
+static struct rte_graph *
+graph_mem_fixup_secondray(struct rte_graph *graph)
+{
+	if (graph == NULL || rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return graph;
+
+	return graph_mem_fixup_node_ctx(graph);
+}
+
+struct rte_graph *
+rte_graph_lookup(const char *name)
+{
+	const struct rte_memzone *mz;
+	struct rte_graph *rc = NULL;
+
+	mz = rte_memzone_lookup(name);
+	if (mz)
+		rc = mz->addr;
+
+	return graph_mem_fixup_secondray(rc);
+}
+
 rte_graph_t
 rte_graph_create(const char *name, struct rte_graph_param *prm)
 {
@@ -342,6 +390,76 @@ rte_graph_destroy(const char *graph_name)
 	return rc;
 }
 
+rte_graph_t
+rte_graph_from_name(const char *name)
+{
+	struct graph *graph;
+
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0)
+			return graph->id;
+
+	return RTE_GRAPH_ID_INVALID;
+}
+
+char *
+rte_graph_id_to_name(rte_graph_t id)
+{
+	struct graph *graph;
+
+	GRAPH_ID_CHECK(id);
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (graph->id == id)
+			return graph->name;
+
+fail:
+	return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get(rte_graph_t gid, uint32_t nid)
+{
+	struct rte_node *node;
+	struct graph *graph;
+	rte_graph_off_t off;
+	rte_node_t count;
+
+	GRAPH_ID_CHECK(gid);
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (graph->id == gid) {
+			rte_graph_foreach_node(count, off, graph->graph,
+						node) {
+				if (node->id == nid)
+					return node;
+			}
+			break;
+		}
+fail:
+	return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get_by_name(const char *graph_name, const char *node_name)
+{
+	struct rte_node *node;
+	struct graph *graph;
+	rte_graph_off_t off;
+	rte_node_t count;
+
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (!strncmp(graph->name, graph_name, RTE_GRAPH_NAMESIZE)) {
+			rte_graph_foreach_node(count, off, graph->graph,
+						node) {
+				if (!strncmp(node->name, node_name,
+					     RTE_NODE_NAMESIZE))
+					return node;
+			}
+			break;
+		}
+
+	return NULL;
+}
+
 void __rte_noinline
 __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 {
@@ -356,3 +474,16 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->size = size;
 	node->realloc_count++;
 }
+
+rte_graph_t
+rte_graph_max_count(void)
+{
+	return graph_id;
+}
+
+RTE_INIT(rte_graph_init_log)
+{
+	rte_graph_logtype = rte_log_register("lib.graph");
+	if (rte_graph_logtype >= 0)
+		rte_log_set_level(rte_graph_logtype, RTE_LOG_INFO);
+}
lib/librte_graph/rte_graph_version.map +8 -0
index dcbd78c02..5a2b13293 100644
@@ -6,6 +6,14 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_from_name;
+	rte_graph_id_to_name;
+	rte_graph_lookup;
+	rte_graph_list_dump;
+	rte_graph_max_count;
+	rte_graph_node_get;
+	rte_graph_node_get_by_name;
+
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#9 lore raw

PATCH 9/26 graph: implement Graphviz export

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding API implementation support exporting the graph object to file. This will export the graph to a file in Graphviz format. It can be viewed in many viewers such as
https://dreampuf.github.io/GraphvizOnline/

---
 lib/librte_graph/graph.c               | 54 ++++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  1 +
 2 files changed, 55 insertions(+)
lib/librte_graph/graph.c +54 -0
index 72a82c2a8..ad58673be 100644
@@ -475,6 +475,60 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->realloc_count++;
 }
 
+static int
+graph_to_dot(FILE *f, struct graph *graph)
+{
+	const char *src_edge_color = " [color=blue]\n";
+	const char *edge_color = "\n";
+	struct graph_node *graph_node;
+	char *node_name;
+	rte_edge_t i;
+	int rc;
+
+	rc = fprintf(f, "Digraph %s {\n\trankdir=LR;\n", graph->name);
+	if (rc < 0)
+		goto end;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		node_name = graph_node->node->name;
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			rc = fprintf(f, "\t\"%s\"->\"%s\"%s", node_name,
+				     graph_node->adjacency_list[i]->node->name,
+				     graph_node->node->flags & RTE_NODE_SOURCE_F
+					     ? src_edge_color
+					     : edge_color);
+			if (rc < 0)
+				goto end;
+		}
+	}
+	rc = fprintf(f, "}\n");
+	if (rc < 0)
+		goto end;
+
+	return 0;
+end:
+	rte_errno = EBADF;
+	return -rte_errno;
+}
+
+rte_graph_t
+rte_graph_export(const char *name, FILE *f)
+{
+	rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+	struct graph *graph;
+
+	STAILQ_FOREACH(graph, &graph_list, next) {
+		if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0) {
+			rc = graph_to_dot(f, graph);
+			goto end;
+		}
+	}
+	rte_errno = ENOENT;
+end:
+	return rc;
+}
+
+
 rte_graph_t
 rte_graph_max_count(void)
 {
lib/librte_graph/rte_graph_version.map +1 -0
index 5a2b13293..2797be044 100644
@@ -6,6 +6,7 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_export;
 	rte_graph_from_name;
 	rte_graph_id_to_name;
 	rte_graph_lookup;
Signature · 1 line
2.25.1
#10 lore raw

PATCH 10/26 graph: implement debug routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for graph specific API to dump the Graph information to a file. This API will dump detailed internal info about node objects and graph objects.

---
 lib/librte_graph/graph.c               | 31 ++++++++++++++
 lib/librte_graph/graph_debug.c         | 59 ++++++++++++++++++++++++++
 lib/librte_graph/graph_private.h       | 13 ++++++
 lib/librte_graph/rte_graph_version.map |  2 +
 4 files changed, 105 insertions(+)
lib/librte_graph/graph.c +31 -0
index ad58673be..cc1e523d9 100644
@@ -528,6 +528,37 @@ rte_graph_export(const char *name, FILE *f)
 	return rc;
 }
 
+static void
+graph_scan_dump(FILE *f, rte_graph_t id, bool all)
+{
+	struct graph *graph;
+
+	RTE_VERIFY(f);
+	GRAPH_ID_CHECK(id);
+
+	STAILQ_FOREACH(graph, &graph_list, next) {
+		if (all == true) {
+			graph_dump(f, graph);
+		} else if (graph->id == id) {
+			graph_dump(f, graph);
+			return;
+		}
+	}
+fail:
+	return;
+}
+
+void
+rte_graph_dump(FILE *f, rte_graph_t id)
+{
+	graph_scan_dump(f, id, false);
+}
+
+void
+rte_graph_list_dump(FILE *f)
+{
+	graph_scan_dump(f, 0, true);
+}
 
 rte_graph_t
 rte_graph_max_count(void)
lib/librte_graph/graph_debug.c +59 -0

68 lines not on this page · raw message

lib/librte_graph/graph_private.h +13 -0
index 051fad53a..397247797 100644
@@ -318,6 +318,19 @@ struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
 struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
 					const char *node_name);
 
+/* Debug functions */
+/**
+ * @internal
+ *
+ * Dump internal graph object data.
+ *
+ * @param f
+ *   FILE pointer to dump the data.
+ * @param g
+ *   Pointer to the internal graph object.
+ */
+void graph_dump(FILE *f, struct graph *g);
+
 /**
  * @internal
  *
lib/librte_graph/rte_graph_version.map +2 -0
index 2797be044..851f4772e 100644
@@ -6,6 +6,7 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_dump;
 	rte_graph_export;
 	rte_graph_from_name;
 	rte_graph_id_to_name;
@@ -14,6 +15,7 @@ EXPERIMENTAL {
 	rte_graph_max_count;
 	rte_graph_node_get;
 	rte_graph_node_get_by_name;
+	rte_graph_obj_dump;
 
 	rte_node_clone;
 	rte_node_dump;
Signature · 1 line
2.25.1
#11 lore raw

PATCH 11/26 graph: implement stats support

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for graph stats collection API. This API will create a cluster for a specified node pattern and aggregate the node runtime stats.

---
 lib/librte_graph/Makefile              |   1 +
 lib/librte_graph/graph_stats.c         | 406 +++++++++++++++++++++++++
 lib/librte_graph/meson.build           |   2 +-
 lib/librte_graph/rte_graph_version.map |   5 +
 4 files changed, 413 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/graph_stats.c
lib/librte_graph/Makefile +1 -0
index 7bfd7d51f..967c8d9bc 100644
@@ -18,6 +18,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_stats.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
 
 # install header files
lib/librte_graph/graph_stats.c new file +406 -0

406 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index fb203a5e2..929a17f84 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_stats.c', 'graph_populate.c')
 headers = files('rte_graph.h', 'rte_graph_worker.h')
 allow_experimental_apis = true
 
lib/librte_graph/rte_graph_version.map +5 -0
index 851f4772e..adf55d406 100644
@@ -17,6 +17,11 @@ EXPERIMENTAL {
 	rte_graph_node_get_by_name;
 	rte_graph_obj_dump;
 
+	rte_graph_cluster_stats_create;
+	rte_graph_cluster_stats_destroy;
+	rte_graph_cluster_stats_get;
+	rte_graph_cluster_stats_reset;
+
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#12 lore raw

PATCH 12/26 graph: implement fastpath API routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for rte_graph_walk() API. This will perform a walk on the circular buffer and call the process function of each node and collect the stats if stats collection is enabled.

---
 doc/api/doxy-api-index.md              |   1 +
 lib/librte_graph/graph.c               |  16 +
 lib/librte_graph/rte_graph_version.map |  10 +
 lib/librte_graph/rte_graph_worker.h    | 434 +++++++++++++++++++++++++
 4 files changed, 461 insertions(+)
doc/api/doxy-api-index.md +1 -0
index 5cc50f750..fd2ff64d7 100644
@@ -160,6 +160,7 @@ The public API headers are grouped by topics:
     [port_in_action]   (@ref rte_port_in_action.h)
     [table_action]     (@ref rte_table_action.h)
   * [graph]            (@ref rte_graph.h):
+    [graph_worker]     (@ref rte_graph_worker.h)
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_graph/graph.c +16 -0
index cc1e523d9..78bc83c4e 100644
@@ -475,6 +475,22 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->realloc_count++;
 }
 
+void __rte_noinline
+__rte_node_stream_alloc_size(struct rte_graph *graph, struct rte_node *node,
+			     uint16_t req_size)
+{
+	uint16_t size = node->size;
+
+	RTE_VERIFY(size != UINT16_MAX);
+	/* Allocate double amount of size to avoid immediate realloc */
+	size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, req_size * 2));
+	node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+					RTE_CACHE_LINE_SIZE, graph->socket);
+	RTE_VERIFY(node->objs);
+	node->size = size;
+	node->realloc_count++;
+}
+
 static int
 graph_to_dot(FILE *f, struct graph *graph)
 {
lib/librte_graph/rte_graph_version.map +10 -0
index adf55d406..13b838752 100644
@@ -3,6 +3,7 @@ EXPERIMENTAL {
 
 	__rte_node_register;
 	__rte_node_stream_alloc;
+	__rte_node_stream_alloc_size;
 
 	rte_graph_create;
 	rte_graph_destroy;
@@ -16,6 +17,7 @@ EXPERIMENTAL {
 	rte_graph_node_get;
 	rte_graph_node_get_by_name;
 	rte_graph_obj_dump;
+	rte_graph_walk;
 
 	rte_graph_cluster_stats_create;
 	rte_graph_cluster_stats_destroy;
@@ -28,10 +30,18 @@ EXPERIMENTAL {
 	rte_node_edge_get;
 	rte_node_edge_shrink;
 	rte_node_edge_update;
+	rte_node_enqueue;
+	rte_node_enqueue_x1;
+	rte_node_enqueue_x2;
+	rte_node_enqueue_x4;
+	rte_node_enqueue_next;
 	rte_node_from_name;
 	rte_node_id_to_name;
 	rte_node_list_dump;
 	rte_node_max_count;
+	rte_node_next_stream_get;
+	rte_node_next_stream_put;
+	rte_node_next_stream_move;
 
 	local: *;
 };
lib/librte_graph/rte_graph_worker.h +434 -0

440 lines not on this page · raw message

Signature · 1 line
2.25.1
#13 lore raw

PATCH 13/26 graph: add unit test case

In reply to [email protected]

From: Kiran Kumar K <[email protected]>

Adding the unit test to test the functionality of node and graph APIs. Testing includes registering a node, cloning a node, creating a graph, perform graph walk, collecting stats and all node and graph debug APIs.

example command to test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

---
 app/test/Makefile     |   4 +
 app/test/meson.build  |  10 +-
 app/test/test_graph.c | 820 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 832 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_graph.c
app/test/Makefile +4 -0
index 1f080d162..065582916 100644
@@ -221,6 +221,10 @@ SRCS-y += test_event_timer_adapter.c
 SRCS-y += test_event_crypto_adapter.c
 endif
 
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
+SRCS-y += test_graph.c
+endif
+
 ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
 SRCS-y += test_rawdev.c
 endif
app/test/meson.build +8 -2
index 0a2ce710f..855403932 100644
@@ -51,6 +51,7 @@ test_sources = files('commands.c',
 	'test_fib6_perf.c',
 	'test_func_reentrancy.c',
 	'test_flow_classify.c',
+	'test_graph.c',
 	'test_hash.c',
 	'test_hash_functions.c',
 	'test_hash_multiwriter.c',
@@ -151,7 +152,9 @@ test_deps = ['acl',
 	'rib',
 	'ring',
 	'stack',
-	'timer'
+	'timer',
+	'graph',
+	'node'
 ]
 
 fast_test_names = [
@@ -183,6 +186,7 @@ fast_test_names = [
         'fib6_autotest',
         'func_reentrancy_autotest',
         'flow_classify_autotest',
+	'graph_autotest',
         'hash_autotest',
         'interrupt_autotest',
         'logs_autotest',
@@ -383,13 +387,15 @@ endforeach
 test_dep_objs += cc.find_library('execinfo', required: false)
 
 link_libs = []
+link_nodes = []
 if get_option('default_library') == 'static'
 	link_libs = dpdk_drivers
+	link_nodes = dpdk_graph_nodes
 endif
 
 dpdk_test = executable('dpdk-test',
 	test_sources,
-	link_whole: link_libs,
+	link_whole: link_libs + link_nodes,
 	dependencies: test_dep_objs,
 	c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
 	install_rpath: driver_install_path,
app/test/test_graph.c new file +820 -0

820 lines not on this page · raw message

Signature · 1 line
2.25.1
#14 lore raw

PATCH 14/26 graph: add performance testcase

In reply to [email protected]

From: Pavan Nikhilesh <[email protected]>

Add unit test framework to create and test performance of various graph models.

example command to test:

echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

---
 app/test/Makefile          |    1 +
 app/test/meson.build       |    1 +
 app/test/test_graph_perf.c | 1057 ++++++++++++++++++++++++++++++++++++
 3 files changed, 1059 insertions(+)
 create mode 100644 app/test/test_graph_perf.c
app/test/Makefile +1 -0
index 065582916..cd2801427 100644
@@ -223,6 +223,7 @@ endif
 
 ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
 SRCS-y += test_graph.c
+SRCS-y += test_graph_perf.c
 endif
 
 ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
app/test/meson.build +1 -0
index 855403932..3fe155778 100644
@@ -52,6 +52,7 @@ test_sources = files('commands.c',
 	'test_func_reentrancy.c',
 	'test_flow_classify.c',
 	'test_graph.c',
+	'test_graph_perf.c',
 	'test_hash.c',
 	'test_hash_functions.c',
 	'test_hash_multiwriter.c',
app/test/test_graph_perf.c new file +1057 -0

1,057 lines not on this page · raw message

Signature · 1 line
2.25.1
#15 lore raw

PATCH 15/26 node: add log infra and null node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add log infra for node specific logging.
Also, add null rte_node that just ignores all the objects directed to it.

---
 MAINTAINERS                          |  5 +++++
 config/common_base                   |  5 +++++
 doc/api/doxy-api.conf.in             |  1 +
 lib/Makefile                         |  3 +++
 lib/librte_node/Makefile             | 20 ++++++++++++++++++++
 lib/librte_node/log.c                | 14 ++++++++++++++
 lib/librte_node/meson.build          |  6 ++++++
 lib/librte_node/node_private.h       | 22 ++++++++++++++++++++++
 lib/librte_node/null.c               | 23 +++++++++++++++++++++++
 lib/librte_node/rte_node_version.map |  6 ++++++
 lib/meson.build                      |  5 ++++-
 meson.build                          |  1 +
 mk/rte.app.mk                        |  1 +
 13 files changed, 111 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/Makefile
 create mode 100644 lib/librte_node/log.c
 create mode 100644 lib/librte_node/meson.build
 create mode 100644 lib/librte_node/node_private.h
 create mode 100644 lib/librte_node/null.c
 create mode 100644 lib/librte_node/rte_node_version.map
MAINTAINERS +5 -0
index 32d0ea032..3959ed19a 100644
@@ -1473,6 +1473,11 @@ M: Jerin Jacob <[email protected]>
 M: Kiran Kumar K <[email protected]>
 F: lib/librte_graph/
 
+Nodes - EXPERIMENTAL
+M: Nithin Dabilpuram <[email protected]>
+M: Pavan Nikhilesh <[email protected]>
+F: lib/librte_node/
+
 
 Test Applications
 -----------------
config/common_base +5 -0
index 04a96aef5..442949ff1 100644
@@ -1082,6 +1082,11 @@ CONFIG_RTE_LIBRTE_GRAPH=y
 CONFIG_RTE_GRAPH_BURST_SIZE=256
 CONFIG_RTE_LIBRTE_GRAPH_STATS=y
 
+#
+# Compile librte_node
+#
+CONFIG_RTE_LIBRTE_NODE=y
+
 #
 # Compile the test application
 #
doc/api/doxy-api.conf.in +1 -0
index 759a7213e..1d4f1a37d 100644
@@ -49,6 +49,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_mempool \
                           @TOPDIR@/lib/librte_meter \
                           @TOPDIR@/lib/librte_metrics \
+                          @TOPDIR@/lib/librte_node \
                           @TOPDIR@/lib/librte_net \
                           @TOPDIR@/lib/librte_pci \
                           @TOPDIR@/lib/librte_pdump \
lib/Makefile +3 -0
index 1f572b659..50d61a338 100644
@@ -122,6 +122,9 @@ DEPDIRS-librte_rcu := librte_eal
 DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
 DEPDIRS-librte_graph := librte_eal
 
+DIRS-$(CONFIG_RTE_LIBRTE_NODE) += librte_node
+DEPDIRS-librte_node := librte_graph librte_lpm librte_ethdev librte_mbuf
+
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
 endif
lib/librte_node/Makefile new file +20 -0
new file mode 100644
index 000000000..5f9be6c2e
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_node.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_graph
+
+EXPORT_MAP := rte_node_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/log.c new file +14 -0
new file mode 100644
index 000000000..f035f91e8
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "node_private.h"
+
+int rte_node_logtype;
+
+RTE_INIT(rte_node_init_log)
+{
+	rte_node_logtype = rte_log_register("lib.node");
+	if (rte_node_logtype >= 0)
+		rte_log_set_level(rte_node_logtype, RTE_LOG_INFO);
+}
lib/librte_node/meson.build new file +6 -0
new file mode 100644
index 000000000..655970614
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+sources = files('null.c', 'log.c')
+allow_experimental_apis = true
+deps += ['graph']
lib/librte_node/node_private.h new file +22 -0
new file mode 100644
index 000000000..f30902a94
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __NODE_PRIVATE_H__
+#define __NODE_PRIVATE_H__
+
+#include <rte_common.h>
+#include <rte_log.h>
+
+extern int rte_node_logtype;
+#define NODE_LOG(level, node_name, ...)                                        \
+	rte_log(RTE_LOG_##level, rte_node_logtype,                             \
+		RTE_FMT("NODE %s: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n",  \
+			node_name, __func__, __LINE__,                         \
+			RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define node_err(node_name, ...) NODE_LOG(ERR, node_name, __VA_ARGS__)
+#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
+#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+
+#endif /* __NODE_PRIVATE_H__ */
lib/librte_node/null.c new file +23 -0
new file mode 100644
index 000000000..c7cd8b6df
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_graph.h>
+
+static uint16_t
+null(struct rte_graph *graph, struct rte_node *node, void **objs,
+	uint16_t nb_objs)
+{
+	RTE_SET_USED(node);
+	RTE_SET_USED(objs);
+	RTE_SET_USED(graph);
+
+	return nb_objs;
+}
+
+static struct rte_node_register null_node = {
+	.name = "null",
+	.process = null,
+};
+
+RTE_NODE_REGISTER(null_node);
lib/librte_node/rte_node_version.map new file +6 -0
new file mode 100644
index 000000000..f87163bb9
@@ -0,0 +1,6 @@
+EXPERIMENTAL {
+	global:
+
+	rte_node_logtype;
+	local: *;
+};
lib/meson.build +4 -1
index c43d86bb9..147129b0b 100644
@@ -30,7 +30,7 @@ libraries = [
 	# add pkt framework libs which use other libs from above
 	'port', 'table', 'pipeline',
 	# flow_classify lib depends on pkt framework table lib
-	'flow_classify', 'bpf', 'graph', 'telemetry']
+	'flow_classify', 'bpf', 'graph', 'node', 'telemetry']
 
 if is_windows
 	libraries = ['kvargs','eal'] # only supported libraries for windows
@@ -186,6 +186,9 @@ foreach l:libraries
 
 			dpdk_libraries = [shared_lib] + dpdk_libraries
 			dpdk_static_libraries = [static_lib] + dpdk_static_libraries
+			if libname == 'rte_node'
+				dpdk_graph_nodes = [static_lib]
+			endif
 		endif # sources.length() > 0
 
 		set_variable('shared_rte_' + name, shared_dep)
meson.build +1 -0
index b7ae9c8d9..811c96421 100644
@@ -16,6 +16,7 @@ cc = meson.get_compiler('c')
 dpdk_conf = configuration_data()
 dpdk_libraries = []
 dpdk_static_libraries = []
+dpdk_graph_nodes = []
 dpdk_driver_classes = []
 dpdk_drivers = []
 dpdk_extra_ldflags = []
mk/rte.app.mk +1 -0
index b1195f09a..68d7806a4 100644
@@ -99,6 +99,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RCU)            += -lrte_rcu
 _LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH)          += -lrte_graph
+_LDLIBS-$(CONFIG_RTE_LIBRTE_NODE)           += -lrte_node
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KNI)            += -lrte_kni
Signature · 1 line
2.25.1
#16 lore raw

PATCH 16/26 node: add ethdev Rx node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add source rte_node ethdev_rx process function and register it. This node is a source node that will be called periodically and when called, performs rte_eth_rx_burst() on a specific (port, queue) pair and enqueue them as stream of objects to next node.

---
 lib/librte_node/Makefile         |   3 +-
 lib/librte_node/ethdev_rx.c      | 221 +++++++++++++++++++++++++++++++
 lib/librte_node/ethdev_rx_priv.h |  81 +++++++++++
 lib/librte_node/meson.build      |   4 +-
 4 files changed, 306 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_rx.c
 create mode 100644 lib/librte_node/ethdev_rx_priv.h
lib/librte_node/Makefile +2 -1
index 5f9be6c2e..5a5f589ba 100644
@@ -9,12 +9,13 @@ LIB = librte_node.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph
+LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
 
 EXPORT_MAP := rte_node_version.map
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_rx.c new file +221 -0

221 lines not on this page · raw message

lib/librte_node/ethdev_rx_priv.h new file +81 -0

81 lines not on this page · raw message

lib/librte_node/meson.build +2 -2
index 655970614..4adfd6580 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c')
 allow_experimental_apis = true
-deps += ['graph']
+deps += ['graph', 'ethdev']
Signature · 1 line
2.25.1
#17 lore raw

PATCH 17/26 node: add ethdev Tx node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add rte_node ethdev_tx process function and register it to graph infra. This node has a specific (port, tx-queue) as context and it enqueue's all the packets received to that specific queue pair. When rte_eth_tx_burst() i.e enqueue to queue pair fails, packets are forwarded to pkt_drop node to be free'd.

---
 lib/librte_node/Makefile         |  3 +-
 lib/librte_node/ethdev_tx.c      | 86 ++++++++++++++++++++++++++++++++
 lib/librte_node/ethdev_tx_priv.h | 62 +++++++++++++++++++++++
 lib/librte_node/meson.build      |  4 +-
 4 files changed, 152 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_tx.c
 create mode 100644 lib/librte_node/ethdev_tx_priv.h
lib/librte_node/Makefile +2 -1
index 5a5f589ba..14a293982 100644
@@ -9,7 +9,7 @@ LIB = librte_node.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
 
 EXPORT_MAP := rte_node_version.map
 
@@ -17,5 +17,6 @@ EXPORT_MAP := rte_node_version.map
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_tx.c new file +86 -0

86 lines not on this page · raw message

lib/librte_node/ethdev_tx_priv.h new file +62 -0

62 lines not on this page · raw message

lib/librte_node/meson.build +2 -2
index 4adfd6580..c3e35276b 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
 allow_experimental_apis = true
-deps += ['graph', 'ethdev']
+deps += ['graph', 'mbuf', 'ethdev']
Signature · 1 line
2.25.1
#18 lore raw

PATCH 18/26 node: add ethdev Rx and Tx node ctrl API

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add ctrl api to setup ethdev_rx and ethdev_tx node. This ctrl api clones 'N' number of ethdev_rx and ethdev_tx nodes with specific (port, queue) pairs updated in their context. All the ethdev ports and queues are setup before this api is called.

---
 doc/api/doxy-api-index.md            |  2 +
 lib/librte_node/Makefile             |  6 +-
 lib/librte_node/ethdev_ctrl.c        | 99 ++++++++++++++++++++++++++++
 lib/librte_node/meson.build          |  5 +-
 lib/librte_node/node_private.h       | 74 +++++++++++++++++++++
 lib/librte_node/rte_node_eth_api.h   | 70 ++++++++++++++++++++
 lib/librte_node/rte_node_version.map |  1 +
 7 files changed, 254 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_ctrl.c
 create mode 100644 lib/librte_node/rte_node_eth_api.h
doc/api/doxy-api-index.md +2 -0
index fd2ff64d7..1784674df 100644
@@ -161,6 +161,8 @@ The public API headers are grouped by topics:
     [table_action]     (@ref rte_table_action.h)
   * [graph]            (@ref rte_graph.h):
     [graph_worker]     (@ref rte_graph_worker.h)
+  * graph_nodes:
+    [eth_node]         (@ref rte_node_eth_api.h),
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_node/Makefile +5 -1
index 14a293982..c7b291c3c 100644
@@ -9,7 +9,7 @@ LIB = librte_node.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
 
 EXPORT_MAP := rte_node_version.map
 
@@ -18,5 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_ctrl.c new file +99 -0

99 lines not on this page · raw message

lib/librte_node/meson.build +3 -2
index c3e35276b..f34162aa7 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
+headers = files('rte_node_eth_api.h')
 allow_experimental_apis = true
-deps += ['graph', 'mbuf', 'ethdev']
+deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
lib/librte_node/node_private.h +74 -0

85 lines not on this page · raw message

lib/librte_node/rte_node_eth_api.h new file +70 -0

70 lines not on this page · raw message

lib/librte_node/rte_node_version.map +1 -0
index f87163bb9..c6c71bd02 100644
@@ -1,6 +1,7 @@
 EXPERIMENTAL {
 	global:
 
+	rte_node_eth_config;
 	rte_node_logtype;
 	local: *;
 };
Signature · 1 line
2.25.1
#19 lore raw

PATCH 19/26 node: ipv4 lookup for arm64

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add arm64 specific IPv4 lookup process function for ip4_lookup node. This node performs LPM lookup on every packet received and forwards it to a next node that is identified by lookup result.

---
 doc/api/doxy-api-index.md          |   1 +
 lib/librte_node/Makefile           |   4 +-
 lib/librte_node/ip4_lookup.c       | 306 +++++++++++++++++++++++++++++
 lib/librte_node/meson.build        |   5 +-
 lib/librte_node/rte_node_ip4_api.h |  43 ++++
 5 files changed, 356 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ip4_lookup.c
 create mode 100644 lib/librte_node/rte_node_ip4_api.h
doc/api/doxy-api-index.md +1 -0
index 1784674df..3908fddde 100644
@@ -163,6 +163,7 @@ The public API headers are grouped by topics:
     [graph_worker]     (@ref rte_graph_worker.h)
   * graph_nodes:
     [eth_node]         (@ref rte_node_eth_api.h),
+    [ip4_node]         (@ref rte_node_ip4_api.h)
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_node/Makefile +3 -1
index c7b291c3c..d85a9e9eb 100644
@@ -9,7 +9,7 @@ LIB = librte_node.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_lpm -lrte_ethdev -lrte_mempool
 
 EXPORT_MAP := rte_node_version.map
 
@@ -19,8 +19,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
 
 # install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ip4_lookup.c new file +306 -0

306 lines not on this page · raw message

lib/librte_node/meson.build +3 -2
index f34162aa7..ddea1237a 100644
@@ -1,7 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
-headers = files('rte_node_eth_api.h')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
+		'ethdev_ctrl.c')
+headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
lib/librte_node/rte_node_ip4_api.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.25.1
#20 lore raw

PATCH 20/26 node: ipv4 lookup for x86

In reply to [email protected]

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

---
 lib/librte_node/ip4_lookup.c | 245 +++++++++++++++++++++++++++++++++++
 1 file changed, 245 insertions(+)
lib/librte_node/ip4_lookup.c +245 -0
index d7fcd1158..c003e9c91 100644
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;
+
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr *) +
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr *) +
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next0,
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next1,
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next2,
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next3,
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0, from[0]);
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node, next_index);
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
Signature · 1 line
2.25.1
#21 lore raw

Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 46 lines · On 18/03/2020 21:35, [email protected] wrote:

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

---
 lib/librte_node/ip4_lookup.c | 245 +++++++++++++++++++++++++++++++++++
 1 file changed, 245 insertions(+)
lib/librte_node/ip4_lookup.c +23 -0
index d7fcd1158..c003e9c91 100644
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 310

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr *) +
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);

Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

Quoted 205 lines
lib/librte_node/ip4_lookup.c +201 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr *) +
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr *) +
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next0,
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next1,
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next2,
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next3,
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0, from[0]);
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node, next_index);
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#22 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 55 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node

*node,

Quoted 11 lines
lib/librte_node/ip4_lookup.c +11 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Ack will remove in v2 for x86.

Quoted 14 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines
lib/librte_node/ip4_lookup.c +4 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 5 lines

Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without unrolling the loop?. Also, in future if we are using rte_rib and fib with a CPU supporting wider SIMD we might need to unroll them further (AVX256 AND 512 currently rte_lpm_lookup uses only 128bit since it is only uses SSE extension).

Quoted 241 lines
Quoted 4 lines
lib/librte_node/ip4_lookup.c +3 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +8 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +5 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 16 lines
lib/librte_node/ip4_lookup.c +13 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next0,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next1,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next2,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 420

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0,

from[0]);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 15 lines
lib/librte_node/ip4_lookup.c +11 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#23 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 87 lines · Pavan Nikhilesh Bhagavatula
Quoted 55 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node

*node,

Quoted 11 lines
lib/librte_node/ip4_lookup.c +11 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Ack will remove in v2 for x86.

Quoted 14 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines
lib/librte_node/ip4_lookup.c +4 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 5 lines

Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

Quoted 4 lines

How would be take the benefit of rte_lpm_lookupx4 without unrolling the loop?. Also, in future if we are using rte_rib and fib with a CPU supporting wider SIMD we might need to unroll them further (AVX256 AND 512 currently rte_lpm_lookup uses only 128bit since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Quoted 242 lines
Quoted 242 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +3 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +8 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +5 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 16 lines
lib/librte_node/ip4_lookup.c +13 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next0,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next1,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next2,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 420

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0,

from[0]);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 15 lines
lib/librte_node/ip4_lookup.c +11 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#24 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 108 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 15 lines · Pavan Nikhilesh Bhagavatula
Quoted 15 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 49 lines
Quoted 23 lines

*node,

Quoted 11 lines
lib/librte_node/ip4_lookup.c +11 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Ack will remove in v2 for x86.

Quoted 14 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines
lib/librte_node/ip4_lookup.c +4 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 7 lines
Quoted 5 lines

Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 9 lines
Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU supporting

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

Quoted 242 lines
Quoted 242 lines
Quoted 242 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +3 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +8 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +5 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 16 lines
lib/librte_node/ip4_lookup.c +13 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next0,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next1,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next2,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 420

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0,

from[0]);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 15 lines
lib/librte_node/ip4_lookup.c +11 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#25 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 112 lines · Pavan Nikhilesh Bhagavatula
Quoted 112 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 15 lines · Pavan Nikhilesh Bhagavatula
Quoted 15 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 49 lines
Quoted 23 lines

*node,

Quoted 11 lines
lib/librte_node/ip4_lookup.c +11 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Ack will remove in v2 for x86.

Quoted 14 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines
lib/librte_node/ip4_lookup.c +4 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 7 lines
Quoted 5 lines

Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU supporting

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

no, it was for all the needless repetition. hand-unrolling loops serve no purpose but to add verbosity.

Quoted 243 lines
Quoted 243 lines
Quoted 243 lines
Quoted 243 lines
Quoted 6 lines
lib/librte_node/ip4_lookup.c +3 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from >= 11)) {

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+				      1);
+			rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+						       struct rte_ether_hdr

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +8 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +5 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 16 lines
lib/librte_node/ip4_lookup.c +13 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next0,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next1,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next2,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node,

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop &

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 420

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0,

from[0]);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 15 lines
lib/librte_node/ip4_lookup.c +11 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#26 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Hi Ray,

I have tried to avoid hand unrolling loops and found the following observations.

1. Although it decreases LOC it also takes away readability too.

	Example: 
	Avoiding unrolled code below

                priv[0].u64[0] = rte_node_mbuf_priv1(mbuf[0])->u;
                priv[0].u64[1] = rte_node_mbuf_priv1(mbuf[1])->u;
                priv[1].u64[0] = rte_node_mbuf_priv1(mbuf[2])->u;
                priv[1].u64[1] = rte_node_mbuf_priv1(mbuf[3])->u;

                /* Increment checksum by one. */
                priv[0].u32[1] += rte_cpu_to_be_16(0x0100);
                priv[0].u32[3] += rte_cpu_to_be_16(0x0100);
                priv[1].u32[1] += rte_cpu_to_be_16(0x0100);
                priv[1].u32[3] += rte_cpu_to_be_16(0x0100);

                d = rte_pktmbuf_mtod(mbuf[0], void *);
                rte_memcpy(d, nh[priv[0].u16[0]].rewrite_data,
                                nh[priv[0].u16[0]].rewrite_len);
                next[0] = nh[priv[0].u16[0]].tx_node;
                ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                                sizeof(struct rte_ether_hdr));
                ip->time_to_live = priv[0].u16[1] - 1;
                ip->hdr_checksum = priv[0].u16[2] + priv[0].u16[3];

                d = rte_pktmbuf_mtod(mbuf[1], void *);
                rte_memcpy(d, nh[priv[0].u16[4]].rewrite_data,
                           nh[priv[0].u16[4]].rewrite_len);
                next[1] = nh[priv[0].u16[4]].tx_node;
                ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                      sizeof(struct rte_ether_hdr));
                ip->time_to_live = priv[0].u16[5] - 1;
                ip->hdr_checksum = priv[0].u16[6] + priv[0].u16[7];

                d = rte_pktmbuf_mtod(mbuf[2], void *);
                rte_memcpy(d, nh[priv[1].u16[0]].rewrite_data,
                           nh[priv[1].u16[0]].rewrite_len);
                next[2] = nh[priv[1].u16[0]].tx_node;
                ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                      sizeof(struct rte_ether_hdr));
                ip->time_to_live = priv[1].u16[1] - 1;
                ip->hdr_checksum = priv[1].u16[2] + priv[1].u16[3];

                d = rte_pktmbuf_mtod(mbuf[3], void *);
                rte_memcpy(d, nh[priv[1].u16[4]].rewrite_data,
                           nh[priv[1].u16[4]].rewrite_len);
                next[3] = nh[priv[1].u16[4]].tx_node;
                ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                      sizeof(struct rte_ether_hdr));
                ip->time_to_live = priv[1].u16[5] - 1;
                ip->hdr_checksum = priv[1].u16[6] + priv[1].u16[7];

	Leads to something like:
	
                for (i = 0, j = 0; i < BUF_PER_LOOP; i += 2, j++) {
                        priv[j].u64[0] = rte_node_mbuf_priv1(mbuf[i])->u;
                        priv[j].u32[1] += rte_cpu_to_be_16(0x0100);
                        d = rte_pktmbuf_mtod(mbuf[i], void *);
                        ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                                            sizeof(struct rte_ether_hdr));
                        ip->time_to_live = priv[j].u16[1] - 1;
                        ip->hdr_checksum = priv[j].u16[2] + priv[j].u16[3];
                        rte_memcpy(d, nh[priv[j].u16[0]].rewrite_data,
                                   nh[priv[j].u16[0]].rewrite_len);

                        next[i] = nh[priv[j].u16[0]].tx_node;

                        priv[j].u64[1] = rte_node_mbuf_priv1(mbuf[i + 1])->u;
                        priv[j].u32[3] += rte_cpu_to_be_16(0x0100);
                        d = rte_pktmbuf_mtod(mbuf[i + 1], void *);
                        ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
                                        sizeof(struct rte_ether_hdr));
                        ip->time_to_live = priv[j].u16[5] - 1;
                        ip->hdr_checksum = priv[j].u16[6] + priv[j].u16[7];
                        rte_memcpy(d, nh[priv[j].u16[4]].rewrite_data,
                                   nh[priv[j].u16[4]].rewrite_len);

                        next[i + 1] = nh[priv[j].u16[4]].tx_node;
                }
	Which is kind of unreadable.

2. Not all compilers are made equal. I found that most of the compilers don’t

     Unroll the loop above even when compiled with `-funroll-all-loops`.
     I have checked with following compilers:
	GCC 9.2.0
	Clang 9.0.1
	Aarch64 GCC 7.3.0
	Aarch64 GCC 9.2.0

3. Performance wise I see a lot of degradation on our platform at least 13%.
On IA with a Broadwell(Xeon E5-2690) and i40e the performance remain same w.r.t Rx/Tx since the hotspot is in the Tx path of the driver which limits the per core capability. But the performance difference in number of cycles per node can be seen below:

Hand unrolling:

+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node                           |calls          |objs           |realloc_count  |objs/call      |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup                     |7765918        |248509344      |1              |32.000         |27.725408      |779.0000   |
|ip4_rewrite                    |7765925        |248509568      |1              |32.000         |27.725408      |425.0000   |
|ethdev_tx-1                    |7765927        |204056223      |1              |26.000         |22.762720      |597.0000   |
|pkt_drop                       |1389170        |44453409       |1              |32.000         |4.962688       |298.0000   |
|ethdev_rx-0-0                  |63604111       |248509792      |2              |32.000         |27.725408      |982.0000   |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+

W/o unrolling:

+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node                           |calls          |objs           |realloc_count  |objs/call      |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup                     |18864640       |603668448      |1              |32.000         |26.051328      |828.0000   |
|ip4_rewrite                    |18864646       |603668640      |1              |32.000         |26.051328      |534.0000   |
|ethdev_tx-1                    |18864648       |527874175      |1              |27.000         |22.780256      |633.0000   |
|pkt_drop                       |2368580        |75794529       |1              |32.000         |3.271072       |286.0000   |
|ethdev_rx-0-0                  |282058226      |603668864      |2              |32.000         |26.051328      |994.0000   |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+

Considering the above findings we would like to continue unrolling the loops by hand.

Regards,
Pavan.

Quoted 424 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Friday, March 20, 2020 2:44 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 6 lines · Pavan Nikhilesh Bhagavatula
Quoted 6 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 39 lines
Quoted 39 lines

for x86

Quoted 15 lines · Pavan Nikhilesh Bhagavatula
Quoted 15 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,

**pkts;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index =

RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 292

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Ack will remove in v2 for x86.

Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 308

Quoted 4 lines
lib/librte_node/ip4_lookup.c +3 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+

rte_prefetch0(rte_pktmbuf_mtod(pkts[i],

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 303

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Suggest you don't reuse the hand-unrolling optimization from

FD.io

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU supporting

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

no, it was for all the needless repetition. hand-unrolling loops serve no purpose but to add verbosity.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +2 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node, · line 335

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 322

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 324

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 325

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 327

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 328

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 330

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 331

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 354

Quoted 6 lines
lib/librte_node/ip4_lookup.c +4 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +1 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum =

ipv4_hdr-

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,

2, 3);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		/* Perform LPM lookup to get NH and next

node */

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]

&

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines
lib/librte_node/ip4_lookup.c +5 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^

next1) |

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			(next_index ^ next2) | (next_index ^

next3);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated

till now

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next0,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next1,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next2,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm,

rte_be_to_cpu_32(ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 5 lines

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop

&

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

0xFFFF;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated

till now

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node,

next0,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

from[0]);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+	rte_node_next_stream_put(graph, node, next_index,

held);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +4 -0
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#27 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 7 lines · Pavan Nikhilesh Bhagavatula

Hi Ray,

I have tried to avoid hand unrolling loops and found the following observations.

1. Although it decreases LOC it also takes away readability too. Example:
Avoiding unrolled code below

[SNIP]

Quoted 1 line

Which is kind of unreadable.

I am confused - isn't it exactly the same code? You still haven't completely unrolled the loop either?

I don't know how one is readable and the other is not.

Quoted 8 lines

2. Not all compilers are made equal. I found that most of the compilers don’t

     Unroll the loop above even when compiled with `-funroll-all-loops`.
     I have checked with following compilers:
	GCC 9.2.0
	Clang 9.0.1
	Aarch64 GCC 7.3.0
	Aarch64 GCC 9.2.0

Compilers have been unrolling fixed length loops for as long time - this isn't new technology.

If the compiler isn't unrolling you are doing something that makes it think it is a bad idea. Hand unrolling the loop isn't the solution, understanding what the compiler is doing is a better idea.

In front of your for loop insert, to indicate to the compiler what you want to do. #pragma unroll BUF_PER_LOOP

With clang you can ask it why it is not unrolling the loop with the following switches. (output is verbose, but the reason is in there).

-Rpass=loop-unroll -Rpass-missed=loop-unroll

Quoted 2 lines

3. Performance wise I see a lot of degradation on our platform at least 13%.

Is the loop being unrolled?

Quoted 456 lines

On IA with a Broadwell(Xeon E5-2690) and i40e the performance remain same w.r.t Rx/Tx since the hotspot is in the Tx path of the driver which limits the per core capability. But the performance difference in number of cycles per node can be seen below:

Hand unrolling:

+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node                           |calls          |objs           |realloc_count  |objs/call      |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup                     |7765918        |248509344      |1              |32.000         |27.725408      |779.0000   |
|ip4_rewrite                    |7765925        |248509568      |1              |32.000         |27.725408      |425.0000   |
|ethdev_tx-1                    |7765927        |204056223      |1              |26.000         |22.762720      |597.0000   |
|pkt_drop                       |1389170        |44453409       |1              |32.000         |4.962688       |298.0000   |
|ethdev_rx-0-0                  |63604111       |248509792      |2              |32.000         |27.725408      |982.0000   |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+

W/o unrolling:

+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node                           |calls          |objs           |realloc_count  |objs/call      |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup                     |18864640       |603668448      |1              |32.000         |26.051328      |828.0000   |
|ip4_rewrite                    |18864646       |603668640      |1              |32.000         |26.051328      |534.0000   |
|ethdev_tx-1                    |18864648       |527874175      |1              |27.000         |22.780256      |633.0000   |
|pkt_drop                       |2368580        |75794529       |1              |32.000         |3.271072       |286.0000   |
|ethdev_rx-0-0                  |282058226      |603668864      |2              |32.000         |26.051328      |994.0000   |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+

Considering the above findings we would like to continue unrolling the loops by hand.

Regards,
Pavan.

Quoted 424 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Friday, March 20, 2020 2:44 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 6 lines · Pavan Nikhilesh Bhagavatula
Quoted 6 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 39 lines
Quoted 39 lines

for x86

Quoted 15 lines · Pavan Nikhilesh Bhagavatula
Quoted 15 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 8 lines

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 5 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 271

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 274

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,

**pkts;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 290

Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index =

RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 292

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any benefit.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Ack will remove in v2 for x86.

Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 302

Quoted 4 lines
lib/librte_node/ip4_lookup.c +3 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+

rte_prefetch0(rte_pktmbuf_mtod(pkts[i],

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 303

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 309

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Suggest you don't reuse the hand-unrolling optimization from

FD.io

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU supporting

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

no, it was for all the needless repetition. hand-unrolling loops serve no purpose but to add verbosity.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +2 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 321

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 322

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 324

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 325

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 327

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 328

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 330

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 331

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 344

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 348

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 354

Quoted 6 lines
lib/librte_node/ip4_lookup.c +4 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 358

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +1 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum =

ipv4_hdr-

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,

2, 3);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		/* Perform LPM lookup to get NH and next

node */

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]

&

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 5 lines
lib/librte_node/ip4_lookup.c +5 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^

next1) |

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			(next_index ^ next2) | (next_index ^

next3);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated

till now

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next0,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next1,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next2,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm,

rte_be_to_cpu_32(ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop

&

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated

till now

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node,

next0,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

from[0]);

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+	rte_node_next_stream_put(graph, node, next_index,

held);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +4 -0
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#28 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 24 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Tuesday, March 24, 2020 8:08 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 3 lines · Pavan Nikhilesh Bhagavatula

Hi Ray,

I have tried to avoid hand unrolling loops and found the following

observations.

Quoted 4 lines

1. Although it decreases LOC it also takes away readability too. Example:
Avoiding unrolled code below

Quoted 7 lines

[SNIP]

Quoted 1 line

Which is kind of unreadable.

I am confused - isn't it exactly the same code? You still haven't completely unrolled the loop either?

I don't know how one is readable and the other is not.

I guess it’s a matter of personal preference.

Quoted 11 lines
Quoted 3 lines

2. Not all compilers are made equal. I found that most of the

compilers don’t

Quoted 6 lines
     Unroll the loop above even when compiled with `-funroll-all-loops`.
     I have checked with following compilers:
	GCC 9.2.0
	Clang 9.0.1
	Aarch64 GCC 7.3.0
	Aarch64 GCC 9.2.0
Quoted 3 lines

Compilers have been unrolling fixed length loops for as long time - this isn't new technology.

In theory, I agree with your view, but even the latest compiler is not doing a decent job on unrolling the loop.
We can revisit this scheme, if and when compiler smart enough to do this as just unrolling the loops is not good enough. It has to do it better than hand unrolling. For example on arm64 GCC doesn’t merge load/stores to load/store pairs when unrolling. (both ldr/str and ldp/stp latency is 3cyc and effectively ldp cost is halved).

Even on x86 we see extra ~100 cycles as mentioned in [1].

Quoted 8 lines

If the compiler isn't unrolling you are doing something that makes it think it is a bad idea.
Hand unrolling the loop isn't the solution, understanding what the compiler is doing is a better idea.

In front of your for loop insert, to indicate to the compiler what you want to do.
#pragma unroll BUF_PER_LOOP

Can you check which versions of compiler which this pragma works on?

Most of the gcc versions that I have tried just spit out the following warning. ../lib/librte_node/ip4_rewrite.c:59: warning: ignoring #pragma unroll BUF_PER_LOOP [-Wunknown-pragmas]
59 | #pragma unroll BUF_PER_LOOP

Quoted 7 lines

With clang you can ask it why it is not unrolling the loop with the following switches.
(output is verbose, but the reason is in there).

-Rpass=loop-unroll -Rpass-missed=loop-unroll

../lib/librte_node/ip4_rewrite.c:57:2: remark: unrolled loop by a factor of 4 with run-time trip count [-Rpass=loop-unroll]

        for (i = 0; i < nb_objs; i++) {
        ^
Quoted 6 lines
Quoted 2 lines

3. Performance wise I see a lot of degradation on our platform at least

13%.

Is the loop being unrolled?

Yes, in a suboptimal way. https://pastebin.com/nkXvzMiW

we decide to stick with hand unrolling based on the test result instead of depending on compilers mercy to do it for us[2]. if you think, it can be improved then please submit a patch.

[2]https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88760

Quoted 7 lines
Quoted 1 line

On IA with a Broadwell(Xeon E5-2690) and i40e the performance

remain same w.r.t Rx/Tx since the

Quoted 1 line

hotspot is in the Tx path of the driver which limits the per core

capability.

Quoted 1 line

But the performance difference in number of cycles per node can

be seen below:

Quoted 1 line

[1]

Quoted 479 lines
Quoted 2 lines

Hand unrolling:
+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|Node |calls |objs |realloc_count |objs/call

|objs/sec(10E6) |cycles/call|

Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|ip4_lookup |7765918 |248509344 |1 |32.000

|27.725408      |779.0000   |
> |ip4_rewrite                    |7765925        |248509568      |1              |32.000
|27.725408      |425.0000   |
> |ethdev_tx-1                    |7765927        |204056223      |1              |26.000
|22.762720      |597.0000   |
> |pkt_drop                       |1389170        |44453409       |1              |32.000
|4.962688       |298.0000   |
> |ethdev_rx-0-0                  |63604111       |248509792      |2              |32.000
|27.725408      |982.0000   |
Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 4 lines

W/o unrolling:

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|Node |calls |objs |realloc_count |objs/call

|objs/sec(10E6) |cycles/call|

Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|ip4_lookup |18864640 |603668448 |1 |32.000

|26.051328      |828.0000   |
> |ip4_rewrite                    |18864646       |603668640      |1              |32.000
|26.051328      |534.0000   |
> |ethdev_tx-1                    |18864648       |527874175      |1              |27.000
|22.780256      |633.0000   |
> |pkt_drop                       |2368580        |75794529       |1              |32.000
|3.271072       |286.0000   |
> |ethdev_rx-0-0                  |282058226      |603668864      |2              |32.000
|26.051328      |994.0000   |
Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 2 lines

Considering the above findings we would like to continue unrolling

the loops by hand.

Quoted 8 lines

Regards,
Pavan.

Quoted 4 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Friday, March 20, 2020 2:44 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 29 lines
Quoted 29 lines

for x86

Quoted 6 lines · Pavan Nikhilesh Bhagavatula
Quoted 6 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 8 lines
Quoted 8 lines

for x86

Quoted 3 lines · Pavan Nikhilesh Bhagavatula
Quoted 3 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 2 lines

From: Pavan Nikhilesh <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Signed-off-by Nithin Dabilpuram

<[email protected]>

Quoted 48 lines
Quoted 48 lines
Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 271

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 274

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,

**pkts;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 290

Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index =

RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 292

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any

benefit.

Quoted 19 lines
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Ack will remove in v2 for x86.

Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 302

Quoted 4 lines
lib/librte_node/ip4_lookup.c +3 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+

rte_prefetch0(rte_pktmbuf_mtod(pkts[i],

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 303

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 309

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Suggest you don't reuse the hand-unrolling optimization from

FD.io

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

VPP.
I have never found any performance benefit from them, and

they

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU

supporting

Quoted 291 lines
Quoted 291 lines
Quoted 11 lines
Quoted 8 lines

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

no, it was for all the needless repetition. hand-unrolling loops serve no purpose but to add verbosity.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +2 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 321

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 322

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 324

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 325

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 327

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 328

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 330

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 331

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 344

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 348

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 354

Quoted 6 lines
lib/librte_node/ip4_lookup.c +4 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 358

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +1 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum =

ipv4_hdr-

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,

2, 3);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		/* Perform LPM lookup to get NH and next

node */

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]

&

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 5 lines
lib/librte_node/ip4_lookup.c +5 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^

next1) |

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			(next_index ^ next2) | (next_index ^

next3);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated

till now

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next0,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next1,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next2,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm,

rte_be_to_cpu_32(ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop

&

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated

till now

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node,

next0,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

from[0]);

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+	rte_node_next_stream_put(graph, node, next_index,

held);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +4 -0
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#29 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 25 lines · Pavan Nikhilesh Bhagavatula
Quoted 25 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Tuesday, March 24, 2020 8:08 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; Nithin Kumar Dabilpuram <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kiran Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86

Quoted 3 lines · Pavan Nikhilesh Bhagavatula

Hi Ray,

I have tried to avoid hand unrolling loops and found the following

observations.

Quoted 4 lines

1. Although it decreases LOC it also takes away readability too. Example:
Avoiding unrolled code below

[SNIP]

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Which is kind of unreadable.

I am confused - isn't it exactly the same code? You still haven't completely unrolled the loop either?

I don't know how one is readable and the other is not.

I guess it’s a matter of personal preference.

You have a rare preference for verbosity, sir.

Quoted 30 lines
Quoted 12 lines
Quoted 4 lines

2. Not all compilers are made equal. I found that most of the

compilers don’t

Quoted 6 lines
     Unroll the loop above even when compiled with `-funroll-all-loops`.
     I have checked with following compilers:
	GCC 9.2.0
	Clang 9.0.1
	Aarch64 GCC 7.3.0
	Aarch64 GCC 9.2.0
Quoted 3 lines

Compilers have been unrolling fixed length loops for as long time - this isn't new technology.

In theory, I agree with your view, but even the latest compiler is not doing a decent job on unrolling the loop.
We can revisit this scheme, if and when compiler smart enough to do this as just unrolling the loops is not good enough. It has to do it better than hand unrolling. For example on arm64 GCC doesn’t merge load/stores to load/store pairs when unrolling. (both ldr/str and ldp/stp latency is 3cyc and effectively ldp cost is halved).

Even on x86 we see extra ~100 cycles as mentioned in [1].

Quoted 5 lines

If the compiler isn't unrolling you are doing something that makes it think it is a bad idea.
Hand unrolling the loop isn't the solution, understanding what the compiler is doing is a better idea.

Quoted 5 lines
Quoted 3 lines

In front of your for loop insert, to indicate to the compiler what you want to do.
#pragma unroll BUF_PER_LOOP

Can you check which versions of compiler which this pragma works on?

https://gcc.gnu.org/onlinedocs/gcc/Loop-Specific-Pragmas.html
https://clang.llvm.org/docs/AttributeReference.htm

Quoted 4 lines

Most of the gcc versions that I have tried just spit out the following warning. ../lib/librte_node/ip4_rewrite.c:59: warning: ignoring #pragma unroll BUF_PER_LOOP [-Wunknown-pragmas]
59 | #pragma unroll BUF_PER_LOOP

Not sure on this one ...

Quoted 11 lines
Quoted 7 lines

With clang you can ask it why it is not unrolling the loop with the following switches.
(output is verbose, but the reason is in there).

-Rpass=loop-unroll -Rpass-missed=loop-unroll

../lib/librte_node/ip4_rewrite.c:57:2: remark: unrolled loop by a factor of 4 with run-time trip count [-Rpass=loop-unroll]

        for (i = 0; i < nb_objs; i++) {
        ^

That is good ...

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

3. Performance wise I see a lot of degradation on our platform at least

13%.

Is the loop being unrolled?

Yes, in a suboptimal way. https://pastebin.com/nkXvzMiW

we decide to stick with hand unrolling based on the test result instead of depending on compilers mercy to do it for us[2].

Why not code in assembler then, taking to it logical conclusion.

Quoted 1 line

if you think, it can be improved then please submit a patch.

I don't have an ARM platform to test on.

From my own work in FD.io VPP, which you are clear borrowing heavily from, to put it kindly. I have never seen an advantage from hand-unrolling compared to letting the compiler do it.

Perhaps this case is the exception.
I would be suspicious though that the compiler is making bad decisions for some reason.

In anycase, if you (and the presumabily the maintainer) are happy with verbose code like this making it's way into the codebase. I won't argue with it.

Quoted 491 lines

[2]https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88760

Quoted 7 lines
Quoted 1 line

On IA with a Broadwell(Xeon E5-2690) and i40e the performance

remain same w.r.t Rx/Tx since the

Quoted 1 line

hotspot is in the Tx path of the driver which limits the per core

capability.

Quoted 1 line

But the performance difference in number of cycles per node can

be seen below:

Quoted 1 line

[1]

Quoted 479 lines
Quoted 2 lines

Hand unrolling:
+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|Node |calls |objs |realloc_count |objs/call

|objs/sec(10E6) |cycles/call|

Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|ip4_lookup |7765918 |248509344 |1 |32.000

|27.725408      |779.0000   |
> |ip4_rewrite                    |7765925        |248509568      |1              |32.000
|27.725408      |425.0000   |
> |ethdev_tx-1                    |7765927        |204056223      |1              |26.000
|22.762720      |597.0000   |
> |pkt_drop                       |1389170        |44453409       |1              |32.000
|4.962688       |298.0000   |
> |ethdev_rx-0-0                  |63604111       |248509792      |2              |32.000
|27.725408      |982.0000   |
Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 4 lines

W/o unrolling:

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|Node |calls |objs |realloc_count |objs/call

|objs/sec(10E6) |cycles/call|

Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 1 line

|ip4_lookup |18864640 |603668448 |1 |32.000

|26.051328      |828.0000   |
> |ip4_rewrite                    |18864646       |603668640      |1              |32.000
|26.051328      |534.0000   |
> |ethdev_tx-1                    |18864648       |527874175      |1              |27.000
|22.780256      |633.0000   |
> |pkt_drop                       |2368580        |75794529       |1              |32.000
|3.271072       |286.0000   |
> |ethdev_rx-0-0                  |282058226      |603668864      |2              |32.000
|26.051328      |994.0000   |
Quoted 1 line

+-------------------------------+---------------+---------------+---------------+-

--------------+---------------+-----------+

Quoted 2 lines

Considering the above findings we would like to continue unrolling

the loops by hand.

Quoted 8 lines

Regards,
Pavan.

Quoted 4 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Friday, March 20, 2020 2:44 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 29 lines
Quoted 29 lines

for x86

Quoted 6 lines · Pavan Nikhilesh Bhagavatula
Quoted 6 lines

-----Original Message-----
From: Ray Kinsella <[email protected]>
Sent: Thursday, March 19, 2020 9:21 PM
To: Pavan Nikhilesh Bhagavatula <[email protected]>;

Jerin

Quoted 1 line
Quoted 1 line

Jacob Kollanukkaran <[email protected]>; Nithin Kumar

Dabilpuram

Quoted 3 lines
Quoted 3 lines

<[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected];

Kiran

Quoted 2 lines
Quoted 2 lines

Kumar Kokkilagadda <[email protected]>
Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4

lookup

Quoted 8 lines
Quoted 8 lines

for x86

Quoted 3 lines · Pavan Nikhilesh Bhagavatula
Quoted 3 lines · On 18/03/2020 21:35, [email protected] wrote:
Quoted 2 lines

From: Pavan Nikhilesh <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Signed-off-by Nithin Dabilpuram

<[email protected]>

Quoted 48 lines
Quoted 48 lines
Quoted 23 lines
Quoted 23 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
---
 lib/librte_node/ip4_lookup.c | 245

+++++++++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines

Review

Quoted 3 lines
 1 file changed, 245 insertions(+)
(quoted diff) +0 -0

b/lib/librte_node/ip4_lookup.c

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct

Quoted 4 lines

index d7fcd1158..c003e9c91 100644

lib/librte_node/ip4_lookup.c +0 -0
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct

rte_graph

Quoted 9 lines
Quoted 9 lines

*graph, struct rte_node *node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 271

Quoted 8 lines
lib/librte_node/ip4_lookup.c +5 -0
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct

rte_node

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*node,

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 274

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,

**pkts;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	struct rte_ether_hdr *eth_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 290

Quoted 8 lines
lib/librte_node/ip4_lookup.c +8 -0
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index =

RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 292

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+	/* Drop node */
+	drop_nh =

((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 299

Quoted 7 lines
lib/librte_node/ip4_lookup.c +7 -0
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;

I doubt this initial prefetch of the first 4 packets has any

benefit.

Quoted 19 lines
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Ack will remove in v2 for x86.

Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 302

Quoted 4 lines
lib/librte_node/ip4_lookup.c +3 -0
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++) {
+

rte_prefetch0(rte_pktmbuf_mtod(pkts[i],

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 303

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 309

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+				      1);
+		}
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node,

next_index, nb_objs);

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Suggest you don't reuse the hand-unrolling optimization from

FD.io

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

VPP.
I have never found any performance benefit from them, and

they

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

make the code unnecessarily verbose.

How would be take the benefit of rte_lpm_lookupx4 without

unrolling the loop?.

Quoted 1 line

Also, in future if we are using rte_rib and fib with a CPU

supporting

Quoted 291 lines
Quoted 291 lines
Quoted 11 lines
Quoted 8 lines

wider SIMD we might

Quoted 1 line

need to unroll them further (AVX256 AND 512 currently

rte_lpm_lookup uses only 128bit

Quoted 1 line

since it is only uses SSE extension).

Let the compiler do it for you, but using a constant vector length. for (int i=0; i < 4; ++i) { ... }

Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?

no, it was for all the needless repetition. hand-unrolling loops serve no purpose but to add verbosity.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
lib/librte_node/ip4_lookup.c +2 -0
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 321

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+		if (likely(n_left_from >= 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from >= 7)) {
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 322

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 324

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 325

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 327

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 328

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 330

Quoted 2 lines
lib/librte_node/ip4_lookup.c +2 -0
+				      1);
+
	rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 331

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+						       struct

rte_ether_hdr

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*) +

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 344

Quoted 13 lines
lib/librte_node/ip4_lookup.c +13 -0
+				      1);
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 348

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 354

Quoted 6 lines
lib/librte_node/ip4_lookup.c +4 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
 time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf1, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c ip4_lookup_node_process(struct · line 358

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum =

ipv4_hdr-

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +1 -0
 hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf2 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf2, struct

rte_ether_hdr *);

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum =

ipv4_hdr-

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +3 -0
+
+		/* Extract DIP of mbuf3 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf3, struct

rte_ether_hdr *);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,

2, 3);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		/* Perform LPM lookup to get NH and next

node */

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]

&

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]

&

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 5 lines
lib/librte_node/ip4_lookup.c +5 -0
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^

next1) |

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			(next_index ^ next2) | (next_index ^

next3);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated

till now

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 12 lines
lib/librte_node/ip4_lookup.c +12 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next0,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next1,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

next2,

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph,

node,

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

next3,

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+

lib/librte_node/ip4_lookup.c

Quoted 10 lines
lib/librte_node/ip4_lookup.c +10 -0
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		eth_hdr = rte_pktmbuf_mtod(mbuf0, struct

rte_ether_hdr *);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_node/ip4_lookup.c

Quoted 3 lines
lib/librte_node/ip4_lookup.c +3 -0
+		ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum =

ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

hdr_checksum;

lib/librte_node/ip4_lookup.c +1 -0
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-

time_to_live;

lib/librte_node/ip4_lookup.c +2 -0
+
+		rc = rte_lpm_lookup(lpm,

rte_be_to_cpu_32(ipv4_hdr-

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_node/ip4_lookup.c

Quoted 5 lines

dst_addr),

lib/librte_node/ip4_lookup.c +4 -0
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop

&

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

0xFFFF;

lib/librte_node/ip4_lookup.c

Quoted 4 lines
lib/librte_node/ip4_lookup.c +4 -0
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated

till now

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

*/

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+			rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node,

next0,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

from[0]);

lib/librte_node/ip4_lookup.c

Quoted 9 lines
lib/librte_node/ip4_lookup.c +9 -0
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node,

next_index);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_node/ip4_lookup.c

Quoted 6 lines
lib/librte_node/ip4_lookup.c +6 -0
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec *

sizeof(from[0]));

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_node/ip4_lookup.c

Quoted 1 line
lib/librte_node/ip4_lookup.c +1 -0
+	rte_node_next_stream_put(graph, node, next_index,

held);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_node/ip4_lookup.c +4 -0
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
#30 lore raw

PATCH 21/26 node: add ipv4 rewrite node

In reply to [email protected]

From: Kiran Kumar K <[email protected]>

Add ip4 rewrite process function for ip4_rewrite rte_node. On every packet received by this node, header is overwritten with new data before forwarding it to next node. Header data to overwrite with is identified by next hop id passed in mbuf priv data by previous node.

---
 lib/librte_node/Makefile           |   1 +
 lib/librte_node/ip4_rewrite.c      | 269 +++++++++++++++++++++++++++++
 lib/librte_node/ip4_rewrite_priv.h |  55 ++++++
 lib/librte_node/meson.build        |   2 +-
 4 files changed, 326 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/ip4_rewrite.c
 create mode 100644 lib/librte_node/ip4_rewrite_priv.h
lib/librte_node/Makefile +1 -0
index d85a9e9eb..be7e7e338 100644
@@ -20,6 +20,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
lib/librte_node/ip4_rewrite.c new file +269 -0

269 lines not on this page · raw message

lib/librte_node/ip4_rewrite_priv.h new file +55 -0

55 lines not on this page · raw message

lib/librte_node/meson.build +1 -1
index ddea1237a..ef056d9af 100644
@@ -2,7 +2,7 @@
 # Copyright(C) 2020 Marvell International Ltd.
 
 sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
-		'ethdev_ctrl.c')
+		'ip4_rewrite.c', 'ethdev_ctrl.c')
 headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
Signature · 1 line
2.25.1
#31 lore raw

PATCH 22/26 node: add ipv4 rewrite and lookup ctrl API

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add ip4_rewrite and ip4_lookup ctrl API. ip4_lookup ctrl API is used to add route entries for LPM lookup with result data containing next hop id and next proto. ip4_rewrite ctrl API is used to add rewrite data for every next hop.

---
 lib/librte_node/ethdev_ctrl.c        | 18 ++++++-
 lib/librte_node/ip4_lookup.c         | 80 ++++++++++++++++++++++++++++
 lib/librte_node/ip4_rewrite.c        | 56 +++++++++++++++++++
 lib/librte_node/ip4_rewrite_priv.h   | 22 ++++++++
 lib/librte_node/rte_node_ip4_api.h   | 44 +++++++++++++++
 lib/librte_node/rte_node_version.map |  2 +
 6 files changed, 221 insertions(+), 1 deletion(-)
lib/librte_node/ethdev_ctrl.c +17 -1
index 971cf0fb8..48ef8746f 100644
@@ -10,6 +10,7 @@
 
 #include "ethdev_rx_priv.h"
 #include "ethdev_tx_priv.h"
+#include "ip4_rewrite_priv.h"
 #include "node_private.h"
 
 static struct ethdev_ctrl {
@@ -20,14 +21,17 @@ int
 rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
 		    uint16_t nb_graphs)
 {
+	struct rte_node_register *ip4_rewrite_node;
 	struct ethdev_tx_node_main *tx_node_data;
 	uint16_t tx_q_used, rx_q_used, port_id;
 	struct rte_node_register *tx_node;
 	char name[RTE_NODE_NAMESIZE];
+	const char *next_nodes = name;
 	struct rte_mempool *mp;
+	int i, j, rc;
 	uint32_t id;
-	int i, j;
 
+	ip4_rewrite_node = ip4_rewrite_node_get();
 	tx_node_data = ethdev_tx_node_data_get();
 	tx_node = ethdev_tx_node_get();
 	for (i = 0; i < nb_confs; i++) {
@@ -92,6 +96,18 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
 
 		node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
 			 name, id);
+
+		/* Prepare the actual name of the cloned node */
+		snprintf(name, sizeof(name), "ethdev_tx-%u", port_id);
+
+		/* Add this tx port node as next to ip4_rewrite_node */
+		rte_node_edge_update(ip4_rewrite_node->id, RTE_EDGE_ID_INVALID,
+				     &next_nodes, 1);
+		/* Assuming edge id is the last one alloc'ed */
+		rc = ip4_rewrite_set_next(
+			port_id, rte_node_edge_count(ip4_rewrite_node->id) - 1);
+		if (rc < 0)
+			return rc;
 	}
 
 	ctrl.nb_graphs = nb_graphs;
lib/librte_node/ip4_lookup.c +80 -0

98 lines not on this page · raw message

lib/librte_node/ip4_rewrite.c +56 -0

66 lines not on this page · raw message

lib/librte_node/ip4_rewrite_priv.h +22 -0
index 420996a03..80f0abdc9 100644
@@ -48,6 +48,28 @@ struct ip4_rewrite_node_main {
 	/**< Next index of each configured port. */
 };
 
+/**
+ * @internal
+ *
+ * Get the ipv4 rewrite node.
+ *
+ * @retrun
+ *   Pointer to the ipv4 rewrite node.
+ */
+struct rte_node_register *ip4_rewrite_node_get(void);
+
+/**
+ * @internal
+ *
+ * Set the Edge index of a given port_id.
+ *
+ * @param port_id
+ *   Ethernet port identifier.
+ * @param next_index
+ *   Edge index of the Given Tx node.
+ */
+int ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index);
+
 #ifdef __cplusplus
 }
 #endif
lib/librte_node/rte_node_ip4_api.h +44 -0

50 lines not on this page · raw message

lib/librte_node/rte_node_version.map +2 -0
index c6c71bd02..a799b0d38 100644
@@ -2,6 +2,8 @@ EXPERIMENTAL {
 	global:
 
 	rte_node_eth_config;
+	rte_node_ip4_route_add;
+	rte_node_ip4_rewrite_add;
 	rte_node_logtype;
 	local: *;
 };
Signature · 1 line
2.25.1
#32 lore raw

PATCH 23/26 node: add pkt drop node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add pkt drop node process function for pkt_drop rte_node. This node simply free's every object received as an rte_mbuf to its rte_pktmbuf pool.

---
 lib/librte_node/Makefile    |  1 +
 lib/librte_node/meson.build |  2 +-
 lib/librte_node/pkt_drop.c  | 26 ++++++++++++++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/pkt_drop.c
lib/librte_node/Makefile +1 -0
index be7e7e338..aaf041580 100644
@@ -21,6 +21,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += pkt_drop.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
lib/librte_node/meson.build +1 -1
index ef056d9af..59e11e5b4 100644
@@ -2,7 +2,7 @@
 # Copyright(C) 2020 Marvell International Ltd.
 
 sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
-		'ip4_rewrite.c', 'ethdev_ctrl.c')
+		'ip4_rewrite.c', 'pkt_drop.c', 'ethdev_ctrl.c')
 headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
lib/librte_node/pkt_drop.c new file +26 -0
new file mode 100644
index 000000000..c35001323
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_graph.h>
+#include <rte_mbuf.h>
+
+static uint16_t
+pkt_drop_process(struct rte_graph *graph, struct rte_node *node, void **objs,
+		 uint16_t nb_objs)
+{
+	RTE_SET_USED(node);
+	RTE_SET_USED(graph);
+
+	rte_pktmbuf_free_bulk((struct rte_mbuf **)objs, nb_objs);
+
+	return nb_objs;
+}
+
+static struct rte_node_register pkt_drop_node = {
+	.process = pkt_drop_process,
+	.name = "pkt_drop",
+};
+
+RTE_NODE_REGISTER(pkt_drop_node);
Signature · 1 line
2.25.1
#33 lore raw

PATCH 24/26 l3fwd-graph: add graph based l3fwd skeleton

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add graph based l3fwd application skeleton with cmdline parsing support inline with normal l3fwd.

---
 MAINTAINERS                      |   3 +
 examples/Makefile                |   3 +
 examples/l3fwd-graph/Makefile    |  58 ++++
 examples/l3fwd-graph/main.c      | 528 +++++++++++++++++++++++++++++++
 examples/l3fwd-graph/meson.build |  13 +
 examples/meson.build             |   6 +-
 6 files changed, 609 insertions(+), 2 deletions(-)
 create mode 100644 examples/l3fwd-graph/Makefile
 create mode 100644 examples/l3fwd-graph/main.c
 create mode 100644 examples/l3fwd-graph/meson.build
MAINTAINERS +3 -0
index 3959ed19a..73debcd6d 100644
@@ -1580,6 +1580,9 @@ F: doc/guides/sample_app_ug/l2_forward_event.rst
 F: examples/l3fwd/
 F: doc/guides/sample_app_ug/l3_forward.rst
 
+M: Nithin Dabilpuram <[email protected]>
+F: examples/l3fwd-graph/
+
 F: examples/link_status_interrupt/
 F: doc/guides/sample_app_ug/link_status_intr.rst
 
examples/Makefile +3 -0
index feff79784..b7e99a2f7 100644
@@ -51,6 +51,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_ACL) += l3fwd-acl
 ifeq ($(CONFIG_RTE_LIBRTE_LPM)$(CONFIG_RTE_LIBRTE_HASH),yy)
 DIRS-$(CONFIG_RTE_LIBRTE_POWER) += l3fwd-power
 endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH),y)
+DIRS-y += l3fwd-graph
+endif
 DIRS-y += link_status_interrupt
 DIRS-y += multi_process
 DIRS-y += ntb
examples/l3fwd-graph/Makefile new file +58 -0

58 lines not on this page · raw message

examples/l3fwd-graph/main.c new file +528 -0

528 lines not on this page · raw message

examples/l3fwd-graph/meson.build new file +13 -0
new file mode 100644
index 000000000..a816bd890
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+deps += ['graph', 'eal', 'lpm', 'ethdev', 'node' ]
+sources = files(
+	'main.c'
+)
+allow_experimental_apis = true
examples/meson.build +4 -2
index 1f2b6f516..3b540012f 100644
@@ -2,8 +2,10 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 driver_libs = []
+node_libs = []
 if get_option('default_library') == 'static'
 	driver_libs = dpdk_drivers
+	node_libs = dpdk_graph_nodes
 endif
 
 execinfo = cc.find_library('execinfo', required: false)
@@ -23,7 +25,7 @@ all_examples = [
 	'l2fwd', 'l2fwd-cat', 'l2fwd-event',
 	'l2fwd-crypto', 'l2fwd-jobstats',
 	'l2fwd-keepalive', 'l3fwd',
-	'l3fwd-acl', 'l3fwd-power',
+	'l3fwd-acl', 'l3fwd-power', 'l3fwd-graph',
 	'link_status_interrupt',
 	'multi_process/client_server_mp/mp_client',
 	'multi_process/client_server_mp/mp_server',
@@ -99,7 +101,7 @@ foreach example: examples
 		endif
 		executable('dpdk-' + name, sources,
 			include_directories: includes,
-			link_whole: driver_libs,
+			link_whole: driver_libs + node_libs,
 			link_args: dpdk_extra_ldflags,
 			c_args: cflags,
 			dependencies: dep_objs)
Signature · 1 line
2.25.1
#34 lore raw

PATCH 25/26 l3fwd-graph: add ethdev configuration changes

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add changes to ethdev port and queue configuration based on command line parameters for l3fwd graph application.

---
 examples/l3fwd-graph/main.c | 350 +++++++++++++++++++++++++++++++++++-
 1 file changed, 349 insertions(+), 1 deletion(-)
examples/l3fwd-graph/main.c +349 -1
index 8cee8bebd..436987877 100644
@@ -19,8 +19,10 @@
 
 #include <rte_branch_prediction.h>
 #include <rte_common.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
 #include <rte_ethdev.h>
+#include <rte_lcore.h>
 #include <rte_log.h>
 #include <rte_mempool.h>
 #include <rte_per_lcore.h>
@@ -48,6 +50,10 @@
 
 #define NB_SOCKETS 8
 
+/* Static global variables used within this file. */
+static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT;
+static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
+
 /**< Ports set in promiscuous mode off by default. */
 static int promiscuous_on;
 
@@ -59,6 +65,7 @@ static volatile bool force_quit;
 
 /* Ethernet addresses of ports */
 static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+static struct rte_ether_addr ports_eth_addr[RTE_MAX_ETHPORTS];
 xmm_t val_eth[RTE_MAX_ETHPORTS];
 
 /* Mask of enabled ports */
@@ -109,6 +116,8 @@ static struct rte_eth_conf port_conf = {
 	},
 };
 
+static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+
 static int
 check_lcore_params(void)
 {
@@ -164,6 +173,27 @@ check_port_config(void)
 	return 0;
 }
 
+static uint8_t
+get_port_n_rx_queues(const uint16_t port)
+{
+	int queue = -1;
+	uint16_t i;
+
+	for (i = 0; i < nb_lcore_params; ++i) {
+		if (lcore_params[i].port_id == port) {
+			if (lcore_params[i].queue_id == queue + 1)
+				queue = lcore_params[i].queue_id;
+			else
+				rte_exit(EXIT_FAILURE,
+					 "Queue ids of the port %d must be"
+					 " in sequence and must start with 0\n",
+					 lcore_params[i].port_id);
+		}
+	}
+
+	return (uint8_t)(++queue);
+}
+
 static int
 init_lcore_rx_queues(void)
 {
@@ -473,6 +503,120 @@ parse_args(int argc, char **argv)
 	return ret;
 }
 
+static void
+print_ethaddr(const char *name, const struct rte_ether_addr *eth_addr)
+{
+	char buf[RTE_ETHER_ADDR_FMT_SIZE];
+	rte_ether_format_addr(buf, RTE_ETHER_ADDR_FMT_SIZE, eth_addr);
+	printf("%s%s", name, buf);
+}
+
+static int
+init_mem(uint16_t portid, uint32_t nb_mbuf)
+{
+	uint32_t lcore_id;
+	int socketid;
+	char s[64];
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+
+		if (numa_on)
+			socketid = rte_lcore_to_socket_id(lcore_id);
+		else
+			socketid = 0;
+
+		if (socketid >= NB_SOCKETS) {
+			rte_exit(EXIT_FAILURE,
+				 "Socket %d of lcore %u is out of range %d\n",
+				 socketid, lcore_id, NB_SOCKETS);
+		}
+
+		if (pktmbuf_pool[portid][socketid] == NULL) {
+			snprintf(s, sizeof(s), "mbuf_pool_%d:%d", portid,
+				 socketid);
+			/* Create a pool with priv size of a cacheline */
+			pktmbuf_pool[portid][socketid] =
+				rte_pktmbuf_pool_create(
+					s, nb_mbuf, MEMPOOL_CACHE_SIZE,
+					RTE_CACHE_LINE_SIZE,
+					RTE_MBUF_DEFAULT_BUF_SIZE, socketid);
+			if (pktmbuf_pool[portid][socketid] == NULL)
+				rte_exit(EXIT_FAILURE,
+					 "Cannot init mbuf pool on socket %d\n",
+					 socketid);
+			else
+				printf("Allocated mbuf pool on socket %d\n",
+				       socketid);
+		}
+	}
+
+	return 0;
+}
+
+/* Check the link status of all ports in up to 9s, and print them finally */
+static void
+check_all_ports_link_status(uint32_t port_mask)
+{
+#define CHECK_INTERVAL 100 /* 100ms */
+#define MAX_CHECK_TIME 90  /* 9s (90 * 100ms) in total */
+	uint8_t count, all_ports_up, print_flag = 0;
+	struct rte_eth_link link;
+	uint16_t portid;
+
+	printf("\nChecking link status");
+	fflush(stdout);
+	for (count = 0; count <= MAX_CHECK_TIME; count++) {
+		if (force_quit)
+			return;
+		all_ports_up = 1;
+		RTE_ETH_FOREACH_DEV(portid)
+		{
+			if (force_quit)
+				return;
+			if ((port_mask & (1 << portid)) == 0)
+				continue;
+			memset(&link, 0, sizeof(link));
+			rte_eth_link_get_nowait(portid, &link);
+			/* Print link status if flag set */
+			if (print_flag == 1) {
+				if (link.link_status)
+					printf("Port%d Link Up. Speed %u Mbps "
+					       "-%s\n",
+					       portid, link.link_speed,
+					       (link.link_duplex ==
+						ETH_LINK_FULL_DUPLEX)
+						       ? ("full-duplex")
+						       : ("half-duplex\n"));
+				else
+					printf("Port %d Link Down\n", portid);
+				continue;
+			}
+			/* Clear all_ports_up flag if any link down */
+			if (link.link_status == ETH_LINK_DOWN) {
+				all_ports_up = 0;
+				break;
+			}
+		}
+		/* After finally printing all link status, get out */
+		if (print_flag == 1)
+			break;
+
+		if (all_ports_up == 0) {
+			printf(".");
+			fflush(stdout);
+			rte_delay_ms(CHECK_INTERVAL);
+		}
+
+		/* Set the print_flag if all ports up or timeout */
+		if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
+			print_flag = 1;
+			printf("Done\n");
+		}
+	}
+}
+
 static void
 signal_handler(int signum)
 {
@@ -486,7 +630,14 @@ signal_handler(int signum)
 int
 main(int argc, char **argv)
 {
-	uint16_t portid;
+	uint8_t nb_rx_queue, queue, socketid;
+	struct rte_eth_dev_info dev_info;
+	uint32_t n_tx_queue, nb_lcores;
+	struct rte_eth_txconf *txconf;
+	uint16_t queueid, portid;
+	struct lcore_conf *qconf;
+	uint32_t lcore_id;
+	uint32_t nb_ports;
 	int ret;
 
 	/* Init EAL */
@@ -522,6 +673,203 @@ main(int argc, char **argv)
 	if (check_port_config() < 0)
 		rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
 
+	nb_ports = rte_eth_dev_count_avail();
+	nb_lcores = rte_lcore_count();
+
+	/* Initialize all ports */
+	RTE_ETH_FOREACH_DEV(portid)
+	{
+		struct rte_eth_conf local_port_conf = port_conf;
+
+		/* Skip ports that are not enabled */
+		if ((enabled_port_mask & (1 << portid)) == 0) {
+			printf("\nSkipping disabled port %d\n", portid);
+			continue;
+		}
+
+		/* Init port */
+		printf("Initializing port %d ... ", portid);
+		fflush(stdout);
+
+		nb_rx_queue = get_port_n_rx_queues(portid);
+		n_tx_queue = nb_lcores;
+		if (n_tx_queue > MAX_TX_QUEUE_PER_PORT)
+			n_tx_queue = MAX_TX_QUEUE_PER_PORT;
+		printf("Creating queues: nb_rxq=%d nb_txq=%u... ",
+		       nb_rx_queue, n_tx_queue);
+
+		rte_eth_dev_info_get(portid, &dev_info);
+		if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
+			local_port_conf.txmode.offloads |=
+				DEV_TX_OFFLOAD_MBUF_FAST_FREE;
+
+		local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
+			dev_info.flow_type_rss_offloads;
+		if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
+		    port_conf.rx_adv_conf.rss_conf.rss_hf) {
+			printf("Port %u modified RSS hash function based on "
+			       "hardware support,"
+			       "requested:%#" PRIx64 " configured:%#" PRIx64
+			       "\n",
+			       portid, port_conf.rx_adv_conf.rss_conf.rss_hf,
+			       local_port_conf.rx_adv_conf.rss_conf.rss_hf);
+		}
+
+		ret = rte_eth_dev_configure(portid, nb_rx_queue,
+					    n_tx_queue, &local_port_conf);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Cannot configure device: err=%d, port=%d\n",
+				 ret, portid);
+
+		ret = rte_eth_dev_adjust_nb_rx_tx_desc(portid, &nb_rxd,
+						       &nb_txd);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Cannot adjust number of descriptors: err=%d, "
+				 "port=%d\n",
+				 ret, portid);
+
+		rte_eth_macaddr_get(portid, &ports_eth_addr[portid]);
+		print_ethaddr(" Address:", &ports_eth_addr[portid]);
+		printf(", ");
+		print_ethaddr(
+			"Destination:",
+			(const struct rte_ether_addr *)&dest_eth_addr[portid]);
+		printf(", ");
+
+		/*
+		 * prepare src MACs for each port.
+		 */
+		rte_ether_addr_copy(
+			&ports_eth_addr[portid],
+			(struct rte_ether_addr *)(val_eth + portid) + 1);
+
+		/* Init memory */
+		if (!per_port_pool) {
+			/* portid = 0; this is *not* signifying the first port,
+			 * rather, it signifies that portid is ignored.
+			 */
+			ret = init_mem(0, NB_MBUF(nb_ports));
+		} else {
+			ret = init_mem(portid, NB_MBUF(1));
+		}
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE, "init_mem() failed\n");
+
+		/* Init one TX queue per couple (lcore,port) */
+		queueid = 0;
+		for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+			if (rte_lcore_is_enabled(lcore_id) == 0)
+				continue;
+
+			qconf = &lcore_conf[lcore_id];
+
+			if (numa_on)
+				socketid = (uint8_t)rte_lcore_to_socket_id(
+					lcore_id);
+			else
+				socketid = 0;
+
+			printf("txq=%u,%d,%d ", lcore_id, queueid, socketid);
+			fflush(stdout);
+
+			txconf = &dev_info.default_txconf;
+			txconf->offloads = local_port_conf.txmode.offloads;
+			ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
+						     socketid, txconf);
+			if (ret < 0)
+				rte_exit(EXIT_FAILURE,
+					 "rte_eth_tx_queue_setup: err=%d, "
+					 "port=%d\n",
+					 ret, portid);
+			queueid++;
+		}
+
+		printf("\n");
+	}
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+		qconf = &lcore_conf[lcore_id];
+		printf("\nInitializing rx queues on lcore %u ... ", lcore_id);
+		fflush(stdout);
+		/* Init RX queues */
+		for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
+			struct rte_eth_rxconf rxq_conf;
+
+			portid = qconf->rx_queue_list[queue].port_id;
+			queueid = qconf->rx_queue_list[queue].queue_id;
+
+			if (numa_on)
+				socketid = (uint8_t)rte_lcore_to_socket_id(
+					lcore_id);
+			else
+				socketid = 0;
+
+			printf("rxq=%d,%d,%d ", portid, queueid, socketid);
+			fflush(stdout);
+
+			rte_eth_dev_info_get(portid, &dev_info);
+			rxq_conf = dev_info.default_rxconf;
+			rxq_conf.offloads = port_conf.rxmode.offloads;
+			if (!per_port_pool)
+				ret = rte_eth_rx_queue_setup(
+					portid, queueid, nb_rxd, socketid,
+					&rxq_conf, pktmbuf_pool[0][socketid]);
+			else
+				ret = rte_eth_rx_queue_setup(
+					portid, queueid, nb_rxd, socketid,
+					&rxq_conf,
+					pktmbuf_pool[portid][socketid]);
+			if (ret < 0)
+				rte_exit(EXIT_FAILURE,
+					 "rte_eth_rx_queue_setup: err=%d, "
+					 "port=%d\n",
+					 ret, portid);
+
+		}
+	}
+
+	printf("\n");
+
+	/* Start ports */
+	RTE_ETH_FOREACH_DEV(portid)
+	{
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+
+		/* Start device */
+		ret = rte_eth_dev_start(portid);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "rte_eth_dev_start: err=%d, port=%d\n", ret,
+				 portid);
+
+		/*
+		 * If enabled, put device in promiscuous mode.
+		 * This allows IO forwarding mode to forward packets
+		 * to itself through 2 cross-connected  ports of the
+		 * target machine.
+		 */
+		if (promiscuous_on)
+			rte_eth_promiscuous_enable(portid);
+	}
+
+	printf("\n");
+
+	check_all_ports_link_status(enabled_port_mask);
+
+	/* Stop ports */
+	RTE_ETH_FOREACH_DEV(portid) {
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+		printf("Closing port %d...", portid);
+		rte_eth_dev_stop(portid);
+		rte_eth_dev_close(portid);
+		printf(" Done\n");
+	}
 	printf("Bye...\n");
 
 	return ret;
Signature · 1 line
2.25.1
#35 lore raw

PATCH 26/26 l3fwd-graph: add graph config and main loop

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add graph creation, configuration logic and graph main loop. This graph main loop is run on every slave lcore and calls rte_graph_walk() to walk over lcore specific rte_graph. Master core accumulates and prints graph walk stats of all the lcore's graph's.

---
 examples/l3fwd-graph/main.c | 241 +++++++++++++++++++++++++++++++++++-
 1 file changed, 239 insertions(+), 2 deletions(-)
examples/l3fwd-graph/main.c +239 -2
index 436987877..104a4d8db 100644
@@ -22,9 +22,13 @@
 #include <rte_cycles.h>
 #include <rte_eal.h>
 #include <rte_ethdev.h>
+#include <rte_graph_worker.h>
+#include <rte_launch.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
 #include <rte_mempool.h>
+#include <rte_node_eth_api.h>
+#include <rte_node_ip4_api.h>
 #include <rte_per_lcore.h>
 #include <rte_string_fns.h>
 #include <rte_vect.h>
@@ -74,12 +78,17 @@ static uint32_t enabled_port_mask;
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
+	char node_name[RTE_NODE_NAMESIZE];
 };
 
 /* Lcore conf */
 struct lcore_conf {
 	uint16_t n_rx_queue;
 	struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+
+	struct rte_graph *graph;
+	char name[RTE_GRAPH_NAMESIZE];
+	rte_graph_t graph_id;
 } __rte_cache_aligned;
 
 static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
@@ -118,6 +127,25 @@ static struct rte_eth_conf port_conf = {
 
 static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
 
+static struct rte_node_ethdev_config ethdev_conf[RTE_MAX_ETHPORTS];
+
+struct ipv4_l3fwd_lpm_route {
+	uint32_t ip;
+	uint8_t depth;
+	uint8_t if_out;
+};
+
+#define IPV4_L3FWD_LPM_NUM_ROUTES                                              \
+	(sizeof(ipv4_l3fwd_lpm_route_array) /                                  \
+	 sizeof(ipv4_l3fwd_lpm_route_array[0]))
+/* 198.18.0.0/16 are set aside for RFC2544 benchmarking. */
+static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
+	{RTE_IPV4(198, 18, 0, 0), 24, 0}, {RTE_IPV4(198, 18, 1, 0), 24, 1},
+	{RTE_IPV4(198, 18, 2, 0), 24, 2}, {RTE_IPV4(198, 18, 3, 0), 24, 3},
+	{RTE_IPV4(198, 18, 4, 0), 24, 4}, {RTE_IPV4(198, 18, 5, 0), 24, 5},
+	{RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7},
+};
+
 static int
 check_lcore_params(void)
 {
@@ -627,17 +655,87 @@ signal_handler(int signum)
 	}
 }
 
+static void
+print_stats(void)
+{
+	const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+	const char clr[] = {27, '[', '2', 'J', '\0'};
+	struct rte_graph_cluster_stats_param s_param;
+	struct rte_graph_cluster_stats *stats;
+	const char *pattern = "worker_*";
+
+	/* Prepare stats object */
+	memset(&s_param, 0, sizeof(s_param));
+	s_param.f = stdout;
+	s_param.socket_id = SOCKET_ID_ANY;
+	s_param.graph_patterns = &pattern;
+	s_param.nb_graph_patterns = 1;
+
+	stats = rte_graph_cluster_stats_create(&s_param);
+	if (stats == NULL)
+		rte_exit(EXIT_FAILURE, "Unable to create stats object\n");
+
+	while (!force_quit) {
+		/* Clear screen and move to top left */
+		printf("%s%s", clr, topLeft);
+		rte_graph_cluster_stats_get(stats, 0);
+		rte_delay_ms(1E3);
+	}
+
+	rte_graph_cluster_stats_destroy(stats);
+}
+
+/* Main processing loop */
+static int
+graph_main_loop(void *conf)
+{
+	struct lcore_conf *qconf;
+	struct rte_graph *graph;
+	uint32_t lcore_id;
+
+	RTE_SET_USED(conf);
+
+	lcore_id = rte_lcore_id();
+	qconf = &lcore_conf[lcore_id];
+	graph = qconf->graph;
+
+	if (!graph) {
+		RTE_LOG(INFO, L3FWD_GRAPH, "Lcore %u has nothing to do\n",
+			lcore_id);
+		return 0;
+	}
+
+	RTE_LOG(INFO, L3FWD_GRAPH,
+		"Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
+		qconf->name, graph);
+
+	while (likely(!force_quit))
+		rte_graph_walk(graph);
+
+	return 0;
+}
+
 int
 main(int argc, char **argv)
 {
+	const char *node_patterns[64] = {
+		"ip4*",
+		"ethdev_tx-*",
+		"pkt_drop",
+	};
+	uint8_t rewrite_data[2 * sizeof(struct rte_ether_addr)];
 	uint8_t nb_rx_queue, queue, socketid;
+	struct rte_graph_param graph_conf;
 	struct rte_eth_dev_info dev_info;
+	uint32_t nb_ports, nb_conf = 0;
 	uint32_t n_tx_queue, nb_lcores;
 	struct rte_eth_txconf *txconf;
-	uint16_t queueid, portid;
+	uint16_t queueid, portid, i;
 	struct lcore_conf *qconf;
+	uint16_t nb_patterns = 3;
+	uint16_t nb_graphs = 0;
+	uint8_t rewrite_len;
 	uint32_t lcore_id;
-	uint32_t nb_ports;
 	int ret;
 
 	/* Init EAL */
@@ -786,6 +884,18 @@ main(int argc, char **argv)
 			queueid++;
 		}
 
+		/* Setup ethdev node config */
+		ethdev_conf[nb_conf].port_id = portid;
+		ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
+		ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
+		if (!per_port_pool)
+			ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
+
+		else
+			ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
+		ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
+
+		nb_conf++;
 		printf("\n");
 	}
 
@@ -829,11 +939,26 @@ main(int argc, char **argv)
 					 "port=%d\n",
 					 ret, portid);
 
+			/* Add this queue node to its graph */
+			snprintf(qconf->rx_queue_list[queue].node_name,
+				 RTE_NODE_NAMESIZE, "ethdev_rx-%u-%u", portid,
+				 queueid);
+		}
+
+		/* Alloc a graph to this lcore only if source exists  */
+		if (qconf->n_rx_queue) {
+			qconf->graph_id = nb_graphs;
+			nb_graphs++;
 		}
 	}
 
 	printf("\n");
 
+	/* Ethdev node config, skip rx queue mapping */
+	ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
+	if (ret)
+		rte_exit(EXIT_FAILURE, "rte_node_eth_config: err=%d\n", ret);
+
 	/* Start ports */
 	RTE_ETH_FOREACH_DEV(portid)
 	{
@@ -861,6 +986,118 @@ main(int argc, char **argv)
 
 	check_all_ports_link_status(enabled_port_mask);
 
+	/* Graph Initialization */
+	memset(&graph_conf, 0, sizeof(graph_conf));
+	graph_conf.node_patterns = node_patterns;
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		rte_graph_t graph_id;
+		rte_edge_t i;
+
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+
+		qconf = &lcore_conf[lcore_id];
+
+		/* Skip graph creation if no source exists */
+		if (!qconf->n_rx_queue)
+			continue;
+
+		/* Add rx node patterns of this lcore */
+		for (i = 0; i < qconf->n_rx_queue; i++) {
+			graph_conf.node_patterns[nb_patterns + i] =
+				qconf->rx_queue_list[i].node_name;
+		}
+
+		graph_conf.nb_node_patterns = nb_patterns + i;
+		graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
+
+		snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+			 lcore_id);
+
+		graph_id = rte_graph_create(qconf->name, &graph_conf);
+		if (graph_id != qconf->graph_id)
+			rte_exit(EXIT_FAILURE,
+				 "rte_graph_create(): graph_id=%d not "
+				 " as expected for lcore %u(%u\n",
+				 graph_id, lcore_id, qconf->graph_id);
+
+		qconf->graph = rte_graph_lookup(qconf->name);
+		if (!qconf->graph)
+			rte_exit(EXIT_FAILURE,
+				 "rte_graph_lookup(): graph %s not found\n",
+				 qconf->name);
+	}
+
+	memset(&rewrite_data, 0, sizeof(rewrite_data));
+	rewrite_len = sizeof(rewrite_data);
+
+	/* Add route to ip4 graph infra */
+	for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
+		char route_str[INET6_ADDRSTRLEN * 4];
+		char abuf[INET6_ADDRSTRLEN];
+		struct in_addr in;
+		uint32_t dst_port;
+		uint16_t next_hop;
+
+		/* Skip unused ports */
+		if ((1 << ipv4_l3fwd_lpm_route_array[i].if_out &
+		     enabled_port_mask) == 0)
+			continue;
+
+		dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
+		next_hop = i;
+
+		in.s_addr = htonl(ipv4_l3fwd_lpm_route_array[i].ip);
+		snprintf(route_str, sizeof(route_str), "%s / %d (%d)",
+			 inet_ntop(AF_INET, &in, abuf, sizeof(abuf)),
+			 ipv4_l3fwd_lpm_route_array[i].depth,
+			 ipv4_l3fwd_lpm_route_array[i].if_out);
+
+		ret = rte_node_ip4_route_add(
+			ipv4_l3fwd_lpm_route_array[i].ip,
+			ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
+			RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
+
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Unable to add ip4 route %s to graph\n",
+				 route_str);
+
+		memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
+
+		/* Add next hop for a given destination */
+		ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
+					       rewrite_len, dst_port);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Unable to add next hop %u for "
+				 "route %s\n",
+				 next_hop, route_str);
+
+		RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
+			route_str, next_hop);
+	}
+
+	/* Launch per-lcore init on every slave lcore */
+	rte_eal_mp_remote_launch(graph_main_loop, NULL, SKIP_MASTER);
+
+	/* Accumulate and print stats on master until exit */
+	if (rte_graph_has_stats_feature())
+		print_stats();
+
+	/* Wait for slave cores to exit */
+	ret = 0;
+	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
+		ret = rte_eal_wait_lcore(lcore_id);
+		/* Destroy graph */
+		rte_graph_destroy(lcore_conf[lcore_id].name);
+		if (ret < 0) {
+			ret = -1;
+			break;
+		}
+	}
+
 	/* Stop ports */
 	RTE_ETH_FOREACH_DEV(portid) {
 		if ((enabled_port_mask & (1 << portid)) == 0)
Signature · 1 line
2.25.1
#36 lore raw

v2 cover graph: introduce graph subsystem

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Using graph traversal for packet processing is a proven architecture that has been implemented in various open source libraries.

Graph architecture for packet processing enables abstracting the data processing functions as “nodes” and “links” them together to create a complex “graph” to create reusable/modular data processing functions.

The patchset further includes performance enhancements and modularity to the DPDK as discussed in more detail below.

v2..v1:
------
1) Added programmer guide/implementation documentation and l3fwd-graph doc

Hosted in netlify for easy reference:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Programmer’s Guide:
https://dpdk-graph.netlify.com/doc/html/guides/prog_guide/graph_lib.html

l3fwd-graph doc:
https://dpdk-graph.netlify.com/doc/html/guides/sample_app_ug/l3_forward_graph.html

API doc:
https://dpdk-graph.netlify.com/doc/html/api/rte__graph_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__graph__worker_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__node__eth__api_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__node__ip4__api_8h.html

2) Added the release notes for the this feature

3) Fix build issues reported by CI for v1:
http://mails.dpdk.org/archives/test-report/2020-March/121326.html

RFC..v1:
--------

1) Split the patch to more logical ones for review.
2) Added doxygen comments for the API
3) Code cleanup
4) Additional performance improvements.
Delta between l3fwd and l3fwd-graph is negligible now. (~1%) on octeontx2.
5) Added SIMD routines for x86 in additional to arm64.

Addional nodes planned for v20.08
----------------------------------
1) Packet classification node
2) Support for IPV6 LPM node

This patchset contains
-----------------------------
1) The API definition to "create" nodes and "link" together to create a "graph" for packet processing. See, lib/librte_graph/rte_graph.h

2) The Fast path API definition for the graph walker and enqueue function used by the workers. See, lib/librte_graph/rte_graph_worker.h

3) Optimized SW implementation for (1) and (2). See, lib/librte_graph/

4) Test case to verify the graph infrastructure functionality See, app/test/test_graph.c

5) Performance test cases to evaluate the cost of graph walker and nodes enqueue fast-path function for various combinations.

See app/test/test_graph_perf.c

6) Packet processing nodes(Null, Rx, Tx, Pkt drop, IPV4 rewrite, IPv4 lookup)
using graph infrastructure. See lib/librte_node/*

7) An example application to showcase l3fwd
(functionality same as existing examples/l3fwd) using graph infrastructure and use packets processing nodes (item (6)). See examples/l3fwd-graph/.

Performance
-----------
1) Graph walk and node enqueue overhead can be tested with performance test case application [1]
# If all packets go from a node to another node (we call it as # "homerun") then it will be just a pointer swap for a burst of packets. # In the worst case, a couple of handful cycles to move an object from a node to another node.

2) Performance comparison with existing l3fwd (The complete static code with out any nodes) vs modular l3fwd-graph with 5 nodes (ip4_lookup, ip4_rewrite, ethdev_tx, ethdev_rx, pkt_drop). Here is graphical representation of the l3fwd-graph as Graphviz dot file:
http://bit.ly/39UPPGm

# l3fwd-graph performance is -1.2% wrt static l3fwd.

# We have simulated the similar test with existing librte_pipeline

# application [4].
ip_pipline application is -48.62% wrt static l3fwd.

The above results are on octeontx2. It may vary on other platforms. The platforms with higher L1 and L2 caches will have further better performance.

Tested architectures:
--------------------
1) AArch64
2) X86

Identified tweaking for better performance on different targets
---------------------------------------------------------------
1) Test with various burst size values (256, 128, 64, 32) using CONFIG_RTE_GRAPH_BURST_SIZE config option.
Based on our testing, on x86 and arm64 servers, The sweet spot is 256 burst size.
While on arm64 embedded SoCs, it is either 64 or 128.

2) Disable node statistics (use CONFIG_RTE_LIBRTE_GRAPH_STATS config option)
if not needed.

3) Use arm64 optimized memory copy for arm64 architecture by selecting CONFIG_RTE_ARCH_ARM64_MEMCPY.

Commands to run tests
---------------------

[1]
perf test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

[2]
functionality test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

[3]
l3fwd-graph:
./l3fwd-graph -c 0x100 -- -p 0x3 --config="(0, 0, 8)" -P

[4]

# ./ip_pipeline --c 0xff0000 -- -s route.cli

Route.cli: (Copy paste to the shell to avoid dos format issues)

https://pastebin.com/raw/B4Ktx7TT

Jerin Jacob (13):
graph: define the public API for graph support
graph: implement node registration
graph: implement node operations
graph: implement node debug routines
graph: implement internal graph operation helpers
graph: populate fastpath memory for graph reel
graph: implement create and destroy APIs
graph: implement graph operation APIs
graph: implement Graphviz export
graph: implement debug routines
graph: implement stats support
graph: implement fastpath API routines
doc: add graph library programmer's guide guide

Kiran Kumar K (2):
graph: add unit test case
node: add ipv4 rewrite node

Nithin Dabilpuram (11):
node: add log infra and null node
node: add ethdev Rx node
node: add ethdev Tx node
node: add ethdev Rx and Tx node ctrl API
node: ipv4 lookup for arm64
node: add ipv4 rewrite and lookup ctrl API
node: add pkt drop node
l3fwd-graph: add graph based l3fwd skeleton
l3fwd-graph: add ethdev configuration changes
l3fwd-graph: add graph config and main loop
doc: add l3fwd graph application user guide

Pavan Nikhilesh (2):
graph: add performance testcase
node: ipv4 lookup for x86

 MAINTAINERS                                   |   14 +
 app/test/Makefile                             |    7 +
 app/test/meson.build                          |   12 +-
 app/test/test_graph.c                         |  819 ++++
 app/test/test_graph_perf.c                    | 1057 ++++++
 config/common_base                            |   12 +
 config/rte_config.h                           |    4 +
 doc/api/doxy-api-index.md                     |    5 +
 doc/api/doxy-api.conf.in                      |    2 +
 doc/guides/prog_guide/graph_lib.rst           |  397 ++
 .../prog_guide/img/anatomy_of_a_node.svg      | 1078 ++++++
 .../prog_guide/img/graph_mem_layout.svg       |  702 ++++
 doc/guides/prog_guide/img/link_the_nodes.svg  | 3330 +++++++++++++++++
 doc/guides/prog_guide/index.rst               |    1 +
 doc/guides/rel_notes/release_20_05.rst        |   32 +
 doc/guides/sample_app_ug/index.rst            |    1 +
 doc/guides/sample_app_ug/intro.rst            |    4 +
 doc/guides/sample_app_ug/l3_forward_graph.rst |  327 ++
 examples/Makefile                             |    3 +
 examples/l3fwd-graph/Makefile                 |   58 +
 examples/l3fwd-graph/main.c                   | 1111 ++++++
 examples/l3fwd-graph/meson.build              |   13 +
 examples/meson.build                          |    6 +-
 lib/Makefile                                  |    6 +
 lib/librte_graph/Makefile                     |   28 +
 lib/librte_graph/graph.c                      |  589 +++
 lib/librte_graph/graph_debug.c                |   84 +
 lib/librte_graph/graph_ops.c                  |  169 +
 lib/librte_graph/graph_populate.c             |  234 ++
 lib/librte_graph/graph_private.h              |  347 ++
 lib/librte_graph/graph_stats.c                |  406 ++
 lib/librte_graph/meson.build                  |   11 +
 lib/librte_graph/node.c                       |  421 +++
 lib/librte_graph/rte_graph.h                  |  786 ++++
 lib/librte_graph/rte_graph_version.map        |   47 +
 lib/librte_graph/rte_graph_worker.h           |  542 +++
 lib/librte_node/Makefile                      |   32 +
 lib/librte_node/ethdev_ctrl.c                 |  116 +
 lib/librte_node/ethdev_rx.c                   |  221 ++
 lib/librte_node/ethdev_rx_priv.h              |   81 +
 lib/librte_node/ethdev_tx.c                   |   86 +
 lib/librte_node/ethdev_tx_priv.h              |   62 +
 lib/librte_node/ip4_lookup.c                  |  619 +++
 lib/librte_node/ip4_rewrite.c                 |  326 ++
 lib/librte_node/ip4_rewrite_priv.h            |   77 +
 lib/librte_node/log.c                         |   14 +
 lib/librte_node/meson.build                   |   10 +
 lib/librte_node/node_private.h                |   96 +
 lib/librte_node/null.c                        |   23 +
 lib/librte_node/pkt_drop.c                    |   26 +
 lib/librte_node/rte_node_eth_api.h            |   70 +
 lib/librte_node/rte_node_ip4_api.h            |   87 +
 lib/librte_node/rte_node_version.map          |    9 +
 lib/meson.build                               |    5 +-
 meson.build                                   |    1 +
 mk/rte.app.mk                                 |    2 +
 56 files changed, 14623 insertions(+), 5 deletions(-)
 create mode 100644 app/test/test_graph.c
 create mode 100644 app/test/test_graph_perf.c
 create mode 100644 doc/guides/prog_guide/graph_lib.rst
 create mode 100644 doc/guides/prog_guide/img/anatomy_of_a_node.svg
 create mode 100644 doc/guides/prog_guide/img/graph_mem_layout.svg
 create mode 100644 doc/guides/prog_guide/img/link_the_nodes.svg
 create mode 100644 doc/guides/sample_app_ug/l3_forward_graph.rst
 create mode 100644 examples/l3fwd-graph/Makefile
 create mode 100644 examples/l3fwd-graph/main.c
 create mode 100644 examples/l3fwd-graph/meson.build
 create mode 100644 lib/librte_graph/Makefile
 create mode 100644 lib/librte_graph/graph.c
 create mode 100644 lib/librte_graph/graph_debug.c
 create mode 100644 lib/librte_graph/graph_ops.c
 create mode 100644 lib/librte_graph/graph_populate.c
 create mode 100644 lib/librte_graph/graph_private.h
 create mode 100644 lib/librte_graph/graph_stats.c
 create mode 100644 lib/librte_graph/meson.build
 create mode 100644 lib/librte_graph/node.c
 create mode 100644 lib/librte_graph/rte_graph.h
 create mode 100644 lib/librte_graph/rte_graph_version.map
 create mode 100644 lib/librte_graph/rte_graph_worker.h
 create mode 100644 lib/librte_node/Makefile
 create mode 100644 lib/librte_node/ethdev_ctrl.c
 create mode 100644 lib/librte_node/ethdev_rx.c
 create mode 100644 lib/librte_node/ethdev_rx_priv.h
 create mode 100644 lib/librte_node/ethdev_tx.c
 create mode 100644 lib/librte_node/ethdev_tx_priv.h
 create mode 100644 lib/librte_node/ip4_lookup.c
 create mode 100644 lib/librte_node/ip4_rewrite.c
 create mode 100644 lib/librte_node/ip4_rewrite_priv.h
 create mode 100644 lib/librte_node/log.c
 create mode 100644 lib/librte_node/meson.build
 create mode 100644 lib/librte_node/node_private.h
 create mode 100644 lib/librte_node/null.c
 create mode 100644 lib/librte_node/pkt_drop.c
 create mode 100644 lib/librte_node/rte_node_eth_api.h
 create mode 100644 lib/librte_node/rte_node_ip4_api.h
 create mode 100644 lib/librte_node/rte_node_version.map
Signature · 1 line
2.25.1
#37 lore raw

PATCH v2 1/28 graph: define the public API for graph support

From: Jerin Jacob <[email protected]>

Graph architecture abstracts the data processing functions as "node" and "link" them together to create a complex "graph" to enable reusable/modular data processing functions.

These APIs enables graph framework operations such as create, lookup, dump and destroy on graph and node operations such as clone, edge update, and edge shrink, etc. The API also allows creating the stats cluster to monitor per graph and per node stats.

This patch defines the public API for graph support. This patch also adds support for the build infrastructure and update the MAINTAINERS file for the graph subsystem.

---
 MAINTAINERS                            |   5 +
 config/common_base                     |   7 +
 config/rte_config.h                    |   4 +
 doc/api/doxy-api-index.md              |   1 +
 doc/api/doxy-api.conf.in               |   1 +
 lib/Makefile                           |   3 +
 lib/librte_graph/Makefile              |  22 +
 lib/librte_graph/graph.c               |   5 +
 lib/librte_graph/meson.build           |  11 +
 lib/librte_graph/rte_graph.h           | 786 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   3 +
 lib/meson.build                        |   2 +-
 mk/rte.app.mk                          |   1 +
 13 files changed, 850 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/Makefile
 create mode 100644 lib/librte_graph/graph.c
 create mode 100644 lib/librte_graph/meson.build
 create mode 100644 lib/librte_graph/rte_graph.h
 create mode 100644 lib/librte_graph/rte_graph_version.map
MAINTAINERS +5 -0
index db235c2cc..bc7085983 100644
@@ -1469,6 +1469,11 @@ F: examples/bpf/
 F: app/test/test_bpf.c
 F: doc/guides/prog_guide/bpf_lib.rst
 
+Graph - EXPERIMENTAL
+M: Jerin Jacob <[email protected]>
+M: Kiran Kumar K <[email protected]>
+F: lib/librte_graph/
+
 
 Test Applications
 -----------------
config/common_base +7 -0
index c31175f9d..32f982136 100644
@@ -1074,6 +1074,13 @@ CONFIG_RTE_LIBRTE_BPF_ELF=n
 #
 CONFIG_RTE_LIBRTE_IPSEC=y
 
+#
+# Compile librte_graph
+#
+CONFIG_RTE_LIBRTE_GRAPH=y
+CONFIG_RTE_GRAPH_BURST_SIZE=256
+CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+
 #
 # Compile the test application
 #
config/rte_config.h +4 -0
index d30786bc0..e9201fd46 100644
@@ -98,6 +98,10 @@
 /* KNI defines */
 #define RTE_KNI_PREEMPT_DEFAULT 1
 
+/* rte_graph defines */
+#define RTE_GRAPH_BURST_SIZE 256
+#define RTE_LIBRTE_GRAPH_STATS 1
+
 /****** driver defines ********/
 
 /* QuickAssist device */
doc/api/doxy-api-index.md +1 -0
index dff496be0..5cc50f750 100644
@@ -159,6 +159,7 @@ The public API headers are grouped by topics:
   * [pipeline]         (@ref rte_pipeline.h)
     [port_in_action]   (@ref rte_port_in_action.h)
     [table_action]     (@ref rte_table_action.h)
+  * [graph]            (@ref rte_graph.h):
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
doc/api/doxy-api.conf.in +1 -0
index 1c4392eec..759a7213e 100644
@@ -33,6 +33,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_eventdev \
                           @TOPDIR@/lib/librte_fib \
                           @TOPDIR@/lib/librte_flow_classify \
+                          @TOPDIR@/lib/librte_graph \
                           @TOPDIR@/lib/librte_gro \
                           @TOPDIR@/lib/librte_gso \
                           @TOPDIR@/lib/librte_hash \
lib/Makefile +3 -0
index 46b91ae1a..1f572b659 100644
@@ -119,6 +119,9 @@ DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
 DIRS-$(CONFIG_RTE_LIBRTE_RCU) += librte_rcu
 DEPDIRS-librte_rcu := librte_eal
 
+DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
+DEPDIRS-librte_graph := librte_eal
+
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
 endif
lib/librte_graph/Makefile new file +22 -0
new file mode 100644
index 000000000..26fe514f3
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_graph.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_graph_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_graph/graph.c new file +5 -0
new file mode 100644
index 000000000..a55bf443a
@@ -0,0 +1,5 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "rte_graph.h"
lib/librte_graph/meson.build new file +11 -0
new file mode 100644
index 000000000..455cf2ba5
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+name = 'graph'
+
+sources = files('graph.c')
+headers = files('rte_graph.h')
+allow_experimental_apis = true
+
+deps += ['eal']
lib/librte_graph/rte_graph.h new file +786 -0

786 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map new file +3 -0
new file mode 100644
index 000000000..55ef35df5
@@ -0,0 +1,3 @@
+EXPERIMENTAL {
+
+};
lib/meson.build +1 -1
index 9c3cc55d5..c43d86bb9 100644
@@ -30,7 +30,7 @@ libraries = [
 	# add pkt framework libs which use other libs from above
 	'port', 'table', 'pipeline',
 	# flow_classify lib depends on pkt framework table lib
-	'flow_classify', 'bpf', 'telemetry']
+	'flow_classify', 'bpf', 'graph', 'telemetry']
 
 if is_windows
 	libraries = ['kvargs','eal'] # only supported libraries for windows
mk/rte.app.mk +1 -0
index d295ca0a5..b1195f09a 100644
@@ -98,6 +98,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RCU)            += -lrte_rcu
+_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH)          += -lrte_graph
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KNI)            += -lrte_kni
Signature · 1 line
2.25.1
#38 lore raw

PATCH v2 2/28 graph: implement node registration

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding rte_node_register() API implementation includes allocating memory for node object, check for duplicate node name and add the allocated node to STAILQ node_list for future use.

---
 lib/librte_graph/Makefile              |   1 +
 lib/librte_graph/graph.c               |  18 +++-
 lib/librte_graph/graph_private.h       |  75 ++++++++++++++++
 lib/librte_graph/meson.build           |   2 +-
 lib/librte_graph/node.c                | 115 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   4 +
 6 files changed, 213 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_private.h
 create mode 100644 lib/librte_graph/node.c
lib/librte_graph/Makefile +1 -0
index 26fe514f3..933d0ee49 100644
@@ -14,6 +14,7 @@ LDLIBS += -lrte_eal
 EXPORT_MAP := rte_graph_version.map
 
 # all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 
 # install header files
lib/librte_graph/graph.c +17 -1
index a55bf443a..a9c124896 100644
@@ -2,4 +2,20 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
-#include "rte_graph.h"
+#include <rte_spinlock.h>
+
+#include "graph_private.h"
+
+static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+
+void
+graph_spinlock_lock(void)
+{
+	rte_spinlock_lock(&graph_lock);
+}
+
+void
+graph_spinlock_unlock(void)
+{
+	rte_spinlock_unlock(&graph_lock);
+}
lib/librte_graph/graph_private.h new file +75 -0

75 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index 455cf2ba5..5754ac23b 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('graph.c')
+sources = files('node.c', 'graph.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
lib/librte_graph/node.c new file +115 -0

115 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map +4 -0
index 55ef35df5..0884c09f1 100644
@@ -1,3 +1,7 @@
 EXPERIMENTAL {
+	global:
 
+	__rte_node_register;
+
+	local: *;
 };
Signature · 1 line
2.25.1
#39 lore raw

PATCH v2 3/28 graph: implement node operations

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding node-specific API implementation like cloning node, updating edges for the node, shrinking edges of a node, retrieving edges of a node.

---
 lib/librte_graph/graph_private.h       |  10 +
 lib/librte_graph/node.c                | 269 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  10 +
 3 files changed, 289 insertions(+)
lib/librte_graph/graph_private.h +10 -0
index 8b9ff5292..7ed6d01b6 100644
@@ -13,6 +13,16 @@
 
 #include "rte_graph.h"
 
+
+#define ID_CHECK(id, id_max)                                                   \
+	do {                                                                   \
+		if ((id) >= (id_max)) {                                        \
+			rte_errno = EINVAL;                                    \
+			goto fail;                                             \
+		}                                                              \
+	} while (0)
+
+
 /**
  * @internal
  *
lib/librte_graph/node.c +269 -0

272 lines not on this page · raw message

lib/librte_graph/rte_graph_version.map +10 -0
index 0884c09f1..412386356 100644
@@ -3,5 +3,15 @@ EXPERIMENTAL {
 
 	__rte_node_register;
 
+	rte_node_clone;
+	rte_node_edge_count;
+	rte_node_edge_get;
+	rte_node_edge_shrink;
+	rte_node_edge_update;
+	rte_node_from_name;
+	rte_node_id_to_name;
+	rte_node_list_dump;
+	rte_node_max_count;
+
 	local: *;
 };
Signature · 1 line
2.25.1
#40 lore raw

PATCH v2 4/28 graph: implement node debug routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding node debug API implementation support to dump single or all the node objects to the given file.

---
 lib/librte_graph/Makefile              |  1 +
 lib/librte_graph/graph_debug.c         | 25 ++++++++++++++++++++
 lib/librte_graph/graph_private.h       | 12 ++++++++++
 lib/librte_graph/meson.build           |  2 +-
 lib/librte_graph/node.c                | 32 ++++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  1 +
 6 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/graph_debug.c
lib/librte_graph/Makefile +1 -0
index 933d0ee49..2a6d86933 100644
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
lib/librte_graph/graph_debug.c new file +25 -0
new file mode 100644
index 000000000..75238e7ca
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_common.h>
+#include <rte_debug.h>
+
+#include "graph_private.h"
+
+void
+node_dump(FILE *f, struct node *n)
+{
+	rte_edge_t i;
+
+	fprintf(f, "node <%s>\n", n->name);
+	fprintf(f, "  id=%" PRIu32 "\n", n->id);
+	fprintf(f, "  flags=0x%" PRIx64 "\n", n->flags);
+	fprintf(f, "  addr=%p\n", n);
+	fprintf(f, "  process=%p\n", n->process);
+	fprintf(f, "  nb_edges=%d\n", n->nb_edges);
+
+	for (i = 0; i < n->nb_edges; i++)
+		fprintf(f, "     edge[%d] <%s>\n", i, n->next_nodes[i]);
+}
+
lib/librte_graph/graph_private.h +12 -0
index 7ed6d01b6..6db04cee7 100644
@@ -82,4 +82,16 @@ void graph_spinlock_lock(void);
  */
 void graph_spinlock_unlock(void);
 
+/**
+ * @internal
+ *
+ * Dump internal node object data.
+ *
+ * @param f
+ *   FILE pointer to dump the info.
+ * @param g
+ *   Pointer to the internal node object.
+ */
+void node_dump(FILE *f, struct node *n);
+
 #endif /* _RTE_GRAPH_PRIVATE_H_ */
lib/librte_graph/meson.build +1 -1
index 5754ac23b..01512182f 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c')
+sources = files('node.c', 'graph.c', 'graph_debug.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
lib/librte_graph/node.c +32 -0
index 8de857889..2f9c2ea4c 100644
@@ -377,6 +377,38 @@ rte_node_edge_get(rte_node_t id, char *next_nodes[])
 	return rc;
 }
 
+static void
+node_scan_dump(FILE *f, rte_node_t id, bool all)
+{
+	struct node *node;
+
+	RTE_ASSERT(f != NULL);
+	NODE_ID_CHECK(id);
+
+	STAILQ_FOREACH(node, &node_list, next) {
+		if (all == true) {
+			node_dump(f, node);
+		} else if (node->id == id) {
+			node_dump(f, node);
+			return;
+		}
+	}
+fail:
+	return;
+}
+
+void
+rte_node_dump(FILE *f, rte_node_t id)
+{
+	node_scan_dump(f, id, false);
+}
+
+void
+rte_node_list_dump(FILE *f)
+{
+	node_scan_dump(f, 0, true);
+}
+
 rte_node_t
 rte_node_max_count(void)
 {
lib/librte_graph/rte_graph_version.map +1 -0
index 412386356..f2c2139c5 100644
@@ -4,6 +4,7 @@ EXPERIMENTAL {
 	__rte_node_register;
 
 	rte_node_clone;
+	rte_node_dump;
 	rte_node_edge_count;
 	rte_node_edge_get;
 	rte_node_edge_shrink;
Signature · 1 line
2.25.1
#41 lore raw

PATCH v2 5/28 graph: implement internal graph operation helpers

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding internal graph API helpers support to check whether a graph has isolated nodes and any node have a loop to itself and BFS algorithm implementation etc.

---
 lib/librte_graph/Makefile        |   1 +
 lib/librte_graph/graph.c         |   2 +-
 lib/librte_graph/graph_ops.c     | 169 ++++++++++++++++++++++++++++++
 lib/librte_graph/graph_private.h | 173 +++++++++++++++++++++++++++++++
 lib/librte_graph/meson.build     |   2 +-
 5 files changed, 345 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_ops.c
lib/librte_graph/Makefile +1 -0
index 2a6d86933..39ecb2652 100644
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
 
 # install header files
lib/librte_graph/graph.c +1 -1
index a9c124896..4c3f2fe7b 100644
@@ -7,7 +7,7 @@
 #include "graph_private.h"
 
 static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
-
+int rte_graph_logtype;
 void
 graph_spinlock_lock(void)
 {
lib/librte_graph/graph_ops.c new file +169 -0

169 lines not on this page · raw message

lib/librte_graph/graph_private.h +173 -0

203 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index 01512182f..16e0625c1 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_debug.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
 headers = files('rte_graph.h')
 allow_experimental_apis = true
 
Signature · 1 line
2.25.1
#42 lore raw

PATCH v2 6/28 graph: populate fastpath memory for graph reel

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding support to create and populate the memory for graph reel. This includes reserving the memory in the memzone, populating the nodes, Allocating memory for node-specific streams to hold objects.

Once it is populated the reel memory contains the following sections.

+---------------------+
|   Graph Header      |
+---------------------+
|   Fence             |
+---------------------+
|   Circular buffer   |
+---------------------+
|   Fence             |
+---------------------+
|   Node Object 0     |
+------------------- -+
|   Node Object 1     |
+------------------- -+
|   Node Object 2     |
+------------------- -+
|   Node Object n     |
+------------------- -+
---
 lib/librte_graph/Makefile              |   2 +
 lib/librte_graph/graph.c               |  16 ++
 lib/librte_graph/graph_populate.c      | 234 +++++++++++++++++++++++++
 lib/librte_graph/graph_private.h       |  64 +++++++
 lib/librte_graph/meson.build           |   4 +-
 lib/librte_graph/node.c                |   5 +
 lib/librte_graph/rte_graph_version.map |   1 +
 lib/librte_graph/rte_graph_worker.h    | 108 ++++++++++++
 8 files changed, 432 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_graph/graph_populate.c
 create mode 100644 lib/librte_graph/rte_graph_worker.h
lib/librte_graph/Makefile +2 -0
index 39ecb2652..7bfd7d51f 100644
@@ -18,8 +18,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph_worker.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_graph/graph.c +16 -0
index 4c3f2fe7b..e1930b7d2 100644
@@ -2,6 +2,7 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
+#include <rte_malloc.h>
 #include <rte_spinlock.h>
 
 #include "graph_private.h"
@@ -19,3 +20,18 @@ graph_spinlock_unlock(void)
 {
 	rte_spinlock_unlock(&graph_lock);
 }
+
+void __rte_noinline
+__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
+{
+	uint16_t size = node->size;
+
+	RTE_VERIFY(size != UINT16_MAX);
+	/* Allocate double amount of size to avoid immediate realloc */
+	size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, size * 2));
+	node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+					RTE_CACHE_LINE_SIZE, graph->socket);
+	RTE_VERIFY(node->objs);
+	node->size = size;
+	node->realloc_count++;
+}
lib/librte_graph/graph_populate.c new file +234 -0

234 lines not on this page · raw message

lib/librte_graph/graph_private.h +64 -0

76 lines not on this page · raw message

lib/librte_graph/meson.build +2 -2
index 16e0625c1..fb203a5e2 100644
@@ -4,8 +4,8 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
-headers = files('rte_graph.h')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+headers = files('rte_graph.h', 'rte_graph_worker.h')
 allow_experimental_apis = true
 
 deps += ['eal']
lib/librte_graph/node.c +5 -0
index 2f9c2ea4c..639269870 100644
@@ -61,6 +61,11 @@ __rte_node_register(const struct rte_node_register *reg)
 	rte_edge_t i;
 	size_t sz;
 
+	/* Limit Node specific metadata to one cacheline on 64B CL machine */
+	RTE_BUILD_BUG_ON((offsetof(struct rte_node, nodes) -
+			  offsetof(struct rte_node, ctx)) !=
+			 RTE_CACHE_LINE_MIN_SIZE);
+
 	graph_spinlock_lock();
 
 	/* Check sanity */
lib/librte_graph/rte_graph_version.map +1 -0
index f2c2139c5..a9fe1b610 100644
@@ -2,6 +2,7 @@ EXPERIMENTAL {
 	global:
 
 	__rte_node_register;
+	__rte_node_stream_alloc;
 
 	rte_node_clone;
 	rte_node_dump;
lib/librte_graph/rte_graph_worker.h new file +108 -0

108 lines not on this page · raw message

Signature · 1 line
2.25.1
#43 lore raw

PATCH v2 7/28 graph: implement create and destroy APIs

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding graph specific API implementations like graph create and graph destroy. This detect loops in the graph, check for isolated nodes and operation to verify the validity of graph.

---
 lib/librte_graph/graph.c               | 320 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   2 +
 2 files changed, 322 insertions(+)
lib/librte_graph/graph.c +320 -0
index e1930b7d2..dc373231e 100644
@@ -2,13 +2,33 @@
  * Copyright(C) 2020 Marvell International Ltd.
  */
 
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
 #include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_spinlock.h>
+#include <rte_string_fns.h>
 
 #include "graph_private.h"
 
+static struct graph_head graph_list = STAILQ_HEAD_INITIALIZER(graph_list);
 static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+static rte_graph_t graph_id;
 int rte_graph_logtype;
+
+#define GRAPH_ID_CHECK(id) ID_CHECK(id, graph_id)
+
+/* Private functions */
+struct graph_head *
+graph_list_head_get(void)
+{
+	return &graph_list;
+}
+
 void
 graph_spinlock_lock(void)
 {
@@ -21,6 +41,306 @@ graph_spinlock_unlock(void)
 	rte_spinlock_unlock(&graph_lock);
 }
 
+static int
+graph_node_add(struct graph *graph, struct node *node)
+{
+	struct graph_node *graph_node;
+	size_t sz;
+
+	/* Skip the duplicate nodes */
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (strncmp(node->name, graph_node->node->name,
+			    RTE_NODE_NAMESIZE) == 0)
+			return 0;
+
+	/* Allocate new graph node object */
+	sz = sizeof(*graph_node) + node->nb_edges * sizeof(struct node *);
+	graph_node = calloc(1, sz);
+
+	if (graph_node == NULL)
+		SET_ERR_JMP(ENOMEM, free, "Failed to calloc %s object",
+			    node->name);
+
+	/* Initialize the graph node */
+	graph_node->node = node;
+
+	/* Add to graph node list */
+	STAILQ_INSERT_TAIL(&graph->node_list, graph_node, next);
+	return 0;
+
+free:
+	free(graph_node);
+	return -rte_errno;
+}
+
+static struct graph_node *
+node_to_graph_node(struct graph *graph, struct node *node)
+{
+	struct graph_node *graph_node;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (graph_node->node == node)
+			return graph_node;
+
+	SET_ERR_JMP(ENODEV, fail, "Found isolated node %s", node->name);
+fail:
+	return NULL;
+}
+
+static int
+graph_node_edges_add(struct graph *graph)
+{
+	struct graph_node *graph_node;
+	struct node *adjacency;
+	const char *next;
+	rte_edge_t i;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			next = graph_node->node->next_nodes[i];
+			adjacency = node_from_name(next);
+			if (adjacency == NULL)
+				SET_ERR_JMP(EINVAL, fail,
+					    "Node %s not registered", next);
+			if (graph_node_add(graph, adjacency))
+				goto fail;
+		}
+	}
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static int
+graph_adjacency_list_update(struct graph *graph)
+{
+	struct graph_node *graph_node, *tmp;
+	struct node *adjacency;
+	const char *next;
+	rte_edge_t i;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			next = graph_node->node->next_nodes[i];
+			adjacency = node_from_name(next);
+			if (adjacency == NULL)
+				SET_ERR_JMP(EINVAL, fail,
+					    "Node %s not registered", next);
+			tmp = node_to_graph_node(graph, adjacency);
+			if (tmp == NULL)
+				goto fail;
+			graph_node->adjacency_list[i] = tmp;
+		}
+	}
+
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static int
+expand_pattern_to_node(struct graph *graph, const char *pattern)
+{
+	struct node_head *node_head = node_list_head_get();
+	bool found = false;
+	struct node *node;
+
+	/* Check for pattern match */
+	STAILQ_FOREACH(node, node_head, next) {
+		if (fnmatch(pattern, node->name, 0) == 0) {
+			if (graph_node_add(graph, node))
+				goto fail;
+			found = true;
+		}
+	}
+	if (found == false)
+		SET_ERR_JMP(EFAULT, fail, "Pattern %s node not found", pattern);
+
+	return 0;
+fail:
+	return -rte_errno;
+}
+
+static void
+graph_cleanup(struct graph *graph)
+{
+	struct graph_node *graph_node;
+
+	while (!STAILQ_EMPTY(&graph->node_list)) {
+		graph_node = STAILQ_FIRST(&graph->node_list);
+		STAILQ_REMOVE_HEAD(&graph->node_list, next);
+		free(graph_node);
+	}
+}
+
+static int
+graph_node_init(struct graph *graph)
+{
+	struct graph_node *graph_node;
+	const char *name;
+	int rc;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		if (graph_node->node->init) {
+			name = graph_node->node->name;
+			rc = graph_node->node->init(
+				graph->graph,
+				graph_node_name_to_ptr(graph->graph, name));
+			if (rc)
+				SET_ERR_JMP(rc, err, "Node %s init() failed",
+					    name);
+		}
+	}
+
+	return 0;
+err:
+	return -rte_errno;
+}
+
+static void
+graph_node_fini(struct graph *graph)
+{
+	struct graph_node *graph_node;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next)
+		if (graph_node->node->fini)
+			graph_node->node->fini(
+				graph->graph,
+				graph_node_name_to_ptr(graph->graph,
+						       graph_node->node->name));
+}
+
+rte_graph_t
+rte_graph_create(const char *name, struct rte_graph_param *prm)
+{
+	struct graph *graph;
+	const char *pattern;
+	uint16_t i;
+
+	graph_spinlock_lock();
+
+	/* Check arguments sanity */
+	if (prm == NULL)
+		SET_ERR_JMP(EINVAL, fail, "Param should not be NULL");
+
+	if (name == NULL)
+		SET_ERR_JMP(EINVAL, fail, "Graph name should not be NULL");
+
+	/* Check for existence of duplicate graph */
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (strncmp(name, graph->name, RTE_GRAPH_NAMESIZE) == 0)
+			SET_ERR_JMP(EEXIST, fail, "Found duplicate graph %s",
+				    name);
+
+	/* Create graph object */
+	graph = calloc(1, sizeof(*graph));
+	if (graph == NULL)
+		SET_ERR_JMP(ENOMEM, fail, "Failed to calloc graph object");
+
+	/* Initialize the graph object */
+	STAILQ_INIT(&graph->node_list);
+	if (rte_strscpy(graph->name, name, RTE_GRAPH_NAMESIZE) < 0)
+		SET_ERR_JMP(E2BIG, free, "Too big name=%s", name);
+
+	/* Expand node pattern and add the nodes to the graph */
+	for (i = 0; i < prm->nb_node_patterns; i++) {
+		pattern = prm->node_patterns[i];
+		if (expand_pattern_to_node(graph, pattern))
+			goto graph_cleanup;
+	}
+
+	/* Go over all the nodes edges and add them to the graph */
+	if (graph_node_edges_add(graph))
+		goto graph_cleanup;
+
+	/* Update adjacency list of all nodes in the graph */
+	if (graph_adjacency_list_update(graph))
+		goto graph_cleanup;
+
+	/* Make sure at least a source node present in the graph */
+	if (!graph_src_nodes_count(graph))
+		goto graph_cleanup;
+
+	/* Make sure no node is pointing to source node */
+	if (graph_node_has_edge_to_src_node(graph))
+		goto graph_cleanup;
+
+	/* Don't allow node has loop to self */
+	if (graph_node_has_loop_edge(graph))
+		goto graph_cleanup;
+
+	/* Do BFS from src nodes on the graph to find isolated nodes */
+	if (graph_has_isolated_node(graph))
+		goto graph_cleanup;
+
+	/* Initialize graph object */
+	graph->socket = prm->socket_id;
+	graph->src_node_count = graph_src_nodes_count(graph);
+	graph->node_count = graph_nodes_count(graph);
+	graph->id = graph_id;
+
+	/* Allocate the Graph fast path memory and populate the data */
+	if (graph_fp_mem_create(graph))
+		goto graph_cleanup;
+
+	/* Call init() of the all the nodes in the graph */
+	if (graph_node_init(graph))
+		goto graph_mem_destroy;
+
+	/* All good, Lets add the graph to the list */
+	graph_id++;
+	STAILQ_INSERT_TAIL(&graph_list, graph, next);
+
+	graph_spinlock_unlock();
+	return graph->id;
+
+graph_mem_destroy:
+	graph_fp_mem_destroy(graph);
+graph_cleanup:
+	graph_cleanup(graph);
+free:
+	free(graph);
+fail:
+	graph_spinlock_unlock();
+	return RTE_GRAPH_ID_INVALID;
+}
+
+rte_graph_t
+rte_graph_destroy(const char *graph_name)
+{
+	rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+	struct graph *graph, *tmp;
+	const char *name;
+
+	graph_spinlock_lock();
+
+	graph = STAILQ_FIRST(&graph_list);
+	while (graph != NULL) {
+		tmp = STAILQ_NEXT(graph, next);
+		name = graph->name;
+		if (strncmp(name, graph_name, RTE_GRAPH_NAMESIZE) == 0) {
+			/* Call fini() of the all the nodes in the graph */
+			graph_node_fini(graph);
+			/* Destroy graph fast path memory */
+			rc = graph_fp_mem_destroy(graph);
+			if (rc)
+				SET_ERR_JMP(rc, done, "MZ %s free failed",
+					    name);
+
+			graph_cleanup(graph);
+			STAILQ_REMOVE(&graph_list, graph, graph, next);
+			rc = graph->id;
+			free(graph);
+			graph_id--;
+			goto done;
+		}
+		graph = tmp;
+	}
+done:
+	graph_spinlock_unlock();
+	return rc;
+}
+
 void __rte_noinline
 __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 {
lib/librte_graph/rte_graph_version.map +2 -0
index a9fe1b610..dcbd78c02 100644
@@ -4,6 +4,8 @@ EXPERIMENTAL {
 	__rte_node_register;
 	__rte_node_stream_alloc;
 
+	rte_graph_create;
+	rte_graph_destroy;
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#44 lore raw

PATCH v2 8/28 graph: implement graph operation APIs

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding support for graph specific API implementation like Graph lookup to get graph object, retrieving graph ID From name and graph name from ID.

---
 lib/librte_graph/graph.c               | 131 +++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |   8 ++
 2 files changed, 139 insertions(+)
lib/librte_graph/graph.c +131 -0
index dc373231e..7c6a7897d 100644
@@ -210,6 +210,54 @@ graph_node_fini(struct graph *graph)
 						       graph_node->node->name));
 }
 
+static struct rte_graph *
+graph_mem_fixup_node_ctx(struct rte_graph *graph)
+{
+	struct rte_node *node;
+	struct node *node_db;
+	rte_graph_off_t off;
+	rte_node_t count;
+	const char *name;
+
+	rte_graph_foreach_node(count, off, graph, node) {
+		if (node->parent_id == RTE_NODE_ID_INVALID) /* Static node */
+			name = node->name;
+		else /* Cloned node */
+			name = node->parent;
+
+		node_db = node_from_name(name);
+		if (node_db == NULL)
+			SET_ERR_JMP(ENOLINK, fail, "Node %s not found", name);
+		node->process = node_db->process;
+	}
+
+	return graph;
+fail:
+	return NULL;
+}
+
+static struct rte_graph *
+graph_mem_fixup_secondray(struct rte_graph *graph)
+{
+	if (graph == NULL || rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return graph;
+
+	return graph_mem_fixup_node_ctx(graph);
+}
+
+struct rte_graph *
+rte_graph_lookup(const char *name)
+{
+	const struct rte_memzone *mz;
+	struct rte_graph *rc = NULL;
+
+	mz = rte_memzone_lookup(name);
+	if (mz)
+		rc = mz->addr;
+
+	return graph_mem_fixup_secondray(rc);
+}
+
 rte_graph_t
 rte_graph_create(const char *name, struct rte_graph_param *prm)
 {
@@ -341,6 +389,76 @@ rte_graph_destroy(const char *graph_name)
 	return rc;
 }
 
+rte_graph_t
+rte_graph_from_name(const char *name)
+{
+	struct graph *graph;
+
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0)
+			return graph->id;
+
+	return RTE_GRAPH_ID_INVALID;
+}
+
+char *
+rte_graph_id_to_name(rte_graph_t id)
+{
+	struct graph *graph;
+
+	GRAPH_ID_CHECK(id);
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (graph->id == id)
+			return graph->name;
+
+fail:
+	return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get(rte_graph_t gid, uint32_t nid)
+{
+	struct rte_node *node;
+	struct graph *graph;
+	rte_graph_off_t off;
+	rte_node_t count;
+
+	GRAPH_ID_CHECK(gid);
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (graph->id == gid) {
+			rte_graph_foreach_node(count, off, graph->graph,
+						node) {
+				if (node->id == nid)
+					return node;
+			}
+			break;
+		}
+fail:
+	return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get_by_name(const char *graph_name, const char *node_name)
+{
+	struct rte_node *node;
+	struct graph *graph;
+	rte_graph_off_t off;
+	rte_node_t count;
+
+	STAILQ_FOREACH(graph, &graph_list, next)
+		if (!strncmp(graph->name, graph_name, RTE_GRAPH_NAMESIZE)) {
+			rte_graph_foreach_node(count, off, graph->graph,
+						node) {
+				if (!strncmp(node->name, node_name,
+					     RTE_NODE_NAMESIZE))
+					return node;
+			}
+			break;
+		}
+
+	return NULL;
+}
+
 void __rte_noinline
 __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 {
@@ -355,3 +473,16 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->size = size;
 	node->realloc_count++;
 }
+
+rte_graph_t
+rte_graph_max_count(void)
+{
+	return graph_id;
+}
+
+RTE_INIT(rte_graph_init_log)
+{
+	rte_graph_logtype = rte_log_register("lib.graph");
+	if (rte_graph_logtype >= 0)
+		rte_log_set_level(rte_graph_logtype, RTE_LOG_INFO);
+}
lib/librte_graph/rte_graph_version.map +8 -0
index dcbd78c02..5a2b13293 100644
@@ -6,6 +6,14 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_from_name;
+	rte_graph_id_to_name;
+	rte_graph_lookup;
+	rte_graph_list_dump;
+	rte_graph_max_count;
+	rte_graph_node_get;
+	rte_graph_node_get_by_name;
+
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#45 lore raw

PATCH v2 9/28 graph: implement Graphviz export

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding API implementation support exporting the graph object to file. This will export the graph to a file in Graphviz format. It can be viewed in many viewers such as
https://dreampuf.github.io/GraphvizOnline/

---
 lib/librte_graph/graph.c               | 54 ++++++++++++++++++++++++++
 lib/librte_graph/rte_graph_version.map |  1 +
 2 files changed, 55 insertions(+)
lib/librte_graph/graph.c +54 -0
index 7c6a7897d..e0c0b71a7 100644
@@ -474,6 +474,60 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->realloc_count++;
 }
 
+static int
+graph_to_dot(FILE *f, struct graph *graph)
+{
+	const char *src_edge_color = " [color=blue]\n";
+	const char *edge_color = "\n";
+	struct graph_node *graph_node;
+	char *node_name;
+	rte_edge_t i;
+	int rc;
+
+	rc = fprintf(f, "Digraph %s {\n\trankdir=LR;\n", graph->name);
+	if (rc < 0)
+		goto end;
+
+	STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+		node_name = graph_node->node->name;
+		for (i = 0; i < graph_node->node->nb_edges; i++) {
+			rc = fprintf(f, "\t\"%s\"->\"%s\"%s", node_name,
+				     graph_node->adjacency_list[i]->node->name,
+				     graph_node->node->flags & RTE_NODE_SOURCE_F
+					     ? src_edge_color
+					     : edge_color);
+			if (rc < 0)
+				goto end;
+		}
+	}
+	rc = fprintf(f, "}\n");
+	if (rc < 0)
+		goto end;
+
+	return 0;
+end:
+	rte_errno = EBADF;
+	return -rte_errno;
+}
+
+rte_graph_t
+rte_graph_export(const char *name, FILE *f)
+{
+	rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+	struct graph *graph;
+
+	STAILQ_FOREACH(graph, &graph_list, next) {
+		if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0) {
+			rc = graph_to_dot(f, graph);
+			goto end;
+		}
+	}
+	rte_errno = ENOENT;
+end:
+	return rc;
+}
+
+
 rte_graph_t
 rte_graph_max_count(void)
 {
lib/librte_graph/rte_graph_version.map +1 -0
index 5a2b13293..2797be044 100644
@@ -6,6 +6,7 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_export;
 	rte_graph_from_name;
 	rte_graph_id_to_name;
 	rte_graph_lookup;
Signature · 1 line
2.25.1
#46 lore raw

PATCH v2 10/28 graph: implement debug routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for graph specific API to dump the Graph information to a file. This API will dump detailed internal info about node objects and graph objects.

---
 lib/librte_graph/graph.c               | 31 ++++++++++++++
 lib/librte_graph/graph_debug.c         | 59 ++++++++++++++++++++++++++
 lib/librte_graph/graph_private.h       | 13 ++++++
 lib/librte_graph/rte_graph_version.map |  2 +
 4 files changed, 105 insertions(+)
lib/librte_graph/graph.c +31 -0
index e0c0b71a7..e96363777 100644
@@ -527,6 +527,37 @@ rte_graph_export(const char *name, FILE *f)
 	return rc;
 }
 
+static void
+graph_scan_dump(FILE *f, rte_graph_t id, bool all)
+{
+	struct graph *graph;
+
+	RTE_VERIFY(f);
+	GRAPH_ID_CHECK(id);
+
+	STAILQ_FOREACH(graph, &graph_list, next) {
+		if (all == true) {
+			graph_dump(f, graph);
+		} else if (graph->id == id) {
+			graph_dump(f, graph);
+			return;
+		}
+	}
+fail:
+	return;
+}
+
+void
+rte_graph_dump(FILE *f, rte_graph_t id)
+{
+	graph_scan_dump(f, id, false);
+}
+
+void
+rte_graph_list_dump(FILE *f)
+{
+	graph_scan_dump(f, 0, true);
+}
 
 rte_graph_t
 rte_graph_max_count(void)
lib/librte_graph/graph_debug.c +59 -0

68 lines not on this page · raw message

lib/librte_graph/graph_private.h +13 -0
index 7fce52e00..f9a85c892 100644
@@ -319,6 +319,19 @@ struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
 struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
 					const char *node_name);
 
+/* Debug functions */
+/**
+ * @internal
+ *
+ * Dump internal graph object data.
+ *
+ * @param f
+ *   FILE pointer to dump the data.
+ * @param g
+ *   Pointer to the internal graph object.
+ */
+void graph_dump(FILE *f, struct graph *g);
+
 /**
  * @internal
  *
lib/librte_graph/rte_graph_version.map +2 -0
index 2797be044..851f4772e 100644
@@ -6,6 +6,7 @@ EXPERIMENTAL {
 
 	rte_graph_create;
 	rte_graph_destroy;
+	rte_graph_dump;
 	rte_graph_export;
 	rte_graph_from_name;
 	rte_graph_id_to_name;
@@ -14,6 +15,7 @@ EXPERIMENTAL {
 	rte_graph_max_count;
 	rte_graph_node_get;
 	rte_graph_node_get_by_name;
+	rte_graph_obj_dump;
 
 	rte_node_clone;
 	rte_node_dump;
Signature · 1 line
2.25.1
#47 lore raw

PATCH v2 11/28 graph: implement stats support

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for graph stats collection API. This API will create a cluster for a specified node pattern and aggregate the node runtime stats.

---
 lib/librte_graph/Makefile              |   1 +
 lib/librte_graph/graph_stats.c         | 406 +++++++++++++++++++++++++
 lib/librte_graph/meson.build           |   2 +-
 lib/librte_graph/rte_graph_version.map |   5 +
 4 files changed, 413 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_graph/graph_stats.c
lib/librte_graph/Makefile +1 -0
index 7bfd7d51f..967c8d9bc 100644
@@ -18,6 +18,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_stats.c
 SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
 
 # install header files
lib/librte_graph/graph_stats.c new file +406 -0

406 lines not on this page · raw message

lib/librte_graph/meson.build +1 -1
index fb203a5e2..929a17f84 100644
@@ -4,7 +4,7 @@
 
 name = 'graph'
 
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_stats.c', 'graph_populate.c')
 headers = files('rte_graph.h', 'rte_graph_worker.h')
 allow_experimental_apis = true
 
lib/librte_graph/rte_graph_version.map +5 -0
index 851f4772e..adf55d406 100644
@@ -17,6 +17,11 @@ EXPERIMENTAL {
 	rte_graph_node_get_by_name;
 	rte_graph_obj_dump;
 
+	rte_graph_cluster_stats_create;
+	rte_graph_cluster_stats_destroy;
+	rte_graph_cluster_stats_get;
+	rte_graph_cluster_stats_reset;
+
 	rte_node_clone;
 	rte_node_dump;
 	rte_node_edge_count;
Signature · 1 line
2.25.1
#48 lore raw

PATCH v2 12/28 graph: implement fastpath API routines

In reply to [email protected]

From: Jerin Jacob <[email protected]>

Adding implementation for rte_graph_walk() API. This will perform a walk on the circular buffer and call the process function of each node and collect the stats if stats collection is enabled.

---
 doc/api/doxy-api-index.md              |   1 +
 lib/librte_graph/graph.c               |  16 +
 lib/librte_graph/rte_graph_version.map |  10 +
 lib/librte_graph/rte_graph_worker.h    | 434 +++++++++++++++++++++++++
 4 files changed, 461 insertions(+)
doc/api/doxy-api-index.md +1 -0
index 5cc50f750..fd2ff64d7 100644
@@ -160,6 +160,7 @@ The public API headers are grouped by topics:
     [port_in_action]   (@ref rte_port_in_action.h)
     [table_action]     (@ref rte_table_action.h)
   * [graph]            (@ref rte_graph.h):
+    [graph_worker]     (@ref rte_graph_worker.h)
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_graph/graph.c +16 -0
index e96363777..d5d816c71 100644
@@ -474,6 +474,22 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
 	node->realloc_count++;
 }
 
+void __rte_noinline
+__rte_node_stream_alloc_size(struct rte_graph *graph, struct rte_node *node,
+			     uint16_t req_size)
+{
+	uint16_t size = node->size;
+
+	RTE_VERIFY(size != UINT16_MAX);
+	/* Allocate double amount of size to avoid immediate realloc */
+	size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, req_size * 2));
+	node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+					RTE_CACHE_LINE_SIZE, graph->socket);
+	RTE_VERIFY(node->objs);
+	node->size = size;
+	node->realloc_count++;
+}
+
 static int
 graph_to_dot(FILE *f, struct graph *graph)
 {
lib/librte_graph/rte_graph_version.map +10 -0
index adf55d406..13b838752 100644
@@ -3,6 +3,7 @@ EXPERIMENTAL {
 
 	__rte_node_register;
 	__rte_node_stream_alloc;
+	__rte_node_stream_alloc_size;
 
 	rte_graph_create;
 	rte_graph_destroy;
@@ -16,6 +17,7 @@ EXPERIMENTAL {
 	rte_graph_node_get;
 	rte_graph_node_get_by_name;
 	rte_graph_obj_dump;
+	rte_graph_walk;
 
 	rte_graph_cluster_stats_create;
 	rte_graph_cluster_stats_destroy;
@@ -28,10 +30,18 @@ EXPERIMENTAL {
 	rte_node_edge_get;
 	rte_node_edge_shrink;
 	rte_node_edge_update;
+	rte_node_enqueue;
+	rte_node_enqueue_x1;
+	rte_node_enqueue_x2;
+	rte_node_enqueue_x4;
+	rte_node_enqueue_next;
 	rte_node_from_name;
 	rte_node_id_to_name;
 	rte_node_list_dump;
 	rte_node_max_count;
+	rte_node_next_stream_get;
+	rte_node_next_stream_put;
+	rte_node_next_stream_move;
 
 	local: *;
 };
lib/librte_graph/rte_graph_worker.h +434 -0

440 lines not on this page · raw message

Signature · 1 line
2.25.1
#49 lore raw

PATCH v2 13/28 graph: add unit test case

In reply to [email protected]

From: Kiran Kumar K <[email protected]>

Adding the unit test to test the functionality of node and graph APIs. Testing includes registering a node, cloning a node, creating a graph, perform graph walk, collecting stats and all node and graph debug APIs.

example command to test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

---
 app/test/Makefile     |   6 +
 app/test/meson.build  |  11 +-
 app/test/test_graph.c | 819 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 834 insertions(+), 2 deletions(-)
 create mode 100644 app/test/test_graph.c
app/test/Makefile +6 -0
index 1f080d162..ce2e08e12 100644
@@ -221,6 +221,10 @@ SRCS-y += test_event_timer_adapter.c
 SRCS-y += test_event_crypto_adapter.c
 endif
 
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
+SRCS-y += test_graph.c
+endif
+
 ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
 SRCS-y += test_rawdev.c
 endif
@@ -240,6 +244,8 @@ endif
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 CFLAGS += -O3
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+CFLAGS += -fno-strict-aliasing
 CFLAGS += $(WERROR_FLAGS)
 
 LDLIBS += -lm
app/test/meson.build +9 -2
index 351d29cb6..0de16290e 100644
@@ -51,6 +51,7 @@ test_sources = files('commands.c',
 	'test_fib6_perf.c',
 	'test_func_reentrancy.c',
 	'test_flow_classify.c',
+	'test_graph.c',
 	'test_hash.c',
 	'test_hash_functions.c',
 	'test_hash_multiwriter.c',
@@ -151,7 +152,9 @@ test_deps = ['acl',
 	'rib',
 	'ring',
 	'stack',
-	'timer'
+	'timer',
+	'graph',
+	'node'
 ]
 
 # Each test is marked with flag true/false
@@ -362,6 +365,8 @@ endif
 
 # specify -D_GNU_SOURCE unconditionally
 cflags += '-D_GNU_SOURCE'
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+cflags += '-fno-strict-aliasing'
 
 test_dep_objs = []
 if dpdk_conf.has('RTE_LIBRTE_COMPRESSDEV')
@@ -385,13 +390,15 @@ endforeach
 test_dep_objs += cc.find_library('execinfo', required: false)
 
 link_libs = []
+link_nodes = []
 if get_option('default_library') == 'static'
 	link_libs = dpdk_drivers
+	link_nodes = dpdk_graph_nodes
 endif
 
 dpdk_test = executable('dpdk-test',
 	test_sources,
-	link_whole: link_libs,
+	link_whole: link_libs + link_nodes,
 	dependencies: test_dep_objs,
 	c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
 	install_rpath: driver_install_path,
app/test/test_graph.c new file +819 -0

819 lines not on this page · raw message

Signature · 1 line
2.25.1
#50 lore raw

PATCH v2 14/28 graph: add performance testcase

In reply to [email protected]

From: Pavan Nikhilesh <[email protected]>

Add unit test framework to create and test performance of various graph models.

example command to test:

echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30

---
 app/test/Makefile          |    1 +
 app/test/meson.build       |    1 +
 app/test/test_graph_perf.c | 1057 ++++++++++++++++++++++++++++++++++++
 3 files changed, 1059 insertions(+)
 create mode 100644 app/test/test_graph_perf.c
app/test/Makefile +1 -0
index ce2e08e12..77276f300 100644
@@ -223,6 +223,7 @@ endif
 
 ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
 SRCS-y += test_graph.c
+SRCS-y += test_graph_perf.c
 endif
 
 ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
app/test/meson.build +1 -0
index 0de16290e..728d20c1f 100644
@@ -52,6 +52,7 @@ test_sources = files('commands.c',
 	'test_func_reentrancy.c',
 	'test_flow_classify.c',
 	'test_graph.c',
+	'test_graph_perf.c',
 	'test_hash.c',
 	'test_hash_functions.c',
 	'test_hash_multiwriter.c',
app/test/test_graph_perf.c new file +1057 -0

1,057 lines not on this page · raw message

Signature · 1 line
2.25.1
#51 lore raw

PATCH v2 15/28 node: add log infra and null node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add log infra for node specific logging.
Also, add null rte_node that just ignores all the objects directed to it.

---
 MAINTAINERS                          |  5 +++++
 config/common_base                   |  5 +++++
 doc/api/doxy-api.conf.in             |  1 +
 lib/Makefile                         |  3 +++
 lib/librte_node/Makefile             | 22 ++++++++++++++++++++++
 lib/librte_node/log.c                | 14 ++++++++++++++
 lib/librte_node/meson.build          |  8 ++++++++
 lib/librte_node/node_private.h       | 22 ++++++++++++++++++++++
 lib/librte_node/null.c               | 23 +++++++++++++++++++++++
 lib/librte_node/rte_node_version.map |  6 ++++++
 lib/meson.build                      |  5 ++++-
 meson.build                          |  1 +
 mk/rte.app.mk                        |  1 +
 13 files changed, 115 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/Makefile
 create mode 100644 lib/librte_node/log.c
 create mode 100644 lib/librte_node/meson.build
 create mode 100644 lib/librte_node/node_private.h
 create mode 100644 lib/librte_node/null.c
 create mode 100644 lib/librte_node/rte_node_version.map
MAINTAINERS +5 -0
index bc7085983..c1acaedab 100644
@@ -1474,6 +1474,11 @@ M: Jerin Jacob <[email protected]>
 M: Kiran Kumar K <[email protected]>
 F: lib/librte_graph/
 
+Nodes - EXPERIMENTAL
+M: Nithin Dabilpuram <[email protected]>
+M: Pavan Nikhilesh <[email protected]>
+F: lib/librte_node/
+
 
 Test Applications
 -----------------
config/common_base +5 -0
index 32f982136..1ed5187dc 100644
@@ -1081,6 +1081,11 @@ CONFIG_RTE_LIBRTE_GRAPH=y
 CONFIG_RTE_GRAPH_BURST_SIZE=256
 CONFIG_RTE_LIBRTE_GRAPH_STATS=y
 
+#
+# Compile librte_node
+#
+CONFIG_RTE_LIBRTE_NODE=y
+
 #
 # Compile the test application
 #
doc/api/doxy-api.conf.in +1 -0
index 759a7213e..1d4f1a37d 100644
@@ -49,6 +49,7 @@ INPUT                   = @TOPDIR@/doc/api/doxy-api-index.md \
                           @TOPDIR@/lib/librte_mempool \
                           @TOPDIR@/lib/librte_meter \
                           @TOPDIR@/lib/librte_metrics \
+                          @TOPDIR@/lib/librte_node \
                           @TOPDIR@/lib/librte_net \
                           @TOPDIR@/lib/librte_pci \
                           @TOPDIR@/lib/librte_pdump \
lib/Makefile +3 -0
index 1f572b659..50d61a338 100644
@@ -122,6 +122,9 @@ DEPDIRS-librte_rcu := librte_eal
 DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
 DEPDIRS-librte_graph := librte_eal
 
+DIRS-$(CONFIG_RTE_LIBRTE_NODE) += librte_node
+DEPDIRS-librte_node := librte_graph librte_lpm librte_ethdev librte_mbuf
+
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
 endif
lib/librte_node/Makefile new file +22 -0
new file mode 100644
index 000000000..dbc8e1d44
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_node.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+CFLAGS += -fno-strict-aliasing
+LDLIBS += -lrte_eal -lrte_graph
+
+EXPORT_MAP := rte_node_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/log.c new file +14 -0
new file mode 100644
index 000000000..f035f91e8
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "node_private.h"
+
+int rte_node_logtype;
+
+RTE_INIT(rte_node_init_log)
+{
+	rte_node_logtype = rte_log_register("lib.node");
+	if (rte_node_logtype >= 0)
+		rte_log_set_level(rte_node_logtype, RTE_LOG_INFO);
+}
lib/librte_node/meson.build new file +8 -0
new file mode 100644
index 000000000..a97813ad4
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+sources = files('null.c', 'log.c')
+allow_experimental_apis = true
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+cflags += '-fno-strict-aliasing'
+deps += ['graph']
lib/librte_node/node_private.h new file +22 -0
new file mode 100644
index 000000000..f30902a94
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __NODE_PRIVATE_H__
+#define __NODE_PRIVATE_H__
+
+#include <rte_common.h>
+#include <rte_log.h>
+
+extern int rte_node_logtype;
+#define NODE_LOG(level, node_name, ...)                                        \
+	rte_log(RTE_LOG_##level, rte_node_logtype,                             \
+		RTE_FMT("NODE %s: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n",  \
+			node_name, __func__, __LINE__,                         \
+			RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define node_err(node_name, ...) NODE_LOG(ERR, node_name, __VA_ARGS__)
+#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
+#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+
+#endif /* __NODE_PRIVATE_H__ */
lib/librte_node/null.c new file +23 -0
new file mode 100644
index 000000000..c7cd8b6df
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_graph.h>
+
+static uint16_t
+null(struct rte_graph *graph, struct rte_node *node, void **objs,
+	uint16_t nb_objs)
+{
+	RTE_SET_USED(node);
+	RTE_SET_USED(objs);
+	RTE_SET_USED(graph);
+
+	return nb_objs;
+}
+
+static struct rte_node_register null_node = {
+	.name = "null",
+	.process = null,
+};
+
+RTE_NODE_REGISTER(null_node);
lib/librte_node/rte_node_version.map new file +6 -0
new file mode 100644
index 000000000..f87163bb9
@@ -0,0 +1,6 @@
+EXPERIMENTAL {
+	global:
+
+	rte_node_logtype;
+	local: *;
+};
lib/meson.build +4 -1
index c43d86bb9..147129b0b 100644
@@ -30,7 +30,7 @@ libraries = [
 	# add pkt framework libs which use other libs from above
 	'port', 'table', 'pipeline',
 	# flow_classify lib depends on pkt framework table lib
-	'flow_classify', 'bpf', 'graph', 'telemetry']
+	'flow_classify', 'bpf', 'graph', 'node', 'telemetry']
 
 if is_windows
 	libraries = ['kvargs','eal'] # only supported libraries for windows
@@ -186,6 +186,9 @@ foreach l:libraries
 
 			dpdk_libraries = [shared_lib] + dpdk_libraries
 			dpdk_static_libraries = [static_lib] + dpdk_static_libraries
+			if libname == 'rte_node'
+				dpdk_graph_nodes = [static_lib]
+			endif
 		endif # sources.length() > 0
 
 		set_variable('shared_rte_' + name, shared_dep)
meson.build +1 -0
index b7ae9c8d9..811c96421 100644
@@ -16,6 +16,7 @@ cc = meson.get_compiler('c')
 dpdk_conf = configuration_data()
 dpdk_libraries = []
 dpdk_static_libraries = []
+dpdk_graph_nodes = []
 dpdk_driver_classes = []
 dpdk_drivers = []
 dpdk_extra_ldflags = []
mk/rte.app.mk +1 -0
index b1195f09a..68d7806a4 100644
@@ -99,6 +99,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RCU)            += -lrte_rcu
 _LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH)          += -lrte_graph
+_LDLIBS-$(CONFIG_RTE_LIBRTE_NODE)           += -lrte_node
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KNI)            += -lrte_kni
Signature · 1 line
2.25.1
#52 lore raw

PATCH v2 16/28 node: add ethdev Rx node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add source rte_node ethdev_rx process function and register it. This node is a source node that will be called periodically and when called, performs rte_eth_rx_burst() on a specific (port, queue) pair and enqueue them as stream of objects to next node.

---
 lib/librte_node/Makefile         |   3 +-
 lib/librte_node/ethdev_rx.c      | 221 +++++++++++++++++++++++++++++++
 lib/librte_node/ethdev_rx_priv.h |  81 +++++++++++
 lib/librte_node/meson.build      |   4 +-
 4 files changed, 306 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_rx.c
 create mode 100644 lib/librte_node/ethdev_rx_priv.h
lib/librte_node/Makefile +2 -1
index dbc8e1d44..314149385 100644
@@ -11,12 +11,13 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph
+LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
 
 EXPORT_MAP := rte_node_version.map
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_rx.c new file +221 -0

221 lines not on this page · raw message

lib/librte_node/ethdev_rx_priv.h new file +81 -0

81 lines not on this page · raw message

lib/librte_node/meson.build +2 -2
index a97813ad4..94caa6c23 100644
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 cflags += '-fno-strict-aliasing'
-deps += ['graph']
+deps += ['graph', 'ethdev']
Signature · 1 line
2.25.1
#53 lore raw

PATCH v2 17/28 node: add ethdev Tx node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add rte_node ethdev_tx process function and register it to graph infra. This node has a specific (port, tx-queue) as context and it enqueue's all the packets received to that specific queue pair. When rte_eth_tx_burst() i.e enqueue to queue pair fails, packets are forwarded to pkt_drop node to be free'd.

---
 lib/librte_node/Makefile         |  3 +-
 lib/librte_node/ethdev_tx.c      | 86 ++++++++++++++++++++++++++++++++
 lib/librte_node/ethdev_tx_priv.h | 62 +++++++++++++++++++++++
 lib/librte_node/meson.build      |  4 +-
 4 files changed, 152 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_tx.c
 create mode 100644 lib/librte_node/ethdev_tx_priv.h
lib/librte_node/Makefile +2 -1
index 314149385..7428f6c43 100644
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
 
 EXPORT_MAP := rte_node_version.map
 
@@ -19,5 +19,6 @@ EXPORT_MAP := rte_node_version.map
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_tx.c new file +86 -0

86 lines not on this page · raw message

lib/librte_node/ethdev_tx_priv.h new file +62 -0

62 lines not on this page · raw message

lib/librte_node/meson.build +2 -2
index 94caa6c23..505c76abd 100644
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 cflags += '-fno-strict-aliasing'
-deps += ['graph', 'ethdev']
+deps += ['graph', 'mbuf', 'ethdev']
Signature · 1 line
2.25.1
#54 lore raw

PATCH v2 18/28 node: add ethdev Rx and Tx node ctrl API

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add ctrl api to setup ethdev_rx and ethdev_tx node. This ctrl api clones 'N' number of ethdev_rx and ethdev_tx nodes with specific (port, queue) pairs updated in their context. All the ethdev ports and queues are setup before this api is called.

---
 doc/api/doxy-api-index.md            |   2 +
 lib/librte_node/Makefile             |   6 +-
 lib/librte_node/ethdev_ctrl.c        | 100 +++++++++++++++++++++++++++
 lib/librte_node/meson.build          |   5 +-
 lib/librte_node/node_private.h       |  74 ++++++++++++++++++++
 lib/librte_node/rte_node_eth_api.h   |  70 +++++++++++++++++++
 lib/librte_node/rte_node_version.map |   1 +
 7 files changed, 255 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ethdev_ctrl.c
 create mode 100644 lib/librte_node/rte_node_eth_api.h
doc/api/doxy-api-index.md +2 -0
index fd2ff64d7..1784674df 100644
@@ -161,6 +161,8 @@ The public API headers are grouped by topics:
     [table_action]     (@ref rte_table_action.h)
   * [graph]            (@ref rte_graph.h):
     [graph_worker]     (@ref rte_graph_worker.h)
+  * graph_nodes:
+    [eth_node]         (@ref rte_node_eth_api.h),
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_node/Makefile +5 -1
index 7428f6c43..ea5fa77f7 100644
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
 
 EXPORT_MAP := rte_node_version.map
 
@@ -20,5 +20,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ethdev_ctrl.c new file +100 -0

100 lines not on this page · raw message

lib/librte_node/meson.build +3 -2
index 505c76abd..af2eb2d91 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
+headers = files('rte_node_eth_api.h')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 cflags += '-fno-strict-aliasing'
-deps += ['graph', 'mbuf', 'ethdev']
+deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
lib/librte_node/node_private.h +74 -0

85 lines not on this page · raw message

lib/librte_node/rte_node_eth_api.h new file +70 -0

70 lines not on this page · raw message

lib/librte_node/rte_node_version.map +1 -0
index f87163bb9..c6c71bd02 100644
@@ -1,6 +1,7 @@
 EXPERIMENTAL {
 	global:
 
+	rte_node_eth_config;
 	rte_node_logtype;
 	local: *;
 };
Signature · 1 line
2.25.1
#55 lore raw

PATCH v2 19/28 node: ipv4 lookup for arm64

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add arm64 specific IPv4 lookup process function for ip4_lookup node. This node performs LPM lookup on every packet received and forwards it to a next node that is identified by lookup result.

---
 doc/api/doxy-api-index.md          |   1 +
 lib/librte_node/Makefile           |   4 +-
 lib/librte_node/ip4_lookup.c       | 301 +++++++++++++++++++++++++++++
 lib/librte_node/meson.build        |   5 +-
 lib/librte_node/rte_node_ip4_api.h |  43 +++++
 5 files changed, 351 insertions(+), 3 deletions(-)
 create mode 100644 lib/librte_node/ip4_lookup.c
 create mode 100644 lib/librte_node/rte_node_ip4_api.h
doc/api/doxy-api-index.md +1 -0
index 1784674df..3908fddde 100644
@@ -163,6 +163,7 @@ The public API headers are grouped by topics:
     [graph_worker]     (@ref rte_graph_worker.h)
   * graph_nodes:
     [eth_node]         (@ref rte_node_eth_api.h),
+    [ip4_node]         (@ref rte_node_ip4_api.h)
 
 - **basic**:
   [approx fraction]    (@ref rte_approx.h),
lib/librte_node/Makefile +3 -1
index ea5fa77f7..ad3f2e349 100644
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS)
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_lpm -lrte_ethdev -lrte_mempool
 
 EXPORT_MAP := rte_node_version.map
 
@@ -21,8 +21,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
 
 # install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_node/ip4_lookup.c new file +301 -0

301 lines not on this page · raw message

lib/librte_node/meson.build +3 -2
index af2eb2d91..702d21a24 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(C) 2020 Marvell International Ltd.
 
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
-headers = files('rte_node_eth_api.h')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
+		'ethdev_ctrl.c')
+headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
 cflags += '-fno-strict-aliasing'
lib/librte_node/rte_node_ip4_api.h new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.25.1
#56 lore raw

PATCH v2 20/28 node: ipv4 lookup for x86

In reply to [email protected]

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

---
 lib/librte_node/ip4_lookup.c | 238 +++++++++++++++++++++++++++++++++++
 1 file changed, 238 insertions(+)
lib/librte_node/ip4_lookup.c +238 -0
index dddabc6c2..a9e204148 100644
@@ -259,6 +259,244 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
 	return nb_objs;
 }
 
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+			void **objs, uint16_t nb_objs)
+{
+	struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+	rte_edge_t next0, next1, next2, next3, next_index;
+	struct rte_ipv4_hdr *ipv4_hdr;
+	uint32_t ip0, ip1, ip2, ip3;
+	void **to_next, **from;
+	uint16_t last_spec = 0;
+	uint16_t n_left_from;
+	struct rte_lpm *lpm;
+	uint16_t held = 0;
+	uint32_t drop_nh;
+	rte_xmm_t dst;
+	__m128i dip; /* SSE register */
+	int rc, i;
+
+	/* Speculative next */
+	next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+	/* Drop node */
+	drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+	/* Get socket specific LPM from ctx */
+	lpm = *((struct rte_lpm **)node->ctx);
+
+	pkts = (struct rte_mbuf **)objs;
+	from = objs;
+	n_left_from = nb_objs;
+
+	if (n_left_from >= 4) {
+		for (i = 0; i < 4; i++)
+			rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[i], void *,
+						sizeof(struct rte_ether_hdr)));
+	}
+
+	/* Get stream for the speculated next node */
+	to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+	while (n_left_from >= 4) {
+		/* Prefetch next-next mbufs */
+		if (likely(n_left_from > 11)) {
+			rte_prefetch0(pkts[8]);
+			rte_prefetch0(pkts[9]);
+			rte_prefetch0(pkts[10]);
+			rte_prefetch0(pkts[11]);
+		}
+
+		/* Prefetch next mbuf data */
+		if (likely(n_left_from > 7)) {
+			rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[4], void *,
+						sizeof(struct rte_ether_hdr)));
+			rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[5], void *,
+						sizeof(struct rte_ether_hdr)));
+			rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[6], void *,
+						sizeof(struct rte_ether_hdr)));
+			rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[7], void *,
+						sizeof(struct rte_ether_hdr)));
+		}
+
+		mbuf0 = pkts[0];
+		mbuf1 = pkts[1];
+		mbuf2 = pkts[2];
+		mbuf3 = pkts[3];
+
+		pkts += 4;
+		n_left_from -= 4;
+
+		/* Extract DIP of mbuf0 */
+		ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+						sizeof(struct rte_ether_hdr));
+		ip0 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf1 */
+		ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf1, struct rte_ipv4_hdr *,
+						sizeof(struct rte_ether_hdr));
+		ip1 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf2 */
+		ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf2, struct rte_ipv4_hdr *,
+						sizeof(struct rte_ether_hdr));
+		ip2 = ipv4_hdr->dst_addr;
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
+
+		/* Extract DIP of mbuf3 */
+		ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf3, struct rte_ipv4_hdr *,
+						sizeof(struct rte_ether_hdr));
+		ip3 = ipv4_hdr->dst_addr;
+
+		/* Prepare for lookup x4 */
+		dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+		/* Byte swap 4 IPV4 addresses. */
+		const __m128i bswap_mask = _mm_set_epi8(
+			12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+		dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
+
+		/* Perform LPM lookup to get NH and next node */
+		rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+		/* Extract next node id and NH */
+		rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
+		next0 = (dst.u32[0] >> 16);
+
+		rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
+		next1 = (dst.u32[1] >> 16);
+
+		rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
+		next2 = (dst.u32[2] >> 16);
+
+		rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
+		next3 = (dst.u32[3] >> 16);
+
+		/* Enqueue four to next node */
+		rte_edge_t fix_spec =
+			(next_index ^ next0) | (next_index ^ next1) |
+			(next_index ^ next2) | (next_index ^ next3);
+
+		if (unlikely(fix_spec)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			/* Next0 */
+			if (next_index == next0) {
+				to_next[0] = from[0];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next0,
+						    from[0]);
+			}
+
+			/* Next1 */
+			if (next_index == next1) {
+				to_next[0] = from[1];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next1,
+						    from[1]);
+			}
+
+			/* Next2 */
+			if (next_index == next2) {
+				to_next[0] = from[2];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next2,
+						    from[2]);
+			}
+
+			/* Next3 */
+			if (next_index == next3) {
+				to_next[0] = from[3];
+				to_next++;
+				held++;
+			} else {
+				rte_node_enqueue_x1(graph, node, next3,
+						    from[3]);
+			}
+
+			from += 4;
+
+		} else {
+			last_spec += 4;
+		}
+	}
+
+	while (n_left_from > 0) {
+		uint32_t next_hop;
+
+		mbuf0 = pkts[0];
+
+		pkts += 1;
+		n_left_from -= 1;
+
+		/* Extract DIP of mbuf0 */
+		ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+						sizeof(struct rte_ether_hdr));
+		/* Extract cksum, ttl as ipv4 hdr is in cache */
+		rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+		rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+		rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+				    &next_hop);
+		next_hop = (rc == 0) ? next_hop : drop_nh;
+
+		rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
+		next0 = (next_hop >> 16);
+
+		if (unlikely(next_index ^ next0)) {
+			/* Copy things successfully speculated till now */
+			rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+			from += last_spec;
+			to_next += last_spec;
+			held += last_spec;
+			last_spec = 0;
+
+			rte_node_enqueue_x1(graph, node, next0, from[0]);
+			from += 1;
+		} else {
+			last_spec += 1;
+		}
+	}
+
+	/* !!! Home run !!! */
+	if (likely(last_spec == nb_objs)) {
+		rte_node_next_stream_move(graph, node, next_index);
+		return nb_objs;
+	}
+
+	held += last_spec;
+	/* Copy things successfully speculated till now */
+	rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+	rte_node_next_stream_put(graph, node, next_index, held);
+
+	return nb_objs;
+}
+
 #else
 
 static uint16_t
Signature · 1 line
2.25.1
#57 lore raw

Re: [dpdk-dev] [PATCH v2 20/28] node: ipv4 lookup for x86

Quoted 9 lines · On Thu, Mar 26, 2020 at 10:30 PM <[email protected]> wrote:

From: Pavan Nikhilesh <[email protected]>

Add IPv4 lookup process function for ip4_lookup rte_node. This node performs LPM lookup using x86_64 vector supported RTE_LPM API on every packet received and forwards it to a next node that is identified by lookup result.

Quoted 7 lines

}

+#elif defined(RTE_ARCH_X86)

# Could you move the below code to ipv4_looup_sse.h and arm64 code to ipv4_looup_neon.h
# In order to make this node work for PPC and ARMv7, please add a scalar version of ip4_lookup_node_process() too

Quoted 19 lines
lib/librte_node/ip4_lookup.c +19 -0
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+                       void **objs, uint16_t nb_objs)
+{
+       struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+       rte_edge_t next0, next1, next2, next3, next_index;
+       struct rte_ipv4_hdr *ipv4_hdr;
+       uint32_t ip0, ip1, ip2, ip3;
+       void **to_next, **from;
+       uint16_t last_spec = 0;
+       uint16_t n_left_from;
+       struct rte_lpm *lpm;
+       uint16_t held = 0;
+       uint32_t drop_nh;
+       rte_xmm_t dst;
+       __m128i dip; /* SSE register */
+       int rc, i;
#58 lore raw

PATCH v2 21/28 node: add ipv4 rewrite node

In reply to [email protected]

From: Kiran Kumar K <[email protected]>

Add ip4 rewrite process function for ip4_rewrite rte_node. On every packet received by this node, header is overwritten with new data before forwarding it to next node. Header data to overwrite with is identified by next hop id passed in mbuf priv data by previous node.

---
 lib/librte_node/Makefile           |   1 +
 lib/librte_node/ip4_rewrite.c      | 270 +++++++++++++++++++++++++++++
 lib/librte_node/ip4_rewrite_priv.h |  55 ++++++
 lib/librte_node/meson.build        |   2 +-
 4 files changed, 327 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/ip4_rewrite.c
 create mode 100644 lib/librte_node/ip4_rewrite_priv.h
lib/librte_node/Makefile +1 -0
index ad3f2e349..ebf473c66 100644
@@ -22,6 +22,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
lib/librte_node/ip4_rewrite.c new file +270 -0

270 lines not on this page · raw message

lib/librte_node/ip4_rewrite_priv.h new file +55 -0

55 lines not on this page · raw message

lib/librte_node/meson.build +1 -1
index 702d21a24..ed78791dd 100644
@@ -2,7 +2,7 @@
 # Copyright(C) 2020 Marvell International Ltd.
 
 sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
-		'ethdev_ctrl.c')
+		'ip4_rewrite.c', 'ethdev_ctrl.c')
 headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
Signature · 1 line
2.25.1
#59 lore raw

PATCH v2 22/28 node: add ipv4 rewrite and lookup ctrl API

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add ip4_rewrite and ip4_lookup ctrl API. ip4_lookup ctrl API is used to add route entries for LPM lookup with result data containing next hop id and next proto. ip4_rewrite ctrl API is used to add rewrite data for every next hop.

---
 lib/librte_node/ethdev_ctrl.c        | 18 ++++++-
 lib/librte_node/ip4_lookup.c         | 80 ++++++++++++++++++++++++++++
 lib/librte_node/ip4_rewrite.c        | 56 +++++++++++++++++++
 lib/librte_node/ip4_rewrite_priv.h   | 22 ++++++++
 lib/librte_node/rte_node_ip4_api.h   | 44 +++++++++++++++
 lib/librte_node/rte_node_version.map |  2 +
 6 files changed, 221 insertions(+), 1 deletion(-)
lib/librte_node/ethdev_ctrl.c +17 -1
index b2ac5e2c4..845d92987 100644
@@ -11,6 +11,7 @@
 
 #include "ethdev_rx_priv.h"
 #include "ethdev_tx_priv.h"
+#include "ip4_rewrite_priv.h"
 #include "node_private.h"
 
 static struct ethdev_ctrl {
@@ -21,14 +22,17 @@ int
 rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
 		    uint16_t nb_graphs)
 {
+	struct rte_node_register *ip4_rewrite_node;
 	struct ethdev_tx_node_main *tx_node_data;
 	uint16_t tx_q_used, rx_q_used, port_id;
 	struct rte_node_register *tx_node;
 	char name[RTE_NODE_NAMESIZE];
+	const char *next_nodes = name;
 	struct rte_mempool *mp;
+	int i, j, rc;
 	uint32_t id;
-	int i, j;
 
+	ip4_rewrite_node = ip4_rewrite_node_get();
 	tx_node_data = ethdev_tx_node_data_get();
 	tx_node = ethdev_tx_node_get();
 	for (i = 0; i < nb_confs; i++) {
@@ -93,6 +97,18 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
 
 		node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
 			 name, id);
+
+		/* Prepare the actual name of the cloned node */
+		snprintf(name, sizeof(name), "ethdev_tx-%u", port_id);
+
+		/* Add this tx port node as next to ip4_rewrite_node */
+		rte_node_edge_update(ip4_rewrite_node->id, RTE_EDGE_ID_INVALID,
+				     &next_nodes, 1);
+		/* Assuming edge id is the last one alloc'ed */
+		rc = ip4_rewrite_set_next(
+			port_id, rte_node_edge_count(ip4_rewrite_node->id) - 1);
+		if (rc < 0)
+			return rc;
 	}
 
 	ctrl.nb_graphs = nb_graphs;
lib/librte_node/ip4_lookup.c +80 -0

98 lines not on this page · raw message

lib/librte_node/ip4_rewrite.c +56 -0

66 lines not on this page · raw message

lib/librte_node/ip4_rewrite_priv.h +22 -0
index 420996a03..80f0abdc9 100644
@@ -48,6 +48,28 @@ struct ip4_rewrite_node_main {
 	/**< Next index of each configured port. */
 };
 
+/**
+ * @internal
+ *
+ * Get the ipv4 rewrite node.
+ *
+ * @retrun
+ *   Pointer to the ipv4 rewrite node.
+ */
+struct rte_node_register *ip4_rewrite_node_get(void);
+
+/**
+ * @internal
+ *
+ * Set the Edge index of a given port_id.
+ *
+ * @param port_id
+ *   Ethernet port identifier.
+ * @param next_index
+ *   Edge index of the Given Tx node.
+ */
+int ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index);
+
 #ifdef __cplusplus
 }
 #endif
lib/librte_node/rte_node_ip4_api.h +44 -0

50 lines not on this page · raw message

lib/librte_node/rte_node_version.map +2 -0
index c6c71bd02..a799b0d38 100644
@@ -2,6 +2,8 @@ EXPERIMENTAL {
 	global:
 
 	rte_node_eth_config;
+	rte_node_ip4_route_add;
+	rte_node_ip4_rewrite_add;
 	rte_node_logtype;
 	local: *;
 };
Signature · 1 line
2.25.1
#60 lore raw

PATCH v2 23/28 node: add pkt drop node

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add pkt drop node process function for pkt_drop rte_node. This node simply free's every object received as an rte_mbuf to its rte_pktmbuf pool.

---
 lib/librte_node/Makefile    |  1 +
 lib/librte_node/meson.build |  2 +-
 lib/librte_node/pkt_drop.c  | 26 ++++++++++++++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_node/pkt_drop.c
lib/librte_node/Makefile +1 -0
index ebf473c66..322b651c8 100644
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
 SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += pkt_drop.c
 
 # install header files
 SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
lib/librte_node/meson.build +1 -1
index ed78791dd..8aa93654b 100644
@@ -2,7 +2,7 @@
 # Copyright(C) 2020 Marvell International Ltd.
 
 sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
-		'ip4_rewrite.c', 'ethdev_ctrl.c')
+		'ip4_rewrite.c', 'pkt_drop.c', 'ethdev_ctrl.c')
 headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
 allow_experimental_apis = true
 # Strict-aliasing rules are violated by uint8_t[] to context size casts.
lib/librte_node/pkt_drop.c new file +26 -0
new file mode 100644
index 000000000..c35001323
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_graph.h>
+#include <rte_mbuf.h>
+
+static uint16_t
+pkt_drop_process(struct rte_graph *graph, struct rte_node *node, void **objs,
+		 uint16_t nb_objs)
+{
+	RTE_SET_USED(node);
+	RTE_SET_USED(graph);
+
+	rte_pktmbuf_free_bulk((struct rte_mbuf **)objs, nb_objs);
+
+	return nb_objs;
+}
+
+static struct rte_node_register pkt_drop_node = {
+	.process = pkt_drop_process,
+	.name = "pkt_drop",
+};
+
+RTE_NODE_REGISTER(pkt_drop_node);
Signature · 1 line
2.25.1
#61 lore raw

PATCH v2 24/28 l3fwd-graph: add graph based l3fwd skeleton

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add graph based l3fwd application skeleton with cmdline parsing support inline with normal l3fwd.

---
 MAINTAINERS                      |   3 +
 examples/Makefile                |   3 +
 examples/l3fwd-graph/Makefile    |  58 ++++
 examples/l3fwd-graph/main.c      | 525 +++++++++++++++++++++++++++++++
 examples/l3fwd-graph/meson.build |  13 +
 examples/meson.build             |   6 +-
 6 files changed, 606 insertions(+), 2 deletions(-)
 create mode 100644 examples/l3fwd-graph/Makefile
 create mode 100644 examples/l3fwd-graph/main.c
 create mode 100644 examples/l3fwd-graph/meson.build
MAINTAINERS +3 -0
index c1acaedab..1d2cf6caa 100644
@@ -1581,6 +1581,9 @@ F: doc/guides/sample_app_ug/l2_forward_event.rst
 F: examples/l3fwd/
 F: doc/guides/sample_app_ug/l3_forward.rst
 
+M: Nithin Dabilpuram <[email protected]>
+F: examples/l3fwd-graph/
+
 F: examples/link_status_interrupt/
 F: doc/guides/sample_app_ug/link_status_intr.rst
 
examples/Makefile +3 -0
index feff79784..b7e99a2f7 100644
@@ -51,6 +51,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_ACL) += l3fwd-acl
 ifeq ($(CONFIG_RTE_LIBRTE_LPM)$(CONFIG_RTE_LIBRTE_HASH),yy)
 DIRS-$(CONFIG_RTE_LIBRTE_POWER) += l3fwd-power
 endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH),y)
+DIRS-y += l3fwd-graph
+endif
 DIRS-y += link_status_interrupt
 DIRS-y += multi_process
 DIRS-y += ntb
examples/l3fwd-graph/Makefile new file +58 -0

58 lines not on this page · raw message

examples/l3fwd-graph/main.c new file +525 -0

525 lines not on this page · raw message

examples/l3fwd-graph/meson.build new file +13 -0
new file mode 100644
index 000000000..a816bd890
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+deps += ['graph', 'eal', 'lpm', 'ethdev', 'node' ]
+sources = files(
+	'main.c'
+)
+allow_experimental_apis = true
examples/meson.build +4 -2
index 1f2b6f516..3b540012f 100644
@@ -2,8 +2,10 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 driver_libs = []
+node_libs = []
 if get_option('default_library') == 'static'
 	driver_libs = dpdk_drivers
+	node_libs = dpdk_graph_nodes
 endif
 
 execinfo = cc.find_library('execinfo', required: false)
@@ -23,7 +25,7 @@ all_examples = [
 	'l2fwd', 'l2fwd-cat', 'l2fwd-event',
 	'l2fwd-crypto', 'l2fwd-jobstats',
 	'l2fwd-keepalive', 'l3fwd',
-	'l3fwd-acl', 'l3fwd-power',
+	'l3fwd-acl', 'l3fwd-power', 'l3fwd-graph',
 	'link_status_interrupt',
 	'multi_process/client_server_mp/mp_client',
 	'multi_process/client_server_mp/mp_server',
@@ -99,7 +101,7 @@ foreach example: examples
 		endif
 		executable('dpdk-' + name, sources,
 			include_directories: includes,
-			link_whole: driver_libs,
+			link_whole: driver_libs + node_libs,
 			link_args: dpdk_extra_ldflags,
 			c_args: cflags,
 			dependencies: dep_objs)
Signature · 1 line
2.25.1
#62 lore raw

PATCH v2 25/28 l3fwd-graph: add ethdev configuration changes

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add changes to ethdev port and queue configuration based on command line parameters for l3fwd graph application.

---
 examples/l3fwd-graph/main.c | 350 +++++++++++++++++++++++++++++++++++-
 1 file changed, 349 insertions(+), 1 deletion(-)
examples/l3fwd-graph/main.c +349 -1
index e0c6f42fa..47d2f2ecb 100644
@@ -20,8 +20,10 @@
 
 #include <rte_branch_prediction.h>
 #include <rte_common.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
 #include <rte_ethdev.h>
+#include <rte_lcore.h>
 #include <rte_log.h>
 #include <rte_mempool.h>
 #include <rte_per_lcore.h>
@@ -49,6 +51,10 @@
 
 #define NB_SOCKETS 8
 
+/* Static global variables used within this file. */
+static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT;
+static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
+
 /**< Ports set in promiscuous mode off by default. */
 static int promiscuous_on;
 
@@ -60,6 +66,7 @@ static volatile bool force_quit;
 
 /* Ethernet addresses of ports */
 static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+static struct rte_ether_addr ports_eth_addr[RTE_MAX_ETHPORTS];
 xmm_t val_eth[RTE_MAX_ETHPORTS];
 
 /* Mask of enabled ports */
@@ -110,6 +117,8 @@ static struct rte_eth_conf port_conf = {
 	},
 };
 
+static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+
 static int
 check_lcore_params(void)
 {
@@ -165,6 +174,27 @@ check_port_config(void)
 	return 0;
 }
 
+static uint8_t
+get_port_n_rx_queues(const uint16_t port)
+{
+	int queue = -1;
+	uint16_t i;
+
+	for (i = 0; i < nb_lcore_params; ++i) {
+		if (lcore_params[i].port_id == port) {
+			if (lcore_params[i].queue_id == queue + 1)
+				queue = lcore_params[i].queue_id;
+			else
+				rte_exit(EXIT_FAILURE,
+					 "Queue ids of the port %d must be"
+					 " in sequence and must start with 0\n",
+					 lcore_params[i].port_id);
+		}
+	}
+
+	return (uint8_t)(++queue);
+}
+
 static int
 init_lcore_rx_queues(void)
 {
@@ -470,6 +500,120 @@ parse_args(int argc, char **argv)
 	return ret;
 }
 
+static void
+print_ethaddr(const char *name, const struct rte_ether_addr *eth_addr)
+{
+	char buf[RTE_ETHER_ADDR_FMT_SIZE];
+	rte_ether_format_addr(buf, RTE_ETHER_ADDR_FMT_SIZE, eth_addr);
+	printf("%s%s", name, buf);
+}
+
+static int
+init_mem(uint16_t portid, uint32_t nb_mbuf)
+{
+	uint32_t lcore_id;
+	int socketid;
+	char s[64];
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+
+		if (numa_on)
+			socketid = rte_lcore_to_socket_id(lcore_id);
+		else
+			socketid = 0;
+
+		if (socketid >= NB_SOCKETS) {
+			rte_exit(EXIT_FAILURE,
+				 "Socket %d of lcore %u is out of range %d\n",
+				 socketid, lcore_id, NB_SOCKETS);
+		}
+
+		if (pktmbuf_pool[portid][socketid] == NULL) {
+			snprintf(s, sizeof(s), "mbuf_pool_%d:%d", portid,
+				 socketid);
+			/* Create a pool with priv size of a cacheline */
+			pktmbuf_pool[portid][socketid] =
+				rte_pktmbuf_pool_create(
+					s, nb_mbuf, MEMPOOL_CACHE_SIZE,
+					RTE_CACHE_LINE_SIZE,
+					RTE_MBUF_DEFAULT_BUF_SIZE, socketid);
+			if (pktmbuf_pool[portid][socketid] == NULL)
+				rte_exit(EXIT_FAILURE,
+					 "Cannot init mbuf pool on socket %d\n",
+					 socketid);
+			else
+				printf("Allocated mbuf pool on socket %d\n",
+				       socketid);
+		}
+	}
+
+	return 0;
+}
+
+/* Check the link status of all ports in up to 9s, and print them finally */
+static void
+check_all_ports_link_status(uint32_t port_mask)
+{
+#define CHECK_INTERVAL 100 /* 100ms */
+#define MAX_CHECK_TIME 90  /* 9s (90 * 100ms) in total */
+	uint8_t count, all_ports_up, print_flag = 0;
+	struct rte_eth_link link;
+	uint16_t portid;
+
+	printf("\nChecking link status");
+	fflush(stdout);
+	for (count = 0; count <= MAX_CHECK_TIME; count++) {
+		if (force_quit)
+			return;
+		all_ports_up = 1;
+		RTE_ETH_FOREACH_DEV(portid)
+		{
+			if (force_quit)
+				return;
+			if ((port_mask & (1 << portid)) == 0)
+				continue;
+			memset(&link, 0, sizeof(link));
+			rte_eth_link_get_nowait(portid, &link);
+			/* Print link status if flag set */
+			if (print_flag == 1) {
+				if (link.link_status)
+					printf("Port%d Link Up. Speed %u Mbps "
+					       "-%s\n",
+					       portid, link.link_speed,
+					       (link.link_duplex ==
+						ETH_LINK_FULL_DUPLEX)
+						       ? ("full-duplex")
+						       : ("half-duplex\n"));
+				else
+					printf("Port %d Link Down\n", portid);
+				continue;
+			}
+			/* Clear all_ports_up flag if any link down */
+			if (link.link_status == ETH_LINK_DOWN) {
+				all_ports_up = 0;
+				break;
+			}
+		}
+		/* After finally printing all link status, get out */
+		if (print_flag == 1)
+			break;
+
+		if (all_ports_up == 0) {
+			printf(".");
+			fflush(stdout);
+			rte_delay_ms(CHECK_INTERVAL);
+		}
+
+		/* Set the print_flag if all ports up or timeout */
+		if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
+			print_flag = 1;
+			printf("Done\n");
+		}
+	}
+}
+
 static void
 signal_handler(int signum)
 {
@@ -483,7 +627,14 @@ signal_handler(int signum)
 int
 main(int argc, char **argv)
 {
-	uint16_t portid;
+	uint8_t nb_rx_queue, queue, socketid;
+	struct rte_eth_dev_info dev_info;
+	uint32_t n_tx_queue, nb_lcores;
+	struct rte_eth_txconf *txconf;
+	uint16_t queueid, portid;
+	struct lcore_conf *qconf;
+	uint32_t lcore_id;
+	uint32_t nb_ports;
 	int ret;
 
 	/* Init EAL */
@@ -519,6 +670,203 @@ main(int argc, char **argv)
 	if (check_port_config() < 0)
 		rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
 
+	nb_ports = rte_eth_dev_count_avail();
+	nb_lcores = rte_lcore_count();
+
+	/* Initialize all ports */
+	RTE_ETH_FOREACH_DEV(portid)
+	{
+		struct rte_eth_conf local_port_conf = port_conf;
+
+		/* Skip ports that are not enabled */
+		if ((enabled_port_mask & (1 << portid)) == 0) {
+			printf("\nSkipping disabled port %d\n", portid);
+			continue;
+		}
+
+		/* Init port */
+		printf("Initializing port %d ... ", portid);
+		fflush(stdout);
+
+		nb_rx_queue = get_port_n_rx_queues(portid);
+		n_tx_queue = nb_lcores;
+		if (n_tx_queue > MAX_TX_QUEUE_PER_PORT)
+			n_tx_queue = MAX_TX_QUEUE_PER_PORT;
+		printf("Creating queues: nb_rxq=%d nb_txq=%u... ",
+		       nb_rx_queue, n_tx_queue);
+
+		rte_eth_dev_info_get(portid, &dev_info);
+		if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
+			local_port_conf.txmode.offloads |=
+				DEV_TX_OFFLOAD_MBUF_FAST_FREE;
+
+		local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
+			dev_info.flow_type_rss_offloads;
+		if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
+		    port_conf.rx_adv_conf.rss_conf.rss_hf) {
+			printf("Port %u modified RSS hash function based on "
+			       "hardware support,"
+			       "requested:%#" PRIx64 " configured:%#" PRIx64
+			       "\n",
+			       portid, port_conf.rx_adv_conf.rss_conf.rss_hf,
+			       local_port_conf.rx_adv_conf.rss_conf.rss_hf);
+		}
+
+		ret = rte_eth_dev_configure(portid, nb_rx_queue,
+					    n_tx_queue, &local_port_conf);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Cannot configure device: err=%d, port=%d\n",
+				 ret, portid);
+
+		ret = rte_eth_dev_adjust_nb_rx_tx_desc(portid, &nb_rxd,
+						       &nb_txd);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Cannot adjust number of descriptors: err=%d, "
+				 "port=%d\n",
+				 ret, portid);
+
+		rte_eth_macaddr_get(portid, &ports_eth_addr[portid]);
+		print_ethaddr(" Address:", &ports_eth_addr[portid]);
+		printf(", ");
+		print_ethaddr(
+			"Destination:",
+			(const struct rte_ether_addr *)&dest_eth_addr[portid]);
+		printf(", ");
+
+		/*
+		 * prepare src MACs for each port.
+		 */
+		rte_ether_addr_copy(
+			&ports_eth_addr[portid],
+			(struct rte_ether_addr *)(val_eth + portid) + 1);
+
+		/* Init memory */
+		if (!per_port_pool) {
+			/* portid = 0; this is *not* signifying the first port,
+			 * rather, it signifies that portid is ignored.
+			 */
+			ret = init_mem(0, NB_MBUF(nb_ports));
+		} else {
+			ret = init_mem(portid, NB_MBUF(1));
+		}
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE, "init_mem() failed\n");
+
+		/* Init one TX queue per couple (lcore,port) */
+		queueid = 0;
+		for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+			if (rte_lcore_is_enabled(lcore_id) == 0)
+				continue;
+
+			qconf = &lcore_conf[lcore_id];
+
+			if (numa_on)
+				socketid = (uint8_t)rte_lcore_to_socket_id(
+					lcore_id);
+			else
+				socketid = 0;
+
+			printf("txq=%u,%d,%d ", lcore_id, queueid, socketid);
+			fflush(stdout);
+
+			txconf = &dev_info.default_txconf;
+			txconf->offloads = local_port_conf.txmode.offloads;
+			ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
+						     socketid, txconf);
+			if (ret < 0)
+				rte_exit(EXIT_FAILURE,
+					 "rte_eth_tx_queue_setup: err=%d, "
+					 "port=%d\n",
+					 ret, portid);
+			queueid++;
+		}
+
+		printf("\n");
+	}
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+		qconf = &lcore_conf[lcore_id];
+		printf("\nInitializing rx queues on lcore %u ... ", lcore_id);
+		fflush(stdout);
+		/* Init RX queues */
+		for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
+			struct rte_eth_rxconf rxq_conf;
+
+			portid = qconf->rx_queue_list[queue].port_id;
+			queueid = qconf->rx_queue_list[queue].queue_id;
+
+			if (numa_on)
+				socketid = (uint8_t)rte_lcore_to_socket_id(
+					lcore_id);
+			else
+				socketid = 0;
+
+			printf("rxq=%d,%d,%d ", portid, queueid, socketid);
+			fflush(stdout);
+
+			rte_eth_dev_info_get(portid, &dev_info);
+			rxq_conf = dev_info.default_rxconf;
+			rxq_conf.offloads = port_conf.rxmode.offloads;
+			if (!per_port_pool)
+				ret = rte_eth_rx_queue_setup(
+					portid, queueid, nb_rxd, socketid,
+					&rxq_conf, pktmbuf_pool[0][socketid]);
+			else
+				ret = rte_eth_rx_queue_setup(
+					portid, queueid, nb_rxd, socketid,
+					&rxq_conf,
+					pktmbuf_pool[portid][socketid]);
+			if (ret < 0)
+				rte_exit(EXIT_FAILURE,
+					 "rte_eth_rx_queue_setup: err=%d, "
+					 "port=%d\n",
+					 ret, portid);
+
+		}
+	}
+
+	printf("\n");
+
+	/* Start ports */
+	RTE_ETH_FOREACH_DEV(portid)
+	{
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+
+		/* Start device */
+		ret = rte_eth_dev_start(portid);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "rte_eth_dev_start: err=%d, port=%d\n", ret,
+				 portid);
+
+		/*
+		 * If enabled, put device in promiscuous mode.
+		 * This allows IO forwarding mode to forward packets
+		 * to itself through 2 cross-connected  ports of the
+		 * target machine.
+		 */
+		if (promiscuous_on)
+			rte_eth_promiscuous_enable(portid);
+	}
+
+	printf("\n");
+
+	check_all_ports_link_status(enabled_port_mask);
+
+	/* Stop ports */
+	RTE_ETH_FOREACH_DEV(portid) {
+		if ((enabled_port_mask & (1 << portid)) == 0)
+			continue;
+		printf("Closing port %d...", portid);
+		rte_eth_dev_stop(portid);
+		rte_eth_dev_close(portid);
+		printf(" Done\n");
+	}
 	printf("Bye...\n");
 
 	return ret;
Signature · 1 line
2.25.1
#63 lore raw

PATCH v2 26/28 l3fwd-graph: add graph config and main loop

In reply to [email protected]

From: Nithin Dabilpuram <[email protected]>

Add graph creation, configuration logic and graph main loop. This graph main loop is run on every slave lcore and calls rte_graph_walk() to walk over lcore specific rte_graph. Master core accumulates and prints graph walk stats of all the lcore's graph's.

---
 examples/l3fwd-graph/main.c | 242 +++++++++++++++++++++++++++++++++++-
 1 file changed, 240 insertions(+), 2 deletions(-)
examples/l3fwd-graph/main.c +240 -2
index 47d2f2ecb..28db947b5 100644
@@ -23,9 +23,13 @@
 #include <rte_cycles.h>
 #include <rte_eal.h>
 #include <rte_ethdev.h>
+#include <rte_graph_worker.h>
+#include <rte_launch.h>
 #include <rte_lcore.h>
 #include <rte_log.h>
 #include <rte_mempool.h>
+#include <rte_node_eth_api.h>
+#include <rte_node_ip4_api.h>
 #include <rte_per_lcore.h>
 #include <rte_string_fns.h>
 #include <rte_vect.h>
@@ -75,12 +79,17 @@ static uint32_t enabled_port_mask;
 struct lcore_rx_queue {
 	uint16_t port_id;
 	uint8_t queue_id;
+	char node_name[RTE_NODE_NAMESIZE];
 };
 
 /* Lcore conf */
 struct lcore_conf {
 	uint16_t n_rx_queue;
 	struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+
+	struct rte_graph *graph;
+	char name[RTE_GRAPH_NAMESIZE];
+	rte_graph_t graph_id;
 } __rte_cache_aligned;
 
 static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
@@ -119,6 +128,25 @@ static struct rte_eth_conf port_conf = {
 
 static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
 
+static struct rte_node_ethdev_config ethdev_conf[RTE_MAX_ETHPORTS];
+
+struct ipv4_l3fwd_lpm_route {
+	uint32_t ip;
+	uint8_t depth;
+	uint8_t if_out;
+};
+
+#define IPV4_L3FWD_LPM_NUM_ROUTES                                              \
+	(sizeof(ipv4_l3fwd_lpm_route_array) /                                  \
+	 sizeof(ipv4_l3fwd_lpm_route_array[0]))
+/* 198.18.0.0/16 are set aside for RFC2544 benchmarking. */
+static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
+	{RTE_IPV4(198, 18, 0, 0), 24, 0}, {RTE_IPV4(198, 18, 1, 0), 24, 1},
+	{RTE_IPV4(198, 18, 2, 0), 24, 2}, {RTE_IPV4(198, 18, 3, 0), 24, 3},
+	{RTE_IPV4(198, 18, 4, 0), 24, 4}, {RTE_IPV4(198, 18, 5, 0), 24, 5},
+	{RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7},
+};
+
 static int
 check_lcore_params(void)
 {
@@ -624,17 +652,87 @@ signal_handler(int signum)
 	}
 }
 
+static void
+print_stats(void)
+{
+	const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+	const char clr[] = {27, '[', '2', 'J', '\0'};
+	struct rte_graph_cluster_stats_param s_param;
+	struct rte_graph_cluster_stats *stats;
+	const char *pattern = "worker_*";
+
+	/* Prepare stats object */
+	memset(&s_param, 0, sizeof(s_param));
+	s_param.f = stdout;
+	s_param.socket_id = SOCKET_ID_ANY;
+	s_param.graph_patterns = &pattern;
+	s_param.nb_graph_patterns = 1;
+
+	stats = rte_graph_cluster_stats_create(&s_param);
+	if (stats == NULL)
+		rte_exit(EXIT_FAILURE, "Unable to create stats object\n");
+
+	while (!force_quit) {
+		/* Clear screen and move to top left */
+		printf("%s%s", clr, topLeft);
+		rte_graph_cluster_stats_get(stats, 0);
+		rte_delay_ms(1E3);
+	}
+
+	rte_graph_cluster_stats_destroy(stats);
+}
+
+/* Main processing loop */
+static int
+graph_main_loop(void *conf)
+{
+	struct lcore_conf *qconf;
+	struct rte_graph *graph;
+	uint32_t lcore_id;
+
+	RTE_SET_USED(conf);
+
+	lcore_id = rte_lcore_id();
+	qconf = &lcore_conf[lcore_id];
+	graph = qconf->graph;
+
+	if (!graph) {
+		RTE_LOG(INFO, L3FWD_GRAPH, "Lcore %u has nothing to do\n",
+			lcore_id);
+		return 0;
+	}
+
+	RTE_LOG(INFO, L3FWD_GRAPH,
+		"Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
+		qconf->name, graph);
+
+	while (likely(!force_quit))
+		rte_graph_walk(graph);
+
+	return 0;
+}
+
 int
 main(int argc, char **argv)
 {
+	uint8_t rewrite_data[2 * sizeof(struct rte_ether_addr)];
+	static const char *node_patterns[] = {
+		"ip4*",
+		"ethdev_tx-*",
+		"pkt_drop",
+	};
 	uint8_t nb_rx_queue, queue, socketid;
+	struct rte_graph_param graph_conf;
 	struct rte_eth_dev_info dev_info;
+	uint32_t nb_ports, nb_conf = 0;
 	uint32_t n_tx_queue, nb_lcores;
 	struct rte_eth_txconf *txconf;
-	uint16_t queueid, portid;
+	uint16_t queueid, portid, i;
 	struct lcore_conf *qconf;
+	uint16_t nb_graphs = 0;
+	uint16_t nb_patterns;
+	uint8_t rewrite_len;
 	uint32_t lcore_id;
-	uint32_t nb_ports;
 	int ret;
 
 	/* Init EAL */
@@ -783,6 +881,18 @@ main(int argc, char **argv)
 			queueid++;
 		}
 
+		/* Setup ethdev node config */
+		ethdev_conf[nb_conf].port_id = portid;
+		ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
+		ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
+		if (!per_port_pool)
+			ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
+
+		else
+			ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
+		ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
+
+		nb_conf++;
 		printf("\n");
 	}
 
@@ -826,11 +936,26 @@ main(int argc, char **argv)
 					 "port=%d\n",
 					 ret, portid);
 
+			/* Add this queue node to its graph */
+			snprintf(qconf->rx_queue_list[queue].node_name,
+				 RTE_NODE_NAMESIZE, "ethdev_rx-%u-%u", portid,
+				 queueid);
+		}
+
+		/* Alloc a graph to this lcore only if source exists  */
+		if (qconf->n_rx_queue) {
+			qconf->graph_id = nb_graphs;
+			nb_graphs++;
 		}
 	}
 
 	printf("\n");
 
+	/* Ethdev node config, skip rx queue mapping */
+	ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
+	if (ret)
+		rte_exit(EXIT_FAILURE, "rte_node_eth_config: err=%d\n", ret);
+
 	/* Start ports */
 	RTE_ETH_FOREACH_DEV(portid)
 	{
@@ -858,6 +983,119 @@ main(int argc, char **argv)
 
 	check_all_ports_link_status(enabled_port_mask);
 
+	/* Graph Initialization */
+	memset(&graph_conf, 0, sizeof(graph_conf));
+	graph_conf.node_patterns = node_patterns;
+	nb_patterns = RTE_DIM(node_patterns);
+
+	for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+		rte_graph_t graph_id;
+		rte_edge_t i;
+
+		if (rte_lcore_is_enabled(lcore_id) == 0)
+			continue;
+
+		qconf = &lcore_conf[lcore_id];
+
+		/* Skip graph creation if no source exists */
+		if (!qconf->n_rx_queue)
+			continue;
+
+		/* Add rx node patterns of this lcore */
+		for (i = 0; i < qconf->n_rx_queue; i++) {
+			graph_conf.node_patterns[nb_patterns + i] =
+				qconf->rx_queue_list[i].node_name;
+		}
+
+		graph_conf.nb_node_patterns = nb_patterns + i;
+		graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
+
+		snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+			 lcore_id);
+
+		graph_id = rte_graph_create(qconf->name, &graph_conf);
+		if (graph_id != qconf->graph_id)
+			rte_exit(EXIT_FAILURE,
+				 "rte_graph_create(): graph_id=%d not "
+				 " as expected for lcore %u(%u\n",
+				 graph_id, lcore_id, qconf->graph_id);
+
+		qconf->graph = rte_graph_lookup(qconf->name);
+		if (!qconf->graph)
+			rte_exit(EXIT_FAILURE,
+				 "rte_graph_lookup(): graph %s not found\n",
+				 qconf->name);
+	}
+
+	memset(&rewrite_data, 0, sizeof(rewrite_data));
+	rewrite_len = sizeof(rewrite_data);
+
+	/* Add route to ip4 graph infra */
+	for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
+		char route_str[INET6_ADDRSTRLEN * 4];
+		char abuf[INET6_ADDRSTRLEN];
+		struct in_addr in;
+		uint32_t dst_port;
+		uint16_t next_hop;
+
+		/* Skip unused ports */
+		if ((1 << ipv4_l3fwd_lpm_route_array[i].if_out &
+		     enabled_port_mask) == 0)
+			continue;
+
+		dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
+		next_hop = i;
+
+		in.s_addr = htonl(ipv4_l3fwd_lpm_route_array[i].ip);
+		snprintf(route_str, sizeof(route_str), "%s / %d (%d)",
+			 inet_ntop(AF_INET, &in, abuf, sizeof(abuf)),
+			 ipv4_l3fwd_lpm_route_array[i].depth,
+			 ipv4_l3fwd_lpm_route_array[i].if_out);
+
+		ret = rte_node_ip4_route_add(
+			ipv4_l3fwd_lpm_route_array[i].ip,
+			ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
+			RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
+
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Unable to add ip4 route %s to graph\n",
+				 route_str);
+
+		memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
+
+		/* Add next hop for a given destination */
+		ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
+					       rewrite_len, dst_port);
+		if (ret < 0)
+			rte_exit(EXIT_FAILURE,
+				 "Unable to add next hop %u for "
+				 "route %s\n",
+				 next_hop, route_str);
+
+		RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
+			route_str, next_hop);
+	}
+
+	/* Launch per-lcore init on every slave lcore */
+	rte_eal_mp_remote_launch(graph_main_loop, NULL, SKIP_MASTER);
+
+	/* Accumulate and print stats on master until exit */
+	if (rte_graph_has_stats_feature())
+		print_stats();
+
+	/* Wait for slave cores to exit */
+	ret = 0;
+	RTE_LCORE_FOREACH_SLAVE(lcore_id) {
+		ret = rte_eal_wait_lcore(lcore_id);
+		/* Destroy graph */
+		rte_graph_destroy(lcore_conf[lcore_id].name);
+		if (ret < 0) {
+			ret = -1;
+			break;
+		}
+	}
+
 	/* Stop ports */
 	RTE_ETH_FOREACH_DEV(portid) {
 		if ((enabled_port_mask & (1 << portid)) == 0)
Signature · 1 line
2.25.1