DPDK User Space: Session onUseability and Ease of Use

All:

Captured white board notes from Jim McNamara's session on Useability and Ease of Use at DPDK User Space today are here:

http://people.redhat.com/therbert/Useability_and_Ease_of_Use_DPDK_User_Space/

--Thomas F Herbert
Red Hat.

Hi Tom,

Thanks for that. Here is my summary of the "Usability and Ease of Use" session from memory and notes. Correction and additions welcome.

* Latest version of the DPDK docs

- (From an earlier session). Add a doc/latest build of the docs to dpdk.org.

* PMD lite

- Do we need a lighter PMD model? Perhaps based on the Mellanox model.

- Vincent suggested be could remove 90% of the code. I'll leave Vincent explain this one.

* OvS issues with Usability

- Discussion of the DPSK usability issues highlighted on the OvS mailing list.

- http://openvswitch.org/pipermail/dev/2015-August/058814.html

* Distributed testing

- There should be some form of distributed testing so patches can be tested on OSes and hardware that a dev doesn't have.

- Suggestion to have an Open Lab/Pod similar to the OPNFV model that participants can use.

* Debuggability

- User expectation for tools like tcpdump, ip, ipconfig to work with DPDK bound NICs.

- Easier in a pipeline application where debug is added as a pipeline stage.

- Maybe add debug hooks via rx/tx callbacks.

- Add/extend a solution based on KNI.

- Use systemd naming algorithm for KNI.

* Create a User Mailing List

- An observation was made that the [email protected] list was very developer orientated and patch heavy.

- Suggestion to add a [email protected] mailing list for people with issues or subjects that aren't development related.

- This seams easy to implement. It may not be well supported however resulting in users cross posting into [email protected].

- Probably worth trying anyway.

* Make the dpdk.org website patchable

- There are already plans to host the dpdk.org code in a git repo.

* Add a Contributing Guide.

- We are at the stage where we need one.

- Suggestion to just use the Kernel guide.

- Tailor it for DPDK.

- Also explain the review process, acks, nacks, etc.

* Add a README .txt or .1st to the root dir.

- This could just include links to the getting started guides and other docs. Either to the online docs or how to build the local html versions.

* EAL annoyances

- Move the EAL to the kernel.

- Have more/better/all default options. EAL figures out its own requirements.

- Have a default for -n.

* Hugepage consumption

- Do not allow DPDK applications to grab all available hugepages.

- Issues with running DPDK apps in tandem with other hugepage hungry apps such as Java/Eclipse.

* rte_malloc()

- Don't use rte_malloc() for non critical objects where malloc() would do.

- Suggestion to allow the type of required memory to be specified by rte_malloc()-like function.

* The Build system

- Make install needs to be improved. Doesn't so what the user expects.

- Use autotools and configure. (There were some objections that this may not be an improvement.)

- Use kconfig.

- Keep going with what we have now until it gets too unwieldy and needs to be changed. Then use kconfig.

- Add better support for cross compilation. Useful for arm target.

* Should DPDK applications be running as root

- Clearly not a great option.

- Currently required due to kernel.

* Mempool debugging

- We need better tools to debug memory leaks in the mempools.

- Suggestion to do this via a valgrind plugin.

* Kernel management of drivers

* Too much duplicated code in the PMDs

- Duplicated code has crept in organically as PMDs have been added.

- Should be moved up to moved up to the ethdev level

* Logging and debugging via a secondary process

- Not a well known technique but very useful/powerful

* Run DPDK as a daemon.

* Issues with config files

- Too many options turned off by default: code paths don't get compiled/tested.

* More sample apps

- Some more examples of using secondary processes.

Of these we the following could be addressed in the near term:

* Latest version of the docs. - Needs support from 6Wind.

* Distributed testing. - Needs support from Intel initially. Some of this is already being rolled out on the [email protected] list for Intel hardware:
http://dpdk.org/ml/archives/test-report/. Other hardware vendors could use the same automated test framework to host something similar.

* Debuggability. - Need some volunteers or workable suggestions.

* Create a User Mailing List. - Needs support from 6Wind.

* Make the dpdk.org website patchable. - Needs support from 6Wind.

* Add a Contributing Guide. - I will submit a doc patch.

* Add a README .txt or .1st to the root dir. - I will submit a doc patch.

* The Build system. - There are some patches on the list for an updated "make install". This could be a start point.

* Mempool debugging. - Any volunteers to add/improve valgrind support for DPDK? See also this patch from Brocade:
https://bugs.kde.org/show_bug.cgi?id=350405

* Too much duplicated code in the PMDs. - Any volunteers to refactor common PMD code up into the ethdev layer?

* Logging and debugging via a secondary process. Any volunteers to add a sample app that demonstrates the technique?

John.

Thanks John for the summary.

2015-10-13 16:36, Mcnamara, John:

Please explain what you mean here. It's difficult to imagine.

OK
It's as simple as a git hook.
Anybody to write and test one locally?
The script could be hosted in the future website git repo.

IBM has first started a daily compilation test.
Others are welcome.
We had a small session dedicated to this topic during the useability session.

OK
Name: [email protected] or other suggestion?

OK
Needs a maintainer, a name for the git repo, the mailing list and the patchwork.

Thanks

We need more details.
Please start a new thread or a RFC.

There is already one:

	http://dpdk.org/browse/dpdk/tree/app/proc_info/main.c#n289

Hi,

Other pages seem to have standardised on "users" - I don't think it matters much, either is fine.

Thanks,
Dave.

Dave Neary - NFV/SDN Community Strategy
Open Source and Standards, Red Hat - http://community.redhat.com
Ph: +1-978-399-2182 / Cell: +1-978-799-3338

+Martin@ARM on it. We were rising this topic with him last week.

[...]

Heh, somebody agrees with me. Patch sent:
http://dpdk.org/ml/archives/dev/2015-October/025045.html

- Panu -

It is not 100% accurate. With some systems, it shall be possible to run DPDK application without running them as root.

It was commented that there is a solution when PMDs are designed differently (verbs API => memory registration of users). Moreover, it would ease the cleanup of resources when a DPDK application does a segfault.

Thomas, John,

thanks for your notes.

Enclosed the contents of our chats,

benefits of 100% userland drivers:
- no dependency on getting kernel patch upstreams first
- it is possible to run on a latest HW even when the Linux host (or
host OS) does not support the new NIC/boards supported by the DPDK

drawbacks of 100% userland drivers:
- redundancy of source code (duplication into the kernel and userland)
=> more maintenance
- DPDK PMDs are in fact outdated while kernel drivers are supporting
latest HW (port management - optics - , latest HW revisions, managing firmware updates) more frequently
- do not run when not root applications
- cannot reuse port management tools from the kernels (ethtool, etc.)

ixgbe pmd
=========
100% lines of userland
must be run as root
~ 40K lines
~ kernel overlaps with
http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/base
~ mostly only
http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx[_vec].c is needed, other files are not needed (6k lines of code)

mlx4 pmd
========
10% lines userland, then using kernel's ones
could be run as a non root process
~ 5.2K lines of code
~ the rxtx.c file is http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/
BUT using infiniband :( => should be cleand up to use bare minimum like,
+ /* Allocate protection domain. */
http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n4828

+ Register memory regions -> ibv_reg_mr()
http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n792

Best regards,
Vincent

Hi Vincent,

is the approach described for the mlx4 driver not dependent upon the hardware having support for infiniband verbs, in which case I don't see how it could be used for other NICs. Has an attempt been made to create a driver for another NIC family using this approach to prove it is generally applicable?

/Bruce

One thing I would appreciate is more basic samples for the more advanced functionality. For example, the qos_sched example is (in my opimion) quite hard to follow since it has both an integrated command line and a config file.

I think it would be better to split the example to multiple smaller samples, which uses static setups to more clearly illustrate some particular functionality of the QoS framework.

// Simon