DPDK 16.04 link changes cause PMD drivers to not be loaded

dpdk-users Aurojit Panda 3 participants 5 messages Apr 21, 2016 – Apr 22, 2016 4 messages not in the archive

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Aurojit Panda

author

2016-04-21 15:01 UTC #4 lore raw

[Cross-posting to dev]

Quoted 10 lines · Panu Matilainen

Quoted 8 lines · Aurojit Panda
I am sorry that is a bit unintuitive considering:
(a) This behavior differs between static and shared builds of DPDK.
- In fact this behavior was identical in 2.2, and even in mainline before 948fd64befc3726 went in.
(b) You already know the EAL_PMD_PATH at build time, and it makes it quite hard to ship scripts or anything to build DPDK, since now the configuration file becomes dependent on location.

Quoted 6 lines

The expectation is that shared libraries are installed to a shared, known location. Otherwise there's not that much point in using shared libraries.
Note that you can also use the EAL option -d to load either single PMDs or entire directories at runtime (regardless of EAL_PMD_PATH) so you dont have to know the path at build time if that's what bothers you most.

I am actually bothered by needing to supply the path at all. You seem to assume that knowing an install path is essential for shared libraries, while you can in reality embed this in the rpath. Shared libraries are used for many reasons, including interop with some runtime, in which case they are not installed at a known place. I would prefer for those PMDs that are included with DPDK that either they be loaded from the "known" build path for them.

Quoted 13 lines

Quoted 4 lines
I wonder if you would consider changing this, as it stands just building DPDK after setting CONFIG_RTE_SHARED_LIB=y results in a testpmd that cannot run.
In shared library configuration, testpmd is not directly runnable regardless of the drivers since its missing all the other libraries too:
[pmatilai@sopuli dpdk]$ build/app/testpmd
build/app/testpmd: error while loading shared libraries: librte_distributor.so.1.1: cannot open shared object file: No such file or directory
[pmatilai@sopuli dpdk]$

Quoted 6 lines

You'll need to get those libraries into linkers path anyway, either by installing them to a common location or by extending LD_LIBRARY_PATH. In either case, you need to know where the libraries are anyway.
All of which is not to say there might not be room for improvement, but the linking behavior is not going to change. Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially for packaging and shipping, at some extra cost in setup complexity.

I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also please note:
(a) This behavior changed recently.
(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required to get any DPDK application to interact with ports.

Quoted 4 lines

For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it. But that's not the way plugins are meant to be used.

That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges. My alternatives neither break people's abilities to use plugins, nor do they impact behavior.

Quoted 5 lines

Oh and BTW, please don't top-post.
- Panu -

Thomas Monjalon

2016-04-21 15:18 UTC #5 lore raw

2016-04-21 08:01, Aurojit Panda:

Quoted 1 line

Panu Matilainen wrote:

[...]

Quoted 7 lines

Quoted 3 lines
Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially
for packaging and shipping, at some extra cost in setup complexity.
I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also please note:
(a) This behavior changed recently.

What changed recently?

Quoted 5 lines

(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required
to get any DPDK application to interact with ports.

?
Are you talking about combined library?

Quoted 9 lines

Quoted 4 lines
For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it. But that's not the way plugins are meant to be used.
That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges.
My alternatives neither break people's abilities to use plugins, nor do they impact behavior.

Please do not hesitate to send some patch to show your solution. Thanks

Aurojit Panda

author

2016-04-21 15:24 UTC #6 lore raw

Quoted 12 lines · Thomas Monjalon

2016-04-21 08:01, Aurojit Panda:
Quoted 1 line
Panu Matilainen wrote:
[...]
Quoted 6 lines
Quoted 3 lines
Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially
for packaging and shipping, at some extra cost in setup complexity.
I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also please note:
(a) This behavior changed recently.
What changed recently?

Change 948fd64befc3726 moved DPDK from building a shared library to using LD linker scripts.

Quoted 8 lines

Quoted 5 lines
(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required
to get any DPDK application to interact with ports.
?
Are you talking about combined library?

Yes, if you build a shared combined library, PMD drivers are not automatically loaded anymore. This implies that programs do not enumerate the set of ports on the machine. (Unless CONFIG_EAL_PMD_PATH is correctly set to the absolute directory where the built DPDK will live).

Quoted 11 lines

Quoted 9 lines
Quoted 5 lines
For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it. But that's not the way plugins are meant to be used.
That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges.
My alternatives neither break people's abilities to use plugins, nor do they impact behavior.
Please do not hesitate to send some patch to show your solution.

My initial e-mail was sent because I don't entirely understand LD linker scripts (the PMD libraries are listed within the built libdpdk.so file), so the only patch I can suggest is undoing 948fd64befc3726 in which case the behavior as I describe.

Quoted 1 line

Thanks

Thanks

Panda

Panu Matilainen

2016-04-22 10:08 UTC #7 lore raw

Quoted 41 lines · Aurojit Panda

Quoted 12 lines · Thomas Monjalon
2016-04-21 08:01, Aurojit Panda:
Quoted 1 line
Panu Matilainen wrote:
[...]
Quoted 6 lines
Quoted 3 lines
Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially
for packaging and shipping, at some extra cost in setup complexity.
I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also please note:
(a) This behavior changed recently.
What changed recently?
Change 948fd64befc3726 moved DPDK from building a shared library to using LD linker scripts.
Quoted 8 lines
Quoted 5 lines
(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required
to get any DPDK application to interact with ports.
?
Are you talking about combined library?
Yes, if you build a shared combined library, PMD drivers are not automatically loaded anymore. This implies that programs do not enumerate the set of ports on the machine. (Unless CONFIG_EAL_PMD_PATH is correctly set to the absolute directory where the built DPDK will live).
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it.
But that's not the way plugins are meant to be used.
That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges.
My alternatives neither break people's abilities to use plugins, nor do they impact behavior.

Quoted 6 lines

Quoted 1 line
Please do not hesitate to send some patch to show your solution.
My initial e-mail was sent because I don't entirely understand LD linker scripts (the PMD libraries are listed within the built libdpdk.so file), so the only patch I can suggest is undoing 948fd64befc3726 in which case the behavior as I describe.

Reverting would be counter-productive, the change was introduced for a reason. Many of them in fact.

Any application wanting to force linkage to plugins always needed to link them inside --whole-archive section with static or individual shared libraries, now that is true for the combined shared library as well.

Within DPDK itself it would be easy enough to add an option to force link-in of the plugins (see the comment about mk/rte.app.mk change), but not sure we want to have yet another build option. And anything else still would need to use --whole-archive if they want that behavior.

So yes it's a behavior change for that particular corner which nobody thought of or brought up during review, and so ended up being undocumented too.

Like said there are always ways to improve things, once the use-case is understood, actual examples of how you are deploying DPDK would perhaps help. One possibility could be attempting to load known plugins from LD_LIBRARY_PATH, but that requires inserting that information somewhere in the binaries where it currently is not needed.

- Panu -

Aurojit Panda

author

2016-04-21 15:29 UTC #8 lore raw

In reply to Thomas Monjalon

[The original report is included below for your convenience]

Quoted 25 lines · Thomas Monjalon

2016-04-21 08:01, Aurojit Panda:
Quoted 1 line
Panu Matilainen wrote:
[...]
Quoted 6 lines
Quoted 3 lines
Again, PMDs are *plugins* that are *meant* to be loaded at runtime. That allows for all sorts of flexibility especially
for packaging and shipping, at some extra cost in setup complexity.
I am all for a plugin architecture, I was merely suggesting that you embed some path infromation at the beginning. Also please note:
(a) This behavior changed recently.
What changed recently?
Quoted 5 lines
(b) This change is entirely undocumented, which is why I was reporting it in the first place.
(c) It is actually quite unintutive, because previously ensuring LD_LIBRARY_PATH was correct was all that was required
to get any DPDK application to interact with ports.
?
Are you talking about combined library?
Quoted 4 lines
Quoted 4 lines
For your own purposes, you can of course tweak the linking settings as much as you like. Look for "plugins" in mk/rte.app.mk and change the shared lib condition on the line above to "y" and there you have it. But that's not the way plugins are meant to be used.

Quoted 7 lines

Quoted 4 lines
That is not a reasonable solution given that it makes it very hard to track future changes to DPDK without merges.
My alternatives neither break people's abilities to use plugins, nor do they impact behavior.
Please do not hesitate to send some patch to show your solution. Thanks

I was trying to run testpmd from DPDK 16.04 on Linux with kernel version 4.4.0-1 (ld version 2.26). My machine has two XL710QDA2 NICs, and I built DPDK as a shared, combined library (i.e., CONFIG_RTE_BUILD_SHARED_LIB=y and CONFIG_RTE_BUILD_COMBINE_LIBS=y in config/common_linuxapp). I found that the issue is due to ld not linking against all libraries with the new linker script (introduced in 948fd64befc3726) but am not sure how to fix this. As evidence for this being caused by this change:

$ LD_TRACE_LOADED_OBJECTS=y ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
          linux-vdso.so.1 (0x00007ffe11b3e000)
          librte_distributor.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_distributor.so.1.1
(0x00007fa55eb85000)
          librte_reorder.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_reorder.so.1.1 (0x00007fa55e982000)
          librte_pipeline.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pipeline.so.3.1 (0x00007fa55e77d000)
          librte_table.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_table.so.2.1 (0x00007fa55e55e000)
          librte_port.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_port.so.2.1 (0x00007fa55e34e000)
          librte_timer.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_timer.so.1.1 (0x00007fa55e145000)
          librte_hash.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_hash.so.2.1 (0x00007fa55df37000)
          librte_jobstats.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_jobstats.so.1.1 (0x00007fa55dd35000)
          librte_lpm.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_lpm.so.2.1 (0x00007fa55db2e000)
          librte_power.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_power.so.1.1 (0x00007fa55d91f000)
          librte_acl.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_acl.so.2.1 (0x00007fa55d705000)
          librte_meter.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_meter.so.1.1 (0x00007fa55d504000)
          librte_sched.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_sched.so.1.1 (0x00007fa55d2fd000)
          librte_vhost.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_vhost.so.2.1 (0x00007fa55d0e6000)
          librte_kvargs.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_kvargs.so.1.1 (0x00007fa55cee4000)
          librte_mbuf.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mbuf.so.2.1 (0x00007fa55cce2000)
          librte_ip_frag.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ip_frag.so.1.1 (0x00007fa55cada000)
          libethdev.so.3.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/libethdev.so.3.1 (0x00007fa55c84f000)
          librte_cryptodev.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cryptodev.so.1.1 (0x00007fa55c647000)
          librte_mempool.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_mempool.so.1.1 (0x00007fa55c444000)
          librte_ring.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_ring.so.1.1 (0x00007fa55c242000)
          librte_eal.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_eal.so.2.1 (0x00007fa55bfe1000)
          librte_cmdline.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cmdline.so.2.1 (0x00007fa55bdd8000)
          librte_cfgfile.so.2.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_cfgfile.so.2.1 (0x00007fa55bbd6000)
          librte_pmd_bond.so.1.1 => /home/apanda/e2d2/3rdparty/dpdk/build/lib/librte_pmd_bond.so.1.1 (0x00007fa55b9c4000)
          libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa55b7ae000)
          libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa55b5aa000)
          libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa55b38d000)
          libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa55afe9000)
          /lib64/ld-linux-x86-64.so.2 (0x00007fa55ed87000)
          libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa55aceb000)
          librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa55aae3000)

As you can see librte_pmd_i40e.so is missing from the list of files. Furthermore, using

$ LD_PRELOAD="../lib/librte_pmd_i40e.so" ./testpmd -c 0x1c00 -n 4 -w 82:00.0 -w 82:00.1 --file-prefix "send" -- -i
correctly enumerates the NICs.

I unfortunately do not how to fix the linker script as checked in, but wanted to report this bug in case someone has a fix.