Skip to contentSkip to conversation

pci: fix missing pci bus with shared library build

PATCH In review Stephen Hemminger 3 participants 23 messages Jul 15, 2019 – Jul 24, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pci: fix missing pci bus with shared library build

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +1 -1
index a277c808ed8e..470b92e4d73e 100644
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched
Signature · 1 line
2.20.1
#1 lore raw
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched

This only seems to happen on 18.11 and if configuration is reduced.

#2 lore raw
In reply to Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters
#3 lore raw
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
#4 lore raw
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters  

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

It is broken with original link order.
With the patch it works fine.

#5 lore raw
In reply to Bruce Richardson
Quoted 7 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

Application should never have to use -d to force driver load! The purpose of all the pmdinfo, and probing logic is to not do this.

#6 lore raw
In reply to Bruce Richardson
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters  

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

There is something else going on.
This patch isn't fixing the problem consistently.

The suggestion of using -d is still wrong. We need to figure out how to get applications to run the same whether using shared library or statically linked.

#7 lore raw
In reply to Bruce Richardson
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters  

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

#8 lore raw

19/07/2019 22:55, Stephen Hemminger:

Quoted 58 lines
Quoted 55 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters  

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

	sudo ./build/l2fwd -c F00000 -- -p 3
Quoted 8 lines
Quoted 3 lines

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.

#9 lore raw
Quoted 66 lines · Thomas Monjalon

19/07/2019 22:55, Stephen Hemminger:

Quoted 65 lines
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched  

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters  

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

Quoted 9 lines
Quoted 1 line

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.

This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to /usr/local or /usr, the EAL PMD path will be correctly configured and properly probe all drivers.

#10 lore raw
Quoted 77 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 66 lines · Thomas Monjalon

19/07/2019 22:55, Stephen Hemminger:

Quoted 65 lines
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…

If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.

This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.

Fixes c752998b5e2e ("pci: introduce library and driver")
---
 mk/rte.app.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +0 -0
index a277c808ed8e..470b92e4d73e 100644

mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97

Quoted 9 lines
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK)          += -lrte_stack
 _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING)   += -lrte_mempool_ring
 _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING)           += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL)            += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI)            += -lrte_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE)        += -lrte_cmdline
 _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER)        += -lrte_reorder
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrte_sched    

It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...

Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
  Cause: Invalid EAL parameters    

I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:

Quoted 5 lines
	sudo ./build/l2fwd -c F00000 -- -p 3

while this succeeds and runs fine

	sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3  

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

Quoted 9 lines
Quoted 1 line

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.

Quoted 4 lines

This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to /usr/local or /usr, the EAL PMD path will be correctly configured and properly probe all drivers.

The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.

One part of fixing this is:

mk/rte.app.mk +5 -0
index df917f946497..46bdff8ec5e8 100644
@@ -130,6 +130,9 @@ ifeq ($(CONFIG_RTE_LIBRTE_FSLMC_BUS),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_COMMON_FSLMC)   += -lrte_common_fslmc
 endif
 
+# Bus devices register in constructor so always link
+_LDLIBS-y      += --no-as-needed
+
 _LDLIBS-$(CONFIG_RTE_LIBRTE_PCI_BUS)        += -lrte_bus_pci
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VDEV_BUS)       += -lrte_bus_vdev
 _LDLIBS-$(CONFIG_RTE_LIBRTE_DPAA_BUS)       += -lrte_bus_dpaa
@@ -137,6 +140,8 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS)      += -lrte_bus_fslmc
 endif
 
+_LDLIBS-y      += --as-needed
+
 ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n)
 # plugins (link only if static libraries)
 
#11 lore raw

22/07/2019 18:43, Stephen Hemminger:

Quoted 16 lines
Quoted 14 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 12 lines · Thomas Monjalon

19/07/2019 22:55, Stephen Hemminger:

Quoted 4 lines

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.

Quoted 7 lines
Quoted 4 lines

This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to /usr/local or /usr, the EAL PMD path will be correctly configured and properly probe all drivers.

The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.

Are the constructors run on dlopen of the bus driver?

#12 lore raw
Quoted 22 lines · On Mon, 22 Jul 2019 19:04:55 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 18:43, Stephen Hemminger:

Quoted 21 lines
Quoted 18 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 4 lines · Thomas Monjalon

19/07/2019 22:55, Stephen Hemminger:

Quoted 3 lines

The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.

Quoted 8 lines
Quoted 1 line

See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library

The constructor is run when calling dlopen, right?

Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.

This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to /usr/local or /usr, the EAL PMD path will be correctly configured and properly probe all drivers.

Quoted 7 lines
Quoted 2 lines

The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

#13 lore raw

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

#14 lore raw
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 6 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.

For distributions, this also matters. Linking with -ldpdk which is a linker script should work.

#15 lore raw

22/07/2019 20:34, Stephen Hemminger:

Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 6 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 2 lines

I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.

The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.

Quoted 2 lines

For distributions, this also matters. Linking with -ldpdk which is a linker script should work.

There is no longer this linker script with meson.

#16 lore raw
Quoted 25 lines · On Tue, 23 Jul 2019 09:59:04 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 20:34, Stephen Hemminger:

Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 6 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 2 lines

I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.

Quoted 9 lines

The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.

Quoted 2 lines

For distributions, this also matters. Linking with -ldpdk which is a linker script should work.

There is no longer this linker script with meson.

Ok, for usability that is a problem.
Requiring user to figure out which DPDK libraries to link with is a serious waste of time. It should be possible to just link with -ldpdk and distribution packages and just get the necessary libraries for the application (no extra rte_foo_bar .so loaded at run time), and the application should just work.

The idea that the user should link with 20 shared libraries, in the right order and pass -d flags to eal_init to load the right PMD is user hostile. It only makes sense if you want to invent yet another layer to manage the ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.

I know it is hard, and I know not all this will make it into 19.08
but let's try and do better. The DPDK already has a reputation as being like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.

#17 lore raw

23/07/2019 20:29, Stephen Hemminger:

Quoted 43 lines
Quoted 25 lines · On Tue, 23 Jul 2019 09:59:04 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 20:34, Stephen Hemminger:

Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 6 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 2 lines

I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.

Quoted 7 lines

The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.

Quoted 2 lines

For distributions, this also matters. Linking with -ldpdk which is a linker script should work.

There is no longer this linker script with meson.

Ok, for usability that is a problem.
Requiring user to figure out which DPDK libraries to link with is a serious waste of time. It should be possible to just link with -ldpdk and distribution packages and just get the necessary libraries for the application (no extra rte_foo_bar .so loaded at run time), and the application should just work.

Quoted 8 lines

The idea that the user should link with 20 shared libraries, in the right order and pass -d flags to eal_init to load the right PMD is user hostile. It only makes sense if you want to invent yet another layer to manage the ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.

I know it is hard, and I know not all this will make it into 19.08
but let's try and do better. The DPDK already has a reputation as being like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.

About the build-time link, it is already improved thanks to the pkgconfig file generated by meson.
The only issue I understand is the runtime need for -d, which is a feature, and could be removed.

#18 lore raw
In reply to Thomas Monjalon
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 8 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

What I think should happen (and isn't is):

1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.

2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.

3. PMD's are linked as --whole-archive, and --as-needed.

4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).

This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.

#19 lore raw
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 8 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 5 lines

What I think should happen (and isn't is):

1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.

This should be applied to the whole of the bus drivers, not just the PCI bus.

Quoted 5 lines

2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.

3. PMD's are linked as --whole-archive, and --as-needed.

I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.

Quoted 9 lines

4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).

This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.

In all this you seem to be assuming that the drivers are not picked up at runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.

Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?

/Bruce

#20 lore raw
Quoted 61 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 8 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 5 lines

What I think should happen (and isn't is):

1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.

This should be applied to the whole of the bus drivers, not just the PCI bus.

Quoted 5 lines

2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.

3. PMD's are linked as --whole-archive, and --as-needed.

I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.

Quoted 9 lines

4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).

This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.

In all this you seem to be assuming that the drivers are not picked up at runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.

Quoted 4 lines

Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?

/Bruce

I was using "make install-runtime" into a local chroot directory like a distribution package builder does.

There are multiple use cases:
1. Developer build DPDK and application together on one machine (and running on another). Or software appliance being built on one machine and run in many environments. Also cross builds are like this.

2. Distribution building a package (and installing in standard library locations /lib etc).

3. Demo machine where build is local and native.

DPDK seems to always focus on #3 which is least interesting for real production.

#21 lore raw
Quoted 69 lines · Stephen Hemminger
Quoted 61 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 8 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 5 lines

What I think should happen (and isn't is):

1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.

This should be applied to the whole of the bus drivers, not just the PCI bus.

Quoted 5 lines

2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.

3. PMD's are linked as --whole-archive, and --as-needed.

I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.

Quoted 9 lines

4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).

This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.

In all this you seem to be assuming that the drivers are not picked up at runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.

Quoted 4 lines

Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?

/Bruce

Quoted 9 lines

I was using "make install-runtime" into a local chroot directory like a distribution package builder does.

There are multiple use cases:
1. Developer build DPDK and application together on one machine (and running on another). Or software appliance being built on one machine and run in many environments. Also cross builds are like this.

Sure, but the expectation here is that the resultant build is installed in system locations, or a known location. As part of the build EAL_PMD_PATH needs to be set to the final driver location and then the driver loading issue requiring "-d" flag goes away.

Quoted 2 lines

2. Distribution building a package (and installing in standard library locations /lib etc).

No issue here, since package building is not going to be a problem and when installed the drivers are all picked up from the EAL_PMD_PATH.

Quoted 3 lines

3. Demo machine where build is local and native.

DPDK seems to always focus on #3 which is least interesting for real production.

You are missing our primary target use-case for the future -
* developer builds application using distro-supplied DPDK.

Individual developers building their own DPDK alongside their app should be a thing of the past. DPDK needs to be come like other libraries e.g. zlib, openssl, where app developers just pick it up from their linux distro and not feel the need to roll their own.

Yes, there will be cases where things are more memory constrained etc. and this model won't work, but despite this, it should be our primary target. To get there, though, there is still work to be done, e.g. in increasing ABI compatibility to allow easier adoption by distros, etc.

/Bruce

#22 lore raw
In reply to Bruce Richardson
Quoted 52 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:

22/07/2019 19:13, Stephen Hemminger:

Quoted 9 lines
Quoted 1 line · Thomas Monjalon

Are the constructors run on dlopen of the bus driver?

Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.

The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.

Quoted 8 lines

OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?

Quoted 5 lines

What I think should happen (and isn't is):

1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.

This should be applied to the whole of the bus drivers, not just the PCI bus.

Quoted 5 lines

2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.

3. PMD's are linked as --whole-archive, and --as-needed.

I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.

Quoted 9 lines

4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).

This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.

Quoted 8 lines

In all this you seem to be assuming that the drivers are not picked up at runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.

Looking at the plugin loading, the problem is it loads every PMD not just those that are going to be used. Isn't this a problem with a distribution model on an embedded system? Not everyone has virtual memory space to burn.