pci: fix missing pci bus with shared library build
PATCH 1/1 pci: fix missing pci bus with shared library build
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.
--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mk/rte.app.mk +1 -1
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
Signature · 1 line
2.20.1
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0
mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
This only seems to happen on 18.11 and if configuration is reduced.
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0
mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
It still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parameters
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parameters
I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
sudo ./build/l2fwd -c F00000 -- -p 3
while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
It is broken with original link order.
With the patch it works fine.
Quoted 7 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
I don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
Application should never have to use -d to force driver load! The purpose of all the pmdinfo, and probing logic is to not do this.
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
There is something else going on.
This patch isn't fixing the problem consistently.
The suggestion of using -d is still wrong. We need to figure out how to get applications to run the same whether using shared library or statically linked.
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
19/07/2019 22:55, Stephen Hemminger:
Quoted 58 lines
Quoted 55 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
sudo ./build/l2fwd -c F00000 -- -p 3
Quoted 8 lines
Quoted 3 lines
while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.
Quoted 66 lines · Thomas Monjalon
19/07/2019 22:55, Stephen Hemminger:
Quoted 65 lines
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
Quoted 9 lines
Quoted 1 line
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.
This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to /usr/local or /usr, the EAL PMD path will be correctly configured and properly probe all drivers.
Quoted 77 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 66 lines · Thomas Monjalon
19/07/2019 22:55, Stephen Hemminger:
Quoted 65 lines
Quoted 53 lines · On Tue, 16 Jul 2019 09:46:04 +0100 Bruce Richardson…
Quoted 37 lines · Stephen Hemminger
Quoted 21 lines · On Mon, 15 Jul 2019 16:41:36 -0700 Stephen Hemminger…
If DPDK is built as a shared library, then any application linked with rte.app.mk will not find any PCI devices. When the application is started no ethernet devices are found.
This is because the link order of libraries on the command line matters. And PCI is before EAL. That causes there to be no dependency on PCI so linker ignores linking the library.
Swapping the order fixes this.Fixes c752998b5e2e ("pci: introduce library and driver")Signed-off-by Stephen Hemminger <[email protected]>--- mk/rte.app.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mk/rte.app.mk +0
-0mk/rte.app.mk _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack · line 97
Quoted 9 lines
mk/rte.app.mk +1
-1@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2 _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_schedIt still happens with 19.08. Testpmd works but only because it is linked with so many things. But l3fwd fails...
Quoted 7 lines
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0 EAL: Detected 8 lcore(s) EAL: Detected 1 NUMA nodes EAL: failed to parse device "02:00.0" EAL: Unable to parse device '02:00.0' EAL: Error - exiting with code: 1 Cause: Invalid EAL parametersI don't think the position of these is going to be the cause here, the more likely cause is that the pci bus driver - and all other drivers - are not linked into apps for shared library builds. You always need to pass "-d" parameter to load drivers at init time (or have them installed in the correct driver path). For example, for me with a shared library build the following gives a no ports error:
Quoted 5 lines
sudo ./build/l2fwd -c F00000 -- -p 3while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
Quoted 9 lines
Quoted 1 line
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.
Quoted 4 lines
This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to
/usr/localor /usr, the EAL PMD path will be correctly configured and properly probe all drivers.
The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.
One part of fixing this is:
mk/rte.app.mk +5 -0
@@ -130,6 +130,9 @@ ifeq ($(CONFIG_RTE_LIBRTE_FSLMC_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_COMMON_FSLMC) += -lrte_common_fslmc endif +# Bus devices register in constructor so always link +_LDLIBS-y += --no-as-needed + _LDLIBS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += -lrte_bus_pci _LDLIBS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += -lrte_bus_vdev _LDLIBS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += -lrte_bus_dpaa
@@ -137,6 +140,8 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += -lrte_bus_fslmc endif +_LDLIBS-y += --as-needed + ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n) # plugins (link only if static libraries)
22/07/2019 18:43, Stephen Hemminger:
Quoted 16 lines
Quoted 14 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 12 lines · Thomas Monjalon
19/07/2019 22:55, Stephen Hemminger:
Quoted 4 lines
The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.
Quoted 7 lines
Quoted 4 lines
This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to
/usr/localor /usr, the EAL PMD path will be correctly configured and properly probe all drivers.The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.
Are the constructors run on dlopen of the bus driver?
Quoted 22 lines · On Mon, 22 Jul 2019 19:04:55 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 18:43, Stephen Hemminger:
Quoted 21 lines
Quoted 18 lines · On Mon, 22 Jul 2019 10:06:11 +0100 Bruce Richardson…
Quoted 4 lines · Thomas Monjalon
19/07/2019 22:55, Stephen Hemminger:
Quoted 3 lines
The root cause is that recent gcc won't run constructor on unused libraries. Testing a patch to take --as-needed off of PCI library.
Quoted 8 lines
Quoted 1 line
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use. If we want an automatic dlopen, like modprobe, then we need more scripts. But I understand you are against the whole dlopen idea.This issue is more of a problem for development systems where we EAL path is not really usable for finding the drivers. For a properly deployed system where we use DPDK installed to
/usr/localor /usr, the EAL PMD path will be correctly configured and properly probe all drivers.
Quoted 7 lines
Quoted 2 lines
The problem is that bus drivers register themselves in constructors and these construtors are not run with as-needed.
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.
The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 6 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.
For distributions, this also matters. Linking with -ldpdk which is a linker script should work.
22/07/2019 20:34, Stephen Hemminger:
Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 6 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
Quoted 2 lines
I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.
The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.
Quoted 2 lines
For distributions, this also matters. Linking with -ldpdk which is a linker script should work.
There is no longer this linker script with meson.
Quoted 25 lines · On Tue, 23 Jul 2019 09:59:04 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 20:34, Stephen Hemminger:
Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 6 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?Quoted 2 lines
I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.
Quoted 9 lines
The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.
Quoted 2 lines
For distributions, this also matters. Linking with -ldpdk which is a linker script should work.
There is no longer this linker script with meson.
Ok, for usability that is a problem.
Requiring user to figure out which DPDK libraries to link with is a serious waste of time. It should be possible to just link with -ldpdk and distribution packages and just get the necessary libraries for the application (no extra rte_foo_bar .so loaded at run time), and the application should just work.
The idea that the user should link with 20 shared libraries, in the right order and pass -d flags to eal_init to load the right PMD is user hostile. It only makes sense if you want to invent yet another layer to manage the ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.
I know it is hard, and I know not all this will make it into 19.08
but let's try and do better. The DPDK already has a reputation as being like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.
23/07/2019 20:29, Stephen Hemminger:
Quoted 43 lines
Quoted 25 lines · On Tue, 23 Jul 2019 09:59:04 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 20:34, Stephen Hemminger:
Quoted 21 lines
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 6 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?Quoted 2 lines
I argue that examples should work the same with dynamic linking. This used to work before the break out of the bus model, so it is a bug.
Quoted 7 lines
The PCI support was part of EAL, yes, but the device drivers were plugins and already required the -d option.
Quoted 2 lines
For distributions, this also matters. Linking with -ldpdk which is a linker script should work.
There is no longer this linker script with meson.
Ok, for usability that is a problem.
Requiring user to figure out which DPDK libraries to link with is a serious waste of time. It should be possible to just link with -ldpdk and distribution packages and just get the necessary libraries for the application (no extrarte_foo_bar.so loaded at run time), and the application should just work.
Quoted 8 lines
The idea that the user should link with 20 shared libraries, in the right order and pass -d flags to
eal_initto load the right PMD is user hostile. It only makes sense if you want to invent yet another layer to manage the ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.I know it is hard, and I know not all this will make it into 19.08
but let's try and do better. The DPDK already has a reputation as being like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.
About the build-time link, it is already improved thanks to the pkgconfig file generated by meson.
The only issue I understand is the runtime need for -d, which is a feature, and could be removed.
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 8 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.
2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 8 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
Quoted 5 lines
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.
This should be applied to the whole of the bus drivers, not just the PCI bus.
Quoted 5 lines
2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.
Quoted 9 lines
4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.
In all this you seem to be assuming that the drivers are not picked up at runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.
Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?
/Bruce
Quoted 61 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 8 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?Quoted 5 lines
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.
This should be applied to the whole of the bus drivers, not just the PCI bus.
Quoted 5 lines
2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.
Quoted 9 lines
4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.
In all this you seem to be assuming that the drivers are not picked up at runtime from the
RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.
Quoted 4 lines
Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?
/Bruce
I was using "make install-runtime" into a local chroot directory like a distribution package builder does.
There are multiple use cases:
1. Developer build DPDK and application together on one machine (and running on another). Or software appliance being built on one machine and run in many environments. Also cross builds are like this.
2. Distribution building a package (and installing in standard library locations /lib etc).
3. Demo machine where build is local and native.
DPDK seems to always focus on #3 which is least interesting for real production.
Quoted 69 lines · Stephen Hemminger
Quoted 61 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 8 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?Quoted 5 lines
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.
This should be applied to the whole of the bus drivers, not just the PCI bus.
Quoted 5 lines
2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.
Quoted 9 lines
4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.
In all this you seem to be assuming that the drivers are not picked up at runtime from the
RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.Quoted 4 lines
Did you try installing DPDK using "ninja install" or "make install" before running any apps using it?
/Bruce
Quoted 9 lines
I was using "make install-runtime" into a local chroot directory like a distribution package builder does.
There are multiple use cases:
1. Developer build DPDK and application together on one machine (and running on another). Or software appliance being built on one machine and run in many environments. Also cross builds are like this.
Sure, but the expectation here is that the resultant build is installed in system locations, or a known location. As part of the build EAL_PMD_PATH needs to be set to the final driver location and then the driver loading issue requiring "-d" flag goes away.
Quoted 2 lines
2. Distribution building a package (and installing in standard library locations /lib etc).
No issue here, since package building is not going to be a problem and when installed the drivers are all picked up from the EAL_PMD_PATH.
Quoted 3 lines
3. Demo machine where build is local and native.
DPDK seems to always focus on #3 which is least interesting for real production.
You are missing our primary target use-case for the future -
* developer builds application using distro-supplied DPDK.
Individual developers building their own DPDK alongside their app should be a thing of the past. DPDK needs to be come like other libraries e.g. zlib, openssl, where app developers just pick it up from their linux distro and not feel the need to roll their own.
Yes, there will be cases where things are more memory constrained etc. and this model won't work, but despite this, it should be our primary target. To get there, though, there is still work to be done, e.g. in increasing ABI compatibility to allow easier adoption by distros, etc.
/Bruce
Quoted 52 lines · On Tue, 23 Jul 2019 13:30:33 +0100 Bruce Richardson…
Quoted 23 lines · Stephen Hemminger
Quoted 11 lines · On Mon, 22 Jul 2019 19:31:08 +0200 Thomas Monjalon <[email protected]> wrote:
22/07/2019 19:13, Stephen Hemminger:
Quoted 9 lines
Quoted 1 line · Thomas Monjalon
Are the constructors run on dlopen of the bus driver?
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.The core principle is that dynamic build of DPDK should act the same as old statically linked DPDK. Otherwise, the user experience is even worse, and all the example documentation is wrong.
Quoted 8 lines
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days. So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?Quoted 5 lines
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed. This causes constructor to be called and register the bus.
This should be applied to the whole of the bus drivers, not just the PCI bus.
Quoted 5 lines
2. As part of the build process all the PCI drivers pmdinfo would get constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
I'm not sure I agree with this change to always link in all the PMDs. It prevents an app from being used with just a subset of the drivers needed.
Quoted 9 lines
4. New code in PCI probe which looks for existing entries (static or -d) for devices. If device is still not found it refers to the table of PMD's (from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without having to know the PMD that is present. It would also solve the problem that currently if applications is linked with -ldpdk linker script then all PMD's get pulled into the application address space.
Quoted 8 lines
In all this you seem to be assuming that the drivers are not picked up at runtime from the
RTE_EAL_PMD_PATH. In real world cases where a user is building an app, and not developing DPDK itself, the DPDK libraries should be installed in /usr(/local)/lib64 and the drivers in .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD drivers are all loaded at runtime for each app, without having any dependency on having a specific one be present, allowing a user to remove any drivers unnecessary for the current hardware.
Looking at the plugin loading, the problem is it loads every PMD not just those that are going to be used. Isn't this a problem with a distribution model on an embedded system? Not everyone has virtual memory space to burn.