Skip to contentSkip to conversation

eal: don't use default library path for static binaries

PATCH Applied Bruce Richardson 4 participants 8 messages Jun 26, 2020 – Jul 5, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eal: don't use default library path for static binaries

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

---
 lib/librte_eal/common/eal_common_options.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_eal/common/eal_common_options.c +9 -2
index 0901b493c..1d958d11e 100644
@@ -447,8 +447,15 @@ eal_plugins_init(void)
 	struct shared_driver *solib = NULL;
 	struct stat sb;
 
-	if (*default_solib_dir != '\0' && stat(default_solib_dir, &sb) == 0 &&
-				S_ISDIR(sb.st_mode))
+	/* if we are not statically linked, add default driver loading
+	 * path if it exists as a directory.
+	 * (Using dlopen with NOLOAD flag on eal, will return NULL if the eal
+	 * shared library is not already loaded i.e. it's statically linked.)
+	 */
+	if (dlopen("librte_eal.so", RTLD_LAZY | RTLD_NOLOAD) != NULL &&
+			*default_solib_dir != '\0' &&
+			stat(default_solib_dir, &sb) == 0 &&
+			S_ISDIR(sb.st_mode))
 		eal_plugin_add(default_solib_dir);
 
 	TAILQ_FOREACH(solib, &solib_list, next) {
Signature · 1 line
2.25.1
#1 lore raw
Quoted 22 lines · On Fri, 26 Jun 2020 15:59:57 +0100 Bruce Richardson…

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

Quoted 7 lines

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

#2 lore raw
Quoted 31 lines · Stephen Hemminger
Quoted 28 lines · On Fri, 26 Jun 2020 15:59:57 +0100 Bruce Richardson…

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

Quoted 5 lines
Quoted 1 line

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

They use the -d flag as now. The only change here is that we don't *automatically* (and silently) attempt to load all drivers from a system location when you have a static binary.

#3 lore raw
Quoted 32 lines · Bruce Richardson
Quoted 31 lines · Stephen Hemminger
Quoted 28 lines · On Fri, 26 Jun 2020 15:59:57 +0100 Bruce Richardson…

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

They use the -d flag as now. The only change here is that we don't *automatically* (and silently) attempt to load all drivers from a system location when you have a static binary.

I'd also make a couple of additional points:
1. If you have a static app and you have extra drivers in your EAL_PMD_PATH directory you have no way of preventing the loading of them.

2. Since all DPDK apps try to load all files in that directory, all one has to do is put a non-loadable file into the DPDK PMD directory and suddenly all DPDK apps on the system will fail to run. [Patchset
http://patches.dpdk.org/project/dpdk/list/?series=10553 will also help here]

3. Since this is a behaviour change, perhaps it needs to be deferred to 20.11? Ideally I think we should fix this now, because I think the current behaviour doesn't make sense and causes more problems than it solves, but if it needs to be deferred, so be it.

/Bruce

#4 lore raw
Quoted 52 lines · On Mon, 29 Jun 2020 10:19:10 +0100 Bruce Richardson…
Quoted 32 lines · Bruce Richardson
Quoted 31 lines · Stephen Hemminger
Quoted 28 lines · On Fri, 26 Jun 2020 15:59:57 +0100 Bruce Richardson…

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

They use the -d flag as now. The only change here is that we don't *automatically* (and silently) attempt to load all drivers from a system location when you have a static binary.

I'd also make a couple of additional points:
1. If you have a static app and you have extra drivers in your EAL_PMD_PATH directory you have no way of preventing the loading of them.

2. Since all DPDK apps try to load all files in that directory, all one has to do is put a non-loadable file into the DPDK PMD directory and suddenly all DPDK apps on the system will fail to run. [Patchset
http://patches.dpdk.org/project/dpdk/list/?series=10553 will also help here]

Quoted 4 lines

3. Since this is a behaviour change, perhaps it needs to be deferred to 20.11? Ideally I think we should fix this now, because I think the current behaviour doesn't make sense and causes more problems than it solves, but if it needs to be deferred, so be it.

Thanks, just trying to poke the corners of this change. Please make sure the documentation and web site match the behavior.

#5 lore raw
Quoted 61 lines · Stephen Hemminger
Quoted 52 lines · On Mon, 29 Jun 2020 10:19:10 +0100 Bruce Richardson…
Quoted 32 lines · Bruce Richardson
Quoted 31 lines · Stephen Hemminger
Quoted 28 lines · On Fri, 26 Jun 2020 15:59:57 +0100 Bruce Richardson…

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

They use the -d flag as now. The only change here is that we don't *automatically* (and silently) attempt to load all drivers from a system location when you have a static binary.

I'd also make a couple of additional points:
1. If you have a static app and you have extra drivers in your EAL_PMD_PATH directory you have no way of preventing the loading of them.

2. Since all DPDK apps try to load all files in that directory, all one has to do is put a non-loadable file into the DPDK PMD directory and suddenly all DPDK apps on the system will fail to run. [Patchset
http://patches.dpdk.org/project/dpdk/list/?series=10553 will also help here]

Quoted 4 lines

3. Since this is a behaviour change, perhaps it needs to be deferred to 20.11? Ideally I think we should fix this now, because I think the current behaviour doesn't make sense and causes more problems than it solves, but if it needs to be deferred, so be it.

Quoted 4 lines

Thanks, just trying to poke the corners of this change. Please make sure the documentation and web site match the behavior.

This appears to be a gap, since I looked through the docs and couldn't find any reference to this behaviour at all. However, I may well have missed something so please flag if so.

Regards,
/Bruce

#6 lore raw

29/06/2020 18:15, Bruce Richardson:

Quoted 59 lines · Stephen Hemminger
Quoted 58 lines
Quoted 55 lines · Bruce Richardson
Quoted 31 lines · Bruce Richardson
Quoted 30 lines · Stephen Hemminger
Quoted 28 lines · Bruce Richardson

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

But what if the majority of the DPDK is statically linked but the application wants also load a dynamically linked driver?

They use the -d flag as now. The only change here is that we don't *automatically* (and silently) attempt to load all drivers from a system location when you have a static binary.

I'd also make a couple of additional points:
1. If you have a static app and you have extra drivers in your EAL_PMD_PATH directory you have no way of preventing the loading of them.

2. Since all DPDK apps try to load all files in that directory, all one has to do is put a non-loadable file into the DPDK PMD directory and suddenly all DPDK apps on the system will fail to run. [Patchset
http://patches.dpdk.org/project/dpdk/list/?series=10553 will also help here]

3. Since this is a behaviour change, perhaps it needs to be deferred to 20.11? Ideally I think we should fix this now, because I think the current behaviour doesn't make sense and causes more problems than it solves, but if it needs to be deferred, so be it.

Quoted 8 lines
Quoted 4 lines

Thanks, just trying to poke the corners of this change. Please make sure the documentation and web site match the behavior.

This appears to be a gap, since I looked through the docs and couldn't find any reference to this behaviour at all. However, I may well have missed something so please flag if so.

Yes we are lacking some doc around the basics of EAL (and also ethdev). This is a behaviour change, but I prefer applying it in 20.08, so we can get some user feedback before the LTS release.

Applied, thanks

Acked-by on v1 1/1
#7 lore raw
In reply to Bruce Richardson
Quoted 22 lines · Bruce Richardson

When using statically linked DPDK binaries, the EAL checks the default PMD path and tries to load any drivers there, despite the fact that all drivers are normally linked into the binary. This behaviour can cause issues if the PMD path and lib dir is configured to a non-standard location which is not in the ld.so.conf paths, e.g. a build with prefix set to a home directory location. In a case such as this, EAL will try and (unnecessarily) load the .so driver files but that load will fail as their dependent libraries, such as ethdev, for example, will not be found.

Because of this, it is better if statically linked DPDK apps do not load drivers from the standard paths automatically. The user can always have this behaviour by explicitly specifying the path using -d flag, if so desired.

Not loading the libraries automatically can also prevent potential issues with a user building and running a statically-linked DPDK binary based off a private copy of DPDK, while there exists on the same machine a system-wide installation of DPDK in the default locations. Without this change, the system-installed drivers will be loaded to the binary alongside the statically-linked drivers, which is not what the user would have intended.

Quoted 8 lines

To detect whether we are in a statically or dynamically linked binary, we can have EAL try to get a dlopen handle to its own shared library, by calling dlopen with the RTLD_NOLOAD flag. This will return NULL if there is no such shared lib loaded i.e. the code is executing from a static library, or a handle to the lib if it is loaded.

---

+1, been bitten by this more than... ummm... well, a lot.

Signature · 2 lines
Thanks,
Anatoly