usertools: fix pmdinfo parsing
PATCH 1/1 usertools: fix pmdinfo parsing
This script was using the librte_pmd prefix has a filter to follow DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.
--- usertools/dpdk-pmdinfo.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py +4 -1
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if words and len(words) >= 3 and words[0] == 'librte' and \ + words[1] in ['baseband', 'compress', 'crypto', 'event', + 'net', 'raw', 'regex', 'vdpa']: library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib")
Signature · 1 line
2.23.0
2020-11-03, David Marchand:
Quoted 17 lines
This script was using the
librte_pmdprefix has a filter to followDT_NEEDEDentries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>--- usertools/dpdk-pmdinfo.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)usertools/dpdk-pmdinfo.py +0
-0
usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456
Quoted 8 lines
usertools/dpdk-pmdinfo.py +4
-1@@ -450,7 +450,10 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if words and len(words) >= 3 and words[0] == 'librte' and \ + words[1] in ['baseband', 'compress', 'crypto', 'event', + 'net', 'raw', 'regex', 'vdpa']:
This code is already ugly and I don't have much better to suggest...
Quoted 2 lines · Robin Jarry
2020-11-03, David Marchand:
Quoted 1 line
This script was using the
librte_pmdprefix has a filter to follow
as*
Quoted 11 lines
Quoted 11 lines
DT_NEEDEDentries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>--- usertools/dpdk-pmdinfo.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 15 lines
usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456
Quoted 13 lines
usertools/dpdk-pmdinfo.py +4
-1@@ -450,7 +450,10 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if words and len(words) >= 3 and words[0] == 'librte' and \ + words[1] in ['baseband', 'compress', 'crypto', 'event', + 'net', 'raw', 'regex', 'vdpa']:This code is already ugly and I don't have much better to suggest...
Less ugly with a regular expression?
if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):
Signature · 1 line
David Marchand
Quoted 33 lines · On Tue, 3 Nov 2020 21:20:43 +0100 David Marchand <[email protected]>…
Quoted 2 lines · Robin Jarry
2020-11-03, David Marchand:
Quoted 1 line
This script was using the
librte_pmdprefix has a filter to followas*
Quoted 11 lines
Quoted 11 lines
DT_NEEDEDentries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>--- usertools/dpdk-pmdinfo.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Quoted 15 lines
usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456
Quoted 13 lines
usertools/dpdk-pmdinfo.py +4
-1@@ -450,7 +450,10 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if words and len(words) >= 3 and words[0] == 'librte' and \ + words[1] in ['baseband', 'compress', 'crypto', 'event', + 'net', 'raw', 'regex', 'vdpa']:This code is already ugly and I don't have much better to suggest...
Quoted 6 lines
Less ugly with a regular expression?
if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):
I prefer the list approach, much less error prone.
The code would read better if the list of drivers was in a variable instead of a long line of code and the string processing was done separate from the test.
Something like
drivers = ['baseband', 'compress'...
prefix = 'librte_pmd'
name = force_unicode(tag.needed)
if name.startswith(prefix):
suffix = name[len(prefix):]
if suffx in drivers:
Quoted 43 lines · Stephen Hemminger
Quoted 33 lines · On Tue, 3 Nov 2020 21:20:43 +0100 David Marchand <[email protected]>…
Quoted 2 lines · Robin Jarry
2020-11-03, David Marchand:
Quoted 1 line
This script was using the
librte_pmdprefix has a filter to followas*
Quoted 11 lines
Quoted 11 lines
DT_NEEDEDentries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>--- usertools/dpdk-pmdinfo.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)Quoted 15 lines
usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456
Quoted 13 lines
usertools/dpdk-pmdinfo.py +4
-1@@ -450,7 +450,10 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if words and len(words) >= 3 and words[0] == 'librte' and \ + words[1] in ['baseband', 'compress', 'crypto', 'event', + 'net', 'raw', 'regex', 'vdpa']:This code is already ugly and I don't have much better to suggest...
Quoted 6 lines
Less ugly with a regular expression?
if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):
Quoted 14 lines
I prefer the list approach, much less error prone.
The code would read better if the list of drivers was in a variable instead of a long line of code and the string processing was done separate from the test.
Something like
drivers = ['baseband', 'compress'...
prefix = 'librte_pmd'
name =force_unicode(tag.needed)
ifname.startswith(prefix):
suffix = name[len(prefix):]
if suffx in drivers:
I agree with Stephen that the list approach and an intermediate variable is clearer than regex. However, I'd keep the split() as in the original patch.
2020-11-03, David Marchand:
Quoted 4 lines
Less ugly with a regular expression?
if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):
No, that's worse :D
As Stephen said, maybe it would be more readable if the list of supported classes were in a constant somewhere.
DRIVER_CLASSES = (
'baseband',
'compress',
'crypto',
'event',
'net',
'raw,
'regex',
'vdpa',)
Signature · 1 line
Robin
PATCH v2 1/1 usertools: fix pmdinfo parsing
In reply to David MarchandThis script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.
---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,
--- usertools/dpdk-pmdinfo.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py +19 -1
@@ -430,6 +430,20 @@ def get_dt_runpath(self, dynsec): return force_unicode(tag.runpath) return "" + dpdk_driver_classes = ( + 'baseband', + 'bus', + 'common', + 'compress', + 'crypto', + 'event', + 'mempool', + 'net', + 'raw', + 'regex', + 'vdpa', + ) + def process_dt_needed_entries(self): """ Look to see if there are any DT_NEEDED entries in the binary And process those if there are
@@ -450,7 +464,11 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if len(words) < 3: + continue + prefix, drv_class = words[:2] + if prefix == 'librte' and drv_class in self.dpdk_driver_classes: library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib")
Signature · 1 line
2.23.0
Quoted 8 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")
Quoted 8 lines
Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- useddpdk_driver_classesvariable name in the hope that someone changing meson will catch this script too,
Good idea, but probably not enough. I think adding a comment to the meson.build file to update this as well would be good.
/Bruce
Quoted 19 lines · Bruce Richardson
Quoted 8 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Quoted 8 lines
Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- useddpdk_driver_classesvariable name in the hope that someone changing meson will catch this script too,
Quoted 2 lines
Good idea, but probably not enough. I think adding a comment to the meson.build file to update this as well would be good.
I am reconsidering all this...
The filtering stage in the pmdinfo script is unneeded. We need to find a PMD_INFO_STRING symbol in any case, which is good enough to find out this is a dpdk driver without relying on the library name.
Parsing all libraries only impact for the user is a debug message print("Scanning %s for pmd information" % library).
We might as well remove it or hide under a verbose option.
Signature · 1 line
David Marchand
Hi,
Quoted 75 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Wednesday, November 04, 2020 17:41
To: [email protected]
Cc: Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; Neil Horman <[email protected]>; Xu, Rosen <[email protected]>; Andrew Rybchenko <[email protected]>; Luca Boccassi <[email protected]>
Subject: [PATCH v2] usertools: fix pmdinfo parsingThis script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- useddpdk_driver_classesvariable name in the hope that someone changing
meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,--- usertools/dpdk-pmdinfo.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)usertools/dpdk-pmdinfo.py index renamed from usertools/dpdk-pmdinfo.py +0
-01661982791..55a55affde 100755
usertools/dpdk-pmdinfo.py +14
-0@@ -430,6 +430,20 @@ def get_dt_runpath(self, dynsec): return force_unicode(tag.runpath) return "" + dpdk_driver_classes = ( + 'baseband', + 'bus', + 'common', + 'compress', + 'crypto', + 'event', + 'mempool', + 'net', + 'raw', + 'regex', + 'vdpa', + ) + def process_dt_needed_entries(self): """ Look to see if there are any DT_NEEDED entries in the binary And process those if there are @@ -450,7 +464,11 @@ def
process_dt_needed_entries(self):
for tag indynsec.iter_tags():
# pyelftools may return byte-strings, force decode them ifforce_unicode(tag.entry.d_tag)== 'DT_NEEDED':usertools/dpdk-pmdinfo.py +5
-2- if 'librte_pmd' in force_unicode(tag.needed): + words = force_unicode(tag.needed).split('_') + if len(words) < 3: + continue + prefix, drv_class = words[:2] + if prefix == 'librte' and drv_class in self.dpdk_driver_classes: library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib") --
Quoted 1 line
2.23.0
Quoted 11 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>
Applied, thanks.
Signature · 1 line
David Marchand
PATCH v3 1/1 usertools: fix pmdinfo parsing
In reply to David MarchandThis script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,
Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the librte_ prefix to identify DPDK libraries: the script later checks for the PMD_INFO_STRING string in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.
---
Changelog since v2:
- revisited the issue and simplified to only filter on librte_ prefix,
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,
--- usertools/dpdk-pmdinfo.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
usertools/dpdk-pmdinfo.py +1 -4
@@ -434,7 +434,6 @@ def process_dt_needed_entries(self): """ Look to see if there are any DT_NEEDED entries in the binary And process those if there are """ - global raw_output runpath = "" ldlibpath = os.environ.get('LD_LIBRARY_PATH') if ldlibpath is None:
@@ -450,13 +449,11 @@ def process_dt_needed_entries(self): for tag in dynsec.iter_tags(): # pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED': - if 'librte_pmd' in force_unicode(tag.needed): + if 'librte_' in force_unicode(tag.needed): library = search_file(force_unicode(tag.needed), runpath + ":" + ldlibpath + ":/usr/lib64:/lib64:/usr/lib:/lib") if library is not None: - if raw_output is False: - print("Scanning %s for pmd information" % library) with io.open(library, 'rb') as file: try: libelf = ReadElf(file, sys.stdout)
Signature · 1 line
2.23.0
Quoted 19 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the
librte_prefix to identify DPDK libraries: the script later checks for thePMD_INFO_STRINGstring in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.
Quoted 8 lines
Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v2:
- revisited the issue and simplified to only filter onlibrte_prefix,
Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.
/Bruce
Quoted 30 lines · Bruce Richardson
Quoted 19 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the
librte_prefix to identify DPDK libraries: the script later checks for thePMD_INFO_STRINGstring in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.Quoted 8 lines
Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v2:
- revisited the issue and simplified to only filter onlibrte_prefix,
Quoted 3 lines
Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.
Ah ah, yes, thought I had put it.
I have values in the 0.40/0.50s range.
Signature · 1 line
David Marchand
Bruce Richardson
Acked-byQuoted 32 lines · David Marchand
Quoted 30 lines · Bruce Richardson
Quoted 27 lines · David Marchand
This script inspects an ELF file (binary or shared library) and its linked dependencies by following
DT_NEEDEDtags.
So far a simplelibrte_pmdprefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the
librte_prefix to identify DPDK libraries: the script later checks for thePMD_INFO_STRINGstring in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.Fixes a20b2c01a7a1 ("build: standardize component names and defines")Signed-off-by David Marchand <[email protected]>Acked-by Robin Jarry <[email protected]>---
Changelog since v2:
- revisited the issue and simplified to only filter onlibrte_prefix,
Quoted 7 lines
Quoted 3 lines
Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.
Ah ah, yes, thought I had put it.
I have values in the 0.40/0.50s range.
Sounds good.