Skip to contentSkip to conversation

usertools: fix pmdinfo parsing

PATCH Superseded David Marchand 6 participants 15 messages Nov 3, 2020 – Nov 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 usertools: fix pmdinfo parsing

This script was using the librte_pmd prefix has a filter to follow DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
---
 usertools/dpdk-pmdinfo.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py +4 -1
index 1661982791..687a9fd032 100755
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if words and len(words) >= 3 and words[0] == 'librte' and \
+                   words[1] in ['baseband', 'compress', 'crypto', 'event',
+                                'net', 'raw', 'regex', 'vdpa']:
                     library = search_file(force_unicode(tag.needed),
                                           runpath + ":" + ldlibpath +
                                           ":/usr/lib64:/lib64:/usr/lib:/lib")
Signature · 1 line
2.23.0
Acked-by on v1 1/1
#1 lore raw

2020-11-03, David Marchand:

Quoted 17 lines

This script was using the librte_pmd prefix has a filter to follow DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
---
 usertools/dpdk-pmdinfo.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py +0 -0
index 1661982791..687a9fd032 100755

usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456

Quoted 8 lines
usertools/dpdk-pmdinfo.py +4 -1
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if words and len(words) >= 3 and words[0] == 'librte' and \
+                   words[1] in ['baseband', 'compress', 'crypto', 'event',
+                                'net', 'raw', 'regex', 'vdpa']:

This code is already ugly and I don't have much better to suggest...

#2 lore raw
Quoted 2 lines · Robin Jarry

2020-11-03, David Marchand:

Quoted 1 line

This script was using the librte_pmd prefix has a filter to follow

as*

Quoted 11 lines
Quoted 11 lines

DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
---
 usertools/dpdk-pmdinfo.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 15 lines

usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456

Quoted 13 lines
usertools/dpdk-pmdinfo.py +4 -1
index 1661982791..687a9fd032 100755
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if words and len(words) >= 3 and words[0] == 'librte' and \
+                   words[1] in ['baseband', 'compress', 'crypto', 'event',
+                                'net', 'raw', 'regex', 'vdpa']:

This code is already ugly and I don't have much better to suggest...

Less ugly with a regular expression?

if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):

Signature · 1 line
David Marchand
#3 lore raw
Quoted 33 lines · On Tue, 3 Nov 2020 21:20:43 +0100 David Marchand <[email protected]>…
Quoted 2 lines · Robin Jarry

2020-11-03, David Marchand:

Quoted 1 line

This script was using the librte_pmd prefix has a filter to follow

as*

Quoted 11 lines
Quoted 11 lines

DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
---
 usertools/dpdk-pmdinfo.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 15 lines

usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456

Quoted 13 lines
usertools/dpdk-pmdinfo.py +4 -1
index 1661982791..687a9fd032 100755
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if words and len(words) >= 3 and words[0] == 'librte' and \
+                   words[1] in ['baseband', 'compress', 'crypto', 'event',
+                                'net', 'raw', 'regex', 'vdpa']:  

This code is already ugly and I don't have much better to suggest...

Quoted 6 lines

Less ugly with a regular expression?

if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):

I prefer the list approach, much less error prone.
The code would read better if the list of drivers was in a variable instead of a long line of code and the string processing was done separate from the test.
Something like
drivers = ['baseband', 'compress'...
prefix = 'librte_pmd'
name = force_unicode(tag.needed)
if name.startswith(prefix):
suffix = name[len(prefix):]
if suffx in drivers:

#4 lore raw
Quoted 43 lines · Stephen Hemminger
Quoted 33 lines · On Tue, 3 Nov 2020 21:20:43 +0100 David Marchand <[email protected]>…
Quoted 2 lines · Robin Jarry

2020-11-03, David Marchand:

Quoted 1 line

This script was using the librte_pmd prefix has a filter to follow

as*

Quoted 11 lines
Quoted 11 lines

DT_NEEDED entries.
Now that we changed the driver names, update this heuristic with an explicit list of device classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
---
 usertools/dpdk-pmdinfo.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 15 lines

usertools/dpdk-pmdinfo.py def process_dt_needed_entries(self): · line 456

Quoted 13 lines
usertools/dpdk-pmdinfo.py +4 -1
index 1661982791..687a9fd032 100755
@@ -450,7 +450,10 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if words and len(words) >= 3 and words[0] == 'librte' and \
+                   words[1] in ['baseband', 'compress', 'crypto', 'event',
+                                'net', 'raw', 'regex', 'vdpa']:  

This code is already ugly and I don't have much better to suggest...

Quoted 6 lines

Less ugly with a regular expression?

if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):

Quoted 14 lines

I prefer the list approach, much less error prone.
The code would read better if the list of drivers was in a variable instead of a long line of code and the string processing was done separate from the test.
Something like
drivers = ['baseband', 'compress'...
prefix = 'librte_pmd'
name = force_unicode(tag.needed)
if name.startswith(prefix):
suffix = name[len(prefix):]
if suffx in drivers:

I agree with Stephen that the list approach and an intermediate variable is clearer than regex. However, I'd keep the split() as in the original patch.

#5 lore raw
In reply to David Marchand

2020-11-03, David Marchand:

Quoted 4 lines

Less ugly with a regular expression?

if re.match(r"^librte_(baseband|compress|crypto|event|net|raw|regex|vdpa)_",
force_unicode(tag.needed)):

No, that's worse :D

As Stephen said, maybe it would be more readable if the list of supported classes were in a constant somewhere.

DRIVER_CLASSES = (

      'baseband',
      'compress',
      'crypto',
      'event',
      'net',
      'raw,
      'regex',
      'vdpa',

)

Signature · 1 line
Robin
#6 lore raw

PATCH v2 1/1 usertools: fix pmdinfo parsing

In reply to David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,

---
 usertools/dpdk-pmdinfo.py | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py +19 -1
index 1661982791..55a55affde 100755
@@ -430,6 +430,20 @@ def get_dt_runpath(self, dynsec):
                 return force_unicode(tag.runpath)
         return ""
 
+    dpdk_driver_classes = (
+        'baseband',
+        'bus',
+        'common',
+        'compress',
+        'crypto',
+        'event',
+        'mempool',
+        'net',
+        'raw',
+        'regex',
+        'vdpa',
+    )
+
     def process_dt_needed_entries(self):
         """ Look to see if there are any DT_NEEDED entries in the binary
             And process those if there are
@@ -450,7 +464,11 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if len(words) < 3:
+                    continue
+                prefix, drv_class = words[:2]
+                if prefix == 'librte' and drv_class in self.dpdk_driver_classes:
                     library = search_file(force_unicode(tag.needed),
                                           runpath + ":" + ldlibpath +
                                           ":/usr/lib64:/lib64:/usr/lib:/lib")
Signature · 1 line
2.23.0
#7 lore raw
Quoted 8 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
Quoted 8 lines

---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,

Good idea, but probably not enough. I think adding a comment to the meson.build file to update this as well would be good.

/Bruce

#8 lore raw
Quoted 19 lines · Bruce Richardson
Quoted 8 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")
Quoted 8 lines

---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,

Quoted 2 lines

Good idea, but probably not enough. I think adding a comment to the meson.build file to update this as well would be good.

I am reconsidering all this...
The filtering stage in the pmdinfo script is unneeded. We need to find a PMD_INFO_STRING symbol in any case, which is good enough to find out this is a dpdk driver without relying on the library name.

Parsing all libraries only impact for the user is a debug message print("Scanning %s for pmd information" % library).
We might as well remove it or hide under a verbose option.

Signature · 1 line
David Marchand
Acked-by on v2 1/1
#9 lore raw
In reply to David Marchand

Hi,

Quoted 75 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Wednesday, November 04, 2020 17:41
To: [email protected]
Cc: Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; Neil Horman <[email protected]>; Xu, Rosen <[email protected]>; Andrew Rybchenko <[email protected]>; Luca Boccassi <[email protected]>
Subject: [PATCH v2] usertools: fix pmdinfo parsing

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing
meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,

---
 usertools/dpdk-pmdinfo.py | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
usertools/dpdk-pmdinfo.py index renamed from usertools/dpdk-pmdinfo.py +0 -0

1661982791..55a55affde 100755

usertools/dpdk-pmdinfo.py +14 -0
@@ -430,6 +430,20 @@ def get_dt_runpath(self, dynsec):
                 return force_unicode(tag.runpath)
         return ""
 
+    dpdk_driver_classes = (
+        'baseband',
+        'bus',
+        'common',
+        'compress',
+        'crypto',
+        'event',
+        'mempool',
+        'net',
+        'raw',
+        'regex',
+        'vdpa',
+    )
+
     def process_dt_needed_entries(self):
         """ Look to see if there are any DT_NEEDED entries in the binary
             And process those if there are @@ -450,7 +464,11 @@ def

process_dt_needed_entries(self):
for tag in dynsec.iter_tags():
# pyelftools may return byte-strings, force decode them if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':

usertools/dpdk-pmdinfo.py +5 -2
-                if 'librte_pmd' in force_unicode(tag.needed):
+                words = force_unicode(tag.needed).split('_')
+                if len(words) < 3:
+                    continue
+                prefix, drv_class = words[:2]
+                if prefix == 'librte' and drv_class in self.dpdk_driver_classes:
                     library = search_file(force_unicode(tag.needed),
                                           runpath + ":" + ldlibpath +
                                           ":/usr/lib64:/lib64:/usr/lib:/lib")
--
Quoted 1 line

2.23.0

author Acked-by Bruce Richardson on v2 1/1 Applied
#10 lore raw
In reply to David Marchand
Quoted 11 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter.
Now that we changed the driver library names, update this heuristic with an explicit list of all driver classes.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")

Applied, thanks.

Signature · 1 line
David Marchand
#11 lore raw

PATCH v3 1/1 usertools: fix pmdinfo parsing

In reply to David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,

Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the librte_ prefix to identify DPDK libraries: the script later checks for the PMD_INFO_STRING string in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v2:
- revisited the issue and simplified to only filter on librte_ prefix,

Changelog since v1:
- moved driver classes list as a class variable and did some cosmetic change for readibility,
- used dpdk_driver_classes variable name in the hope that someone changing meson will catch this script too,
- added bus, common, mempool and raw driver classes as some of them do carry some pmdinfo stuff and were skipped so far but I found no indication this skipping was intended,

---
 usertools/dpdk-pmdinfo.py | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
usertools/dpdk-pmdinfo.py +1 -4
index 1661982791..95fb0111d8 100755
@@ -434,7 +434,6 @@ def process_dt_needed_entries(self):
         """ Look to see if there are any DT_NEEDED entries in the binary
             And process those if there are
         """
-        global raw_output
         runpath = ""
         ldlibpath = os.environ.get('LD_LIBRARY_PATH')
         if ldlibpath is None:
@@ -450,13 +449,11 @@ def process_dt_needed_entries(self):
         for tag in dynsec.iter_tags():
             # pyelftools may return byte-strings, force decode them
             if force_unicode(tag.entry.d_tag) == 'DT_NEEDED':
-                if 'librte_pmd' in force_unicode(tag.needed):
+                if 'librte_' in force_unicode(tag.needed):
                     library = search_file(force_unicode(tag.needed),
                                           runpath + ":" + ldlibpath +
                                           ":/usr/lib64:/lib64:/usr/lib:/lib")
                     if library is not None:
-                        if raw_output is False:
-                            print("Scanning %s for pmd information" % library)
                         with io.open(library, 'rb') as file:
                             try:
                                 libelf = ReadElf(file, sys.stdout)
Signature · 1 line
2.23.0
#12 lore raw
Quoted 19 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,

Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the librte_ prefix to identify DPDK libraries: the script later checks for the PMD_INFO_STRING string in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.

Quoted 8 lines
Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v2:
- revisited the issue and simplified to only filter on librte_ prefix,

Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.

/Bruce

#13 lore raw
Quoted 30 lines · Bruce Richardson
Quoted 19 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,

Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the librte_ prefix to identify DPDK libraries: the script later checks for the PMD_INFO_STRING string in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.

Quoted 8 lines
Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v2:
- revisited the issue and simplified to only filter on librte_ prefix,

Quoted 3 lines

Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.

Ah ah, yes, thought I had put it.
I have values in the 0.40/0.50s range.

Signature · 1 line
David Marchand
#14 lore raw
Quoted 32 lines · David Marchand
Quoted 30 lines · Bruce Richardson
Quoted 27 lines · David Marchand

This script inspects an ELF file (binary or shared library) and its linked dependencies by following DT_NEEDED tags.
So far a simple librte_pmd prefix was used as a filter to only parse DPDK drivers dependencies.
While the reason is not clear from the commitlog of the patch that introduced this filter, it was probably added for performance reasons, since going through all dependencies can be quite long. Testing with a DPDK built before the driver name changes:
- running the script takes ~0.3s with the filter,
- running the script takes ~9s without the filter,

Now that we changed the driver library names, it becomes more difficult to identify only DPDK drivers, but we can just filter on the librte_ prefix to identify DPDK libraries: the script later checks for the PMD_INFO_STRING string in .rodata and it is enough to differentiate the DPDK drivers from the other DPDK libraries.
A debug message was logged for each inspected file, it gives no useful information and is removed.

Fixes a20b2c01a7a1 ("build: standardize component names and defines")

---
Changelog since v2:
- revisited the issue and simplified to only filter on librte_ prefix,

Quoted 7 lines
Quoted 3 lines

Given you provided some original perf numbers of 0.3 vs 9 seconds above, rather than leave us all in suspense :-), can you perhaps provide the numbers for this version too.

Ah ah, yes, thought I had put it.
I have values in the 0.40/0.50s range.

Sounds good.