usertools: rewrite pmdinfo
PATCH 1/1 usertools: rewrite pmdinfo
dpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "dmadev_idxd_pci")'
{
"name": "dmadev_idxd_pci",
"params": "max_queues=0",
"kmod": "vfio-pci",
"devices": [
{
"vendor_id": "8086",
"device_id": "0b25",
"subsystem_device_id": "ffff",
"subsystem_system_id": "ffff"
} ]
}
Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "net_i40e").kmod'
"* igb_uio | uio_pci_generic | vfio-pci"
Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.devices[] | .vendor_id == "15b3" and .device_id == "1013").kmod'
"* ib_uverbs & mlx5_core & mlx5_ib"
Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}'
...
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
---
There were multiple compatibility issues with this script in the past years. Also, the style and complexity may be unsettling for python developers. After this patch, maintenance should be much easier.
doc/guides/rel_notes/release_22_11.rst | 5 + usertools/dpdk-pmdinfo.py | 856 ++++++++----------------- 2 files changed, 261 insertions(+), 600 deletions(-)
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -84,6 +84,11 @@ API Changes Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' ABI Changes -----------
usertools/dpdk-pmdinfo.py +256 -600
882 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 68 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "dmadev_idxd_pci")' { "name": "dmadev_idxd_pci", "params": "max_queues=0", "kmod": "vfio-pci", "devices": [ { "vendor_id": "8086", "device_id": "0b25", "subsystem_device_id": "ffff", "subsystem_system_id": "ffff" } ] }Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "net_i40e").kmod' "* igb_uio | uio_pci_generic | vfio-pci"Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.devices[] | .vendor_id == "15b3" and .device_id == "1013").kmod' "* ib_uverbs & mlx5_core & mlx5_ib"Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}' ...The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26.
Quoted 7 lines
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz<[email protected]>Signed-off-by Robin Jarry<[email protected]>
Hi Robin,
Thanks for the work.
One of the major usecase of the script is to get information from binary drivers. So intentions of the script is to run it on drivers more than applications (dpdk-testpmd).
When I run it on one of the .so drivers, it is generating some warnings [1], is this expected?
[1]
$ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_net_ixgbe.so
warning: cannot find librte_ethdev.so.23
warning: cannot find librte_eal.so.23
warning: cannot find librte_kvargs.so.23
warning: cannot find librte_telemetry.so.23
warning: cannot find librte_net.so.23
warning: cannot find librte_mbuf.so.23
warning: cannot find librte_mempool.so.23
warning: cannot find librte_ring.so.23
warning: cannot find librte_meter.so.23
warning: cannot find librte_bus_pci.so.23
warning: cannot find librte_pci.so.23
warning: cannot find librte_bus_vdev.so.23
warning: cannot find librte_hash.so.23
warning: cannot find librte_rcu.so.23
warning: cannot find librte_security.so.23
warning: cannot find librte_cryptodev.so.23 [
{
"name": "net_ixgbe_vf",
"params": "pflink_fullchk=<0|1>",
"kmod": "* igb_uio | vfio-pci",
"devices": [...
...
Ferruh Yigit, Sep 13, 2022 at 13:29:
Quoted 32 lines
Hi Robin,
Thanks for the work.
One of the major usecase of the script is to get information from binary drivers. So intentions of the script is to run it on drivers more than applications (dpdk-testpmd).
When I run it on one of the .so drivers, it is generating some warnings [1], is this expected?
[1]
$ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_net_ixgbe.sowarning: cannot find librte_ethdev.so.23
warning: cannot find librte_eal.so.23
warning: cannot find librte_kvargs.so.23
warning: cannot find librte_telemetry.so.23
warning: cannot find librte_net.so.23
warning: cannot find librte_mbuf.so.23
warning: cannot find librte_mempool.so.23
warning: cannot find librte_ring.so.23
warning: cannot find librte_meter.so.23
warning: cannot find librte_bus_pci.so.23
warning: cannot find librte_pci.so.23
warning: cannot find librte_bus_vdev.so.23
warning: cannot find librte_hash.so.23
warning: cannot find librte_rcu.so.23
warning: cannot find librte_security.so.23
warning: cannot find librte_cryptodev.so.23
[
Quoted 7 lines
{ "name": "net_ixgbe_vf", "params": "pflink_fullchk=<0|1>", "kmod": "* igb_uio | vfio-pci", "devices": [...
...
Hi Ferruh,
yes it tries to parse all required (DT_NEEDED) dynamic libraries as did the previous version of the script. The warnings are displayed when a needed lib is not found.
You can fix that by exporting LD_LIBRARY_PATH:
$ LD_LIBRARY_PATH=build/lib/:build/drivers/ usertools/dpdk-pmdinfo.py build/drivers/librte_net_ixgbe.so | head
[
{
"name": "net_ixgbe_vf",
"params": "pflink_fullchk=<0|1>",
"kmod": "* igb_uio | vfio-pci",
"devices": [
{
"vendor_id": "8086",
"device_id": "10ed",
"subsystem_device_id": "ffff",
...
...If the libraries are installed in a standard path, it should not be necessary to export LD_LIBRARY_PATH.
Quoted 66 lines · Robin Jarry
CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
Ferruh Yigit, Sep 13, 2022 at 13:29:
Quoted 32 lines
Hi Robin,
Thanks for the work.
One of the major usecase of the script is to get information from binary drivers. So intentions of the script is to run it on drivers more than applications (dpdk-testpmd).
When I run it on one of the .so drivers, it is generating some warnings [1], is this expected?
[1]
$ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_net_ixgbe.sowarning: cannot find librte_ethdev.so.23
warning: cannot find librte_eal.so.23
warning: cannot find librte_kvargs.so.23
warning: cannot find librte_telemetry.so.23
warning: cannot find librte_net.so.23
warning: cannot find librte_mbuf.so.23
warning: cannot find librte_mempool.so.23
warning: cannot find librte_ring.so.23
warning: cannot find librte_meter.so.23
warning: cannot find librte_bus_pci.so.23
warning: cannot find librte_pci.so.23
warning: cannot find librte_bus_vdev.so.23
warning: cannot find librte_hash.so.23
warning: cannot find librte_rcu.so.23
warning: cannot find librte_security.so.23
warning: cannot find librte_cryptodev.so.23
[Quoted 7 lines
{ "name": "net_ixgbe_vf", "params": "pflink_fullchk=<0|1>", "kmod": "* igb_uio | vfio-pci", "devices": [...
...Hi Ferruh,
yes it tries to parse all required (
DT_NEEDED) dynamic libraries as did the previous version of the script. The warnings are displayed when a needed lib is not found.You can fix that by exporting
LD_LIBRARY_PATH:$ LD_LIBRARY_PATH=build/lib/:build/drivers/ usertools/dpdk-pmdinfo.py build/drivers/librte_net_ixgbe.so | head [ { "name": "net_ixgbe_vf", "params": "pflink_fullchk=<0|1>", "kmod": "* igb_uio | vfio-pci", "devices": [ { "vendor_id": "8086", "device_id": "10ed", "subsystem_device_id": "ffff", ... ...
Quoted 2 lines
If the libraries are installed in a standard path, it should not be necessary to export
LD_LIBRARY_PATH.
I confirm warnings are gone when LD_LIBRARY_PATH is provided, but why current version doesn't require LD_LIBRARY_PATH?
Also can it be possible to parse 'DT_NEEDED' for applications, but ignore it for dynamic libraries?
Ferruh Yigit, Sep 13, 2022 at 15:50:
Quoted 2 lines
I confirm warnings are gone when
LD_LIBRARY_PATHis provided, but why current version doesn't requireLD_LIBRARY_PATH?
It does but I assume no warning is displayed when the required libs are not found. I could silence the warnings unless a -v/--verbose flag is specified.
Quoted 2 lines
Also can it be possible to parse '
DT_NEEDED' for applications, but ignore it for dynamic libraries?
Yes it is possible. I'll include that in v2.
Quoted 9 lines · Robin Jarry
Ferruh Yigit, Sep 13, 2022 at 15:50:
Quoted 2 lines
I confirm warnings are gone when
LD_LIBRARY_PATHis provided, but why current version doesn't requireLD_LIBRARY_PATH?It does but I assume no warning is displayed when the required libs are not found. I could silence the warnings unless a -v/--verbose flag is specified.
+1 to silence the warnings by default.
Quoted 4 lines
Quoted 2 lines
Also can it be possible to parse '
DT_NEEDED' for applications, but ignore it for dynamic libraries?Yes it is possible. I'll include that in v2.
Cool, thanks.
Quoted 9 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Thanks for the rework. Comment inline below.
/Bruce
<snip>
usertools/dpdk-pmdinfo.py line 172
Quoted 6 lines
usertools/dpdk-pmdinfo.py +6
-0+ :raises FileNotFoundError: + """ + folders = [] + if "LD_LIBRARY_PATH" in os.environ: + folders += os.environ["LD_LIBRARY_PATH"].split(":") + folders += ["/usr/lib64", "/lib64", "/usr/lib", "/lib"]
This is a standard set of folders for Redhat and similar based distributions, but not for Ubuntu and Debian ones - which use e.g. "/lib/x86_64-linux-gnu". It's also missing path options from /usr/local, which is a likely location for a compiled and installed DPDK.
Ideally, the script would parse the entries from /etc/ld.so.conf, or /etc/ld.so.conf.d/* files to get a full list of directories to search on the system. Frankly, though, that seems like too much work. :-) Two ideas I'd suggest, though both involve shelling out to other commands:
* Call "ldd" to get the paths to the relevant libraries for the current object, rather than parsing the DT_NEEDED entries from the elf file in the script
* Use "ldconfig -p" to get the list of libs and paths for the script and use that for matching the DT_NEEDED entries.
PATCH v2 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "dmadev_idxd_pci")'
{
"name": "dmadev_idxd_pci",
"params": "max_queues=0",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "8086",
"device": "0b25",
"subsystem_vendor": "ffff",
"subsystem_device": "ffff"
} ]
}
Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "net_i40e").kmod'
"* igb_uio | uio_pci_generic | vfio-pci"
Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod'
"* ib_uverbs & mlx5_core & mlx5_ib"
Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}'
...
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
---
v1 -> v2:
* update release notes and minimal python version requirement
* hide warnings by default (-v/--verbose to show them)
* show debug messages with -vv
* also search libs in folders listed in /etc/ld.so.conf/*.conf
* only search for DT_NEEDED on executables, not on dynamic libraries
* take DT_RUNPATH into account for searching libraries
* fix weird broken pipe error
* fix some typos:
s/begining/beginning/ s/subsystem_device/subsystem_vendor/ s/subsystem_system/subsystem_device/
* change field names for pci_ids elements (remove _id suffixes)
* DT_NEEDED of all files are analyzed. There is no way to differentiate between dynamically linked executables and dynamic libraries.
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 5 + usertools/dpdk-pmdinfo.py | 913 +++++++++---------------- 3 files changed, 313 insertions(+), 607 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -84,6 +84,11 @@ API Changes Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' ABI Changes -----------
usertools/dpdk-pmdinfo.py +307 -606
933 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 74 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "dmadev_idxd_pci")' { "name": "dmadev_idxd_pci", "params": "max_queues=0", "kmod": "vfio-pci", "pci_ids": [ { "vendor": "8086", "device": "0b25", "subsystem_vendor": "ffff", "subsystem_device": "ffff" } ] }Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "net_i40e").kmod' "* igb_uio | uio_pci_generic | vfio-pci"Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod' "* ib_uverbs & mlx5_core & mlx5_ib"Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}' ...The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
Quoted 6 lines
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
Some of the drivers doesn't provide PCI ids, but script is listing them empty, like [1], is it better to omit the output for that case, as done to 'params' & 'kmod'?
Except from above note,
[1]
{
"name": "net_enetfec",
"pci_ids": []},
Ferruh Yigit, Sep 13, 2022 at 22:54:
Quoted 3 lines
Some of the drivers doesn't provide PCI ids, but script is listing them empty, like [1], is it better to omit the output for that case, as done to 'params' & 'kmod'?
[snip]
Quoted 5 lines
[1]
{ "name": "net_enetfec", "pci_ids": []},
I could indeed omit the drivers that only report their name.
However, this raises another question: why do these drivers report a PMD_INFO_STRING in the first place? Should buildtools/pmdinfogen.py be modified as well?
Quoted 14 lines · Robin Jarry
CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
Ferruh Yigit, Sep 13, 2022 at 22:54:
Quoted 3 lines
Some of the drivers doesn't provide PCI ids, but script is listing them empty, like [1], is it better to omit the output for that case, as done to 'params' & 'kmod'?
[snip]
Quoted 5 lines
[1]
{ "name": "net_enetfec", "pci_ids": [] },
Quoted 6 lines
I could indeed omit the drivers that only report their name.
However, this raises another question: why do these drivers report a
PMD_INFO_STRINGin the first place? Shouldbuildtools/pmdinfogen.pybe modified as well?
I think better to display name, if there is nothing else to display, comparing to not display anything at all. Command can be run on the driver object, .so, so user expects to see some output.
For above enetfec driver, it is virtual driver and doesn't support pci_ids, and it seems doesn't have parameters or kmod requirements. So in this case it is OK to have only name.
Ferruh Yigit, Sep 14, 2022 at 13:46:
Quoted 8 lines
I think better to display name, if there is nothing else to display, comparing to not display anything at all.
Command can be run on the driver object, .so, so user expects to see some output.For above enetfec driver, it is virtual driver and doesn't support
pci_ids, and it seems doesn't have parameters or kmod requirements. So in this case it is OK to have only name.
Ok, was there something else to change before I send a v3?
PATCH v3 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "dmadev_idxd_pci")'
{
"name": "dmadev_idxd_pci",
"params": "max_queues=0",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "8086",
"device": "0b25",
"subsystem_vendor": "ffff",
"subsystem_device": "ffff"
} ]
}
Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "net_i40e").kmod'
"* igb_uio | uio_pci_generic | vfio-pci"
Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod'
"* ib_uverbs & mlx5_core & mlx5_ib"
Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}'
...
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
---
v2 -> v3:
* strip "pci_ids" when it is empty (some drivers do not support any pci devices)
v1 -> v2:
* update release notes and minimal python version requirement
* hide warnings by default (-v/--verbose to show them)
* show debug messages with -vv
* also search libs in folders listed in /etc/ld.so.conf/*.conf
* only search for DT_NEEDED on executables, not on dynamic libraries
* take DT_RUNPATH into account for searching libraries
* fix weird broken pipe error
* fix some typos:
s/begining/beginning/ s/subsystem_device/subsystem_vendor/ s/subsystem_system/subsystem_device/
* change field names for pci_ids elements (remove _id suffixes)
* DT_NEEDED of files are analyzed. There is no way to differentiate between dynamically linked executables and dynamic libraries.
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 5 + usertools/dpdk-pmdinfo.py | 914 +++++++++---------------- 3 files changed, 314 insertions(+), 607 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -84,6 +84,11 @@ API Changes Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' ABI Changes -----------
usertools/dpdk-pmdinfo.py +308 -606
934 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 76 lines · Robin Jarry
CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "dmadev_idxd_pci")' { "name": "dmadev_idxd_pci", "params": "max_queues=0", "kmod": "vfio-pci", "pci_ids": [ { "vendor": "8086", "device": "0b25", "subsystem_vendor": "ffff", "subsystem_device": "ffff" } ] }Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "net_i40e").kmod' "* igb_uio | uio_pci_generic | vfio-pci"Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod' "* ib_uverbs & mlx5_core & mlx5_ib"Print the list of drivers which define multiple parameters without string separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}' ...The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
Quoted 6 lines
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
For 'subsystem_vendor' & 'subsystem_device', the value "ffff" means it is not explicitly defined, so it gets default value. What do you think to omit those as well, when value is "ffff", to reduce noise on the output?
BTW, I have detected some duplicates in the output, like [1], [2] & [3]. It seems related to the duplicates in the code, cc'ed maintainers.
[1]:
{
"name": "net_qede",
"kmod": "* igb_uio | uio_pci_generic | vfio-pci",
"pci_ids": [
{
"vendor": "1077",
"device": "1634",
"subsystem_vendor": "ffff",
"subsystem_device": "ffff"
},
{
"vendor": "1077",
"device": "1629",
"subsystem_vendor": "ffff",
"subsystem_device": "ffff"
},
{
"vendor": "1077",
"device": "1634",
"subsystem_vendor": "ffff",
"subsystem_device": "ffff"
},...
[2]
{
"name": "mempool_cnxk",
"params": "max_pools=<128-1048576>",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "177d",
"device": "a0fb",
"subsystem_vendor": "ffff",
"subsystem_device": "b900"
},
{
"vendor": "177d",
"device": "a0fb",
"subsystem_vendor": "ffff",
"subsystem_device": "b900"
},...
[3]
{
"name": "net_cn10k",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "177d",
"device": "a063",
"subsystem_vendor": "ffff",
"subsystem_device": "b900"
},
{
"vendor": "177d",
"device": "a063",
"subsystem_vendor": "ffff",
"subsystem_device": "b900"
},...
Ferruh Yigit, Sep 20, 2022 at 12:10:
Quoted 4 lines
For '
subsystem_vendor' & 'subsystem_device', the value "ffff" means it is not explicitly defined, so it gets default value.
What do you think to omit those as well, when value is "ffff", to reduce noise on the output?
Sure, I could strip those as well.
PATCH v4 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "dmadev_idxd_pci")'
{
"name": "dmadev_idxd_pci",
"params": "max_queues=0",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "8086",
"device": "0b25"
} ]
}
Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.name == "net_i40e").kmod'
"* igb_uio | uio_pci_generic | vfio-pci"
Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod'
"* ib_uverbs & mlx5_core & mlx5_ib"
Print the list of drivers which define multiple parameters without space separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \
jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}'
...
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
---
v3 -> v4:
* also strip pci_id fields when they have the wildcard 0xFFFF value.
v2 -> v3:
* strip "pci_ids" when it is empty (some drivers do not support any pci devices)
v1 -> v2:
* update release notes and minimal python version requirement
* hide warnings by default (-v/--verbose to show them)
* show debug messages with -vv
* also search libs in folders listed in /etc/ld.so.conf/*.conf
* only search for DT_NEEDED on executables, not on dynamic libraries
* take DT_RUNPATH into account for searching libraries
* fix weird broken pipe error
* fix some typos:
s/begining/beginning/ s/subsystem_device/subsystem_vendor/ s/subsystem_system/subsystem_device/
* change field names for pci_ids elements (remove _id suffixes)
* DT_NEEDED of files are analyzed. There is no way to differentiate between dynamically linked executables and dynamic libraries.
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 5 + usertools/dpdk-pmdinfo.py | 924 +++++++++---------------- 3 files changed, 324 insertions(+), 607 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +5 -0
@@ -84,6 +84,11 @@ API Changes Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' ABI Changes -----------
usertools/dpdk-pmdinfo.py +318 -606
944 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 71 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "dmadev_idxd_pci")' { "name": "dmadev_idxd_pci", "params": "max_queues=0", "kmod": "vfio-pci", "pci_ids": [ { "vendor": "8086", "device": "0b25" } ]}
Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "net_i40e").kmod'"*
igb_uio|uio_pci_generic| vfio-pci"Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod'"*
ib_uverbs&mlx5_core&mlx5_ib"Print the list of drivers which define multiple parameters without space separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}'...
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
Quoted 6 lines
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
Quoted 78 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "dmadev_idxd_pci")' { "name": "dmadev_idxd_pci", "params": "max_queues=0", "kmod": "vfio-pci", "pci_ids": [ { "vendor": "8086", "device": "0b25" } ] }Get only the required kernel modules for a given driver:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.name == "net_i40e").kmod' "* igb_uio | uio_pci_generic | vfio-pci"Get only the required kernel modules for a given device:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod' "* ib_uverbs & mlx5_core & mlx5_ib"Print the list of drivers which define multiple parameters without space separators:
~$ usertools/dpdk-pmdinfo.py build/app/dpdk-testpmd | \ jq '.[] | select(.params!=null and (.params|test("=[^ ]+="))) | {name, params}' ...The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
<...>
Quoted 5 lines
doc/guides/rel_notes/release_22_11.rst +0
-0
doc/guides/rel_notes/release_22_11.rst API Changes · line 92
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst +5
-0@@ -84,6 +84,11 @@ API Changes Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'
Empty line is missing (in case there will be a new version for some other reason).
Thanks for the update,
Quoted 103 lines · Ferruh Yigit
Quoted 78 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
Here are some examples of use with jq:
Get the complete info for a given driver:
~$
usertools/dpdk-pmdinfo.pybuild/app/dpdk-testpmd | \
jq '.[] |select(.name == "dmadev_idxd_pci")' {
"name": "dmadev_idxd_pci",
"params": "max_queues=0",
"kmod": "vfio-pci",
"pci_ids": [
{
"vendor": "8086",
"device": "0b25"
}
]
}Get only the required kernel modules for a given driver:
~$
usertools/dpdk-pmdinfo.pybuild/app/dpdk-testpmd | \
jq '.[] |select(.name == "net_i40e").kmod' "*igb_uio|uio_pci_generic| vfio-pci"Get only the required kernel modules for a given device:
~$
usertools/dpdk-pmdinfo.pybuild/app/dpdk-testpmd | \
jq '.[] |select(.pci_ids[] | .vendor == "15b3" and .device == "1013").kmod' "*ib_uverbs&mlx5_core&mlx5_ib"Print the list of drivers which define multiple parameters without space separators:
~$
usertools/dpdk-pmdinfo.pybuild/app/dpdk-testpmd | \
jq '.[] |select(.params!=null and (.params|test("=[^ ]+=")))| {name, params}'
...The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 3.6 ok ok ok ok ok ok ok ok 3.7 ok ok ok ok ok ok ok ok Python 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module> from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements and release notes.
NB: The output produced by the legacy -r/--raw flag can be obtained with the following command:
strings build/app/dpdk-testpmd | sed -n 's/^PMD_INFO_STRING= //p'
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]><...>
Quoted 8 lines
doc/guides/rel_notes/release_22_11.rst +0
-0@@ -84,6 +84,11 @@ API ChangesAlso, make sure to start the actual text at the margin.
=======================================================Quoted 8 lines
+* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
+ PCI-IDs parsing has been removed.
+ To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command::
+
+ strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'Empty line is missing (in case there will be a new version for some other reason).
Quoted 3 lines
Thanks for the update,
Tested-by Ferruh Yigit <[email protected]>
Thomas, John,
Should we have documentation for usertools, since they are user facing, what do you think? Can it be possible to find resource for it?
Thanks,
ferruh
20/09/2022 19:50, Ferruh Yigit:
Quoted 4 lines
Thomas, John,
Should we have documentation for usertools, since they are user facing, what do you think?
We have doc/guides/tools/
Quoted 1 line
Can it be possible to find resource for it?
Quoted 8 lines · Thomas Monjalon
20/09/2022 19:50, Ferruh Yigit:
Quoted 4 lines
Thomas, John,
Should we have documentation for usertools, since they are user facing, what do you think?
We have doc/guides/tools/
Indeed, and there is even 'doc/guides/tools/pmdinfo.rst', so @Robin that documentation also needs to be updated with this patch.
I wonder if that documentation should have some pmdinfo technical details (not just dpdk-pmdinfo.py but overall feature)?
Quoted 3 lines
Quoted 1 line
Can it be possible to find resource for it?
Ferruh Yigit, Sep 20, 2022 at 19:48:
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst API Changes · line 91
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'Empty line is missing (in case there will be a new version for some other reason).
What do you mean?
Quoted 12 lines · Robin Jarry
Ferruh Yigit, Sep 20, 2022 at 19:48:
Quoted 9 lines
doc/guides/rel_notes/release_22_11.rst API Changes · line 91
Quoted 6 lines
doc/guides/rel_notes/release_22_11.rst +5
-0+* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command:: + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'Empty line is missing (in case there will be a new version for some other reason).
Quoted 2 lines
What do you mean?
just nit picking, there needs to be two empty lines before "ABI Changes", I assume committer can fix it while merging if there won't be any new version.
Quoted 16 lines · Ferruh Yigit
Quoted 14 lines · Robin Jarry
Ferruh Yigit, Sep 20, 2022 at 19:48:
Quoted 11 lines
Quoted 8 lines
+* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only.
+ PCI-IDs parsing has been removed.
+ To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command::
+
+ strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p'Empty line is missing (in case there will be a new version for some other reason).
Quoted 4 lines
Quoted 2 lines
What do you mean?
just nit picking, there needs to be two empty lines before "ABI Changes", I assume committer can fix it while merging if there won't be any new version.
Also looking twice, this release note update is in "API Changes" section, I think that is not correct place, "New Features" section can be better place.
PATCH v5 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
---
v4 -> v5:
* fixed doc/guides/rel_notes/release_22_11.rst
* updated doc/guides/tools/pmdinfo.rst with examples that were in the commit message
v3 -> v4:
* also strip pci_id fields when they have the wildcard 0xFFFF value.
v2 -> v3:
* strip "pci_ids" when it is empty (some drivers do not support any pci devices)
v1 -> v2:
* update release notes and minimal python version requirement
* hide warnings by default (-v/--verbose to show them)
* show debug messages with -vv
* also search libs in folders listed in /etc/ld.so.conf/*.conf
* only search for DT_NEEDED on executables, not on dynamic libraries
* take DT_RUNPATH into account for searching libraries
* fix weird broken pipe error
* fix some typos:
s/begining/beginning/ s/subsystem_device/subsystem_vendor/ s/subsystem_system/subsystem_device/
* change field names for pci_ids elements (remove _id suffixes)
* DT_NEEDED of files are analyzed. There is no way to differentiate between dynamically linked executables and dynamic libraries.
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 8 + doc/guides/tools/pmdinfo.rst | 86 ++- usertools/dpdk-pmdinfo.py | 924 +++++++++---------------- 4 files changed, 400 insertions(+), 620 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +8 -0
@@ -55,6 +55,14 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command: + + .. code-block:: sh + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' + Removed Items -------------
doc/guides/tools/pmdinfo.rst +73 -13
98 lines not on this page · raw message
usertools/dpdk-pmdinfo.py +318 -606
944 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 27 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Quoted 8 lines
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
Quoted 27 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Quoted 8 lines
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
Cc Olivier Matz<[email protected]>Cc Ferruh Yigit<[email protected]>Cc Bruce Richardson<[email protected]>Signed-off-by Robin Jarry<[email protected]>
Quoted 27 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Quoted 8 lines
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>
Robin Jarry, Sep 22, 2022 at 13:58:
usertools/dpdk-pmdinfo.py line 278
Quoted 7 lines
usertools/dpdk-pmdinfo.py +7
-0+# ---------------------------------------------------------------------------- +def ld_so_path() -> Iterator[str]: + """ + Return the list of directories where dynamic libraries are loaded based + on the contents of /etc/ld.so.conf/*.conf. + """ + for conf in glob.iglob("/etc/ld.so.conf/*.conf"):
I just noticed that this folder path is invalid. I did not encounter any errors since I did not test with librte_*.so libs installed in /usr/*.
usertools/dpdk-pmdinfo.py line 289
Quoted 11 lines
usertools/dpdk-pmdinfo.py +11
-0+ try: + with open(conf, "r", encoding="utf-8") as f: + for line in f: + line = line.strip() + if os.path.isdir(line): + yield line + except OSError: + pass + + +LD_SO_CONF_PATH = ld_so_path()
Also, this is stupid. The iterator will be exhausted after iterating once over it and the SO path folders will be empty on subsequent lookups. I'll submit a v6 with a fix for these two bugs.
PATCH v6 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok ok ok ok
3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
---
v5 -> v6:
* fixed typo: /etc/ld.so.conf/*.conf -> /etc/ld.so.conf.d/*.conf
* changed ld_so_path() to return an actual list to allow iterating multiple times
* include standard /lib & /usr/lib folders in LD_SO_PATH
v4 -> v5:
* fixed doc/guides/rel_notes/release_22_11.rst
* updated doc/guides/tools/pmdinfo.rst with examples that were in the commit message
v3 -> v4:
* also strip pci_id fields when they have the wildcard 0xFFFF value.
v2 -> v3:
* strip "pci_ids" when it is empty (some drivers do not support any pci devices)
v1 -> v2:
* update release notes and minimal python version requirement
* hide warnings by default (-v/--verbose to show them)
* show debug messages with -vv
* also search libs in folders listed in /etc/ld.so.conf/*.conf
* only search for DT_NEEDED on executables, not on dynamic libraries
* take DT_RUNPATH into account for searching libraries
* fix weird broken pipe error
* fix some typos:
s/begining/beginning/ s/subsystem_device/subsystem_vendor/ s/subsystem_system/subsystem_device/
* change field names for pci_ids elements (remove _id suffixes)
* DT_NEEDED of files are analyzed. There is no way to differentiate between dynamically linked executables and dynamic libraries.
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 8 + doc/guides/tools/pmdinfo.rst | 86 ++- usertools/dpdk-pmdinfo.py | 927 +++++++++---------------- 4 files changed, 403 insertions(+), 620 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +8 -0
@@ -55,6 +55,14 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command: + + .. code-block:: sh + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' + Removed Items -------------
doc/guides/tools/pmdinfo.rst +73 -13
98 lines not on this page · raw message
usertools/dpdk-pmdinfo.py +321 -606
947 lines not on this page · raw message
Signature · 1 line
2.37.3
Quoted 31 lines · Robin Jarry
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok ok ok ok 3.10 fail fail fail fail ok ok ok okAll failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later. Update the minimal system requirements, docs and release notes.
Quoted 8 lines
Cc Olivier Matz <[email protected]>Cc Ferruh Yigit <[email protected]>Cc Bruce Richardson <[email protected]>Signed-off-by Robin Jarry <[email protected]>---
v5 -> v6:* fixed typo:
/etc/ld.so.conf/*.conf->/etc/ld.so.conf.d/*.conf
I am a little uncertain about doing this parsing, and worried it may be a bit fragile. The main file for ld.so still is ld.so.conf, which, on my system anyway, does indeed just have an include for *.conf in the .d directory. However, is it possible that there are systems out there that still have entries in ld.so.conf and possibly elsewhere?
I think my preference would still be to shell out to ldconfig and query its database, or to shell out to ldd to get the dependencies of a .so from there. I just think it may be more robust, but at the cost of running some shell commands.
However, I don't feel strongly about this, so if others prefer the pure-python ld.so.conf parsing approach better, I'm ok with that.
/Bruce
Bruce Richardson, Sep 26, 2022 at 17:17:
Quoted 8 lines
Quoted 1 line
* fixed typo:
/etc/ld.so.conf/*.conf->/etc/ld.so.conf.d/*.confI am a little uncertain about doing this parsing, and worried it may be a bit fragile. The main file for ld.so still is ld.so.conf, which, on my system anyway, does indeed just have an include for *.conf in the .d directory. However, is it possible that there are systems out there that still have entries in ld.so.conf and possibly elsewhere?
Quoted 7 lines
I think my preference would still be to shell out to ldconfig and query its database, or to shell out to ldd to get the dependencies of a .so from there. I just think it may be more robust, but at the cost of running some shell commands.
However, I don't feel strongly about this, so if others prefer the pure-python ld.so.conf parsing approach better, I'm ok with that.
I was also concerned with parsing ld.so.conf files. However, I did not find a way to get ldconfig simply to print the folders that are to be analyzed. This would require some regexp parsing of ldconfig output:
ldconfig -vNX 2>/dev/null | sed -nre 's,^(/.*): \(from .*\)$,\1,p'
I don't know which way is the least hacky.
Quoted 17 lines · Robin Jarry
Bruce Richardson, Sep 26, 2022 at 17:17:
Quoted 8 lines
Quoted 1 line
* fixed typo:
/etc/ld.so.conf/*.conf->/etc/ld.so.conf.d/*.confI am a little uncertain about doing this parsing, and worried it may be a bit fragile. The main file for ld.so still is ld.so.conf, which, on my system anyway, does indeed just have an include for *.conf in the .d directory. However, is it possible that there are systems out there that still have entries in ld.so.conf and possibly elsewhere?
Quoted 7 lines
I think my preference would still be to shell out to ldconfig and query its database, or to shell out to ldd to get the dependencies of a .so from there. I just think it may be more robust, but at the cost of running some shell commands.
However, I don't feel strongly about this, so if others prefer the pure-python ld.so.conf parsing approach better, I'm ok with that.
Quoted 8 lines
I was also concerned with parsing ld.so.conf files. However, I did not find a way to get ldconfig simply to print the folders that are to be analyzed. This would require some regexp parsing of ldconfig output:
ldconfig -vNX 2>/dev/null | sed -nre 's,^(/.*): \(from .*\)$,\1,p'
I don't know which way is the least hacky.
How about "ldconfig -p" and just using the list of libraries given to match against those requested in the elf file, rather than worrying about directories at all?
Bruce Richardson, Sep 28, 2022 at 12:53:
Quoted 3 lines
How about "ldconfig -p" and just using the list of libraries given to match against those requested in the elf file, rather than worrying about directories at all?
I could do that but then, the DT_RUNPATH and LD_LIBRARY_PATH directories must be searched at runtime anyways. It seems more consistent to use a single method to resolve dynamic libraries.
PATCH v7 1/1 usertools: rewrite pmdinfo
In reply to Robin Jarrydpdk-pmdinfo.py does not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extracting PMD_INFO_STRING from .rodata ELF sections. Rewrite it so that it can produce valid JSON.
Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok ok
Python 3.8 ok ok ok ok ok ok ok ok
3.9 ok ok ok ok ok *ok ok ok
3.10 fail fail fail fail ok ok ok ok* Also tested on FreeBSD
All failures with python 3.10 are related to the same issue:
File "elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'
Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later.
Update the minimal system requirements, docs and release notes.
---
v6 -> v7:
* replaced hacky /etc/ld.so.conf.d/*.conf parsing with ldd invocation (makes code actually shorter)
* tested that it works on FreeBSD and Linux
* rebased on latest main (v22.07-486-gd5262b521d09)
doc/guides/linux_gsg/sys_reqs.rst | 2 +- doc/guides/rel_notes/release_22_11.rst | 10 + doc/guides/tools/pmdinfo.rst | 82 ++- usertools/dpdk-pmdinfo.py | 896 ++++++++----------------- 4 files changed, 370 insertions(+), 620 deletions(-)
doc/guides/linux_gsg/sys_reqs.rst +1 -1
@@ -41,7 +41,7 @@ Compilation of the DPDK resulting in statically linked applications not being linked properly. Use an updated version of ``pkg-config`` or ``pkgconf`` instead when building applications -* Python 3.5 or later. +* Python 3.6 or later. * Meson (version 0.49.2+) and ninja
doc/guides/rel_notes/release_22_11.rst +10 -0
@@ -123,6 +123,16 @@ New Features into single event containing ``rte_event_vector`` whose event type is ``RTE_EVENT_TYPE_CRYPTODEV_VECTOR``. +* **Rewritten dpdk-pmdinfo.py script.** + + The ``dpdk-pmdinfo.py`` script was rewritten to produce valid JSON only. + PCI-IDs parsing has been removed. + To get a similar output to the (now removed) ``-r/--raw`` flag, you may use the following command: + + .. code-block:: sh + + strings $dpdk_binary_or_driver | sed -n 's/^PMD_INFO_STRING= //p' + Removed Items -------------
doc/guides/tools/pmdinfo.rst +69 -13
94 lines not on this page · raw message
usertools/dpdk-pmdinfo.py +290 -606
916 lines not on this page · raw message
Signature · 1 line
2.37.3
Thomas Monjalon
Applied04/10/2022 21:29, Robin Jarry:
Quoted 32 lines
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok *ok ok ok 3.10 fail fail fail fail ok ok ok ok* Also tested on FreeBSD
All failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later.
Quoted 6 lines
Update the minimal system requirements, docs and release notes.
Signed-off-by Robin Jarry <[email protected]>Tested-by Ferruh Yigit <[email protected]>Tested-by Olivier Matz <[email protected]>Acked-by Bruce Richardson <[email protected]>
Applied, thanks.
Hi,
Quoted 33 lines · Thomas Monjalon
04/10/2022 21:29, Robin Jarry:
Quoted 32 lines
dpdk-pmdinfo.pydoes not produce any parseable output. The -r/--raw flag merely prints multiple independent JSON lines which cannot be fed directly to any JSON parser. Moreover, the script complexity is rather high for such a simple task: extractingPMD_INFO_STRINGfrom .rodata ELF sections. Rewrite it so that it can produce valid JSON.Remove the PCI database parsing for PCI-ID to Vendor-Device names conversion. This should be done by external scripts (if really needed).
The script passes flake8, black, isort and pylint checks.
I have tested this with a matrix of python/pyelftools versions:
pyelftools
0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29
3.6 ok ok ok ok ok ok ok ok
3.7 ok ok ok ok ok ok ok okPython 3.8 ok ok ok ok ok ok ok ok 3.9 ok ok ok ok ok *ok ok ok 3.10 fail fail fail fail ok ok ok ok* Also tested on FreeBSD
All failures with python 3.10 are related to the same issue:
File "
elftools/construct/lib/container.py", line 5, in <module>
from collections import MutableMapping
ImportError: cannot import name 'MutableMapping' from 'collections'Python 3.10 support is only available since pyelftools 0.26. The script will only work with Python 3.6 and later.
Quoted 8 lines
Quoted 6 lines
Update the minimal system requirements, docs and release notes.
Signed-off-by Robin Jarry <[email protected]>Tested-by Ferruh Yigit <[email protected]>Tested-by Olivier Matz <[email protected]>Acked-by Bruce Richardson <[email protected]>Applied, thanks.
As discussed off-list with Robin, it appears that "ldd" is not available on buildroot-based images. See:
http://lists.busybox.net/pipermail/buildroot/2013-July/074927.html
The link is quite old but it seems it's still true today if we don't build the toolchain.
Robin suggested this patch:
usertools/dpdk-pmdinfo.py +3 -1
@@ -290,8 +290,10 @@ def get_needed_libs(path: Path) -> Iterator[Path]: """ Extract the dynamic library dependencies from an ELF executable. """ + env = os.environ.copy() + env["LD_TRACE_LOADED_OBJECTS"] = "1" with subprocess.Popen( - ["ldd", str(path)], stdout=subprocess.PIPE, stderr=subprocess.PIPE + [str(path)], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env ) as proc: out, err = proc.communicate() if proc.returncode != 0:
One subtle difference is that the patched version won't work on non-executable files, but I don't think it can happen in real-life.
An alternative for us is to provide a simple "ldd" shell script in our buildroot-based images.
I don't have a strong opinion, I'll tend to say that the patch is a better option. Any comment?
Thanks,
Olivier
12/10/2022 17:16, Olivier Matz:
Quoted 25 lines · Thomas Monjalon
As discussed off-list with Robin, it appears that "ldd" is not available on buildroot-based images. See:
http://lists.busybox.net/pipermail/buildroot/2013-July/074927.htmlThe link is quite old but it seems it's still true today if we don't build the toolchain.
Robin suggested this patch:
usertools/dpdk-pmdinfo.py +3
-1@@ -290,8 +290,10 @@ def get_needed_libs(path: Path) -> Iterator[Path]: """ Extract the dynamic library dependencies from an ELF executable. """ + env = os.environ.copy() + env["LD_TRACE_LOADED_OBJECTS"] = "1" with subprocess.Popen( - ["ldd", str(path)], stdout=subprocess.PIPE, stderr=subprocess.PIPE + [str(path)], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env ) as proc: out, err = proc.communicate() if proc.returncode != 0:
Quoted 8 lines
One subtle difference is that the patched version won't work on non-executable files, but I don't think it can happen in real-life.
An alternative for us is to provide a simple "ldd" shell script in our buildroot-based images.
I don't have a strong opinion, I'll tend to say that the patch is a better option. Any comment?
What about implementing both?
If ldd is available, use it,
otherwise use LD_TRACE_LOADED_OBJECTS variable.
Thomas Monjalon, Oct 12, 2022 at 18:16:
Quoted 3 lines
What about implementing both?
If ldd is available, use it,
otherwise useLD_TRACE_LOADED_OBJECTSvariable.
This is a bit overkill in my opinion. Also it would make the behaviour somewhat different whether ldd is available and/or the analyzed binaries are executable.
I think it is an OK limitation to require that the analyzed ELF files are executable.
By the way, the following command:
LD_TRACE_LOADED_OBJECTS=1 build/app/dpdk-test-pmd
Works both on Linux and FreeBSD and produces similar outputs.
12/10/2022 18:30, Robin Jarry:
Quoted 8 lines
Thomas Monjalon, Oct 12, 2022 at 18:16:
Quoted 3 lines
What about implementing both?
If ldd is available, use it,
otherwise useLD_TRACE_LOADED_OBJECTSvariable.This is a bit overkill in my opinion. Also it would make the behaviour somewhat different whether ldd is available and/or the analyzed binaries are executable.
Yes, different behaviour is not desirable.
Quoted 8 lines
I think it is an OK limitation to require that the analyzed ELF files are executable.
By the way, the following command:
LD_TRACE_LOADED_OBJECTS=1 build/app/dpdk-test-pmd
Works both on Linux and FreeBSD and produces similar outputs.
OK for executable,
but can we expect .so to be always executable?
Thomas Monjalon, Oct 12, 2022 at 18:44:
Quoted 2 lines
OK for executable,
but can we expect .so to be always executable?
I think this is the default when linking. Whether dynamic libraries or executable programs.
Also, the "must be executable" limitation only applies on the files specified on the command line. Perhaps this is acceptable? We can make it obvious in the docs and return an explicit error so that users are not confused.
12/10/2022 18:48, Robin Jarry:
Quoted 11 lines
Thomas Monjalon, Oct 12, 2022 at 18:44:
Quoted 2 lines
OK for executable,
but can we expect .so to be always executable?I think this is the default when linking. Whether dynamic libraries or executable programs.
Also, the "must be executable" limitation only applies on the files specified on the command line. Perhaps this is acceptable? We can make it obvious in the docs and return an explicit error so that users are not confused.
Yes good idea to add a check with a warning.