Skip to contentSkip to conversation

pmdinfogen: rewrite in Python

RFC PATCH Superseded Dmitry Kozlyuk 12 participants 88 messages Jun 22, 2020 – Feb 2, 2021 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover pmdinfogen: rewrite in Python

This is a PoC rewrite of pmdinfogen in Python with missing bits described below and in commits. Community input is desired.

Pros:

1. Simpler build process without host apps.
2. Less build requirements (host libelf).
3. Easier debugging and maintenance with a high-level language.
4. Easier porting on Windows (only add new object format).

Cons:

1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).
2. struct rte_pci_id must be synchronized with header file (it's a few lines that never change).

There are no built-in or widely used Python libraries for ELF or COFF. Some ELF-parsing libraries exist on PyPI, but they're not very handy for the task and their installation would complicate build requirements. Thus, elf.py implements its own parsing.

COFF support is underway, it's just not included in this RFC. Amount of code is similar to elf.py.

Build is only tested on Linux x64_64.

If the community deems this RFC worth finishing, there are a few opens:

1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?

2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.

3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?

---
Dmitry Kozlyuk (2):
pmdinfogen: prototype in Python
build: use Python pmdinfogen

 buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   3 +-
 buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
 drivers/meson.build      |   2 +-
 4 files changed, 340 insertions(+), 3 deletions(-)
 create mode 100644 buildtools/elf.py
 create mode 100755 buildtools/pmdinfogen.py
Signature · 1 line
2.25.4
#1 lore raw

RFC PATCH 1/2 pmdinfogen: prototype in Python

* No user-friendly error handling and no bounds checking yet.
* No support for >65K sections case (is it needed?).
* The order of definitions is reversed.

---
 buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
 buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
 2 files changed, 338 insertions(+)
 create mode 100644 buildtools/elf.py
 create mode 100755 buildtools/pmdinfogen.py
buildtools/elf.py new file +194 -0
new file mode 100644
index 000000000..4085d547b
@@ -0,0 +1,194 @@
+import ctypes
+
+
+class ElfIdent(ctypes.Structure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_char * 4),
+        ("class_", ctypes.c_uint8),
+        ("data", ctypes.c_uint8),
+        ("version", ctypes.c_uint8),
+        ("os_abi", ctypes.c_uint8),
+        ("abi_version", ctypes.c_uint8),
+        ("pad", ctypes.c_uint8 * 7),
+    ]
+
+    @property
+    def is_magic_ok(self):
+        return self.magic.value == b"\x7fELF"
+
+    @property
+    def is_32bit(self):
+        return self.class_ == 1
+
+    @property
+    def is_big_endian(self):
+        return self.data == 2
+
+    def define_structures(self):
+        base_type = ctypes.LittleEndianStructure
+        if self.is_big_endian:
+            base = ctypes.BigEndianStructure
+
+        size_type = ctypes.c_uint64
+        if self.is_32bit:
+            size_type = ctypes.c_uint32
+
+        class FileHeader(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("e_ident", ElfIdent),
+                ("e_type", ctypes.c_uint16),
+                ("e_machine", ctypes.c_uint16),
+                ("e_version", ctypes.c_uint32),
+                ("e_entry", size_type),
+                ("e_phoff", size_type),
+                ("e_shoff", size_type),
+                ("e_flags", ctypes.c_uint32),
+                ("e_ehsize", ctypes.c_uint16),
+                ("e_phentsize", ctypes.c_uint16),
+                ("e_phnum", ctypes.c_uint16),
+                ("e_shentsize", ctypes.c_uint16),
+                ("e_shnum", ctypes.c_uint16),
+                ("e_shstrndx", ctypes.c_uint16),
+            ]
+
+        class SectionHeader(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("sh_name", ctypes.c_uint32),
+                ("sh_type", ctypes.c_uint32),
+                ("sh_flags", size_type),
+                ("sh_addr", size_type),
+                ("sh_offset", size_type),
+                ("sh_size", size_type),
+                ("sh_link", ctypes.c_uint32),
+                ("sh_info", ctypes.c_uint32),
+                ("sh_addralign", size_type),
+                ("sh_entsize", size_type),
+            ]
+
+        class Symbol32(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("st_name", ctypes.c_uint32),
+                ("st_value", ctypes.c_uint32),
+                ("st_size", ctypes.c_uint32),
+                ("st_info", ctypes.c_uint8),
+                ("st_other", ctypes.c_uint8),
+                ("st_shndx", ctypes.c_uint16),
+            ]
+
+        class Symbol64(base_type):
+            _pack_ = True
+            _fields_ = [
+                ("st_name", ctypes.c_uint32),
+                ("st_info", ctypes.c_uint8),
+                ("st_other", ctypes.c_uint8),
+                ("st_shndx", ctypes.c_uint16),
+                ("st_value", ctypes.c_uint64),
+                ("st_size", ctypes.c_uint64),
+            ]
+
+        Symbol = Symbol32 if self.is_32bit else Symbol64
+
+        return FileHeader, SectionHeader, Symbol
+
+
+class Symbol:
+    def __init__(self, image, elf):
+        self._image = image
+        self._elf = elf
+
+
+    @property
+    def address(self):
+        base = self._image._sections[self._elf.st_shndx].sh_offset
+        offset = base + self._elf.st_value
+        memory = ctypes.c_char.from_buffer(self._image._data, offset)
+        return ctypes.addressof(memory)
+
+
+class Image:
+    def __init__(self, data):
+        SHN_UNDEF = 0x0000
+        SHN_XINDEX = 0xFFFF
+
+        ident = ElfIdent.from_buffer(data)
+        ElfFileHeader, ElfSectionHeader, ElfSymbol = ident.define_structures()
+
+        header = ElfFileHeader.from_buffer(data)
+
+        if header.e_shnum == SHN_UNDEF:
+            section = ElfSectionHeader.from_buffer(data, header.e_shoff)
+            sections_num = section.sh_size
+        else:
+            sections_num = header.e_shnum
+        sections_desc = ElfSectionHeader * sections_num
+        sections = sections_desc.from_buffer(data, header.e_shoff)
+
+        if header.e_shstrndx == SHN_XINDEX:
+            strings_index = sections[0].sh_link
+        else:
+            strings_index = header.e_shstrndx
+
+        symtab, strtab = Image._find_symbol_table(data, sections, ElfSymbol)
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._strings = sections[strings_index]
+        self._symtab = symtab
+        self._strtab = strtab
+
+
+    @staticmethod
+    def _find_symbol_table(data, sections, symbol_type):
+        SHT_SYMTAB = 2
+        SHT_SYMTAB_SHNDX = 18
+
+        for section in sections:
+            if section.sh_type == SHT_SYMTAB:
+                symbol_count = section.sh_size // ctypes.sizeof(symbol_type)
+                symtab_desc = symbol_type * symbol_count
+                symtab_offset = section.sh_offset
+                symtab = symtab_desc.from_buffer(data, symtab_offset)
+
+                strtab = sections[section.sh_link].sh_offset
+
+                return symtab, strtab
+
+            # TODO: SHT_SYMTAB_SHNDX?
+
+        raise Exception('no symbol table')
+
+
+    @property
+    def is_big_endian(self):
+        return self._header.e_ident.is_big_endian
+
+
+    def find_symbol(self, name: str, start: Symbol = None):
+        name_size = len(name)
+        name_desc = ctypes.c_char * name_size
+
+        i = 0
+        if start:
+            table_address = ctypes.addressof(self._symtab)
+            start_address = ctypes.addressof(start._elf)
+            i = (start_address - table_address) // ctypes.sizeof(start._elf) + 1
+
+        while i < len(self._symtab):
+            symbol = self._symtab[i]
+
+            name_offset = self._strtab + symbol.st_name
+            if name_offset + name_size > len(self._data):
+                break
+
+            symbol_name = name_desc.from_buffer(self._data, name_offset).value
+            if symbol_name == name.encode():
+                return Symbol(self, symbol)
+
+            i += 1
+
+        return None
buildtools/pmdinfogen.py new file +144 -0

144 lines not on this page · raw message

Signature · 1 line
2.25.4
#2 lore raw

RFC PATCH 2/2 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

* C pmdinfogen not removed.
* No Makefile support yet.

---
 buildtools/meson.build | 3 +--
 drivers/meson.build    | 2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)
buildtools/meson.build +1 -2
index d5f8291be..1af835a9c 100644
@@ -1,9 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-subdir('pmdinfogen')
-
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
+pmdinfogen = find_program('pmdinfogen.py')
 pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
drivers/meson.build +1 -1
index cfb6a833c..be521398e 100644
@@ -122,7 +122,7 @@ foreach class:dpdk_driver_classes
 					command: [pmdinfo, tmp_lib.full_path(),
 						'@OUTPUT@', pmdinfogen],
 					output: out_filename,
-					depends: [pmdinfogen, tmp_lib])
+					depends: [tmp_lib])
 
 			version_map = '@0@/@1@/@2@_version.map'.format(
 					meson.current_source_dir(),
Signature · 1 line
2.25.4
#3 lore raw
In reply to Dmitry Kozlyuk
Quoted 10 lines · Dmitry Kozlyuk

This is a PoC rewrite of pmdinfogen in Python with missing bits described below and in commits. Community input is desired.

Pros:

1. Simpler build process without host apps.
2. Less build requirements (host libelf).
3. Easier debugging and maintenance with a high-level language.
4. Easier porting on Windows (only add new object format).

Front matter: It seems reasonable to me to implement this in python, just for ease of maintenence.

Quoted 4 lines

Cons:

1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).

Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden

Quoted 3 lines

2. struct rte_pci_id must be synchronized with header file (it's a few lines that never change).

I think you can just use ctypes and the native python C interface to just import the rte_pci_id structure from the native dpdk header to avoid this, no?

Quoted 8 lines

There are no built-in or widely used Python libraries for ELF or COFF. Some ELF-parsing libraries exist on PyPI, but they're not very handy for the task and their installation would complicate build requirements. Thus, elf.py implements its own parsing.

COFF support is underway, it's just not included in this RFC. Amount of code is similar to elf.py.

Quoted 7 lines

Build is only tested on Linux x64_64.

If the community deems this RFC worth finishing, there are a few opens:

1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?

Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.

Quoted 5 lines

2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.

I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.

Quoted 6 lines

3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?

I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?

Quoted 17 lines

---
Dmitry Kozlyuk (2):
pmdinfogen: prototype in Python
build: use Python pmdinfogen

 buildtools/elf.py        | 194 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   3 +-
 buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++
 drivers/meson.build      |   2 +-
 4 files changed, 340 insertions(+), 3 deletions(-)
 create mode 100644 buildtools/elf.py
 create mode 100755 buildtools/pmdinfogen.py
Signature · 1 line
2.25.4
#4 lore raw
Quoted 1 line · Neil Horman

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:

[snip]

Quoted 6 lines
Quoted 2 lines

1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).

Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden

I must have underestimated pyelftools spread. I'll look into using it. The less new code the better, I agree here. Windows users can get it via PyPI.

Quoted 5 lines
Quoted 3 lines

2. struct rte_pci_id must be synchronized with header file (it's a few lines that never change).

I think you can just use ctypes and the native python C interface to just import the rte_pci_id structure from the native dpdk header to avoid this, no?

Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id in Python. If you're suggesting to create a Python module in C just to include a header with a single small structure, I'd say it's not worth the trouble.

Quoted 8 lines
Quoted 3 lines

1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?

Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.

I was talking about these fields of struct elf_info:

	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
	 * take shndx from symtab_shndx_start[N] instead
	 */
	Elf32_Word   *symtab_shndx_start;
	Elf32_Word   *symtab_shndx_stop;

They are not used after being filled in parse_elf() and their endianness fixed in the end despite the comment.

Quoted 8 lines
Quoted 5 lines

2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.

I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.

Consider a snippet from pmdinfogen.c:

	/* Check if file offset is correct */
	if (hdr->e_shoff > info->size) {
		fprintf(stderr, "section header offset=%lu in file '%s' "
		      "is bigger than filesize=%lu\n",
		      (unsigned long)hdr->e_shoff,
		      filename, info->size);
		return 0;
	}

It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.

Quoted 11 lines
Quoted 6 lines

3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?

I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?

WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):

Host     Toolchain  Archive  Script      Extractor
-------  ---------  -------  ---------   ---------
Linux    MinGW      .a       sh          ar
Windows  MinGW      .a       PowerShell  ar
Windows  Clang      .lib     PowerShell  lib
Signature · 1 line
Dmitry Kozlyuk
#5 lore raw
Quoted 5 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:

[snip]

Quoted 2 lines
Quoted 2 lines

1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).

Quoted 8 lines
Quoted 4 lines

Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden

I must have underestimated pyelftools spread. I'll look into using it. The less new code the better, I agree here. Windows users can get it via PyPI.

Ack.

Quoted 11 lines
Quoted 6 lines
Quoted 4 lines

2. struct rte_pci_id must be synchronized with header file (it's a few lines that never change).

I think you can just use ctypes and the native python C interface to just import the rte_pci_id structure from the native dpdk header to avoid this, no?

Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id in Python. If you're suggesting to create a Python module in C just to include a header with a single small structure, I'd say it's not worth the trouble.

Yeah, apologies, i misread what you were saying here, and didn't bother to check the code. what you're doing makes sense.

Quoted 12 lines
Quoted 9 lines
Quoted 4 lines

1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?

Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.

I was talking about these fields of struct elf_info:

Quoted 9 lines
	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
	 * take shndx from symtab_shndx_start[N] instead
	 */
	Elf32_Word   *symtab_shndx_start;
	Elf32_Word   *symtab_shndx_stop;

They are not used after being filled in parse_elf() and their endianness fixed in the end despite the comment.

Its been a while since I wrote this, so I need to go back and look closely, but as you say, these values aren't used after parse_elf completes, but they are(I think) used to correct the endianess of the section header index table, so that get_sym_value works properly. You would (again I think), only note a problem if you were parsing an ELF file for an architecture with an endianess opposite that of what you are building on (i.e. an x86 system cross compiling for a powerpc target).

Quoted 21 lines
Quoted 9 lines
Quoted 6 lines

2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.

I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.

Consider a snippet from pmdinfogen.c:

	/* Check if file offset is correct */
	if (hdr->e_shoff > info->size) {
		fprintf(stderr, "section header offset=%lu in file '%s' "
		      "is bigger than filesize=%lu\n",
		      (unsigned long)hdr->e_shoff,
		      filename, info->size);
		return 0;
	}
Quoted 7 lines

It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.

Hmm, I'd defer to others opinions on that. As a build tool it may be ok to just crash with a backtrace, but I'm personally not a fan of that. Id rather see a diagnostic error and have the tool exits with an appropriate exit code, but lets see what others say.

Quoted 19 lines
Quoted 7 lines
Quoted 7 lines

3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?

Quoted 5 lines

I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?

WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):

Quoted 5 lines
Host     Toolchain  Archive  Script      Extractor
-------  ---------  -------  ---------   ---------
Linux    MinGW      .a       sh          ar
Windows  MinGW      .a       PowerShell  ar
Windows  Clang      .lib     PowerShell  lib

I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?

Neil

Quoted 4 lines
Signature · 2 lines
-- 
Dmitry Kozlyuk
#6 lore raw
Quoted 33 lines · Neil Horman
Quoted 25 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:

[snip]

Quoted 8 lines
Quoted 5 lines

2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.

I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.

Consider a snippet from pmdinfogen.c:

	/* Check if file offset is correct */
	if (hdr->e_shoff > info->size) {
		fprintf(stderr, "section header offset=%lu in file '%s' "
		      "is bigger than filesize=%lu\n",
		      (unsigned long)hdr->e_shoff,
		      filename, info->size);
		return 0;
	}
Quoted 7 lines

It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.

Quoted 5 lines

Hmm, I'd defer to others opinions on that. As a build tool it may be ok to just crash with a backtrace, but I'm personally not a fan of that. Id rather see a diagnostic error and have the tool exits with an appropriate exit code, but lets see what others say.

I'd take a wait-and-see approach here. Obviously individualized errors messages are better, but for a case like the above where we have a malformed elf header, I would think that this is such a rare event that just letting python exception handling manage it should be ok.

Quoted 24 lines
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines

3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?

Quoted 5 lines

I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?

WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):

Quoted 5 lines
Host     Toolchain  Archive  Script      Extractor
-------  ---------  -------  ---------   ---------
Linux    MinGW      .a       sh          ar
Windows  MinGW      .a       PowerShell  ar
Windows  Clang      .lib     PowerShell  lib
Quoted 6 lines

I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?

It might be better just to do that detection in the python script itself, rather than in meson, since the script may have to do other behavioural changes based on the platform.

Regards,
/Bruce

#7 lore raw
In reply to Neil Horman
Quoted 3 lines · On Tue, 23 Jun 2020 07:28:06 -0400 Neil Horman wrote:
Quoted 2 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman

On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:

[snip]

Quoted 11 lines
Quoted 11 lines

I was talking about these fields of struct elf_info:

	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
	 * take shndx from symtab_shndx_start[N] instead
	 */
	Elf32_Word   *symtab_shndx_start;
	Elf32_Word   *symtab_shndx_stop;

They are not used after being filled in parse_elf() and their endianness fixed in the end despite the comment.

Quoted 7 lines

Its been a while since I wrote this, so I need to go back and look closely, but as you say, these values aren't used after parse_elf completes, but they are(I think) used to correct the endianess of the section header index table, so that get_sym_value works properly. You would (again I think), only note a problem if you were parsing an ELF file for an architecture with an endianess opposite that of what you are building on (i.e. an x86 system cross compiling for a powerpc target).

Thanks for the explanation. Anyway, it's yet another point to prefer pyelftools to parsing ELF manually.

[snip]

Quoted 10 lines
Quoted 10 lines

Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):

Host     Toolchain  Archive  Script      Extractor
-------  ---------  -------  ---------   ---------
Linux    MinGW      .a       sh          ar
Windows  MinGW      .a       PowerShell  ar
Windows  Clang      .lib     PowerShell  lib  
Quoted 5 lines

I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?

As I said, PowerShell cannot consume the same shell script. My suggestion was to have two scripts, one existing for shell and ar (used by Linux host), and one new PowerShell script for both lib.exe and ar (used by Windows host). Meson then would select which script to call. Another option is to have one wrapper Python script for all hosts: a few extra lines to launch archiver, but less files and no need to know shell/PowerShell when changing this part. (I believe Bruce means the same.) But it will still be a separate script.

Signature · 1 line
Dmitry Kozlyuk
#8 lore raw

v2 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

The goals are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Only logic errors, like missing symbols or invalid content, are checked. This seems enough for debugging and, to my understanding, there are no strong objections. All errors raise exceptions printed to stderr.

While discussion of future wrapper logic and Windows support continues, this patchset is ready for testing and review.

Dmitry Kozlyuk (3):
pmdinfogen: prototype in Python
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen

 GNUmakefile                       |   2 +-
 buildtools/Makefile               |   9 --
 buildtools/gen-pmdinfo-cfile.sh   |   3 +-
 buildtools/meson.build            |   3 +-
 buildtools/pmdinfogen.py          | 191 ++++++++++++++++++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |   6 +
 drivers/meson.build               |   2 +-
 mk/internal/rte.compile-pre.mk    |   2 +-
 mk/rte.sdkinstall.mk              |   2 -
 9 files changed, 202 insertions(+), 18 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py
Signature · 1 line
2.25.4
#9 lore raw

RFC PATCH v2 1/3 pmdinfogen: prototype in Python

Requires pyelfutils, but no docs yet.

---
 buildtools/pmdinfogen.py | 191 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 191 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +191 -0
new file mode 100755
index 000000000..05252e7ce
@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        if table_symbol.size % pci_id_size != 0:
+            raise Exception(
+                "PCI table size (%d) is not a multiple of rte_pci_id size (%d)"
+                % (table_symbol.size, pci_id_size)
+            )
+
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.25.4
#10 lore raw

RFC PATCH v2 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Also don't suppress stderr from ar and pmdinfogen.

---
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 3 +--
 buildtools/meson.build          | 3 +--
 drivers/meson.build             | 2 +-
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 7 files changed, 5 insertions(+), 18 deletions(-)
 delete mode 100644 buildtools/Makefile
GNUmakefile +1 -1
index e8de422df..242d30d2e 100644
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
deleted file mode 100644
index 7f76fd7d6..000000000
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +1 -2
index 43059cf36..eeeab678e 100755
@@ -9,6 +9,5 @@ pmdinfogen=$3
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +1 -2
index f9d2fdf74..88c238ca4 100644
@@ -1,9 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-subdir('pmdinfogen')
-
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
+pmdinfogen = find_program('pmdinfogen.py')
 pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
drivers/meson.build +1 -1
index e78c76c55..37822537a 100644
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 			version_map = '@0@/@1@/@2@_version.map'.format(
 					meson.current_source_dir(),
mk/internal/rte.compile-pre.mk +1 -1
index df05b5576..bb2ab0725 100644
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected]
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected]
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected]
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
index 32bed5d95..875a64f04 100644
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
Signature · 1 line
2.25.4
#11 lore raw

RFC PATCH v2 3/3 doc/linux_gsg: require pyelftools for pmdinfogen

In reply to Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
#12 lore raw

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

Quoted 4 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file

Neil

Quoted 26 lines
---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
#13 lore raw

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

Quoted 8 lines · Neil Horman
Quoted 4 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file

Windows uses COFF instead of ELF. I'm planning to add COFF support to pmdinfogen in a separate patch and also put wrapper script there. In this series I'd like to focus on just rewriting existing functionality.

Signature · 1 line
Dmitry Kozlyuk
#14 lore raw

Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen

Quoted 6 lines · Dmitry Kozlyuk
Quoted 5 lines · Neil Horman
Quoted 4 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

Quoted 8 lines
Quoted 3 lines

Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file

Windows uses COFF instead of ELF. I'm planning to add COFF support to pmdinfogen in a separate patch and also put wrapper script there. In this series I'd like to focus on just rewriting existing functionality.

Ah, understood, thanks for the clarification

Neil

Quoted 3 lines
Signature · 1 line
Dmitry Kozlyuk
#15 lore raw

v3 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 GNUmakefile                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/Makefile                |   9 -
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 194 ++++++++++++
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 mk/internal/rte.compile-pre.mk     |   2 +-
 mk/rte.sdkinstall.mk               |   2 -
 16 files changed, 209 insertions(+), 629 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.25.4
#16 lore raw

PATCH v3 1/4 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 194 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 194 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +194 -0
new file mode 100755
index 000000000..c32654a4a
@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        if table_symbol.size % pci_id_size != 0:
+            raise Exception(
+                "PCI table size (%d) is not a multiple of rte_pci_id size (%d)"
+                % (table_symbol.size, pci_id_size)
+            )
+
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.25.4
#17 lore raw

PATCH v3 2/4 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add python3-pyelftools to CI configuration.

---
 .travis.yml                     | 2 +-
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 6 +++---
 buildtools/meson.build          | 1 +
 drivers/meson.build             | 2 +-
 meson.build                     | 1 -
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 9 files changed, 8 insertions(+), 19 deletions(-)
 delete mode 100644 buildtools/Makefile
.travis.yml +1 -1
index 14f812423..28b559a25 100644
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
GNUmakefile +1 -1
index e8de422df..242d30d2e 100644
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
deleted file mode 100644
index 7f76fd7d6..000000000
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +1 -0
index 04808dabc..3a64b28b7 100644
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
drivers/meson.build +1 -1
index 161cfda04..3c4d4b700 100644
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
mk/internal/rte.compile-pre.mk +1 -1
index df05b5576..bb2ab0725 100644
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected]
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected]
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected]
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
index 32bed5d95..875a64f04 100644
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
Signature · 1 line
2.25.4
#18 lore raw

PATCH v3 3/4 doc/linux_gsg: require pyelftools for pmdinfogen

In reply to Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
#19 lore raw

PATCH v3 4/4 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.

---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 5 files changed, 1 insertion(+), 610 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
index 6ba02d73a..93d209101 100644
@@ -157,7 +157,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/Makefile deleted +0 -20
deleted file mode 100644
index a97a7648f..000000000
@@ -1,20 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-#
-# library name
-#
-HOSTAPP = dpdk-pmdinfogen
-
-#
-# all sources are stored in SRCS-y
-#
-SRCS-y += pmdinfogen.c
-
-HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g
-HOST_CFLAGS += -I$(RTE_OUTPUT)/include
-
-include $(RTE_SDK)/mk/rte.hostapp.mk
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.25.4
#20 lore raw

v4 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 GNUmakefile                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/Makefile                |   9 -
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 188 ++++++++++++
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 mk/internal/rte.compile-pre.mk     |   2 +-
 mk/rte.sdkinstall.mk               |   2 -
 16 files changed, 203 insertions(+), 629 deletions(-)
 delete mode 100644 buildtools/Makefile
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.25.4
#21 lore raw

PATCH v4 1/4 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +188 -0
new file mode 100755
index 000000000..a9c2643a4
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.25.4
#22 lore raw

PATCH v4 2/4 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add python3-pyelftools to CI configuration.

---
 .travis.yml                     | 2 +-
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 6 +++---
 buildtools/meson.build          | 1 +
 drivers/meson.build             | 2 +-
 meson.build                     | 1 -
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 9 files changed, 8 insertions(+), 19 deletions(-)
 delete mode 100644 buildtools/Makefile
.travis.yml +1 -1
index 14f812423..28b559a25 100644
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
GNUmakefile +1 -1
index e8de422df..242d30d2e 100644
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
deleted file mode 100644
index 7f76fd7d6..000000000
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +1 -0
index 04808dabc..3a64b28b7 100644
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
drivers/meson.build +1 -1
index 161cfda04..3c4d4b700 100644
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
mk/internal/rte.compile-pre.mk +1 -1
index df05b5576..bb2ab0725 100644
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected]
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected]
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected]
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
index 32bed5d95..875a64f04 100644
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
Signature · 1 line
2.25.4
Reviewed-by on v4 2/4
#23 lore raw

Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen

Quoted 69 lines · Dmitry Kozlyuk

Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add python3-pyelftools to CI configuration.

---
 .travis.yml                     | 2 +-
 GNUmakefile                     | 2 +-
 buildtools/Makefile             | 9 ---------
 buildtools/gen-pmdinfo-cfile.sh | 6 +++---
 buildtools/meson.build          | 1 +
 drivers/meson.build             | 2 +-
 meson.build                     | 1 -
 mk/internal/rte.compile-pre.mk  | 2 +-
 mk/rte.sdkinstall.mk            | 2 --
 9 files changed, 8 insertions(+), 19 deletions(-)
 delete mode 100644 buildtools/Makefile
.travis.yml +1 -1
index 14f812423..28b559a25 100644
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
GNUmakefile +1 -1
index e8de422df..242d30d2e 100644
@@ -12,6 +12,6 @@ export RTE_SDK
 # directory list
 #
 
-ROOTDIRS-y := buildtools lib kernel drivers app
+ROOTDIRS-y := lib kernel drivers app
 
 include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
deleted file mode 100644
index 7f76fd7d6..000000000
@@ -1,9 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-DIRS-y += pmdinfogen
-
-include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +0 -1
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3

buildtools/gen-pmdinfo-cfile.sh line 14

Quoted 10 lines
buildtools/gen-pmdinfo-cfile.sh +3 -2
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0

For 20.11, since make build system is being removed, I think we should look to merge this into the python script rather than having an extra layer of wrapper and having meson pass the command from one script to another. However, since out-of-the-box python doesn't seem to have support for reading .a files, this is fine for now.

Quoted 70 lines
buildtools/meson.build +1 -0
index 04808dabc..3a64b28b7 100644
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
drivers/meson.build +1 -1
index 161cfda04..3c4d4b700 100644
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
mk/internal/rte.compile-pre.mk +1 -1
index df05b5576..bb2ab0725 100644
@@ -59,7 +59,7 @@ endif
 CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh
 CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@
 
-PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected]
+PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected]
 PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected]
 PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@
 PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
index 32bed5d95..875a64f04 100644
@@ -98,7 +98,6 @@ install-runtime:
 	$(Q)cp $(CP_FLAGS)    $O/lib/* $(DESTDIR)$(libdir)
 	$(Q)$(call rte_mkdir, $(DESTDIR)$(bindir))
 	$(Q)tar -cf -      -C $O --exclude 'app/*.map' \
-		--exclude app/dpdk-pmdinfogen \
 		--exclude 'app/cmdline*' --exclude app/test \
 		--exclude app/testacl --exclude app/testpipeline app | \
 	    tar -xf -      -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk:
 	$(Q)cp $(CP_FLAGS)      $(RTE_SDK)/buildtools    $(DESTDIR)$(sdkdir)
 	$(Q)$(call rte_mkdir,                            $(DESTDIR)$(targetdir)/app)
 	$(Q)cp $(CP_FLAGS)      $O/.config               $(DESTDIR)$(targetdir)
-	$(Q)cp $(CP_FLAGS)      $O/app/dpdk-pmdinfogen   $(DESTDIR)$(targetdir)/app
 	$(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include)
 	$(Q)$(call rte_symlink, $(DESTDIR)$(libdir),     $(DESTDIR)$(targetdir)/lib)
 
Signature · 1 line
2.25.4

For the meson build changes here:

#24 lore raw

Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen

Quoted 1 line · Bruce Richardson

On Thu, Jul 09, 2020 at 12:23:33AM +0300, Dmitry Kozlyuk wrote:

[snip]

Quoted 12 lines
Quoted 12 lines
buildtools/gen-pmdinfo-cfile.sh +2 -1
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
Quoted 13 lines

buildtools/gen-pmdinfo-cfile.sh line 14

Quoted 7 lines
buildtools/gen-pmdinfo-cfile.sh +1 -2
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0  

For 20.11, since make build system is being removed, I think we should look to merge this into the python script rather than having an extra layer of wrapper and having meson pass the command from one script to another. However, since out-of-the-box python doesn't seem to have support for reading .a files, this is fine for now.

Or Meson can only tell the Python script whether to process ELF of COFF and the path to an archiver tool (ar or lib.exe). This way we'd get rid of shell (which is non-portable to Windows) and need no libraries to read .a or .lib.

#25 lore raw

PATCH v4 3/4 doc/linux_gsg: require pyelftools for pmdinfogen

In reply to Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
#26 lore raw

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

Quoted 14 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +0 -0
index a124656bc..ef2037ca7 100644

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 64

Quoted 9 lines
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+

If you want for completeness you can add:
* For FreeBSD it can be installed using ``pkg install py37-pyelftools``, where py37 refers to the python version in use.

Quoted 6 lines
doc/guides/linux_gsg/sys_reqs.rst +0 -0
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
#27 lore raw

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

Quoted 18 lines · Bruce Richardson
Quoted 17 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +0 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
Quoted 11 lines

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 64

Quoted 6 lines
doc/guides/linux_gsg/sys_reqs.rst +6 -0
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+

If you want for completeness you can add:
* For FreeBSD it can be installed using ``pkg install py37-pyelftools``, where py37 refers to the python version in use.

Sorry, missed the fact that this was the Linux GSG. Never mind...

#28 lore raw

Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen

In reply to Dmitry Kozlyuk
Quoted 17 lines · Dmitry Kozlyuk

The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.

---
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +0 -0
index a124656bc..ef2037ca7 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 67

Quoted 10 lines
doc/guides/linux_gsg/sys_reqs.rst +6 -0
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;

This can be merged into patch 2, I think.

#29 lore raw

PATCH v4 4/4 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.

---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/Makefile     |  20 --
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 5 files changed, 1 insertion(+), 610 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/Makefile
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
index 2bb8583f3..68046aaad 100644
@@ -157,7 +157,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/Makefile deleted +0 -20
deleted file mode 100644
index a97a7648f..000000000
@@ -1,20 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2016 Neil Horman <[email protected]>
-# All rights reserved.
-
-include $(RTE_SDK)/mk/rte.vars.mk
-
-#
-# library name
-#
-HOSTAPP = dpdk-pmdinfogen
-
-#
-# all sources are stored in SRCS-y
-#
-SRCS-y += pmdinfogen.c
-
-HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g
-HOST_CFLAGS += -I$(RTE_OUTPUT)/include
-
-include $(RTE_SDK)/mk/rte.hostapp.mk
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.25.4
Acked-by on the v4 series
#30 lore raw
In reply to Dmitry Kozlyuk
Quoted 28 lines · Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Quoted 7 lines

---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation

I'm not currently setup to test with cross compiling, but for direct targets (tested on power and x86), it seems to work well

#31 lore raw
In reply to Dmitry Kozlyuk
Quoted 28 lines · Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Quoted 7 lines

---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation

I've done builds with this patchset on linux and FreeBSD and checked the results with dpdk-pmdinfo script. All looks ok to me.

#32 lore raw

v5 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

From v4:

---
Changes in v5:

1. Rebase on ToT, remove Makefile changes.
2. Squash doc changes into patch 2/3.

Changes in v4:

Fix build by removing redundant check.

Changes in v3:

1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.

Changes in v2:

1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   2 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 188 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 201 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
#33 lore raw

PATCH v5 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +188 -0
new file mode 100755
index 000000000..a9c2643a4
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        if len(value):
+            return value[:-1].decode()
+        return ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        if symbol:
+            return ELFSymbol(self, symbol[0])
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    def _load_pci_ids(self, image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            self.pci_ids.append(
+                [
+                    pci_id.vendor_id,
+                    pci_id.device_id,
+                    pci_id.subsystem_vendor_id,
+                    pci_id.subsystem_device_id,
+                ]
+            )
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.28.0
#34 lore raw

Re: [dpdk-dev] [PATCH v5 1/3] pmdinfogen: add Python implementation

Quoted 5 lines · On Mon, 28 Sep 2020 00:47:30 +0300 Dmitry Kozlyuk <[email protected]>…

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

Looks good, a couple little nits.

Yapf flags this as unnecessary spacing:

+ @property
+ def value(self):

+        data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+        base = self._symbol["st_value"]
+        return data[base : base + self.size]
                         ^^^

Python lint complains about missing docstrings, which doesn't matter but it also shows:

$ pylint3 buildtools/pmdinfogen.py  | grep -v docstring
************* Module pmdinfogen
buildtools/pmdinfogen.py:27:15: W0212: Access to a protected member _image of a client class (protected-access)
buildtools/pmdinfogen.py:34:11: C1801: Do not use `len(SEQUENCE)` to determine if a sequence is empty (len-as-condition)
buildtools/pmdinfogen.py:50:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
buildtools/pmdinfogen.py:107:12: W0212: Access to a protected member _load_pci_ids of a client class (protected-access)

------------------------------------------------------------------
Your code has been rated at 7.93/10 (previous run: 7.93/10, +0.00)

#35 lore raw

PATCH v5 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---
 .travis.yml                       | 2 +-
 buildtools/gen-pmdinfo-cfile.sh   | 6 +++---
 buildtools/meson.build            | 1 +
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 drivers/meson.build               | 2 +-
 meson.build                       | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)
.travis.yml +1 -1
index d6eeab371..226db7590 100644
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +1 -0
index 04808dabc..3a64b28b7 100644
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index 6b9f573fc..200212081 100644
@@ -56,6 +56,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index 5f9526557..88364259d 100644
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.28.0
#36 lore raw

PATCH v5 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.

---
 MAINTAINERS                        |   2 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 1 insertion(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
index 652712ccb..38e994388 100644
@@ -135,7 +135,7 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.28.0
#37 lore raw

Re: [dpdk-dev] [PATCH v5 3/3] pmdinfogen: remove C implementation

Quoted 3 lines

Delete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.

Dmitry, don't you think you should be co-maintainer?

#38 lore raw
In reply to Dmitry Kozlyuk

Hello Dmitry,

Quoted 4 lines · Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* support pmdinfo on Windows ? :-)

Quoted 6 lines

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

This new requirement will impact mainly distribution packagers, developers and the CI.
While I think this is fine for the first two categories (python-elftools is now available on most distributions, and you can also pip install it), the last category is a concern.

If we want to move forward with this patch, this must be addressed in a timely manner as we don't want to break the CI.

Signature · 1 line
David Marchand
#39 lore raw

v6 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v6:

1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 203 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
#40 lore raw

PATCH v6 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
new file mode 100755
index 000000000..0cca47ff1
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.28.0
#41 lore raw

PATCH v6 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---
 .travis.yml                       | 2 +-
 buildtools/gen-pmdinfo-cfile.sh   | 6 +++---
 buildtools/meson.build            | 1 +
 doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++
 drivers/meson.build               | 2 +-
 meson.build                       | 1 -
 6 files changed, 12 insertions(+), 6 deletions(-)
.travis.yml +1 -1
index 5e12db23b..6744f6fc1 100644
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +1 -0
index 04808dabc..3a64b28b7 100644
@@ -17,3 +17,4 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index 6ecdc04aa..89c9f2570 100644
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index 5f9526557..88364259d 100644
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.28.0
#42 lore raw

PATCH v6 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index 681093d94..24a8a160c 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.28.0
#43 lore raw
In reply to Dmitry Kozlyuk

Hi Dmitry,

Quoted 24 lines · Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v6:

1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

Quoted 17 lines
 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |   1 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 203 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h

Ninja fails when pyelftools is not installed. Shouldn't meson detect and report it before build time?

Thanks,
Maxime

#45 lore raw

v7 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v7:

Add configure-time check for elftools module (Maxime Coquelin).

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
#46 lore raw

PATCH v7 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
new file mode 100755
index 000000000..0cca47ff1
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.28.0
#47 lore raw

PATCH v7 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add configure-time check for elftools Python module.

Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)
.travis.yml +1 -1
index 5e12db23b..6744f6fc1 100644
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +15 -0
index 04808dabc..0c668eb46 100644
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = ['elftools']
+foreach module : python3_required_modules
+	script = \
+		'import importlib.util;' + \
+		'import sys;' + \
+		'exit(importlib.util.find_spec("@0@") is None)'.format(module)
+	if run_command(py3, '-c', script).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index 6ecdc04aa..89c9f2570 100644
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index 5f9526557..88364259d 100644
@@ -137,7 +137,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.28.0
#48 lore raw

PATCH v7 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index 499c7d297..5f5c599d2 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.28.0
#49 lore raw
In reply to Dmitry Kozlyuk

Hello Dmitry,

Quoted 4 lines · Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

Quoted 8 lines

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478

Build machine cpu: x86_64
buildtools/meson.build:27:10: ERROR: Expecting endforeach got eol_cont.

script = \
          ^

For a block that started at 26,0
foreach module : python3_required_modules
^

- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html

ninja: error: loading 'build.ninja': No such file or directory

- As far as Intel CI systems are concerned, there might be an issue with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.html

OS: WIN10-64
Target: x86_64-windows-clang
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (c:\program files\python37\python.exe)

buildtools\meson.build:32:2: ERROR: Problem encountered: missing python module: elftools

A full log can be found at C:\windows-dpdk\dpdk\build\meson-logs\meson-log.txt
ninja: Entering directory `build'
DPDK STV team

Signature · 1 line
David Marchand
#50 lore raw

Hello David,

Quoted 10 lines

- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478

Build machine cpu: x86_64
buildtools/meson.build:27:10: ERROR: Expecting endforeach got eol_cont.

script = \
          ^

For a block that started at 26,0
foreach module : python3_required_modules
^

My bad, meson 0.47.1 doesn't understand this indeed, fixed in v8.

Quoted 8 lines

- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html

ninja: error: loading 'build.ninja': No such file or directory

Makes sense.

Quoted 11 lines

- As far as Intel CI systems are concerned, there might be an issue with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.html

OS: WIN10-64
Target: x86_64-windows-clang
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (c:\program files\python37\python.exe)

buildtools\meson.build:32:2: ERROR: Problem encountered: missing python module: elftools

This is the check added in the last revision per Maxime's suggestion. Disabled the check on Windows in v8 until pmdinfogen supports COFF (and this will probably introduce an alternative dependency).

I'll watch new CI results, haven't been doing so closely because failures were expected for these patches. Tested v8 build on Linux->Linux, Linux->Windows, Windows->Windows (GCC & clang, without pyelftools installed).

#51 lore raw
In reply to David Marchand

Hi IOL team,

The following error persists in v8:

Quoted 8 lines

- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html

ninja: error: loading 'build.ninja': No such file or directory

https://inbox.dpdk.org/test-report/[email protected]/T/#u

The message isn't very informative. I suspect meson failed to configure, but there's no way to tell what went wrong. How can I get more information about this failure or specification of your environment to reproduce it?

#52 lore raw

v8 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.

---
Changes in v8:

* Fix syntax for meson 0.47.1 (CI).
* Disable configure-time check for pyelftools for Windows (CI).

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .travis.yml                        |   2 +-
 MAINTAINERS                        |   3 +-
 buildtools/gen-pmdinfo-cfile.sh    |   6 +-
 buildtools/meson.build             |  15 +
 buildtools/pmdinfogen.py           | 189 ++++++++++++
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 doc/guides/linux_gsg/sys_reqs.rst  |   6 +
 drivers/meson.build                |   2 +-
 meson.build                        |   1 -
 11 files changed, 217 insertions(+), 596 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
#53 lore raw

PATCH v8 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
new file mode 100755
index 000000000..0cca47ff1
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.28.0
#54 lore raw

PATCH v8 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)
.travis.yml +1 -1
index 5e12db23b..6744f6fc1 100644
@@ -11,7 +11,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +15 -0
index 04808dabc..dd4c0f640 100644
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index 6ecdc04aa..89c9f2570 100644
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index a5a6fed06..94403f7ac 100644
@@ -152,7 +152,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 61d9a4f5f..a6e8c09c2 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.28.0
Acked-by on v8 2/3
#55 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 6 lines · Dmitry Kozlyuk

Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Quoted 7 lines

Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---

From a meson viewpoint, this all looks ok. I'd hope in future that we could merge the shell script wrapper into the python script directly, rather than having two levels of scripting, but it's not something with any urgency.

Quoted 8 lines
 .travis.yml                       |  2 +-
 buildtools/gen-pmdinfo-cfile.sh   |  6 +++---
 buildtools/meson.build            | 15 +++++++++++++++
 doc/guides/linux_gsg/sys_reqs.rst |  6 ++++++
 drivers/meson.build               |  2 +-
 meson.build                       |  1 -
 6 files changed, 26 insertions(+), 6 deletions(-)
#56 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Few more comments below:

20/10/2020 19:44, Dmitry Kozlyuk:

buildtools/meson.build else · line 26

Quoted 3 lines
buildtools/meson.build +1 -0
+if host_machine.system() != 'windows'

You can use "is_windows".

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55

Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1 -0
+*   ``pyelftools`` (version 0.22+)

This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst

meson.build subdir('buildtools') · line 48

Quoted 3 lines
meson.build +0 -1
-subdir('buildtools/pmdinfogen')

This could be in patch 3 (removing the code).

#57 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 6 lines · Thomas Monjalon

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Will investigate and fix ASAP.

Quoted 8 lines

Few more comments below:

20/10/2020 19:44, Dmitry Kozlyuk:

buildtools/meson.build else · line 26

Quoted 3 lines
buildtools/meson.build +1 -0
+if host_machine.system() != 'windows'  

You can use "is_windows".

It's defined by config/meson.build, which is processed after buidtools/meson.build, because of the dependency, if swapped:

	config/x86/meson.build:6:1: ERROR: Unknown variable
	"binutils_avx512_check".
Quoted 5 lines

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55

Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1 -0
+*   ``pyelftools`` (version 0.22+)  

This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst

OK.

Quoted 5 lines

meson.build subdir('buildtools') · line 48

Quoted 3 lines
meson.build +0 -1
-subdir('buildtools/pmdinfogen')  

This could be in patch 3 (removing the code).

It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?

#58 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 19 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Will investigate and fix ASAP.

Quoted 8 lines

Few more comments below:

20/10/2020 19:44, Dmitry Kozlyuk:

buildtools/meson.build else · line 26

Quoted 3 lines
buildtools/meson.build +1 -0
+if host_machine.system() != 'windows'  

You can use "is_windows".

Quoted 5 lines

It's defined by config/meson.build, which is processed after buidtools/meson.build, because of the dependency, if swapped:

	config/x86/meson.build:6:1: ERROR: Unknown variable
	"binutils_avx512_check".

OK

Quoted 8 lines
Quoted 5 lines

doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55

Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1 -0
+*   ``pyelftools`` (version 0.22+)  

This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst

OK.

Quoted 8 lines
Quoted 5 lines

meson.build subdir('buildtools') · line 48

Quoted 3 lines
meson.build +0 -1
-subdir('buildtools/pmdinfogen')  

This could be in patch 3 (removing the code).

It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?

Just trying to find the best patch split.
If needed, OK to keep as is.

#59 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 10 lines · Thomas Monjalon

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 9 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

meson.build subdir('buildtools') · line 48

Quoted 3 lines
meson.build +0 -1
-subdir('buildtools/pmdinfogen')    

This could be in patch 3 (removing the code).

It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?

Just trying to find the best patch split.
If needed, OK to keep as is.

I even don't mind squashing all three commits into one. The split is done to ease the review.

#60 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

meson.build subdir('buildtools') · line 48

Quoted 3 lines
meson.build +0 -1
-subdir('buildtools/pmdinfogen')    

This could be in patch 3 (removing the code).

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?

Just trying to find the best patch split.
If needed, OK to keep as is.

I even don't mind squashing all three commits into one. The split is done to ease the review.

I think the split is good as is.

#61 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 20 lines · Thomas Monjalon

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

#62 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

#63 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 8 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

Send v10 with suppression.

Such check, however, *can* be implemented: at ABI check stage we have two install directories that dpdk-pmdinfo.py can inspect. Then a script can check that diff contains only additions, i.e. no device support being removed.

#64 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

In reply to Thomas Monjalon
Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 9 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

Ray K

#65 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 48 lines · Ray
Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 9 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

Quoted 7 lines

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

Ray K

Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?

#66 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 55 lines · Ray
Quoted 51 lines · Ray
Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 9 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Quoted 7 lines
Quoted 4 lines

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

Ray K

Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?

The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used by usertools/dpdk-pmdinfo.py and maybe some other parsing tool.

I guess a static symbol would be enough:

(quoted diff) +0 -0

b/buildtools/pmdinfogen/pmdinfogen.c
index a68d1ea999..14bf7d9f42 100644

buildtools/pmdinfogen/pmdinfogen.c +0 -0
@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info

*info, char *outfile)

        drv = info->drivers;

        while (drv) {
-               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
+               fprintf(ofd, "static const char %s_pmd_info[]
__attribute__((used)) = "
                        "\"PMD_INFO_STRING= {",
                        drv->name);
                fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);

We will need an exception for the v21 ABI though.

Signature · 1 line
David Marchand
#67 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 83 lines · David Marchand
Quoted 55 lines · Ray
Quoted 51 lines · Ray
Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 9 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Quoted 7 lines
Quoted 4 lines

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

Ray K

Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?

The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used by usertools/dpdk-pmdinfo.py and maybe some other parsing tool.

I guess a static symbol would be enough:

buildtools/pmdinfogen/pmdinfogen.c +1 -1
index a68d1ea999..14bf7d9f42 100644
@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
 *info, char *outfile)
        drv = info->drivers;
 
        while (drv) {
-               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
+               fprintf(ofd, "static const char %s_pmd_info[]

__attribute__((used)) = "

Quoted 7 lines
                        "\"PMD_INFO_STRING= {",
                        drv->name);
                fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);

We will need an exception for the v21 ABI though.

Good suggestion +1

#68 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

25/01/2021 11:46, Kinsella, Ray:

Quoted 25 lines · David Marchand
Quoted 24 lines

The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used by usertools/dpdk-pmdinfo.py and maybe some other parsing tool.

I guess a static symbol would be enough:

buildtools/pmdinfogen/pmdinfogen.c +1 -1
index a68d1ea999..14bf7d9f42 100644
@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
 *info, char *outfile)
        drv = info->drivers;
 
        while (drv) {
-               fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
+               fprintf(ofd, "static const char %s_pmd_info[]

__attribute__((used)) = "

                        "\"PMD_INFO_STRING= {",
                        drv->name);
                fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
Quoted 4 lines
Quoted 2 lines

We will need an exception for the v21 ABI though.

Good suggestion +1

Yes +1 for adding static on *_pmd_info

#69 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

In reply to Kinsella, Ray
Quoted 46 lines · Ray
Quoted 35 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 9 lines
Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

Quoted 5 lines

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place. As for content, checking that no PCI IDs are removed is out of scope for libabigail anyway. Technically we could fix _pmd_info size, but this still allows breaking changes to pass the check with no benefit.

#70 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 47 lines · Dmitry Kozlyuk
Quoted 46 lines · Ray
Quoted 44 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

Quoted 8 lines
Quoted 5 lines

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place.

Right - then is belongs in INTERNAL, I would say.

Quoted 3 lines

As for
content, checking that no PCI IDs are removed is out of scope for libabigail anyway.

Lets be clear PCI IDs - are _nothing_ to do with ABI.

Quoted 2 lines

Technically we could fix _pmd_info size, but this still allows breaking changes to pass the check with no benefit.

ABI changes or other, please explain?

#71 lore raw

Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen

Quoted 57 lines · Ray
Quoted 47 lines · Dmitry Kozlyuk
Quoted 46 lines · Ray
Quoted 44 lines · Thomas Monjalon

22/01/2021 23:24, Dmitry Kozlyuk:

Quoted 34 lines · Thomas Monjalon
Quoted 20 lines

22/01/2021 21:31, Dmitry Kozlyuk:

Quoted 10 lines · Thomas Monjalon
Quoted 9 lines

20/01/2021 08:23, Dmitry Kozlyuk:

Quoted 8 lines · Thomas Monjalon
Quoted 6 lines

This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:

'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Will investigate and fix ASAP.

Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.

You're right.

Quoted 7 lines
Quoted 2 lines

"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.

If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.

This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.

Quoted 4 lines

To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.

Quoted 8 lines
Quoted 5 lines

So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.

Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?

I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place.

Quoted 7 lines

Right - then is belongs in INTERNAL, I would say.

Quoted 3 lines

As for
content, checking that no PCI IDs are removed is out of scope for libabigail anyway.

Lets be clear PCI IDs - are _nothing_ to do with ABI.

Technically, yes, but they're referred to in abi_policy.rst, because DPDK behavior depends on them. Same issue as with as return values: no formats change, yet compatibility is broken.

Quoted 4 lines
Quoted 2 lines

Technically we could fix _pmd_info size, but this still allows breaking changes to pass the check with no benefit.

ABI changes or other, please explain?

Behavioral changes via PCI ID removal, see above.

#72 lore raw

PATCH v8 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index a3d192747..996f732fd 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.28.0
Tested-by on v8 3/3
#73 lore raw

Re: [dpdk-dev] [PATCH v8 3/3] pmdinfogen: remove C implementation

Quoted 637 lines · Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index a3d192747..996f732fd 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
deleted file mode 100644
index a68d1ea99..000000000
@@ -1,456 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <[email protected]>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <limits.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <libgen.h>
-
-#include <rte_common.h>
-#include "pmdinfogen.h"
-
-#ifdef RTE_ARCH_64
-#define ADDR_SIZE 64
-#else
-#define ADDR_SIZE 32
-#endif
-
-static int use_stdin, use_stdout;
-
-static const char *sym_name(struct elf_info *elf, Elf_Sym *sym)
-{
-	if (sym)
-		return elf->strtab + sym->st_name;
-	else
-		return "(unknown)";
-}
-
-static void *grab_file(const char *filename, unsigned long *size)
-{
-	struct stat st;
-	void *map = MAP_FAILED;
-	int fd = -1;
-
-	if (!use_stdin) {
-		fd = open(filename, O_RDONLY);
-		if (fd < 0)
-			return NULL;
-	} else {
-		/* from stdin, use a temporary file to mmap */
-		FILE *infile;
-		char buffer[1024];
-		int n;
-
-		infile = tmpfile();
-		if (infile == NULL) {
-			perror("tmpfile");
-			return NULL;
-		}
-		fd = dup(fileno(infile));
-		fclose(infile);
-		if (fd < 0)
-			return NULL;
-
-		n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		while (n > 0) {
-			if (write(fd, buffer, n) != n)
-				goto failed;
-			n = read(STDIN_FILENO, buffer, sizeof(buffer));
-		}
-	}
-
-	if (fstat(fd, &st))
-		goto failed;
-
-	*size = st.st_size;
-	map = mmap(NULL, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
-
-failed:
-	close(fd);
-	if (map == MAP_FAILED)
-		return NULL;
-	return map;
-}
-
-/**
-  * Return a copy of the next line in a mmap'ed file.
-  * spaces in the beginning of the line is trimmed away.
-  * Return a pointer to a static buffer.
-  **/
-static void release_file(void *file, unsigned long size)
-{
-	munmap(file, size);
-}
-
-
-static void *get_sym_value(struct elf_info *info, const Elf_Sym *sym)
-{
-	return RTE_PTR_ADD(info->hdr,
-		info->sechdrs[sym->st_shndx].sh_offset + sym->st_value);
-}
-
-static Elf_Sym *find_sym_in_symtab(struct elf_info *info,
-				   const char *name, Elf_Sym *last)
-{
-	Elf_Sym *idx;
-	if (last)
-		idx = last+1;
-	else
-		idx = info->symtab_start;
-
-	for (; idx < info->symtab_stop; idx++) {
-		const char *n = sym_name(info, idx);
-		if (!strncmp(n, name, strlen(name)))
-			return idx;
-	}
-	return NULL;
-}
-
-static int parse_elf(struct elf_info *info, const char *filename)
-{
-	unsigned int i;
-	Elf_Ehdr *hdr;
-	Elf_Shdr *sechdrs;
-	Elf_Sym  *sym;
-	int endian;
-	unsigned int symtab_idx = ~0U, symtab_shndx_idx = ~0U;
-
-	hdr = grab_file(filename, &info->size);
-	if (!hdr) {
-		perror(filename);
-		exit(1);
-	}
-	info->hdr = hdr;
-	if (info->size < sizeof(*hdr)) {
-		/* file too small, assume this is an empty .o file */
-		return 0;
-	}
-	/* Is this a valid ELF file? */
-	if ((hdr->e_ident[EI_MAG0] != ELFMAG0) ||
-	    (hdr->e_ident[EI_MAG1] != ELFMAG1) ||
-	    (hdr->e_ident[EI_MAG2] != ELFMAG2) ||
-	    (hdr->e_ident[EI_MAG3] != ELFMAG3)) {
-		/* Not an ELF file - silently ignore it */
-		return 0;
-	}
-
-	if (!hdr->e_ident[EI_DATA]) {
-		/* Unknown endian */
-		return 0;
-	}
-
-	endian = hdr->e_ident[EI_DATA];
-
-	/* Fix endianness in ELF header */
-	hdr->e_type      = TO_NATIVE(endian, 16, hdr->e_type);
-	hdr->e_machine   = TO_NATIVE(endian, 16, hdr->e_machine);
-	hdr->e_version   = TO_NATIVE(endian, 32, hdr->e_version);
-	hdr->e_entry     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_entry);
-	hdr->e_phoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_phoff);
-	hdr->e_shoff     = TO_NATIVE(endian, ADDR_SIZE, hdr->e_shoff);
-	hdr->e_flags     = TO_NATIVE(endian, 32, hdr->e_flags);
-	hdr->e_ehsize    = TO_NATIVE(endian, 16, hdr->e_ehsize);
-	hdr->e_phentsize = TO_NATIVE(endian, 16, hdr->e_phentsize);
-	hdr->e_phnum     = TO_NATIVE(endian, 16, hdr->e_phnum);
-	hdr->e_shentsize = TO_NATIVE(endian, 16, hdr->e_shentsize);
-	hdr->e_shnum     = TO_NATIVE(endian, 16, hdr->e_shnum);
-	hdr->e_shstrndx  = TO_NATIVE(endian, 16, hdr->e_shstrndx);
-
-	sechdrs = RTE_PTR_ADD(hdr, hdr->e_shoff);
-	info->sechdrs = sechdrs;
-
-	/* Check if file offset is correct */
-	if (hdr->e_shoff > info->size) {
-		fprintf(stderr, "section header offset=%lu in file '%s' "
-		      "is bigger than filesize=%lu\n",
-		      (unsigned long)hdr->e_shoff,
-		      filename, info->size);
-		return 0;
-	}
-
-	if (hdr->e_shnum == SHN_UNDEF) {
-		/*
-		 * There are more than 64k sections,
-		 * read count from .sh_size.
-		 */
-		info->num_sections =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[0].sh_size);
-	} else {
-		info->num_sections = hdr->e_shnum;
-	}
-	if (hdr->e_shstrndx == SHN_XINDEX)
-		info->secindex_strings =
-			TO_NATIVE(endian, 32, sechdrs[0].sh_link);
-	else
-		info->secindex_strings = hdr->e_shstrndx;
-
-	/* Fix endianness in section headers */
-	for (i = 0; i < info->num_sections; i++) {
-		sechdrs[i].sh_name      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_name);
-		sechdrs[i].sh_type      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_type);
-		sechdrs[i].sh_flags     =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_flags);
-		sechdrs[i].sh_addr      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addr);
-		sechdrs[i].sh_offset    =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_offset);
-		sechdrs[i].sh_size      =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_size);
-		sechdrs[i].sh_link      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_link);
-		sechdrs[i].sh_info      =
-			TO_NATIVE(endian, 32, sechdrs[i].sh_info);
-		sechdrs[i].sh_addralign =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_addralign);
-		sechdrs[i].sh_entsize   =
-			TO_NATIVE(endian, ADDR_SIZE, sechdrs[i].sh_entsize);
-	}
-	/* Find symbol table. */
-	for (i = 1; i < info->num_sections; i++) {
-		int nobits = sechdrs[i].sh_type == SHT_NOBITS;
-
-		if (!nobits && sechdrs[i].sh_offset > info->size) {
-			fprintf(stderr, "%s is truncated. "
-			      "sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
-			      filename, (unsigned long)sechdrs[i].sh_offset,
-			      sizeof(*hdr));
-			return 0;
-		}
-
-		if (sechdrs[i].sh_type == SHT_SYMTAB) {
-			unsigned int sh_link_idx;
-			symtab_idx = i;
-			info->symtab_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-			sh_link_idx = sechdrs[i].sh_link;
-			info->strtab       = RTE_PTR_ADD(hdr,
-				sechdrs[sh_link_idx].sh_offset);
-		}
-
-		/* 32bit section no. table? ("more than 64k sections") */
-		if (sechdrs[i].sh_type == SHT_SYMTAB_SHNDX) {
-			symtab_shndx_idx = i;
-			info->symtab_shndx_start = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset);
-			info->symtab_shndx_stop  = RTE_PTR_ADD(hdr,
-				sechdrs[i].sh_offset + sechdrs[i].sh_size);
-		}
-	}
-	if (!info->symtab_start)
-		fprintf(stderr, "%s has no symtab?\n", filename);
-	else {
-		/* Fix endianness in symbols */
-		for (sym = info->symtab_start; sym < info->symtab_stop; sym++) {
-			sym->st_shndx = TO_NATIVE(endian, 16, sym->st_shndx);
-			sym->st_name  = TO_NATIVE(endian, 32, sym->st_name);
-			sym->st_value = TO_NATIVE(endian, ADDR_SIZE, sym->st_value);
-			sym->st_size  = TO_NATIVE(endian, ADDR_SIZE, sym->st_size);
-		}
-	}
-
-	if (symtab_shndx_idx != ~0U) {
-		Elf32_Word *p;
-		if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link)
-			fprintf(stderr,
-			      "%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n",
-			      filename, sechdrs[symtab_shndx_idx].sh_link,
-			      symtab_idx);
-		/* Fix endianness */
-		for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop;
-		     p++)
-			*p = TO_NATIVE(endian, 32, *p);
-	}
-
-	return 1;
-}
-
-static void parse_elf_finish(struct elf_info *info)
-{
-	struct pmd_driver *tmp, *idx = info->drivers;
-	release_file(info->hdr, info->size);
-	while (idx) {
-		tmp = idx->next;
-		free(idx);
-		idx = tmp;
-	}
-}
-
-struct opt_tag {
-	const char *suffix;
-	const char *json_id;
-};
-
-static const struct opt_tag opt_tags[] = {
-	{"_param_string_export", "params"},
-	{"_kmod_dep_export", "kmod"},
-};
-
-static int complete_pmd_entry(struct elf_info *info, struct pmd_driver *drv)
-{
-	const char *tname;
-	int i;
-	char tmpsymname[128];
-	Elf_Sym *tmpsym;
-
-	drv->name = get_sym_value(info, drv->name_sym);
-
-	for (i = 0; i < PMD_OPT_MAX; i++) {
-		memset(tmpsymname, 0, 128);
-		sprintf(tmpsymname, "__%s%s", drv->name, opt_tags[i].suffix);
-		tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-		if (!tmpsym)
-			continue;
-		drv->opt_vals[i] = get_sym_value(info, tmpsym);
-	}
-
-	memset(tmpsymname, 0, 128);
-	sprintf(tmpsymname, "__%s_pci_tbl_export", drv->name);
-
-	tmpsym = find_sym_in_symtab(info, tmpsymname, NULL);
-
-
-	/*
-	 * If this returns NULL, then this is a PMD_VDEV, because
-	 * it has no pci table reference
-	 */
-	if (!tmpsym) {
-		drv->pci_tbl = NULL;
-		return 0;
-	}
-
-	tname = get_sym_value(info, tmpsym);
-	tmpsym = find_sym_in_symtab(info, tname, NULL);
-	if (!tmpsym)
-		return -ENOENT;
-
-	drv->pci_tbl = (struct rte_pci_id *)get_sym_value(info, tmpsym);
-	if (!drv->pci_tbl)
-		return -ENOENT;
-
-	return 0;
-}
-
-static int locate_pmd_entries(struct elf_info *info)
-{
-	Elf_Sym *last = NULL;
-	struct pmd_driver *new;
-
-	info->drivers = NULL;
-
-	do {
-		new = calloc(sizeof(struct pmd_driver), 1);
-		if (new == NULL) {
-			fprintf(stderr, "Failed to calloc memory\n");
-			return -1;
-		}
-		new->name_sym = find_sym_in_symtab(info, "this_pmd_name", last);
-		last = new->name_sym;
-		if (!new->name_sym)
-			free(new);
-		else {
-			if (complete_pmd_entry(info, new)) {
-				fprintf(stderr,
-					"Failed to complete pmd entry\n");
-				free(new);
-			} else {
-				new->next = info->drivers;
-				info->drivers = new;
-			}
-		}
-	} while (last);
-
-	return 0;
-}
-
-static void output_pmd_info_string(struct elf_info *info, char *outfile)
-{
-	FILE *ofd;
-	struct pmd_driver *drv;
-	struct rte_pci_id *pci_ids;
-	int idx = 0;
-
-	if (use_stdout)
-		ofd = stdout;
-	else {
-		ofd = fopen(outfile, "w+");
-		if (!ofd) {
-			fprintf(stderr, "Unable to open output file\n");
-			return;
-		}
-	}
-
-	drv = info->drivers;
-
-	while (drv) {
-		fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
-			"\"PMD_INFO_STRING= {",
-			drv->name);
-		fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
-
-		for (idx = 0; idx < PMD_OPT_MAX; idx++) {
-			if (drv->opt_vals[idx])
-				fprintf(ofd, "\\\"%s\\\" : \\\"%s\\\", ",
-					opt_tags[idx].json_id,
-					drv->opt_vals[idx]);
-		}
-
-		pci_ids = drv->pci_tbl;
-		fprintf(ofd, "\\\"pci_ids\\\" : [");
-
-		while (pci_ids && pci_ids->device_id) {
-			fprintf(ofd, "[%d, %d, %d, %d]",
-				pci_ids->vendor_id, pci_ids->device_id,
-				pci_ids->subsystem_vendor_id,
-				pci_ids->subsystem_device_id);
-			pci_ids++;
-			if (pci_ids->device_id)
-				fprintf(ofd, ",");
-			else
-				fprintf(ofd, " ");
-		}
-		fprintf(ofd, "]}\";\n");
-		drv = drv->next;
-	}
-
-	fclose(ofd);
-}
-
-int main(int argc, char **argv)
-{
-	struct elf_info info = {0};
-	int rc = 1;
-
-	if (argc < 3) {
-		fprintf(stderr,
-			"usage: %s <object file> <c output file>\n",
-			basename(argv[0]));
-		exit(127);
-	}
-	use_stdin = !strcmp(argv[1], "-");
-	use_stdout = !strcmp(argv[2], "-");
-	parse_elf(&info, argv[1]);
-
-	if (locate_pmd_entries(&info) < 0)
-		exit(1);
-
-	if (info.drivers) {
-		output_pmd_info_string(&info, argv[2]);
-		rc = 0;
-	} else {
-		fprintf(stderr, "No drivers registered\n");
-	}
-
-	parse_elf_finish(&info);
-	exit(rc);
-}
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
deleted file mode 100644
index 38fd3ddcc..000000000
@@ -1,119 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0
- * Postprocess pmd object files to export hw support
- *
- * Copyright 2016 Neil Horman <[email protected]>
- * Based in part on modpost.c from the linux kernel
- */
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <string.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#ifdef __linux__
-#include <endian.h>
-#else
-#include <sys/endian.h>
-#endif
-#include <fcntl.h>
-#include <unistd.h>
-#include <elf.h>
-#include <rte_pci.h>
-
-/* On BSD-alike OSes elf.h defines these according to host's word size */
-#undef ELF_ST_BIND
-#undef ELF_ST_TYPE
-#undef ELF_R_SYM
-#undef ELF_R_TYPE
-
-/*
- * Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit
- * flavors in elf.h.  This makes our code a bit more generic between arches
- * and allows us to support 32 bit code in the future should we ever want to
- */
-#ifdef RTE_ARCH_64
-#define Elf_Ehdr    Elf64_Ehdr
-#define Elf_Shdr    Elf64_Shdr
-#define Elf_Sym     Elf64_Sym
-#define Elf_Addr    Elf64_Addr
-#define Elf_Sword   Elf64_Sxword
-#define Elf_Section Elf64_Half
-#define ELF_ST_BIND ELF64_ST_BIND
-#define ELF_ST_TYPE ELF64_ST_TYPE
-
-#define Elf_Rel     Elf64_Rel
-#define Elf_Rela    Elf64_Rela
-#define ELF_R_SYM   ELF64_R_SYM
-#define ELF_R_TYPE  ELF64_R_TYPE
-#else
-#define Elf_Ehdr    Elf32_Ehdr
-#define Elf_Shdr    Elf32_Shdr
-#define Elf_Sym     Elf32_Sym
-#define Elf_Addr    Elf32_Addr
-#define Elf_Sword   Elf32_Sxword
-#define Elf_Section Elf32_Half
-#define ELF_ST_BIND ELF32_ST_BIND
-#define ELF_ST_TYPE ELF32_ST_TYPE
-
-#define Elf_Rel     Elf32_Rel
-#define Elf_Rela    Elf32_Rela
-#define ELF_R_SYM   ELF32_R_SYM
-#define ELF_R_TYPE  ELF32_R_TYPE
-#endif
-
-
-/*
- * Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro
- * below.  We do this because the values passed to TO_NATIVE may themselves be
- * macros and need both macros here to get expanded.  Specifically its the width
- * variable we are concerned with, because it needs to get expanded prior to
- * string concatenation
- */
-#define CONVERT_NATIVE(fend, width, x) ({ \
-typeof(x) ___x; \
-if ((fend) == ELFDATA2LSB) \
-	___x = le##width##toh(x); \
-else \
-	___x = be##width##toh(x); \
-___x; \
-})
-
-#define TO_NATIVE(fend, width, x) CONVERT_NATIVE(fend, width, x)
-
-enum opt_params {
-	PMD_PARAM_STRING = 0,
-	PMD_KMOD_DEP,
-	PMD_OPT_MAX
-};
-
-struct pmd_driver {
-	Elf_Sym *name_sym;
-	const char *name;
-	struct rte_pci_id *pci_tbl;
-	struct pmd_driver *next;
-
-	const char *opt_vals[PMD_OPT_MAX];
-};
-
-struct elf_info {
-	unsigned long size;
-	Elf_Ehdr     *hdr;
-	Elf_Shdr     *sechdrs;
-	Elf_Sym      *symtab_start;
-	Elf_Sym      *symtab_stop;
-	char         *strtab;
-
-	/* support for 32bit section numbers */
-
-	unsigned int num_sections; /* max_secindex + 1 */
-	unsigned int secindex_strings;
-	/* if Nth symbol table entry has .st_shndx = SHN_XINDEX,
-	 * take shndx from symtab_shndx_start[N] instead
-	 */
-	Elf32_Word   *symtab_shndx_start;
-	Elf32_Word   *symtab_shndx_stop;
-
-	struct pmd_driver *drivers;
-};
Signature · 1 line
2.28.0
#74 lore raw

v9 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Canonical JSON formatting of generated strings raises ABI warnings. There are no meaningful changes, which can be checked by comparing output of pmdinfo before and after the patch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

---
Changes in v9:

* Document pyelftools requirement for FreeBSD (Thomas).
* Add pyelftools to GitHub workflow.

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .github/workflows/build.yml           |   4 +-
 .travis.yml                           |   2 +-
 MAINTAINERS                           |   3 +-
 buildtools/gen-pmdinfo-cfile.sh       |   6 +-
 buildtools/meson.build                |  15 +
 buildtools/pmdinfogen.py              | 189 +++++++++++
 buildtools/pmdinfogen/meson.build     |  14 -
 buildtools/pmdinfogen/pmdinfogen.c    | 456 --------------------------
 buildtools/pmdinfogen/pmdinfogen.h    | 119 -------
 doc/guides/freebsd_gsg/build_dpdk.rst |   3 +-
 doc/guides/linux_gsg/sys_reqs.rst     |   6 +
 drivers/meson.build                   |   2 +-
 meson.build                           |   1 -
 13 files changed, 221 insertions(+), 599 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.29.2
#75 lore raw

PATCH v9 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
new file mode 100755
index 000000000..0cca47ff1
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.29.2
#76 lore raw

PATCH v9 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.

Add configure-time check for elftools Python module for Unix hosts.

Add pyelftools to CI configuration and build requirements for Linux and FreeBSD. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.

---
 .github/workflows/build.yml           |  4 ++--
 .travis.yml                           |  2 +-
 buildtools/gen-pmdinfo-cfile.sh       |  6 +++---
 buildtools/meson.build                | 15 +++++++++++++++
 doc/guides/freebsd_gsg/build_dpdk.rst |  3 ++-
 doc/guides/linux_gsg/sys_reqs.rst     |  6 ++++++
 drivers/meson.build                   |  2 +-
 meson.build                           |  1 -
 8 files changed, 30 insertions(+), 9 deletions(-)
.github/workflows/build.yml +2 -2
index 0b72df0eb..a5b579add 100644
@@ -91,8 +91,8 @@ jobs:
       run: sudo apt update
     - name: Install packages
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
-        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
-        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
+        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -1
index 5aa7ad49f..4391af1d5 100644
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +15 -0
index 04808dabc..dd4c0f640 100644
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
doc/guides/freebsd_gsg/build_dpdk.rst +2 -1
index e3005a7f3..bed353473 100644
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK:
 * meson
 * ninja
 * pkgconf
+* py37-pyelftools
 
 These can be installed using (as root)::
 
-  pkg install meson pkgconf
+  pkg install meson pkgconf py37-pyelftools
 
 To compile the required kernel modules for memory management and working
 with physical NIC devices, the kernel sources for FreeBSD also
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index be714adf2..a05b5bd81 100644
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index 77f65fa90..ff5cdb952 100644
@@ -132,7 +132,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 45d974cd2..2b9c37eb4 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.29.2
#77 lore raw

PATCH v9 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index aa973a396..65f6fffd1 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.29.2
#78 lore raw

v10 cover pmdinfogen: rewrite in Python

In reply to Dmitry Kozlyuk

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Identity of generated PMD information is checked by comparing output of pmdinfo before and after the patch:

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

---
Changes in v10:

* Suppress ABI warnings for generated strings (Thomas).

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

 .github/workflows/build.yml           |   4 +-
 .travis.yml                           |   2 +-
 MAINTAINERS                           |   3 +-
 buildtools/gen-pmdinfo-cfile.sh       |   6 +-
 buildtools/meson.build                |  15 +
 buildtools/pmdinfogen.py              | 189 +++++++++++
 buildtools/pmdinfogen/meson.build     |  14 -
 buildtools/pmdinfogen/pmdinfogen.c    | 456 --------------------------
 buildtools/pmdinfogen/pmdinfogen.h    | 119 -------
 devtools/libabigail.abignore          |   4 +
 doc/guides/freebsd_gsg/build_dpdk.rst |   3 +-
 doc/guides/linux_gsg/sys_reqs.rst     |   6 +
 drivers/meson.build                   |   2 +-
 meson.build                           |   1 -
 14 files changed, 225 insertions(+), 599 deletions(-)
 create mode 100755 buildtools/pmdinfogen.py
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.29.2
#79 lore raw

PATCH v10 1/3 pmdinfogen: add Python implementation

Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.

---
 buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 189 insertions(+)
 create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
new file mode 100755
index 000000000..0cca47ff1
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2016 Neil Horman <[email protected]>
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import argparse
+import ctypes
+import json
+import sys
+import tempfile
+
+from elftools.elf.elffile import ELFFile
+from elftools.elf.sections import SymbolTableSection
+
+
+class ELFSymbol:
+    def __init__(self, image, symbol):
+        self._image = image
+        self._symbol = symbol
+
+    @property
+    def size(self):
+        return self._symbol["st_size"]
+
+    @property
+    def value(self):
+        data = self._image.get_section_data(self._symbol["st_shndx"])
+        base = self._symbol["st_value"]
+        return data[base:base + self.size]
+
+    @property
+    def string_value(self):
+        value = self.value
+        return value[:-1].decode() if value else ""
+
+
+class ELFImage:
+    def __init__(self, data):
+        self._image = ELFFile(data)
+        self._symtab = self._image.get_section_by_name(".symtab")
+        if not isinstance(self._symtab, SymbolTableSection):
+            raise Exception(".symtab section is not a symbol table")
+
+    @property
+    def is_big_endian(self):
+        return not self._image.little_endian
+
+    def get_section_data(self, name):
+        return self._image.get_section(name).data()
+
+    def find_by_name(self, name):
+        symbol = self._symtab.get_symbol_by_name(name)
+        return ELFSymbol(self, symbol[0]) if symbol else None
+
+    def find_by_prefix(self, prefix):
+        for i in range(self._symtab.num_symbols()):
+            symbol = self._symtab.get_symbol(i)
+            if symbol.name.startswith(prefix):
+                yield ELFSymbol(self, symbol)
+
+
+def define_rte_pci_id(is_big_endian):
+    base_type = ctypes.LittleEndianStructure
+    if is_big_endian:
+        base_type = ctypes.BigEndianStructure
+
+    class rte_pci_id(base_type):
+        _pack_ = True
+        _fields_ = [
+            ("class_id", ctypes.c_uint32),
+            ("vendor_id", ctypes.c_uint16),
+            ("device_id", ctypes.c_uint16),
+            ("subsystem_vendor_id", ctypes.c_uint16),
+            ("subsystem_device_id", ctypes.c_uint16),
+        ]
+
+    return rte_pci_id
+
+
+class Driver:
+    OPTIONS = [
+        ("params", "_param_string_export"),
+        ("kmod", "_kmod_dep_export"),
+    ]
+
+    def __init__(self, name, options):
+        self.name = name
+        for key, value in options.items():
+            setattr(self, key, value)
+        self.pci_ids = []
+
+    @classmethod
+    def load(cls, image, symbol):
+        name = symbol.string_value
+
+        options = {}
+        for key, suffix in cls.OPTIONS:
+            option_symbol = image.find_by_name("__%s%s" % (name, suffix))
+            if option_symbol:
+                value = option_symbol.string_value
+                options[key] = value
+
+        driver = cls(name, options)
+
+        pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name)
+        if pci_table_name_symbol:
+            driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol)
+
+        return driver
+
+    @staticmethod
+    def _load_pci_ids(image, table_name_symbol):
+        table_name = table_name_symbol.string_value
+        table_symbol = image.find_by_name(table_name)
+        if not table_symbol:
+            raise Exception("PCI table declared but not defined: %d" % table_name)
+
+        rte_pci_id = define_rte_pci_id(image.is_big_endian)
+
+        pci_id_size = ctypes.sizeof(rte_pci_id)
+        pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size)
+        pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value)
+        result = []
+        for pci_id in pci_ids:
+            if not pci_id.device_id:
+                break
+            result.append([
+                pci_id.vendor_id,
+                pci_id.device_id,
+                pci_id.subsystem_vendor_id,
+                pci_id.subsystem_device_id,
+                ])
+        return result
+
+    def dump(self, file):
+        dumped = json.dumps(self.__dict__)
+        escaped = dumped.replace('"', '\\"')
+        print(
+            'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";'
+            % (self.name, escaped),
+            file=file,
+        )
+
+
+def load_drivers(image):
+    drivers = []
+    for symbol in image.find_by_prefix("this_pmd_name"):
+        drivers.append(Driver.load(image, symbol))
+    return drivers
+
+
+def dump_drivers(drivers, file):
+    # Keep legacy order of definitions.
+    for driver in reversed(drivers):
+        driver.dump(file)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument("output", help="output C file path or '-' for stdout")
+    return parser.parse_args()
+
+
+def open_input(path):
+    if path == "-":
+        temp = tempfile.TemporaryFile()
+        temp.write(sys.stdin.buffer.read())
+        return temp
+    return open(path, "rb")
+
+
+def open_output(path):
+    if path == "-":
+        return sys.stdout
+    return open(path, "w")
+
+
+def main():
+    args = parse_args()
+    infile = open_input(args.input)
+    image = ELFImage(infile)
+    drivers = load_drivers(image)
+    output = open_output(args.output)
+    dump_drivers(drivers, output)
+
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.29.2
#80 lore raw

PATCH v10 2/3 build: use Python pmdinfogen

In reply to Dmitry Kozlyuk

Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen. Add configure-time check for elftools Python module for Unix hosts.

Add pyelftools to CI configuration and build requirements for Linux and FreeBSD. Windows targets are not currently using pmdinfogen.

Suppress ABI warnings about generated PMD information strings.

---
 .github/workflows/build.yml           |  4 ++--
 .travis.yml                           |  2 +-
 buildtools/gen-pmdinfo-cfile.sh       |  6 +++---
 buildtools/meson.build                | 15 +++++++++++++++
 devtools/libabigail.abignore          |  4 ++++
 doc/guides/freebsd_gsg/build_dpdk.rst |  3 ++-
 doc/guides/linux_gsg/sys_reqs.rst     |  6 ++++++
 drivers/meson.build                   |  2 +-
 meson.build                           |  1 -
 9 files changed, 34 insertions(+), 9 deletions(-)
.github/workflows/build.yml +2 -2
index 0b72df0eb..a5b579add 100644
@@ -91,8 +91,8 @@ jobs:
       run: sudo apt update
     - name: Install packages
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
-        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
-        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
+        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -1
index 5aa7ad49f..4391af1d5 100644
@@ -14,7 +14,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
 
 _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
index 43059cf36..109ee461e 100755
@@ -4,11 +4,11 @@
 
 arfile=$1
 output=$2
-pmdinfogen=$3
+shift 2
+pmdinfogen=$*
 
 # The generated file must not be empty if compiled in pedantic mode
 echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
 for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null
+	ar p $arfile $ofile | $pmdinfogen - - >> $output
 done
-exit 0
buildtools/meson.build +15 -0
index 04808dabc..dd4c0f640 100644
@@ -17,3 +17,18 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+pmdinfogen = py3 + files('pmdinfogen.py')
+
+# TODO: starting from Meson 0.51.0 use
+# 	python3 = import('python').find_installation('python',
+#		modules : python3_required_modules)
+python3_required_modules = []
+if host_machine.system() != 'windows'
+	python3_required_modules = ['elftools']
+endif
+foreach module : python3_required_modules
+	script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)'
+	if run_command(py3, '-c', script.format(module)).returncode() != 0
+		error('missing python module: @0@'.format(module))
+	endif
+endforeach
devtools/libabigail.abignore +4 -0
index 1dc84fa74..05afccc1a 100644
@@ -16,3 +16,7 @@
 [suppress_type]
         name = rte_cryptodev
         has_data_member_inserted_between = {0, 1023}
+
+; Ignore all changes in generated PMD information strings.
+[suppress_variable]
+        name_regex = _pmd_info$
doc/guides/freebsd_gsg/build_dpdk.rst +2 -1
index e3005a7f3..bed353473 100644
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK:
 * meson
 * ninja
 * pkgconf
+* py37-pyelftools
 
 These can be installed using (as root)::
 
-  pkg install meson pkgconf
+  pkg install meson pkgconf py37-pyelftools
 
 To compile the required kernel modules for memory management and working
 with physical NIC devices, the kernel sources for FreeBSD also
doc/guides/linux_gsg/sys_reqs.rst +6 -0
index be714adf2..a05b5bd81 100644
@@ -52,6 +52,12 @@ Compilation of the DPDK
     * If the packaged version is below the minimum version, the latest versions
       can be installed from Python's "pip" repository: ``pip3 install meson ninja``
 
+*   ``pyelftools`` (version 0.22+)
+
+    * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools``
+
+    * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools``
+
 *   Library for handling NUMA (Non Uniform Memory Access).
 
     * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
index 77f65fa90..ff5cdb952 100644
@@ -132,7 +132,7 @@ foreach subpath:subdirs
 						command: [pmdinfo, tmp_lib.full_path(),
 							'@OUTPUT@', pmdinfogen],
 						output: out_filename,
-						depends: [pmdinfogen, tmp_lib])
+						depends: [tmp_lib])
 			endif
 
 			# now build the static driver
meson.build +0 -1
index 45d974cd2..2b9c37eb4 100644
@@ -45,7 +45,6 @@ subdir('buildtools')
 subdir('config')
 
 # build libs and drivers
-subdir('buildtools/pmdinfogen')
 subdir('lib')
 subdir('drivers')
 
Signature · 1 line
2.29.2
#81 lore raw

Re: [dpdk-dev] [PATCH v10 2/3] build: use Python pmdinfogen

24/01/2021 21:51, Dmitry Kozlyuk:

devtools/libabigail.abignore line 22

Quoted 5 lines
devtools/libabigail.abignore +3 -0
+; Ignore all changes in generated PMD information strings.
+[suppress_variable]
+        name_regex = _pmd_info$

I will fix this rule as "name_regexp" so it works.

#82 lore raw

PATCH v10 3/3 pmdinfogen: remove C implementation

In reply to Dmitry Kozlyuk

Delete the files no longer used in build process.
Add myself as maintainer of new implementation.

---
 MAINTAINERS                        |   3 +-
 buildtools/pmdinfogen/meson.build  |  14 -
 buildtools/pmdinfogen/pmdinfogen.c | 456 -----------------------------
 buildtools/pmdinfogen/pmdinfogen.h | 119 --------
 4 files changed, 2 insertions(+), 590 deletions(-)
 delete mode 100644 buildtools/pmdinfogen/meson.build
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c
 delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
index aa973a396..65f6fffd1 100644
@@ -133,7 +133,8 @@ F: lib/*/*.map
 
 Driver information
 M: Neil Horman <[email protected]>
-F: buildtools/pmdinfogen/
+M: Dmitry Kozlyuk <[email protected]>
+F: buildtools/pmdinfogen.py
 F: usertools/dpdk-pmdinfo.py
 F: doc/guides/tools/pmdinfo.rst
 
buildtools/pmdinfogen/meson.build deleted +0 -14
deleted file mode 100644
index 670528fac..000000000
@@ -1,14 +0,0 @@
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-if is_windows
-	subdir_done()
-endif
-
-pmdinfogen_inc = [global_inc]
-pmdinfogen_inc += include_directories('../../lib/librte_eal/include')
-pmdinfogen_inc += include_directories('../../lib/librte_pci')
-pmdinfogen = executable('pmdinfogen',
-	'pmdinfogen.c',
-	include_directories: pmdinfogen_inc,
-	native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456

456 lines not on this page · raw message

buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119

119 lines not on this page · raw message

Signature · 1 line
2.29.2
#83 lore raw
In reply to Dmitry Kozlyuk
Quoted 4 lines

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

Applied, thanks.

Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:

	http://mails.dpdk.org/archives/ci/2021-January/000939.html
	http://mails.dpdk.org/archives/dev/2021-January/197814.html

If not already done, please UPDATE your environment! Thanks

#84 lore raw

Hi Thomas,

We have updated all UNH-IOL environments to include pyelftools.

Thanks,
Brandon

Quoted 16 lines · Thomas Monjalon
Quoted 5 lines

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

Applied, thanks.

Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:

        http://mails.dpdk.org/archives/ci/2021-January/000939.html
        http://mails.dpdk.org/archives/dev/2021-January/197814.html

If not already done, please UPDATE your environment! Thanks

Signature · 9 lines
Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

[email protected]

www.iol.unh.edu
#85 lore raw
Quoted 8 lines

Subject: Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

External email: Use caution opening links or attachments

Hi Thomas,

We have updated all UNH-IOL environments to include pyelftools.

FreeBSD compilation test is still failing on the missing dependency:

buildtools/meson.build:45:2: ERROR: Problem encountered: missing python module: elftools

Quoted 66 lines

Thanks,
Brandon

Quoted 11 lines · Thomas Monjalon
Quoted 5 lines

Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation

Applied, thanks.

Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:

https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.
dpdk.org%2Farchives%2Fci%2F2021- January%2F000939.html&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C70 5c7ce391d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a %7C0%7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIj oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 000&amp;sdata=Lei7o0Mtgsb3WZJ14lGolduVJK3fA7EbYuw8ojVCb3g%3D&a mp;reserved=0

Quoted 2 lines

https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails

Quoted 1 line

.dpdk.org%2Farchives%2Fdev%2F2021-

January%2F197814.html&amp;data=04%7C

Quoted 1 line

01%7Ctalshn%40nvidia.com%7C705c7ce391d1429bc94508d8c14b9880%7C430 83d15

Quoted 1 line

727340c1b7db39efd9ccc17a%7C0%7C0%7C637471877725119466%7CUnknow n%7CTWFp

Quoted 1 line

bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC I6Mn

Quoted 1 line

0%3D%7C1000&amp;sdata=74Zp6%2B6YB8OCVfo%2F0Gb7C9C43QWAo1YD4 AUANneBGdA%

Quoted 5 lines

3D&amp;reserved=0

If not already done, please UPDATE your environment! Thanks

--

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

[email protected]

https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
.iol.unh.edu%2F&amp;data=04%7C01%7Ctalshn%40nvidia.com%7C705c7ce3 91d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a%7C0% 7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4 wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a mp;sdata=2Fc4epSpG%2FhLFdxj9D1kVoafaxlfUhaZvpEm4xwHFd4%3D&amp; reserved=0

#86 lore raw
In reply to Dmitry Kozlyuk

Hi,

Quoted 20 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Dmitry Kozlyuk
Sent: Sunday, January 24, 2021 10:52 PM
To: [email protected]
Cc: Maxime Coquelin <[email protected]>; Bruce Richardson <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Dmitry Kozlyuk <[email protected]>; Neil Horman <[email protected]>; Jie Zhou <[email protected]>
Subject: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python

This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:

* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.

Identity of generated PMD information is checked by comparing output of pmdinfo before and after the patch:

Quoted 6 lines

find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py

---

Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED: drivers/rte_common_cpt.pmd.c
/usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf Traceback (most recent call last):
File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
[ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
"""

I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

Thanks,
Ali

#87 lore raw

Hi Ali,

Quoted 13 lines

Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED: drivers/rte_common_cpt.pmd.c
/usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf Traceback (most recent call last):
File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
[ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
"""

Quoted 1 line

I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).

If fixed the error you encountered and also discovered a bunch of other issues, all of them hopefully fixed in the following series:

http://patchwork.dpdk.org/project/dpdk/list/?series=14938