pmdinfogen: rewrite in Python
cover pmdinfogen: rewrite in Python
This is a PoC rewrite of pmdinfogen in Python with missing bits described below and in commits. Community input is desired.
Pros:
1. Simpler build process without host apps.
2. Less build requirements (host libelf).
3. Easier debugging and maintenance with a high-level language.
4. Easier porting on Windows (only add new object format).
Cons:
1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).
2. struct rte_pci_id must be synchronized with header file (it's a few lines that never change).
There are no built-in or widely used Python libraries for ELF or COFF. Some ELF-parsing libraries exist on PyPI, but they're not very handy for the task and their installation would complicate build requirements. Thus, elf.py implements its own parsing.
COFF support is underway, it's just not included in this RFC. Amount of code is similar to elf.py.
Build is only tested on Linux x64_64.
If the community deems this RFC worth finishing, there are a few opens:
1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?
2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.
3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?
---
Dmitry Kozlyuk (2):
pmdinfogen: prototype in Python
build: use Python pmdinfogen
buildtools/elf.py | 194 +++++++++++++++++++++++++++++++++++++++ buildtools/meson.build | 3 +- buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++ drivers/meson.build | 2 +- 4 files changed, 340 insertions(+), 3 deletions(-) create mode 100644 buildtools/elf.py create mode 100755 buildtools/pmdinfogen.py
Signature · 1 line
2.25.4
RFC PATCH 1/2 pmdinfogen: prototype in Python
* No user-friendly error handling and no bounds checking yet.
* No support for >65K sections case (is it needed?).
* The order of definitions is reversed.
--- buildtools/elf.py | 194 +++++++++++++++++++++++++++++++++++++++ buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++ 2 files changed, 338 insertions(+) create mode 100644 buildtools/elf.py create mode 100755 buildtools/pmdinfogen.py
buildtools/elf.py new file +194 -0
@@ -0,0 +1,194 @@ +import ctypes + + +class ElfIdent(ctypes.Structure): + _pack_ = True + _fields_ = [ + ("magic", ctypes.c_char * 4), + ("class_", ctypes.c_uint8), + ("data", ctypes.c_uint8), + ("version", ctypes.c_uint8), + ("os_abi", ctypes.c_uint8), + ("abi_version", ctypes.c_uint8), + ("pad", ctypes.c_uint8 * 7), + ] + + @property + def is_magic_ok(self): + return self.magic.value == b"\x7fELF" + + @property + def is_32bit(self): + return self.class_ == 1 + + @property + def is_big_endian(self): + return self.data == 2 + + def define_structures(self): + base_type = ctypes.LittleEndianStructure + if self.is_big_endian: + base = ctypes.BigEndianStructure + + size_type = ctypes.c_uint64 + if self.is_32bit: + size_type = ctypes.c_uint32 + + class FileHeader(base_type): + _pack_ = True + _fields_ = [ + ("e_ident", ElfIdent), + ("e_type", ctypes.c_uint16), + ("e_machine", ctypes.c_uint16), + ("e_version", ctypes.c_uint32), + ("e_entry", size_type), + ("e_phoff", size_type), + ("e_shoff", size_type), + ("e_flags", ctypes.c_uint32), + ("e_ehsize", ctypes.c_uint16), + ("e_phentsize", ctypes.c_uint16), + ("e_phnum", ctypes.c_uint16), + ("e_shentsize", ctypes.c_uint16), + ("e_shnum", ctypes.c_uint16), + ("e_shstrndx", ctypes.c_uint16), + ] + + class SectionHeader(base_type): + _pack_ = True + _fields_ = [ + ("sh_name", ctypes.c_uint32), + ("sh_type", ctypes.c_uint32), + ("sh_flags", size_type), + ("sh_addr", size_type), + ("sh_offset", size_type), + ("sh_size", size_type), + ("sh_link", ctypes.c_uint32), + ("sh_info", ctypes.c_uint32), + ("sh_addralign", size_type), + ("sh_entsize", size_type), + ] + + class Symbol32(base_type): + _pack_ = True + _fields_ = [ + ("st_name", ctypes.c_uint32), + ("st_value", ctypes.c_uint32), + ("st_size", ctypes.c_uint32), + ("st_info", ctypes.c_uint8), + ("st_other", ctypes.c_uint8), + ("st_shndx", ctypes.c_uint16), + ] + + class Symbol64(base_type): + _pack_ = True + _fields_ = [ + ("st_name", ctypes.c_uint32), + ("st_info", ctypes.c_uint8), + ("st_other", ctypes.c_uint8), + ("st_shndx", ctypes.c_uint16), + ("st_value", ctypes.c_uint64), + ("st_size", ctypes.c_uint64), + ] + + Symbol = Symbol32 if self.is_32bit else Symbol64 + + return FileHeader, SectionHeader, Symbol + + +class Symbol: + def __init__(self, image, elf): + self._image = image + self._elf = elf + + + @property + def address(self): + base = self._image._sections[self._elf.st_shndx].sh_offset + offset = base + self._elf.st_value + memory = ctypes.c_char.from_buffer(self._image._data, offset) + return ctypes.addressof(memory) + + +class Image: + def __init__(self, data): + SHN_UNDEF = 0x0000 + SHN_XINDEX = 0xFFFF + + ident = ElfIdent.from_buffer(data) + ElfFileHeader, ElfSectionHeader, ElfSymbol = ident.define_structures() + + header = ElfFileHeader.from_buffer(data) + + if header.e_shnum == SHN_UNDEF: + section = ElfSectionHeader.from_buffer(data, header.e_shoff) + sections_num = section.sh_size + else: + sections_num = header.e_shnum + sections_desc = ElfSectionHeader * sections_num + sections = sections_desc.from_buffer(data, header.e_shoff) + + if header.e_shstrndx == SHN_XINDEX: + strings_index = sections[0].sh_link + else: + strings_index = header.e_shstrndx + + symtab, strtab = Image._find_symbol_table(data, sections, ElfSymbol) + + self._data = data + self._header = header + self._sections = sections + self._strings = sections[strings_index] + self._symtab = symtab + self._strtab = strtab + + + @staticmethod + def _find_symbol_table(data, sections, symbol_type): + SHT_SYMTAB = 2 + SHT_SYMTAB_SHNDX = 18 + + for section in sections: + if section.sh_type == SHT_SYMTAB: + symbol_count = section.sh_size // ctypes.sizeof(symbol_type) + symtab_desc = symbol_type * symbol_count + symtab_offset = section.sh_offset + symtab = symtab_desc.from_buffer(data, symtab_offset) + + strtab = sections[section.sh_link].sh_offset + + return symtab, strtab + + # TODO: SHT_SYMTAB_SHNDX? + + raise Exception('no symbol table') + + + @property + def is_big_endian(self): + return self._header.e_ident.is_big_endian + + + def find_symbol(self, name: str, start: Symbol = None): + name_size = len(name) + name_desc = ctypes.c_char * name_size + + i = 0 + if start: + table_address = ctypes.addressof(self._symtab) + start_address = ctypes.addressof(start._elf) + i = (start_address - table_address) // ctypes.sizeof(start._elf) + 1 + + while i < len(self._symtab): + symbol = self._symtab[i] + + name_offset = self._strtab + symbol.st_name + if name_offset + name_size > len(self._data): + break + + symbol_name = name_desc.from_buffer(self._data, name_offset).value + if symbol_name == name.encode(): + return Symbol(self, symbol) + + i += 1 + + return None
buildtools/pmdinfogen.py new file +144 -0
144 lines not on this page · raw message
Signature · 1 line
2.25.4
RFC PATCH 2/2 build: use Python pmdinfogen
In reply to Dmitry Kozlyuk* C pmdinfogen not removed.
* No Makefile support yet.
--- buildtools/meson.build | 3 +-- drivers/meson.build | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-)
buildtools/meson.build +1 -2
@@ -1,9 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017-2019 Intel Corporation -subdir('pmdinfogen') - pkgconf = find_program('pkg-config', 'pkgconf', required: false) +pmdinfogen = find_program('pmdinfogen.py') pmdinfo = find_program('gen-pmdinfo-cfile.sh') list_dir_globs = find_program('list-dir-globs.py') check_symbols = find_program('check-symbols.sh')
drivers/meson.build +1 -1
@@ -122,7 +122,7 @@ foreach class:dpdk_driver_classes command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) version_map = '@0@/@1@/@2@_version.map'.format( meson.current_source_dir(),
Signature · 1 line
2.25.4
Quoted 10 lines · Dmitry Kozlyuk
This is a PoC rewrite of pmdinfogen in Python with missing bits described below and in commits. Community input is desired.
Pros:
1. Simpler build process without host apps.
2. Less build requirements (host libelf).
3. Easier debugging and maintenance with a high-level language.
4. Easier porting on Windows (only add new object format).
Front matter: It seems reasonable to me to implement this in python, just for ease of maintenence.
Quoted 4 lines
Cons:
1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).
Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden
Quoted 3 lines
2. struct
rte_pci_idmust be synchronized with header file (it's a few lines that never change).
I think you can just use ctypes and the native python C interface to just import the rte_pci_id structure from the native dpdk header to avoid this, no?
Quoted 8 lines
There are no built-in or widely used Python libraries for ELF or COFF. Some ELF-parsing libraries exist on PyPI, but they're not very handy for the task and their installation would complicate build requirements. Thus,
elf.pyimplements its own parsing.COFF support is underway, it's just not included in this RFC. Amount of code is similar to
elf.py.
Quoted 7 lines
Build is only tested on Linux
x64_64.If the community deems this RFC worth finishing, there are a few opens:
1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?
Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.
Quoted 5 lines
2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.
I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.
Quoted 6 lines
3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?
I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?
Quoted 17 lines
---
Dmitry Kozlyuk (2):
pmdinfogen: prototype in Python
build: use Python pmdinfogenbuildtools/elf.py | 194 +++++++++++++++++++++++++++++++++++++++ buildtools/meson.build | 3 +- buildtools/pmdinfogen.py | 144 +++++++++++++++++++++++++++++ drivers/meson.build | 2 +- 4 files changed, 340 insertions(+), 3 deletions(-) create mode 100644 buildtools/elf.py create mode 100755 buildtools/pmdinfogen.pySignature · 1 line
2.25.4
Quoted 1 line · Neil Horman
On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
[snip]
Quoted 6 lines
Quoted 2 lines
1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden
I must have underestimated pyelftools spread. I'll look into using it. The less new code the better, I agree here. Windows users can get it via PyPI.
Quoted 5 lines
Quoted 3 lines
2. struct
rte_pci_idmust be synchronized with header file (it's a few lines that never change).I think you can just use ctypes and the native python C interface to just import the
rte_pci_idstructure from the native dpdk header to avoid this, no?
Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id in Python. If you're suggesting to create a Python module in C just to include a header with a single small structure, I'd say it's not worth the trouble.
Quoted 8 lines
Quoted 3 lines
1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?
Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.
I was talking about these fields of struct elf_info:
/* if Nth symbol table entry has .st_shndx = SHN_XINDEX, * take shndx from symtab_shndx_start[N] instead */ Elf32_Word *symtab_shndx_start; Elf32_Word *symtab_shndx_stop;
They are not used after being filled in parse_elf() and their endianness fixed in the end despite the comment.
Quoted 8 lines
Quoted 5 lines
2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.
I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.
Consider a snippet from pmdinfogen.c:
/* Check if file offset is correct */
if (hdr->e_shoff > info->size) {
fprintf(stderr, "section header offset=%lu in file '%s' "
"is bigger than filesize=%lu\n",
(unsigned long)hdr->e_shoff,
filename, info->size);
return 0;
}It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.
Quoted 11 lines
Quoted 6 lines
3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?
I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?
WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):
Host Toolchain Archive Script Extractor ------- --------- ------- --------- --------- Linux MinGW .a sh ar Windows MinGW .a PowerShell ar Windows Clang .lib PowerShell lib
Signature · 1 line
Dmitry Kozlyuk
Quoted 5 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman
On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
[snip]
Quoted 2 lines
Quoted 2 lines
1. No standard ELF or COFF module for Python
(amount of Python code without libelf on par with C code using it).
Quoted 8 lines
Quoted 4 lines
Thats not really true, pyelftools is pretty widely used, and packaged for most(all) major distributions. Requiring this kicks the can for (2) above down the road a bit, but I would prefer to see that used rather than a custom implementation, just to reduce the maint. burden
I must have underestimated pyelftools spread. I'll look into using it. The less new code the better, I agree here. Windows users can get it via PyPI.
Ack.
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
2. struct
rte_pci_idmust be synchronized with header file (it's a few lines that never change).I think you can just use ctypes and the native python C interface to just import the
rte_pci_idstructure from the native dpdk header to avoid this, no?Not sure I follow. RFC script uses ctypes to describe struct
rte_pci_idin Python. If you're suggesting to create a Python module in C just to include a header with a single small structure, I'd say it's not worth the trouble.
Yeah, apologies, i misread what you were saying here, and didn't bother to check the code. what you're doing makes sense.
Quoted 12 lines
Quoted 9 lines
Quoted 4 lines
1. Support for >65K section headers seems present in current pmdinfogen. However, the data it reads is not used after. Is it really needed?
Its used.
The support you're referring to is primarily the ability to extract the true number of section headers from the ELF file (in the event that its more than 64k). Without that, on very large binaries, you may not be able to find the symbol table in the ELF file.I was talking about these fields of struct
elf_info:
Quoted 9 lines
/* if Nth symbol table entry has .st_shndx = SHN_XINDEX, * take shndx from symtab_shndx_start[N] instead */ Elf32_Word *symtab_shndx_start; Elf32_Word *symtab_shndx_stop;They are not used after being filled in
parse_elf()and their endianness fixed in the end despite the comment.
Its been a while since I wrote this, so I need to go back and look closely, but as you say, these values aren't used after parse_elf completes, but they are(I think) used to correct the endianess of the section header index table, so that get_sym_value works properly. You would (again I think), only note a problem if you were parsing an ELF file for an architecture with an endianess opposite that of what you are building on (i.e. an x86 system cross compiling for a powerpc target).
Quoted 21 lines
Quoted 9 lines
Quoted 6 lines
2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.
I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.
Consider a snippet from
pmdinfogen.c:/* Check if file offset is correct */ if (hdr->e_shoff > info->size) { fprintf(stderr, "section header offset=%lu in file '%s' " "is bigger than filesize=%lu\n", (unsigned long)hdr->e_shoff, filename, info->size); return 0; }
Quoted 7 lines
It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.
Hmm, I'd defer to others opinions on that. As a build tool it may be ok to just crash with a backtrace, but I'm personally not a fan of that. Id rather see a diagnostic error and have the tool exits with an appropriate exit code, but lets see what others say.
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?
Quoted 5 lines
I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?
WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):
Quoted 5 lines
Host Toolchain Archive Script Extractor ------- --------- ------- --------- --------- Linux MinGW .a sh ar Windows MinGW .a PowerShell ar Windows Clang .lib PowerShell lib
I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?
Neil
Quoted 4 lines
Signature · 2 lines
-- Dmitry Kozlyuk
Quoted 33 lines · Neil Horman
Quoted 25 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman
On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
[snip]
Quoted 8 lines
Quoted 5 lines
2. How much error-handling is required? This is a build-time tool, and Python gives nice stacktraces. However, segfaults are possible in Python version due to pointer handling. IMO, error checking must be just sufficient to prevent silent segfaults.
I'm not really sure what the question is here. You need to handle errors in the parsing process, we can't just have the tool crashing during the build. How thats handled is an implementation detail I would think.
Consider a snippet from
pmdinfogen.c:/* Check if file offset is correct */ if (hdr->e_shoff > info->size) { fprintf(stderr, "section header offset=%lu in file '%s' " "is bigger than filesize=%lu\n", (unsigned long)hdr->e_shoff, filename, info->size); return 0; }Quoted 7 lines
It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple.
Quoted 5 lines
Hmm, I'd defer to others opinions on that. As a build tool it may be ok to just crash with a backtrace, but I'm personally not a fan of that. Id rather see a diagnostic error and have the tool exits with an appropriate exit code, but lets see what others say.
I'd take a wait-and-see approach here. Obviously individualized errors messages are better, but for a case like the above where we have a malformed elf header, I would think that this is such a rare event that just letting python exception handling manage it should be ok.
Quoted 24 lines
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
3. On Unix, pmdinfogen is called for each object file extracted with ar from an .a library by a shell script. On Windows, other tools have to be used, shell script will not work. On the other hand, COFF library format is quite simple. Would it be appropriate for pmdinfogen to handle it to avoid intermediate script?
Quoted 5 lines
I suppose you could, but extracting objects from archives seems outside the scope of what pmdinfogen normally does. What is the problem with using a shell script on windows? I thought WSL had support for executing bash syntax shell scripts there? Why not key off an environment variable to make the relevant scripts do the right thing on your environment?
WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):
Quoted 5 lines
Host Toolchain Archive Script Extractor ------- --------- ------- --------- --------- Linux MinGW .a sh ar Windows MinGW .a PowerShell ar Windows Clang .lib PowerShell lib
Quoted 6 lines
I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?
It might be better just to do that detection in the python script itself, rather than in meson, since the script may have to do other behavioural changes based on the platform.
Regards,
/Bruce
Quoted 3 lines · On Tue, 23 Jun 2020 07:28:06 -0400 Neil Horman wrote:
Quoted 2 lines · Dmitry Kozlyuk
Quoted 1 line · Neil Horman
On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote:
[snip]
Quoted 11 lines
Quoted 11 lines
I was talking about these fields of struct
elf_info:/* if Nth symbol table entry has .st_shndx = SHN_XINDEX, * take shndx from symtab_shndx_start[N] instead */ Elf32_Word *symtab_shndx_start; Elf32_Word *symtab_shndx_stop;They are not used after being filled in
parse_elf()and their endianness fixed in the end despite the comment.
Quoted 7 lines
Its been a while since I wrote this, so I need to go back and look closely, but as you say, these values aren't used after
parse_elfcompletes, but they are(I think) used to correct the endianess of the section header index table, so thatget_sym_valueworks properly. You would (again I think), only note a problem if you were parsing an ELF file for an architecture with an endianess opposite that of what you are building on (i.e. an x86 system cross compiling for a powerpc target).
Thanks for the explanation. Anyway, it's yet another point to prefer pyelftools to parsing ELF manually.
[snip]
Quoted 10 lines
Quoted 10 lines
Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF):
Host Toolchain Archive Script Extractor ------- --------- ------- --------- --------- Linux MinGW .a sh ar Windows MinGW .a PowerShell ar Windows Clang .lib PowerShell lib
Quoted 5 lines
I think if I read you right, what you're suggesting here is that meson setup a build time marco $ARCHIVETOOL, and set it to ar or lib.exe depending on the build environment, and just have the script in question use $ARCHIVER? If so, I'd be good with that. Do we have to worry about the use of divergent command line options for each tool as well?
As I said, PowerShell cannot consume the same shell script. My suggestion was to have two scripts, one existing for shell and ar (used by Linux host), and one new PowerShell script for both lib.exe and ar (used by Windows host). Meson then would select which script to call. Another option is to have one wrapper Python script for all hosts: a few extra lines to launch archiver, but less files and no need to know shell/PowerShell when changing this part. (I believe Bruce means the same.) But it will still be a separate script.
Signature · 1 line
Dmitry Kozlyuk
v2 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThe goals are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.
Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
Only logic errors, like missing symbols or invalid content, are checked. This seems enough for debugging and, to my understanding, there are no strong objections. All errors raise exceptions printed to stderr.
While discussion of future wrapper logic and Windows support continues, this patchset is ready for testing and review.
Dmitry Kozlyuk (3):
pmdinfogen: prototype in Python
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
GNUmakefile | 2 +- buildtools/Makefile | 9 -- buildtools/gen-pmdinfo-cfile.sh | 3 +- buildtools/meson.build | 3 +- buildtools/pmdinfogen.py | 191 ++++++++++++++++++++++++++++++ doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 - 9 files changed, 202 insertions(+), 18 deletions(-) delete mode 100644 buildtools/Makefile create mode 100755 buildtools/pmdinfogen.py
Signature · 1 line
2.25.4
RFC PATCH v2 1/3 pmdinfogen: prototype in Python
Requires pyelfutils, but no docs yet.
--- buildtools/pmdinfogen.py | 191 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +191 -0
@@ -0,0 +1,191 @@ +#!/usr/bin/env python3 + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image._image.get_section(self._symbol["st_shndx"]).data() + base = self._symbol["st_value"] + return data[base : base + self.size] + + @property + def string_value(self): + value = self.value + if len(value): + return value[:-1].decode() + return "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + if symbol: + return ELFSymbol(self, symbol[0]) + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver._load_pci_ids(image, pci_table_name_symbol) + + return driver + + def _load_pci_ids(self, image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + if table_symbol.size % pci_id_size != 0: + raise Exception( + "PCI table size (%d) is not a multiple of rte_pci_id size (%d)" + % (table_symbol.size, pci_id_size) + ) + + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + for pci_id in pci_ids: + if not pci_id.device_id: + break + self.pci_ids.append( + [ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ] + ) + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.25.4
RFC PATCH v2 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukAlso don't suppress stderr from ar and pmdinfogen.
--- GNUmakefile | 2 +- buildtools/Makefile | 9 --------- buildtools/gen-pmdinfo-cfile.sh | 3 +-- buildtools/meson.build | 3 +-- drivers/meson.build | 2 +- mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 -- 7 files changed, 5 insertions(+), 18 deletions(-) delete mode 100644 buildtools/Makefile
GNUmakefile +1 -1
@@ -12,6 +12,6 @@ export RTE_SDK # directory list # -ROOTDIRS-y := buildtools lib kernel drivers app +ROOTDIRS-y := lib kernel drivers app include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
@@ -1,9 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -DIRS-y += pmdinfogen - -include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +1 -2
@@ -9,6 +9,5 @@ pmdinfogen=$3 # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +1 -2
@@ -1,9 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017-2019 Intel Corporation -subdir('pmdinfogen') - pkgconf = find_program('pkg-config', 'pkgconf', required: false) +pmdinfogen = find_program('pmdinfogen.py') pmdinfo = find_program('gen-pmdinfo-cfile.sh') list_dir_globs = find_program('list-dir-globs.py') check_symbols = find_program('check-symbols.sh')
drivers/meson.build +1 -1
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif version_map = '@0@/@1@/@2@_version.map'.format( meson.current_source_dir(),
mk/internal/rte.compile-pre.mk +1 -1
@@ -59,7 +59,7 @@ endif CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@ -PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected] +PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected] PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected] PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@ PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
@@ -98,7 +98,6 @@ install-runtime: $(Q)cp $(CP_FLAGS) $O/lib/* $(DESTDIR)$(libdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(bindir)) $(Q)tar -cf - -C $O --exclude 'app/*.map' \ - --exclude app/dpdk-pmdinfogen \ --exclude 'app/cmdline*' --exclude app/test \ --exclude app/testacl --exclude app/testpipeline app | \ tar -xf - -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk: $(Q)cp $(CP_FLAGS) $(RTE_SDK)/buildtools $(DESTDIR)$(sdkdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(targetdir)/app) $(Q)cp $(CP_FLAGS) $O/.config $(DESTDIR)$(targetdir) - $(Q)cp $(CP_FLAGS) $O/app/dpdk-pmdinfogen $(DESTDIR)$(targetdir)/app $(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include) $(Q)$(call rte_symlink, $(DESTDIR)$(libdir), $(DESTDIR)$(targetdir)/lib)
Signature · 1 line
2.25.4
RFC PATCH v2 3/3 doc/linux_gsg: require pyelftools for pmdinfogen
In reply to Dmitry KozlyukThe package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen
Quoted 4 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file
Neil
Quoted 26 lines
Signed-off-by Dmitry Kozlyuk <[email protected]>--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)doc/guides/linux_gsg/sys_reqs.rst +6
-0@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;Signature · 1 line
2.25.4
Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen
Quoted 8 lines · Neil Horman
Quoted 4 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file
Windows uses COFF instead of ELF. I'm planning to add COFF support to pmdinfogen in a separate patch and also put wrapper script there. In this series I'd like to focus on just rewriting existing functionality.
Signature · 1 line
Dmitry Kozlyuk
Re: [dpdk-dev] [RFC PATCH v2 3/3] doc/linux_gsg: require pyelftools for pmdinfogen
Quoted 6 lines · Dmitry Kozlyuk
Quoted 5 lines · Neil Horman
Quoted 4 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Quoted 8 lines
Quoted 3 lines
Why doesn't windows use pmdinfogen? That tool doesn't relate to DSO function versioning, it just generates additional symbols for binaries so that we can see what pmd support is in a given ELF file
Windows uses COFF instead of ELF. I'm planning to add COFF support to pmdinfogen in a separate patch and also put wrapper script there. In this series I'd like to focus on just rewriting existing functionality.
Ah, understood, thanks for the clarification
Neil
Quoted 3 lines
Signature · 1 line
Dmitry Kozlyuk
v3 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v3:
1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.
Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- GNUmakefile | 2 +- MAINTAINERS | 2 +- buildtools/Makefile | 9 - buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 1 + buildtools/pmdinfogen.py | 194 ++++++++++++ buildtools/pmdinfogen/Makefile | 20 -- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 - 16 files changed, 209 insertions(+), 629 deletions(-) delete mode 100644 buildtools/Makefile create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/Makefile delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.25.4
PATCH v3 1/4 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 194 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +194 -0
@@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image._image.get_section(self._symbol["st_shndx"]).data() + base = self._symbol["st_value"] + return data[base : base + self.size] + + @property + def string_value(self): + value = self.value + if len(value): + return value[:-1].decode() + return "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + if symbol: + return ELFSymbol(self, symbol[0]) + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver._load_pci_ids(image, pci_table_name_symbol) + + return driver + + def _load_pci_ids(self, image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + if table_symbol.size % pci_id_size != 0: + raise Exception( + "PCI table size (%d) is not a multiple of rte_pci_id size (%d)" + % (table_symbol.size, pci_id_size) + ) + + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + for pci_id in pci_ids: + if not pci_id.device_id: + break + self.pci_ids.append( + [ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ] + ) + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.25.4
PATCH v3 2/4 build: use Python pmdinfogen
In reply to Dmitry KozlyukLike for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add python3-pyelftools to CI configuration.
--- .travis.yml | 2 +- GNUmakefile | 2 +- buildtools/Makefile | 9 --------- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 1 + drivers/meson.build | 2 +- meson.build | 1 - mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 -- 9 files changed, 8 insertions(+), 19 deletions(-) delete mode 100644 buildtools/Makefile
.travis.yml +1 -1
@@ -14,7 +14,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
GNUmakefile +1 -1
@@ -12,6 +12,6 @@ export RTE_SDK # directory list # -ROOTDIRS-y := buildtools lib kernel drivers app +ROOTDIRS-y := lib kernel drivers app include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
@@ -1,9 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -DIRS-y += pmdinfogen - -include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +1 -0
@@ -17,3 +17,4 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py')
drivers/meson.build +1 -1
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
mk/internal/rte.compile-pre.mk +1 -1
@@ -59,7 +59,7 @@ endif CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@ -PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected] +PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected] PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected] PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@ PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
@@ -98,7 +98,6 @@ install-runtime: $(Q)cp $(CP_FLAGS) $O/lib/* $(DESTDIR)$(libdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(bindir)) $(Q)tar -cf - -C $O --exclude 'app/*.map' \ - --exclude app/dpdk-pmdinfogen \ --exclude 'app/cmdline*' --exclude app/test \ --exclude app/testacl --exclude app/testpipeline app | \ tar -xf - -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk: $(Q)cp $(CP_FLAGS) $(RTE_SDK)/buildtools $(DESTDIR)$(sdkdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(targetdir)/app) $(Q)cp $(CP_FLAGS) $O/.config $(DESTDIR)$(targetdir) - $(Q)cp $(CP_FLAGS) $O/app/dpdk-pmdinfogen $(DESTDIR)$(targetdir)/app $(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include) $(Q)$(call rte_symlink, $(DESTDIR)$(libdir), $(DESTDIR)$(targetdir)/lib)
Signature · 1 line
2.25.4
PATCH v3 3/4 doc/linux_gsg: require pyelftools for pmdinfogen
In reply to Dmitry KozlyukThe package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
PATCH v3 4/4 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.
--- MAINTAINERS | 2 +- buildtools/pmdinfogen/Makefile | 20 -- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 5 files changed, 1 insertion(+), 610 deletions(-) delete mode 100644 buildtools/pmdinfogen/Makefile delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
@@ -157,7 +157,7 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/Makefile deleted +0 -20
@@ -1,20 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -# -# library name -# -HOSTAPP = dpdk-pmdinfogen - -# -# all sources are stored in SRCS-y -# -SRCS-y += pmdinfogen.c - -HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g -HOST_CFLAGS += -I$(RTE_OUTPUT)/include - -include $(RTE_SDK)/mk/rte.hostapp.mk
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.25.4
v4 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v4:
Fix build by removing redundant check.
Changes in v3:
1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.
Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- GNUmakefile | 2 +- MAINTAINERS | 2 +- buildtools/Makefile | 9 - buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 1 + buildtools/pmdinfogen.py | 188 ++++++++++++ buildtools/pmdinfogen/Makefile | 20 -- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 - 16 files changed, 203 insertions(+), 629 deletions(-) delete mode 100644 buildtools/Makefile create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/Makefile delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.25.4
PATCH v4 1/4 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +188 -0
@@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image._image.get_section(self._symbol["st_shndx"]).data() + base = self._symbol["st_value"] + return data[base : base + self.size] + + @property + def string_value(self): + value = self.value + if len(value): + return value[:-1].decode() + return "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + if symbol: + return ELFSymbol(self, symbol[0]) + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver._load_pci_ids(image, pci_table_name_symbol) + + return driver + + def _load_pci_ids(self, image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + for pci_id in pci_ids: + if not pci_id.device_id: + break + self.pci_ids.append( + [ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ] + ) + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.25.4
PATCH v4 2/4 build: use Python pmdinfogen
In reply to Dmitry KozlyukLike for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add python3-pyelftools to CI configuration.
--- .travis.yml | 2 +- GNUmakefile | 2 +- buildtools/Makefile | 9 --------- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 1 + drivers/meson.build | 2 +- meson.build | 1 - mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 -- 9 files changed, 8 insertions(+), 19 deletions(-) delete mode 100644 buildtools/Makefile
.travis.yml +1 -1
@@ -14,7 +14,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
GNUmakefile +1 -1
@@ -12,6 +12,6 @@ export RTE_SDK # directory list # -ROOTDIRS-y := buildtools lib kernel drivers app +ROOTDIRS-y := lib kernel drivers app include $(RTE_SDK)/mk/rte.sdkroot.mk
buildtools/Makefile deleted +0 -9
@@ -1,9 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -DIRS-y += pmdinfogen - -include $(RTE_SDK)/mk/rte.subdir.mk
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +1 -0
@@ -17,3 +17,4 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py')
drivers/meson.build +1 -1
@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
mk/internal/rte.compile-pre.mk +1 -1
@@ -59,7 +59,7 @@ endif CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@ -PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected] +PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected] PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected] PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@ PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
mk/rte.sdkinstall.mk +0 -2
@@ -98,7 +98,6 @@ install-runtime: $(Q)cp $(CP_FLAGS) $O/lib/* $(DESTDIR)$(libdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(bindir)) $(Q)tar -cf - -C $O --exclude 'app/*.map' \ - --exclude app/dpdk-pmdinfogen \ --exclude 'app/cmdline*' --exclude app/test \ --exclude app/testacl --exclude app/testpipeline app | \ tar -xf - -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)
@@ -134,7 +133,6 @@ install-sdk: $(Q)cp $(CP_FLAGS) $(RTE_SDK)/buildtools $(DESTDIR)$(sdkdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(targetdir)/app) $(Q)cp $(CP_FLAGS) $O/.config $(DESTDIR)$(targetdir) - $(Q)cp $(CP_FLAGS) $O/app/dpdk-pmdinfogen $(DESTDIR)$(targetdir)/app $(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include) $(Q)$(call rte_symlink, $(DESTDIR)$(libdir), $(DESTDIR)$(targetdir)/lib)
Signature · 1 line
2.25.4
Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen
Quoted 69 lines · Dmitry Kozlyuk
Like for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add python3-pyelftools to CI configuration.
Signed-off-by Dmitry Kozlyuk <[email protected]>--- .travis.yml | 2 +- GNUmakefile | 2 +- buildtools/Makefile | 9 --------- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 1 + drivers/meson.build | 2 +- meson.build | 1 - mk/internal/rte.compile-pre.mk | 2 +- mk/rte.sdkinstall.mk | 2 -- 9 files changed, 8 insertions(+), 19 deletions(-) delete mode 100644 buildtools/Makefile.travis.yml +1
-1@@ -14,7 +14,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packagesGNUmakefile +1
-1@@ -12,6 +12,6 @@ export RTE_SDK # directory list # -ROOTDIRS-y := buildtools lib kernel drivers app +ROOTDIRS-y := lib kernel drivers app include $(RTE_SDK)/mk/rte.sdkroot.mkbuildtools/Makefile deleted +0
-9@@ -1,9 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -DIRS-y += pmdinfogen - -include $(RTE_SDK)/mk/rte.subdir.mkbuildtools/gen-pmdinfo-cfile.sh +0
-1@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3
buildtools/gen-pmdinfo-cfile.sh line 14
Quoted 10 lines
buildtools/gen-pmdinfo-cfile.sh +3
-2+shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
For 20.11, since make build system is being removed, I think we should look to merge this into the python script rather than having an extra layer of wrapper and having meson pass the command from one script to another. However, since out-of-the-box python doesn't seem to have support for reading .a files, this is fine for now.
Quoted 70 lines
buildtools/meson.build +1
-0@@ -17,3 +17,4 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py')drivers/meson.build +1
-1@@ -119,7 +119,7 @@ foreach class:dpdk_driver_classes command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static drivermeson.build +0
-1@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')mk/internal/rte.compile-pre.mk +1
-1@@ -59,7 +59,7 @@ endif CHECK_SYMBOLS_SCRIPT = $(RTE_SDK)/buildtools/check-symbols.sh CHECK_SYMBOLS = $(CHECK_SYMBOLS_SCRIPT) $(SRCDIR)/$(EXPORT_MAP) $@ -PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ [email protected] +PMDINFO_GEN = $(RTE_SDK)/buildtools/pmdinfogen.py $@ [email protected] PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o [email protected] [email protected] PMDINFO_LD = $(CROSS)ld -r $(filter-out -export-dynamic,$(LDFLAGS)) -o [email protected] [email protected] $@ PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \mk/rte.sdkinstall.mk +0
-2@@ -98,7 +98,6 @@ install-runtime: $(Q)cp $(CP_FLAGS) $O/lib/* $(DESTDIR)$(libdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(bindir)) $(Q)tar -cf - -C $O --exclude 'app/*.map' \ - --exclude app/dpdk-pmdinfogen \ --exclude 'app/cmdline*' --exclude app/test \ --exclude app/testacl --exclude app/testpipeline app | \ tar -xf - -C $(DESTDIR)$(bindir) $(TAR_X_FLAGS)@@ -134,7 +133,6 @@ install-sdk: $(Q)cp $(CP_FLAGS) $(RTE_SDK)/buildtools $(DESTDIR)$(sdkdir) $(Q)$(call rte_mkdir, $(DESTDIR)$(targetdir)/app) $(Q)cp $(CP_FLAGS) $O/.config $(DESTDIR)$(targetdir) - $(Q)cp $(CP_FLAGS) $O/app/dpdk-pmdinfogen $(DESTDIR)$(targetdir)/app $(Q)$(call rte_symlink, $(DESTDIR)$(includedir), $(DESTDIR)$(targetdir)/include) $(Q)$(call rte_symlink, $(DESTDIR)$(libdir), $(DESTDIR)$(targetdir)/lib)Signature · 1 line
2.25.4
For the meson build changes here:
Re: [dpdk-dev] [PATCH v4 2/4] build: use Python pmdinfogen
Quoted 1 line · Bruce Richardson
On Thu, Jul 09, 2020 at 12:23:33AM +0300, Dmitry Kozlyuk wrote:
[snip]
Quoted 12 lines
Quoted 12 lines
buildtools/gen-pmdinfo-cfile.sh +2
-1@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$*
Quoted 13 lines
buildtools/gen-pmdinfo-cfile.sh line 14
Quoted 7 lines
buildtools/gen-pmdinfo-cfile.sh +1
-2# The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0For 20.11, since make build system is being removed, I think we should look to merge this into the python script rather than having an extra layer of wrapper and having meson pass the command from one script to another. However, since out-of-the-box python doesn't seem to have support for reading .a files, this is fine for now.
Or Meson can only tell the Python script whether to process ELF of COFF and the path to an archiver tool (ar or lib.exe). This way we'd get rid of shell (which is non-portable to Windows) and need no libraries to read .a or .lib.
PATCH v4 3/4 doc/linux_gsg: require pyelftools for pmdinfogen
In reply to Dmitry KozlyukThe package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
Signature · 1 line
2.25.4
Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen
Quoted 14 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Signed-off-by Dmitry Kozlyuk <[email protected]>--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)doc/guides/linux_gsg/sys_reqs.rst +0
-0
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 64
Quoted 9 lines
doc/guides/linux_gsg/sys_reqs.rst +6
-0@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` +
If you want for completeness you can add:
* For FreeBSD it can be installed using ``pkg install py37-pyelftools``, where py37 refers to the python version in use.
Quoted 6 lines
doc/guides/linux_gsg/sys_reqs.rst +0
-0* Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;Signature · 1 line
2.25.4
Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen
Quoted 18 lines · Bruce Richardson
Quoted 17 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Signed-off-by Dmitry Kozlyuk <[email protected]>--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)doc/guides/linux_gsg/sys_reqs.rst +0
-0@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja``
Quoted 11 lines
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 64
Quoted 6 lines
doc/guides/linux_gsg/sys_reqs.rst +6
-0+* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` +If you want for completeness you can add:
* For FreeBSD it can be installed using ``pkg install py37-pyelftools``, where py37 refers to the python version in use.
Sorry, missed the fact that this was the Linux GSG. Never mind...
Re: [dpdk-dev] [PATCH v4 3/4] doc/linux_gsg: require pyelftools for pmdinfogen
In reply to Dmitry KozlyukQuoted 17 lines · Dmitry Kozlyuk
The package is available on all major distributions. FreeBSD has no system requirements section in its GSG, Windows currently neither uses pmdinfogen not is supported by it.
Signed-off-by Dmitry Kozlyuk <[email protected]>--- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ 1 file changed, 6 insertions(+)doc/guides/linux_gsg/sys_reqs.rst +0
-0@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja``
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 67
Quoted 10 lines
doc/guides/linux_gsg/sys_reqs.rst +6
-0+* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
This can be merged into patch 2, I think.
PATCH v4 4/4 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.
--- MAINTAINERS | 2 +- buildtools/pmdinfogen/Makefile | 20 -- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 5 files changed, 1 insertion(+), 610 deletions(-) delete mode 100644 buildtools/pmdinfogen/Makefile delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
@@ -157,7 +157,7 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/Makefile deleted +0 -20
@@ -1,20 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Neil Horman <[email protected]> -# All rights reserved. - -include $(RTE_SDK)/mk/rte.vars.mk - -# -# library name -# -HOSTAPP = dpdk-pmdinfogen - -# -# all sources are stored in SRCS-y -# -SRCS-y += pmdinfogen.c - -HOST_CFLAGS += $(HOST_WERROR_FLAGS) -g -HOST_CFLAGS += -I$(RTE_OUTPUT)/include - -include $(RTE_SDK)/mk/rte.hostapp.mk
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.25.4
Quoted 28 lines · Dmitry Kozlyuk
This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v4:Fix build by removing redundant check.
Changes in v3:
1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
Quoted 7 lines
---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation
I'm not currently setup to test with cross compiling, but for direct targets (tested on power and x86), it seems to work well
Quoted 28 lines · Dmitry Kozlyuk
This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v4:Fix build by removing redundant check.
Changes in v3:
1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
Quoted 7 lines
---
Dmitry Kozlyuk (4):
pmdinfogen: add Python implementation
build: use Python pmdinfogen doc/linux_gsg: require pyelftools for pmdinfogen
pmdinfogen: remove C implementation
I've done builds with this patchset on linux and FreeBSD and checked the results with dpdk-pmdinfo script. All looks ok to me.
v5 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
From v4:
---
Changes in v5:
1. Rebase on ToT, remove Makefile changes.
2. Squash doc changes into patch 2/3.
Changes in v4:
Fix build by removing redundant check.
Changes in v3:
1. Rebase on ToT, add package to CI.
2. Add copyrights, fixup maintainers file.
3. Remove C implementation.
Changes in v2:
1. pyelftools is used instead of hand-written ELF parser.
2. Makefiles are adjusted.
3. f-strings replaced with % formatting to support Python 3.5.
4. Wrapper script does not hide stderr from ar and pmdinfogen.
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- MAINTAINERS | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 1 + buildtools/pmdinfogen.py | 188 ++++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 11 files changed, 201 insertions(+), 596 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
PATCH v5 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 188 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +188 -0
@@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image._image.get_section(self._symbol["st_shndx"]).data() + base = self._symbol["st_value"] + return data[base : base + self.size] + + @property + def string_value(self): + value = self.value + if len(value): + return value[:-1].decode() + return "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + if symbol: + return ELFSymbol(self, symbol[0]) + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver._load_pci_ids(image, pci_table_name_symbol) + + return driver + + def _load_pci_ids(self, image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + for pci_id in pci_ids: + if not pci_id.device_id: + break + self.pci_ids.append( + [ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ] + ) + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.28.0
Re: [dpdk-dev] [PATCH v5 1/3] pmdinfogen: add Python implementation
Quoted 5 lines · On Mon, 28 Sep 2020 00:47:30 +0300 Dmitry Kozlyuk <[email protected]>…
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
Signed-off-by Dmitry Kozlyuk <[email protected]>
Looks good, a couple little nits.
Yapf flags this as unnecessary spacing:
+ @property
+ def value(self):
+ data = self._image._image.get_section(self._symbol["st_shndx"]).data()
+ base = self._symbol["st_value"]
+ return data[base : base + self.size]
^^^Python lint complains about missing docstrings, which doesn't matter but it also shows:
$ pylint3 buildtools/pmdinfogen.py | grep -v docstring ************* Module pmdinfogen buildtools/pmdinfogen.py:27:15: W0212: Access to a protected member _image of a client class (protected-access) buildtools/pmdinfogen.py:34:11: C1801: Do not use `len(SEQUENCE)` to determine if a sequence is empty (len-as-condition) buildtools/pmdinfogen.py:50:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements) buildtools/pmdinfogen.py:107:12: W0212: Access to a protected member _load_pci_ids of a client class (protected-access)
------------------------------------------------------------------
Your code has been rated at 7.93/10 (previous run: 7.93/10, +0.00)
PATCH v5 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukLike for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
--- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 1 + doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 6 files changed, 12 insertions(+), 6 deletions(-)
.travis.yml +1 -1
@@ -11,7 +11,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +1 -0
@@ -17,3 +17,4 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py')
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -56,6 +56,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -137,7 +137,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.28.0
PATCH v5 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.
--- MAINTAINERS | 2 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 1 insertion(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +1 -1
@@ -135,7 +135,7 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.28.0
Re: [dpdk-dev] [PATCH v5 3/3] pmdinfogen: remove C implementation
Quoted 3 lines
Delete the files no longer used in build process. Keep Neil Horman maintainer of the entire "Driver information" section, only adjust paths.
Dmitry, don't you think you should be co-maintainer?
Hello Dmitry,
Quoted 4 lines · Dmitry Kozlyuk
This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* support pmdinfo on Windows ? :-)
Quoted 6 lines
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf.Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
This new requirement will impact mainly distribution packagers, developers and the CI.
While I think this is fine for the first two categories (python-elftools is now available on most distributions, and you can also pip install it), the last category is a concern.
If we want to move forward with this patch, this must be addressed in a timely manner as we don't want to break the CI.
Signature · 1 line
David Marchand
v6 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v6:
1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 1 + buildtools/pmdinfogen.py | 189 ++++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 11 files changed, 203 insertions(+), 596 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
PATCH v6 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image.get_section_data(self._symbol["st_shndx"]) + base = self._symbol["st_value"] + return data[base:base + self.size] + + @property + def string_value(self): + value = self.value + return value[:-1].decode() if value else "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def get_section_data(self, name): + return self._image.get_section(name).data() + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + return ELFSymbol(self, symbol[0]) if symbol else None + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol) + + return driver + + @staticmethod + def _load_pci_ids(image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + result = [] + for pci_id in pci_ids: + if not pci_id.device_id: + break + result.append([ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ]) + return result + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.28.0
PATCH v6 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukLike for other build scripts, use Python interpreter to run pmdinfogen. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
--- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 1 + doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 6 files changed, 12 insertions(+), 6 deletions(-)
.travis.yml +1 -1
@@ -11,7 +11,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +1 -0
@@ -17,3 +17,4 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py')
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -52,6 +52,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -137,7 +137,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.28.0
PATCH v6 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process.
Add myself as maintainer of new implementation.
--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.28.0
Hi Dmitry,
Quoted 24 lines · Dmitry Kozlyuk
This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
Acked-by Neil Horman <[email protected]>---
Changes in v6:1. Address yapf and pylint warnings (Stephen Hemminger).
2. Add myself as new implementation maintainer (Thomas Monjalon).Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
Quoted 17 lines
.travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 1 + buildtools/pmdinfogen.py | 189 ++++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 11 files changed, 203 insertions(+), 596 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Ninja fails when pyelftools is not installed. Shouldn't meson detect and report it before build time?
Thanks,
Maxime
Hi Maxime,
Quoted 2 lines
Ninja fails when pyelftools is not installed. Shouldn't meson detect and report it before build time?
Will add in v7. Meson 0.51 introduced a terse way to check for Python dependencies, but it's not hard even with Meson 0.47.1.
https://github.com/mesonbuild/meson/issues/3551#issuecomment-518320640
v7 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v7:
Add configure-time check for elftools module (Maxime Coquelin).
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 15 + buildtools/pmdinfogen.py | 189 ++++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 11 files changed, 217 insertions(+), 596 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
PATCH v7 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image.get_section_data(self._symbol["st_shndx"]) + base = self._symbol["st_value"] + return data[base:base + self.size] + + @property + def string_value(self): + value = self.value + return value[:-1].decode() if value else "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def get_section_data(self, name): + return self._image.get_section(name).data() + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + return ELFSymbol(self, symbol[0]) if symbol else None + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol) + + return driver + + @staticmethod + def _load_pci_ids(image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + result = [] + for pci_id in pci_ids: + if not pci_id.device_id: + break + result.append([ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ]) + return result + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.28.0
PATCH v7 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukUse the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add configure-time check for elftools Python module.
Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
--- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 15 +++++++++++++++ doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 6 files changed, 26 insertions(+), 6 deletions(-)
.travis.yml +1 -1
@@ -11,7 +11,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +15 -0
@@ -17,3 +17,18 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py') + +# TODO: starting from Meson 0.51.0 use +# python3 = import('python').find_installation('python', +# modules : python3_required_modules) +python3_required_modules = ['elftools'] +foreach module : python3_required_modules + script = \ + 'import importlib.util;' + \ + 'import sys;' + \ + 'exit(importlib.util.find_spec("@0@") is None)'.format(module) + if run_command(py3, '-c', script).returncode() != 0 + error('missing python module: @0@'.format(module)) + endif +endforeach
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -52,6 +52,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -137,7 +137,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.28.0
PATCH v7 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process.
Add myself as maintainer of new implementation.
--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.28.0
Hello Dmitry,
Quoted 4 lines · Dmitry Kozlyuk
This patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
Quoted 8 lines
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478
Build machine cpu: x86_64
buildtools/meson.build:27:10: ERROR: Expecting endforeach got eol_cont.
script = \
^For a block that started at 26,0
foreach module : python3_required_modules
^
- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.html
ninja: error: loading 'build.ninja': No such file or directory
- As far as Intel CI systems are concerned, there might be an issue with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.html
OS: WIN10-64
Target: x86_64-windows-clang
Program binutils-avx512-check.sh found: YES
Program python3 found: YES (c:\program files\python37\python.exe)
buildtools\meson.build:32:2: ERROR: Problem encountered: missing python module: elftools
A full log can be found at C:\windows-dpdk\dpdk\build\meson-logs\meson-log.txt
ninja: Entering directory `build'
DPDK STV team
Signature · 1 line
David Marchand
Hello David,
Quoted 10 lines
- Travis reported an error in meson:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/399762460#L478Build machine cpu:
x86_64
buildtools/meson.build:27:10: ERROR: Expecting endforeach goteol_cont.script = \ ^For a block that started at 26,0
foreach module :python3_required_modules
^
My bad, meson 0.47.1 doesn't understand this indeed, fixed in v8.
Quoted 8 lines
- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.htmlninja: error: loading 'build.ninja': No such file or directory
Makes sense.
Quoted 11 lines
- As far as Intel CI systems are concerned, there might be an issue with the Windows system:
http://mails.dpdk.org/archives/test-report/2020-October/159123.htmlOS: WIN10-64
Target: x86_64-windows-clang
Programbinutils-avx512-check.shfound: YES
Program python3 found: YES (c:\program files\python37\python.exe)buildtools\meson.build:32:2: ERROR: Problem encountered: missing python module: elftools
This is the check added in the last revision per Maxime's suggestion. Disabled the check on Windows in v8 until pmdinfogen supports COFF (and this will probably introduce an alternative dependency).
I'll watch new CI results, haven't been doing so closely because failures were expected for these patches. Tested v8 build on Linux->Linux, Linux->Windows, Windows->Windows (GCC & clang, without pyelftools installed).
Hi IOL team,
The following error persists in v8:
Quoted 8 lines
- In a thread on [email protected]
http://inbox.dpdk.org/ci/CAOeXdvbosqRGouSX06_phVTCam57XUt0s=sf_kpe-VmiwFTgMA@mail.gmail.com/T/#m9bb193c74ca979c151f2dbfa35664f95093a9e71,
UNH people confirmed that python3-elftools should be available, but the last test had an error too, which might be linked to incorrect meson syntax:
http://mails.dpdk.org/archives/test-report/2020-October/159038.htmlninja: error: loading 'build.ninja': No such file or directory
https://inbox.dpdk.org/test-report/[email protected]/T/#u
The message isn't very informative. I suspect meson failed to configure, but there's no way to tell what went wrong. How can I get more information about this failure or specification of your environment to reproduce it?
v8 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.
Travis CI script is adjusted to install python3-pyelftools, but other CI systems may need similar tweaking. Particularly, testing on FreeBSD and big-endian targets is desired.
---
Changes in v8:
* Fix syntax for meson 0.47.1 (CI).
* Disable configure-time check for pyelftools for Windows (CI).
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 15 + buildtools/pmdinfogen.py | 189 ++++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 11 files changed, 217 insertions(+), 596 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.28.0
PATCH v8 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image.get_section_data(self._symbol["st_shndx"]) + base = self._symbol["st_value"] + return data[base:base + self.size] + + @property + def string_value(self): + value = self.value + return value[:-1].decode() if value else "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def get_section_data(self, name): + return self._image.get_section(name).data() + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + return ELFSymbol(self, symbol[0]) if symbol else None + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol) + + return driver + + @staticmethod + def _load_pci_ids(image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + result = [] + for pci_id in pci_ids: + if not pci_id.device_id: + break + result.append([ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ]) + return result + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.28.0
PATCH v8 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukUse the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add configure-time check for elftools Python module for Unix hosts.
Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
--- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 15 +++++++++++++++ doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 6 files changed, 26 insertions(+), 6 deletions(-)
.travis.yml +1 -1
@@ -11,7 +11,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +15 -0
@@ -17,3 +17,18 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py') + +# TODO: starting from Meson 0.51.0 use +# python3 = import('python').find_installation('python', +# modules : python3_required_modules) +python3_required_modules = [] +if host_machine.system() != 'windows' + python3_required_modules = ['elftools'] +endif +foreach module : python3_required_modules + script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)' + if run_command(py3, '-c', script.format(module)).returncode() != 0 + error('missing python module: @0@'.format(module)) + endif +endforeach
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -52,6 +52,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -152,7 +152,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.28.0
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 6 lines · Dmitry Kozlyuk
Use the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add configure-time check for elftools Python module for Unix hosts.
Quoted 7 lines
Add python3-pyelftools to CI configuration. The package is available on all major distributions. FreeBSD has no system requirements section in its GSG. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
Signed-off-by Dmitry Kozlyuk <[email protected]>---
From a meson viewpoint, this all looks ok. I'd hope in future that we could merge the shell script wrapper into the python script directly, rather than having two levels of scripting, but it's not something with any urgency.
Quoted 8 lines
.travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 15 +++++++++++++++ doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 6 files changed, 26 insertions(+), 6 deletions(-)
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
In reply to Dmitry KozlyukThis is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' at rte_common_mlx5.pmd.c
Few more comments below:
20/10/2020 19:44, Dmitry Kozlyuk:
buildtools/meson.build else · line 26
Quoted 3 lines
buildtools/meson.build +1
-0+if host_machine.system() != 'windows'
You can use "is_windows".
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55
Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1
-0+* ``pyelftools`` (version 0.22+)
This requirement is missing in doc/guides/freebsd_gsg/build_dpdk.rst
meson.build subdir('buildtools') · line 48
Quoted 3 lines
meson.build +0
-1-subdir('buildtools/pmdinfogen')
This could be in patch 3 (removing the code).
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 6 lines · Thomas Monjalon
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.c
Will investigate and fix ASAP.
Quoted 8 lines
Few more comments below:
20/10/2020 19:44, Dmitry Kozlyuk:
buildtools/meson.build else · line 26
Quoted 3 lines
buildtools/meson.build +1
-0+if host_machine.system() != 'windows'You can use "
is_windows".
It's defined by config/meson.build, which is processed after buidtools/meson.build, because of the dependency, if swapped:
config/x86/meson.build:6:1: ERROR: Unknown variable "binutils_avx512_check".
Quoted 5 lines
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55
Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1
-0+* ``pyelftools`` (version 0.22+)This requirement is missing in
doc/guides/freebsd_gsg/build_dpdk.rst
OK.
Quoted 5 lines
meson.build subdir('buildtools') · line 48
Quoted 3 lines
meson.build +0
-1-subdir('buildtools/pmdinfogen')This could be in patch 3 (removing the code).
It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 19 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cWill investigate and fix ASAP.
Quoted 8 lines
Few more comments below:
20/10/2020 19:44, Dmitry Kozlyuk:
buildtools/meson.build else · line 26
Quoted 3 lines
buildtools/meson.build +1
-0+if host_machine.system() != 'windows'You can use "
is_windows".
Quoted 5 lines
It's defined by config/meson.build, which is processed after buidtools/meson.build, because of the dependency, if swapped:
config/x86/meson.build:6:1: ERROR: Unknown variable "binutils_avx512_check".
OK
Quoted 8 lines
Quoted 5 lines
doc/guides/linux_gsg/sys_reqs.rst Compilation of the DPDK · line 55
Quoted 3 lines
doc/guides/linux_gsg/sys_reqs.rst +1
-0+* ``pyelftools`` (version 0.22+)This requirement is missing in
doc/guides/freebsd_gsg/build_dpdk.rstOK.
Quoted 8 lines
Quoted 5 lines
meson.build subdir('buildtools') · line 48
Quoted 3 lines
meson.build +0
-1-subdir('buildtools/pmdinfogen')This could be in patch 3 (removing the code).
It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?
Just trying to find the best patch split.
If needed, OK to keep as is.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 10 lines · Thomas Monjalon
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 9 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cWill investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is 28c9a7d7b48e ("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.
"xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
meson.build subdir('buildtools') · line 48
Quoted 3 lines
meson.build +0
-1-subdir('buildtools/pmdinfogen')This could be in patch 3 (removing the code).
It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?
Just trying to find the best patch split.
If needed, OK to keep as is.
I even don't mind squashing all three commits into one. The split is done to ease the review.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.c
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.
You're right.
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.
If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
meson.build subdir('buildtools') · line 48
Quoted 3 lines
meson.build +0
-1-subdir('buildtools/pmdinfogen')This could be in patch 3 (removing the code).
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
It would redefine "pmdinfogen" variable to old pmdinfogen. Besides, why build what's not used at this patch already?
Just trying to find the best patch split.
If needed, OK to keep as is.I even don't mind squashing all three commits into one. The split is done to ease the review.
I think the split is good as is.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 20 lines · Thomas Monjalon
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with RTE_DEVICE_IN in patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.py
For now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.
Quoted 8 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
Send v10 with suppression.
Such check, however, *can* be implemented: at ABI check stage we have two install directories that dpdk-pmdinfo.py can inspect. Then a script can check that diff contains only additions, i.e. no device support being removed.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
In reply to Thomas MonjalonQuoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.
Quoted 9 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Would it make sense to just fix the size of _pmd_info to some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
Ray K
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 48 lines · Ray
Quoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 9 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
Quoted 7 lines
So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?Ray K
Another point is - shouldn't _pmd_info probably live in "INTERNAL" is anycase?
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 55 lines · Ray
Quoted 51 lines · Ray
Quoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 9 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Quoted 7 lines
Quoted 4 lines
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?Ray K
Another point is - shouldn't
_pmd_infoprobably live in "INTERNAL" is anycase?
The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used by usertools/dpdk-pmdinfo.py and maybe some other parsing tool.
I guess a static symbol would be enough:
(quoted diff) +0 -0
b/buildtools/pmdinfogen/pmdinfogen.c
index a68d1ea999..14bf7d9f42 100644
buildtools/pmdinfogen/pmdinfogen.c +0 -0
@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info
*info, char *outfile)
drv = info->drivers;
while (drv) {
- fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = "
+ fprintf(ofd, "static const char %s_pmd_info[]
__attribute__((used)) = "
"\"PMD_INFO_STRING= {",
drv->name);
fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);We will need an exception for the v21 ABI though.
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 83 lines · David Marchand
Quoted 55 lines · Ray
Quoted 51 lines · Ray
Quoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 9 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Quoted 7 lines
Quoted 4 lines
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?Ray K
Another point is - shouldn't
_pmd_infoprobably live in "INTERNAL" is anycase?The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used byusertools/dpdk-pmdinfo.pyand maybe some other parsing tool.I guess a static symbol would be enough:
buildtools/pmdinfogen/pmdinfogen.c +1
-1@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info *info, char *outfile) drv = info->drivers;while (drv) { - fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = " + fprintf(ofd, "static const char %s_pmd_info[]__attribute__((used)) = "
Quoted 7 lines
"\"PMD_INFO_STRING= {", drv->name); fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);We will need an exception for the v21 ABI though.
Good suggestion +1
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
25/01/2021 11:46, Kinsella, Ray:
Quoted 25 lines · David Marchand
Quoted 24 lines
The symbol itself can be hidden from the ABeyes.
It is only a placeholder for the PMD_INFO_STRING= string used byusertools/dpdk-pmdinfo.pyand maybe some other parsing tool.I guess a static symbol would be enough:
buildtools/pmdinfogen/pmdinfogen.c +1
-1@@ -393,7 +393,7 @@ static void output_pmd_info_string(struct elf_info *info, char *outfile) drv = info->drivers;while (drv) { - fprintf(ofd, "const char %s_pmd_info[] __attribute__((used)) = " + fprintf(ofd, "static const char %s_pmd_info[]__attribute__((used)) = "
"\"PMD_INFO_STRING= {", drv->name); fprintf(ofd, "\\\"name\\\" : \\\"%s\\\", ", drv->name);
Quoted 4 lines
Quoted 2 lines
We will need an exception for the v21 ABI though.
Good suggestion +1
Yes +1 for adding static on *_pmd_info
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
In reply to Kinsella, RayQuoted 46 lines · Ray
Quoted 35 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 9 lines
Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
Quoted 5 lines
So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?
I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place. As for content, checking that no PCI IDs are removed is out of scope for libabigail anyway. Technically we could fix _pmd_info size, but this still allows breaking changes to pass the check with no benefit.
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 47 lines · Dmitry Kozlyuk
Quoted 46 lines · Ray
Quoted 44 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.
Quoted 8 lines
Quoted 5 lines
So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place.
Right - then is belongs in INTERNAL, I would say.
Quoted 3 lines
As for
content, checking that no PCI IDs are removed is out of scope for libabigail anyway.
Lets be clear PCI IDs - are _nothing_ to do with ABI.
Quoted 2 lines
Technically we could fix
_pmd_infosize, but this still allows breaking changes to pass the check with no benefit.
ABI changes or other, please explain?
Re: [dpdk-dev] [PATCH v8 2/3] build: use Python pmdinfogen
Quoted 57 lines · Ray
Quoted 47 lines · Dmitry Kozlyuk
Quoted 46 lines · Ray
Quoted 44 lines · Thomas Monjalon
22/01/2021 23:24, Dmitry Kozlyuk:
Quoted 34 lines · Thomas Monjalon
Quoted 20 lines
22/01/2021 21:31, Dmitry Kozlyuk:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
20/01/2021 08:23, Dmitry Kozlyuk:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
This is now the right timeframe to introduce this change with the new Python module dependency. Unfortunately, the ABI check is returning an issue:
'const char mlx5_common_pci_pmd_info[62]' was changed
to 'const char mlx5_common_pci_pmd_info[60]' atrte_common_mlx5.pmd.cQuoted 6 lines
Quoted 1 line
Quoted 1 line
Will investigate and fix ASAP.
Now that I think of it: strings like this change every time new PCI IDs are added to a PMD, but AFAIK adding PCI IDs is not considered an ABI breakage, is it? One example is
28c9a7d7b48e("net/mlx5: add ConnectX-6 Lx device ID") added 2020-07-08, i.e. clearly outside of ABI change window.You're right.
Quoted 7 lines
Quoted 2 lines
"
xxx_pmd_info" changes are due to JSON formatting (new is more canonical), which can be worked around easily, if the above is wrong.If the new format is better, please keep it. What we need is an exception for the pmdinfo symbols in the file devtools/libabigail.abignore. You can probably use a regex for these symbols.
This would allow real breakages to pass ABI check, abidiff doesn't analyze variable content and it's not easy to compare. Maybe later a script can be added that checks lines with
RTE_DEVICE_INin patches. There are at most 32 of 5494 relevant commits between 19.11 and 20.11, though.Quoted 4 lines
To verify there are no meaningful changes I ensured empty diff between results of the following command for "main" and the branch:
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyFor now we cannot do such check as part of the ABI checker. And we cannot merge this patch if the ABI check fails.
I think the only solution is to allow any change in the pmdinfo variables.Quoted 8 lines
Quoted 5 lines
So my 2c on this is that this is an acceptable work-around for the v21 (DPDK v20.11) ABI. However we are going to end up carrying this rule in libabigail.ignore indefinitely.
Would it make sense to just fix the size of
_pmd_infoto some reasonably large value - say 128 bytes, to allow us to drop the rule in the DPDK 21.11 v22 release?I don't think so. This is a JSON *string to be parsed;* considering its size as part of application *binary* interface is wrong in the first place.
Quoted 7 lines
Right - then is belongs in INTERNAL, I would say.
Quoted 3 lines
As for
content, checking that no PCI IDs are removed is out of scope for libabigail anyway.Lets be clear PCI IDs - are _nothing_ to do with ABI.
Technically, yes, but they're referred to in abi_policy.rst, because DPDK behavior depends on them. Same issue as with as return values: no formats change, yet compatibility is broken.
Quoted 4 lines
Quoted 2 lines
Technically we could fix
_pmd_infosize, but this still allows breaking changes to pass the check with no benefit.ABI changes or other, please explain?
Behavioral changes via PCI ID removal, see above.
PATCH v8 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process.
Add myself as maintainer of new implementation.
--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.28.0
Re: [dpdk-dev] [PATCH v8 3/3] pmdinfogen: remove C implementation
Quoted 637 lines · Dmitry Kozlyuk
Delete the files no longer used in build process.
Add myself as maintainer of new implementation.Signed-off-by Dmitry Kozlyuk <[email protected]>--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.hMAINTAINERS +2
-1@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rstbuildtools/pmdinfogen/meson.build deleted +0
-14@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)buildtools/pmdinfogen/pmdinfogen.c deleted +0
-456@@ -1,456 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 -* Postprocess pmd object files to export hw support -* -* Copyright 2016 Neil Horman <[email protected]> -* Based in part on modpost.c from the linux kernel -*/ - -#include <stdio.h> -#include <ctype.h> -#include <string.h> -#include <limits.h> -#include <stdbool.h> -#include <errno.h> -#include <libgen.h> - -#include <rte_common.h> -#include "pmdinfogen.h" - -#ifdef RTE_ARCH_64 -#define ADDR_SIZE 64 -#else -#define ADDR_SIZE 32 -#endif - -static int use_stdin, use_stdout; - -static const char *sym_name (struct elf_info *elf,Elf_Sym *sym) -{ -if (sym) -return elf->strtab + sym->st_name; -else -return "(unknown)" ; -} - -static void *grab_file (const char *filename,unsigned long *size) -{ -struct stat st; -void *map =MAP_FAILED ; -int fd =-1 ; - -if (!use_stdin) { - fd =open (filename,O_RDONLY ); -if (fd <0 ) -return NULL ; - }else { -/* from stdin, use a temporary file to mmap */ -FILE *infile; -char buffer[1024 ]; -int n; - - infile =tmpfile (); -if (infile ==NULL ) { -perror ("tmpfile" ); -return NULL ; - } - fd =dup (fileno (infile)); -fclose (infile); -if (fd <0 ) -return NULL ; - - n =read (STDIN_FILENO , buffer,sizeof (buffer)); -while (n >0 ) { -if (write (fd, buffer, n) != n) - goto failed; - n =read (STDIN_FILENO , buffer,sizeof (buffer)); - } - } - -if (fstat (fd, &st)) - goto failed; - - *size = st.st_size; - map =mmap (NULL , *size,PROT_READ |PROT_WRITE ,MAP_PRIVATE , fd,0 ); - -failed: -close (fd); -if (map ==MAP_FAILED ) -return NULL ; -return map; -} - -/** -* Return a copy of the next line in a mmap'ed file. -* spaces in the beginning of the line is trimmed away. -* Return a pointer to a static buffer. -**/ -static void release_file (void *file,unsigned long size) -{ -munmap (file, size); -} - - -static void *get_sym_value (struct elf_info *info,const Elf_Sym *sym) -{ -return RTE_PTR_ADD (info->hdr, - info->sechdrs[sym->st_shndx].sh_offset + sym->st_value); -} - -static Elf_Sym *find_sym_in_symtab (struct elf_info *info, -const char *name,Elf_Sym *last) -{ -Elf_Sym *idx; -if (last) - idx = last+1 ; -else - idx = info->symtab_start; - -for (; idx < info->symtab_stop; idx++) { -const char *n =sym_name (info, idx); -if (!strncmp (n, name,strlen (name))) -return idx; - } -return NULL ; -} - -static int parse_elf (struct elf_info *info,const char *filename) -{ -unsigned int i; -Elf_Ehdr *hdr; -Elf_Shdr *sechdrs; -Elf_Sym *sym; -int endian; -unsigned int symtab_idx = ~0U , symtab_shndx_idx = ~0U ; - - hdr =grab_file (filename, &info->size); -if (!hdr) { -perror (filename); -exit (1 ); - } - info->hdr = hdr; -if (info->size <sizeof (*hdr)) { -/* file too small, assume this is an empty .o file */ -return 0 ; - } -/* Is this a valid ELF file? */ -if ((hdr->e_ident[EI_MAG0 ] !=ELFMAG0 ) || - (hdr->e_ident[EI_MAG1 ] !=ELFMAG1 ) || - (hdr->e_ident[EI_MAG2 ] !=ELFMAG2 ) || - (hdr->e_ident[EI_MAG3 ] !=ELFMAG3 )) { -/* Not an ELF file - silently ignore it */ -return 0 ; - } - -if (!hdr->e_ident[EI_DATA ]) { -/* Unknown endian */ -return 0 ; - } - - endian = hdr->e_ident[EI_DATA ]; - -/* Fix endianness in ELF header */ - hdr->e_type =TO_NATIVE (endian,16 , hdr->e_type); - hdr->e_machine =TO_NATIVE (endian,16 , hdr->e_machine); - hdr->e_version =TO_NATIVE (endian,32 , hdr->e_version); - hdr->e_entry =TO_NATIVE (endian,ADDR_SIZE , hdr->e_entry); - hdr->e_phoff =TO_NATIVE (endian,ADDR_SIZE , hdr->e_phoff); - hdr->e_shoff =TO_NATIVE (endian,ADDR_SIZE , hdr->e_shoff); - hdr->e_flags =TO_NATIVE (endian,32 , hdr->e_flags); - hdr->e_ehsize =TO_NATIVE (endian,16 , hdr->e_ehsize); - hdr->e_phentsize =TO_NATIVE (endian,16 , hdr->e_phentsize); - hdr->e_phnum =TO_NATIVE (endian,16 , hdr->e_phnum); - hdr->e_shentsize =TO_NATIVE (endian,16 , hdr->e_shentsize); - hdr->e_shnum =TO_NATIVE (endian,16 , hdr->e_shnum); - hdr->e_shstrndx =TO_NATIVE (endian,16 , hdr->e_shstrndx); - - sechdrs =RTE_PTR_ADD (hdr, hdr->e_shoff); - info->sechdrs = sechdrs; - -/* Check if file offset is correct */ -if (hdr->e_shoff > info->size) { -fprintf (stderr,"section header offset=%lu in file '%s' " -"is bigger than filesize=%lu\n" , - (unsigned long )hdr->e_shoff, - filename, info->size); -return 0 ; - } - -if (hdr->e_shnum ==SHN_UNDEF ) { -/* -* There are more than 64k sections, -* read count from .sh_size. -*/ - info->num_sections = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[0 ].sh_size); - }else { - info->num_sections = hdr->e_shnum; - } -if (hdr->e_shstrndx ==SHN_XINDEX ) - info->secindex_strings = -TO_NATIVE (endian,32 , sechdrs[0 ].sh_link); -else - info->secindex_strings = hdr->e_shstrndx; - -/* Fix endianness in section headers */ -for (i =0 ; i < info->num_sections; i++) { - sechdrs[i].sh_name = -TO_NATIVE (endian,32 , sechdrs[i].sh_name); - sechdrs[i].sh_type = -TO_NATIVE (endian,32 , sechdrs[i].sh_type); - sechdrs[i].sh_flags = -TO_NATIVE (endian,32 , sechdrs[i].sh_flags); - sechdrs[i].sh_addr = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[i].sh_addr); - sechdrs[i].sh_offset = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[i].sh_offset); - sechdrs[i].sh_size = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[i].sh_size); - sechdrs[i].sh_link = -TO_NATIVE (endian,32 , sechdrs[i].sh_link); - sechdrs[i].sh_info = -TO_NATIVE (endian,32 , sechdrs[i].sh_info); - sechdrs[i].sh_addralign = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[i].sh_addralign); - sechdrs[i].sh_entsize = -TO_NATIVE (endian,ADDR_SIZE , sechdrs[i].sh_entsize); - } -/* Find symbol table. */ -for (i =1 ; i < info->num_sections; i++) { -int nobits = sechdrs[i].sh_type ==SHT_NOBITS ; - -if (!nobits && sechdrs[i].sh_offset > info->size) { -fprintf (stderr,"%s is truncated. " -"sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n" , - filename, (unsigned long )sechdrs[i].sh_offset, -sizeof (*hdr)); -return 0 ; - } - -if (sechdrs[i].sh_type ==SHT_SYMTAB ) { -unsigned int sh_link_idx; - symtab_idx = i; - info->symtab_start =RTE_PTR_ADD (hdr, - sechdrs[i].sh_offset); - info->symtab_stop =RTE_PTR_ADD (hdr, - sechdrs[i].sh_offset + sechdrs[i].sh_size); - sh_link_idx = sechdrs[i].sh_link; - info->strtab =RTE_PTR_ADD (hdr, - sechdrs[sh_link_idx].sh_offset); - } - -/* 32bit section no. table? ("more than 64k sections") */ -if (sechdrs[i].sh_type ==SHT_SYMTAB_SHNDX ) { - symtab_shndx_idx = i; - info->symtab_shndx_start =RTE_PTR_ADD (hdr, - sechdrs[i].sh_offset); - info->symtab_shndx_stop =RTE_PTR_ADD (hdr, - sechdrs[i].sh_offset + sechdrs[i].sh_size); - } - } -if (!info->symtab_start) -fprintf (stderr,"%s has no symtab?\n" , filename); -else { -/* Fix endianness in symbols */ -for (sym = info->symtab_start; sym < info->symtab_stop; sym++) { - sym->st_shndx =TO_NATIVE (endian,16 , sym->st_shndx); - sym->st_name =TO_NATIVE (endian,32 , sym->st_name); - sym->st_value =TO_NATIVE (endian,ADDR_SIZE , sym->st_value); - sym->st_size =TO_NATIVE (endian,ADDR_SIZE , sym->st_size); - } - } - -if (symtab_shndx_idx != ~0U ) { -Elf32_Word *p; -if (symtab_idx != sechdrs[symtab_shndx_idx].sh_link) -fprintf (stderr, -"%s: SYMTAB_SHNDX has bad sh_link: %u!=%u\n" , - filename, sechdrs[symtab_shndx_idx].sh_link, - symtab_idx); -/* Fix endianness */ -for (p = info->symtab_shndx_start; p < info->symtab_shndx_stop; - p++) - *p =TO_NATIVE (endian,32 , *p); - } - -return 1 ; -} - -static void parse_elf_finish (struct elf_info *info) -{ -struct pmd_driver *tmp, *idx = info->drivers; -release_file (info->hdr, info->size); -while (idx) { - tmp = idx->next; -free (idx); - idx = tmp; - } -} - -struct opt_tag { -const char *suffix; -const char *json_id; -}; - -static const struct opt_tag opt_tags[] = { - {"_param_string_export" ,"params" }, - {"_kmod_dep_export" ,"kmod" }, -}; - -static int complete_pmd_entry (struct elf_info *info,struct pmd_driver *drv) -{ -const char *tname; -int i; -char tmpsymname[128 ]; -Elf_Sym *tmpsym; - - drv->name =get_sym_value (info, drv->name_sym); - -for (i =0 ; i <PMD_OPT_MAX ; i++) { -memset (tmpsymname,0 ,128 ); -sprintf (tmpsymname,"__%s%s" , drv->name, opt_tags[i].suffix); - tmpsym =find_sym_in_symtab (info, tmpsymname,NULL ); -if (!tmpsym) -continue ; - drv->opt_vals[i] =get_sym_value (info, tmpsym); - } - -memset (tmpsymname,0 ,128 ); -sprintf (tmpsymname,"__%s_pci_tbl_export" , drv->name); - - tmpsym =find_sym_in_symtab (info, tmpsymname,NULL ); - - -/* -* If this returns NULL, then this is a PMD_VDEV, because -* it has no pci table reference -*/ -if (!tmpsym) { - drv->pci_tbl =NULL ; -return 0 ; - } - - tname =get_sym_value (info, tmpsym); - tmpsym =find_sym_in_symtab (info, tname,NULL ); -if (!tmpsym) -return -ENOENT ; - - drv->pci_tbl = (struct rte_pci_id *)get_sym_value (info, tmpsym); -if (!drv->pci_tbl) -return -ENOENT ; - -return 0 ; -} - -static int locate_pmd_entries (struct elf_info *info) -{ -Elf_Sym *last =NULL ; -struct pmd_driver *new; - - info->drivers =NULL ; - -do { - new =calloc (sizeof (struct pmd_driver ),1 ); -if (new ==NULL ) { -fprintf (stderr,"Failed to calloc memory\n" ); -return -1 ; - } - new->name_sym =find_sym_in_symtab (info,"this_pmd_name" , last); - last = new->name_sym; -if (!new->name_sym) -free (new); -else { -if (complete_pmd_entry (info, new)) { -fprintf (stderr, -"Failed to complete pmd entry\n" ); -free (new); - }else { - new->next = info->drivers; - info->drivers = new; - } - } - }while (last); - -return 0 ; -} - -static void output_pmd_info_string (struct elf_info *info,char *outfile) -{ -FILE *ofd; -struct pmd_driver *drv; -struct rte_pci_id *pci_ids; -int idx =0 ; - -if (use_stdout) - ofd = stdout; -else { - ofd =fopen (outfile,"w+" ); -if (!ofd) { -fprintf (stderr,"Unable to open output file\n" ); -return ; - } - } - - drv = info->drivers; - -while (drv) { -fprintf (ofd,"const char %s_pmd_info[] __attribute__((used)) = " -"\"PMD_INFO_STRING= {" , - drv->name); -fprintf (ofd,"\\\"name\\\" : \\\"%s\\\", " , drv->name); - -for (idx =0 ; idx <PMD_OPT_MAX ; idx++) { -if (drv->opt_vals[idx]) -fprintf (ofd,"\\\"%s\\\" : \\\"%s\\\", " , - opt_tags[idx].json_id, - drv->opt_vals[idx]); - } - - pci_ids = drv->pci_tbl; -fprintf (ofd,"\\\"pci_ids\\\" : [" ); - -while (pci_ids && pci_ids->device_id) { -fprintf (ofd,"[%d, %d, %d, %d]" , - pci_ids->vendor_id, pci_ids->device_id, - pci_ids->subsystem_vendor_id, - pci_ids->subsystem_device_id); - pci_ids++; -if (pci_ids->device_id) -fprintf (ofd,"," ); -else -fprintf (ofd," " ); - } -fprintf (ofd,"]}\";\n" ); - drv = drv->next; - } - -fclose (ofd); -} - -int main (int argc,char **argv) -{ -struct elf_info info = {0 }; -int rc =1 ; - -if (argc <3 ) { -fprintf (stderr, -"usage: %s <object file> <c output file>\n" , -basename (argv[0 ])); -exit (127 ); - } - use_stdin = !strcmp (argv[1 ],"-" ); - use_stdout = !strcmp (argv[2 ],"-" ); -parse_elf (&info, argv[1 ]); - -if (locate_pmd_entries (&info) <0 ) -exit (1 ); - -if (info.drivers) { -output_pmd_info_string (&info, argv[2 ]); - rc =0 ; - }else { -fprintf (stderr,"No drivers registered\n" ); - } - -parse_elf_finish (&info); -exit (rc); -}buildtools/pmdinfogen/pmdinfogen.h deleted +0
-119@@ -1,119 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 -* Postprocess pmd object files to export hw support -* -* Copyright 2016 Neil Horman <[email protected]> -* Based in part on modpost.c from the linux kernel -*/ - -#include <stdio.h> -#include <stdlib.h> -#include <stdarg.h> -#include <string.h> -#include <sys/types.h> -#include <sys/stat.h> -#include <sys/mman.h> -#ifdef __linux__ -#include <endian.h> -#else -#include <sys/endian.h> -#endif -#include <fcntl.h> -#include <unistd.h> -#include <elf.h> -#include <rte_pci.h> - -/* On BSD-alike OSes elf.h defines these according to host's word size */ -#undef ELF_ST_BIND -#undef ELF_ST_TYPE -#undef ELF_R_SYM -#undef ELF_R_TYPE - -/* -* Define ELF64_* to ELF_*, the latter being defined in both 32 and 64 bit -* flavors in elf.h. This makes our code a bit more generic between arches -* and allows us to support 32 bit code in the future should we ever want to -*/ -#ifdef RTE_ARCH_64 -#define Elf_Ehdr Elf64_Ehdr -#define Elf_Shdr Elf64_Shdr -#define Elf_Sym Elf64_Sym -#define Elf_Addr Elf64_Addr -#define Elf_Sword Elf64_Sxword -#define Elf_Section Elf64_Half -#define ELF_ST_BIND ELF64_ST_BIND -#define ELF_ST_TYPE ELF64_ST_TYPE - -#define Elf_Rel Elf64_Rel -#define Elf_Rela Elf64_Rela -#define ELF_R_SYM ELF64_R_SYM -#define ELF_R_TYPE ELF64_R_TYPE -#else -#define Elf_Ehdr Elf32_Ehdr -#define Elf_Shdr Elf32_Shdr -#define Elf_Sym Elf32_Sym -#define Elf_Addr Elf32_Addr -#define Elf_Sword Elf32_Sxword -#define Elf_Section Elf32_Half -#define ELF_ST_BIND ELF32_ST_BIND -#define ELF_ST_TYPE ELF32_ST_TYPE - -#define Elf_Rel Elf32_Rel -#define Elf_Rela Elf32_Rela -#define ELF_R_SYM ELF32_R_SYM -#define ELF_R_TYPE ELF32_R_TYPE -#endif - - -/* -* Note, it seems odd that we have both a CONVERT_NATIVE and a TO_NATIVE macro -* below. We do this because the values passed to TO_NATIVE may themselves be -* macros and need both macros here to get expanded. Specifically its the width -* variable we are concerned with, because it needs to get expanded prior to -* string concatenation -*/ -#define CONVERT_NATIVE (fend, width, x) ({ \ -typeof(x) ___x; \ -if ((fend) == ELFDATA2LSB) \ - ___x = le##width##toh(x); \ -else \ - ___x = be##width##toh(x); \ -___x; \ -}) - -#define TO_NATIVE (fend, width, x) CONVERT_NATIVE(fend, width, x) - -enum opt_params { -PMD_PARAM_STRING =0 , -PMD_KMOD_DEP , -PMD_OPT_MAX -}; - -struct pmd_driver { -Elf_Sym *name_sym; -const char *name; -struct rte_pci_id *pci_tbl; -struct pmd_driver *next; - -const char *opt_vals[PMD_OPT_MAX ]; -}; - -struct elf_info { -unsigned long size; -Elf_Ehdr *hdr; -Elf_Shdr *sechdrs; -Elf_Sym *symtab_start; -Elf_Sym *symtab_stop; -char *strtab; - -/* support for 32bit section numbers */ - -unsigned int num_sections;/* max_secindex + 1 */ -unsigned int secindex_strings; -/* if Nth symbol table entry has .st_shndx = SHN_XINDEX, -* take shndx from symtab_shndx_start[N] instead -*/ -Elf32_Word *symtab_shndx_start; -Elf32_Word *symtab_shndx_stop; - -struct pmd_driver *drivers; -};Signature · 1 line
2.28.0
v9 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.
Canonical JSON formatting of generated strings raises ABI warnings. There are no meaningful changes, which can be checked by comparing output of pmdinfo before and after the patch:
find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
---
Changes in v9:
* Document pyelftools requirement for FreeBSD (Thomas).
* Add pyelftools to GitHub workflow.
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.github/workflows/build.yml | 4 +- .travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 15 + buildtools/pmdinfogen.py | 189 +++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 -------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 ------- doc/guides/freebsd_gsg/build_dpdk.rst | 3 +- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 13 files changed, 221 insertions(+), 599 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.29.2
PATCH v9 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image.get_section_data(self._symbol["st_shndx"]) + base = self._symbol["st_value"] + return data[base:base + self.size] + + @property + def string_value(self): + value = self.value + return value[:-1].decode() if value else "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def get_section_data(self, name): + return self._image.get_section(name).data() + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + return ELFSymbol(self, symbol[0]) if symbol else None + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol) + + return driver + + @staticmethod + def _load_pci_ids(image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + result = [] + for pci_id in pci_ids: + if not pci_id.device_id: + break + result.append([ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ]) + return result + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.29.2
PATCH v9 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukUse the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen.
Add configure-time check for elftools Python module for Unix hosts.
Add pyelftools to CI configuration and build requirements for Linux and FreeBSD. Currently neither Windows uses pmdinfogen, nor is COFF (PE) supported.
--- .github/workflows/build.yml | 4 ++-- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 15 +++++++++++++++ doc/guides/freebsd_gsg/build_dpdk.rst | 3 ++- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 8 files changed, 30 insertions(+), 9 deletions(-)
.github/workflows/build.yml +2 -2
@@ -91,8 +91,8 @@ jobs: run: sudo apt update - name: Install packages run: sudo apt install -y ccache libnuma-dev python3-setuptools - python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev - libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev + python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev + libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev - name: Install libabigail build dependencies if no cache is available if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true' run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -1
@@ -14,7 +14,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +15 -0
@@ -17,3 +17,18 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py') + +# TODO: starting from Meson 0.51.0 use +# python3 = import('python').find_installation('python', +# modules : python3_required_modules) +python3_required_modules = [] +if host_machine.system() != 'windows' + python3_required_modules = ['elftools'] +endif +foreach module : python3_required_modules + script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)' + if run_command(py3, '-c', script.format(module)).returncode() != 0 + error('missing python module: @0@'.format(module)) + endif +endforeach
doc/guides/freebsd_gsg/build_dpdk.rst +2 -1
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK: * meson * ninja * pkgconf +* py37-pyelftools These can be installed using (as root):: - pkg install meson pkgconf + pkg install meson pkgconf py37-pyelftools To compile the required kernel modules for memory management and working with physical NIC devices, the kernel sources for FreeBSD also
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -52,6 +52,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -132,7 +132,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.29.2
PATCH v9 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process.
Add myself as maintainer of new implementation.
--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.29.2
v10 cover pmdinfogen: rewrite in Python
In reply to Dmitry KozlyukThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.
Identity of generated PMD information is checked by comparing output of pmdinfo before and after the patch:
find build/drivers -name '*.so' -exec usertools/dpdk-pmdinfo.py
---
Changes in v10:
* Suppress ABI warnings for generated strings (Thomas).
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
.github/workflows/build.yml | 4 +- .travis.yml | 2 +- MAINTAINERS | 3 +- buildtools/gen-pmdinfo-cfile.sh | 6 +- buildtools/meson.build | 15 + buildtools/pmdinfogen.py | 189 +++++++++++ buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 -------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 ------- devtools/libabigail.abignore | 4 + doc/guides/freebsd_gsg/build_dpdk.rst | 3 +- doc/guides/linux_gsg/sys_reqs.rst | 6 + drivers/meson.build | 2 +- meson.build | 1 - 14 files changed, 225 insertions(+), 599 deletions(-) create mode 100755 buildtools/pmdinfogen.py delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
Signature · 1 line
2.29.2
PATCH v10 1/3 pmdinfogen: add Python implementation
Using a high-level, interpreted language simplifies maintenance and build process. Furthermore, ELF handling is delegated to pyelftools package. Original logic is kept, the copyright recognizes that.
--- buildtools/pmdinfogen.py | 189 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100755 buildtools/pmdinfogen.py
buildtools/pmdinfogen.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2016 Neil Horman <[email protected]> +# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]> + +import argparse +import ctypes +import json +import sys +import tempfile + +from elftools.elf.elffile import ELFFile +from elftools.elf.sections import SymbolTableSection + + +class ELFSymbol: + def __init__(self, image, symbol): + self._image = image + self._symbol = symbol + + @property + def size(self): + return self._symbol["st_size"] + + @property + def value(self): + data = self._image.get_section_data(self._symbol["st_shndx"]) + base = self._symbol["st_value"] + return data[base:base + self.size] + + @property + def string_value(self): + value = self.value + return value[:-1].decode() if value else "" + + +class ELFImage: + def __init__(self, data): + self._image = ELFFile(data) + self._symtab = self._image.get_section_by_name(".symtab") + if not isinstance(self._symtab, SymbolTableSection): + raise Exception(".symtab section is not a symbol table") + + @property + def is_big_endian(self): + return not self._image.little_endian + + def get_section_data(self, name): + return self._image.get_section(name).data() + + def find_by_name(self, name): + symbol = self._symtab.get_symbol_by_name(name) + return ELFSymbol(self, symbol[0]) if symbol else None + + def find_by_prefix(self, prefix): + for i in range(self._symtab.num_symbols()): + symbol = self._symtab.get_symbol(i) + if symbol.name.startswith(prefix): + yield ELFSymbol(self, symbol) + + +def define_rte_pci_id(is_big_endian): + base_type = ctypes.LittleEndianStructure + if is_big_endian: + base_type = ctypes.BigEndianStructure + + class rte_pci_id(base_type): + _pack_ = True + _fields_ = [ + ("class_id", ctypes.c_uint32), + ("vendor_id", ctypes.c_uint16), + ("device_id", ctypes.c_uint16), + ("subsystem_vendor_id", ctypes.c_uint16), + ("subsystem_device_id", ctypes.c_uint16), + ] + + return rte_pci_id + + +class Driver: + OPTIONS = [ + ("params", "_param_string_export"), + ("kmod", "_kmod_dep_export"), + ] + + def __init__(self, name, options): + self.name = name + for key, value in options.items(): + setattr(self, key, value) + self.pci_ids = [] + + @classmethod + def load(cls, image, symbol): + name = symbol.string_value + + options = {} + for key, suffix in cls.OPTIONS: + option_symbol = image.find_by_name("__%s%s" % (name, suffix)) + if option_symbol: + value = option_symbol.string_value + options[key] = value + + driver = cls(name, options) + + pci_table_name_symbol = image.find_by_name("__%s_pci_tbl_export" % name) + if pci_table_name_symbol: + driver.pci_ids = cls._load_pci_ids(image, pci_table_name_symbol) + + return driver + + @staticmethod + def _load_pci_ids(image, table_name_symbol): + table_name = table_name_symbol.string_value + table_symbol = image.find_by_name(table_name) + if not table_symbol: + raise Exception("PCI table declared but not defined: %d" % table_name) + + rte_pci_id = define_rte_pci_id(image.is_big_endian) + + pci_id_size = ctypes.sizeof(rte_pci_id) + pci_ids_desc = rte_pci_id * (table_symbol.size // pci_id_size) + pci_ids = pci_ids_desc.from_buffer_copy(table_symbol.value) + result = [] + for pci_id in pci_ids: + if not pci_id.device_id: + break + result.append([ + pci_id.vendor_id, + pci_id.device_id, + pci_id.subsystem_vendor_id, + pci_id.subsystem_device_id, + ]) + return result + + def dump(self, file): + dumped = json.dumps(self.__dict__) + escaped = dumped.replace('"', '\\"') + print( + 'const char %s_pmd_info[] __attribute__((used)) = "PMD_INFO_STRING= %s";' + % (self.name, escaped), + file=file, + ) + + +def load_drivers(image): + drivers = [] + for symbol in image.find_by_prefix("this_pmd_name"): + drivers.append(Driver.load(image, symbol)) + return drivers + + +def dump_drivers(drivers, file): + # Keep legacy order of definitions. + for driver in reversed(drivers): + driver.dump(file) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument("input", help="input object file path or '-' for stdin") + parser.add_argument("output", help="output C file path or '-' for stdout") + return parser.parse_args() + + +def open_input(path): + if path == "-": + temp = tempfile.TemporaryFile() + temp.write(sys.stdin.buffer.read()) + return temp + return open(path, "rb") + + +def open_output(path): + if path == "-": + return sys.stdout + return open(path, "w") + + +def main(): + args = parse_args() + infile = open_input(args.input) + image = ELFImage(infile) + drivers = load_drivers(image) + output = open_output(args.output) + dump_drivers(drivers, output) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.29.2
PATCH v10 2/3 build: use Python pmdinfogen
In reply to Dmitry KozlyukUse the same interpreter to run pmdinfogen as for other build scripts. Adjust wrapper script accordingly and also don't suppress stderr from ar and pmdinfogen. Add configure-time check for elftools Python module for Unix hosts.
Add pyelftools to CI configuration and build requirements for Linux and FreeBSD. Windows targets are not currently using pmdinfogen.
Suppress ABI warnings about generated PMD information strings.
--- .github/workflows/build.yml | 4 ++-- .travis.yml | 2 +- buildtools/gen-pmdinfo-cfile.sh | 6 +++--- buildtools/meson.build | 15 +++++++++++++++ devtools/libabigail.abignore | 4 ++++ doc/guides/freebsd_gsg/build_dpdk.rst | 3 ++- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++++ drivers/meson.build | 2 +- meson.build | 1 - 9 files changed, 34 insertions(+), 9 deletions(-)
.github/workflows/build.yml +2 -2
@@ -91,8 +91,8 @@ jobs: run: sudo apt update - name: Install packages run: sudo apt install -y ccache libnuma-dev python3-setuptools - python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev - libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev + python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev + libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev - name: Install libabigail build dependencies if no cache is available if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true' run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -1
@@ -14,7 +14,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build] - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev] _aarch64_packages: &aarch64_packages
buildtools/gen-pmdinfo-cfile.sh +3 -3
@@ -4,11 +4,11 @@ arfile=$1 output=$2 -pmdinfogen=$3 +shift 2 +pmdinfogen=$* # The generated file must not be empty if compiled in pedantic mode echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output for ofile in `ar t $arfile` ; do - ar p $arfile $ofile | $pmdinfogen - - >> $output 2> /dev/null + ar p $arfile $ofile | $pmdinfogen - - >> $output done -exit 0
buildtools/meson.build +15 -0
@@ -17,3 +17,18 @@ else endif map_to_win_cmd = py3 + files('map_to_win.py') sphinx_wrapper = py3 + files('call-sphinx-build.py') +pmdinfogen = py3 + files('pmdinfogen.py') + +# TODO: starting from Meson 0.51.0 use +# python3 = import('python').find_installation('python', +# modules : python3_required_modules) +python3_required_modules = [] +if host_machine.system() != 'windows' + python3_required_modules = ['elftools'] +endif +foreach module : python3_required_modules + script = 'import importlib.util; import sys; exit(importlib.util.find_spec("@0@") is None)' + if run_command(py3, '-c', script.format(module)).returncode() != 0 + error('missing python module: @0@'.format(module)) + endif +endforeach
devtools/libabigail.abignore +4 -0
@@ -16,3 +16,7 @@ [suppress_type] name = rte_cryptodev has_data_member_inserted_between = {0, 1023} + +; Ignore all changes in generated PMD information strings. +[suppress_variable] + name_regex = _pmd_info$
doc/guides/freebsd_gsg/build_dpdk.rst +2 -1
@@ -14,10 +14,11 @@ The following FreeBSD packages are required to build DPDK: * meson * ninja * pkgconf +* py37-pyelftools These can be installed using (as root):: - pkg install meson pkgconf + pkg install meson pkgconf py37-pyelftools To compile the required kernel modules for memory management and working with physical NIC devices, the kernel sources for FreeBSD also
doc/guides/linux_gsg/sys_reqs.rst +6 -0
@@ -52,6 +52,12 @@ Compilation of the DPDK * If the packaged version is below the minimum version, the latest versions can be installed from Python's "pip" repository: ``pip3 install meson ninja`` +* ``pyelftools`` (version 0.22+) + + * For RHEL/Fedora systems it can be installed using ``dnf install python-pyelftools`` + + * For Ubuntu/Debian it can be installed using ``apt install python3-pyelftools`` + * Library for handling NUMA (Non Uniform Memory Access). * ``numactl-devel`` in RHEL/Fedora;
drivers/meson.build +1 -1
@@ -132,7 +132,7 @@ foreach subpath:subdirs command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, - depends: [pmdinfogen, tmp_lib]) + depends: [tmp_lib]) endif # now build the static driver
meson.build +0 -1
@@ -45,7 +45,6 @@ subdir('buildtools') subdir('config') # build libs and drivers -subdir('buildtools/pmdinfogen') subdir('lib') subdir('drivers')
Signature · 1 line
2.29.2
Re: [dpdk-dev] [PATCH v10 2/3] build: use Python pmdinfogen
24/01/2021 21:51, Dmitry Kozlyuk:
devtools/libabigail.abignore line 22
Quoted 5 lines
devtools/libabigail.abignore +3
-0+; Ignore all changes in generated PMD information strings. +[suppress_variable] + name_regex = _pmd_info$
I will fix this rule as "name_regexp" so it works.
PATCH v10 3/3 pmdinfogen: remove C implementation
In reply to Dmitry KozlyukDelete the files no longer used in build process.
Add myself as maintainer of new implementation.
--- MAINTAINERS | 3 +- buildtools/pmdinfogen/meson.build | 14 - buildtools/pmdinfogen/pmdinfogen.c | 456 ----------------------------- buildtools/pmdinfogen/pmdinfogen.h | 119 -------- 4 files changed, 2 insertions(+), 590 deletions(-) delete mode 100644 buildtools/pmdinfogen/meson.build delete mode 100644 buildtools/pmdinfogen/pmdinfogen.c delete mode 100644 buildtools/pmdinfogen/pmdinfogen.h
MAINTAINERS +2 -1
@@ -133,7 +133,8 @@ F: lib/*/*.map Driver information M: Neil Horman <[email protected]> -F: buildtools/pmdinfogen/ +M: Dmitry Kozlyuk <[email protected]> +F: buildtools/pmdinfogen.py F: usertools/dpdk-pmdinfo.py F: doc/guides/tools/pmdinfo.rst
buildtools/pmdinfogen/meson.build deleted +0 -14
@@ -1,14 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2017 Intel Corporation - -if is_windows - subdir_done() -endif - -pmdinfogen_inc = [global_inc] -pmdinfogen_inc += include_directories('../../lib/librte_eal/include') -pmdinfogen_inc += include_directories('../../lib/librte_pci') -pmdinfogen = executable('pmdinfogen', - 'pmdinfogen.c', - include_directories: pmdinfogen_inc, - native: true)
buildtools/pmdinfogen/pmdinfogen.c deleted +0 -456
456 lines not on this page · raw message
buildtools/pmdinfogen/pmdinfogen.h deleted +0 -119
119 lines not on this page · raw message
Signature · 1 line
2.29.2
Thomas Monjalon
AppliedQuoted 4 lines
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementation
Applied, thanks.
Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:
http://mails.dpdk.org/archives/ci/2021-January/000939.html http://mails.dpdk.org/archives/dev/2021-January/197814.html
If not already done, please UPDATE your environment! Thanks
Hi Thomas,
We have updated all UNH-IOL environments to include pyelftools.
Thanks,
Brandon
Quoted 16 lines · Thomas Monjalon
Quoted 5 lines
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementationApplied, thanks.
Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:
http://mails.dpdk.org/archives/ci/2021-January/000939.html http://mails.dpdk.org/archives/dev/2021-January/197814.htmlIf not already done, please UPDATE your environment! Thanks
Signature · 9 lines
Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 [email protected] www.iol.unh.edu
Quoted 8 lines
Subject: Re: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in Python
External email: Use caution opening links or attachments
Hi Thomas,
We have updated all UNH-IOL environments to include pyelftools.
FreeBSD compilation test is still failing on the missing dependency:
buildtools/meson.build:45:2: ERROR: Problem encountered: missing python module: elftools
Quoted 66 lines
Thanks,
BrandonQuoted 11 lines · Thomas Monjalon
Quoted 5 lines
Dmitry Kozlyuk (3):
pmdinfogen: add Python implementation
build: use Python pmdinfogen
pmdinfogen: remove C implementationApplied, thanks.
Reminder: it is adding a NEW DEPENDENCY on pyelftools, as highlighted already in CI and maintainers meetings:
https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.
dpdk.org%2Farchives%2Fci%2F2021- January%2F000939.html&data=04%7C01%7Ctalshn%40nvidia.com%7C70 5c7ce391d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a %7C0%7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIj oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 000&sdata=Lei7o0Mtgsb3WZJ14lGolduVJK3fA7EbYuw8ojVCb3g%3D&a mp;reserved=0Quoted 2 lines
https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails
Quoted 1 line
.dpdk.org%2Farchives%2Fdev%2F2021-
January%2F197814.html&data=04%7C
Quoted 1 line
01%7Ctalshn%40nvidia.com%7C705c7ce391d1429bc94508d8c14b9880%7C430 83d15
Quoted 1 line
727340c1b7db39efd9ccc17a%7C0%7C0%7C637471877725119466%7CUnknow n%7CTWFp
Quoted 1 line
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC I6Mn
Quoted 1 line
0%3D%7C1000&sdata=74Zp6%2B6YB8OCVfo%2F0Gb7C9C43QWAo1YD4 AUANneBGdA%
Quoted 5 lines
3D&reserved=0
If not already done, please UPDATE your environment! Thanks
--
Brandon Lo
UNH InterOperability Laboratory
21 Madbury Rd, Suite 100, Durham, NH 03824
https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww
.iol.unh.edu%2F&data=04%7C01%7Ctalshn%40nvidia.com%7C705c7ce3 91d1429bc94508d8c14b9880%7C43083d15727340c1b7db39efd9ccc17a%7C0% 7C0%7C637471877725119466%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4 wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&a mp;sdata=2Fc4epSpG%2FhLFdxj9D1kVoafaxlfUhaZvpEm4xwHFd4%3D& reserved=0
Hi,
Quoted 20 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Dmitry Kozlyuk
Sent: Sunday, January 24, 2021 10:52 PM
To: [email protected]
Cc: Maxime Coquelin <[email protected]>; Bruce Richardson <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Dmitry Kozlyuk <[email protected]>; Neil Horman <[email protected]>; Jie Zhou <[email protected]>
Subject: [dpdk-dev] [PATCH v10 0/3] pmdinfogen: rewrite in PythonThis patchset implements existing pmdinfogen logic in Python, replaces and removes the old code. The goals of rewriting are:
* easier maintenance by using a more high-level language,
* simpler build process without host application and libelf,
* foundation for adding Windows support.Identity of generated PMD information is checked by comparing output of pmdinfo before and after the patch:
Quoted 6 lines
find build/drivers -name '*.so' -exec
usertools/dpdk-pmdinfo.pyAcked-by Neil Horman <[email protected]>Tested-by Jie Zhou <[email protected]>---
Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED: drivers/rte_common_cpt.pmd.c/usr/bin/python3 ../buildtools/gen-pmdinfo-cfile.py ar /home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.a drivers/rte_common_cpt.pmd.c /usr/bin/python3 ../buildtools/pmdinfogen.py elf Traceback (most recent call last):
File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
[ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'
"""
I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).
Thanks,
Ali
Hi Ali,
Quoted 13 lines
Please see the following build errors:
https://github.com/DPDK/dpdk/actions/runs/509771118
"""
FAILED:drivers/rte_common_cpt.pmd.c/usr/bin/python3../buildtools/gen-pmdinfo-cfile.pyar/home/runner/work/dpdk/dpdk/build/drivers/libtmp_rte_common_cpt.adrivers/rte_common_cpt.pmd.c/usr/bin/python3../buildtools/pmdinfogen.pyelf Traceback (most recent call last):
File "../buildtools/gen-pmdinfo-cfile.py", line 14, in <module>
[ar, "xv",os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
File "/usr/lib/python3.6/subprocess.py", line 423, in run
withPopen(*popenargs, **kwargs)as process:
TypeError:__init__()got an unexpected keyword argument 'capture_output'
"""
Quoted 1 line
I also see the exception in Ubuntu 16.04 (Python 3.5.2, pyelftools 0.27), CentOS 7 (Python 3.6.8, pyelftools 0.27), CentOS 8 (Python 3.6.8, pyelftools 0.27) and OpenSUSE Leap 15.2 (Python 3.6.12, pyelftools 0.26).
If fixed the error you encountered and also discovered a bunch of other issues, all of them hopefully fixed in the following series: