Skip to contentSkip to conversation

pmdinfogen: support Windows

PATCH Superseded Dmitry Kozlyuk 3 participants 13 messages Dec 12, 2020 – Jan 25, 2021 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover pmdinfogen: support Windows

Based on Python version of pmdinfogen, deferred until 21.02:

    http://patchwork.dpdk.org/project/dpdk/list/?series=13153

There are a few Python libraries for PE/COFF, none of which suits the need, so a custom COFF parser is used. Advice is welcome, options considered:

* py-coff (https://github.com/jeppeter/py-coff): doesn't give access to symbol values, Python 2 code inside, not very popular.

* pefile (https://github.com/erocarrera/pefile): for PE (executables and libraries), not COFF (objects); most popular.

* pype32-py3 (https://github.com/crackinglandia/pype32): ditto, less popular.

A script to extract object files from library is still required. Meson has extract_all_objects(), but they can't be passed as inputs to custom_target() until 0.52.0 (commit f431cff809).

Depends-on: series-13153 ("pmdinfogen: rewrite in Python")

Dmitry Kozlyuk (4):
pmdinfogen: support COFF
pmdinfogen: allow multiple input files
buildtools: support object file extraction for Windows
build: enable pmdinfogen for Windows

 buildtools/coff.py              | 154 ++++++++++++++++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.py |  19 ++++
 buildtools/gen-pmdinfo-cfile.sh |  14 ---
 buildtools/meson.build          |  15 +++-
 buildtools/pmdinfogen.py        | 126 +++++++++++++++++++-------
 drivers/meson.build             |  26 +++---
 6 files changed, 294 insertions(+), 60 deletions(-)
 create mode 100644 buildtools/coff.py
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh
Signature · 1 line
2.29.2
#1 lore raw

PATCH 1/4 pmdinfogen: support COFF

Common Object File Format (COFF) is used on Windows in place of ELF.

Add COFF parser to pmdinfogen. Also add an argument to specify input file format, which is selected at configure time based on the target.

---
 buildtools/coff.py       | 154 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   7 ++
 buildtools/pmdinfogen.py | 107 ++++++++++++++++++++-------
 3 files changed, 240 insertions(+), 28 deletions(-)
 create mode 100644 buildtools/coff.py
buildtools/coff.py new file +154 -0
new file mode 100644
index 000000000..86fb0602b
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import ctypes
+
+# x86_64 little-endian
+COFF_MAGIC = 0x8664
+
+# Names up to this length are stored immediately in symbol table entries.
+COFF_NAMELEN = 8
+
+# Special "section numbers" changing the meaning of symbol table entry.
+COFF_SN_UNDEFINED = 0
+COFF_SN_ABSOLUTE = -1
+COFF_SN_DEBUG = -2
+
+
+class CoffFileHeader(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_uint16),
+        ("section_count", ctypes.c_uint16),
+        ("timestamp", ctypes.c_uint32),
+        ("symbol_table_offset", ctypes.c_uint32),
+        ("symbol_count", ctypes.c_uint32),
+        ("optional_header_size", ctypes.c_uint16),
+        ("flags", ctypes.c_uint16),
+    ]
+
+
+class CoffName(ctypes.Union):
+    class Reference(ctypes.LittleEndianStructure):
+        _pack_ = True
+        _fields_ = [
+            ("zeroes", ctypes.c_uint32),
+            ("offset", ctypes.c_uint32),
+        ]
+
+    Immediate = ctypes.c_char * 8
+
+    _pack_ = True
+    _fields_ = [
+        ("immediate", Immediate),
+        ("reference", Reference),
+    ]
+
+
+class CoffSection(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("physical_address", ctypes.c_uint32),
+        ("physical_address", ctypes.c_uint32),
+        ("size", ctypes.c_uint32),
+        ("data_offset", ctypes.c_uint32),
+        ("relocations_offset", ctypes.c_uint32),
+        ("line_numbers_offset", ctypes.c_uint32),
+        ("relocation_count", ctypes.c_uint16),
+        ("line_number_count", ctypes.c_uint16),
+        ("flags", ctypes.c_uint32),
+    ]
+
+
+class CoffSymbol(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("value", ctypes.c_uint32),
+        ("section_number", ctypes.c_int16),
+        ("type", ctypes.c_uint16),
+        ("storage_class", ctypes.c_uint8),
+        ("auxiliary_count", ctypes.c_uint8),
+    ]
+
+
+class Symbol:
+    def __init__(self, image, symbol: CoffSymbol):
+        self._image = image
+        self._coff = symbol
+
+    @property
+    def name(self):
+        if self._coff.name.reference.zeroes:
+            return decode_asciiz(bytes(self._coff.name.immediate))
+
+        offset = self._coff.name.reference.offset
+        offset -= ctypes.sizeof(ctypes.c_uint32)
+        return self._image.get_string(offset)
+
+    def get_value(self, offset):
+        section_number = self._coff.section_number
+
+        if section_number == COFF_SN_UNDEFINED:
+            return None
+
+        if section_number == COFF_SN_DEBUG:
+            return None
+
+        if section_number == COFF_SN_ABSOLUTE:
+            return bytes(ctypes.c_uint32(self._coff.value))
+
+        section_data = self._image.get_section_data(section_number)
+        section_offset = self._coff.value + offset
+        return section_data[section_offset:]
+
+
+class Image:
+    def __init__(self, data):
+        header = CoffFileHeader.from_buffer_copy(data)
+        header_size = ctypes.sizeof(header) + header.optional_header_size
+
+        sections_desc = CoffSection * header.section_count
+        sections = sections_desc.from_buffer_copy(data, header_size)
+
+        symbols_desc = CoffSymbol * header.symbol_count
+        symbols = symbols_desc.from_buffer_copy(data, header.symbol_table_offset)
+
+        strings_offset = header.symbol_table_offset + ctypes.sizeof(symbols)
+        strings = Image._parse_strings(data[strings_offset:])
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._symbols = symbols
+        self._strings = strings
+
+    @staticmethod
+    def _parse_strings(data):
+        full_size = ctypes.c_uint32.from_buffer_copy(data)
+        header_size = ctypes.sizeof(full_size)
+        return data[header_size : full_size.value]
+
+    @property
+    def symbols(self):
+        i = 0
+        while i < self._header.symbol_count:
+            symbol = self._symbols[i]
+            yield Symbol(self, symbol)
+            i += symbol.auxiliary_count + 1
+
+    def get_section_data(self, number):
+        # section numbers are 1-based
+        section = self._sections[number - 1]
+        base = section.data_offset
+        return self._data[base : base + section.size]
+
+    def get_string(self, offset):
+        return decode_asciiz(self._strings[offset:])
+
+
+def decode_asciiz(data):
+    index = data.find(b'\x00')
+    end = index if index >= 0 else len(data)
+    return data[:end].decode()
buildtools/meson.build +7 -0
index dd4c0f640..23cefd4be 100644
@@ -17,7 +17,14 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+
+# select object file format
 pmdinfogen = py3 + files('pmdinfogen.py')
+if host_machine.system() == 'windows'
+	pmdinfogen += 'coff'
+else
+	pmdinfogen += 'elf'
+endif
 
 # TODO: starting from Meson 0.51.0 use
 # 	python3 = import('python').find_installation('python',
buildtools/pmdinfogen.py +79 -28

163 lines not on this page · raw message

Signature · 1 line
2.29.2
#2 lore raw

PATCH 2/4 pmdinfogen: allow multiple input files

In reply to Dmitry Kozlyuk

Process any number of input object files and write a unified output.

---
Used in the next patch, separated for clarity.

 buildtools/pmdinfogen.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
buildtools/pmdinfogen.py +10 -4
index 65d78b872..3209510eb 100755
@@ -194,7 +194,9 @@ def dump_drivers(drivers, file):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument("format", help="object file format, 'elf' or 'coff'")
-    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument(
+        "input", nargs='+', help="input object file path or '-' for stdin"
+    )
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
 
@@ -230,10 +232,14 @@ def open_output(path):
 
 def main():
     args = parse_args()
-    image = load_image(args.format, args.input)
-    drivers = load_drivers(image)
+    if args.input.count('-') > 1:
+        raise Exception("'-' input cannot be used multiple times")
+
     output = open_output(args.output)
-    dump_drivers(drivers, output)
+    for path in args.input:
+        image = load_image(args.format, path)
+        drivers = load_drivers(image)
+        dump_drivers(drivers, output)
 
 
 if __name__ == "__main__":
Signature · 1 line
2.29.2
#3 lore raw

PATCH 3/4 buildtools: support object file extraction for Windows

In reply to Dmitry Kozlyuk

clang archiver tool is llvm-ar on Windows and ar on other platforms. MinGW always uses ar. Replace shell script (Unix-only) that calls ar with a Python script (OS-independent) that calls an appropriate archiver tool selected at configuration time. Move the logic not to generate empty sources into pmdinfogen.

---
Stdin and stdout are not longer used for input and output. Code to handle that could be removed, but maybe it's useful for someone.

 buildtools/gen-pmdinfo-cfile.py | 19 +++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.sh | 14 --------------
 buildtools/meson.build          | 10 ++++++++--
 buildtools/pmdinfogen.py        |  7 +++++++
 4 files changed, 34 insertions(+), 16 deletions(-)
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh
buildtools/gen-pmdinfo-cfile.py new file +19 -0
new file mode 100644
index 000000000..f1f289ffe
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import os
+import subprocess
+import sys
+import tempfile
+
+_, ar, archive, output, *pmdinfogen = sys.argv
+with tempfile.TemporaryDirectory() as temp:
+    proc = subprocess.run(
+        # Don't use "ar p", because its output is corrupted on Windows.
+        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
+    )
+    lines = proc.stdout.decode().splitlines()
+    names = [line[len("x - ") :] for line in lines]
+    paths = [os.path.join(temp, name) for name in names]
+    subprocess.run(pmdinfogen + paths + [output], check=True)
buildtools/gen-pmdinfo-cfile.sh deleted +0 -14
deleted file mode 100755
index 109ee461e..000000000
@@ -1,14 +0,0 @@
-#! /bin/sh
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-arfile=$1
-output=$2
-shift 2
-pmdinfogen=$*
-
-# The generated file must not be empty if compiled in pedantic mode
-echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
-for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output
-done
buildtools/meson.build +8 -2
index 23cefd4be..0a2e91a7b 100644
@@ -2,7 +2,6 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
-pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
 ldflags_ibverbs_static = find_program('options-ibverbs-static.sh')
@@ -18,11 +17,18 @@ endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
 
-# select object file format
+# select library and object file format
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
+	if cc.get_id() == 'gcc'
+		pmdinfo += 'ar'
+	else
+		pmdinfo += 'llvm-ar'
+	endif
 	pmdinfogen += 'coff'
 else
+	pmdinfo += 'ar'
 	pmdinfogen += 'elf'
 endif
 
buildtools/pmdinfogen.py +7 -0
index 3209510eb..56f5f488c 100755
@@ -230,12 +230,19 @@ def open_output(path):
     return open(path, "w")
 
 
+def write_header(output):
+    output.write(
+        "static __attribute__((unused)) const char *generator = \"%s\";\n" % sys.argv[0]
+    )
+
+
 def main():
     args = parse_args()
     if args.input.count('-') > 1:
         raise Exception("'-' input cannot be used multiple times")
 
     output = open_output(args.output)
+    write_header(output)
     for path in args.input:
         image = load_image(args.format, path)
         drivers = load_drivers(image)
Signature · 1 line
2.29.2
#4 lore raw

PATCH 4/4 build: enable pmdinfogen for Windows

In reply to Dmitry Kozlyuk

Remove platform restriction when building drivers.

---
 drivers/meson.build | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)
drivers/meson.build +12 -14
index f49d4f79b..1dfa8738f 100644
@@ -143,20 +143,18 @@ foreach subpath:subdirs
 			# lib and then running pmdinfogen on the contents of
 			# that lib. The final lib reuses the object files and
 			# adds in the new source file.
-			if not is_windows
-				out_filename = lib_name + '.pmd.c'
-				tmp_lib = static_library('tmp_' + lib_name,
-						sources,
-						include_directories: includes,
-						dependencies: static_deps,
-						c_args: cflags)
-				objs += tmp_lib.extract_all_objects()
-				sources = custom_target(out_filename,
-						command: [pmdinfo, tmp_lib.full_path(),
-							'@OUTPUT@', pmdinfogen],
-						output: out_filename,
-						depends: [tmp_lib])
-			endif
+			out_filename = lib_name + '.pmd.c'
+			tmp_lib = static_library('tmp_' + lib_name,
+					sources,
+					include_directories: includes,
+					dependencies: static_deps,
+					c_args: cflags)
+			objs += tmp_lib.extract_all_objects()
+			sources = custom_target(out_filename,
+					command: [pmdinfo, tmp_lib.full_path(),
+						'@OUTPUT@', pmdinfogen],
+					output: out_filename,
+					depends: [tmp_lib])
 
 			# now build the static driver
 			static_lib = static_library(lib_name,
Signature · 1 line
2.29.2
#5 lore raw

v2 cover pmdinfogen: support Windows

In reply to Dmitry Kozlyuk

Based on Python version of pmdinfogen, deferred until 21.02:

    http://patchwork.dpdk.org/project/dpdk/list/?series=13153

There are a few Python libraries for PE/COFF, none of which suits the need, so a custom COFF parser is used. Advice is welcome, options considered:

* py-coff (https://github.com/jeppeter/py-coff): doesn't give access to symbol values, Python 2 code inside, not very popular.

* pefile (https://github.com/erocarrera/pefile): for PE (executables and libraries), not COFF (objects); most popular.

* pype32-py3 (https://github.com/crackinglandia/pype32): ditto, less popular.

A script to extract object files from library is still required. Meson has extract_all_objects(), but they can't be passed as inputs to custom_target() until 0.52.0 (commit f431cff809).

v2:
Don't require elftools on Windows (Jie Zhou).

Dmitry Kozlyuk (4):
pmdinfogen: support COFF
pmdinfogen: allow multiple input files
buildtools: support object file extraction for Windows
build: enable pmdinfogen for Windows

 buildtools/coff.py              | 154 ++++++++++++++++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.py |  19 ++++
 buildtools/gen-pmdinfo-cfile.sh |  14 ---
 buildtools/meson.build          |  15 +++-
 buildtools/pmdinfogen.py        | 135 +++++++++++++++++++++-------
 drivers/meson.build             |  26 +++---
 6 files changed, 301 insertions(+), 62 deletions(-)
 create mode 100644 buildtools/coff.py
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh
Signature · 1 line
2.29.2
#6 lore raw

PATCH v2 1/4 pmdinfogen: support COFF

Common Object File Format (COFF) is used on Windows in place of ELF.

Add COFF parser to pmdinfogen. Also add an argument to specify input file format, which is selected at configure time based on the target.

---
Depends-on: series-13153 ("pmdinfogen: rewrite in Python")

 buildtools/coff.py       | 154 +++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build   |   7 ++
 buildtools/pmdinfogen.py | 117 +++++++++++++++++++++--------
 3 files changed, 248 insertions(+), 30 deletions(-)
 create mode 100644 buildtools/coff.py
buildtools/coff.py new file +154 -0
new file mode 100644
index 000000000..86fb0602b
@@ -0,0 +1,154 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import ctypes
+
+# x86_64 little-endian
+COFF_MAGIC = 0x8664
+
+# Names up to this length are stored immediately in symbol table entries.
+COFF_NAMELEN = 8
+
+# Special "section numbers" changing the meaning of symbol table entry.
+COFF_SN_UNDEFINED = 0
+COFF_SN_ABSOLUTE = -1
+COFF_SN_DEBUG = -2
+
+
+class CoffFileHeader(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("magic", ctypes.c_uint16),
+        ("section_count", ctypes.c_uint16),
+        ("timestamp", ctypes.c_uint32),
+        ("symbol_table_offset", ctypes.c_uint32),
+        ("symbol_count", ctypes.c_uint32),
+        ("optional_header_size", ctypes.c_uint16),
+        ("flags", ctypes.c_uint16),
+    ]
+
+
+class CoffName(ctypes.Union):
+    class Reference(ctypes.LittleEndianStructure):
+        _pack_ = True
+        _fields_ = [
+            ("zeroes", ctypes.c_uint32),
+            ("offset", ctypes.c_uint32),
+        ]
+
+    Immediate = ctypes.c_char * 8
+
+    _pack_ = True
+    _fields_ = [
+        ("immediate", Immediate),
+        ("reference", Reference),
+    ]
+
+
+class CoffSection(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("physical_address", ctypes.c_uint32),
+        ("physical_address", ctypes.c_uint32),
+        ("size", ctypes.c_uint32),
+        ("data_offset", ctypes.c_uint32),
+        ("relocations_offset", ctypes.c_uint32),
+        ("line_numbers_offset", ctypes.c_uint32),
+        ("relocation_count", ctypes.c_uint16),
+        ("line_number_count", ctypes.c_uint16),
+        ("flags", ctypes.c_uint32),
+    ]
+
+
+class CoffSymbol(ctypes.LittleEndianStructure):
+    _pack_ = True
+    _fields_ = [
+        ("name", CoffName),
+        ("value", ctypes.c_uint32),
+        ("section_number", ctypes.c_int16),
+        ("type", ctypes.c_uint16),
+        ("storage_class", ctypes.c_uint8),
+        ("auxiliary_count", ctypes.c_uint8),
+    ]
+
+
+class Symbol:
+    def __init__(self, image, symbol: CoffSymbol):
+        self._image = image
+        self._coff = symbol
+
+    @property
+    def name(self):
+        if self._coff.name.reference.zeroes:
+            return decode_asciiz(bytes(self._coff.name.immediate))
+
+        offset = self._coff.name.reference.offset
+        offset -= ctypes.sizeof(ctypes.c_uint32)
+        return self._image.get_string(offset)
+
+    def get_value(self, offset):
+        section_number = self._coff.section_number
+
+        if section_number == COFF_SN_UNDEFINED:
+            return None
+
+        if section_number == COFF_SN_DEBUG:
+            return None
+
+        if section_number == COFF_SN_ABSOLUTE:
+            return bytes(ctypes.c_uint32(self._coff.value))
+
+        section_data = self._image.get_section_data(section_number)
+        section_offset = self._coff.value + offset
+        return section_data[section_offset:]
+
+
+class Image:
+    def __init__(self, data):
+        header = CoffFileHeader.from_buffer_copy(data)
+        header_size = ctypes.sizeof(header) + header.optional_header_size
+
+        sections_desc = CoffSection * header.section_count
+        sections = sections_desc.from_buffer_copy(data, header_size)
+
+        symbols_desc = CoffSymbol * header.symbol_count
+        symbols = symbols_desc.from_buffer_copy(data, header.symbol_table_offset)
+
+        strings_offset = header.symbol_table_offset + ctypes.sizeof(symbols)
+        strings = Image._parse_strings(data[strings_offset:])
+
+        self._data = data
+        self._header = header
+        self._sections = sections
+        self._symbols = symbols
+        self._strings = strings
+
+    @staticmethod
+    def _parse_strings(data):
+        full_size = ctypes.c_uint32.from_buffer_copy(data)
+        header_size = ctypes.sizeof(full_size)
+        return data[header_size : full_size.value]
+
+    @property
+    def symbols(self):
+        i = 0
+        while i < self._header.symbol_count:
+            symbol = self._symbols[i]
+            yield Symbol(self, symbol)
+            i += symbol.auxiliary_count + 1
+
+    def get_section_data(self, number):
+        # section numbers are 1-based
+        section = self._sections[number - 1]
+        base = section.data_offset
+        return self._data[base : base + section.size]
+
+    def get_string(self, offset):
+        return decode_asciiz(self._strings[offset:])
+
+
+def decode_asciiz(data):
+    index = data.find(b'\x00')
+    end = index if index >= 0 else len(data)
+    return data[:end].decode()
buildtools/meson.build +7 -0
index dd4c0f640..23cefd4be 100644
@@ -17,7 +17,14 @@ else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+
+# select object file format
 pmdinfogen = py3 + files('pmdinfogen.py')
+if host_machine.system() == 'windows'
+	pmdinfogen += 'coff'
+else
+	pmdinfogen += 'elf'
+endif
 
 # TODO: starting from Meson 0.51.0 use
 # 	python3 = import('python').find_installation('python',
buildtools/pmdinfogen.py +87 -30

173 lines not on this page · raw message

Signature · 1 line
2.29.2
#7 lore raw

PATCH v2 2/4 pmdinfogen: allow multiple input files

In reply to Dmitry Kozlyuk

Process any number of input object files and write a unified output.

---
 buildtools/pmdinfogen.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
buildtools/pmdinfogen.py +9 -4
index 474168f21..965c08945 100755
@@ -197,7 +197,9 @@ def dump_drivers(drivers, file):
 def parse_args():
     parser = argparse.ArgumentParser()
     parser.add_argument("format", help="object file format, 'elf' or 'coff'")
-    parser.add_argument("input", help="input object file path or '-' for stdin")
+    parser.add_argument(
+        "input", nargs='+', help="input object file path or '-' for stdin"
+    )
     parser.add_argument("output", help="output C file path or '-' for stdout")
     return parser.parse_args()
 
@@ -233,13 +235,16 @@ def open_output(path):
 
 def main():
     args = parse_args()
+    if args.input.count('-') > 1:
+        raise Exception("'-' input cannot be used multiple times")
     if args.format == "elf" and "ELFFile" not in globals():
         raise Exception("elftools module not found")
 
-    image = load_image(args.format, args.input)
-    drivers = load_drivers(image)
     output = open_output(args.output)
-    dump_drivers(drivers, output)
+    for path in args.input:
+        image = load_image(args.format, path)
+        drivers = load_drivers(image)
+        dump_drivers(drivers, output)
 
 
 if __name__ == "__main__":
Signature · 1 line
2.29.2
#8 lore raw

PATCH v2 3/4 buildtools: support object file extraction for Windows

In reply to Dmitry Kozlyuk

clang archiver tool is llvm-ar on Windows and ar on other platforms. MinGW always uses ar. Replace shell script (Unix-only) that calls ar with a Python script (OS-independent) that calls an appropriate archiver tool selected at configuration time. Move the logic not to generate empty sources into pmdinfogen.

---
 buildtools/gen-pmdinfo-cfile.py | 19 +++++++++++++++++++
 buildtools/gen-pmdinfo-cfile.sh | 14 --------------
 buildtools/meson.build          | 10 ++++++++--
 buildtools/pmdinfogen.py        |  7 +++++++
 4 files changed, 34 insertions(+), 16 deletions(-)
 create mode 100644 buildtools/gen-pmdinfo-cfile.py
 delete mode 100755 buildtools/gen-pmdinfo-cfile.sh
buildtools/gen-pmdinfo-cfile.py new file +19 -0
new file mode 100644
index 000000000..f1f289ffe
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2020 Dmitry Kozlyuk <[email protected]>
+
+import os
+import subprocess
+import sys
+import tempfile
+
+_, ar, archive, output, *pmdinfogen = sys.argv
+with tempfile.TemporaryDirectory() as temp:
+    proc = subprocess.run(
+        # Don't use "ar p", because its output is corrupted on Windows.
+        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
+    )
+    lines = proc.stdout.decode().splitlines()
+    names = [line[len("x - ") :] for line in lines]
+    paths = [os.path.join(temp, name) for name in names]
+    subprocess.run(pmdinfogen + paths + [output], check=True)
buildtools/gen-pmdinfo-cfile.sh deleted +0 -14
deleted file mode 100755
index 109ee461e..000000000
@@ -1,14 +0,0 @@
-#! /bin/sh
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright(c) 2017 Intel Corporation
-
-arfile=$1
-output=$2
-shift 2
-pmdinfogen=$*
-
-# The generated file must not be empty if compiled in pedantic mode
-echo 'static __attribute__((unused)) const char *generator = "'$0'";' > $output
-for ofile in `ar t $arfile` ; do
-	ar p $arfile $ofile | $pmdinfogen - - >> $output
-done
buildtools/meson.build +8 -2
index 23cefd4be..0a2e91a7b 100644
@@ -2,7 +2,6 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 pkgconf = find_program('pkg-config', 'pkgconf', required: false)
-pmdinfo = find_program('gen-pmdinfo-cfile.sh')
 list_dir_globs = find_program('list-dir-globs.py')
 check_symbols = find_program('check-symbols.sh')
 ldflags_ibverbs_static = find_program('options-ibverbs-static.sh')
@@ -18,11 +17,18 @@ endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
 
-# select object file format
+# select library and object file format
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
+	if cc.get_id() == 'gcc'
+		pmdinfo += 'ar'
+	else
+		pmdinfo += 'llvm-ar'
+	endif
 	pmdinfogen += 'coff'
 else
+	pmdinfo += 'ar'
 	pmdinfogen += 'elf'
 endif
 
buildtools/pmdinfogen.py +7 -0
index 965c08945..893a6c571 100755
@@ -233,6 +233,12 @@ def open_output(path):
     return open(path, "w")
 
 
+def write_header(output):
+    output.write(
+        "static __attribute__((unused)) const char *generator = \"%s\";\n" % sys.argv[0]
+    )
+
+
 def main():
     args = parse_args()
     if args.input.count('-') > 1:
@@ -241,6 +247,7 @@ def main():
         raise Exception("elftools module not found")
 
     output = open_output(args.output)
+    write_header(output)
     for path in args.input:
         image = load_image(args.format, path)
         drivers = load_drivers(image)
Signature · 1 line
2.29.2
#9 lore raw

Re: [dpdk-dev] [PATCH v2 3/4] buildtools: support object file extraction for Windows

08/01/2021 03:47, Dmitry Kozlyuk:

Quoted 2 lines
buildtools/gen-pmdinfo-cfile.py new file +0 -0

[...]

buildtools/gen-pmdinfo-cfile.py line 15

Quoted 5 lines
buildtools/gen-pmdinfo-cfile.py new file +5 -0
+with tempfile.TemporaryDirectory() as temp:
+    proc = subprocess.run(
+        # Don't use "ar p", because its output is corrupted on Windows.
+        [ar, "xv", os.path.abspath(archive)], capture_output=True, check=True, cwd=temp
+    )

Not sure where is this temp file.
Please can you make sure it created inside the build directory? It can be a follow-up patch, thanks.

#10 lore raw

PATCH v2 4/4 build: enable pmdinfogen for Windows

In reply to Dmitry Kozlyuk

Remove platform restriction when building drivers.

---
 drivers/meson.build | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)
drivers/meson.build +12 -14
index f49d4f79b..1dfa8738f 100644
@@ -143,20 +143,18 @@ foreach subpath:subdirs
 			# lib and then running pmdinfogen on the contents of
 			# that lib. The final lib reuses the object files and
 			# adds in the new source file.
-			if not is_windows
-				out_filename = lib_name + '.pmd.c'
-				tmp_lib = static_library('tmp_' + lib_name,
-						sources,
-						include_directories: includes,
-						dependencies: static_deps,
-						c_args: cflags)
-				objs += tmp_lib.extract_all_objects()
-				sources = custom_target(out_filename,
-						command: [pmdinfo, tmp_lib.full_path(),
-							'@OUTPUT@', pmdinfogen],
-						output: out_filename,
-						depends: [tmp_lib])
-			endif
+			out_filename = lib_name + '.pmd.c'
+			tmp_lib = static_library('tmp_' + lib_name,
+					sources,
+					include_directories: includes,
+					dependencies: static_deps,
+					c_args: cflags)
+			objs += tmp_lib.extract_all_objects()
+			sources = custom_target(out_filename,
+					command: [pmdinfo, tmp_lib.full_path(),
+						'@OUTPUT@', pmdinfogen],
+					output: out_filename,
+					depends: [tmp_lib])
 
 			# now build the static driver
 			static_lib = static_library(lib_name,
Signature · 1 line
2.29.2
#12 lore raw
In reply to Dmitry Kozlyuk

08/01/2021 03:47, Dmitry Kozlyuk:

Quoted 5 lines

Dmitry Kozlyuk (4):
pmdinfogen: support COFF
pmdinfogen: allow multiple input files
buildtools: support object file extraction for Windows
build: enable pmdinfogen for Windows

Applied, thanks.

Please follow-up on temporary file location.