Skip to contentSkip to conversation

Support compressed firmwares

PATCH Superseded David Marchand 8 participants 46 messages Jun 2, 2021 – Jul 7, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Support compressed firmwares

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Signature · 27 lines
David Marchand

David Marchand (2):
  net/ice: factorize firmware loading
  eal: handle compressed firmwares

 .github/workflows/build.yml      |   1 +
 .travis.yml                      |   1 +
 config/meson.build               |   9 ++
 drivers/net/bnx2x/bnx2x.c        |  35 +++----
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++---------------
 drivers/net/ice/ice_ethdev.c     | 175 ++++++++++++-------------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 drivers/net/nfp/nfp_net.c        |  57 +++-------
 drivers/net/qede/qede_main.c     |  44 +++-----
 lib/eal/include/rte_firmware.h   |  35 +++++++
 lib/eal/unix/eal_firmware.c      | 106 +++++++++++++++++++
 lib/eal/unix/meson.build         |   1 +
 lib/eal/version.map              |   2 +
 lib/eal/windows/eal.c            |   9 ++
 15 files changed, 283 insertions(+), 298 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

-- 
2.23.0
#1 lore raw

PATCH 1/2 net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/base/ice_osdep.h +0 -6
index 878c5597d4..78093adb00 100644
@@ -74,12 +74,6 @@ typedef uint64_t        s64;
 #define min(a, b) RTE_MIN(a, b)
 #define max(a, b) RTE_MAX(a, b)
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 #define FIELD_SIZEOF(t, f) RTE_SIZEOF_FIELD(t, f)
 #define ARRAY_SIZE(arr) RTE_DIM(arr)
 
drivers/net/ice/ice_dcf_parent.c +9 -88

125 lines not on this page · raw message

drivers/net/ice/ice_ethdev.c +77 -84

228 lines not on this page · raw message

drivers/net/ice/ice_ethdev.h +2 -1
index 2a8a8169d5..1997f9296f 100644
@@ -531,7 +531,8 @@ struct ice_vsi_vlan_pvid_info {
 #define ICE_PF_TO_ETH_DEV(pf) \
 	(((struct ice_pf *)pf)->adapter->eth_dev)
 
-enum ice_pkg_type ice_load_pkg_type(struct ice_hw *hw);
+int
+ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn);
 struct ice_vsi *
 ice_setup_vsi(struct ice_pf *pf, enum ice_vsi_type type);
 int
Signature · 1 line
2.23.0
#2 lore raw

PATCH 2/2 eal: handle compressed firmwares

In reply to David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |   9 +++
 drivers/net/bnx2x/bnx2x.c      |  35 ++++-------
 drivers/net/ice/ice_ethdev.c   |  60 ++++---------------
 drivers/net/nfp/nfp_net.c      |  57 ++++--------------
 drivers/net/qede/qede_main.c   |  44 ++++++--------
 lib/eal/include/rte_firmware.h |  35 +++++++++++
 lib/eal/unix/eal_firmware.c    | 106 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   2 +
 lib/eal/windows/eal.c          |   9 +++
 12 files changed, 218 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +9 -0
index 017bb2efbb..337daa2719 100644
@@ -166,6 +166,15 @@ if fdt_dep.found() and cc.has_header('fdt.h')
     dpdk_extra_ldflags += '-lfdt'
 endif
 
+has_libarchive = 0
+archive_dep = cc.find_library('libarchive', required: false)
+if archive_dep.found() and cc.has_header('archive.h')
+    dpdk_conf.set10('RTE_HAS_LIBARCHIVE', true)
+    has_libarchive = 1
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 libexecinfo = cc.find_library('libexecinfo', required: false)
 if libexecinfo.found() and cc.has_header('execinfo.h')
     add_project_link_arguments('-lexecinfo', language: 'c')
drivers/net/bnx2x/bnx2x.c +13 -22
index 654878d9de..60292753e2 100644
@@ -26,7 +26,9 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
@@ -9655,44 +9657,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +12 -48

110 lines not on this page · raw message

drivers/net/nfp/nfp_net.c +12 -45

96 lines not on this page · raw message

drivers/net/qede/qede_main.c +17 -27

74 lines not on this page · raw message

lib/eal/include/rte_firmware.h new file +35 -0
new file mode 100644
index 0000000000..d2a7bc8083
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param name
+ *      Firmware filename to load.
+ * @param buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for freeing the buffer.
+ * @param bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +106 -0

106 lines not on this page · raw message

lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +2 -0
index fe5c3dac98..020c058e5f 100644
@@ -423,11 +423,13 @@ EXPERIMENTAL {
 	rte_version_release; # WINDOWS_NO_EXPORT
 	rte_version_suffix; # WINDOWS_NO_EXPORT
 	rte_version_year; # WINDOWS_NO_EXPORT
+
 };
 
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 28c787c0b0..938ace92f5 100644
@@ -21,6 +21,7 @@
 #include <eal_private.h>
 #include <rte_service_component.h>
 #include <rte_vfio.h>
+#include <rte_firmware.h>
 
 #include "eal_hugepages.h"
 #include "eal_trace.h"
@@ -463,3 +464,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
Signature · 1 line
2.23.0
#3 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

Quoted 24 lines · David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

+ */

lib/eal/include/rte_firmware.h new file +8 -0
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+

lib/eal/include/rte_firmware.h line 21

Quoted 10 lines
lib/eal/include/rte_firmware.h new file +10 -0
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param name
+ *      Firmware filename to load.
+ * @param buf

Adding out to express the output useful. i.e
@param[out] buf

Quoted 2 lines

+ * Buffer allocated by this function. If this function succeeds, the
+ * caller is responsible for freeing the buffer.

I think, we can chnange to "freeing the buffer using free()" to avoid confusion with rte_free()

Quoted 1 line

+ * @param bufsz

@param[out] bufsz

Quoted 30 lines

+ * Size of the data in the buffer.
+ *
+ * @return
+ * 0 if successful.
+ * Negative otherwise, buf and bufsize contents are invalid.
+ */

lib/eal/include/rte_firmware.h new file +5 -0
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +13 -0
new file mode 100644
index 0000000000..ea66fecfe9
@@ -0,0 +1,106 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifdef RTE_HAS_LIBARCHIVE
+#include <archive.h>
+#endif
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <rte_common.h>

lib/eal/unix/eal_firmware.c line 23

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+#include <rte_firmware.h>
+#include <rte_log.h>
+
+static int
+firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+       const size_t blocksize = 4096;
+       int ret = -1;
+       int err;
+#ifdef RTE_HAS_LIBARCHIVE

I think, better to have small inline functions for libarchive variant vs normal file accessors
in the group for open, access, read etc to avoid the ifdef clutter and manage with one ifdef.

lib/eal/unix/eal_firmware.c line 33

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+       struct archive_entry *entry;
+       struct archive *a;
+#else
+       int fd;
+#endif
+
+       *buf = NULL;
+       *bufsz = 0;
+
+#ifdef RTE_HAS_LIBARCHIVE

See above

Quoted 15 lines
lib/eal/unix/eal_firmware.c new file +15 -0
+       a = archive_read_new();
+       if (a == NULL || archive_read_support_format_raw(a) != ARCHIVE_OK ||
+                       archive_read_support_filter_xz(a) != ARCHIVE_OK ||
+                       archive_read_open_filename(a, name, blocksize) != ARCHIVE_OK ||
+                       archive_read_next_header(a, &entry) != ARCHIVE_OK)
+               goto out;
+#else
+       fd = open(name, O_RDONLY);
+       if (fd < 0)
+               goto out;
+#endif
+
+       do {
+               void *tmp;
+

lib/eal/unix/eal_firmware.c line 58

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+               tmp = realloc(*buf, *bufsz + blocksize);
+               if (tmp == NULL) {
+                       free(*buf);
+                       *buf = NULL;
+                       *bufsz = 0;
+                       break;
+               }
+               *buf = tmp;
+
+#ifdef RTE_HAS_LIBARCHIVE

See above

Quoted 8 lines
lib/eal/unix/eal_firmware.c new file +8 -0
+               err = archive_read_data(a, RTE_PTR_ADD(*buf, *bufsz), blocksize);
+#else
+               err = read(fd, RTE_PTR_ADD(*buf, *bufsz), blocksize);
+#endif
+               if (err < 0) {
+                       free(*buf);
+                       *buf = NULL;
+                       *bufsz = 0;

lib/eal/unix/eal_firmware.c line 76

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+                       break;
+               }
+               *bufsz += err;
+
+       } while (err != 0);
+
+       if (*buf != NULL)
+               ret = 0;
+out:
+#ifdef RTE_HAS_LIBARCHIVE

See above

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+       if (a != NULL)
+               archive_read_free(a);
+#else
+       if (fd >= 0)
+               close(fd);
+#endif
+       return ret;
+}
+
+int

lib/eal/unix/eal_firmware.c line 96

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+rte_firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+       char path[PATH_MAX];
+       int ret;
+
+       ret = firmware_read(name, buf, bufsz);
+       if (ret < 0) {
+               snprintf(path, sizeof(path), "%s.xz", name);
+               path[PATH_MAX - 1] = '\0';
+#ifndef RTE_HAS_LIBARCHIVE

See above

Quoted 8 lines
lib/eal/unix/eal_firmware.c new file +7 -0
+               if (access(path, F_OK) == 0) {
+                       RTE_LOG(WARNING, EAL, "libarchive not available, %s cannot be decompressed\n",
+                               path);
+               }
+#else
+               ret = firmware_read(path, buf, bufsz);
+#endif
#4 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

Quoted 5 lines · Jerin Jacob
Quoted 5 lines
lib/eal/unix/eal_firmware.c new file +5 -0
+static int
+firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+       const size_t blocksize = 4096;
+       int ret = -1;
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+       int err;
+#ifdef RTE_HAS_LIBARCHIVE

I think, better to have small inline functions for libarchive variant vs normal file accessors
in the group for open, access, read etc to avoid the ifdef clutter and manage with one ifdef.

That may be a bit artificial, since there is no reuse of such helpers for now. I'll have a try and see how it looks.

Quoted 5 lines
Quoted 5 lines
lib/eal/unix/eal_firmware.c new file +5 -0
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+       char path[PATH_MAX];
+       int ret;
Quoted 8 lines

lib/eal/unix/eal_firmware.c line 96

Quoted 6 lines
lib/eal/unix/eal_firmware.c new file +6 -0
+
+       ret = firmware_read(name, buf, bufsz);
+       if (ret < 0) {
+               snprintf(path, sizeof(path), "%s.xz", name);
+               path[PATH_MAX - 1] = '\0';
+#ifndef RTE_HAS_LIBARCHIVE

See above

There is nothing to abstract here.

If you don't have libarchive, returning the .xz content to a driver is wrong. I prefer to leave this block as is.

Quoted 10 lines
Quoted 9 lines
lib/eal/unix/eal_firmware.c new file +7 -0
+               if (access(path, F_OK) == 0) {
+                       RTE_LOG(WARNING, EAL, "libarchive not available, %s cannot be decompressed\n",
+                               path);
+               }
+#else
+               ret = firmware_read(path, buf, bufsz);
+#endif
Signature · 1 line
David Marchand
Reviewed-by on v1 2/2
#5 lore raw

Re: [dpdk-dev] [EXT] [PATCH 2/2] eal: handle compressed firmwares

In reply to David Marchand
Quoted 7 lines

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

Quoted 7 lines

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

for QEDE,

Devendra, please give it a try when possible.

Regards,
Igor

#6 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

In reply to David Marchand

2021-06-02 11:58 (UTC+0200), David Marchand:

Quoted 11 lines

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

JFYI, it doesn't seem hard to implement as a follow-up if need be. libarchive is available for Windows and even provides a .pc file.

[...]

Quoted 7 lines
config/meson.build +0 -0
index 017bb2efbb..337daa2719 100644
@@ -166,6 +166,15 @@ if fdt_dep.found() and cc.has_header('fdt.h')
     dpdk_extra_ldflags += '-lfdt'
 endif

config/meson.build if fdt_dep.found() and cc.has_header('fdt.h') · line 177

Quoted 10 lines
config/meson.build +9 -0
 
+has_libarchive = 0
+archive_dep = cc.find_library('libarchive', required: false)
+if archive_dep.found() and cc.has_header('archive.h')
+    dpdk_conf.set10('RTE_HAS_LIBARCHIVE', true)
+    has_libarchive = 1
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+

Why not use pkg-config?
has_libarchive is unused.

Quoted 3 lines
config/meson.build +0 -0
 libexecinfo = cc.find_library('libexecinfo', required: false)
 if libexecinfo.found() and cc.has_header('execinfo.h')
     add_project_link_arguments('-lexecinfo', language: 'c')

[...]

lib/eal/version.map EXPERIMENTAL { · line 427

Quoted 10 lines
lib/eal/version.map +1 -0
index fe5c3dac98..020c058e5f 100644
@@ -423,11 +423,13 @@ EXPERIMENTAL {
 	rte_version_release; # WINDOWS_NO_EXPORT
 	rte_version_suffix; # WINDOWS_NO_EXPORT
 	rte_version_year; # WINDOWS_NO_EXPORT
+
 };

Unnecessary empty line.

Quoted 8 lines
lib/eal/version.map +1 -0
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
#7 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

Quoted 8 lines · Dmitry Kozlyuk

2021-06-02 11:58 (UTC+0200), David Marchand:

Quoted 6 lines

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

Quoted 8 lines
Quoted 5 lines

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

JFYI, it doesn't seem hard to implement as a follow-up if need be. libarchive is available for Windows and even provides a .pc file.

I noticed.
But I am not sure the current API is abstracted enough since we pass a firmware filename.

Quoted 10 lines

[...]

Quoted 8 lines
config/meson.build +0 -0
index 017bb2efbb..337daa2719 100644
@@ -166,6 +166,15 @@ if fdt_dep.found() and cc.has_header('fdt.h')
     dpdk_extra_ldflags += '-lfdt'
 endif
 
Quoted 12 lines

config/meson.build if fdt_dep.found() and cc.has_header('fdt.h') · line 177

Quoted 9 lines
config/meson.build +9 -0
+has_libarchive = 0
+archive_dep = cc.find_library('libarchive', required: false)
+if archive_dep.found() and cc.has_header('archive.h')
+    dpdk_conf.set10('RTE_HAS_LIBARCHIVE', true)
+    has_libarchive = 1
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+

Why not use pkg-config?
has_libarchive is unused.

Frankly, I just copied the logic already present in config/meson.build. If there is better to do, please advise.

Quoted 5 lines
Quoted 4 lines
config/meson.build +0 -0
 libexecinfo = cc.find_library('libexecinfo', required: false)
 if libexecinfo.found() and cc.has_header('execinfo.h')
     add_project_link_arguments('-lexecinfo', language: 'c')

[...]

Quoted 12 lines

lib/eal/version.map EXPERIMENTAL { · line 427

Quoted 10 lines
lib/eal/version.map +1 -0
index fe5c3dac98..020c058e5f 100644
@@ -423,11 +423,13 @@ EXPERIMENTAL {
      rte_version_release; # WINDOWS_NO_EXPORT
      rte_version_suffix; # WINDOWS_NO_EXPORT
      rte_version_year; # WINDOWS_NO_EXPORT
+
 };

Unnecessary empty line.

Yep, already fixed in my local branch.
Thanks.

Signature · 1 line
David Marchand
#8 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

Quoted 21 lines · David Marchand
Quoted 8 lines
Quoted 8 lines
config/meson.build +0 -0
index 017bb2efbb..337daa2719 100644
@@ -166,6 +166,15 @@ if fdt_dep.found() and cc.has_header('fdt.h')
     dpdk_extra_ldflags += '-lfdt'
 endif
 
Quoted 12 lines

config/meson.build if fdt_dep.found() and cc.has_header('fdt.h') · line 177

Quoted 9 lines
config/meson.build +9 -0
+has_libarchive = 0
+archive_dep = cc.find_library('libarchive', required: false)
+if archive_dep.found() and cc.has_header('archive.h')
+    dpdk_conf.set10('RTE_HAS_LIBARCHIVE', true)
+    has_libarchive = 1
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+

Why not use pkg-config?
has_libarchive is unused.

Quoted 2 lines

Frankly, I just copied the logic already present in config/meson.build. If there is better to do, please advise.

Ah ok, I think I understand.
Do you mean to align on libbsd?

Signature · 1 line
David Marchand
#9 lore raw

Re: [dpdk-dev] [PATCH 2/2] eal: handle compressed firmwares

Quoted 16 lines · David Marchand
Quoted 14 lines · David Marchand
Quoted 13 lines
Quoted 11 lines
config/meson.build index renamed from config/meson.build +0 -0

017bb2efbb..337daa2719 100644 --- a/config/meson.build +++ b/config/meson.build @@ -166,6 +166,15 @@ if fdt_dep.found() and cc.has_header('fdt.h') dpdk_extra_ldflags += '-lfdt' endif

+has_libarchive = 0 +archive_dep = cc.find_library('libarchive',
required: false) +if archive_dep.found() and cc.has_header('archive.h') +
dpdk_conf.set10('RTE_HAS_LIBARCHIVE', true) + has_libarchive = 1
+ add_project_link_arguments('-larchive', language: 'c') +
dpdk_extra_ldflags += '-larchive' +endif +

Why not use pkg-config? has_libarchive is unused.

Quoted 5 lines
Quoted 2 lines

Frankly, I just copied the logic already present in config/meson.build. If there is better to do, please advise.

Ah ok, I think I understand. Do you mean to align on libbsd?

Yep. "dependency()" should be used for libs/packages that have a pkg-config file, while "find_library()" is a fallback for literally just trying to find a library that doesn't have a .pc file.

When specifying dependencies, it's best to explicitly state the lookup method as pkg-config (only), because if not found via pkg-config, meson will by default then attempt to use cmake (if present) to find the requested package, and that can cause issues with cmake finding the incorrect package version e.g. when doing 32-bit builds on 64-bit systems. There is almost certainly a way to configure cmake not to do this, but by limiting search to pkg-config it saves us having to document how to configure proper library search paths for multiple tools. Setting PKG_CONFIG_LIBDIR should be enough!

/Bruce

#10 lore raw
In reply to David Marchand
Quoted 12 lines

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

Quoted 5 lines

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Hi David,

We (Marvell QED) already provide packed version of FW in linux-firmware:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/qed

But thats a custom name, and zlib.

I'm just wondering if its a good solution to try transparently load .xz variant? User may not expect that. M.b. through this api, give a driver an option to specify archive format? Or even autodetect it from content?

Regards,
Igor

#11 lore raw
Quoted 4 lines · Igor Russkikh

We (Marvell QED) already provide packed version of FW in linux-firmware:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/qed

But thats a custom name, and zlib.

Whatever binary blob is available in linux-firmware upstream repo, it ends up compressed on Fedora 34 (and later).

# fc33
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115649&filename=/usr/lib/firmware/qed/qed_init_values-8.10.9.0.bin
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115649&filename=/usr/lib/firmware/qed/qed_init_values_zipped-8.10.10.0.bin
# fc34
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115322&filename=/usr/lib/firmware/qed/qed_init_values-8.10.9.0.bin.xz
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115322&filename=/usr/lib/firmware/qed/qed_init_values_zipped-8.10.10.0.bin.xz
# fc35
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115234&filename=/usr/lib/firmware/qed/qed_init_values-8.10.9.0.bin.xz
https://koji.fedoraproject.org/koji/fileinfo?rpmID=26115234&filename=/usr/lib/firmware/qed/qed_init_values_zipped-8.10.10.0.bin.xz

Did you try the qede pmd on fc34?

Quoted 2 lines

I'm just wondering if its a good solution to try transparently load .xz variant?

The linux kernel (since v5.2, I think) uncompresses this transparently without kernel drivers knowing.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=82fd7a8142a10b8eb41313074b3859d82c0857dc

Quoted 2 lines

User may not expect that. M.b. through this api, give a driver an option to specify archive format? Or even autodetect it from content?

User should (do ?) not care about firmwares, only drivers do.

If not doing transparently, each driver will have to implement (or ask a common helper for) support of compressed blobs.
While this issue is common to all drivers.

As for autodetecting compression, libarchive can do this, I added only xz support because this is the only supported compression in the Linux kernel loader.

Signature · 1 line
David Marchand
#12 lore raw
Quoted 10 lines · David Marchand
Quoted 4 lines · Igor Russkikh

We (Marvell QED) already provide packed version of FW in linux-firmware:

But thats a custom name, and zlib.

Whatever binary blob is available in linux-firmware upstream repo, it ends up compressed on Fedora 34 (and later).

Did you try the qede pmd on fc34?

Got it, thanks.
No, have not tried, but think qede pmd will fail now on fedora without your patch.

Quoted 2 lines

The linux kernel (since v5.2, I think) uncompresses this transparently without kernel drivers knowing.

Was not aware of that, thanks.
In that perspective your patch fits good with the existing linux kernel behavior, agree.

Adding more people from my team.

Regards,
Igor

#13 lore raw

v2 cover Support compressed firmwares

In reply to David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Signature · 31 lines
David Marchand

Changes since v1:
- address comments on patch2,


David Marchand (2):
  net/ice: factorize firmware loading
  eal: handle compressed firmwares

 .github/workflows/build.yml      |   1 +
 .travis.yml                      |   1 +
 config/meson.build               |   7 ++
 drivers/net/bnx2x/bnx2x.c        |  35 +++----
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++---------------
 drivers/net/ice/ice_ethdev.c     | 175 ++++++++++++-------------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 drivers/net/nfp/nfp_net.c        |  57 +++-------
 drivers/net/qede/qede_main.c     |  45 ++++----
 lib/eal/include/rte_firmware.h   |  32 ++++++
 lib/eal/unix/eal_firmware.c      | 149 ++++++++++++++++++++++++++
 lib/eal/unix/meson.build         |   1 +
 lib/eal/version.map              |   1 +
 lib/eal/windows/eal.c            |   9 ++
 15 files changed, 321 insertions(+), 298 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

-- 
2.23.0
#14 lore raw

PATCH v2 1/2 net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/base/ice_osdep.h +0 -6
index 878c5597d4..78093adb00 100644
@@ -74,12 +74,6 @@ typedef uint64_t        s64;
 #define min(a, b) RTE_MIN(a, b)
 #define max(a, b) RTE_MAX(a, b)
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 #define FIELD_SIZEOF(t, f) RTE_SIZEOF_FIELD(t, f)
 #define ARRAY_SIZE(arr) RTE_DIM(arr)
 
drivers/net/ice/ice_dcf_parent.c +9 -88

125 lines not on this page · raw message

drivers/net/ice/ice_ethdev.c +77 -84

228 lines not on this page · raw message

drivers/net/ice/ice_ethdev.h +2 -1
index 2a8a8169d5..1997f9296f 100644
@@ -531,7 +531,8 @@ struct ice_vsi_vlan_pvid_info {
 #define ICE_PF_TO_ETH_DEV(pf) \
 	(((struct ice_pf *)pf)->adapter->eth_dev)
 
-enum ice_pkg_type ice_load_pkg_type(struct ice_hw *hw);
+int
+ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn);
 struct ice_vsi *
 ice_setup_vsi(struct ice_pf *pf, enum ice_vsi_type type);
 int
Signature · 1 line
2.23.0
#15 lore raw

Re: [dpdk-dev] [PATCH v2 1/2] net/ice: factorize firmware loading

Quoted 9 lines · David Marchand

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

Quoted 8 lines

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

Please, can you confirm nothing breaks for net/ice with this series? Thanks.

Signature · 1 line
David Marchand
#16 lore raw

PATCH v2 2/2 eal: handle compressed firmwares

In reply to David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |   7 ++
 drivers/net/bnx2x/bnx2x.c      |  35 +++-----
 drivers/net/ice/ice_ethdev.c   |  60 +++----------
 drivers/net/nfp/nfp_net.c      |  57 +++----------
 drivers/net/qede/qede_main.c   |  45 ++++------
 lib/eal/include/rte_firmware.h |  32 +++++++
 lib/eal/unix/eal_firmware.c    | 149 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   1 +
 lib/eal/windows/eal.c          |   9 ++
 12 files changed, 256 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +7 -0
index 017bb2efbb..c6985139b4 100644
@@ -172,6 +172,13 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 # check for libbsd
 libbsd = dependency('libbsd', required: false, method: 'pkg-config')
 if libbsd.found()
drivers/net/bnx2x/bnx2x.c +13 -22
index 654878d9de..60292753e2 100644
@@ -26,7 +26,9 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
@@ -9655,44 +9657,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +12 -48

110 lines not on this page · raw message

drivers/net/nfp/nfp_net.c +12 -45

96 lines not on this page · raw message

drivers/net/qede/qede_main.c +18 -27

76 lines not on this page · raw message

lib/eal/include/rte_firmware.h new file +32 -0
new file mode 100644
index 0000000000..3389e60ca0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param[in] name
+ *      Firmware filename to load.
+ * @param[out] buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for calling free() on this buffer.
+ * @param[out] bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +149 -0

149 lines not on this page · raw message

lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +1 -0
index fe5c3dac98..2df65c6903 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 28c787c0b0..938ace92f5 100644
@@ -21,6 +21,7 @@
 #include <eal_private.h>
 #include <rte_service_component.h>
 #include <rte_vfio.h>
+#include <rte_firmware.h>
 
 #include "eal_hugepages.h"
 #include "eal_trace.h"
@@ -463,3 +464,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
Signature · 1 line
2.23.0
#17 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] eal: handle compressed firmwares

2021-06-03 18:55 (UTC+0200), David Marchand:
[...]

Quoted 5 lines
config/meson.build +0 -0
index 017bb2efbb..c6985139b4 100644

config/meson.build if libexecinfo.found() and cc.has_header('execinfo.h') · line 181

Quoted 10 lines
config/meson.build +7 -0
@@ -172,6 +172,13 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+

Suggestion:

config/meson.build +0 -1
index c6985139b4..c3668798c1 100644
@@ -175,7 +175,6 @@ endif
 libarchive = dependency('libarchive', required: false, method: 'pkg-config')
 if libarchive.found()
     dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
-    add_project_link_arguments('-larchive', language: 'c')
     dpdk_extra_ldflags += '-larchive'
 endif
 
lib/eal/meson.build +1 -0
index 1722924f67..5a018d97d6 100644
@@ -16,6 +16,7 @@ subdir(exec_env)
 subdir(arch_subdir)
 
 deps += ['kvargs']
+ext_deps += libarchive
 if not is_windows
     deps += ['telemetry']
 endif
#18 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] eal: handle compressed firmwares

Quoted 39 lines · Dmitry Kozlyuk

2021-06-03 18:55 (UTC+0200), David Marchand:
[...]

Quoted 5 lines
config/meson.build +0 -0
index 017bb2efbb..c6985139b4 100644

config/meson.build if libexecinfo.found() and cc.has_header('execinfo.h') · line 181

Quoted 10 lines
config/meson.build +7 -0
@@ -172,6 +172,13 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+

Suggestion:

config/meson.build +0 -1
index c6985139b4..c3668798c1 100644
@@ -175,7 +175,6 @@ endif
 libarchive = dependency('libarchive', required: false, method: 'pkg-config')
 if libarchive.found()
     dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
-    add_project_link_arguments('-larchive', language: 'c')
     dpdk_extra_ldflags += '-larchive'
 endif
 
lib/eal/meson.build +0 -0
index 1722924f67..5a018d97d6 100644

lib/eal/meson.build subdir(exec_env) · line 22

Quoted 7 lines
lib/eal/meson.build +1 -0
@@ -16,6 +16,7 @@ subdir(exec_env)
 subdir(arch_subdir)
 
 deps += ['kvargs']
+ext_deps += libarchive
 if not is_windows
     deps += ['telemetry']
 endif

I had tried something close when preparing v2 (only keeping RTE_HAS_LIBARCHIVE in config/meson.build and putting extra_ldflags and ext_deps in lib/eal/unix/meson.build) but both my try and your suggestion break static compilation for the helloworld example.

$ ./devtools/test-meson-builds.sh -vv
...
## Building helloworld
gmake: Entering directory
'/home/dmarchan/builds/build-x86-generic/install/usr/local/share/dpdk/examples/helloworld'
rm -f build/helloworld build/helloworld-static build/helloworld-shared
test -d build && rmdir -p build || true
cc  -I/home/dmarchan/intel-ipsec-mb/install/include -O3
-I/home/dmarchan/builds/build-x86-generic/install/usr/local/include
-include rte_config.h -march=nehalem -I/usr/usr/include
-I/opt/isa-l/include  -DALLOW_EXPERIMENTAL_API main.c -o
build/helloworld-shared  -L/home/dmarchan/intel-ipsec-mb/install/lib
-Wl,--as-needed
-L/home/dmarchan/builds/build-x86-generic/install/usr/local/lib
-lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline
-lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack
-lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev
-lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm
-lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso
-lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev
-lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev
-lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci
-lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu
-lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lbsd
ln -sf helloworld-shared build/helloworld
cc  -I/home/dmarchan/intel-ipsec-mb/install/include -O3
-I/home/dmarchan/builds/build-x86-generic/install/usr/local/include
-include rte_config.h -march=nehalem -I/usr/usr/include
-I/opt/isa-l/include  -DALLOW_EXPERIMENTAL_API main.c -o
build/helloworld-static  -L/home/dmarchan/intel-ipsec-mb/install/lib
-Wl,--whole-archive
-L/home/dmarchan/builds/build-x86-generic/install/usr/local/lib
-l:librte_common_cpt.a -l:librte_common_dpaax.a
-l:librte_common_iavf.a -l:librte_common_octeontx.a
-l:librte_common_octeontx2.a -l:librte_bus_dpaa.a
-l:librte_bus_fslmc.a -l:librte_bus_ifpga.a -l:librte_bus_pci.a
-l:librte_bus_vdev.a -l:librte_bus_vmbus.a -l:librte_common_cnxk.a
-l:librte_common_mlx5.a -l:librte_common_qat.a
-l:librte_common_sfc_efx.a -l:librte_mempool_bucket.a
-l:librte_mempool_cnxk.a -l:librte_mempool_dpaa.a
-l:librte_mempool_dpaa2.a -l:librte_mempool_octeontx.a
-l:librte_mempool_octeontx2.a -l:librte_mempool_ring.a
-l:librte_mempool_stack.a -l:librte_net_af_packet.a
-l:librte_net_af_xdp.a -l:librte_net_ark.a -l:librte_net_atlantic.a
-l:librte_net_avp.a -l:librte_net_axgbe.a -l:librte_net_bnx2x.a
-l:librte_net_bnxt.a -l:librte_net_bond.a -l:librte_net_cxgbe.a
-l:librte_net_dpaa.a -l:librte_net_dpaa2.a -l:librte_net_e1000.a
-l:librte_net_ena.a -l:librte_net_enetc.a -l:librte_net_enic.a
-l:librte_net_failsafe.a -l:librte_net_fm10k.a -l:librte_net_hinic.a
-l:librte_net_hns3.a -l:librte_net_i40e.a -l:librte_net_iavf.a
-l:librte_net_ice.a -l:librte_net_igc.a -l:librte_net_ionic.a
-l:librte_net_ipn3ke.a -l:librte_net_ixgbe.a -l:librte_net_kni.a
-l:librte_net_liquidio.a -l:librte_net_memif.a -l:librte_net_mlx4.a
-l:librte_net_mlx5.a -l:librte_net_netvsc.a -l:librte_net_nfp.a
-l:librte_net_null.a -l:librte_net_octeontx.a
-l:librte_net_octeontx2.a -l:librte_net_octeontx_ep.a
-l:librte_net_pcap.a -l:librte_net_pfe.a -l:librte_net_qede.a
-l:librte_net_ring.a -l:librte_net_sfc.a -l:librte_net_softnic.a
-l:librte_net_szedata2.a -l:librte_net_tap.a -l:librte_net_thunderx.a
-l:librte_net_txgbe.a -l:librte_net_vdev_netvsc.a
-l:librte_net_vhost.a -l:librte_net_virtio.a -l:librte_net_vmxnet3.a
-l:librte_raw_dpaa2_cmdif.a -l:librte_raw_dpaa2_qdma.a
-l:librte_raw_ifpga.a -l:librte_raw_ioat.a -l:librte_raw_ntb.a
-l:librte_raw_octeontx2_dma.a -l:librte_raw_octeontx2_ep.a
-l:librte_raw_skeleton.a -l:librte_crypto_aesni_gcm.a
-l:librte_crypto_aesni_mb.a -l:librte_crypto_bcmfs.a
-l:librte_crypto_caam_jr.a -l:librte_crypto_ccp.a
-l:librte_crypto_dpaa_sec.a -l:librte_crypto_dpaa2_sec.a
-l:librte_crypto_kasumi.a -l:librte_crypto_nitrox.a
-l:librte_crypto_null.a -l:librte_crypto_octeontx.a
-l:librte_crypto_octeontx2.a -l:librte_crypto_openssl.a
-l:librte_crypto_scheduler.a -l:librte_crypto_snow3g.a
-l:librte_crypto_virtio.a -l:librte_crypto_zuc.a
-l:librte_compress_isal.a -l:librte_compress_mlx5.a
-l:librte_compress_octeontx.a -l:librte_compress_zlib.a
-l:librte_regex_mlx5.a -l:librte_regex_octeontx2.a
-l:librte_vdpa_ifc.a -l:librte_vdpa_mlx5.a -l:librte_event_cnxk.a
-l:librte_event_dlb2.a -l:librte_event_dpaa.a -l:librte_event_dpaa2.a
-l:librte_event_dsw.a -l:librte_event_octeontx2.a
-l:librte_event_opdl.a -l:librte_event_skeleton.a -l:librte_event_sw.a
-l:librte_event_octeontx.a -l:librte_baseband_acc100.a
-l:librte_baseband_fpga_5gnr_fec.a -l:librte_baseband_fpga_lte_fec.a
-l:librte_baseband_null.a -l:librte_baseband_turbo_sw.a
-l:librte_node.a -l:librte_graph.a -l:librte_bpf.a
-l:librte_flow_classify.a -l:librte_pipeline.a -l:librte_table.a
-l:librte_port.a -l:librte_fib.a -l:librte_ipsec.a -l:librte_vhost.a
-l:librte_stack.a -l:librte_security.a -l:librte_sched.a
-l:librte_reorder.a -l:librte_rib.a -l:librte_regexdev.a
-l:librte_rawdev.a -l:librte_pdump.a -l:librte_power.a
-l:librte_member.a -l:librte_lpm.a -l:librte_latencystats.a
-l:librte_kni.a -l:librte_jobstats.a -l:librte_ip_frag.a
-l:librte_gso.a -l:librte_gro.a -l:librte_eventdev.a -l:librte_efd.a
-l:librte_distributor.a -l:librte_cryptodev.a -l:librte_compressdev.a
-l:librte_cfgfile.a -l:librte_bitratestats.a -l:librte_bbdev.a
-l:librte_acl.a -l:librte_timer.a -l:librte_hash.a -l:librte_metrics.a
-l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a
-l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a
-l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a
-l:librte_kvargs.a -Wl,--no-whole-archive -Wl,--export-dynamic
-lIPSec_MB /usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/librt.so
-Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify
-lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec
-lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder
-lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power
-lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats
-lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd
-lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile
-lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash
-lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter
-lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal
-lrte_telemetry -lrte_kvargs -pthread -lm -ldl -lnuma -lfdt -larchive
-lbsd -larchive -lacl -llzma -lzstd -llz4 -lbz2 -lz -lxml2
-L/usr/usr/lib64 -lmlx5 -lpthread -L/usr/usr/lib64 -lpthread -libverbs
-lpthread -lcrypto -lz -ldl -pthread -lbpf -lz -lmlx4 -lpthread
-L/usr/usr/lib64 -libverbs -lpthread -lpcap -Wl,-R/usr/lib64 -lsze2
-L/opt/isa-l/lib -lisal -lelf -lz -ljansson
/usr/bin/ld: cannot find -lacl
/usr/bin/ld: cannot find -llz4
/usr/bin/ld: cannot find -lbz2
collect2: error: ld returned 1 exit status
gmake: *** [Makefile:43: build/helloworld-static] Error 1
gmake: Leaving directory
'/home/dmarchan/builds/build-x86-generic/install/usr/local/share/dpdk/examples/helloworld'

If it helps, here is what pkg-config reports:

$ PKG_CONFIG_PATH=/opt/isa-l/lib/pkgconfig:$HOME/builds/build-x86-generic/install/usr/local/lib/pkgconfig
pkg-config --static --libs libarchive
-larchive -lcrypto -lacl -llzma -lzstd -llz4 -lbz2 -lz -lxml2
$ ls /lib*/lib{acl,lz4,bz2}*
/lib64/libacl.so.1  /lib64/libacl.so.1.1.2253  /lib64/libbz2.so.1
/lib64/libbz2.so.1.0.8  /lib64/liblz4.so.1  /lib64/liblz4.so.1.9.1
$ PKG_CONFIG_PATH=/opt/isa-l/lib/pkgconfig:$HOME/builds/build-x86-generic/install/usr/local/lib/pkgconfig
pkg-config --libs libdpdk
-Wl,--as-needed -L/usr/local/lib -lrte_node -lrte_graph -lrte_bpf
-lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_fib
-lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched
-lrte_reorder -lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump
-lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni
-lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev
-lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev
-lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer
-lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev
-lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring
-lrte_eal -lrte_telemetry -lrte_kvargs -lbsd
[dmarchan@wsfd-netdev66 dpdk]$
PKG_CONFIG_PATH=/opt/isa-l/lib/pkgconfig:$HOME/builds/build-x86-generic/install/usr/local/lib/pkgconfig
pkg-config --static --libs libdpdk
-Wl,--whole-archive -L/usr/local/lib -l:librte_common_cpt.a
-l:librte_common_dpaax.a -l:librte_common_iavf.a
-l:librte_common_octeontx.a -l:librte_common_octeontx2.a
-l:librte_bus_dpaa.a -l:librte_bus_fslmc.a -l:librte_bus_ifpga.a
-l:librte_bus_pci.a -l:librte_bus_vdev.a -l:librte_bus_vmbus.a
-l:librte_common_cnxk.a -l:librte_common_mlx5.a -l:librte_common_qat.a
-l:librte_common_sfc_efx.a -l:librte_mempool_bucket.a
-l:librte_mempool_cnxk.a -l:librte_mempool_dpaa.a
-l:librte_mempool_dpaa2.a -l:librte_mempool_octeontx.a
-l:librte_mempool_octeontx2.a -l:librte_mempool_ring.a
-l:librte_mempool_stack.a -l:librte_net_af_packet.a
-l:librte_net_af_xdp.a -l:librte_net_ark.a -l:librte_net_atlantic.a
-l:librte_net_avp.a -l:librte_net_axgbe.a -l:librte_net_bnx2x.a
-l:librte_net_bnxt.a -l:librte_net_bond.a -l:librte_net_cxgbe.a
-l:librte_net_dpaa.a -l:librte_net_dpaa2.a -l:librte_net_e1000.a
-l:librte_net_ena.a -l:librte_net_enetc.a -l:librte_net_enic.a
-l:librte_net_failsafe.a -l:librte_net_fm10k.a -l:librte_net_hinic.a
-l:librte_net_hns3.a -l:librte_net_i40e.a -l:librte_net_iavf.a
-l:librte_net_ice.a -l:librte_net_igc.a -l:librte_net_ionic.a
-l:librte_net_ipn3ke.a -l:librte_net_ixgbe.a -l:librte_net_kni.a
-l:librte_net_liquidio.a -l:librte_net_memif.a -l:librte_net_mlx4.a
-l:librte_net_mlx5.a -l:librte_net_netvsc.a -l:librte_net_nfp.a
-l:librte_net_null.a -l:librte_net_octeontx.a
-l:librte_net_octeontx2.a -l:librte_net_octeontx_ep.a
-l:librte_net_pcap.a -l:librte_net_pfe.a -l:librte_net_qede.a
-l:librte_net_ring.a -l:librte_net_sfc.a -l:librte_net_softnic.a
-l:librte_net_szedata2.a -l:librte_net_tap.a -l:librte_net_thunderx.a
-l:librte_net_txgbe.a -l:librte_net_vdev_netvsc.a
-l:librte_net_vhost.a -l:librte_net_virtio.a -l:librte_net_vmxnet3.a
-l:librte_raw_dpaa2_cmdif.a -l:librte_raw_dpaa2_qdma.a
-l:librte_raw_ifpga.a -l:librte_raw_ioat.a -l:librte_raw_ntb.a
-l:librte_raw_octeontx2_dma.a -l:librte_raw_octeontx2_ep.a
-l:librte_raw_skeleton.a -l:librte_crypto_aesni_gcm.a
-l:librte_crypto_aesni_mb.a -l:librte_crypto_bcmfs.a
-l:librte_crypto_caam_jr.a -l:librte_crypto_ccp.a
-l:librte_crypto_dpaa_sec.a -l:librte_crypto_dpaa2_sec.a
-l:librte_crypto_kasumi.a -l:librte_crypto_nitrox.a
-l:librte_crypto_null.a -l:librte_crypto_octeontx.a
-l:librte_crypto_octeontx2.a -l:librte_crypto_openssl.a
-l:librte_crypto_scheduler.a -l:librte_crypto_snow3g.a
-l:librte_crypto_virtio.a -l:librte_crypto_zuc.a
-l:librte_compress_isal.a -l:librte_compress_mlx5.a
-l:librte_compress_octeontx.a -l:librte_compress_zlib.a
-l:librte_regex_mlx5.a -l:librte_regex_octeontx2.a
-l:librte_vdpa_ifc.a -l:librte_vdpa_mlx5.a -l:librte_event_cnxk.a
-l:librte_event_dlb2.a -l:librte_event_dpaa.a -l:librte_event_dpaa2.a
-l:librte_event_dsw.a -l:librte_event_octeontx2.a
-l:librte_event_opdl.a -l:librte_event_skeleton.a -l:librte_event_sw.a
-l:librte_event_octeontx.a -l:librte_baseband_acc100.a
-l:librte_baseband_fpga_5gnr_fec.a -l:librte_baseband_fpga_lte_fec.a
-l:librte_baseband_null.a -l:librte_baseband_turbo_sw.a
-l:librte_node.a -l:librte_graph.a -l:librte_bpf.a
-l:librte_flow_classify.a -l:librte_pipeline.a -l:librte_table.a
-l:librte_port.a -l:librte_fib.a -l:librte_ipsec.a -l:librte_vhost.a
-l:librte_stack.a -l:librte_security.a -l:librte_sched.a
-l:librte_reorder.a -l:librte_rib.a -l:librte_regexdev.a
-l:librte_rawdev.a -l:librte_pdump.a -l:librte_power.a
-l:librte_member.a -l:librte_lpm.a -l:librte_latencystats.a
-l:librte_kni.a -l:librte_jobstats.a -l:librte_ip_frag.a
-l:librte_gso.a -l:librte_gro.a -l:librte_eventdev.a -l:librte_efd.a
-l:librte_distributor.a -l:librte_cryptodev.a -l:librte_compressdev.a
-l:librte_cfgfile.a -l:librte_bitratestats.a -l:librte_bbdev.a
-l:librte_acl.a -l:librte_timer.a -l:librte_hash.a -l:librte_metrics.a
-l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a
-l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a
-l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a
-l:librte_kvargs.a -Wl,--no-whole-archive -Wl,--export-dynamic
-lIPSec_MB /usr/lib/gcc/x86_64-redhat-linux/10/../../../../lib64/librt.so
-Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify
-lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec
-lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder
-lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power
-lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats
-lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd
-lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile
-lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash
-lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter
-lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal
-lrte_telemetry -lrte_kvargs -pthread -lm -ldl -lnuma -lfdt -larchive
-lbsd -larchive -lacl -llzma -lzstd -llz4 -lbz2 -lz -lxml2
-L/usr/usr/lib64 -lmlx5 -lpthread -L/usr/usr/lib64 -lpthread -libverbs
-lpthread -lcrypto -lz -ldl -pthread -lbpf -lz -lmlx4 -lpthread
-L/usr/usr/lib64 -libverbs -lpthread -lpcap -Wl,-R/usr/lib64 -lsze2
-L/opt/isa-l/lib -lisal -lelf -lz -ljansson
Signature · 1 line
David Marchand
#19 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] eal: handle compressed firmwares

2021-06-04 09:27 (UTC+0200), David Marchand:

Quoted 54 lines · Dmitry Kozlyuk
Quoted 39 lines

2021-06-03 18:55 (UTC+0200), David Marchand:
[...]

Quoted 5 lines
config/meson.build +0 -0
index 017bb2efbb..c6985139b4 100644

config/meson.build if libexecinfo.found() and cc.has_header('execinfo.h') · line 181

Quoted 10 lines
config/meson.build +7 -0
@@ -172,6 +172,13 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+  

Suggestion:

config/meson.build +0 -1
index c6985139b4..c3668798c1 100644
@@ -175,7 +175,6 @@ endif
 libarchive = dependency('libarchive', required: false, method: 'pkg-config')
 if libarchive.found()
     dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
-    add_project_link_arguments('-larchive', language: 'c')
     dpdk_extra_ldflags += '-larchive'
 endif
 
lib/eal/meson.build +0 -0
index 1722924f67..5a018d97d6 100644

lib/eal/meson.build subdir(exec_env) · line 22

Quoted 7 lines
lib/eal/meson.build +1 -0
@@ -16,6 +16,7 @@ subdir(exec_env)
 subdir(arch_subdir)
 
 deps += ['kvargs']
+ext_deps += libarchive
 if not is_windows
     deps += ['telemetry']
 endif

I had tried something close when preparing v2 (only keeping RTE_HAS_LIBARCHIVE in config/meson.build and putting extra_ldflags and ext_deps in lib/eal/unix/meson.build) but both my try and your suggestion break static compilation for the helloworld example.

Quoted 3 lines
$ ./devtools/test-meson-builds.sh -vv
...
## Building helloworld

[snip]

Thanks for details.
Indeed, libarchive.pc lists all libraries present at libarchive build time in Libs.private, despite that libarchive static linkage doesn't require them. We'll have to go your way, sorry for misdirection.
Maybe it's worth a comment.

From libarchive README:

I've attempted to minimize static link pollution. If you don't explicitly invoke a particular feature (such as support for a particular compression or format), it won't get pulled in to statically-linked programs. In particular, if you don't explicitly enable a particular compression or decompression support, you won't need to link against the corresponding compression or decompression libraries. This also reduces the size of statically-linked binaries in environments where that matters.

#20 lore raw

Re: [dpdk-dev] [PATCH v2 2/2] eal: handle compressed firmwares

Quoted 19 lines · Dmitry Kozlyuk
Quoted 9 lines

I had tried something close when preparing v2 (only keeping RTE_HAS_LIBARCHIVE in config/meson.build and putting extra_ldflags and ext_deps in lib/eal/unix/meson.build) but both my try and your suggestion break static compilation for the helloworld example.

$ ./devtools/test-meson-builds.sh -vv
...
## Building helloworld

[snip]

Thanks for details.
Indeed, libarchive.pc lists all libraries present at libarchive build time in Libs.private, despite that libarchive static linkage doesn't require them. We'll have to go your way, sorry for misdirection.
Maybe it's worth a comment.

From libarchive README:

Quoted 8 lines

I've attempted to minimize static link pollution. If you don't explicitly invoke a particular feature (such as support for a particular compression or format), it won't get pulled in to statically-linked programs. In particular, if you don't explicitly enable a particular compression or decompression support, you won't need to link against the corresponding compression or decompression libraries. This also reduces the size of statically-linked binaries in environments where that matters.

Yes, I can explain this in v3, but I'll wait a bit to see if issues with this series are discovered.
Thanks Dmitry.

Signature · 1 line
David Marchand
#21 lore raw
In reply to David Marchand
Quoted 12 lines · David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

Quoted 5 lines

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

- Trying with direct mails to maintainers of drivers impacted by this.

- I also want opinions on backporting this change to stable releases.

F34 just got released and will live until 2022-05-17.
F34 got released with 20.11 dpdk.
Afaiu, F35 and maybe F36 will also use dpdk 20.11 (since the dpdk package is aligned on the version used by the ovs package on Fedora). Fedora will need this in downstream 20.11 for net/bnx2x, net/ice, net/nfp and net/qede drivers to work with firmwares in /lib/firmware.

Not sure about Debian/Ubuntu.

Christian, Luca, Timothy?

Thanks!

Signature · 1 line
David Marchand
#22 lore raw

v3 cover Support compressed firmwares

In reply to David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Signature · 34 lines
David Marchand

Changes since v2:
- update comment on libarchive link dependency,

Changes since v1:
- address comments on patch2,


David Marchand (2):
  net/ice: factorize firmware loading
  eal: handle compressed firmwares

 .github/workflows/build.yml      |   1 +
 .travis.yml                      |   1 +
 config/meson.build               |  10 ++
 drivers/net/bnx2x/bnx2x.c        |  35 +++----
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++---------------
 drivers/net/ice/ice_ethdev.c     | 175 ++++++++++++-------------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 drivers/net/nfp/nfp_net.c        |  57 +++-------
 drivers/net/qede/qede_main.c     |  45 ++++----
 lib/eal/include/rte_firmware.h   |  32 ++++++
 lib/eal/unix/eal_firmware.c      | 149 ++++++++++++++++++++++++++
 lib/eal/unix/meson.build         |   1 +
 lib/eal/version.map              |   1 +
 lib/eal/windows/eal.c            |   9 ++
 15 files changed, 324 insertions(+), 298 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

-- 
2.23.0
#23 lore raw

PATCH v3 1/2 net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/base/ice_osdep.h +0 -6
index 878c5597d4..78093adb00 100644
@@ -74,12 +74,6 @@ typedef uint64_t        s64;
 #define min(a, b) RTE_MIN(a, b)
 #define max(a, b) RTE_MAX(a, b)
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 #define FIELD_SIZEOF(t, f) RTE_SIZEOF_FIELD(t, f)
 #define ARRAY_SIZE(arr) RTE_DIM(arr)
 
drivers/net/ice/ice_dcf_parent.c +9 -88

125 lines not on this page · raw message

drivers/net/ice/ice_ethdev.c +77 -84

228 lines not on this page · raw message

drivers/net/ice/ice_ethdev.h +2 -1
index 7f3c26fb6f..edafdf168b 100644
@@ -534,7 +534,8 @@ struct ice_vsi_vlan_pvid_info {
 #define ICE_PF_TO_ETH_DEV(pf) \
 	(((struct ice_pf *)pf)->adapter->eth_dev)
 
-enum ice_pkg_type ice_load_pkg_type(struct ice_hw *hw);
+int
+ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn);
 struct ice_vsi *
 ice_setup_vsi(struct ice_pf *pf, enum ice_vsi_type type);
 int
Signature · 1 line
2.23.0
#24 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Hi David,

Quoted 56 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/ice_ethdev.c +22 -0
+	if (!use_dsn)
+		goto no_dsn;
+
+	memset(opt_ddp_filename, 0, ICE_MAX_PKG_FILENAME_SIZE);
+	snprintf(opt_ddp_filename, ICE_MAX_PKG_FILENAME_SIZE,
+		"ice-%016" PRIx64 ".pkg", dsn);
+	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_UPDATES,
+		ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+		goto load_fw;
+
+	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_DEFAULT,
+		ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+		goto load_fw;
+
+no_dsn:
+	strncpy(pkg_file, ICE_PKG_FILE_UPDATES, ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(pkg_file, 0))
+		goto load_fw;
+	strncpy(pkg_file, ICE_PKG_FILE_DEFAULT, ICE_MAX_PKG_FILENAME_SIZE);
+	if (ice_access(pkg_file, 0)) {
 		PMD_INIT_LOG(ERR, "failed to search file path\n");

drivers/net/ice/ice_ethdev.c ice_load_pkg_type(struct ice_hw *hw) · line 1724

Quoted 10 lines
drivers/net/ice/ice_ethdev.c +2 -1
-		return err;
+		return -1;
 	}
 
+load_fw:
 	file = fopen(pkg_file, "rb");
 	if (!file)  {
 		PMD_INIT_LOG(ERR, "failed to open file: %s\n", pkg_file);
 		return -1;
 	}

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

Quoted 1 line

2.23.0

#25 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Hi David & Qi,

Quoted 80 lines

-----Original Message-----
From: Wang, Haiyue
Sent: Monday, July 5, 2021 09:43
To: David Marchand <[email protected]>; [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Hi David,

Quoted 56 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/ice_ethdev.c +22 -0
+	if (!use_dsn)
+		goto no_dsn;
+
+	memset(opt_ddp_filename, 0, ICE_MAX_PKG_FILENAME_SIZE);
+	snprintf(opt_ddp_filename, ICE_MAX_PKG_FILENAME_SIZE,
+		"ice-%016" PRIx64 ".pkg", dsn);
+	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_UPDATES,
+		ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+		goto load_fw;
+
+	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_DEFAULT,
+		ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+		goto load_fw;
+
+no_dsn:
+	strncpy(pkg_file, ICE_PKG_FILE_UPDATES, ICE_MAX_PKG_FILENAME_SIZE);
+	if (!ice_access(pkg_file, 0))
+		goto load_fw;
+	strncpy(pkg_file, ICE_PKG_FILE_DEFAULT, ICE_MAX_PKG_FILENAME_SIZE);
+	if (ice_access(pkg_file, 0)) {
 		PMD_INIT_LOG(ERR, "failed to search file path\n");

drivers/net/ice/ice_ethdev.c ice_load_pkg_type(struct ice_hw *hw) · line 1724

Quoted 10 lines
drivers/net/ice/ice_ethdev.c +2 -1
-		return err;
+		return -1;
 	}
 
+load_fw:
 	file = fopen(pkg_file, "rb");
 	if (!file)  {
 		PMD_INIT_LOG(ERR, "failed to open file: %s\n", pkg_file);
 		return -1;
 	}

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

Quoted 3 lines

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

We need to update the default/comms search method:

I try the F34:

tree /lib/firmware/intel/ice/
/lib/firmware/intel/ice/

├── ddp
│   ├── ice-1.3.16.0.pkg.xz
│   └── ice.pkg.xz -> ice-1.3.16.0.pkg.xz
└── ddp-comms
    └── ice_comms-1.3.20.0.pkg.xz

It matches the upstream version:

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/intel/ice

d--------- ddp-comms 50 logstatsplain
d--------- ddp 44 logstatsplain

Quoted 2 lines
Quoted 2 lines

2.23.0

#26 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

In reply to Wang, Haiyue
Quoted 74 lines · Haiyue

Hi David,

Quoted 56 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/ice_ethdev.c +22 -0
+     if (!use_dsn)
+             goto no_dsn;
+
+     memset(opt_ddp_filename, 0, ICE_MAX_PKG_FILENAME_SIZE);
+     snprintf(opt_ddp_filename, ICE_MAX_PKG_FILENAME_SIZE,
+             "ice-%016" PRIx64 ".pkg", dsn);
+     strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_UPDATES,
+             ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+             goto load_fw;
+
+     strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_DEFAULT,
+             ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+             goto load_fw;
+
+no_dsn:
+     strncpy(pkg_file, ICE_PKG_FILE_UPDATES, ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(pkg_file, 0))
+             goto load_fw;
+     strncpy(pkg_file, ICE_PKG_FILE_DEFAULT, ICE_MAX_PKG_FILENAME_SIZE);
+     if (ice_access(pkg_file, 0)) {
              PMD_INIT_LOG(ERR, "failed to search file path\n");

drivers/net/ice/ice_ethdev.c ice_load_pkg_type(struct ice_hw *hw) · line 1724

Quoted 10 lines
drivers/net/ice/ice_ethdev.c +2 -1
-             return err;
+             return -1;
      }
 
+load_fw:
      file = fopen(pkg_file, "rb");
      if (!file)  {
              PMD_INIT_LOG(ERR, "failed to open file: %s\n", pkg_file);
              return -1;
      }

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

Quoted 2 lines

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Is there a change in behavior with this patch?

Signature · 1 line
David Marchand
#27 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 82 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 5, 2021 15:08
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 74 lines · Haiyue

Hi David,

Quoted 56 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/ice_ethdev.c +22 -0
+     if (!use_dsn)
+             goto no_dsn;
+
+     memset(opt_ddp_filename, 0, ICE_MAX_PKG_FILENAME_SIZE);
+     snprintf(opt_ddp_filename, ICE_MAX_PKG_FILENAME_SIZE,
+             "ice-%016" PRIx64 ".pkg", dsn);
+     strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_UPDATES,
+             ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+             goto load_fw;
+
+     strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_DEFAULT,
+             ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+             goto load_fw;
+
+no_dsn:
+     strncpy(pkg_file, ICE_PKG_FILE_UPDATES, ICE_MAX_PKG_FILENAME_SIZE);
+     if (!ice_access(pkg_file, 0))
+             goto load_fw;
+     strncpy(pkg_file, ICE_PKG_FILE_DEFAULT, ICE_MAX_PKG_FILENAME_SIZE);
+     if (ice_access(pkg_file, 0)) {
              PMD_INIT_LOG(ERR, "failed to search file path\n");

drivers/net/ice/ice_ethdev.c ice_load_pkg_type(struct ice_hw *hw) · line 1724

Quoted 10 lines
drivers/net/ice/ice_ethdev.c +2 -1
-             return err;
+             return -1;
      }
 
+load_fw:
      file = fopen(pkg_file, "rb");
      if (!file)  {
              PMD_INIT_LOG(ERR, "failed to open file: %s\n", pkg_file);
              return -1;
      }

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

Quoted 6 lines
Quoted 2 lines

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Since the pkg file has the *.xz, now the search method doesn't work. You fix the read only. ;-)

Quoted 6 lines

Is there a change in behavior with this patch?

--
David Marchand

#28 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 7 lines · Haiyue
Quoted 7 lines
Quoted 6 lines

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

Quoted 6 lines
Quoted 3 lines

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Since the pkg file has the *.xz, now the search method doesn't work. You fix the read only. ;-)

This patch fixes nothing wrt to F34.
It simply prepares for the next patch, because I did not want to fix in two places with the same change.
I can squash everything in a single patch if you prefer.

In the end, I end up with the same question, but for the whole series:

Quoted 1 line
Quoted 1 line

Is there a change in behavior with this patch?

Signature · 1 line
David Marchand
#29 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 24 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 5, 2021 4:34 PM
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 6 lines · Haiyue
Quoted 6 lines
Quoted 6 lines

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES

"/lib/firmware/updates/intel/ice/ddp/ice.pkg"

Quoted 5 lines
Quoted 4 lines

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Quoted 6 lines
Quoted 2 lines

Since the pkg file has the *.xz, now the search method doesn't work. You fix the read only. ;-)

This patch fixes nothing wrt to F34.
It simply prepares for the next patch, because I did not want to fix in two places with the same change.

I agree with this , the patch just do code refactory, it looks good to have , no matter if we need this for F34 or not.

Haiyue, did you see any problem on F34 with both patches be applied or just this one be applied?

Quoted 11 lines

I can squash everything in a single patch if you prefer.

In the end, I end up with the same question, but for the whole series:

Quoted 1 line
Quoted 1 line

Is there a change in behavior with this patch?

--
David Marchand

#30 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 34 lines

-----Original Message-----
From: Zhang, Qi Z <[email protected]>
Sent: Monday, July 5, 2021 18:00
To: David Marchand <[email protected]>; Wang, Haiyue <[email protected]>
Cc: [email protected]; Yang, Qiming <[email protected]>
Subject: RE: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 25 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 5, 2021 4:34 PM
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 6 lines · Haiyue
Quoted 6 lines
Quoted 6 lines

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES

"/lib/firmware/updates/intel/ice/ddp/ice.pkg"

Quoted 7 lines
Quoted 4 lines

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Since the pkg file has the *.xz, now the search method doesn't work. You fix the read only. ;-)

Quoted 8 lines
Quoted 3 lines

This patch fixes nothing wrt to F34.
It simply prepares for the next patch, because I did not want to fix in two places with the same change.

I agree with this , the patch just do code refactory, it looks good to have , no matter if we need this for F34 or not.

Haiyue, did you see any problem on F34 with both patches be applied or just this one be applied?

Just install a VM to see the firmware name, not test yet. ;-)

Quoted 15 lines
Quoted 14 lines

I can squash everything in a single patch if you prefer.

In the end, I end up with the same question, but for the whole series:

Quoted 1 line
Quoted 1 line

Is there a change in behavior with this patch?

--
David Marchand

#31 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

In reply to David Marchand
Quoted 22 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 5, 2021 16:34
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 7 lines · Haiyue
Quoted 7 lines
Quoted 6 lines

I'm wondering what's full name for ice firmware in F34, has any *.xz postfix ? If so, the search method will also needs to be updated, since we will check each file can be accessed:

#define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"

Quoted 6 lines
Quoted 3 lines

This first patch is a preparation to have a single helper to select/open the firmware.
I don't get what you mean.

Since the pkg file has the *.xz, now the search method doesn't work. You fix the read only. ;-)

Quoted 8 lines

This patch fixes nothing wrt to F34.
It simply prepares for the next patch, because I did not want to fix in two places with the same change.
I can squash everything in a single patch if you prefer.

In the end, I end up with the same question, but for the whole series:

Quoted 1 line
Quoted 1 line

Is there a change in behavior with this patch?

Sorry, after applied the whole patch to read the change, I get the real design.

Looks like one patch is clear, since I'm lost in Patch#1, I thought it used '*.pkg' to get the pkg name.... after applied, I read the source code directly, yes, it’s clear in ice_load_pkg: the rte_firmware_read has handled "*.xz" right.

static int
ice_dcf_load_pkg(struct ice_adapter *adapter)

{
	struct ice_dcf_adapter *dcf_adapter =
			container_of(&adapter->hw, struct ice_dcf_adapter, parent.hw);
	struct virtchnl_pkg_info pkg_info;
	struct dcf_virtchnl_cmd vc_cmd;
	bool use_dsn;
	uint64_t dsn = 0;

	vc_cmd.v_op = VIRTCHNL_OP_DCF_GET_PKG_INFO;
	vc_cmd.req_msglen = 0;
	vc_cmd.req_msg = NULL;
	vc_cmd.rsp_buflen = sizeof(pkg_info);
	vc_cmd.rsp_msgbuf = (uint8_t *)&pkg_info;

	use_dsn = ice_dcf_execute_virtchnl_cmd(&dcf_adapter->real_hw, &vc_cmd) == 0;
	if (use_dsn)
		rte_memcpy(&dsn, pkg_info.dsn, sizeof(dsn));

	return ice_load_pkg(adapter, use_dsn, dsn);
}
Quoted 4 lines

--
David Marchand

Acked-by on v3 1/2
#32 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

In reply to David Marchand
Quoted 22 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

Quoted 8 lines
---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)

Squash into a single patch seems great help to understand the compress firmware context. ;-)

But it's OK as two patches.

Quoted 2 lines

--
2.23.0

#33 lore raw

Re: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Quoted 32 lines · Haiyue
Quoted 23 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/2] net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

Quoted 8 lines
---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
Quoted 6 lines

Squash into a single patch seems great help to understand the compress firmware context. ;-)

But it's OK as two patches.

Cool, thanks for the review and test!

Signature · 1 line
David Marchand
#34 lore raw

PATCH v3 2/2 eal: handle compressed firmwares

In reply to David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |  10 +++
 drivers/net/bnx2x/bnx2x.c      |  35 +++-----
 drivers/net/ice/ice_ethdev.c   |  60 +++----------
 drivers/net/nfp/nfp_net.c      |  57 +++----------
 drivers/net/qede/qede_main.c   |  45 ++++------
 lib/eal/include/rte_firmware.h |  32 +++++++
 lib/eal/unix/eal_firmware.c    | 149 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   1 +
 lib/eal/windows/eal.c          |   9 ++
 12 files changed, 259 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +10 -0
index 017bb2efbb..639aa74e12 100644
@@ -172,6 +172,16 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    # Push libarchive link dependency at the project level to support
+    # statically linking dpdk apps. Details at:
+    # https://inbox.dpdk.org/dev/20210605004024.660267a1@sovereign/
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 # check for libbsd
 libbsd = dependency('libbsd', required: false, method: 'pkg-config')
 if libbsd.found()
drivers/net/bnx2x/bnx2x.c +13 -22
index 654878d9de..60292753e2 100644
@@ -26,7 +26,9 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
@@ -9655,44 +9657,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +12 -48

110 lines not on this page · raw message

drivers/net/nfp/nfp_net.c +12 -45

96 lines not on this page · raw message

drivers/net/qede/qede_main.c +18 -27

76 lines not on this page · raw message

lib/eal/include/rte_firmware.h new file +32 -0
new file mode 100644
index 0000000000..3389e60ca0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param[in] name
+ *      Firmware filename to load.
+ * @param[out] buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for calling free() on this buffer.
+ * @param[out] bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +149 -0

149 lines not on this page · raw message

lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +1 -0
index fe5c3dac98..2df65c6903 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 8483f6b02c..6fe2bdd282 100644
@@ -21,6 +21,7 @@
 #include <eal_private.h>
 #include <rte_service_component.h>
 #include <rte_vfio.h>
+#include <rte_firmware.h>
 
 #include "eal_hugepages.h"
 #include "eal_trace.h"
@@ -465,3 +466,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
Signature · 1 line
2.23.0
Acked-by on v3 2/2
#35 lore raw

Re: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

Quoted 634 lines · David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |  10 +++
 drivers/net/bnx2x/bnx2x.c      |  35 +++-----
 drivers/net/ice/ice_ethdev.c   |  60 +++----------
 drivers/net/nfp/nfp_net.c      |  57 +++----------
 drivers/net/qede/qede_main.c   |  45 ++++------
 lib/eal/include/rte_firmware.h |  32 +++++++
 lib/eal/unix/eal_firmware.c    | 149 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   1 +
 lib/eal/windows/eal.c          |   9 ++
 12 files changed, 259 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +10 -0
index 017bb2efbb..639aa74e12 100644
@@ -172,6 +172,16 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    # Push libarchive link dependency at the project level to support
+    # statically linking dpdk apps. Details at:
+    # https://inbox.dpdk.org/dev/20210605004024.660267a1@sovereign/
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 # check for libbsd
 libbsd = dependency('libbsd', required: false, method: 'pkg-config')
 if libbsd.found()
drivers/net/bnx2x/bnx2x.c +13 -22
index 654878d9de..60292753e2 100644
@@ -26,7 +26,9 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
@@ -9655,44 +9657,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +12 -48
index d180b73c32..06da1bbd94 100644
@@ -10,6 +10,7 @@
 #include <sys/stat.h>
 #include <unistd.h>
 
+#include <rte_firmware.h>
 #include <rte_tailq.h>
 
 #include "base/ice_sched.h"
@@ -1673,22 +1674,14 @@ ice_load_pkg_type(struct ice_hw *hw)
 	return package_type;
 }
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 int ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn)
 {
 	struct ice_hw *hw = &adapter->hw;
 	char pkg_file[ICE_MAX_PKG_FILENAME_SIZE];
 	char opt_ddp_filename[ICE_MAX_PKG_FILENAME_SIZE];
+	void *buf;
+	size_t bufsz;
 	int err;
-	uint8_t *buf = NULL;
-	int buf_len;
-	FILE *file;
-	struct stat fstat;
 
 	if (!use_dsn)
 		goto no_dsn;
@@ -1698,57 +1691,31 @@ int ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn)
 		"ice-%016" PRIx64 ".pkg", dsn);
 	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_UPDATES,
 		ICE_MAX_PKG_FILENAME_SIZE);
-	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+	strcat(pkg_file, opt_ddp_filename);
+	if (rte_firmware_read(pkg_file, &buf, &bufsz) == 0)
 		goto load_fw;
 
 	strncpy(pkg_file, ICE_PKG_FILE_SEARCH_PATH_DEFAULT,
 		ICE_MAX_PKG_FILENAME_SIZE);
-	if (!ice_access(strcat(pkg_file, opt_ddp_filename), 0))
+	strcat(pkg_file, opt_ddp_filename);
+	if (rte_firmware_read(pkg_file, &buf, &bufsz) == 0)
 		goto load_fw;
 
 no_dsn:
 	strncpy(pkg_file, ICE_PKG_FILE_UPDATES, ICE_MAX_PKG_FILENAME_SIZE);
-	if (!ice_access(pkg_file, 0))
+	if (rte_firmware_read(pkg_file, &buf, &bufsz) == 0)
 		goto load_fw;
+
 	strncpy(pkg_file, ICE_PKG_FILE_DEFAULT, ICE_MAX_PKG_FILENAME_SIZE);
-	if (ice_access(pkg_file, 0)) {
+	if (rte_firmware_read(pkg_file, &buf, &bufsz) < 0) {
 		PMD_INIT_LOG(ERR, "failed to search file path\n");
 		return -1;
 	}
 
 load_fw:
-	file = fopen(pkg_file, "rb");
-	if (!file)  {
-		PMD_INIT_LOG(ERR, "failed to open file: %s\n", pkg_file);
-		return -1;
-	}
-
 	PMD_INIT_LOG(DEBUG, "DDP package name: %s", pkg_file);
 
-	err = stat(pkg_file, &fstat);
-	if (err) {
-		PMD_INIT_LOG(ERR, "failed to get file stats\n");
-		goto out;
-	}
-
-	buf_len = fstat.st_size;
-	buf = rte_malloc(NULL, buf_len, 0);
-
-	if (!buf) {
-		PMD_INIT_LOG(ERR, "failed to allocate buf of size %d for package\n",
-				buf_len);
-		err = -1;
-		goto out;
-	}
-
-	err = fread(buf, buf_len, 1, file);
-	if (err != 1) {
-		PMD_INIT_LOG(ERR, "failed to read package data\n");
-		err = -1;
-		goto out;
-	}
-
-	err = ice_copy_and_init_pkg(hw, buf, buf_len);
+	err = ice_copy_and_init_pkg(hw, buf, bufsz);
 	if (err) {
 		PMD_INIT_LOG(ERR, "ice_copy_and_init_hw failed: %d\n", err);
 		goto out;
@@ -1758,13 +1725,10 @@ int ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn)
 	adapter->active_pkg_type = ice_load_pkg_type(hw);
 
 out:
-	fclose(file);
-	rte_free(buf);
+	free(buf);
 	return err;
 }
 
-#undef ice_access
-
 static void
 ice_base_queue_get(struct ice_pf *pf)
 {
drivers/net/nfp/nfp_net.c +12 -45
index 2ee88fbfc7..879da7ef59 100644
@@ -29,6 +29,7 @@
 #include <rte_alarm.h>
 #include <rte_spinlock.h>
 #include <rte_service_component.h>
+#include <rte_firmware.h>
 
 #include "nfpcore/nfp_cpp.h"
 #include "nfpcore/nfp_nffw.h"
@@ -3366,12 +3367,10 @@ static int
 nfp_fw_upload(struct rte_pci_device *dev, struct nfp_nsp *nsp, char *card)
 {
 	struct nfp_cpp *cpp = nsp->cpp;
-	int fw_f;
-	char *fw_buf;
+	void *fw_buf;
 	char fw_name[125];
 	char serial[40];
-	struct stat file_stat;
-	off_t fsize, bytes;
+	size_t fsize;
 
 	/* Looking for firmware file in order of priority */
 
@@ -3384,66 +3383,34 @@ nfp_fw_upload(struct rte_pci_device *dev, struct nfp_nsp *nsp, char *card)
 
 	snprintf(fw_name, sizeof(fw_name), "%s/%s.nffw", DEFAULT_FW_PATH,
 			serial);
-
 	PMD_DRV_LOG(DEBUG, "Trying with fw file: %s", fw_name);
-	fw_f = open(fw_name, O_RDONLY);
-	if (fw_f >= 0)
-		goto read_fw;
+	if (rte_firmware_read(fw_name, &fw_buf, &fsize) == 0)
+		goto load_fw;
 
 	/* Then try the PCI name */
 	snprintf(fw_name, sizeof(fw_name), "%s/pci-%s.nffw", DEFAULT_FW_PATH,
 			dev->device.name);
-
 	PMD_DRV_LOG(DEBUG, "Trying with fw file: %s", fw_name);
-	fw_f = open(fw_name, O_RDONLY);
-	if (fw_f >= 0)
-		goto read_fw;
+	if (rte_firmware_read(fw_name, &fw_buf, &fsize) == 0)
+		goto load_fw;
 
 	/* Finally try the card type and media */
 	snprintf(fw_name, sizeof(fw_name), "%s/%s", DEFAULT_FW_PATH, card);
 	PMD_DRV_LOG(DEBUG, "Trying with fw file: %s", fw_name);
-	fw_f = open(fw_name, O_RDONLY);
-	if (fw_f < 0) {
+	if (rte_firmware_read(fw_name, &fw_buf, &fsize) < 0) {
 		PMD_DRV_LOG(INFO, "Firmware file %s not found.", fw_name);
 		return -ENOENT;
 	}
 
-read_fw:
-	if (fstat(fw_f, &file_stat) < 0) {
-		PMD_DRV_LOG(INFO, "Firmware file %s size is unknown", fw_name);
-		close(fw_f);
-		return -ENOENT;
-	}
-
-	fsize = file_stat.st_size;
-	PMD_DRV_LOG(INFO, "Firmware file found at %s with size: %" PRIu64 "",
-			    fw_name, (uint64_t)fsize);
-
-	fw_buf = malloc((size_t)fsize);
-	if (!fw_buf) {
-		PMD_DRV_LOG(INFO, "malloc failed for fw buffer");
-		close(fw_f);
-		return -ENOMEM;
-	}
-	memset(fw_buf, 0, fsize);
-
-	bytes = read(fw_f, fw_buf, fsize);
-	if (bytes != fsize) {
-		PMD_DRV_LOG(INFO, "Reading fw to buffer failed."
-				   "Just %" PRIu64 " of %" PRIu64 " bytes read",
-				   (uint64_t)bytes, (uint64_t)fsize);
-		free(fw_buf);
-		close(fw_f);
-		return -EIO;
-	}
+load_fw:
+	PMD_DRV_LOG(INFO, "Firmware file found at %s with size: %zu",
+		fw_name, fsize);
 
 	PMD_DRV_LOG(INFO, "Uploading the firmware ...");
-	nfp_nsp_load_fw(nsp, fw_buf, bytes);
+	nfp_nsp_load_fw(nsp, fw_buf, fsize);
 	PMD_DRV_LOG(INFO, "Done");
 
 	free(fw_buf);
-	close(fw_f);
-
 	return 0;
 }
 
drivers/net/qede/qede_main.c +18 -27
index caa9d1d4f6..504e2c66a0 100644
@@ -5,7 +5,9 @@
  */
 
 #include <limits.h>
+
 #include <rte_alarm.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #include "qede_ethdev.h"
@@ -127,51 +129,40 @@ static void qed_free_stream_mem(struct ecore_dev *edev)
 #ifdef CONFIG_ECORE_BINARY_FW
 static int qed_load_firmware_data(struct ecore_dev *edev)
 {
-	int fd;
-	struct stat st;
 	const char *fw = RTE_LIBRTE_QEDE_FW;
+	void *buf;
+	size_t bufsz;
+	int ret;
 
 	if (strcmp(fw, "") == 0)
 		strcpy(qede_fw_file, QEDE_DEFAULT_FIRMWARE);
 	else
 		strcpy(qede_fw_file, fw);
 
-	fd = open(qede_fw_file, O_RDONLY);
-	if (fd < 0) {
-		DP_ERR(edev, "Can't open firmware file\n");
-		return -ENOENT;
-	}
-
-	if (fstat(fd, &st) < 0) {
-		DP_ERR(edev, "Can't stat firmware file\n");
-		close(fd);
+	if (rte_firmware_read(qede_fw_file, &buf, &bufsz) < 0) {
+		DP_ERR(edev, "Can't read firmware data: %s\n", qede_fw_file);
 		return -1;
 	}
 
-	edev->firmware = rte_zmalloc("qede_fw", st.st_size,
-				    RTE_CACHE_LINE_SIZE);
+	edev->firmware = rte_zmalloc("qede_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!edev->firmware) {
 		DP_ERR(edev, "Can't allocate memory for firmware\n");
-		close(fd);
-		return -ENOMEM;
+		ret = -ENOMEM;
+		goto out;
 	}
 
-	if (read(fd, edev->firmware, st.st_size) != st.st_size) {
-		DP_ERR(edev, "Can't read firmware data\n");
-		close(fd);
-		return -1;
-	}
-
-	edev->fw_len = st.st_size;
+	memcpy(edev->firmware, buf, bufsz);
+	edev->fw_len = bufsz;
 	if (edev->fw_len < 104) {
 		DP_ERR(edev, "Invalid fw size: %" PRIu64 "\n",
 			  edev->fw_len);
-		close(fd);
-		return -EINVAL;
+		ret = -EINVAL;
+		goto out;
 	}
-
-	close(fd);
-	return 0;
+	ret = 0;
+out:
+	free(buf);
+	return ret;
 }
 #endif
 
lib/eal/include/rte_firmware.h new file +32 -0
new file mode 100644
index 0000000000..3389e60ca0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param[in] name
+ *      Firmware filename to load.
+ * @param[out] buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for calling free() on this buffer.
+ * @param[out] bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +131 -0
new file mode 100644
index 0000000000..494cd5f058
@@ -0,0 +1,149 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifdef RTE_HAS_LIBARCHIVE
+#include <archive.h>
+#endif
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_firmware.h>
+#include <rte_log.h>
+
+#ifdef RTE_HAS_LIBARCHIVE
+
+struct firmware_read_ctx {
+	struct archive *a;
+};
+
+static int
+firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)
+{
+	struct archive_entry *e;
+
+	ctx->a = archive_read_new();
+	if (ctx->a == NULL)
+		return -1;
+	if (archive_read_support_format_raw(ctx->a) != ARCHIVE_OK ||
+			archive_read_support_filter_xz(ctx->a) != ARCHIVE_OK ||
+			archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK ||
+			archive_read_next_header(ctx->a, &e) != ARCHIVE_OK) {
+		archive_read_free(ctx->a);
+		ctx->a = NULL;
+		return -1;
+	}
+	return 0;
+}
+
+static ssize_t
+firmware_read_block(struct firmware_read_ctx *ctx, void *buf, size_t count)
+{
+	return archive_read_data(ctx->a, buf, count);
+}
+
+static void
+firmware_close(struct firmware_read_ctx *ctx)
+{
+	archive_read_free(ctx->a);
+	ctx->a = NULL;
+}
+
+#else /* !RTE_HAS_LIBARCHIVE */
+
+struct firmware_read_ctx {
+	int fd;
+};
+
+static int
+firmware_open(struct firmware_read_ctx *ctx, const char *name,
+	__rte_unused size_t blocksize)
+{
+	ctx->fd = open(name, O_RDONLY);
+	if (ctx->fd < 0)
+		return -1;
+	return 0;
+}
+
+static ssize_t
+firmware_read_block(struct firmware_read_ctx *ctx, void *buf, size_t count)
+{
+	return read(ctx->fd, buf, count);
+}
+
+static void
+firmware_close(struct firmware_read_ctx *ctx)
+{
+	close(ctx->fd);
+	ctx->fd = -1;
+}
+
+#endif /* !RTE_HAS_LIBARCHIVE */
+
+static int
+firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+	const size_t blocksize = 4096;
+	struct firmware_read_ctx ctx;
+	int ret = -1;
+	int err;
+
+	*buf = NULL;
+	*bufsz = 0;
+
+	if (firmware_open(&ctx, name, blocksize) < 0)
+		return -1;
+
+	do {
+		void *tmp;
+
+		tmp = realloc(*buf, *bufsz + blocksize);
+		if (tmp == NULL) {
+			free(*buf);
+			*buf = NULL;
+			*bufsz = 0;
+			goto out;
+		}
+		*buf = tmp;
+
+		err = firmware_read_block(&ctx, RTE_PTR_ADD(*buf, *bufsz), blocksize);
+		if (err < 0) {
+			free(*buf);
+			*buf = NULL;
+			*bufsz = 0;
+			goto out;
+		}
+		*bufsz += err;
+
+	} while (err != 0);
+
+	ret = 0;
+out:
+	firmware_close(&ctx);
+	return ret;
+}
+
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz)
+{

lib/eal/unix/eal_firmware.c line 141

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+	char path[PATH_MAX];
+	int ret;
+
+	ret = firmware_read(name, buf, bufsz);
+	if (ret < 0) {
+		snprintf(path, sizeof(path), "%s.xz", name);
+		path[PATH_MAX - 1] = '\0';
+#ifndef RTE_HAS_LIBARCHIVE
+		if (access(path, F_OK) == 0) {
+			RTE_LOG(WARNING, EAL, "libarchive not available, %s cannot be decompressed\n",

Maybe 'not linked' instead of 'not available'. A user might decide to install libarchive and then be confused that the error message still pops up (without realizing they need to re-compile).

Otherwise,

Quoted 55 lines
lib/eal/unix/eal_firmware.c new file +8 -0
+				path);
+		}
+#else
+		ret = firmware_read(path, buf, bufsz);
+#endif
+	}
+	return ret;
+}
lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +1 -0
index fe5c3dac98..2df65c6903 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 8483f6b02c..6fe2bdd282 100644
@@ -21,6 +21,7 @@
 #include <eal_private.h>
 #include <rte_service_component.h>
 #include <rte_vfio.h>
+#include <rte_firmware.h>
 
 #include "eal_hugepages.h"
 #include "eal_trace.h"
@@ -465,3 +466,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
#36 lore raw

Re: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

In reply to David Marchand
Quoted 67 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Igor Russkikh <[email protected]>; Aaron Conole <[email protected]>; Michael Santana <[email protected]>; Richardson, Bruce <[email protected]>; Rasesh Mody <[email protected]>; Shahed Shaikh <[email protected]>; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>; Heinrich Kuhn <[email protected]>; Devendra Singh Rawat <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>; Dmitry Kozlyuk <[email protected]>; Narcisa Ana Maria Vasile <[email protected]>; Dmitry Malloy <[email protected]>; Kadam, Pallavi <[email protected]>
Subject: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |  10 +++
 drivers/net/bnx2x/bnx2x.c      |  35 +++-----
 drivers/net/ice/ice_ethdev.c   |  60 +++----------
 drivers/net/nfp/nfp_net.c      |  57 +++----------
 drivers/net/qede/qede_main.c   |  45 ++++------
 lib/eal/include/rte_firmware.h |  32 +++++++
 lib/eal/unix/eal_firmware.c    | 149 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   1 +
 lib/eal/windows/eal.c          |   9 ++
 12 files changed, 259 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
lib/eal/unix/eal_firmware.c new file +11 -0
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+	char path[PATH_MAX];
+	int ret;
+
+	ret = firmware_read(name, buf, bufsz);
+	if (ret < 0) {
+		snprintf(path, sizeof(path), "%s.xz", name);
+		path[PATH_MAX - 1] = '\0';
+#ifndef RTE_HAS_LIBARCHIVE

lib/eal/unix/eal_firmware.c line 149

Quoted 10 lines
lib/eal/unix/eal_firmware.c new file +10 -0
+		if (access(path, F_OK) == 0) {
+			RTE_LOG(WARNING, EAL, "libarchive not available, %s cannot be decompressed\n",
+				path);
+		}
+#else
+		ret = firmware_read(path, buf, bufsz);
+#endif
+	}
+	return ret;
+}

Since ice PMD needs to check if the firmware file with different name can be accessed by some kind of order, before doing the final firmware selection. Should we also add the firmware access API for handling this ?

bool
rte_firmware_access(const char *name)

{
	char path[PATH_MAX];

	if (access(name, F_OK) == 0)
		return true;

	snprintf(path, sizeof(path), "%s.xz", name);
	path[PATH_MAX - 1] = '\0';
	if (access(path, F_OK) == 0)
		return true;

	return false;
}
Quoted 2 lines

--
2.23.0

#37 lore raw

Re: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

Quoted 18 lines · Haiyue
Quoted 18 lines
lib/eal/unix/eal_firmware.c new file +18 -0
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz)
+{
+     char path[PATH_MAX];
+     int ret;
+
+     ret = firmware_read(name, buf, bufsz);
+     if (ret < 0) {
+             snprintf(path, sizeof(path), "%s.xz", name);
+             path[PATH_MAX - 1] = '\0';
+#ifndef RTE_HAS_LIBARCHIVE
+             if (access(path, F_OK) == 0) {
+                     RTE_LOG(WARNING, EAL, "libarchive not available, %s cannot be decompressed\n",
+                             path);
+             }
+#else
+             ret = firmware_read(path, buf, bufsz);
+#endif
Quoted 8 lines

lib/eal/unix/eal_firmware.c line 127

Quoted 3 lines
lib/eal/unix/eal_firmware.c new file +3 -0
+     }
+     return ret;
+}

Since ice PMD needs to check if the firmware file with different name can be accessed by some kind of order, before doing the final firmware selection. Should we also add the firmware access API for handling this ?

I don't see the need.
Is the behavior changed for net/ice with this patch?

Signature · 1 line
David Marchand
Tested-by on v3 2/2
#38 lore raw

Re: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

In reply to David Marchand
Quoted 37 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of David Marchand
Sent: Tuesday, June 29, 2021 16:07
To: [email protected]
Cc: Igor Russkikh <[email protected]>; Aaron Conole <[email protected]>; Michael Santana <[email protected]>; Richardson, Bruce <[email protected]>; Rasesh Mody <[email protected]>; Shahed Shaikh <[email protected]>; Yang, Qiming <[email protected]>; Zhang, Qi Z <[email protected]>; Heinrich Kuhn <[email protected]>; Devendra Singh Rawat <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>; Dmitry Kozlyuk <[email protected]>; Narcisa Ana Maria Vasile <[email protected]>; Dmitry Malloy <[email protected]>; Kadam, Pallavi <[email protected]>
Subject: [dpdk-dev] [PATCH v3 2/2] eal: handle compressed firmwares

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

Quoted 6 lines
---
 .github/workflows/build.yml    |   1 +
 .travis.yml                    |   1 +
 config/meson.build             |  10 +++
 drivers/net/bnx2x/bnx2x.c      |  35 +++-----
 drivers/net/ice/ice_ethdev.c   |  60 +++----------

For ice PMD:

Quoted 15 lines
 drivers/net/nfp/nfp_net.c      |  57 +++----------
 drivers/net/qede/qede_main.c   |  45 ++++------
 lib/eal/include/rte_firmware.h |  32 +++++++
 lib/eal/unix/eal_firmware.c    | 149 +++++++++++++++++++++++++++++++++
 lib/eal/unix/meson.build       |   1 +
 lib/eal/version.map            |   1 +
 lib/eal/windows/eal.c          |   9 ++
 12 files changed, 259 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

--
2.23.0

#39 lore raw

v4 cover Support compressed firmwares

In reply to David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Signature · 38 lines
David Marchand

Changes since v3:
- add release note update,

Changes since v2:
- update comment on libarchive link dependency,

Changes since v1:
- address comments on patch2,


David Marchand (2):
  net/ice: factorize firmware loading
  eal: handle compressed firmwares

 .github/workflows/build.yml            |   1 +
 .travis.yml                            |   1 +
 config/meson.build                     |  10 ++
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/bnx2x/bnx2x.c              |  35 ++---
 drivers/net/ice/base/ice_osdep.h       |   6 -
 drivers/net/ice/ice_dcf_parent.c       |  97 ++------------
 drivers/net/ice/ice_ethdev.c           | 175 ++++++++++---------------
 drivers/net/ice/ice_ethdev.h           |   3 +-
 drivers/net/nfp/nfp_net.c              |  57 ++------
 drivers/net/qede/qede_main.c           |  45 +++----
 lib/eal/include/rte_firmware.h         |  32 +++++
 lib/eal/unix/eal_firmware.c            | 149 +++++++++++++++++++++
 lib/eal/unix/meson.build               |   1 +
 lib/eal/version.map                    |   1 +
 lib/eal/windows/eal.c                  |   9 ++
 16 files changed, 330 insertions(+), 298 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

-- 
2.23.0
#40 lore raw

PATCH v4 1/2 net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/base/ice_osdep.h +0 -6
index 878c5597d4..78093adb00 100644
@@ -74,12 +74,6 @@ typedef uint64_t        s64;
 #define min(a, b) RTE_MIN(a, b)
 #define max(a, b) RTE_MAX(a, b)
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 #define FIELD_SIZEOF(t, f) RTE_SIZEOF_FIELD(t, f)
 #define ARRAY_SIZE(arr) RTE_DIM(arr)
 
drivers/net/ice/ice_dcf_parent.c +9 -88

125 lines not on this page · raw message

drivers/net/ice/ice_ethdev.c +77 -84

228 lines not on this page · raw message

drivers/net/ice/ice_ethdev.h +2 -1
index 7f3c26fb6f..edafdf168b 100644
@@ -534,7 +534,8 @@ struct ice_vsi_vlan_pvid_info {
 #define ICE_PF_TO_ETH_DEV(pf) \
 	(((struct ice_pf *)pf)->adapter->eth_dev)
 
-enum ice_pkg_type ice_load_pkg_type(struct ice_hw *hw);
+int
+ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn);
 struct ice_vsi *
 ice_setup_vsi(struct ice_pf *pf, enum ice_vsi_type type);
 int
Signature · 1 line
2.23.0
#41 lore raw

PATCH v4 2/2 eal: handle compressed firmwares

In reply to David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmwares by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmwares have been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v3:
- added release note update,
- updated warning message when uncompressing is unavailable,

Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml            |   1 +
 .travis.yml                            |   1 +
 config/meson.build                     |  10 ++
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/bnx2x/bnx2x.c              |  35 +++---
 drivers/net/ice/ice_ethdev.c           |  60 ++--------
 drivers/net/nfp/nfp_net.c              |  57 ++--------
 drivers/net/qede/qede_main.c           |  45 +++-----
 lib/eal/include/rte_firmware.h         |  32 ++++++
 lib/eal/unix/eal_firmware.c            | 149 +++++++++++++++++++++++++
 lib/eal/unix/meson.build               |   1 +
 lib/eal/version.map                    |   1 +
 lib/eal/windows/eal.c                  |   9 ++
 13 files changed, 265 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/include/rte_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +10 -0
index 017bb2efbb..639aa74e12 100644
@@ -172,6 +172,16 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    # Push libarchive link dependency at the project level to support
+    # statically linking dpdk apps. Details at:
+    # https://inbox.dpdk.org/dev/20210605004024.660267a1@sovereign/
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 # check for libbsd
 libbsd = dependency('libbsd', required: false, method: 'pkg-config')
 if libbsd.found()
doc/guides/rel_notes/release_21_08.rst +6 -0
index 0a05cb02fa..df6f9ebafc 100644
@@ -61,6 +61,12 @@ New Features
   representing sub-domains of functionality. Each auxiliary device
   represents a part of its parent functionality.
 
+* **Added XZ compressed firmware support.**
+
+  Using ``rte_firmware_read``, a driver can now handle XZ compressed firmwares
+  in a transparent way, with EAL uncompressing using libarchive if this library
+  is available when building DPDK.
+
 * **Added Baseband PHY CNXK PMD.**
 
   Added Baseband PHY PMD which allows to configure BPHY hardware block
drivers/net/bnx2x/bnx2x.c +13 -22
index 654878d9de..60292753e2 100644
@@ -26,7 +26,9 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
+#include <rte_firmware.h>
 #include <rte_string_fns.h>
 
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
@@ -9655,44 +9657,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +12 -48

110 lines not on this page · raw message

drivers/net/nfp/nfp_net.c +12 -45

96 lines not on this page · raw message

drivers/net/qede/qede_main.c +18 -27

76 lines not on this page · raw message

lib/eal/include/rte_firmware.h new file +32 -0
new file mode 100644
index 0000000000..3389e60ca0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef __RTE_FIRMWARE_H__
+#define __RTE_FIRMWARE_H__
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param[in] name
+ *      Firmware filename to load.
+ * @param[out] buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for calling free() on this buffer.
+ * @param[out] bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif
lib/eal/unix/eal_firmware.c new file +149 -0

149 lines not on this page · raw message

lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +1 -0
index fe5c3dac98..2df65c6903 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 8483f6b02c..6fe2bdd282 100644
@@ -21,6 +21,7 @@
 #include <eal_private.h>
 #include <rte_service_component.h>
 #include <rte_vfio.h>
+#include <rte_firmware.h>
 
 #include "eal_hugepages.h"
 #include "eal_trace.h"
@@ -465,3 +466,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
Signature · 1 line
2.23.0
#42 lore raw

v5 cover Support compressed firmwares

In reply to David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

Signature · 41 lines
David Marchand

Changes since v4:
- unexport header,
- s/firmwares/firmware/ from Bruce,

Changes since v3:
- add release note update,

Changes since v2:
- update comment on libarchive link dependency,

Changes since v1:
- address comments on patch2,

David Marchand (2):
  net/ice: factorize firmware loading
  eal: handle compressed firmware

 .github/workflows/build.yml            |   1 +
 .travis.yml                            |   1 +
 config/meson.build                     |  10 ++
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/bnx2x/bnx2x.c              |  36 ++---
 drivers/net/ice/base/ice_osdep.h       |   6 -
 drivers/net/ice/ice_dcf_parent.c       |  97 ++------------
 drivers/net/ice/ice_ethdev.c           | 176 ++++++++++---------------
 drivers/net/ice/ice_ethdev.h           |   3 +-
 drivers/net/nfp/nfp_net.c              |  58 ++------
 drivers/net/qede/qede_main.c           |  46 +++----
 lib/eal/common/eal_firmware.h          |  32 +++++
 lib/eal/unix/eal_firmware.c            | 150 +++++++++++++++++++++
 lib/eal/unix/meson.build               |   1 +
 lib/eal/version.map                    |   1 +
 lib/eal/windows/eal.c                  |   9 ++
 16 files changed, 335 insertions(+), 298 deletions(-)
 create mode 100644 lib/eal/common/eal_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c

-- 
2.23.0
#43 lore raw

PATCH v5 1/2 net/ice: factorize firmware loading

Both "normal" and "dcf" inits have their copy of some firmware loading code.

The DSN query is moved in specific parts for the "normal" and "dcf" init.

A common helper ice_load_pkg is then introduced and takes an adapter pointer as its main input.

This helper takes care of finding the right firmware file and loading it.
The adapter active_pkg_type field is set by this helper.

The ice_access macro is removed from the osdep.h header: osdep.h should only hosts wrappers for base driver code.

---
 drivers/net/ice/base/ice_osdep.h |   6 --
 drivers/net/ice/ice_dcf_parent.c |  97 ++-----------------
 drivers/net/ice/ice_ethdev.c     | 161 +++++++++++++++----------------
 drivers/net/ice/ice_ethdev.h     |   3 +-
 4 files changed, 88 insertions(+), 179 deletions(-)
drivers/net/ice/base/ice_osdep.h +0 -6
index 878c5597d4..78093adb00 100644
@@ -74,12 +74,6 @@ typedef uint64_t        s64;
 #define min(a, b) RTE_MIN(a, b)
 #define max(a, b) RTE_MAX(a, b)
 
-#ifdef RTE_EXEC_ENV_WINDOWS
-#define ice_access _access
-#else
-#define ice_access access
-#endif
-
 #define FIELD_SIZEOF(t, f) RTE_SIZEOF_FIELD(t, f)
 #define ARRAY_SIZE(arr) RTE_DIM(arr)
 
drivers/net/ice/ice_dcf_parent.c +9 -88

125 lines not on this page · raw message

drivers/net/ice/ice_ethdev.c +77 -84

228 lines not on this page · raw message

drivers/net/ice/ice_ethdev.h +2 -1
index 7f3c26fb6f..edafdf168b 100644
@@ -534,7 +534,8 @@ struct ice_vsi_vlan_pvid_info {
 #define ICE_PF_TO_ETH_DEV(pf) \
 	(((struct ice_pf *)pf)->adapter->eth_dev)
 
-enum ice_pkg_type ice_load_pkg_type(struct ice_hw *hw);
+int
+ice_load_pkg(struct ice_adapter *adapter, bool use_dsn, uint64_t dsn);
 struct ice_vsi *
 ice_setup_vsi(struct ice_pf *pf, enum ice_vsi_type type);
 int
Signature · 1 line
2.23.0
#44 lore raw

PATCH v5 2/2 eal: handle compressed firmware

In reply to David Marchand

Introduce an internal firmware loading helper to remove code duplication in our drivers and handle xz compressed firmware by calling libarchive.

This helper tries to look for .xz suffixes so that drivers are not aware the firmware has been compressed.

libarchive is set as an optional dependency: without libarchive, a runtime warning is emitted so that users know there is a compressed firmware.

Windows implementation is left as an empty stub.

---
Changes since v4:
- moved header to internal,
- s/firmwares/firmware/g,

Changes since v3:
- added release note update,
- updated warning message when uncompressing is unavailable,

Changes since v2:
- added a comment on libarchive link dependency,

Changes since v1:
- used pkg-config for libarchive detection,
- updated doxygen annotations,
- added internal helpers in eal_firmware.c to enhance readability,
- dropped whitespace damage in version.map,

---
 .github/workflows/build.yml            |   1 +
 .travis.yml                            |   1 +
 config/meson.build                     |  10 ++
 doc/guides/rel_notes/release_21_08.rst |   6 +
 drivers/net/bnx2x/bnx2x.c              |  36 +++---
 drivers/net/ice/ice_ethdev.c           |  61 +++-------
 drivers/net/nfp/nfp_net.c              |  58 +++-------
 drivers/net/qede/qede_main.c           |  46 ++++----
 lib/eal/common/eal_firmware.h          |  32 ++++++
 lib/eal/unix/eal_firmware.c            | 150 +++++++++++++++++++++++++
 lib/eal/unix/meson.build               |   1 +
 lib/eal/version.map                    |   1 +
 lib/eal/windows/eal.c                  |   9 ++
 13 files changed, 270 insertions(+), 142 deletions(-)
 create mode 100644 lib/eal/common/eal_firmware.h
 create mode 100644 lib/eal/unix/eal_firmware.c
.github/workflows/build.yml +1 -0
index 7c4d6dcdbf..7dac20ddeb 100644
@@ -93,6 +93,7 @@ jobs:
       run: sudo apt install -y ccache libnuma-dev python3-setuptools
         python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
         libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev
     - name: Install libabigail build dependencies if no cache is available
       if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
       run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
.travis.yml +1 -0
index 5b702cc9bb..23067d9e3c 100644
@@ -16,6 +16,7 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, python3-pyelftools, ninja-build]
       - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson-dev]
+      - [libarchive-dev]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
config/meson.build +10 -0
index 017bb2efbb..639aa74e12 100644
@@ -172,6 +172,16 @@ if libexecinfo.found() and cc.has_header('execinfo.h')
     dpdk_extra_ldflags += '-lexecinfo'
 endif
 
+libarchive = dependency('libarchive', required: false, method: 'pkg-config')
+if libarchive.found()
+    dpdk_conf.set('RTE_HAS_LIBARCHIVE', 1)
+    # Push libarchive link dependency at the project level to support
+    # statically linking dpdk apps. Details at:
+    # https://inbox.dpdk.org/dev/20210605004024.660267a1@sovereign/
+    add_project_link_arguments('-larchive', language: 'c')
+    dpdk_extra_ldflags += '-larchive'
+endif
+
 # check for libbsd
 libbsd = dependency('libbsd', required: false, method: 'pkg-config')
 if libbsd.found()
doc/guides/rel_notes/release_21_08.rst +6 -0
index cd02820e68..96bd907e53 100644
@@ -61,6 +61,12 @@ New Features
   representing sub-domains of functionality. Each auxiliary device
   represents a part of its parent functionality.
 
+* **Added XZ compressed firmware support.**
+
+  Using ``rte_firmware_read``, a driver can now handle XZ compressed firmware
+  in a transparent way, with EAL uncompressing using libarchive if this library
+  is available when building DPDK.
+
 * **Updated Intel iavf driver.**
 
   * Added Tx QoS VF queue TC mapping.
drivers/net/bnx2x/bnx2x.c +14 -22
index 654878d9de..7ee805bd0d 100644
@@ -26,9 +26,12 @@
 #include <arpa/inet.h>
 #include <fcntl.h>
 #include <zlib.h>
+
 #include <rte_bitops.h>
 #include <rte_string_fns.h>
 
+#include "eal_firmware.h"
+
 #define BNX2X_PMD_VER_PREFIX "BNX2X PMD"
 #define BNX2X_PMD_VERSION_MAJOR 1
 #define BNX2X_PMD_VERSION_MINOR 1
@@ -9655,44 +9658,33 @@ static void bnx2x_init_rte(struct bnx2x_softc *sc)
 void bnx2x_load_firmware(struct bnx2x_softc *sc)
 {
 	const char *fwname;
-	int f;
-	struct stat st;
+	void *buf;
+	size_t bufsz;
 
 	fwname = sc->devinfo.device_id == CHIP_NUM_57711
 		? FW_NAME_57711 : FW_NAME_57810;
-	f = open(fwname, O_RDONLY);
-	if (f < 0) {
+	if (rte_firmware_read(fwname, &buf, &bufsz) != 0) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't open firmware file");
 		return;
 	}
 
-	if (fstat(f, &st) < 0) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't stat firmware file");
-		close(f);
-		return;
-	}
-
-	sc->firmware = rte_zmalloc("bnx2x_fw", st.st_size, RTE_CACHE_LINE_SIZE);
+	sc->firmware = rte_zmalloc("bnx2x_fw", bufsz, RTE_CACHE_LINE_SIZE);
 	if (!sc->firmware) {
 		PMD_DRV_LOG(NOTICE, sc, "Can't allocate memory for firmware");
-		close(f);
-		return;
+		goto out;
 	}
 
-	if (read(f, sc->firmware, st.st_size) != st.st_size) {
-		PMD_DRV_LOG(NOTICE, sc, "Can't read firmware data");
-		close(f);
-		return;
-	}
-	close(f);
-
-	sc->fw_len = st.st_size;
+	sc->fw_len = bufsz;
 	if (sc->fw_len < FW_HEADER_LEN) {
 		PMD_DRV_LOG(NOTICE, sc,
 			    "Invalid fw size: %" PRIu64, sc->fw_len);
-		return;
+		goto out;
 	}
+
+	memcpy(sc->firmware, buf, sc->fw_len);
 	PMD_DRV_LOG(DEBUG, sc, "fw_len = %" PRIu64, sc->fw_len);
+out:
+	free(buf);
 }
 
 static void
drivers/net/ice/ice_ethdev.c +13 -48

111 lines not on this page · raw message

drivers/net/nfp/nfp_net.c +13 -45

97 lines not on this page · raw message

drivers/net/qede/qede_main.c +19 -27

79 lines not on this page · raw message

lib/eal/common/eal_firmware.h new file +32 -0
new file mode 100644
index 0000000000..96ec98458f
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 Red Hat, Inc.
+ */
+
+#ifndef _EAL_FIRMWARE_H_
+#define _EAL_FIRMWARE_H_
+
+#include <sys/types.h>
+
+#include <rte_compat.h>
+
+/**
+ * Load a firmware in a dynamically allocated buffer, dealing with compressed
+ * files if libarchive is available.
+ *
+ * @param[in] name
+ *      Firmware filename to load.
+ * @param[out] buf
+ *      Buffer allocated by this function. If this function succeeds, the
+ *      caller is responsible for calling free() on this buffer.
+ * @param[out] bufsz
+ *      Size of the data in the buffer.
+ *
+ * @return
+ *      0 if successful.
+ *      Negative otherwise, buf and bufsize contents are invalid.
+ */
+__rte_internal
+int
+rte_firmware_read(const char *name, void **buf, size_t *bufsz);
+
+#endif /* _EAL_FIRMWARE_H_ */
lib/eal/unix/eal_firmware.c new file +150 -0

150 lines not on this page · raw message

lib/eal/unix/meson.build +1 -0
index dc711b4240..e3ecd3e956 100644
@@ -5,5 +5,6 @@ sources += files(
         'eal_file.c',
         'eal_unix_memory.c',
         'eal_unix_timer.c',
+        'eal_firmware.c',
         'rte_thread.c',
 )
lib/eal/version.map +1 -0
index fe5c3dac98..2df65c6903 100644
@@ -428,6 +428,7 @@ EXPERIMENTAL {
 INTERNAL {
 	global:
 
+	rte_firmware_read;
 	rte_mem_lock;
 	rte_mem_map;
 	rte_mem_page_size;
lib/eal/windows/eal.c +9 -0
index 8483f6b02c..5413d4d87f 100644
@@ -22,6 +22,7 @@
 #include <rte_service_component.h>
 #include <rte_vfio.h>
 
+#include "eal_firmware.h"
 #include "eal_hugepages.h"
 #include "eal_trace.h"
 #include "eal_log.h"
@@ -465,3 +466,11 @@ rte_vfio_container_dma_unmap(__rte_unused int container_fd,
 {
 	return -1;
 }
+
+int
+rte_firmware_read(__rte_unused const char *name,
+			__rte_unused void **buf,
+			__rte_unused size_t *bufsz)
+{
+	return -1;
+}
Signature · 1 line
2.23.0
#45 lore raw
In reply to David Marchand
Quoted 32 lines · David Marchand

Fedora 34 only provides compressed firmwares.

Introduce an internal driver helper to handle transparently compression.

I chose libarchive for decompressing as it seems widely available and DPDK had used it in the past.

Windows support only matters for net/ice and firmware loading was skipped in this driver before this series. Since I don't know if/how we want to load firmwares on Windows, I let an empty stub for this OS.

This series has been compile tested on Linux (I'll trust the CI for others OSes).
I only tested basic init with a net/ice device (no DCF test).

So please drivers maintainers, check nothing is broken.

--
David Marchand

Changes since v4:
- unexport header,
- s/firmwares/firmware/ from Bruce,

Changes since v3:
- add release note update,

Changes since v2:
- update comment on libarchive link dependency,

Quoted 6 lines

Changes since v1:
- address comments on patch2,

David Marchand (2):
net/ice: factorize firmware loading
eal: handle compressed firmware

Series applied.

Signature · 1 line
David Marchand