Skip to contentSkip to conversation

package_manager/ipk: skip checksums for unsigned local feeds

PATCH In review [email protected] 2 participants 6 messages Jul 30

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 package_manager/ipk: skip checksums for unsigned local feeds

From: Etienne Cordonnier <[email protected]>

Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every do_rootfs run.

Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.

For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.

On a test image with 7000+ packages (including a 2.3 GB debug package): write_index time reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.

See https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
("Enable sha256 checksums in opkg indexer", 2019) for the original rationale.

AI-Generated: Claude Sonnet 4.6

---
 meta/lib/oe/package_manager/ipk/__init__.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
meta/lib/oe/package_manager/ipk/__init__.py +9 -5
index 344e0852177..93a12fad6a3 100644
@@ -12,7 +12,7 @@ from oe.package_manager import *
 from oe.package_manager.common_deb_ipk import OpkgDpkgPM
 
 class OpkgIndexer(Indexer):
-    def write_index(self):
+    def write_index(self, build_checksums=True):
         arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS",
                      "SDK_PACKAGE_ARCHS",
                      ]
@@ -44,8 +44,9 @@ class OpkgIndexer(Indexer):
                 if not os.path.exists(pkgs_file):
                     open(pkgs_file, "w").close()
 
-                index_cmds.add('%s --checksum md5 --checksum sha256 -r %s -p %s -m %s %s' %
-                                  (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params))
+                checksum_args = '--checksum md5 --checksum sha256 ' if build_checksums else ''
+                index_cmds.add('%s %s-r %s -p %s -m %s %s' %
+                                  (opkg_index_cmd, checksum_args, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params))
 
                 index_sign_files.add(pkgs_file)
 
@@ -103,8 +104,12 @@ class OpkgPM(OpkgDpkgPM):
         self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), ipk_repo_workdir)
         self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
         self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg")
+        self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1"
+        self._build_checksums = self.from_feeds or self.d.getVar('PACKAGE_FEED_SIGN') == '1'
         self.opkg_args = ['--volatile-cache', '-f', config_file, '-t', self.d.expand('${T}/ipktemp/'), '-o', target_rootfs]
         self.opkg_args.extend(shlex.split(self.d.getVar("OPKG_ARGS")))
+        if not self._build_checksums:
+            self.opkg_args.append('--force-checksum')
 
         if prepare_index:
             create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
@@ -116,7 +121,6 @@ class OpkgPM(OpkgDpkgPM):
         if not os.path.exists(self.d.expand('${T}/saved')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved'))
 
-        self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1"
         if self.from_feeds:
             self._create_custom_config()
         else:
@@ -346,7 +350,7 @@ class OpkgPM(OpkgDpkgPM):
     def write_index(self):
         self.deploy_dir_lock()
 
-        result = self.indexer.write_index()
+        result = self.indexer.write_index(build_checksums=self._build_checksums)
 
         self.deploy_dir_unlock()
 
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 2/2 opkg: enable sha256 checksum verification by default

From: Etienne Cordonnier <[email protected]>

WITH_SHA256=ON adds no external dependency: sha256.c is a self-contained 528-line pure C implementation already present in the source tree.

oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.

Once SHA256 is universally available, --checksum md5 can also be
dropped from OpkgIndexer.write_index() -- MD5 is collision-prone and
should not be relied upon for integrity checking -- completing the
deprecation noted in https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
(2019).

AI-Generated: Claude Sonnet 4.6

---
 meta/recipes-devtools/opkg/opkg_0.10.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.10.0.bb +1 -1
index debcd94a300..e3c2dd790dd 100644
@@ -30,7 +30,7 @@ inherit cmake pkgconfig ptest
 target_localstatedir := "${localstatedir}"
 OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv ${@bb.utils.filter('DISTRO_FEATURES', 'acl xattr', d)}"
+PACKAGECONFIG ??= "libsolv sha256 ${@bb.utils.filter('DISTRO_FEATURES', 'acl xattr', d)}"
 
 PACKAGECONFIG[gpg] = "-DWITH_GPGME=ON,-DWITH_GPGME=OFF,\
     gnupg gpgme libgpg-error,\
Signature · 1 line
2.43.0
#2 lore raw

Re: [OE-core][PATCH v2 2/2] opkg: enable sha256 checksum verification by default

Quoted 11 lines · Etienne Cordonnier via lists.openembedded.org

From: Etienne Cordonnier <[email protected]>

WITH_SHA256=ON adds no external dependency: sha256.c is a self-contained 528-line pure C implementation already present in the source tree.

oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.

Quoted 8 lines

Once SHA256 is universally available, --checksum md5 can also be
dropped from OpkgIndexer.write_index() -- MD5 is collision-prone and
should not be relied upon for integrity checking -- completing the
deprecation noted in https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
(2019).

AI-Generated: Claude Sonnet 4.6

Hi,

Claude likes to include irrelevant information in commit messages, please re-write this one to focus on the relevant bits.

Otherwise this LGTM.

Best regards,

Signature · 1 line
Paul Barker
#3 lore raw

Re: [OE-core][PATCH v2 2/2] opkg: enable sha256 checksum verification by default

hi Paul,
I've already manually revised the commit message to include the relevant context justifying the increase in size of opkg. What should I remove?

Étienne

Quoted 36 lines · Paul Barker
Quoted 11 lines · Etienne Cordonnier via lists.openembedded.org

From: Etienne Cordonnier <[email protected]>

WITH_SHA256=ON adds no external dependency: sha256.c is a self-contained 528-line pure C implementation already present in the source tree.

oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.

Quoted 4 lines

Once SHA256 is universally available, --checksum md5 can also be dropped from OpkgIndexer.write_index() -- MD5 is collision-prone and should not be relied upon for integrity checking -- completing the deprecation noted in

https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796

Quoted 4 lines

(2019).

AI-Generated: Claude Sonnet 4.6

Hi,

Claude likes to include irrelevant information in commit messages, please re-write this one to focus on the relevant bits.

Otherwise this LGTM.

Best regards,

--
Paul Barker

#4 lore raw
In reply to [email protected]
Quoted 22 lines · Etienne Cordonnier via lists.openembedded.org

From: Etienne Cordonnier <[email protected]>

Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every do_rootfs run.

Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.

For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.

On a test image with 7000+ packages (including a 2.3 GB debug package): write_index time reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.

Quoted 5 lines

See https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
("Enable sha256 checksums in opkg indexer", 2019) for the original rationale.

AI-Generated: Claude Sonnet 4.6

Hi,

The package feed may be made available over a network and/or externally signed after it has been generated, so I don't think we should disable writing checksums into the feed by default.

How about adding a variable to control this? It can be enabled by default in OE-core and you can disable it in a local or distro config if you know the signatures won't be needed in your use case.

Best regards,

Signature · 1 line
Paul Barker
#5 lore raw

Hi Paul,
my understanding is that my change does not impact the package feed. Checksums are still always calculated for the normal package feed after my change. There are two different code paths:

1. do_package_index (package-index.bb -> generate_index_files() -> OpkgIndexer(d, DEPLOY_DIR_IPK).write_index()): operates on the global DEPLOY_DIR_IPK. This is the feed that can be published, externally signed, or served over a network. This path is not touched by this commit: it still always writes checksums.
This is the write_index() call at meta/lib/oe/package_manager/__init__.py:579

2. do_rootfs (OpkgPM.write_index()): operates on WORKDIR/oe-rootfs-repo, a throwaway local copy of packages assembled exclusively for the current rootfs build. This directory is never published or signed. This is the only path the commit changes.
This is the write_index() call
at meta/lib/oe/package_manager/ipk/rootfs.py:281

Or are you saying that there is a use-case where
e.g. core-image-minimal/1.0/oe-rootfs-repo gets published over the network, rather than build/tmp/deploy/ipk? I was referring to yocto_docs where the example given is publishing packages from tmp/deploy:

$ cd bitbake-builds/build/tmp/deploy/rpm
$ python3 -m http.server

Étienne

Quoted 49 lines · Paul Barker
Quoted 5 lines · Etienne Cordonnier via lists.openembedded.org

From: Etienne Cordonnier <[email protected]>

Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every do_rootfs

run.

Quoted 13 lines

Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.

For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.

Quoted 5 lines

On a test image with 7000+ packages (including a 2.3 GB debug package): write_index time reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.

See

https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796

Quoted 1 line

("Enable sha256 checksums in opkg indexer", 2019) for the original

rationale.

Quoted 3 lines

AI-Generated: Claude Sonnet 4.6

Hi,

The package feed may be made available over a network and/or externally signed after it has been generated, so I don't think we should disable writing checksums into the feed by default.

How about adding a variable to control this? It can be enabled by default in OE-core and you can disable it in a local or distro config if you know the signatures won't be needed in your use case.

Best regards,

--
Paul Barker