package_manager/ipk: skip checksums for unsigned local feeds
PATCH v2 1/2 package_manager/ipk: skip checksums for unsigned local feeds
From: Etienne Cordonnier <[email protected]>
Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every do_rootfs run.
Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.
For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.
On a test image with 7000+ packages (including a 2.3 GB debug package): write_index time reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.
See https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
("Enable sha256 checksums in opkg indexer", 2019) for the original rationale.
AI-Generated: Claude Sonnet 4.6
--- meta/lib/oe/package_manager/ipk/__init__.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
meta/lib/oe/package_manager/ipk/__init__.py +9 -5
@@ -12,7 +12,7 @@ from oe.package_manager import * from oe.package_manager.common_deb_ipk import OpkgDpkgPM class OpkgIndexer(Indexer): - def write_index(self): + def write_index(self, build_checksums=True): arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", "SDK_PACKAGE_ARCHS", ]
@@ -44,8 +44,9 @@ class OpkgIndexer(Indexer): if not os.path.exists(pkgs_file): open(pkgs_file, "w").close() - index_cmds.add('%s --checksum md5 --checksum sha256 -r %s -p %s -m %s %s' % - (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params)) + checksum_args = '--checksum md5 --checksum sha256 ' if build_checksums else '' + index_cmds.add('%s %s-r %s -p %s -m %s %s' % + (opkg_index_cmd, checksum_args, pkgs_file, pkgs_file, pkgs_dir, opkg_index_cmd_extra_params)) index_sign_files.add(pkgs_file)
@@ -103,8 +104,12 @@ class OpkgPM(OpkgDpkgPM): self.deploy_dir = oe.path.join(self.d.getVar('WORKDIR'), ipk_repo_workdir) self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") self.opkg_cmd = bb.utils.which(os.getenv('PATH'), "opkg") + self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1" + self._build_checksums = self.from_feeds or self.d.getVar('PACKAGE_FEED_SIGN') == '1' self.opkg_args = ['--volatile-cache', '-f', config_file, '-t', self.d.expand('${T}/ipktemp/'), '-o', target_rootfs] self.opkg_args.extend(shlex.split(self.d.getVar("OPKG_ARGS"))) + if not self._build_checksums: + self.opkg_args.append('--force-checksum') if prepare_index: create_packages_dir(self.d, self.deploy_dir, d.getVar("DEPLOY_DIR_IPK"), "package_write_ipk", filterbydependencies)
@@ -116,7 +121,6 @@ class OpkgPM(OpkgDpkgPM): if not os.path.exists(self.d.expand('${T}/saved')): bb.utils.mkdirhier(self.d.expand('${T}/saved')) - self.from_feeds = (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") == "1" if self.from_feeds: self._create_custom_config() else:
@@ -346,7 +350,7 @@ class OpkgPM(OpkgDpkgPM): def write_index(self): self.deploy_dir_lock() - result = self.indexer.write_index() + result = self.indexer.write_index(build_checksums=self._build_checksums) self.deploy_dir_unlock()
Signature · 1 line
2.43.0
PATCH v2 2/2 opkg: enable sha256 checksum verification by default
From: Etienne Cordonnier <[email protected]>
WITH_SHA256=ON adds no external dependency: sha256.c is a self-contained 528-line pure C implementation already present in the source tree.
oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.
Once SHA256 is universally available, --checksum md5 can also be
dropped from OpkgIndexer.write_index() -- MD5 is collision-prone and
should not be relied upon for integrity checking -- completing the
deprecation noted in https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
(2019).
AI-Generated: Claude Sonnet 4.6
--- meta/recipes-devtools/opkg/opkg_0.10.0.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.10.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit cmake pkgconfig ptest target_localstatedir := "${localstatedir}" OPKGLIBDIR ??= "${target_localstatedir}/lib" -PACKAGECONFIG ??= "libsolv ${@bb.utils.filter('DISTRO_FEATURES', 'acl xattr', d)}" +PACKAGECONFIG ??= "libsolv sha256 ${@bb.utils.filter('DISTRO_FEATURES', 'acl xattr', d)}" PACKAGECONFIG[gpg] = "-DWITH_GPGME=ON,-DWITH_GPGME=OFF,\ gnupg gpgme libgpg-error,\
Signature · 1 line
2.43.0
Re: [OE-core][PATCH v2 2/2] opkg: enable sha256 checksum verification by default
Quoted 11 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
WITH_SHA256=ON adds no external dependency:
sha256.cis a self-contained 528-line pure C implementation already present in the source tree.oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.
Quoted 8 lines
Once SHA256 is universally available, --checksum md5 can also be
dropped fromOpkgIndexer.write_index()-- MD5 is collision-prone and
should not be relied upon for integrity checking -- completing the
deprecation noted in https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
(2019).AI-Generated: Claude Sonnet 4.6
Signed-off-by Etienne Cordonnier <[email protected]>
Hi,
Claude likes to include irrelevant information in commit messages, please re-write this one to focus on the relevant bits.
Otherwise this LGTM.
Best regards,
Signature · 1 line
Paul Barker
Re: [OE-core][PATCH v2 2/2] opkg: enable sha256 checksum verification by default
hi Paul,
I've already manually revised the commit message to include the relevant context justifying the increase in size of opkg. What should I remove?
Étienne
Quoted 36 lines · Paul Barker
Quoted 11 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
WITH_SHA256=ON adds no external dependency:
sha256.cis a self-contained 528-line pure C implementation already present in the source tree.oe-core already generates SHA256Sum fields in the Packages index (via --checksum sha256 in OpkgIndexer.write_index), but opkg silently ignores them unless compiled with WITH_SHA256=ON. Enabling sha256 by default makes those checksums actually verified at install time, improving package integrity checking at zero dependency cost.
Quoted 4 lines
Once SHA256 is universally available, --checksum md5 can also be dropped from
OpkgIndexer.write_index()-- MD5 is collision-prone and should not be relied upon for integrity checking -- completing the deprecation noted inhttps://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
Quoted 4 lines
(2019).
AI-Generated: Claude Sonnet 4.6
Signed-off-by Etienne Cordonnier <[email protected]>Hi,
Claude likes to include irrelevant information in commit messages, please re-write this one to focus on the relevant bits.
Otherwise this LGTM.
Best regards,
--
Paul Barker
Quoted 22 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every
do_rootfsrun.Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.
For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.
On a test image with 7000+ packages (including a 2.3 GB debug package):
write_indextime reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.
Quoted 5 lines
See https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
("Enable sha256 checksums in opkg indexer", 2019) for the original rationale.AI-Generated: Claude Sonnet 4.6
Signed-off-by Etienne Cordonnier <[email protected]>
Hi,
The package feed may be made available over a network and/or externally signed after it has been generated, so I don't think we should disable writing checksums into the feed by default.
How about adding a variable to control this? It can be enabled by default in OE-core and you can disable it in a local or distro config if you know the signatures won't be needed in your use case.
Best regards,
Signature · 1 line
Paul Barker
Hi Paul,
my understanding is that my change does not impact the package feed. Checksums are still always calculated for the normal package feed after my change. There are two different code paths:
1. do_package_index (package-index.bb -> generate_index_files() -> OpkgIndexer(d, DEPLOY_DIR_IPK).write_index()): operates on the global DEPLOY_DIR_IPK. This is the feed that can be published, externally signed, or served over a network. This path is not touched by this commit: it still always writes checksums.
This is the write_index() call at meta/lib/oe/package_manager/__init__.py:579
2. do_rootfs (OpkgPM.write_index()): operates on WORKDIR/oe-rootfs-repo, a throwaway local copy of packages assembled exclusively for the current rootfs build. This directory is never published or signed. This is the only path the commit changes.
This is the write_index() call
at meta/lib/oe/package_manager/ipk/rootfs.py:281
Or are you saying that there is a use-case where
e.g. core-image-minimal/1.0/oe-rootfs-repo gets published over the network, rather than build/tmp/deploy/ipk? I was referring to yocto_docs where the example given is publishing packages from tmp/deploy:
$ cd bitbake-builds/build/tmp/deploy/rpm $ python3 -m http.server
Étienne
Quoted 49 lines · Paul Barker
Quoted 5 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
Computing checksums in the opkg Packages index requires reading every .ipk file in full. For a large image with 7000+ packages (including multi-gigabyte debug packages), this adds 150-300s to every
do_rootfsrun.
Quoted 13 lines
Checksums in the Packages index are only meaningful for signed feeds (PACKAGE_FEED_SIGN=1): the GPG signature covers the Packages index which contains the SHA256Sum of each .ipk, forming a chain of trust that prevents tampered packages being swapped on a remote feed.
For unsigned local file:// feeds the packages are installed directly from the build host filesystem where there is no tampering risk. Skip all checksum generation in that case by passing no --checksum flags to opkg-make-index (the tool's default behaviour when given no flags). Pass --force-checksum to opkg so it does not error on the absent checksum fields.
Quoted 5 lines
On a test image with 7000+ packages (including a 2.3 GB debug package):
write_indextime reduced from ~180s to ~22s (8x speedup) when opkg-make-index is configured to produce no checksums by default.See
https://git.openembedded.org/openembedded-core/commit/?id=e462f47489f35902b6972f9837d9adfa542fc796
Quoted 1 line
("Enable sha256 checksums in opkg indexer", 2019) for the original
rationale.
Quoted 3 lines
AI-Generated: Claude Sonnet 4.6
Signed-off-by Etienne Cordonnier <[email protected]>Hi,
The package feed may be made available over a network and/or externally signed after it has been generated, so I don't think we should disable writing checksums into the feed by default.
How about adding a variable to control this? It can be enabled by default in OE-core and you can disable it in a local or distro config if you know the signatures won't be needed in your use case.
Best regards,
--
Paul Barker