Skip to contentSkip to conversation

debian packaging: set md5sums for conffiles

PATCH In review Rusty Howell 4 participants 9 messages Jul 26, 2022 – Jul 27, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 debian packaging: set md5sums for conffiles

Currently in an image created with debian package management, dpkg is missing the md5sums in /var/lib/dpkg/status for files designated as conffiles. They are just listed as "newconffile", which is an intermediate state when dpkg installs a package with an conf file. Also, when the device first boots the script /usr/sbin/run-postinsts does not cause /var/lib/dpkg/status to update those hashes either.

The problem here is that when using a package feed, all the conf files are listed as "newconffile", which will indicate to dpkg to replace them all on the first update. dpkg will not follow the rules for confold, confnew, confdef.

---
 meta/lib/oe/package_manager/deb/__init__.py | 42 +++++++++++++++++++++
 meta/lib/oe/package_manager/deb/rootfs.py   |  2 +
 2 files changed, 44 insertions(+)
meta/lib/oe/package_manager/deb/__init__.py +42 -0
index 2ee68fefb1..fabb955558 100644
@@ -3,6 +3,7 @@
 #
 
 import re
+import glob
 import subprocess
 from oe.package_manager import *
 
@@ -216,6 +217,47 @@ class DpkgPM(OpkgDpkgPM):
 
         os.rename(status_file + ".tmp", status_file)
 
+    def set_conffile_hashes(self):
+        """
+        This function will populate the md5sums for all the conffiles listed in /var/lib/dpkg/status
+        """
+        status_file = self.target_rootfs + "/var/lib/dpkg/status"
+
+        with open(status_file, "r") as sf:
+            with open(status_file + ".tmp", "w+") as tmp_sf:
+                status = sf.read()
+
+                conffiles = glob.glob(self.target_rootfs + "/var/lib/dpkg/info/*.conffiles")
+                for cf in conffiles:
+                    fname = os.path.basename(cf)
+                    pkg = fname.split('.')[0]
+
+                    bb.note("Populating md5sums for pkg %s" % pkg)
+                    conffiles = "%s/var/lib/dpkg/info/%s.conffiles" % (self.target_rootfs, pkg)
+                    md5sums = "%s/var/lib/dpkg/info/%s.md5sums" % (self.target_rootfs, pkg)
+                    if os.path.exists(conffiles) and os.path.exists(md5sums):
+                        conf_hashes = {}
+                        with open(md5sums, "r") as md5s:
+                            for line in md5s.read().split('\n'):
+                                m = re.match(r"^([a-f0-9]+)\s+(\S+)", line)
+                                if m:
+                                    hash_ = m.group(1)
+                                    fpath = m.group(2)
+                                    if not fpath.startswith('/'):
+                                        fpath = '/' + fpath
+                                    conf_hashes[fpath] = hash_
+
+                        with open(conffiles, "r") as cf:
+                            for fpath in cf.read().split('\n'):
+                                hash_ = conf_hashes.get(fpath, None)
+                                if hash_:
+                                    status = re.sub("%s newconffile" % fpath, "%s %s" % (fpath, hash_), status)
+
+
+                tmp_sf.write(status)
+
+        os.rename(status_file + ".tmp", status_file)
+
     def run_pre_post_installs(self, package_name=None):
         """
         Run the pre/post installs for package "package_name". If package_name is
meta/lib/oe/package_manager/deb/rootfs.py +2 -0
index 8fbaca11d6..06e99158de 100644
@@ -186,6 +186,8 @@ class PkgRootfs(DpkgOpkgRootfs):
 
         execute_pre_post_process(self.d, deb_post_process_cmds)
 
+        self.pm.set_conffile_hashes()
+
         if self.progress_reporter:
             self.progress_reporter.next_stage()
 
Signature · 1 line
2.25.1
#1 lore raw

Hello Rusty,

Quoted 9 lines · On Tue, 26 Jul 2022 15:13:28 -0600 "Rusty Howell" <[email protected]> wrote:

Currently in an image created with debian package management, dpkg is missing the md5sums in /var/lib/dpkg/status for files designated as conffiles. They are just listed as "newconffile", which is an intermediate state when dpkg installs a package with an conf file. Also, when the device first boots the script /usr/sbin/run-postinsts does not cause /var/lib/dpkg/status to update those hashes either.

The problem here is that when using a package feed, all the conf files are listed as "newconffile", which will indicate to dpkg to replace them all on the first update. dpkg will not follow the rules for confold, confnew, confdef.

This patch does not apply on the current master branch of oe-core. Maybe you wrote it on an older commit or on a branch (e.g. kirkstone)?

Also we require you so sign-off the patches you send on the mailing list ('git commit -s' does that automatically).

Best regards.

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

Also, I'm not sure I understand why this needs to be fixed with custom code. Isn't this something that dpkg/apt take care of themselves?

Alex

Quoted 36 lines · Luca Ceresoli via lists.openembedded.org

Hello Rusty,

Quoted 9 lines · On Tue, 26 Jul 2022 15:13:28 -0600 "Rusty Howell" <[email protected]> wrote:

Currently in an image created with debian package management, dpkg is missing the md5sums in /var/lib/dpkg/status for files designated as conffiles. They are just listed as "newconffile", which is an intermediate state when dpkg installs a package with an conf file. Also, when the device first boots the script /usr/sbin/run-postinsts does not cause /var/lib/dpkg/status to update those hashes either.

The problem here is that when using a package feed, all the conf files are listed as "newconffile", which will indicate to dpkg to replace them all on the first update. dpkg will not follow the rules for confold, confnew, confdef.

This patch does not apply on the current master branch of oe-core. Maybe you wrote it on an older commit or on a branch (e.g. kirkstone)?

Also we require you so sign-off the patches you send on the mailing list ('git commit -s' does that automatically).

Best regards.
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168552): https://lists.openembedded.org/g/openembedded-core/message/168552
Mute This Topic: https://lists.openembedded.org/mt/92637328/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
Quoted 5 lines · Alexander Kanavin via lists.openembedded.org

Also, I'm not sure I understand why this needs to be fixed with custom code. Isn't this something that dpkg/apt take care of themselves?

Yes, I suspect this is a problem with how we’re generating the DEBIAN/.

Ross

#4 lore raw

So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.

Ross

#5 lore raw
Quoted 1 line

So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.

dpkg —configure does this, but we’re neutering it by renaming all foo.dpkg-new files to foo in the deb install() function.

If we’re going to rename foo.dpkg-new to foo, then that is where we go and update the status file.

Alternatively, we don’t do that, and every package needs to run its configure step on first boot.

Ross

#6 lore raw

Thanks for the quick responses. It took us a while to figure out why we
were seeing weird behaviour with our products, and we ultimately figured
out the conffiles are being replaced during the update. I am working off
of Hardknott currently, so that's my mistake not switching to master. And I guess I misunderstood who is allowed to "sign off" on patches. I assumed I couldn't since I'm not a regular. But in any case, it looks like Ross Burton may be on to a better solution than what I have to fix the issue.

Rusty

Quoted 27 lines · Ross Burton
Quoted 1 line

So we’re definitely not generating the DEBIAN/md5sums files that

dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.

dpkg —configure does this, but we’re neutering it by renaming all foo.dpkg-new files to foo in the deb install() function.

If we’re going to rename foo.dpkg-new to foo, then that is where we go and update the status file.

Alternatively, we don’t do that, and every package needs to run its configure step on first boot.

Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#168563):
https://lists.openembedded.org/g/openembedded-core/message/168563
Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#7 lore raw

I think Ross is merely helping you to reach the correct solution, but everyone would appreciate if you write and submit the actual fix :-)

Alex

Quoted 24 lines · Rusty Howell

Thanks for the quick responses. It took us a while to figure out why we were seeing weird behaviour with our products, and we ultimately figured out the conffiles are being replaced during the update. I am working off of Hardknott currently, so that's my mistake not switching to master. And I guess I misunderstood who is allowed to "sign off" on patches. I assumed I couldn't since I'm not a regular. But in any case, it looks like Ross Burton may be on to a better solution than what I have to fix the issue.

Rusty

Quoted 18 lines · Ross Burton
Quoted 2 lines

So we’re definitely not generating the DEBIAN/md5sums files that dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.

dpkg —configure does this, but we’re neutering it by renaming all foo.dpkg-new files to foo in the deb install() function.

If we’re going to rename foo.dpkg-new to foo, then that is where we go and update the status file.

Alternatively, we don’t do that, and every package needs to run its configure step on first boot.

Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#168563): https://lists.openembedded.org/g/openembedded-core/message/168563
Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#8 lore raw

Will do.

Quoted 46 lines · Alexander Kanavin

I think Ross is merely helping you to reach the correct solution, but everyone would appreciate if you write and submit the actual fix :-)

Alex

Quoted 2 lines · Rusty Howell

Thanks for the quick responses. It took us a while to figure out why

we were seeing weird behaviour with our products, and we ultimately figured
out the conffiles are being replaced during the update. I am working off
of Hardknott currently, so that's my mistake not switching to master. And I guess I misunderstood who is allowed to "sign off" on patches. I assumed I couldn't since I'm not a regular. But in any case, it looks like Ross Burton may be on to a better solution than what I have to fix the issue.

Quoted 6 lines

Rusty

Quoted 2 lines · Ross Burton
Quoted 2 lines

So we’re definitely not generating the DEBIAN/md5sums files that

dpkg-deb likes to have. However, this file in proper Debian packages doesn’t contain hashes for the conffiles, so I can only imagine that dpkg generates them on install, which leads to the question of where that is meant to happen.

Quoted 2 lines
Quoted 2 lines

dpkg —configure does this, but we’re neutering it by renaming all

foo.dpkg-new files to foo in the deb install() function.

Quoted 2 lines
Quoted 2 lines

If we’re going to rename foo.dpkg-new to foo, then that is where we go

and update the status file.

Quoted 2 lines
Quoted 2 lines

Alternatively, we don’t do that, and every package needs to run its

configure step on first boot.

Quoted 5 lines
Quoted 5 lines

Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#168563):

https://lists.openembedded.org/g/openembedded-core/message/168563

Quoted 3 lines
Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/92637328/4800265
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [

[email protected]]

Quoted 2 lines
Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-