Skip to contentSkip to conversation

rootfs: Add debugfs package db file copy and cleanup

PATCH In review Alex Kiernan 2 participants 5 messages Jul 20, 2023 – Aug 8, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/3 rootfs: Add debugfs package db file copy and cleanup

When copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.

This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.

---

Changes in v2:
- New in v2

 meta/lib/oe/rootfs.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
meta/lib/oe/rootfs.py +14 -6
index 890ba5f03984..1a48ed10b3f6 100644
@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
     def _cleanup(self):
         pass
 
-    def _setup_dbg_rootfs(self, dirs):
+    def _setup_dbg_rootfs(self, package_paths):
         gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
         if gen_debugfs != '1':
            return
@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
         bb.utils.mkdirhier(self.image_rootfs)
 
         bb.note("  Copying back package database...")
-        for dir in dirs:
-            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
-                continue
-            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
-            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
+        for path in package_paths:
+            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
+            if os.path.isdir(self.image_rootfs + '-orig' + path):
+                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
+            elif os.path.isfile(self.image_rootfs + '-orig' + path):
+                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
 
         # Copy files located in /usr/lib/debug or /usr/src/debug
         for dir in ["/usr/lib/debug", "/usr/src/debug"]:
@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
             bb.note("  Install extra debug packages...")
             self.pm.install(extra_debug_pkgs.split(), True)
 
+        bb.note("  Removing package database...")
+        for path in package_paths:
+            if os.path.isdir(self.image_rootfs + path):
+                shutil.rmtree(self.image_rootfs + path)
+            elif os.path.isfile(self.image_rootfs + path):
+                os.remove(self.image_rootfs + path)
+
         bb.note("  Rename debug rootfs...")
         try:
             shutil.rmtree(self.image_rootfs + '-dbg')
Signature · 1 line
2.39.0
#1 lore raw

PATCH v2 2/3 rpm: Pick debugfs package db files/dirs explicitly

Rather than copying the entire /etc hierarchy, specify the pieces we actually need.

---

Changes in v2:
- New in v2

 meta/lib/oe/package_manager/rpm/rootfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/package_manager/rpm/rootfs.py +1 -1
index d4c415f68cad..3ba539632063 100644
@@ -110,7 +110,7 @@ class PkgRootfs(Rootfs):
         if self.progress_reporter:
             self.progress_reporter.next_stage()
 
-        self._setup_dbg_rootfs(['/etc', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
+        self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
 
         execute_pre_post_process(self.d, rpm_post_process_cmds)
 
Signature · 1 line
2.39.0
#2 lore raw

PATCH v2 3/3 eudev: Add group sgx to eudev package

In reply to Alex Kiernan

Fix startup warning:

udevd[171]: specified group 'sgx' unknown

This mirrors the change in bab455cd9b1b ("systemd: add group sgx to udev package") for systemd-udev.

---

Changes in v2:
- Rework rpm handling so that the debugfs doesn't include a spurious copy of /etc which caused the gdbserver selftest to fail because of a duplicate /etc/gshadow file

 meta/recipes-core/udev/eudev_3.2.12.bb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-core/udev/eudev_3.2.12.bb +4 -1
index 572ccecafd0c..4268bcc2c5de 100644
@@ -18,7 +18,7 @@ SRC_URI[sha256sum] = "ccdd64ec3c381d3c3ed0e99d2e70d1f62988c7763de89ca7bdffafa5ea
 
 GITHUB_BASE_URI = "https://github.com/eudev-project/eudev/releases"
 
-inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases
+inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases useradd
 
 CONFLICT_DISTRO_FEATURES = "systemd"
 
@@ -85,3 +85,6 @@ pkg_postinst:${PN}-hwdb () {
 pkg_prerm:${PN}-hwdb () {
 	rm -f $D${sysconfdir}/udev/hwdb.bin
 }
+
+USERADD_PACKAGES = "${PN}"
+GROUPADD_PARAM:${PN} = "-r sgx"
Signature · 1 line
2.39.0
#3 lore raw
In reply to Alex Kiernan
Quoted 57 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alex Kiernan
Sent: den 20 juli 2023 12:20
To: [email protected]
Cc: Alex Kiernan <[email protected]>
Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

When copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.

This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.

---

Changes in v2:
- New in v2

 meta/lib/oe/rootfs.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
meta/lib/oe/rootfs.py +7 -6
index 890ba5f03984..1a48ed10b3f6 100644
@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
     def _cleanup(self):
         pass
 
-    def _setup_dbg_rootfs(self, dirs):
+    def _setup_dbg_rootfs(self, package_paths):
         gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
         if gen_debugfs != '1':
            return
@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
         bb.utils.mkdirhier(self.image_rootfs)
 
         bb.note("  Copying back package database...")
-        for dir in dirs:
-            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
-                continue
-            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
-            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
+        for path in package_paths:
+            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
+            if os.path.isdir(self.image_rootfs + '-orig' + path):
+                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
+            elif os.path.isfile(self.image_rootfs + '-orig' + path):
+                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
 
         # Copy files located in /usr/lib/debug or /usr/src/debug
         for dir in ["/usr/lib/debug", "/usr/src/debug"]:

meta/lib/oe/rootfs.py class Rootfs(object, metaclass=ABCMeta): · line 171

Quoted 9 lines
meta/lib/oe/rootfs.py +6 -0
@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
             bb.note("  Install extra debug packages...")
             self.pm.install(extra_debug_pkgs.split(), True)
 
+        bb.note("  Removing package database...")
+        for path in package_paths:
+            if os.path.isdir(self.image_rootfs + path):
+                shutil.rmtree(self.image_rootfs + path)
+            elif os.path.isfile(self.image_rootfs + path):
+                os.remove(self.image_rootfs + path)

What's the reason to do it like this rather than calling self.pm.remove_packaging_data()?

I also just noticed that we apparently have a local change that does the above, but where we also do:

# Remove /etc as it may clash with rootfs. Also saves some space. bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)

I do not know if that would be appropriate to do here as well?

Quoted 6 lines
meta/lib/oe/rootfs.py +1 -1
+
         bb.note("  Rename debug rootfs...")
         try:
             shutil.rmtree(self.image_rootfs + '-dbg')
--

2.39.0

//Peter

#4 lore raw
Quoted 61 lines · Peter Kjellerstedt
Quoted 61 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alex Kiernan
Sent: den 20 juli 2023 12:20
To: [email protected]
Cc: Alex Kiernan <[email protected]>
Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

When copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.

This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.

---

Changes in v2:
- New in v2

 meta/lib/oe/rootfs.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
meta/lib/oe/rootfs.py +7 -6
index 890ba5f03984..1a48ed10b3f6 100644
@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
     def _cleanup(self):
         pass
 
-    def _setup_dbg_rootfs(self, dirs):
+    def _setup_dbg_rootfs(self, package_paths):
         gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
         if gen_debugfs != '1':
            return
@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
         bb.utils.mkdirhier(self.image_rootfs)
 
         bb.note("  Copying back package database...")
-        for dir in dirs:
-            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
-                continue
-            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
-            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
+        for path in package_paths:
+            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
+            if os.path.isdir(self.image_rootfs + '-orig' + path):
+                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
+            elif os.path.isfile(self.image_rootfs + '-orig' + path):
+                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
 
         # Copy files located in /usr/lib/debug or /usr/src/debug
         for dir in ["/usr/lib/debug", "/usr/src/debug"]:
@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
             bb.note("  Install extra debug packages...")
             self.pm.install(extra_debug_pkgs.split(), True)
 
Quoted 10 lines

meta/lib/oe/rootfs.py class Rootfs(object, metaclass=ABCMeta): · line 171

Quoted 6 lines
meta/lib/oe/rootfs.py +6 -0
+        bb.note("  Removing package database...")
+        for path in package_paths:
+            if os.path.isdir(self.image_rootfs + path):
+                shutil.rmtree(self.image_rootfs + path)
+            elif os.path.isfile(self.image_rootfs + path):
+                os.remove(self.image_rootfs + path)

What's the reason to do it like this rather than calling self.pm.remove_packaging_data()?

Just looking now, I suspect only that I didn't notice that it existed! Though looking across the other packaging implementations, it looks like you'd get different things removed than were passed initially. Certainly looks like a refactor would be in order.

Quoted 8 lines

I also just noticed that we apparently have a local change that does the above, but where we also do:

# Remove /etc as it may clash with rootfs. Also saves some space. bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)

I do not know if that would be appropriate to do here as well?

The clash was exactly what I was fixing here (gshadow), but prior to these two changes you got the whole of /etc copied, so a clash was exactly what you were in danger of running into - anything that clashes now, I think wants investigating as to why, rather than just wiping out /etc entirely.

Signature · 1 line
Alex Kiernan