rootfs: Add debugfs package db file copy and cleanup
PATCH v2 1/3 rootfs: Add debugfs package db file copy and cleanup
When copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.
This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.
---
Changes in v2:
- New in v2
meta/lib/oe/rootfs.py | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
meta/lib/oe/rootfs.py +14 -6
@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta): def _cleanup(self): pass - def _setup_dbg_rootfs(self, dirs): + def _setup_dbg_rootfs(self, package_paths): gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0' if gen_debugfs != '1': return
@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta): bb.utils.mkdirhier(self.image_rootfs) bb.note(" Copying back package database...") - for dir in dirs: - if not os.path.isdir(self.image_rootfs + '-orig' + dir): - continue - bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir)) - shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True) + for path in package_paths: + bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path)) + if os.path.isdir(self.image_rootfs + '-orig' + path): + shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True) + elif os.path.isfile(self.image_rootfs + '-orig' + path): + shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path) # Copy files located in /usr/lib/debug or /usr/src/debug for dir in ["/usr/lib/debug", "/usr/src/debug"]:
@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta): bb.note(" Install extra debug packages...") self.pm.install(extra_debug_pkgs.split(), True) + bb.note(" Removing package database...") + for path in package_paths: + if os.path.isdir(self.image_rootfs + path): + shutil.rmtree(self.image_rootfs + path) + elif os.path.isfile(self.image_rootfs + path): + os.remove(self.image_rootfs + path) + bb.note(" Rename debug rootfs...") try: shutil.rmtree(self.image_rootfs + '-dbg')
Signature · 1 line
2.39.0
PATCH v2 2/3 rpm: Pick debugfs package db files/dirs explicitly
Rather than copying the entire /etc hierarchy, specify the pieces we actually need.
---
Changes in v2:
- New in v2
meta/lib/oe/package_manager/rpm/rootfs.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/package_manager/rpm/rootfs.py +1 -1
@@ -110,7 +110,7 @@ class PkgRootfs(Rootfs): if self.progress_reporter: self.progress_reporter.next_stage() - self._setup_dbg_rootfs(['/etc', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) + self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf']) execute_pre_post_process(self.d, rpm_post_process_cmds)
Signature · 1 line
2.39.0
PATCH v2 3/3 eudev: Add group sgx to eudev package
In reply to Alex KiernanFix startup warning:
udevd[171]: specified group 'sgx' unknown
This mirrors the change in bab455cd9b1b ("systemd: add group sgx to udev package") for systemd-udev.
---
Changes in v2:
- Rework rpm handling so that the debugfs doesn't include a spurious copy of /etc which caused the gdbserver selftest to fail because of a duplicate /etc/gshadow file
meta/recipes-core/udev/eudev_3.2.12.bb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-core/udev/eudev_3.2.12.bb +4 -1
@@ -18,7 +18,7 @@ SRC_URI[sha256sum] = "ccdd64ec3c381d3c3ed0e99d2e70d1f62988c7763de89ca7bdffafa5ea GITHUB_BASE_URI = "https://github.com/eudev-project/eudev/releases" -inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases +inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases useradd CONFLICT_DISTRO_FEATURES = "systemd"
@@ -85,3 +85,6 @@ pkg_postinst:${PN}-hwdb () { pkg_prerm:${PN}-hwdb () { rm -f $D${sysconfdir}/udev/hwdb.bin } + +USERADD_PACKAGES = "${PN}" +GROUPADD_PARAM:${PN} = "-r sgx"
Signature · 1 line
2.39.0
Quoted 57 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alex Kiernan
Sent: den 20 juli 2023 12:20
To: [email protected]
Cc: Alex Kiernan <[email protected]>
Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanupWhen copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.
This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.
Signed-off-by Alex Kiernan <[email protected]>---
Changes in v2:
- New in v2meta/lib/oe/rootfs.py | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)meta/lib/oe/rootfs.py +7
-6@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta): def _cleanup(self): pass - def _setup_dbg_rootfs(self, dirs): + def _setup_dbg_rootfs(self, package_paths): gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0' if gen_debugfs != '1': return@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta): bb.utils.mkdirhier(self.image_rootfs) bb.note(" Copying back package database...") - for dir in dirs: - if not os.path.isdir(self.image_rootfs + '-orig' + dir): - continue - bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir)) - shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True) + for path in package_paths: + bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path)) + if os.path.isdir(self.image_rootfs + '-orig' + path): + shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True) + elif os.path.isfile(self.image_rootfs + '-orig' + path): + shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path) # Copy files located in /usr/lib/debug or /usr/src/debug for dir in ["/usr/lib/debug", "/usr/src/debug"]:
meta/lib/oe/rootfs.py class Rootfs(object, metaclass=ABCMeta): · line 171
Quoted 9 lines
meta/lib/oe/rootfs.py +6
-0@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta): bb.note(" Install extra debug packages...") self.pm.install(extra_debug_pkgs.split(), True) + bb.note(" Removing package database...") + for path in package_paths: + if os.path.isdir(self.image_rootfs + path): + shutil.rmtree(self.image_rootfs + path) + elif os.path.isfile(self.image_rootfs + path): + os.remove(self.image_rootfs + path)
What's the reason to do it like this rather than calling self.pm.remove_packaging_data()?
I also just noticed that we apparently have a local change that does the above, but where we also do:
# Remove /etc as it may clash with rootfs. Also saves some space. bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)
I do not know if that would be appropriate to do here as well?
Quoted 6 lines
meta/lib/oe/rootfs.py +1
-1+ bb.note(" Rename debug rootfs...") try: shutil.rmtree(self.image_rootfs + '-dbg') --2.39.0
//Peter
Quoted 61 lines · Peter Kjellerstedt
Quoted 61 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alex Kiernan
Sent: den 20 juli 2023 12:20
To: [email protected]
Cc: Alex Kiernan <[email protected]>
Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanupWhen copying the package database files for the debugfs, add individual file copy as well as tree copying. After the debug rootfs has been created, cleanup the package files.
This then allows us to avoid a problem where (for rpm at least) extraneous files in the debug rootfs would cause failures during oe-selftest because some files existed in both regular and debugfs images.
Signed-off-by Alex Kiernan <[email protected]>---
Changes in v2:
- New in v2meta/lib/oe/rootfs.py | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)meta/lib/oe/rootfs.py +7
-6@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta): def _cleanup(self): pass - def _setup_dbg_rootfs(self, dirs): + def _setup_dbg_rootfs(self, package_paths): gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0' if gen_debugfs != '1': return@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta): bb.utils.mkdirhier(self.image_rootfs) bb.note(" Copying back package database...") - for dir in dirs: - if not os.path.isdir(self.image_rootfs + '-orig' + dir): - continue - bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir)) - shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True) + for path in package_paths: + bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path)) + if os.path.isdir(self.image_rootfs + '-orig' + path): + shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True) + elif os.path.isfile(self.image_rootfs + '-orig' + path): + shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path) # Copy files located in /usr/lib/debug or /usr/src/debug for dir in ["/usr/lib/debug", "/usr/src/debug"]:@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta): bb.note(" Install extra debug packages...") self.pm.install(extra_debug_pkgs.split(), True)
Quoted 10 lines
meta/lib/oe/rootfs.py class Rootfs(object, metaclass=ABCMeta): · line 171
Quoted 6 lines
meta/lib/oe/rootfs.py +6
-0+ bb.note(" Removing package database...") + for path in package_paths: + if os.path.isdir(self.image_rootfs + path): + shutil.rmtree(self.image_rootfs + path) + elif os.path.isfile(self.image_rootfs + path): + os.remove(self.image_rootfs + path)What's the reason to do it like this rather than calling
self.pm.remove_packaging_data()?
Just looking now, I suspect only that I didn't notice that it existed! Though looking across the other packaging implementations, it looks like you'd get different things removed than were passed initially. Certainly looks like a refactor would be in order.
Quoted 8 lines
I also just noticed that we apparently have a local change that does the above, but where we also do:
# Remove /etc as it may clash with rootfs. Also saves some space.
bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)I do not know if that would be appropriate to do here as well?
The clash was exactly what I was fixing here (gshadow), but prior to these two changes you got the whole of /etc copied, so a clash was exactly what you were in danger of running into - anything that clashes now, I think wants investigating as to why, rather than just wiping out /etc entirely.
Signature · 1 line
Alex Kiernan