Skip to contentSkip to conversation

package/dosfstools: introduce sanitation step during install

PATCH New Markus Mayer 3 participants 5 messages May 29, 2019 – May 30, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/dosfstools: introduce sanitation step during install

We can't install dosfstools into the target directory directly, because it'll install *all* binaries, even the disabled ones.

Also, we can't just delete dosfstools binaries from the target directory after installing them, because some tools of the same name may be provided by other packages (specifically Busybox). When a tool is disabled in dosfstools, it only means that dosfstools shouldn't install its own copy, not that it is okay to remove a tool that might already be there.

To avoid any issues, we install dosfstools into a temporary location, remove the binaries that have been disabled in the Buildroot configuration, and only then copy the rest to the target location.

We also extend sanitation to include man pages, so only man pages for enabled tools are copied to the target.

---

We discovered the issue at hand because we are using Busybox for most of our tools, including FAT related tools. When we needed to add fsck.fat to our root file system, we had to enable DOSFSTOOLS, because Busybox doesn't provide it. And suddenly all our FAT related tools (except fsck.fat) had disappeared from the rootfs! That was not at all the idea.

The reason, as it turned out, was DOSFSTOOLS doing it sanitation directly inside the target directory, which indiscriminately removed binaries, even if they were enabled via a different package (BUSYBOX in our case).

Hence the approach using a temporary install location.

 package/dosfstools/dosfstools.mk | 29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)
package/dosfstools/dosfstools.mk +26 -3
index 6eb0851d0e23..30a35e3f9b8f 100644
@@ -10,6 +10,7 @@ DOSFSTOOLS_SITE = https://github.com/dosfstools/dosfstools/releases/download/v$(
 DOSFSTOOLS_LICENSE = GPL-3.0+
 DOSFSTOOLS_LICENSE_FILES = COPYING
 DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks --exec-prefix=/
+DOSFSTOOLS_TEMP=$(BASE_DIR)/build/dosfstools-install
 HOST_DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks
 
 ifeq ($(BR2_PACKAGE_HAS_UDEV),y)
@@ -24,26 +25,48 @@ DOSFSTOOLS_CONF_OPTS += LIBS="-liconv"
 DOSFSTOOLS_DEPENDENCIES += libiconv
 endif
 
+# Install into temporary location
+DOSFSTOOLS_INSTALL_TARGET_OPTS = \
+	DESTDIR=$(DOSFSTOOLS_TEMP) \
+	install
+
+# Clean out our temporary install location before installation
+define DOSFSTOOLS_CLEAR_TEMP
+	rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_PRE_INSTALL_TARGET_HOOKS += DOSFSTOOLS_CLEAR_TEMP
+
+# Sanitize temporary install location after installation
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FATLABEL),)
 define DOSFSTOOLS_REMOVE_FATLABEL
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,dosfslabel fatlabel)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,dosfslabel fatlabel)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,dosfslabel.8 fatlabel.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FATLABEL
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FSCK_FAT),)
 define DOSFSTOOLS_REMOVE_FSCK_FAT
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,fsck.fat.8 dosfsck.8 fsck.msdos.8 fsck.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FSCK_FAT
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_MKFS_FAT),)
 define DOSFSTOOLS_REMOVE_MKFS_FAT
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,mkfs.fat.8 mkdosfs.8 mkfs.msdos.8 mkfs.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_MKFS_FAT
 endif
 
+# Install what's left into the actual target directory
+define DOSFSTOOLS_TARGET_INSTALL
+	tar -C $(DOSFSTOOLS_TEMP) -c -f - . | tar -C $(TARGET_DIR) -x -f -
+	rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_TARGET_INSTALL
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 package/dosfstools: introduce sanitation step during install

Hi Markus,

Quoted 10 lines · Markus Mayer

We can't install dosfstools into the target directory directly, because it'll install *all* binaries, even the disabled ones.

Also, we can't just delete dosfstools binaries from the target directory after installing them, because some tools of the same name may be provided by other packages (specifically Busybox). When a tool is disabled in dosfstools, it only means that dosfstools shouldn't install its own copy, not that it is okay to remove a tool that might already be there.

Quoted 3 lines

To avoid any issues, we install dosfstools into a temporary location, remove the binaries that have been disabled in the Buildroot configuration, and only then copy the rest to the target location.

Not a bad idea.

Quoted 2 lines

We also extend sanitation to include man pages, so only man pages for enabled tools are copied to the target.

Since man pages anyway get removed in target-finalize, there isn't much point
to doing this.

Quoted 69 lines

---

We discovered the issue at hand because we are using Busybox for most of our tools, including FAT related tools. When we needed to add fsck.fat to our root file system, we had to enable DOSFSTOOLS, because Busybox doesn't provide it. And suddenly all our FAT related tools (except fsck.fat) had disappeared from the rootfs! That was not at all the idea.

The reason, as it turned out, was DOSFSTOOLS doing it sanitation directly inside the target directory, which indiscriminately removed binaries, even if they were enabled via a different package (BUSYBOX in our case).

Hence the approach using a temporary install location.

 package/dosfstools/dosfstools.mk | 29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)
package/dosfstools/dosfstools.mk +17 -2
index 6eb0851d0e23..30a35e3f9b8f 100644
@@ -10,6 +10,7 @@ DOSFSTOOLS_SITE = https://github.com/dosfstools/dosfstools/releases/download/v$(
 DOSFSTOOLS_LICENSE = GPL-3.0+
 DOSFSTOOLS_LICENSE_FILES = COPYING
 DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks --exec-prefix=/
+DOSFSTOOLS_TEMP=$(BASE_DIR)/build/dosfstools-install
 HOST_DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks
 
 ifeq ($(BR2_PACKAGE_HAS_UDEV),y)
@@ -24,26 +25,48 @@ DOSFSTOOLS_CONF_OPTS += LIBS="-liconv"
 DOSFSTOOLS_DEPENDENCIES += libiconv
 endif
 
+# Install into temporary location
+DOSFSTOOLS_INSTALL_TARGET_OPTS = \
+	DESTDIR=$(DOSFSTOOLS_TEMP) \
+	install
+
+# Clean out our temporary install location before installation
+define DOSFSTOOLS_CLEAR_TEMP
+	rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_PRE_INSTALL_TARGET_HOOKS += DOSFSTOOLS_CLEAR_TEMP
+
+# Sanitize temporary install location after installation
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FATLABEL),)
 define DOSFSTOOLS_REMOVE_FATLABEL
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,dosfslabel fatlabel)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,dosfslabel fatlabel)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,dosfslabel.8 fatlabel.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FATLABEL
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FSCK_FAT),)
 define DOSFSTOOLS_REMOVE_FSCK_FAT
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,fsck.fat.8 dosfsck.8 fsck.msdos.8 fsck.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FSCK_FAT
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_MKFS_FAT),)
 define DOSFSTOOLS_REMOVE_MKFS_FAT

package/dosfstools/dosfstools.mk DOSFSTOOLS_CONF_OPTS += LIBS="-liconv" · line 66

Quoted 10 lines
package/dosfstools/dosfstools.mk +5 -1
-	rm -f $(addprefix $(TARGET_DIR)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+	rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,mkfs.fat.8 mkdosfs.8 mkfs.msdos.8 mkfs.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_MKFS_FAT
 endif
 
+# Install what's left into the actual target directory
+define DOSFSTOOLS_TARGET_INSTALL
+	tar -C $(DOSFSTOOLS_TEMP) -c -f - . | tar -C $(TARGET_DIR) -x -f -

We generally use rsync to copy files, not the tar trick.

However, I'm wondering... dosfstools only installs things in /sbin,
/usr/share/man and /usr/share/doc. The latter two get deleted in target-finalize. Wouldn't it be a lot easier then to manually install the programs we need to sbin, instead of installing everything and removing what is not needed?

Regards,
Arnout

Quoted 7 lines
package/dosfstools/dosfstools.mk +4 -0
+	rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_TARGET_INSTALL
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
#2 lore raw

PATCH 1/1 package/dosfstools: introduce sanitation step during install

Quoted 18 lines · Arnout Vandecappelle

Hi Markus,

Quoted 10 lines · Markus Mayer

We can't install dosfstools into the target directory directly, because it'll install *all* binaries, even the disabled ones.

Also, we can't just delete dosfstools binaries from the target directory after installing them, because some tools of the same name may be provided by other packages (specifically Busybox). When a tool is disabled in dosfstools, it only means that dosfstools shouldn't install its own copy, not that it is okay to remove a tool that might already be there.

Quoted 3 lines

To avoid any issues, we install dosfstools into a temporary location, remove the binaries that have been disabled in the Buildroot configuration, and only then copy the rest to the target location.

Quoted 8 lines

Not a bad idea.

Quoted 2 lines

We also extend sanitation to include man pages, so only man pages for enabled tools are copied to the target.

Since man pages anyway get removed in target-finalize, there isn't much point
to doing this.

No problem. I'll remove that part.

Quoted 75 lines
Quoted 75 lines

---

We discovered the issue at hand because we are using Busybox for most of our tools, including FAT related tools. When we needed to add fsck.fat to our root file system, we had to enable DOSFSTOOLS, because Busybox doesn't provide it. And suddenly all our FAT related tools (except fsck.fat) had disappeared from the rootfs! That was not at all the idea.

The reason, as it turned out, was DOSFSTOOLS doing it sanitation directly inside the target directory, which indiscriminately removed binaries, even if they were enabled via a different package (BUSYBOX in our case).

Hence the approach using a temporary install location.

 package/dosfstools/dosfstools.mk | 29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)
package/dosfstools/dosfstools.mk +19 -3
index 6eb0851d0e23..30a35e3f9b8f 100644
@@ -10,6 +10,7 @@ DOSFSTOOLS_SITE = https://github.com/dosfstools/dosfstools/releases/download/v$(
 DOSFSTOOLS_LICENSE = GPL-3.0+
 DOSFSTOOLS_LICENSE_FILES = COPYING
 DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks --exec-prefix=/
+DOSFSTOOLS_TEMP=$(BASE_DIR)/build/dosfstools-install
 HOST_DOSFSTOOLS_CONF_OPTS = --enable-compat-symlinks
 
 ifeq ($(BR2_PACKAGE_HAS_UDEV),y)
@@ -24,26 +25,48 @@ DOSFSTOOLS_CONF_OPTS += LIBS="-liconv"
 DOSFSTOOLS_DEPENDENCIES += libiconv
 endif
 
+# Install into temporary location
+DOSFSTOOLS_INSTALL_TARGET_OPTS = \
+     DESTDIR=$(DOSFSTOOLS_TEMP) \
+     install
+
+# Clean out our temporary install location before installation
+define DOSFSTOOLS_CLEAR_TEMP
+     rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_PRE_INSTALL_TARGET_HOOKS += DOSFSTOOLS_CLEAR_TEMP
+
+# Sanitize temporary install location after installation
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FATLABEL),)
 define DOSFSTOOLS_REMOVE_FATLABEL
-     rm -f $(addprefix $(TARGET_DIR)/sbin/,dosfslabel fatlabel)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,dosfslabel fatlabel)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,dosfslabel.8 fatlabel.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FATLABEL
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_FSCK_FAT),)
 define DOSFSTOOLS_REMOVE_FSCK_FAT
-     rm -f $(addprefix $(TARGET_DIR)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,fsck.fat dosfsck fsck.msdos fsck.vfat)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,fsck.fat.8 dosfsck.8 fsck.msdos.8 fsck.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_FSCK_FAT
 endif
 
 ifeq ($(BR2_PACKAGE_DOSFSTOOLS_MKFS_FAT),)
 define DOSFSTOOLS_REMOVE_MKFS_FAT
-     rm -f $(addprefix $(TARGET_DIR)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/sbin/,mkfs.fat mkdosfs mkfs.msdos mkfs.vfat)
+     rm -f $(addprefix $(DOSFSTOOLS_TEMP)/usr/share/man/man8/,mkfs.fat.8 mkdosfs.8 mkfs.msdos.8 mkfs.vfat.8)
 endef
 DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_REMOVE_MKFS_FAT
 endif
 
Quoted 5 lines

package/dosfstools/dosfstools.mk DOSFSTOOLS_CONF_OPTS += LIBS="-liconv" · line 66

Quoted 3 lines
package/dosfstools/dosfstools.mk +3 -0
+# Install what's left into the actual target directory
+define DOSFSTOOLS_TARGET_INSTALL
+     tar -C $(DOSFSTOOLS_TEMP) -c -f - . | tar -C $(TARGET_DIR) -x -f -

We generally use rsync to copy files, not the tar trick.

Sure. I'll change that.

Quoted 5 lines

However, I'm wondering... dosfstools only installs things in /sbin,
/usr/share/man and /usr/share/doc. The latter two get deleted in target-finalize. Wouldn't it be a lot easier then to manually install the programs we need to sbin, instead of installing everything and removing what is not needed?

I had that first, actually. But then I was thinking that would duplicate the install logic (or at least a part of it) in the package makefile. It is unlikely, but it could lead to some unexpected dependencies between upstream tar-ball and Buildroot makefile after an update of dosfstools. But maybe that isn't a big problem or there wouldn't be any more of a dependency than with the approach above.

I'm open to either approach. Does anybody have any thoughts one way or another?

Quoted 10 lines

Regards,
Arnout

Quoted 7 lines
package/dosfstools/dosfstools.mk +4 -0
+     rm -rf $(DOSFSTOOLS_TEMP)
+endef
+DOSFSTOOLS_POST_INSTALL_TARGET_HOOKS += DOSFSTOOLS_TARGET_INSTALL
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
#3 lore raw

PATCH 1/1 package/dosfstools: introduce sanitation step during install

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Markus" == Markus Mayer <[email protected]> writes:

Hi,

Quoted 6 lines
Quoted 5 lines

However, I'm wondering... dosfstools only installs things in /sbin, /usr/share/man and /usr/share/doc. The latter two get deleted in target-finalize. Wouldn't it be a lot easier then to manually install the programs we need to sbin, instead of installing everything and removing what is not needed?

Quoted 8 lines

I had that first, actually. But then I was thinking that would duplicate the install logic (or at least a part of it) in the package makefile. It is unlikely, but it could lead to some unexpected dependencies between upstream tar-ball and Buildroot makefile after an update of dosfstools. But maybe that isn't a big problem or there wouldn't be any more of a dependency than with the approach above.

I'm open to either approach. Does anybody have any thoughts one way or another?

Manually installing the specific binaries we want seems like the simplest / clearest option to me.

Care to send a patch doing that? Thanks!

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH 1/1 package/dosfstools: introduce sanitation step during install

Quoted 18 lines · Peter Korsgaard
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

"Markus" == Markus Mayer <[email protected]> writes:

Hi,

Quoted 12 lines
Quoted 5 lines

However, I'm wondering... dosfstools only installs things in /sbin, /usr/share/man and /usr/share/doc. The latter two get deleted in target-finalize. Wouldn't it be a lot easier then to manually install the programs we need to sbin, instead of installing everything and removing what is not needed?

I had that first, actually. But then I was thinking that would duplicate the install logic (or at least a part of it) in the package makefile. It is unlikely, but it could lead to some unexpected dependencies between upstream tar-ball and Buildroot makefile after an update of dosfstools. But maybe that isn't a big problem or there wouldn't be any more of a dependency than with the approach above.

Quoted 6 lines
Quoted 1 line

I'm open to either approach. Does anybody have any thoughts one way or another?

Manually installing the specific binaries we want seems like the simplest / clearest option to me.

Care to send a patch doing that? Thanks!

Done.

Regards,
-Markus

Quoted 2 lines

--
Bye, Peter Korsgaard