Skip to contentSkip to conversation

udev providers: compile the hwdb at build time (branch yem/host-eudev-2)

PATCH New Yann E. MORIN 3 participants 12 messages Oct 29, 2019 – Nov 6, 2019

lore
Share & tools
lore
#0 lore raw

v2 cover udev providers: compile the hwdb at build time (branch yem/host-eudev-2)

Hello All!

The *.hwdb files installed by eudev or systemd are "source" files; they need to be compiled into a binary form to be useful.

This needs udevadm to be available, so we introduce a eudev host variant to provide it, and use that from both eudev and systemd to compile the database during target-finalize.

---
Changes v1 -> v2:
- do a custom configure and install of host-eudev, to simplify the hooks (Carlos)

Regards,
Yann E. MORIN.

The following changes since commit 1f89ab5c0cf44017955ebf72c89c31df60b1b0a5

package/uclibc-ng-test: fix build with latest glibc (2019-10-29 22:55:36 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 803cedc8ae9597d0ae287e65d873cb9e0f6a91d9

packagesystemd: generate the hwdb.bin (2019-10-29 23:06:24 +0100)

----------------------------------------------------------------
Yann E. MORIN (3):
package/eudev: introduce a host variant
package/eudev: generate the hwdb.bin
packagesystemd: generate the hwdb.bin

 package/Config.in.host       |  1 +
 package/eudev/Config.in      |  3 ++-
 package/eudev/Config.in.host |  3 +++
 package/eudev/eudev.mk       | 36 ++++++++++++++++++++++++++++++++++++
 package/systemd/Config.in    |  1 +
 5 files changed, 43 insertions(+), 1 deletion(-)
 create mode 100644 package/eudev/Config.in.host
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v2 1/3 package/eudev: introduce a host variant

eudev and systemd provide a hardware database (hwdb) as a set of multiple files. Various other utilities may also use that database.

Those files have to be "compiled" into a binary to be useful; libudev (and thus all utilities based on it) only use the compiled hwdb.

Compiling the hwdb is done with udevadm, using the hwdb sub-command:

    udevadm hwdb --update

Provide a simple host-variant of eudev, so that we can call udevadm at build time.

When it is configured, eudev will shoehorn its --prefix path as the base location where the .hwdb file will be searched from, as well as where the hwdb.bin will be generated in. This means that with the usual --prefix=$(HOST_DIR), it would look into there.

udevadm also accepts a --root=/path option at runtime, which prepends /path to all the paths it uses to find and generate files.

Obviously, combining --root=$(TARGET_DIR) and --prefix=$(HOST_DIR) would not do what we want: all files would be searched for, and generated, in $(HOST_DIR)$(TARGET_DIR)/ . Avoiding use of --root would not help much either, as files would still searched in $(HOST_DIR) (we could use a trick to copy files there, generate and then move the hwdb.bin, but that's not nice).

However, since we only need udevadm, and since udevadm has no internal and no external dependency, we can use a less dirty trick, an configure host-eudev with --prefix=/usr (and similar for the other paths), manually copy udevadm to HOST_DIR, and then use --root when calling it.

Then, we get a udevadm that can read files from, and generate files into $(TARGET_DIR). We register a target-finalize hook to generate the hwdb.bin, so that any pakage may install its .hwdb files (currently only eudev and systemd do, but other packages might (e.g. sane is known to do so on standard desktop distros))

The *.hwdb source files consume a lot of space, roughly the same as the generated database, i.e. ~8MiB as of today, and they are totally useless on the target; only the generated hwdb.bin is useful. So we want to get rid of them.

However, we also want to be able to complete a build (e.g. make foo-reinstall to reinstall more hwdb files), so we don't want to irremediably lose them. As such, we register a pre-rootfs-cmd hook, that removes them just before assembling the filesystems, when we're only using a copy of the target directory.

Note that this is the first host package to register a target-finalize hook, and also the first to register a pre-rootfs-cmd hook. This avoids duplicating these hooks logic in both eudev and systemd.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- tweak and trick the configure and install (Carlos)
- include the hooks to avoid duplication (Peter)
- expand the commit log accordingly

---
 package/Config.in.host       |  1 +
 package/eudev/Config.in.host |  3 +++
 package/eudev/eudev.mk       | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+)
 create mode 100644 package/eudev/Config.in.host
package/Config.in.host +1 -0
index 914491e210..c0ac4fbbcc 100644
@@ -17,6 +17,7 @@ menu "Host utilities"
 	source "package/dtc/Config.in.host"
 	source "package/e2fsprogs/Config.in.host"
 	source "package/e2tools/Config.in.host"
+	source "package/eudev/Config.in.host"
 	source "package/f2fs-tools/Config.in.host"
 	source "package/faketime/Config.in.host"
 	source "package/fatcat/Config.in.host"
package/eudev/Config.in.host new file +3 -0
new file mode 100644
index 0000000000..87dfc43912
@@ -0,0 +1,3 @@
+# Select this if you need host eudev tools (e.g. udevadm)
+config BR2_PACKAGE_HOST_EUDEV
+	bool
package/eudev/eudev.mk +36 -0
index 2a3adb849c..99b5befd09 100644
@@ -55,4 +55,40 @@ define EUDEV_USERS
 	- - kvm -1 * - - - kvm nodes
 endef
 
+HOST_EUDEV_DEPENDENCIES = host-gperf host-pkgconf
+
+HOST_EUDEV_SYSCONFDIR = $(if $(BR2_PACKAGE_SYSTEMD),/usr/lib,/etc)
+HOST_EUDEV_CONF_OPTS = \
+	--prefix=/usr \
+	--sbindir=/sbin \
+	--libexecdir=/lib \
+	--with-rootlibdir=/lib \
+	--sysconfdir=$(HOST_EUDEV_SYSCONFDIR) \
+	--disable-blkid \
+	--disable-introspection \
+	--disable-kmod \
+	--disable-manpages \
+	--disable-rule-generator \
+	--disable-selinux \
+	--enable-hwdb
+
+define HOST_EUDEV_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/src/udev/udevadm \
+		$(HOST_DIR)/bin/udevadm
+endef
+
+define HOST_EUDEV_BUILD_HWDB
+	$(HOST_DIR)/bin/udevadm hwdb --update --root $(TARGET_DIR)
+endef
+HOST_EUDEV_TARGET_FINALIZE_HOOKS += HOST_EUDEV_BUILD_HWDB
+
+# Note: this will run in the filesystem context, so will use a copy
+# of tharget/, not the real one, so the files are still available on
+# re-builds (foo-rebuild, etc...)
+define HOST_EUDEV_RM_HWDB_SRC
+	rm -rf $(TARGET_DIR)/$(HOST_EUDEV_SYSCONFDIR)/udev/hwdb.d/
+endef
+HOST_EUDEV_ROOTFS_PRE_CMD_HOOKS += HOST_EUDEV_RM_HWDB_SRC
+
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
Signature · 1 line
2.20.1
#2 lore raw

PATCH v2 1/3 package/eudev: introduce a host variant

Quoted 99 lines · Yann E. MORIN

eudev and systemd provide a hardware database (hwdb) as a set of multiple files. Various other utilities may also use that database.

Those files have to be "compiled" into a binary to be useful; libudev (and thus all utilities based on it) only use the compiled hwdb.

Compiling the hwdb is done with udevadm, using the hwdb sub-command:

    udevadm hwdb --update

Provide a simple host-variant of eudev, so that we can call udevadm at build time.

When it is configured, eudev will shoehorn its --prefix path as the base location where the .hwdb file will be searched from, as well as where the hwdb.bin will be generated in. This means that with the usual --prefix=$(HOST_DIR), it would look into there.

udevadm also accepts a --root=/path option at runtime, which prepends /path to all the paths it uses to find and generate files.

Obviously, combining --root=$(TARGET_DIR) and --prefix=$(HOST_DIR) would not do what we want: all files would be searched for, and generated, in $(HOST_DIR)$(TARGET_DIR)/ . Avoiding use of --root would not help much either, as files would still searched in $(HOST_DIR) (we could use a trick to copy files there, generate and then move the hwdb.bin, but that's not nice).

However, since we only need udevadm, and since udevadm has no internal and no external dependency, we can use a less dirty trick, an configure host-eudev with --prefix=/usr (and similar for the other paths), manually copy udevadm to HOST_DIR, and then use --root when calling it.

Then, we get a udevadm that can read files from, and generate files into $(TARGET_DIR). We register a target-finalize hook to generate the hwdb.bin, so that any pakage may install its .hwdb files (currently only eudev and systemd do, but other packages might (e.g. sane is known to do so on standard desktop distros))

The *.hwdb source files consume a lot of space, roughly the same as the generated database, i.e. ~8MiB as of today, and they are totally useless on the target; only the generated hwdb.bin is useful. So we want to get rid of them.

However, we also want to be able to complete a build (e.g. make foo-reinstall to reinstall more hwdb files), so we don't want to irremediably lose them. As such, we register a pre-rootfs-cmd hook, that removes them just before assembling the filesystems, when we're only using a copy of the target directory.

Note that this is the first host package to register a target-finalize hook, and also the first to register a pre-rootfs-cmd hook. This avoids duplicating these hooks logic in both eudev and systemd.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- tweak and trick the configure and install (Carlos)
- include the hooks to avoid duplication (Peter)
- expand the commit log accordingly

---
 package/Config.in.host       |  1 +
 package/eudev/Config.in.host |  3 +++
 package/eudev/eudev.mk       | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+)
 create mode 100644 package/eudev/Config.in.host
package/Config.in.host +1 -0
index 914491e210..c0ac4fbbcc 100644
@@ -17,6 +17,7 @@ menu "Host utilities"
        source "package/dtc/Config.in.host"
        source "package/e2fsprogs/Config.in.host"
        source "package/e2tools/Config.in.host"
+       source "package/eudev/Config.in.host"
        source "package/f2fs-tools/Config.in.host"
        source "package/faketime/Config.in.host"
        source "package/fatcat/Config.in.host"
package/eudev/Config.in.host new file +3 -0
new file mode 100644
index 0000000000..87dfc43912
@@ -0,0 +1,3 @@
+# Select this if you need host eudev tools (e.g. udevadm)
+config BR2_PACKAGE_HOST_EUDEV
+       bool
package/eudev/eudev.mk +0 -0
index 2a3adb849c..99b5befd09 100644
@@ -55,4 +55,40 @@ define EUDEV_USERS
        - - kvm -1 * - - - kvm nodes
 endef

package/eudev/eudev.mk define EUDEV_USERS · line 66

Quoted 10 lines
package/eudev/eudev.mk +9 -0
 
+HOST_EUDEV_DEPENDENCIES = host-gperf host-pkgconf
+
+HOST_EUDEV_SYSCONFDIR = $(if $(BR2_PACKAGE_SYSTEMD),/usr/lib,/etc)
+HOST_EUDEV_CONF_OPTS = \
+       --prefix=/usr \
+       --sbindir=/sbin \
+       --libexecdir=/lib \
+       --with-rootlibdir=/lib \
+       --sysconfdir=$(HOST_EUDEV_SYSCONFDIR) \

We don't actually need this, since /etc/udev/udevdb.bin is a valid location for systemd too. It was my idea, but I take it back. :-)

Quoted 31 lines
package/eudev/eudev.mk +27 -1
+       --disable-blkid \
+       --disable-introspection \
+       --disable-kmod \
+       --disable-manpages \
+       --disable-rule-generator \
+       --disable-selinux \
+       --enable-hwdb
+
+define HOST_EUDEV_INSTALL_CMDS
+       $(INSTALL) -D -m 0755 $(@D)/src/udev/udevadm \
+               $(HOST_DIR)/bin/udevadm
+endef
+
+define HOST_EUDEV_BUILD_HWDB
+       $(HOST_DIR)/bin/udevadm hwdb --update --root $(TARGET_DIR)
+endef
+HOST_EUDEV_TARGET_FINALIZE_HOOKS += HOST_EUDEV_BUILD_HWDB
+
+# Note: this will run in the filesystem context, so will use a copy
+# of tharget/, not the real one, so the files are still available on
+# re-builds (foo-rebuild, etc...)
+define HOST_EUDEV_RM_HWDB_SRC
+       rm -rf $(TARGET_DIR)/$(HOST_EUDEV_SYSCONFDIR)/udev/hwdb.d/
+endef
+HOST_EUDEV_ROOTFS_PRE_CMD_HOOKS += HOST_EUDEV_RM_HWDB_SRC
+
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
--

2.20.1

Signature · 1 line
Carlos Santos <[email protected]>
#3 lore raw

PATCH v2 1/3 package/eudev: introduce a host variant

Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Carlos" == Carlos Santos <[email protected]> writes:

+++ b/package/eudev/eudev.mk

(quoted diff) +0 -1
@@ -55,4 +55,40 @@ define EUDEV_USERS
- - kvm -1 * - - - kvm nodes

endef

Quoted 12 lines

package/eudev/eudev.mk define EUDEV_USERS · line 66

Quoted 9 lines
package/eudev/eudev.mk +9 -0
+HOST_EUDEV_DEPENDENCIES = host-gperf host-pkgconf
+
+HOST_EUDEV_SYSCONFDIR = $(if $(BR2_PACKAGE_SYSTEMD),/usr/lib,/etc)
+HOST_EUDEV_CONF_OPTS = \
+       --prefix=/usr \
+       --sbindir=/sbin \
+       --libexecdir=/lib \
+       --with-rootlibdir=/lib \
+       --sysconfdir=$(HOST_EUDEV_SYSCONFDIR) \

We don't actually need this, since /etc/udev/udevdb.bin is a valid location for systemd too. It was my idea, but I take it back. :-)

Are you sure that works? The sysconfdir is used both for the input and output location. If you use systemd then the files provided by systemd-udev presumably gets installed under /usr/lib/udev/hwdb.d as well?

Signature · 1 line
Bye, Peter Korsgaard
#4 lore raw

PATCH v2 1/3 package/eudev: introduce a host variant

In reply to Yann E. MORIN
Quoted 51 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

eudev and systemd provide a hardware database (hwdb) as a set of multiple files. Various other utilities may also use that database.

Those files have to be "compiled" into a binary to be useful; libudev (and thus all utilities based on it) only use the compiled hwdb.

Compiling the hwdb is done with udevadm, using the hwdb sub-command:

    udevadm hwdb --update

Provide a simple host-variant of eudev, so that we can call udevadm at build time.

When it is configured, eudev will shoehorn its --prefix path as the base location where the .hwdb file will be searched from, as well as where the hwdb.bin will be generated in. This means that with the usual --prefix=$(HOST_DIR), it would look into there.

udevadm also accepts a --root=/path option at runtime, which prepends /path to all the paths it uses to find and generate files.

Obviously, combining --root=$(TARGET_DIR) and --prefix=$(HOST_DIR) would not do what we want: all files would be searched for, and generated, in $(HOST_DIR)$(TARGET_DIR)/ . Avoiding use of --root would not help much either, as files would still searched in $(HOST_DIR) (we could use a trick to copy files there, generate and then move the hwdb.bin, but that's not nice).

However, since we only need udevadm, and since udevadm has no internal and no external dependency, we can use a less dirty trick, an configure host-eudev with --prefix=/usr (and similar for the other paths), manually copy udevadm to HOST_DIR, and then use --root when calling it.

Then, we get a udevadm that can read files from, and generate files into $(TARGET_DIR). We register a target-finalize hook to generate the hwdb.bin, so that any pakage may install its .hwdb files (currently only eudev and systemd do, but other packages might (e.g. sane is known to do so on standard desktop distros))

The *.hwdb source files consume a lot of space, roughly the same as the generated database, i.e. ~8MiB as of today, and they are totally useless on the target; only the generated hwdb.bin is useful. So we want to get rid of them.

However, we also want to be able to complete a build (e.g. make foo-reinstall to reinstall more hwdb files), so we don't want to irremediably lose them. As such, we register a pre-rootfs-cmd hook, that removes them just before assembling the filesystems, when we're only using a copy of the target directory.

Quoted 8 lines

Note that this is the first host package to register a target-finalize hook, and also the first to register a pre-rootfs-cmd hook. This avoids duplicating these hooks logic in both eudev and systemd.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#5 lore raw

PATCH v2 1/3 package/eudev: introduce a host variant

In reply to Yann E. MORIN
Quoted 60 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

eudev and systemd provide a hardware database (hwdb) as a set of multiple files. Various other utilities may also use that database.

Those files have to be "compiled" into a binary to be useful; libudev (and thus all utilities based on it) only use the compiled hwdb.

Compiling the hwdb is done with udevadm, using the hwdb sub-command:

    udevadm hwdb --update

Provide a simple host-variant of eudev, so that we can call udevadm at build time.

When it is configured, eudev will shoehorn its --prefix path as the base location where the .hwdb file will be searched from, as well as where the hwdb.bin will be generated in. This means that with the usual --prefix=$(HOST_DIR), it would look into there.

udevadm also accepts a --root=/path option at runtime, which prepends /path to all the paths it uses to find and generate files.

Obviously, combining --root=$(TARGET_DIR) and --prefix=$(HOST_DIR) would not do what we want: all files would be searched for, and generated, in $(HOST_DIR)$(TARGET_DIR)/ . Avoiding use of --root would not help much either, as files would still searched in $(HOST_DIR) (we could use a trick to copy files there, generate and then move the hwdb.bin, but that's not nice).

However, since we only need udevadm, and since udevadm has no internal and no external dependency, we can use a less dirty trick, an configure host-eudev with --prefix=/usr (and similar for the other paths), manually copy udevadm to HOST_DIR, and then use --root when calling it.

Then, we get a udevadm that can read files from, and generate files into $(TARGET_DIR). We register a target-finalize hook to generate the hwdb.bin, so that any pakage may install its .hwdb files (currently only eudev and systemd do, but other packages might (e.g. sane is known to do so on standard desktop distros))

The *.hwdb source files consume a lot of space, roughly the same as the generated database, i.e. ~8MiB as of today, and they are totally useless on the target; only the generated hwdb.bin is useful. So we want to get rid of them.

However, we also want to be able to complete a build (e.g. make foo-reinstall to reinstall more hwdb files), so we don't want to irremediably lose them. As such, we register a pre-rootfs-cmd hook, that removes them just before assembling the filesystems, when we're only using a copy of the target directory.

Note that this is the first host package to register a target-finalize hook, and also the first to register a pre-rootfs-cmd hook. This avoids duplicating these hooks logic in both eudev and systemd.

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>
Quoted 5 lines

---
Changes v1 -> v2:
- tweak and trick the configure and install (Carlos)
- include the hooks to avoid duplication (Peter)
- expand the commit log accordingly

Committed to 2019.02.x and 2019.08.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#6 lore raw

PATCH v2 2/3 package/eudev: generate the hwdb.bin

In reply to Yann E. MORIN
Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- drop the post hooks

---
 package/eudev/Config.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/eudev/Config.in +2 -1
index 2220265a55..b1d2f37a72 100644
@@ -29,8 +29,9 @@ config BR2_PACKAGE_EUDEV_RULES_GEN
 config BR2_PACKAGE_EUDEV_ENABLE_HWDB
 	bool "enable hwdb installation"
 	default y
+	select BR2_PACKAGE_HOST_EUDEV # for udevadm, during target-finalize
 	help
-	  Enables hardware database installation to /etc/udev/hwdb.d
+	  Enables hardware database installation to /etc/udev/hwdb.bin
 
 endif
 
Signature · 1 line
2.20.1
#8 lore raw

PATCH v2 2/3 package/eudev: generate the hwdb.bin

In reply to Yann E. MORIN
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- drop the post hooks

Committed to 2019.02.x and 2019.08.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#9 lore raw

PATCH v2 3/3 packagesystemd: generate the hwdb.bin

In reply to Yann E. MORIN
Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- drop the post hooks

---
 package/systemd/Config.in | 1 +
 1 file changed, 1 insertion(+)
package/systemd/Config.in +1 -0
index 601398bf96..aef39abe27 100644
@@ -199,6 +199,7 @@ config BR2_PACKAGE_SYSTEMD_HOSTNAMED
 config BR2_PACKAGE_SYSTEMD_HWDB
 	bool "enable hwdb installation"
 	default y
+	select BR2_PACKAGE_HOST_EUDEV # for udevadm, during target-finalize
 	help
 	  Enables hardware database installation to /usr/lib/udev/hwdb.d
 
Signature · 1 line
2.20.1
#11 lore raw

PATCH v2 3/3 packagesystemd: generate the hwdb.bin

In reply to Yann E. MORIN
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Carlos Santos <[email protected]>

---
Changes v1 -> v2:
- drop the post hooks

Committed to 2019.02.x and 2019.08.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard