Skip to contentSkip to conversation

Add compressed ID database support to (usb|pci)-db

PATCH In review Piter PUNK 7 participants 15 messages Sep 22, 2009 – Sep 30, 2009

lore
Share & tools
lore

Piter PUNK

author
#0 lore raw

PATCH 1/1 Add compressed ID database support to (usb|pci)-db

Hi

usbutils and pciutils have support to gzip'ed ID databases. But, if I use usb.ids.gz or pci.ids.gz, usb-db and pci-db can't show anything, giving only an error of file not found. Looking in the code, they didn't have any support to compressed ID files.

This patch add support to gzip'ed database files to usb-db.c and add the option --disable-zlib to configure.ac. Most of code here come from usbutils names.c file with little modifications to work with udev.

Bye,

Piter PUNK

configure.ac +17 -1
index 3afb94a..5a273e6 100644
@@ -6,7 +6,7 @@ AC_DISABLE_STATIC
 AC_USE_SYSTEM_EXTENSIONS
 AC_SYS_LARGEFILE
 AC_CONFIG_MACRO_DIR([m4])
-LT_INIT
+m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL])
 AC_PROG_AWK
 GTK_DOC_CHECK(1.10)
 AC_PREFIX_DEFAULT([/usr])
@@ -69,6 +69,22 @@ if test "x$enable_extras" = xyes; then
 	PKG_CHECK_MODULES(USBUTILS, usbutils >= 0.82)
 	AC_SUBST([USB_DATABASE], [$($PKG_CONFIG --variable=usbids usbutils)])
 
+	USE_ZLIB=yes
+	AC_ARG_ENABLE(zlib,
+		AS_HELP_STRING(--disable-zlib,disable support for zlib),
+		[
+			if eval "test x$enable_zlib = xno"; then
+				USE_ZLIB=
+			fi
+		])
+	if test "$USE_ZLIB" = "yes" ; then
+		AC_CHECK_LIB(z, inflateEnd)
+		if test "${ac_cv_lib_z_inflateEnd}" = "yes" ; then
+			HAVE_ZLIB="yes"
+		fi
+	fi
+	AM_CONDITIONAL(HAVE_ZLIB, test x$HAVE_ZLIB = xyes)
+
 	AC_CHECK_FILES([/usr/share/pci.ids], [pciids=/usr/share/pci.ids])
 	AC_CHECK_FILES([/usr/share/hwdata/pci.ids], [pciids=/usr/share/hwdata/pci.ids])
 	AC_CHECK_FILES([/usr/share/misc/pci.ids], [pciids=/usr/share/misc/pci.ids])
extras/usb-db/usb-db.c +31 -12

74 lines not on this page · raw message

#1 lore raw
Quoted 7 lines · Piter PUNK

Hi

This patch add support to gzip'ed database files to usb-db.c and add the option
--disable-zlib to configure.ac. Most of code here come from usbutils names.c
file with little modifications to work with udev.

Looking at the code it seems that enabling zlib support sets HAVE_ZLIB which assumes that the DBs are always compressed with gzip which removes the support for uncompressed db files. Logically, both should be supported with some kind of file format detection.

Ignore this if I wrongly understand the intention :)

Regards,

author
#2 lore raw
In reply to Piter PUNK
Quoted 8 lines · Ozan Çağlayan
Quoted 6 lines · Piter PUNK

This patch add support to gzip'ed database files to usb-db.c and add the option
--disable-zlib to configure.ac. Most of code here come from usbutils names.c
file with little modifications to work with udev.

Quoted 5 lines

Looking at the code it seems that enabling zlib support sets HAVE_ZLIB which assumes that the DBs are always compressed with gzip which removes the support for uncompressed db files. Logically, both should be supported with some kind of file format detection.

zlib functions can read compressed and uncompressed files. That's why I try to
open DATABASE.gz first and, if failed, open DATABASE when zlib support is enabled. The support of both file formats is OK when zlib is enabled.

When disabled, only uncompressed files are supported. Almost no changes there. The only drawback of enable zlib support is the need of zlib shared library. But, who are using gzip'ed ID db files already needs zlib to have working lsusb and lspci.

Quoted 2 lines

Ignore this if I wrongly understand the intention :)

Oh, i think is better to explain. Maybe someone else have the same doubt.

Hope it's all OK now -:)

Piter Punk

#3 lore raw
In reply to Piter PUNK
Quoted 1 line · Piter PUNK

Hi

Heya,

Quoted 9 lines

usbutils and pciutils have support to gzip'ed ID databases. But, if I use usb.ids.gz or pci.ids.gz, usb-db and pci-db can't show anything, giving only an error of file not found. Looking in the code, they didn't have any support to compressed ID files.

This patch add support to gzip'ed database files to usb-db.c and add the option --disable-zlib to configure.ac. Most of code here come from usbutils names.c file with little modifications to work with udev.

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

That said, generally speaking it's bad enough that we do a linar search for the vid/pid. Compressing things won't make things faster.

Is there actually a distribution that uses a compressed pci.ids file?

configure.ac AC_DISABLE_STATIC · line 9

Quoted 10 lines
configure.ac +1 -1
index 3afb94a..5a273e6 100644
@@ -6,7 +6,7 @@ AC_DISABLE_STATIC
 AC_USE_SYSTEM_EXTENSIONS
 AC_SYS_LARGEFILE
 AC_CONFIG_MACRO_DIR([m4])
-LT_INIT
+m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL])

Uh, this shouldn't be sneaked in here.

Lennart

Signature · 3 lines
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4
#4 lore raw
In reply to Piter PUNK
Quoted 2 lines · Lennart Poettering

That said, generally speaking it's bad enough that we do a linar search for the vid/pid.

You could do a binary search kind of deal, the entries are sorted. Not sure it's worth the effort due to how things are laid out - you'd need to search back or forward to find the first line not starting with white-space.

Quoted 3 lines

Compressing things won't make things faster.

Is there actually a distribution that uses a compressed pci.ids file?

None of the major distros does this kind of thing. And that is a good thing - see below.

FWIW, this thing came up a couple of years ago on the hal list and back then I very strongly refused to support compressed ids files. Mostly on the grounds that it makes it impossible to mmap the files. E.g. you need to allocate memory for the uncompressed image.

In udev this is even worse because we'd be doing all this work over and over again - for every freaking event. In HAL, we mmaped the ids files at start-up, built a simple look-up table and did binary lookups on every event.

Just so you know, I'm still very much against this and I think it would be a mistake to support compressed ids files in in udev. Please don't do it.

David

#5 lore raw
In reply to Piter PUNK
Quoted 5 lines · David Zeuthen

FWIW, this thing came up a couple of years ago on the hal list and back then I very strongly refused to support compressed ids files. Mostly on the grounds that it makes it impossible to mmap the files. E.g. you need to allocate memory for the uncompressed image.

Quoted 8 lines

In udev this is even worse because we'd be doing all this work over and over again - for every freaking event. In HAL, we mmaped the ids files at start-up, built a simple look-up table and did binary lookups on every event.

Just so you know, I'm still very much against this and I think it would be a mistake to support compressed ids files in in udev. Please don't do it.

Ok, that makes sense to me. Sorry, Piter, it is unlikely that we will merge your patch.

Even though your work will not be merged it is really appreciated!

Sorry,

Lennart

Signature · 3 lines
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4
#6 lore raw
In reply to Piter PUNK
Quoted 4 lines · Lennart Poettering

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

The macro orgy is from usb-utils code -;)

Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

After installed, the distribution uses uncompressed pci.ids and usb.ids. I am doing some tests with usb-utils and pciutils packages and got the errors from usb-db and pci-db. Because that i wrote the patch.

Quoted 4 lines
Quoted 4 lines
configure.ac +0 -0
index 3afb94a..5a273e6 100644
Quoted 8 lines

configure.ac AC_DISABLE_STATIC · line 9

Quoted 6 lines
configure.ac +1 -1
@@ -6,7 +6,7 @@ AC_DISABLE_STATIC
 AC_USE_SYSTEM_EXTENSIONS
 AC_SYS_LARGEFILE
 AC_CONFIG_MACRO_DIR([m4])
-LT_INIT
+m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL])

Uh, this shouldn't be sneaked in here.

Sorry! I put this here to compile udev from git on Slackware. Forgot the change is there (well, this is very useful for me).

Thanks

Piter PUNK

#7 lore raw
In reply to Piter PUNK
Quoted 4 lines · Lennart Poettering

Ok, that makes sense to me. Sorry, Piter, it is unlikely that we will merge your patch.

Even though your work will not be merged it is really appreciated!

Ok! Better luck for me next time -:)

Piter PUNK

#8 lore raw
In reply to Piter PUNK
Quoted 6 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

Quoted 6 lines

The macro orgy is from usb-utils code -;)

Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.

Lennart

Signature · 3 lines
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4
#9 lore raw
In reply to Piter PUNK
Quoted 10 lines · Lennart Poettering
Quoted 8 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

The macro orgy is from usb-utils code -;)

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.

s/initialization/installation/

Lennart

Signature · 3 lines
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4
#10 lore raw
In reply to Piter PUNK
Quoted 6 lines · Lennart Poettering
Quoted 2 lines

Slackware uses it on installer but the udev use there is very restrict.

AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.

Good. I know our udev use on installation is very limited, only to create /dev and load network modules. Good to see that even if our udev use increase, the (pci|usb).ids.gz support isn´t a problem at all.

Thanks again!

Piter PUNK

#11 lore raw
In reply to Piter PUNK
Quoted 8 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

The macro orgy is from usb-utils code -;)

And I'm seriously considering removing it from the usbutils package as no distro ships compressed ids that I know of.

Quoted 3 lines
Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

Why would an installer need an .ids file?

thanks,

greg k-h

#12 lore raw
In reply to Piter PUNK
Quoted 5 lines · Greg KH
Quoted 3 lines
Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

Why would an installer need an .ids file?

Because people wants to use lspci and lsusb to know the hardware they are using and both commands search the names on .ids files. The lspci helps me more than once to look some board and found tips to finnish installation or fix some installer bug (usually some boot parameter or kernel compilation option).

Piter PUNK

#13 lore raw
In reply to Piter PUNK
Quoted 8 lines · Greg KH
Quoted 7 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering

Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.

Quoted 4 lines
Quoted 1 line

The macro orgy is from usb-utils code -;)

And I'm seriously considering removing it from the usbutils package as no distro ships compressed ids that I know of.

We used to do that (for pci.ids) but had to disable it when HAL started to mmap the file and pciutils couldn't handle having file in both compressed and uncompressed form.

Signature · 2 lines
Frederic Crozat <[email protected]>
Mandriva
#14 lore raw
In reply to Piter PUNK
Quoted 11 lines · Piter PUNK
Quoted 5 lines · Greg KH
Quoted 3 lines
Quoted 1 line

Is there actually a distribution that uses a compressed pci.ids file?

Slackware uses it on installer but the udev use there is very restrict.

Why would an installer need an .ids file?

Because people wants to use lspci and lsusb to know the hardware they are using and both commands search the names on .ids files.

You should not need the id file for USB as the strings in the device are almost always correct. So you might wish to change your installer to remove that dependancy.

thanks,

greg k-h