Add compressed ID database support to (usb|pci)-db
PATCH 1/1 Add compressed ID database support to (usb|pci)-db
Hi
usbutils and pciutils have support to gzip'ed ID databases. But, if I use usb.ids.gz or pci.ids.gz, usb-db and pci-db can't show anything, giving only an error of file not found. Looking in the code, they didn't have any support to compressed ID files.
This patch add support to gzip'ed database files to usb-db.c and add the option --disable-zlib to configure.ac. Most of code here come from usbutils names.c file with little modifications to work with udev.
Bye,
Piter PUNK
configure.ac +17 -1
@@ -6,7 +6,7 @@ AC_DISABLE_STATIC AC_USE_SYSTEM_EXTENSIONS AC_SYS_LARGEFILE AC_CONFIG_MACRO_DIR([m4]) -LT_INIT +m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL]) AC_PROG_AWK GTK_DOC_CHECK(1.10) AC_PREFIX_DEFAULT([/usr])
@@ -69,6 +69,22 @@ if test "x$enable_extras" = xyes; then PKG_CHECK_MODULES(USBUTILS, usbutils >= 0.82) AC_SUBST([USB_DATABASE], [$($PKG_CONFIG --variable=usbids usbutils)]) + USE_ZLIB=yes + AC_ARG_ENABLE(zlib, + AS_HELP_STRING(--disable-zlib,disable support for zlib), + [ + if eval "test x$enable_zlib = xno"; then + USE_ZLIB= + fi + ]) + if test "$USE_ZLIB" = "yes" ; then + AC_CHECK_LIB(z, inflateEnd) + if test "${ac_cv_lib_z_inflateEnd}" = "yes" ; then + HAVE_ZLIB="yes" + fi + fi + AM_CONDITIONAL(HAVE_ZLIB, test x$HAVE_ZLIB = xyes) + AC_CHECK_FILES([/usr/share/pci.ids], [pciids=/usr/share/pci.ids]) AC_CHECK_FILES([/usr/share/hwdata/pci.ids], [pciids=/usr/share/hwdata/pci.ids]) AC_CHECK_FILES([/usr/share/misc/pci.ids], [pciids=/usr/share/misc/pci.ids])
extras/usb-db/usb-db.c +31 -12
74 lines not on this page · raw message
Quoted 7 lines · Piter PUNK
Hi
This patch add support to gzip'ed database files to
usb-db.cand add the option
--disable-zlib to configure.ac. Most of code here come from usbutilsnames.c
file with little modifications to work with udev.
Looking at the code it seems that enabling zlib support sets HAVE_ZLIB which assumes that the DBs are always compressed with gzip which removes the support for uncompressed db files. Logically, both should be supported with some kind of file format detection.
Ignore this if I wrongly understand the intention :)
Regards,
Quoted 8 lines · Ozan Çağlayan
Quoted 6 lines · Piter PUNK
This patch add support to gzip'ed database files to
usb-db.cand add the option
--disable-zlib to configure.ac. Most of code here come from usbutilsnames.c
file with little modifications to work with udev.
Quoted 5 lines
Looking at the code it seems that enabling zlib support sets
HAVE_ZLIBwhich assumes that the DBs are always compressed with gzip which removes the support for uncompressed db files. Logically, both should be supported with some kind of file format detection.
zlib functions can read compressed and uncompressed files. That's why I try to
open DATABASE.gz first and, if failed, open DATABASE when zlib support is enabled. The support of both file formats is OK when zlib is enabled.
When disabled, only uncompressed files are supported. Almost no changes there. The only drawback of enable zlib support is the need of zlib shared library. But, who are using gzip'ed ID db files already needs zlib to have working lsusb and lspci.
Quoted 2 lines
Ignore this if I wrongly understand the intention :)
Oh, i think is better to explain. Maybe someone else have the same doubt.
Hope it's all OK now -:)
Piter Punk
Quoted 1 line · Piter PUNK
Hi
Heya,
Quoted 9 lines
usbutils and pciutils have support to gzip'ed ID databases. But, if I use usb.ids.gz or pci.ids.gz, usb-db and pci-db can't show anything, giving only an error of file not found. Looking in the code, they didn't have any support to compressed ID files.
This patch add support to gzip'ed database files to
usb-db.cand add the option --disable-zlib to configure.ac. Most of code here come from usbutilsnames.cfile with little modifications to work with udev.
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
That said, generally speaking it's bad enough that we do a linar search for the vid/pid. Compressing things won't make things faster.
Is there actually a distribution that uses a compressed pci.ids file?
configure.ac AC_DISABLE_STATIC · line 9
Quoted 10 lines
configure.ac +1
-1@@ -6,7 +6,7 @@ AC_DISABLE_STATIC AC_USE_SYSTEM_EXTENSIONS AC_SYS_LARGEFILE AC_CONFIG_MACRO_DIR([m4]) -LT_INIT +m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL])
Uh, this shouldn't be sneaked in here.
Lennart
Signature · 3 lines
Lennart Poettering Red Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Quoted 2 lines · Lennart Poettering
That said, generally speaking it's bad enough that we do a linar search for the vid/pid.
You could do a binary search kind of deal, the entries are sorted. Not sure it's worth the effort due to how things are laid out - you'd need to search back or forward to find the first line not starting with white-space.
Quoted 3 lines
Compressing things won't make things faster.
Is there actually a distribution that uses a compressed pci.ids file?
None of the major distros does this kind of thing. And that is a good thing - see below.
FWIW, this thing came up a couple of years ago on the hal list and back then I very strongly refused to support compressed ids files. Mostly on the grounds that it makes it impossible to mmap the files. E.g. you need to allocate memory for the uncompressed image.
In udev this is even worse because we'd be doing all this work over and over again - for every freaking event. In HAL, we mmaped the ids files at start-up, built a simple look-up table and did binary lookups on every event.
Just so you know, I'm still very much against this and I think it would be a mistake to support compressed ids files in in udev. Please don't do it.
David
Quoted 5 lines · David Zeuthen
FWIW, this thing came up a couple of years ago on the hal list and back then I very strongly refused to support compressed ids files. Mostly on the grounds that it makes it impossible to mmap the files. E.g. you need to allocate memory for the uncompressed image.
Quoted 8 lines
In udev this is even worse because we'd be doing all this work over and over again - for every freaking event. In HAL, we mmaped the ids files at start-up, built a simple look-up table and did binary lookups on every event.
Just so you know, I'm still very much against this and I think it would be a mistake to support compressed ids files in in udev. Please don't do it.
Ok, that makes sense to me. Sorry, Piter, it is unlikely that we will merge your patch.
Even though your work will not be merged it is really appreciated!
Sorry,
Lennart
Signature · 3 lines
Lennart Poettering Red Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Quoted 4 lines · Lennart Poettering
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
The macro orgy is from usb-utils code -;)
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
After installed, the distribution uses uncompressed pci.ids and usb.ids. I am doing some tests with usb-utils and pciutils packages and got the errors from usb-db and pci-db. Because that i wrote the patch.
Quoted 4 lines
Quoted 4 lines
configure.ac +0
-0
Quoted 8 lines
configure.ac AC_DISABLE_STATIC · line 9
Quoted 6 lines
configure.ac +1
-1@@ -6,7 +6,7 @@ AC_DISABLE_STATIC AC_USE_SYSTEM_EXTENSIONS AC_SYS_LARGEFILE AC_CONFIG_MACRO_DIR([m4]) -LT_INIT +m4_ifdef([LT_INIT],[LT_INIT],[AC_PROG_LIBTOOL])Uh, this shouldn't be sneaked in here.
Sorry! I put this here to compile udev from git on Slackware. Forgot the change is there (well, this is very useful for me).
Thanks
Piter PUNK
Quoted 4 lines · Lennart Poettering
Ok, that makes sense to me. Sorry, Piter, it is unlikely that we will merge your patch.
Even though your work will not be merged it is really appreciated!
Ok! Better luck for me next time -:)
Piter PUNK
Quoted 6 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
Quoted 6 lines
The macro orgy is from usb-utils code -;)
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.
Lennart
Signature · 3 lines
Lennart Poettering Red Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Quoted 10 lines · Lennart Poettering
Quoted 8 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
The macro orgy is from usb-utils code -;)
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.
s/initialization/installation/
Lennart
Signature · 3 lines
Lennart Poettering Red Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4
Quoted 6 lines · Lennart Poettering
Quoted 2 lines
Slackware uses it on installer but the udev use there is very restrict.
AFAIK currently the data generated by usb-db/pci-db is used only by PulseAudio, NetworkManager and ModemManager. I am not sure how either of these would matter during initialization.
Good. I know our udev use on installation is very limited, only to create /dev and load network modules. Good to see that even if our udev use increase, the (pci|usb).ids.gz support isn´t a problem at all.
Thanks again!
Piter PUNK
Quoted 8 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
The macro orgy is from usb-utils code -;)
And I'm seriously considering removing it from the usbutils package as no distro ships compressed ids that I know of.
Quoted 3 lines
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
Why would an installer need an .ids file?
thanks,
greg k-h
Quoted 5 lines · Greg KH
Quoted 3 lines
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
Why would an installer need an .ids file?
Because people wants to use lspci and lsusb to know the hardware they are using and both commands search the names on .ids files. The lspci helps me more than once to look some board and found tips to finnish installation or fix some installer bug (usually some boot parameter or kernel compilation option).
Piter PUNK
Quoted 8 lines · Greg KH
Quoted 7 lines · Piter PUNK
Quoted 4 lines · Lennart Poettering
Patch looks mstly good. However after discussing this a little with Kay we'd prefer if we could do without the macro orgy and just depend unconditionally on zlib. Given that this is in extras this should not be a problem.
Quoted 4 lines
Quoted 1 line
The macro orgy is from usb-utils code -;)
And I'm seriously considering removing it from the usbutils package as no distro ships compressed ids that I know of.
We used to do that (for pci.ids) but had to disable it when HAL started to mmap the file and pciutils couldn't handle having file in both compressed and uncompressed form.
Signature · 2 lines
Frederic Crozat <[email protected]> Mandriva
Quoted 11 lines · Piter PUNK
Quoted 5 lines · Greg KH
Quoted 3 lines
Quoted 1 line
Is there actually a distribution that uses a compressed pci.ids file?
Slackware uses it on installer but the udev use there is very restrict.
Why would an installer need an .ids file?
Because people wants to use lspci and lsusb to know the hardware they are using and both commands search the names on .ids files.
You should not need the id file for USB as the strings in the device are almost always correct. So you might wish to change your installer to remove that dependancy.
thanks,
greg k-h