man/info: fix not support xz/bz2 compression

PATCH Superseded Hongxu Jia 4 participants 13 messages Sep 23, 2014 – Sep 24, 2014

lore

Share & tools

lore

Hongxu Jia

author

2014-09-23 14:20 UTC #0 lore raw

cover man/info: fix not support xz/bz2 compression

Test Case:

vim local.conf
..
IMAGE_INSTALL_append = " man man-doc texinfo texinfo-doc info info-doc" INHERIT += "compress_doc"
DOC_COMPRESS = "xz" or DOC_COMPRESS = "bz2"
..

bitbake core-image-minimal

runqemu qemux86

On target,
...

root@qemux86:~# man man

root@qemux86:~# info info
...

//Hongxu

The following changes since commit afdbe3112b940987f3f3cddcb32c91a5e17a297d:

bitbake: bitbake-worker: Fix bitbake -n (2014-09-22 15:54:40 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-doc
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-doc

Hongxu Jia (3):
busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default
man: fix not support xz/bz2 compression
texinfo: fix info not work

 meta/recipes-core/busybox/busybox/defconfig  | 6 +++---
 meta/recipes-extended/man/man/man.conf       | 3 ++-
 meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
 3 files changed, 12 insertions(+), 5 deletions(-)

Signature · 1 line

1.9.1

Hongxu Jia

author

2014-09-23 14:20 UTC #1 lore raw

PATCH 1/3 busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

It was required by info/man command to compress/decompress doc pages with xz/bz2 compression.

[YOCTO #6750]
[YOCTO #6751]

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-core/busybox/busybox/defconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

meta/recipes-core/busybox/busybox/defconfig +3 -3

index 8394067..676f770 100644
@@ -134,7 +134,7 @@ CONFIG_AR=y
 # CONFIG_FEATURE_AR_LONG_FILENAMES is not set
 # CONFIG_FEATURE_AR_CREATE is not set
 CONFIG_BUNZIP2=y
-# CONFIG_BZIP2 is not set
+CONFIG_BZIP2=y
 CONFIG_CPIO=y
 # CONFIG_FEATURE_CPIO_O is not set
 # CONFIG_FEATURE_CPIO_P is not set

@@ -165,8 +165,8 @@ CONFIG_FEATURE_TAR_NOPRESERVE_TIME=y
 # CONFIG_UNLZMA is not set
 # CONFIG_FEATURE_LZMA_FAST is not set
 # CONFIG_LZMA is not set
-# CONFIG_UNXZ is not set
-# CONFIG_XZ is not set
+CONFIG_UNXZ=y
+CONFIG_XZ=y
 CONFIG_UNZIP=y
 
 #

Signature · 1 line

1.9.1

Burton, Ross

2014-09-23 14:47 UTC #2 lore raw

Re: [PATCH 1/3] busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

Quoted 2 lines · Hongxu Jia

It was required by info/man command to compress/decompress doc pages with xz/bz2 compression.

When changing busybox configuration, it's helpful to list the size change.

So for anyone else curious, buildhistory reports the package size (not the binary size) as:

-PKGSIZE = 573632
+PKGSIZE = 585144

Ross

Hongxu Jia

author

2014-09-24 10:46 UTC #3 lore raw

Re: [PATCH 1/3] busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

Review

Quoted 10 lines · Ross

Quoted 2 lines · Hongxu Jia
It was required by info/man command to compress/decompress doc pages with xz/bz2 compression.
When changing busybox configuration, it's helpful to list the size change.
So for anyone else curious, buildhistory reports the package size (not the binary size) as:
(quoted diff) +1 -1
-PKGSIZE = 573632
+PKGSIZE = 585144

Since we care about the size, and affect of the fix is global.

How about conditionally add 'xz/bz2/gzip' to info/man's RDEPENDS according DOC_COMPRESS value.

V2 incoming

//Hongxu

Quoted 1 line

Ross

Hongxu Jia

author

2014-09-23 14:20 UTC #4 lore raw

PATCH 2/3 man: fix not support xz/bz2 compression

In reply to Hongxu Jia

In oe-core, bunzip and unzx located in /usr/bin/
rather than /usr, so tweak man's config.

[YOCTO #6750]

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-extended/man/man/man.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

meta/recipes-extended/man/man/man.conf +2 -1

index bb3c697..fc10be2 100644
@@ -133,7 +133,8 @@ MANSECT		1:1p:8:2:3:3p:4:5:6:7:9:0p:tcl:n:l:p:o
 # The command given must act as a filter.
 #
 .gz		/bin/gunzip -c
-.bz2		/bin/bzip2 -c -d
+.bz2		/usr/bin/bunzip2 -c
+.xz		/usr/bin/unxz -c
 .z		
 .Z		/bin/zcat
 .F

Signature · 1 line

1.9.1

Hongxu Jia

author

2014-09-23 14:20 UTC #5 lore raw

PATCH 3/3 texinfo: fix info not work

In reply to Hongxu Jia

In texinfo, we didn't have info/dir entries, which caused info not work.
...

root@qemux86:~# info info
info: dir: No such file or directory
...
Invoke install-info to create info/dir entries in pkg_postinst_info.

Also fix a QA Issue
...
ERROR: QA Issue: texinfo: Files/directories were installed but not shipped
/usr/share/info/info.info.bz2 /usr/share/info/info-stnd.info.bz2 [installed-vs-shipped]
...

[YOCTO #6751]

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

meta/recipes-extended/texinfo/texinfo_5.2.bb +7 -1

index be90aba..91552a9 100644
@@ -51,7 +51,7 @@ do_install_append_class-native() {
 PACKAGES += "info info-doc"
 
 FILES_info = "${bindir}/info ${bindir}/infokey ${bindir}/install-info"
-FILES_info-doc = "${infodir}/info.info ${infodir}/dir ${infodir}/info-*.info \
+FILES_info-doc = "${infodir}/info.info* ${infodir}/dir ${infodir}/info-*.info* \
                   ${mandir}/man1/info.1* ${mandir}/man5/info.5* \
                   ${mandir}/man1/infokey.1* ${mandir}/man1/install-info.1*"

@@ -61,4 +61,10 @@ FILES_${PN}-doc = "${infodir}/texinfo* \
                    ${datadir}/${tex_texinfo} \
                    ${mandir}/man1 ${mandir}/man5"
 
+pkg_postinst_info () {
+    for f in `find $D/${infodir} -name "*.info*"`; do
+        install-info $f $D/${infodir}/dir
+    done
+}
+
 BBCLASSEXTEND = "native"

Signature · 1 line

1.9.1

Burton, Ross

2014-09-23 14:52 UTC #6 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

meta/recipes-extended/texinfo/texinfo_5.2.bb FILES_${PN}-doc = "${infodir}/texinfo* \ · line 68

Quoted 5 lines · Hongxu Jia

meta/recipes-extended/texinfo/texinfo_5.2.bb +5 -0

+pkg_postinst_info () {
+    for f in `find $D/${infodir} -name "*.info*"`; do
+        install-info $f $D/${infodir}/dir
+    done
+}

For future, this seems like the sort of thing we need a class for. There is a texinfo.bbclass now...

Ross

Richard Purdie

2014-09-23 20:37 UTC #7 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

In reply to Hongxu Jia

Quoted 36 lines · Hongxu Jia

In texinfo, we didn't have info/dir entries, which caused info not work.
...

root@qemux86:~# info info
info: dir: No such file or directory
...
Invoke install-info to create info/dir entries in pkg_postinst_info.

Also fix a QA Issue
...
ERROR: QA Issue: texinfo: Files/directories were installed but not shipped
/usr/share/info/info.info.bz2 /usr/share/info/info-stnd.info.bz2 [installed-vs-shipped]
...

[YOCTO #6751]

Signed-off-by Hongxu Jia <[email protected]>

---
 meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

meta/recipes-extended/texinfo/texinfo_5.2.bb +1 -1

index be90aba..91552a9 100644
@@ -51,7 +51,7 @@ do_install_append_class-native() {
 PACKAGES += "info info-doc"
 
 FILES_info = "${bindir}/info ${bindir}/infokey ${bindir}/install-info"
-FILES_info-doc = "${infodir}/info.info ${infodir}/dir ${infodir}/info-*.info \
+FILES_info-doc = "${infodir}/info.info* ${infodir}/dir ${infodir}/info-*.info* \
                   ${mandir}/man1/info.1* ${mandir}/man5/info.5* \
                   ${mandir}/man1/infokey.1* ${mandir}/man1/install-info.1*"

meta/recipes-extended/texinfo/texinfo_5.2.bb FILES_${PN}-doc = "${infodir}/texinfo* \ · line 70

Quoted 10 lines

meta/recipes-extended/texinfo/texinfo_5.2.bb +6 -0

@@ -61,4 +61,10 @@ FILES_${PN}-doc = "${infodir}/texinfo* \
                    ${datadir}/${tex_texinfo} \
                    ${mandir}/man1 ${mandir}/man5"
 
+pkg_postinst_info () {
+    for f in `find $D/${infodir} -name "*.info*"`; do
+        install-info $f $D/${infodir}/dir
+    done
+}
+
 BBCLASSEXTEND = "native"

I think there are a few issues here. Firstly, this postinst can run at build time which means there is a dependency on texinfo-native. I guess if this fails, it will just run on target so its probably "harmless" but not deterministic. Secondly, anything installing an info directory really needs to update the dir files with its own postinst. For optimal handling at rootfs time, that will need trigger/intercept support.

So this probably can merge as it improves things however we need some enhacement bugs opening related to this, we may want to force the postinst to run on target for now so things are deterministic too.

Cheers,

Richard

Burton, Ross

2014-09-23 20:41 UTC #8 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

Quoted 10 lines · Richard Purdie

I think there are a few issues here. Firstly, this postinst can run at build time which means there is a dependency on texinfo-native. I guess if this fails, it will just run on target so its probably "harmless" but not deterministic. Secondly, anything installing an info directory really needs to update the dir files with its own postinst. For optimal handling at rootfs time, that will need trigger/intercept support.
So this probably can merge as it improves things however we need some enhacement bugs opening related to this, we may want to force the postinst to run on target for now so things are deterministic too.

There's a texinfo.bbclass which all recipes installing info files are probably inheriting anyway, so moving the postinst there should solve the race on the directory. That class says that texinfo-native is assume-provided but you'll want to check that usual configurations ship install-info alongside makeinfo on the distributions we support.

Ross

Hongxu Jia

author

2014-09-24 09:16 UTC #9 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

Quoted 9 lines · Ross

Quoted 7 lines · Richard Purdie
I think there are a few issues here. Firstly, this postinst can run at build time which means there is a dependency on texinfo-native. I guess if this fails, it will just run on target so its probably "harmless" but not deterministic. Secondly, anything installing an info directory really needs to update the dir files with its own postinst. For optimal handling at rootfs time, that will need trigger/intercept support.

Quoted 8 lines

Quoted 3 lines
So this probably can merge as it improves things however we need some enhacement bugs opening related to this, we may want to force the postinst to run on target for now so things are deterministic too.
There's a texinfo.bbclass which all recipes installing info files are probably inheriting anyway, so moving the postinst there should solve the race on the directory. That class says that texinfo-native is assume-provided but you'll want to check that usual configurations ship install-info alongside makeinfo on the distributions we support.

Hi Ross,

I am afraid moving the postinst to texinfo.bbclass will cause some potential issues:

- It will be invoked many times if multiple packages installed.

- We could not use a generic way to name pkg_postinst. Such as texinfo recipe, we need pkg_postinst_info.

I will refer what Ubuntu/Debian did, add postinst/postrm to info packages.

//Hongxu

Quoted 1 line

Ross

Burton, Ross

2014-09-24 10:50 UTC #10 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

Quoted 4 lines · Hongxu Jia

I am afraid moving the postinst to texinfo.bbclass will cause some potential issues:
- It will be invoked many times if multiple packages installed.

Sure, we could make it a trigger in the future.

Quoted 2 lines

- We could not use a generic way to name pkg_postinst. Such as texinfo recipe, we need pkg_postinst_info.

Many classes extend the postinst scripts, so this isn't a problem.

Ross

Hongxu Jia

author

2014-09-24 08:58 UTC #11 lore raw

Re: [PATCH 3/3] texinfo: fix info not work

In reply to Richard Purdie

Quoted 6 lines · Richard Purdie

I think there are a few issues here. Firstly, this postinst can run at build time which means there is a dependency on texinfo-native. I guess if this fails, it will just run on target so its probably "harmless" but not deterministic. Secondly, anything installing an info directory really needs to update the dir files with its own postinst. For optimal handling at rootfs time, that will need trigger/intercept support.

Hi Richard,

Yes, you are right, I will refer Ubuntu to add a script to postinst_intercept to do that, run it after all packages have been installed at build time.

And about trigger, but I have no idea about how to implement it. It seems we support pkg_postinst/pkg_preinst but don't support pkg_triggers, Refer triggers in install-info_5.2.0.dfsg.1-4_amd64.deb, we got:
...
interest-noawait /usr/share/info
...

V2 incoming

//Hongxu

Quoted 9 lines

So this probably can merge as it improves things however we need some enhacement bugs opening related to this, we may want to force the postinst to run on target for now so things are deterministic too.
Cheers,
Richard

Mark Hatle

2014-09-23 15:16 UTC #12 lore raw

In reply to Hongxu Jia

Quoted 8 lines · Hongxu Jia

Test Case:
vim local.conf
..
IMAGE_INSTALL_append = " man man-doc texinfo texinfo-doc info info-doc" INHERIT += "compress_doc"
DOC_COMPRESS = "xz" or DOC_COMPRESS = "bz2"
..

Patch 1/3 the busybox change. I'm not sure we really want to make that change as it will increase the size of busybox.

(following is me thinking aloud and hopefully someone will have a suggest on how best to unify this..)

I'm wondering if instead of forcing the change for everyone via the busybox change, if it makes sense to add a image validation routine or something that verifies if the DOC_COMPRESS is set, that the corresponding tool was installed into the target environment (along w/ man, info or other appropriate items.. and the pkgs-docs was set as well.. if any of those are missing then a warning should probably be presented.)

I'm sure there is probably a better way to set this via a distribution configuration switch or something as well.. maybe something like?

DISTRO += "man info"

That could then force the man and info systems to add a dependency on bz2 (or xz or whatever else was set in DOC_COMPRESS).. and also add man, info and the associated man and info pages to the image itself.

(As I see it, DOC_COMPRESS is really a distribution wide setting, so we should treat it as such...)

--Mark

Quoted 32 lines

bitbake core-image-minimal
runqemu qemux86
On target,
...
root@qemux86:~# man man
root@qemux86:~# info info
...
//Hongxu
The following changes since commit afdbe3112b940987f3f3cddcb32c91a5e17a297d:
   bitbake: bitbake-worker: Fix bitbake -n (2014-09-22 15:54:40 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/fix-doc
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-doc
Hongxu Jia (3):
busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default
man: fix not support xz/bz2 compression
texinfo: fix info not work
  meta/recipes-core/busybox/busybox/defconfig  | 6 +++---
  meta/recipes-extended/man/man/man.conf       | 3 ++-
  meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
  3 files changed, 12 insertions(+), 5 deletions(-)