Automatically generate package repos for rpm and deb [bug #1024]

From: Dexuan Cui <[email protected]>

This was made to address http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024.
Please comment.

This version moved the package_update_index_* into package_*.bbclass.

As to RPM, both package_update_index_rpm(creating solvedb) and createrepo(creating rpm-metadata) seems necessary. Mark, please comment on this.

As to DEB, I found currently in target, some important config files, like /etc/apt/sources.list, for deb/apt are missing. As a result, we can't install/remove package in target at all.
I'm trying to resolve this by add "deb http://192.168.7.1/deb/all/ ./" into the file, but later "apt-get update" failes -- it tries to find a file named Packages.gz, but the direcory only has Packages.bz2. Not sure what's wrong. Investigating.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: dcui/package-index
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/package-index

Thanks,
Dexuan Cui <[email protected]>
---

Dexuan Cui (1):
package-index.bb: add support for deb and rpm.

 meta/classes/package_deb.bbclass        |    9 +++++++++
 meta/classes/package_ipk.bbclass        |   12 ++++++++++++
 meta/classes/package_rpm.bbclass        |    7 +++++++
 meta/recipes-core/meta/package-index.bb |    8 --------
 4 files changed, 28 insertions(+), 8 deletions(-)

1.7.2

From: Dexuan Cui <[email protected]>

[YOCTO #1024]
Currently package-index.bb only supports ipk. This commit addes the support for rpm and deb, too.

------------------------------
How to generate and use repos:

1) run "bitbake package-index" after building some target, e.g., core-image-sato-sdk;

2) export ${DEPLOY_DIR_RPM}, ${DEPLOY_DIR_IPK} and ${DEPLOY_DIR_DEB} by a webserver on the host, assuming the host IP is 192.168.7.1, at
http://192.168.7.1/rpm
http://192.168.7.1/ipk
http://192.168.7.1/deb

3) inside the target, according to the packaging system (rpm, ipk or deb) used when we generate the target image, we can use different ways to manage packages:

3.1) RPM
run "zypper addrepo http://192.168.7.1/rpm main; zypper refresh" to retrieve info about the repo; next, we can use "zypper install/remove" to manage packages.

3.2) IPK
add the repo info into opkg config file, i.e., in /etc/opkg/arch.conf, we can add something like
"src i586 http://192.168.7.1/ipk/i586", and next, we run "opkg update" to make opkg update the list of available packages. And later, we can use "opkg install/remove" to manage packages.

3.3) DEB
Currently in target, some important config files, like
/etc/apt/sources.list, for deb/apt are missing. So we can't install/remove package in target at present.

---
 meta/classes/package_deb.bbclass        |    9 +++++++++
 meta/classes/package_ipk.bbclass        |   12 ++++++++++++
 meta/classes/package_rpm.bbclass        |    7 +++++++
 meta/recipes-core/meta/package-index.bb |    8 --------
 4 files changed, 28 insertions(+), 8 deletions(-)

index 4faeb4a..4cc0b69 100644
@@ -431,3 +431,12 @@ python do_package_write_deb () {
 do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
 addtask package_write_deb before do_package_write after do_package
 
+
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
+do_package_index[recrdeptask] += "package_update_index_deb"
+
+do_package_index() {
+   set -ex
+   package_update_index_deb
+   set +ex
+}

index 3c2472b..2a2d9c3 100644
@@ -386,3 +386,15 @@ python do_package_write_ipk () {
 }
 do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
 addtask package_write_ipk before do_package_write after do_package
+
+
+do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
+do_package_index[depends] += "opkg-native:do_populate_sysroot"
+do_package_index[recrdeptask] += "package_update_index_ipk"
+
+do_package_index() {
+   set -ex
+   package_update_index_ipk
+   set +ex
+}
+

index 70170d1..d9470d6 100644
@@ -814,3 +814,10 @@ python do_package_write_rpm () {
 do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
 addtask package_write_rpm before do_package_write after do_package
 
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+do_package_index() {
+    set -ex
+    package_update_index_rpm
+    createrepo "${DEPLOY_DIR_RPM}"
+    set +ex
+}

index 3c642cb..3f1caea 100644
@@ -19,14 +19,6 @@ do_package_write_deb[noexec] = "1"
 do_populate_sysroot[noexec] = "1"
 
 do_package_index[nostamp] = "1"
-do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
-do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
-do_package_index[depends] += "opkg-native:do_populate_sysroot"
 
-do_package_index() {
-	set -ex
-	package_update_index_ipk
-	set +ex
-}
 addtask do_package_index before do_build
 EXCLUDE_FROM_WORLD = "1"

1.7.2

Sorry, I didn't use the recrdeptask flag and I guess I still don't quite understand it.

Here removing the do_package_index() I will get such an ERROR:
ERROR: Task do_package_index from /distro/dcui/pc1/meta/recipes-core/meta/package-index.bb seems to be empty?!########################## | ETA: 00:00:00
ERROR: Error parsing /distro/dcui/pc1/meta/recipes-core/meta/package-index.bb: md5() argument 1 must be string or read-only buffer, not None | ETA: 00:00:00

About the recrdeptask flag: "These are specified with the 'recrdeptask' flag and is used signify the task(s) of each RDEPENDS which must have completed before that task can be executed. It applies recursively so also, the RDEPENDS of each item in the original RDEPENDS must be met and so on. It also runs all DEPENDS first too". I even added RDEPENDS += " package-index" in package-index.bb, but no change.

I'm trying to figure this out. Please point out my issue once this is obvious to you. :-)

BTW: using the recrdeptask method, we have no chance to do "set -ex" and as a result we lose some debug info in the log file. Is this ok?

Yes, I should have added a new function including package_update_index_rpm and createrepo "${DEPLOY_DIR_RPM}". and set do_package_index[recrdeptask] to it.

Thanks,
-- Dexuan

Hi Dexuan,

When I read Saul's original emails I thought this approach would work although it wasn't the first one that came to my mind.

Looking at the code you have, the problem is you have nowhere you can call the "addtask" for the individual tasks and we don't want to the overhead system wide for every recipe as it makes no sense.

I'd therefore like to suggest we tweak this approach a little. Each package_xxx.bbclass defines a function which updates the index for that packaging backend:

package_ipk.bbclass:

PACKAGEINDEXES += "package_index_ipk();" PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot" PACKAGEINDEXDEPS += "opkg-native:do_populate_sysroot" package_index_ipk[dirs] = "${DEPLOY_DIR_IPK}"

package_index_ipk() {
set -ex
package_update_index_ipk
set +ex
}

The key is the addition to the PACKAGEINDEXES variable and its dependencies in PACKAGEINDEXDEPS.

In package-index.bb we can then just have:

do_package_index[depends] += "${PACKAGEINDEXDEPS}"

do_package_index() {
	${PACKAGEINDEXES}
}

and these variables should always have the things we need in them depending on the package backends available.

Ideally we should cover the case there is no package backend enabled and just return too rather than error (or bbwarn "No package backends enabled, doing nothing").

Cheers,

Richard

Yes, I realized this later.

I agree. I've actually made a patch in branch dcui/package-index, but the below implementation you suggested is obviously much better! :-) Thank you very much for the detailed suggestion!

Oh, I didn't realize this... I'll remember to handle this, too.

Thanks,
-- Dexuan