Automatically generate package repos for rpm and deb [bug #1024]

From: Dexuan Cui <[email protected]>

This was made to address http://bugzilla.yoctoproject.org/show_bug.cgi?id=1024.
Please comment.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: dcui/master
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dcui/master

Thanks,
Dexuan Cui <[email protected]>
---

Dexuan Cui (1):
package-index-{ipk, rpm, deb}.bb: add the package-index for rpm and
deb

 meta/recipes-core/meta/package-index-deb.bb        |   10 ++++++++++
 meta/recipes-core/meta/package-index-ipk.bb        |   11 +++++++++++
 meta/recipes-core/meta/package-index-rpm.bb        |   10 ++++++++++
 .../meta/{package-index.bb => package-index.inc}   |    8 --------
 4 files changed, 31 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/meta/package-index-deb.bb
 create mode 100644 meta/recipes-core/meta/package-index-ipk.bb
 create mode 100644 meta/recipes-core/meta/package-index-rpm.bb
 rename meta/recipes-core/meta/{package-index.bb => package-index.inc} (69%)

1.7.2

From: Dexuan Cui <[email protected]>

[YOCTO #1024]

The commit renamed package-index.bb to package-index.inc and added 3 bb files for ipk, rpm and deb.

-------------------------------

How to generate and use repo:

RPM:
1) run "bitbake package-index-rpm" after building some target;
2) export the ${DEPLOY_DIR_RPM} by a webserver on the host (let's assume
the IP is 192.168.7.1) at http://192.168.7.1/rpm;
3) inside the target, run "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
to retrieve info about the repo; next, we can use zypper install/remove to manage packages.

IPK:
1) run "bitbake package-index-ipk" after building some target;
2) export the ${DEPLOY_DIR_IPK} by a webserver on the host (let's assume
the IP is 192.168.7.1) at http://192.168.7.1/ipk;
3) inside the target, add the repo info into opkg config file, i.e.,
in /etc/opkg/arch.conf, we can add something like
src i586 http://192.168.7.1/ipk/i586, and next we run "opkg update" to make opkg update the list of available packages, and next, we can use opkg install/remove to manage packages.

DEB: To be added.

---
 meta/recipes-core/meta/package-index-deb.bb        |   10 ++++++++++
 meta/recipes-core/meta/package-index-ipk.bb        |   11 +++++++++++
 meta/recipes-core/meta/package-index-rpm.bb        |   10 ++++++++++
 .../meta/{package-index.bb => package-index.inc}   |    8 --------
 4 files changed, 31 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/meta/package-index-deb.bb
 create mode 100644 meta/recipes-core/meta/package-index-ipk.bb
 create mode 100644 meta/recipes-core/meta/package-index-rpm.bb
 rename meta/recipes-core/meta/{package-index.bb => package-index.inc} (69%)

new file mode 100644
index 0000000..db52e2a
@@ -0,0 +1,10 @@
+require package-index.inc
+
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    package_update_index_deb
+    set +ex
+}
+

new file mode 100644
index 0000000..5eddea1
@@ -0,0 +1,11 @@
+require package-index.inc
+
+do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
+do_package_index[depends] += "opkg-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    package_update_index_ipk
+    set +ex
+}
+

new file mode 100644
index 0000000..605a638
@@ -0,0 +1,10 @@
+require package-index.inc
+
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+
+do_package_index() {
+    set -ex
+    createrepo "${DEPLOY_DIR_RPM}"
+    set +ex
+}
+

similarity index 69%
rename from meta/recipes-core/meta/package-index.bb
rename to meta/recipes-core/meta/package-index.inc
index 3c642cb..3f1caea 100644
@@ -19,14 +19,6 @@ do_package_write_deb[noexec] = "1"
 do_populate_sysroot[noexec] = "1"
 
 do_package_index[nostamp] = "1"
-do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
-do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
-do_package_index[depends] += "opkg-native:do_populate_sysroot"
 
-do_package_index() {
-	set -ex
-	package_update_index_ipk
-	set +ex
-}
 addtask do_package_index before do_build
 EXCLUDE_FROM_WORLD = "1"

1.7.2

This is an improved and simple version that keeps 1 package-index recipe and updates the 3 kinds of packages at the same time:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=dcui/package-index&id=fb26e81dcb6e27e2908294e406a3011baed0120d
This avoids forcing the user to remember which package type they enabled when running "bitbake package-index". (I also past the new version at the end of this mail for easy reviewing)

Thanks!
-- Dexuan

commit fb26e81dcb6e27e2908294e406a3011baed0120d

Author: Dexuan Cui <[email protected]>
Date:   Thu May 12 19:15:59 2011 +0800

package-index.bb: also add the support for rpm/deb

[YOCTO #1024]

-------------------------------
(I'll make a new patch to add the below description to the manual).

How to generate and use repos:

1) run "bitbake package-index" after building some target;
2) export ${DEPLOY_DIR_RPM} and ${DEPLOY_DIR_IPK} by a webserver on the host (let's assume the host IP is 192.168.7.1) at
http://192.168.7.1/rpm
http://192.168.7.1/ipk
3) inside the target, according to the packaging system (ipk, rpm, deb) used when we generate the target image, we can use different ways to manage packages:
3.1) RPM: "zypper addrepo http://192.168.7.1/rpm main; zypper refresh"
to retrieve info about the repo; next, we can use zypper install/remove to manage packages.
3.2) IPK: add the repo info into opkg config file, i.e., in
/etc/opkg/arch.conf, we can add something like
src i586 http://192.168.7.1/ipk/i586, and next we run "opkg update" to make opkg update the list of available packages, and next we can use opkg install/remove to manage packages.
3.3) DEB:(To be added)

index 3c642cb..969430b 100644
@@ -22,10 +22,14 @@ do_package_index[nostamp] = "1"
 do_package_index[dirs] = "${DEPLOY_DIR_IPK}"
 do_package_index[depends] += "opkg-utils-native:do_populate_sysroot"
 do_package_index[depends] += "opkg-native:do_populate_sysroot"
+do_package_index[depends] += "createrepo-native:do_populate_sysroot"
+do_package_index[depends] += "dpkg-native:do_populate_sysroot apt-native:do_populate_sysroot"
 
 do_package_index() {
    set -ex
    package_update_index_ipk
+   createrepo "${DEPLOY_DIR_RPM}"
+   package_update_index_deb
    set +ex
 }
 addtask do_package_index before do_build

Dexuan,

I am not sure that this is the correct approach. I think that you should look at how the packages get generated based on which package classes are included via PACKAGE_CLASSES, then in each of those classes set it up so that when the package-index task is called you know which indexer to run. This would be moving what you have in the package-index-<packager>.bb files to their respective package_<packager>.bbclass files and use

do_package_index[recrdeptask] += package_index_<packager>

in the case of ipk it would be in package_ipk.bbclass

do_package_index[recrdeptask] += "package_update_index_ipk"

You will also need to move the do_package_index[depends]

Another thing to verify is the current package_update_index_* correct for the different package types? Looking at package_update_index_rpm, I am not sure since it does not call createrepo, there might be a reason for this that Mark H or Qing can comment on.

Hope that made sense, I am sure Richard will correct me if I got something wrong!

Sau!

OK, thanks very much for the detailed suggestion! I'll change to this better method.

My understanding about package_update_index_rpm vs. createrepo is:
both can be used to generate rpm repo;
package_update_index_rpm generates the metadata about the packages in solvedb to have rpm install the packages into target rootfs. This is the current method used in do_rootfs for rpm. This is complex and the generated repo is not suitable to be exported (e.g., via http) for general use; createrepo generated the metadata in .xml files and the generated repo can be easily exported via http and can be used easily by standard zypper (and yum) tools.
So in package-index.bb, I adopt createrepo for rpm.

Let me Cc Qing and Mark for comments about this.

Thanks!
-- Dexuan