Skip to contentSkip to conversation

toaster: Y9137 get packages from sstate

PATCH In review brian avery 3 participants 5 messages Mar 8, 2016 – Mar 10, 2016

lore
Share & tools
lore

brian avery

author
#0 lore raw

cover toaster: Y9137 get packages from sstate

This addresses the missing packages part of Y9137. This patch allows us to see
packages that came from a setscene.
Testing it is a little hard:
1) You can add the following to your local.conf in a toaster dir:
SSTATE_MIRRORS="file://.* http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master/sstate-cache/PATH"
This might also work:
SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
and build dropbear in a LOCAL project. You should see ~322 pkgs built.

The hard way to test this is to:
1) start toaster
2) build dropbear as a local project
3) stop toaster
4) rm everything except sstate-cache/ and conf/
5) restart toaster
6) build dropbearas a local project. it should come from sstate and downloads should remain empty.

Note, there is still a big difference between the pkgs we get from a build ~2844 and the ones we get from setscening. The setscening doesn't package:
*locales
glibc-charmap*
glibc-gconv*
perl-module*
ncurses-lib*
as they seem to be unnecessary in order to be able to package dropbear.

---
This does not address the need to change "packages built" to something more generic like packages in the UI.

-b
an Intel employee

The following changes since commit 325a35c43df83d4268fc54a8955ee785f6a1f6fe:

toaster: update the meta-yocto toaster configuration file (2016-03-07 21:36:03 -0800)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/toaster/getPkgsFromSstate-Y9137V2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/getPkgsFromSstate-Y9137V2

brian avery (1):
toaster: show packages that were setscened into existence too

 meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

--
1.9.1

author
#1 lore raw

PATCH 1/1 toaster: show packages that were setscened into existence too

We were previously ignoring pkgs that came in from a setscene (like from an sstate mirror). With this patch we can use pkgs than come from sstate for image customisation as well.
[YOCTO #9137]

Signed-off-by brian avery <[email protected]>
---
 meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
meta/classes/toaster.bbclass +23 -2
index 51a4c74..4afaf1c 100644
@@ -112,6 +112,26 @@ def _toaster_load_pkgdatafile(dirpath, filepath):
                 pass    # ignore lines without valid key: value pairs
     return pkgdata
 
+python toaster_package_dumpdata_setscene() {
+    """
+    Dumps the data created by package_setscene
+    """
+    # replicate variables from the package.bbclass
+    packages = d.getVar('PACKAGES', True)
+    pkgdatadir = d.getVar('PKGDATA_DIR', True)
+    # scan and send data for each package
+    lpkgdata = {}
+    for pkg in packages.split():
+        try:
+          lpkgdata = _toaster_load_pkgdatafile(pkgdatadir + "/runtime/", pkg)
+        except:
+          # these are typically foo-locale which actually point into foo-locale-<language> in runtime-rprovides
+          bb.note("toaster_package_dumpdata_setscene: failed to load pkg information for: %s:%s"%(pkg,sys.exc_info()[0]))
+        # Fire an event containing the pkg data
+        bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
+
+}
+
 
 python toaster_package_dumpdata() {
     """

@@ -120,8 +140,6 @@ python toaster_package_dumpdata() {
     # replicate variables from the package.bbclass
 
     packages = d.getVar('PACKAGES', True)
-    pkgdest = d.getVar('PKGDEST', True)
-
     pkgdatadir = d.getVar('PKGDESTWORK', True)
 
     # scan and send data for each package

@@ -362,6 +380,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
 addhandler toaster_buildhistory_dump
 toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 
+do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata_setscene "
+do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata_setscene "
+
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
Signature · 1 line
1.9.1
#3 lore raw
In reply to brian avery
Quoted 3 lines · On 08/03/2016 05:50, "[email protected] on behalf of brian…

This addresses the missing packages part of Y9137. This patch allows us to see
packages that came from a setscene.

This works for me, which is great. But in my innocence, I somehow assumed it would also perform the miracle in those builds where no sstate is used (the task output from a previous build is used. We call those tasks 'Prebuilt' in Toaster, although I am not sure how they are called in the non-Toaster world). The kind of build you do when, after step 6 below, you rebuild dropbear yet again.

This might be completely unrealistic though.

Cheers

Belén

Quoted 63 lines

Testing it is a little hard:
1) You can add the following to your local.conf in a toaster dir:
SSTATE_MIRRORS="file://.*
http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master
/sstate-cache/PATH"
This might also work:
SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
and build dropbear in a LOCAL project. You should see ~322 pkgs built.

The hard way to test this is to:
1) start toaster
2) build dropbear as a local project
3) stop toaster
4) rm everything except sstate-cache/ and conf/
5) restart toaster
6) build dropbearas a local project. it should come from sstate and downloads should remain empty.

Note, there is still a big difference between the pkgs we get from a build ~2844 and the ones we get from setscening.
The setscening doesn't package:
*locales
glibc-charmap*
glibc-gconv*
perl-module*
ncurses-lib*
as they seem to be unnecessary in order to be able to package dropbear.

---
This does not address the need to change "packages built" to something more generic like packages in the UI.

-b
an Intel employee

The following changes since commit
325a35c43df83d4268fc54a8955ee785f6a1f6fe:

toaster: update the meta-yocto toaster configuration file (2016-03-07
21:36:03 -0800)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
bavery/toaster/getPkgsFromSstate-Y9137V2

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/ge
tPkgsFromSstate-Y9137V2

brian avery (1):
toaster: show packages that were setscened into existence too

meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--

1 file changed, 23 insertions(+), 2 deletions(-)

--
1.9.1

Signature · 4 lines
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
author
#4 lore raw

in those cases, bitbake itself doesn't do anything, so the 0 is correct, albeit perhaps confusing. however, since it did do something once before the package is available for ic. My main fear was having packages be available but toaster being unable to add them into a CI. -b
an intel employee

Quoted 88 lines · Belen
Quoted 3 lines · On 08/03/2016 05:50, "[email protected] on behalf of brian…

This addresses the missing packages part of Y9137. This patch allows us to see
packages that came from a setscene.

This works for me, which is great. But in my innocence, I somehow assumed it would also perform the miracle in those builds where no sstate is used (the task output from a previous build is used. We call those tasks 'Prebuilt' in Toaster, although I am not sure how they are called in the non-Toaster world). The kind of build you do when, after step 6 below, you rebuild dropbear yet again.

This might be completely unrealistic though.

Cheers

Belén

Quoted 63 lines

Testing it is a little hard:
1) You can add the following to your local.conf in a toaster dir:
SSTATE_MIRRORS="file://.*
http://downloads.yoctoproject.org/tools/support/toaster/sstate-test/master
/sstate-cache/PATH"
This might also work:
SSTATE_MIRRORS="file://.* http://sstate.yoctoproject.org/dev/PATH"
and build dropbear in a LOCAL project. You should see ~322 pkgs built.

The hard way to test this is to:
1) start toaster
2) build dropbear as a local project
3) stop toaster
4) rm everything except sstate-cache/ and conf/
5) restart toaster
6) build dropbearas a local project. it should come from sstate and downloads should remain empty.

Note, there is still a big difference between the pkgs we get from a build ~2844 and the ones we get from setscening.
The setscening doesn't package:
*locales
glibc-charmap*
glibc-gconv*
perl-module*
ncurses-lib*
as they seem to be unnecessary in order to be able to package dropbear.

---
This does not address the need to change "packages built" to something more generic like packages in the UI.

-b
an Intel employee

The following changes since commit
325a35c43df83d4268fc54a8955ee785f6a1f6fe:

toaster: update the meta-yocto toaster configuration file (2016-03-07
21:36:03 -0800)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
bavery/toaster/getPkgsFromSstate-Y9137V2

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/ge
tPkgsFromSstate-Y9137V2

brian avery (1):
toaster: show packages that were setscened into existence too

meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--

1 file changed, 23 insertions(+), 2 deletions(-)

--
1.9.1
--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

#2 lore raw

Re: [PATCH 1/1] toaster: show packages that were setscened into existence too

In reply to brian avery

Thanks sent with some commit messages fixes to openembedded-core http://lists.openembedded.org/pipermail/openembedded-core/2016-March/118796.html and pushed to toaster-next

Quoted 60 lines · brian avery

We were previously ignoring pkgs that came in from a setscene (like from an sstate mirror). With this patch we can use pkgs than come from sstate for image customisation as well.
[YOCTO #9137]

Signed-off-by brian avery <[email protected]>
---
  meta/classes/toaster.bbclass | 25 +++++++++++++++++++++++--
  1 file changed, 23 insertions(+), 2 deletions(-)
meta/classes/toaster.bbclass +23 -2
index 51a4c74..4afaf1c 100644
@@ -112,6 +112,26 @@ def _toaster_load_pkgdatafile(dirpath, filepath):
                  pass    # ignore lines without valid key: value pairs
      return pkgdata
 
+python toaster_package_dumpdata_setscene() {
+    """
+    Dumps the data created by package_setscene
+    """
+    # replicate variables from the package.bbclass
+    packages = d.getVar('PACKAGES', True)
+    pkgdatadir = d.getVar('PKGDATA_DIR', True)
+    # scan and send data for each package
+    lpkgdata = {}
+    for pkg in packages.split():
+        try:
+          lpkgdata = _toaster_load_pkgdatafile(pkgdatadir + "/runtime/", pkg)
+        except:
+          # these are typically foo-locale which actually point into foo-locale-<language> in runtime-rprovides
+          bb.note("toaster_package_dumpdata_setscene: failed to load pkg information for: %s:%s"%(pkg,sys.exc_info()[0]))
+        # Fire an event containing the pkg data
+        bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
+
+}
+
 
  python toaster_package_dumpdata() {
      """

@@ -120,8 +140,6 @@ python toaster_package_dumpdata() {
      # replicate variables from the package.bbclass
 
      packages = d.getVar('PACKAGES', True)
-    pkgdest = d.getVar('PKGDEST', True)
-
      pkgdatadir = d.getVar('PKGDESTWORK', True)
 
      # scan and send data for each package

@@ -362,6 +380,9 @@ toaster_collect_task_stats[eventmask] = "bb.event.BuildCompleted bb.build.TaskSu
  addhandler toaster_buildhistory_dump
  toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 
+do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata_setscene "
+do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata_setscene "
+
  do_package[postfuncs] += "toaster_package_dumpdata "
  do_package[vardepsexclude] += "toaster_package_dumpdata "