state.bbclass: check before open the manifest

PATCH In review Robert Yang 3 participants 8 messages Aug 29, 2013 – Sep 2, 2013

lore

Share & tools

lore

Robert Yang

author

2013-08-29 13:13 UTC #0 lore raw

cover state.bbclass: check before open the manifest

The following changes since commit 5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f:

buildtools-tarball: Add python-pkgutil (2013-08-29 00:23:20 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/sstate

Robert Yang (1):
sstate.bbclass: check before open the manifest

 meta/classes/sstate.bbclass |    3 +++
 1 file changed, 3 insertions(+)

Signature · 1 line

1.7.10.4

Robert Yang

author

2013-08-29 13:13 UTC #1 lore raw

PATCH 1/1 sstate.bbclass: check before open the manifest

The python stack trace would be printed if we:

$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81
[snip]
 *** 0004:    mfile = open(manifest)
     0005:    entries = mfile.readlines()
     0006:    mfile.close()
     0007:
     0008:    for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]

This because the make-3.81 and make-3.82 are being built at the same time, the manifest may have been removed by make-3.82, but make-3.81 still opens it, so the error happens.

Check before open the manifest would fix the problem.

[YOCTO #5067]

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/sstate.bbclass |    3 +++
 1 file changed, 3 insertions(+)

meta/classes/sstate.bbclass +3 -0

index c86f393..6e71086 100644
@@ -302,6 +302,9 @@ def sstate_clean_cachefiles(d):
 def sstate_clean_manifest(manifest, d):
     import oe.path
 
+    if not os.path.exists(manifest):
+        return True
+
     mfile = open(manifest)
     entries = mfile.readlines()
     mfile.close()

Signature · 1 line

1.7.10.4

Richard Purdie

2013-08-29 17:00 UTC #2 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Quoted 5 lines · Robert Yang

The python stack trace would be printed if we:
$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81

This is at best hiding the real problem. Why is bitbake building make-3.82 when you asked for make-3.81?

I tried this here and also see the warning:

ERROR: Multiple .bb files are due to be built which each provide make (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
This usually means one provides something the other doesn't and should.

so bitbake is basically telling you there is a problem already. So there are two issues:

a) Why is make 3.82 being built?
b) If multiple identical PNs are being built we probably should hard error out since its not supported in the slightest. The sstate race you mention is the least of the problems :(.

Cheers,

Richard

Robert Yang

author

2013-08-30 01:24 UTC #3 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Quoted 10 lines · Richard Purdie

Quoted 5 lines · Robert Yang
The python stack trace would be printed if we:
$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81
This is at best hiding the real problem. Why is bitbake building make-3.82 when you asked for make-3.81?

Quoted 9 lines

I tried this here and also see the warning:
ERROR: Multiple .bb files are due to be built which each provide make (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
This usually means one provides something the other doesn't and should.
so bitbake is basically telling you there is a problem already. So there are two issues:

What I thought was that let the user know the normal error, but we can do more to fix it as you pointed out.

Quoted 1 line

a) Why is make 3.82 being built?

Ah, yes, that's problem, I will do more investigation.

Quoted 4 lines

b) If multiple identical PNs are being built we probably should hard error out since its not supported in the slightest. The sstate race you mention is the least of the problems :(.

OK, I will try to fix such a case:

bitbake make-3.81 make-3.82

// Robert

Quoted 6 lines

Cheers,
Richard

Richard Purdie

2013-08-30 15:29 UTC #4 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Quoted 29 lines · Robert Yang

Quoted 10 lines · Richard Purdie
Quoted 5 lines · Robert Yang
The python stack trace would be printed if we:
$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81
This is at best hiding the real problem. Why is bitbake building make-3.82 when you asked for make-3.81?
Quoted 9 lines
I tried this here and also see the warning:
ERROR: Multiple .bb files are due to be built which each provide make (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
This usually means one provides something the other doesn't and should.
so bitbake is basically telling you there is a problem already. So there are two issues:
What I thought was that let the user know the normal error, but we can do more to fix it as you pointed out.
Quoted 1 line
a) Why is make 3.82 being built?
Ah, yes, that's problem, I will do more investigation.

Quoted 8 lines

Quoted 4 lines
b) If multiple identical PNs are being built we probably should hard error out since its not supported in the slightest. The sstate race you mention is the least of the problems :(.
OK, I will try to fix such a case:
bitbake make-3.81 make-3.82

I think this should just give an error.

Cheers,

Richard

Robert Yang

author

2013-08-31 10:51 UTC #5 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Hi RP,

Thanks, I've made a draft patch for fixing why make-3.82 is being built, the problem is that when we run "bitbake make-3.81":
- For 'make' itself, it will build make_3.81.bb
- But for make-dev/dbg, it will build make_3.82.bb since there is no PREFERRED_VERSION and it will use the highest version.

This draft patch sets the PREFERRED_VERSION for the pkg and will fix the problem, I will send it to bitbake-devel later.

bitbake/lib/bb/taskdata.py +16 -0

index 58fe199..fc8ee6d 100644
@@ -429,6 +429,22 @@ class TaskData:
             return
 
         all_p = dataCache.providers[item]
+        logger.warn("pppreferred: %s" % dataCache.preferred)
+
+        # Check whether item has the one of following formats and set the
+        # PREFERRED_VERSION if it does:
+        # - <pn>-<pv>
+        # - <pn>-<pv>-<pr> (when no PE)
+        # - <pn>_<pe>-<pv>-<pr> (when PE)
+        for fn in all_p:
+            pn = dataCache.pkg_fn[fn]
+            pe = dataCache.pkg_pepvpr[fn][0]
+            pv = dataCache.pkg_pepvpr[fn][1]
+            pr = dataCache.pkg_pepvpr[fn][2]
+            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s" % (pn, pv, pr)):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
+            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
 
         eligible, foundUnique = bb.providers.filterProviders(all_p, item,

cfgData, dataCache)
eligible = [p for p in eligible if not self.getfn_id(p) in
self.failed_fnids]

// Robert

Quoted 47 lines · Richard Purdie

Quoted 29 lines · Robert Yang
Quoted 10 lines · Richard Purdie
Quoted 5 lines · Robert Yang
The python stack trace would be printed if we:
$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81
This is at best hiding the real problem. Why is bitbake building make-3.82 when you asked for make-3.81?
Quoted 9 lines
I tried this here and also see the warning:
ERROR: Multiple .bb files are due to be built which each provide make (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
This usually means one provides something the other doesn't and should.
so bitbake is basically telling you there is a problem already. So there are two issues:
What I thought was that let the user know the normal error, but we can do more to fix it as you pointed out.
Quoted 1 line
a) Why is make 3.82 being built?
Ah, yes, that's problem, I will do more investigation.
Quoted 8 lines
Quoted 4 lines
b) If multiple identical PNs are being built we probably should hard error out since its not supported in the slightest. The sstate race you mention is the least of the problems :(.
OK, I will try to fix such a case:
bitbake make-3.81 make-3.82
I think this should just give an error.
Cheers,
Richard

Otavio Salvador

2013-08-31 22:07 UTC #6 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Quoted 11 lines · Robert Yang

Hi RP,
Thanks, I've made a draft patch for fixing why make-3.82 is being built, the problem is that when we run "bitbake make-3.81":
- For 'make' itself, it will build make_3.81.bb
- But for make-dev/dbg, it will build make_3.82.bb since there is no PREFERRED_VERSION and it will use the highest version.
This draft patch sets the PREFERRED_VERSION for the pkg and will fix the problem, I will send it to bitbake-devel later.

Nice discovery :-) This does seem to address a core issue.

Quoted 22 lines

bitbake/lib/bb/taskdata.py +13 -0

index 58fe199..fc8ee6d 100644
@@ -429,6 +429,22 @@ class TaskData:
             return
 
         all_p = dataCache.providers[item]
+        logger.warn("pppreferred: %s" % dataCache.preferred)
+
+        # Check whether item has the one of following formats and set the
+        # PREFERRED_VERSION if it does:
+        # - <pn>-<pv>
+        # - <pn>-<pv>-<pr> (when no PE)
+        # - <pn>_<pe>-<pv>-<pr> (when PE)
+        for fn in all_p:
+            pn = dataCache.pkg_fn[fn]
+            pe = dataCache.pkg_pepvpr[fn][0]
+            pv = dataCache.pkg_pepvpr[fn][1]
+            pr = dataCache.pkg_pepvpr[fn][2]
+            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s"

% (pn, pv, pr)):

bitbake/lib/bb/taskdata.py class TaskData: · line 447

Quoted 3 lines

bitbake/lib/bb/taskdata.py +3 -0

+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
+            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)

I understand the problem here but maybe this should be checked in bitbake and avoid iterate in all packages?

Signature · 3 lines

Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Robert Yang

author

2013-09-02 02:00 UTC #7 lore raw

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

Quoted 35 lines · Otavio Salvador

Quoted 11 lines · Robert Yang
Hi RP,
Thanks, I've made a draft patch for fixing why make-3.82 is being built, the problem is that when we run "bitbake make-3.81":
- For 'make' itself, it will build make_3.81.bb
- But for make-dev/dbg, it will build make_3.82.bb since there is no PREFERRED_VERSION and it will use the highest version.
This draft patch sets the PREFERRED_VERSION for the pkg and will fix the problem, I will send it to bitbake-devel later.
Nice discovery :-) This does seem to address a core issue.
Quoted 20 lines
bitbake/lib/bb/taskdata.py +12 -0
index 58fe199..fc8ee6d 100644
@@ -429,6 +429,22 @@ class TaskData:
              return
 
          all_p = dataCache.providers[item]
+        logger.warn("pppreferred: %s" % dataCache.preferred)
+
+        # Check whether item has the one of following formats and set the
+        # PREFERRED_VERSION if it does:
+        # - <pn>-<pv>
+        # - <pn>-<pv>-<pr> (when no PE)
+        # - <pn>_<pe>-<pv>-<pr> (when PE)
+        for fn in all_p:
+            pn = dataCache.pkg_fn[fn]
+            pe = dataCache.pkg_pepvpr[fn][0]
+            pv = dataCache.pkg_pepvpr[fn][1]
+            pr = dataCache.pkg_pepvpr[fn][2]

Quoted 9 lines

bitbake/lib/bb/taskdata.py class TaskData: · line 447
Quoted 5 lines
bitbake/lib/bb/taskdata.py +1 -0
+            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s"
% (pn, pv, pr)):
bitbake/lib/bb/taskdata.py +3 -0
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
+            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
I understand the problem here but maybe this should be checked in bitbake and avoid iterate in all packages?

Yes, you are right, it should be sent to bitbake-devel mailing list, and I'm trying to avoid iterating in all packages.

// Robert