Skip to contentSkip to conversation

bitbake-whatchanged: print what is about to happen

PATCH Superseded Robert Yang 3 participants 10 messages Aug 7, 2012 – Aug 8, 2012

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover bitbake-whatchanged: print what is about to happen

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/whatchanged
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Robert Yang (1):
bitbake-whatchanged: print what is about to happen

 bitbake/bin/bitbake-whatchanged | 334 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged
Signature · 1 line
1.7.11.2
#1 lore raw

PATCH 1/1 bitbake-whatchanged: print what is about to happen

* Contents:
- Summary
- Usage
- Implementation summary
- Output
- TODO

* Summary:
This is used for printing out what is about to happen between the current and last builds, for example:

  $ bitbake core-image-sato
  # Edit some recipes
  $ bitbake-whatchanged core-image-sato

The changes will be printed.

* Usage:
bitbake-whatchanged [[opts] recipe]

* Implementation summary:
- Move the tmp/stamps to tmp/stamps.old
- Use the "bitbake -S recipe" to generate the new stamps

- When "the -v" (verbose) is not specified:

Quoted 2 lines

Figure out the newly added tasks
Figure out the PV (including PE) and PR changeds "PN", I think

that we the tasks details here, since all the task will change when PV or PR changes.
> The left tasks are the ones that the "DEPENDS"(Or dependency might
be better?) changed ones.

- When "-v" is specified:

Quoted 2 lines

Figure out the newly added tasks
Use bb.siggen.compare_sigfiles to figure out the details

* Output, for example (between core-image-sato and meta-toolchain-sdk, and with recipes upgraded):
- without "-v":

    Figuring out the TMPDIR ...
    Moving the stamps to stamps.old ...
    Generating the new stamps ... (need several minutes)
    === Newly added tasks: (807 tasks)
      libpthread-stubs-nativesdk: do_install do_build do_fetch do_package_write_rpm [snip]
      telepathy-mission-control: do_unpack do_build do_package_write_rpm do_package [snip]
      [snip]

    === PV changed: (48 tasks)
      bison-native: 2.5 -> 2.5.1
      atk: 2.2.0 -> 2.4.0
      [snip]

    === PR changed: (276 tasks)
      gdk-pixbuf-native: r3 -> r4
      db-native: r7 -> r8
      [snip]

    === DEPENDS changed: (2522 tasks)
      speex: do_package_write_rpm do_configure do_package do_populate_sysroot [snip]
      libgpg-error-native: do_build do_package_write do_package do_package_write_rpm [snip]
      [snip]

    === Summary: (3653 changed, 1927 unchanged)
    Newly added: 807, PV changed: 48, PR changed: 276, Depends changed: 2522

    Removing the newly generated stamps ...
    Moving the stamps.old back to stamps ...

- with "-v": (No PV or PR changes summary)

    === Newly added tasks: (807 tasks)
      libpthread-stubs-nativesdk: do_install do_build do_fetch do_package_write_rpm [snip]
      telepathy-mission-control: do_unpack do_build do_package_write_rpm do_package [snip]
      [snip]

    === The verbose changes of libice:
    Dependency on task bash_4.2.bb.do_package_write was added with hash d03d35205f8c897b2d37c1ebf6249fbf
    Dependency on task bash_3.2.48.bb.do_package_write was removed with hash 4bdef77351789cb078742430fd034212
    [snip]

    === The verbose changes of bash:
    basehash changed from 2a3301fad820eaad859ae738b2cdb613 to 8086533b1abb4ab5b8507ff5e4a3046f
    List of dependencies for variable sbindir changed from set(['exec_prefix']) to set([])
    List of dependencies for variable bindir changed from set(['exec_prefix']) to set([])
    [snip]

    === Summary: (3653 changed, 1927 unchanged)
    Newly added: 807, Depends changed: 2846

    Removing the newly generated stamps ...
    Moving the stamps.old back to stamps ...

* TODO
- It seems that the "bitabke -S core-image-sato" has bugs, it would always report errors, but doesn't fatal errors

- "bitbake -S" always generate the nativesdk stamps regardless of what we build.

- The gcc-cross' stamps are in tmp/stamps/work-shared, but the "bitbake -S" doesn't put the stamps in work-shared. This doesn't matter to the "bitbake-whatchanged" since it uses the os.walk.

- Print the ones which can be installed from the sstate.

[YOCTO #1659]

---
 bitbake/bin/bitbake-whatchanged | 334 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged
bitbake/bin/bitbake-whatchanged new file +334 -0
new file mode 100755
index 0000000..6e2f71b
@@ -0,0 +1,334 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+
+# Copyright (c) 2012 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+from __future__ import print_function
+import os
+import sys
+import getopt
+import shutil
+import re
+from optparse import OptionParser
+
+# Need the bb.process.run and bb.siggen
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+import bb.process
+import bb.siggen
+
+# Match the stamp's filename
+# group(1): PN
+# group(2): PE_PV (may no PE)
+# group(3): PR
+# group(4): TASK
+# group(5): HASH (not used here)
+fre = re.compile("(?P<pn>.*)-(?P<pv>.*)-(?P<pr>r\d+)\.(?P<task>do_\w+)\.sigdata\.(?P<hash>.*)")
+fre_sdk = re.compile("(.*)-nativesdk-(.*)|(.*)-crosssdk-(.*)")
+
+def gen_dict(stamps, sdk):
+    """
+    Generate the dict from the stamps dir.
+    The output dict format is:
+    {FN: {pn: PN, pv: PV, pr: PR, task: TASK, path: PATH}}
+    """
+    # The member of the sub dict (a "path" will be added later)
+    sub_mem = ("pn", "pv", "pr", "task")
+    d = {}
+    for dirpath, _, files in os.walk(stamps):
+        for fn in files:
+            subdict = {}
+            if fre.match(fn):
+                tmp = fre.match(fn)
+                # Nothing especially should be done if sdk, just add it to the dict.
+                if sdk:
+                    for i in sub_mem:
+                        subdict[i] = tmp.group(i)
+                elif not fre_sdk.match(fn):
+                    for i in sub_mem:
+                        subdict[i] = tmp.group(i)
+                if len(subdict) != 0:
+                    # The path will be used by os.stat() and bb.siggen
+                    subdict['path'] = dirpath
+                    d[fn] = subdict
+    return d
+
+# Re-construct the dict
+def recon_dict(dict_in):
+    """
+    The output dict format is:
+    {PN_TASK: {fn: FN, pv: PV, pr: PR, path: PATH}}
+    """
+    dict_out = {}
+    for k in dict_in.keys():
+        subdict = {}
+        # The key
+        pn_task = "%s_%s" % (dict_in.get(k).get('pn'), dict_in.get(k).get('task'))
+        # If more than one stamps are found, use the latest one.
+        if pn_task in dict_out:
+            full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
+            full_path_cur = "%s/%s" % (dict_in.get(k).get('path'), k)
+            if os.stat(full_path_pre).st_mtime > os.stat(full_path_cur).st_mtime:
+                continue
+        subdict['fn'] = k
+        subdict['pv'] = dict_in.get(k).get('pv')
+        subdict['pr'] = dict_in.get(k).get('pr')
+        subdict['path'] = dict_in.get(k).get('path')
+        dict_out[pn_task] = subdict
+
+    return dict_out
+
+def split_pntask(s):
+    """
+    Split the pn_task in to (pn, task) and return it
+    """
+    tmp = re.match("(.*)_(do_.*)", s)
+    return (tmp.group(1), tmp.group(2))
+
+
+def print_added(d_new = None, d_old = None):
+    """
+    Print the newly added tasks
+    """
+    added = {}
+    for k in d_new.keys():
+        if k not in d_old:
+            # Add the new one to added dict, and remove it from
+            # d_new, so the remaining ones are the changed ones
+            added[k] = d_new.get(k)
+            del(d_new[k])
+
+    if not added:
+        return 0
+
+    # Format the output, the dict format is:
+    # {pn: task1, task2 ...}
+    added_format = {}
+    counter = 0
+    for k in added.keys():
+        pn, task = split_pntask(k)
+        if pn in added_format:
+            # Append the value
+            added_format[pn] = "%s %s" % (added_format.get(pn), task)
+        else:
+            added_format[pn] = task
+        counter += 1
+    print("=== Newly added tasks: (%s tasks)" % counter)
+    for k in added_format.keys():
+        print("  %s: %s" % (k, added_format.get(k)))
+
+    return counter
+
+def print_vrchanged(d_new = None, d_old = None, vr = None):
+    """
+    Print the pv or pr changed tasks.
+    The arg "vr" is "pv" or "pr"
+    """
+    pvchanged = {}
+    counter = 0
+    for k in d_new.keys():
+        if d_new.get(k).get(vr) != d_old.get(k).get(vr):
+            counter += 1
+            pn, task = split_pntask(k)
+            if pn not in pvchanged:
+                # Format the output, we only print pn (no task) since
+                # all the tasks would be changed when pn or pr changed,
+                # the dict format is:
+                # {pn: pv/pr_old -> pv/pr_new}
+                pvchanged[pn] = "%s -> %s" % (d_old.get(k).get(vr), d_new.get(k).get(vr))
+            del(d_new[k])
+
+    if not pvchanged:
+        return 0
+
+    print("\n=== %s changed: (%s tasks)" % (vr.upper(), counter))
+    for k in pvchanged.keys():
+        print("  %s: %s" % (k, pvchanged.get(k)))
+
+    return counter
+
+def print_depchanged(d_new = None, d_old = None, verbose = False):
+    """
+    Print the dependency changes
+    """
+    depchanged = {}
+    counter = 0
+    for k in d_new.keys():
+        counter += 1
+        pn, task = split_pntask(k)
+        if (verbose):
+            full_path_old = "%s/%s" % (d_old.get(k).get("path"), d_old.get(k).get('fn'))
+            full_path_new = "%s/%s" % (d_new.get(k).get("path"), d_new.get(k).get('fn'))
+            # No counter since it is not ready here
+            print("\n=== The verbose changes of %s:" % pn)
+            bb.siggen.compare_sigfiles(full_path_old, full_path_new)
+        else:
+            # Format the output, the format is:
+            # {pn: task1, task2, ...}
+            if pn in depchanged:
+                depchanged[pn] = "%s %s" % (depchanged.get(pn), task)
+            else:
+                depchanged[pn] = task
+
+    if len(depchanged) > 0:
+        print("\n=== DEPENDS changed: (%s tasks)" % counter)
+        for k in depchanged.keys():
+            print("  %s: %s" % (k, depchanged[k]))
+
+    return counter
+
+def main():
+    """
+    Print what will be done between the current and last builds:
+    1) Save the stamps to stamps.old
+    2) Run "bitbake -S recipe" to re-generate the stamps
+    3) Figure out what are newly added and changed, can't figure out
+       what are removed since we can't know the previous stamps
+       clearly, for example, if there are several builds, we can't know
+       exactly which stamps the last build has used.
+    4) Use bb.siggen.compare_sigfiles to diff the old and new stamps
+    """
+
+    parser = OptionParser(
+        version = "1.0",
+        usage = """%prog [options] [package ...]
+print what will be done between the current and last builds, for example:
+
+    $ bitbake core-image-sato
+    # Edit the recipes
+    $ bitbake-whatchanged core-image-sato
+
+The changes will be printed"
+
+Note: The amount of tasks is not accurate when the task is "do_build" since
+      it usually depends on other tasks
+"""
+)
+    parser.add_option("-v", "--verbose", help = "print the verbose changes",
+               action = "store_true", dest = "verbose")
+
+    options, args = parser.parse_args(sys.argv)
+
+    verbose = options.verbose
+
+    if len(args) != 2:
+        parser.error("Incorrect number of arguments")
+    else:
+        recipe = args[1]
+
+    # FIXME
+    # The bitbake -S will always generate the stamp for the nativesdk,
+    # so we should check whether it really builds sdk or not:
+    if recipe.endswith('sdk'):
+        sdk = True
+    else:
+        sdk = False
+
+    # Get TMPDIR
+    print("Figuring out the TMPDIR ...")
+    cmdline = "bitbake -e %s | sed -ne 's/^TMPDIR=\"\(.*\)\"/\\1/p'" % recipe
+    tmpdir, err = bb.process.run(cmdline)
+    if not tmpdir:
+        print("No TMPDIR found for '%s'" % recipe, file=sys.stderr)
+        return 2
+
+    # Get the top stamps dir, TMPDIR/stamps
+    new_stamps = tmpdir.rstrip("\n") + "/stamps"
+    if not os.path.isdir(new_stamps):
+        print("ERROR: no stamps directory found!", file=sys.stderr)
+        return 2
+
+    old_stamps = new_stamps + ".old"
+    if os.path.exists(old_stamps):
+        print ("ERROR: %s is not empty!" % old_stamps, file=sys.stderr)
+        return 2
+
+    try:
+        # Rename the stamps to stamps.old:
+        print("Moving the stamps to stamps.old ...")
+        os.rename(new_stamps, old_stamps)
+
+        # Generate the new stamps dir
+        print("Generating the new stamps ... (need several minutes)")
+        cmdline = "bitbake -S %s" % recipe
+        # FIXME
+        # The "bitbake -S" may fail, not fatal error, the stamps will still
+        # be generated, This might be a bug of "bitbake -S".
+        try:
+            bb.process.run(cmdline)
+        except:
+            pass
+
+        # The dict for the new and old stamps.
+        new_dict = gen_dict(new_stamps, sdk)
+        old_dict = gen_dict(old_stamps, sdk)
+
+        # Remove the same one from both stamps.
+        cnt_unchanged = 0
+        for k in new_dict.keys():
+            if k in old_dict:
+                cnt_unchanged += 1
+                del(new_dict[k])
+                del(old_dict[k])
+
+        # Re-construct the dict to easily find out what is added or changed.
+        # The dict format is:
+        # {PN_TASK: {fn: FN, pv: PV, pr: PR, path:PATH}}
+        new_recon = recon_dict(new_dict)
+        old_recon = recon_dict(old_dict)
+
+        del new_dict
+        del old_dict
+
+        # Figure out what are changed, the new_recon would be changed
+        # by the print_xxx function.
+        # Newly added
+        cnt_added = print_added(new_recon, old_recon)
+
+        # PV (including PE) and PR changed
+        # Let the bb.siggen handle them if verbose
+        cnt_rv = {}
+        if not verbose:
+            for i in ('pv', 'pr'):
+               cnt_rv[i] = print_vrchanged(new_recon, old_recon, i)
+
+        # Depends changed (use bitbake-diffsigs)
+        cnt_dep = print_depchanged(new_recon, old_recon, verbose)
+
+        total_changed = cnt_added + (cnt_rv.get('pv') or 0) + (cnt_rv.get('pr') or 0) + cnt_dep
+
+        print("\n=== Summary: (%s changed, %s unchanged)" % (total_changed, cnt_unchanged))
+        if verbose:
+            print("Newly added: %s, Depends changed: %s\n" % \
+                (cnt_added, cnt_dep))
+        else:
+            print("Newly added: %s, PV changed: %s, PR changed: %s, Depends changed: %s\n" % \
+                (cnt_added, cnt_rv.get('pv') or 0, cnt_rv.get('pr') or 0, cnt_dep))
+    except:
+        print("ERROR occurred!!! Recovering the stamps dir ...", file=sys.stderr)
+        raise
+    finally:
+        # Remove the newly generated stamps dir
+        print("Removing the newly generated stamps ...")
+        shutil.rmtree(new_stamps)
+
+        # Rename the stamps.old back
+        print("Moving the stamps.old back to stamps ...")
+        os.rename(old_stamps, new_stamps)
+
+if __name__ == "__main__":
+    sys.exit(main())
+
Signature · 1 line
1.7.11.2
#2 lore raw
In reply to Robert Yang
Quoted 10 lines · On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
+0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/whatchanged
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Quoted 6 lines

Robert Yang (1):
bitbake-whatchanged: print what is about to happen

 bitbake/bin/bitbake-whatchanged | 334
++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the following:

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>

    sys.exit(main())
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
    old_recon = recon_dict(old_dict)
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
    full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)

NameError: global name 'fn' is not defined
-------------- snip ----------------

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#3 lore raw
Quoted 40 lines · Paul Eggleton
Quoted 10 lines · On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
+0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/whatchanged
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Quoted 6 lines

Robert Yang (1):
bitbake-whatchanged: print what is about to happen

 bitbake/bin/bitbake-whatchanged | 334
++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
 create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the following:

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>

    sys.exit(main())
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
    old_recon = recon_dict(old_dict)
  File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
    full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)

NameError: global name 'fn' is not defined
-------------- snip ----------------

A couple of other things:

Quoted 7 lines

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Cheers,

Richard

#4 lore raw
Quoted 22 lines · Richard Purdie
Quoted 21 lines · Paul Eggleton
Quoted 10 lines · On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
+0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/whatchanged
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Quoted 6 lines

Robert Yang (1):
bitbake-whatchanged: print what is about to happen

  bitbake/bin/bitbake-whatchanged | 334

++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the following:

Quoted 15 lines
Quoted 9 lines

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in <module>

Quoted 6 lines
     sys.exit(main())
   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in main
     old_recon = recon_dict(old_dict)
   File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in recon_dict
     full_path_pre = "%s/%s" % (dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)

NameError: global name 'fn' is not defined

Thanks Paul, I've fixed this one, my testing didn't get here before, it should be the string 'fn', not the variable fn.

Quoted 8 lines
Quoted 8 lines

-------------- snip ----------------

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

I'm sorry, I don't understand what did you mean here. it seems that what I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at the beginning, then use it elsewhere.

Quoted 7 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

I'd like to send a patch for 2) if you are OK with it.

// Robert

Quoted 9 lines

Cheers,

Richard

#5 lore raw
Quoted 11 lines · On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
Quoted 6 lines · Richard Purdie
Quoted 5 lines · Paul Eggleton

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

I'm sorry, I don't understand what did you mean here. it seems that what I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at the beginning, then use it elsewhere.

Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e, bitbake has to go through a parse of the recipes (retrieving from the cache if available of course, but even that still takes a few seconds). I'm suggesting you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save quite a bit of time.

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

Quoted 7 lines

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

I have to admit I'm not sure what -S is currently being used for; so it's hard for me to comment on what might get broken if we change this. I suspect it's not being used very much at all though.

Quoted 6 lines

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

I'd like to send a patch for 2) if you are OK with it.

We already have a mechanism to allow through variables from the environment - BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in preference to os.getenv(). I think we would not want this change in stamp writing behaviour to take effect unless -S is being used though, which suggests it ought to be a new variable that is not checked unless -S has been specified.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw
Quoted 13 lines · Paul Eggleton
Quoted 11 lines · On Wednesday 08 August 2012 11:40:14 Robert Yang wrote:
Quoted 6 lines · Richard Purdie
Quoted 5 lines · Paul Eggleton

A couple of other things:

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

I'm sorry, I don't understand what did you mean here. it seems that what I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at the beginning, then use it elsewhere.

Quoted 17 lines

Sorry, I wasn't being very clear. If you specify the recipe with bitbake -e, bitbake has to go through a parse of the recipes (retrieving from the cache if available of course, but even that still takes a few seconds). I'm suggesting you don't specify the recipe as TMPDIR shouldn't be recipe-specific, and save quite a bit of time.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

Got it, thanks.

Quoted 20 lines
Quoted 10 lines

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

I have to admit I'm not sure what -S is currently being used for; so it's hard for me to comment on what might get broken if we change this. I suspect it's not being used very much at all though.

Quoted 5 lines

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve
it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

Quoted 8 lines
Quoted 1 line

I'd like to send a patch for 2) if you are OK with it.

We already have a mechanism to allow through variables from the environment - BB_ENV_WHITELIST / BB_ENV_EXTRAWHITE, and we should make use of that in preference to os.getenv(). I think we would not want this change in stamp writing behaviour to take effect unless -S is being used though, which suggests it ought to be a new variable that is not checked unless -S has been specified.

Thank you very much, I will add a BB_DUMP_STAMP.

// Robert

Quoted 3 lines

Cheers,
Paul

#7 lore raw
In reply to Robert Yang
Quoted 51 lines · Robert Yang
Quoted 33 lines · Richard Purdie
Quoted 32 lines · Paul Eggleton
Quoted 10 lines · On Tuesday 07 August 2012 23:48:55 Robert Yang wrote:

The following changes since commit c86d26cb976e665b1516e72153f3f686f62dedf9:

subversion: Add missing build dependency on sqlite3 (2012-08-06 16:14:06
+0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib robert/whatchanged
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/whatchanged

Quoted 6 lines

Robert Yang (1):
bitbake-whatchanged: print what is about to happen

  bitbake/bin/bitbake-whatchanged | 334

++++++++++++++++++++++++++++++++++++++++ 1 file changed, 334 insertions(+)
create mode 100755 bitbake/bin/bitbake-whatchanged

I tried this on top of latest master, unfortunately what I got was the following:

-------------- snip ----------------
Figuring out the TMPDIR ...
Moving the stamps to stamps.old ...
Generating the new stamps ... (need several minutes)
ERROR occurred!!! Recovering the stamps dir ...
Removing the newly generated stamps ...
Moving the stamps.old back to stamps ...
Traceback (most recent call last):
File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 333, in
<module>

     sys.exit(main())
Quoted 8 lines
Quoted 8 lines

File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 291, in
main
old_recon = recon_dict(old_dict)
File "/home/paul/poky/poky/bitbake/bin/bitbake-whatchanged", line 82, in
recon_dict

     full_path_pre = "%s/%s" %

(dict_in.get(dict_out.get(pn_task).get(fn)).get('path'), k)
NameError: global name 'fn' is not defined

Thanks Paul, I've fixed this one, my testing didn't get here before, it should be the string 'fn', not the variable fn.

Quoted 4 lines
Quoted 4 lines

-------------- snip ----------------

A couple of other things:

Quoted 16 lines
Quoted 4 lines
Quoted 4 lines

1) We ought to be able to assume that TMPDIR is the same regardless of the recipe specified; this avoids having to parse all of the recipes just to get the value of this variable.

I'm sorry, I don't understand what did you mean here. it seems that what I did is the same as you said: Use "bitbake -e" to figure out the TMPDIR at the beginning, then use it elsewhere.

Quoted 7 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

I just notices that added the variable STAMP to BB_ENV_EXTRAWHITE would make it work, thanks, I missed this before, I will send an updated sooner.

// Robert

Quoted 43 lines

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

I'd like to send a patch for 2) if you are OK with it.

// Robert

Quoted 9 lines

Cheers,

Richard

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

#8 lore raw
In reply to Robert Yang
Quoted 15 lines · Robert Yang
Quoted 7 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

Quoted 7 lines

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

Hmm, we probably could change this option as long as we update the manuals too. Ideally I would like some way to say "use the default stamps directory" without having to put a full path in.

Quoted 6 lines

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

I'd like to send a patch for 2) if you are OK with it.

Please don't use TMPDIR, bitbake has no knowledge of that variable, nor should it have and it will change the cache directory. I'd suggest using STAMP, that is what the variable is designed for. We just need to allow it from the environment which Paul mentions and this is what I originally proposed.

I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the variable is called) updated accordingly.

Cheers,

Richard

#9 lore raw
Quoted 35 lines · Richard Purdie
Quoted 15 lines · Robert Yang
Quoted 7 lines
Quoted 3 lines

2) I'm a little concerned with the general approach - is there no way of avoiding having to copy and move around the stamps directory? It seems a little risky if nothing else.

I think adding a parameter to -S would be a good move for this, its something people likely want in conjunction with that.

Yes, add a parameter to "bitbake -S recipe" would be the correct way, but as far as I know, the "-S" is a bool option currently, it doesn't accept an argument, I think that we have the following 2 solutions:

1) Modify the "-S" to accept an argument, but this may break the the usage of the "bitbake -S", the currently usage is:

Quoted 7 lines

bitbake -S <recipe>

We may change it to:

bitbake -S <tmpdir>(or stampsdir) <recipe>

But it seems that it's not easy differentiate the argument behind "-S".

Hmm, we probably could change this option as long as we update the manuals too. Ideally I would like some way to say "use the default stamps directory" without having to put a full path in.

Quoted 6 lines

2) Use "TMPDIR(or STAMP)=<path> bitbake -S recipe", but we don't support it currently, but we can add an os.getenv("TMPDIR") in bitbake to achieve it, the BB_TMPDIR or BB_STAMP would be better, but I'm not sure whether it will cause other problems.

I'd like to send a patch for 2) if you are OK with it.

Quoted 9 lines

Please don't use TMPDIR, bitbake has no knowledge of that variable, nor should it have and it will change the cache directory. I'd suggest using STAMP, that is what the variable is designed for. We just need to allow it from the environment which Paul mentions and this is what I originally proposed.

I'm leaning towards 2) and STAMP with BB_ENV_WHITELIST (or whatever the variable is called) updated accordingly.

Thank you very much, the STAMP is:

STAMP = "${TMPDIR}/stamps/${MULTIMACH_TARGET_SYS}/${PF}"

How about add a variable STAMP_DIR, it would be:

STAMP_DIR ?= "${TMPDIR}/stamps/"
STAMP = "${STAMP_DIR}/${MULTIMACH_TARGET_SYS}/${PF}"

Then add STAMP_DIR to BB_ENV_WHITELIST and we can run bitbake -S like:
STAMP_DIR=/what/we/want bitbake -S

Thanks

// Robert

Quoted 6 lines

Cheers,

Richard