File inclusion and variable assignment tracking

This is the assignment/inclusion tracking stuff I've had floating around. In further testing, I found one missing [1] that made it fail for "bitbake -e [specific package]". Haven't had any other troubles with it.

This could totally use improvement, but I think this has reached a point where:
1. It's more useful to have it than not.
2. It will be better improved by feedback from live usage.

Impact outside of the bitbake -e case is basically nil, and in the -e case it's a HUGE improvement in usability.

The following changes since commit d316f28ed725ff40daa8771c1aa224ac46d5b224:
Richard Purdie (1):
methodpool: Improve method already seen error message

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib seebs/tracking
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=seebs/tracking

Peter Seebach (2):
data_smart.py: Provide (optional) logging of variable modifications data_smart.py: Track configuration file inclusions

 lib/bb/cooker.py         |    2 +
 lib/bb/data.py           |   65 +++++++++++++++++-----
 lib/bb/data_smart.py     |  140 +++++++++++++++++++++++++++++++++++++--------
 lib/bb/parse/__init__.py |    6 ++-
 lib/bb/parse/ast.py      |   65 ++++++++++++----------
 5 files changed, 209 insertions(+), 69 deletions(-)

This is very preliminary, and probably not very good, but it provides a much-wanted feature: An explanation of how variables got their current values.

How it works:

1. We create a history dict in the data_smart object, the members of which are lists of tuples. It's three, count them, THREE unrelated data types in a single object!
2. There is an eventLog(...) function which allows recording things which happen to variables.
3. All over the place, tons of setVar and related functions are updated with two optional arguments, filename and lineno.
4. We log pretty much everything if logging is enabled.
5. You can query the history of a thing with data.getHistory(var)

Tracking must be specifically turned on. If it's not, the history object is a single empty table (cheap) and the calls to eventLog() and extra argument passing are also probably quite cheap. As a proof of concept, the showEnvironment function used by bitbake -e now uses this if available.

---
 lib/bb/cooker.py     |    2 +
 lib/bb/data.py       |   49 +++++++++++++++++-------
 lib/bb/data_smart.py |  103 ++++++++++++++++++++++++++++++++++++++------------
 lib/bb/parse/ast.py  |   65 +++++++++++++++++--------------
 4 files changed, 152 insertions(+), 67 deletions(-)

index 928b600..8feac99 100644
@@ -825,6 +825,8 @@ class BBCooker:
 
     def parseConfigurationFiles(self, prefiles, postfiles):
         data = self.configuration.data
+        if self.configuration.show_environment:
+            data.enableTracking()
         bb.parse.init_parser(data)
 
         # Parse files for loading *before* bitbake.conf and any includes

1.7.0.4

This is a preliminary attempt to create a structured representation of the processing of include files. It ignores line numbers because they're meaningless. It could probably be updated to, say, distinguish between includes and requires. Output is added to bitbake -e.

---
 lib/bb/data.py           |   16 ++++++++++++++++
 lib/bb/data_smart.py     |   41 ++++++++++++++++++++++++++++++++++++++---
 lib/bb/parse/__init__.py |    6 +++++-
 3 files changed, 59 insertions(+), 4 deletions(-)

index bd03e84..4db0280 100644
@@ -253,10 +253,26 @@ def emit_env(o=sys.__stdout__, d = init(), all=False):
     isfunc = lambda key: bool(d.getVarFlag(key, "func"))
     keys = sorted((key for key in d.keys() if not key.startswith("__")), key=isfunc)
     grouped = groupby(keys, isfunc)
+    # Include history!
+    if d.tracking():
+        o.write('#\n# INCLUDE HISTORY:\n#\n')
+        emit_history(o, d.getIncludeHistory())
+        
     for isfunc, keys in grouped:
         for key in keys:
             emit_var(key, o, d, all and not isfunc) and o.write('\n')
 
+def emit_history(o, h, depth = 0):
+    if not h:
+        return
+    for event in h:
+        o.write("# %*s%s" % (depth * 2, "", event[0]))
+        if event[1]:
+            o.write(" includes:\n")
+            emit_history(o, event[1], depth + 1)
+        else:
+            o.write("\n")
+
 def exported_keys(d):
     return (key for key in d.keys() if not key.startswith('__') and
                                       d.getVarFlag(key, 'export') and

index 7b9c47e..0ea6a55 100644
@@ -88,7 +88,11 @@ def handle(fn, data, include = 0):
     """Call the handler that is appropriate for this file"""
     for h in handlers:
         if h['supports'](fn, data):
-            return h['handle'](fn, data, include)
+            data.includeLog(fn)
+            try:
+                return h['handle'](fn, data, include)
+            finally:
+                data.includeLogDone(fn)
     raise ParseError("not a BitBake file", fn)
 
 def init(fn, data):

1.7.0.4