Skip to contentSkip to conversation

Variable/include tracking (rebased)

PATCH New Peter Seebach 1 participant 3 messages Aug 8, 2012

lore
Share & tools
lore
#0 lore raw

cover Variable/include tracking (rebased)

This is a set of changes so that "bitbake -e" will report a list of all the files processed (and from what they were included), and prefix the variable values with their assignment history.

The following changes since commit 23bd5300b4a99218a15f4f6b0ab4091d63a602a5:
Richard Purdie (1):
data_smart: Fix unanchored regexp causing strange parsing issue

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib seebs/ntracking
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=seebs/ntracking

Peter Seebach (2):
data_smart.py: Provide (optional) logging of variable modifications data_smart.py: Track configuration file inclusions

 lib/bb/cooker.py         |    2 +
 lib/bb/data.py           |   65 +++++++++++++++++-----
 lib/bb/data_smart.py     |  140 +++++++++++++++++++++++++++++++++++++--------
 lib/bb/parse/__init__.py |    6 ++-
 lib/bb/parse/ast.py      |   65 ++++++++++++----------
 5 files changed, 209 insertions(+), 69 deletions(-)
#1 lore raw

PATCH 1/2 data_smart.py: Provide (optional) logging of variable modifications

This is very preliminary, and probably not very good, but it provides a much-wanted feature: An explanation of how variables got their current values.

How it works:

1. We create a history dict in the data_smart object, the members of which are lists of tuples. It's three, count them, THREE unrelated data types in a single object!
2. There is an eventLog(...) function which allows recording things which happen to variables.
3. All over the place, tons of setVar and related functions are updated with two optional arguments, filename and lineno.
4. We log pretty much everything if logging is enabled.
5. You can query the history of a thing with data.getHistory(var)

Tracking must be specifically turned on. If it's not, the history object is a single empty table (cheap) and the calls to eventLog() and extra argument passing are also probably quite cheap. As a proof of concept, the showEnvironment function used by bitbake -e now uses this if available.

---
 lib/bb/cooker.py     |    2 +
 lib/bb/data.py       |   49 +++++++++++++++++-------
 lib/bb/data_smart.py |  103 ++++++++++++++++++++++++++++++++++++++------------
 lib/bb/parse/ast.py  |   65 +++++++++++++++++--------------
 4 files changed, 152 insertions(+), 67 deletions(-)
lib/bb/cooker.py +2 -0
index 23fffc9..8d59774 100644
@@ -830,6 +830,8 @@ class BBCooker:
 
     def parseConfigurationFiles(self, prefiles, postfiles):
         data = self.configuration.data
+        if self.configuration.show_environment:
+            data.enableTracking()
         bb.parse.init_parser(data)
 
         # Parse files for loading *before* bitbake.conf and any includes
lib/bb/data.py +35 -14

97 lines not on this page · raw message

lib/bb/data_smart.py +79 -24

253 lines not on this page · raw message

lib/bb/parse/ast.py +36 -29

163 lines not on this page · raw message

Signature · 1 line
1.7.0.4
#2 lore raw

PATCH 2/2 data_smart.py: Track configuration file inclusions

In reply to Peter Seebach

This is a preliminary attempt to create a structured representation of the processing of include files. It ignores line numbers because they're meaningless. It could probably be updated to, say, distinguish between includes and requires. Output is added to bitbake -e.

---
 lib/bb/data.py           |   16 ++++++++++++++++
 lib/bb/data_smart.py     |   41 ++++++++++++++++++++++++++++++++++++++---
 lib/bb/parse/__init__.py |    6 +++++-
 3 files changed, 59 insertions(+), 4 deletions(-)
lib/bb/data.py +16 -0
index 64bf9a9..a762a40 100644
@@ -253,10 +253,26 @@ def emit_env(o=sys.__stdout__, d = init(), all=False):
     isfunc = lambda key: bool(d.getVarFlag(key, "func"))
     keys = sorted((key for key in d.keys() if not key.startswith("__")), key=isfunc)
     grouped = groupby(keys, isfunc)
+    # Include history!
+    if d.tracking():
+        o.write('#\n# INCLUDE HISTORY:\n#\n')
+        emit_history(o, d.getIncludeHistory())
+        
     for isfunc, keys in grouped:
         for key in keys:
             emit_var(key, o, d, all and not isfunc) and o.write('\n')
 
+def emit_history(o, h, depth = 0):
+    if not h:
+        return
+    for event in h:
+        o.write("# %*s%s" % (depth * 2, "", event[0]))
+        if event[1]:
+            o.write(" includes:\n")
+            emit_history(o, event[1], depth + 1)
+        else:
+            o.write("\n")
+
 def exported_keys(d):
     return (key for key in d.keys() if not key.startswith('__') and
                                       d.getVarFlag(key, 'export') and
lib/bb/data_smart.py +38 -3

65 lines not on this page · raw message

lib/bb/parse/__init__.py +5 -1
index 7b9c47e..0ea6a55 100644
@@ -88,7 +88,11 @@ def handle(fn, data, include = 0):
     """Call the handler that is appropriate for this file"""
     for h in handlers:
         if h['supports'](fn, data):
-            return h['handle'](fn, data, include)
+            data.includeLog(fn)
+            try:
+                return h['handle'](fn, data, include)
+            finally:
+                data.includeLogDone(fn)
     raise ParseError("not a BitBake file", fn)
 
 def init(fn, data):
Signature · 1 line
1.7.0.4