Skip to contentSkip to conversation

runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

PATCH In review Richard Purdie 3 participants 16 messages Dec 18, 2013 – Jan 27, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

Its useful to understand where the delta starts against an existing sstate cache for a given target. Adding this to the output of the -S option seems like a natural fit.

We use the hashvalidate function to figure this out and assume it can find siginfo files for more than just the setscene tasks.

---

bitbake/lib/bb/runqueue.py +77 -0
index 428cff1..b7a602b 100644
@@ -1039,6 +1039,7 @@ class RunQueue:
 
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
+                self.print_diffscenetasks()
                 self.dump_signatures()
             else:
                 self.start_worker()
@@ -1123,6 +1124,82 @@ class RunQueue:
 
         return
 
+    def print_diffscenetasks(self):
+
+        valid = []
+        sq_hash = []
+        sq_hashfn = []
+        sq_fn = []
+        sq_taskname = []
+        sq_task = []
+        noexec = []
+        stamppresent = []
+        valid_new = set()
+
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
+            taskname = self.rqdata.runq_task[task]
+            taskdep = self.rqdata.dataCache.task_deps[fn]
+
+            if 'noexec' in taskdep and taskname in taskdep['noexec']:
+                noexec.append(task)
+                continue
+
+            sq_fn.append(fn)
+            sq_hashfn.append(self.rqdata.dataCache.hashfn[fn])
+            sq_hash.append(self.rqdata.runq_hash[task])
+            sq_taskname.append(taskname)
+            sq_task.append(task)
+        call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
+        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
+        valid = bb.utils.better_eval(call, locs)
+        for v in valid:
+            valid_new.add(sq_task[v])
+
+        # Tasks which are both setscene and noexec never care about dependencies
+        # We therefore find tasks which are setscene and noexec and mark their
+        # unique dependencies as valid.
+        for task in noexec:
+            if task not in self.rqdata.runq_setscene:
+                continue
+            for dep in self.rqdata.runq_depends[task]:
+                hasnoexecparents = True
+                for dep2 in self.rqdata.runq_revdeps[dep]:
+                    if dep2 in self.rqdata.runq_setscene and dep2 in noexec:
+                        continue
+                    hasnoexecparents = False
+                    break
+                if hasnoexecparents:
+                    valid_new.add(dep)
+
+        invalidtasks = set()
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            if task not in valid_new and task not in noexec:
+                invalidtasks.add(task)
+
+        found = set()
+        processed = set()
+        for task in invalidtasks:
+            toprocess = set([task])
+            while toprocess:
+                next = set()
+                for t in toprocess:
+                    for dep in self.rqdata.runq_depends[t]:
+                        if dep in invalidtasks:
+                            found.add(task)
+                        if dep not in processed:
+                            processed.add(dep)
+                            next.add(dep)
+                toprocess = next
+                if task in found:
+                    toprocess = set()
+
+        tasklist = []
+        for task in invalidtasks.difference(found):
+            tasklist.append(self.rqdata.get_user_idstring(task))
+
+        if tasklist:
+            bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
 
 class RunQueueExecute:
 
#1 lore raw
Quoted 6 lines · Richard Purdie

Its useful to understand where the delta starts against an existing sstate cache for a given target. Adding this to the output of the -S option seems like a natural fit.

We use the hashvalidate function to figure this out and assume it can find siginfo files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

NOTE: Reparsing files to collect dependency data

Quoted 1 line

self.write_diffscenetasks(invalidtasks)

self.state = runQueueComplete
File "/OE/webos/owub/build-starfish-master/bitbake/lib/bb/runqueue.py", line 1227, in RunQueue.write_diffscenetasks(invalidtasks=set([16694, 11264, 5803, 4100, 2054, 16391, 2060, 16, 14354, 4115, 22, 14360, 6148, 8218, 2075, 10268, 13658, 8224, 4130, 18467, 16421, 38, 2087, 10283, 6188, 18687, 4145, 6194, 12686, 2105, 10298, 12347, 62, 14399, 4160, 12353, 16452, 14406, 7231, 6218, 13312, 77, 6224, 2130, 83, 14421, 4184, 15756, 4190, 6239, 2145, 98, 10939, 14436, 10598, 2151, 8296, 4202, 8302, 11665, 6257, 6163, 16763, 14949, 123, 4220, 14461, 2175, 10368, 129, 14468, 2181, 8326, 6849, 12507, 16748, 4235, 6286, 10383, 14480, 13336, 146, 8341, 10390, 10391, 4250, 6302, 161, 14498, 2211, 8357, 11735, 16552, 4265, 8363, 6316, 8562, 12462, 10415, 14513, 2226, 10421, 16567, 8378, 4283, 4286, 8565, 5152, 10436, 14710, 12486, 4295, 200, 8393, 2250, 16588, 12322, 206, 14543, 10452, 6357, 4310, 8408, 6363, 221, 14558, 2272, 7888, 10619, 6375, 17623, 12522, 14375, 10476, 4334, 2287, 8432, 16784, 16627, 18677, 8438, 16633, 4349, 2302, 17409, 10497, 6402, 4355, 7851, 13697, 6408, 12554, 268, 2317, 8462, 13357, 13818, 8468, 16376, 14650, 16663, 6424, 283, 2332, 15823, 10528, 8483, 16648, 6439, 2347, 16192, 4401, 8498, 14644, 6454, 10552, 313, 14023, 7903, 9285, 8513, 2371, 6469, 13718, 14741, 2377, 16446, 14668, 10573, 4431, 337, 6484, 12629, 343, 11741, 16733, 4446, 8544, 14689, 6499, 2404, 358, 1425, 10604, 12656, 6514, 2419, 8254, 374, 6889, 12692, 14665, 5183, 8574, 12671, 6529, 2434, 16107, 389, 4487, 2440, 4493, 8590, 14735, 6331, 10644, 405, 406, 2455, 6553, 4508, 8605, 3141, 420, 12709, 2470, 10667, 17938, 8621, 6574, 12153, 4532, 10313, 4379, 16825, 4538, 6559, 444, 6589, 13381, 7584, 450, 14787, 2500, 8645, 17814, 10697, 8651, 6604, 8269, 4562, 2515, 16856, 10713, 474, 6620, 477, 13733, 12291, 14817, 4578, 12710, 4584, 2539, 12370, 12382, 14832, 2545, 498, 12371, 501, 4599, 8696, 15444, 8814, 6653, 510, 14847, 6656, 2561, 12033, 6665, 8714, 8717, 2575, 10328, 4629, 535, 8728, 7599, 18567, 541, 2590, 12831, 6689, 9307, 4645, 8742, 6695, 14888, 7943, 556, 2605, 17584, 7262, 6710, 12855, 12724, 571, 10813, 14910, 15991, 10821, 2630, 16823, 2636, 4685, 17273, 15459, 596, 8789, 10838, 10839, 15460, 12891, 4700, 10682, 16300, 13111, 14955, 613, 11068, 2664, 619, 6764, 16146, 2670, 6770, 8820, 5566, 14970, 9663, 2685, 13251, 3861, 4736, 6786, 643, 12932, 10006, 14985, 7523, 10893, 4751, 658, 12947, 10900, 14038, 15000, 12740, 8859, 2716, 6815, 8865, 674, 10915, 12093, 10353, 14108, 13426, 689, 12978, 2740, 15030, 695, 8889, 2746, 4795, 8895, 4801, 710, 2761, 10954, 13004, 13005, 8910, 8529, 4816, 13971, 15060, 6873, 13019, 735, 10976, 2785, 16507, 10822, 741, 8934, 2791, 4841, 16528, 13035, 15084, 8943, 8946, 12755, 9777, 5588, 4858, 6271, 8956, 15146, 11007, 2816, 12416, 15106, 772, 2822, 16513, 4873, 6922, 9687, 11022, 4879, 6928, 15121, 787, 2837, 14620, 13080, 17348, 8666, 4894, 12779, 802, 2852, 11047, 6952, 7644, 9002, 9003, 15090, 11053, 6621, 13104, 817, 2868, 4918, 6967, 13112, 5940, 17690, 4924, 14772, 832, 9026, 15499, 9029, 15176, 17483, 2891, 10262, 15182, 847, 13136, 6997, 15161, 4954, 12431, 15197, 862, 2912, 13153, 9062, 4969, 877, 10767, 10728, 7029, 11128, 4985, 7038, 2943, 13184, 7041, 6635, 9093, 902, 13191, 5000, 15243, 908, 11153, 17300, 11159, 5016, 5017, 15258, 9117, 13215, 2976, 7075, 933, 5031, 15273, 9132, 8690, 2991, 7090, 15435, 948, 11189, 16603, 5047, 7096, 3003, 957, 16467, 4939, 11204, 5062, 7111, 9162, 971, 3021, 15312, 12785, 5077, 7126, 17368, 9177, 986, 3036, 7674, 9183, 11234, 5092, 7141, 1001, 16799, 3051, 9199, 11249, 5107, 7157, 1016, 3066, 12117, 9216, 10752, 7171, 15364, 17765, 3081, 5131, 9228, 12446, 1040, 5137, 7186, 11696, 15380, 3096, 11289, 9243, 2906, 13342, 1056, 15392, 1062, 3111, 7210, 9261, 11311, 7216, 15410, 13402, 1078, 17615, 5177, 14799, 16709, 11326, 3135, 16483, 9056, 11098, 1093, 9291, 7246, 10083, 13396, 1109, 11350, 5208, 9991, 4623, 11356, 3165, 5214, 16229, 3171, 1124, 16406, 15474, 9322, 8039, 5229, 13501, 3186, 1139, 8835, 16006, 13432, 9337, 15551, 5244, 10773, 1156, 1157, 7302, 13447, 3210, 5259, 11478, 1171, 7317, 13462, 9367, 7320, 3225, 5274, 15515, 3231, 15521, 11426, 16582, 9383, 5289, 7340, 7341, 13486, 6041, 7973, 1202, 9398, 3255, 5304, 15545, 7355, 3261, 18557, 15221, 1218, 12815, 13510, 11463, 13513, 7372, 7373, 12316, 9423, 11596, 3285, 5334, 2937, 1242, 3291, 11083, 1248, 13538, 9445, 5350, 17639, 7402, 13863, 15596, 3309, 11502, 14704, 9460, 6064, 1273, 13562, 15611, 5374, 12501, 11520, 13569, 9475, 9430, 11526, 5383, 3336, 5386, 15575, 1294, 10797, 5395, 11541, 7447, 3352, 3354, 9499, 1309, 13598, 15237, 5410, 7387, 11556, 7477, 7462, 7729, 3368, 9514, 16022, 11571, 1333, 13622, 15672, 9529, 1339, 13628, 11296, 9535, 3392, 5441, 7493, 3398, 15687, 13643, 8774, 9550, 8024, 11602, 1363, 7508, 3413, 15702, 13881, 5466, 5472, 11617, 7820, 1379, 3428, 15590, 9575, 13673, 17770, 13276, 5487, 1394, 3443, 17784, 14247, 11643, 7548, 9597, 3648, 1410, 10208, 13703, 17644, 11658, 3467, 9612, 5518, 3473, 15762, 11667, 14871, 14574, 12800, 1431, 9627, 3488, 16283, 7750, 1447, 11690, 15787, 5550, 3503, 5552, 15793, 9651, 9660, 13757, 3518, 1471, 15808, 13763, 7623, 9672, 12876, 17868, 7629, 1486, 14575, 14017, 5585, 13778, 1492, 11711, 3542, 7417, 3548, 5597, 13221, 17888, 13793, 10960, 9702, 15848, 7659, 5612, 1517, 253, 11371, 3572, 9717, 11766, 6740, 3578, 5627, 11772, 1533, 15870, 13824, 1539, 9732, 3593, 5642, 11787, 15885, 13839, 1554, 9747, 5654, 15961, 3608, 7705, 11802, 11183, 17948, 1569, 9762, 17958, 3623, 5672, 11441, 14599, 15916, 9138, 7554, 1288, 11829, 7735, 13236, 15931, 15285, 1600, 5697, 11844, 14262, 3654, 13896, 12215, 13902, 9807, 5712, 11859, 7765, 5718, 1624, 11865, 17679, 16177, 13917, 1630, 10853, 955, 5733, 9831, 16316, 13932, 7789, 1646, 11889, 15635, 5748, 3701, 9846, 11895, 13947, 7804, 4715, 3716, 1670, 11911, 9864, 18028, 3722, 1676, 18061, 9870, 12568, 7826, 10558, 4718, 11926, 15741, 13977, 13050, 5787, 9885, 16034, 3747, 1701, 3753, 1707, 9901, 11950, 5809, 14002, 14605, 16052, 3765, 12916, 1722, 9916, 17354, 15717, 7873, 15947, 11972, 18038, 3783, 1737, 9931, 16076, 15650, 5840, 3798, 9946, 16091, 11996, 14925, 5855, 13592, 1761, 12002, 18644, 3814, 1767, 9961, 12583, 5870, 10877, 15656, 12018, 3828, 14069, 7928, 16122, 1792, 9985, 3843, 14084, 5895, 16140, 7949, 1807, 12048, 7698, 5763, 1813, 5910, 3864, 17028, 5916, 12063, 3873, 16162, 1829, 11228, 7979, 12078, 3888, 14129, 1844, 5944, 15327, 5949, 10046, 3903, 14144, 7277, 8003, 1860, 12598, 14756, 5961, 12269, 13282, 3918, 14159, 17720, 8018, 10068, 1877, 5976, 16217, 3933, 14174, 15333, 16226, 5435, 12132, 1893, 5991, 13065, 16362, 10098, 12147, 1908, 3957, 8054, 14199, 10160, 3963, 16253, 6015, 1923, 8069, 14216, 12169, 10123, 16268, 14222, 6031, 1938, 3987, 8084, 6039, 12184, 3993, 10139, 17591, 12613, 1953, 9792, 13297, 4008, 6812, 10154, 8108, 1968, 12209, 8114, 6070, 4425, 14268, 16317, 12861, 1983, 4033, 8130, 6085, 12230, 4039, 8727, 10185, 14283, 1998, 10193, 10194, 12963, 16340, 12245, 4055, 16346, 6112, 8163, 14308, 4069, 6118, 2023, 12547, 14314, 15015, 18071, 2029, 10223, 7569, 8178, 6133, 14327, 4088, 4091, 16844, 12285, 10238])):

                 matches = {k : v for k, v in matches.iteritems() if h not in k}
    >            latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
                 prevh = __find_md5__.search(latestmatch).group(0)

IndexError: list index out of range

Quoted 104 lines

---

bitbake/lib/bb/runqueue.py +77 -0
index 428cff1..b7a602b 100644
@@ -1039,6 +1039,7 @@ class RunQueue:
 
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
+                self.print_diffscenetasks()
                 self.dump_signatures()
             else:
                 self.start_worker()
@@ -1123,6 +1124,82 @@ class RunQueue:
 
         return
 
+    def print_diffscenetasks(self):
+
+        valid = []
+        sq_hash = []
+        sq_hashfn = []
+        sq_fn = []
+        sq_taskname = []
+        sq_task = []
+        noexec = []
+        stamppresent = []
+        valid_new = set()
+
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
+            taskname = self.rqdata.runq_task[task]
+            taskdep = self.rqdata.dataCache.task_deps[fn]
+
+            if 'noexec' in taskdep and taskname in taskdep['noexec']:
+                noexec.append(task)
+                continue
+
+            sq_fn.append(fn)
+            sq_hashfn.append(self.rqdata.dataCache.hashfn[fn])
+            sq_hash.append(self.rqdata.runq_hash[task])
+            sq_taskname.append(taskname)
+            sq_task.append(task)
+        call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
+        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
+        valid = bb.utils.better_eval(call, locs)
+        for v in valid:
+            valid_new.add(sq_task[v])
+
+        # Tasks which are both setscene and noexec never care about dependencies
+        # We therefore find tasks which are setscene and noexec and mark their
+        # unique dependencies as valid.
+        for task in noexec:
+            if task not in self.rqdata.runq_setscene:
+                continue
+            for dep in self.rqdata.runq_depends[task]:
+                hasnoexecparents = True
+                for dep2 in self.rqdata.runq_revdeps[dep]:
+                    if dep2 in self.rqdata.runq_setscene and dep2 in noexec:
+                        continue
+                    hasnoexecparents = False
+                    break
+                if hasnoexecparents:
+                    valid_new.add(dep)
+
+        invalidtasks = set()
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            if task not in valid_new and task not in noexec:
+                invalidtasks.add(task)
+
+        found = set()
+        processed = set()
+        for task in invalidtasks:
+            toprocess = set([task])
+            while toprocess:
+                next = set()
+                for t in toprocess:
+                    for dep in self.rqdata.runq_depends[t]:
+                        if dep in invalidtasks:
+                            found.add(task)
+                        if dep not in processed:
+                            processed.add(dep)
+                            next.add(dep)
+                toprocess = next
+                if task in found:
+                    toprocess = set()
+
+        tasklist = []
+        for task in invalidtasks.difference(found):
+            tasklist.append(self.rqdata.get_user_idstring(task))
+
+        if tasklist:
+            bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
 
 class RunQueueExecute:
 

_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#2 lore raw
Quoted 11 lines · Martin Jansa
Quoted 1 line · Richard Purdie

Its useful to understand where the delta starts against an existing

sstate cache

Quoted 1 line

for a given target. Adding this to the output of the -S option seems

like a

Quoted 3 lines

natural fit.

We use the hashvalidate function to figure this out and assume it can

find siginfo

Quoted 1 line

files for more than just the setscene tasks.

Quoted 7 lines

I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

I’m hitting this now as well.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#3 lore raw

RFC PATCH 1/1 runqueue: Don't fail without match found

* WIP, because I haven't checked the reasons why this happens I'm using this now only to see which recipes/tasks are failing like this

---
 lib/bb/runqueue.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
lib/bb/runqueue.py +8 -4
index 4577236..413b95e 100644
@@ -1224,10 +1224,14 @@ class RunQueue:
             if match is None:
                 bb.fatal("Can't find a task we're supposed to have written out? (hash: %s)?" % h)
             matches = {k : v for k, v in matches.iteritems() if h not in k}
-            latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
-            prevh = __find_md5__.search(latestmatch).group(0)
-            output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
-            bb.plain("\nTask %s:%s couldn't be used from the cache because:\n  We need hash %s, closest matching task was %s\n  " % (pn, taskname, h, prevh) + '\n  '.join(output))
+            sortedmatches=sorted(matches.keys(), key=lambda f: matches[f])
+            if sortedmatches:
+                latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
+                prevh = __find_md5__.search(latestmatch).group(0)
+                output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
+                bb.plain("\nTask %s:%s couldn't be used from the cache because:\n  We need hash %s, closest matching task was %s\n  " % (pn, taskname, h, prevh) + '\n  '.join(output))
+            else:
+                bb.plain("\nTask %s:%s couldn't find any matches for hash %s" % (pn, taskname, h))
              
 
 
Signature · 1 line
1.8.4.3
#4 lore raw
In reply to Chris Larson
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie

Its useful to understand where the delta starts against an existing

sstate cache

Quoted 1 line

for a given target. Adding this to the output of the -S option seems

like a

Quoted 3 lines

natural fit.

We use the hashvalidate function to figure this out and assume it can

find siginfo

Quoted 1 line

files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with

Quoted 7 lines
Quoted 4 lines

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

I’m hitting this now as well.

To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.

Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#5 lore raw
Quoted 30 lines · Chris Larson
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie

Its useful to understand where the delta starts against an existing

sstate cache

Quoted 1 line

for a given target. Adding this to the output of the -S option seems

like a

Quoted 3 lines

natural fit.

We use the hashvalidate function to figure this out and assume it can

find siginfo

Quoted 1 line

files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with

Quoted 7 lines
Quoted 4 lines

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

I’m hitting this now as well.

To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.

Quoted 7 lines

Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?

Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.sh

and this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).

It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).

I see it for many do_fetch and do_rm_work_all tasks.

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#6 lore raw
Quoted 46 lines · Martin Jansa
Quoted 30 lines · Chris Larson
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie

Its useful to understand where the delta starts against an existing

sstate cache

Quoted 1 line

for a given target. Adding this to the output of the -S option seems

like a

Quoted 3 lines

natural fit.

We use the hashvalidate function to figure this out and assume it can

find siginfo

Quoted 1 line

files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with

Quoted 7 lines
Quoted 4 lines

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

I’m hitting this now as well.

To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.

Quoted 7 lines

Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?

Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.sh

and this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).

Quoted 5 lines

It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).

I see it for many do_fetch and do_rm_work_all tasks.

To put things in context, I was trying to work with the existing "whatchanged" type tools or more specifically I wanted to know "why isn't my sstate cache being used". Having to do a two step process for any of these enquires seems a little odd. There are also two specific and as it turns our quite different work flows:

a) What changed since my last build?
b) What is the least delta between my current build and whatever I have cached?

Those experiments revealed some big holes in the sstate metadata and some good improvements there. It also highlights that there are two different questions which have different answers.

In the process, I found bitbake does have some state in runqueue which can help pinpoint the "whatchanged" part of the problem much more accurately/quickly. Obviously that state is lost after bitbake -S finishes so I started printing out some extra data.

Clearly bitbake shouldn't crash, that is a bug and I think Martin is heading in the right direction with the other patch.

As for bitbake -S being kept fast, I think (but I need to double check with measurements) that the parsing part of bitbake -S is the slow part and this extra data is effectively "for free".

So basically consider the current state an experiment to see if we can make -S more useful/interesting (without much cost) and to try to better understand where/what functionality we need going forward.

One possible outcome might be we generate an extra file (or files) (like we do with task-depends.dot) which contains this extra tree data which some other command can then parse and display to the user. I'm still struggling to get a complete grip on the exact data and format that file needs to be. Part of the reason for having -S print something was to see if it printed what the user wanted to know or if not, we could then try and figure out what the right thing was so we could improve it.

So sorry if this is causing pain, the code is easy to disable by commenting out the call, or reverting the patch adding it. Equally, I think we need to make sstate hashes and reasons for rebuilds more accessible to users and I think this does move us closer to that assuming we fix the crash.

I'm trying to take a bit of a break over the holiday period (been riding a motorcycle in driving snow/rain today and am rather damp/cold) but I'll get the crash fixed sometime next week at the latest.

Cheers,

Richard

#7 lore raw
Quoted 85 lines · Richard Purdie
Quoted 2 lines · Martin Jansa
Quoted 1 line · Chris Larson

On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <[email protected]<javascript:;>>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines

On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <

Quoted 3 lines · [email protected] <javascript:;>>wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Richard Purdie
Quoted 1 line

Its useful to understand where the delta starts against an

existing

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

sstate cache

Quoted 1 line

for a given target. Adding this to the output of the -S option

seems

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

like a

Quoted 3 lines

natural fit.

We use the hashvalidate function to figure this out and assume it

can

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

find siginfo

Quoted 1 line

files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code

where

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

it happends, but in bigger builds I have it fails with

I'll try to reproduce and narrow it a bit, but if you have tip where to look first..

IndexError: list index out of range:

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

I’m hitting this now as well.

To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.

Actually, this is getting in my way trying to do something I don’t

want it

Quoted 1 line
Quoted 1 line

doing anyway. I want to dump the signature data from one config, and

use

Quoted 1 line
Quoted 1 line

bitbake-whatchanged from another build. If I wanted to compare

something

Quoted 12 lines
Quoted 4 lines

right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?

Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.sh

and this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).

Quoted 5 lines

It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).

I see it for many do_fetch and do_rm_work_all tasks.

To put things in context, I was trying to work with the existing "whatchanged" type tools or more specifically I wanted to know "why isn't my sstate cache being used". Having to do a two step process for any of these enquires seems a little odd. There are also two specific and as it turns our quite different work flows:

a) What changed since my last build?
b) What is the least delta between my current build and whatever I have cached?

Those experiments revealed some big holes in the sstate metadata and some good improvements there. It also highlights that there are two different questions which have different answers.

In the process, I found bitbake does have some state in runqueue which can help pinpoint the "whatchanged" part of the problem much more accurately/quickly. Obviously that state is lost after bitbake -S finishes so I started printing out some extra data.

Clearly bitbake shouldn't crash, that is a bug and I think Martin is heading in the right direction with the other patch.

Quoted 8 lines

As for bitbake -S being kept fast, I think (but I need to double check with measurements) that the parsing part of bitbake -S is the slow part and this extra data is effectively "for free".

So basically consider the current state an experiment to see if we can make -S more useful/interesting (without much cost) and to try to better understand where/what functionality we need going forward.

I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My SSTATE_DIR has weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this?

-Chris

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#8 lore raw
Quoted 8 lines · Chris Larson

I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My SSTATE_DIR has weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm

Quoted 8 lines

misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this?

Nothing changed in the data -S writes out.

Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.

On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.

So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).

That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.

I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).

I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.

Cheers,

Richard

#9 lore raw
Quoted 5 lines · Richard Purdie

So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).

I've finally got some numbers to support my claim that it "feels" slower with this added functionality.

My script is calling
openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;

which in turn runs
bitbake -S world
three times (once for each machine).

The script on jenkins doing world builds is completely crazy (I've killed it in last run assuming it was stuck, but it wasn't just terribly slow).
http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
Running 21 hours already and doing 2nd machine from 3 (maybe because longer SSTATE_MIRROR?). There is one python process using 100% cpu.

In local builds without SSTATE_MIRROR it's not so bad, but still significantly slower with each release (dylan-dora-master).

Master builds have that WIP patch included in order to finish.

It's not very accurate, because it was running on my desktop and each one has different metadata, but at least the layers included are similar (only systems which got separated in dora and removal
of meta-webos-backports which isn't needed for dora and newer).

At least I've executed each test twice.

dylan:

real   9m37.271s
user   6m37.104s
sys    3m12.625s
real   10m5.787s
user   6m37.832s
sys    3m23.614s

dora:

real   26m11.374s
user   9m7.325s
sys    3m0.048s
real   28m10.606s
user   8m27.895s
sys    2m57.497s

master:

real   38m17.641s
user   17m46.905s
sys    5m12.061s
real    36m12.748s
user    16m32.961s
sys     4m43.103s

master-without-S-changes:

real    27m34.228s
user    8m6.811s
sys     2m53.724s
real    27m49.752s
user    8m25.026s
sys     2m50.921s
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#10 lore raw
Quoted 17 lines · Martin Jansa
Quoted 5 lines · Richard Purdie

So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).

I've finally got some numbers to support my claim that it "feels" slower with this added functionality.

My script is calling
openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;

which in turn runs
bitbake -S world
three times (once for each machine).

Quoted 6 lines

The script on jenkins doing world builds is completely crazy (I've killed it in last run assuming it was stuck, but it wasn't just terribly slow).
http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
Running 21 hours already and doing 2nd machine from 3 (maybe because longer SSTATE_MIRROR?). There is one python process using 100% cpu.

FWIW: this jenkins job was completed yesterday after 16 days

small part of strace:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/log.strace

complete log:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/signatures.sstate-diff-machines

Previous runs of the same job took 25-40 minutes

Quoted 50 lines

In local builds without SSTATE_MIRROR it's not so bad, but still significantly slower with each release (dylan-dora-master).

Master builds have that WIP patch included in order to finish.

It's not very accurate, because it was running on my desktop and each one has different metadata, but at least the layers included are similar (only systems which got separated in dora and removal
of meta-webos-backports which isn't needed for dora and newer).

At least I've executed each test twice.

dylan:

real   9m37.271s
user   6m37.104s
sys    3m12.625s
real   10m5.787s
user   6m37.832s
sys    3m23.614s

dora:

real   26m11.374s
user   9m7.325s
sys    3m0.048s
real   28m10.606s
user   8m27.895s
sys    2m57.497s

master:

real   38m17.641s
user   17m46.905s
sys    5m12.061s
real    36m12.748s
user    16m32.961s
sys     4m43.103s

master-without-S-changes:

real    27m34.228s
user    8m6.811s
sys     2m53.724s
real    27m49.752s
user    8m25.026s
sys     2m50.921s
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#11 lore raw
In reply to Richard Purdie
Quoted 30 lines · Richard Purdie

Nothing changed in the data -S writes out.

Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.

On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.

So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).

That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.

I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).

Quoted 4 lines

I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.

I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.

With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.

I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.

I don't have any code to do this yet but at least we have a plan assuming no strong objections.

Cheers,

Richard

#12 lore raw
Quoted 45 lines · Richard Purdie
Quoted 30 lines · Richard Purdie

Nothing changed in the data -S writes out.

Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.

On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.

So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).

That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.

I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).

Quoted 4 lines

I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.

I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.

With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.

Quoted 6 lines

I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.

I don't have any code to do this yet but at least we have a plan assuming no strong objections.

I strongly support this plan :).

Cheers,

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#13 lore raw
In reply to Richard Purdie
Quoted 4 lines · Richard Purdie

I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.

Quoted 8 lines

With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.

I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.

So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.

It has good reason for this as its near impossible for optparse to tell the difference between:

bitbake image -S tracesigs
and
bitbake image -S image2

So our options are:

a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different name

In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.

Any opinions?

Cheers,

Richard

#14 lore raw
Quoted 33 lines · Richard Purdie
Quoted 4 lines · Richard Purdie

I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.

Quoted 8 lines

With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.

I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.

So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.

It has good reason for this as its near impossible for optparse to tell the difference between:

bitbake image -S tracesigs
and
bitbake image -S image2

So our options are:

a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different name

Quoted 6 lines

In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.

Any opinions?

What about using some existing parameter in combination with -S?

-S + -v shows tracesigs
-S without -v old behavior

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#15 lore raw
Quoted 43 lines · Martin Jansa
Quoted 33 lines · Richard Purdie
Quoted 4 lines · Richard Purdie

I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.

Quoted 8 lines

With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.

I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.

So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.

It has good reason for this as its near impossible for optparse to tell the difference between:

bitbake image -S tracesigs
and
bitbake image -S image2

So our options are:

a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different name

Quoted 6 lines

In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.

Any opinions?

What about using some existing parameter in combination with -S?

Review

Quoted 2 lines
(quoted diff) +0 -2
-S + -v shows tracesigs
-S without -v old behavior

I have some use cases locally (such as locked sigs) which could really benefit from a string parameter which gets passed into the siggen code. I can see us needing more than two "behaviours" of -S...

I also think using something like -v will lead to more confusion and the code won't be particularly nice either :/.

Cheers,

Richard