runqueue: Add output for -S option for listing the changepoints compared with an sstate cache
PATCH 1/1 runqueue: Add output for -S option for listing the changepoints compared with an sstate cache
Its useful to understand where the delta starts against an existing sstate cache for a given target. Adding this to the output of the -S option seems like a natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo files for more than just the setscene tasks.
---
bitbake/lib/bb/runqueue.py +77 -0
@@ -1039,6 +1039,7 @@ class RunQueue: if self.state is runQueueSceneInit: if self.cooker.configuration.dump_signatures: + self.print_diffscenetasks() self.dump_signatures() else: self.start_worker()
@@ -1123,6 +1124,82 @@ class RunQueue: return + def print_diffscenetasks(self): + + valid = [] + sq_hash = [] + sq_hashfn = [] + sq_fn = [] + sq_taskname = [] + sq_task = [] + noexec = [] + stamppresent = [] + valid_new = set() + + for task in xrange(len(self.rqdata.runq_fnid)): + fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]] + taskname = self.rqdata.runq_task[task] + taskdep = self.rqdata.dataCache.task_deps[fn] + + if 'noexec' in taskdep and taskname in taskdep['noexec']: + noexec.append(task) + continue + + sq_fn.append(fn) + sq_hashfn.append(self.rqdata.dataCache.hashfn[fn]) + sq_hash.append(self.rqdata.runq_hash[task]) + sq_taskname.append(taskname) + sq_task.append(task) + call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)" + locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data } + valid = bb.utils.better_eval(call, locs) + for v in valid: + valid_new.add(sq_task[v]) + + # Tasks which are both setscene and noexec never care about dependencies + # We therefore find tasks which are setscene and noexec and mark their + # unique dependencies as valid. + for task in noexec: + if task not in self.rqdata.runq_setscene: + continue + for dep in self.rqdata.runq_depends[task]: + hasnoexecparents = True + for dep2 in self.rqdata.runq_revdeps[dep]: + if dep2 in self.rqdata.runq_setscene and dep2 in noexec: + continue + hasnoexecparents = False + break + if hasnoexecparents: + valid_new.add(dep) + + invalidtasks = set() + for task in xrange(len(self.rqdata.runq_fnid)): + if task not in valid_new and task not in noexec: + invalidtasks.add(task) + + found = set() + processed = set() + for task in invalidtasks: + toprocess = set([task]) + while toprocess: + next = set() + for t in toprocess: + for dep in self.rqdata.runq_depends[t]: + if dep in invalidtasks: + found.add(task) + if dep not in processed: + processed.add(dep) + next.add(dep) + toprocess = next + if task in found: + toprocess = set() + + tasklist = [] + for task in invalidtasks.difference(found): + tasklist.append(self.rqdata.get_user_idstring(task)) + + if tasklist: + bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) class RunQueueExecute:
Quoted 6 lines · Richard Purdie
Its useful to understand where the delta starts against an existing sstate cache for a given target. Adding this to the output of the -S option seems like a natural fit.
We use the hashvalidate function to figure this out and assume it can find siginfo files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
NOTE: Reparsing files to collect dependency data
Quoted 1 line
self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
File "/OE/webos/owub/build-starfish-master/bitbake/lib/bb/runqueue.py", line 1227, in RunQueue.write_diffscenetasks(invalidtasks=set([16694, 11264, 5803, 4100, 2054, 16391, 2060, 16, 14354, 4115, 22, 14360, 6148, 8218, 2075, 10268, 13658, 8224, 4130, 18467, 16421, 38, 2087, 10283, 6188, 18687, 4145, 6194, 12686, 2105, 10298, 12347, 62, 14399, 4160, 12353, 16452, 14406, 7231, 6218, 13312, 77, 6224, 2130, 83, 14421, 4184, 15756, 4190, 6239, 2145, 98, 10939, 14436, 10598, 2151, 8296, 4202, 8302, 11665, 6257, 6163, 16763, 14949, 123, 4220, 14461, 2175, 10368, 129, 14468, 2181, 8326, 6849, 12507, 16748, 4235, 6286, 10383, 14480, 13336, 146, 8341, 10390, 10391, 4250, 6302, 161, 14498, 2211, 8357, 11735, 16552, 4265, 8363, 6316, 8562, 12462, 10415, 14513, 2226, 10421, 16567, 8378, 4283, 4286, 8565, 5152, 10436, 14710, 12486, 4295, 200, 8393, 2250, 16588, 12322, 206, 14543, 10452, 6357, 4310, 8408, 6363, 221, 14558, 2272, 7888, 10619, 6375, 17623, 12522, 14375, 10476, 4334, 2287, 8432, 16784, 16627, 18677, 8438, 16633, 4349, 2302, 17409, 10497, 6402, 4355, 7851, 13697, 6408, 12554, 268, 2317, 8462, 13357, 13818, 8468, 16376, 14650, 16663, 6424, 283, 2332, 15823, 10528, 8483, 16648, 6439, 2347, 16192, 4401, 8498, 14644, 6454, 10552, 313, 14023, 7903, 9285, 8513, 2371, 6469, 13718, 14741, 2377, 16446, 14668, 10573, 4431, 337, 6484, 12629, 343, 11741, 16733, 4446, 8544, 14689, 6499, 2404, 358, 1425, 10604, 12656, 6514, 2419, 8254, 374, 6889, 12692, 14665, 5183, 8574, 12671, 6529, 2434, 16107, 389, 4487, 2440, 4493, 8590, 14735, 6331, 10644, 405, 406, 2455, 6553, 4508, 8605, 3141, 420, 12709, 2470, 10667, 17938, 8621, 6574, 12153, 4532, 10313, 4379, 16825, 4538, 6559, 444, 6589, 13381, 7584, 450, 14787, 2500, 8645, 17814, 10697, 8651, 6604, 8269, 4562, 2515, 16856, 10713, 474, 6620, 477, 13733, 12291, 14817, 4578, 12710, 4584, 2539, 12370, 12382, 14832, 2545, 498, 12371, 501, 4599, 8696, 15444, 8814, 6653, 510, 14847, 6656, 2561, 12033, 6665, 8714, 8717, 2575, 10328, 4629, 535, 8728, 7599, 18567, 541, 2590, 12831, 6689, 9307, 4645, 8742, 6695, 14888, 7943, 556, 2605, 17584, 7262, 6710, 12855, 12724, 571, 10813, 14910, 15991, 10821, 2630, 16823, 2636, 4685, 17273, 15459, 596, 8789, 10838, 10839, 15460, 12891, 4700, 10682, 16300, 13111, 14955, 613, 11068, 2664, 619, 6764, 16146, 2670, 6770, 8820, 5566, 14970, 9663, 2685, 13251, 3861, 4736, 6786, 643, 12932, 10006, 14985, 7523, 10893, 4751, 658, 12947, 10900, 14038, 15000, 12740, 8859, 2716, 6815, 8865, 674, 10915, 12093, 10353, 14108, 13426, 689, 12978, 2740, 15030, 695, 8889, 2746, 4795, 8895, 4801, 710, 2761, 10954, 13004, 13005, 8910, 8529, 4816, 13971, 15060, 6873, 13019, 735, 10976, 2785, 16507, 10822, 741, 8934, 2791, 4841, 16528, 13035, 15084, 8943, 8946, 12755, 9777, 5588, 4858, 6271, 8956, 15146, 11007, 2816, 12416, 15106, 772, 2822, 16513, 4873, 6922, 9687, 11022, 4879, 6928, 15121, 787, 2837, 14620, 13080, 17348, 8666, 4894, 12779, 802, 2852, 11047, 6952, 7644, 9002, 9003, 15090, 11053, 6621, 13104, 817, 2868, 4918, 6967, 13112, 5940, 17690, 4924, 14772, 832, 9026, 15499, 9029, 15176, 17483, 2891, 10262, 15182, 847, 13136, 6997, 15161, 4954, 12431, 15197, 862, 2912, 13153, 9062, 4969, 877, 10767, 10728, 7029, 11128, 4985, 7038, 2943, 13184, 7041, 6635, 9093, 902, 13191, 5000, 15243, 908, 11153, 17300, 11159, 5016, 5017, 15258, 9117, 13215, 2976, 7075, 933, 5031, 15273, 9132, 8690, 2991, 7090, 15435, 948, 11189, 16603, 5047, 7096, 3003, 957, 16467, 4939, 11204, 5062, 7111, 9162, 971, 3021, 15312, 12785, 5077, 7126, 17368, 9177, 986, 3036, 7674, 9183, 11234, 5092, 7141, 1001, 16799, 3051, 9199, 11249, 5107, 7157, 1016, 3066, 12117, 9216, 10752, 7171, 15364, 17765, 3081, 5131, 9228, 12446, 1040, 5137, 7186, 11696, 15380, 3096, 11289, 9243, 2906, 13342, 1056, 15392, 1062, 3111, 7210, 9261, 11311, 7216, 15410, 13402, 1078, 17615, 5177, 14799, 16709, 11326, 3135, 16483, 9056, 11098, 1093, 9291, 7246, 10083, 13396, 1109, 11350, 5208, 9991, 4623, 11356, 3165, 5214, 16229, 3171, 1124, 16406, 15474, 9322, 8039, 5229, 13501, 3186, 1139, 8835, 16006, 13432, 9337, 15551, 5244, 10773, 1156, 1157, 7302, 13447, 3210, 5259, 11478, 1171, 7317, 13462, 9367, 7320, 3225, 5274, 15515, 3231, 15521, 11426, 16582, 9383, 5289, 7340, 7341, 13486, 6041, 7973, 1202, 9398, 3255, 5304, 15545, 7355, 3261, 18557, 15221, 1218, 12815, 13510, 11463, 13513, 7372, 7373, 12316, 9423, 11596, 3285, 5334, 2937, 1242, 3291, 11083, 1248, 13538, 9445, 5350, 17639, 7402, 13863, 15596, 3309, 11502, 14704, 9460, 6064, 1273, 13562, 15611, 5374, 12501, 11520, 13569, 9475, 9430, 11526, 5383, 3336, 5386, 15575, 1294, 10797, 5395, 11541, 7447, 3352, 3354, 9499, 1309, 13598, 15237, 5410, 7387, 11556, 7477, 7462, 7729, 3368, 9514, 16022, 11571, 1333, 13622, 15672, 9529, 1339, 13628, 11296, 9535, 3392, 5441, 7493, 3398, 15687, 13643, 8774, 9550, 8024, 11602, 1363, 7508, 3413, 15702, 13881, 5466, 5472, 11617, 7820, 1379, 3428, 15590, 9575, 13673, 17770, 13276, 5487, 1394, 3443, 17784, 14247, 11643, 7548, 9597, 3648, 1410, 10208, 13703, 17644, 11658, 3467, 9612, 5518, 3473, 15762, 11667, 14871, 14574, 12800, 1431, 9627, 3488, 16283, 7750, 1447, 11690, 15787, 5550, 3503, 5552, 15793, 9651, 9660, 13757, 3518, 1471, 15808, 13763, 7623, 9672, 12876, 17868, 7629, 1486, 14575, 14017, 5585, 13778, 1492, 11711, 3542, 7417, 3548, 5597, 13221, 17888, 13793, 10960, 9702, 15848, 7659, 5612, 1517, 253, 11371, 3572, 9717, 11766, 6740, 3578, 5627, 11772, 1533, 15870, 13824, 1539, 9732, 3593, 5642, 11787, 15885, 13839, 1554, 9747, 5654, 15961, 3608, 7705, 11802, 11183, 17948, 1569, 9762, 17958, 3623, 5672, 11441, 14599, 15916, 9138, 7554, 1288, 11829, 7735, 13236, 15931, 15285, 1600, 5697, 11844, 14262, 3654, 13896, 12215, 13902, 9807, 5712, 11859, 7765, 5718, 1624, 11865, 17679, 16177, 13917, 1630, 10853, 955, 5733, 9831, 16316, 13932, 7789, 1646, 11889, 15635, 5748, 3701, 9846, 11895, 13947, 7804, 4715, 3716, 1670, 11911, 9864, 18028, 3722, 1676, 18061, 9870, 12568, 7826, 10558, 4718, 11926, 15741, 13977, 13050, 5787, 9885, 16034, 3747, 1701, 3753, 1707, 9901, 11950, 5809, 14002, 14605, 16052, 3765, 12916, 1722, 9916, 17354, 15717, 7873, 15947, 11972, 18038, 3783, 1737, 9931, 16076, 15650, 5840, 3798, 9946, 16091, 11996, 14925, 5855, 13592, 1761, 12002, 18644, 3814, 1767, 9961, 12583, 5870, 10877, 15656, 12018, 3828, 14069, 7928, 16122, 1792, 9985, 3843, 14084, 5895, 16140, 7949, 1807, 12048, 7698, 5763, 1813, 5910, 3864, 17028, 5916, 12063, 3873, 16162, 1829, 11228, 7979, 12078, 3888, 14129, 1844, 5944, 15327, 5949, 10046, 3903, 14144, 7277, 8003, 1860, 12598, 14756, 5961, 12269, 13282, 3918, 14159, 17720, 8018, 10068, 1877, 5976, 16217, 3933, 14174, 15333, 16226, 5435, 12132, 1893, 5991, 13065, 16362, 10098, 12147, 1908, 3957, 8054, 14199, 10160, 3963, 16253, 6015, 1923, 8069, 14216, 12169, 10123, 16268, 14222, 6031, 1938, 3987, 8084, 6039, 12184, 3993, 10139, 17591, 12613, 1953, 9792, 13297, 4008, 6812, 10154, 8108, 1968, 12209, 8114, 6070, 4425, 14268, 16317, 12861, 1983, 4033, 8130, 6085, 12230, 4039, 8727, 10185, 14283, 1998, 10193, 10194, 12963, 16340, 12245, 4055, 16346, 6112, 8163, 14308, 4069, 6118, 2023, 12547, 14314, 15015, 18071, 2029, 10223, 7569, 8178, 6133, 14327, 4088, 4091, 16844, 12285, 10238])):
matches = {k : v for k, v in matches.iteritems() if h not in k}
> latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
prevh = __find_md5__.search(latestmatch).group(0)IndexError: list index out of range
Quoted 104 lines
Signed-off-by Richard Purdie <[email protected]>---
bitbake/lib/bb/runqueue.py +77
-0@@ -1039,6 +1039,7 @@ class RunQueue: if self.state is runQueueSceneInit: if self.cooker.configuration.dump_signatures: + self.print_diffscenetasks() self.dump_signatures() else: self.start_worker()@@ -1123,6 +1124,82 @@ class RunQueue: return + def print_diffscenetasks(self): + + valid = [] + sq_hash = [] + sq_hashfn = [] + sq_fn = [] + sq_taskname = [] + sq_task = [] + noexec = [] + stamppresent = [] + valid_new = set() + + for task in xrange(len(self.rqdata.runq_fnid)): + fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]] + taskname = self.rqdata.runq_task[task] + taskdep = self.rqdata.dataCache.task_deps[fn] + + if 'noexec' in taskdep and taskname in taskdep['noexec']: + noexec.append(task) + continue + + sq_fn.append(fn) + sq_hashfn.append(self.rqdata.dataCache.hashfn[fn]) + sq_hash.append(self.rqdata.runq_hash[task]) + sq_taskname.append(taskname) + sq_task.append(task) + call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)" + locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data } + valid = bb.utils.better_eval(call, locs) + for v in valid: + valid_new.add(sq_task[v]) + + # Tasks which are both setscene and noexec never care about dependencies + # We therefore find tasks which are setscene and noexec and mark their + # unique dependencies as valid. + for task in noexec: + if task not in self.rqdata.runq_setscene: + continue + for dep in self.rqdata.runq_depends[task]: + hasnoexecparents = True + for dep2 in self.rqdata.runq_revdeps[dep]: + if dep2 in self.rqdata.runq_setscene and dep2 in noexec: + continue + hasnoexecparents = False + break + if hasnoexecparents: + valid_new.add(dep) + + invalidtasks = set() + for task in xrange(len(self.rqdata.runq_fnid)): + if task not in valid_new and task not in noexec: + invalidtasks.add(task) + + found = set() + processed = set() + for task in invalidtasks: + toprocess = set([task]) + while toprocess: + next = set() + for t in toprocess: + for dep in self.rqdata.runq_depends[t]: + if dep in invalidtasks: + found.add(task) + if dep not in processed: + processed.add(dep) + next.add(dep) + toprocess = next + if task in found: + toprocess = set() + + tasklist = [] + for task in invalidtasks.difference(found): + tasklist.append(self.rqdata.get_user_idstring(task)) + + if tasklist: + bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) class RunQueueExecute:_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 11 lines · Martin Jansa
Quoted 1 line · Richard Purdie
Its useful to understand where the delta starts against an existing
sstate cache
Quoted 1 line
for a given target. Adding this to the output of the -S option seems
like a
Quoted 3 lines
natural fit.
We use the hashvalidate function to figure this out and assume it can
find siginfo
Quoted 1 line
files for more than just the setscene tasks.
Quoted 7 lines
I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
I’m hitting this now as well.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
RFC PATCH 1/1 runqueue: Don't fail without match found
* WIP, because I haven't checked the reasons why this happens I'm using this now only to see which recipes/tasks are failing like this
--- lib/bb/runqueue.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
lib/bb/runqueue.py +8 -4
@@ -1224,10 +1224,14 @@ class RunQueue: if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s)?" % h) matches = {k : v for k, v in matches.iteritems() if h not in k} - latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] - prevh = __find_md5__.search(latestmatch).group(0) - output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) - bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) + sortedmatches=sorted(matches.keys(), key=lambda f: matches[f]) + if sortedmatches: + latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] + prevh = __find_md5__.search(latestmatch).group(0) + output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) + bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) + else: + bb.plain("\nTask %s:%s couldn't find any matches for hash %s" % (pn, taskname, h))
Signature · 1 line
1.8.4.3
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie
Its useful to understand where the delta starts against an existing
sstate cache
Quoted 1 line
for a given target. Adding this to the output of the -S option seems
like a
Quoted 3 lines
natural fit.
We use the hashvalidate function to figure this out and assume it can
find siginfo
Quoted 1 line
files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with
Quoted 7 lines
Quoted 4 lines
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
I’m hitting this now as well.
To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.
Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 30 lines · Chris Larson
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie
Its useful to understand where the delta starts against an existing
sstate cache
Quoted 1 line
for a given target. Adding this to the output of the -S option seems
like a
Quoted 3 lines
natural fit.
We use the hashvalidate function to figure this out and assume it can
find siginfo
Quoted 1 line
files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with
Quoted 7 lines
Quoted 4 lines
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
I’m hitting this now as well.
To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.
Quoted 7 lines
Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?
Same here, I'm using bitbake -S fromoe-core/scripts/sstate-diff-machines.sh
and this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).
It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).
I see it for many do_fetch and do_rm_work_all tasks.
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 46 lines · Martin Jansa
Quoted 30 lines · Chris Larson
Quoted 16 lines · Chris Larson
Quoted 14 lines · Martin Jansa
Quoted 1 line · Richard Purdie
Its useful to understand where the delta starts against an existing
sstate cache
Quoted 1 line
for a given target. Adding this to the output of the -S option seems
like a
Quoted 3 lines
natural fit.
We use the hashvalidate function to figure this out and assume it can
find siginfo
Quoted 1 line
files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with
Quoted 7 lines
Quoted 4 lines
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
I’m hitting this now as well.
To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.
Quoted 7 lines
Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?
Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.shand this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).
Quoted 5 lines
It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).
I see it for many
do_fetchanddo_rm_work_alltasks.
To put things in context, I was trying to work with the existing "whatchanged" type tools or more specifically I wanted to know "why isn't my sstate cache being used". Having to do a two step process for any of these enquires seems a little odd. There are also two specific and as it turns our quite different work flows:
a) What changed since my last build?
b) What is the least delta between my current build and whatever I have cached?
Those experiments revealed some big holes in the sstate metadata and some good improvements there. It also highlights that there are two different questions which have different answers.
In the process, I found bitbake does have some state in runqueue which can help pinpoint the "whatchanged" part of the problem much more accurately/quickly. Obviously that state is lost after bitbake -S finishes so I started printing out some extra data.
Clearly bitbake shouldn't crash, that is a bug and I think Martin is heading in the right direction with the other patch.
As for bitbake -S being kept fast, I think (but I need to double check with measurements) that the parsing part of bitbake -S is the slow part and this extra data is effectively "for free".
So basically consider the current state an experiment to see if we can make -S more useful/interesting (without much cost) and to try to better understand where/what functionality we need going forward.
One possible outcome might be we generate an extra file (or files) (like we do with task-depends.dot) which contains this extra tree data which some other command can then parse and display to the user. I'm still struggling to get a complete grip on the exact data and format that file needs to be. Part of the reason for having -S print something was to see if it printed what the user wanted to know or if not, we could then try and figure out what the right thing was so we could improve it.
So sorry if this is causing pain, the code is easy to disable by commenting out the call, or reverting the patch adding it. Equally, I think we need to make sstate hashes and reasons for rebuilds more accessible to users and I think this does move us closer to that assuming we fix the crash.
I'm trying to take a bit of a break over the holiday period (been riding a motorcycle in driving snow/rain today and am rather damp/cold) but I'll get the crash fixed sometime next week at the latest.
Cheers,
Richard
Quoted 85 lines · Richard Purdie
Quoted 2 lines · Martin Jansa
Quoted 1 line · Chris Larson
On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <[email protected]<javascript:;>>
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <
Quoted 3 lines · [email protected] <javascript:;>>wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Richard Purdie
Quoted 1 line
Its useful to understand where the delta starts against an
existing
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
sstate cache
Quoted 1 line
for a given target. Adding this to the output of the -S option
seems
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
like a
Quoted 3 lines
natural fit.
We use the hashvalidate function to figure this out and assume it
can
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
find siginfo
Quoted 1 line
files for more than just the setscene tasks.
I haven't tried to find smaller reproducer or exact part of code
where
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
it happends, but in bigger builds I have it fails with
I'll try to reproduce and narrow it a bit, but if you have tip where to look first..
IndexError: list index out of range:
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
I’m hitting this now as well.
To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base.
Actually, this is getting in my way trying to do something I don’t
want it
Quoted 1 line
Quoted 1 line
doing anyway. I want to dump the signature data from one config, and
use
Quoted 1 line
Quoted 1 line
bitbake-whatchanged from another build. If I wanted to compare
something
Quoted 12 lines
Quoted 4 lines
right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare?
Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.shand this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple).
Quoted 5 lines
It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image).
I see it for many
do_fetchanddo_rm_work_alltasks.To put things in context, I was trying to work with the existing "whatchanged" type tools or more specifically I wanted to know "why isn't my sstate cache being used". Having to do a two step process for any of these enquires seems a little odd. There are also two specific and as it turns our quite different work flows:
a) What changed since my last build?
b) What is the least delta between my current build and whatever I have cached?Those experiments revealed some big holes in the sstate metadata and some good improvements there. It also highlights that there are two different questions which have different answers.
In the process, I found bitbake does have some state in runqueue which can help pinpoint the "whatchanged" part of the problem much more accurately/quickly. Obviously that state is lost after bitbake -S finishes so I started printing out some extra data.
Clearly bitbake shouldn't crash, that is a bug and I think Martin is heading in the right direction with the other patch.
Quoted 8 lines
As for bitbake -S being kept fast, I think (but I need to double check with measurements) that the parsing part of bitbake -S is the slow part and this extra data is effectively "for free".
So basically consider the current state an experiment to see if we can make -S more useful/interesting (without much cost) and to try to better understand where/what functionality we need going forward.
I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My SSTATE_DIR has weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this?
-Chris
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 8 lines · Chris Larson
I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My
SSTATE_DIRhas weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm
Quoted 8 lines
misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate
STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this?
Nothing changed in the data -S writes out.
Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.
On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.
So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).
That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.
I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).
I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.
Cheers,
Richard
Quoted 5 lines · Richard Purdie
So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).
I've finally got some numbers to support my claim that it "feels" slower with this added functionality.
My script is callingopenembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;
which in turn runs
bitbake -S world
three times (once for each machine).
The script on jenkins doing world builds is completely crazy (I've killed it in last run assuming it was stuck, but it wasn't just terribly slow).
http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
Running 21 hours already and doing 2nd machine from 3 (maybe because longer SSTATE_MIRROR?). There is one python process using 100% cpu.
In local builds without SSTATE_MIRROR it's not so bad, but still significantly slower with each release (dylan-dora-master).
Master builds have that WIP patch included in order to finish.
It's not very accurate, because it was running on my desktop and each one has different metadata, but at least the layers included are similar (only systems which got separated in dora and removal
of meta-webos-backports which isn't needed for dora and newer).
At least I've executed each test twice.
dylan:
real 9m37.271s user 6m37.104s sys 3m12.625s
real 10m5.787s user 6m37.832s sys 3m23.614s
dora:
real 26m11.374s user 9m7.325s sys 3m0.048s
real 28m10.606s user 8m27.895s sys 2m57.497s
master:
real 38m17.641s user 17m46.905s sys 5m12.061s
real 36m12.748s user 16m32.961s sys 4m43.103s
master-without-S-changes:
real 27m34.228s user 8m6.811s sys 2m53.724s
real 27m49.752s user 8m25.026s sys 2m50.921s
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 30 lines · Richard Purdie
Nothing changed in the data -S writes out.
Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.
On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.
So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).
That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.
I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).
Quoted 4 lines
I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.
I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.
With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.
I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.
I don't have any code to do this yet but at least we have a plan assuming no strong objections.
Cheers,
Richard
Quoted 45 lines · Richard Purdie
Quoted 30 lines · Richard Purdie
Nothing changed in the data -S writes out.
Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now.
On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against.
So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).
That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data.
I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of).
Quoted 4 lines
I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(.
I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.
With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.
Quoted 6 lines
I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.
I don't have any code to do this yet but at least we have a plan assuming no strong objections.
I strongly support this plan :).
Cheers,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 17 lines · Martin Jansa
Quoted 5 lines · Richard Purdie
So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose).
I've finally got some numbers to support my claim that it "feels" slower with this added functionality.
My script is calling
openembedded-core/scripts/sstate-diff-machines.sh--machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;which in turn runs
bitbake -S world
three times (once for each machine).
Quoted 6 lines
The script on jenkins doing world builds is completely crazy (I've killed it in last run assuming it was stuck, but it wasn't just terribly slow).
http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
Running 21 hours already and doing 2nd machine from 3 (maybe because longerSSTATE_MIRROR?). There is one python process using 100% cpu.
FWIW: this jenkins job was completed yesterday after 16 days
small part of strace:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/log.strace
Previous runs of the same job took 25-40 minutes
Quoted 50 lines
In local builds without
SSTATE_MIRRORit's not so bad, but still significantly slower with each release (dylan-dora-master).Master builds have that WIP patch included in order to finish.
It's not very accurate, because it was running on my desktop and each one has different metadata, but at least the layers included are similar (only systems which got separated in dora and removal
of meta-webos-backports which isn't needed for dora and newer).At least I've executed each test twice.
dylan:
real 9m37.271s user 6m37.104s sys 3m12.625sreal 10m5.787s user 6m37.832s sys 3m23.614sdora:
real 26m11.374s user 9m7.325s sys 3m0.048sreal 28m10.606s user 8m27.895s sys 2m57.497smaster:
real 38m17.641s user 17m46.905s sys 5m12.061sreal 36m12.748s user 16m32.961s sys 4m43.103smaster-without-S-changes:
real 27m34.228s user 8m6.811s sys 2m53.724sreal 27m49.752s user 8m25.026s sys 2m50.921sSignature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 4 lines · Richard Purdie
I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.
Quoted 8 lines
With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.
I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.
So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.
It has good reason for this as its near impossible for optparse to tell the difference between:
bitbake image -S tracesigs
and
bitbake image -S image2
So our options are:
a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different name
In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.
Any opinions?
Cheers,
Richard
Quoted 33 lines · Richard Purdie
Quoted 4 lines · Richard Purdie
I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.
Quoted 8 lines
With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.
I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.
So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.
It has good reason for this as its near impossible for optparse to tell the difference between:
bitbake image -S tracesigs
and
bitbake image -S image2So our options are:
a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different name
Quoted 6 lines
In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.
Any opinions?
What about using some existing parameter in combination with -S?
-S + -v shows tracesigs
-S without -v old behavior
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 43 lines · Martin Jansa
Quoted 33 lines · Richard Purdie
Quoted 4 lines · Richard Purdie
I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically.
Quoted 8 lines
With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block.
I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people.
So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified.
It has good reason for this as its near impossible for optparse to tell the difference between:
bitbake image -S tracesigs
and
bitbake image -S image2So our options are:
a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy
c) Add a new option with a different nameQuoted 6 lines
In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed.
Any opinions?
What about using some existing parameter in combination with -S?
Review
Quoted 2 lines
(quoted diff) +0
-2-S + -v shows tracesigs -S without -v old behavior
I have some use cases locally (such as locked sigs) which could really benefit from a string parameter which gets passed into the siggen code. I can see us needing more than two "behaviours" of -S...
I also think using something like -v will lead to more confusion and the code won't be particularly nice either :/.
Cheers,
Richard