Skip to contentSkip to conversation

Build profiling, now in pictures

PATCH Richard Purdie 2 participants 4 messages Feb 15, 2011 – Feb 17, 2011

lore
Share & tools
lore
#0 lore raw

We have just merged Beth's initial buildstats logging work. I was sitting wondering how to actually evaluate the numbers as I wanted to know "where are we spending the time?".

It occurred to me that I wanted a graph very similar to that generated by bootchart. I looked around and found pyboootchartgui and then hacked it around a bit and coerced it to start producing charts like:

http://tim.rpsys.net/bootchart.png

which is the initial "pseudo-native" part of the build. This was simple enough to test with.

I then tried graphing a poky-image-sato. To get a graph I could actually read, I stripped out any task taking less than 8 seconds and scaled the x axis from 25 units per second to one unit per second. The result was:

http://tim.rpsys.net/bootchart2.png
(warning this is a 2.7MB png)

I also added in a little bit of colour coding for the second chart. Interestingly it looks like there is more yellow than green meaning configure is a bigger drain on the build time not that its unexpected :/.

I quite enjoyed playing with this and on a serious note, the gradient of the task graph makes me a little suspicious of whether the overhead of launching tasks in bitbake itself is having some effect on build time. Certainly on the first graph there are some interesting latencies showing up.

Anyhow, I think this is the first time bitbake's task execution has been visualised and there are some interesting things we can learn from it. I'm hoping this is a start of a much more detailed understanding of the build process with respect to performance.

Cheers,

Richard

draw.py +91 -39
Index: pybootchartgui/draw.py
===================================================================
@@ -53,7 +53,7 @@
 	
 # Process border color.
 PROC_BORDER_COLOR = (0.71, 0.71, 0.71, 1.0)
-# Waiting process color.
+
 PROC_COLOR_D = (0.76, 0.48, 0.48, 0.125)
 # Running process color.
 PROC_COLOR_R = CPU_COLOR
@@ -88,6 +88,17 @@
 # Process description date format.
 DESC_TIME_FORMAT = "mm:ss.SSS"
 
+# Configure task color
+TASK_COLOR_CONFIGURE = (1.0, 1.0, 0.00, 1.0)
+# Compile task color.
+TASK_COLOR_COMPILE = (0.0, 1.00, 0.00, 1.0)
+# Install task color
+TASK_COLOR_INSTALL = (1.0, 0.00, 1.00, 1.0)
+# Package task color
+TASK_COLOR_PACKAGE = (0.0, 1.00, 1.00, 1.0)
+# Sysroot task color
+TASK_COLOR_SYSROOT = (0.0, 0.00, 1.00, 1.0)
+
 # Process states
 STATE_UNDEFINED = 0
 STATE_RUNNING   = 1
@@ -147,7 +158,7 @@
 def draw_5sec_labels(ctx, rect, sec_w):
         ctx.set_font_size(AXIS_FONT_SIZE)
 	for i in range(0, rect[2] + 1, sec_w):
-		if ((i / sec_w) % 5 == 0) :
+		if ((i / sec_w) % 30 == 0) :
 			label = "%ds" % (i / sec_w)
 			label_w = ctx.text_extents(label)[2]
 			draw_text(ctx, label, TEXT_COLOR, rect[0] + i - label_w/2, rect[1] - 2)
@@ -159,7 +170,7 @@
 	ctx.set_line_cap(cairo.LINE_CAP_SQUARE)
 
 	for i in range(sec_w, rect[2] + 1, sec_w):
-		if ((i / sec_w) % 5 == 0) :
+		if ((i / sec_w) % 30 == 0) :
 			ctx.set_source_rgba(*TICK_COLOR_BOLD)
 		else :
 			ctx.set_source_rgba(*TICK_COLOR)
@@ -204,23 +215,26 @@
 bar_h = 55
 # offsets
 off_x, off_y = 10, 10
-sec_w = 25 # the width of a second
+sec_w = 1 # the width of a second
 proc_h = 16 # the height of a process
 leg_s = 10
 MIN_IMG_W = 800
 OPTIONS = None
 
 
-def extents(headers, cpu_stats, disk_stats, proc_tree):
-	w = (proc_tree.duration * sec_w / 100) + 2*off_x
-	h = proc_h * proc_tree.num_proc + header_h + 2*off_y
+def extents(res):
+        start = min(res.start.keys())
+        end = max(res.end.keys())
+
+	w = ((end - start) * sec_w) + 2*off_x
+	h = proc_h * len(res.processes) + header_h + 2*off_y
 	return (w,h)
 
 #
 # Render the chart.
 # 
-def render(ctx, options, headers, cpu_stats, disk_stats, proc_tree):
-	(w, h) = extents(headers, cpu_stats, disk_stats, proc_tree)
+def render(ctx, options, res):
+	(w, h) = extents(res)
 
 	global OPTIONS
 	OPTIONS = options
@@ -231,65 +245,103 @@
 	draw_fill_rect(ctx, WHITE, (0, 0, max(w, MIN_IMG_W), h))
 	w -= 2*off_x    
 	# draw the title and headers
-	curr_y = draw_header(ctx, headers, off_x, proc_tree.duration)
+	#curr_y = draw_header(ctx, headers, off_x, proc_tree.duration)
+        curr_y = 0
     
 	# render bar legend
 	ctx.set_font_size(LEGEND_FONT_SIZE)
 
-	draw_legend_box(ctx, "CPU (user+sys)", CPU_COLOR, off_x, curr_y+20, leg_s)
-	draw_legend_box(ctx, "I/O (wait)", IO_COLOR, off_x + 120, curr_y+20, leg_s)
+        #print "w, h %s %s" % (w, h)
 
+	#draw_legend_box(ctx, "CPU (user+sys)", CPU_COLOR, off_x, curr_y+20, leg_s)
+	#draw_legend_box(ctx, "I/O (wait)", IO_COLOR, off_x + 120, curr_y+20, leg_s)
+
 	# render I/O wait
-        chart_rect = (off_x, curr_y+30, w, bar_h)
-	draw_box_ticks(ctx, chart_rect, sec_w)
-	draw_chart(ctx, IO_COLOR, True, chart_rect, [(sample.time, sample.user + sample.sys + sample.io) for sample in cpu_stats], proc_tree) 
+        #chart_rect = (off_x, curr_y+30, w, bar_h)
+	#draw_box_ticks(ctx, chart_rect, sec_w)
+	#draw_chart(ctx, IO_COLOR, True, chart_rect, [(sample.time, sample.user + sample.sys + sample.io) for sample in cpu_stats], proc_tree) 
 	# render CPU load
-	draw_chart(ctx, CPU_COLOR, True, chart_rect, [(sample.time, sample.user + sample.sys) for sample in cpu_stats], proc_tree)
+	#draw_chart(ctx, CPU_COLOR, True, chart_rect, [(sample.time, sample.user + sample.sys) for sample in cpu_stats], proc_tree)
 
-	curr_y = curr_y + 30 + bar_h
+	#curr_y = curr_y + 30 + bar_h
 
 	# render second chart
-	draw_legend_line(ctx, "Disk throughput", DISK_TPUT_COLOR, off_x, curr_y+20, leg_s)
-	draw_legend_box(ctx, "Disk utilization", IO_COLOR, off_x + 120, curr_y+20, leg_s)
+	#draw_legend_line(ctx, "Disk throughput", DISK_TPUT_COLOR, off_x, curr_y+20, leg_s)
+	#draw_legend_box(ctx, "Disk utilization", IO_COLOR, off_x + 120, curr_y+20, leg_s)
 
         # render I/O utilization
-	chart_rect = (off_x, curr_y+30, w, bar_h)
-	draw_box_ticks(ctx, chart_rect, sec_w)			
-	draw_chart(ctx, IO_COLOR, True, chart_rect, [(sample.time, sample.util) for sample in disk_stats], proc_tree)
+	#chart_rect = (off_x, curr_y+30, w, bar_h)
+	#draw_box_ticks(ctx, chart_rect, sec_w)			
+	#draw_chart(ctx, IO_COLOR, True, chart_rect, [(sample.time, sample.util) for sample in disk_stats], proc_tree)
 				
 	# render disk throughput
-	max_sample = max(disk_stats, key=lambda s: s.tput)
-	draw_chart(ctx, DISK_TPUT_COLOR, False, chart_rect, [(sample.time, sample.tput) for sample in disk_stats], proc_tree)
+	#max_sample = max(disk_stats, key=lambda s: s.tput)
+	#draw_chart(ctx, DISK_TPUT_COLOR, False, chart_rect, [(sample.time, sample.tput) for sample in disk_stats], proc_tree)
 	
-	pos_x = off_x + ((max_sample.time - proc_tree.start_time) * w / proc_tree.duration)
+	#pos_x = off_x + ((max_sample.time - proc_tree.start_time) * w / proc_tree.duration)
+        pos_x = off_x
 
 	shift_x, shift_y = -20, 20
 	if (pos_x < off_x + 245):
 		shift_x, shift_y = 5, 40
        				
-	label = "%dMB/s" % round((max_sample.tput) / 1024.0)
-	draw_text(ctx, label, DISK_TPUT_COLOR, pos_x + shift_x, curr_y + shift_y)
+	#label = "%dMB/s" % round((max_sample.tput) / 1024.0)
+	#draw_text(ctx, label, DISK_TPUT_COLOR, pos_x + shift_x, curr_y + shift_y)
 
 
-	# draw process boxes
-	draw_process_bar_chart(ctx, proc_tree, curr_y + bar_h, w, h)
 
-	ctx.set_font_size(SIG_FONT_SIZE)
-	draw_text(ctx, SIGNATURE, SIG_COLOR, off_x + 5, h - off_y - 5)
+        chart_rect = [off_x, curr_y+60, w, h - 2 * off_y - (curr_y+60) + proc_h]
+
+	draw_legend_box(ctx, "Configure", 	TASK_COLOR_CONFIGURE, off_x    , curr_y + 45, leg_s)		
+	draw_legend_box(ctx, "Compile", 	TASK_COLOR_COMPILE, off_x+120, curr_y + 45, leg_s)
+	draw_legend_box(ctx, "Install", 	TASK_COLOR_INSTALL, off_x+240, curr_y + 45, leg_s)
+	draw_legend_box(ctx, "Package", 	TASK_COLOR_PACKAGE, off_x+360, curr_y + 45, leg_s)
+	draw_legend_box(ctx, "Populate Sysroot",	TASK_COLOR_SYSROOT, off_x+480, curr_y + 45, leg_s)
 	
-def draw_process_bar_chart(ctx, proc_tree, curr_y, w, h):
-	draw_legend_box(ctx, "Running (%cpu)", 		PROC_COLOR_R, off_x    , curr_y + 45, leg_s)		
-	draw_legend_box(ctx, "Unint.sleep (I/O)", 	PROC_COLOR_D, off_x+120, curr_y + 45, leg_s)
-	draw_legend_box(ctx, "Sleeping", 		PROC_COLOR_S, off_x+240, curr_y + 45, leg_s)
-	draw_legend_box(ctx, "Zombie", 			PROC_COLOR_Z, off_x+360, curr_y + 45, leg_s)
-	
-	chart_rect = [off_x, curr_y+60, w, h - 2 * off_y - (curr_y+60) + proc_h]
 	ctx.set_font_size(PROC_TEXT_FONT_SIZE)
 	
 	draw_box_ticks(ctx, chart_rect, sec_w)
 	draw_5sec_labels(ctx, chart_rect, sec_w)
-        
+
 	y = curr_y+60
+
+        offset = min(res.start.keys())
+        for s in sorted(res.start.keys()):
+            task = res.start[s].split(":")[1]
+            #print res.start[s]
+            #print res.processes[res.start[s]][1]
+            #print s
+            x = (s - offset) * sec_w
+            w = ((res.processes[res.start[s]][1] - s) * sec_w)
+
+            #print "proc at %s %s %s %s" % (x, y, w, proc_h)
+            col = None
+            if task == "do_compile":
+                col = TASK_COLOR_COMPILE
+            elif task == "do_configure":
+                col = TASK_COLOR_CONFIGURE
+            elif task == "do_install":
+                col = TASK_COLOR_INSTALL
+            elif task == "do_package":
+                col = TASK_COLOR_PACKAGE
+            elif task == "do_populate_sysroot":
+                col = TASK_COLOR_SYSROOT
+
+            draw_rect(ctx, PROC_BORDER_COLOR, (x, y, w, proc_h))
+            if col:
+ 		draw_fill_rect(ctx, col, (x, y, w, proc_h))
+
+            draw_label_in_box(ctx, PROC_TEXT_COLOR, res.start[s], x, y + proc_h - 4, w, proc_h)
+            y = y + proc_h
+
+	# draw process boxes
+	#draw_process_bar_chart(ctx, proc_tree, curr_y + bar_h, w, h)
+
+	ctx.set_font_size(SIG_FONT_SIZE)
+	draw_text(ctx, SIGNATURE, SIG_COLOR, off_x + 5, h - off_y - 5)
+	
+def draw_process_bar_chart(ctx, proc_tree, curr_y, w, h):
+
 	for root in proc_tree.process_tree:        
 		draw_processes_recursively(ctx, root, proc_tree, y, proc_h, chart_rect)
 		y  = y + proc_h * proc_tree.num_nodes([root])
batch.py +2 -2
Index: pybootchartgui/batch.py
===================================================================
@@ -34,11 +34,11 @@
         return 10
 
     make_surface, write_surface = handlers[format]
-    w,h = draw.extents(*res)
+    w,h = draw.extents(res)
     w = max(w, draw.MIN_IMG_W)
     surface = make_surface(w,h)
     ctx = cairo.Context(surface)
-    draw.render(ctx, options, *res)
+    draw.render(ctx, options, res)
     write_surface(surface)
     writer.status("bootchart written to '%s'" % filename)
 
parsing.py +26 -28

78 lines not on this page · raw message

main.py +2 -0
Index: pybootchartgui/main.py
===================================================================
@@ -102,6 +102,8 @@
 			
 
 		res = parsing.parse(writer, args, options.prune)
+
+                #print str(res)
 		
 		if options.interactive or options.output == None:
 			gui.show(res, options)
#1 lore raw

This is a chart of pseudo-native as of master before the last few bitbake commits:

http://tim.rpsys.net/bootchart-pseudo1.png

and after the recent bitbake commits, this is the result:

http://tim.rpsys.net/bootchart-pseudo1.png

Whats interesting in the gradient of the rapid fire tasks at the start of the graph. There is now much less "free" space between the tasks and task execution and throughput was therefore improved and the gradient much sharper.

Looking at the graphs, the new task execution order is suboptimal though, mainly as I changed something there that before realising there was another problem that was making this hurt more than it should.

I think reverting:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=974ea1a190167dcfd831ba1fc5f733e0dc9a6fda

should help parallelism a little further.

I'm running a full build to see what this means in read world numbers. It certainly executed all the non-critical path tasks faster with these changes. I pulled the graph of the first 1000 tasks and they executed in a faster time period, its difficult to say what the effect on the overall build will be at this point though.

Interesting what some numbers can do to point out problems though.

Just for the record, I used a dummy recipe of empty tasks to test how fast the execution code was combined with bitbake profiling and some suitable tweaks to the looping delays to make it clear when bitbake was sleeping and shouldn't be. These changes dropped the test time from 9 seconds to 5 (where the timing includes bitbake startup and cache loading overheads).

Cheers,

Richard

#2 lore raw
In reply to Richard Purdie
Quoted 31 lines · Richard Purdie

We have just merged Beth's initial buildstats logging work. I was sitting wondering how to actually evaluate the numbers as I wanted to know "where are we spending the time?".

It occurred to me that I wanted a graph very similar to that generated by bootchart. I looked around and found pyboootchartgui and then hacked it around a bit and coerced it to start producing charts like:

http://tim.rpsys.net/bootchart.png

which is the initial "pseudo-native" part of the build. This was simple enough to test with.

I then tried graphing a poky-image-sato. To get a graph I could actually read, I stripped out any task taking less than 8 seconds and scaled the x axis from 25 units per second to one unit per second. The result was:

http://tim.rpsys.net/bootchart2.png
(warning this is a 2.7MB png)

I also added in a little bit of colour coding for the second chart. Interestingly it looks like there is more yellow than green meaning configure is a bigger drain on the build time not that its unexpected :/.

I quite enjoyed playing with this and on a serious note, the gradient of the task graph makes me a little suspicious of whether the overhead of launching tasks in bitbake itself is having some effect on build time. Certainly on the first graph there are some interesting latencies showing up.

Quoted 5 lines

Anyhow, I think this is the first time bitbake's task execution has been visualised and there are some interesting things we can learn from it. I'm hoping this is a start of a much more detailed understanding of the build process with respect to performance.

Per Richard's request, I ran this on a large Westmere machine with 40 physical cores (10 per socket).

BB_NUMBER_THREADS = "40"
PARALLEL_MAKE = "-j 40"
MACHINE=qemux86

Here's the poky-image-sato "buildchart" (1.6M):

http://dvhart.com/darren/yocto/buildchart-sato.png

For reference, total build time:

real    68m57.268s
user    424m23.566s
sys     211m30.272s

The higher core count results in a steeper gradient and makes the serialized build sections a bit more prominent. The two obvious serialization points being "eglibc" and the "openssl,ncurses,gettext,readline,sqlite,augeas,python,boost,perl" sequence.

--
Darren

Quoted 10 lines

Cheers,

Richard

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#3 lore raw
Quoted 55 lines · Darren Hart
Quoted 31 lines · Richard Purdie

We have just merged Beth's initial buildstats logging work. I was sitting wondering how to actually evaluate the numbers as I wanted to know "where are we spending the time?".

It occurred to me that I wanted a graph very similar to that generated by bootchart. I looked around and found pyboootchartgui and then hacked it around a bit and coerced it to start producing charts like:

http://tim.rpsys.net/bootchart.png

which is the initial "pseudo-native" part of the build. This was simple enough to test with.

I then tried graphing a poky-image-sato. To get a graph I could actually read, I stripped out any task taking less than 8 seconds and scaled the x axis from 25 units per second to one unit per second. The result was:

http://tim.rpsys.net/bootchart2.png
(warning this is a 2.7MB png)

I also added in a little bit of colour coding for the second chart. Interestingly it looks like there is more yellow than green meaning configure is a bigger drain on the build time not that its unexpected :/.

I quite enjoyed playing with this and on a serious note, the gradient of the task graph makes me a little suspicious of whether the overhead of launching tasks in bitbake itself is having some effect on build time. Certainly on the first graph there are some interesting latencies showing up.

Quoted 5 lines

Anyhow, I think this is the first time bitbake's task execution has been visualised and there are some interesting things we can learn from it. I'm hoping this is a start of a much more detailed understanding of the build process with respect to performance.

Per Richard's request, I ran this on a large Westmere machine with 40 physical cores (10 per socket).

BB_NUMBER_THREADS = "40"
PARALLEL_MAKE = "-j 40"
MACHINE=qemux86

Here's the poky-image-sato "buildchart" (1.6M):

http://dvhart.com/darren/yocto/buildchart-sato.png

For reference, total build time:

real 68m57.268s
user 424m23.566s
sys 211m30.272s

Quoted 5 lines

The higher core count results in a steeper gradient and makes the serialized build sections a bit more prominent. The two obvious serialization points being "eglibc" and the "openssl,ncurses,gettext,readline,sqlite,augeas,python,boost,perl" sequence.

Note: I am not the only one running jobs on the 40-way, but the CPU utilization has been under 30% during my periodic spot checks.

With poky/master 27894c58b74fb01530aa7144b8e8c7a2298187b4 with perl-native forced to use PARALLEL_MAKE="" we get:

http://dvhart.com/darren/yocto/buildchart-pre-perl.png

real    58m28.575s
user    421m47.013s
sys     193m12.428s

The second pinch point (gettext,python,etc.) seems to be less pronounced, and we're 10 minutes faster (but that shouldn't be taken too literally with other jobs running on this system).

With contrib/nitin/misc 42a0ae1220f9add4078f207086ed7f9c859de0af which includes a fix for perl-native PARALLEL_MAKE:

http://dvhart.com/darren/yocto/buildchart-post-perl.png

real    57m42.992s
user    423m5.026s
sys     197m50.407s

Visual inspections suggest the second pinch point is further reduced here.

--
Darren

Quoted 16 lines

--
Darren

Quoted 10 lines

Cheers,

Richard

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel