Skip to contentSkip to conversation

utils/show-progress: add tool to show build progress at runtime

RFC New Vadim Kochan 2 participants 3 messages Jul 18, 2019 – Aug 6, 2019

lore
Share & tools
lore
#0 lore raw

RFC v2 1/1 utils/show-progress: add tool to show build progress at runtime

This might be useful to watch the amount of built and selected packages and some progress about it. So added python script which prints progress and build stats. The sample of output is:

Press Ctrl-C to exit ...

Building: output/qemu_x86_64
##################################------------------------------ [ 42.42% 14/33]

---

RFC v2:
1) filter rootfs package type
2) rename build-progress -> show-progress
3) check empty version

 utils/show-progress | 91 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100755 utils/show-progress
utils/show-progress new file +91 -0
new file mode 100755
index 0000000000..ee1e1da289
@@ -0,0 +1,91 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2019 Vadim Kochan <[email protected]>
+
+import os
+import sys
+import json
+import time
+import logging
+import subprocess
+
+do_exit = False
+
+def usage():
+    print("""Usage: show-progress [-h] [PATH]
+show-progress shows progress about selected and built packages.
+If PATH is no specified then the current working one will be used as
+build output directory.
+
+Example usage:
+ $ show-progress output/qemu_x86_64
+
+""")
+    sys.exit(0)
+
+def get_pkgs_list(build_dir):
+    cmd = ["make", "-C", build_dir, "-s", "--no-print-directory", "show-info"]
+    results = []
+
+    with open(os.devnull, 'wb') as devnull:
+        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull,
+                             universal_newlines=True)
+        pkg_list = json.loads(p.communicate()[0])
+
+        for p in pkg_list:
+            if pkg_list[p]["type"] == "rootfs":
+                continue
+
+            ver = "" if pkg_list[p]["virtual"] else pkg_list[p]["version"]
+            ver = "-" + ver if ver != "" else ver
+
+            results.append(p + ver)
+
+    return results
+
+def is_pkg_built(build_dir, p):
+    for s in ["host", "target"]:
+        if os.path.exists(build_dir + "/build/" + p + "/.stamp_" + s + "_installed"):
+            return True
+
+    return False
+
+def progress(ready, total):
+    perc = ready / total
+    fill = round(perc * 80)
+    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')
+    sys.stdout.flush()
+
+def main():
+    build_dir = "."
+
+    if "-h" in sys.argv or "--help" in sys.argv:
+        usage()
+
+    if len(sys.argv) > 1:
+        build_dir = sys.argv[1]
+
+    pkgs = get_pkgs_list(build_dir)
+    total = len(pkgs)
+
+    print("Press Ctrl-C to exit ...\n")
+    print("Building: " + build_dir)
+
+    while not do_exit:
+        ready = 0
+
+        for p in pkgs:
+            if is_pkg_built(build_dir, p):
+                ready = ready + 1
+
+        progress(ready, total)
+        time.sleep(1)
+
+        if ready == total:
+            print("Done")
+            break
+
+try:
+    main()
+except KeyboardInterrupt:
+    do_exit = True
Signature · 1 line
2.22.0
#1 lore raw

RFC v2 1/1 utils/show-progress: add tool to show build progress at runtime

Vadim, All,

On 2019-07-18 10:32 +0300, Vadim Kochan spake thusly:

Quoted 8 lines

This might be useful to watch the amount of built and selected packages and some progress about it. So added python script which prints progress and build stats. The sample of output is:

Press Ctrl-C to exit ...

Building: output/qemu_x86_64
##################################------------------------------ [ 42.42% 14/33]

With this defconfig:

    BR2_arm=y
    BR2_cortex_a7=y
    BR2_TOOLCHAIN_EXTERNAL=y
    # BR2_TARGET_ROOTFS_TAR is not set

and building out-of-tree, the script never reaches 100%:

    $ ../buildroot/utils/show-progress
    Press Ctrl-C to exit ...
    Building: .
     #########################################################################------- [ 91.67% 11/12]

Even though the build did finish...

Quoted 1 line

[--SNIP--]

utils/show-progress line 1

Quoted 7 lines
utils/show-progress new file +1 -0
new file mode 100755
index 0000000000..ee1e1da289
@@ -0,0 +1,91 @@
+#!/usr/bin/env python

This is actually a python3 script, and is not a valid pyhon2 one.

utils/show-progress line 3

Quoted 1 line
utils/show-progress new file +1 -0
+# Copyright (C) 2019 Vadim Kochan <[email protected]>

There is no reason nowadays to add copyright notices. The copyright is defacto by the Bern convention. Furthermore, in an opensource project, this notice is wrong as ssoon as other contributors do substantial modification to the file.

And the git log is a much better way to extract authroship of a file.

[--SNIP--]

utils/show-progress line 56

Quoted 4 lines
utils/show-progress new file +4 -0
+def progress(ready, total):
+    perc = ready / total
+    fill = round(perc * 80)
+    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')

I've locally changed it to:

    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {: >2d}/{}]'.format(perc, ready, total), end='')

utils/show-progress line 72

Quoted 2 lines
utils/show-progress new file +2 -0
+    print("Press Ctrl-C to exit ...\n")
+    print("Building: " + build_dir)

This print '.' as the build directory when doing an out-of-tree build (see above). I've locally changed it to:

print("Building: " + os.path.realpath(build_dir))

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

RFC v2 1/1 utils/show-progress: add tool to show build progress at runtime

Yann,

Quoted 63 lines · Yann E. MORIN

Vadim, All,

On 2019-07-18 10:32 +0300, Vadim Kochan spake thusly:

Quoted 8 lines

This might be useful to watch the amount of built and selected packages and some progress about it. So added python script which prints progress and build stats. The sample of output is:

Press Ctrl-C to exit ...

Building: output/qemu_x86_64
##################################------------------------------ [ 42.42% 14/33]

With this defconfig:

    BR2_arm=y
    BR2_cortex_a7=y
    BR2_TOOLCHAIN_EXTERNAL=y
    # BR2_TARGET_ROOTFS_TAR is not set

and building out-of-tree, the script never reaches 100%:

    $ ../buildroot/utils/show-progress
    Press Ctrl-C to exit ...
    Building: .
     #########################################################################------- [ 91.67% 11/12]

Even though the build did finish...

Quoted 1 line

[--SNIP--]

utils/show-progress line 1

Quoted 7 lines
utils/show-progress new file +1 -0
new file mode 100755
index 0000000000..ee1e1da289
@@ -0,0 +1,91 @@
+#!/usr/bin/env python

This is actually a python3 script, and is not a valid pyhon2 one.

utils/show-progress line 3

Quoted 1 line
utils/show-progress new file +1 -0
+# Copyright (C) 2019 Vadim Kochan <[email protected]>

There is no reason nowadays to add copyright notices. The copyright is defacto by the Bern convention. Furthermore, in an opensource project, this notice is wrong as ssoon as other contributors do substantial modification to the file.

And the git log is a much better way to extract authroship of a file.

[--SNIP--]

utils/show-progress line 56

Quoted 4 lines
utils/show-progress new file +4 -0
+def progress(ready, total):
+    perc = ready / total
+    fill = round(perc * 80)
+    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')

I've locally changed it to:

    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {: >2d}/{}]'.format(perc, ready, total), end='')

utils/show-progress line 72

Quoted 2 lines
utils/show-progress new file +2 -0
+    print("Press Ctrl-C to exit ...\n")
+    print("Building: " + build_dir)
Quoted 8 lines

This print '.' as the build directory when doing an out-of-tree build (see above). I've locally changed it to:

print("Building: " + os.path.realpath(build_dir))

Regards,
Yann E. MORIN.

Yes, sorry for wasting your time. It really does not work if there is package which has no installation
for target but only installed to the staging. I will fix this case after "install_" properties will be merged to the "show-info" make helper.

I forgot to mark this patch for ignoring.

Regards,
Vadim Kochan