Progress fixes

PATCH In review Paul Eggleton 2 participants 7 messages Aug 11, 2016 – Aug 16, 2016

Share & tools

Paul Eggleton

author

2016-08-11 03:37 UTC #0 lore raw

cover Progress fixes

Fixes for minor issues in the progress bar display, plus a patch for some unclosed files that I noticed at the same time as working on these.

The following changes since commit 0a9b5d7d9655dbb09d458fc6e330e932f0f9dab6:

toaster: buildinfohelper Add handling local layers (i.e. non-git) layers (2016-08-11 00:08:18 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib paule/bb-progress-fixes2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-progress-fixes2

Paul Eggleton (4):
runqueue: fix two minor issues with the initialising tasks progress
knotty: fix task progress bar not starting at 0%
knotty: don't show number of running tasks in quiet mode
siggen: properly close files rather than opening them inline

 lib/bb/runqueue.py  |  5 +++--
 lib/bb/siggen.py    | 15 +++++++++------
 lib/bb/ui/knotty.py | 11 ++++++++---
 3 files changed, 20 insertions(+), 11 deletions(-)

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-11 03:37 UTC #1 lore raw

PATCH 1/4 runqueue: fix two minor issues with the initialising tasks progress

A couple of fixes for the "Initialising tasks" progress bar behaviour:
* Properly finish the progress bar when using bitbake -S
* Finish the progress bar before calling BB_HASHCHECK_FUNCTION (so that in OE when that shows its own "Checking sstate mirror object availability" progress bar it gets shown on the next line as it should).

Signed-off-by Paul Eggleton <[email protected]>

---
 lib/bb/runqueue.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

lib/bb/runqueue.py +3 -2

index aa939d0..3a593b6 100644
@@ -1148,6 +1148,7 @@ class RunQueue:
         if self.state is runQueueSceneInit:
             dump = self.cooker.configuration.dump_signatures
             if dump:
+                self.rqdata.init_progress_reporter.finish()
                 if 'printdiff' in dump:
                     invalidtasks = self.print_diffscenetasks()
                 self.dump_signatures(dump)

@@ -1967,6 +1968,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
             if len(self.sq_revdeps[tid]) == 0:
                 self.runq_buildable.add(tid)
 
+        self.rqdata.init_progress_reporter.finish()
+
         self.outrightfail = []
         if self.rq.hashvalidate:
             sq_hash = []

@@ -2018,8 +2021,6 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                     logger.debug(2, 'No package found, so skipping setscene task %s', tid)
                     self.outrightfail.append(tid)
 
-        self.rqdata.init_progress_reporter.finish()
-
         logger.info('Executing SetScene Tasks')
 
         self.rq.state = runQueueSceneRun

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-11 03:37 UTC #2 lore raw

PATCH 2/4 knotty: fix task progress bar not starting at 0%

In reply to Paul Eggleton

If we have the task number here we need to subtract 1 to get the number of tasks completed.

Signed-off-by Paul Eggleton <[email protected]>

---
 lib/bb/ui/knotty.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

lib/bb/ui/knotty.py +5 -2

index 1723a72..f89f7b3 100644
@@ -260,13 +260,16 @@ class TerminalFilter(object):
                 content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
             else:
                 content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)
-            maxtask = self.helper.tasknumber_total + 1
+            maxtask = self.helper.tasknumber_total
             if not self.main_progress or self.main_progress.maxval != maxtask:
                 widgets = [' ', progressbar.Percentage(), ' ', progressbar.Bar()]
                 self.main_progress = BBProgress("Running tasks", maxtask, widgets=widgets)
                 self.main_progress.start(False)
             self.main_progress.setmessage(content)
-            self.main_progress.update(self.helper.tasknumber_current)
+            progress = self.helper.tasknumber_current - 1
+            if progress < 0:
+                progress = 0
+            self.main_progress.update(progress)
             print('')
         lines = 1 + int(len(content) / (self.columns + 1))
         if not self.quiet:

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-11 03:37 UTC #3 lore raw

PATCH 3/4 knotty: don't show number of running tasks in quiet mode

In reply to Paul Eggleton

There's not a whole lot of point showing how many tasks are running when we're in quiet mode, it just looks a bit strange particularly when it's not running any tasks.

Signed-off-by Paul Eggleton <[email protected]>

---
 lib/bb/ui/knotty.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

lib/bb/ui/knotty.py +3 -1

index f89f7b3..b30135b 100644
@@ -256,7 +256,9 @@ class TerminalFilter(object):
             content = "Waiting for %s running tasks to finish:" % len(activetasks)
             print(content)
         else:
-            if not len(activetasks):
+            if self.quiet:
+                content = "Running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
+            elif not len(activetasks):
                 content = "No currently running tasks (%s of %s)" % (self.helper.tasknumber_current, self.helper.tasknumber_total)
             else:
                 content = "Currently %2s running tasks (%s of %s)" % (len(activetasks), self.helper.tasknumber_current, self.helper.tasknumber_total)

Signature · 1 line

2.5.5

Paul Eggleton

author

2016-08-11 03:37 UTC #4 lore raw

PATCH 4/4 siggen: properly close files rather than opening them inline

In reply to Paul Eggleton

If you don't do this, with Python 3 you get a warning on exit under some circumstances.

Signed-off-by Paul Eggleton <[email protected]>

---
 lib/bb/siggen.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

lib/bb/siggen.py +9 -6

index 5d8a253..9b2f658 100644
@@ -363,10 +363,12 @@ def clean_basepaths_list(a):
 def compare_sigfiles(a, b, recursecb = None):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
-    p2 = pickle.Unpickler(open(b, "rb"))
-    b_data = p2.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(ff)
+        a_data = p1.load()
+    with open(b, 'rb') as f:
+        p2 = pickle.Unpickler(f)
+        b_data = p2.load()
 
     def dict_diff(a, b, whitelist=set()):
         sa = set(a.keys())

@@ -563,8 +565,9 @@ def calc_taskhash(sigdata):
 def dump_sigfile(a):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(f)
+        a_data = p1.load()
 
     output.append("basewhitelist: %s" % (a_data['basewhitelist']))

Signature · 1 line

2.5.5

Peter Kjellerstedt

2016-08-16 21:16 UTC #5 lore raw

Re: [PATCH 4/4] siggen: properly close files rather than opening them inline

Quoted 21 lines

-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 11 augusti 2016 05:37
To: [email protected]
Subject: [bitbake-devel] [PATCH 4/4] siggen: properly close files rather than opening them inline
If you don't do this, with Python 3 you get a warning on exit under some circumstances.
Signed-off-by Paul Eggleton <[email protected]>
---
 lib/bb/siggen.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
lib/bb/siggen.py +0 -0
index 5d8a253..9b2f658 100644

lib/bb/siggen.py def clean_basepaths_list(a): · line 367

Quoted 9 lines

lib/bb/siggen.py +2 -4

@@ -363,10 +363,12 @@ def clean_basepaths_list(a):
 def compare_sigfiles(a, b, recursecb = None):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
-    p2 = pickle.Unpickler(open(b, "rb"))
-    b_data = p2.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(ff)

^^

Shouldn't that be:

p1 = pickle.Unpickler(f)

Quoted 20 lines

lib/bb/siggen.py +7 -3

+        a_data = p1.load()
+    with open(b, 'rb') as f:
+        p2 = pickle.Unpickler(f)
+        b_data = p2.load()
 
     def dict_diff(a, b, whitelist=set()):
         sa = set(a.keys())

@@ -563,8 +565,9 @@ def calc_taskhash(sigdata):
 def dump_sigfile(a):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(f)
+        a_data = p1.load()
 
     output.append("basewhitelist: %s" % (a_data['basewhitelist']))
 
--

Quoted 1 line

2.5.5

//Peter

Paul Eggleton

author

2016-08-16 22:30 UTC #6 lore raw

Re: [PATCH 4/4] siggen: properly close files rather than opening them inline

Quoted 34 lines · On Tue, 16 Aug 2016 21:16:17 Peter Kjellerstedt wrote:

Quoted 23 lines
-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Paul Eggleton
Sent: den 11 augusti 2016 05:37
To: [email protected]
Subject: [bitbake-devel] [PATCH 4/4] siggen: properly close files rather than opening them inline
If you don't do this, with Python 3 you get a warning on exit under some circumstances.
Signed-off-by Paul Eggleton <[email protected]>
---
 lib/bb/siggen.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
lib/bb/siggen.py +0 -0
index 5d8a253..9b2f658 100644
lib/bb/siggen.py def clean_basepaths_list(a): · line 367
Quoted 9 lines
lib/bb/siggen.py +2 -4
@@ -363,10 +363,12 @@ def clean_basepaths_list(a):
 def compare_sigfiles(a, b, recursecb = None):
     output = []
 
-    p1 = pickle.Unpickler(open(a, "rb"))
-    a_data = p1.load()
-    p2 = pickle.Unpickler(open(b, "rb"))
-    b_data = p2.load()
+    with open(a, 'rb') as f:
+        p1 = pickle.Unpickler(ff)
                                 ^^

Quoted 4 lines

Shouldn't that be:
p1 = pickle.Unpickler(f)

Oops - yes it should be. I guess it just proves that even "simple, obvious" changes like this one do actually need to be tested. I'll send a v2.

Thanks,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre