Skip to contentSkip to conversation

sstate: add a LockedSet class as python set() is not thread safe

RFC PATCH In review Jose Quaresma 2 participants 8 messages Apr 18, 2022 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/4 sstate: add a LockedSet class as python set() is not thread safe

With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.

This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +22 -2
index 1c0cae4893..a3ba748a1e 100644
@@ -918,8 +918,28 @@ sstate_unpack_package () {
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
 def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs):
-    found = set()
-    missed = set()
+    # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
+    import threading
+    class LockedSet(set):
+        """A set where add(), remove(), and 'in' operator are thread-safe"""
+        def __init__(self, *args, **kwargs):
+            self._lock = threading.Lock()
+            super(LockedSet, self).__init__(*args, **kwargs)
+
+        def add(self, elem):
+            with self._lock:
+                super(LockedSet, self).add(elem)
+
+        def remove(self, elem):
+            with self._lock:
+                super(LockedSet, self).remove(elem)
+
+        def __contains__(self, elem):
+            with self._lock:
+                super(LockedSet, self).__contains__(elem)
+
+    found = LockedSet()
+    missed = LockedSet()
 
     def gethash(task):
         return sq_data['unihash'][task]
Signature · 1 line
2.35.3
#1 lore raw

RFC PATCH v2 2/4 sstate: store the tasklist in a LockedSet

With this aproach the progress bar will be independent of the ThreadedPool.

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
meta/classes/sstate.bbclass +11 -9
index a3ba748a1e..b9e9ef8e26 100644
@@ -1026,29 +1026,31 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
                 bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc()))
 
             if progress:
-                bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d)
+                tasks.remove(arg)
+                bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d)
 
-        tasklist = []
+        tasks = LockedSet()
         for tid in missed:
             sstatefile = d.expand(getsstatefile(tid, siginfo, d))
-            tasklist.append((tid, sstatefile))
+            tasks.add((tid, sstatefile))
 
-        if tasklist:
-            nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist))
+        total_tasks = len(tasks)
+        if total_tasks:
+            nproc = min(int(d.getVar("BB_NUMBER_THREADS")), total_tasks)
 
-            progress = len(tasklist) >= 100
+            progress = total_tasks >= 100
             if progress:
                 msg = "Checking sstate mirror object availability"
-                bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d)
+                bb.event.fire(bb.event.ProcessStarted(msg, total_tasks), d)
 
             # Have to setup the fetcher environment here rather than in each thread as it would race
             fetcherenv = bb.fetch2.get_fetcher_environment(d)
             with bb.utils.environment(**fetcherenv):
                 bb.event.enable_threadlock()
-                pool = oe.utils.ThreadedPool(nproc, len(tasklist),
+                pool = oe.utils.ThreadedPool(nproc, total_tasks,
                         worker_init=checkstatus_init, worker_end=checkstatus_end,
                         name="sstate_checkhashes-")
-                for t in tasklist:
+                for t in tasks:
                     pool.add_task(checkstatus, t)
                 pool.start()
                 pool.wait_completion()
Signature · 1 line
2.35.3
#2 lore raw

RFC PATCH v2 3/4 sstate: use the python3 ThreadPoolExecutor instead of the OE ThreadedPool

In reply to Jose Quaresma

for the FetchConnectionCache use a queue where each thread can get an unsed connection_cache that is properly initialized before we fireup the ThreadPoolExecutor.

for the progress bar we need an adictional task counter that is protected with thread lock as it runs inside the ThreadPoolExecutor.

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)
meta/classes/sstate.bbclass +19 -13
index b9e9ef8e26..7a18d9d554 100644
@@ -997,15 +997,19 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             localdata.delVar('BB_NO_NETWORK')
 
         from bb.fetch2 import FetchConnectionCache
-        def checkstatus_init(thread_worker):
-            thread_worker.connection_cache = FetchConnectionCache()
+        def checkstatus_init():
+            while not connection_cache_pool.full():
+                connection_cache_pool.put(FetchConnectionCache())
 
-        def checkstatus_end(thread_worker):
-            thread_worker.connection_cache.close_connections()
+        def checkstatus_end():
+            while not connection_cache_pool.empty():
+                connection_cache = connection_cache_pool.get()
+                connection_cache.close_connections()
 
-        def checkstatus(thread_worker, arg):
+        def checkstatus(arg):
             (tid, sstatefile) = arg
 
+            connection_cache = connection_cache_pool.get()
             localdata2 = bb.data.createCopy(localdata)
             srcuri = "file://" + sstatefile
             localdata2.setVar('SRC_URI', srcuri)
@@ -1015,7 +1019,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
 
             try:
                 fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2,
-                            connection_cache=thread_worker.connection_cache)
+                            connection_cache=connection_cache)
                 fetcher.checkstatus()
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 found.add(tid)
@@ -1025,6 +1029,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             except Exception as e:
                 bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc()))
 
+            connection_cache_pool.put(connection_cache)
+
             if progress:
                 tasks.remove(arg)
                 bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d)
@@ -1047,13 +1053,13 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             fetcherenv = bb.fetch2.get_fetcher_environment(d)
             with bb.utils.environment(**fetcherenv):
                 bb.event.enable_threadlock()
-                pool = oe.utils.ThreadedPool(nproc, total_tasks,
-                        worker_init=checkstatus_init, worker_end=checkstatus_end,
-                        name="sstate_checkhashes-")
-                for t in tasks:
-                    pool.add_task(checkstatus, t)
-                pool.start()
-                pool.wait_completion()
+                import concurrent.futures
+                from queue import Queue
+                connection_cache_pool = Queue(nproc)
+                checkstatus_init()
+                with concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor:
+                    executor.map(checkstatus, tasks)
+                checkstatus_end()
                 bb.event.disable_threadlock()
 
             if progress:
Signature · 1 line
2.35.3
#3 lore raw

RFC PATCH v2 4/4 oe/utils: remove the ThreadedPool

In reply to Jose Quaresma

The ThreadedPool in OE-core is mainly because python2 doesn't have threaded pools but python2 is dead for some time now and python3 have a ThreadPoolExecutor.

The only local in OE-core where this ThreadedPool is in use is on the sstate.bbclass that is ported to the python3 ThreadPoolExecutor.

---
 meta/lib/oe/utils.py | 64 --------------------------------------------
 1 file changed, 64 deletions(-)
meta/lib/oe/utils.py +0 -64
index 46fc76c261..1ee947d584 100644
@@ -473,70 +473,6 @@ def get_multilib_datastore(variant, d):
         localdata.setVar("MLPREFIX", "")
     return localdata
 
-#
-# Python 2.7 doesn't have threaded pools (just multiprocessing)
-# so implement a version here
-#
-
-from queue import Queue
-from threading import Thread
-
-class ThreadedWorker(Thread):
-    """Thread executing tasks from a given tasks queue"""
-    def __init__(self, tasks, worker_init, worker_end, name=None):
-        Thread.__init__(self, name=name)
-        self.tasks = tasks
-        self.daemon = True
-
-        self.worker_init = worker_init
-        self.worker_end = worker_end
-
-    def run(self):
-        from queue import Empty
-
-        if self.worker_init is not None:
-            self.worker_init(self)
-
-        while True:
-            try:
-                func, args, kargs = self.tasks.get(block=False)
-            except Empty:
-                if self.worker_end is not None:
-                    self.worker_end(self)
-                break
-
-            try:
-                func(self, *args, **kargs)
-            except Exception as e:
-                # Eat all exceptions
-                bb.mainlogger.debug("Worker task raised %s" % e, exc_info=e)
-            finally:
-                self.tasks.task_done()
-
-class ThreadedPool:
-    """Pool of threads consuming tasks from a queue"""
-    def __init__(self, num_workers, num_tasks, worker_init=None, worker_end=None, name="ThreadedPool-"):
-        self.tasks = Queue(num_tasks)
-        self.workers = []
-
-        for i in range(num_workers):
-            worker = ThreadedWorker(self.tasks, worker_init, worker_end, name=name + str(i))
-            self.workers.append(worker)
-
-    def start(self):
-        for worker in self.workers:
-            worker.start()
-
-    def add_task(self, func, *args, **kargs):
-        """Add a task to the queue"""
-        self.tasks.put((func, args, kargs))
-
-    def wait_completion(self):
-        """Wait for completion of all the tasks in the queue"""
-        self.tasks.join()
-        for worker in self.workers:
-            worker.join()
-
 class ImageQAFailed(Exception):
     def __init__(self, description, name=None, logfile=None):
         self.description = description
Signature · 1 line
2.35.3

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 62 lines · Richard Purdie via lists.openembedded.org
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 38 lines · Jose Quaresma

With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.

This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +15 -2
index 1c0cae4893..a3ba748a1e 100644
@@ -918,8 +918,28 @@ sstate_unpack_package () {
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
 def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs):
-    found = set()
-    missed = set()
+    # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
+    import threading
+    class LockedSet(set):
+        """A set where add(), remove(), and 'in' operator are thread-safe"""
+        def __init__(self, *args, **kwargs):
+            self._lock = threading.Lock()
+            super(LockedSet, self).__init__(*args, **kwargs)
+
+        def add(self, elem):
+            with self._lock:
+                super(LockedSet, self).add(elem)
+
+        def remove(self, elem):
+            with self._lock:
+                super(LockedSet, self).remove(elem)

meta/classes/sstate.bbclass sstate_unpack_package () { · line 945

Quoted 10 lines
meta/classes/sstate.bbclass +7 -0
+
+        def __contains__(self, elem):
+            with self._lock:
+                super(LockedSet, self).__contains__(elem)
+
+    found = LockedSet()
+    missed = LockedSet()
 
     def gethash(task):
         return sq_data['unihash'][task]

The series blew up on the autobuilder:

Quoted 6 lines

testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535

Quoted 2 lines

That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539

meta/classes/sstate.bbclass +1 -1
index 7a18d9d554..99d125b8c5 100644
@@ -1058,7 +1058,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
                 connection_cache_pool = Queue(nproc)
                 checkstatus_init()
                 with concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor:
-                    executor.map(checkstatus, tasks)
+                    executor.map(checkstatus, tasks.copy())
                 checkstatus_end()
                 bb.event.disable_threadlock()
 

fixes things.

Cheers,

Richard

#7 lore raw

Richard Purdie <[email protected]> escreveu no dia domingo, 17/04/2022 à(s) 21:09:

Quoted 92 lines
Quoted 1 line · Richard Purdie via lists.openembedded.org

On Sun, 2022-04-17 at 14:39 +0100, Richard Purdie via

lists.openembedded.org

Quoted 7 lines
Quoted 6 lines · wrote:
Quoted 5 lines · Jose Quaresma

With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.

This piece of code is taen from the stackoverflow

https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Fixes [YOCTO #14775] --

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

Quoted 7 lines
Quoted 7 lines

Review

Quoted 7 lines
---
 meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/meta/classes/sstate.bbclass

Quoted 7 lines
Quoted 7 lines

meta/classes/sstate.bbclass sstate_unpack_package () { · line 920

Quoted 7 lines

index 1c0cae4893..a3ba748a1e 100644

meta/classes/sstate.bbclass +0 -0
@@ -918,8 +918,28 @@ sstate_unpack_package () {
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
 def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,

summary=True, **kwargs):

Quoted 3 lines
Quoted 3 lines

meta/classes/sstate.bbclass sstate_unpack_package () { · line 921

Quoted 3 lines
meta/classes/sstate.bbclass +1 -2
-    found = set()
-    missed = set()
+    #

https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Quoted 3 lines
Quoted 3 lines

meta/classes/sstate.bbclass sstate_unpack_package () { · line 924

Quoted 3 lines
meta/classes/sstate.bbclass +3 -0
+    import threading
+    class LockedSet(set):
+        """A set where add(), remove(), and 'in' operator are

thread-safe"""

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
meta/classes/sstate.bbclass +18 -0
+        def __init__(self, *args, **kwargs):
+            self._lock = threading.Lock()
+            super(LockedSet, self).__init__(*args, **kwargs)
+
+        def add(self, elem):
+            with self._lock:
+                super(LockedSet, self).add(elem)
+
+        def remove(self, elem):
+            with self._lock:
+                super(LockedSet, self).remove(elem)
+
+        def __contains__(self, elem):
+            with self._lock:
+                super(LockedSet, self).__contains__(elem)
+
+    found = LockedSet()
+    missed = LockedSet()
 
Quoted 8 lines
Quoted 8 lines

meta/classes/sstate.bbclass sstate_unpack_package () { · line 945

Quoted 2 lines
meta/classes/sstate.bbclass +0 -0
     def gethash(task):
         return sq_data['unihash'][task]

The series blew up on the autobuilder:

testsdkest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio

Quoted 2 lines
Quoted 2 lines

oe-selftest:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio

Quoted 5 lines
Quoted 2 lines

and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535

That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539

meta/classes/sstate.bbclass +0 -0
index 7a18d9d554..99d125b8c5 100644
@@ -1058,7 +1058,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False,
 currentcount=0, summary=True,
                 connection_cache_pool = Queue(nproc)
                 checkstatus_init()
                 with

concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor:

meta/classes/sstate.bbclass +1 -1
-                    executor.map(checkstatus, tasks)
+                    executor.map(checkstatus, tasks.copy())
                 checkstatus_end()
                 bb.event.disable_threadlock()
Quoted 7 lines

fixes things.

Cheers,

Richard

Thanks for testing and fixing, this can be because the tasks collection has been changed inside the thread pool.
I'll wait a few more days to see if there are any more comments and I hope to be able to finish during this week.

Signature · 3 lines
Best regards,

José Quaresma
#8 lore raw
In reply to a message not in the archive
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 38 lines · Jose Quaresma

With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.

This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +15 -2
index 1c0cae4893..a3ba748a1e 100644
@@ -918,8 +918,28 @@ sstate_unpack_package () {
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
 def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs):
-    found = set()
-    missed = set()
+    # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
+    import threading
+    class LockedSet(set):
+        """A set where add(), remove(), and 'in' operator are thread-safe"""
+        def __init__(self, *args, **kwargs):
+            self._lock = threading.Lock()
+            super(LockedSet, self).__init__(*args, **kwargs)
+
+        def add(self, elem):
+            with self._lock:
+                super(LockedSet, self).add(elem)
+
+        def remove(self, elem):
+            with self._lock:
+                super(LockedSet, self).remove(elem)

meta/classes/sstate.bbclass sstate_unpack_package () { · line 945

Quoted 10 lines
meta/classes/sstate.bbclass +7 -0
+
+        def __contains__(self, elem):
+            with self._lock:
+                super(LockedSet, self).__contains__(elem)
+
+    found = LockedSet()
+    missed = LockedSet()
 
     def gethash(task):
         return sq_data['unihash'][task]

The series blew up on the autobuilder:

Quoted 6 lines

testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535

That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539

Cheers,

Richard

#9 lore raw
In reply to Jose Quaresma
Quoted 38 lines · Jose Quaresma

With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.

This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe

Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775

---
 meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +15 -2
index 1c0cae4893..a3ba748a1e 100644
@@ -918,8 +918,28 @@ sstate_unpack_package () {
 BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 
 def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs):
-    found = set()
-    missed = set()
+    # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
+    import threading
+    class LockedSet(set):
+        """A set where add(), remove(), and 'in' operator are thread-safe"""
+        def __init__(self, *args, **kwargs):
+            self._lock = threading.Lock()
+            super(LockedSet, self).__init__(*args, **kwargs)
+
+        def add(self, elem):
+            with self._lock:
+                super(LockedSet, self).add(elem)
+
+        def remove(self, elem):
+            with self._lock:
+                super(LockedSet, self).remove(elem)

meta/classes/sstate.bbclass sstate_unpack_package () { · line 945

Quoted 10 lines
meta/classes/sstate.bbclass +7 -0
+
+        def __contains__(self, elem):
+            with self._lock:
+                super(LockedSet, self).__contains__(elem)
+
+    found = LockedSet()
+    missed = LockedSet()
 
     def gethash(task):
         return sq_data['unihash'][task]

The series blew up on the autobuilder:

testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535

Cheers,

Richard