sstate: add a LockedSet class as python set() is not thread safe
RFC PATCH v2 1/4 sstate: add a LockedSet class as python set() is not thread safe
With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.
This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe
Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
--- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
meta/classes/sstate.bbclass +22 -2
@@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): - found = set() - missed = set() + # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe + import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem) + + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet() def gethash(task): return sq_data['unihash'][task]
Signature · 1 line
2.35.3
RFC PATCH v2 2/4 sstate: store the tasklist in a LockedSet
With this aproach the progress bar will be independent of the ThreadedPool.
Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
--- meta/classes/sstate.bbclass | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
meta/classes/sstate.bbclass +11 -9
@@ -1026,29 +1026,31 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc())) if progress: - bb.event.fire(bb.event.ProcessProgress(msg, len(tasklist) - thread_worker.tasks.qsize()), d) + tasks.remove(arg) + bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d) - tasklist = [] + tasks = LockedSet() for tid in missed: sstatefile = d.expand(getsstatefile(tid, siginfo, d)) - tasklist.append((tid, sstatefile)) + tasks.add((tid, sstatefile)) - if tasklist: - nproc = min(int(d.getVar("BB_NUMBER_THREADS")), len(tasklist)) + total_tasks = len(tasks) + if total_tasks: + nproc = min(int(d.getVar("BB_NUMBER_THREADS")), total_tasks) - progress = len(tasklist) >= 100 + progress = total_tasks >= 100 if progress: msg = "Checking sstate mirror object availability" - bb.event.fire(bb.event.ProcessStarted(msg, len(tasklist)), d) + bb.event.fire(bb.event.ProcessStarted(msg, total_tasks), d) # Have to setup the fetcher environment here rather than in each thread as it would race fetcherenv = bb.fetch2.get_fetcher_environment(d) with bb.utils.environment(**fetcherenv): bb.event.enable_threadlock() - pool = oe.utils.ThreadedPool(nproc, len(tasklist), + pool = oe.utils.ThreadedPool(nproc, total_tasks, worker_init=checkstatus_init, worker_end=checkstatus_end, name="sstate_checkhashes-") - for t in tasklist: + for t in tasks: pool.add_task(checkstatus, t) pool.start() pool.wait_completion()
Signature · 1 line
2.35.3
RFC PATCH v2 3/4 sstate: use the python3 ThreadPoolExecutor instead of the OE ThreadedPool
In reply to Jose Quaresmafor the FetchConnectionCache use a queue where each thread can get an unsed connection_cache that is properly initialized before we fireup the ThreadPoolExecutor.
for the progress bar we need an adictional task counter that is protected with thread lock as it runs inside the ThreadPoolExecutor.
Fixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
--- meta/classes/sstate.bbclass | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-)
meta/classes/sstate.bbclass +19 -13
@@ -997,15 +997,19 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, localdata.delVar('BB_NO_NETWORK') from bb.fetch2 import FetchConnectionCache - def checkstatus_init(thread_worker): - thread_worker.connection_cache = FetchConnectionCache() + def checkstatus_init(): + while not connection_cache_pool.full(): + connection_cache_pool.put(FetchConnectionCache()) - def checkstatus_end(thread_worker): - thread_worker.connection_cache.close_connections() + def checkstatus_end(): + while not connection_cache_pool.empty(): + connection_cache = connection_cache_pool.get() + connection_cache.close_connections() - def checkstatus(thread_worker, arg): + def checkstatus(arg): (tid, sstatefile) = arg + connection_cache = connection_cache_pool.get() localdata2 = bb.data.createCopy(localdata) srcuri = "file://" + sstatefile localdata2.setVar('SRC_URI', srcuri)
@@ -1015,7 +1019,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, try: fetcher = bb.fetch2.Fetch(srcuri.split(), localdata2, - connection_cache=thread_worker.connection_cache) + connection_cache=connection_cache) fetcher.checkstatus() bb.debug(2, "SState: Successful fetch test for %s" % srcuri) found.add(tid)
@@ -1025,6 +1029,8 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, except Exception as e: bb.error("SState: cannot test %s: %s\n%s" % (srcuri, repr(e), traceback.format_exc())) + connection_cache_pool.put(connection_cache) + if progress: tasks.remove(arg) bb.event.fire(bb.event.ProcessProgress(msg, total_tasks - len(tasks)), d)
@@ -1047,13 +1053,13 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, fetcherenv = bb.fetch2.get_fetcher_environment(d) with bb.utils.environment(**fetcherenv): bb.event.enable_threadlock() - pool = oe.utils.ThreadedPool(nproc, total_tasks, - worker_init=checkstatus_init, worker_end=checkstatus_end, - name="sstate_checkhashes-") - for t in tasks: - pool.add_task(checkstatus, t) - pool.start() - pool.wait_completion() + import concurrent.futures + from queue import Queue + connection_cache_pool = Queue(nproc) + checkstatus_init() + with concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor: + executor.map(checkstatus, tasks) + checkstatus_end() bb.event.disable_threadlock() if progress:
Signature · 1 line
2.35.3
RFC PATCH v2 4/4 oe/utils: remove the ThreadedPool
In reply to Jose QuaresmaThe ThreadedPool in OE-core is mainly because python2 doesn't have threaded pools but python2 is dead for some time now and python3 have a ThreadPoolExecutor.
The only local in OE-core where this ThreadedPool is in use is on the sstate.bbclass that is ported to the python3 ThreadPoolExecutor.
--- meta/lib/oe/utils.py | 64 -------------------------------------------- 1 file changed, 64 deletions(-)
meta/lib/oe/utils.py +0 -64
@@ -473,70 +473,6 @@ def get_multilib_datastore(variant, d): localdata.setVar("MLPREFIX", "") return localdata -# -# Python 2.7 doesn't have threaded pools (just multiprocessing) -# so implement a version here -# - -from queue import Queue -from threading import Thread - -class ThreadedWorker(Thread): - """Thread executing tasks from a given tasks queue""" - def __init__(self, tasks, worker_init, worker_end, name=None): - Thread.__init__(self, name=name) - self.tasks = tasks - self.daemon = True - - self.worker_init = worker_init - self.worker_end = worker_end - - def run(self): - from queue import Empty - - if self.worker_init is not None: - self.worker_init(self) - - while True: - try: - func, args, kargs = self.tasks.get(block=False) - except Empty: - if self.worker_end is not None: - self.worker_end(self) - break - - try: - func(self, *args, **kargs) - except Exception as e: - # Eat all exceptions - bb.mainlogger.debug("Worker task raised %s" % e, exc_info=e) - finally: - self.tasks.task_done() - -class ThreadedPool: - """Pool of threads consuming tasks from a queue""" - def __init__(self, num_workers, num_tasks, worker_init=None, worker_end=None, name="ThreadedPool-"): - self.tasks = Queue(num_tasks) - self.workers = [] - - for i in range(num_workers): - worker = ThreadedWorker(self.tasks, worker_init, worker_end, name=name + str(i)) - self.workers.append(worker) - - def start(self): - for worker in self.workers: - worker.start() - - def add_task(self, func, *args, **kargs): - """Add a task to the queue""" - self.tasks.put((func, args, kargs)) - - def wait_completion(self): - """Wait for completion of all the tasks in the queue""" - self.tasks.join() - for worker in self.workers: - worker.join() - class ImageQAFailed(Exception): def __init__(self, description, name=None, logfile=None): self.description = description
Signature · 1 line
2.35.3
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 62 lines · Richard Purdie via lists.openembedded.org
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 38 lines · Jose Quaresma
With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.
This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safeFixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
Signed-off-by Jose Quaresma <[email protected]>--- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)meta/classes/sstate.bbclass +15
-2@@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): - found = set() - missed = set() + # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe + import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem)meta/classes/sstate.bbclass sstate_unpack_package () { · line 945
Quoted 10 lines
meta/classes/sstate.bbclass +7
-0+ + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet() def gethash(task): return sq_data['unihash'][task]The series blew up on the autobuilder:
Quoted 6 lines
testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535
Quoted 2 lines
That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539
meta/classes/sstate.bbclass +1 -1
@@ -1058,7 +1058,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, connection_cache_pool = Queue(nproc) checkstatus_init() with concurrent.futures.ThreadPoolExecutor(max_workers=nproc) as executor: - executor.map(checkstatus, tasks) + executor.map(checkstatus, tasks.copy()) checkstatus_end() bb.event.disable_threadlock()
fixes things.
Cheers,
Richard
Richard Purdie <[email protected]> escreveu no dia domingo, 17/04/2022 à(s) 21:09:
Quoted 92 lines
Quoted 1 line · Richard Purdie via lists.openembedded.org
On Sun, 2022-04-17 at 14:39 +0100, Richard Purdie via
lists.openembedded.org
Quoted 7 lines
Quoted 6 lines · wrote:
Quoted 5 lines · Jose Quaresma
With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.
This piece of code is taen from the stackoverflow
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Fixes [YOCTO #14775] --
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
Quoted 7 lines
Quoted 7 lines
Review
Quoted 7 lines
Signed-off-by Jose Quaresma <[email protected]>--- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)(quoted diff) +0
-0b/meta/classes/sstate.bbclass
Quoted 7 lines
Quoted 7 lines
meta/classes/sstate.bbclass sstate_unpack_package () { · line 920
Quoted 7 lines
index
1c0cae4893..a3ba748a1e100644meta/classes/sstate.bbclass +0
-0@@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,summary=True, **kwargs):
Quoted 3 lines
Quoted 3 lines
meta/classes/sstate.bbclass sstate_unpack_package () { · line 921
Quoted 3 lines
meta/classes/sstate.bbclass +1
-2- found = set() - missed = set() + #Quoted 3 lines
Quoted 3 lines
meta/classes/sstate.bbclass sstate_unpack_package () { · line 924
Quoted 3 lines
meta/classes/sstate.bbclass +3
-0+ import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator arethread-safe"""
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
meta/classes/sstate.bbclass +18
-0+ def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem) + + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet()Quoted 8 lines
Quoted 8 lines
meta/classes/sstate.bbclass sstate_unpack_package () { · line 945
Quoted 2 lines
meta/classes/sstate.bbclass +0
-0def gethash(task): return sq_data['unihash'][task]The series blew up on the autobuilder:
testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
Quoted 2 lines
Quoted 2 lines
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
Quoted 5 lines
Quoted 2 lines
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539meta/classes/sstate.bbclass +0
-0@@ -1058,7 +1058,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, connection_cache_pool = Queue(nproc) checkstatus_init() with
concurrent.futures.ThreadPoolExecutor(max_workers=nproc)as executor:meta/classes/sstate.bbclass +1
-1- executor.map(checkstatus, tasks) + executor.map(checkstatus, tasks.copy()) checkstatus_end() bb.event.disable_threadlock()
Quoted 7 lines
fixes things.
Cheers,
Richard
Thanks for testing and fixing, this can be because the tasks collection has been changed inside the thread pool.
I'll wait a few more days to see if there are any more comments and I hope to be able to finish during this week.
Signature · 3 lines
Best regards, José Quaresma
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 38 lines · Jose Quaresma
With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.
This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safeFixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
Signed-off-by Jose Quaresma <[email protected]>--- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)meta/classes/sstate.bbclass +15
-2@@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): - found = set() - missed = set() + # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe + import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem)meta/classes/sstate.bbclass sstate_unpack_package () { · line 945
Quoted 10 lines
meta/classes/sstate.bbclass +7
-0+ + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet() def gethash(task): return sq_data['unihash'][task]The series blew up on the autobuilder:
Quoted 6 lines
testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535
That last one should be:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3539
Cheers,
Richard
Quoted 38 lines · Jose Quaresma
With this LockedSet python class we can call the 'add' and 'remove' safely inside the ThreadedPool.
This piece of code is taen from the stackoverflow
https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safeFixes [YOCTO #14775] -- https://bugzilla.yoctoproject.org/show_bug.cgi?id=14775
Signed-off-by Jose Quaresma <[email protected]>--- meta/classes/sstate.bbclass | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)meta/classes/sstate.bbclass +15
-2@@ -918,8 +918,28 @@ sstate_unpack_package () { BB_HASHCHECK_FUNCTION = "sstate_checkhashes" def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, **kwargs): - found = set() - missed = set() + # https://stackoverflow.com/questions/13610654/how-to-make-built-in-containers-sets-dicts-lists-thread-safe + import threading + class LockedSet(set): + """A set where add(), remove(), and 'in' operator are thread-safe""" + def __init__(self, *args, **kwargs): + self._lock = threading.Lock() + super(LockedSet, self).__init__(*args, **kwargs) + + def add(self, elem): + with self._lock: + super(LockedSet, self).add(elem) + + def remove(self, elem): + with self._lock: + super(LockedSet, self).remove(elem)
meta/classes/sstate.bbclass sstate_unpack_package () { · line 945
Quoted 10 lines
meta/classes/sstate.bbclass +7
-0+ + def __contains__(self, elem): + with self._lock: + super(LockedSet, self).__contains__(elem) + + found = LockedSet() + missed = LockedSet() def gethash(task): return sq_data['unihash'][task]
The series blew up on the autobuilder:
testsdkest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/5079/steps/17/logs/stdio
oe-selftest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/3461/steps/15/logs/stdio
and many more, see:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3535
Cheers,
Richard