Skip to contentSkip to conversation

support/scripts/pkg-stats: clear multiprocessing pools after use

PATCH New Titouan Christophe 2 participants 2 messages Mar 1, 2020 – Mar 2, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 support/scripts/pkg-stats: clear multiprocessing pools after use

During the CVE checking phase, we can still see a huge amount of Python processes (actually 128) running on the host, even though the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the packages URL statuses and the latest versions from release-monitoring. This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

The problem was already there before the CVE matching step was introduced, but because pkg-stat was terminating right after the release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from the Package class, as this is not needed.

---
 support/scripts/pkg-stats | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +5 -2
index 7721d98459..4f653a6ad9 100755
@@ -402,11 +402,13 @@ def check_url_status_worker(url, url_status):
 
 
 def check_package_urls(packages):
-    Package.pool = Pool(processes=64)
+    pool = Pool(processes=64)
     for pkg in packages:
-        pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
+        pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
     for pkg in packages:
         pkg.url_status = pkg.url_worker.get(timeout=3600)
+        del pkg.url_worker
+    pool.terminate()
 
 
 def release_monitoring_get_latest_version_by_distro(pool, name):
@@ -479,6 +481,7 @@ def check_package_latest_version(packages):
     results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
     for pkg, r in zip(packages, results):
         pkg.latest_version = r
+    worker_pool.terminate()
     del http_pool
 
 
Signature · 1 line
2.24.1
#1 lore raw

PATCH 1/1 support/scripts/pkg-stats: clear multiprocessing pools after use

Quoted 11 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Titouan" == Titouan Christophe <[email protected]> writes:

During the CVE checking phase, we can still see a huge amount of Python processes (actually 128) running on the host, even though the CVE step is entirely ran in the main thread.

These are actually the worker processes spawned to check for the packages URL statuses and the latest versions from release-monitoring. This is because of an issue in Python's multiprocessing implementation:
https://bugs.python.org/issue34172

Quoted 8 lines

The problem was already there before the CVE matching step was introduced, but because pkg-stat was terminating right after the release-monitoring step, it went unnoticed.

Also, do not hold a reference to the multiprocessing pool from the Package class, as this is not needed.

Committed, thanks.

Signature · 1 line
Bye, Peter Korsgaard