Skip to contentSkip to conversation

sepolicy: generate man pages in parallel

PATCH Superseded Ondrej Mosnacek 3 participants 6 messages Oct 14, 2019 – Oct 18, 2019 SELinux security module

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sepolicy: generate man pages in parallel

Generating man pages takes a lot of time. Do it in parallel to speed up the process.

---
 python/sepolicy/sepolicy.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
python/sepolicy/sepolicy.py +10 -4
index 1934cd86..02094013 100755
@@ -25,6 +25,7 @@ import os
 import sys
 import selinux
 import sepolicy
+from concurrent.futures import ProcessPoolExecutor
 from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
 import argparse
 PROGNAME = "policycoreutils"
@@ -326,8 +327,13 @@ def gen_gui_args(parser):
     gui.set_defaults(func=gui_run)
 
 
+def manpage_work(domain, path, root, source_files, web):
+    from sepolicy.manpage import ManPage
+    m = ManPage(domain, path, root, source_files, web)
+    print(m.get_man_page_path())
+
 def manpage(args):
-    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
+    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
 
     path = args.path
     if not args.policy and args.root != "/":
@@ -340,9 +346,9 @@ def manpage(args):
     else:
         test_domains = args.domain
 
-    for domain in test_domains:
-        m = ManPage(domain, path, args.root, args.source_files, args.web)
-        print(m.get_man_page_path())
+    with ProcessPoolExecutor() as e:
+        for domain in test_domains:
+            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
 
     if args.web:
         HTMLManPages(manpage_roles, manpage_domains, path, args.os)
Signature · 1 line
2.21.0
Acked-by on v1 1/1
#1 lore raw
Quoted 4 lines · Ondrej Mosnacek

Generating man pages takes a lot of time. Do it in parallel to speed up the process.

Quoted 45 lines
---
  python/sepolicy/sepolicy.py | 14 ++++++++++----
  1 file changed, 10 insertions(+), 4 deletions(-)
python/sepolicy/sepolicy.py +10 -4
index 1934cd86..02094013 100755
@@ -25,6 +25,7 @@ import os
  import sys
  import selinux
  import sepolicy
+from concurrent.futures import ProcessPoolExecutor
  from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
  import argparse
  PROGNAME = "policycoreutils"
@@ -326,8 +327,13 @@ def gen_gui_args(parser):
      gui.set_defaults(func=gui_run)
 
 
+def manpage_work(domain, path, root, source_files, web):
+    from sepolicy.manpage import ManPage
+    m = ManPage(domain, path, root, source_files, web)
+    print(m.get_man_page_path())
+
  def manpage(args):
-    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
+    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
 
      path = args.path
      if not args.policy and args.root != "/":
@@ -340,9 +346,9 @@ def manpage(args):
      else:
          test_domains = args.domain
 
-    for domain in test_domains:
-        m = ManPage(domain, path, args.root, args.source_files, args.web)
-        print(m.get_man_page_path())
+    with ProcessPoolExecutor() as e:
+        for domain in test_domains:
+            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
 
      if args.web:
          HTMLManPages(manpage_roles, manpage_domains, path, args.os)
#2 lore raw
Quoted 7 lines · Stephen Smalley
Quoted 4 lines · Ondrej Mosnacek

Generating man pages takes a lot of time. Do it in parallel to speed up the process.

Thank you for the ack, however I discovered that after this change it becomes more difficult to end the program via KeyboardInterrupt (SIGINT). The first interrupt only stops the main process and you need to send several more to take down the background processes as well...

I found a different way (multiprocessing.Pool) to do the same, which ends the processing gracefully on interrupt, but that one behaves even worse under Python 2 (each interrupt only cancels one work item and the processing happily continues...). Since there are plans to support only Python 3 in 3.0+ this may not be an issue, but I could also add a few lines to fallback to sequential execution under Python 2 for the sake of compatibility. Would that be OK or should I not bother?

Either way I'd like to send a v2 that uses multiprocessing instead of concurrent.futures, so please don't merge this yet :)

FYI, here is a preliminary diff for a switch to multiprocessing.Pool:
https://github.com/WOnder93/selinux/commit/a33acec8c298c112f5412b8b61b5b09058a267ee

...and here is what the Python 2 fallback would look like:
https://github.com/WOnder93/selinux/commit/b39a12120656b50eb0a1ee01227646ba3cd63f15

Quoted 47 lines
Quoted 46 lines
---
  python/sepolicy/sepolicy.py | 14 ++++++++++----
  1 file changed, 10 insertions(+), 4 deletions(-)
python/sepolicy/sepolicy.py +10 -4
index 1934cd86..02094013 100755
@@ -25,6 +25,7 @@ import os
  import sys
  import selinux
  import sepolicy
+from concurrent.futures import ProcessPoolExecutor
  from sepolicy import get_os_version, get_conditionals, get_conditionals_format_text
  import argparse
  PROGNAME = "policycoreutils"
@@ -326,8 +327,13 @@ def gen_gui_args(parser):
      gui.set_defaults(func=gui_run)
 
 
+def manpage_work(domain, path, root, source_files, web):
+    from sepolicy.manpage import ManPage
+    m = ManPage(domain, path, root, source_files, web)
+    print(m.get_man_page_path())
+
  def manpage(args):
-    from sepolicy.manpage import ManPage, HTMLManPages, manpage_domains, manpage_roles, gen_domains
+    from sepolicy.manpage import HTMLManPages, manpage_domains, manpage_roles, gen_domains
 
      path = args.path
      if not args.policy and args.root != "/":
@@ -340,9 +346,9 @@ def manpage(args):
      else:
          test_domains = args.domain
 
-    for domain in test_domains:
-        m = ManPage(domain, path, args.root, args.source_files, args.web)
-        print(m.get_man_page_path())
+    with ProcessPoolExecutor() as e:
+        for domain in test_domains:
+            e.submit(manpage_work, domain, path, args.root, args.source_files, args.web)
 
      if args.web:
          HTMLManPages(manpage_roles, manpage_domains, path, args.os)
Signature · 3 lines
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.
#3 lore raw
Quoted 4 lines · Ondrej Mosnacek

Since there are plans to support
only Python 3 in 3.0+ this may not be an issue, but I could also add a few lines to fallback to sequential execution under Python 2 for the sake of compatibility. Would that be OK or should I not bother?

Python 2 end of life is in less than 2 months. Please don't add new code only for Python 2 compatibility.

Signature · 1 line
Chris PeBenito
#4 lore raw
Quoted 7 lines · Chris PeBenito
Quoted 4 lines · Ondrej Mosnacek

Since there are plans to support
only Python 3 in 3.0+ this may not be an issue, but I could also add a few lines to fallback to sequential execution under Python 2 for the sake of compatibility. Would that be OK or should I not bother?

Python 2 end of life is in less than 2 months.  Please don't add new code only for Python 2 compatibility.

I can't count. It's a little over 2 months. The point still stands :)

Signature · 1 line
Chris PeBenito
#5 lore raw
Quoted 11 lines · Chris PeBenito
Quoted 8 lines · Chris PeBenito
Quoted 4 lines · Ondrej Mosnacek

Since there are plans to support
only Python 3 in 3.0+ this may not be an issue, but I could also add a few lines to fallback to sequential execution under Python 2 for the sake of compatibility. Would that be OK or should I not bother?

Python 2 end of life is in less than 2 months. Please don't add new code only for Python 2 compatibility.

I can't count. It's a little over 2 months. The point still stands :)

OK, I posted a v2 without the fallback.

Signature · 3 lines
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.