Skip to contentSkip to conversation

genboardscfg: limit to 240 jobs

PATCH In review Andre Przywara 3 participants 4 messages Jan 11, 2022 – Jan 13, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 genboardscfg: limit to 240 jobs

When genboardscfg.py is run on machines with 255 or more cores, the process will consume more than 1024 file descriptors, which is a common standard ulimit for user processes. As a consequence it will fail with a lenghty Python trace, with the almost hidden message:
OSError: [Errno 24] Too many open files

It's somewhat questionable whether that level of parallelity is actually useful for genboardscfg, so we limit the *default* number of jobs to the safe number of 240, to avoid the problem.
If a user persists, she can still force a higher number via the -j parameter - hopefully having raised the ulimit accordingly beforehand.

---
 tools/genboardscfg.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/genboardscfg.py +1 -1
index 4ee7aa1f89..07bf681d1d 100755
@@ -430,7 +430,7 @@ def main():
     # Add options here
     parser.add_option('-f', '--force', action="store_true", default=False,
                       help='regenerate the output even if it is new')
-    parser.add_option('-j', '--jobs', type='int', default=cpu_count,
+    parser.add_option('-j', '--jobs', type='int', default=min(cpu_count, 240),
                       help='the number of jobs to run simultaneously')
     parser.add_option('-o', '--output', default=OUTPUT_FILE,
                       help='output file [default=%s]' % OUTPUT_FILE)
Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 10 lines · Andre Przywara

When genboardscfg.py is run on machines with 255 or more cores, the process will consume more than 1024 file descriptors, which is a common standard ulimit for user processes. As a consequence it will fail with a lenghty Python trace, with the almost hidden message:
OSError: [Errno 24] Too many open files

It's somewhat questionable whether that level of parallelity is actually useful for genboardscfg, so we limit the *default* number of jobs to the safe number of 240, to avoid the problem.
If a user persists, she can still force a higher number via the -j

Nits:
%s/persists/insists/
%s/she/they/ if you don't specifically mean a female user.

Otherwise

Quoted 20 lines

parameter - hopefully having raised the ulimit accordingly beforehand.

---
  tools/genboardscfg.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/genboardscfg.py +1 -1
index 4ee7aa1f89..07bf681d1d 100755
@@ -430,7 +430,7 @@ def main():
      # Add options here
      parser.add_option('-f', '--force', action="store_true", default=False,
                        help='regenerate the output even if it is new')
-    parser.add_option('-j', '--jobs', type='int', default=cpu_count,
+    parser.add_option('-j', '--jobs', type='int', default=min(cpu_count, 240),
                        help='the number of jobs to run simultaneously')
      parser.add_option('-o', '--output', default=OUTPUT_FILE,
                        help='output file [default=%s]' % OUTPUT_FILE)
Reviewed-by on v1 1/1
#2 lore raw
In reply to Andre Przywara
Quoted 13 lines · Andre Przywara

When genboardscfg.py is run on machines with 255 or more cores, the process will consume more than 1024 file descriptors, which is a common standard ulimit for user processes. As a consequence it will fail with a lenghty Python trace, with the almost hidden message:
OSError: [Errno 24] Too many open files

It's somewhat questionable whether that level of parallelity is actually useful for genboardscfg, so we limit the *default* number of jobs to the safe number of 240, to avoid the problem.
If a user persists, she can still force a higher number via the -j parameter - hopefully having raised the ulimit accordingly beforehand.

Quoted 4 lines
---
 tools/genboardscfg.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Please send such machine. Thank you.

Reviewed-by
#3 lore raw
Quoted 13 lines · Andre Przywara

When genboardscfg.py is run on machines with 255 or more cores, the process will consume more than 1024 file descriptors, which is a common standard ulimit for user processes. As a consequence it will fail with a lenghty Python trace, with the almost hidden message:
OSError: [Errno 24] Too many open files

It's somewhat questionable whether that level of parallelity is actually useful for genboardscfg, so we limit the *default* number of jobs to the safe number of 240, to avoid the problem.
If a user persists, she can still force a higher number via the -j parameter - hopefully having raised the ulimit accordingly beforehand.

Quoted 4 lines
---
 tools/genboardscfg.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Please send such machine. Thank you.

Applied to u-boot-dm, thanks!