Skip to contentSkip to conversation

tools/genboardscfg.py: Don't rely on subprocess.check_output

PATCH New Tom Rini 3 participants 8 messages Jul 30, 2014

lore
Share & tools
lore

Tom Rini

author
#0 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +6 -3
index 805e4eb..6588392 100755
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:
         try:
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1
Signature · 1 line
1.7.0.4
#1 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

Hi Tom,

Quoted 6 lines · Tom Rini

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

Looks fine as it is - see a few optional comments below.

Quoted 9 lines
---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +0 -0
index 805e4eb..6588392 100755

tools/genboardscfg.py def check_top_directory(): · line 87

Quoted 6 lines
tools/genboardscfg.py +2 -1
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()

Sometimes people use underscore for unused variables, like:

make_cmd, _ = process.communicate()

but in this case you could also do:

make_cmd = process.communicate()[0]

Quoted 17 lines
tools/genboardscfg.py +4 -3
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:
         try:
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1
--
Quoted 1 line

1.7.0.4

Regards,
Simon

author
#2 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

Quoted 38 lines · Simon Glass

Hi Tom,

Quoted 6 lines · Tom Rini

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

Looks fine as it is - see a few optional comments below.

Quoted 9 lines
---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +0 -0
index 805e4eb..6588392 100755

tools/genboardscfg.py def check_top_directory(): · line 87

Quoted 6 lines
tools/genboardscfg.py +2 -1
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()

Sometimes people use underscore for unused variables, like:

make_cmd, _ = process.communicate()

but in this case you could also do:

make_cmd = process.communicate()[0]

Quoted 16 lines
Quoted 6 lines
tools/genboardscfg.py +0 -0
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:

tools/genboardscfg.py def main(): · line 503

Quoted 10 lines
tools/genboardscfg.py +4 -2
         try:
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1

Ah, OK, thanks. I've cleaned up both to just do communicate()[0] (and condensed the jobs part back to a single line). One last sanity check build going on now.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140730/0a90a949/attachment.pgp>
#3 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

In reply to Tom Rini

Hi Tom,

2014-07-30 23:24 GMT+09:00 Tom Rini <[email protected]>:

Quoted 5 lines

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

Unfortunately.. :-(
Anyway, thanks for catching this issue.

Quoted 10 lines
---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +0 -0
index 805e4eb..6588392 100755

tools/genboardscfg.py def check_top_directory(): · line 90

Quoted 9 lines
tools/genboardscfg.py +2 -1
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)

No good.

Unlike check_output(), the communicate() method never throws CalledProcessError exception,
which means the lines in except are never run.

When scripts/show-gnu-make fails, the function will not error out, but just return a null string.
To know if the subprocess succeeded or not, 'returncode' should be checked.

The correct code should be something like this:

def get_make_cmd():
"""Get the command name of GNU Make."""
process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) ret = process.communicate()
if process.returncode:
print >> sys.stderr, 'GNU Make not found' sys.exit(1)
return ret[0].strip()

Quoted 5 lines
tools/genboardscfg.py +0 -0
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:
         try:

tools/genboardscfg.py def main():

Quoted 10 lines
tools/genboardscfg.py +4 -3
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1
--

Ditto.
'except subprocess.CalledProcessError:' is meaningless and never catches an exception.

In this case, 'getconf' may not exist on some systems.

If the 'getconf' command is not found, Popen() will throw OSError exception. If the command is found but fails by some reason, int() will throw ValueError. We cannot handle the other exceptions.

So, we can write the code something like this:

...
else:
try:
jobs = int(subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
stdout=subprocess.PIPE).communicate()[0])
except (OSError, ValueError):
# getconf command not found or fails
print 'info: failed to get the number of CPUs. Set jobs to 1' jobs = 1
gen_boards_cfg(jobs)

Best Regards
Masahiro Yamada

author
#4 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

Quoted 89 lines · Masahiro YAMADA

Hi Tom,

2014-07-30 23:24 GMT+09:00 Tom Rini <[email protected]>:

Quoted 5 lines

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

Unfortunately.. :-(
Anyway, thanks for catching this issue.

Quoted 10 lines
---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +0 -0
index 805e4eb..6588392 100755

tools/genboardscfg.py def check_top_directory(): · line 90

Quoted 9 lines
tools/genboardscfg.py +2 -1
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)

No good.

Unlike check_output(), the communicate() method never throws CalledProcessError exception,
which means the lines in except are never run.

When scripts/show-gnu-make fails, the function will not error out, but just return a null string.
To know if the subprocess succeeded or not, 'returncode' should be checked.

The correct code should be something like this:

def get_make_cmd():
"""Get the command name of GNU Make."""
process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) ret = process.communicate()
if process.returncode:
print >> sys.stderr, 'GNU Make not found' sys.exit(1)
return ret[0].strip()

Quoted 5 lines
tools/genboardscfg.py +0 -0
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:
         try:

tools/genboardscfg.py def main():

Quoted 10 lines
tools/genboardscfg.py +4 -3
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1
--

Ditto.
'except subprocess.CalledProcessError:' is meaningless and never catches an exception.

In this case, 'getconf' may not exist on some systems.

If the 'getconf' command is not found, Popen() will throw OSError exception. If the command is found but fails by some reason, int() will throw ValueError. We cannot handle the other exceptions.

So, we can write the code something like this:

Quoted 10 lines

...
else:
try:
jobs = int(subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
stdout=subprocess.PIPE).communicate()[0])
except (OSError, ValueError):
# getconf command not found or fails
print 'info: failed to get the number of CPUs. Set jobs to 1' jobs = 1
gen_boards_cfg(jobs)

Arg. Do you want me to fold / rework things like that or do you want to post a v9 of just this patch, adapted as you've shown?

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140730/62dea442/attachment.pgp>
author
#5 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

Quoted 101 lines · Tom Rini
Quoted 89 lines · Masahiro YAMADA

Hi Tom,

2014-07-30 23:24 GMT+09:00 Tom Rini <[email protected]>:

Quoted 5 lines

The function subprocess.check_output() only exists in Python 2.7 and later (and there are long term supported distributions that ship with 2.6.x). Replace this with a call to subprocess.Popen() and then checking output via communicate()

Unfortunately.. :-(
Anyway, thanks for catching this issue.

Quoted 10 lines
---
 tools/genboardscfg.py |    9 ++++++---
 1 files changed, 6 insertions(+), 3 deletions(-)
tools/genboardscfg.py +0 -0
index 805e4eb..6588392 100755

tools/genboardscfg.py def check_top_directory(): · line 90

Quoted 9 lines
tools/genboardscfg.py +2 -1
@@ -83,7 +83,8 @@ def check_top_directory():
 def get_make_cmd():
     """Get the command name of GNU Make."""
     try:
-        make_cmd = subprocess.check_output([SHOW_GNU_MAKE])
+        process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
+        make_cmd, unused = process.communicate()
     except subprocess.CalledProcessError:
         print >> sys.stderr, 'GNU Make not found'
         sys.exit(1)

No good.

Unlike check_output(), the communicate() method never throws CalledProcessError exception,
which means the lines in except are never run.

When scripts/show-gnu-make fails, the function will not error out, but just return a null string.
To know if the subprocess succeeded or not, 'returncode' should be checked.

The correct code should be something like this:

def get_make_cmd():
"""Get the command name of GNU Make."""
process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) ret = process.communicate()
if process.returncode:
print >> sys.stderr, 'GNU Make not found' sys.exit(1)
return ret[0].strip()

Quoted 5 lines
tools/genboardscfg.py +0 -0
@@ -493,8 +494,10 @@ def main():
             sys.exit(1)
     else:
         try:

tools/genboardscfg.py def main():

Quoted 10 lines
tools/genboardscfg.py +4 -3
-            jobs = int(subprocess.check_output(['getconf',
-                                                '_NPROCESSORS_ONLN']))
+            process = subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
+                                       stdout=subprocess.PIPE)
+            jobstr, unused = process.communicate()
+            jobs = int(jobstr)
         except subprocess.CalledProcessError:
             print 'info: failed to get the number of CPUs. Set jobs to 1'
             jobs = 1
--

Ditto.
'except subprocess.CalledProcessError:' is meaningless and never catches an exception.

In this case, 'getconf' may not exist on some systems.

If the 'getconf' command is not found, Popen() will throw OSError exception. If the command is found but fails by some reason, int() will throw ValueError. We cannot handle the other exceptions.

So, we can write the code something like this:

Quoted 10 lines

...
else:
try:
jobs = int(subprocess.Popen(['getconf', '_NPROCESSORS_ONLN'],
stdout=subprocess.PIPE).communicate()[0])
except (OSError, ValueError):
# getconf command not found or fails
print 'info: failed to get the number of CPUs. Set jobs to 1' jobs = 1
gen_boards_cfg(jobs)

Quoted 2 lines

Arg. Do you want me to fold / rework things like that or do you want to post a v9 of just this patch, adapted as you've shown?

... doing it this way now, testing, will move from there :)

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140730/57eacf01/attachment.pgp>
#6 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

In reply to Tom Rini

Hi Tom,

2014-07-31 3:04 GMT+09:00 Tom Rini <[email protected]>:

Quoted 2 lines

Arg. Do you want me to fold / rework things like that or do you want to post a v9 of just this patch, adapted as you've shown?

Either is fine for me.

If you were about to apply v8 (I saw you are doing one last build check now), could you rework things, please?

No more updates from me.

Best Regards
Masahiro Yamada

#7 lore raw

PATCH 1/1 tools/genboardscfg.py: Don't rely on subprocess.check_output

In reply to Tom Rini

Tom,

Quoted 4 lines

Arg. Do you want me to fold / rework things like that or do you want to post a v9 of just this patch, adapted as you've shown?

Either is fine for me.

But if you were about to apply v8
(it looks like you are doing the final test),
could you rework things, please?

No more updates from me. Thanks!

Best Regards
Masahiro Yamada