Skip to contentSkip to conversation

utils: Use rm -rf in remove()

PATCH In review Richard Purdie 4 participants 7 messages Feb 8, 2013 – Feb 16, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 utils: Use rm -rf in remove()

Whilst shutils.rmtree() is pythonic, its also slow. Its faster to use rm -rf which makes optimal use of the right syscalls.

---

bitbake/lib/bb/utils.py +4 -2
index 484fb2d..94ef447 100644
@@ -533,13 +533,15 @@ def remove(path, recurse=False):
     """Equivalent to rm -f or rm -rf"""
     if not path:
         return
-    import os, errno, shutil, glob
+    import os, errno, glob, subprocess
     for name in glob.glob(path):
         try:
             os.unlink(name)
         except OSError as exc:
             if recurse and exc.errno == errno.EISDIR:
-                shutil.rmtree(name)
+                # shutil.rmtree(name) would be ideal but its too slow
+                subprocess.call('rm -rf %s' % path, shell=True)
+                return
             elif exc.errno != errno.ENOENT:
                 raise
 
#1 lore raw

bitbake/lib/bb/utils.py def remove(path, recurse=False): · line 543

Quoted 4 lines · Richard Purdie
bitbake/lib/bb/utils.py +2 -1
-                shutil.rmtree(name)
+                # shutil.rmtree(name) would be ideal but its too slow
+                subprocess.call('rm -rf %s' % path, shell=True)

This is a good idea, but I'm curious about forking off a shell process for it. I'd think this would work as well: subprocess.call(['rm', '-rf', path])

Signature · 1 line
Christopher Larson
#6 lore raw
In reply to Richard Purdie
Quoted 19 lines

-----Original Message-----
From: [email protected] [mailto:bitbake- [email protected]] On Behalf Of Richard Purdie
Sent: den 8 februari 2013 00:55
To: bitbake-devel
Subject: [bitbake-devel] [PATCH] utils: Use rm -rf in remove()

Whilst shutils.rmtree() is pythonic, its also slow. Its faster to use rm -rf which makes optimal use of the right syscalls.

---

bitbake/lib/bb/utils.py +0 -0
index 484fb2d..94ef447 100644
@@ -533,13 +533,15 @@ def remove(path, recurse=False):
     """Equivalent to rm -f or rm -rf"""
     if not path:

bitbake/lib/bb/utils.py def remove(path, recurse=False): · line 542

Quoted 10 lines
bitbake/lib/bb/utils.py +2 -2
         return
-    import os, errno, shutil, glob
+    import os, errno, glob, subprocess
     for name in glob.glob(path):
         try:
             os.unlink(name)
         except OSError as exc:
             if recurse and exc.errno == errno.EISDIR:
-                shutil.rmtree(name)
+                # shutil.rmtree(name) would be ideal but its too slow

its -> it is

bitbake/lib/bb/utils.py def remove(path, recurse=False): · line 544

Quoted 2 lines
bitbake/lib/bb/utils.py +2 -0
+                subprocess.call('rm -rf %s' % path, shell=True)
+                return

Shouldn't the last two lines above be:

subprocess.call('rm -rf %s' % name, shell=True)

to maintain behavior.

bitbake/lib/bb/utils.py def remove(path, recurse=False): · line 546

Quoted 2 lines
bitbake/lib/bb/utils.py +0 -0
             elif exc.errno != errno.ENOENT:
                 raise

Alternatively, wouldn't it be better to check for recurse first:

if recurse:
import subprocess
subprocess.call('rm -rf %s' % path, shell=True)
else:
import os, errno, glob
for name in glob.glob(path):
try:
os.unlink(name)
except OSError as exc:
if exc.errno != errno.ENOENT:
raise

//Peter

#2 lore raw
In reply to Chris Larson
Quoted 9 lines · Chris Larson

On Thu, Feb 7, 2013 at 4:55 PM, Richard Purdie
<[email protected]> wrote:
- shutil.rmtree(name)
+ # shutil.rmtree(name) would be ideal but its
too slow
+ subprocess.call('rm -rf %s' % path,
shell=True)

Quoted 3 lines

This is a good idea, but I'm curious about forking off a shell process for it. I'd think this would work as well: subprocess.call(['rm', '-rf', path])

path can have wildcards in it. The code wasn't entirely obvious so I've tweaked it after your/Peter's comments. I'm hoping it will help the problems Martin was seeing too.

Cheers,

Richard

#3 lore raw
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Chris Larson

On Thu, Feb 7, 2013 at 4:55 PM, Richard Purdie
<[email protected]> wrote:
- shutil.rmtree(name)
+ # shutil.rmtree(name) would be ideal but its
too slow
+ subprocess.call('rm -rf %s' % path,
shell=True)

This is a good idea, but I'm curious about forking off a shell process for it. I'd think this would work as well: subprocess.call(['rm', '-rf', path])

Quoted 9 lines

path can have wildcards in it. The code wasn't entirely obvious so I've tweaked it after your/Peter's comments. I'm hoping it will help the problems Martin was seeing too.

Cheers,

Richard

I think this is a good idea as well. One thing I would question though, is to have this in the except clause. Why not:

for name in glob.glob(path):
try:
if recurse and os.path.isdir(path):
subprocess.call('rm -rf %s' % path, shell=True) return
os.unlink(name)
except OSError as exc:
if exc.errno != errno.ENOENT:
raise

Personally I feel it is cleaner to have all the remove possibilities in the try, and let the exceptions be in except. Kind of feel it is wrong to let the recursive case be handled in the exception, as I don't see it being an exceptional case to delete a folder.

Also, unfortunately your patch would not fix osx, as EISDIR is not the error that is returned there. What I get is either a EPERM or an EACCES (not on a mac now, so can't check it until monday). So osx would still fall through to the raise.

#4 lore raw
Quoted 41 lines · Martin Ertsås
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Chris Larson

On Thu, Feb 7, 2013 at 4:55 PM, Richard Purdie
<[email protected]> wrote:
- shutil.rmtree(name)
+ # shutil.rmtree(name) would be ideal but its
too slow
+ subprocess.call('rm -rf %s' % path,
shell=True)

This is a good idea, but I'm curious about forking off a shell process for it. I'd think this would work as well: subprocess.call(['rm', '-rf', path])

Quoted 9 lines

path can have wildcards in it. The code wasn't entirely obvious so I've tweaked it after your/Peter's comments. I'm hoping it will help the problems Martin was seeing too.

Cheers,

Richard

I think this is a good idea as well. One thing I would question though, is to have this in the except clause. Why not:

for name in glob.glob(path):
try:
if recurse and os.path.isdir(path):
subprocess.call('rm -rf %s' % path, shell=True) return
os.unlink(name)
except OSError as exc:
if exc.errno != errno.ENOENT:
raise

Personally I feel it is cleaner to have all the remove possibilities in the try, and let the exceptions be in except. Kind of feel it is wrong to let the recursive case be handled in the exception, as I don't see it being an exceptional case to delete a folder.

Quoted 4 lines

Also, unfortunately your patch would not fix osx, as EISDIR is not the error that is returned there. What I get is either a EPERM or an EACCES (not on a mac now, so can't check it until monday). So osx would still fall through to the raise.

Can you take a look at the patch I ended up committing? Its basically as you describe above expect I moved the subprocess outside the for loop as well.

(http://git.openembedded.org/bitbake/commit/?id=96088ebdec08e49ba9e8dbcac437bfcdc21f5983)

Cheers,

Richard

#5 lore raw
Quoted 50 lines · Richard Purdie
Quoted 41 lines · Martin Ertsås
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Chris Larson

On Thu, Feb 7, 2013 at 4:55 PM, Richard Purdie
<[email protected]> wrote:
- shutil.rmtree(name)
+ # shutil.rmtree(name) would be ideal but its
too slow
+ subprocess.call('rm -rf %s' % path,
shell=True)

This is a good idea, but I'm curious about forking off a shell process for it. I'd think this would work as well: subprocess.call(['rm', '-rf', path])

Quoted 9 lines

path can have wildcards in it. The code wasn't entirely obvious so I've tweaked it after your/Peter's comments. I'm hoping it will help the problems Martin was seeing too.

Cheers,

Richard

I think this is a good idea as well. One thing I would question though, is to have this in the except clause. Why not:

for name in glob.glob(path):
try:
if recurse and os.path.isdir(path):
subprocess.call('rm -rf %s' % path, shell=True) return
os.unlink(name)
except OSError as exc:
if exc.errno != errno.ENOENT:
raise

Personally I feel it is cleaner to have all the remove possibilities in the try, and let the exceptions be in except. Kind of feel it is wrong to let the recursive case be handled in the exception, as I don't see it being an exceptional case to delete a folder.

Quoted 4 lines

Also, unfortunately your patch would not fix osx, as EISDIR is not the error that is returned there. What I get is either a EPERM or an EACCES (not on a mac now, so can't check it until monday). So osx would still fall through to the raise.

Can you take a look at the patch I ended up committing? Its basically as you describe above expect I moved the subprocess outside the for loop as well.

Quoted 8 lines

(http://git.openembedded.org/bitbake/commit/?id=96088ebdec08e49ba9e8dbcac437bfcdc21f5983)

Cheers,

Richard

That is great! Will fix up the mac problems without any problems, at least it looks like. I'll try it out on monday, and come back complaining if it doesn't fix it ;)

- Martin