Skip to contentSkip to conversation

scripts/autobuild-run: properly delete output dir even with write-protected folders

PATCH Superseded Thomas Petazzoni 2 participants 4 messages Jul 18, 2020 – Aug 8, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders

We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which shutil.rmtree() doesn't do.

However "rm -rf" fails on write protected folders.

So instead, we do a recursive chmod +w, and then use shutil.rmtree() since we are now sure that all folders and files will have write permissions.

This solves an autobuild-run failure that occurs when the output directory cannot be deleted due to write-protected directories. This is happening with mender-artefact Go modules for example.

---
 scripts/autobuild-run | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/autobuild-run +4 -3
index e475ea8..212e751 100755
@@ -384,13 +384,14 @@ class Builder:
 
         # Create an empty output directory. We remove it first, in case a previous build was aborted.
         if os.path.exists(self.outputdir):
-            # shutil.rmtree doesn't remove write-protected files
-            subprocess.call(["rm", "-rf", self.outputdir])
+            subprocess.call(["chmod", "-R", "+w", self.outputdir])
+            shutil.rmtree(self.outputdir)
         os.mkdir(self.outputdir)
 
         # If it exists, remove the other output directory used for reproducibility testing
         if os.path.exists(self.outputdir_2):
-            subprocess.call(["rm", "-rf", self.outputdir_2])
+            subprocess.call(["chmod", "-R", "+w", self.outputdir_2])
+            shutil.rmtree(self.outputdir_2)
         with open(os.path.join(self.outputdir, "branch"), "w") as branchf:
             branchf.write(branch)
 
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders

Thomas, All,

On 2020-07-18 22:18 +0200, Thomas Petazzoni spake thusly:

Quoted 24 lines

We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which shutil.rmtree() doesn't do.

However "rm -rf" fails on write protected folders.

So instead, we do a recursive chmod +w, and then use shutil.rmtree() since we are now sure that all folders and files will have write permissions.

This solves an autobuild-run failure that occurs when the output directory cannot be deleted due to write-protected directories. This is happening with mender-artefact Go modules for example.

---
 scripts/autobuild-run | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/autobuild-run +0 -0
index e475ea8..212e751 100755

scripts/autobuild-run class Builder: · line 388

Quoted 7 lines
scripts/autobuild-run +2 -2
@@ -384,13 +384,14 @@ class Builder:
 
         # Create an empty output directory. We remove it first, in case a previous build was aborted.
         if os.path.exists(self.outputdir):
-            # shutil.rmtree doesn't remove write-protected files
-            subprocess.call(["rm", "-rf", self.outputdir])
+            subprocess.call(["chmod", "-R", "+w", self.outputdir])
+            shutil.rmtree(self.outputdir)

Why can't we do as suggested in the docs:

    https://docs.python.org/3/library/shutil.html#rmtree-example

    def rm_ro(f, p, _):
        os.chmod(p, stat.S_IWRITE)
        f(p)

    shutil.rmtree(self.outputdir, onerror=rm_ro)

This is IMHO cleaner than calling to the shell...

I'm pretty sure a better python developper would write it with a lambda, which would be even better! ;-)

Regards,
Yann E. MORIN.

Quoted 17 lines
scripts/autobuild-run +2 -1
         os.mkdir(self.outputdir)
 
         # If it exists, remove the other output directory used for reproducibility testing
         if os.path.exists(self.outputdir_2):
-            subprocess.call(["rm", "-rf", self.outputdir_2])
+            subprocess.call(["chmod", "-R", "+w", self.outputdir_2])
+            shutil.rmtree(self.outputdir_2)
         with open(os.path.join(self.outputdir, "branch"), "w") as branchf:
             branchf.write(branch)
 
Signature · 6 lines
2.26.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders

Thomas, All,

On 2020-07-18 22:55 +0200, Yann E. MORIN spake thusly:

Quoted 6 lines

On 2020-07-18 22:18 +0200, Thomas Petazzoni spake thusly:

Quoted 5 lines

We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which shutil.rmtree() doesn't do.

However "rm -rf" fails on write protected folders.

[--SNIP--]

Quoted 6 lines

scripts/autobuild-run class Builder: · line 388

Quoted 5 lines

if os.path.exists(self.outputdir):

scripts/autobuild-run +2 -2
-            # shutil.rmtree doesn't remove write-protected files
-            subprocess.call(["rm", "-rf", self.outputdir])
+            subprocess.call(["chmod", "-R", "+w", self.outputdir])
+            shutil.rmtree(self.outputdir)
Quoted 8 lines

Why can't we do as suggested in the docs:

    https://docs.python.org/3/library/shutil.html#rmtree-example

    def rm_ro(f, p, _):
        os.chmod(p, stat.S_IWRITE)
        f(p)

    shutil.rmtree(self.outputdir, onerror=rm_ro)

We might need to be a little bit more careful, though:

    def rm_ro(f, p, _):
        os.chmod(os.path.dirname(p), stat.S_IWRITE)
        os.chmod(p, stat.S_IWRITE)
        f(p)

    shutil.rmtree(self.outputdir, onerror=rm_ro)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders

Hello,

Quoted 5 lines · On Sun, 19 Jul 2020 14:15:24 +0200 "Yann E. MORIN" <[email protected]>…

We might need to be a little bit more careful, though:

    def rm_ro(f, p, _):
        os.chmod(os.path.dirname(p), stat.S_IWRITE)
        os.chmod(p, stat.S_IWRITE)

Thanks for the suggestion! This didn't work as is, since os.chmod() sets the permissions, it does not *add* to the existing permissions. So doing os.chmod(foo, stat.S_IWRITE) makes the "foo" directory only writable, no longer readable/executable.

So I fixed that, and just sent a v2.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com