scripts/autobuild-run: properly delete output dir even with write-protected folders
PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders
We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which shutil.rmtree() doesn't do.
However "rm -rf" fails on write protected folders.
So instead, we do a recursive chmod +w, and then use shutil.rmtree() since we are now sure that all folders and files will have write permissions.
This solves an autobuild-run failure that occurs when the output directory cannot be deleted due to write-protected directories. This is happening with mender-artefact Go modules for example.
--- scripts/autobuild-run | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/autobuild-run +4 -3
@@ -384,13 +384,14 @@ class Builder: # Create an empty output directory. We remove it first, in case a previous build was aborted. if os.path.exists(self.outputdir): - # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", self.outputdir]) + subprocess.call(["chmod", "-R", "+w", self.outputdir]) + shutil.rmtree(self.outputdir) os.mkdir(self.outputdir) # If it exists, remove the other output directory used for reproducibility testing if os.path.exists(self.outputdir_2): - subprocess.call(["rm", "-rf", self.outputdir_2]) + subprocess.call(["chmod", "-R", "+w", self.outputdir_2]) + shutil.rmtree(self.outputdir_2) with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch)
Signature · 1 line
2.26.2
PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders
Thomas, All,
On 2020-07-18 22:18 +0200, Thomas Petazzoni spake thusly:
Quoted 24 lines
We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which
shutil.rmtree()doesn't do.However "rm -rf" fails on write protected folders.
So instead, we do a recursive chmod +w, and then use
shutil.rmtree()since we are now sure that all folders and files will have write permissions.This solves an autobuild-run failure that occurs when the output directory cannot be deleted due to write-protected directories. This is happening with mender-artefact Go modules for example.
Signed-off-by Thomas Petazzoni <[email protected]>--- scripts/autobuild-run | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)scripts/autobuild-run +0
-0
scripts/autobuild-run class Builder: · line 388
Quoted 7 lines
scripts/autobuild-run +2
-2@@ -384,13 +384,14 @@ class Builder: # Create an empty output directory. We remove it first, in case a previous build was aborted. if os.path.exists(self.outputdir): - # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", self.outputdir]) + subprocess.call(["chmod", "-R", "+w", self.outputdir]) + shutil.rmtree(self.outputdir)
Why can't we do as suggested in the docs:
https://docs.python.org/3/library/shutil.html#rmtree-example def rm_ro(f, p, _): os.chmod(p, stat.S_IWRITE) f(p) shutil.rmtree(self.outputdir, onerror=rm_ro)
This is IMHO cleaner than calling to the shell...
I'm pretty sure a better python developper would write it with a lambda, which would be even better! ;-)
Regards,
Yann E. MORIN.
Quoted 17 lines
scripts/autobuild-run +2
-1os.mkdir(self.outputdir) # If it exists, remove the other output directory used for reproducibility testing if os.path.exists(self.outputdir_2): - subprocess.call(["rm", "-rf", self.outputdir_2]) + subprocess.call(["chmod", "-R", "+w", self.outputdir_2]) + shutil.rmtree(self.outputdir_2) with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch)Signature · 6 lines
2.26.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders
Thomas, All,
On 2020-07-18 22:55 +0200, Yann E. MORIN spake thusly:
Quoted 6 lines
On 2020-07-18 22:18 +0200, Thomas Petazzoni spake thusly:
Quoted 5 lines
We're already deleting the output directory before a new build with "rm -rf" so that write protected files are removed as well, which
shutil.rmtree()doesn't do.However "rm -rf" fails on write protected folders.
[--SNIP--]
Quoted 6 lines
scripts/autobuild-run class Builder: · line 388
Quoted 5 lines
if
os.path.exists(self.outputdir):scripts/autobuild-run +2
-2- # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", self.outputdir]) + subprocess.call(["chmod", "-R", "+w", self.outputdir]) + shutil.rmtree(self.outputdir)
Quoted 8 lines
Why can't we do as suggested in the docs:
https://docs.python.org/3/library/shutil.html#rmtree-example def rm_ro(f, p, _): os.chmod(p, stat.S_IWRITE) f(p) shutil.rmtree(self.outputdir, onerror=rm_ro)
We might need to be a little bit more careful, though:
def rm_ro(f, p, _):
os.chmod(os.path.dirname(p), stat.S_IWRITE)
os.chmod(p, stat.S_IWRITE)
f(p)
shutil.rmtree(self.outputdir, onerror=rm_ro)Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 scripts/autobuild-run: properly delete output dir even with write-protected folders
Hello,
Quoted 5 lines · On Sun, 19 Jul 2020 14:15:24 +0200 "Yann E. MORIN" <[email protected]>…
We might need to be a little bit more careful, though:
def rm_ro(f, p, _): os.chmod(os.path.dirname(p), stat.S_IWRITE) os.chmod(p, stat.S_IWRITE)
Thanks for the suggestion! This didn't work as is, since os.chmod() sets the permissions, it does not *add* to the existing permissions. So doing os.chmod(foo, stat.S_IWRITE) makes the "foo" directory only writable, no longer readable/executable.
So I fixed that, and just sent a v2.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com