Badly formatted subpath for git SRC_URI deletes portions of hard drive.
Hi
I was writing a bitbake recipe for a project hosted on a git-server. I applied subpath in order to limit the checkout to a specific directory.
When I ran the recipe it failed in do_fetch (or do_unpack) complaining about missing files. It had deleted the entire Yocto directory and selected directories from my home-dir like ssh-keys, my dir where I keep my code, and some configuration files for Thunderbird and my terminal.
Since I had no Yocto directory left I was unable to debug the recipe.
After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially the SRC_URI looked like this:SRC_URI = "git://git@<server+repo>;protocol=ssh;subpath=path/"
Running the recipe resulted in do_fetch (or do_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.
I was keeping bitbake in my home-dir both of these times.
The poky version I am using is Daisy.
Best regards // John Ernberg
Hi John,
Quoted 24 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:
I was writing a bitbake recipe for a project hosted on a git-server. I applied subpath in order to limit the checkout to a specific directory.
When I ran the recipe it failed in
do_fetch(ordo_unpack) complaining about missing files. It had deleted the entire Yocto directory and selected directories from my home-dir like ssh-keys, my dir where I keep my code, and some configuration files for Thunderbird and my terminal.Since I had no Yocto directory left I was unable to debug the recipe.
After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially theSRC_URIlooked like this:SRC_URI= "git://git@<server+repo>;protocol=ssh;subpath=path/"Running the recipe resulted in
do_fetch(ordo_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.
Quoted 3 lines
I was keeping bitbake in my home-dir both of these times.
The poky version I am using is Daisy.
Very sorry about this :(
Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.
It may be worth noting in case it wasn't clear that when it is working correctly subpath= doesn't do a shallow clone, it still clones the full repository, it's just that you only get the specified subdirectory unpacked into the work directory after that.
- Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 16 lines · Paul Eggleton
Hi John,
Quoted 13 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:
After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially theSRC_URIlooked like this:SRC_URI= "git://git@<server+repo>;protocol=ssh;subpath=path/"Running the recipe resulted in
do_fetch(ordo_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.
It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...
Quoted 1 line
Very sorry about this :(
Yeah, too bad...
Quoted 3 lines
Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.
The issue with having '/' at the end ought to be solved by adding .rstrip('/') like:
----------------------------
bitbake/lib/bb/fetch2/git.py +1 -1
@@ -246,7 +246,7 @@ class Git(FetchMethod): subdir = ud.parm.get("subpath", "") if subdir != "": readpathspec = ":%s" % (subdir) - def_destsuffix = "%s/" % os.path.basename(subdir) + def_destsuffix = "%s/" %
os.path.basename(subdir.rstrip('/'))
else:
readpathspec = ""
def_destsuffix = "git/"
---------------------------
Probably, the first issue could be solved by replacing subdir with subdir.strip('/') on the line above that.
Though I might be missing something...
I've only looked for a moment on the patch above, which at least seems to create correct path's when inspecting the output.
Cheers,
Anders
Signature · 2 lines
Anders Darander ChargeStorm AB / eStorm AB
Quoted 9 lines · Anders Darander
Quoted 1 line
Quoted 1 line
Running the recipe resulted in
do_fetch(ordo_unpack) failing againand
Quoted 4 lines
Quoted 4 lines
complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.
It's most likely a call to
bb.utils.prunedir(destdir), with destdir being set to '/'...
ouch... this is a bit scary ;-)
maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..
hopefully nobody does sudo bitbake ;-)
Quoted 15 lines · Nicolas Dechesne
Quoted 9 lines · Anders Darander
Quoted 1 line
Quoted 1 line
Running the recipe resulted in
do_fetch(ordo_unpack) failing againand
Quoted 4 lines
Quoted 4 lines
complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.
It's most likely a call to
bb.utils.prunedir(destdir), with destdir being set to '/'...
Quoted 7 lines
ouch... this is a bit scary ;-)
maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..
hopefully nobody does sudo bitbake ;-)
This is scary indeed. We definitely need to be more careful about calls to dirname/basename with paths with a trailing or leading / in general, and especially for removals, and just need to do more path sanitization/normalization probably..
Quoted 1 line
Quoted 1 line
Quoted 1 line
def dirsplit(p):
... return os.path.dirname(p), os.path.basename(p)
...
Quoted 1 line
Quoted 1 line
Quoted 1 line
dirsplit('/usr/bin/foo')
('/usr/bin', 'foo')
Quoted 1 line
Quoted 1 line
Quoted 1 line
dirsplit('/usr/bin/')
('/usr/bin', '')
Quoted 1 line
Quoted 1 line
Quoted 1 line
dirsplit('/usr/bin')
('/usr', 'bin')
Quoted 1 line
Quoted 1 line
Quoted 1 line
dirsplit('/bin')
('/', 'bin')
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 49 lines · On Thursday 16 April 2015 15:11:44 Anders Darander wrote:
Quoted 16 lines · Paul Eggleton
Hi John,
Quoted 13 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:
After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially theSRC_URIlooked like this:SRC_URI= "git://git@<server+repo>;protocol=ssh;subpath=path/"Running the recipe resulted in
do_fetch(ordo_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.It's most likely a call to
bb.utils.prunedir(destdir), with destdir being set to '/'...Quoted 1 line
Very sorry about this :(
Yeah, too bad...
Quoted 4 lines
Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.
The issue with having '/' at the end ought to be solved by adding .rstrip('/') like:
----------------------------bitbake/lib/bb/fetch2/git.py +1
-1@@ -246,7 +246,7 @@ class Git(FetchMethod): subdir = ud.parm.get("subpath", "") if subdir != "": readpathspec = ":%s" % (subdir) - def_destsuffix = "%s/" % os.path.basename(subdir) + def_destsuffix = "%s/" % os.path.basename(subdir.rstrip('/')) else: readpathspec = ""
def_destsuffix= "git/"
---------------------------
Quoted 7 lines
Probably, the first issue could be solved by replacing subdir with
subdir.strip('/')on the line above that.Though I might be missing something...
I've only looked for a moment on the patch above, which at least seems to create correct path's when inspecting the output.
That looks like a reasonable solution to me, thanks.
FYI I've created the following bug to track this issue:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7620
I'll send a fix ASAP probably based on the above.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 24 lines · On Thursday 16 April 2015 07:35:40 Christopher Larson wrote:
On Thu, Apr 16, 2015 at 7:25 AM, Nicolas Dechesne <
Quoted 21 lines · [email protected]> wrote:
On Thu, Apr 16, 2015 at 3:11 PM, Anders Darander <[email protected]>
Quoted 4 lines · wrote:
Quoted 1 line
Quoted 1 line
Running the recipe resulted in
do_fetch(ordo_unpack) failing againand
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this
time was re-installing the machine.It's most likely a call to
bb.utils.prunedir(destdir), with destdir being set to '/'...ouch... this is a bit scary ;-)
maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..
Quoted 6 lines
Quoted 1 line
hopefully nobody does sudo bitbake ;-)
This is scary indeed. We definitely need to be more careful about calls to dirname/basename with paths with a trailing or leading / in general, and especially for removals, and just need to do more path sanitization/normalization probably..
I have sent an additional patch to the bitbake list to try to protect against future instances of this issue, at least for bb.utils.prunedir() and bb.utils.remove():
http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005664.html
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 2 lines · Paul Eggleton
I have sent an additional patch to the bitbake list to try to protect
against
Quoted 4 lines
future instances of this issue, at least for
bb.utils.prunedir()andbb.utils.remove():
http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005664.html
Good! That should give us some extra confidence for the future.
All the patches mentioned in this thread ought to be cherry-picked to all currently supported Poky releases.
Thanks and cheers,
Anders