Skip to contentSkip to conversation

Badly formatted subpath for git SRC_URI deletes portions of hard drive.

DISCUSSION John Ernberg 5 participants 8 messages Apr 16, 2015 – Apr 18, 2015

lore
Share & tools
lore
#0 lore raw

Hi

I was writing a bitbake recipe for a project hosted on a git-server. I applied subpath in order to limit the checkout to a specific directory.

When I ran the recipe it failed in do_fetch (or do_unpack) complaining about missing files. It had deleted the entire Yocto directory and selected directories from my home-dir like ssh-keys, my dir where I keep my code, and some configuration files for Thunderbird and my terminal.

Since I had no Yocto directory left I was unable to debug the recipe.

After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially the SRC_URI looked like this:
SRC_URI = "git://git@<server+repo>;protocol=ssh;subpath=path/"

Running the recipe resulted in do_fetch (or do_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

I was keeping bitbake in my home-dir both of these times.

The poky version I am using is Daisy.

Best regards // John Ernberg

#1 lore raw

Hi John,

Quoted 24 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:

I was writing a bitbake recipe for a project hosted on a git-server. I applied subpath in order to limit the checkout to a specific directory.

When I ran the recipe it failed in do_fetch (or do_unpack) complaining about missing files. It had deleted the entire Yocto directory and selected directories from my home-dir like ssh-keys, my dir where I keep my code, and some configuration files for Thunderbird and my terminal.

Since I had no Yocto directory left I was unable to debug the recipe.

After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially the SRC_URI looked like this:
SRC_URI = "git://git@<server+repo>;protocol=ssh;subpath=path/"

Running the recipe resulted in do_fetch (or do_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

Quoted 3 lines

I was keeping bitbake in my home-dir both of these times.

The poky version I am using is Daisy.

Very sorry about this :(

Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.

It may be worth noting in case it wasn't clear that when it is working correctly subpath= doesn't do a shallow clone, it still clones the full repository, it's just that you only get the specified subdirectory unpacked into the work directory after that.

- Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#2 lore raw
Quoted 16 lines · Paul Eggleton

Hi John,

Quoted 13 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:

After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially the SRC_URI looked like this:
SRC_URI = "git://git@<server+repo>;protocol=ssh;subpath=path/"

Running the recipe resulted in do_fetch (or do_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...

Quoted 1 line

Very sorry about this :(

Yeah, too bad...

Quoted 3 lines

Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.

The issue with having '/' at the end ought to be solved by adding .rstrip('/') like:
----------------------------

bitbake/lib/bb/fetch2/git.py +1 -1
index 44fc271..0fd9bee 100644
@@ -246,7 +246,7 @@ class Git(FetchMethod):
         subdir = ud.parm.get("subpath", "")
         if subdir != "":
             readpathspec = ":%s" % (subdir)
-            def_destsuffix = "%s/" % os.path.basename(subdir)
+            def_destsuffix = "%s/" %

os.path.basename(subdir.rstrip('/'))
else:
readpathspec = ""
def_destsuffix = "git/"
---------------------------

Probably, the first issue could be solved by replacing subdir with subdir.strip('/') on the line above that.

Though I might be missing something...

I've only looked for a moment on the patch above, which at least seems to create correct path's when inspecting the output.

Cheers,
Anders

Signature · 2 lines
Anders Darander
ChargeStorm AB / eStorm AB
#3 lore raw
Quoted 9 lines · Anders Darander
Quoted 1 line
Quoted 1 line

Running the recipe resulted in do_fetch (or do_unpack) failing again

and

Quoted 4 lines
Quoted 4 lines

complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...

ouch... this is a bit scary ;-)

maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..

hopefully nobody does sudo bitbake ;-)

#4 lore raw
Quoted 15 lines · Nicolas Dechesne
Quoted 9 lines · Anders Darander
Quoted 1 line
Quoted 1 line

Running the recipe resulted in do_fetch (or do_unpack) failing again

and

Quoted 4 lines
Quoted 4 lines

complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...

Quoted 7 lines

ouch... this is a bit scary ;-)

maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..

hopefully nobody does sudo bitbake ;-)

This is scary indeed. We definitely need to be more careful about calls to dirname/basename with paths with a trailing or leading / in general, and especially for removals, and just need to do more path sanitization/normalization probably..

Quoted 1 line
Quoted 1 line
Quoted 1 line

def dirsplit(p):

... return os.path.dirname(p), os.path.basename(p)
...

Quoted 1 line
Quoted 1 line
Quoted 1 line

dirsplit('/usr/bin/foo')

('/usr/bin', 'foo')

Quoted 1 line
Quoted 1 line
Quoted 1 line

dirsplit('/usr/bin/')

('/usr/bin', '')

Quoted 1 line
Quoted 1 line
Quoted 1 line

dirsplit('/usr/bin')

('/usr', 'bin')

Quoted 1 line
Quoted 1 line
Quoted 1 line

dirsplit('/bin')

('/', 'bin')

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#7 lore raw
In reply to Anders Darander
Quoted 49 lines · On Thursday 16 April 2015 15:11:44 Anders Darander wrote:
Quoted 16 lines · Paul Eggleton

Hi John,

Quoted 13 lines · On Thursday 16 April 2015 11:52:48 John Ernberg wrote:

After restoring everything I re-created the recipe, this time I also saved a copy of the recipe at a safe location.
At the end of the path given to subpath I had added a '/' as it was not specified what the format of the path should be. I figured I would see an error if the path was wrong, and technically I did. Essentially the SRC_URI looked like this:
SRC_URI = "git://git@<server+repo>;protocol=ssh;subpath=path/"

Running the recipe resulted in do_fetch (or do_unpack) failing again and complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this time was re-installing the machine.

It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...

Quoted 1 line

Very sorry about this :(

Yeah, too bad...

Quoted 4 lines

Looking at the code I can see at least two problems we need to fix relating to this issue. It seems that having '/' either at the start or the end of the value was unexpected and would trigger undesirable behaviour.

The issue with having '/' at the end ought to be solved by adding .rstrip('/') like:
----------------------------

bitbake/lib/bb/fetch2/git.py +1 -1
index 44fc271..0fd9bee 100644
@@ -246,7 +246,7 @@ class Git(FetchMethod):
         subdir = ud.parm.get("subpath", "")
         if subdir != "":
             readpathspec = ":%s" % (subdir)
-            def_destsuffix = "%s/" % os.path.basename(subdir)
+            def_destsuffix = "%s/" %
 os.path.basename(subdir.rstrip('/'))
         else:
             readpathspec = ""

def_destsuffix = "git/"
---------------------------

Quoted 7 lines

Probably, the first issue could be solved by replacing subdir with subdir.strip('/') on the line above that.

Though I might be missing something...

I've only looked for a moment on the patch above, which at least seems to create correct path's when inspecting the output.

That looks like a reasonable solution to me, thanks.

FYI I've created the following bug to track this issue:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=7620

I'll send a fix ASAP probably based on the above.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#5 lore raw
In reply to Christopher Larson
Quoted 24 lines · On Thursday 16 April 2015 07:35:40 Christopher Larson wrote:

On Thu, Apr 16, 2015 at 7:25 AM, Nicolas Dechesne <

Quoted 21 lines · [email protected]> wrote:

On Thu, Apr 16, 2015 at 3:11 PM, Anders Darander <[email protected]>

Quoted 4 lines · wrote:
Quoted 1 line
Quoted 1 line

Running the recipe resulted in do_fetch (or do_unpack) failing again

and

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

complaining about missing files. This time the deletion was on a much greater scale, it had deleted so much of my home-dir that my user account was rendered entirely useless. The fastest way to recover this
time was re-installing the machine.

It's most likely a call to bb.utils.prunedir(destdir), with destdir being set to '/'...

ouch... this is a bit scary ;-)

maybe we should assert the folder name is 'sane' when we do destructive operations like that. e.g. check that we are in TMPDIR, or TOPDIR..

Quoted 6 lines
Quoted 1 line

hopefully nobody does sudo bitbake ;-)

This is scary indeed. We definitely need to be more careful about calls to dirname/basename with paths with a trailing or leading / in general, and especially for removals, and just need to do more path sanitization/normalization probably..

I have sent an additional patch to the bitbake list to try to protect against future instances of this issue, at least for bb.utils.prunedir() and bb.utils.remove():

http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005664.html

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw
Quoted 2 lines · Paul Eggleton

I have sent an additional patch to the bitbake list to try to protect

against

Quoted 4 lines

future instances of this issue, at least for bb.utils.prunedir() and bb.utils.remove():

http://lists.openembedded.org/pipermail/bitbake-devel/2015-April/005664.html

Good! That should give us some extra confidence for the future.

All the patches mentioned in this thread ought to be cherry-picked to all currently supported Poky releases.

Thanks and cheers,
Anders