Skip to contentSkip to conversation

Installing GDB or the saga of getting RPM/Zypp to run on JFFS2

DISCUSSION Holger Hans Peter Freyther 3 participants 7 messages Jul 1, 2011 – Jul 12, 2011

lore
Share & tools
lore
#0 lore raw

Hi,

I would like to share some more bits of my poky/yocto experience. We have started to move our software from the desktop to the device and we had some minor issues and wanted to use/install gdb. So far we are based on a -minimal image.

So the next step was to include the variable for the package manager in our custom image and then the fun starts.

a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...

b) libzypp didn't find gettext... (general ramblings against CMake and not providing information like autoconf), so somehow I had a mix of gettext 0.17 and 0.18 (no idea where I got a temp 0.18 build from, maybe due a rebase that i aborted or such).

$ bitbake -cclean virtual/gettext... fixed that for me.

For my colleague it somehow ended up not finding udev... but I have no other information about it, the issue fixed 'itself' by having another reason to rebuild udev.

c) then we found that RPM does not work on jffs2, or more precisely that Berkeley DB opens the db files with MAP_READ|MAP_WRITE which is not supported by jffs2. Googling reveals this[1], which didn't fix the problem for me. I know have:

$ cat /etc/rpm/macros
%__dbi_txn      create lock log txn auto_commit nommap private.

The key here is 'private', the NO_MMAP is only for readonly data (a nice red herring). I found an explanation for private here[2], I have also tested this with 'shared' (which maps to DB_SYSTEM_MEM).

Now I wonder how to fix that, should we add a /etc/rpm/macros when jffs2 is in the filesystem list? which mode should one use by default? for my target audience private is an acceptable option. Using ubifs is not something I want to try right now.

d) we also had a very weird issue of packages-split/libgudev.lock being there and packaging failing, it might be due the OE tree moving and bblayers.conf pointing to another tree.

e) gdb 7.2 on ARM (but should apply to others too) Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] 'import site' failed; use -v for traceback

Traceback (most recent call last):
File "<string>", line 32, in <module>
ImportError: No module named os.path
...

Quoted 2 lines

b
mmap

Function "mmap" not defined.
Make breakpoint pending on future shared library load? (y or [n]) \n ...[answered N; input not from terminal]

I assume that GDB needs
1.) an RDEPENDS on libthread_db
2.) besides libpython it needs some minimal python support, I am not
sure if libpython should have this RDEPENDS
3.) either there is a GLIBC with isatty... or I am missing some kernel
option... or something weird in libreadline/ncurses (e.g. missing terminfo) but it would be nice to be able to answer questions..

f) The size of zypper
libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?

While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...
- oprofile installs the data for all platforms, would be nice if we could only install the files that are needed by the given platform.

Okay this is a lot, but it all started from the 'I would like to install GDB' task. Feel free to start new top level threads for the separate issues or point me to the bugzilla and I can create tickets.

holger

[1] http://elinux.org/RPM_jffs2_issue
[2] http://www.mathematik.uni-ulm.de/help/BerkeleyDB/ref/env/region.html

#1 lore raw

Op 1 jul 2011, om 14:48 heeft Holger Hans Peter Freyther het volgende geschreven:

Quoted 4 lines

While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...

The udev in meta-oe should have the above 2 issues address and is a lot faster as well

#2 lore raw
In reply to Holger Hans Peter Freyther
Quoted 25 lines · Holger Hans Peter Freyther

Hi,

I would like to share some more bits of my poky/yocto experience. We have started to move our software from the desktop to the device and we had some minor issues and wanted to use/install gdb. So far we are based on a -minimal image.

So the next step was to include the variable for the package manager in our custom image and then the fun starts.

a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...

b) libzypp didn't find gettext... (general ramblings against CMake and not providing information like autoconf), so somehow I had a mix of gettext 0.17 and 0.18 (no idea where I got a temp 0.18 build from, maybe due a rebase that i aborted or such).

$ bitbake -cclean virtual/gettext... fixed that for me.

For my colleague it somehow ended up not finding udev... but I have no other information about it, the issue fixed 'itself' by having another reason to rebuild udev.

Quoted 7 lines

c) then we found that RPM does not work on jffs2, or more precisely that Berkeley DB opens the db files with MAP_READ|MAP_WRITE which is not supported by jffs2. Googling reveals this[1], which didn't fix the problem for me. I know have:

$ cat /etc/rpm/macros
%__dbi_txn      create lock log txn auto_commit nommap private.

There is also a database configuration file:

/usr/lib/rpm/DB_CONFIG

This contains database specific settings such as the memory pool and similar.

Note, I've never tried RPM on a jffs2 before. I'd not expected someone to use it there either.. ;) ipk is likely more appropriate simply due to cached file sizes and such.

See bug http://bugzilla.yoctoproject.org/show_bug.cgi?id=1174

RPM, as currently configured, ends up gobbling up a lot of disk space. We're working on a solution to this (changing the values in DB_CONFIG).. but it hasn't yet been implemented.

Quoted 8 lines

The key here is 'private', the NO_MMAP is only for readonly data (a nice red herring). I found an explanation for private here[2], I have also tested this with 'shared' (which maps to DB_SYSTEM_MEM).

Now I wonder how to fix that, should we add a /etc/rpm/macros when jffs2 is in the filesystem list? which mode should one use by default? for my target audience private is an acceptable option. Using ubifs is not something I want to try right now.

macro files are being loaded from:

%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros

The key above is the /etc/rpm/macros.*

I would recommend a new file be generated called /etc/rpm/macros.jffs2 that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)

Quoted 17 lines

d) we also had a very weird issue of packages-split/libgudev.lock being there and packaging failing, it might be due the OE tree moving and bblayers.conf pointing to another tree.

e) gdb 7.2 on ARM (but should apply to others too) Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] 'import site' failed; use -v for traceback

Traceback (most recent call last):
File "<string>", line 32, in <module>
ImportError: No module named os.path
...

Quoted 2 lines

b
mmap

Quoted 7 lines

Function "mmap" not defined.
Make breakpoint pending on future shared library load? (y or [n]) \n ...[answered N; input not from terminal]

I assume that GDB needs
1.) an RDEPENDS on libthread_db

libpthread_db is only needed if you are debugging more then one thread.

Quoted 2 lines

2.) besides libpython it needs some minimal python support, I am not
sure if libpython should have this RDEPENDS

GDB includes some python scripts to make it easier to walk various structures and similar. I don't remember off hand where these get placed, but you should see in your FS something of the format gdb...py, if you remove these it'll no longer load the python helpers.

Quoted 8 lines

3.) either there is a GLIBC with isatty... or I am missing some kernel
option... or something weird in libreadline/ncurses (e.g. missing terminfo) but it would be nice to be able to answer questions..

f) The size of zypper
libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?

zypper is somewhat of a pig. Again this wasn't really designed for really small systems.

I've used smart in the past, it's worth looking into... but it's been a lack of time. (Plus smart, at least used to, require python.. which itself is a pig, if you don't already need python.)

Quoted 4 lines

While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...

These should definitely be fixed.

Quoted 8 lines

- oprofile installs the data for all platforms, would be nice if we could only install the files that are needed by the given platform.

Okay this is a lot, but it all started from the 'I would like to install GDB' task. Feel free to start new top level threads for the separate issues or point me to the bugzilla and I can create tickets.

Most of my debugging so far with GDB on the target has been on ext3 and "large" filesystems. Alternatively I've run gdbserver on fairly small systems -- no RPM or .debug on the target.. the .debug stuff was on the host side.

--Mark

Quoted 11 lines

holger

[1] http://elinux.org/RPM_jffs2_issue
[2] http://www.mathematik.uni-ulm.de/help/BerkeleyDB/ref/env/region.html

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#3 lore raw
Quoted 11 lines · Mark Hatle
Quoted 4 lines

f) The size of zypper

Quoted 3 lines

libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?

zypper is somewhat of a pig. Again this wasn't really designed for really small systems.

I've used smart in the past, it's worth looking into... but it's been a lack of time. (Plus smart, at least used to, require python.. which itself is a pig, if you don't already need python.)

Hi again,

I looked a bit around and I found:
- zypper
- SMART
- yum
- apt-rpm

zypper/libzypp:
- I tried to produce a static libzypp and hoped the compiler would make it small, but I ended up in unresolved symbols in zypper and didn't feel like continuing (and using -Wl,--begin-group.. )

SMART:
- Oh, I assume that the python installation needed will bringt it to the ballpark of libzypp/zypper

yum:
- It is slow on my laptop, it will not fly on a 300Mhz device..

apt-rpm:
- I stumbled over a port for rpm 5.x (rpm is such a mess), it compiles (after patching out a json dependency, for a library using Scons, highlighting broken scons support).
- Even with things in the package we don't need the installed size is < 2MB
- genbasedir requires our RPMs to be in RPMS.all/, RPMS.armv5te/ format... and it is segfaulting.
- I have attached my current bb file (against my companies bblayer)

anyone wants to jump in with trying to get apt-rpm to replace zypper? The net-saving will be probably around 5-6MB...

meta-sysmocom/recipes-system/apt-rpm/apt-rpm_0.5.bb new file +22 -0
new file mode 100644
index 0000000..1681104
@@ -0,0 +1,22 @@
+DESCRIPTION = "apt-rpm for rpm based distros"
+HOMEPAGE = "http://gitorious.org/rpm5distro/apt-rpm"
+SRC_URI = "git://gitorious.org/rpm5distro/apt-rpm.git;protocol=git \
+	   file://hacks.patch"
+SRCREV = "2a6bd7a847e9a7ad269c637a1b673dafb71d26dd"
+LICENSE = "GPL"
+LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3"
+
+S = "${WORKDIR}/git"
+
+DEPENDS = "rpm"
+DEPENDS_virtclass-native = "rpm-native"
+PR = "r1"
+
+EXTRA_OECONF += " --disable-scripts"
+
+FILES_${PN} += "${libdir}/apt/methods/*"
+FILES_${PN}-dbg += "${libdir}/apt/methods/.debug"
+
+inherit autotools gettext
+
+BBCLASSEXTEND = "native"
meta-sysmocom/recipes-system/apt-rpm/files/hacks.patch new file +98 -0

98 lines not on this page · raw message

#4 lore raw
In reply to Mark Hatle
Quoted 11 lines · Mark Hatle

macro files are being loaded from:

%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros

The key above is the /etc/rpm/macros.*

I would recommend a new file be generated called /etc/rpm/macros.jffs2 that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)

What do you think about something like the change below? Alternatively one could write a post-inst script that checks if one is on jffs2 and then creates the config file.

meta/classes/rootfs_rpm.bbclass +6 -0
index 3a11858..70459e5 100644
@@ -139,6 +139,14 @@ EOF
        install -d ${IMAGE_ROOTFS}/${sysconfdir}
        echo ${BUILDNAME} > ${IMAGE_ROOTFS}/${sysconfdir}/version
 
+
+       # check if there is a jffs2 install, it requires a workaround
+       # due lacking support for MMAP read/write.
+       (echo "${IMAGE_FSTYPES}" | grep "jffs2" > /dev/null)
+       if [ $? == 0 ]; then
+               echo "%__dbi_txn      create lock log txn auto_commit nommap

private" > ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2
+ fi
+

        ${RPM_POSTPROCESS_COMMANDS}
        ${ROOTFS_POSTPROCESS_COMMAND}

@@ -164,6 +172,7 @@ EOF
}

remove_packaging_data_files() {
+ rm -rf ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2

        rm -rf ${IMAGE_ROOTFS}${rpmlibdir}
        rm -rf ${IMAGE_ROOTFS}${opkglibdir}

}

#5 lore raw
Quoted 9 lines · Holger Hans Peter Freyther
Quoted 7 lines · Mark Hatle

macro files are being loaded from:

%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros

The key above is the /etc/rpm/macros.*

Quoted 8 lines
Quoted 4 lines

I would recommend a new file be generated called /etc/rpm/macros.jffs2 that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)

What do you think about something like the change below? Alternatively one could write a post-inst script that checks if one is on jffs2 and then creates the config file.

Sorry for the late reply. I'm back from vacation now.

The below is fine with me. I suggest it be submitted as a patch to oe-core. I'll be happy to ack it.

--Mark

Quoted 33 lines
meta/classes/rootfs_rpm.bbclass +6 -0
index 3a11858..70459e5 100644
@@ -139,6 +139,14 @@ EOF
        install -d ${IMAGE_ROOTFS}/${sysconfdir}
        echo ${BUILDNAME} > ${IMAGE_ROOTFS}/${sysconfdir}/version
 
+
+       # check if there is a jffs2 install, it requires a workaround
+       # due lacking support for MMAP read/write.
+       (echo "${IMAGE_FSTYPES}" | grep "jffs2" > /dev/null)
+       if [ $? == 0 ]; then
+               echo "%__dbi_txn      create lock log txn auto_commit nommap

private" > ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2

meta/classes/rootfs_rpm.bbclass +3 -0
+       fi
+
        ${RPM_POSTPROCESS_COMMANDS}
        ${ROOTFS_POSTPROCESS_COMMAND}
@@ -164,6 +172,7 @@ EOF
 }
 
 remove_packaging_data_files() {
+       rm -rf ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2
        rm -rf ${IMAGE_ROOTFS}${rpmlibdir}
        rm -rf ${IMAGE_ROOTFS}${opkglibdir}
 }

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

#6 lore raw
In reply to Holger Hans Peter Freyther
Quoted 6 lines · Holger Hans Peter Freyther

Hi,

a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...

Okay, I am doing an oe-core build using this patch right now. Any comments?

holger

From 89e38923bdc01f2ef3f5ea81e5fd435e416b2ba6 Mon Sep 17 00:00:00 2001
From: Holger Hans Peter Freyther <[email protected]>
Date: Fri, 1 Jul 2011 22:12:52 +0800
Subject: [PATCH] misc: Replace os.chdir with bb.utils.checked_chdir

The Python os.chdir does not return the status, in some rare condition it seems possible that a Keyboard interrupt can interrupt the chdir and continue with the execution from the wrong directory. Introduce the bb.utils.checked_chdir to verify the chdir operation.

The users in bitbake of chdir are a bit sloppy, it is possible that someone is using a relative directory, uses a trailing slash or double slashes (e.g. license-destdir//).

---
 lib/bb/build.py           |    4 ++--
 lib/bb/cache.py           |    4 ++--
 lib/bb/fetch/bzr.py       |    6 +++---
 lib/bb/fetch/cvs.py       |   10 +++++-----
 lib/bb/fetch/git.py       |   18 +++++++++---------
 lib/bb/fetch/hg.py        |    8 ++++----
 lib/bb/fetch/osc.py       |    6 +++---
 lib/bb/fetch/perforce.py  |    2 +-
 lib/bb/fetch/repo.py      |    4 ++--
 lib/bb/fetch/svk.py       |    4 ++--
 lib/bb/fetch/svn.py       |    6 +++---
 lib/bb/fetch2/__init__.py |    6 +++---
 lib/bb/fetch2/bzr.py      |    6 +++---
 lib/bb/fetch2/cvs.py      |   10 +++++-----
 lib/bb/fetch2/git.py      |   14 +++++++-------
 lib/bb/fetch2/hg.py       |    8 ++++----
 lib/bb/fetch2/osc.py      |    6 +++---
 lib/bb/fetch2/perforce.py |    2 +-
 lib/bb/fetch2/repo.py     |    4 ++--
 lib/bb/fetch2/svk.py      |    4 ++--
 lib/bb/fetch2/svn.py      |    6 +++---
 lib/bb/pysh/builtin.py    |    4 ++--
 lib/bb/pysh/interp.py     |    4 ++--
 lib/bb/utils.py           |   17 +++++++++++++++++
 24 files changed, 90 insertions(+), 73 deletions(-)
lib/bb/build.py +2 -2
index 1c73ae2..0980e7a 100644
@@ -191,7 +191,7 @@ def exec_func_python(func, d, runfile, cwd=None):
             olddir = os.getcwd()
         except OSError:
             olddir = None
-        os.chdir(cwd)
+        bb.utils.checked_chdir(cwd)
 
     try:
         comp = utils.better_compile(code, func, bbfile)
@@ -204,7 +204,7 @@ def exec_func_python(func, d, runfile, cwd=None):
     finally:
         if cwd and olddir:
             try:
-                os.chdir(olddir)
+                bb.utils.checked_chdir(olddir)
             except OSError:
                 pass
 
lib/bb/cache.py +2 -2
index 99d7395..81d24ab 100644
@@ -657,11 +657,11 @@ class Cache(object):
                 data.setVar('__BBAPPEND', " ".join(appends), bb_data)
             bb_data = parse.handle(bbfile, bb_data)
             if chdir_back:
-                os.chdir(oldpath)
+                bb.utils.checked_chdir(oldpath)
             return bb_data
         except:
             if chdir_back:
-                os.chdir(oldpath)
+                bb.utils.checked_chdir(oldpath)
             raise
 
 
lib/bb/fetch/bzr.py +3 -3
index 85a9294..9774c65 100644
@@ -88,18 +88,18 @@ class Bzr(Fetch):
         if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK):
             bzrcmd = self._buildbzrcommand(ud, d, "update")
             logger.debug(1, "BZR Update %s", loc)
-            os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
+            bb.utils.checked_chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
             runfetchcmd(bzrcmd, d)
         else:
             bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
             bzrcmd = self._buildbzrcommand(ud, d, "fetch")
             logger.debug(1, "BZR Checkout %s", loc)
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", bzrcmd)
             runfetchcmd(bzrcmd, d)
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
lib/bb/fetch/cvs.py +5 -5
index 64450af..9f9f2fd 100644
@@ -132,13 +132,13 @@ class Cvs(Fetch):
         if os.access(os.path.join(moddir, 'CVS'), os.R_OK):
             logger.info("Update " + loc)
             # update sources there
-            os.chdir(moddir)
+            bb.utils.checked_chdir(moddir)
             myret = os.system(cvsupdatecmd)
         else:
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(pkgdir)
-            os.chdir(pkgdir)
+            bb.utils.checked_chdir(pkgdir)
             logger.debug(1, "Running %s", cvscmd)
             myret = os.system(cvscmd)
 
@@ -157,11 +157,11 @@ class Cvs(Fetch):
 
         # tar them up to a defined filename
         if 'fullpath' in ud.parm:
-            os.chdir(pkgdir)
+            bb.utils.checked_chdir(pkgdir)
             myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir))
         else:
-            os.chdir(moddir)
-            os.chdir('..')
+            bb.utils.checked_chdir(moddir)
+            bb.utils.checked_chdir('..')
             myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir)))
 
         if myret != 0:
lib/bb/fetch/git.py +9 -9
index 49c1cfe..0c1f170 100644
@@ -132,14 +132,14 @@ class Git(Fetch):
         # If the checkout doesn't exist and the mirror tarball does, extract it
         if not os.path.exists(ud.clonedir) and os.path.exists(repofile):
             bb.utils.mkdirhier(ud.clonedir)
-            os.chdir(ud.clonedir)
+            bb.utils.checked_chdir(ud.clonedir)
             runfetchcmd("tar -xzf %s" % (repofile), d)
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
             runfetchcmd("%s clone -n %s://%s%s%s %s" % (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir), d)
 
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         # Update the checkout if needed
         if not self._contains_ref(ud.tag, d) or 'fullclone' in ud.parm:
             # Remove all but the .git directory
@@ -153,7 +153,7 @@ class Git(Fetch):
             runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
 
         # Generate a mirror tarball if needed
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         mirror_tarballs = data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True)
         if mirror_tarballs != "0" or 'fullclone' in ud.parm:
             logger.info("Creating tarball of git repository")
@@ -185,19 +185,19 @@ class Git(Fetch):
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
             runfetchcmd("%s clone -n %s %s" % (ud.basecmd, ud.clonedir, coprefix), d)
-            os.chdir(coprefix)
+            bb.utils.checked_chdir(coprefix)
             runfetchcmd("%s checkout -q -f %s%s" % (ud.basecmd, ud.tag, readpathspec), d)
         else:
             bb.utils.mkdirhier(codir)
-            os.chdir(ud.clonedir)
+            bb.utils.checked_chdir(ud.clonedir)
             runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.tag, readpathspec), d)
             runfetchcmd("%s checkout-index -q -f --prefix=%s -a" % (ud.basecmd, coprefix), d)
 
-        os.chdir(codir)
+        bb.utils.checked_chdir(codir)
         logger.info("Creating tarball of git checkout")
         runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
 
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         bb.utils.prunedir(codir)
 
     def supports_srcrev(self):
@@ -327,12 +327,12 @@ class Git(Fetch):
                 return None
 
 
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         if not self._contains_ref(rev, d):
             self.go(None, ud, d)
 
         output = runfetchcmd("%s rev-list %s -- 2> /dev/null | wc -l" % (ud.basecmd, rev), d, quiet=True)
-        os.chdir(cwd)
+        bb.utils.checked_chdir(cwd)
 
         buildindex = "%s" % output.split()[0]
         logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
lib/bb/fetch/hg.py +4 -4
index 2b3aec5..84a94b5 100644
@@ -123,7 +123,7 @@ class Hg(Fetch):
             updatecmd = self._buildhgcommand(ud, d, "pull")
             logger.info("Update " + loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", updatecmd)
             runfetchcmd(updatecmd, d)
 
@@ -132,14 +132,14 @@ class Hg(Fetch):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", fetchcmd)
             runfetchcmd(fetchcmd, d)
 
         # Even when we clone (fetch), we still need to update as hg's clone
         # won't checkout the specified revision if its on a branch
         updatecmd = self._buildhgcommand(ud, d, "update")
-        os.chdir(ud.moddir)
+        bb.utils.checked_chdir(ud.moddir)
         logger.debug(1, "Running %s", updatecmd)
         runfetchcmd(updatecmd, d)
 
@@ -149,7 +149,7 @@ class Hg(Fetch):
         else:
             tar_flags = "--exclude '.hg' --exclude '.hgrags'"
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
         try:
             runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d)
         except:
lib/bb/fetch/osc.py +3 -3
index 32237b9..3a69846 100644
@@ -91,7 +91,7 @@ class Osc(Fetch):
             oscupdatecmd = self._buildosccommand(ud, d, "update")
             logger.info("Update "+ loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", oscupdatecmd)
             runfetchcmd(oscupdatecmd, d)
         else:
@@ -99,11 +99,11 @@ class Osc(Fetch):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", oscfetchcmd)
             runfetchcmd(oscfetchcmd, d)
 
-        os.chdir(os.path.join(ud.pkgdir + ud.path))
+        bb.utils.checked_chdir(os.path.join(ud.pkgdir + ud.path))
         # tar them up to a defined filename
         try:
             runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
lib/bb/fetch/perforce.py +1 -1
index e933d27..b1e79ce 100644
@@ -168,7 +168,7 @@ class Perforce(Fetch):
             cset = Perforce.getcset(d, depot, host, user, pswd, parm)
             depot = "%s@%s" % (depot, cset)
 
-        os.chdir(tmpfile)
+        bb.utils.checked_chdir(tmpfile)
         logger.info("Fetch " + loc)
         logger.info("%s%s files %s", p4cmd, p4opt, depot)
         p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
lib/bb/fetch/repo.py +2 -2
index 512fffb..cda36bf 100644
@@ -72,12 +72,12 @@ class Repo(Fetch):
             username = ""
 
         bb.utils.mkdirhier(os.path.join(codir, "repo"))
-        os.chdir(os.path.join(codir, "repo"))
+        bb.utils.checked_chdir(os.path.join(codir, "repo"))
         if not os.path.exists(os.path.join(codir, "repo", ".repo")):
             runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d)
 
         runfetchcmd("repo sync", d)
-        os.chdir(codir)
+        bb.utils.checked_chdir(codir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
lib/bb/fetch/svk.py +2 -2
index dc818d2..d67ae16 100644
@@ -80,7 +80,7 @@ class Svk(Fetch):
             raise FetchError(ud.module)
 
         # check out sources there
-        os.chdir(tmpfile)
+        bb.utils.checked_chdir(tmpfile)
         logger.info("Fetch " + loc)
         logger.debug(1, "Running %s", svkcmd)
         myret = os.system(svkcmd)
@@ -91,7 +91,7 @@ class Svk(Fetch):
                 pass
             raise FetchError(ud.module)
 
-        os.chdir(os.path.join(tmpfile, os.path.dirname(ud.module)))
+        bb.utils.checked_chdir(os.path.join(tmpfile, os.path.dirname(ud.module)))
         # tar them up to a defined filename
         myret = os.system("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.module)))
         if myret != 0:
lib/bb/fetch/svn.py +3 -3
index 6c2a118..25c13a4 100644
@@ -139,7 +139,7 @@ class Svn(Fetch):
             svnupdatecmd = self._buildsvncommand(ud, d, "update")
             logger.info("Update " + loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", svnupdatecmd)
             runfetchcmd(svnupdatecmd, d)
         else:
@@ -147,7 +147,7 @@ class Svn(Fetch):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", svnfetchcmd)
             runfetchcmd(svnfetchcmd, d)
 
@@ -157,7 +157,7 @@ class Svn(Fetch):
         else:
             tar_flags = "--exclude '.svn'"
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
         # tar them up to a defined filename
         try:
             runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d)
lib/bb/fetch2/__init__.py +3 -3
index e9a64c5..e325f6e 100644
@@ -739,17 +739,17 @@ class FetchMethod(object):
 
         # Change to subdir before executing command
         save_cwd = os.getcwd();
-        os.chdir(rootdir)
+        bb.utils.checked_chdir(rootdir)
         if 'subdir' in urldata.parm:
             newdir = ("%s/%s" % (rootdir, urldata.parm.get('subdir')))
             bb.utils.mkdirhier(newdir)
-            os.chdir(newdir)
+            bb.utils.checked_chdir(newdir)
 
         cmd = "PATH=\"%s\" %s" % (bb.data.getVar('PATH', data, True), cmd)
         bb.note("Unpacking %s to %s/" % (file, os.getcwd()))
         ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True)
 
-        os.chdir(save_cwd)
+        bb.utils.checked_chdir(save_cwd)
 
         if ret != 0:
             raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url)
lib/bb/fetch2/bzr.py +3 -3
index 0d10eb4..404d004 100644
@@ -88,7 +88,7 @@ class Bzr(FetchMethod):
             bzrcmd = self._buildbzrcommand(ud, d, "update")
             logger.debug(1, "BZR Update %s", loc)
             bb.fetch2.check_network_access(d, bzrcmd, ud.url)
-            os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
+            bb.utils.checked_chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
             runfetchcmd(bzrcmd, d)
         else:
             bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
@@ -96,11 +96,11 @@ class Bzr(FetchMethod):
             bb.fetch2.check_network_access(d, bzrcmd, ud.url)
             logger.debug(1, "BZR Checkout %s", loc)
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", bzrcmd)
             runfetchcmd(bzrcmd, d)
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
lib/bb/fetch2/cvs.py +5 -5
index a111467..0710b32 100644
@@ -134,13 +134,13 @@ class Cvs(FetchMethod):
             logger.info("Update " + loc)
             bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url)
             # update sources there
-            os.chdir(moddir)
+            bb.utils.checked_chdir(moddir)
             cmd = cvsupdatecmd
         else:
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(pkgdir)
-            os.chdir(pkgdir)
+            bb.utils.checked_chdir(pkgdir)
             logger.debug(1, "Running %s", cvscmd)
             bb.fetch2.check_network_access(d, cvscmd, ud.url)
             cmd = cvscmd
@@ -158,11 +158,11 @@ class Cvs(FetchMethod):
 
         # tar them up to a defined filename
         if 'fullpath' in ud.parm:
-            os.chdir(pkgdir)
+            bb.utils.checked_chdir(pkgdir)
             cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir)
         else:
-            os.chdir(moddir)
-            os.chdir('..')
+            bb.utils.checked_chdir(moddir)
+            bb.utils.checked_chdir('..')
             cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir))
 
         runfetchcmd(cmd, d, cleanup = [ud.localpath])
lib/bb/fetch2/git.py +7 -7
index 534c87d..d4f841b 100644
@@ -135,7 +135,7 @@ class Git(FetchMethod):
     def need_update(self, u, ud, d):
         if not os.path.exists(ud.clonedir):
             return True
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         for name in ud.names:
             if not self._contains_ref(ud.revisions[name], d):
                 return True
@@ -165,7 +165,7 @@ class Git(FetchMethod):
         # If the checkout doesn't exist and the mirror tarball does, extract it
         if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
             bb.utils.mkdirhier(ud.clonedir)
-            os.chdir(ud.clonedir)
+            bb.utils.checked_chdir(ud.clonedir)
             runfetchcmd("tar -xzf %s" % (ud.fullmirror), d)
 
         # If the repo still doesn't exist, fallback to cloning it
@@ -175,7 +175,7 @@ class Git(FetchMethod):
             bb.fetch2.check_network_access(d, clone_cmd)
             runfetchcmd(clone_cmd, d)
 
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         # Update the checkout if needed
         needupdate = False
         for name in ud.names:
@@ -199,7 +199,7 @@ class Git(FetchMethod):
     def build_mirror_data(self, url, ud, d):
         # Generate a mirror tarball if needed
         if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
-            os.chdir(ud.clonedir)
+            bb.utils.checked_chdir(ud.clonedir)
             logger.info("Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
 
@@ -219,7 +219,7 @@ class Git(FetchMethod):
 
         runfetchcmd("git clone -s -n %s %s" % (ud.clonedir, destdir), d)
         if not ud.nocheckout:
-            os.chdir(destdir)
+            bb.utils.checked_chdir(destdir)
             if subdir != "":
                 runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d)
                 runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
@@ -289,12 +289,12 @@ class Git(FetchMethod):
                 return None
 
 
-        os.chdir(ud.clonedir)
+        bb.utils.checked_chdir(ud.clonedir)
         if not self._contains_ref(rev, d):
             self.download(None, ud, d)
 
         output = runfetchcmd("%s rev-list %s -- 2> /dev/null | wc -l" % (ud.basecmd, rev), d, quiet=True)
-        os.chdir(cwd)
+        bb.utils.checked_chdir(cwd)
 
         buildindex = "%s" % output.split()[0]
         logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
lib/bb/fetch2/hg.py +4 -4
index 793831a..f8a3b1e 100644
@@ -124,7 +124,7 @@ class Hg(FetchMethod):
             updatecmd = self._buildhgcommand(ud, d, "pull")
             logger.info("Update " + loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", updatecmd)
             bb.fetch2.check_network_access(d, updatecmd, ud.url)
             runfetchcmd(updatecmd, d)
@@ -134,7 +134,7 @@ class Hg(FetchMethod):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", fetchcmd)
             bb.fetch2.check_network_access(d, fetchcmd, ud.url)
             runfetchcmd(fetchcmd, d)
@@ -142,7 +142,7 @@ class Hg(FetchMethod):
         # Even when we clone (fetch), we still need to update as hg's clone
         # won't checkout the specified revision if its on a branch
         updatecmd = self._buildhgcommand(ud, d, "update")
-        os.chdir(ud.moddir)
+        bb.utils.checked_chdir(ud.moddir)
         logger.debug(1, "Running %s", updatecmd)
         runfetchcmd(updatecmd, d)
 
@@ -152,7 +152,7 @@ class Hg(FetchMethod):
         else:
             tar_flags = "--exclude '.hg' --exclude '.hgrags'"
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
         runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath])
 
     def supports_srcrev(self):
lib/bb/fetch2/osc.py +3 -3
index a16a53e..1e22ca9 100644
@@ -88,7 +88,7 @@ class Osc(FetchMethod):
             oscupdatecmd = self._buildosccommand(ud, d, "update")
             logger.info("Update "+ loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", oscupdatecmd)
             bb.fetch2.check_network_access(d, oscupdatecmd, ud.url)
             runfetchcmd(oscupdatecmd, d)
@@ -97,12 +97,12 @@ class Osc(FetchMethod):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", oscfetchcmd)
             bb.fetch2.check_network_access(d, oscfetchcmd, ud.url)
             runfetchcmd(oscfetchcmd, d)
 
-        os.chdir(os.path.join(ud.pkgdir + ud.path))
+        bb.utils.checked_chdir(os.path.join(ud.pkgdir + ud.path))
         # tar them up to a defined filename
         runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d, cleanup = [ud.localpath])
 
lib/bb/fetch2/perforce.py +1 -1
index cbdc848..bb13594 100644
@@ -165,7 +165,7 @@ class Perforce(FetchMethod):
             cset = Perforce.getcset(d, depot, host, user, pswd, parm)
             depot = "%s@%s" % (depot, cset)
 
-        os.chdir(tmpfile)
+        bb.utils.checked_chdir(tmpfile)
         logger.info("Fetch " + loc)
         logger.info("%s%s files %s", p4cmd, p4opt, depot)
         p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
lib/bb/fetch2/repo.py +2 -2
index 8300da8..e267917 100644
@@ -70,14 +70,14 @@ class Repo(FetchMethod):
             username = ""
 
         bb.utils.mkdirhier(os.path.join(codir, "repo"))
-        os.chdir(os.path.join(codir, "repo"))
+        bb.utils.checked_chdir(os.path.join(codir, "repo"))
         if not os.path.exists(os.path.join(codir, "repo", ".repo")):
             bb.fetch2.check_network_access(d, "repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
             runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d)
 
         bb.fetch2.check_network_access(d, "repo sync %s" % ud.url, ud.url)
         runfetchcmd("repo sync", d)
-        os.chdir(codir)
+        bb.utils.checked_chdir(codir)
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata == "keep":
lib/bb/fetch2/svk.py +2 -2
index 9d34abf..a2fb864 100644
@@ -84,12 +84,12 @@ class Svk(FetchMethod):
             raise FetchError("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.", loc)
 
         # check out sources there
-        os.chdir(tmpfile)
+        bb.utils.checked_chdir(tmpfile)
         logger.info("Fetch " + loc)
         logger.debug(1, "Running %s", svkcmd)
         runfetchcmd(svkcmd, d, cleanup = [tmpfile])
 
-        os.chdir(os.path.join(tmpfile, os.path.dirname(ud.module)))
+        bb.utils.checked_chdir(os.path.join(tmpfile, os.path.dirname(ud.module)))
         # tar them up to a defined filename
         runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.module)), d, cleanup = [ud.localpath])
 
lib/bb/fetch2/svn.py +3 -3
index 59d7ccb..0bc5203 100644
@@ -116,7 +116,7 @@ class Svn(FetchMethod):
             svnupdatecmd = self._buildsvncommand(ud, d, "update")
             logger.info("Update " + loc)
             # update sources there
-            os.chdir(ud.moddir)
+            bb.utils.checked_chdir(ud.moddir)
             logger.debug(1, "Running %s", svnupdatecmd)
             bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
             runfetchcmd(svnupdatecmd, d)
@@ -125,7 +125,7 @@ class Svn(FetchMethod):
             logger.info("Fetch " + loc)
             # check out sources there
             bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
+            bb.utils.checked_chdir(ud.pkgdir)
             logger.debug(1, "Running %s", svnfetchcmd)
             bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
             runfetchcmd(svnfetchcmd, d)
@@ -136,7 +136,7 @@ class Svn(FetchMethod):
         else:
             tar_flags = "--exclude '.svn'"
 
-        os.chdir(ud.pkgdir)
+        bb.utils.checked_chdir(ud.pkgdir)
         # tar them up to a defined filename
         runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath])
 
lib/bb/pysh/builtin.py +2 -2
index b748e4a..693becf 100644
@@ -581,11 +581,11 @@ def utility_sort(name, args, interp, env, stdin, stdout, stderr, debugflags):
     # Load all files lines
     curdir = os.getcwd()
     try:
-        os.chdir(env['PWD'])
+        bb.utils.checked_chdir(env['PWD'])
         for path in args:
             alllines += sort(path)
     finally:
-        os.chdir(curdir)
+        bb.utils.checked_chdir(curdir)
             
     alllines.sort()
     for line in alllines:
lib/bb/pysh/interp.py +2 -2
index 25d8c92..da3aa21 100644
@@ -1261,11 +1261,11 @@ class Interpreter:
             
         def pwd_glob(pattern):
             cwd = os.getcwd()
-            os.chdir(self._env['PWD'])
+            bb.utils.checked_chdir(self._env['PWD'])
             try:
                 return glob.glob(pattern) 
             finally:
-                os.chdir(cwd)    
+                bb.utils.checked_chdir(cwd)
             
         #TODO: check working directory issues here wrt relative patterns
         try:
lib/bb/utils.py +17 -0
index 82e5dc4..99d3d8a 100644
@@ -856,3 +856,20 @@ def to_boolean(string, default=None):
         return False
     else:
         raise ValueError("Invalid value for to_boolean: %s" % string)
+
+def checked_chdir(name):
+    """I change the directory and verify that it was changed, if
+    not I am raising an exception."""
+
+    # We are a bit sloppy with '/' or using a relative dir like
+    # .. in the fetchers. We resolve the new path here, remove a
+    # trailing slash and see if things match now.
+    old_dir = os.getcwd()
+    new_dir = os.path.join(old_dir, name)
+    new_dir = new_dir.replace('//', '/')
+    if new_dir[-1] == '/':
+        new_dir = new_dir[:-1]
+    os.chdir(new_dir)
+
+    if os.getcwd() != new_dir:
+        raise OSError('Failed to change to directory: \'%s\'. You are in \'%s\', resolved \'%s\'' % (new_dir, os.getcwd(), new_dir))
Signature · 1 line
1.7.4.1