Installing GDB or the saga of getting RPM/Zypp to run on JFFS2
Hi,
I would like to share some more bits of my poky/yocto experience. We have started to move our software from the desktop to the device and we had some minor issues and wanted to use/install gdb. So far we are based on a -minimal image.
So the next step was to include the variable for the package manager in our custom image and then the fun starts.
a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...
b) libzypp didn't find gettext... (general ramblings against CMake and not providing information like autoconf), so somehow I had a mix of gettext 0.17 and 0.18 (no idea where I got a temp 0.18 build from, maybe due a rebase that i aborted or such).
$ bitbake -cclean virtual/gettext... fixed that for me.
For my colleague it somehow ended up not finding udev... but I have no other information about it, the issue fixed 'itself' by having another reason to rebuild udev.
c) then we found that RPM does not work on jffs2, or more precisely that Berkeley DB opens the db files with MAP_READ|MAP_WRITE which is not supported by jffs2. Googling reveals this[1], which didn't fix the problem for me. I know have:
$ cat /etc/rpm/macros %__dbi_txn create lock log txn auto_commit nommap private.
The key here is 'private', the NO_MMAP is only for readonly data (a nice red herring). I found an explanation for private here[2], I have also tested this with 'shared' (which maps to DB_SYSTEM_MEM).
Now I wonder how to fix that, should we add a /etc/rpm/macros when jffs2 is in the filesystem list? which mode should one use by default? for my target audience private is an acceptable option. Using ubifs is not something I want to try right now.
d) we also had a very weird issue of packages-split/libgudev.lock being there and packaging failing, it might be due the OE tree moving and bblayers.conf pointing to another tree.
e) gdb 7.2 on ARM (but should apply to others too) Could not find platform independent libraries <prefix> Could not find platform dependent libraries <exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] 'import site' failed; use -v for traceback
Traceback (most recent call last):
File "<string>", line 32, in <module>
ImportError: No module named os.path
...
Quoted 2 lines
b
mmap
Function "mmap" not defined.
Make breakpoint pending on future shared library load? (y or [n]) \n ...[answered N; input not from terminal]
I assume that GDB needs
1.) an RDEPENDS on libthread_db
2.) besides libpython it needs some minimal python support, I am not
sure if libpython should have this RDEPENDS
3.) either there is a GLIBC with isatty... or I am missing some kernel
option... or something weird in libreadline/ncurses (e.g. missing terminfo) but it would be nice to be able to answer questions..
f) The size of zypper
libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?
While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...
- oprofile installs the data for all platforms, would be nice if we could only install the files that are needed by the given platform.
Okay this is a lot, but it all started from the 'I would like to install GDB' task. Feel free to start new top level threads for the separate issues or point me to the bugzilla and I can create tickets.
holger
[1] http://elinux.org/RPM_jffs2_issue
[2] http://www.mathematik.uni-ulm.de/help/BerkeleyDB/ref/env/region.html
Op 1 jul 2011, om 14:48 heeft Holger Hans Peter Freyther het volgende geschreven:
Quoted 4 lines
While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...
The udev in meta-oe should have the above 2 issues address and is a lot faster as well
Quoted 25 lines · Holger Hans Peter Freyther
Hi,
I would like to share some more bits of my poky/yocto experience. We have started to move our software from the desktop to the device and we had some minor issues and wanted to use/install gdb. So far we are based on a -minimal image.
So the next step was to include the variable for the package manager in our custom image and then the fun starts.
a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...
b) libzypp didn't find gettext... (general ramblings against CMake and not providing information like autoconf), so somehow I had a mix of gettext 0.17 and 0.18 (no idea where I got a temp 0.18 build from, maybe due a rebase that i aborted or such).
$ bitbake -cclean virtual/gettext... fixed that for me.For my colleague it somehow ended up not finding udev... but I have no other information about it, the issue fixed 'itself' by having another reason to rebuild udev.
Quoted 7 lines
c) then we found that RPM does not work on jffs2, or more precisely that Berkeley DB opens the db files with MAP_READ|MAP_WRITE which is not supported by jffs2. Googling reveals this[1], which didn't fix the problem for me. I know have:
$ cat /etc/rpm/macros %__dbi_txn create lock log txn auto_commit nommap private.
There is also a database configuration file:
/usr/lib/rpm/DB_CONFIG
This contains database specific settings such as the memory pool and similar.
Note, I've never tried RPM on a jffs2 before. I'd not expected someone to use it there either.. ;) ipk is likely more appropriate simply due to cached file sizes and such.
See bug http://bugzilla.yoctoproject.org/show_bug.cgi?id=1174
RPM, as currently configured, ends up gobbling up a lot of disk space. We're working on a solution to this (changing the values in DB_CONFIG).. but it hasn't yet been implemented.
Quoted 8 lines
The key here is 'private', the
NO_MMAPis only for readonly data (a nice red herring). I found an explanation for private here[2], I have also tested this with 'shared' (which maps toDB_SYSTEM_MEM).Now I wonder how to fix that, should we add a
/etc/rpm/macroswhen jffs2 is in the filesystem list? which mode should one use by default? for my target audience private is an acceptable option. Using ubifs is not something I want to try right now.
macro files are being loaded from:
%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros
The key above is the /etc/rpm/macros.*
I would recommend a new file be generated called /etc/rpm/macros.jffs2 that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)
Quoted 17 lines
d) we also had a very weird issue of packages-split/libgudev.lock being there and packaging failing, it might be due the OE tree moving and bblayers.conf pointing to another tree.
e) gdb 7.2 on ARM (but should apply to others too) Could not find platform independent libraries <prefix> Could not find platform dependent libraries <
exec_prefix> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] 'import site' failed; use -v for tracebackTraceback (most recent call last):
File "<string>", line 32, in <module>
ImportError: No module named os.path
...Quoted 2 lines
b
mmap
Quoted 7 lines
Function "mmap" not defined.
Make breakpoint pending on future shared library load? (y or [n]) \n ...[answered N; input not from terminal]I assume that GDB needs
1.) an RDEPENDS onlibthread_db
libpthread_db is only needed if you are debugging more then one thread.
Quoted 2 lines
2.) besides libpython it needs some minimal python support, I am not
sure if libpython should have this RDEPENDS
GDB includes some python scripts to make it easier to walk various structures and similar. I don't remember off hand where these get placed, but you should see in your FS something of the format gdb...py, if you remove these it'll no longer load the python helpers.
Quoted 8 lines
3.) either there is a GLIBC with isatty... or I am missing some kernel
option... or something weird in libreadline/ncurses (e.g. missing terminfo) but it would be nice to be able to answer questions..f) The size of zypper
libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?
zypper is somewhat of a pig. Again this wasn't really designed for really small systems.
I've used smart in the past, it's worth looking into... but it's been a lack of time. (Plus smart, at least used to, require python.. which itself is a pig, if you don't already need python.)
Quoted 4 lines
While looking into the size I also found that:
- udev drags in libgobject/libgthread due libgudev
- udev installs keymaps even when i have no keyboard in my machine flags...
These should definitely be fixed.
Quoted 8 lines
- oprofile installs the data for all platforms, would be nice if we could only install the files that are needed by the given platform.
Okay this is a lot, but it all started from the 'I would like to install GDB' task. Feel free to start new top level threads for the separate issues or point me to the bugzilla and I can create tickets.
Most of my debugging so far with GDB on the target has been on ext3 and "large" filesystems. Alternatively I've run gdbserver on fairly small systems -- no RPM or .debug on the target.. the .debug stuff was on the host side.
--Mark
Quoted 11 lines
holger
[1] http://elinux.org/RPM_jffs2_issue
[2] http://www.mathematik.uni-ulm.de/help/BerkeleyDB/ref/env/region.html_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Quoted 11 lines · Mark Hatle
Quoted 4 lines
f) The size of zypper
Quoted 3 lines
libzypp is a _huge_ library, did anyone look into shrinking it? maybe by making zypper link statically to it? Did anyone look into the alternatives like SMART?
zypper is somewhat of a pig. Again this wasn't really designed for really small systems.
I've used smart in the past, it's worth looking into... but it's been a lack of time. (Plus smart, at least used to, require python.. which itself is a pig, if you don't already need python.)
Hi again,
I looked a bit around and I found:
- zypper
- SMART
- yum
- apt-rpm
zypper/libzypp:
- I tried to produce a static libzypp and hoped the compiler would make it small, but I ended up in unresolved symbols in zypper and didn't feel like continuing (and using -Wl,--begin-group.. )
SMART:
- Oh, I assume that the python installation needed will bringt it to the ballpark of libzypp/zypper
yum:
- It is slow on my laptop, it will not fly on a 300Mhz device..
apt-rpm:
- I stumbled over a port for rpm 5.x (rpm is such a mess), it compiles (after patching out a json dependency, for a library using Scons, highlighting broken scons support).
- Even with things in the package we don't need the installed size is < 2MB
- genbasedir requires our RPMs to be in RPMS.all/, RPMS.armv5te/ format... and it is segfaulting.
- I have attached my current bb file (against my companies bblayer)
anyone wants to jump in with trying to get apt-rpm to replace zypper? The net-saving will be probably around 5-6MB...
meta-sysmocom/recipes-system/apt-rpm/apt-rpm_0.5.bb new file +22 -0
@@ -0,0 +1,22 @@ +DESCRIPTION = "apt-rpm for rpm based distros" +HOMEPAGE = "http://gitorious.org/rpm5distro/apt-rpm" +SRC_URI = "git://gitorious.org/rpm5distro/apt-rpm.git;protocol=git \ + file://hacks.patch" +SRCREV = "2a6bd7a847e9a7ad269c637a1b673dafb71d26dd" +LICENSE = "GPL" +LIC_FILES_CHKSUM = "file://COPYING.GPL;md5=0636e73ff0215e8d672dc4c32c317bb3" + +S = "${WORKDIR}/git" + +DEPENDS = "rpm" +DEPENDS_virtclass-native = "rpm-native" +PR = "r1" + +EXTRA_OECONF += " --disable-scripts" + +FILES_${PN} += "${libdir}/apt/methods/*" +FILES_${PN}-dbg += "${libdir}/apt/methods/.debug" + +inherit autotools gettext + +BBCLASSEXTEND = "native"
meta-sysmocom/recipes-system/apt-rpm/files/hacks.patch new file +98 -0
98 lines not on this page · raw message
Quoted 11 lines · Mark Hatle
macro files are being loaded from:
%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros
The key above is the
/etc/rpm/macros.*I would recommend a new file be generated called
/etc/rpm/macros.jffs2that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)
What do you think about something like the change below? Alternatively one could write a post-inst script that checks if one is on jffs2 and then creates the config file.
meta/classes/rootfs_rpm.bbclass +6 -0
@@ -139,6 +139,14 @@ EOF install -d ${IMAGE_ROOTFS}/${sysconfdir} echo ${BUILDNAME} > ${IMAGE_ROOTFS}/${sysconfdir}/version + + # check if there is a jffs2 install, it requires a workaround + # due lacking support for MMAP read/write. + (echo "${IMAGE_FSTYPES}" | grep "jffs2" > /dev/null) + if [ $? == 0 ]; then + echo "%__dbi_txn create lock log txn auto_commit nommap
private" > ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2
+ fi
+
${RPM_POSTPROCESS_COMMANDS}
${ROOTFS_POSTPROCESS_COMMAND}@@ -164,6 +172,7 @@ EOF
}
remove_packaging_data_files() {
+ rm -rf ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2
rm -rf ${IMAGE_ROOTFS}${rpmlibdir}
rm -rf ${IMAGE_ROOTFS}${opkglibdir}}
Quoted 9 lines · Holger Hans Peter Freyther
Quoted 7 lines · Mark Hatle
macro files are being loaded from:
%{_usrlibrpm}/macros:%{_usrlibrpm}/poky/macros:%{_usrlibrpm}/poky/%{_target}/macros:%{_etcrpm}/macros.*:%{_etcrpm}/macros:%{_etcrpm}/%{_target}/macros:~/.oerpmmacros
The key above is the
/etc/rpm/macros.*
Quoted 8 lines
Quoted 4 lines
I would recommend a new file be generated called
/etc/rpm/macros.jffs2that changes the setting as appropriate for that filesystem. (How it's placed into the filesystem I'm not sure. I think it all comes down to detecting we're building a jffs2 filesystem and doing it there. Perhaps in the rootfs_rpm.bbclass?)What do you think about something like the change below? Alternatively one could write a post-inst script that checks if one is on jffs2 and then creates the config file.
Sorry for the late reply. I'm back from vacation now.
The below is fine with me. I suggest it be submitted as a patch to oe-core. I'll be happy to ack it.
--Mark
Quoted 33 lines
meta/classes/rootfs_rpm.bbclass +6
-0@@ -139,6 +139,14 @@ EOF install -d ${IMAGE_ROOTFS}/${sysconfdir} echo ${BUILDNAME} > ${IMAGE_ROOTFS}/${sysconfdir}/version + + # check if there is a jffs2 install, it requires a workaround + # due lacking support for MMAP read/write. + (echo "${IMAGE_FSTYPES}" | grep "jffs2" > /dev/null) + if [ $? == 0 ]; then + echo "%__dbi_txn create lock log txn auto_commit nommapprivate" > ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2
meta/classes/rootfs_rpm.bbclass +3
-0+ fi + ${RPM_POSTPROCESS_COMMANDS} ${ROOTFS_POSTPROCESS_COMMAND}@@ -164,6 +172,7 @@ EOF } remove_packaging_data_files() { + rm -rf ${IMAGE_ROOTFS}/etc/rpm/macros.jffs2 rm -rf ${IMAGE_ROOTFS}${rpmlibdir} rm -rf ${IMAGE_ROOTFS}${opkglibdir} }_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Quoted 6 lines · Holger Hans Peter Freyther
Hi,
a) I CTRL+C'ed the build once and later I noticed that my poky repo had the git origin changed. I think bitbake should check the result of the os.chdir... I will prepare a patch for that...
Okay, I am doing an oe-core build using this patch right now. Any comments?
holger
From 89e38923bdc01f2ef3f5ea81e5fd435e416b2ba6 Mon Sep 17 00:00:00 2001
From: Holger Hans Peter Freyther <[email protected]>
Date: Fri, 1 Jul 2011 22:12:52 +0800
Subject: [PATCH] misc: Replace os.chdir with bb.utils.checked_chdir
The Python os.chdir does not return the status, in some rare condition it seems possible that a Keyboard interrupt can interrupt the chdir and continue with the execution from the wrong directory. Introduce the bb.utils.checked_chdir to verify the chdir operation.
The users in bitbake of chdir are a bit sloppy, it is possible that someone is using a relative directory, uses a trailing slash or double slashes (e.g. license-destdir//).
--- lib/bb/build.py | 4 ++-- lib/bb/cache.py | 4 ++-- lib/bb/fetch/bzr.py | 6 +++--- lib/bb/fetch/cvs.py | 10 +++++----- lib/bb/fetch/git.py | 18 +++++++++--------- lib/bb/fetch/hg.py | 8 ++++---- lib/bb/fetch/osc.py | 6 +++--- lib/bb/fetch/perforce.py | 2 +- lib/bb/fetch/repo.py | 4 ++-- lib/bb/fetch/svk.py | 4 ++-- lib/bb/fetch/svn.py | 6 +++--- lib/bb/fetch2/__init__.py | 6 +++--- lib/bb/fetch2/bzr.py | 6 +++--- lib/bb/fetch2/cvs.py | 10 +++++----- lib/bb/fetch2/git.py | 14 +++++++------- lib/bb/fetch2/hg.py | 8 ++++---- lib/bb/fetch2/osc.py | 6 +++--- lib/bb/fetch2/perforce.py | 2 +- lib/bb/fetch2/repo.py | 4 ++-- lib/bb/fetch2/svk.py | 4 ++-- lib/bb/fetch2/svn.py | 6 +++--- lib/bb/pysh/builtin.py | 4 ++-- lib/bb/pysh/interp.py | 4 ++-- lib/bb/utils.py | 17 +++++++++++++++++ 24 files changed, 90 insertions(+), 73 deletions(-)
lib/bb/build.py +2 -2
@@ -191,7 +191,7 @@ def exec_func_python(func, d, runfile, cwd=None): olddir = os.getcwd() except OSError: olddir = None - os.chdir(cwd) + bb.utils.checked_chdir(cwd) try: comp = utils.better_compile(code, func, bbfile)
@@ -204,7 +204,7 @@ def exec_func_python(func, d, runfile, cwd=None): finally: if cwd and olddir: try: - os.chdir(olddir) + bb.utils.checked_chdir(olddir) except OSError: pass
lib/bb/cache.py +2 -2
@@ -657,11 +657,11 @@ class Cache(object): data.setVar('__BBAPPEND', " ".join(appends), bb_data) bb_data = parse.handle(bbfile, bb_data) if chdir_back: - os.chdir(oldpath) + bb.utils.checked_chdir(oldpath) return bb_data except: if chdir_back: - os.chdir(oldpath) + bb.utils.checked_chdir(oldpath) raise
lib/bb/fetch/bzr.py +3 -3
@@ -88,18 +88,18 @@ class Bzr(Fetch): if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK): bzrcmd = self._buildbzrcommand(ud, d, "update") logger.debug(1, "BZR Update %s", loc) - os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path))) + bb.utils.checked_chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path))) runfetchcmd(bzrcmd, d) else: bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True) bzrcmd = self._buildbzrcommand(ud, d, "fetch") logger.debug(1, "BZR Checkout %s", loc) bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", bzrcmd) runfetchcmd(bzrcmd, d) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
lib/bb/fetch/cvs.py +5 -5
@@ -132,13 +132,13 @@ class Cvs(Fetch): if os.access(os.path.join(moddir, 'CVS'), os.R_OK): logger.info("Update " + loc) # update sources there - os.chdir(moddir) + bb.utils.checked_chdir(moddir) myret = os.system(cvsupdatecmd) else: logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(pkgdir) - os.chdir(pkgdir) + bb.utils.checked_chdir(pkgdir) logger.debug(1, "Running %s", cvscmd) myret = os.system(cvscmd)
@@ -157,11 +157,11 @@ class Cvs(Fetch): # tar them up to a defined filename if 'fullpath' in ud.parm: - os.chdir(pkgdir) + bb.utils.checked_chdir(pkgdir) myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir)) else: - os.chdir(moddir) - os.chdir('..') + bb.utils.checked_chdir(moddir) + bb.utils.checked_chdir('..') myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir))) if myret != 0:
lib/bb/fetch/git.py +9 -9
@@ -132,14 +132,14 @@ class Git(Fetch): # If the checkout doesn't exist and the mirror tarball does, extract it if not os.path.exists(ud.clonedir) and os.path.exists(repofile): bb.utils.mkdirhier(ud.clonedir) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) runfetchcmd("tar -xzf %s" % (repofile), d) # If the repo still doesn't exist, fallback to cloning it if not os.path.exists(ud.clonedir): runfetchcmd("%s clone -n %s://%s%s%s %s" % (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir), d) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) # Update the checkout if needed if not self._contains_ref(ud.tag, d) or 'fullclone' in ud.parm: # Remove all but the .git directory
@@ -153,7 +153,7 @@ class Git(Fetch): runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d) # Generate a mirror tarball if needed - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) mirror_tarballs = data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) if mirror_tarballs != "0" or 'fullclone' in ud.parm: logger.info("Creating tarball of git repository")
@@ -185,19 +185,19 @@ class Git(Fetch): scmdata = ud.parm.get("scmdata", "") if scmdata == "keep": runfetchcmd("%s clone -n %s %s" % (ud.basecmd, ud.clonedir, coprefix), d) - os.chdir(coprefix) + bb.utils.checked_chdir(coprefix) runfetchcmd("%s checkout -q -f %s%s" % (ud.basecmd, ud.tag, readpathspec), d) else: bb.utils.mkdirhier(codir) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.tag, readpathspec), d) runfetchcmd("%s checkout-index -q -f --prefix=%s -a" % (ud.basecmd, coprefix), d) - os.chdir(codir) + bb.utils.checked_chdir(codir) logger.info("Creating tarball of git checkout") runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) bb.utils.prunedir(codir) def supports_srcrev(self):
@@ -327,12 +327,12 @@ class Git(Fetch): return None - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) if not self._contains_ref(rev, d): self.go(None, ud, d) output = runfetchcmd("%s rev-list %s -- 2> /dev/null | wc -l" % (ud.basecmd, rev), d, quiet=True) - os.chdir(cwd) + bb.utils.checked_chdir(cwd) buildindex = "%s" % output.split()[0] logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
lib/bb/fetch/hg.py +4 -4
@@ -123,7 +123,7 @@ class Hg(Fetch): updatecmd = self._buildhgcommand(ud, d, "pull") logger.info("Update " + loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", updatecmd) runfetchcmd(updatecmd, d)
@@ -132,14 +132,14 @@ class Hg(Fetch): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", fetchcmd) runfetchcmd(fetchcmd, d) # Even when we clone (fetch), we still need to update as hg's clone # won't checkout the specified revision if its on a branch updatecmd = self._buildhgcommand(ud, d, "update") - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", updatecmd) runfetchcmd(updatecmd, d)
@@ -149,7 +149,7 @@ class Hg(Fetch): else: tar_flags = "--exclude '.hg' --exclude '.hgrags'" - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) try: runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d) except:
lib/bb/fetch/osc.py +3 -3
@@ -91,7 +91,7 @@ class Osc(Fetch): oscupdatecmd = self._buildosccommand(ud, d, "update") logger.info("Update "+ loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", oscupdatecmd) runfetchcmd(oscupdatecmd, d) else:
@@ -99,11 +99,11 @@ class Osc(Fetch): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", oscfetchcmd) runfetchcmd(oscfetchcmd, d) - os.chdir(os.path.join(ud.pkgdir + ud.path)) + bb.utils.checked_chdir(os.path.join(ud.pkgdir + ud.path)) # tar them up to a defined filename try: runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
lib/bb/fetch/perforce.py +1 -1
@@ -168,7 +168,7 @@ class Perforce(Fetch): cset = Perforce.getcset(d, depot, host, user, pswd, parm) depot = "%s@%s" % (depot, cset) - os.chdir(tmpfile) + bb.utils.checked_chdir(tmpfile) logger.info("Fetch " + loc) logger.info("%s%s files %s", p4cmd, p4opt, depot) p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
lib/bb/fetch/repo.py +2 -2
@@ -72,12 +72,12 @@ class Repo(Fetch): username = "" bb.utils.mkdirhier(os.path.join(codir, "repo")) - os.chdir(os.path.join(codir, "repo")) + bb.utils.checked_chdir(os.path.join(codir, "repo")) if not os.path.exists(os.path.join(codir, "repo", ".repo")): runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d) runfetchcmd("repo sync", d) - os.chdir(codir) + bb.utils.checked_chdir(codir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
lib/bb/fetch/svk.py +2 -2
@@ -80,7 +80,7 @@ class Svk(Fetch): raise FetchError(ud.module) # check out sources there - os.chdir(tmpfile) + bb.utils.checked_chdir(tmpfile) logger.info("Fetch " + loc) logger.debug(1, "Running %s", svkcmd) myret = os.system(svkcmd)
@@ -91,7 +91,7 @@ class Svk(Fetch): pass raise FetchError(ud.module) - os.chdir(os.path.join(tmpfile, os.path.dirname(ud.module))) + bb.utils.checked_chdir(os.path.join(tmpfile, os.path.dirname(ud.module))) # tar them up to a defined filename myret = os.system("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.module))) if myret != 0:
lib/bb/fetch/svn.py +3 -3
@@ -139,7 +139,7 @@ class Svn(Fetch): svnupdatecmd = self._buildsvncommand(ud, d, "update") logger.info("Update " + loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", svnupdatecmd) runfetchcmd(svnupdatecmd, d) else:
@@ -147,7 +147,7 @@ class Svn(Fetch): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", svnfetchcmd) runfetchcmd(svnfetchcmd, d)
@@ -157,7 +157,7 @@ class Svn(Fetch): else: tar_flags = "--exclude '.svn'" - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) # tar them up to a defined filename try: runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d)
lib/bb/fetch2/__init__.py +3 -3
@@ -739,17 +739,17 @@ class FetchMethod(object): # Change to subdir before executing command save_cwd = os.getcwd(); - os.chdir(rootdir) + bb.utils.checked_chdir(rootdir) if 'subdir' in urldata.parm: newdir = ("%s/%s" % (rootdir, urldata.parm.get('subdir'))) bb.utils.mkdirhier(newdir) - os.chdir(newdir) + bb.utils.checked_chdir(newdir) cmd = "PATH=\"%s\" %s" % (bb.data.getVar('PATH', data, True), cmd) bb.note("Unpacking %s to %s/" % (file, os.getcwd())) ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True) - os.chdir(save_cwd) + bb.utils.checked_chdir(save_cwd) if ret != 0: raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), urldata.url)
lib/bb/fetch2/bzr.py +3 -3
@@ -88,7 +88,7 @@ class Bzr(FetchMethod): bzrcmd = self._buildbzrcommand(ud, d, "update") logger.debug(1, "BZR Update %s", loc) bb.fetch2.check_network_access(d, bzrcmd, ud.url) - os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path))) + bb.utils.checked_chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path))) runfetchcmd(bzrcmd, d) else: bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
@@ -96,11 +96,11 @@ class Bzr(FetchMethod): bb.fetch2.check_network_access(d, bzrcmd, ud.url) logger.debug(1, "BZR Checkout %s", loc) bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", bzrcmd) runfetchcmd(bzrcmd, d) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
lib/bb/fetch2/cvs.py +5 -5
@@ -134,13 +134,13 @@ class Cvs(FetchMethod): logger.info("Update " + loc) bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url) # update sources there - os.chdir(moddir) + bb.utils.checked_chdir(moddir) cmd = cvsupdatecmd else: logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(pkgdir) - os.chdir(pkgdir) + bb.utils.checked_chdir(pkgdir) logger.debug(1, "Running %s", cvscmd) bb.fetch2.check_network_access(d, cvscmd, ud.url) cmd = cvscmd
@@ -158,11 +158,11 @@ class Cvs(FetchMethod): # tar them up to a defined filename if 'fullpath' in ud.parm: - os.chdir(pkgdir) + bb.utils.checked_chdir(pkgdir) cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir) else: - os.chdir(moddir) - os.chdir('..') + bb.utils.checked_chdir(moddir) + bb.utils.checked_chdir('..') cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir)) runfetchcmd(cmd, d, cleanup = [ud.localpath])
lib/bb/fetch2/git.py +7 -7
@@ -135,7 +135,7 @@ class Git(FetchMethod): def need_update(self, u, ud, d): if not os.path.exists(ud.clonedir): return True - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) for name in ud.names: if not self._contains_ref(ud.revisions[name], d): return True
@@ -165,7 +165,7 @@ class Git(FetchMethod): # If the checkout doesn't exist and the mirror tarball does, extract it if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror): bb.utils.mkdirhier(ud.clonedir) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) runfetchcmd("tar -xzf %s" % (ud.fullmirror), d) # If the repo still doesn't exist, fallback to cloning it
@@ -175,7 +175,7 @@ class Git(FetchMethod): bb.fetch2.check_network_access(d, clone_cmd) runfetchcmd(clone_cmd, d) - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) # Update the checkout if needed needupdate = False for name in ud.names:
@@ -199,7 +199,7 @@ class Git(FetchMethod): def build_mirror_data(self, url, ud, d): # Generate a mirror tarball if needed if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)): - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) logger.info("Creating tarball of git repository") runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
@@ -219,7 +219,7 @@ class Git(FetchMethod): runfetchcmd("git clone -s -n %s %s" % (ud.clonedir, destdir), d) if not ud.nocheckout: - os.chdir(destdir) + bb.utils.checked_chdir(destdir) if subdir != "": runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d) runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
@@ -289,12 +289,12 @@ class Git(FetchMethod): return None - os.chdir(ud.clonedir) + bb.utils.checked_chdir(ud.clonedir) if not self._contains_ref(rev, d): self.download(None, ud, d) output = runfetchcmd("%s rev-list %s -- 2> /dev/null | wc -l" % (ud.basecmd, rev), d, quiet=True) - os.chdir(cwd) + bb.utils.checked_chdir(cwd) buildindex = "%s" % output.split()[0] logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
lib/bb/fetch2/hg.py +4 -4
@@ -124,7 +124,7 @@ class Hg(FetchMethod): updatecmd = self._buildhgcommand(ud, d, "pull") logger.info("Update " + loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", updatecmd) bb.fetch2.check_network_access(d, updatecmd, ud.url) runfetchcmd(updatecmd, d)
@@ -134,7 +134,7 @@ class Hg(FetchMethod): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", fetchcmd) bb.fetch2.check_network_access(d, fetchcmd, ud.url) runfetchcmd(fetchcmd, d)
@@ -142,7 +142,7 @@ class Hg(FetchMethod): # Even when we clone (fetch), we still need to update as hg's clone # won't checkout the specified revision if its on a branch updatecmd = self._buildhgcommand(ud, d, "update") - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", updatecmd) runfetchcmd(updatecmd, d)
@@ -152,7 +152,7 @@ class Hg(FetchMethod): else: tar_flags = "--exclude '.hg' --exclude '.hgrags'" - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath]) def supports_srcrev(self):
lib/bb/fetch2/osc.py +3 -3
@@ -88,7 +88,7 @@ class Osc(FetchMethod): oscupdatecmd = self._buildosccommand(ud, d, "update") logger.info("Update "+ loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", oscupdatecmd) bb.fetch2.check_network_access(d, oscupdatecmd, ud.url) runfetchcmd(oscupdatecmd, d)
@@ -97,12 +97,12 @@ class Osc(FetchMethod): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", oscfetchcmd) bb.fetch2.check_network_access(d, oscfetchcmd, ud.url) runfetchcmd(oscfetchcmd, d) - os.chdir(os.path.join(ud.pkgdir + ud.path)) + bb.utils.checked_chdir(os.path.join(ud.pkgdir + ud.path)) # tar them up to a defined filename runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d, cleanup = [ud.localpath])
lib/bb/fetch2/perforce.py +1 -1
@@ -165,7 +165,7 @@ class Perforce(FetchMethod): cset = Perforce.getcset(d, depot, host, user, pswd, parm) depot = "%s@%s" % (depot, cset) - os.chdir(tmpfile) + bb.utils.checked_chdir(tmpfile) logger.info("Fetch " + loc) logger.info("%s%s files %s", p4cmd, p4opt, depot) p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
lib/bb/fetch2/repo.py +2 -2
@@ -70,14 +70,14 @@ class Repo(FetchMethod): username = "" bb.utils.mkdirhier(os.path.join(codir, "repo")) - os.chdir(os.path.join(codir, "repo")) + bb.utils.checked_chdir(os.path.join(codir, "repo")) if not os.path.exists(os.path.join(codir, "repo", ".repo")): bb.fetch2.check_network_access(d, "repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url) runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d) bb.fetch2.check_network_access(d, "repo sync %s" % ud.url, ud.url) runfetchcmd("repo sync", d) - os.chdir(codir) + bb.utils.checked_chdir(codir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
lib/bb/fetch2/svk.py +2 -2
@@ -84,12 +84,12 @@ class Svk(FetchMethod): raise FetchError("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.", loc) # check out sources there - os.chdir(tmpfile) + bb.utils.checked_chdir(tmpfile) logger.info("Fetch " + loc) logger.debug(1, "Running %s", svkcmd) runfetchcmd(svkcmd, d, cleanup = [tmpfile]) - os.chdir(os.path.join(tmpfile, os.path.dirname(ud.module))) + bb.utils.checked_chdir(os.path.join(tmpfile, os.path.dirname(ud.module))) # tar them up to a defined filename runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.module)), d, cleanup = [ud.localpath])
lib/bb/fetch2/svn.py +3 -3
@@ -116,7 +116,7 @@ class Svn(FetchMethod): svnupdatecmd = self._buildsvncommand(ud, d, "update") logger.info("Update " + loc) # update sources there - os.chdir(ud.moddir) + bb.utils.checked_chdir(ud.moddir) logger.debug(1, "Running %s", svnupdatecmd) bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) runfetchcmd(svnupdatecmd, d)
@@ -125,7 +125,7 @@ class Svn(FetchMethod): logger.info("Fetch " + loc) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) logger.debug(1, "Running %s", svnfetchcmd) bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) runfetchcmd(svnfetchcmd, d)
@@ -136,7 +136,7 @@ class Svn(FetchMethod): else: tar_flags = "--exclude '.svn'" - os.chdir(ud.pkgdir) + bb.utils.checked_chdir(ud.pkgdir) # tar them up to a defined filename runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath])
lib/bb/pysh/builtin.py +2 -2
@@ -581,11 +581,11 @@ def utility_sort(name, args, interp, env, stdin, stdout, stderr, debugflags): # Load all files lines curdir = os.getcwd() try: - os.chdir(env['PWD']) + bb.utils.checked_chdir(env['PWD']) for path in args: alllines += sort(path) finally: - os.chdir(curdir) + bb.utils.checked_chdir(curdir) alllines.sort() for line in alllines:
lib/bb/pysh/interp.py +2 -2
@@ -1261,11 +1261,11 @@ class Interpreter: def pwd_glob(pattern): cwd = os.getcwd() - os.chdir(self._env['PWD']) + bb.utils.checked_chdir(self._env['PWD']) try: return glob.glob(pattern) finally: - os.chdir(cwd) + bb.utils.checked_chdir(cwd) #TODO: check working directory issues here wrt relative patterns try:
lib/bb/utils.py +17 -0
@@ -856,3 +856,20 @@ def to_boolean(string, default=None): return False else: raise ValueError("Invalid value for to_boolean: %s" % string) + +def checked_chdir(name): + """I change the directory and verify that it was changed, if + not I am raising an exception.""" + + # We are a bit sloppy with '/' or using a relative dir like + # .. in the fetchers. We resolve the new path here, remove a + # trailing slash and see if things match now. + old_dir = os.getcwd() + new_dir = os.path.join(old_dir, name) + new_dir = new_dir.replace('//', '/') + if new_dir[-1] == '/': + new_dir = new_dir[:-1] + os.chdir(new_dir) + + if os.getcwd() != new_dir: + raise OSError('Failed to change to directory: \'%s\'. You are in \'%s\', resolved \'%s\'' % (new_dir, os.getcwd(), new_dir))
Signature · 1 line
1.7.4.1