Skip to contentSkip to conversation

Update cleanup-workdir

PATCH Applied Kang Kai 2 participants 6 messages Jun 15, 2012 – Jun 18, 2012

lore
Share & tools
lore

Kang Kai

author
#0 lore raw

cover Update cleanup-workdir

Hi,

This series of patches update the cleanup-workdir:
1 update the help text
2 update the way to check obsolete directories 3 only clean the directories related to current arch 4 replace module commands with subprocess

Regards,
Kai

The following changes since commit 5ed855d12cddf2de535c3f6d05d3dfe85d69d99d:

openjade-native: Ensure we reautoconf the package (2012-06-12 16:34:30 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib kangkai/distro
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro

Kang Kai (4):
cleanup-workdir: update the way to check obsolete dirs
cleanup-workdir: replace commands with subprocess
cleanup-workdir: only deal dirs related to current arch
cleanup-workdir: update help text

 scripts/cleanup-workdir |  128 ++++++++++++++++++++++++++++++-----------------
 1 files changed, 82 insertions(+), 46 deletions(-)
Signature · 1 line
1.7.5.4
author
#1 lore raw

PATCH 1/4 cleanup-workdir: update the way to check obsolete dirs

Update the way to check obsolete directories.

According to package and its version construct a list of all packages' current build directory. If any directory under $WORKDIR/*/ is not in the list will be removed.

At same time, all the files(vs. directory) under $WORKDIR and $WORKDIR/*/ will be removed because they are not created by poky.

---
 scripts/cleanup-workdir |   59 ++++++++++++++++++----------------------------
 1 files changed, 23 insertions(+), 36 deletions(-)
scripts/cleanup-workdir +23 -36
index b77e8c6..3739a00 100755
@@ -22,7 +22,7 @@ import re
 import commands
 import shutil
 
-versions = {}
+pkg_cur_dirs = []
 obsolete_dirs = []
 parser = None
 
@@ -39,15 +39,6 @@ def parse_version(verstr):
     else:
         return epoch + '_' + elems[1]
 
-def parse_dir(match, pkgabsdir):
-    pkg_name = match.group(1)
-    pkg_version = match.group(2)
-    if pkg_name in versions:
-        if pkg_version != versions[pkg_name]:
-            obsolete_dirs.append(pkgabsdir)
-        return True
-    return False
-
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -89,7 +80,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
             version = parse_version(elems[1])
         else:
             version = parse_version(elems[2])
-        versions[elems[0]] = version
+        pkg_cur_dirs.append(elems[0] + '-' + version)
 
     cmd = "bitbake -e | grep ^TMPDIR"
     (ret, output) = commands.getstatusoutput(cmd)
@@ -103,31 +94,27 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
         print "WORKDIR %s does NOT exist. Quit." % workdir
         return 1
 
-    for archdir in os.listdir(workdir):
-        archdir = os.path.join(workdir, archdir)
-        if not os.path.isdir(archdir):
-            pass
-
-        for pkgdir in sorted(os.listdir(archdir)):
-            pkgabsdir = os.path.join(archdir, pkgdir)
-            if not os.path.isdir(pkgabsdir):
-                pass
-
-            # parse the package directory names
-            # parse native/nativesdk packages first
-            match = re.match('(.*?-native.*?)-(.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
-
-            # parse package names which ends with numbers such as 'glib-2.0'
-            match = re.match('(.*?-[\.\d]+)-(\d.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
-
-            # other packages
-            match = re.match('(.*?)-(\d.*)', pkgdir)
-            if match and parse_dir(match, pkgabsdir):
-                continue
+    for workroot, dirs, files in os.walk(workdir):
+        # For the files, they should NOT exist in WORKDIR. Romve them.
+        for f in files:
+            obsolete_dirs.append(os.path.join(workroot, f))
+
+        for d in dirs:
+            for pkgroot, pkgdirs, filenames in os.walk(os.path.join(workroot, d)):
+                for f in filenames:
+                    obsolete_dirs.append(os.path.join(pkgroot, f))
+
+                for pkgdir in sorted(pkgdirs):
+                    if pkgdir not in pkg_cur_dirs:
+                        obsolete_dirs.append(os.path.join(pkgroot, pkgdir))
+
+                # just process the top dir of every package under tmp/work/*/,
+                # then jump out of the above os.walk()
+                break
+
+        # it is convenient to use os.walk() to get dirs and files at same time
+        # both of them have been dealed in the loop, so jump out
+        break
 
     for d in obsolete_dirs:
         print "Deleleting %s" % d
Signature · 1 line
1.7.5.4
author
#2 lore raw

PATCH 2/4 cleanup-workdir: replace commands with subprocess

In reply to Kang Kai

Use modules subprocess to run command instead of module commands.

---
 scripts/cleanup-workdir |   22 ++++++++++++----------
 1 files changed, 12 insertions(+), 10 deletions(-)
scripts/cleanup-workdir +12 -10
index 3739a00..1e9c56d 100755
@@ -19,7 +19,7 @@ import os
 import sys
 import optparse
 import re
-import commands
+import subprocess
 import shutil
 
 pkg_cur_dirs = []
@@ -39,6 +39,14 @@ def parse_version(verstr):
     else:
         return epoch + '_' + elems[1]
 
+def run_command(cmd):
+    pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
+    output = pipe.communicate()[0]
+    if pipe.returncode != 0:
+        print "Execute command '%s' failed." % cmd
+        sys.exit(1)
+    return output
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -49,7 +57,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
 
     options, args = parser.parse_args(sys.argv)
 
-    builddir = commands.getoutput('echo $BUILDDIR')
+    builddir = run_command('echo $BUILDDIR').strip()
     if len(builddir) == 0:
         err_quit("Please source file \"oe-init-build-env\" first.\n")
 
@@ -58,10 +66,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
 
     print 'Updating bitbake caches...'
     cmd = "bitbake -s"
-    (ret, output) = commands.getstatusoutput(cmd)
-    if ret != 0:
-        print "Execute 'bitbake -s' failed. Can't get packages' versions."
-        return 1
+    output = run_command(cmd)
 
     output = output.split('\n')
     index = 0
@@ -83,10 +88,7 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
         pkg_cur_dirs.append(elems[0] + '-' + version)
 
     cmd = "bitbake -e | grep ^TMPDIR"
-    (ret, output) = commands.getstatusoutput(cmd)
-    if ret != 0:
-        print "Execute 'bitbke -e' failed. Can't get TMPDIR."
-        return 1
+    output = run_command(cmd)
 
     tmpdir = output.split('"')[1]
     workdir = os.path.join(tmpdir, 'work')
Signature · 1 line
1.7.5.4
author
#3 lore raw

PATCH 3/4 cleanup-workdir: only deal dirs related to current arch

In reply to Kang Kai

Some users may build for different archs under same workdir, so they don't want to clean the dirs not related to current arch.

Run command 'bitbake -e' with selected packages to get the dirs related to current arch then clean them.

Update the way to get the WORKDIR by parsing the IMAGE_ROOTFS by the way.

---
 scripts/cleanup-workdir |   54 ++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 49 insertions(+), 5 deletions(-)
scripts/cleanup-workdir +49 -5
index 1e9c56d..156a259 100755
@@ -47,6 +47,19 @@ def run_command(cmd):
         sys.exit(1)
     return output
 
+def get_cur_arch_dirs(workdir, arch_dirs):
+    pattern = workdir + '/(.*?)/'
+
+    # select thest 5 packages to get the dirs of current arch
+    pkgs = ['hicolor-icon-theme', 'base-files', 'acl-native', 'binutils-crosssdk', 'autoconf-nativesdk']
+
+    for pkg in pkgs:
+        cmd = "bitbake -e " + pkg + " | grep ^IMAGE_ROOTFS="
+        output = run_command(cmd)
+        output = output.split('"')[1]
+        m = re.match(pattern, output)
+        arch_dirs.append(m.group(1))
+
 def main():
     global parser
     parser = optparse.OptionParser(
@@ -87,21 +100,52 @@ This script must be ran under BUILDDIR after source file \"oe-init-build-env\"."
             version = parse_version(elems[2])
         pkg_cur_dirs.append(elems[0] + '-' + version)
 
-    cmd = "bitbake -e | grep ^TMPDIR"
+    cmd = "bitbake -e"
     output = run_command(cmd)
 
-    tmpdir = output.split('"')[1]
-    workdir = os.path.join(tmpdir, 'work')
-    if not os.path.exists(workdir):
-        print "WORKDIR %s does NOT exist. Quit." % workdir
+    tmpdir = None
+    image_rootfs = None
+    output = output.split('\n')
+    for line in output:
+        if tmpdir and image_rootfs:
+            break
+
+        if not tmpdir:
+            m = re.match('TMPDIR="(.*)"', line)
+            if m:
+                tmpdir = m.group(1)
+
+        if not image_rootfs:
+            m = re.match('IMAGE_ROOTFS="(.*)"', line)
+            if m:
+                image_rootfs = m.group(1)
+
+    # won't fail just in case
+    if not tmpdir or not image_rootfs:
+        print "Can't get TMPDIR or IMAGE_ROOTFS."
+        return 1
+
+    pattern = tmpdir + '/(.*?)/(.*?)/'
+    m = re.match(pattern, image_rootfs)
+    if not m:
+        print "Can't get WORKDIR."
         return 1
 
+    workdir = os.path.join(tmpdir, m.group(1))
+
+    # we only deal the dirs of current arch, total numbers of dirs are 6
+    cur_arch_dirs = [m.group(2)]
+    get_cur_arch_dirs(workdir, cur_arch_dirs)
+
     for workroot, dirs, files in os.walk(workdir):
         # For the files, they should NOT exist in WORKDIR. Romve them.
         for f in files:
             obsolete_dirs.append(os.path.join(workroot, f))
 
         for d in dirs:
+            if d not in cur_arch_dirs:
+                continue
+
             for pkgroot, pkgdirs, filenames in os.walk(os.path.join(workroot, d)):
                 for f in filenames:
                     obsolete_dirs.append(os.path.join(pkgroot, f))
Signature · 1 line
1.7.5.4
author
#4 lore raw

PATCH 4/4 cleanup-workdir: update help text

In reply to Kang Kai

Update the help text to tell user that the files and dirs under WORKDIR which are not created by Yocto will be deleted.

---
 scripts/cleanup-workdir |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
scripts/cleanup-workdir +5 -2
index 156a259..c748846 100755
@@ -65,8 +65,11 @@ def main():
     parser = optparse.OptionParser(
         usage = """%prog
 
-Remove the obsolete packages' build directories in WORKDIR.
-This script must be ran under BUILDDIR after source file \"oe-init-build-env\".""")
+%prog removes the obsolete packages' build directories in WORKDIR.
+This script must be ran under BUILDDIR after source file \"oe-init-build-env\".
+
+Any file or directory under WORKDIR which is not created by Yocto
+will be deleted. Be CAUTIOUS.""")
 
     options, args = parser.parse_args(sys.argv)
 
Signature · 1 line
1.7.5.4
Applied
#5 lore raw
In reply to Kang Kai
Quoted 19 lines · Kang Kai

Hi,

This series of patches update the cleanup-workdir:
1 update the help text
2 update the way to check obsolete directories 3 only clean the directories related to current arch 4 replace module commands with subprocess

Regards,
Kai

The following changes since commit 5ed855d12cddf2de535c3f6d05d3dfe85d69d99d:

   openjade-native: Ensure we reautoconf the package (2012-06-12 16:34:30 +0100)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib kangkai/distro
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/distro

Quoted 9 lines

Kang Kai (4):
cleanup-workdir: update the way to check obsolete dirs
cleanup-workdir: replace commands with subprocess
cleanup-workdir: only deal dirs related to current arch
cleanup-workdir: update help text

  scripts/cleanup-workdir |  128 ++++++++++++++++++++++++++++++-----------------
  1 files changed, 82 insertions(+), 46 deletions(-)

Merged into OE-Core

Thanks
Sau!