staging: Strip files in sysroot

Add functionality to strip binaries/libraries going into the sysroot. Whilst this does fractionally slow down the build, it also significantly reduces the size of the sstate cache files.

Signed-off-by Richard Purdie <[email protected]>

meta/classes/staging.bbclass +96 -0

index 10198fd..7732e16 100644
@@ -57,6 +57,101 @@ sysroot_stage_all() {
        sysroot_stage_dirs ${D} ${SYSROOT_DESTDIR}
 }
 
+python sysroot_strip () {
+    import stat, errno
+
+    dvar = d.getVar('SYSROOT_DESTDIR', True)
+    pn = d.getVar('PN', True)
+
+    os.chdir(dvar)
+
+    # Return type (bits):
+    # 0 - not elf
+    # 1 - ELF
+    # 2 - stripped
+    # 4 - executable
+    # 8 - shared library
+    # 16 - kernel module
+    def isELF(path):
+        type = 0
+        ret, result = oe.utils.getstatusoutput("file \"%s\"" % path.replace("\"", "\\\""))
+
+        if ret:
+            bb.error("split_and_strip_files: 'file %s' failed" % path)
+            return type
+
+        # Not stripped
+        if "ELF" in result:
+            type |= 1
+            if "not stripped" not in result:
+                type |= 2
+            if "executable" in result:
+                type |= 4
+            if "shared" in result:
+                type |= 8
+        return type
+
+
+    elffiles = {}
+    inodes = {}
+    libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir", True))
+    baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True))
+    if (d.getVar('INHIBIT_SYSROOT_STRIP', True) != '1'):
+        #
+        # First lets figure out all of the files we may have to process
+        #
+        for root, dirs, files in os.walk(dvar):
+            for f in files:
+                file = os.path.join(root, f)
+
+                try:
+                    ltarget = oe.path.realpath(file, dvar, False)
+                    s = os.lstat(ltarget)
+                except OSError as e:
+                    (err, strerror) = e.args
+                    if err != errno.ENOENT:
+                        raise
+                    # Skip broken symlinks
+                    continue
+                if not s:
+                    continue
+                # Check its an excutable
+                if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and ".so" in f):
+                    # If it's a symlink, and points to an ELF file, we capture the readlink target
+                    if os.path.islink(file):
+                        continue
+
+                    # It's a file (or hardlink), not a link
+                    # ...but is it ELF, and is it already stripped?
+                    elf_file = isELF(file)
+                    if elf_file & 1:
+                        if elf_file & 2:
+                            bb.warn("File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn))
+                            continue
+
+                        if s.st_ino in inodes:
+                            os.unlink(file)
+                            os.link(inodes[s.st_ino], file)
+                        else:
+                            inodes[s.st_ino] = file
+                            # break hardlink
+                            bb.utils.copyfile(file, file)
+                            elffiles[file] = elf_file
+
+        #
+        # Now strip them (in parallel)
+        #
+        strip = d.getVar("STRIP", True)
+        sfiles = []
+        for file in elffiles:
+            elf_file = int(elffiles[file])
+            #bb.note("Strip %s" % file)
+            sfiles.append((file, elf_file, strip))
+
+        oe.utils.multiprocess_exec(sfiles, oe.package.runstrip)
+}
+
 do_populate_sysroot[dirs] = "${SYSROOT_DESTDIR}"
 do_populate_sysroot[umask] = "022"
 

@@ -90,6 +185,7 @@ def sysroot_checkhashes(covered, tasknames, fnids, fns, d, invalidtasks = None):
 
 python do_populate_sysroot () {
     bb.build.exec_func("sysroot_stage_all", d)
+    bb.build.exec_func("sysroot_strip", d)
     for f in (d.getVar('SYSROOT_PREPROCESS_FUNCS', True) or '').split():
         bb.build.exec_func(f, d)
     pn = d.getVar("PN", True)

meta/recipes-core/glibc/glibc-package.inc +0 -0

index 984362e..8ea5915 100644

Quoted 128 lines · Richard Purdie

Add functionality to strip binaries/libraries going into the sysroot. Whilst this does fractionally slow down the build, it also significantly reduces the size of the sstate cache files.

Signed-off-by Richard Purdie <[email protected]>

meta/classes/staging.bbclass +96 -0

index 10198fd..7732e16 100644
@@ -57,6 +57,101 @@ sysroot_stage_all() {
        sysroot_stage_dirs ${D} ${SYSROOT_DESTDIR}
 }
 
+python sysroot_strip () {
+    import stat, errno
+
+    dvar = d.getVar('SYSROOT_DESTDIR', True)
+    pn = d.getVar('PN', True)
+
+    os.chdir(dvar)
+
+    # Return type (bits):
+    # 0 - not elf
+    # 1 - ELF
+    # 2 - stripped
+    # 4 - executable
+    # 8 - shared library
+    # 16 - kernel module
+    def isELF(path):
+        type = 0
+        ret, result = oe.utils.getstatusoutput("file \"%s\"" % path.replace("\"", "\\\""))
+
+        if ret:
+            bb.error("split_and_strip_files: 'file %s' failed" % path)
+            return type
+
+        # Not stripped
+        if "ELF" in result:
+            type |= 1
+            if "not stripped" not in result:
+                type |= 2
+            if "executable" in result:
+                type |= 4
+            if "shared" in result:
+                type |= 8
+        return type
+
+
+    elffiles = {}
+    inodes = {}
+    libdir = os.path.abspath(dvar + os.sep + d.getVar("libdir", True))
+    baselibdir = os.path.abspath(dvar + os.sep + d.getVar("base_libdir", True))
+    if (d.getVar('INHIBIT_SYSROOT_STRIP', True) != '1'):
+        #
+        # First lets figure out all of the files we may have to process
+        #
+        for root, dirs, files in os.walk(dvar):
+            for f in files:
+                file = os.path.join(root, f)
+
+                try:
+                    ltarget = oe.path.realpath(file, dvar, False)
+                    s = os.lstat(ltarget)
+                except OSError as e:
+                    (err, strerror) = e.args
+                    if err != errno.ENOENT:
+                        raise
+                    # Skip broken symlinks
+                    continue
+                if not s:
+                    continue
+                # Check its an excutable
+                if (s[stat.ST_MODE] & stat.S_IXUSR) or (s[stat.ST_MODE] & stat.S_IXGRP) or (s[stat.ST_MODE] & stat.S_IXOTH) \
+                        or ((file.startswith(libdir) or file.startswith(baselibdir)) and ".so" in f):
+                    # If it's a symlink, and points to an ELF file, we capture the readlink target
+                    if os.path.islink(file):
+                        continue
+
+                    # It's a file (or hardlink), not a link
+                    # ...but is it ELF, and is it already stripped?
+                    elf_file = isELF(file)
+                    if elf_file & 1:
+                        if elf_file & 2:
+                            bb.warn("File '%s' from %s was already stripped, this will prevent future debugging!" % (file[len(dvar):], pn))
+                            continue
+
+                        if s.st_ino in inodes:
+                            os.unlink(file)
+                            os.link(inodes[s.st_ino], file)
+                        else:
+                            inodes[s.st_ino] = file
+                            # break hardlink
+                            bb.utils.copyfile(file, file)
+                            elffiles[file] = elf_file
+
+        #
+        # Now strip them (in parallel)
+        #
+        strip = d.getVar("STRIP", True)
+        sfiles = []
+        for file in elffiles:
+            elf_file = int(elffiles[file])
+            #bb.note("Strip %s" % file)
+            sfiles.append((file, elf_file, strip))
+
+        oe.utils.multiprocess_exec(sfiles, oe.package.runstrip)
+}
+
 do_populate_sysroot[dirs] = "${SYSROOT_DESTDIR}"
 do_populate_sysroot[umask] = "022"
 

@@ -90,6 +185,7 @@ def sysroot_checkhashes(covered, tasknames, fnids, fns, d, invalidtasks = None):
 
 python do_populate_sysroot () {
     bb.build.exec_func("sysroot_stage_all", d)
+    bb.build.exec_func("sysroot_strip", d)
     for f in (d.getVar('SYSROOT_PREPROCESS_FUNCS', True) or '').split():
         bb.build.exec_func(f, d)
     pn = d.getVar("PN", True)

meta/recipes-core/glibc/glibc-package.inc +0 -0

index 984362e..8ea5915 100644
@@ -17,6 +17,8 @@ python __anonymous () {
 # Set this to zero if you don't want ldconfig in the output package
 USE_LDCONFIG ?= "1"
 

meta/recipes-core/glibc/glibc-package.inc python __anonymous () { · line 24

Quoted 6 lines

meta/recipes-core/glibc/glibc-package.inc +2 -0

+INHIBIT_SYSROOT_STRIP = "1"
+
 PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode ${PN}-utils glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH

Do I understand this right: Stripping is disabled by default (because it would cause trouble for remote debugging)?

Quoted 2 lines

Do I understand this right: Stripping is disabled by default (because it would cause trouble for remote debugging)?

No, this is only stripping of the sysroot files, not target. Its due to the way locale packaging happens for glibc. It does mean some files in the sysroot are not stripped when they could be but its no change from the existing behaviour before the patch.

Cheers,

Richard

Quoted 12 lines · Richard Purdie

Quoted 2 lines

Do I understand this right: Stripping is disabled by default (because it would cause trouble for remote debugging)?

No, this is only stripping of the sysroot files, not target. Its due to the way locale packaging happens for glibc. It does mean some files in the sysroot are not stripped when they could be but its no change from the existing behaviour before the patch.

Cheers,

Richard

Sorry for insisting - maybe I need more explanation:

From my understanding staging sequence is:

* do_install installs to recipe's image folder
* do_populate_sysroot -> sysroot_stage_all() creates hardlinks to recipe's sysroot-destdir folder
* do_populate_sysroot -> sysroot_strip() strips files in recipe's sysroot-destdir (=files in recipe's image folder)

* through sstate_install the files are hard linked to MACHINE's sysroot (STAGING_DIR_HOST)

I see two issues activating stripping

1. If do_package starts after stripping only stripped packages are packed also for dbg. I think nothing gaurantees that packaging occurs before do_populate_sysroot.

2. When remote debugging I tell debugger: set sysroot <content of STAGING_DIR_HOST>. Where do I find a unstripped sysroot now?

Quoted 16 lines · Andreas Müller

Quoted 12 lines · Richard Purdie

Quoted 2 lines

Do I understand this right: Stripping is disabled by default (because it would cause trouble for remote debugging)?

No, this is only stripping of the sysroot files, not target. Its due to the way locale packaging happens for glibc. It does mean some files in the sysroot are not stripped when they could be but its no change from the existing behaviour before the patch.

Cheers,

Richard

Sorry for insisting - maybe I need more explanation:

Quoted 7 lines

From my understanding staging sequence is:

* do_install installs to recipe's image folder
* do_populate_sysroot -> sysroot_stage_all() creates hardlinks to recipe's sysroot-destdir folder
* do_populate_sysroot -> sysroot_strip() strips files in recipe's sysroot-destdir (=files in recipe's image folder)

When it strips files, the hardlink is broken and replaced with a stripped copy.

Quoted 8 lines

* through sstate_install the files are hard linked to MACHINE's sysroot (STAGING_DIR_HOST)

I see two issues activating stripping

1. If do_package starts after stripping only stripped packages are packed also for dbg. I think nothing gaurantees that packaging occurs before do_populate_sysroot.

Nothing does but the copy in image remains unstripped as above.

Quoted 2 lines

2. When remote debugging I tell debugger: set sysroot <content of STAGING_DIR_HOST>. Where do I find a unstripped sysroot now?

That is not really the way the sysroot has been intended to be used. Inhibiting the sysroot stripping for target files would seem to be the best thing to do if you do wish to use the sysroot that way:

INHIBIT_SYSROOT_STRIP_class-target = "1"

Cheers,