Skip to contentSkip to conversation

relocate_sdk.py: improvements

PATCH Superseded Jason Wessel 2 participants 8 messages Feb 11, 2013 – Feb 12, 2013

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover relocate_sdk.py: improvements

Now that I have had to debug the SDK relocator on multiple occasions I figure it might be nice to get the patches upstreamed. This last time through fixing the python 2.4.x compatibility was very easy using the prior two patches to allow me to do so.

The idea here is to increase the portability of the relocator because once the SDK is installed/relocated, generally speaking it is working well on old and new hosts.

Thanks,
Jason.

#1 lore raw

PATCH v2 1/3 relocate_sdk.py: Fix corruption of sdk binaries

There are two cases of corruption that the relocate_sdk.py was not correctly dealing with.

1) SDK Extras should be left alone
Extra external binaries included in an SDK that were linked against the host's version of /usr/lib/ld-so.so should not get a relocation applied. In the case that was discovered these were LSB compliant binaries that already worked on many hosts.

2) If the interp section is too small generate an error
In the case of the qemu user code, it was using its own .ld file to link the executables which overrides the default in the nativesdk binutils. This generated host executables which had a interp section that was too small to relocate.

Now the relocate_sdk.py will print an error and continue on such that the error can be fixed by a developer without having to do the difficult task of debugging why it is crashing or not loading correctly.

---
 scripts/relocate_sdk.py |   15 ++++++++++++---
 1 files changed, 12 insertions(+), 3 deletions(-)
scripts/relocate_sdk.py +12 -3
index 74bb7a5..6f3530d 100755
@@ -66,7 +66,7 @@ def parse_elf_header():
     e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx =\
         hdr_struct.unpack(elf_header[16:hdr_size])
 
-def change_interpreter():
+def change_interpreter(elf_file_name):
     if arch == 32:
         ph_struct = struct.Struct("<IIIIIIII")
     else:
@@ -89,7 +89,16 @@ def change_interpreter():
         if p_type == 3:
             # PT_INTERP section
             f.seek(p_offset)
-            dl_path = new_dl_path + "\0" * (p_filesz - len(new_dl_path))
+            dl_path = new_dl_path + "\0"
+            # External SDKs with mixed pre-compiled binaries should not get
+            # relocated so look for some variant of /lib
+            fname = f.read(11)
+            if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"):
+                break
+            if (len(dl_path) >= p_memsz):
+                print "ERROR: could not relocate %s, interp size = %i and %i is needed." % (elf_file_name, p_memsz, len(dl_path))
+                break
+            f.seek(p_offset)
             f.write(dl_path)
             break
 
@@ -199,7 +208,7 @@ for e in executables_list:
     arch = get_arch()
     if arch:
         parse_elf_header()
-        change_interpreter()
+        change_interpreter(e)
         change_dl_sysdirs()
 
     """ change permissions back """
Signature · 1 line
1.7.1
#2 lore raw

PATCH v2 2/3 populate_sdk_base.bbclass: Improve debugging capabilities for SDK installer

In reply to Jason Wessel

After having to debug the SDK installer a few times in addition to the relocation code the following patch was created to improve the capabilities around debugging the SDK installer.

1) Add a verbose mode -D which set a set -x to see what the SDK installer is doing.

2) Add a mode -S to save the relocation scripts for the purpose of debugging them in conjunction with -D

3) Add a mode -R to not execute the relocation scripts for the purpose of debugging the relocations.

---
 meta/classes/populate_sdk_base.bbclass |   40 +++++++++++++++++++++++++++----
 1 files changed, 34 insertions(+), 6 deletions(-)
meta/classes/populate_sdk_base.bbclass +34 -6
index c025d40..310b7cf 100644
@@ -136,7 +136,10 @@ DEFAULT_INSTALL_DIR="${SDKPATH}"
 SUDO_EXEC=""
 target_sdk_dir=""
 answer=""
-while getopts ":yd:" OPT; do
+relocate=1
+savescripts=0
+verbose=0
+while getopts ":yd:DRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
@@ -145,15 +148,33 @@ while getopts ":yd:" OPT; do
 	d)
 		target_sdk_dir=$OPTARG
 		;;
+	D)
+		verbose=1
+		;;
+	R)
+		relocate=0
+		savescripts=1
+		;;
+	S)
+		savescripts=1
+		;;
 	*)
 		echo "Usage: $(basename $0) [-y] [-d <dir>]"
 		echo "  -y         Automatic yes to all prompts"
 		echo "  -d <dir>   Install the SDK to <dir>"
+		echo "======== Advanced DEBUGGING ONLY OPTIONS ========"
+		echo "  -S         Save relocation scripts"
+		echo "  -R         Do not relocate executables"
+		echo "  -D         use set -x to see what is going on"
 		exit 1
 		;;
 	esac
 done
 
+if [ $verbose = 1 ] ; then
+	set -x
+fi
+
 printf "Enter target directory for SDK (default: $DEFAULT_INSTALL_DIR): "
 if [ "$target_sdk_dir" = "" ]; then
 	read target_sdk_dir
@@ -231,10 +252,15 @@ if [ "$dl_path" = "" ] ; then
 	exit 1
 fi
 executable_files=$($SUDO_EXEC find $native_sysroot -type f -perm +111)
-$SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files
-if [ $? -ne 0 ]; then
-	echo "SDK could not be set up. Relocate script failed. Abort!"
-	exit 1
+echo "#!/bin/bash" > ${env_setup_script%/*}/relocate_sdk.sh
+echo exec $SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files >> ${env_setup_script%/*}/relocate_sdk.sh
+chmod 755 ${env_setup_script%/*}/relocate_sdk.sh
+if [ $relocate = 1 ] ; then
+	${env_setup_script%/*}/relocate_sdk.sh
+	if [ $? -ne 0 ]; then
+		echo "SDK could not be set up. Relocate script failed. Abort!"
+		exit 1
+	fi
 fi
 
 # replace ${SDKPATH} with the new prefix in all text files: configs/scripts/etc
@@ -249,7 +275,9 @@ echo done
 
 # delete the relocating script, so that user is forced to re-run the installer
 # if he/she wants another location for the sdk
-$SUDO_EXEC rm ${env_setup_script%/*}/relocate_sdk.py
+if [ $savescripts = 0 ] ; then
+	$SUDO_EXEC rm ${env_setup_script%/*}/relocate_sdk.py ${env_setup_script%/*}/relocate_sdk.sh
+fi
 
 echo "SDK has been successfully set up and is ready to be used."
 
Signature · 1 line
1.7.1
#3 lore raw

PATCH v2 3/3 relocate_sdk.py: allow relocate_sdk.py to work with python 2.4.x

In reply to Jason Wessel

Avoid the chicken / egg problem of an SDK that provides a working python but requires that version of python to extract itself. The RHEL 5.x systems and some other enterprise Linux systems ship with python 2.4.x as the default python. We need to at least be able to extract work executables even if we never use the the host provided python again.

---
 scripts/relocate_sdk.py |   26 +++++++++++++-------------
 1 files changed, 13 insertions(+), 13 deletions(-)
scripts/relocate_sdk.py +13 -13
index 6f3530d..b7e579d 100755
@@ -55,22 +55,22 @@ def parse_elf_header():
 
     if arch == 32:
         # 32bit
-        hdr_struct = struct.Struct("<HHILLLIHHHHHH")
+        hdr_fmt = "<HHILLLIHHHHHH"
         hdr_size = 52
     else:
         # 64bit
-        hdr_struct = struct.Struct("<HHIQQQIHHHHHH")
+        hdr_fmt = "<HHIQQQIHHHHHH"
         hdr_size = 64
 
     e_type, e_machine, e_version, e_entry, e_phoff, e_shoff, e_flags,\
     e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx =\
-        hdr_struct.unpack(elf_header[16:hdr_size])
+        struct.unpack(hdr_fmt, elf_header[16:hdr_size])
 
 def change_interpreter(elf_file_name):
     if arch == 32:
-        ph_struct = struct.Struct("<IIIIIIII")
+        ph_fmt = "<IIIIIIII"
     else:
-        ph_struct = struct.Struct("<IIQQQQQQ")
+        ph_fmt = "<IIQQQQQQ"
 
     """ look for PT_INTERP section """
     for i in range(0,e_phnum):
@@ -79,11 +79,11 @@ def change_interpreter(elf_file_name):
         if arch == 32:
             # 32bit
             p_type, p_offset, p_vaddr, p_paddr, p_filesz,\
-                p_memsz, p_flags, p_align = ph_struct.unpack(ph_hdr)
+                p_memsz, p_flags, p_align = struct.unpack(ph_fmt, ph_hdr)
         else:
             # 64bit
             p_type, p_flags, p_offset, p_vaddr, p_paddr, \
-            p_filesz, p_memsz, p_align = ph_struct.unpack(ph_hdr)
+            p_filesz, p_memsz, p_align = struct.unpack(ph_fmt, ph_hdr)
 
         """ change interpreter """
         if p_type == 3:
@@ -104,9 +104,9 @@ def change_interpreter(elf_file_name):
 
 def change_dl_sysdirs():
     if arch == 32:
-        sh_struct = struct.Struct("<IIIIIIIIII")
+        sh_fmt = "<IIIIIIIIII"
     else:
-        sh_struct = struct.Struct("<IIQQQQIIQQ")
+        sh_fmt = "<IIQQQQIIQQ"
 
     """ read section string table """
     f.seek(e_shoff + e_shstrndx * e_shentsize)
@@ -127,7 +127,7 @@ def change_dl_sysdirs():
         sh_hdr = f.read(e_shentsize)
 
         sh_name, sh_type, sh_flags, sh_addr, sh_offset, sh_size, sh_link,\
-            sh_info, sh_addralign, sh_entsize = sh_struct.unpack(sh_hdr)
+            sh_info, sh_addralign, sh_entsize = struct.unpack(sh_fmt, sh_hdr)
 
         name = sh_strtab[sh_name:sh_strtab.find("\0", sh_name)]
 
@@ -181,7 +181,7 @@ def change_dl_sysdirs():
 
 # MAIN
 if len(sys.argv) < 4:
-    exit(-1)
+    sys.exit(-1)
 
 new_prefix = sys.argv[1]
 new_dl_path = sys.argv[2]
@@ -196,14 +196,14 @@ for e in executables_list:
 
     try:
         f = open(e, "r+b")
-    except IOError as ioex:
+    except IOError, ioex:
         if ioex.errno == errno.ETXTBSY:
             print("Could not open %s. File used by another process.\nPlease "\
                   "make sure you exit all processes that might use any SDK "\
                   "binaries." % e)
         else:
             print("Could not open %s: %s(%d)" % (e, ioex.strerror, ioex.errno))
-        exit(-1)
+        sys.exit(-1)
 
     arch = get_arch()
     if arch:
Signature · 1 line
1.7.1
#4 lore raw
In reply to Jason Wessel
Quoted 2 lines · Jason Wessel

Now that I have had to debug the SDK relocator on multiple occasions I figure it might be nice to get the patches upstreamed.

But, before that, did you see my comments on the previous patchset? It looks like they went unnoticed as they were not addressed.

Here is what I replied to your previous patches:
http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034868.html
http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034869.html

Thanks,
Laurentiu

#5 lore raw
Quoted 11 lines · Laurentiu Palcu
Quoted 2 lines · Jason Wessel

Now that I have had to debug the SDK relocator on multiple occasions I figure it might be nice to get the patches upstreamed.

But, before that, did you see my comments on the previous patchset? It looks like they went unnoticed as they were not addressed.

Here is what I replied to your previous patches:
http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034868.html
http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034869.html

For what ever reason I never received the original mails, else I absolutely would have responded. This is the first response I have received from the oe-core list in months in fact.

To answer your question you posed in the threads above, I'll do it right here due to the lack of the originals.

Quoted 7 lines
Quoted 1 line

- dl_path = new_dl_path + "\0" * (p_filesz - len(new_dl_path))

Personally, I would prefer you left the zero padding in place. Otherwise, if installing in a location like /opt/test the .interp section would look like below. Technically, the dynamic loader would not care but it would be nice to have a clean .interp section, without leftover strings in it...

Quoted 6 lines

This is how it would look like after relocation:

$ readelf -p .interp qemu-arm

String dump of section '.interp':
[ 0] /opt/test/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2 [ 41] -x86-64.so.2

I completely agree with you. This will be fixed in v3.

Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+echo "#!/bin/bash" > ${env_setup_script%/*}/relocate_sdk.sh
+echo exec $SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files >> ${env_setup_script%/*}/relocate_sdk.sh
+chmod 755 ${env_setup_script%/*}/relocate_sdk.sh

The last 3 lines will certainly fail if installation takes place in a location you don't have rights... So, you'll end up with no relocate_sdk.sh script.

Easy enough to fix, I'll fix this shortly. Many thanks for your comments.

Cheers,
Jason.

#6 lore raw
Quoted 1 line · Jason Wessel

For what ever reason I never received the original mails, else I absolutely would have responded. This is the first response I have received from the oe-core list in months in fact.

I believe you... It happened to me too not to receive replies to my own patches. :/

Thanks,
Laurentiu

#7 lore raw
Quoted 5 lines · Laurentiu Palcu
Quoted 1 line · Jason Wessel

For what ever reason I never received the original mails, else I absolutely would have responded. This is the first response I have received from the oe-core list in months in fact.

I believe you... It happened to me too not to receive replies to my own patches. :/

Hard to say where the mail went other than the black hole. We too have had black hole issues, but I had always assumed it was on anti-spam filter side with our local mail server.

As a side point it occurred to me that I never answered your other question, which was about what qemu we were using. At the time I hit the problem with the silent .interp corruption it was the qemu from denzil. Silent corruption is never fun to find or deal with, we had just ended up with binary that segmentation faulted some of the time. One time through trying to figure out what the root of the problem was more than enough. I figured I never wanted anyone else to have to debug that problem again, which the origin of these patches. ;-)

Cheers,
Jason.