Skip to contentSkip to conversation

livepatch-gcc: Allow toolchain command with versions

PATCH In review Pawel Wieczorkiewicz 3 participants 12 messages Apr 8, 2019 – Apr 29, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 livepatch-gcc: Allow toolchain command with versions

Xen build system may enforce particular gcc version (e.g. gcc72). Make sure the livepatch-gcc script accepts all input toolchain gcc commands with or without version specified.

---
 livepatch-gcc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
livepatch-gcc +2 -1
index 634157a..617f865 100755
@@ -25,7 +25,8 @@ shift
 declare -a args=("$@")
 keep=no
 
-if [[ "$TOOLCHAINCMD" = "gcc" ]] ; then
+declare -r GCC_RE='gcc.*'
+if [[ "$TOOLCHAINCMD" =~ $GCC_RE ]] ; then
     while [ "$#" -gt 0 ]; do
         if [ "$1" = "-o" ]; then
             obj=$2
Signature · 18 lines
2.16.5




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B



_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 2/4 livepatch-gcc: Ignore built_in.o and prelink.o object files

Do not copy over the built_in.o and prelink.o object files when they get rebuilt as they are used for transient linking by Xen's build system.

---
 livepatch-gcc | 2 ++
 1 file changed, 2 insertions(+)
livepatch-gcc +2 -0
index 617f865..01e4b8c 100755
@@ -35,6 +35,8 @@ if [[ "$TOOLCHAINCMD" =~ $GCC_RE ]] ; then
             version.o|\
             debug.o|\
             *.xen-syms.*.o|\
+            built_in.o|\
+            prelink.o|\
             .*.o)
                 break
                 ;;
Signature · 18 lines
2.16.5




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B



_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 2/4
#2 lore raw

Re: [livepatch-build-tools 2/4] livepatch-gcc: Ignore built_in.o and prelink.o object files

Quoted 7 lines · Pawel Wieczorkiewicz

Do not copy over the built_in.o and prelink.o object files when they get rebuilt as they are used for transient linking by Xen's build system.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

PATCH 3/4 livepatch-build: Do not follow every symlink for patch file

In reply to Pawel Wieczorkiewicz

In some build systems symlinks might be used for patch file names to point from target directories to actual patches. Following those symlinks breaks naming convention as the resulting built modules would be named after the actual hardlink insteads of the symlink.

Livepatch-build obtains hotpatch name from the patch file, so it should not canonicalize the file path resolving all the symlinks to not lose the original symlink name.

---
 livepatch-build | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
livepatch-build +3 -1
index c057fa1..796838c 100755
@@ -265,7 +265,9 @@ done
 [ -z "$DEPENDS" ] && die "Build-id dependency not given"
 
 SRCDIR="$(readlink -m -- "$srcarg")"
-PATCHFILE="$(readlink -m -- "$patcharg")"
+# We need an absolute path because we move around, but we need to
+# retain the name of the symlink (= realpath -s)
+PATCHFILE="$(readlink -f "$(dirname "$patcharg")")/$(basename "$patcharg")"
 CONFIGFILE="$(readlink -m -- "$configarg")"
 OUTPUT="$(readlink -m -- "$outputarg")"
 
Signature · 18 lines
2.16.5




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B



_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

Re: [livepatch-build-tools 3/4] livepatch-build: Do not follow every symlink for patch file

Quoted 23 lines · Pawel Wieczorkiewicz

In some build systems symlinks might be used for patch file names to point from target directories to actual patches. Following those symlinks breaks naming convention as the resulting built modules would be named after the actual hardlink insteads of the symlink.

Livepatch-build obtains hotpatch name from the patch file, so it should not canonicalize the file path resolving all the symlinks to not lose the original symlink name.

---
  livepatch-build | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
livepatch-build +0 -0
index c057fa1..796838c 100755

livepatch-build done · line 273

Quoted 10 lines
livepatch-build +3 -1
@@ -265,7 +265,9 @@ done
  [ -z "$DEPENDS" ] && die "Build-id dependency not given"
 
  SRCDIR="$(readlink -m -- "$srcarg")"
-PATCHFILE="$(readlink -m -- "$patcharg")"
+# We need an absolute path because we move around, but we need to
+# retain the name of the symlink (= realpath -s)
+PATCHFILE="$(readlink -f "$(dirname "$patcharg")")/$(basename "$patcharg")"
  CONFIGFILE="$(readlink -m -- "$configarg")"
  OUTPUT="$(readlink -m -- "$outputarg")"
 

This works, but would it not be simpler to just pass $patcharg into make_patch_name()?

Signature · 6 lines
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

Re: [livepatch-build-tools 3/4] livepatch-build: Do not follow every symlink for patch file

Quoted 26 lines · Ross Lagerwall
Quoted 22 lines · Pawel Wieczorkiewicz

In some build systems symlinks might be used for patch file names to point from target directories to actual patches. Following those symlinks breaks naming convention as the resulting built modules would be named after the actual hardlink insteads of the symlink. Livepatch-build obtains hotpatch name from the patch file, so it should not canonicalize the file path resolving all the symlinks to not lose the original symlink name.

---
 livepatch-build | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
livepatch-build +0 -1
index c057fa1..796838c 100755
@@ -265,7 +265,9 @@ done
 [ -z "$DEPENDS" ] && die "Build-id dependency not given"
   SRCDIR="$(readlink -m -- "$srcarg")"
-PATCHFILE="$(readlink -m -- "$patcharg")"
Quoted 9 lines

livepatch-build done · line 273

Quoted 6 lines
livepatch-build +3 -0
+# We need an absolute path because we move around, but we need to
+# retain the name of the symlink (= realpath -s)
+PATCHFILE="$(readlink -f "$(dirname "$patcharg")")/$(basename "$patcharg")"
 CONFIGFILE="$(readlink -m -- "$configarg")"
 OUTPUT="$(readlink -m -- "$outputarg")"
 

This works, but would it not be simpler to just pass $patcharg into make_patch_name()?

No strong opinion here, but the readlink change felt less invasive (no changes to the existing semantics).

Thank you for looking into this.

Quoted 3 lines
Signature · 1 line
Ross Lagerwall

Best Regards,
Pawel Wieczorkiewicz

Amazon Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

PATCH 4/4 livepatch-build: Handle newly created object files

In reply to Pawel Wieczorkiewicz

Up to now the livepatch-build ignores newly created object files. When patch applies new .c file and augments its Makefile to build it the resulting object file is not taken into account for final linking step.

Such newly created object files can be detected by comparing patched/ and original/ directories and copied over to the output directory for the final linking step.

---
 livepatch-build | 6 ++++++
 1 file changed, 6 insertions(+)
livepatch-build +6 -0
index 796838c..b43730c 100755
@@ -146,6 +146,12 @@ function create_patch()
         fi
     done
 
+    NEW_FILES=$(comm -23 <(find patched -type f -name '*.o' | cut -f2- -d'/' | sort -u) <(find original -type f -name '*.o' | cut -f2- -d'/' | sort -u))
+    for i in $NEW_FILES; do
+        cp "patched/$i" "output/$i"
+        CHANGED=1
+    done
+
     if [[ $ERROR -ne 0 ]]; then
         die "$ERROR error(s) encountered"
     fi
Signature · 18 lines
2.16.5




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B



_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [livepatch-build-tools 4/4] livepatch-build: Handle newly created object files

Quoted 5 lines · Pawel Wieczorkiewicz

Up to now the livepatch-build ignores newly created object files. When patch applies new .c file and augments its Makefile to build it the resulting object file is not taken into account for final linking step.

Quoted 8 lines

Such newly created object files can be detected by comparing patched/ and original/ directories and copied over to the output directory for the final linking step.

This looks to resolve:

https://xenproject.atlassian.net/browse/XEN-90

(I think?)

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [livepatch-build-tools 4/4] livepatch-build: Handle newly created object files

Quoted 15 lines · Andrew Cooper
Quoted 5 lines · Pawel Wieczorkiewicz

Up to now the livepatch-build ignores newly created object files. When patch applies new .c file and augments its Makefile to build it the resulting object file is not taken into account for final linking step.

Quoted 8 lines

Such newly created object files can be detected by comparing patched/ and original/ directories and copied over to the output directory for the final linking step.

Quoted 6 lines

This looks to resolve:

https://xenproject.atlassian.net/browse/XEN-90

(I think?)

Yes, I expect it would.

Signature · 6 lines
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

Re: [livepatch-build-tools 4/4] livepatch-build: Handle newly created object files

In reply to Pawel Wieczorkiewicz
Quoted 22 lines · Pawel Wieczorkiewicz

Up to now the livepatch-build ignores newly created object files. When patch applies new .c file and augments its Makefile to build it the resulting object file is not taken into account for final linking step.

Such newly created object files can be detected by comparing patched/ and original/ directories and copied over to the output directory for the final linking step.

---
  livepatch-build | 6 ++++++
  1 file changed, 6 insertions(+)
livepatch-build +0 -0
index 796838c..b43730c 100755

livepatch-build function create_patch() · line 149

Quoted 4 lines
livepatch-build +1 -0
@@ -146,6 +146,12 @@ function create_patch()
          fi
      done
 
+    NEW_FILES=$(comm -23 <(find patched -type f -name '*.o' | cut -f2- -d'/' | sort -u) <(find original -type f -name '*.o' | cut -f2- -d'/' | sort -u))

The paths should be patched/xen and original/xen so that only hypervisor changes are processed. It is done this way earlier (see FILES="$(find xen ...)").

Since process substitution creates a subshell, it might be simpler to do <(cd patched/xen && find . ...) and then drop the cut. Also, the -u argument to sort seems unnecessary.

livepatch-build function create_patch() · line 153

Quoted 4 lines
livepatch-build +4 -0
+    for i in $NEW_FILES; do
+        cp "patched/$i" "output/$i"
+        CHANGED=1
+    done

If the live patch for whatever reason has no "patched" object files and only "new" object files, then it is not going to do anything useful since nothing will use the new functions. This should fail to build with an error. E.g. set NEW=1 above and then later check for CHANGED=0 && NEW=1.

Previously all object files that were linked into the livepatch were from the output of create-diff-object. This change just includes the newly built object files directly. I wonder if there are any issues or subtle differences when doing this? A couple of differences off the top of my head:

1) The object file will include _everything_ (e.g. potentially unused functions) while create-diff-object generally only includes the minimum that is needed.

2) Hooks and ignored functions/sections in the new object file would not be processed at all.

The was previously a patch on xen-devel which took a different approach (https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg03532.html). Perhaps you could look at it and see which approach would be better?

Thanks,

Signature · 6 lines
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

Re: [livepatch-build-tools 4/4] livepatch-build: Handle newly created object files

Quoted 20 lines · Ross Lagerwall
Quoted 16 lines · Pawel Wieczorkiewicz

Up to now the livepatch-build ignores newly created object files. When patch applies new .c file and augments its Makefile to build it the resulting object file is not taken into account for final linking step.
Such newly created object files can be detected by comparing patched/ and original/ directories and copied over to the output directory for the final linking step.

---
 livepatch-build | 6 ++++++
 1 file changed, 6 insertions(+)
livepatch-build +0 -0
index 796838c..b43730c 100755
Quoted 8 lines

livepatch-build function create_patch() · line 148

Quoted 6 lines
livepatch-build +0 -0
@@ -146,6 +146,12 @@ function create_patch()
         fi
     done
 +    NEW_FILES=$(comm -23 <(find patched -type f -name '*.o' | cut -f2- -d'/' | sort -u) <(find original -type f -name '*.o' | cut -f2- -d'/' | sort -u))

The paths should be patched/xen and original/xen so that only hypervisor changes are processed. It is done this way earlier (see FILES="$(find xen ...)").

ACK. Will fix.

Quoted 2 lines

Since process substitution creates a subshell, it might be simpler to do <(cd patched/xen && find . ...) and then drop the cut. Also, the -u argument to sort seems unnecessary.

ACK. Will fix.
The -u for sort is just my paranoia.

Quoted 8 lines

livepatch-build function create_patch() · line 153

Quoted 5 lines
livepatch-build +4 -0
+    for i in $NEW_FILES; do
+        cp "patched/$i" "output/$i"
+        CHANGED=1
+    done

If the live patch for whatever reason has no "patched" object files and only "new" object files, then it is not going to do anything useful since nothing will use the new functions. This should fail to build with an error. E.g. set NEW=1 above and then later check for CHANGED=0 && NEW=1.

I disagree here. Inline assembly patching can use new functions even without having any "patched" objects. Also hotpatch modules with hooks (functionality I will send upstream soon) can use the "new" objects provided functionality for various reasons (re-registrating a callback for instance, or some testing related activities).

Quoted 3 lines

Previously all object files that were linked into the livepatch were from the output of create-diff-object. This change just includes the newly built object files directly. I wonder if there are any issues or subtle differences when doing this? A couple of differences off the top of my head:

1) The object file will include _everything_ (e.g. potentially unused functions) while create-diff-object generally only includes the minimum that is needed.

Yes, that’s right. When a patch defines new files (and thereby new object files) it should do it wisely.

Quoted 2 lines

2) Hooks and ignored functions/sections in the new object file would not be processed at all.

Not sure I follow the point here, but hooks and functions properly defined in new object files will be processed and used during link time.

I have a test hotpatch that does that.

Quoted 2 lines

The was previously a patch on xen-devel which took a different approach (https://lists.xenproject.org/archives/html/xen-devel/2017-06/msg03532.html). Perhaps you could look at it and see which approach would be better?

Taking a look. Thanks.

Quoted 6 lines

Thanks,

Signature · 1 line
Ross Lagerwall

Best Regards,
Pawel Wieczorkiewicz

Amazon Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrer: Christian Schlaeger, Ralf Herbrich
Ust-ID: DE 289 237 879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by on v1 1/4
#11 lore raw
In reply to Pawel Wieczorkiewicz
Quoted 7 lines · Pawel Wieczorkiewicz

Xen build system may enforce particular gcc version (e.g. gcc72). Make sure the livepatch-gcc script accepts all input toolchain gcc commands with or without version specified.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel