Problems with livepatching

PATCH In review M A Young 5 participants 9 messages Nov 23, 2016 – Nov 24, 2016

Share & tools

M A Young

author

2016-11-23 23:06 UTC #0 lore raw

I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?

Michael Young

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Andrew Cooper

2016-11-23 23:35 UTC #1 lore raw

Quoted 5 lines · M A Young

I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output

Quoted 8 lines

Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?

This looks like a problem generating the livepatch itself, not the livepatching mechanism.

Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.

With that one bug fixed, I can confirm that all of the recent XSAs are properly livepatchable with the available tools, and confirmed to work.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

M A Young

author

2016-11-24 00:13 UTC #2 lore raw

Quoted 15 lines · Andrew Cooper

Quoted 5 lines · M A Young
I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Quoted 8 lines
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?

Quoted 8 lines

This looks like a problem generating the livepatch itself, not the livepatching mechanism.
Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.

Where is this latest version available from? I had checked the git repo on xenbits
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
but the last commit was 4 months ago.

Michael Young

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2016-11-24 01:08 UTC #3 lore raw

Quoted 26 lines · M A Young

Quoted 15 lines · Andrew Cooper
Quoted 5 lines · M A Young
I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Quoted 8 lines
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?
Quoted 8 lines
This looks like a problem generating the livepatch itself, not the livepatching mechanism.
Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.

Quoted 4 lines

Where is this latest version available from? I had checked the git repo on xenbits
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
but the last commit was 4 months ago.

Here is the patch that Ross has been working on - it is being first reviewed for merge in the upstream version of kpatch. Hence hasn't been posted. But posting it here.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Konrad Rzeszutek Wilk

2016-11-24 01:08 UTC #4 lore raw

PATCH 1/1 create-diff-object: Update fixup offsets in .rela.ex_table

From: Ross Lagerwall <[email protected]>

When pruning entries from the fixup table, update the offsets in .rela.ex_table otherwise the relas might point to the wrong fixup entry or even out of the .fixup section.

Signed-off-by Ross Lagerwall <[email protected]>

Signed-off-by Konrad Rzeszutek Wilk <[email protected]>

---
v2: Squashed fix from kpatch upstream
[see https://github.com/dynup/kpatch/pull/616]

---
 create-diff-object.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

create-diff-object.c +41 -0

index 69bcd88..58f7c4c 100644
@@ -1057,6 +1057,34 @@ static int should_keep_rela_group(struct section *sec, int start, int size)
 	return found;
 }
 
+/*
+ * When updating .fixup, the corresponding addends in .ex_table need to be
+ * updated too. Stash the result in rela.r_addend so that the calculation in
+ * fixup_group_size() is not affected.
+ */
+void kpatch_update_ex_table_addend(struct kpatch_elf *kelf,
+				   struct special_section *special,
+				   int src_offset, int dest_offset,
+				   int group_size)
+{
+	struct rela *rela;
+	struct section *sec;
+
+	if (strcmp(special->name, ".fixup"))
+		return;
+
+	sec = find_section_by_name(&kelf->sections, ".rela.ex_table");
+	if (!sec)
+		ERROR("missing .rela.ex_table section");
+
+	list_for_each_entry(rela, &sec->relas, list) {
+		if (!strcmp(rela->sym->name, ".fixup") &&
+		    rela->addend >= src_offset &&
+		    rela->addend < src_offset + group_size)
+			rela->rela.r_addend = rela->addend - (src_offset - dest_offset);
+	}
+}
+
 static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				              struct special_section *special,
 				              struct section *sec)

@@ -1073,6 +1101,14 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 	if (!dest)
 		ERROR("malloc");
 
+	/* Restore the stashed r_addend from kpatch_update_ex_table_addend. */
+	if (!strcmp(special->name, ".ex_table")) {
+		list_for_each_entry(rela, &sec->relas, list) {
+			if (!strcmp(rela->sym->name, ".fixup"))
+				rela->addend = rela->rela.r_addend;
+		}
+	}
+
 	group_size = 0;
 	src_offset = 0;
 	dest_offset = 0;

@@ -1100,6 +1136,11 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				rela->rela.r_offset = rela->offset;
 
 				rela->sym->include = 1;
+
+				kpatch_update_ex_table_addend(kelf, special,
+							      src_offset,
+							      dest_offset,
+							      group_size);
 			}
 		}

Signature · 7 lines

2.9.3


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Ross Lagerwall

2016-11-24 08:05 UTC #5 lore raw

Re: [PATCH LIVEPATCH-BUILD-TOOLS] create-diff-object: Update fixup offsets in .rela.ex_table

Quoted 28 lines · Konrad Rzeszutek Wilk

From: Ross Lagerwall <[email protected]>
When pruning entries from the fixup table, update the offsets in .rela.ex_table otherwise the relas might point to the wrong fixup entry or even out of the .fixup section.
Signed-off-by Ross Lagerwall <[email protected]>
Signed-off-by Konrad Rzeszutek Wilk <[email protected]>
---
v2: Squashed fix from kpatch upstream
[see https://github.com/dynup/kpatch/pull/616]
---
 create-diff-object.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
create-diff-object.c +5 -0
index 69bcd88..58f7c4c 100644
@@ -1057,6 +1057,34 @@ static int should_keep_rela_group(struct section *sec, int start, int size)
 	return found;
 }
 
+/*
+ * When updating .fixup, the corresponding addends in .ex_table need to be
+ * updated too. Stash the result in rela.r_addend so that the calculation in
+ * fixup_group_size() is not affected.
+ */

create-diff-object.c static int should_keep_rela_group(struct section *sec, int start, int size) · line 1074

Quoted 10 lines

create-diff-object.c +10 -0

+void kpatch_update_ex_table_addend(struct kpatch_elf *kelf,
+				   struct special_section *special,
+				   int src_offset, int dest_offset,
+				   int group_size)
+{
+	struct rela *rela;
+	struct section *sec;
+
+	if (strcmp(special->name, ".fixup"))
+		return;

As suggested by kpatch upstream, the code would be clearer if the above check is in kpatch_regenerate_special_section() rather than this function.

I'll update and resend.

Quoted 44 lines

create-diff-object.c +26 -0

+
+	sec = find_section_by_name(&kelf->sections, ".rela.ex_table");
+	if (!sec)
+		ERROR("missing .rela.ex_table section");
+
+	list_for_each_entry(rela, &sec->relas, list) {
+		if (!strcmp(rela->sym->name, ".fixup") &&
+		    rela->addend >= src_offset &&
+		    rela->addend < src_offset + group_size)
+			rela->rela.r_addend = rela->addend - (src_offset - dest_offset);
+	}
+}
+
 static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				              struct special_section *special,
 				              struct section *sec)

@@ -1073,6 +1101,14 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 	if (!dest)
 		ERROR("malloc");
 
+	/* Restore the stashed r_addend from kpatch_update_ex_table_addend. */
+	if (!strcmp(special->name, ".ex_table")) {
+		list_for_each_entry(rela, &sec->relas, list) {
+			if (!strcmp(rela->sym->name, ".fixup"))
+				rela->addend = rela->rela.r_addend;
+		}
+	}
+
 	group_size = 0;
 	src_offset = 0;
 	dest_offset = 0;

@@ -1100,6 +1136,11 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				rela->rela.r_offset = rela->offset;
 
 				rela->sym->include = 1;
+
+				kpatch_update_ex_table_addend(kelf, special,
+							      src_offset,
+							      dest_offset,
+							      group_size);
 			}
 		}

Signature · 6 lines

Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Ross Lagerwall

2016-11-24 08:52 UTC #6 lore raw

In reply to Konrad Rzeszutek Wilk

Quoted 27 lines · Konrad Rzeszutek Wilk

Quoted 26 lines · M A Young
Quoted 23 lines · Andrew Cooper
Quoted 5 lines · M A Young
I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Quoted 8 lines
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?
This looks like a problem generating the livepatch itself, not the livepatching mechanism.
Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.

Quoted 9 lines

Quoted 4 lines
Where is this latest version available from? I had checked the git repo on xenbits
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
but the last commit was 4 months ago.
Here is the patch that Ross has been working on - it is being first reviewed for merge in the upstream version of kpatch. Hence hasn't been posted. But posting it here.

I've also got a patch to fix the build with GCC 6 which I shall send out shortly. This should fix the 'Unknown symbol' issue.

Signature · 6 lines

Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Lars Kurth

2016-11-24 09:24 UTC #7 lore raw

In reply to Konrad Rzeszutek Wilk

Quoted 30 lines · Konrad Rzeszutek Wilk

Quoted 26 lines · M A Young
Quoted 23 lines · Andrew Cooper
Quoted 5 lines · M A Young
I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Quoted 8 lines
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?
This looks like a problem generating the livepatch itself, not the livepatching mechanism.
Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.

Quoted 8 lines

Quoted 4 lines
Where is this latest version available from? I had checked the git repo on xenbits
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
but the last commit was 4 months ago.
Here is the patch that Ross has been working on - it is being first reviewed for merge in the upstream version of kpatch. Hence hasn't been posted. But posting it here.

I think the link is missing.
Lars
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Ross Lagerwall

2016-11-24 09:55 UTC #8 lore raw

Quoted 35 lines · Lars Kurth

Quoted 35 lines · Konrad Rzeszutek Wilk
Quoted 30 lines · M A Young
Quoted 15 lines · Andrew Cooper
Quoted 5 lines · M A Young
I have been experimenting with live patching with the recent batch of security updates on Fedora xen with very limited success. I had most attempts with a test build of xen-4.8.0-rc6, and of the updates I have tried only xsa192.patch uploads successfully. For example with xsa191.patch the upload fails with the output
Quoted 8 lines
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes) Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such file or directory)!
and in xl dmesg a long debugging output ends with the line (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0 with a similar line (mostly with .LC0 but with .LC3 in one case) for the other failed attempts. Am I doing something wrong or is there a problem with live patching in this case?
Quoted 8 lines
This looks like a problem generating the livepatch itself, not the livepatching mechanism.
Make sure you are completely up to date with the livepatch tools userspace. There was one bug in livepatch generation which was discovered due to XSA-191 and fixed (actually a preexisting bug even in Linux), but its symptoms were innocuous until you patched, at which vcpu context switch blew up.
Where is this latest version available from? I had checked the git repo on xenbits
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
but the last commit was 4 months ago.

Quoted 7 lines

Quoted 3 lines
Here is the patch that Ross has been working on - it is being first reviewed for merge in the upstream version of kpatch. Hence hasn't been posted. But posting it here.
I think the link is missing.
Lars

Konrad posted the patch to xen-devel.

For what it's worth, I sometimes push changes that I'm working on to a branch at https://github.com/rosslagerwall/livepatch-build-tools

Signature · 6 lines

Ross Lagerwall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel