Skip to contentSkip to conversation

kprobes/x86: Fix to copy RIP relative instruction correctly

PATCH In review Masami Hiramatsu 4 participants 7 messages Aug 23, 2018 – Dec 4, 2018 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kprobes/x86: Fix to copy RIP relative instruction correctly

Since copy_optimized_instructions() misses to update real RIP address while copying several instructions to working buffer, it adjusts RIP-relative instruction with wrong RIP address for the 2nd and subsequent instructions.

This may break the kernel (like kernel freeze) because probed instruction can refer a wrong data. For example, putting kprobes on cpumask_next hit this bug.

cpumask_next is normally like below if CONFIG_CPUMASK_OFFSTACK=y (in this case nr_cpumask_bits is an alias of nr_cpu_ids)

<cpumask_next>:

       48 89 f0                mov    %rsi,%rax
       8b 35 7b fb e2 00       mov    0xe2fb7b(%rip),%esi
        # ffffffff82db9e64 <nr_cpu_ids>
       55                      push   %rbp
...

If we put a kprobe on it and optimized with jump, it becomes like this.

	e9 95 7d 07 1e	jmpq   0xffffffffa000207a
	7b fb	jnp    0xffffffff81f8a2e2 <cpumask_next+2>
	e2 00	loop   0xffffffff81f8a2e9 <cpumask_next+9>
	55	push   %rbp

This shows first 2 "mov" instructions are copied to trampoline buffer at 0xffffffffa000207a. Here is the disassembled result. (skipped optprobe template instructions)

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +2 -1
index eaf02f2e7300..e92672b8b490 100644
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);
 		if (!ret || !can_boost(&insn, src + len))
 			return -EINVAL;
 		len += ret;
Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · On Fri, 24 Aug 2018 02:16:12 +0900 Masami Hiramatsu <[email protected]> wrote:

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>
Quoted 3 lines

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Ouch! Nice catch.

Quoted 17 lines

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +0 -0
index eaf02f2e7300..e92672b8b490 100644

arch/x86/kernel/kprobes/opt.c static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real) · line 196

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -1
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);
 		if (!ret || !can_boost(&insn, src + len))
 			return -EINVAL;
 		len += ret;

Looking at the change that broke this we have:

arch/x86/kernel/kprobes/opt.c

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-static int copy_optimized_instructions(u8 *dest, u8 *src)
+static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 {
        struct insn insn;
        int len = 0, ret;
 
        while (len < RELATIVEJUMP_SIZE) {
-               ret = __copy_instruction(dest + len, src + len, &insn);
+               ret = __copy_instruction(dest + len, src + len, real, &insn);

Where "real" was added as a parameter to __copy_instruction. Note that we pass in "dest + len" but not "real + len" as you patch fixes. __copy_instruction was changed by the bad commit with:

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 391

Quoted 11 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
+int __copy_instruction(u8 *dest, u8 *src, u8 *real, struct insn *insn)
 {
        kprobe_opcode_t buf[MAX_INSN_SIZE];
        unsigned long recovered_insn =
@@ -387,11 +388,11 @@ int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
                 * have given.
                 */
                newdisp = (u8 *) src + (s64) insn->displacement.value
-                         - (u8 *) dest;
+                         - (u8 *) real;

"real" replaces "dest", which was the first parameter to __copy_instruction.

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 392

Quoted 1 line
arch/x86/kernel/kprobes/opt.c +0 -0
                        return 0;

And:

Quoted 28 lines
arch/x86/kernel/kprobes/opt.c +2 -2
 int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,
                                  struct kprobe *__unused)
 {
-       u8 *buf;
-       int ret;
+       u8 *buf = NULL, *slot;
+       int ret, len;
        long rel;
 
        if (!can_optimize((unsigned long)op->kp.addr))
                return -EILSEQ;
arch/x86/kernel/kprobes/opt.c +8 -2
-       op->optinsn.insn = get_optinsn_slot();
-       if (!op->optinsn.insn)
+       buf = kzalloc(MAX_OPTINSN_SIZE, GFP_KERNEL);
+       if (!buf)
                return -ENOMEM;
 
+       op->optinsn.insn = slot = get_optinsn_slot();
+       if (!slot) {
+               ret = -ENOMEM;
+               goto out;
+       }
+
        /*
         * Verify if the address gap is in 2GB range, because this uses
         * a relative jump.
         */

arch/x86/kernel/kprobes/opt.c

Quoted 10 lines
arch/x86/kernel/kprobes/opt.c +3 -4
-       rel = (long)op->optinsn.insn - (long)op->kp.addr + RELATIVEJUMP_SIZE;
+       rel = (long)slot - (long)op->kp.addr + RELATIVEJUMP_SIZE;
        if (abs(rel) > 0x7fffffff) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return -ERANGE;
+               ret = -ERANGE;
+               goto err;
        }
 
-       buf = (u8 *)op->optinsn.insn;

"slot" is equivalent to the old "buf".

arch/x86/kernel/kprobes/opt.c

Quoted 12 lines
arch/x86/kernel/kprobes/opt.c +4 -6
-       set_memory_rw((unsigned long)buf & PAGE_MASK, 1);
+       /* Copy arch-dep-instance from template */
+       memcpy(buf, &optprobe_template_entry, TMPL_END_IDX);
 
        /* Copy instructions into the out-of-line buffer */
-       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr);
-       if (ret < 0) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return ret;
-       }
+       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr,
+                                         slot + TMPL_END_IDX);

We pass in "real" as "slot + TMPL_END_IDX" and "dest" as "buf + TMPL_END_IDX", thus to make it be equivalent to the code before this commit, "real" should have "+ len" added to it in order to be equivalent to what was there before.

That said...

-- Steve

Quoted 2 lines
arch/x86/kernel/kprobes/opt.c +2 -0
+       if (ret < 0)
+               goto err;
#2 lore raw
Quoted 133 lines · On Thu, 23 Aug 2018 21:41:09 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 9 lines · On Fri, 24 Aug 2018 02:16:12 +0900 Masami Hiramatsu <[email protected]> wrote:

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>
Quoted 3 lines

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Ouch! Nice catch.

Quoted 17 lines

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +0 -0
index eaf02f2e7300..e92672b8b490 100644

arch/x86/kernel/kprobes/opt.c static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real) · line 196

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -1
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);
 		if (!ret || !can_boost(&insn, src + len))
 			return -EINVAL;
 		len += ret;

Looking at the change that broke this we have:

arch/x86/kernel/kprobes/opt.c

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-static int copy_optimized_instructions(u8 *dest, u8 *src)
+static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 {
        struct insn insn;
        int len = 0, ret;
 
        while (len < RELATIVEJUMP_SIZE) {
-               ret = __copy_instruction(dest + len, src + len, &insn);
+               ret = __copy_instruction(dest + len, src + len, real, &insn);

Where "real" was added as a parameter to __copy_instruction. Note that we pass in "dest + len" but not "real + len" as you patch fixes. __copy_instruction was changed by the bad commit with:

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 391

Quoted 11 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
+int __copy_instruction(u8 *dest, u8 *src, u8 *real, struct insn *insn)
 {
        kprobe_opcode_t buf[MAX_INSN_SIZE];
        unsigned long recovered_insn =
@@ -387,11 +388,11 @@ int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
                 * have given.
                 */
                newdisp = (u8 *) src + (s64) insn->displacement.value
-                         - (u8 *) dest;
+                         - (u8 *) real;

"real" replaces "dest", which was the first parameter to __copy_instruction.

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 392

Quoted 1 line
arch/x86/kernel/kprobes/opt.c +0 -0
                        return 0;

And:

Quoted 28 lines
arch/x86/kernel/kprobes/opt.c +2 -2
 int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,
                                  struct kprobe *__unused)
 {
-       u8 *buf;
-       int ret;
+       u8 *buf = NULL, *slot;
+       int ret, len;
        long rel;
 
        if (!can_optimize((unsigned long)op->kp.addr))
                return -EILSEQ;
arch/x86/kernel/kprobes/opt.c +8 -2
-       op->optinsn.insn = get_optinsn_slot();
-       if (!op->optinsn.insn)
+       buf = kzalloc(MAX_OPTINSN_SIZE, GFP_KERNEL);
+       if (!buf)
                return -ENOMEM;
 
+       op->optinsn.insn = slot = get_optinsn_slot();
+       if (!slot) {
+               ret = -ENOMEM;
+               goto out;
+       }
+
        /*
         * Verify if the address gap is in 2GB range, because this uses
         * a relative jump.
         */

arch/x86/kernel/kprobes/opt.c

Quoted 10 lines
arch/x86/kernel/kprobes/opt.c +3 -4
-       rel = (long)op->optinsn.insn - (long)op->kp.addr + RELATIVEJUMP_SIZE;
+       rel = (long)slot - (long)op->kp.addr + RELATIVEJUMP_SIZE;
        if (abs(rel) > 0x7fffffff) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return -ERANGE;
+               ret = -ERANGE;
+               goto err;
        }
 
-       buf = (u8 *)op->optinsn.insn;

"slot" is equivalent to the old "buf".

arch/x86/kernel/kprobes/opt.c

Quoted 12 lines
arch/x86/kernel/kprobes/opt.c +4 -6
-       set_memory_rw((unsigned long)buf & PAGE_MASK, 1);
+       /* Copy arch-dep-instance from template */
+       memcpy(buf, &optprobe_template_entry, TMPL_END_IDX);
 
        /* Copy instructions into the out-of-line buffer */
-       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr);
-       if (ret < 0) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return ret;
-       }
+       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr,
+                                         slot + TMPL_END_IDX);
Quoted 4 lines

We pass in "real" as "slot + TMPL_END_IDX" and "dest" as "buf + TMPL_END_IDX", thus to make it be equivalent to the code before this commit, "real" should have "+ len" added to it in order to be equivalent to what was there before.

Right! The broken commit splits trampoline buffer into "temporary" destination buffer and "real" trampoline buffer, and use the "real" address for RIP-relative adjustment. However, I forgot to introduce update the "real" address in the copying loop.

Quoted 4 lines

That said...

Thanks!

Signature · 1 line
Masami Hiramatsu <[email protected]>
#3 lore raw

Hi Ingo,

Could you pick this fix to urgent branch?

Thank you,

Quoted 157 lines · On Fri, 24 Aug 2018 16:57:19 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 133 lines · On Thu, 23 Aug 2018 21:41:09 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 9 lines · On Fri, 24 Aug 2018 02:16:12 +0900 Masami Hiramatsu <[email protected]> wrote:

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>
Quoted 3 lines

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Ouch! Nice catch.

Quoted 17 lines

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +0 -0
index eaf02f2e7300..e92672b8b490 100644

arch/x86/kernel/kprobes/opt.c static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real) · line 196

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -1
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);
 		if (!ret || !can_boost(&insn, src + len))
 			return -EINVAL;
 		len += ret;

Looking at the change that broke this we have:

arch/x86/kernel/kprobes/opt.c

Quoted 9 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-static int copy_optimized_instructions(u8 *dest, u8 *src)
+static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 {
        struct insn insn;
        int len = 0, ret;
 
        while (len < RELATIVEJUMP_SIZE) {
-               ret = __copy_instruction(dest + len, src + len, &insn);
+               ret = __copy_instruction(dest + len, src + len, real, &insn);

Where "real" was added as a parameter to __copy_instruction. Note that we pass in "dest + len" but not "real + len" as you patch fixes. __copy_instruction was changed by the bad commit with:

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 391

Quoted 11 lines
arch/x86/kernel/kprobes/opt.c +2 -2
-int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
+int __copy_instruction(u8 *dest, u8 *src, u8 *real, struct insn *insn)
 {
        kprobe_opcode_t buf[MAX_INSN_SIZE];
        unsigned long recovered_insn =
@@ -387,11 +388,11 @@ int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
                 * have given.
                 */
                newdisp = (u8 *) src + (s64) insn->displacement.value
-                         - (u8 *) dest;
+                         - (u8 *) real;

"real" replaces "dest", which was the first parameter to __copy_instruction.

arch/x86/kernel/kprobes/opt.c int __copy_instruction(u8 *dest, u8 *src, struct insn *insn) · line 392

Quoted 1 line
arch/x86/kernel/kprobes/opt.c +0 -0
                        return 0;

And:

Quoted 28 lines
arch/x86/kernel/kprobes/opt.c +2 -2
 int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,
                                  struct kprobe *__unused)
 {
-       u8 *buf;
-       int ret;
+       u8 *buf = NULL, *slot;
+       int ret, len;
        long rel;
 
        if (!can_optimize((unsigned long)op->kp.addr))
                return -EILSEQ;
arch/x86/kernel/kprobes/opt.c +8 -2
-       op->optinsn.insn = get_optinsn_slot();
-       if (!op->optinsn.insn)
+       buf = kzalloc(MAX_OPTINSN_SIZE, GFP_KERNEL);
+       if (!buf)
                return -ENOMEM;
 
+       op->optinsn.insn = slot = get_optinsn_slot();
+       if (!slot) {
+               ret = -ENOMEM;
+               goto out;
+       }
+
        /*
         * Verify if the address gap is in 2GB range, because this uses
         * a relative jump.
         */

arch/x86/kernel/kprobes/opt.c

Quoted 10 lines
arch/x86/kernel/kprobes/opt.c +3 -4
-       rel = (long)op->optinsn.insn - (long)op->kp.addr + RELATIVEJUMP_SIZE;
+       rel = (long)slot - (long)op->kp.addr + RELATIVEJUMP_SIZE;
        if (abs(rel) > 0x7fffffff) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return -ERANGE;
+               ret = -ERANGE;
+               goto err;
        }
 
-       buf = (u8 *)op->optinsn.insn;

"slot" is equivalent to the old "buf".

arch/x86/kernel/kprobes/opt.c

Quoted 12 lines
arch/x86/kernel/kprobes/opt.c +4 -6
-       set_memory_rw((unsigned long)buf & PAGE_MASK, 1);
+       /* Copy arch-dep-instance from template */
+       memcpy(buf, &optprobe_template_entry, TMPL_END_IDX);
 
        /* Copy instructions into the out-of-line buffer */
-       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr);
-       if (ret < 0) {
-               __arch_remove_optimized_kprobe(op, 0);
-               return ret;
-       }
+       ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr,
+                                         slot + TMPL_END_IDX);
Quoted 4 lines

We pass in "real" as "slot + TMPL_END_IDX" and "dest" as "buf + TMPL_END_IDX", thus to make it be equivalent to the code before this commit, "real" should have "+ len" added to it in order to be equivalent to what was there before.

Right! The broken commit splits trampoline buffer into "temporary" destination buffer and "real" trampoline buffer, and use the "real" address for RIP-relative adjustment. However, I forgot to introduce update the "real" address in the copying loop.

Quoted 4 lines

That said...

Thanks!

Signature · 1 line
Masami Hiramatsu <[email protected]>
Signature · 1 line
Masami Hiramatsu <[email protected]>
#4 lore raw
In reply to Masami Hiramatsu
Quoted 60 lines · Masami Hiramatsu

Since copy_optimized_instructions() misses to update real RIP address while copying several instructions to working buffer, it adjusts RIP-relative instruction with wrong RIP address for the 2nd and subsequent instructions.

This may break the kernel (like kernel freeze) because probed instruction can refer a wrong data. For example, putting kprobes on cpumask_next hit this bug.

cpumask_next is normally like below if CONFIG_CPUMASK_OFFSTACK=y (in this case nr_cpumask_bits is an alias of nr_cpu_ids)

<cpumask_next>:

       48 89 f0                mov    %rsi,%rax
       8b 35 7b fb e2 00       mov    0xe2fb7b(%rip),%esi
        # ffffffff82db9e64 <nr_cpu_ids>
       55                      push   %rbp
...

If we put a kprobe on it and optimized with jump, it becomes like this.

	e9 95 7d 07 1e	jmpq   0xffffffffa000207a
	7b fb	jnp    0xffffffff81f8a2e2 <cpumask_next+2>
	e2 00	loop   0xffffffff81f8a2e9 <cpumask_next+9>
	55	push   %rbp

This shows first 2 "mov" instructions are copied to trampoline buffer at 0xffffffffa000207a. Here is the disassembled result. (skipped optprobe template instructions)

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +0 -0
index eaf02f2e7300..e92672b8b490 100644

arch/x86/kernel/kprobes/opt.c static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real) · line 193

Quoted 6 lines
arch/x86/kernel/kprobes/opt.c +2 -1
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);

I applied this, except that I unbroke the line: please ignore checkpatch in such cases where the cure is worse than the disease ...

I.e. even if slightly over 80 cols, this is more readable:

		ret = __copy_instruction(dest + len, src + len, real + len, &insn);

Thanks,

Ingo

#5 lore raw
Quoted 65 lines · On Tue, 4 Dec 2018 09:13:35 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 62 lines · Masami Hiramatsu

Since copy_optimized_instructions() misses to update real RIP address while copying several instructions to working buffer, it adjusts RIP-relative instruction with wrong RIP address for the 2nd and subsequent instructions.

This may break the kernel (like kernel freeze) because probed instruction can refer a wrong data. For example, putting kprobes on cpumask_next hit this bug.

cpumask_next is normally like below if CONFIG_CPUMASK_OFFSTACK=y (in this case nr_cpumask_bits is an alias of nr_cpu_ids)

<cpumask_next>:

       48 89 f0                mov    %rsi,%rax
       8b 35 7b fb e2 00       mov    0xe2fb7b(%rip),%esi
        # ffffffff82db9e64 <nr_cpu_ids>
       55                      push   %rbp
...

If we put a kprobe on it and optimized with jump, it becomes like this.

	e9 95 7d 07 1e	jmpq   0xffffffffa000207a
	7b fb	jnp    0xffffffff81f8a2e2 <cpumask_next+2>
	e2 00	loop   0xffffffff81f8a2e9 <cpumask_next+9>
	55	push   %rbp

This shows first 2 "mov" instructions are copied to trampoline buffer at 0xffffffffa000207a. Here is the disassembled result. (skipped optprobe template instructions)

Dump of assembler code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54	push   %rsp
...
	48 83 c4 08	add    $0x8,%rsp
	9d	popfq
	48 89 f0	mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi
        # 0xffffffff82db9e67 <nr_cpu_ids+3>

As it shows, the 2nd mov accesses *(nr_cpu_ids+3) instead of *nr_cpu_ids. This leads a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ended.

Fixing this by adding len correctly to real RIP address while copying.

Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +0 -0
index eaf02f2e7300..e92672b8b490 100644
@@ -189,7 +189,8 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
Quoted 7 lines

arch/x86/kernel/kprobes/opt.c static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real) · line 193

Quoted 4 lines
arch/x86/kernel/kprobes/opt.c +2 -1
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len,
+			&insn);

I applied this, except that I unbroke the line: please ignore checkpatch in such cases where the cure is worse than the disease ...

Thanks Ingo!

Quoted 4 lines

I.e. even if slightly over 80 cols, this is more readable:

		ret = __copy_instruction(dest + len, src + len, real + len, &insn);

Got it.

BTW, this ugry "+ len" repeat would be avoided. I would better make a new inline function to wrap it.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#6 lore raw

[tip:perf/urgent] kprobes/x86: Fix instruction patching corruption when copying more than one RIP-relative instruction

In reply to Masami Hiramatsu
kprobes/x86: Fix instruction patching corruption when copying more than one RIP-relative instruction
Commit-ID:  43a1b0cb4cd6dbfd3cd9c10da663368394d299d8
Gitweb:     https://git.kernel.org/tip/43a1b0cb4cd6dbfd3cd9c10da663368394d299d8
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Fri, 24 Aug 2018 02:16:12 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 4 Dec 2018 09:35:20 +0100

kprobes/x86: Fix instruction patching corruption when copying more than one RIP-relative instruction

After copy_optimized_instructions() copies several instructions to the working buffer it tries to fix up the real RIP address, but it adjusts the RIP-relative instruction with an incorrect RIP address for the 2nd and subsequent instructions due to a bug in the logic.

This will break the kernel pretty badly (with likely outcomes such as a kernel freeze, a crash, or worse) because probed instructions can refer to the wrong data.

For example putting kprobes on cpumask_next() typically hits this bug.

cpumask_next() is normally like below if CONFIG_CPUMASK_OFFSTACK=y (in this case nr_cpumask_bits is an alias of nr_cpu_ids):

<cpumask_next>:

	48 89 f0		mov    %rsi,%rax
	8b 35 7b fb e2 00	mov    0xe2fb7b(%rip),%esi # ffffffff82db9e64 <nr_cpu_ids>
	55			push   %rbp

...

If we put a kprobe on it and it gets jump-optimized, it gets patched by the kprobes code like this:

<cpumask_next>:

	e9 95 7d 07 1e		jmpq   0xffffffffa000207a
	7b fb			jnp    0xffffffff81f8a2e2 <cpumask_next+2>
	e2 00			loop   0xffffffff81f8a2e9 <cpumask_next+9>
	55			push   %rbp

This shows that the first two MOV instructions were copied to a trampoline buffer at 0xffffffffa000207a.

Here is the disassembled result of the trampoline, skipping the optprobe template instructions:

	# Dump of assembly code from 0xffffffffa000207a to 0xffffffffa00020ea:

	54			push   %rsp
	...
	48 83 c4 08		add    $0x8,%rsp
	9d			popfq
	48 89 f0		mov    %rsi,%rax
	8b 35 82 7d db e2	mov    -0x1d24827e(%rip),%esi # 0xffffffff82db9e67 <nr_cpu_ids+3>

This dump shows that the second MOV accesses *(nr_cpu_ids+3) instead of the original *nr_cpu_ids. This leads to a kernel freeze because cpumask_next() always returns 0 and for_each_cpu() never ends.

Fix this by adding 'len' correctly to the real RIP address while copying.

[ mingo: Improved the changelog. ]

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ravi Bangoria <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc [email protected] # v4.15+
Fixes 63fef14fc98a ("kprobes/x86: Make insn buffer always ROX and use text_poke()")
---
 arch/x86/kernel/kprobes/opt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/kprobes/opt.c +1 -1
index 40b16b270656..6adf6e6c2933 100644
@@ -189,7 +189,7 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 	int len = 0, ret;
 
 	while (len < RELATIVEJUMP_SIZE) {
-		ret = __copy_instruction(dest + len, src + len, real, &insn);
+		ret = __copy_instruction(dest + len, src + len, real + len, &insn);
 		if (!ret || !can_boost(&insn, src + len))
 			return -EINVAL;
 		len += ret;