Strange hang with gcc 8 of kprobe multiple_kprobes test
Hi Masami,
I started testing some of my new code and the system got into a strange state. Debugging further, I found the cause came from the kprobe tests. It became stranger to me that I could reproduce it with older kernels. I went back as far as 4.16 and it triggered. I thought this very strange because I ran this test on all those kernels in the past.
After a bit of hair pulling, I figured out what changed. I upgraded to gcc 8.1 (and I reproduce it with 8.2 as well). I convert back to gcc 7 and the tests pass without issue.
The issue that I notice when the system gets into this strange state is that I can't log into the box. Nor can I reboot. Basically it's anything to do with systemd just doesn't work (insert your jokes here now, and then let's move on).
I was able to narrow down what the exact function was that caused the issues and it is: update_vsyscall()
gcc 7 looks like this:
ffffffff81004bf0 <update_vsyscall>:ffffffff81004bf0: e8 0b cc 9f 00 callq ffffffff81a01800 <__fentry__>
ffffffff81004bf1: R_X86_64_PC32 __fentry__-0x4
ffffffff81004bf5: 48 8b 07 mov (%rdi),%rax
ffffffff81004bf8: 8b 15 96 5f 34 01 mov 0x1345f96(%rip),%edx # ffffffff8234ab94 <vclocks_used>
ffffffff81004bfa: R_X86_64_PC32 vclocks_used-0x4
ffffffff81004bfe: 83 05 7b 84 6f 01 01 addl $0x1,0x16f847b(%rip) # ffffffff826fd080 <vsyscall_gtod_data>
ffffffff81004c00: R_X86_64_PC32 vsyscall_gtod_data-0x5
ffffffff81004c05: 8b 48 24 mov 0x24(%rax),%ecx
ffffffff81004c08: b8 01 00 00 00 mov $0x1,%eax
ffffffff81004c0d: d3 e0 shl %cl,%eaxAnd gcc 8 looks like this:
ffffffff81004c90 <update_vsyscall>:ffffffff81004c90: e8 6b cb 9f 00 callq ffffffff81a01800 <__fentry__>
ffffffff81004c91: R_X86_64_PC32 __fentry__-0x4
ffffffff81004c95: 48 8b 07 mov (%rdi),%rax
ffffffff81004c98: 83 05 e1 93 6f 01 01 addl $0x1,0x16f93e1(%rip) # ffffffff826fe080 <vsyscall_gtod_data>
ffffffff81004c9a: R_X86_64_PC32 vsyscall_gtod_data-0x5
ffffffff81004c9f: 8b 50 24 mov 0x24(%rax),%edx
ffffffff81004ca2: 8b 05 ec 5e 34 01 mov 0x1345eec(%rip),%eax # ffffffff8234ab94 <vclocks_used>
ffffffff81004ca4: R_X86_64_PC32 vclocks_used-0x4The test adds a kprobe (optimized) at udpate_vsyscall+5. And will insert a jump on the two instructions after fentry. The difference between v7 and v8 is that v7 is touching vclocks_used and v8 is touching vsyscall_gtod_data.
Is there some black magic going on with the vsyscall area with vsyscall_gtod_data that is causing havoc when a kprobe is added there?
I can dig a little more into this, but I'm currently at my HQ office with a lot of other objectives that I must get done, and I can't work on this much more this week.
I included my config (for my virt machine, which I was also able to trigger it with).
The test that triggers this bug is:
tools/testing/selftests/ftrace/test.d/kprobe/multiple_kprobes.tc
It runs the test fine, but other things just start to act up after I run it.
I notice that when I get into the state, journald and the dbus_daemon are constantly running. Perhaps the userspace time keeping went bad?
-- Steve
Hi Steve,
Quoted 9 lines · On Mon, 3 Dec 2018 21:18:07 -0500 Steven Rostedt <[email protected]> wrote:
Hi Masami,
I started testing some of my new code and the system got into a strange state. Debugging further, I found the cause came from the kprobe tests. It became stranger to me that I could reproduce it with older kernels. I went back as far as 4.16 and it triggered. I thought this very strange because I ran this test on all those kernels in the past.
Quoted 3 lines
After a bit of hair pulling, I figured out what changed. I upgraded to gcc 8.1 (and I reproduce it with 8.2 as well). I convert back to gcc 7 and the tests pass without issue.
OK, let me see.
Quoted 22 lines
The issue that I notice when the system gets into this strange state is that I can't log into the box. Nor can I reboot. Basically it's anything to do with systemd just doesn't work (insert your jokes here now, and then let's move on).
I was able to narrow down what the exact function was that caused the issues and it is:
update_vsyscall()gcc 7 looks like this:
ffffffff81004bf0<update_vsyscall>:ffffffff81004bf0: e8 0b cc 9f 00 callqffffffff81a01800<__fentry__>ffffffff81004bf1: R_X86_64_PC32 __fentry__-0x4 ffffffff81004bf5: 48 8b 07 mov (%rdi),%rax ffffffff81004bf8: 8b 15 96 5f 34 01 mov 0x1345f96(%rip),%edx # ffffffff8234ab94 <vclocks_used> ffffffff81004bfa: R_X86_64_PC32 vclocks_used-0x4 ffffffff81004bfe: 83 05 7b 84 6f 01 01 addl $0x1,0x16f847b(%rip) # ffffffff826fd080 <vsyscall_gtod_data> ffffffff81004c00: R_X86_64_PC32 vsyscall_gtod_data-0x5 ffffffff81004c05: 8b 48 24 mov 0x24(%rax),%ecx ffffffff81004c08: b8 01 00 00 00 mov $0x1,%eax ffffffff81004c0d: d3 e0 shl %cl,%eax
Quoted 7 lines
And gcc 8 looks like this:
ffffffff81004c90<update_vsyscall>:ffffffff81004c90: e8 6b cb 9f 00 callqffffffff81a01800<__fentry__>ffffffff81004c91: R_X86_64_PC32 __fentry__-0x4 ffffffff81004c95: 48 8b 07 mov (%rdi),%rax ffffffff81004c98: 83 05 e1 93 6f 01 01 addl $0x1,0x16f93e1(%rip) # ffffffff826fe080 <vsyscall_gtod_data>
Hm this is a RIP relative instruction, it should be modified by kprobes.
Quoted 5 lines
ffffffff81004c9a: R_X86_64_PC32 vsyscall_gtod_data-0x5
ffffffff81004c9f: 8b 50 24 mov 0x24(%rax),%edx
ffffffff81004ca2: 8b 05 ec 5e 34 01 mov 0x1345eec(%rip),%eax # ffffffff8234ab94 <vclocks_used>
ffffffff81004ca4: R_X86_64_PC32 vclocks_used-0x4Quoted 7 lines
The test adds a kprobe (optimized) at udpate_vsyscall+5. And will insert a jump on the two instructions after fentry. The difference between v7 and v8 is that v7 is touching
vclocks_usedand v8 is touchingvsyscall_gtod_data.Is there some black magic going on with the vsyscall area with
vsyscall_gtod_datathat is causing havoc when a kprobe is added there?
I think it might miss something when preprocessing RIP relative instruction. Could you disable jump optimization as below and test what happen on update_vsyscall+5 AND update_vsyscall+8? (RIP relative preprocess must happen even if the jump optimization is disabled)
# echo 0 > /proc/sys/debug/kprobes-optimization
Quoted 3 lines
I can dig a little more into this, but I'm currently at my HQ office with a lot of other objectives that I must get done, and I can't work on this much more this week.
OK, let me try to reproduce it in my environment.
Quoted 3 lines
I included my config (for my virt machine, which I was also able to trigger it with).
Thanks, but I think it should not depend on the kconfig.
Quoted 7 lines
The test that triggers this bug is:
tools/testing/selftests/ftrace/test.d/kprobe/multiple_kprobes.tcIt runs the test fine, but other things just start to act up after I run it.
Yeah, thank you for digging it down. It is now much easier to me.
Quoted 3 lines
I notice that when I get into the state, journald and the
dbus_daemonare constantly running. Perhaps the userspace time keeping went bad?
Yeah, I think so. Maybe addl instruction becomes broken.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Quoted 62 lines · On Tue, 4 Dec 2018 16:40:07 +0900 Masami Hiramatsu <[email protected]> wrote:
Hi Steve,
Quoted 9 lines · On Mon, 3 Dec 2018 21:18:07 -0500 Steven Rostedt <[email protected]> wrote:
Hi Masami,
I started testing some of my new code and the system got into a strange state. Debugging further, I found the cause came from the kprobe tests. It became stranger to me that I could reproduce it with older kernels. I went back as far as 4.16 and it triggered. I thought this very strange because I ran this test on all those kernels in the past.
Quoted 3 lines
After a bit of hair pulling, I figured out what changed. I upgraded to gcc 8.1 (and I reproduce it with 8.2 as well). I convert back to gcc 7 and the tests pass without issue.
OK, let me see.
Quoted 22 lines
The issue that I notice when the system gets into this strange state is that I can't log into the box. Nor can I reboot. Basically it's anything to do with systemd just doesn't work (insert your jokes here now, and then let's move on).
I was able to narrow down what the exact function was that caused the issues and it is:
update_vsyscall()gcc 7 looks like this:
ffffffff81004bf0<update_vsyscall>:ffffffff81004bf0: e8 0b cc 9f 00 callqffffffff81a01800<__fentry__>ffffffff81004bf1: R_X86_64_PC32 __fentry__-0x4 ffffffff81004bf5: 48 8b 07 mov (%rdi),%rax ffffffff81004bf8: 8b 15 96 5f 34 01 mov 0x1345f96(%rip),%edx # ffffffff8234ab94 <vclocks_used> ffffffff81004bfa: R_X86_64_PC32 vclocks_used-0x4 ffffffff81004bfe: 83 05 7b 84 6f 01 01 addl $0x1,0x16f847b(%rip) # ffffffff826fd080 <vsyscall_gtod_data> ffffffff81004c00: R_X86_64_PC32 vsyscall_gtod_data-0x5 ffffffff81004c05: 8b 48 24 mov 0x24(%rax),%ecx ffffffff81004c08: b8 01 00 00 00 mov $0x1,%eax ffffffff81004c0d: d3 e0 shl %cl,%eaxQuoted 7 lines
And gcc 8 looks like this:
ffffffff81004c90<update_vsyscall>:ffffffff81004c90: e8 6b cb 9f 00 callqffffffff81a01800<__fentry__>ffffffff81004c91: R_X86_64_PC32 __fentry__-0x4 ffffffff81004c95: 48 8b 07 mov (%rdi),%rax ffffffff81004c98: 83 05 e1 93 6f 01 01 addl $0x1,0x16f93e1(%rip) # ffffffff826fe080 <vsyscall_gtod_data>Hm this is a RIP relative instruction, it should be modified by kprobes.
Quoted 10 lines
ffffffff81004c9a: R_X86_64_PC32 vsyscall_gtod_data-0x5 ffffffff81004c9f: 8b 50 24 mov 0x24(%rax),%edx ffffffff81004ca2: 8b 05 ec 5e 34 01 mov 0x1345eec(%rip),%eax # ffffffff8234ab94 <vclocks_used> ffffffff81004ca4: R_X86_64_PC32 vclocks_used-0x4The test adds a kprobe (optimized) at udpate_vsyscall+5. And will insert a jump on the two instructions after fentry. The difference between v7 and v8 is that v7 is touching
vclocks_usedand v8 is touchingvsyscall_gtod_data.
Quoted 4 lines
Quoted 2 lines
Is there some black magic going on with the vsyscall area with
vsyscall_gtod_datathat is causing havoc when a kprobe is added there?I think it might miss something when preprocessing RIP relative instruction.
Ah, I got it. I thought it had been fixed, but not...
In arch/x86/kernel/kprobes/opt.c, copy_optimized_instructions() does a copy loop, but only update src and dest cursors, but not update real address which is used for adjusting RIP relative instruction.
while (len < RELATIVEJUMP_SIZE) {
ret = __copy_instruction(dest + len, src + len, real, &insn);
if (!ret || !can_boost(&insn, src + len))
return -EINVAL;
len += ret;
}I remember I have fixed this, and actually WE did it :-D
https://lkml.org/lkml/2018/8/23/1203
Ah, we hit a same bug...
Ingo, could you pick the patch? Should I resend it?
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Quoted 7 lines · Masami Hiramatsu
I remember I have fixed this, and actually WE did it :-D
https://lkml.org/lkml/2018/8/23/1203
Ah, we hit a same bug...
Ingo, could you pick the patch? Should I resend it?
Indeed: I just picked it up into tip:perf/urgent.
It's my bad: I missed the original submission due to Steve's feedback which I mistook as a request for another iteration, while he only commented on the reason for the original breakage and gave his Reviewed-by ...
Thanks,
Ingo
Quoted 8 lines · On Tue, 4 Dec 2018 09:15:06 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Masami Hiramatsu
I remember I have fixed this, and actually WE did it :-D
https://lkml.org/lkml/2018/8/23/1203
Ah, we hit a same bug...
Quoted 9 lines
Quoted 1 line
Ingo, could you pick the patch? Should I resend it?
Indeed: I just picked it up into tip:perf/urgent.
It's my bad: I missed the original submission due to Steve's feedback which I mistook as a request for another iteration, while he only commented on the reason for the original breakage and gave his Reviewed-by ...
Sorry for the confusion. The patch and code was quite complex, and I was documenting what I thought of the patch (and the bug), so that my Reviewed-by had a bit more meaning.
-- Steve
Quoted 21 lines · Steven Rostedt
Quoted 8 lines · On Tue, 4 Dec 2018 09:15:06 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 6 lines · Masami Hiramatsu
I remember I have fixed this, and actually WE did it :-D
https://lkml.org/lkml/2018/8/23/1203
Ah, we hit a same bug...
Quoted 9 lines
Quoted 1 line
Ingo, could you pick the patch? Should I resend it?
Indeed: I just picked it up into tip:perf/urgent.
It's my bad: I missed the original submission due to Steve's feedback which I mistook as a request for another iteration, while he only commented on the reason for the original breakage and gave his Reviewed-by ...
Quoted 3 lines
Sorry for the confusion. The patch and code was quite complex, and I was documenting what I thought of the patch (and the bug), so that my Reviewed-by had a bit more meaning.
It was totally welcome when I read it the second time - and it's my bad I was reading too quickly first time around ;-)
So please keep doing it whenever you find the time! :)
Thanks,
Ingo