ckpt-16-dev BUG() - bisected

DISCUSSION Nathan Lynch 2 participants 4 messages Jun 3, 2009 – Jun 10, 2009 2 messages not in the archive

Subsystems (2)

x86 x86 MM

lore

Share & tools

lore

Nathan Lynch

author

2009-06-03 22:00 UTC #0 lore raw

Pulled latest updates today, started seeing the bug below when doing simple bash shell checkpoint/restart. Git bisect blames:

commit 505d9949cac385c5111bbd8ff3d2e123d5c468d8

Author: Oren Laadan <[email protected]>
Date:   Sun May 31 12:59:40 2009 -0400

c/r: [X86_32] validate task's cpu registers and tls during restart

Check the eflags, segment registers and tls entries from the checkpoint image to prevent breakpoints inside the kernel code.

Fixes in restore_thread() and load_cpu_regs():
/* TODO: ADD SANITY CHECKS TO VERIFY VALIDITY OF VALUES */

Based on patch from Alexey Dobriyan:
https://lists.linux-foundation.org/pipermail/containers/2009-May/017886.html
Segment registers are abstracted to allow i386 => x86_64 migration (BTW, I'm not so sure if just making 32-bit selectors the same will achieve same effect)

kernel BUG at arch/x86/mm/checkpoint.c:86! invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC last sysfs file: /sys/block/sda/size

Modules linked in:

Pid: 2236, comm: ckpt Not tainted (2.6.30-rc7-00054-g505d994 #55)
EIP: 0060:[<c021c3dd>] EFLAGS: 00010202 CPU: 0
EIP is at encode_segment+0x51/0x57
EAX: 1fffe009 EBX: df03a720 ECX: 00000130 EDX: 1fffe00f
ESI: df996fb4 EDI: def3f250 EBP: de8abf18 ESP: de8abf18
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process ckpt (pid: 2236, ti=de8ab000 task=de8e28e0 task.ti=de8ab000)

Stack:
 de8abf38 c021cc07 decf51c0 def3f250 00000062 dc99a440 00000000 def3f250
 de8abf58 c038eb3c decf51c0 decf54e8 00000007 00000000 00000000 def3f250
 de8abf98 c038d7bf c07a69e0 00000001 00000001 00000000 def73ee0 c07ad224
Call Trace:
 [<c021cc07>] ? checkpoint_cpu+0x11a/0x426
 [<c038eb3c>] ? checkpoint_task+0x226/0x239
 [<c038d7bf>] ? do_checkpoint+0x404/0x511
 [<c038c669>] ? sys_checkpoint+0x6d/0x83
 [<c0202ce5>] ? syscall_call+0x7/0xb
Code: 74 2d 83 fa 7b b8 02 00 00 00 74 23 f6 c2 04 74 0a 89 d0 c1 e8 03 80 cc 80 eb 14 c1 ea 03 8d 42 fa 83 f8 02 77 05 80 cc 40 eb 04 <0f> 0b eb fe 5d c3 55 89 e5 e8 f9 72 fe ff 85 c0 74 40 83 f8 01 
EIP: [<c021c3dd>] encode_segment+0x51/0x57 SS:ESP 0068:de8abf18
---[ end trace 270dba57a69e79e1 ]---

A message that is not in the archive (sent off-list or not yet received): [email protected]

Oren Laadan

2009-06-04 00:31 UTC #2 lore raw

Hi,

I couldn't reproduce the problem on my system - your script works fine (checkpoint and restart) on my kernel.

The actual logic is borrowed from Alexey's i386 patches.

My best guess is that either the issue occurs because you are running under KVM which uses different segments (?)

Either that, or something is wrong with Alexey's original code.

Alexey - any idea ?

Also, you can you apply this patch so we know what was the segment's value:

----------------------------------

arch/x86/mm/checkpoint.c +4 -0

index 7242762..18e957a 100644
@@ -83,6 +83,8 @@ static __u32 encode_segment(unsigned long seg)
 	seg >>= 3;
 	if (GDT_ENTRY_TLS_MIN <= seg && seg <= GDT_ENTRY_TLS_MAX)
 		return CKPT_X86_SEG_TLS | (seg - GDT_ENTRY_TLS_MIN);
+
+	printk(KERN_ERR "bad segment %#lx\n", seg);
 	BUG();
 }

@@ -103,6 +105,8 @@ static unsigned long decode_segment(__u32 seg)
 		seg &= ~CKPT_X86_SEG_LDT;
 		return (seg << 3) | 7;
 	}
+
+	printk(KERN_ERR "bad segment %#lx\n", seg);
 	BUG();
 }

----------------------------------

Oren.

Quoted 49 lines · Nathan Lynch

Pulled latest updates today, started seeing the bug below when doing simple bash shell checkpoint/restart. Git bisect blames:
commit 505d9949cac385c5111bbd8ff3d2e123d5c468d8
Author: Oren Laadan <[email protected]>
Date:   Sun May 31 12:59:40 2009 -0400
c/r: [X86_32] validate task's cpu registers and tls during restart
Check the eflags, segment registers and tls entries from the checkpoint image to prevent breakpoints inside the kernel code.
Fixes in restore_thread() and load_cpu_regs():
/* TODO: ADD SANITY CHECKS TO VERIFY VALIDITY OF VALUES */
Based on patch from Alexey Dobriyan:
https://lists.linux-foundation.org/pipermail/containers/2009-May/017886.html
Segment registers are abstracted to allow i386 => x86_64 migration (BTW, I'm not so sure if just making 32-bit selectors the same will achieve same effect)
kernel BUG at arch/x86/mm/checkpoint.c:86! invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC last sysfs file: /sys/block/sda/size
Modules linked in:
Pid: 2236, comm: ckpt Not tainted (2.6.30-rc7-00054-g505d994 #55)
EIP: 0060:[<c021c3dd>] EFLAGS: 00010202 CPU: 0
EIP is at encode_segment+0x51/0x57
EAX: 1fffe009 EBX: df03a720 ECX: 00000130 EDX: 1fffe00f
ESI: df996fb4 EDI: def3f250 EBP: de8abf18 ESP: de8abf18
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process ckpt (pid: 2236, ti=de8ab000 task=de8e28e0 task.ti=de8ab000)
Stack:
 de8abf38 c021cc07 decf51c0 def3f250 00000062 dc99a440 00000000 def3f250
 de8abf58 c038eb3c decf51c0 decf54e8 00000007 00000000 00000000 def3f250
 de8abf98 c038d7bf c07a69e0 00000001 00000001 00000000 def73ee0 c07ad224
Call Trace:
 [<c021cc07>] ? checkpoint_cpu+0x11a/0x426
 [<c038eb3c>] ? checkpoint_task+0x226/0x239
 [<c038d7bf>] ? do_checkpoint+0x404/0x511
 [<c038c669>] ? sys_checkpoint+0x6d/0x83
 [<c0202ce5>] ? syscall_call+0x7/0xb
Code: 74 2d 83 fa 7b b8 02 00 00 00 74 23 f6 c2 04 74 0a 89 d0 c1 e8 03 80 cc 80 eb 14 c1 ea 03 8d 42 fa 83 f8 02 77 05 80 cc 40 eb 04 <0f> 0b eb fe 5d c3 55 89 e5 e8 f9 72 fe ff 85 c0 74 40 83 f8 01 
EIP: [<c021c3dd>] encode_segment+0x51/0x57 SS:ESP 0068:de8abf18
---[ end trace 270dba57a69e79e1 ]---

A message that is not in the archive (sent off-list or not yet received): Pine.LNX.4.64.0906032027370.22069-CXF6herHY6ykSYb+qCZC/1i27PF6R63G9nwVQlTi/[email protected]

Nathan Lynch

author

2009-06-04 18:08 UTC #4 lore raw

Quoted 32 lines · Oren Laadan

Hi,
I couldn't reproduce the problem on my system - your script works fine (checkpoint and restart) on my kernel.
The actual logic is borrowed from Alexey's i386 patches.
My best guess is that either the issue occurs because you are running under KVM which uses different segments (?)
Either that, or something is wrong with Alexey's original code.
Alexey - any idea ?
Also, you can you apply this patch so we know what was the segment's value:
----------------------------------
arch/x86/mm/checkpoint.c +2 -0
index 7242762..18e957a 100644
@@ -83,6 +83,8 @@ static __u32 encode_segment(unsigned long seg)
 	seg >>= 3;
 	if (GDT_ENTRY_TLS_MIN <= seg && seg <= GDT_ENTRY_TLS_MAX)
 		return CKPT_X86_SEG_TLS | (seg - GDT_ENTRY_TLS_MIN);
+
+	printk(KERN_ERR "bad segment %#lx\n", seg);
 	BUG();
 }
 

arch/x86/mm/checkpoint.c static unsigned long decode_segment(__u32 seg) · line 111

Quoted 7 lines

arch/x86/mm/checkpoint.c +2 -0

@@ -103,6 +105,8 @@ static unsigned long decode_segment(__u32 seg)
 		seg &= ~CKPT_X86_SEG_LDT;
 		return (seg << 3) | 7;
 	}
+
+	printk(KERN_ERR "bad segment %#lx\n", seg);
 	BUG();
 }

bad segment 0x1fffe00f

------------[ cut here ]------------
kernel BUG at arch/x86/mm/checkpoint.c:88!
invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/class/net/br0/ifindex
Modules linked in:

Pid: 3117, comm: ckpt Not tainted (2.6.30-rc7-00058-gb0ddfc4-dirty #58)
EIP: 0060:[<c021c622>] EFLAGS: 00010246 CPU: 0
EIP is at encode_segment+0x5c/0x64
EAX: 0000001a EBX: de86f300 ECX: 00000000 EDX: 00000003
ESI: df1f8fb4 EDI: dfad7020 EBP: df1faf18 ESP: df1faf10
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process ckpt (pid: 3117, ti=df1fa000 task=df9f0000 task.ti=df1fa000)

Stack:
 c06f9756 1fffe00f df1faf38 c021cc21 de810000 dfad7020 00000044 df280200
 00000000 dfad7020 df1faf58 c038edee de810000 de810328 00000007 00000000
 00000000 dfad7020 df1faf98 c038d88c c07ab9e0 00000001 00000001 00000000
Call Trace:
 [<c021cc21>] ? checkpoint_cpu+0x11a/0x426
 [<c038edee>] ? checkpoint_task+0x226/0x239
 [<c038d88c>] ? do_checkpoint+0x459/0x551
 [<c038c689>] ? sys_checkpoint+0x6d/0x83
 [<c0202ce5>] ? syscall_call+0x7/0xb
Code: 30 f6 c2 04 74 0a 89 d0 c1 e8 03 80 cc 80 eb 21 c1 ea 03 8d 42 fa 83 f8 02 77 05 80 cc 40 eb 11 52 68 56 97 6f c0 e8 76 c2 3c 00 <0f> 0b 58 5a eb fe c9 c3 55 89 e5 57 56 53 83 ec 10 e8 ac 70 fe 
EIP: [<c021c622>] encode_segment+0x5c/0x64 SS:ESP 0068:df1faf10
---[ end trace 0c5f24ea38f32c3f ]---

Nathan Lynch

author

2009-06-10 06:07 UTC #5 lore raw

In reply to a message not in the archive

Quoted 8 lines · Nathan Lynch

Pulled latest updates today, started seeing the bug below when doing simple bash shell checkpoint/restart. Git bisect blames:
commit 505d9949cac385c5111bbd8ff3d2e123d5c468d8
Author: Oren Laadan <[email protected]>
Date:   Sun May 31 12:59:40 2009 -0400
c/r: [X86_32] validate task's cpu registers and tls during restart

...

Quoted 1 line

kernel BUG at arch/x86/mm/checkpoint.c:86!

This seems to be fixed in latest ckpt-v16-dev, presumably by f5867c9c8383c2d2d5361512e79e70df4b730fe9 ("c/r: [X86_32] used __u16/short to encode/decode segment registers").