pv 2.6.31 (kernel.org) and save/migrate
Subsystems (3)
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
I'd assume migrate and live migrate would fail also but haven't tried them.
Thanks,
Dan
P.S. This is with very recent xen-unstable, c/s 20399.
Quoted 9 lines · Dan Magenheimer
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 3 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Just checking.. are you running the latest 2.6.31.5 ? I think there has been multiple xen related bugfixes in the 2.6.31.X releases.
-- Pasi
Quoted 14 lines · Dan Magenheimer
Quoted 9 lines
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 3 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Quoted 5 lines
Just checking.. are you running the latest 2.6.31.5 ? I think there has
been multiple xen related bugfixes in the 2.6.31.X releases.-- Pasi
No it was plain 2.6.31. But I downloaded/built 2.6.31.5 and can't even get it to boot (and no console or VNC output at all). Are CONFIG changes required betwen 2.6.31 and 2.6.31.5 for Xen? (I checked and I am using the same .config.)
Trying to reproduce on a different machine, just to verify.
Dan
Quoted 18 lines · Dan Magenheimer
Quoted 18 lines · Dan Magenheimer
Quoted 9 lines
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 3 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Just checking.. are you running the latest 2.6.31.5 ? I think there has
been multiple xen related bugfixes in the 2.6.31.X releases.
Quoted 9 lines
Quoted 1 line
-- Pasi
No it was plain 2.6.31. But I downloaded/built 2.6.31.5 and can't even get it to boot (and no console or VNC output at all). Are CONFIG changes required betwen 2.6.31 and 2.6.31.5 for Xen? (I checked and I am using the same .config.)
Trying to reproduce on a different machine, just to verify.
There shouldn't be any .config changes needed.
Can you paste the full domU console output? Does it crash or?
-- Pasi
Quoted 26 lines · Dan Magenheimer
Quoted 25 lines
Quoted 14 lines · Dan Magenheimer
Quoted 9 lines
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 3 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Quoted 5 lines
Just checking.. are you running the latest 2.6.31.5 ? I think there has
been multiple xen related bugfixes in the 2.6.31.X releases.-- Pasi
No it was plain 2.6.31. But I downloaded/built 2.6.31.5 and can't even get it to boot (and no console or VNC output at all). Are CONFIG changes required betwen 2.6.31 and 2.6.31.5 for Xen? (I checked and I am using the same .config.)
Quoted 7 lines
Quoted 1 line
Trying to reproduce on a different machine, just to verify.
There shouldn't be any .config changes needed.
Can you paste the full domU console output? Does it crash or?
-- Pasi
Well, first, I got 2.6.31.5 to boot in a PV guest in another machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)
(On the machine I couldn't boot 2.6.31.5 as a PV guest, there was absolutely no console output. However, I think tools are out-of-date on that machine so ignore that.)
Quoted 32 lines · Dan Magenheimer
Quoted 32 lines · Dan Magenheimer
Quoted 18 lines
Quoted 18 lines · Dan Magenheimer
Quoted 9 lines
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 3 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Just checking.. are you running the latest 2.6.31.5 ? I think there has
been multiple xen related bugfixes in the 2.6.31.X releases.Quoted 8 lines
Quoted 1 line
-- Pasi
No it was plain 2.6.31. But I downloaded/built 2.6.31.5 and can't even get it to boot (and no console or VNC output at all). Are CONFIG changes required betwen 2.6.31 and 2.6.31.5 for Xen? (I checked and I am using the same .config.)
Trying to reproduce on a different machine, just to verify.
There shouldn't be any .config changes needed.
Can you paste the full domU console output? Does it crash or?
Quoted 8 lines
Quoted 1 line
-- Pasi
Well, first, I got 2.6.31.5 to boot in a PV guest in another machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)
I'll have to try it later today..
Quoted 3 lines
(On the machine I couldn't boot 2.6.31.5 as a PV guest, there was absolutely no console output. However, I think tools are out-of-date on that machine so ignore that.)
Did you have "console=hvc0 earlyprintk=xen" in the domU kernel parameters?
You might also change the xen guest cfgfile so that you have on_crash=preserve and then when the PV guest is crashed run this:
/usr/lib/xen/bin/xenctx -s System.map-domUkernelversion <domid>
(if you have 64b host the xenctx binary might be under /usr/lib64/)
to get a stack trace..
-- Pasi
Quoted 7 lines
Quoted 5 lines
Well, first, I got 2.6.31.5 to boot in a PV guest in another machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)I'll have to try it later today..
Let me know.
Quoted 6 lines
Quoted 3 lines
(On the machine I couldn't boot 2.6.31.5 as a PV guest, there was absolutely no console output. However, I think tools are out-of-date on that machine so ignore that.)
Did you have "console=hvc0 earlyprintk=xen" in the domU kernel parameters?
No, but that didn't work either.
Quoted 8 lines
You might also change the xen guest cfgfile so that you have on_crash=preserve and then when the PV guest is crashed run this:
/usr/lib/xen/bin/xenctx-s System.map-domUkernelversion <domid>(if you have 64b host the xenctx binary might be under
/usr/lib64/)to get a stack trace..
Very interesting and useful! I was completely unaware of xenctx and could have used it many times in tmem development!
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.
cs:eip: e019:c040342d xen_cpuid+0x46
flags: 00001206 i nz p
ss:esp: e021:c0779ee4
eax: 00000001 ebx: 00000002 ecx: 00000100 edx: 00000001
esi: c0779f1c edi: c0779f18 ebp: c0779f24
ds: e021 es: e021 fs: 00d8 gs: 0000
Code (instr addr c040342d)
24 04 8b 15 a4 02 7c c0 89 54 24 08 8b 0e 0f 0b 78 65 6e 0f a2 <89> 45 00 8b 04 24 89 18 89 0e 89
Stack: c0779f20 ffffffff ffffffff c07c0360 c0779f18 c0779f1c c0779f20 c066fd0f c0779f18 c0779f24 00000002 16aee301 00000001 00000001 16aee301 00000002 0000000b c07c03cc c07c0360 c07c0360 c07c03d8 c0670ed8 c0779f58 00000001 c07c0360 c0779f60 c066fe6a c0779f60 c0779f60 00000003 00000001 00000000 Call Trace: [<c040342d>] xen_cpuid+0x46 <-- [<c066fd0f>] detect_extended_topology+0xae [<c0670ed8>] init_intel+0x140 [<c066fe6a>] init_scattered_cpuid_features+0x82 [<c06705e2>] identify_cpu+0x22d [<c040584c>] xen_force_evtchn_callback+0xc [<c0405e78>] check_events+0x8 [<c07c9dec>] identify_boot_cpu+0xa [<c07c9e9a>] check_bugs+0x8 [<c07c27bd>] start_kernel+0x2a0 [<c07c5206>] xen_start_kernel+0x340
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
Quoted 10 lines · Dan Magenheimer
Quoted 7 lines
Quoted 5 lines
Well, first, I got 2.6.31.5 to boot in a PV guest in another machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)I'll have to try it later today..
Let me know.
Ok. I just tried with a Fedora 12 (rawhide) PV guest. I was able to "xm save" and "xm restore" it without problems.
But I noticed there was a BUG printed on the guest console:
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86_64-saverestore.txt
BUG: sleeping function called from invalid context at kernel/mutex.c:94 in_atomic(): 0, irqs_disabled(): 1, pid: 1052, name: kstop/0 Pid: 1052, comm: kstop/0 Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: [<ffffffff8104021f>] __might_sleep+0xe6/0xe8 [<ffffffff81419c84>] mutex_lock+0x22/0x4e [<ffffffff812afdce>] dpm_resume_noirq+0x21/0x11f [<ffffffff81272b05>] xen_suspend+0xca/0xd1 [<ffffffff8108c172>] stop_cpu+0x8c/0xd2 [<ffffffff8106350c>] worker_thread+0x18a/0x224 [<ffffffff81067ae7>] ? autoremove_wake_function+0x0/0x39 [<ffffffff8141ab29>] ? _spin_unlock_irqrestore+0x19/0x1b [<ffffffff81063382>] ? worker_thread+0x0/0x224 [<ffffffff81067765>] kthread+0x91/0x99 [<ffffffff81012daa>] child_rip+0xa/0x20 [<ffffffff81011f97>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff8101271d>] ? retint_restore_args+0x5/0x6 [<ffffffff81012da0>] ? child_rip+0x0/0x20
More information about my setup:
Host/dom0: Fedora 12 (latest rawhide) with included Xen 3.4.1-5 and custom 2.6.31.5 x86_64 pv_ops dom0 kernel (a couple of days old).
Guest/domU: Fedora 12 (latest rawhide) with the included/default 2.6.31.5-122.fc12.x86_64 kernel.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
(On the machine I couldn't boot 2.6.31.5 as a PV guest, there was absolutely no console output. However, I think tools are out-of-date on that machine so ignore that.)
Did you have "console=hvc0 earlyprintk=xen" in the domU kernel parameters?
No, but that didn't work either.
Ok.. then it crashes really early.
Quoted 9 lines
Quoted 8 lines
You might also change the xen guest cfgfile so that you have on_crash=preserve and then when the PV guest is crashed run this:
/usr/lib/xen/bin/xenctx-s System.map-domUkernelversion <domid>(if you have 64b host the xenctx binary might be under
/usr/lib64/)to get a stack trace..
Quoted 9 lines
Very interesting and useful! I was completely unaware of xenctx and could have used it many times in tmem development!
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.
Did you try with that patch applied?
-- Pasi
Quoted 28 lines
cs:eip: e019:c040342d
xen_cpuid+0x46
flags: 00001206 i nz p
ss:esp: e021:c0779ee4
eax: 00000001 ebx: 00000002 ecx: 00000100 edx: 00000001
esi:c0779f1cedi:c0779f18ebp:c0779f24
ds: e021 es: e021 fs: 00d8 gs: 0000
Code (instr addrc040342d)
24 04 8b 15 a4 02 7c c0 89 54 24 08 8b 0e 0f 0b 78 65 6e 0f a2 <89> 45 00 8b 04 24 89 18 89 0e 89Stack: c0779f20 ffffffff ffffffff c07c0360 c0779f18 c0779f1c c0779f20 c066fd0f c0779f18 c0779f24 00000002 16aee301 00000001 00000001 16aee301 00000002 0000000b c07c03cc c07c0360 c07c0360 c07c03d8 c0670ed8 c0779f58 00000001 c07c0360 c0779f60 c066fe6a c0779f60 c0779f60 00000003 00000001 00000000 Call Trace: [<c040342d>] xen_cpuid+0x46 <-- [<c066fd0f>] detect_extended_topology+0xae [<c0670ed8>] init_intel+0x140 [<c066fe6a>] init_scattered_cpuid_features+0x82 [<c06705e2>] identify_cpu+0x22d [<c040584c>] xen_force_evtchn_callback+0xc [<c0405e78>] check_events+0x8 [<c07c9dec>] identify_boot_cpu+0xa [<c07c9e9a>] check_bugs+0x8 [<c07c27bd>] start_kernel+0x2a0 [<c07c5206>] xen_start_kernel+0x340
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG
Quoted 28 lines · Pasi Kärkkäinen
Quoted 10 lines · Dan Magenheimer
Quoted 7 lines
Quoted 5 lines
Well, first, I got 2.6.31.5 to boot in a PV guest in another machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)I'll have to try it later today..
Let me know.
Ok. I just tried with a Fedora 12 (rawhide) PV guest. I was able to "xm save" and "xm restore" it without problems.
But I noticed there was a BUG printed on the guest console:
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86_64-saverestore.txtBUG: sleeping function called from invalid context at kernel/mutex.c:94 in_atomic(): 0, irqs_disabled(): 1, pid: 1052, name: kstop/0 Pid: 1052, comm: kstop/0 Not tainted 2.6.31.5-122.fc12.x86_64 #1 Call Trace: [<ffffffff8104021f>] __might_sleep+0xe6/0xe8 [<ffffffff81419c84>] mutex_lock+0x22/0x4e [<ffffffff812afdce>] dpm_resume_noirq+0x21/0x11f [<ffffffff81272b05>] xen_suspend+0xca/0xd1 [<ffffffff8108c172>] stop_cpu+0x8c/0xd2 [<ffffffff8106350c>] worker_thread+0x18a/0x224
Quoted 9 lines
[<ffffffff81067ae7>] ? autoremove_wake_function+0x0/0x39 [<ffffffff8141ab29>] ? _spin_unlock_irqrestore+0x19/0x1b [<ffffffff81063382>] ? worker_thread+0x0/0x224 [<ffffffff81067765>] kthread+0x91/0x99 [<ffffffff81012daa>] child_rip+0xa/0x20 [<ffffffff81011f97>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff8101271d>] ? retint_restore_args+0x5/0x6 [<ffffffff81012da0>] ? child_rip+0x0/0x20
Oh, I forgot to mention that this BUG is non-fatal. The guest still works after that..
-- Pasi
Quoted 80 lines
More information about my setup:
Host/dom0: Fedora 12 (latest rawhide) with included Xen 3.4.1-5 and custom 2.6.31.5
x86_64pv_opsdom0 kernel (a couple of days old).Guest/domU: Fedora 12 (latest rawhide) with the included/default 2.6.31.5-122.fc12.x86_64 kernel.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
(On the machine I couldn't boot 2.6.31.5 as a PV guest, there was absolutely no console output. However, I think tools are out-of-date on that machine so ignore that.)
Did you have "console=hvc0 earlyprintk=xen" in the domU kernel parameters?
No, but that didn't work either.
Ok.. then it crashes really early.
Quoted 9 lines
Quoted 8 lines
You might also change the xen guest cfgfile so that you have on_crash=preserve and then when the PV guest is crashed run this:
/usr/lib/xen/bin/xenctx-s System.map-domUkernelversion <domid>(if you have 64b host the xenctx binary might be under
/usr/lib64/)to get a stack trace..
Quoted 9 lines
Very interesting and useful! I was completely unaware of xenctx and could have used it many times in tmem development!
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
-- Pasi
Quoted 20 lines
cs:eip: e019:c040342d
xen_cpuid+0x46
flags: 00001206 i nz p
ss:esp: e021:c0779ee4
eax: 00000001 ebx: 00000002 ecx: 00000100 edx: 00000001
esi:c0779f1cedi:c0779f18ebp:c0779f24
ds: e021 es: e021 fs: 00d8 gs: 0000
Code (instr addrc040342d)
24 04 8b 15 a4 02 7c c0 89 54 24 08 8b 0e 0f 0b 78 65 6e 0f a2 <89> 45 00 8b 04 24 89 18 89 0e 89Stack: c0779f20 ffffffff ffffffff c07c0360 c0779f18 c0779f1c c0779f20 c066fd0f c0779f18 c0779f24 00000002 16aee301 00000001 00000001 16aee301 00000002 0000000b c07c03cc c07c0360 c07c0360 c07c03d8 c0670ed8 c0779f58 00000001 c07c0360 c0779f60 c066fe6a c0779f60 c0779f60 00000003 00000001 00000000 Call Trace: [<c040342d>] xen_cpuid+0x46 <-- [<c066fd0f>] detect_extended_topology+0xae [<c0670ed8>] init_intel+0x140Quoted 8 lines
[<c066fe6a>] init_scattered_cpuid_features+0x82 [<c06705e2>] identify_cpu+0x22d [<c040584c>] xen_force_evtchn_callback+0xc [<c0405e78>] check_events+0x8 [<c07c9dec>] identify_boot_cpu+0xa [<c07c9e9a>] check_bugs+0x8 [<c07c27bd>] start_kernel+0x2a0 [<c07c5206>] xen_start_kernel+0x340_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
RE: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
In reply to Pasi KärkkäinenQuoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)Ok. I just tried with a Fedora 12 (rawhide) PV guest. I was able to "xm save" and "xm restore" it without problems.
Quoted 7 lines
But I noticed there was a BUG printed on the guest console:
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86
_64-saverestore.txtBUG: sleeping function called from invalid context at
kernel/mutex.c:94in_atomic(): 0,irqs_disabled(): 1, pid: 1052, name: kstop/0
Pid: 1052, comm: kstop/0 Not tainted 2.6.31.5-122.fc12.x86_64 #1
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Quoted 7 lines
Quoted 5 lines
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
No, the patch wasn't posted, just a pull request to Linus, so I don't have the patch (and am not a git expert so am not sure how to get it).
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00182.html
So I'll try it again when .6 or .7 is available.
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
Quoted 16 lines · Dan Magenheimer
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)Ok. I just tried with a Fedora 12 (rawhide) PV guest. I was able to "xm save" and "xm restore" it without problems.
Quoted 7 lines
But I noticed there was a BUG printed on the guest console:
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86
_64-saverestore.txtBUG: sleeping function called from invalid context at
kernel/mutex.c:94in_atomic(): 0,irqs_disabled(): 1, pid: 1052, name: kstop/0
Pid: 1052, comm: kstop/0 Not tainted 2.6.31.5-122.fc12.x86_64 #1
Quoted 5 lines
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Yeah, fedora kernel has some patches, but it should be pretty close to upstream kernel..
btw was your guest UP or SMP? Mine was UP..
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
Quoted 7 lines
No, the patch wasn't posted, just a pull request to Linus, so I don't have the patch (and am not a git expert so am not sure how to get it).
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00182.html
So I'll try it again when .6 or .7 is available.
See here for changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=bugfix
You can get the diffs/patches from there using the links..
-- Pasi
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
Quoted 23 lines · Pasi Kärkkäinen
Quoted 16 lines · Dan Magenheimer
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
machine and it fails to save also. Are you able to save 2.6.31{,.5} successfully? On latest xen-unstable? (NOTE: Yes, I do have
CONFIG_XEN_SAVE_RESTORE=y... don't know if that is important.)Ok. I just tried with a Fedora 12 (rawhide) PV guest. I was able to "xm save" and "xm restore" it without problems.
Quoted 7 lines
But I noticed there was a BUG printed on the guest console:
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86
_64-saverestore.txtBUG: sleeping function called from invalid context at
kernel/mutex.c:94in_atomic(): 0,irqs_disabled(): 1, pid: 1052, name: kstop/0
Pid: 1052, comm: kstop/0 Not tainted 2.6.31.5-122.fc12.x86_64 #1Quoted 5 lines
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Quoted 5 lines
Yeah, fedora kernel has some patches, but it should be pretty close to upstream kernel..
btw was your guest UP or SMP? Mine was UP..
Ok.. saving SMP guest fails for me too:
[2009-11-09 23:44:38 1353] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib64/xen/bin/xc_save 28 2 0 0 0 [2009-11-09 23:44:38 1353] INFO (XendCheckpoint:417) xc_save: failed to get the suspend evtchn port
Jeremy: Ideas what's causing that? "xm save" for UP 2.6.31.5 guest works OK, but for SMP guest it fails with the error above.
-- Pasi
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
Quoted 7 lines · Pasi Kärkkäinen
Ok.. saving SMP guest fails for me too:
[2009-11-09 23:44:38 1353] DEBUG (XendCheckpoint:110) [
xc_save]:/usr/lib64/xen/bin/xc_save28 2 0 0 0 [2009-11-09 23:44:38 1353] INFO (XendCheckpoint:417)xc_save: failed to get the suspend evtchn portJeremy: Ideas what's causing that? "xm save" for UP 2.6.31.5 guest works OK, but for SMP guest it fails with the error above.
There's no "suspend evtchn port" in a pvops kernel. That looks like a Remus thing. I think.
J
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
Quoted 11 lines · Jeremy Fitzhardinge
Quoted 7 lines · Pasi Kärkkäinen
Ok.. saving SMP guest fails for me too:
[2009-11-09 23:44:38 1353] DEBUG (XendCheckpoint:110) [
xc_save]:/usr/lib64/xen/bin/xc_save28 2 0 0 0 [2009-11-09 23:44:38 1353] INFO (XendCheckpoint:417)xc_save: failed to get the suspend evtchn portJeremy: Ideas what's causing that? "xm save" for UP 2.6.31.5 guest works OK, but for SMP guest it fails with the error above.
There's no "suspend evtchn port" in a pvops kernel. That looks like a Remus thing. I think.
This is only an INFO-level message, because xc_save falls back to the old xenstore method if it can't find a suspend event channel. I don't know the context here, but this particular message ought to be harmless.
The event channel was made for Remus, but regular xc_save also uses it to reduce the downtime at the end of live migration.
RE: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
In reply to Pasi KärkkäinenQuoted 10 lines
Quoted 5 lines
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Yeah, fedora kernel has some patches, but it should be pretty close to upstream kernel..
btw was your guest UP or SMP? Mine was UP..
Mine was SMP... switching to UP I can now save. BUT... restore doesn't seem to quite work. The restore completes but I get no response from the VNC console. When I use a tty console, after restore, I am getting
an infinite dump of
WARNING: at arch/x86/time.c:180 xen_sched_clock+0x2b
(see attached).
Did you try restore on Fedora 12?
Quoted 12 lines
Quoted 12 lines
Quoted 7 lines
Quoted 5 lines
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
No, the patch wasn't posted, just a pull request to Linus, so I don't have the patch (and am not a git expert so am not sure how to get it).
Quoted 8 lines
Quoted 3 lines
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00182.html
So I'll try it again when .6 or .7 is available.
See here for changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=bugfixYou can get the diffs/patches from there using the links..
Thanks. Yes, Jeremy's patch allows 2.6.31.5 (in a PV domain) to completely boot on my Nehalem box.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG
Quoted 17 lines · Dan Magenheimer
Quoted 10 lines
Quoted 5 lines
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Yeah, fedora kernel has some patches, but it should be pretty close to upstream kernel..
btw was your guest UP or SMP? Mine was UP..
Mine was SMP... switching to UP I can now save. BUT... restore doesn't seem to quite work. The restore completes but I get no response from the VNC console. When I use a tty console, after restore, I am getting
an infinite dump of
Quoted 6 lines
WARNING: at arch/x86/time.c:180 xen_sched_clock+0x2b(see attached).
Did you try restore on Fedora 12?
Yeah. save+restore for UP F12 guest works for me
(except I get that non-fatal BUG on the guest).
SMP guest doesn't work.. save crashes it.
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 7 lines
Quoted 5 lines
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
No, the patch wasn't posted, just a pull request to Linus, so I don't have the patch (and am not a git expert so am not sure how to get it).
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00182.html
So I'll try it again when .6 or .7 is available.
Quoted 7 lines
Quoted 4 lines
See here for changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=bugfixYou can get the diffs/patches from there using the links..
Thanks. Yes, Jeremy's patch allows 2.6.31.5 (in a PV domain) to completely boot on my Nehalem box.
Ok. But I guess those doesn't help for the save+restore problem..
-- Pasi
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Hello,
Jeremy: Here's summary about these save/restore problems using upstream Linux 2.6.31.5 PV guest.
For me:
- I can "xm save" + "xm restore" UP guest, but I get non-fatal
BUG in the guest kernel, see [1].
- "xm save" fails for SMP guest with "failed to get the suspend evtchn port", see [2].
For Dan:
- "xm save" works for UP guest, but "xm restore" doesn't, giving infinite xen_sched_clock related dumps in the guest kernel, see [3].
- "xm save" for SMP guest fails, it never ends. I suspect this is the same problem I'm seeing.
[1] non-fatal BUG on the guest kernel after "xm restore":
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86_64-saverestore.txt
[2] "xm log" contains:
[2009-11-09 23:44:38 1353] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib64/xen/bin/xc_save 28 2 0 0 0 [2009-11-09 23:44:38 1353] INFO (XendCheckpoint:417) xc_save: failed to get the suspend evtchn port
[3] See the attachment in this email:
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00391.html
Any tips how to debug these?
-- Pasi
Quoted 58 lines · Pasi Kärkkäinen
Quoted 17 lines · Dan Magenheimer
Quoted 10 lines
Quoted 5 lines
Ok, so it appears there is something problematic with saving an upstream kernel. It might be (partially) fixed in Fedora 12 or maybe there is some other environmental difference which makes save fail entirely on my system.
Yeah, fedora kernel has some patches, but it should be pretty close to upstream kernel..
btw was your guest UP or SMP? Mine was UP..
Mine was SMP... switching to UP I can now save. BUT... restore doesn't seem to quite work. The restore completes but I get no response from the VNC console. When I use a tty console, after restore, I am getting
an infinite dump ofQuoted 6 lines
WARNING: at arch/x86/time.c:180 xen_sched_clock+0x2b(see attached).
Did you try restore on Fedora 12?
Yeah. save+restore for UP F12 guest works for me
(except I get that non-fatal BUG on the guest).SMP guest doesn't work.. save crashes it.
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 7 lines
Quoted 5 lines
The results explain why I can get it to run on
one machine (an older laptop) and not run on another machine (a Nehalem system)... looks like this is maybe related to the cpuid-extended-topology-leaf bug that Jeremy sent a fix for upstream recently.Did you try with that patch applied?
No, the patch wasn't posted, just a pull request to Linus, so I don't have the patch (and am not a git expert so am not sure how to get it).
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00182.html
So I'll try it again when .6 or .7 is available.
Quoted 7 lines
Quoted 4 lines
See here for changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=bugfixYou can get the diffs/patches from there using the links..
Thanks. Yes, Jeremy's patch allows 2.6.31.5 (in a PV domain) to completely boot on my Nehalem box.
Ok. But I guess those doesn't help for the save+restore problem..
-- Pasi
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 17 lines · Pasi Kärkkäinen
Hello,
Jeremy: Here's summary about these save/restore problems using upstream Linux 2.6.31.5 PV guest.
For me:
- I can "xm save" + "xm restore" UP guest, but I get non-fatal
BUG in the guest kernel, see [1].
- "xm save" fails for SMP guest with "failed to get the suspend evtchn port", see [2].For Dan:
- "xm save" works for UP guest, but "xm restore" doesn't, giving infinitexen_sched_clockrelated dumps in the guest kernel, see [3].
- "xm save" for SMP guest fails, it never ends. I suspect this is the same problem I'm seeing.
Quoted 3 lines
[1] non-fatal BUG on the guest kernel after "xm restore":
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86_64-saverestore.txt
Does this help:
drivers/xen/manage.c +3 -2
@@ -43,7 +43,6 @@ static int xen_suspend(void *data)if (err) {printk (KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n" , err); -dpm_resume_noirq (PMSG_RESUME );return err; }
@@ -69,7 +68,6 @@ static int xen_suspend(void *data) }sysdev_resume (); -dpm_resume_noirq (PMSG_RESUME );return 0 ; }
@@ -108,6 +106,9 @@ static void do_suspend(void) } err =stop_machine (xen_suspend, &cancelled,cpumask_of (0 )); + +dpm_resume_noirq (PMSG_RESUME ); +if (err) {printk (KERN_ERR "failed to start xen_suspend: %d\n" , err); goto out;
Quoted 4 lines
[2] "xm log" contains:
[2009-11-09 23:44:38 1353] DEBUG (XendCheckpoint:110) [xc_save]:/usr/lib64/xen/bin/xc_save28 2 0 0 0 [2009-11-09 23:44:38 1353] INFO (XendCheckpoint:417)xc_save: failed to get the suspend evtchn port
I think this may be a Remus side-effect.
Quoted 3 lines
[3] See the attachment in this email:
http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00391.html
No idea about this one. Needs a closer look.
J
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 18 lines · Jeremy Fitzhardinge
Quoted 17 lines · Pasi Kärkkäinen
Hello,
Jeremy: Here's summary about these save/restore problems using upstream Linux 2.6.31.5 PV guest.
For me:
- I can "xm save" + "xm restore" UP guest, but I get non-fatal
BUG in the guest kernel, see [1].
- "xm save" fails for SMP guest with "failed to get the suspend evtchn port", see [2].For Dan:
- "xm save" works for UP guest, but "xm restore" doesn't, giving infinitexen_sched_clockrelated dumps in the guest kernel, see [3].
- "xm save" for SMP guest fails, it never ends. I suspect this is the same problem I'm seeing.
Quoted 5 lines
Quoted 3 lines
[1] non-fatal BUG on the guest kernel after "xm restore":
http://pasik.reaktio.net/xen/debug/dmesg-2.6.31.5-122.fc12.x86_64-saverestore.txtDoes this help:
It does for me. There's another dpm_resume_noirq(PMSG_RESUME) a little later in do_suspend() which I think needs to be dropped as well.
I'm still seeing other problems with resume, the system is hung on restore and the RCU stall detection logic is triggering, unfortunately arch_trigger_all_cpu_backtrace is not Xen compatible (uses APIC directly) so I don't get much useful info out of it. It's most likely a symptom of the actual problem rather than a problem with RCU per-se anyhow.
drivers/xen/manage.c +3 -4
@@ -43,7 +43,6 @@ static int xen_suspend(void *data)if (err) {printk (KERN_ERR "xen_suspend: sysdev_suspend failed: %d\n" , err); -dpm_resume_noirq (PMSG_RESUME );return err; }
@@ -69,7 +68,6 @@ static int xen_suspend(void *data) }sysdev_resume (); -dpm_resume_noirq (PMSG_RESUME );return 0 ; }
@@ -108,6 +106,9 @@ static void do_suspend(void) } err =stop_machine (xen_suspend, &cancelled,cpumask_of (0 )); + +dpm_resume_noirq (PMSG_RESUME ); +if (err) {printk (KERN_ERR "failed to start xen_suspend: %d\n" , err); goto out;
@@ -119,8 +120,6 @@ static void do_suspend(void) }else xs_suspend_cancel (); -dpm_resume_noirq (PMSG_RESUME ); - resume_devices:dpm_resume_end (PMSG_RESUME );
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 7 lines · Ian Campbell
I'm still seeing other problems with resume, the system is hung on restore and the RCU stall detection logic is triggering, unfortunately
arch_trigger_all_cpu_backtraceis not Xen compatible (uses APIC directly) so I don't get much useful info out of it. It's most likely a symptom of the actual problem rather than a problem with RCU per-se anyhow.
tick_resume() is never called on secondary processors. Presumably this is because they are offlined for suspend on native and so this is normally taken care of in the CPU onlining path. Under Xen we keep all CPUs online over a suspend.
This patch papers over the issue for me but I will investigate a more generic, less hacky, way of doing to the same.
tick_suspend is also only called on the boot CPU which I presume should be fixed too.
Ian.
arch/x86/xen/suspend.c +14 -1
@@ -1,4 +1,5 @@#include <linux/types.h> +#include <linux/clockchips.h> #include <xen/interface/xen.h> #include <xen/grant_table.h>
@@ -46,7 +50,19 @@ void xen_post_suspend(int suspend_cancelled) } +static void xen_vcpu_notify_restore (void *data) +{ +unsigned long reason = (unsigned long )data; + +/* Boot processor notified via generic timekeeping_resume() */ +if (smp_processor_id () ==0 ) +return ; + +clockevents_notify (reason,NULL ); +} +void xen_arch_resume (void ) { -/* nothing */ +smp_call_function_many (cpu_online_mask, xen_vcpu_notify_restore, + (void *)CLOCK_EVT_NOTIFY_RESUME ,1 ); }
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 11 lines · Ian Campbell
tick_resume()is never called on secondary processors. Presumably this is because they are offlined for suspend on native and so this is normally taken care of in the CPU onlining path. Under Xen we keep all CPUs online over a suspend.This patch papers over the issue for me but I will investigate a more generic, less hacky, way of doing to the same.
tick_suspendis also only called on the boot CPU which I presume should be fixed too.
Yep. I wonder how it ever worked? There's been a fair amount of change in the PM code, so that could have changed things. I don't know if there's a deep reason for not calling tick_resume() on all processors.
Rafael, tglx: suspend/resume under Xen doesn't need to hot unplug all the CPUs, so we don't; the hypervisor can manage the context save/restore for all CPUs. Is there a deep reason why timekeeping_resume() can't call the CLOCK_EVT_NOTIFY_RESUME notifier on all online CPUs?
arch/x86/xen/suspend.c void xen_post_suspend(int suspend_cancelled) · line 68
Quoted 7 lines
void
xen_arch_resume(void)
{arch/x86/xen/suspend.c +2
-1-/* nothing */ +smp_call_function_many (cpu_online_mask, xen_vcpu_notify_restore, + (void *)CLOCK_EVT_NOTIFY_RESUME ,1 ); }
This is equivalent to smp_call_function().
J
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 17 lines · Jeremy Fitzhardinge
Quoted 11 lines · Ian Campbell
tick_resume()is never called on secondary processors. Presumably this is because they are offlined for suspend on native and so this is normally taken care of in the CPU onlining path. Under Xen we keep all CPUs online over a suspend.This patch papers over the issue for me but I will investigate a more generic, less hacky, way of doing to the same.
tick_suspendis also only called on the boot CPU which I presume should be fixed too.Yep. I wonder how it ever worked? There's been a fair amount of change in the PM code, so that could have changed things. I don't know if there's a deep reason for not calling
tick_resume()on all processors.
Quoted 5 lines
Rafael, tglx: suspend/resume under Xen doesn't need to hot unplug all the CPUs, so we don't; the hypervisor can manage the context save/restore for all CPUs. Is there a deep reason why
timekeeping_resume()can't call theCLOCK_EVT_NOTIFY_RESUMEnotifier on all online CPUs?
Interrupts are disabled at that point where it currently calls the notifier, so none of the SMP function call primitives work.
Quoted 9 lines
arch/x86/xen/suspend.c void xen_post_suspend(int suspend_cancelled) · line 68
Quoted 7 lines
void
xen_arch_resume(void)
{arch/x86/xen/suspend.c +2
-1-/* nothing */ +smp_call_function_many (cpu_online_mask, xen_vcpu_notify_restore, + (void *)CLOCK_EVT_NOTIFY_RESUME ,1 ); }This is equivalent to
smp_call_function().
Oh yeah.
Ian.
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 4 lines · Ian Campbell
Quoted 4 lines
Yep. I wonder how it ever worked? There's been a fair amount of change in the PM code, so that could have changed things. I don't know if there's a deep reason for not calling
tick_resume()on all processors.
Quoted 9 lines
Quoted 6 lines
Rafael, tglx: suspend/resume under Xen doesn't need to hot unplug all the CPUs, so we don't; the hypervisor can manage the context save/restore for all CPUs. Is there a deep reason why
timekeeping_resume()can't call theCLOCK_EVT_NOTIFY_RESUMEnotifier on all online CPUs?Interrupts are disabled at that point where it currently calls the notifier, so none of the SMP function call primitives work.
That does make it pretty awkward.
J
PATCH 1/1 xen: improve error handling in do_suspend.
In reply to Ian CampbellThe existing error handling has a few issues:
- If freeze_processes() fails it exits with shutting_down = SHUTDOWN_SUSPEND.
- If dpm_suspend_noirq() fails it exits without resuming xenbus.
- If stop_machine() fails it exits without resuming xenbus or calling dpm_resume_end().
- xs_suspend()/xs_resume() and dpm_suspend_noirq()/dpm_resume_noirq() were not nested in the obvious way.
Fix by ensuring each failure case goto's the correct label. Treat a failure of stop_machine() as a cancelled suspend in order to follow the correct resume path.
--- drivers/xen/manage.c | 20 +++++++++++--------- 1 files changed, 11 insertions(+), 9 deletions(-)
drivers/xen/manage.c +11 -9
@@ -86,32 +86,32 @@ static void do_suspend(void) err =freeze_processes ();if (err) {printk (KERN_ERR "xen suspend: freeze failed %d\n" , err); -return ; + goto out; }#endif err =dpm_suspend_start (PMSG_SUSPEND );if (err) {printk (KERN_ERR "xen suspend: dpm_suspend_start %d\n" , err); - goto out; + goto out_thaw; } -printk (KERN_DEBUG "suspending xenstore...\n" ); -xs_suspend (); - err =dpm_suspend_noirq (PMSG_SUSPEND );if (err) {printk (KERN_ERR "dpm_suspend_noirq failed: %d\n" , err); - goto resume_devices; + goto out_resume; } +printk (KERN_DEBUG "suspending xenstore...\n" ); +xs_suspend (); + err =stop_machine (xen_suspend, &cancelled,cpumask_of (0 ));dpm_resume_noirq (PMSG_RESUME );if (err) {printk (KERN_ERR "failed to start xen_suspend: %d\n" , err); - goto out; + cancelled =1 ; }if (!cancelled) {
@@ -120,15 +120,17 @@ static void do_suspend(void) } elsexs_suspend_cancel (); -resume_devices: +out_resume:dpm_resume_end (PMSG_RESUME );/* Make sure timer events get retriggered on all CPUs */ clock_was_set (); -out: + +out_thaw: #ifdef CONFIG_PREEMPTthaw_processes ();#endif +out: shutting_down =SHUTDOWN_INVALID ; }#endif /* CONFIG_PM_SLEEP */
Signature · 1 line
1.5.6.5
PATCH 1/1 xen: explicitly create/destroy stop_machine workqueues outside suspend/resume region.
I have observed cases where the implicit stop_machine_destroy() done by stop_machine() hangs while destroying the workqueues, specifically in kthread_stop(). This seems to be because timer ticks are not restarted until after stop_machine() returns.
Fortunately stop_machine provides a facility to pre-create/post-destroy the workqueues so use this to ensure that workqueues are only destroyed after everything is really up and running again.
I only actually observed this failure with 2.6.30. It seems that newer kernels are somehow more robust against doing kthread_stop() without timer interrupts (I tried some backports of some likely looking candidates but did not track down the commit which added this robustness). However this change seems like a reasonable belt&braces thing to do.
--- drivers/xen/manage.c | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-)
drivers/xen/manage.c +11 -1
@@ -79,6 +79,12 @@ static void do_suspend(void) shutting_down =SHUTDOWN_SUSPEND ; + err =stop_machine_create (); +if (err) { +printk (KERN_ERR "xen suspend: failed to setup stop_machine %d\n" , err); + goto out; + } +#ifdef CONFIG_PREEMPT /* If the kernel is preemptible, we need to freeze all the processes to prevent them from being in the middle of a pagetable update
@@ -86,7 +92,7 @@ static void do_suspend(void) err =freeze_processes ();if (err) {printk (KERN_ERR "xen suspend: freeze failed %d\n" , err); - goto out; + goto out_destroy_sm; }#endif
@@ -129,7 +135,11 @@ out_resume: out_thaw: #ifdef CONFIG_PREEMPTthaw_processes (); + +out_destroy_sm: #endif +stop_machine_destroy (); + out: shutting_down =SHUTDOWN_INVALID ; }
Signature · 1 line
1.5.6.5
Re: [PATCH] xen: explicitly create/destroy stop_machine workqueues outside suspend/resume region.
Quoted 5 lines · Ian Campbell
I have observed cases where the implicit
stop_machine_destroy()done bystop_machine()hangs while destroying the workqueues, specifically inkthread_stop(). This seems to be because timer ticks are not restarted until afterstop_machine()returns.
Thanks for these - applied.
J
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
In reply to Pasi KärkkäinenQuoted 5 lines · Pasi Kärkkäinen
Hello,
Jeremy: Here's summary about these save/restore problems using upstream Linux 2.6.31.5 PV guest.
Quoted 8 lines
For me:
- I can "xm save" + "xm restore" UP guest, but I get non-fatal
BUG in the guest kernel, see [1].
- "xm save" fails for SMP guest with "failed to get the suspend evtchn port", see [2].For Dan:
- "xm save" works for UP guest, but "xm restore" doesn't, giving infinitexen_sched_clockrelated dumps in the guest kernel, see [3].
The runstate fix I sent to the list last week should help with this one.
Quoted 2 lines
- "xm save" for SMP guest fails, it never ends. I suspect this is the same problem I'm seeing.
I'm seeing this (or something very like it) too. At the moment it looks as if drivers/xen/manage.c:do_suspend is getting as far as the stop_machine() call but I am never seeing to the xen_suspend() callback.
Ian.
Re: pv 2.6.31 (kernel.org) and save/migrate fails, domU BUG
Quoted 15 lines · Ian Campbell
Quoted 5 lines · Pasi Kärkkäinen
Hello,
Jeremy: Here's summary about these save/restore problems using upstream Linux 2.6.31.5 PV guest.
Quoted 8 lines
For me:
- I can "xm save" + "xm restore" UP guest, but I get non-fatal
BUG in the guest kernel, see [1].
- "xm save" fails for SMP guest with "failed to get the suspend evtchn port", see [2].For Dan:
- "xm save" works for UP guest, but "xm restore" doesn't, giving infinitexen_sched_clockrelated dumps in the guest kernel, see [3].
Quoted 8 lines
The runstate fix I sent to the list last week should help with this one.
Quoted 2 lines
- "xm save" for SMP guest fails, it never ends. I suspect this is the same problem I'm seeing.
I'm seeing this (or something very like it) too. At the moment it looks as if drivers/xen/manage.c:do_suspend is getting as far as the
stop_machine()call but I am never seeing to thexen_suspend()callback.
See "xen: register timer interrupt with IRQF_TIMER" that I just sent to the list for the fix.
Ian.
Re: pv 2.6.31 (kernel.org) and save/migrate, domU BUG()
In reply to Dan MagenheimerQuoted 8 lines · Dan Magenheimer
Mine was SMP... switching to UP I can now save. BUT... restore doesn't seem to quite work. The restore completes but I get no response from the VNC console. When I use a tty console, after restore, I am getting
an infinite dump ofWARNING: at arch/x86/time.c:180 xen_sched_clock+0x2b
That means that the test to see that the CPU its currently running on is not currently running according to Xen... It's hard to imagine how it got into that state...
J
Quoted 11 lines · Pasi Kärkkäinen
Quoted 9 lines · Dan Magenheimer
Sorry for another possibly stupid question:
I've observed that for a pv domain that's been updated to a 2.6.31 kernel (straight from kernel.org), "xm save" never completes. When the older kernel (2.6.18) is booted, "xm save" works fine. Is this a known problem... or perhaps xm save has never worked with an upstream pv kernel and I've never noticed?
Quoted 7 lines
Quoted 4 lines
I'd assume migrate and live migrate would fail also but haven't tried them.
Just checking.. are you running the latest 2.6.31.5 ? I think there has been multiple xen related bugfixes in the 2.6.31.X releases.
Nothing relating to save/restore. Does it work for you?
J