domU suspend issue - freeze processes failed - Linux 6.16
Hi,
When suspending domU I get the following issue:
Freezing user space processes
Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006
Call Trace:
<TASK>
__schedule+0x2f3/0x780
schedule+0x27/0x80
schedule_preempt_disabled+0x15/0x30
__mutex_lock.constprop.0+0x49f/0x880
unregister_xenbus_watch+0x216/0x230
xenbus_write_watch+0xb9/0x220
xenbus_file_write+0x131/0x1b0
vfs_writev+0x26c/0x3d0
? do_writev+0xeb/0x110
do_writev+0xeb/0x110
do_syscall_64+0x84/0x2c0
? do_syscall_64+0x200/0x2c0
? generic_handle_irq+0x3f/0x60
? syscall_exit_work+0x108/0x140
? do_syscall_64+0x200/0x2c0
? __irq_exit_rcu+0x4c/0xe0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x79b618138642
RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642
RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014
RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014
R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000
</TASK>
OOM killer enabled.
Restarting tasks: Starting
Restarting tasks: Done
xen:manage: do_suspend: freeze processes failed -16The process in question is xl devd daemon. It's a domU serving a xenvif backend.
I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16
Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157
Especially, there is another call trace with panic_on_warn enabled - slightly different, but looks related.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Jürgen Groß
maintainerQuoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.
I'm pretty sure the PV variant for suspending is just wrong: it is calling dpm_suspend_start() from do_suspend() without taking the required system_transition_mutex, resulting in the WARN() in pm_restrict_gfp_mask().
It might be as easy as just adding the mutex() call to do_suspend(), but I'm really not sure that will be a proper fix.
Juergen
Quoted 55 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.
Quoted 6 lines
I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.
Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().
Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason why do_suspend() is not using pm_suspend() and register Xen-specific actions via platform_suspend_ops (and maybe syscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
[1] https://github.com/marmarek/linux/commit/47cfdb991c85566c9c333570511e67bf477a5da6
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.
Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) Hybernation
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).
the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if platform_suspend_ops can be used.
Generic suspend stages list
- freeze
- prepare
- suspend
- suspend_late
- suspend_noirq (SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
- disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
- syscore_suspend
[rest here]
- platform->enter() (suspended)
You can't just overwrite platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const struct platform_suspend_ops psci_suspend_ops = {
As an option, some Xen components could be converted to use syscore_ops (but not xenstore), and some might need to use DD(dev_pm_ops).
Signature · 2 lines
Best regards, -grygorii
Quoted 85 lines · Grygorii Strashko
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) Hybernation
Quoted 7 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through
pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()
IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.
Quoted 3 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 18 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)
Quoted 4 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {
Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 10 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore), and some might need to useDD(dev_pm_ops).Signature · 2 lines
Best regards, -grygorii
[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 142 lines · Marek Marczykowski-Górecki
Quoted 85 lines · Grygorii Strashko
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 7 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through
pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 3 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 18 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)Quoted 4 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 4 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore), and some might need to useDD(dev_pm_ops).Quoted 1 line
Quoted 9 lines
Quoted 6 lines
--
Best regards,
-grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
On my setup I get a weird behavior when trying to suspend (s2idle) a Linux guest.
Doing echo freeze > /sys/power/state in the guest seems to "freeze" the guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge control request: -9
shutdown failed (rc=-9)
Do you think it's related to your current issue?
Regards,
--
--
Yann Sionneau | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 165 lines · Yann Sionneau
Quoted 142 lines · Marek Marczykowski-Górecki
Quoted 85 lines · Grygorii Strashko
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 7 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through
pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 3 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 18 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)Quoted 4 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 4 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore), and some might need to useDD(dev_pm_ops).Quoted 1 line
Quoted 9 lines
Quoted 6 lines
--
Best regards,
-grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
On my setup I get a weird behavior when trying to suspend (s2idle) a Linux guest.
Doing echo freeze >/sys/power/statein the guest seems to "freeze" the guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge control request: -9
shutdown failed (rc=-9)
Quoted 1 line
Do you think it's related to your current issue?
Maybe? Is it a HVM guest or PV?
Regarding s2idle, we have another patch:
https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
but it fixes wakeup, not really going to sleep.
It was posted also to this ML, but still waiting for review (see link in the patch header).
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 165 lines · Yann Sionneau
Quoted 142 lines · Marek Marczykowski-Górecki
Quoted 85 lines · Grygorii Strashko
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 7 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through
pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 3 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 18 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)Quoted 4 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 4 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore), and some might need to useDD(dev_pm_ops).Quoted 1 line
Quoted 9 lines
Quoted 6 lines
Signature · 2 lines
Best regards, -grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
On my setup I get a weird behavior when trying to suspend (s2idle) a Linux guest.
Doing echo freeze >/sys/power/statein the guest seems to "freeze" the guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge control request: -9
shutdown failed (rc=-9)
Quoted 1 line
Do you think it's related to your current issue?
idle=halt on the Linux command line addresses the 100% CPU usage. Or alternatively C2 needs to be implemented for guest vcpus. I forget preceisely, but I think the 100% CPU is because there are no C-states available and Linux/cpuidle won't use halt by default.
To wake up, you need a wake up source. The ACPI buttons presses will do that:
xl trigger $dom power
xl trigger $dom sleep
However, I think without changes, domU s2idle/S3 will detach all its PV devices. Naturally they don't get reconnected on resume. You can hack around that to skip the detach.
Actually, maybe we just need:
drivers/xen/xenbus/xenbus_probe_frontend.c +1 -3
@@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct device *_dev) }static const struct dev_pm_ops xenbus_pm_ops = { - .suspend = xenbus_dev_suspend, - .resume = xenbus_frontend_dev_resume, .freeze = xenbus_dev_suspend, .thaw = xenbus_dev_cancel, - .restore = xenbus_dev_resume, + .restore = xenbus_frontend_dev_resume,
};
b3e96c0c7562 ("xen: use freeze/restore/thaw PM events for suspend/resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks remained. But freeze and suspend being identical doesn't seem correct.
This would leave xl save/restore/migrate using the hibernate freeze/thaw/resume. S3/s2idle would no touch the PV devices, so they would still be present on resume. Maybe there are cases I am not thinking of though.
Regards,
Jason
Quoted 234 lines · Jason Andryuk
Quoted 196 lines · Yann Sionneau
Quoted 174 lines · Marek Marczykowski-Górecki
Quoted 106 lines · Grygorii Strashko
Quoted 92 lines · Marek Marczykowski-Górecki
Quoted 74 lines · Marek Marczykowski- Górecki
Quoted 73 lines · Jürgen Groß
Quoted 58 lines · Marek Marczykowski-Górecki
Quoted 47 lines · Marek Marczykowski- Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes
Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0):
task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX:0000000000000014
RAX: ffffffffffffffda RBX:00000000024fd490RCX:000079b618138642
RDX: 0000000000000003 RSI:00007fff9a193120RDI:
0000000000000014
RBP:00007fff9a193000R08: 0000000000000000 R09:
0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000014
R13:00007fff9a193120R14: 0000000000000003 R15:
0000000000000000</TASK>OOM killer enabled.
Restarting tasks: Starting
Restarting tasks: Done
xen:manage:do_suspend: freeze processes failed -16Quoted 8 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with
6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
seemingly no relevant changes between rc2 and rc6).Quoted 7 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled -
slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 7 lines
Quoted 3 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm
really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure
quoted here above, I think?While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops
(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 5 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 8 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
started throughpm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 6 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 20 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)Quoted 5 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter
suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 8 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore),
and some might need to useDD(dev_pm_ops).Quoted 3 lines
[1] https://github.com/marmarek/linux/
commit/47cfdb991c85566c9c333570511e67bf477a5da6Quoted 8 lines
Quoted 4 lines
--
Best regards,
-grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-
use-suspend.patchOn my setup I get a weird behavior when trying to suspend (s2idle) a Linux guest.
Doing echo freeze >/sys/power/statein the guest seems to "freeze" the guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge control request: -9
shutdown failed (rc=-9)Quoted 1 line
Do you think it's related to your current issue?
idle=halt on the Linux command line addresses the 100% CPU usage. Or alternatively C2 needs to be implemented for guest vcpus. I forget preceisely, but I think the 100% CPU is because there are no C-states available and Linux/cpuidle won't use halt by default.
To wake up, you need a wake up source. The ACPI buttons presses will do that:
xl trigger $dom power
xl trigger $dom sleepHowever, I think without changes, domU s2idle/S3 will detach all its PV devices. Naturally they don't get reconnected on resume. You can hack around that to skip the detach.
Actually, maybe we just need:
drivers/xen/xenbus/xenbus_probe_frontend.c +1
-3@@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct device *_dev) } static const struct dev_pm_ops xenbus_pm_ops = { - .suspend = xenbus_dev_suspend, - .resume = xenbus_frontend_dev_resume, .freeze = xenbus_dev_suspend, .thaw = xenbus_dev_cancel, - .restore = xenbus_dev_resume, + .restore = xenbus_frontend_dev_resume, };
b3e96c0c7562("xen: use freeze/restore/thaw PM events for suspend/ resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to PMSG_FREEZE/ PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks remained. But freeze and suspend being identical doesn't seem correct.
Quoted 8 lines
This would leave xl save/restore/migrate using the hibernate freeze/ thaw/resume. S3/s2idle would no touch the PV devices, so they would still be present on resume. Maybe there are cases I am not thinking of though.
Regards,
Jason
Hi all,
I finally took time to try what you guyz advised, thanks for all the answers!
So, I tried this on a clean Debian 13 VM install on a XCP-ng 8.3 (Xen 4.17.5 with local patches) host.
1/ Just booting with idle=halt/nomwait/poll didn't help with waking up the VM, it kept being in a weird unwakable state. Even providing xl trigger power/sleep events. Although I reckon you were saying it would help with the CPU@100% and indeed I don't see this anymore.
2/ Just applying QubesOS patch
https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
did help *a bit*. Hitting a key would wake the VM, then show again the Debian desktop, with some flickering, allowing to move the mouse and bit and hitting some keys on the keyboard ... then it would very quickly completely freeze and stay dead.
3/ on top of previous patch, applying your modification (removing .suspend, .resume and changing .restore to xenbus_frontend_dev_resume):
then tadaaa I can wake the VM and it stays alive afterward. It seems it fixes the issue (I did not test for very very long though).
Thanks for the help on this!
What do you think we should do to move this forward? Submit those changes to upstream Linux? That will take long to end up in distros...
--
--
Yann Sionneau | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 249 lines · Yann Sionneau
Quoted 234 lines · Jason Andryuk
Quoted 196 lines · Yann Sionneau
Quoted 174 lines · Marek Marczykowski-Górecki
Quoted 106 lines · Grygorii Strashko
Quoted 92 lines · Marek Marczykowski-Górecki
Quoted 74 lines · Marek Marczykowski- Górecki
Quoted 73 lines · Jürgen Groß
Quoted 58 lines · Marek Marczykowski-Górecki
Quoted 47 lines · Marek Marczykowski- Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes
Freezing user space processes failed after 20.004
seconds (1 tasks refusing to freeze, wq_busy=0):
task:xl state:D stack:0 pid:466
tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX:0000000000000014
RAX: ffffffffffffffda RBX:00000000024fd490RCX:000079b618138642
RDX: 0000000000000003 RSI:00007fff9a193120RDI:
0000000000000014
RBP:00007fff9a193000R08: 0000000000000000 R09:
0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000014
R13:00007fff9a193120R14: 0000000000000003 R15:
0000000000000000</TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 8 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with
6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given
seemingly no relevant changes between rc2 and rc6).Quoted 7 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled -
slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 7 lines
Quoted 3 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm
really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure
quoted here above, I think?While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops
(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 5 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I
surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 8 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always
started throughpm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 6 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM",
but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE.
As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 20 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)Quoted 5 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter
suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
Quoted 8 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore),
and some might need to useDD(dev_pm_ops).Quoted 3 lines
[1] https://github.com/marmarek/linux/
commit/47cfdb991c85566c9c333570511e67bf477a5da6Quoted 8 lines
Quoted 4 lines
Signature · 2 lines
Best regards, -grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-
use-suspend.patchOn my setup I get a weird behavior when trying to suspend (s2idle) a Linux guest.
Doing echo freeze >/sys/power/statein the guest seems to "freeze" the guest for good, I could not unfreeze it afterward.
VCPU goes to 100% according to XenOrchestra
xl list shows state "r" but xl console blocks forever
xl shutdown would block for some time and then print:
Shutting down domain 721
?ibxl: error: libxl_domain.c:848:pvcontrol_cb: guest didn't acknowledge control request: -9
shutdown failed (rc=-9)Quoted 1 line
Do you think it's related to your current issue?
idle=halt on the Linux command line addresses the 100% CPU usage. Or alternatively C2 needs to be implemented for guest vcpus. I forget preceisely, but I think the 100% CPU is because there are no C-states available and Linux/cpuidle won't use halt by default.
To wake up, you need a wake up source. The ACPI buttons presses will do that:
xl trigger $dom power
xl trigger $dom sleepHowever, I think without changes, domU s2idle/S3 will detach all its PV devices. Naturally they don't get reconnected on resume. You can hack around that to skip the detach.
Actually, maybe we just need:
drivers/xen/xenbus/xenbus_probe_frontend.c +1
-3@@ -148,8 +148,6 @@ static void xenbus_frontend_dev_shutdown(struct device *_dev) } static const struct dev_pm_ops xenbus_pm_ops = { - .suspend = xenbus_dev_suspend, - .resume = xenbus_frontend_dev_resume, .freeze = xenbus_dev_suspend, .thaw = xenbus_dev_cancel, - .restore = xenbus_dev_resume, + .restore = xenbus_frontend_dev_resume, };
b3e96c0c7562("xen: use freeze/restore/thaw PM events for suspend/ resume/chkpt") changed from PMSG_SUSPEND/PMSG_RESUME to PMSG_FREEZE/ PMSG_THAW/PMSG_RESTORE, but the suspend/resume callbacks remained. But freeze and suspend being identical doesn't seem correct.Quoted 8 lines
This would leave xl save/restore/migrate using the hibernate freeze/ thaw/resume. S3/s2idle would no touch the PV devices, so they would still be present on resume. Maybe there are cases I am not thinking of though.
Regards,
JasonHi all,
I finally took time to try what you guyz advised, thanks for all the answers!
Quoted 7 lines
So, I tried this on a clean Debian 13 VM install on a XCP-ng 8.3 (Xen 4.17.5 with local patches) host.
1/ Just booting with idle=halt/nomwait/poll didn't help with waking up the VM, it kept being in a weird unwakable state. Even providing xl trigger power/sleep events. Although I reckon you were saying it would help with the CPU@100% and indeed I don't see this anymore.
Yes, just 100% CPU.
Quoted 6 lines
2/ Just applying QubesOS patch
https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-events-Add-wakeup-support-to-xen-pirq.patch
did help *a bit*. Hitting a key would wake the VM, then show again the Debian desktop, with some flickering, allowing to move the mouse and bit and hitting some keys on the keyboard ... then it would very quickly completely freeze and stay dead.
I think this is needed for dom0 s2idle, but not for domU.
Quoted 8 lines
3/ on top of previous patch, applying your modification (removing .suspend, .resume and changing .restore to
xenbus_frontend_dev_resume):
then tadaaa I can wake the VM and it stays alive afterward. It seems it fixes the issue (I did not test for very very long though).Thanks for the help on this!
What do you think we should do to move this forward? Submit those changes to upstream Linux? That will take long to end up in distros...
I submitted the patch for #3. Yes, it'll take a little while to get to distros, but I don't see any other choice here.
Regards,
Jason
Quoted 129 lines · Marek Marczykowski-Górecki
Quoted 85 lines · Grygorii Strashko
Quoted 73 lines · Marek Marczykowski-Górecki
Quoted 60 lines · Marek Marczykowski-Górecki
Quoted 59 lines · Jürgen Groß
Quoted 47 lines · Marek Marczykowski-Górecki
Quoted 39 lines · Marek Marczykowski-Górecki
Hi,
When suspending domU I get the following issue:
Freezing user space processes Freezing user space processes failed after 20.004 seconds (1 tasks refusing to freeze, wq_busy=0): task:xl state:D stack:0 pid:466 tgid:466 ppid:1 task_flags:0x400040 flags:0x00004006 Call Trace: <TASK> __schedule+0x2f3/0x780 schedule+0x27/0x80 schedule_preempt_disabled+0x15/0x30 __mutex_lock.constprop.0+0x49f/0x880 unregister_xenbus_watch+0x216/0x230 xenbus_write_watch+0xb9/0x220 xenbus_file_write+0x131/0x1b0 vfs_writev+0x26c/0x3d0 ? do_writev+0xeb/0x110 do_writev+0xeb/0x110 do_syscall_64+0x84/0x2c0 ? do_syscall_64+0x200/0x2c0 ? generic_handle_irq+0x3f/0x60 ? syscall_exit_work+0x108/0x140 ? do_syscall_64+0x200/0x2c0 ? __irq_exit_rcu+0x4c/0xe0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x79b618138642 RSP: 002b:00007fff9a192fc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014 RAX: ffffffffffffffda RBX: 00000000024fd490 RCX: 000079b618138642 RDX: 0000000000000003 RSI: 00007fff9a193120 RDI: 0000000000000014 RBP: 00007fff9a193000 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000014 R13: 00007fff9a193120 R14: 0000000000000003 R15: 0000000000000000 </TASK> OOM killer enabled. Restarting tasks: Starting Restarting tasks: Done xen:manage: do_suspend: freeze processes failed -16Quoted 6 lines
The process in question is
xl devddaemon. It's a domU serving a xenvif backend.I noticed it on 6.16.1, but looking at earlier test logs I see it with 6.16-rc6 already (but interestingly, not 6.16-rc2 yet? feels weird given seemingly no relevant changes between rc2 and rc6).
Quoted 6 lines
I forgot to include link for (a little) more details:
https://github.com/QubesOS/qubes-linux-kernel/pull/1157Especially, there is another call trace with
panic_on_warnenabled - slightly different, but looks related.I'm pretty sure the PV variant for suspending is just wrong: it is calling
dpm_suspend_start()fromdo_suspend()without taking the requiredsystem_transition_mutex, resulting in theWARN()inpm_restrict_gfp_mask().Quoted 5 lines
Quoted 2 lines
It might be as easy as just adding the
mutex()call todo_suspend(), but I'm really not sure that will be a proper fix.Hm, this might explain the second call trace, but not the freeze failure quoted here above, I think?
While the patch I sent appears to fix this particular issue, it made me
wonder: is there any fundamental reason whydo_suspend()is not usingpm_suspend()and register Xen-specific actions viaplatform_suspend_ops(and maybesyscore_ops)? From a brief look at the code, it should theoretically be possible, and should avoid issues like this.Quoted 4 lines
I tried to do a quick&dirty attempt at that[1], and it failed (panic). I surely made several mistakes there (and also left a ton of todo comments). But before spending any more time at that, I'd like to ask if this is a viable option at all.
I think it might, but be careful with this, because there are two "System Low power" paths in Linux
1) Suspend2RAM and Co
2) HybernationQuoted 7 lines
While "Suspend2RAM and Co" path is relatively straight forward and expected to be always started through
pm_suspend(). In general, it's expected to happen
- from sysfs (User space)
- from autosuspend (wakelocks).the "hibernation" path is more complicated:(
- Genuine Linux hybernation hibernate()/hibernate_quiet_exec()IIUC hibernation is very different as it puts Linux in charge of dumping all the state to the disk. In case of Xen, the primary use case for suspend is preparing VM for Xen toolstack serializing its state to disk (or migrating to another host).
Additionally, VM suspend may be used as preparation for host suspend (this is what I actually do here). This is especially relevant if the VM has some PCI passthrough - to properly suspend (and resume) devices across host suspend.Quoted 3 lines
I'm not sure what path Xen originally implemented :( It seems like "suspend2RAM", but, at the same time "hybernation" specific staff is used, like PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE. As result, Linux suspend/hybernation code moves forward while Xen stays behind and unsync.
Yeah, I think it's supposed to be suspend2RAM. TBH the PMSG_FREEZE/PMSG_THAW/PMSG_RESTORE confuses me too and Qubes OS has a patch[2] to switch it to PMSG_SUSPEND/PMSG_RESUME.
Quoted 18 lines
So it sounds reasonable to avoid custom implementation, but may be not easy :(
Suspending Xen features can be split between suspend stages, but not sure if
platform_suspend_opscan be used.Generic suspend stages list
- freeze
- prepare
- suspend
-suspend_late
-suspend_noirq(SPIs disabled, except wakeups)
[most of Xen specific staff has to be suspended at this point]
-disable_secondary_cpus
- arch disable IRQ (from this point no IRQs allowed, no timers, no scheduling)
-syscore_suspend
[rest here]
-platform->enter()(suspended)
Quoted 8 lines
Quoted 4 lines
You can't just overwrite
platform_suspend_ops, because ARM64 is expected to enter suspend through PSCI FW interface:drivers/firmware/psci/psci.c
static const structplatform_suspend_opspsci_suspend_ops= {Does this apply to a VM on ARM64 too? At least on x86, the VM is supposed to make a hypercall to tell Xen it suspended (the hypercall will return only on resume).
On ARM64 Guest expected to trigger PSCI(SYSTEM_SUSPEND) (which is HVC - trap, similar to hypercall on x86) PSCI is Arm Power State Coordination Interface which defines unified FW interface for PM. So if you overwrite platform_suspend_ops (which is Singleton) it will break ARM Guest suspend.
Quoted 14 lines
Quoted 11 lines
As an option, some Xen components could be converted to use
syscore_ops(but not xenstore), and some might need to useDD(dev_pm_ops).Signature · 2 lines
Best regards, -grygorii[2] https://github.com/QubesOS/qubes-linux-kernel/blob/main/xen-pm-use-suspend.patch
Signature · 2 lines
Best regards, -grygorii