move hyperv CHANNELMSG_UNLOAD from crashed kernel to kdump kernel
KY,
if a hyperv VM crashes alot of work must be done to prepare the environment for the kdump kernel. This approach is different compared to all the other VM types, or baremetal. Since the just crashed kernel is per definition unreliable all that work should be done within the kdump kernel because I think a reliable environment exists only there.
Was it ever considered to do the CHANNELMSG_UNLOAD / CHANNELMSG_UNLOAD_RESPONSE work during boot, instead of doing it before starting the kexec/kdump kernel?
What would it take to prepare the runtime environment during boot? Does the newly booted kernel need any info from the previous kernel, something that cant be determined during boot? If yes, how can such info be passed from the old kernel to the new kernel?
Olaf
Quoted 24 lines
-----Original Message-----
From: Olaf Hering [mailto:[email protected]]
Sent: Wednesday, December 7, 2016 12:51 AM
To: KY Srinivasan <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: move hypervCHANNELMSG_UNLOADfrom crashed kernel to kdump kernelKY,
if a hyperv VM crashes alot of work must be done to prepare the environment for the kdump kernel. This approach is different compared to all the other VM types, or baremetal. Since the just crashed kernel is per definition unreliable all that work should be done within the kdump kernel because I think a reliable environment exists only there.
Was it ever considered to do the
CHANNELMSG_UNLOAD/CHANNELMSG_UNLOAD_RESPONSEwork during boot, instead of doing it before
starting the kexec/kdump kernel?
Quoted 4 lines
What would it take to prepare the runtime environment during boot? Does the newly booted kernel need any info from the previous kernel, something that cant be determined during boot? If yes, how can such info be passed from the old kernel to the new kernel?
Yes; I had played with this approach a while ago. The issue is that the host knows about a bunch of in memory state that will be different in the kexec kernel. For instance if we did all the cleanup as part of the boot sequence, we will need access to all the interrupt/messaging infrastructure that was set up in the previous kernel.
K. Y
Quoted 2 lines
Olaf
Am 7. Dezember 2016 16:04:29 MEZ, schrieb KY Srinivasan <[email protected]>:
Quoted 7 lines
Yes; I had played with this approach a while ago. The issue is that the host knows about a
bunch of in memory state that will be different in the kexec kernel. For instance if we did all
the cleanup as part of the boot sequence, we will need access to all the interrupt/messaging
infrastructure that was set up in the previous kernel.
Where is that stored? Perhaps it should be put into one place, so that the new kernel can find and use it.
Olaf
Quoted 23 lines
-----Original Message-----
From: Olaf Hering [mailto:[email protected]]
Sent: Wednesday, December 7, 2016 7:47 AM
To: KY Srinivasan <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: move hypervCHANNELMSG_UNLOADfrom crashed kernel to kdump kernelAm 7. Dezember 2016 16:04:29 MEZ, schrieb KY Srinivasan <[email protected]>:
Quoted 7 lines
Yes; I had played with this approach a while ago. The issue is that the host knows about a
bunch of in memory state that will be different in the kexec kernel. For instance if we did all
the cleanup as part of the boot sequence, we will need access to all the interrupt/messaging
infrastructure that was set up in the previous kernel.
Quoted 2 lines
Where is that stored? Perhaps it should be put into one place, so that the new kernel can find and use it.
All the state required to handle interrupts is needed as well as the state needed
to issue unload. Look at vmbus_isr() for the needed interrupt handling state. The entire code path from the isr routine will have to be executed. Is there a mechanism for stashing away state that can be retrieved in the context of the execed kernel.
K. Y
Quoted 2 lines
Olaf
Quoted 2 lines · KY Srinivasan
Is there a mechanism for stashing away state that can be retrieved in the context of the execed kernel.
I have to find out. To simplify things the new approach may only be used in the kdump case, which already passes various info in cmdline. Most likely there is a way to preserve a few gpfns with the relevant data.
Olaf
Quoted 11 lines
-----Original Message-----
From: Olaf Hering [mailto:[email protected]]
Sent: Wednesday, December 7, 2016 8:19 AM
To: KY Srinivasan <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: move hypervCHANNELMSG_UNLOADfrom crashed kernel to kdump kernel
Quoted 8 lines
Quoted 2 lines · KY Srinivasan
Is there a mechanism for stashing away state that can be retrieved in the context of the execed kernel.
I have to find out. To simplify things the new approach may only be used in the kdump case, which already passes various info in cmdline. Most likely there is a way to preserve a few gpfns with the relevant data.
May be a better solution might be to have a new mechanism to indicate to the host that all state of the previous incarnation of the kernel needs to be cleaned up. This will be close to what we have on hardware.
K. Y
Quoted 2 lines
Olaf
Quoted 4 lines · KY Srinivasan
May be a better solution might be to have a new mechanism to indicate to the host that all state of the previous incarnation of the kernel needs to be cleaned up. This will be close to what we have on hardware.
That would be cool, but until this appears and until its deployed in Azure and elsewhere I will likely lose the remaining hairs.
Olaf
Quoted 13 lines
-----Original Message-----
From: Olaf Hering [mailto:[email protected]]
Sent: Wednesday, December 7, 2016 8:40 AM
To: KY Srinivasan <[email protected]>
Cc: [email protected]; [email protected]; linux- [email protected]; [email protected]
Subject: Re: move hypervCHANNELMSG_UNLOADfrom crashed kernel to kdump kernelOn Wed, Dec 07, KY Srinivasan wrote:
Quoted 7 lines
Quoted 4 lines
May be a better solution might be to have a new mechanism to indicate to the host that all state of the previous incarnation of the kernel needs to be cleaned up. This will be close to what we have on hardware.
That would be cool, but until this appears and until its deployed in Azure and elsewhere I will likely lose the remaining hairs.
I think the code we have upstream works; I have not seen any issues reported. I will try to get the host guys to implement this - hopefully it won't be too late!
K. Y
Quoted 2 lines
Olaf
Quoted 11 lines · Olaf Hering
KY,
if a hyperv VM crashes alot of work must be done to prepare the environment for the kdump kernel. This approach is different compared to all the other VM types, or baremetal. Since the just crashed kernel is per definition unreliable all that work should be done within the kdump kernel because I think a reliable environment exists only there.
Was it ever considered to do the
CHANNELMSG_UNLOAD/CHANNELMSG_UNLOAD_RESPONSEwork during boot, instead of doing it before starting the kexec/kdump kernel?
Sorry guys I missed the discussion, I was on vacation.
I see a number of minor but at least one major issue against such move:
At least for some Hyper-V versions (2012R2 for example) CHANNELMSG_UNLOAD_RESPONSE is delivered to the CPU which initially sent CHANNELMSG_REQUESTOFFERS and on kdump we may not have this CPU up as we usually do kdump with nr_cpus=1 (and on the CPU which crashed).
Minor issue is the necessity preserve the information about message/events pages across kexec.
Quoted 8 lines
What would it take to prepare the runtime environment during boot? Does the newly booted kernel need any info from the previous kernel, something that cant be determined during boot? If yes, how can such info be passed from the old kernel to the new kernel?
Olaf
Signature · 1 line
Vitaly
Quoted 11 lines · Olaf Hering
Quoted 5 lines · Vitaly Kuznetsov
I see a number of minor but at least one major issue against such move:
At least for some Hyper-V versions (2012R2 for example)CHANNELMSG_UNLOAD_RESPONSEis delivered to the CPU which initially sentCHANNELMSG_REQUESTOFFERSand on kdump we may not have this CPU up as we usually do kdump with nr_cpus=1 (and on the CPU which crashed).Since the kdump or kexec kernel will send the unload during boot I would expect the response to arrive where it was sent, independent from the number of cpus.
Wait, I just noticed that "REQUESTOFFERS" now. That might be a reason why my suggestion will not work.
Olaf
Quoted 5 lines · Vitaly Kuznetsov
I see a number of minor but at least one major issue against such move:
At least for some Hyper-V versions (2012R2 for example)CHANNELMSG_UNLOAD_RESPONSEis delivered to the CPU which initially sentCHANNELMSG_REQUESTOFFERSand on kdump we may not have this CPU up as we usually do kdump with nr_cpus=1 (and on the CPU which crashed).
Since the kdump or kexec kernel will send the unload during boot I would expect the response to arrive where it was sent, independent from the number of cpus.
Quoted 2 lines
Minor issue is the necessity preserve the information about message/events pages across kexec.
I guess this info is stored somewhere, and the relevant gfns can be preserved across kernels, if we try really hard.
But after looking further at the involved code paths it seems that the implemnted polling might be good enough to snatch the response. Was the mdelay(10) just an arbitrary decision? I interpret the comments in vmbus_signal_eom such that the host may overwrite the response. Perhaps such thing may happen during the mdelay?
Olaf
Quoted 8 lines · Olaf Hering
Quoted 5 lines · Vitaly Kuznetsov
I see a number of minor but at least one major issue against such move:
At least for some Hyper-V versions (2012R2 for example)CHANNELMSG_UNLOAD_RESPONSEis delivered to the CPU which initially sentCHANNELMSG_REQUESTOFFERSand on kdump we may not have this CPU up as we usually do kdump with nr_cpus=1 (and on the CPU which crashed).
Quoted 4 lines
Since the kdump or kexec kernel will send the unload during boot I would expect the response to arrive where it was sent, independent from the number of cpus.
We actually need to read the reply and empty the message slot to make unload happen. And reading on a different CPU may not work, see:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2016-December/097330.html
Quoted 9 lines
Quoted 2 lines
Minor issue is the necessity preserve the information about message/events pages across kexec.
I guess this info is stored somewhere, and the relevant gfns can be preserved across kernels, if we try really hard.
But after looking further at the involved code paths it seems that the implemnted polling might be good enough to snatch the response. Was the mdelay(10) just an arbitrary decision?
I observed delays up to several seconds (!) before CHANNELMSG_UNLOAD_RESPONSE is delivered.
Quoted 2 lines
I interpret the comments in
vmbus_signal_eomsuch that the host may overwrite the response. Perhaps such thing may happen during the mdelay?
No, (at least in theory) the host is never supposed to overwrite messages, it waits for the guest to clean the slot and do wrmsr.
Signature · 1 line
Vitaly
Quoted 10 lines · Olaf Hering
Quoted 8 lines · Olaf Hering
Quoted 5 lines · Vitaly Kuznetsov
I see a number of minor but at least one major issue against such move:
At least for some Hyper-V versions (2012R2 for example)CHANNELMSG_UNLOAD_RESPONSEis delivered to the CPU which initially sentCHANNELMSG_REQUESTOFFERSand on kdump we may not have this CPU up as we usually do kdump with nr_cpus=1 (and on the CPU which crashed).
Quoted 7 lines
Quoted 3 lines
Since the kdump or kexec kernel will send the unload during boot I would expect the response to arrive where it was sent, independent from the number of cpus.
Wait, I just noticed that "REQUESTOFFERS" now. That might be a reason why my suggestion will not work.
Yep, that's what I meant to say.
Signature · 1 line
Vitaly
Quoted 4 lines · Vitaly Kuznetsov
We actually need to read the reply and empty the message slot to make unload happen. And reading on a different CPU may not work, see:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2016-December/097330.html
Does the following sentences mean the vmbus_signal_eom in vmbus_wait_for_unload is a noop because the wrmsrl() is expected to happen on the other cpu instead of the current cpu?
...
- When we read the message we need to clear the slot and signal the fact to the hypervisor. In case there are more messages to this CPU pending the hypervisor will deliver the next message. The signaling is done by writing to an MSR so this can only be done on the appropriate CPU.
...
Olaf
Quoted 11 lines · Olaf Hering
Quoted 4 lines · Vitaly Kuznetsov
We actually need to read the reply and empty the message slot to make unload happen. And reading on a different CPU may not work, see:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2016-December/097330.html
Does the following sentences mean the
vmbus_signal_eominvmbus_wait_for_unloadis a noop because thewrmsrl()is expected to happen on the other cpu instead of the current cpu?
Quoted 7 lines
...
- When we read the message we need to clear the slot and signal the fact to the hypervisor. In case there are more messages to this CPU pending the hypervisor will deliver the next message. The signaling is done by writing to an MSR so this can only be done on the appropriate CPU.
...
-> K. Y., but these words were written before I implemented vmbus_wait_for_unload(), to me they just explain how we read messages.
vmbus_wait_for_unload() may be receiving a message (not necessarily the CHANNELMSG_UNLOAD_RESPONSE, we may see some other message) on the same CPU it runs and in this case wrmsrl() makes sense. In other cases it does nothing (neither good nor bad).
Signature · 1 line
Vitaly
Quoted 4 lines · Vitaly Kuznetsov
vmbus_wait_for_unload()may be receiving a message (not necessarily theCHANNELMSG_UNLOAD_RESPONSE, we may see some other message) on the same CPU it runs and in this casewrmsrl()makes sense. In other cases it does nothing (neither good nor bad).
If that other cpu has interrupts disabled it may not process a pending msg (the response may be stuck in the host queue?), and the loop can not kick the other cpus queue if a wrmsrl is just valid for the current cpu. If thats true, the response will not arrive in the loop.
Olaf
Quoted 7 lines · Olaf Hering
Quoted 4 lines · Vitaly Kuznetsov
vmbus_wait_for_unload()may be receiving a message (not necessarily theCHANNELMSG_UNLOAD_RESPONSE, we may see some other message) on the same CPU it runs and in this casewrmsrl()makes sense. In other cases it does nothing (neither good nor bad).
Quoted 5 lines
If that other cpu has interrupts disabled it may not process a pending msg (the response may be stuck in the host queue?), and the loop can not kick the other cpus queue if a wrmsrl is just valid for the current cpu. If thats true, the response will not arrive in the loop.
In case interrupts get permanently disabled on the CPU which is supposed to receive the CHANNELMSG_UNLOAD_RESPONSE message *and* there is some other message pedning in the slot for that CPU we'll hang. We may try to overcome this by sending NMIs but this is getting more and more complicated...
I'd like to see a simple fix from Hyper-V host team: always deliver CHANNELMSG_UNLOAD_RESPONSE reply to the cpu which sent CHANNELMSG_UNLOAD request. This would allow us to remove all the craziness.
Signature · 1 line
Vitaly
Quoted 2 lines · Vitaly Kuznetsov
-> K. Y., but these words were written before I implemented
vmbus_wait_for_unload(), to me they just explain how we read messages.
Another question for KY:
In my testing, while busy-looping in vmbus_wait_for_unload, I see a few "message_type==1, hdr->msgtype==2" in the hv_context.synic_message_page of the cpu which will deliver CHANNELMSG_UNLOAD_RESPONSE. These values are not listed in their enum lists. Any idea what these values mean?
Olaf
Quoted 35 lines
-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Thursday, December 15, 2016 6:32 AM
To: Olaf Hering <[email protected]>
Cc: KY Srinivasan <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: Re: move hypervCHANNELMSG_UNLOADfrom crashed kernel to kdump kernelQuoted 3 lines · Olaf Hering
Quoted 1 line · Vitaly Kuznetsov
vmbus_wait_for_unload()may be receiving a message (not necessarilythe
Quoted 1 line
Quoted 1 line
CHANNELMSG_UNLOAD_RESPONSE, we may see some other message)on the same
Quoted 8 lines
Quoted 2 lines
CPU it runs and in this case
wrmsrl()makes sense. In other cases it does nothing (neither good nor bad).If that other cpu has interrupts disabled it may not process a pending msg (the response may be stuck in the host queue?), and the loop can not kick the other cpus queue if a wrmsrl is just valid for the current cpu. If thats true, the response will not arrive in the loop.
In case interrupts get permanently disabled on the CPU which is supposed to receive the
CHANNELMSG_UNLOAD_RESPONSEmessage *and* there is some
other message pedning in the slot for that CPU we'll hang. We may try to overcome this by sending NMIs but this is getting more and more complicated...
Quoted 4 lines
I'd like to see a simple fix from Hyper-V host team: always deliver
CHANNELMSG_UNLOAD_RESPONSEreply to the cpu which sentCHANNELMSG_UNLOAD
request. This would allow us to remove all the craziness.
Agreed; I will give this feedback to the Hyper-V guys.
K. Y
Quoted 3 lines
--
Vitaly