From: Yanfei Zhang Subject: Re: [PATCH v2 0/5] Export offsets of VMCS fields as note information for kdump Date: Mon, 28 May 2012 13:25:20 +0800 Message-ID: <4FC30C40.80500@cn.fujitsu.com> References: <4FB35C48.30708@cn.fujitsu.com> <4FB92D5A.3060507@redhat.com> <4FB9A92D.7050108@cn.fujitsu.com> <4FB9FE08.4050905@redhat.com> <4FBA05F6.8070804@cn.fujitsu.com> <4FBA0C8A.2050003@redhat.com> <4FBB0ACA.2040907@cn.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: mtosatti@redhat.com, ebiederm@xmission.com, luto@mit.edu, Joerg Roedel , dzickus@redhat.com, paul.gortmaker@windriver.com, ludwig.nussel@suse.de, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kexec@lists.infradead.org, Greg KH To: Avi Kivity Return-path: In-Reply-To: <4FBB0ACA.2040907@cn.fujitsu.com> Sender: linux-kernel-owner@vger.kernel.org List-Id: kvm.vger.kernel.org Hello Avi, =D3=DA 2012=C4=EA05=D4=C222=C8=D5 11:40, Yanfei Zhang =D0=B4=B5=C0: > =D3=DA 2012=C4=EA05=D4=C221=C8=D5 17:36, Avi Kivity =D0=B4=B5=C0: >> On 05/21/2012 12:08 PM, Yanfei Zhang wrote: >>> =D3=DA 2012=C4=EA05=D4=C221=C8=D5 16:34, Avi Kivity =D0=B4=B5=C0: >>>> On 05/21/2012 05:32 AM, Yanfei Zhang wrote: >>>>> =D3=DA 2012=C4=EA05=D4=C221=C8=D5 01:43, Avi Kivity =D0=B4=B5=C0: >>>>>> On 05/16/2012 10:50 AM, zhangyanfei wrote: >>>>>>> This patch set exports offsets of VMCS fields as note informati= on for >>>>>>> kdump. We call it VMCSINFO. The purpose of VMCSINFO is to retri= eve >>>>>>> runtime state of guest machine image, such as registers, in hos= t >>>>>>> machine's crash dump as VMCS format. The problem is that VMCS i= nternal >>>>>>> is hidden by Intel in its specification. So, we slove this prob= lem >>>>>>> by reverse engineering implemented in this patch set. The VMCSI= NFO >>>>>>> is exported via sysfs to kexec-tools just like VMCOREINFO. >>>>>>> >>>>>>> Here are two usercases for two features that we want. >>>>>>> >>>>>>> 1) Create guest machine's crash dumpfile from host machine's cr= ash dumpfile >>>>>>> >>>>>>> In general, we want to use this feature on failure analysis for= the system >>>>>>> where the processing depends on the communication between host = and guest >>>>>>> machines to look into the system from both machines's viewpoint= s. >>>>>>> >>>>>>> As a concrete situation, consider where there's heartbeat monit= oring >>>>>>> feature on the guest machine's side, where we need to determine= in >>>>>>> which machine side the cause of heartbeat stop lies. In our act= ual >>>>>>> experiments, we encountered such situation and we found the cau= se of >>>>>>> the bug was in host's process schedular so guest machine's vcpu= stopped >>>>>>> for a long time and then led to heartbeat stop. >>>>>>> >>>>>>> The module that judges heartbeat stop is on guest machine, so w= e need >>>>>>> to debug guest machine's data. But if the cause lies in host ma= chine >>>>>>> side, we need to look into host machine's crash dump. >>>>>> >>>>>> Do you mean, that a heartbeat failure in the guest lead to host = panic? >>>>>> >>>>>> My expectation is that a problem in the guest will cause the gue= st to >>>>>> panic and perhaps produce a dump; the host will remain up. >>>>>> >>>>> >>>>> The point is that before our investigation, we didn't know which = side=20 >>>>> leads to this buggy situation. Maybe a bug in host machine or the= guest >>>>> machine itself causes a heartbeat failure. >>>> >>>> How can a guest bug cause a host panic? >>>> >>>>> So we want to get both host machine's crash dump and guest machin= e's >>>>> crash dump *at the same time*. Then we could use userspace tools = to >>>>> get guest machine crash dump from host machine's and analyse them >>>>> separately to find which side causes the problem. >>>>> >>>> >>>> If the guest caused the problem, there would be no panic; therefor= e >>>> there was a host bug. >>>> >>> >>> Yes, a guest bug cannot cause a host panic. When heartbeat stops in= guest >>> machine, we could trigger the host dump mechanism to work. This is = because >>> we want to get the status of both host and guest machine at the sam= e time >>> when heartbeat stops in guest machine. Then we can look for bug rea= sons >>> from both host machine's and guest machine's views. >> >> That sounds like a bad idea. Can you explain in what situation it ma= kes >> sense for a guest to stop the host (and all other guests running on = it) >> rather than just restarting the failed services (on the host or othe= r >> guests)? >> >=20 > We never do this on customer's environment which maybe a host with ma= ny guests > running on it. We do this on another environment to reproduce the bug= gy > situation; or we do this in testing phase on development environment = towards > production one on the customer's site. >=20 >>>>>>> Without this feature, we first create guest machine's dump and = then >>>>>>> create host mahine's, but there's only a short time between two >>>>>>> processings, during which it's unlikely that buggy situation re= mains. >>>>>>> >>>>>>> So, we think the feature is useful to debug both guest machine'= s and >>>>>>> host machine's sides at the same time, and expect we can make f= ailure >>>>>>> analysis efficiently. >>>>>>> >>>>>>> Of course, we believe this feature is commonly useful on the si= tuation >>>>>>> where guest machine doesn't work well due to something of host = machine's. >>>>>>> >>>>>>> 2) Get offsets of VMCS information on the CPU running on the ho= st machine >>>>>>> >>>>>>> If kdump doesn't work well, then it means we cannot use kvm API= to get >>>>>>> register values of guest machine and they are still left on its= vmcs >>>>>>> region. In the case, we use crash dump mechanism running outsid= e of >>>>>>> linux kernel, such as sadump, a firmware-based crash dump. Then= VMCS >>>>>>> information is then necessary. >>>>>> >>>>>> Shouldn't sadump then expose the VMCS offsets? Perhaps bundling = them >>>>>> into its dump file? >>>>>> >>>>> >>>>> Firmware-based crash dump doesn't concern the os running on the m= achine. >>>>> So it will not do any os handling when machine crashes. >>>> >>>> Seems to me the VMCS offsets are OS independent. >>>> >>> Hmm, you mean we could get VMCS offsets in sadump itself? >>> But I think if we just export VMCS offsets in kernel, we could use = the current >>> existing dump tools with no or just very tiny change. I think this = could be >>> a more general mechanism than making changes in all kinds of dump t= ools. >> >> The sadump tool generates a core file with the OS image, right? Can = it >> not attach the offsets to a note, just like you propose for kdump? >> >=20 > Both are right. Dou you have any comments about this patch set? Thanks Zhang Yanfei