From: Bandan Das Subject: Re: Nested VMX - L1 hangs on running L2 Date: Wed, 20 Jul 2011 16:42:29 -0400 Message-ID: <20110720204229.GB1725@stratus.com> References: <20110708184053.GA23958@stratus.com> <20110718182628.GB5324@amt.cnet> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Zachary Amsden , Bandan Das , KVM Mailing List , Marcelo Tosatti To: "Nadav Har'El" Return-path: Received: from mailhub4.stratus.com ([134.111.1.17]:42493 "EHLO mailhub4.stratus.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751368Ab1GTUnB (ORCPT ); Wed, 20 Jul 2011 16:43:01 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On 0, Nadav Har'El wrote: > > No, both patches are wrong. >=20 > Guys, thanks for looking into this bug. I'm afraid I'm still at a los= s at > why a TSC bug would even cause a guest lockup :( >=20 > When Avi Kivity saw my nested TSC handling code he remarked "this is > probably wrong". When I asked him where it was wrong, he basically sa= id > that he didn't know where, but TSC handling code is always wrong ;-) > And it turns out he was right. >=20 > > The correct fix is to make kvm_get_msr() return the L1 guest TSC at= all > times. > > =C2=A0We are serving the L1 guest in this hypervisor, not the L2 gu= est, and so >=20 > > should never read its TSC for any reason. > ... > > allows the L2 guest to overwrite the L1 guest TSC, which at first s= eems > wrong, > > but is in fact the correct virtualization of a security hole in the= L1 > guest. >=20 > I think I'm beginning to see the error in my ways... >=20 > When L1 lets L2 (using the MSR bitmap) direct read/write access to th= e TSC, > it doesn't want L0 to be "clever" and give L2 its own separate TSC (l= ike > I do now), but rather gives it full control over L1's TSC - so readin= g or > writing it should actually return L1's TSC, and the TSC_OFFSET in vmc= s12 > is to be ignored. >=20 > So basically, if I understand correctly, what I need to change is > in prepare_vmcs02(), if the MSR_IA32_TSC is on the MSR bitmap (read? > write?), instead of doing > vmcs_write64(TSC_OFFSET, > vmx->nested.vmcs01_tsc_offset + vmcs12->tsc_offset); > I just need to do > vmcs_write64(TSC_OFFSET, > vmx->nested.vmcs01_tsc_offset); > thereby giving L2 exactly the same TSC that L1 had. > Brandan, if I remember correctly you once tried this sort of fix and > it actually worked? That is correct. That is still my "workaround fix" that I have been usi= ng on my systems. But as you have mentioned above (and below), I am still = struggling with two questions : 1. Why does L1 hang even if the TSC has wrong values. 2. I see this on a Dell R610 and I don't know why you and some others = don't see this. I assumed from the symptoms that this should be fairly easy to reproduc= e on any system. Bandan=20 > Then, guest_read_tsc() will return (without need to change this code) > the correct L1 TSC. >=20 > And vmx_write_tsc_offset() should do in the is_guest_mode() not what > it does now (vmcs12->tsc_offset is of no important when the TSC MSR > is passed through) but rather set vmcs01_tsc_offset (which will be > applied on the next exit to L1). >=20 > Is my analysis correct? Or perhaps completely wrong? ;-) > Am I missing anything else that should be change? >=20 > In any case, I don't understand why on my machine I never encountered > these problems, and nothing broke even if I replaced the TSC nesting > code with randomly broken code. Are the people who are seeing this > brakage actually passed the MSR from L1 to L2 - using the MSR bitmap = - > like I guessed above? Or am I missing something completely different? >=20 > Sorry, but I'm really becoming confused by these TSC issues... >=20 > Thanks, > Nadav. >=20