Skip to contentSkip to conversation

New: nVMX: Add nested EPT

Bug 53611 [email protected] 6 participants 26 messages Feb 11, 2013 – Apr 8, 2015 KVM

lore
Share & tools
lore

[email protected]

author
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Summary: nVMX: Add nested EPT
Product: Virtualization
Version: unspecified
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Created an attachment (id=93101)
--> (https://bugzilla.kernel.org/attachment.cgi?id=93101)
Nested EPT patches, v2

Nested EPT means emulating EPT for an L1 guest, allowing it to use EPT when running a nested guest L2. When L1 uses EPT, it allows the L2 guest to set its own cr3 and take its own page faults without either of L0 or L1 getting involved. In many workloads this significanlty improves L2's performance over the previous two alternatives (shadow page tables over ept, and shadow page tables over shadow page tables). As an example, I measured a single-threaded "make", which has a lot of context switches and page faults, on the three options:

shadow over shadow: 105 seconds
shadow over EPT: 87 seconds (this is the default currently)
EPT over EPT: 29 seconds

single-level virtualization (with EPT): 25 seconds

So clearly nested EPT would be a big win for such workloads.

I attach a patch set which I worked on and allowed me to measure the above results. This is the same patch set I sent to KVM mailing list on August 1st, 2012, titled "nEPT v2: Nested EPT support for Nested VMX".

This patch set still needs some work: it is known to only work in some setups but not others, and the file "announce" in the attached tar lists 5 things which definitely need to be done. There were a few additional comments in the mailing list - see
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/95395

Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
author
#1 lore raw

[Bug 53611] nVMX: Add nested EPT

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Nadav Har'El <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Blocks|                            |53601
Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
#2 lore raw
In reply to [email protected]
Quoted 42 lines · On 2013-02-11 13:49, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Summary: nVMX: Add nested EPT
Product: Virtualization
Version: unspecified
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Created an attachment (id=93101)
--> (https://bugzilla.kernel.org/attachment.cgi?id=93101)
Nested EPT patches, v2

Nested EPT means emulating EPT for an L1 guest, allowing it to use EPT when running a nested guest L2. When L1 uses EPT, it allows the L2 guest to set its own cr3 and take its own page faults without either of L0 or L1 getting involved. In many workloads this significanlty improves L2's performance over the previous two alternatives (shadow page tables over ept, and shadow page tables over shadow page tables). As an example, I measured a single-threaded "make", which has a lot of context switches and page faults, on the three options:

shadow over shadow: 105 seconds
shadow over EPT: 87 seconds (this is the default currently)
EPT over EPT: 29 seconds

single-level virtualization (with EPT): 25 seconds

So clearly nested EPT would be a big win for such workloads.

I attach a patch set which I worked on and allowed me to measure the above results. This is the same patch set I sent to KVM mailing list on August 1st, 2012, titled "nEPT v2: Nested EPT support for Nested VMX".

Quoted 6 lines

This patch set still needs some work: it is known to only work in some setups but not others, and the file "announce" in the attached tar lists 5 things which definitely need to be done. There were a few additional comments in the mailing list - see
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/95395

I suppose they do not apply anymore as well. Do you have a recent tree around somewhere or plan to resume work on it?

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#3 lore raw

Hi,

On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 3 lines · On 2013-02-11 13:49, [email protected] wrote:
Quoted 2 lines

https://bugzilla.kernel.org/show_bug.cgi?id=53611
Summary: nVMX: Add nested EPT

Yikes, I didn't realize that these bugzilla edits all get spammed to the entire mailing list :( Sorry about those...

Quoted 2 lines

I suppose they do not apply anymore as well. Do you have a recent tree around somewhere or plan to resume work on it?

Unfortunately, no - I did not have time to work on these patches since August.

The reason I'm now stuffing these things into the bug tracker is that at the end of this month I am leaving IBM to a new job, so I'm pretty sure I won't have time myself to continue any work on nested VMX, and would like for the missing nested-VMX features to be documented in case someone else comes along and wants to improve it. So unfortunately, you should expect more of this bugzilla spam on the mailing list...

Nadav.

Signature · 4 lines
Nadav Har'El                        |         Monday, Feb 11 2013, 1 Adar 5773
[email protected]             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |The message above is just this
http://nadav.harel.org.il           |signature's way of propagating itself.
#4 lore raw
Quoted 16 lines · Nadav Har'El

Hi,

On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 3 lines · On 2013-02-11 13:49, [email protected] wrote:
Quoted 2 lines

https://bugzilla.kernel.org/show_bug.cgi?id=53611
Summary: nVMX: Add nested EPT

Yikes, I didn't realize that these bugzilla edits all get spammed to the entire mailing list :( Sorry about those...

Quoted 2 lines

I suppose they do not apply anymore as well. Do you have a recent tree around somewhere or plan to resume work on it?

Unfortunately, no - I did not have time to work on these patches since August.

Quoted 6 lines

The reason I'm now stuffing these things into the bug tracker is that at the end of this month I am leaving IBM to a new job, so I'm pretty sure I won't have time myself to continue any work on nested VMX, and would like for the missing nested-VMX features to be documented in case someone else comes along and wants to improve it. So unfortunately, you should expect more of this bugzilla spam on the mailing list...

A pity that you cannot finish this great work. But documenting the open issues is definitely helpful and welcome.

Best wishes,
Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#5 lore raw
In reply to Nadav Har'El
Quoted 16 lines · Nadav Har'El

Hi,

On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 3 lines · On 2013-02-11 13:49, [email protected] wrote:
Quoted 2 lines

https://bugzilla.kernel.org/show_bug.cgi?id=53611
Summary: nVMX: Add nested EPT

Yikes, I didn't realize that these bugzilla edits all get spammed to the entire mailing list :( Sorry about those...

Quoted 2 lines

I suppose they do not apply anymore as well. Do you have a recent tree around somewhere or plan to resume work on it?

Unfortunately, no - I did not have time to work on these patches since August.

Quoted 7 lines

The reason I'm now stuffing these things into the bug tracker is that at the end of this month I am leaving IBM to a new job, so I'm pretty sure I won't have time myself to continue any work on nested VMX, and would like for the missing nested-VMX features to be documented in case someone else comes along and wants to improve it. So unfortunately, you should expect more of this bugzilla spam on the mailing list...

I looked at your (old) patches, and they seem to be very useful although some of them require rebasing or rewriting. We are interested in completing the nested-VMX features.

Signature · 2 lines
Jun
Intel Open Source Technology Center
#6 lore raw
Quoted 25 lines · Jun
Quoted 16 lines · Nadav Har'El

Hi,

On Mon, Feb 11, 2013, Jan Kiszka wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 3 lines · On 2013-02-11 13:49, [email protected] wrote:
Quoted 2 lines

https://bugzilla.kernel.org/show_bug.cgi?id=53611
Summary: nVMX: Add nested EPT

Yikes, I didn't realize that these bugzilla edits all get spammed to the entire mailing list :( Sorry about those...

Quoted 2 lines

I suppose they do not apply anymore as well. Do you have a recent tree around somewhere or plan to resume work on it?

Unfortunately, no - I did not have time to work on these patches since August.

Quoted 7 lines

The reason I'm now stuffing these things into the bug tracker is that at the end of this month I am leaving IBM to a new job, so I'm pretty sure I won't have time myself to continue any work on nested VMX, and would like for the missing nested-VMX features to be documented in case someone else comes along and wants to improve it. So unfortunately, you should expect more of this bugzilla spam on the mailing list...

Quoted 3 lines

I looked at your (old) patches, and they seem to be very useful although some of them require rebasing or rewriting. We are interested in completing the nested-VMX features.

That's great news. Can you estimate when you will be able to work on it?

I will have a use case for nEPT soon - testing purposes. But working into the KVM MMU and doing the port myself may unfortunately consume too much time here.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#7 lore raw
Quoted 8 lines · Jan Kiszka
Quoted 3 lines · Jun

I looked at your (old) patches, and they seem to be very useful although some of them require rebasing or rewriting. We are interested in completing the nested-VMX features.

That's great news. Can you estimate when you will be able to work on it?

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

Quoted 6 lines

I will have a use case for nEPT soon - testing purposes. But working into the KVM MMU and doing the port myself may unfortunately consume too much time here.

Jan

--
Jun
Intel Open Source Technology Center

#8 lore raw

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 8 lines

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Quoted 8 lines

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33, while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

I've heard another report of my patch not working with newer L1/L2 - the report said that L2 failed to boot (like you reported), and also that L1 became unstable (running anything in it gave a memory fault). So it is very likely that this code still has bugs - but since I already know of errors and holes that need to be plugged (see the announcement file together with the patches), it's not very surprising :( These patches definitely need some lovin', but it's easier than starting from scratch.

Nadav.

Signature · 4 lines
Nadav Har'El                        |       Tuesday, Feb 26 2013, 16 Adar 5773
[email protected]             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |I think therefore I am. My computer
http://nadav.harel.org.il           |thinks for me, therefore I am not.
#9 lore raw
Quoted 24 lines · Nadav Har'El

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 8 lines

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Quoted 8 lines

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33, while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

Quoted 7 lines

I've heard another report of my patch not working with newer L1/L2 - the report said that L2 failed to boot (like you reported), and also that L1 became unstable (running anything in it gave a memory fault). So it is very likely that this code still has bugs - but since I already know of errors and holes that need to be plugged (see the announcement file together with the patches), it's not very surprising :( These patches definitely need some lovin', but it's easier than starting from scratch.

FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for 3.6) for a while. They work OK for my use case (static mapping) but apparently lock up L2 when starting KVM on KVM, just as reported. I didn't look into any details there, still busy with fixing other issues like CR0/CR4 handling (which I came across while adding unrestricted guest support on top of EPT).

Given that I'm porting now patches between that branch and "next" back and forth (I depend on EPT), it would be really great if someone familiar with the KVM MMU (or enough time) could port the series to the current git head. That would not solve remaining bugs but could trigger more development, maybe also help me jumping into this.

Thanks,
Jan

#10 lore raw
Quoted 40 lines · Jan Kiszka
Quoted 24 lines · Nadav Har'El

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 8 lines

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Quoted 8 lines

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33, while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

Quoted 7 lines

I've heard another report of my patch not working with newer L1/L2 - the report said that L2 failed to boot (like you reported), and also that L1 became unstable (running anything in it gave a memory fault). So it is very likely that this code still has bugs - but since I already know of errors and holes that need to be plugged (see the announcement file together with the patches), it's not very surprising :( These patches definitely need some lovin', but it's easier than starting from scratch.

FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for 3.6) for a while. They work OK for my use case (static mapping) but apparently lock up L2 when starting KVM on KVM, just as reported. I didn't look into any details there, still busy with fixing other issues like CR0/CR4 handling (which I came across while adding unrestricted guest support on top of EPT).

Quoted 6 lines

Given that I'm porting now patches between that branch and "next" back and forth (I depend on EPT), it would be really great if someone familiar with the KVM MMU (or enough time) could port the series to the current git head. That would not solve remaining bugs but could trigger more development, maybe also help me jumping into this.

I'd like to do that. See if I'll have time...

--
Gleb.

author
#23 lore raw

[Bug 53611] nVMX: Add nested EPT

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=53611

--- Comment #1 from Nadav Har'El <[email protected]> 2013-02-27 08:14:13 --- In addition to the known issues list in the "announce" file attached above, I thought of several more issues that should be considered:

1. When switching back and forth between L1 and L2 it will be a waste to throw away the EPT table already built. So I hope (need to check...) that the EPT table is cached. But what is the cache key - the cr3? But cr3 has a different meaning in L2 and L1, so it might not be correct to use that as the key.

2. When L0 swaps out pages, it needs to remove these entries in all EPT tables, including the cached EPT02 even if not currently used. Does this happen correctly?

3. If L1 uses EPT ("nested EPT") and gives us a malformed EPT12 table, we may need to inject an EPT_MISCONFIGURATION exit when building the merged EPT02 entry. Typically, we do this building (see "fetch" in paging_tmpl.h) when handling an EPT violation exit from L2, so if we encounter this problem instead of reentering L2 immediately, we should exit to L1 with an EPT misconfigration. I'm not sure exactly how to notice this problem. Perhaps the pagetable walking code, which in our case walks EPT12 already notices a problem and does something (#GP perhaps?) and we need to have it do the EPT misconfig instead. But it is possible we need to add additional tests that are not done for normal page tables - in particularly regarding reserved bits, and especially bit 5 (in EPT it is reserved, in normal page tables it is the accessed bit). This issue is low priority, as it only deals with the error path; A well-written L1 will not caused EPT configurations anyway.

Signature · 3 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
#11 lore raw
In reply to Jan Kiszka
Quoted 33 lines · Jan Kiszka
Quoted 24 lines · Nadav Har'El

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 8 lines

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Quoted 8 lines

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33, while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

Quoted 7 lines

I've heard another report of my patch not working with newer L1/L2 - the report said that L2 failed to boot (like you reported), and also that L1 became unstable (running anything in it gave a memory fault). So it is very likely that this code still has bugs - but since I already know of errors and holes that need to be plugged (see the announcement file together with the patches), it's not very surprising :( These patches definitely need some lovin', but it's easier than starting from scratch.

Quoted 6 lines

FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for 3.6) for a while. They work OK for my use case (static mapping) but apparently lock up L2 when starting KVM on KVM, just as reported. I didn't look into any details there, still busy with fixing other issues like CR0/CR4 handling (which I came across while adding unrestricted guest support on top of EPT).

I have some updates on this. We rebased the patched to the latest KVM (L0). It turned out that the version of L1 KVM/Linux matters. At that time, actually I used v3.7 kernel for L1, and the L2 didn't work as I described above. If I use v3.5 or older for L1, L2 works with the EPT patches. So, I guess some changes made to v3.6 might have exposed a bug with the nested EPT patches or somewhere. We are looking at the changes to root-cause it.

Quoted 10 lines

Given that I'm porting now patches between that branch and "next" back and forth (I depend on EPT), it would be really great if someone familiar with the KVM MMU (or enough time) could port the series to the current git head. That would not solve remaining bugs but could trigger more development, maybe also help me jumping into this.

Thanks,
Jan

Signature · 2 lines
Jun
Intel Open Source Technology Center
#12 lore raw
Quoted 41 lines · Jun
Quoted 33 lines · Jan Kiszka
Quoted 24 lines · Nadav Har'El

On Thu, Feb 14, 2013, Nakajima, Jun wrote about "Re: [Bug 53611] New: nVMX: Add nested EPT":

Quoted 8 lines

We have started looking at the pataches first. But I couldn't reproduce the results by simply applying the original patches to v3.6:
- L2 Ubuntu 12.04 (64-bit) (smp 2)
- L1 Ubuntu 12.04 (64-bit) KVM (smp 2)
- L0 Ubuntu 12.04 (64-bit)-based. kernel/KVM is v3.6 + patches (the ones in nept-v2.tgz).
https://bugzilla.kernel.org/attachment.cgi?id=93101

Quoted 8 lines

Without the patches, the L2 guest works. With it, it hangs at boot time (just black screen):
- EPT was detected by L1 KVM.
- UP L2 didn't help.
- Looks like it's looping at EPT_walk_add_generic at the same address in L0.

Will take a closer look. It would be helpful if the test configuration (e.g kernel/commit id used, L1/L2 guests) was documented as well.

I sent the patches in August 1st, and they applied to commit ade38c311a0ad8c32e902fe1d0ae74d0d44bc71e from a week earlier.

In most of my tests, L1 and L2 were old images - L1 had Linux 2.6.33, while L2 had Linux 2.6.28. In most of my tests both L1 and L2 were UP.

Quoted 7 lines

I've heard another report of my patch not working with newer L1/L2 - the report said that L2 failed to boot (like you reported), and also that L1 became unstable (running anything in it gave a memory fault). So it is very likely that this code still has bugs - but since I already know of errors and holes that need to be plugged (see the announcement file together with the patches), it's not very surprising :( These patches definitely need some lovin', but it's easier than starting from scratch.

Quoted 6 lines

FWIW, I'm playing with them on top of kvm-3.6-2 (second pull request for 3.6) for a while. They work OK for my use case (static mapping) but apparently lock up L2 when starting KVM on KVM, just as reported. I didn't look into any details there, still busy with fixing other issues like CR0/CR4 handling (which I came across while adding unrestricted guest support on top of EPT).

Quoted 7 lines

I have some updates on this. We rebased the patched to the latest KVM (L0). It turned out that the version of L1 KVM/Linux matters. At that time, actually I used v3.7 kernel for L1, and the L2 didn't work as I described above. If I use v3.5 or older for L1, L2 works with the EPT patches. So, I guess some changes made to v3.6 might have exposed a bug with the nested EPT patches or somewhere. We are looking at the changes to root-cause it.

Great to hear! Would you mind to share your work early, even when it's not yet stable?

At least regarding lockups or misbehaviors of L1 and L2, some of the patches I posted recently may help. Did you try to merge them as well?

Thanks,
Jan

#13 lore raw
In reply to Nakajima, Jun
Quoted 8 lines · Jun

I have some updates on this. We rebased the patched to the latest KVM (L0). It turned out that the version of L1 KVM/Linux matters. At that time, actually I used v3.7 kernel for L1, and the L2 didn't work as I described above. If I use v3.5 or older for L1, L2 works with the EPT patches. So, I guess some changes made to v3.6 might have exposed a bug with the nested EPT patches or somewhere. We are looking at the changes to root-cause it.

Finally I've had more time to work on this, and I think I've fixed this. The problem was that the exit qualification for EPT violation (to L1) was not accurate (enough). And I needed to save the exit qualification upon EPT violation somewhere. Today, that information is converted to error_code (see below), and we lose the information. We need to use at least the lower 3 bits when injecting EPT violation to the L1 VMM. I tried to use the upper bytes of error_code to pass part of the exit qualification, but it didn't work well. Any suggestion for the place to store the value? kvm_vcpu?

       ...
        /* It is a write fault? */
        error_code = exit_qualification & (1U << 1);
        /* ept page table is present? */
        error_code |= (exit_qualification >> 3) & 0x1;

        return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);

--
Jun
Intel Open Source Technology Center

#14 lore raw
Quoted 20 lines · Jun
Quoted 8 lines · Jun

I have some updates on this. We rebased the patched to the latest KVM (L0). It turned out that the version of L1 KVM/Linux matters. At that time, actually I used v3.7 kernel for L1, and the L2 didn't work as I described above. If I use v3.5 or older for L1, L2 works with the EPT patches. So, I guess some changes made to v3.6 might have exposed a bug with the nested EPT patches or somewhere. We are looking at the changes to root-cause it.

Finally I've had more time to work on this, and I think I've fixed this. The problem was that the exit qualification for EPT violation (to L1) was not accurate (enough). And I needed to save the exit qualification upon EPT violation somewhere. Today, that information is converted to error_code (see below), and we lose the information. We need to use at least the lower 3 bits when injecting EPT violation to the L1 VMM. I tried to use the upper bytes of error_code to pass part of the exit qualification, but it didn't work well. Any suggestion for the place to store the value? kvm_vcpu?

Quoted 7 lines
       ...
        /* It is a write fault? */
        error_code = exit_qualification & (1U << 1);
        /* ept page table is present? */
        error_code |= (exit_qualification >> 3) & 0x1;

        return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);

I don't have a full picture (already asked you to post / git-push your intermediate state), but nested related states typically go to nested_vmx, thus vcpu_vmx.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#15 lore raw
Quoted 27 lines · Jan Kiszka
Quoted 26 lines · Jun
Quoted 8 lines · Jun

I have some updates on this. We rebased the patched to the latest KVM (L0). It turned out that the version of L1 KVM/Linux matters. At that time, actually I used v3.7 kernel for L1, and the L2 didn't work as I described above. If I use v3.5 or older for L1, L2 works with the EPT patches. So, I guess some changes made to v3.6 might have exposed a bug with the nested EPT patches or somewhere. We are looking at the changes to root-cause it.

Finally I've had more time to work on this, and I think I've fixed this. The problem was that the exit qualification for EPT violation (to L1) was not accurate (enough). And I needed to save the exit qualification upon EPT violation somewhere. Today, that information is converted to error_code (see below), and we lose the information. We need to use at least the lower 3 bits when injecting EPT violation to the L1 VMM. I tried to use the upper bytes of error_code to pass part of the exit qualification, but it didn't work well. Any suggestion for the place to store the value? kvm_vcpu?

       ...
        /* It is a write fault? */
        error_code = exit_qualification & (1U << 1);
        /* ept page table is present? */
        error_code |= (exit_qualification >> 3) & 0x1;
Quoted 5 lines
Quoted 1 line
        return kvm_mmu_page_fault(vcpu, gpa, error_code, NULL, 0);

I don't have a full picture (already asked you to post / git-push your intermediate state), but nested related states typically go to nested_vmx, thus vcpu_vmx.

Ping regarding publication. I'm about to redo your porting work as we are making no progress.

Jan

#16 lore raw
Quoted 8 lines · Jan Kiszka
Quoted 4 lines

I don't have a full picture (already asked you to post / git-push your intermediate state), but nested related states typically go to nested_vmx, thus vcpu_vmx.

Ping regarding publication. I'm about to redo your porting work as we are making no progress.

Sorry about the slow progress. We've been distracted by some priority things. The patches are ready (i.e. working), but we are cleaning them up. I'll send what we have today.

--
Jun
Intel Open Source Technology Center

#17 lore raw
Quoted 6 lines · Jun
Quoted 5 lines · Jan Kiszka
Quoted 4 lines

I don't have a full picture (already asked you to post / git-push your intermediate state), but nested related states typically go to nested_vmx, thus vcpu_vmx.

Quoted 7 lines
Quoted 3 lines

Ping regarding publication. I'm about to redo your porting work as we are making no progress.

Sorry about the slow progress. We've been distracted by some priority things. The patches are ready (i.e. working), but we are cleaning them up. I'll send what we have today.

Great news, thanks a lot!

Jan

#18 lore raw
In reply to Nakajima, Jun
Quoted 3 lines · Jun

Sorry about the slow progress. We've been distracted by some priority things. The patches are ready (i.e. working), but we are cleaning them up. I'll send what we have today.

So, I have sent them, and frankly we are still cleaning up. Please bear with us.
We are also sending one more patchset to deal with EPT misconfiguration, but Linux should run in L2 on top of L1 KVM.

--
Jun
Intel Open Source Technology Center

#19 lore raw
Quoted 10 lines · Jun
Quoted 3 lines · Jun

Sorry about the slow progress. We've been distracted by some priority things. The patches are ready (i.e. working), but we are cleaning them up. I'll send what we have today.

So, I have sent them, and frankly we are still cleaning up. Please bear with us.
We are also sending one more patchset to deal with EPT misconfiguration, but Linux should run in L2 on top of L1 KVM.

The patches are mangled and unreadable. Please resend using "git send-email".

--
Gleb.

#20 lore raw
In reply to Nakajima, Jun
Quoted 9 lines · Jun
Quoted 3 lines · Jun

Sorry about the slow progress. We've been distracted by some priority things. The patches are ready (i.e. working), but we are cleaning them up. I'll send what we have today.

So, I have sent them, and frankly we are still cleaning up. Please bear with us.
We are also sending one more patchset to deal with EPT misconfiguration, but Linux should run in L2 on top of L1 KVM.

That's great but - as Gleb already said - unfortunately not yet usable. I'd like to rebase my fixes and enhancements (unrestricted guest mode specifically) on top these days, and also run some tests with a non-KVM guest. So, if git send-email is not yet working there, I would also be happy about a public git repository.

Thanks,
Jan

#21 lore raw
Quoted 6 lines · Jan Kiszka

That's great but - as Gleb already said - unfortunately not yet usable. I'd like to rebase my fixes and enhancements (unrestricted guest mode specifically) on top these days, and also run some tests with a non-KVM guest. So, if git send-email is not yet working there, I would also be happy about a public git repository.

I re-submitted the patches last night using git send-email this time. We had some email problems at that time, and I needed to use a workaround (imap-send) at that time (and it didn't work well).

Signature · 2 lines
Jun
Intel Open Source Technology Center
#22 lore raw
Quoted 9 lines · Jun
Quoted 6 lines · Jan Kiszka

That's great but - as Gleb already said - unfortunately not yet usable. I'd like to rebase my fixes and enhancements (unrestricted guest mode specifically) on top these days, and also run some tests with a non-KVM guest. So, if git send-email is not yet working there, I would also be happy about a public git repository.

Quoted 3 lines

I re-submitted the patches last night using git send-email this time. We had some email problems at that time, and I needed to use a workaround (imap-send) at that time (and it didn't work well).

I've picked them up (except for Xinhao's follow-up patch) and rebased them over next + my pending patches:

git://git.kiszka.org/linux-kvm.git queues/nept

Some patches required a bit massaging to apply, and the last one had a trivial style issue. Feel free to integrate the changes. I didn't look into functional details yet.

Instead, I've rebased my unrestricted guest mode patch plus a bunch of fixes around nEPT and that feature. See the branch above. I'm currently testing them, and it looks very good so far. Unrestricted guest mode speeds up L2's BIOS and boot loader phase noticeably.

Jan

author
#25 lore raw

[Bug 53611] nVMX: Add nested EPT

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=53611

Paolo Bonzini <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
                 CC|                            |[email protected]
     Kernel Version|                            |3.19
         Resolution|---                         |CODE_FIX

--- Comment #2 from Paolo Bonzini <[email protected]> ---
Fixed by commit afa61f752ba6 (Advertise the support of EPT to the L1 guest, through the appropriate MSR., 2013-08-07)

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.