Allow removing writable mappings from splintered page tables.
PATCH 1/1 Allow removing writable mappings from splintered page tables.
Hello,
The moving of the pagetable mapping in the linux kernel exposed the fact that under the linux kernel sh_rm_write_access_from_sl1p was always failing.
Linux seems to use big pages to access page tables, so we should instruct the shadow code to be able to remove writable mappings from splintered pagetables as well, avoiding using OS heuristic (which were failing in 2.6.27 before George patch, leading to brute-force search at each resync).
Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.
Thanks,
Gianluca
xen/arch/x86/mm/shadow/common.c +7 -4
@@ -2462,22 +2462,25 @@ int sh_remove_write_access_from_sl1p(strASSERT (mfn_valid (smfn));ASSERT (mfn_valid (gmfn)); -if ( sp->type == SH_type_l1_32_shadow ) +if ( sp->type == SH_type_l1_32_shadow + || sp->type == SH_type_fl1_32_shadow ) {return SHADOW_INTERNAL_NAME (sh_rm_write_access_from_sl1p,2 ) (v, gmfn, smfn, off); } #if CONFIG_PAGING_LEVELS >= 3 - else if ( sp->type == SH_type_l1_pae_shadow ) + else if ( sp->type == SH_type_l1_pae_shadow + || sp->type == SH_type_fl1_pae_shadow )return SHADOW_INTERNAL_NAME (sh_rm_write_access_from_sl1p,3 ) (v, gmfn, smfn, off); #if CONFIG_PAGING_LEVELS >= 4 - else if ( sp->type == SH_type_l1_64_shadow ) + else if ( sp->type == SH_type_l1_64_shadow + || sp->type == SH_type_fl1_64_shadow )return SHADOW_INTERNAL_NAME (sh_rm_write_access_from_sl1p,4 ) (v, gmfn, smfn, off);#endif #endif - + return 0; }#endif
xen/arch/x86/mm/shadow/multi.c +2 -1
@@ -4539,7 +4539,8 @@ int sh_rm_write_access_from_sl1p(struct sp =mfn_to_shadow_page (smfn);if ( sp->mbz !=0 - || (sp->type != SH_type_l1_shadow) ) + || (sp->type != SH_type_l1_shadow + && sp->type != SH_type_fl1_shadow) ) goto fail; sl1p =sh_map_domain_page (smfn);
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 5 lines · Gianluca Guida
Hello,
The moving of the pagetable mapping in the linux kernel exposed the fact that under the linux kernel
sh_rm_write_access_from_sl1pwas always failing.
Quoted 9 lines
Linux seems to use big pages to access page tables, so we should instruct the shadow code to be able to remove writable mappings from splintered pagetables as well, avoiding using OS heuristic (which were failing in 2.6.27 before George patch, leading to brute-force search at each resync).
Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.
Sure, I can give it a try. Should I also apply George's patch right?
I am not using ioemu stub-domains yet, would you recommend them in a general sense or are they more of a special case?
Cheers,
Todd
Quoted 6 lines · Todd Deshane
Quoted 4 lines
Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.
Sure, I can give it a try. Should I also apply George's patch right?
Yes.
Quoted 2 lines
I am not using ioemu stub-domains yet, would you recommend them in a general sense or are they more of a special case?
I think they're in general recommendable. Both Samuel and Stefano will be able to give more information about it.
Thanks,
Gianluca
Quoted 6 lines · Gianluca Guida, le Fri 12 Sep 2008 18:14:17 +0100, a écrit :
Quoted 3 lines
I am not using ioemu stub-domains yet, would you recommend them in a general
sense or are they more of a special case?I think they're in general recommendable. Both Samuel and Stefano will be able to give more information about it.
That brings more security and performance, so yes it's recommended :)
Samuel
Quoted 4 lines · Gianluca Guida
Quoted 1 line · Todd Deshane
Quoted 1 line
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
Also, are you using ioemu stub-domains? That should also give a performance
gain.Sure, I can give it a try. Should I also apply George's patch right?
Much faster, I'll also test on debian 4.0 for a regression test.
Average Optimal load -j 4 Run (std deviation):
Elapsed Time 519.056 (0.098387)
User Time 426.226 (0.815555)
System Time 92.716 (0.858563)
Percent CPU 99 (0)
Context Switches 99125.2 (71.3667) Sleeps 31199.2 (41.3062)
I also want to look into the ioemu stub domains after reading more about them.
I'll report the debian (older kernel results) when they come out.
Todd
Quoted 1 line
I'll report the debian (older kernel results) when they come out.
Roughly the same as before, so the patches look good.
HVM guest (Debian 4.0 vanilla 2.6.18.6-amd64)
Average Optimal load -j 4 Run (std deviation):
Elapsed Time 492.826 (1.89446)
User Time 390.316 (0.571341)
System Time 101.396 (2.09772)
Percent CPU 99 (0)
Context Switches 29514.4 (53.8266) Sleeps 37991.4 (572.437)
Signature · 3 lines
Todd Deshane http://todddeshane.net check out our book: http://runningxen.com
And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!
If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.
-George
Quoted 37 lines · Todd Deshane
Quoted 4 lines · Gianluca Guida
Quoted 1 line · Todd Deshane
Quoted 1 line
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
Also, are you using ioemu stub-domains? That should also give a performance
gain.Sure, I can give it a try. Should I also apply George's patch right?
Much faster, I'll also test on debian 4.0 for a regression test.
Average Optimal load -j 4 Run (std deviation):
Elapsed Time 519.056 (0.098387)
User Time 426.226 (0.815555)
System Time 92.716 (0.858563)
Percent CPU 99 (0)
Context Switches 99125.2 (71.3667) Sleeps 31199.2 (41.3062)I also want to look into the ioemu stub domains after reading more about them.
I'll report the debian (older kernel results) when they come out.
Todd
_______________________________________________
Xen-devel mailing list [email protected]
http://lists.xensource.com/xen-devel
Quoted 4 lines · George Dunlap
And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!
KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.html
I think the competition is a good thing.
Quoted 4 lines
If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.
Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.
Todd
Signature · 3 lines
Todd Deshane http://todddeshane.net check out our book: http://runningxen.com
Heh... the blatant copying is flattering and annoying at the same time. :-) Ah, the beauty of open-source...
I've got your trace, and I'll take a look at it tomorrow. Thanks!
-George
Quoted 32 lines · Todd Deshane
Quoted 4 lines · George Dunlap
And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!
KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.htmlI think the competition is a good thing.
Quoted 4 lines
If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.
Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.
Todd
--
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Hmm, no really obvious low-hanging fruit. The Xen-HVM was about 9% slower than your reported numbers for Xen-PV, and the trace shows that the guest spent about that much inside the hypervisor. The breakdown:
* 3.6% propagating page faults to guest
* 3.0% pulling through entries from out-of-sync guest pt's to shadow pagetables
* 1.4% Marking pages out of sync (of which 75% was in unsyncs that had to re-sync another page)
* 0.9% cr3 switches
* 0.9% handling I/O
(Rounding may cause the numbers not to add up exactly.)
So one of the biggest things, really, is that Linux seems to insist on mapping pages one-at-a-time as they're demand-faulted, rather than doing a batch of them. Unfortunately, having pages out-of-sync means that we must use the slow propagate path rather than the fast-propagate path, which is at least 25% slower.
The only avenues for optimization I can see are:
* See if there's a way to reduce the number of unsyncs that cause resyncs. Allowing more pages to go out-of-sync *might* do this; or it might just shift the same overhead into cr3 switch.
* Reduce the time of "hot paths" through the hypervisor by profiling, &c.
-George
Quoted 41 lines · George Dunlap
Heh... the blatant copying is flattering and annoying at the same time. :-) Ah, the beauty of open-source...
I've got your trace, and I'll take a look at it tomorrow. Thanks!
-George
Quoted 32 lines · Todd Deshane
Quoted 4 lines · George Dunlap
And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!
KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.htmlI think the competition is a good thing.
Quoted 4 lines
If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.
Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.
Todd
--
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel