Skip to contentSkip to conversation

Allow removing writable mappings from splintered page tables.

PATCH In review Gianluca Guida 4 participants 10 messages Sep 12, 2008 – Sep 16, 2008

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Allow removing writable mappings from splintered page tables.

Hello,

The moving of the pagetable mapping in the linux kernel exposed the fact that under the linux kernel sh_rm_write_access_from_sl1p was always failing.

Linux seems to use big pages to access page tables, so we should instruct the shadow code to be able to remove writable mappings from splintered pagetables as well, avoiding using OS heuristic (which were failing in 2.6.27 before George patch, leading to brute-force search at each resync).

Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.

Thanks,
Gianluca

xen/arch/x86/mm/shadow/common.c +7 -4
diff -r dbac9ee4d761 xen/arch/x86/mm/shadow/common.c
@@ -2462,22 +2462,25 @@ int sh_remove_write_access_from_sl1p(str
     ASSERT(mfn_valid(smfn));
     ASSERT(mfn_valid(gmfn));
     
-    if ( sp->type == SH_type_l1_32_shadow )
+    if ( sp->type == SH_type_l1_32_shadow
+         || sp->type == SH_type_fl1_32_shadow )
     {
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,2)
             (v, gmfn, smfn, off);
     }
 #if CONFIG_PAGING_LEVELS >= 3
-    else if ( sp->type == SH_type_l1_pae_shadow )
+    else if ( sp->type == SH_type_l1_pae_shadow
+              || sp->type == SH_type_fl1_pae_shadow )
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,3)
             (v, gmfn, smfn, off);
 #if CONFIG_PAGING_LEVELS >= 4
-    else if ( sp->type == SH_type_l1_64_shadow )
+    else if ( sp->type == SH_type_l1_64_shadow
+              || sp->type == SH_type_fl1_64_shadow )
         return SHADOW_INTERNAL_NAME(sh_rm_write_access_from_sl1p,4)
             (v, gmfn, smfn, off);
 #endif
 #endif
-    
+
     return 0;
 }
 #endif 
xen/arch/x86/mm/shadow/multi.c +2 -1
diff -r dbac9ee4d761 xen/arch/x86/mm/shadow/multi.c
@@ -4539,7 +4539,8 @@ int sh_rm_write_access_from_sl1p(struct 
     sp = mfn_to_shadow_page(smfn);
 
     if ( sp->mbz != 0
-         || (sp->type != SH_type_l1_shadow) )
+         || (sp->type != SH_type_l1_shadow
+             && sp->type != SH_type_fl1_shadow) )
         goto fail;
 
     sl1p = sh_map_domain_page(smfn);

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw
Quoted 5 lines · Gianluca Guida

Hello,

The moving of the pagetable mapping in the linux kernel exposed the fact that under the linux kernel sh_rm_write_access_from_sl1p was always failing.

Quoted 9 lines

Linux seems to use big pages to access page tables, so we should instruct the shadow code to be able to remove writable mappings from splintered pagetables as well, avoiding using OS heuristic (which were failing in 2.6.27 before George patch, leading to brute-force search at each resync).

Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.

Sure, I can give it a try. Should I also apply George's patch right?

I am not using ioemu stub-domains yet, would you recommend them in a general sense or are they more of a special case?

Cheers,
Todd

#2 lore raw
Quoted 6 lines · Todd Deshane
Quoted 4 lines

Todd: Can you try try the linux-2.6.27 kernbench with the following patch? Also, are you using ioemu stub-domains? That should also give a performance gain.

Sure, I can give it a try. Should I also apply George's patch right?

Yes.

Quoted 2 lines

I am not using ioemu stub-domains yet, would you recommend them in a general sense or are they more of a special case?

I think they're in general recommendable. Both Samuel and Stefano will be able to give more information about it.

Thanks,
Gianluca

#3 lore raw
Quoted 6 lines · Gianluca Guida, le Fri 12 Sep 2008 18:14:17 +0100, a écrit :
Quoted 3 lines

I am not using ioemu stub-domains yet, would you recommend them in a general
sense or are they more of a special case?

I think they're in general recommendable. Both Samuel and Stefano will be able to give more information about it.

That brings more security and performance, so yes it's recommended :)

Samuel

#4 lore raw
In reply to Gianluca Guida
Quoted 4 lines · Gianluca Guida
Quoted 1 line · Todd Deshane
Quoted 1 line
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
Also, are you using ioemu stub-domains? That should also give a performance
gain.

Sure, I can give it a try. Should I also apply George's patch right?

Much faster, I'll also test on debian 4.0 for a regression test.

Average Optimal load -j 4 Run (std deviation):
Elapsed Time 519.056 (0.098387)
User Time 426.226 (0.815555)
System Time 92.716 (0.858563)
Percent CPU 99 (0)
Context Switches 99125.2 (71.3667) Sleeps 31199.2 (41.3062)

I also want to look into the ioemu stub domains after reading more about them.

I'll report the debian (older kernel results) when they come out.

Todd

#5 lore raw
Quoted 1 line

I'll report the debian (older kernel results) when they come out.

Roughly the same as before, so the patches look good.

HVM guest (Debian 4.0 vanilla 2.6.18.6-amd64)

Average Optimal load -j 4 Run (std deviation):
Elapsed Time 492.826 (1.89446)
User Time 390.316 (0.571341)
System Time 101.396 (2.09772)
Percent CPU 99 (0)
Context Switches 29514.4 (53.8266) Sleeps 37991.4 (572.437)

Signature · 3 lines
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com
#6 lore raw
In reply to Todd Deshane

And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!

If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.

-George

Quoted 37 lines · Todd Deshane
Quoted 4 lines · Gianluca Guida
Quoted 1 line · Todd Deshane
Quoted 1 line
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

Todd: Can you try try the linux-2.6.27 kernbench with the following patch?
Also, are you using ioemu stub-domains? That should also give a performance
gain.

Sure, I can give it a try. Should I also apply George's patch right?

Much faster, I'll also test on debian 4.0 for a regression test.

Average Optimal load -j 4 Run (std deviation):
Elapsed Time 519.056 (0.098387)
User Time 426.226 (0.815555)
System Time 92.716 (0.858563)
Percent CPU 99 (0)
Context Switches 99125.2 (71.3667) Sleeps 31199.2 (41.3062)

I also want to look into the ioemu stub domains after reading more about them.

I'll report the debian (older kernel results) when they come out.

Todd

_______________________________________________
Xen-devel mailing list [email protected]
http://lists.xensource.com/xen-devel

#7 lore raw
Quoted 4 lines · George Dunlap

And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!

KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.html

I think the competition is a good thing.

Quoted 4 lines

If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.

Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.

Todd

Signature · 3 lines
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com
#8 lore raw

Heh... the blatant copying is flattering and annoying at the same time. :-) Ah, the beauty of open-source...

I've got your trace, and I'll take a look at it tomorrow. Thanks!

-George

Quoted 32 lines · Todd Deshane
Quoted 4 lines · George Dunlap

And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!

KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.html

I think the competition is a good thing.

Quoted 4 lines

If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.

Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.

Todd

--
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#9 lore raw

Hmm, no really obvious low-hanging fruit. The Xen-HVM was about 9% slower than your reported numbers for Xen-PV, and the trace shows that the guest spent about that much inside the hypervisor. The breakdown:
* 3.6% propagating page faults to guest
* 3.0% pulling through entries from out-of-sync guest pt's to shadow pagetables
* 1.4% Marking pages out of sync (of which 75% was in unsyncs that had to re-sync another page)
* 0.9% cr3 switches
* 0.9% handling I/O

(Rounding may cause the numbers not to add up exactly.)

So one of the biggest things, really, is that Linux seems to insist on mapping pages one-at-a-time as they're demand-faulted, rather than doing a batch of them. Unfortunately, having pages out-of-sync means that we must use the slow propagate path rather than the fast-propagate path, which is at least 25% slower.

The only avenues for optimization I can see are:
* See if there's a way to reduce the number of unsyncs that cause resyncs. Allowing more pages to go out-of-sync *might* do this; or it might just shift the same overhead into cr3 switch.
* Reduce the time of "hot paths" through the hypervisor by profiling, &c.

-George

Quoted 41 lines · George Dunlap

Heh... the blatant copying is flattering and annoying at the same time. :-) Ah, the beauty of open-source...

I've got your trace, and I'll take a look at it tomorrow. Thanks!

-George

Quoted 32 lines · Todd Deshane
Quoted 4 lines · George Dunlap

And your original numbers showed elapsed time to be 527s for KVM, so now Xen is 8 seconds in the lead for HVM Linux. :-) Thanks for the help tracking this down!

KVM is also working on improved page table algorithms
http://www.mail-archive.com/[email protected]/msg03562.html

I think the competition is a good thing.

Quoted 4 lines

If you have time, could you take another 30-second trace with the new changes in, just for fun? I'll take a quick look and see if there's any other low-hanging fruit to grab.

Sent the trace to you with another service called sendspace, since, for some reason, the trace file was much bigger.

Todd

--
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel