Status of SMP? · kernelthread

Gregory Haskins

author

2007-03-13 12:47 UTC #0 lore raw

Hi all,
So what is the status of SMP support and is someone working on this? I have some ideas about how to implement this and can spend some time with it if it would be helpful. I just dont want to duplicate efforts.

-Greg

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Avi Kivity

2007-03-13 13:16 UTC #2 lore raw

Quoted 3 lines · Gregory Haskins

Hi all,
So what is the status of SMP support and is someone working on this? I have some ideas about how to implement this and can spend some time with it if it would be helpful. I just dont want to duplicate efforts.

SMP is planned, though of course help is appreciated. I'd like to start on an NPT capable processor so that there's no need to worry about the mmu and bootstrap/ipi simultaneously.

If you have ideas, please share them!

Signature · 8 lines

error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Gregory Haskins

author

2007-03-13 15:47 UTC #8 lore raw

BTW: Please excuse my stupid confusion of uP (microprocessor) with UP (uniprocessor). Duh!

Quoted 1 line

Quoted 1 line
Quoted 1 line
On Tue, Mar 13, 2007 at 11:43 AM, in message <[email protected]>,

Quoted 45 lines · Gregory Haskins

Quoted 1 line
Quoted 1 line
Quoted 1 line
On Tue, Mar 13, 2007 at 11:09 AM, in message <45F6BEC0.1010906-atKUWr5tajBWk0Htik3J/[email protected]>,
Quoted 6 lines · Avi Kivity
Quoted 3 lines · Gregory Haskins
Does this sound reasonable?
Quoted 7 lines
Much simpler (at least to start with) is to send a signal. The signal will interrupt the guest if it is in guest mode, or if it sleeping, it will wake up the thread. Upon receiving the signal, the thread can inject an interrupt.
Later on, we may wish to optimize this to avoid the signal if we're certain to be in guest mode, and only send it if we're asleep.
I will investigate it from this perspective then...but I currently dont understand how a signal delivery would cause the guest to be interrupted (unless you guys have already put some trickery in the kvm-kmod ;).
I can follow the logic for a uP system (I think, therefore I am.....e.g. you cannot be both host and guest mode at the same time). However, in an SMP system it would be conceivable for one task to be in host context, and another to be in guest context simultaneously. If a guest was already in VM context, wouldnt the signal being delivered simply be posted to the pending-signals for the task and get evaluated at the next VMEXIT? The method I was describing would actually invoke the VMEXIT to begin with. Perhaps the best solution is a combination of both (post a signal and an IPI which evaluates to no-op in host mode). Forgive me if I am ignorant and/or being obtuse. :)
Is there something I am missing? Or is there an issue with SMP with this approach?
Quoted 11 lines
Quoted 3 lines
The thing I cant quite wrap my head around is how this would work in the PV
driver sense. For SMP, its clear...one VCPU thread might be in QEMU context in the LAPIC model and send the FORCE_EXIT ioctl against another VCPU to deliever the IPI. How does this work in a uP setup for PV? Is there some other thread in QEMU that might be waiting for AIO completions (etc) that would then be able to invoke the ioctl? Otherwise, our one and only thread is tied up in VM context so I dont get how it would work.
Quoted 1 line

Quoted 10 lines

Quoted 3 lines
The scenario is a host- side kernel- mode device receiving a network packet and wishing to interrupt a guest. Here, the device receive handler is a separate thread of execution.
I see. That makes sense.
-Greg

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Nakajima, Jun

2007-03-14 21:09 UTC #10 lore raw

Quoted 8 lines · Avi Kivity

Quoted 6 lines · Gregory Haskins
Hi all,
So what is the status of SMP support and is someone working on
this? I have some ideas about how to implement this and can spend some time with it if it would be helpful. I just dont want to duplicate efforts.

Quoted 5 lines

SMP is planned, though of course help is appreciated. I'd like to start on an NPT capable processor so that there's no need to worry about the mmu and bootstrap/ipi simultaneously.
If you have ideas, please share them!

The key is the virtual MMU; getting bootstrap/IPI working is straightforward. Without NPT or EPT, we need to have an SMP-capable shadow pagetable code. I think one of the most efficient ways is to re-use the code from Xen. The current one is the third generation of the shadow page table, and it works well.

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Avi Kivity

2007-03-15 08:43 UTC #12 lore raw

Quoted 2 lines · Jun

The key is the virtual MMU; getting bootstrap/IPI working is straightforward.

I'm glad to hear you say that.

Quoted 5 lines

Without NPT or EPT, we need to have an SMP-capable
shadow pagetable code. I think one of the most efficient ways is to re-use the code from Xen. The current one is the third generation of the shadow page table, and it works well.

I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would be rejected. Linux development requires incremental patches.

I'm not familiar with the current Xen shadow code, but a cursory look gave the following impressions:

- it is single threaded, so there's no reason to expect more scalability out of it
- it is lighter weight, by not maintaining a full reverse mapping. on the other hand, it is less generic (requires special handling for linearized page tables)
- does order 2 allocations (which work in Linux, but get less reliable as uptime increases)

I don't see a compelling reason to change here, but of course I may have missed something and I'm also biased.

Getting the kvm mmu to do smp involves the following, as far as I can tell:

- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have touched the domain, and later, for cpus that have touched the pagetable)

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Nakajima, Jun

2007-03-15 22:08 UTC #14 lore raw

Quoted 7 lines · Avi Kivity

Quoted 2 lines · Jun
The key is the virtual MMU; getting bootstrap/IPI working is straightforward.
I'm glad to hear you say that.

Quoted 8 lines

Quoted 5 lines
Without NPT or EPT, we need to have an SMP-capable
shadow pagetable code. I think one of the most efficient ways is to re-use the code from Xen. The current one is the third generation of the shadow page table, and it works well.
I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would be rejected. Linux development requires incremental patches.

I assumed that the current virutal MMU code was not designed to support SMP (when you said "there's no need to worry about the mmu"). After reviewing the code, I think the current code is good enough, especially in the sense that it emulates writes to guest page tables. Actually the code at least the emulator is from Xen ;-)

This particular code keeps the shadow page table in sync with the guest right away, avoiding major incosistencies in the SMP environment.

		spin_lock(&vcpu->kvm->lock);
		r = kvm_mmu_page_fault(vcpu, cr2, error_code);
		if (r < 0) {
			spin_unlock(&vcpu->kvm->lock);
			return r;
		}
		if (!r) {
			spin_unlock(&vcpu->kvm->lock);
			return 1;
		}

		er = emulate_instruction(vcpu, kvm_run, cr2,
error_code);
		spin_unlock(&vcpu->kvm->lock);

Quoted 11 lines

Getting the kvm mmu to do smp involves the following, as far as I can tell:
- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have touched the domain, and later, for cpus that have touched the pagetable)

The other thing is how do you detect when the guest page tables become just data (this is not an SMP issue, though)? If you don't unshadow such pages, we'll tend to get (frequent) unnecessary #PF.

These are mostly performance optimization issues, but SMP guests will use local and IO APICs (and other misc timers), and we'll see more frequent VM exits in such a guest. We may need to move them to the KVM side if we see performance issues.

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Avi Kivity

2007-03-16 06:46 UTC #16 lore raw

Quoted 2 lines · Jun

I assumed that the current virutal MMU code was not designed to support SMP (when you said "there's no need to worry about the mmu").

What I meant was that there is no need debug both the core smp support and the mmu smp support at the same time. Certainly the mmu was written with smp support in mind.

Quoted 4 lines

The other thing is how do you detect when the guest page tables become just data (this is not an SMP issue, though)? If you don't unshadow such pages, we'll tend to get (frequent) unnecessary #PF.

There are a few heuristics in place for zapping shadow page tables:

- user mode writes (fix_write_pf)
- unaligned writes (kvm_mmu_pre_write)
- lots of writes to the same page (kvm_mmu_pre_write)

We could probably also add a heuristic that checks that the contents look like a pte if PT_PRESENT is set. This is useful mostly on pae where there are many reserved bits.

Quoted 5 lines

These are mostly performance optimization issues, but SMP guests will use local and IO APICs (and other misc timers), and we'll see more frequent VM exits in such a guest. We may need to move them to the KVM side if we see performance issues.

We've actually done this, and found that performance was not affected. The reason is that the kernel->user transition is very fast compared to the guest->kernel transition. It is still worthwhile to do this if we have to use the apic to inject interrupts from in-kernel paravirtualized devices, where it makes little sense to go back to userspace in order to inject the interrupt.

I'd also like to add a read-only memory type so that apic reads can proceed natively rather than be emulated (the TODO has this now).

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Nakajima, Jun

2007-03-16 15:21 UTC #18 lore raw

Quoted 3 lines · Avi Kivity

I'd also like to add a read-only memory type so that apic reads can proceed natively rather than be emulated (the TODO has this now).

I don't understand how this can work. Instead, we should use the hardware-based virtualization for the local APIC.

Jun
---
Intel Open Source Technology Center

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Avi Kivity

2007-03-18 05:48 UTC #20 lore raw

Quoted 7 lines · Jun

Quoted 3 lines · Avi Kivity
I'd also like to add a read-only memory type so that apic reads can proceed natively rather than be emulated (the TODO has this now).
I don't understand how this can work.

You're right - it can't. I thought all read accesses are static and side-effect-free, but reading the timer is not static.

Quoted 3 lines

Instead, we should use the
hardware-based virtualization for the local APIC.

Can you point me at documentation for that? What I have only mentions tpr/cr8 shadow, but not much else.

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Huang2, Wei

2007-03-16 14:35 UTC #21 lore raw

In reply to a message not in the archive

I am currently working on supporting nested paging for KVM. As of now, I am able to boot 32-bit Linux guest (ttylinux) under AMD-V nested paging. My plan is to support a full-fledged of Linux guests and Windows guests in the next few weeks.

The current KVM MMU code was not designed with hardware assisted paging (i.e. nested paging or extended paging) in mind. For instance, some functions in kvm_main.c (i.e., write_cr functions) need to become become function pointers. Guest page table walking also needs some re-work. It will be nice to have both SMP and hardware assisted paging (NPT and EPT) in place for the next phase of KVM MMU code. I will be happy to provide help and work on these these areas. :P

Best,

-Wei

Operating System Research Center
Advanced Micro Devices (AMD)

-----Original Message-----
From: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org [mailto:kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of Avi Kivity
Sent: Thursday, March 15, 2007 3:43 AM
To: Nakajima, Jun
Cc: [email protected]
Subject: Re: [kvm-devel] Status of SMP?

Quoted 2 lines · Jun

The key is the virtual MMU; getting bootstrap/IPI working is straightforward.

I'm glad to hear you say that.

Quoted 5 lines

Without NPT or EPT, we need to have an SMP-capable shadow pagetable code. I think one of the most efficient ways is to re-use the code from Xen. The current one is the third generation of the shadow page table, and it works well.

I'm fairly certain a rip-it-the-old-and-plug-in-the-new approach would be rejected. Linux development requires incremental patches.

I'm not familiar with the current Xen shadow code, but a cursory look gave the following impressions:

- it is single threaded, so there's no reason to expect more scalability out of it
- it is lighter weight, by not maintaining a full reverse mapping. on the other hand, it is less generic (requires special handling for linearized page tables)
- does order 2 allocations (which work in Linux, but get less reliable as uptime increases)

I don't see a compelling reason to change here, but of course I may have missed something and I'm also biased.

Getting the kvm mmu to do smp involves the following, as far as I can tell:

- verifying that all the locking in place is correct
- make sure that spurious faults (due to races) are handled well
- install shadow ptes atomically instead of building them incrementally
- replace local tlb flushes by remote tlb flushes (for cpus that have touched the domain, and later, for cpus that have touched the pagetable)

--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Huang2, Wei

2007-03-16 17:51 UTC #22 lore raw

In reply to a message not in the archive

I am currently working on supporting nested paging for KVM. As of now, I am able to boot 32-bit Linux guest (ttylinux) under AMD-V nested paging. My plan is to support a full-fledged of Linux guests and Windows guests in the next few weeks.

The current KVM MMU code was not designed with hardware assisted paging (i.e. nested paging or extended paging) in mind. For instance, some functions in kvm_main.c (i.e., write_cr functions) need to become become function pointers. Guest page table walking also needs some re-work. It will be nice to have both SMP and hardware assisted paging (NPT and EPT) in place for the next phase of KVM MMU code. I will be happy to provide help and work on these these areas. :P

Best,

-Wei
-----Original Message-----
From: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org [mailto:kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org] On Behalf Of Avi Kivity
Sent: Thursday, March 15, 2007 3:43 AM
To: Nakajima, Jun
Cc: [email protected]
Subject: Re: [kvm-devel] Status of SMP?