Q: What is the struct kvm srcu protecting?

DISCUSSION David Daney 3 participants 5 messages May 2, 2013 – May 5, 2013 KVM

Share & tools

David Daney

author

2013-05-02 18:22 UTC #0 lore raw

Hi,

I am working on the MIPS KVM port, and am trying to figure out under which circumstances do I need to srcu_read_lock()/srcu_read_unlock() the kvm->srcu.

I am looking at implementing something similar to arch/x86/kvm/x86.c at __msr_io(), where we see:

.
.
.

	idx = srcu_read_lock(&vcpu->kvm->srcu);
	for (i = 0; i < msrs->nmsrs; ++i)
		if (do_msr(vcpu, entries[i].index, &entries[i].data))
			break;
	srcu_read_unlock(&vcpu->kvm->srcu, idx);

.
.
.

Why is the srcu_read_lock() taken here? I see no srcu_dereference() in the code path that would indicate the need for obtaining the lock.

I have a feeling that I am missing some essential concept about the design of this code, but I don't know what it is.

Can someone explain what is happening here?

Thanks in advance,
David Daney

Marcelo Tosatti

2013-05-03 01:49 UTC #1 lore raw

Quoted 5 lines · David Daney

Hi,
I am working on the MIPS KVM port, and am trying to figure out under which circumstances do I need to srcu_read_lock()/srcu_read_unlock() the kvm->srcu.

For x86: kvm->srcu protects memory slot information (kvm->memslots) and in-kernel MMIO/PIO address->device structure mapping (kvm->buses). Search for synchronize_srcu_expedited() in virt/kvm/ to locate the updaters.

Quoted 11 lines

I am looking at implementing something similar to arch/x86/kvm/x86.c at __msr_io(), where we see:
.
.
.
	idx = srcu_read_lock(&vcpu->kvm->srcu);
	for (i = 0; i < msrs->nmsrs; ++i)
		if (do_msr(vcpu, entries[i].index, &entries[i].data))
			break;
	srcu_read_unlock(&vcpu->kvm->srcu, idx);

Quoted 7 lines

.
.
.
Why is the srcu_read_lock() taken here? I see no srcu_dereference() in the code path that would indicate the need for obtaining the lock.

        case KVM_GET_MSRS:
                r = msr_io(vcpu, argp, kvm_get_msr, 1);
                break;
        case KVM_SET_MSRS:
                r = msr_io(vcpu, argp, do_set_msr, 0);
                break;

int kvm_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata)

{
        return kvm_x86_ops->get_msr(vcpu, msr_index, pdata);
}

Somewhere down that path memslot information must be accessed.

Quoted 4 lines

I have a feeling that I am missing some essential concept about the design of this code, but I don't know what it is.
Can someone explain what is happening here?

For x86 the usage is:

VCPU CODE PATH
--------------

IOCTL(KVM_FD, KVM_VCPU_RUN)
ENTER-KERNEL
SRCU_READ_LOCK()
... large parte of vcpu context code performed with srcu lock held, so that memory slot information can be used to access guest memory (gfn_to_memslot for example).
SRCU_READ_UNLOCK()
VMENTER
while in guest mode srcu is not held so that updaters can make progress
VMEXIT
SRCU_READ_LOCK()
back to vcpu context code

SRCU_READ_UNLOCK before return to userspace.

Also, when emulating HALT (kvm_vcpu_block) srcu is dropped.

UPDATERS
--------
See synchronize_srcu_expedited in virt/kvm/

Gleb Natapov

2013-05-03 10:51 UTC #2 lore raw

Hi David,

Quoted 7 lines · Marcelo Tosatti

Quoted 5 lines · David Daney
Hi,
I am working on the MIPS KVM port, and am trying to figure out under which circumstances do I need to srcu_read_lock()/srcu_read_unlock() the kvm->srcu.

Is your work somehow related to the work of Sanjay Lal that can be found
here: https://git.linux-mips.org/?p=ralf/upstream-sfr.git;a=summary?

Some clarification/addition to what Marcelo said below.

Quoted 4 lines

For x86: kvm->srcu protects memory slot information (kvm->memslots) and in-kernel MMIO/PIO address->device structure mapping (kvm->buses). Search for synchronize_srcu_expedited() in virt/kvm/ to locate the updaters.

This is not only for x86. Any arch code that access memslots have to be srcu read protected.

Quoted 27 lines

Quoted 12 lines
I am looking at implementing something similar to arch/x86/kvm/x86.c at __msr_io(), where we see:
.
.
.
	idx = srcu_read_lock(&vcpu->kvm->srcu);
	for (i = 0; i < msrs->nmsrs; ++i)
		if (do_msr(vcpu, entries[i].index, &entries[i].data))
			break;
	srcu_read_unlock(&vcpu->kvm->srcu, idx);
Quoted 7 lines
.
.
.
Why is the srcu_read_lock() taken here? I see no srcu_dereference() in the code path that would indicate the need for obtaining the lock.
        case KVM_GET_MSRS:
                r = msr_io(vcpu, argp, kvm_get_msr, 1);
                break;
        case KVM_SET_MSRS:
                r = msr_io(vcpu, argp, do_set_msr, 0);
                break;

Quoted 9 lines

to
int kvm_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata)
{
        return kvm_x86_ops->get_msr(vcpu, msr_index, pdata);
}
Somewhere down that path memslot information must be accessed.

As Marcelo explained bellow on X86 srcu lock is held while vcpu loop is running, so all the code that is usually called from vcpu does not do its own locking. In rare cases that this code is called not from vcpu main loop the locking is done explicitly on the high level.

This design is architecture choice. Other arches access memslot much less and do locking as close to memslot access as possible.

Quoted 28 lines

Quoted 4 lines
I have a feeling that I am missing some essential concept about the design of this code, but I don't know what it is.
Can someone explain what is happening here?
For x86 the usage is:
VCPU CODE PATH
--------------
IOCTL(KVM_FD, KVM_VCPU_RUN)
ENTER-KERNEL
SRCU_READ_LOCK()
... large parte of vcpu context code performed with srcu lock held, so that memory slot information can be used to access guest memory (gfn_to_memslot for example).
SRCU_READ_UNLOCK()
VMENTER
while in guest mode srcu is not held so that updaters can make progress
VMEXIT
SRCU_READ_LOCK()
back to vcpu context code
SRCU_READ_UNLOCK before return to userspace.
Also, when emulating HALT (kvm_vcpu_block) srcu is dropped.

Quoted 8 lines

UPDATERS
--------
See synchronize_srcu_expedited in virt/kvm/
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Gleb.

David Daney

author

2013-05-03 17:21 UTC #3 lore raw

Quoted 4 lines · Gleb Natapov

Hi David,
Quoted 1 line · Marcelo Tosatti
On Thu, May 02, 2013 at 11:22:52AM -0700, David Daney wrote:

Quoted 9 lines

Quoted 6 lines
Quoted 5 lines
Hi,
I am working on the MIPS KVM port, and am trying to figure out under which circumstances do I need to srcu_read_lock()/srcu_read_unlock() the kvm->srcu.
Is your work somehow related to the work of Sanjay Lal that can be found
here: https://git.linux-mips.org/?p=ralf/upstream-sfr.git;a=summary?

It is related in that a single asm/uaip/kvm.h must be shared between the implementations. It differs in that it is based on the MIPS-VZ hardware virtualization feature, where as Sanjay's code is a pure software solution.

If possible some code might be shared between the two, but it may end up looking somewhat like the x86 implementation where there are separate VMX and SVM implementations.

Quoted 1 line

Some clarification/addition to what Marcelo said below.

.
.
.

Thanks to all for setting me straight here.

David Daney

Gleb Natapov

2013-05-05 07:38 UTC #4 lore raw

Quoted 15 lines · David Daney

Quoted 4 lines · Gleb Natapov
Hi David,
Quoted 1 line · Marcelo Tosatti
On Thu, May 02, 2013 at 11:22:52AM -0700, David Daney wrote:
Quoted 9 lines
Quoted 6 lines
Quoted 5 lines
Hi,
I am working on the MIPS KVM port, and am trying to figure out under which circumstances do I need to srcu_read_lock()/srcu_read_unlock() the kvm->srcu.
Is your work somehow related to the work of Sanjay Lal that can be found
here: https://git.linux-mips.org/?p=ralf/upstream-sfr.git;a=summary?

Quoted 9 lines

It is related in that a single asm/uaip/kvm.h must be shared between the implementations. It differs in that it is based on the MIPS-VZ hardware virtualization feature, where as Sanjay's code is a pure software solution.
If possible some code might be shared between the two, but it may end up looking somewhat like the x86 implementation where there are separate VMX and SVM implementations.

Sanjay code already has such design and the reason he gave for it was upcoming HW virtualization support.

--
Gleb.