Improve PIO latency

RFC PATCH In review Anthony Liguori 3 participants 5 messages Feb 4, 2007 – Feb 6, 2007 3 messages not in the archive KVM

lore

Share & tools

lore

Anthony Liguori

author

2007-02-04 04:31 UTC #0 lore raw

RFC PATCH 1/1 Improve PIO latency

The attached patch modifies libkvmctl to only make SET_REG/GET_REG ioctls when needed for PIO instructions. I was only able to do this for out instructions because I didn't want to break the kernel ABI.

I think we should change the API though so we can do this with other types of IO instructions. Before this patch, the time line for a PIO instruction looks something like this:

All times in nanoseconds and are round trips from the guests perspective for an out instruction on an AMD X2 4200.

1015 - immediately after restoring saving guest registers 1991 - handled within the kernel in io_interception 2294 - libkvmctl returns immediately
2437 - w/ patch
3311 - w/o patch

The first data point is the best we could possible do. The only work being done after the VMRUN is a VMSAVE/VMLOAD, saving the guest registers, and restoring the host registers. The VMSAVE/VMLOAD is needed so that vmcb->save.eip can be updated.[1] I played around reducing the register savings but the differences weren't noticable.

I suspect that more intelligent handling of things like FPU save/restore should be able to reduce the second data point. This will also improve some other exit paths (like shadow paging). We save/restore an awful lot of state considering that we probably return back to the guest for the vast majority of exits.

On this system, a sysenter based syscall is roughly 100 nsec so I'm pretty happy with the third data point. This is just what one would expect.

With the attached patch, we reduce the time we spend in QEMU by eliminating unnecessary ioctl()s. This cuts down the total trip time by about 1/3. We should be able to do this for in{b,w,l}s too.

With these patches, we get an improvement in disk performance.

virtbench can measure disk latency and small disk read speeds (16kb). According to virt bench:

w/o patch
80% of native - latency
61% of native - bandwidth

w/ patch
96% of native - latency
99% of native - bandwidth

Before getting too excited, we're still only 25% of native with dbench.
We see a small improvement with the patch (around 10%) but there's
an awful lot of variability.

There are quite a few things that should improve disk performance in QEMU. Moving to an asynchronous IO model (QEMU CVS), and utilizing linux-aio should make a pretty significant difference.

The last interesting bit is that the native latency for an IDE PIO operation is around 750 nsec on this system. Theoretically, we should be able to get pretty close to native IDE performance with emulation. At least, that's the theory :-)

Regards,

Anthony Liguori

Avoid making system calls for out{b,w,l} instructions since it is not necessary to sync GP registers.

Signed-off-by Anthony Liguori <[email protected]>

user/kvmctl.c +13 -6

diff -r 29119439ef33 user/kvmctl.c
@@ -234,11 +234,14 @@ static int handle_io(kvm_context_t kvm, 
 	int first_time = 1;
 	int delta;
 	struct translation_cache tr;
+	int _in = (run->io.direction == KVM_EXIT_IO_IN);
 
 	translation_cache_init(&tr);
 
-	regs.vcpu = run->vcpu;
-	ioctl(kvm->fd, KVM_GET_REGS, &regs);
+	if (run->io.string || _in) {
+		regs.vcpu = run->vcpu;
+		ioctl(kvm->fd, KVM_GET_REGS, &regs);
+	}
 
 	delta = run->io.string_down ? -run->io.size : run->io.size;

@@ -246,9 +249,12 @@ static int handle_io(kvm_context_t kvm, 
 		void *value_addr;
 		int r;
 
-		if (!run->io.string)
-			value_addr = &regs.rax;
-		else {
+		if (!run->io.string) {
+			if (_in)
+				value_addr = &regs.rax;
+			else
+				value_addr = &run->io.value;
+		} else {
 			r = translate(kvm, run->vcpu, &tr, run->io.address, 
 				      &value_addr);
 			if (r) {

@@ -326,7 +332,8 @@ static int handle_io(kvm_context_t kvm, 
 		}
 	}
 
-	ioctl(kvm->fd, KVM_SET_REGS, &regs);
+	if (run->io.string || _in)
+		ioctl(kvm->fd, KVM_SET_REGS, &regs);
 	run->emulated = 1;
 	return 0;
 }

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

A message that is not in the archive (sent off-list or not yet received): [email protected]

Avi Kivity

2007-02-06 10:46 UTC #2 lore raw

Quoted 23 lines · Anthony Liguori

The attached patch modifies libkvmctl to only make SET_REG/GET_REG ioctls when needed for PIO instructions. I was only able to do this for out instructions because I didn't want to break the kernel ABI.
I think we should change the API though so we can do this with other types of IO instructions. Before this patch, the time line for a PIO instruction looks something like this:
All times in nanoseconds and are round trips from the guests perspective for an out instruction on an AMD X2 4200.
1015 - immediately after restoring saving guest registers 1991 - handled within the kernel in io_interception 2294 - libkvmctl returns immediately
2437 - w/ patch
3311 - w/o patch
The first data point is the best we could possible do. The only work being done after the VMRUN is a VMSAVE/VMLOAD, saving the guest registers, and restoring the host registers. The VMSAVE/VMLOAD is needed so that vmcb->save.eip can be updated.[1] I played around reducing the register savings but the differences weren't noticable.

Quoted 6 lines

I suspect that more intelligent handling of things like FPU save/restore should be able to reduce the second data point. This will also improve some other exit paths (like shadow paging). We save/restore an awful lot of state considering that we probably return back to the guest for the vast majority of exits.

These are very encouraging numbers. I'd expected the vmexit to be more expensive, and fpu save/restore to be less expensive. Since, as you say, we can eliminate the fpu save/restore in many cases, we have a net win :)

The planned userspace api changes will eliminate registers and virtual addresses for pio. This will both improve performance and make the api more architecture agnostic.

I'm applying the patch now, even though it will be obsoleted soon, as it's always nice to have a performance improvement.

ps. that [1] is a dangling reference?

Signature · 8 lines

Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

A message that is not in the archive (sent off-list or not yet received): 45C85C6D.5030501-atKUWr5tajBWk0Htik3J/[email protected]

Dor Laor

2007-02-06 11:56 UTC #4 lore raw

Quoted 24 lines

Quoted 23 lines · Anthony Liguori
The attached patch modifies libkvmctl to only make SET_REG/GET_REG ioctls when needed for PIO instructions. I was only able to do this for out instructions because I didn't want to break the kernel ABI.
I think we should change the API though so we can do this with other types of IO instructions. Before this patch, the time line for a PIO instruction looks something like this:
All times in nanoseconds and are round trips from the guests perspective for an out instruction on an AMD X2 4200.
1015 - immediately after restoring saving guest registers 1991 - handled within the kernel in io_interception 2294 - libkvmctl returns immediately
2437 - w/ patch
3311 - w/o patch
The first data point is the best we could possible do. The only work being done after the VMRUN is a VMSAVE/VMLOAD, saving the guest registers, and restoring the host registers. The VMSAVE/VMLOAD is needed so that vmcb->save.eip can be updated.[1] I played around reducing the register savings but the differences weren't noticable.

Quoted 4 lines

Quoted 4 lines
I suspect that more intelligent handling of things like FPU save/restore should be able to reduce the second data point. This will also improve some other exit paths (like shadow paging). We save/restore an awful lot of state considering that we probably

return

Quoted 5 lines

Quoted 2 lines
back to the guest for the vast majority of exits.
These are very encouraging numbers. I'd expected the vmexit to be more expensive, and fpu save/restore to be less expensive. Since, as you

I tried to see the performance gain using dd if=/file iflag=direct ... And didn't get any visible gain. So maybe all the vmexit/disk latency are shadowing the performance gain?
Anthony can you please send the virt bench patch?

Quoted 7 lines

say, we can eliminate the fpu save/restore in many cases, we have a net win :)
The planned userspace api changes will eliminate registers and virtual addresses for pio. This will both improve performance and make the api more architecture agnostic.

Quoted 8 lines

I'm applying the patch now, even though it will be obsoleted soon, as it's always nice to have a performance improvement.
ps. that [1] is a dangling reference?
--
Do not meddle in the internals of kernels, for they are subtle and

quick to

Quoted 4 lines

panic.
-----------------------------------------------------------------------

Quoted 1 line

Using Tomcat but need to do more? Need to support web services,

security?

Quoted 3 lines

Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache

Geronimo

Quoted 1 line

http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=12164

Quoted 4 lines

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

A message that is not in the archive (sent off-list or not yet received): 64F9B87B6B770947A9F8391472E032160A4D2550-yEcIvxbTEBqsx+V+t5oei8rau4O3wl8o3fe8/T/[email protected]

Anthony Liguori

author

2007-02-06 14:38 UTC #6 lore raw

Quoted 35 lines · Dor Laor

Quoted 25 lines
Quoted 23 lines · Anthony Liguori
The attached patch modifies libkvmctl to only make SET_REG/GET_REG ioctls when needed for PIO instructions. I was only able to do this for out instructions because I didn't want to break the kernel ABI.
I think we should change the API though so we can do this with other types of IO instructions. Before this patch, the time line for a PIO instruction looks something like this:
All times in nanoseconds and are round trips from the guests perspective for an out instruction on an AMD X2 4200.
1015 - immediately after restoring saving guest registers 1991 - handled within the kernel in io_interception 2294 - libkvmctl returns immediately
2437 - w/ patch
3311 - w/o patch
The first data point is the best we could possible do. The only work being done after the VMRUN is a VMSAVE/VMLOAD, saving the guest registers, and restoring the host registers. The VMSAVE/VMLOAD is needed so that vmcb->save.eip can be updated.[1] I played around reducing the register savings but the differences weren't noticable.
Quoted 5 lines
Quoted 5 lines
I suspect that more intelligent handling of things like FPU save/restore should be able to reduce the second data point. This will also improve some other exit paths (like shadow paging). We save/restore an awful lot of state considering that we probably
return
Quoted 3 lines
Quoted 3 lines
back to the guest for the vast majority of exits.

Quoted 7 lines

Quoted 3 lines
These are very encouraging numbers. I'd expected the vmexit to be more expensive, and fpu save/restore to be less expensive. Since, as you
I tried to see the performance gain using dd if=/file iflag=direct ...

Have you compared against the host? I used dbench.

Quoted 4 lines

And didn't get any visible gain. So maybe all the vmexit/disk latency are shadowing the performance gain?
Anthony can you please send the virt bench patch?

You don't need a patched version of virtbench. Just try use:

virtbench local pio

http://ozlabs.org/~rusty/virtbench

Regards,

Anthony Liguori

Quoted 33 lines

Quoted 7 lines
say, we can eliminate the fpu save/restore in many cases, we have a net win :)
The planned userspace api changes will eliminate registers and virtual addresses for pio. This will both improve performance and make the api more architecture agnostic.
Quoted 9 lines
I'm applying the patch now, even though it will be obsoleted soon, as it's always nice to have a performance improvement.
ps. that [1] is a dangling reference?
--
Do not meddle in the internals of kernels, for they are subtle and
quick to
Quoted 5 lines
panic.
-----------------------------------------------------------------------
--
Quoted 2 lines
Using Tomcat but need to do more? Need to support web services,
security?
Quoted 4 lines
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache

Quoted 11 lines

Geronimo
Quoted 2 lines
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=12164
2
Quoted 5 lines
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Anthony Liguori

author

2007-02-06 14:35 UTC #7 lore raw

In reply to a message not in the archive

Quoted 31 lines · Avi Kivity

Quoted 23 lines · Anthony Liguori
The attached patch modifies libkvmctl to only make SET_REG/GET_REG ioctls when needed for PIO instructions. I was only able to do this for out instructions because I didn't want to break the kernel ABI.
I think we should change the API though so we can do this with other types of IO instructions. Before this patch, the time line for a PIO instruction looks something like this:
All times in nanoseconds and are round trips from the guests perspective for an out instruction on an AMD X2 4200.
1015 - immediately after restoring saving guest registers 1991 - handled within the kernel in io_interception 2294 - libkvmctl returns immediately
2437 - w/ patch
3311 - w/o patch
The first data point is the best we could possible do. The only work being done after the VMRUN is a VMSAVE/VMLOAD, saving the guest registers, and restoring the host registers. The VMSAVE/VMLOAD is needed so that vmcb->save.eip can be updated.[1] I played around reducing the register savings but the differences weren't noticable.
Quoted 6 lines
I suspect that more intelligent handling of things like FPU save/restore should be able to reduce the second data point. This will also improve some other exit paths (like shadow paging). We save/restore an awful lot of state considering that we probably return back to the guest for the vast majority of exits.

Quoted 8 lines

These are very encouraging numbers. I'd expected the vmexit to be more expensive, and fpu save/restore to be less expensive. Since, as you say, we can eliminate the fpu save/restore in many cases, we have a net win :)
The planned userspace api changes will eliminate registers and virtual addresses for pio. This will both improve performance and make the api more architecture agnostic.

Cool!

Quoted 5 lines

I'm applying the patch now, even though it will be obsoleted soon, as it's always nice to have a performance improvement.
ps. that [1] is a dangling reference?

Sorry, it was supposed to be something like:

[1] I have no idea why there is are VM{SAVE,LOAD} instructions in the first place. AFAIK, there is nothing useful you can do after a VMRUN without doing a VMSAVE since you cannot re-enter the VM without updating the VMCB.

Regards,

Anthony Liguori