SVM: vmload/vmsave-free VM exits?
Hi,
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsave
B) wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Hi Jan,
Quoted 18 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)
Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 31 lines · Valentine Sinitsyn
Hi Jan,
Quoted 18 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Quoted 6 lines
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
/*
* VM exit benchmark using a hypercall
*
* Copyright (c) Siemens AG, 2015
*
* Authors:
* Jan Kiszka <[email protected]>
*
* This work is licensed under the terms of the GNU GPL, version 2. See
* the COPYING file in the top-level directory.
*/
#ifndef __x86_64__
#error only x86-64 supported
#endif
#include <stdbool.h>
#include <stdio.h>
#define LOOPS 1000000
#define X86_FEATURE_VMX (1UL << 5)
static inline unsigned long cpuid_ecx(void)
{
unsigned long val;
asm volatile("cpuid" : "=c" (val) : "a" (1) : "ebx", "edx");
return val;
}static inline __attribute__((always_inline)) unsigned long long read_tsc(void)
{
unsigned long long hi, lo;
asm volatile("rdtsc" : "=a" (lo), "=d" (hi));
return (hi << 32) | lo;
}int main(int argc, char *argv[])
{
bool use_vmcall = !!(cpuid_ecx() & X86_FEATURE_VMX);
unsigned long long start, sum = 0;
unsigned int n;
for (n = 0; n < LOOPS; n++) {
if (use_vmcall) {
start = read_tsc();
asm volatile("vmcall" : : "a" (-1));
sum += read_tsc() - start;
} else {
start = read_tsc();
asm volatile("vmmcall" : : "a" (-1));
sum += read_tsc() - start;
}
}
printf("Null hypercall: %llu cycles\n", sum / LOOPS);
return 0;
}Hi Jan,
Quoted 36 lines · Jan Kiszka
Quoted 35 lines · Valentine Sinitsyn
Hi Jan,
Quoted 18 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Quoted 5 lines
Quoted 2 lines
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Great, thanks. Could you post absolute numbers, i.e how long do A and B take on your CPU?
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 45 lines · Valentine Sinitsyn
Hi Jan,
Quoted 42 lines · Jan Kiszka
Quoted 40 lines · Valentine Sinitsyn
Hi Jan,
Quoted 19 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
Quoted 5 lines
Quoted 3 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Great, thanks. Could you post absolute numbers, i.e how long do A and B take on your CPU?
A is around 1910 cycles, B about 1750.
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 7 lines · Jan Kiszka
Quoted 5 lines
Quoted 3 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Great, thanks. Could you post absolute numbers, i.e how long do A and B take on your CPU?
A is around 1910 cycles, B about 1750.
It's with hot caches I guess? Not bad anyways, it's a pity I didn't observe this and didn't include this optimization from the day one.
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 10 lines · Valentine Sinitsyn
Quoted 7 lines · Jan Kiszka
Quoted 5 lines
Quoted 3 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Great, thanks. Could you post absolute numbers, i.e how long do A and B take on your CPU?
A is around 1910 cycles, B about 1750.
It's with hot caches I guess? Not bad anyways, it's a pity I didn't observe this and didn't include this optimization from the day one.
Yes, that is with the unmodified benchmark I sent. When I add, say usleep(1000) to that loop body, the cycles jumped to 4k (IIRC).
BTW, this is the Jailhouse patch:
https://github.com/siemens/jailhouse/commit/dbf2fe479ac07a677462dfa87e008e37a4e72858
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 9 lines · Jan Kiszka
Quoted 8 lines · Valentine Sinitsyn
Quoted 7 lines · Jan Kiszka
Quoted 5 lines
Quoted 3 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Great, thanks. Could you post absolute numbers, i.e how long do A and B take on your CPU?
A is around 1910 cycles, B about 1750.
Quoted 8 lines
Quoted 2 lines
It's with hot caches I guess? Not bad anyways, it's a pity I didn't observe this and didn't include this optimization from the day one.
Yes, that is with the unmodified benchmark I sent. When I add, say usleep(1000) to that loop body, the cycles jumped to 4k (IIRC).
BTW, this is the Jailhouse patch:
https://github.com/siemens/jailhouse/commit/dbf2fe479ac07a677462dfa87e008e37a4e72858
I guess, it's getting off-topic here, but wouldn't it be cleaner to simply use wrmsr and rdmsr instead of vmload and vmsave in svm-vmexit.S? This would require less changes and will keep all entry/exit setup code in one place.
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 18 lines · Valentine Sinitsyn
Quoted 17 lines · Jan Kiszka
Quoted 5 lines · Valentine Sinitsyn
Quoted 4 lines · Jan Kiszka
Quoted 4 lines
Quoted 4 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more
towards 600 if they are colder (added some usleep to each loop in the test).Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Great, thanks. Could you post absolute numbers, i.e how long do A and B
take on your CPU?A is around 1910 cycles, B about 1750.
It's with hot caches I guess? Not bad anyways, it's a pity I didn't observe this and didn't include this optimization from the day one.
Yes, that is with the unmodified benchmark I sent. When I add, say usleep(1000) to that loop body, the cycles jumped to 4k (IIRC).
Quoted 7 lines
Quoted 3 lines
BTW, this is the Jailhouse patch:
https://github.com/siemens/jailhouse/commit/dbf2fe479ac07a677462dfa87e008e37a4e72858I guess, it's getting off-topic here, but wouldn't it be cleaner to simply use wrmsr and rdmsr instead of vmload and vmsave in
svm-vmexit.S? This would require less changes and will keep all entry/exit setup code in one place.
It's a tradeoff between assembly lines and C statements. My feeling is that it's easier done in C, but you can prove me wrong.
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 39 lines · Jan Kiszka
Quoted 31 lines · Valentine Sinitsyn
Hi Jan,
Quoted 18 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Quoted 6 lines
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
Quoted 6 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
Joel, can you comment on the benefit of variant B) for the various AMD CPUs? Is it always positive?
Thanks,
Jan
[1] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/61455
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux
Quoted 39 lines · Jan Kiszka
Quoted 38 lines · Jan Kiszka
Quoted 37 lines · Valentine Sinitsyn
Hi Jan,
Quoted 18 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.
So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
Quoted 8 lines
Quoted 6 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
Quoted 8 lines
Joel, can you comment on the benefit of variant B) for the various AMD CPUs? Is it always positive?
Thanks,
Jan[1] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/61455
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 47 lines · Avi Kivity
Quoted 46 lines · Jan Kiszka
Quoted 45 lines · Jan Kiszka
Quoted 33 lines · Valentine Sinitsyn
Hi Jan,
Quoted 19 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Quoted 7 lines
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Jan
Signature · 7 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 51 lines · Jan Kiszka
Quoted 50 lines · Avi Kivity
Quoted 49 lines · Jan Kiszka
Quoted 48 lines · Jan Kiszka
Quoted 33 lines · Valentine Sinitsyn
Hi Jan,
Quoted 19 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Quoted 7 lines
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Exceptions that utilize the IST. I found a writeup [17] that describes this, but I think it's even more impossible than that writeup implies.
[17] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/26712/
Quoted 2 lines
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 54 lines · Avi Kivity
Quoted 53 lines · Jan Kiszka
Quoted 52 lines · Avi Kivity
Quoted 51 lines · Jan Kiszka
Quoted 41 lines · Jan Kiszka
Quoted 33 lines · Valentine Sinitsyn
Hi Jan,
Quoted 19 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 5 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to the additional MSR accesses.
Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Quoted 7 lines
Note however that how AMD chips store host state during VM switches are implementation-specific. I did my quick experiments on one CPU only, so your mileage may vary.
Regarding your question, I feel B will be faster anyways but again I'm afraid that the gain could be within statistical error of the experiment.
Quoted 7 lines
It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
I've tested via vmmcall from guest userspace under Jailhouse. KVM should be adjustable in a similar way. Attached the benchmark, patch will be in the Jailhouse next branch soon. We need to check more CPU types, though.
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Exceptions that utilize the IST. I found a writeup [17] that describes this, but I think it's even more impossible than that writeup implies.
I think that Xen does (or did) something along the lines of disabling IST usage (by playing with the descriptors in the IDT) and then re-enabling them when exiting to userspace.
Quoted 6 lines
[17] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/26712/
Quoted 2 lines
Jan
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 75 lines · Avi Kivity
Quoted 74 lines · Avi Kivity
Quoted 72 lines · Jan Kiszka
Quoted 64 lines · Avi Kivity
Quoted 63 lines · Jan Kiszka
Quoted 62 lines · Jan Kiszka
Quoted 46 lines · Valentine Sinitsyn
Hi Jan,
Quoted 22 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use
these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no
user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also
still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 6 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to
the additional MSR accesses.Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing
Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it
didn't seem to affect the latency in any noticeable way. That's why I
decided not to push the patch (in fact, I was even unable to find it now).Note however that how AMD chips store host state during VM switches are
implementation-specific. I did my quick experiments on one CPU only, so
your mileage may vary.Quoted 4 lines
Regarding your question, I feel B will be faster anyways but again I'm
afraid that the gain could be within statistical error of the experiment.It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more
towards 600 if they are colder (added some usleep to each loop in the test).I've tested via vmmcall from guest userspace under Jailhouse. KVM should
be adjustable in a similar way. Attached the benchmark, patch will be in
the Jailhouse next branch soon. We need to check more CPU types, though.Quoted 4 lines
Quoted 2 lines
Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Quoted 8 lines
Quoted 4 lines
Exceptions that utilize the IST. I found a writeup [17] that describes this, but I think it's even more impossible than that writeup implies.
I think that Xen does (or did) something along the lines of disabling IST usage (by playing with the descriptors in the IDT) and then re-enabling them when exiting to userspace.
So we would reuse that active stack for the current IST users until then. But I bet there are subtle details that prevent a simple switch at IDT level. Hmm, no low-hanging fruit it seems...
Quoted 3 lines
Quoted 3 lines
[17] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/26712/
That thread proposed the complete IST removal. But, given that we still have it 7 years later, I suppose that was not very welcome in general.
Thanks,
Jan
PS: For the Jailhouse readers: we don't use IST.
Signature · 7 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 6 lines · Jan Kiszka
Quoted 3 lines · Avi Kivity
I think that Xen does (or did) something along the lines of disabling IST usage (by playing with the descriptors in the IDT) and then re-enabling them when exiting to userspace.
So we would reuse that active stack for the current IST users until then.
Yes.
Quoted 2 lines
But I bet there are subtle details that prevent a simple switch at IDT level. Hmm, no low-hanging fruit it seems...
For sure. It's not insurmountable, but fairly hard.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
[17] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/26712/
That thread proposed the complete IST removal. But, given that we still have it 7 years later,
Well, it's not as if a crack team of kernel hackers was laboring night and day to remove it, but...
Quoted 1 line
I suppose that was not very welcome in general.
Simply removing it is impossible, or an NMI happening immediately after SYSCALL will hit user-provided %rsp.
Quoted 5 lines
Thanks,
JanPS: For the Jailhouse readers: we don't use IST.
You don't have userspace, yes? Only guests?
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 29 lines · Avi Kivity
Quoted 6 lines · Jan Kiszka
Quoted 3 lines · Avi Kivity
I think that Xen does (or did) something along the lines of disabling IST usage (by playing with the descriptors in the IDT) and then re-enabling them when exiting to userspace.
So we would reuse that active stack for the current IST users until then.
Yes.
Quoted 2 lines
But I bet there are subtle details that prevent a simple switch at IDT level. Hmm, no low-hanging fruit it seems...
For sure. It's not insurmountable, but fairly hard.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
[17] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/26712/
That thread proposed the complete IST removal. But, given that we still have it 7 years later,
Well, it's not as if a crack team of kernel hackers was laboring night and day to remove it, but...
Quoted 1 line
I suppose that was not very welcome in general.
Simply removing it is impossible, or an NMI happening immediately after SYSCALL will hit user-provided %rsp.
Quoted 7 lines
Quoted 5 lines
Thanks,
JanPS: For the Jailhouse readers: we don't use IST.
You don't have userspace, yes? Only guests?
Exactly. The day someone adds userspace, I guess I'll have to create a new hypervisor.
Jan
Signature · 7 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Hi all,
Quoted 63 lines · Avi Kivity
Quoted 62 lines · Jan Kiszka
Quoted 61 lines · Avi Kivity
Quoted 60 lines · Jan Kiszka
Quoted 51 lines · Jan Kiszka
Quoted 42 lines · Valentine Sinitsyn
Hi Jan,
Quoted 21 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no
user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also
still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 6 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to
the additional MSR accesses.Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it didn't seem to affect the latency in any noticeable way. That's why I decided not to push the patch (in fact, I was even unable to find it now).
Note however that how AMD chips store host state during VM switches are
implementation-specific. I did my quick experiments on one CPU only, so
your mileage may vary.Quoted 4 lines
Regarding your question, I feel B will be faster anyways but again I'm
afraid that the gain could be within statistical error of the experiment.It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more towards 600 if they are colder (added some usleep to each loop in the test).
Quoted 6 lines
I've tested via vmmcall from guest userspace under Jailhouse. KVM should
be adjustable in a similar way. Attached the benchmark, patch will be in
the Jailhouse next branch soon. We need to check more CPU types, though.Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Exceptions that utilize the IST. I found a writeup [17] that describes this, but I think it's even more impossible than that writeup implies.
Pardon my slowness, but how does it affect Jailhouse running on AMD? For NMI, we do #VMEXIT, but we can disable IST (I'm not sure it's enabled already, in fact). Double faults don't cause #VMEXIT, so there is no VMLOAD/VMSAVE issue. I'm not sure about MCE, but for now they are sort of flawed in Jailhouse anyways IIRC.
What am I missing here?
Thanks,
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 79 lines · Valentine Sinitsyn
Hi all,
Quoted 68 lines · Avi Kivity
Quoted 67 lines · Jan Kiszka
Quoted 66 lines · Avi Kivity
Quoted 65 lines · Jan Kiszka
Quoted 56 lines · Jan Kiszka
Quoted 46 lines · Valentine Sinitsyn
Hi Jan,
Quoted 22 lines · Jan Kiszka
studying the VM exit logic of Jailhouse, I was wondering when AMD's vmload/vmsave can be avoided. Jailhouse as well as KVM currently use
these instructions unconditionally. However, I think both only need GS.base, i.e. the per-cpu base address, to be saved and restored if no
user space exit or no CPU migration is involved (both is always true for
Jailhouse). Xen avoids vmload/vmsave on lightweight exits but it also
still uses rsp-based per-cpu variables.So the question boils down to what is generally faster:
A) vmload
vmrun
vmsaveB)
wrmsrl(MSR_GS_BASE, guest_gs_base)
vmrun
rdmsrl(MSR_GS_BASE, guest_gs_base)Quoted 6 lines
Of course, KVM also has to take into account that heavyweight exits still require vmload/vmsave, thus become more expensive with B) due to
the additional MSR accesses.Any thoughts or results of previous experiments?
That's a good question, I also thought about it when I was finalizing
Jailhouse AMD port. I tried "lightweight exits" with apic-demo but it
didn't seem to affect the latency in any noticeable way. That's why I
decided not to push the patch (in fact, I was even unable to find it now).Note however that how AMD chips store host state during VM switches are
implementation-specific. I did my quick experiments on one CPU only, so
your mileage may vary.Quoted 4 lines
Regarding your question, I feel B will be faster anyways but again I'm
afraid that the gain could be within statistical error of the experiment.It is, at least 160 cycles with hot caches on an AMD A6-5200 APU, more
towards 600 if they are colder (added some usleep to each loop in the test).Quoted 6 lines
I've tested via vmmcall from guest userspace under Jailhouse. KVM should
be adjustable in a similar way. Attached the benchmark, patch will be in
the Jailhouse next branch soon. We need to check more CPU types, though.Avi, I found some preparatory patches of yours from 2010 [1]. Do you happen to remember if it was never completed for a technical reason?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
IIRC, I came to the conclusion that it was impossible. Something about TR.size not receiving a reasonable value. Let me see.
To my understanding, TR doesn't play a role until we leave ring 0 again. Or what could make the CPU look for any of the fields in the 64-bit TSS before that?
Exceptions that utilize the IST. I found a writeup [17] that describes this, but I think it's even more impossible than that writeup implies.
Quoted 7 lines
Pardon my slowness, but how does it affect Jailhouse running on AMD? For NMI, we do #VMEXIT, but we can disable IST (I'm not sure it's enabled already, in fact). Double faults don't cause #VMEXIT, so there is no VMLOAD/VMSAVE issue. I'm not sure about MCE, but for now they are sort of flawed in Jailhouse anyways IIRC.
What am I missing here?
Nothing. As I said in the other branch of this thread, Jailhouse is not affected as it doesn't use the IST. Only KVM is because Linux - in host mode - requires it for the cases Avi mentioned.
Jan
Signature · 7 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Quoted 5 lines · Jan Kiszka
Quoted 1 line
What am I missing here?
Nothing. As I said in the other branch of this thread, Jailhouse is not affected as it doesn't use the IST. Only KVM is because Linux - in host mode - requires it for the cases Avi mentioned.
Then what I am missing is correct discussion branch. :) Thanks for explaining.
Valentine
Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.