List of unaccessible x86 states
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
- nmi_masked
- nmi_pending
- nmi_injected
- kvm_queued_exception (whole struct content)
- KVM_REQ_TRIPLE_FAULT (from vcpu.requests)
Unclear points (for me) from the last discussion:
- sipi_vector
- MCE (covered via kvm_queued_exception, or does it require more?)
Please extend or correct the list as required.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)
Quoted 6 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
Alex
Quoted 20 lines · Alexander Graf
Quoted 18 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
Quoted 7 lines
Quoted 4 lines
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
OK. Whole hflags or just the GIF bit?
If we allow access to all bits, can user space cause any problems (beyond screwing up its guests) by passing weird patterns?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Alexander Graf
Quoted 22 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
Quoted 8 lines
Quoted 2 lines
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
OK. Whole hflags or just the GIF bit?
If we allow access to all bits, can user space cause any problems (beyond screwing up its guests) by passing weird patterns?
HF_NMI_MASK should be migrated too. Destination should enable IRET intercept if HF_NMI_MASK is set. Or we can assume that migration in the middle of NMI will never happen :)
--
Gleb.
Quoted 28 lines · Gleb Natapov
Quoted 27 lines · Jan Kiszka
Quoted 26 lines · Alexander Graf
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 6 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
Quoted 8 lines
Quoted 5 lines
OK. Whole hflags or just the GIF bit?
If we allow access to all bits, can user space cause any problems (beyond screwing up its guests) by passing weird patterns?
HF_NMI_MASKshould be migrated too. Destination should enable IRET intercept ifHF_NMI_MASKis set. Or we can assume that migration in the middle of NMI will never happen :)
HF_NMI_MASK is redundant to the vendor-agnostic nmi_masked and would therefore likely be masked out.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 34 lines · Jan Kiszka
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Jan Kiszka
Quoted 18 lines · Alexander Graf
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 8 lines
Quoted 6 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
OK. Whole hflags or just the GIF bit?
If we allow access to all bits, can user space cause any problems (beyond screwing up its guests) by passing weird patterns?
Quoted 7 lines
Quoted 3 lines
HF_NMI_MASKshould be migrated too. Destination should enable IRET intercept ifHF_NMI_MASKis set. Or we can assume that migration in the middle of NMI will never happen :)
HF_NMI_MASKis redundant to the vendor-agnosticnmi_maskedand would therefore likely be masked out.
Correct. We can restore HF_NMI_MASK from nmi_masked.
--
Gleb.
Quoted 24 lines · Jan Kiszka
Quoted 23 lines · Alexander Graf
Quoted 21 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
OK. Whole hflags or just the GIF bit?
agraf@busu:~/git/kvm> grep -R HF_ arch/x86/include/asm/*kvm*
arch/x86/include/asm/kvm_host.h:#define HF_GIF_MASK (1 << 0)
arch/x86/include/asm/kvm_host.h:#define HF_HIF_MASK (1 << 1)
arch/x86/include/asm/kvm_host.h:#define HF_VINTR_MASK (1 << 2)
arch/x86/include/asm/kvm_host.h:#define HF_NMI_MASK (1 << 3)
arch/x86/include/asm/kvm_host.h:#define HF_IRET_MASK (1 << 4)
I can only talk for GIF here and that should be fine. Not knowing about the others does seem like we could get race conditions though.
Quoted 2 lines
If we allow access to all bits, can user space cause any problems (beyond screwing up its guests) by passing weird patterns?
IMHO the hflags should be converted between userspace and kernel representation. There's a good chance we run older userspace that doesn't know about certain flags yet and I'd like to keep the bits as flexible as possible.
Alex
Quoted 20 lines · Alexander Graf
Quoted 18 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
Quoted 7 lines
Quoted 4 lines
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
BTW, GIF is related to svm nesting, right?
Orit, are there any additional states arriving on the vmx side as well with your nesting patches?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 24 lines · Jan Kiszka
Quoted 23 lines · Alexander Graf
Quoted 21 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a nested VM. We can just #VMEXIT just before migrating with a VMEXIT_INTR intercept.
Now just after #VMEXIT we're in a state that's pure host context, but has GIF=0. So we need to know about that in userspace to support migration.
Alex
Quoted 32 lines · Alexander Graf
Quoted 29 lines · Jan Kiszka
Quoted 28 lines · Alexander Graf
Quoted 25 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Quoted 3 lines
nested VM. We can just #VMEXIT just before migrating with a
VMEXIT_INTRintercept.
We don't notify kernel about migration currently. CPU state is migrated when VM is already paused, how we can exit nested guest at this point?
Quoted 5 lines
Now just after #VMEXIT we're in a state that's pure host context, but has GIF=0. So we need to know about that in userspace to support migration.
Alex
--
Gleb.
Quoted 36 lines · Gleb Natapov
Quoted 35 lines · Alexander Graf
Quoted 32 lines · Jan Kiszka
Quoted 30 lines · Alexander Graf
Quoted 19 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 6 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to sync it.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
Quoted 5 lines
Quoted 3 lines
nested VM. We can just #VMEXIT just before migrating with a
VMEXIT_INTRintercept.We don't notify kernel about migration currently. CPU state is migrated when VM is already paused, how we can exit nested guest at this point?
Hm - introduce a new ioctl? I haven't fully thought it through yet :-).
Alex
Quoted 42 lines · Alexander Graf
Quoted 39 lines · Gleb Natapov
Quoted 38 lines · Alexander Graf
Quoted 34 lines · Jan Kiszka
Quoted 23 lines · Alexander Graf
Quoted 21 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
Quoted 8 lines
Quoted 4 lines
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to
sync it.BTW, GIF is related to svm nesting, right?
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
But the amount of information is constant no matter how l2 guest there are. Correct? We can expose it as separate substate.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
nested VM. We can just #VMEXIT just before migrating with a
VMEXIT_INTRintercept.We don't notify kernel about migration currently. CPU state is migrated
when VM is already paused, how we can exit nested guest at this point?Hm - introduce a new ioctl? I haven't fully thought it through yet :-).
There is not software problem that can't be solved by introducing new ioctl :)
--
Gleb.
Quoted 47 lines · Gleb Natapov
Quoted 46 lines · Alexander Graf
Quoted 43 lines · Gleb Natapov
Quoted 42 lines · Alexander Graf
Quoted 28 lines · Jan Kiszka
Quoted 27 lines · Alexander Graf
Quoted 25 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to
sync it.BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Why not?
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
But the amount of information is constant no matter how l2 guest there are. Correct? We can expose it as separate substate.
Or we can just not migrate while in a nested guest :-). Which will make everything a lot easier.
Alex
Quoted 56 lines · Alexander Graf
Quoted 53 lines · Gleb Natapov
Quoted 52 lines · Alexander Graf
Quoted 38 lines · Gleb Natapov
Quoted 37 lines · Alexander Graf
Quoted 34 lines · Jan Kiszka
Quoted 32 lines · Alexander Graf
Quoted 19 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 7 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to
sync it.Quoted 6 lines
Quoted 5 lines
Quoted 1 line
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
Quoted 6 lines
Quoted 2 lines
But the amount of information is constant no matter how l2 guest there are. Correct? We can expose it as separate substate.
Or we can just not migrate while in a nested guest :-). Which will make everything a lot easier.
Suppose we have a l2 guest that handles interrupt/nmis by itself how can we force it to exit? I don't think requesting certain cpu state before migration is the right thing to do. What if user paused a VM and then decided to migrate? Or VM was paused automatically because of shortage of disk space and management want to migrate VM to other host with bigger disk?
--
Gleb.
Quoted 57 lines · Gleb Natapov
Quoted 56 lines · Alexander Graf
Quoted 53 lines · Gleb Natapov
Quoted 52 lines · Alexander Graf
Quoted 38 lines · Gleb Natapov
Quoted 37 lines · Alexander Graf
Quoted 34 lines · Jan Kiszka
Quoted 32 lines · Alexander Graf
Quoted 19 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 7 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to
sync it.Quoted 6 lines
Quoted 5 lines
Quoted 1 line
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
But the amount of information is constant no matter how l2 guest there are. Correct? We can expose it as separate substate.
Or we can just not migrate while in a nested guest :-). Which will make everything a lot easier.
Suppose we have a l2 guest that handles interrupt/nmis by itself how can we force it to exit?
If the nested hypervisor doesn't intercept INTR we don't support it anyways.
Quoted 3 lines
I don't think requesting certain cpu state before migration is the right thing to do. What if user paused a VM and then decided to migrate?
So pausing has to make it go out of nested guest context too? Then we're not in the nested guest context, right? :)
Quoted 3 lines
Or VM was paused automatically because of shortage
of disk space and management want to migrate VM to other host with bigger disk?
Same as before.
Really, pushing the whole nesting state over is not a good idea.
Alex
Quoted 68 lines · Alexander Graf
Quoted 65 lines · Gleb Natapov
Quoted 64 lines · Alexander Graf
Quoted 50 lines · Gleb Natapov
Quoted 49 lines · Alexander Graf
Quoted 45 lines · Gleb Natapov
Quoted 37 lines · Alexander Graf
Quoted 34 lines · Jan Kiszka
Quoted 32 lines · Alexander Graf
Quoted 19 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM,
this is an attempt to collect the precise requirements for additional
state fields. Once everyone feels the list is complete, we can decide
how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 7 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
hflags. Qemu supports GIF, kvm supports GIF, but no side knows how to
sync it.Quoted 6 lines
Quoted 1 line
BTW, GIF is related to svm nesting, right?
Yes and no. It's an architecture addition that came with SVM, yes.
The problem is that I don't want to support migrating while in a
Why not?
Quoted 7 lines
Quoted 4 lines
Because then we'd have to transfer the whole host cpu cache and the merged intercept bitmaps to userspace as well. That's just too many internals to expose IMHO.
But the amount of information is constant no matter how l2 guest there
are. Correct? We can expose it as separate substate.Or we can just not migrate while in a nested guest :-). Which will make everything a lot easier.
Quoted 7 lines
Quoted 3 lines
Suppose we have a l2 guest that handles interrupt/nmis by itself how can we
force it to exit?If the nested hypervisor doesn't intercept INTR we don't support it anyways.
Why? I looked at the code briefly and it looks like we just inject interrupt as usual instead of do nested exit if l2 does not intercept INTR. Have I miss interpreted the code. Even if I have why not support it?
Quoted 5 lines
Quoted 3 lines
I don't think requesting certain cpu state before migration is the right thing to do. What if user paused a VM and then decided to migrate?
So pausing has to make it go out of nested guest context too?
Probably.
Quoted 7 lines
Then we're not in the nested guest context, right? :)
Quoted 3 lines
Or VM was paused automatically because of shortage
of disk space and management want to migrate VM to other host with bigger disk?Same as before.
What do you mean?
Quoted 4 lines
Really, pushing the whole nesting state over is not a good idea.
May be just disallow migration with nested guest running then? Cross vendor migration is not possible anyway.
--
Gleb.
Quoted 2 lines · Alexander Graf
If the nested hypervisor doesn't intercept INTR we don't support it anyways.
That's a bug.
Quoted 1 line
Really, pushing the whole nesting state over is not a good idea.
Isn't the entire state just one bit? Everything else should be saved to guest memory.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 25.10.2009 um 10:46 schrieb Avi Kivity <[email protected]>:
Quoted 5 lines · Alexander Graf
Quoted 2 lines
If the nested hypervisor doesn't intercept INTR we don't support it anyways.
That's a bug.
It's a question of how accurate we want to be.
Quoted 4 lines
Quoted 2 lines
Really, pushing the whole nesting state over is not a good idea.
Isn't the entire state just one bit? Everything else should be saved to guest memory.
It's not. We can't use the guest memory for hsave because then the guest could break the l1 state, so a malicious hypervisor could break us.
Alex
Quoted 4 lines
Signature · 2 lines
-- error compiling committee.c: too many arguments to function
Quoted 10 lines · Alexander Graf
Am 25.10.2009 um 10:46 schrieb Avi Kivity <[email protected]>:
Quoted 5 lines · Alexander Graf
Quoted 2 lines
If the nested hypervisor doesn't intercept INTR we don't support it anyways.
That's a bug.
It's a question of how accurate we want to be.
Even if we don't implement it immediately, it's still a bug. It won't matter much until we hit a guest that needs it.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Really, pushing the whole nesting state over is not a good idea.
Isn't the entire state just one bit? Everything else should be saved to guest memory.
It's not. We can't use the guest memory for hsave because then the guest could break the l1 state, so a malicious hypervisor could break us.
Guest hsave should be used for storing guest state when switching into the nested guest, not host state. Host state is not part of the save/restore state in any case.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 25.10.2009 um 15:08 schrieb Avi Kivity <[email protected]>:
Quoted 14 lines · Alexander Graf
Quoted 10 lines
Am 25.10.2009 um 10:46 schrieb Avi Kivity <[email protected]>:
Quoted 5 lines · Alexander Graf
Quoted 2 lines
If the nested hypervisor doesn't intercept INTR we don't support it anyways.
That's a bug.
It's a question of how accurate we want to be.
Even if we don't implement it immediately, it's still a bug. It won't matter much until we hit a guest that needs it.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Really, pushing the whole nesting state over is not a good idea.
Isn't the entire state just one bit? Everything else should be saved to guest memory.
It's not. We can't use the guest memory for hsave because then the guest could break the l1 state, so a malicious hypervisor could break us.
Guest hsave should be used for storing guest state when switching into the nested guest, not host state. Host state is not part of the save/restore state in any case.
No it's not.
When going in an l2 guest, we need to save the l1 state in the hsave. Now if we'd use the l1 given hsave, the l2 guest could modify the hsave.
That means the l2 guest could rewrite the intercept bitmap to 0 and compromize the host.
That's why we're storing the hsave data in a host allocated page.
Of course, we could save the whole hsave are off to the host on migeation...
Alex
Quoted 1 line
Quoted 10 lines · Alexander Graf
Quoted 3 lines
Quoted 1 line
It's not. We can't use the guest memory for hsave because then the guest could break the l1 state, so a malicious hypervisor could break us.
Guest hsave should be used for storing guest state when switching into the nested guest, not host state. Host state is not part of the save/restore state in any case.
No it's not.
When going in an l2 guest, we need to save the l1 state in the hsave. Now if we'd use the l1 given hsave, the l2 guest could modify the hsave.
That means the l2 guest could rewrite the intercept bitmap to 0 and compromize the host.
L1 hsave stores the architected state saved by vmrun, e.g. cs.sel, next_rip, cr0, cr3, etc. The host intercept bitmap is not state since it is calculated from the L1 intercept bitmap and host code. Indeed it can be different from host to host even with the same guest state.
Quoted 3 lines
That's why we're storing the hsave data in a host allocated page.
Of course, we could save the whole hsave are off to the host on migeation...
Sorry, -ENOPARSE.
Signature · 1 line
error compiling committee.c: too many arguments to function
Am 26.10.2009 um 09:33 schrieb Avi Kivity <[email protected]>:
Quoted 6 lines · Alexander Graf
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
It's not. We can't use the guest memory for hsave because then the guest could break the l1 state, so a malicious hypervisor could break us.
Guest hsave should be used for storing guest state when switching into the nested guest, not host state. Host state is not part of the save/restore state in any case.
Quoted 7 lines
Quoted 5 lines
No it's not.
When going in an l2 guest, we need to save the l1 state in the hsave. Now if we'd use the l1 given hsave, the l2 guest could modify the hsave.
That means the l2 guest could rewrite the intercept bitmap to 0 and compromize the host.
L1 hsave stores the architected state saved by vmrun, e.g. cs.sel,
next_rip, cr0, cr3, etc. The host intercept bitmap is not state since it is calculated from the L1 intercept bitmap and host code. Indeed it can be different from host to host even with the same guest state.
Ah, so you'd only save off the cpu state parts of the vmcb.
Currently we save off control parts too, so we can easily swap them in on #vmexit.
So if we'd migrate off when inside the nested guest, we'd have to save off the resume control state, OR them again with the guest vmcb control states and be inside the nested guest.
Wouldn't it be much easier to not migrate / save state when inside a nested guest? I'm afraid the code will become overly complex if we do allow migration while in a nested context.
Alex
Quoted 1 line
Quoted 6 lines · Alexander Graf
Quoted 1 line
L1 hsave stores the architected state saved by vmrun, e.g. cs.sel,
next_rip, cr0, cr3, etc. The host intercept bitmap is not state since it is calculated from the L1 intercept bitmap and host code. Indeed it can be different from host to host even with the same guest state.Ah, so you'd only save off the cpu state parts of the vmcb.
Currently we save off control parts too, so we can easily swap them in on #vmexit.
These can still be saved in a host memory area as an optimization, and regenerated if needed.
Quoted 1 line
So if we'd migrate off when inside the nested guest, we'd have to save off the resume control state, OR them again with the guest vmcb control states and be inside the nested guest.
Right, if the new state bit (guest mode) is set, we look at the control bits and OR them into the vmcb. That part can be reused with the VMRUN code.
Quoted 2 lines
Wouldn't it be much easier to not migrate / save state when inside a nested guest? I'm afraid the code will become overly complex if we do allow migration while in a nested context.
I can't really see why but then I don't know the code as well as you do. The current code won't work for guests which don't intercept external interrupts (probably only malware). For nested vmx it may be necessary since vmx has a mode where interrupts are acknowledged during #VMEXIT and the interrupt vector is saved into a register; you can't fake an interrupt #VMEXIT since you can't fake the vector. Xen is one guest which uses this mode.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)
Quoted 3 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
Should be migrated.
Quoted 9 lines
- MCE (covered via
kvm_queued_exception, or does it require more?)Please extend or correct the list as required.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
--
Gleb.
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)
Quoted 4 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)
Should save/restore the MCE MSRs (its contents are currently lost/overwritten AFAICS).
MTRR contents are also dropped.
Quoted 19 lines · Marcelo Tosatti
Quoted 18 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
Quoted 7 lines
Quoted 2 lines
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Should save/restore the MCE MSRs (its contents are currently lost/overwritten AFAICS).
MTRR contents are also dropped.
Hmm, the code path is winding, but aren't they already available to user space via GET/SET_MSRS?
Jan
Quoted 23 lines · Jan Kiszka
Quoted 22 lines · Marcelo Tosatti
Quoted 20 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)
Quoted 7 lines
Quoted 4 lines
Should save/restore the MCE MSRs (its contents are currently lost/overwritten AFAICS).
MTRR contents are also dropped.
Hmm, the code path is winding, but aren't they already available to user space via GET/SET_MSRS?
Yes, nevermind, irrelevant to the current discussion.
Quoted 23 lines · Marcelo Tosatti
Quoted 22 lines · Jan Kiszka
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Should save/restore the MCE MSRs (its contents are currently lost/overwritten AFAICS).
MTRR contents are also dropped.
Hmm, the code path is winding, but aren't they already available to user space via GET/SET_MSRS?
Yes, nevermind, irrelevant to the current discussion.
Oh, then I misunderstood your original reply as "we need to add them to the list as well". Even better.
Jan
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)
Quoted 7 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
Here is a wrap-up of what has been reported so far:
- NMI
o nmi_masked
o nmi_pending
o nmi_injected
- queued exception
o kvm_queued_exception
o triple_fault
- SVM
o gif
(Are we sure that there is really nothing more here?)
- sipi_vector
So the next question is how to map these on substates. I'm currently leaning towards this organization:
- KVM_X86_VCPU_STATE_EVENTS
o NMI states
o pending exception
o sipi_vector
o pending interrupt?
(would be redundant to kvm_sregs.interrupt_bitmap, but that struct may be obsoleted one day)
- KVM_X86_VCPU_STATE_SVM
o gif
Any concerns or better suggestions?
Jan
Quoted 27 lines · Jan Kiszka
Quoted 16 lines · Jan Kiszka
Hi all,
as the list of yet user-unaccessible x86 states is a bit volatile ATM, this is an attempt to collect the precise requirements for additional state fields. Once everyone feels the list is complete, we can decide how to partition it into one ore more substates for the new KVM_GET/SET_VCPU_STATE interface.
What I read so far (or tried to patch already):
-
nmi_masked
-nmi_pending
-nmi_injected
-kvm_queued_exception(whole struct content)
-KVM_REQ_TRIPLE_FAULT(from vcpu.requests)Quoted 7 lines
Unclear points (for me) from the last discussion:
-
sipi_vector
- MCE (covered viakvm_queued_exception, or does it require more?)Please extend or correct the list as required.
Here is a wrap-up of what has been reported so far:
Quoted 10 lines
- NMI
onmi_masked
onmi_pending
onmi_injected
- queued exception
okvm_queued_exception
otriple_fault
- SVM
o gif
(Are we sure that there is really nothing more here?)
Hm, thinking about this again, it might be useful to have an "currently in nested VM" flag here. That way userspace can decide if it needs to get out of the nested state (for migration) or if it just doesn't care.
Quoted 5 lines
-
sipi_vectorSo the next question is how to map these on substates. I'm currently leaning towards this organization:
Quoted 10 lines
-
KVM_X86_VCPU_STATE_EVENTS
o NMI states
o pending exception
osipi_vector
o pending interrupt?
(would be redundant to kvm_sregs.interrupt_bitmap, but that
struct
may be obsoleted one day)
-KVM_X86_VCPU_STATE_SVM
o gif
Can we make this an "svm_flags" or so u32? And then we'd just set bits?
Alex
Quoted 2 lines · Alexander Graf
Hm, thinking about this again, it might be useful to have an "currently in nested VM" flag here. That way userspace can decide if it needs to get out of the nested state (for migration) or if it just doesn't care.
Getting out of nested state involves modifying state (both memory and registers). Nor can we in the general case force it. The guest can set up a situation where it is impossible to #vmexit.
Quoted 5 lines
Quoted 2 lines
-
KVM_X86_VCPU_STATE_SVM
o gifCan we make this an "
svm_flags" or so u32? And then we'd just set bits?
Or individual flags as u8s, so we don't get trapped into a specific encoding which is really an implementation detail.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 5 lines · Alexander Graf
Hm, thinking about this again, it might be useful to have an "currently in nested VM" flag here. That way userspace can decide if it needs to get out of the nested state (for migration) or if it just doesn't care.
Getting out of nested state involves modifying state (both memory and registers). Nor can we in the general case force it. The guest can set up a situation where it is impossible to #vmexit.
There is actually more than that. If the guest runs in guest mode itself we also need to report the host state to be able to do an #vmexit after migration.
In nested SVM the host state is not saved in the guest memory to prevent the guest from modifying it and break out of its virtualization jail.
Joerg
Quoted 13 lines · Joerg Roedel
Quoted 11 lines · Avi Kivity
Quoted 5 lines · Alexander Graf
Hm, thinking about this again, it might be useful to have an "currently in nested VM" flag here. That way userspace can decide if it needs to get out of the nested state (for migration) or if it just doesn't care.
Getting out of nested state involves modifying state (both memory and registers). Nor can we in the general case force it. The guest can set up a situation where it is impossible to #vmexit.
Quoted 6 lines
There is actually more than that. If the guest runs in guest mode itself we also need to report the host state to be able to do an #vmexit after migration.
In nested SVM the host state is not saved in the guest memory to prevent the guest from modifying it and break out of its virtualization jail.
Which host state? As far as I can tell, it can all be regenerated.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 9 lines · Joerg Roedel
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Alexander Graf
Hm, thinking about this again, it might be useful to have an "currently in nested VM" flag here. That way userspace can decide if it needs to get out of the nested state (for migration) or if it just doesn't care.
Getting out of nested state involves modifying state (both memory and registers). Nor can we in the general case force it. The guest can set up a situation where it is impossible to #vmexit.
Quoted 7 lines
Quoted 5 lines
There is actually more than that. If the guest runs in guest mode itself we also need to report the host state to be able to do an #vmexit after migration.
In nested SVM the host state is not saved in the guest memory to prevent the guest from modifying it and break out of its virtualization jail.Which host state? As far as I can tell, it can all be regenerated.
The state which is loaded into the vcpu when a #vmexit is emulated. This includes segments, control registers and the host rip for example.
Joerg
Quoted 6 lines · Joerg Roedel
Quoted 3 lines
Which host state? As far as I can tell, it can all be regenerated.
The state which is loaded into the vcpu when a #vmexit is emulated. This includes segments, control registers and the host rip for example.
All of this state does not change between nested guest and normal guest mode.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Joerg Roedel
Quoted 2 lines
Which host state? As far as I can tell, it can all be regenerated.
The state which is loaded into the vcpu when a #vmexit is emulated. This includes segments, control registers and the host rip for example.
All of this state does not change between nested guest and normal guest mode.
I am talking about all the state that is saved in svm->nested.hsave. When we migrate a guest vcpu while it is running in guest mode itself (without forcing a nested #vmexit) this state is required when a #vmexit needs to be emulated on this vcpu after migration.
Same is true for the nested intercept conditions.
Joerg
Quoted 10 lines · Joerg Roedel
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Joerg Roedel
Quoted 3 lines
Which host state? As far as I can tell, it can all be regenerated.
The state which is loaded into the vcpu when a #vmexit is emulated. This includes segments, control registers and the host rip for example.
Quoted 9 lines
Quoted 3 lines
All of this state does not change between nested guest and normal guest mode.
I am talking about all the state that is saved in
svm->nested.hsave. When we migrate a guest vcpu while it is running in guest mode itself (without forcing a nested #vmexit) this state is required when a #vmexit needs to be emulated on this vcpu after migration.
Same is true for the nested intercept conditions.
The state that is saved by VMRUN can be saved to guest memory and migrated. Extra state (like the intercepts for the previous mode) must be saved to host memory and not migrated; host intercepts can be regenerated.
Concretely:
hsave->save.es = vmcb->save.es;
hsave->save.cs = vmcb->save.cs;
hsave->save.ss = vmcb->save.ss;
hsave->save.ds = vmcb->save.ds;
hsave->save.gdtr = vmcb->save.gdtr;
hsave->save.idtr = vmcb->save.idtr;
hsave->save.efer = svm->vcpu.arch.shadow_efer;
hsave->save.cr0 = svm->vcpu.arch.cr0;
hsave->save.cr4 = svm->vcpu.arch.cr4;
hsave->save.rflags = vmcb->save.rflags;
hsave->save.rip = svm->next_rip;
hsave->save.rsp = vmcb->save.rsp;
hsave->save.rax = vmcb->save.rax;
if (npt_enabled)
hsave->save.cr3 = vmcb->save.cr3;
else
hsave->save.cr3 = svm->vcpu.arch.cr3;Can all be saved to guest memory.
copy_vmcb_control_area(hsave, vmcb);
Must not be saved into guest memory. On the other hand, it is not needed for migration.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 14 lines · Avi Kivity
Quoted 5 lines · Joerg Roedel
Quoted 4 lines · Avi Kivity
Quoted 3 lines · Joerg Roedel
Quoted 1 line
Which host state? As far as I can tell, it can all be regenerated.
The state which is loaded into the vcpu when a #vmexit is emulated. This includes segments, control registers and the host rip for example.
Quoted 7 lines
Quoted 2 lines
All of this state does not change between nested guest and normal guest mode.
I am talking about all the state that is saved in
svm->nested.hsave. When we migrate a guest vcpu while it is running in guest mode itself (without forcing a nested #vmexit) this state is required when a #vmexit needs to be emulated on this vcpu after migration.
Same is true for the nested intercept conditions.
Quoted 4 lines
The state that is saved by VMRUN can be saved to guest memory and migrated. Extra state (like the intercepts for the previous mode) must be saved to host memory and not migrated; host intercepts can be regenerated.
Ok, parts of the state can be saved in guest memory. But thats currently not done. This will need some care to not introduce a security hole. But it shouldn't be too difficult.
The state thats not reproducible in an sane way is the intercept bitmap for the l2 guest.
Quoted 1 line
From the nested state what needs to be exposed to userspace for
migration is:
* guest mode flag (as returned by is_nested)
* nested vmcb address
* nested hsave msr
* nested intercepts
* for nested nested paging: guest nested cr3 value
Another state which needs exposure is the last branch record related state.
Off-topic question: Will the new migration protocol include some kind
handshake to find out if migration is possible at all?
Joerg
Quoted 9 lines · Joerg Roedel
Ok, parts of the state can be saved in guest memory. But thats currently not done. This will need some care to not introduce a security hole. But it shouldn't be too difficult.
The state thats not reproducible in an sane way is the intercept bitmap for the l2 guest.
From the nested state what needs to be exposed to userspace for
migration is:
Quoted 3 lines
* guest mode flag (as returned by
is_nested)
* nested vmcb address
Yes, forgot that. We can store it in the hsave area (note the hsave area format becomes an ABI).
Quoted 2 lines
* nested hsave msr
That's already saved.
Quoted 2 lines
* nested intercepts
These are part of the guest vmcb. The host nested intercepts can be recalculated, no?
Quoted 2 lines
* for nested nested paging: guest nested cr3 value
Part of the guest vmcb.
Quoted 3 lines
Another state which needs exposure is the last branch record related state.
Aren't those just more MSRs?
Quoted 4 lines
Off-topic question: Will the new migration protocol include some kind
handshake to find out if migration is possible at all?
It's assumed that migration always works for a newer qemu version, and that the management tools don't attempt backward migration.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 1 line · Joerg Roedel
* nested intercepts
These are part of the guest vmcb. The host nested intercepts can be recalculated, no?
Quoted 1 line
* for nested nested paging: guest nested cr3 value
Part of the guest vmcb.
This will work is most cases. But its not architecturally sane because real hardware caches this information in the cpu. So software is free to modify the vmcb without impacting the in-cpu state until the next #vmexit. I don't know any software which relies on that so it may be not an issue.
Quoted 6 lines
Quoted 3 lines
Off-topic question: Will the new migration protocol include some kind
handshake to find out if migration is possible at all?It's assumed that migration always works for a newer qemu version, and that the management tools don't attempt backward migration.
I think such a handshake would make sense to just prevent that a nested svm hypervisor is migrated to an intel machine or vice versa (just an example, there are more like sse*, nested nested paging, ...).
Joerg