About the behavior of HLT in VMX guest mode

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

Any suggestion would be greatly appreciated, thanks!

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 8 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

Quoted 5 lines

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

Quoted 2 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

Quoted 1 line

Any suggestion would be greatly appreciated, thanks!

Mostly just more questions, sorry ...

I'd look at temperature sensors while halting inside guests on all cores to see if they really enter a power saving mode -- I expect a noticeable difference from idle=poll. :)

Hi, Radim,

Quoted 9 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 8 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

Quoted 8 lines

Quoted 5 lines

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

Quoted 7 lines

Quoted 2 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------

			(vcpu2)idle=poll	(vcpu2)idle=halt

(HLT exiting=1)
vcpu1 costs 3800931 1900209

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Quoted 9 lines

Quoted 1 line

Any suggestion would be greatly appreciated, thanks!

Mostly just more questions, sorry ...

I'd look at temperature sensors while halting inside guests on all cores to see if they really enter a power saving mode -- I expect a noticeable difference from idle=poll. :)

Hi, Radim,

Quoted 15 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 13 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Quoted 2 lines

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

Quoted 8 lines

Quoted 3 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------
(vcpu2)idle=poll (vcpu2)idle=halt
(HLT exiting=1)
vcpu1 costs 3800931 1900209

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Quoted 9 lines

Quoted 1 line

Any suggestion would be greatly appreciated, thanks!

Mostly just more questions, sorry ...

I'd look at temperature sensors while halting inside guests on all cores to see if they really enter a power saving mode -- I expect a noticeable difference from idle=poll. :)

--
Regards,
Longpeng(Mike)

2017-03-16 10:08 GMT+08:00 Longpeng (Mike) <[email protected]>:

Quoted 19 lines

Hi, Radim,

Quoted 15 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 13 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Quoted 6 lines

Quoted 2 lines

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

Execute hlt will enter C1 state, just mwait can enter C1E state.

Regards,
Wanpeng Li

Quoted 53 lines

Quoted 8 lines

Quoted 3 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------
(vcpu2)idle=poll (vcpu2)idle=halt
(HLT exiting=1)
vcpu1 costs 3800931 1900209

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Quoted 9 lines

Quoted 1 line

Any suggestion would be greatly appreciated, thanks!

Mostly just more questions, sorry ...

I'd look at temperature sensors while halting inside guests on all cores to see if they really enter a power saving mode -- I expect a noticeable difference from idle=poll. :)

Quoted 4 lines

--
Regards,
Longpeng(Mike)

.

Hi, Radim,

Quoted 12 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 11 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

Quoted 5 lines

Quoted 2 lines

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

Quoted 7 lines

Quoted 2 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------

			(vcpu2)idle=poll	(vcpu2)idle=halt

(HLT exiting=1)
vcpu1 costs 3800931 1900209

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

2017-03-16 10:08+0800, Longpeng (Mike):

Quoted 16 lines

Hi, Radim,

Quoted 12 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 11 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

I'd go for the thermal monitoring (ideally with constant fan speed) if CPU counters are lacking. Thermal sensors are easily accessible and far more trustworthy for measuring power saving. :)

Quoted 29 lines

Quoted 7 lines

Quoted 2 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------

			(vcpu2)idle=poll	(vcpu2)idle=halt

(HLT exiting=1)
vcpu1 costs 3800931 1900209

Quoted 3 lines

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

Oh, great results.
I wonder if the slightly better time on HLT exiting=1 is because the other hyper-thread goes into deeper sleep after exit.

Btw. does adding pause() into the while loop bring the performance close to halt?

Quoted 8 lines

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Yes, and using MWAIT for idle is even better than HLT (you can be woken

up without IPI) -- any reason to prefer HLT?

Thanks.

.

Hi Radim,

Quoted 59 lines · Radim Krčmář

2017-03-16 10:08+0800, Longpeng (Mike):

Quoted 16 lines

Hi, Radim,

Quoted 12 lines · Radim Krčmář

2017-03-13 15:12+0800, Longpeng (Mike):

Quoted 11 lines

Hi guys,

I'm confusing about the behavior of HLT instruction in VMX guest mode.

I set "hlt exiting" bit to 0 in VMCS, and the vcpu didn't vmexit when execute HLT as expected. However, I used powertop/cpupower on host to watch the pcpu's c-states, it seems that the pcpu didn't enter C1/C1E state during this period.

I searched the Intel spec vol-3, and only found that guest MWAIT won't entering a low-power sleep state under certain conditions(ch 25.3), but not mentioned HLT.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

My questions are
1) Does executing HLT instruction in guest-mode won't enter C1/C1E state ?

Do you get a different result when running HLT outside VMX?

Yep, I'm sure that executing HLT in host will enter C1/C1E state, but it won't when executing in guest.

I'd go for the thermal monitoring (ideally with constant fan speed) if CPU counters are lacking. Thermal sensors are easily accessible and far more trustworthy for measuring power saving. :)

Quoted 29 lines

Quoted 7 lines

Quoted 2 lines

2) If it won't, then whether it would release the hardware resources shared with another hyper-thread ?

No idea. Aren't hyperthreaded resources scheduled dynamically, so even a nop-spinning VCPU won't hinder the other hyper-thread?

I had wrote a testcase in kvm-unit-tests, and it seems that guest-mode HLT-ed vcpu won't compete the hardware resources( maybe including the pipeline ) any more.

My testcase is: binding vcpu1 and vcpu2 to a core's 2 hyper-threads, and

(vcpu1)
t1 = rdtsc();
for (int i = 0; i < 10000000; ++i) ;
t2 = rdtsc();
costs = t2 - t1;

(vcpu2)
"halt" or "while (1) ;"

The result is:
-----------------------------------------------------------------------

			(vcpu2)idle=poll	(vcpu2)idle=halt

(HLT exiting=1)
vcpu1 costs 3800931 1900209

Quoted 7 lines

Quoted 3 lines

(HLT exiting=0)
vcpu1 costs 3800193 1913514
-----------------------------------------------------------------------

Oh, great results.
I wonder if the slightly better time on HLT exiting=1 is because the other hyper-thread goes into deeper sleep after exit.

Yes, maybe.

Another potential reason is maybe the host's overhead is lower.
For "HLT exiting=1 && idle=halt" the host is idle and the cpu-usage close to 0%, while for "HLT exiting=0 && idle=halt" the host is actually very busy and the cpu-usage close to 100%. Maybe host kernel would do more work when it's busy.

Quoted 2 lines

Btw. does adding pause() into the while loop bring the performance close to halt?

Good suggestion! :)

I tested pause() into poll loop and set "ple_gap=0" just now, the performance is much better than "while(1) ;", but it's still obvious slower than halt.
-----------------------------------------------------------------------
(vcpu2)poll (vcpu2)pause (vcpu2)halt
(HLT exiting=1)
vcpu1 costs 3800931 2572812 1916724

(HLT exiting=0)
vcpu1 costs 3800193 2573685 1912443
-----------------------------------------------------------------------

Quoted 10 lines

Quoted 8 lines

I found that https://www.spinics.net/lists/kvm-commits/msg00137.html had maked "HLT exiting" configurable, while
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html removed it due to redundant with CFS hardlimit.

I focus on the VM's performance. According the result, I think running HLT in guest-mode is better than idle=poll with HLT-exiting in *certain* scenarios.

Yes, and using MWAIT for idle is even better than HLT (you can be woken

Yes, agree.

Quoted 2 lines

up without IPI) -- any reason to prefer HLT?

In my humble opinion:

1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes entering deeper sleep) under certain conditions.
Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache. This is insecurity if we don't take other protective measures(such as limit the guest's max-cstate, it's fortunately that power subsystem isn't supported by QEMU, but we should be careful for some special-purpose in case). While HLT in guest mode can't cause hardware into sleep.

2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in guest mode can't work as perfect as in host sometimes.
For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor address will be cleaned, so the MWAIT won't be waken up by a store operation to the monitored address any more.

But I'm glad to do some tests if time permits, thanks :)

Radim, how about to make HLT-exiting configurable again in upstream ? If you like it, there is a problem should be resolved, asynpf is conflict with "HLT-exiting = 0" in certain situations.

2017-03-17 13:22+0800, Longpeng (Mike):

Quoted 1 line

Hi Radim,

Quoted 9 lines

In my humble opinion:

1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes entering deeper sleep) under certain conditions.
Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache. This is insecurity if we don't take other protective measures(such as limit the guest's max-cstate, it's fortunately that power subsystem isn't supported by QEMU, but we should be careful for some special-purpose in case). While HLT in guest mode can't cause hardware into sleep.

Good point. I'm not aware of any VMX capabilities to prevent deeper C-states, so we'd always hope that guests obey provided information.

Quoted 6 lines

2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in guest mode can't work as perfect as in host sometimes.
For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor address will be cleaned, so the MWAIT won't be waken up by a store operation to the monitored address any more.

It's not as perfect, but should not cause a bug (well, there is a discussion with suspicious MWAIT behavior :]).
MWAIT on all Intels I tested would just behave as a nop if exit happened between MONITOR and MWAIT, like it does if you skip the MONITOR (MWAIT instruction desciption):

If the preceding MONITOR instruction did not successfully arm an address range or if the MONITOR instruction has not been executed prior to executing MWAIT, then the processor will not enter the implementation-dependent-optimized state. Execution will resume at the instruction following the MWAIT.

Quoted 5 lines

But I'm glad to do some tests if time permits, thanks :)

Radim, how about to make HLT-exiting configurable again in upstream ? If you like it, there is a problem should be resolved, asynpf is conflict with "HLT-exiting = 0" in certain situations.

Go ahead. KVM should provide access to hardware features and no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a good case) capability. I'm interested in the asyncpf conflict.

Thanks.

.

Hi Radim,

Quoted 2 lines · Radim Krčmář

2017-03-17 13:22+0800, Longpeng (Mike):

Quoted 1 line

Hi Radim,

...

Quoted 10 lines

Quoted 9 lines

In my humble opinion:

1) As "Intel sdm vol3 ch25.3" says, MWAIT operates normally (I think includes entering deeper sleep) under certain conditions.
Some deeper sleep modes(such as C4E/C6/C7) will clear the L1/L2/L3 cache. This is insecurity if we don't take other protective measures(such as limit the guest's max-cstate, it's fortunately that power subsystem isn't supported by QEMU, but we should be careful for some special-purpose in case). While HLT in guest mode can't cause hardware into sleep.

Quoted 3 lines

Good point. I'm not aware of any VMX capabilities to prevent deeper C-states, so we'd always hope that guests obey provided information.

I'll do some tests this weekend.
I plan to use MWAIT to enter deeper C-states in a testcase of kvm-unit-tests, and start a memory-sensitive workload on another hyper-thread, then use intel-pcm or perf to observe the count of cache miss on that core.

Quoted 13 lines

Quoted 6 lines

2) According to the "Intel sdm vol3 ch26.3.3 & ch27.5.6", I think MONITOR in guest mode can't work as perfect as in host sometimes.
For example, a vcpu MONITOR a address and then MWAIT, if a external-intr(suppose this intr won't cause to inject any virtual events ) cause VMEXIT, the monitor address will be cleaned, so the MWAIT won't be waken up by a store operation to the monitored address any more.

It's not as perfect, but should not cause a bug (well, there is a discussion with suspicious MWAIT behavior :]).
MWAIT on all Intels I tested would just behave as a nop if exit happened between MONITOR and MWAIT, like it does if you skip the MONITOR (MWAIT instruction desciption):

Quoted 6 lines

If the preceding MONITOR instruction did not successfully arm an address range or if the MONITOR instruction has not been executed prior to executing MWAIT, then the processor will not enter the implementation-dependent-optimized state. Execution will resume at the instruction following the MWAIT.

OK. :)

Quoted 9 lines

Quoted 5 lines

But I'm glad to do some tests if time permits, thanks :)

Radim, how about to make HLT-exiting configurable again in upstream ? If you like it, there is a problem should be resolved, asynpf is conflict with "HLT-exiting = 0" in certain situations.

Go ahead. KVM should provide access to hardware features and no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a good case) capability. I'm interested in the asyncpf conflict.

I had did some offline discussion with Wanpeng Li, he's interesting to write a path for this feature. :)

2017-03-21 9:47 GMT+08:00 Longpeng (Mike) <[email protected]>:

Quoted 9 lines · Radim Krčmář

Quoted 8 lines

2017-03-17 13:22+0800, Longpeng (Mike):

Quoted 3 lines

Radim, how about to make HLT-exiting configurable again in upstream ? If you like it, there is a problem should be resolved, asynpf is conflict with "HLT-exiting = 0" in certain situations.

Go ahead. KVM should provide access to hardware features and no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a good case) capability. I'm interested in the asyncpf conflict.

After async pf setup successfully, there is a broadcast wakeup w/ special token 0xffffffff which tells vCPU that it should wake up all processes waiting for APFs though there is no real process waiting at the moment.

Refer to SDM 26.3.1.5:

HLT. The only events allowed are the following:

— Those with interruption type external interrupt or non-maskable interrupt (NMI).
— Those with interruption type hardware exception and vector 1 (debug exception) or vector 18 (machine-check exception).
— Those with interruption type other event and vector 0 (pending MTF VM exit).

So if the guest activity state is hlt and delivery #PF event during vmentry, the vmentry will fail.

Refer to the original "KVM: VMX: add module parameter to avoid trapping HLT instructions"
https://www.spinics.net/lists/kvm-commits/msg00137.html, it will set guest activity state to active if it is hlt before manually. Actually I wonder who set guest activity state to active when there is HLT-exiting.

In addition, what's your design for per-VM non HLT-exiting capability?

2017-03-21 14:21+0800, Wanpeng Li:

Quoted 20 lines

2017-03-21 9:47 GMT+08:00 Longpeng (Mike) <[email protected]>:

Quoted 9 lines · Radim Krčmář

Quoted 8 lines

2017-03-17 13:22+0800, Longpeng (Mike):

Quoted 3 lines

Radim, how about to make HLT-exiting configurable again in upstream ? If you like it, there is a problem should be resolved, asynpf is conflict with "HLT-exiting = 0" in certain situations.

Go ahead. KVM should provide access to hardware features and no-HLT-exiting is reasonable as a per-VM (even per-VCPU if you make a good case) capability. I'm interested in the asyncpf conflict.

After async pf setup successfully, there is a broadcast wakeup w/ special token 0xffffffff which tells vCPU that it should wake up all processes waiting for APFs though there is no real process waiting at the moment.

Refer to SDM 26.3.1.5:

HLT. The only events allowed are the following:

Quoted 8 lines

— Those with interruption type external interrupt or non-maskable interrupt (NMI).
— Those with interruption type hardware exception and vector 1 (debug exception) or vector 18 (machine-check exception).
— Those with interruption type other event and vector 0 (pending MTF VM exit).

So if the guest activity state is hlt and delivery #PF event during vmentry, the vmentry will fail.

I see, so we also need to change the activity state to ACTIVE when setting most interrupts in VM_ENTRY_INTR_INFO_FIELD ... This adds overhead to the common case and gets a little tricky with cancel_injection(), but should turn out to be bearable.

No word about GUEST_INTR_STATUS, so posted interrupts hopefully work without additional changes.

Quoted 7 lines

Refer to the original "KVM: VMX: add module parameter to avoid trapping HLT instructions"
https://www.spinics.net/lists/kvm-commits/msg00137.html, it will set guest activity state to active if it is hlt before manually. Actually I wonder who set guest activity state to active when there is HLT-exiting.

Quoted 1 line

In addition, what's your design for per-VM non HLT-exiting capability?

I would go with an enableable per-VM KVM_CAP_X86_GUEST_HLT that can be used by userspace to set a variable in struct kvm.

Also, please take a look at SVM -- I think that errata #415 for family 0x10 forces us to introduce a vendor specific KVM_CHECK_EXTENSION and KVM_ENABLE_CAP, because HLT must be intercepted in that case.

I observed that the hang during the boot of guest since pvspinlock depends on trapping the halt as mentioned in this patch:
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order to disable the pvspinlock in guest from kvm. So if disable the pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and pCPU 1:1 pin can be acceptable?

2017-07-04 12:24+0800, Wanpeng Li:

Quoted 7 lines

I observed that the hang during the boot of guest since pvspinlock depends on trapping the halt as mentioned in this patch:
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order to disable the pvspinlock in guest from kvm. So if disable the pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and pCPU 1:1 pin can be acceptable?

I think the hang happened because one VCPU did a halt with disabled interrupts and then ignored KVM_HC_KICK_CPU from another VCPU. The target VCPU does a VM exit, so handling the PV unhalt should be a simple setting of GUEST_ACTIVITY_ACTIVE upon arch.pv.pv_unhalted.

If it turns out to be complicated, then forbidding the userspace to configure both is acceptable, (like you said, it currently doesn't make much sense to combine them)

2017-07-11 1:08 GMT+08:00 Radim Krčmář <[email protected]>:

Quoted 14 lines

2017-07-04 12:24+0800, Wanpeng Li:

Quoted 7 lines

I observed that the hang during the boot of guest since pvspinlock depends on trapping the halt as mentioned in this patch:
http://lkml.iu.edu/hypermail/linux/kernel/1202.0/03309.html，the hang disappears when I remove the KVM_FEATURE_PV_UNHALT CPUID flag in order to disable the pvspinlock in guest from kvm. So if disable the pvspinlock in the low latency scenario(idle=poll in guest) w/ vCPU and pCPU 1:1 pin can be acceptable?

I think the hang happened because one VCPU did a halt with disabled interrupts and then ignored KVM_HC_KICK_CPU from another VCPU. The target VCPU does a VM exit, so handling the PV unhalt should be a simple setting of GUEST_ACTIVITY_ACTIVE upon arch.pv.pv_unhalted.

Quoted 3 lines

If it turns out to be complicated, then forbidding the userspace to configure both is acceptable, (like you said, it currently doesn't make much sense to combine them)

kvm_spinlock_init() just check KVM_FEATURE_PV_UNHALT once to determine whether to enable pvspinlock, however, there is no easy way to check per-VM halt-exiting flag in the function __do_cpuid_ent().