lots of interrupts injected to vm when pressing some key w/o releasing

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.
And, linux guest has no this problem.

Quoted 7 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.
And, linux guest has no this problem.

Do you have a trace for this? What version of QEMU and what UI backend?

Paolo

Quoted 11 lines · Zhang Haoyu

Quoted 7 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.
And, linux guest has no this problem.

Do you have a trace for this? What version of QEMU and what UI backend?

Sorry for forgetting to mention test environment from the start.
Host: rhel7 with kernel-3.10.0-121
QEMU: qemu-2.0.2
Guest: win7(bad),win2008(bad),linux-kernel-3.10.0-121(good)

No UI backend, directly start the VM via qemu command.

perf top data when above problem happening:

Quoted 16 lines

Quoted 11 lines · Zhang Haoyu

Quoted 7 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.
And, linux guest has no this problem.

Do you have a trace for this? What version of QEMU and what UI backend?

Sorry for forgetting to mention test environment from the start.
Host: rhel7 with kernel-3.10.0-121
QEMU: qemu-2.0.2
Guest: win7(bad),win2008(bad),linux-kernel-3.10.0-121(good)

Quoted 3 lines

No UI backend, directly start the VM via qemu command.

perf top data when above problem happening:

Trace it like this: http://www.linux-kvm.org/page/Tracing

Zhang Haoyu wrote on 2014-11-20:

Quoted 8 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay. And, linux guest has no this problem.

Quoted 6 lines

If I remove the commit of 0bc830b05c667218d703f2026ec866c49df974fc, then the problem disappeared, but win7 guest got stuck at booting stage. And so strange that If the vm has only one vcpu, then the problem also disappeared.

Any ideas?

It looks commit 0bc830 doesn't do the right thing. The right point to clear an edge triggered interrupt in ioapic->irr is after userspace changes the irq line status. Otherwise, there may cause interrupt storm if a device sets the irq line in a fix edge continuously.

See below code:
ioapic_set_irq:

	.....
    old_irr = ioapic->irr;
    ioapic->irr |= mask;
    if ((edge && old_irr == ioapic->irr) ||
        (!edge && entry.fields.remote_irr)) { 
        ret = 0;						 // normally, we should break from here. But we never go to here due to (edge && old_irr != ioapic->irr) now.
        goto out;
    }   

Quoted 3 lines

The IRR register means an interrupt was received and not serviced yet, similar to the LAPIC or PIC register. It is not the same thing as the interrupt line level (it happens to be for level-triggered interrupts).

Yes, but commit(0bc830b05) changes the behavior: before it , ioapic->irr is cleared only when userspace lower the irq level. With it, it is cleared after the edge interrupt is serviced by ioapic. As you mentioned below, if QEMU tried to set a line twice, than there will be two interrupts for guest which only one interrupt before commit(0bc830b05).

According the feedback from Haoyu on my test patch which skipping the interrupt injection if irq line is active (See another thread), it seems QEMU does not follow the rule. But my patch is just a workaround. I guess we should have more though on this issue to find a better solution.

Quoted 5 lines · Yang Z

According the feedback from Haoyu on my test patch which skipping the interrupt injection if irq line is active (See another thread), it seems QEMU does not follow the rule. But my patch is just a workaround. I guess we should have more though on this issue to find a better solution.

Yes, I agree.

Paolo

Hi Paolo, Yang
What's the status of this problem?

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.

And, linux guest has no this problem.

If I remove the commit of 0bc830b05c667218d703f2026ec866c49df974fc, then the problem disappeared, but win7 guest got stuck at booting stage.
And so strange that If the vm has only one vcpu, then the problem also disappeared.

Any ideas?

Thanks,
Zhang Haoyu

Quoted 6 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.

Sorry for wrong description, the interrupt rate is normal, but huge numbers of vmexit induced by PIO were produced.

Quoted 7 lines

Quoted 6 lines

Hi all,

If I press the one of "Insert/Delete/Home/End/PageUp/PageDown/UpArrow/ DownArrow/LeftArrow/RightArrow" key w/o releasing, then lots of interrupts will be injected to vm(win7/win2008), about 8000/s, the system become very slow, bringing very bad experience. But the other keys are okay.

Quoted 2 lines

Sorry for wrong description, the interrupt rate is normal, but huge numbers of vmexit induced by PIO were produced.

This is expected when running Windows without paravirtualized time counter (-cpu ...,hv_time).

Paolo

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second,
bringing very bad experience, just like being stuck.

Quoted 3 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second,
bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

Paolo

Quoted 6 lines

Quoted 3 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second,
bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

# perf kvm stat report --event=vmexit
Warning:
Processed 4912765 events and lost 81 chunks!
Check IO/CPU overload!

Analyze events for all VCPUs:

VM-EXIT Samples Samples% Time% Avg time

   PAUSE_INSTRUCTION    1248620    60.74%    23.33%      0.83us ( +-   0.02% )
      IO_INSTRUCTION     799619    38.89%    74.49%      4.13us ( +-   6.93% )
  EXTERNAL_INTERRUPT       2629     0.13%     0.47%      7.93us ( +-   1.27% )
              VMCALL       1785     0.09%     0.04%      0.98us ( +-   0.30% )
         APIC_ACCESS       1495     0.07%     0.11%      3.38us ( +-   1.65% )
       EXCEPTION_NMI       1188     0.06%     0.03%      1.26us ( +-   1.66% )
 TPR_BELOW_THRESHOLD        273     0.01%     0.01%      1.57us ( +-   1.32% )
   PENDING_INTERRUPT        226     0.01%     0.01%      1.38us ( +-   2.59% )
                 HLT          6     0.00%     1.51%  11153.81us ( +-  15.24% )
       EPT_MISCONFIG          2     0.00%     0.00%     15.46us ( +-  23.93% )

Total Samples:2055843, Total events handled time:4438794.59us.

# perf kvm stat report --event=ioport
Warning:
Processed 4912765 events and lost 81 chunks!
Check IO/CPU overload!

Analyze events for all VCPUs:

      IO Port Access    Samples  Samples%     Time%         Avg time

            0x64:PIN     799032    99.93%    99.84%      3.42us ( +-   8.35% )
           0x3fd:PIN        300     0.04%     0.08%      6.88us ( +-   3.45% )
            0x60:PIN        274     0.03%     0.09%      8.53us ( +-   3.07% )
         0xb000:POUT          2     0.00%     0.00%      7.96us ( +-   3.16% )
          0xb000:PIN          2     0.00%     0.00%      8.16us ( +-   2.31% )
          0xb008:PIN          2     0.00%     0.00%      1.53us ( +-  13.66% )
          0xafe0:PIN          2     0.00%     0.00%      3.62us ( +-   0.84% )
          0xafe1:PIN          2     0.00%     0.00%      2.76us ( +-   2.99% )

Total Samples:799616, Total events handled time:2739038.53us.

# trace-cmd report
version = 6
CPU 4 is empty
CPU 5 is empty
CPU 6 is empty
CPU 7 is empty
CPU 9 is empty
CPU 11 is empty
CPU 12 is empty
CPU 13 is empty
cpus=16
             kvm-22783 [003]   721.255618: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255619: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255622: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22784 [014]   721.255623: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255623: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255623: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255623: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255625: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255626: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255626: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255627: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255628: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255628: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255628: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255629: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255630: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255631: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22784 [014]   721.255632: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22783 [003]   721.255632: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255632: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255632: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255634: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22784 [014]   721.255634: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255635: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255636: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255636: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255636: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255637: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255637: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255638: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255640: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255640: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255640: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255640: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255641: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255642: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22784 [014]   721.255643: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255643: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255645: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255645: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255645: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255645: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255646: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255647: kvm_entry:            vcpu 1
             kvm-22784 [014]   721.255649: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255649: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255649: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255649: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255650: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255651: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255652: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22783 [003]   721.255653: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255653: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255653: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22784 [014]   721.255654: kvm_userspace_exit:   reason KVM_EXIT_IO (2)
             kvm-22783 [003]   721.255655: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
             kvm-22784 [014]   721.255655: kvm_entry:            vcpu 1
             kvm-22783 [003]   721.255655: kvm_entry:            vcpu 0
             kvm-22784 [014]   721.255657: kvm_exit:             reason IO_INSTRUCTION rip 0xfffff88000cdd23b info 640008 0
             kvm-22784 [014]   721.255658: kvm_pio:              pio_read at 0x64 size 1 count 1
             kvm-22783 [003]   721.255658: kvm_exit:             reason PAUSE_INSTRUCTION rip 0xfffff80001688a9e info 0 0
...

And, if I remove the commit of 0bc830b0, the problem disappeared.

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second, bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

And, if I remove the commit of 0bc830b0, the problem disappeared.

Quoted 5 lines

Quoted 2 lines

Please send the full trace file. If you compress it, it should be small.

See the attach 1, please.

Quoted 1 line

Paolo

Can you try the follow draft patch to see whether it solve your problem? This patch is based on commit 0bc830b0.

virt/kvm/ioapic.c +8 -0

index 25e16a6..8f4e211 100644
@@ -305,6 +305,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
       return ret;
 }
 
+static int irq_status[256];
 int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                      int level, bool line_status)
 {

@@ -312,10 +313,13 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
       u32 mask = 1 << irq;
       union kvm_ioapic_redirect_entry entry;
       int ret, irq_level;
+       int old_irq;
 
       BUG_ON(irq < 0 || irq >= IOAPIC_NUM_PINS);
 
       spin_lock(&ioapic->lock);
+       old_irq = irq_status[irq];
+       irq_status[irq] = level;
       old_irr = ioapic->irr;
       irq_level = __kvm_irq_line_state(&ioapic->irq_states[irq],
                                        irq_source_id, level);

@@ -332,6 +336,10 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                       goto out;
               }
               ioapic->irr |= mask;
+               if (edge && old_irq == irq_status[irq] && old_irq) {
+                       ret = 0;
+                       goto out;
+               }
               if ((edge && old_irr != ioapic->irr) ||
                   (!edge && !entry.fields.remote_irr))
                       ret = ioapic_service(ioapic, irq, line_status);

Best regards,
Yang

Quoted 20 lines

Quoted 20 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 4 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second, bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

Quoted 7 lines

Quoted 4 lines

Quoted 1 line

And, if I remove the commit of 0bc830b0, the problem disappeared.

Please send the full trace file. If you compress it, it should be small.

See the attach 1, please.

Quoted 1 line

Paolo

Can you try the follow draft patch to see whether it solve your problem? This patch is based on commit 0bc830b0.

Quoted 1 line

After applying this patch, VM got stuck with black-screen at boot

It is works well in my side. And this patch does fix the issue in my box.
I am using window 2008 R2 for testing, here is the qemu cmdline:
qemu-system-x86_64 -enable-kvm -m 4G -smp 4 -net nic,macaddr=00:12:41:41:13:41 -net tap,script=/etc/kvm/qemu-ifup win2k8.qcow

Can you provide your configuration for me to test?

Quoted 51 lines

stage, # trace-cmd report:
version = 6
CPU 0 is empty
CPU 1 is empty
CPU 2 is empty
CPU 3 is empty
CPU 5 is empty
CPU 7 is empty
cpus=8

             kvm-1266  [004] 14399.834397: kvm_set_irq:          gsi 9
             level 1 source 0 kvm-1266  [004] 14399.834403:
             kvm_pic_set_irq:      chip 1 pin 1 (edge|masked) kvm-1266 
             [004] 14399.834411: kvm_apic_accept_irq: apicid 0 vec 177
             (LowPrio|level) kvm-1266  [004] 14399.834412:
             kvm_ioapic_set_irq:   pin 9 dst 3 vec=177
             (LowPrio|logical|level) kvm-1266  [004] 14402.180013:
             kvm_set_irq:          gsi 9 level 1 source 0 kvm-1266 
             [004] 14402.180019: kvm_pic_set_irq:      chip 1 pin 1
             (edge|masked) kvm-1266  [004] 14402.180028:
             kvm_apic_accept_irq: apicid 1 vec 177 (LowPrio|level)
             kvm-1266  [004] 14402.180029: kvm_ioapic_set_irq:   pin 9
             dst 3 vec=177 (LowPrio|logical|level) kvm-1266  [004]
             14404.525627: kvm_set_irq:          gsi 9 level 1 source 0
             kvm-1266  [004] 14404.525634: kvm_pic_set_irq:      chip 1
             pin 1 (edge|masked) kvm-1266  [004] 14404.525641:
             kvm_apic_accept_irq: apicid 0 vec 177 (LowPrio|level)
             kvm-1266  [004] 14404.525642: kvm_ioapic_set_irq:   pin 9
             dst 3 vec=177 (LowPrio|logical|level) kvm-1266  [004]
             14406.871238: kvm_set_irq:          gsi 9 level 1 source 0
             kvm-1266  [004] 14406.871245: kvm_pic_set_irq:      chip 1
             pin 1 (edge|masked) kvm-1266  [004] 14406.871254:
             kvm_apic_accept_irq: apicid 1 vec 177 (LowPrio|level)
             kvm-1266  [004] 14406.871256: kvm_ioapic_set_irq:   pin 9
             dst 3 vec=177 (LowPrio|logical|level) kvm-1266  [006]
             14409.216849: kvm_set_irq:          gsi 9 level 1 source 0
             kvm-1266  [006] 14409.216855: kvm_pic_set_irq:      chip 1
             pin 1 (edge|masked) kvm-1266  [006] 14409.216862:
             kvm_apic_accept_irq: apicid 0 vec 177 (LowPrio|level)
             kvm-1266  [006] 14409.216863: kvm_ioapic_set_irq:   pin 9
             dst 3 vec=177 (LowPrio|logical|level) kvm-1266  [006]
             14411.562475: kvm_set_irq:          gsi 9 level 1 source 0
             kvm-1266  [006] 14411.562481: kvm_pic_set_irq:      chip 1
             pin 1 (edge|masked) kvm-1266  [006] 14411.562489:
             kvm_apic_accept_irq: apicid 1 vec 177 (LowPrio|level)
             kvm-1266  [006] 14411.562491: kvm_ioapic_set_irq:   pin 9
             dst 3 vec=177 (LowPrio|logical|level) kvm-1266  [004]
             14413.908074: kvm_set_irq:          gsi 9 level 1 source 0
             kvm-1266  [004] 14413.908080: kvm_pic_set_irq:      chip 1
             pin 1 (edge|masked) kvm-1266  [004] 14413.908088:
             kvm_apic_accept_irq: apicid 0 vec 177 (LowPrio|level)
             kvm-1266  [004] 14413.908089: kvm_ioapic_set_irq:   pin 9

Quoted 5 lines

dst 3 vec=177 (LowPrio|logical|level)

Thanks,
Zhang Haoyu

Best regards,
Yang

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second, bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

And, if I remove the commit of 0bc830b0, the problem disappeared.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

Please send the full trace file. If you compress it, it should be small.

See the attach 1, please.

Quoted 1 line

Paolo

Can you try the follow draft patch to see whether it solve your problem? This patch is based on commit 0bc830b0.

After applying this patch, VM got stuck with black-screen at boot stage,

# trace-cmd report:
version = 6
CPU 0 is empty
CPU 1 is empty
CPU 2 is empty
CPU 3 is empty
CPU 5 is empty
CPU 7 is empty
cpus=8
            kvm-1266  [004] 14399.834397: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14399.834403: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14399.834411: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14399.834412: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14402.180013: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14402.180019: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14402.180028: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [004] 14402.180029: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14404.525627: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14404.525634: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14404.525641: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14404.525642: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14406.871238: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14406.871245: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14406.871254: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [004] 14406.871256: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [006] 14409.216849: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [006] 14409.216855: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [006] 14409.216862: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [006] 14409.216863: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [006] 14411.562475: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [006] 14411.562481: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [006] 14411.562489: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [006] 14411.562491: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14413.908074: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14413.908080: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14413.908088: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14413.908089: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)

Thanks,
Zhang Haoyu

Quoted 8 lines

virt/kvm/ioapic.c +0 -0

index 25e16a6..8f4e211 100644
@@ -305,6 +305,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
       return ret;
 }
 

virt/kvm/ioapic.c int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id, · line 317

Quoted 10 lines

virt/kvm/ioapic.c +2 -0

+static int irq_status[256];
 int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                      int level, bool line_status)
 {

@@ -312,10 +313,13 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
       u32 mask = 1 << irq;
       union kvm_ioapic_redirect_entry entry;
       int ret, irq_level;
+       int old_irq;
 

Quoted 20 lines

virt/kvm/ioapic.c +6 -0

       BUG_ON(irq < 0 || irq >= IOAPIC_NUM_PINS);
 
       spin_lock(&ioapic->lock);
+       old_irq = irq_status[irq];
+       irq_status[irq] = level;
       old_irr = ioapic->irr;
       irq_level = __kvm_irq_line_state(&ioapic->irq_states[irq],
                                        irq_source_id, level);

@@ -332,6 +336,10 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                       goto out;
               }
               ioapic->irr |= mask;
+               if (edge && old_irq == irq_status[irq] && old_irq) {
+                       ret = 0;
+                       goto out;
+               }
               if ((edge && old_irr != ioapic->irr) ||
                   (!edge && !entry.fields.remote_irr))
                       ret = ioapic_service(ioapic, irq, line_status);

Quoted 2 lines

Best regards,
Yang

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi all,

Quoted 77 lines · Zhang Haoyu

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

I tested win-server-2008 with "-cpu core2duo,hv_spinlocks=0xffff,hv_relaxed,hv_time", this problem still happened, about 200,000 vmexits per-second, bringing very bad experience, just like being stuck.

Please upload a full trace somewhere, or at least the "perf report" output.

And, if I remove the commit of 0bc830b0, the problem disappeared.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

Please send the full trace file. If you compress it, it should be small.

See the attach 1, please.

Quoted 1 line

Paolo

Can you try the follow draft patch to see whether it solve your problem? This patch is based on commit 0bc830b0.

After applying this patch, VM got stuck with black-screen at boot stage,

# trace-cmd report:
version = 6
CPU 0 is empty
CPU 1 is empty
CPU 2 is empty
CPU 3 is empty
CPU 5 is empty
CPU 7 is empty
cpus=8
            kvm-1266  [004] 14399.834397: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14399.834403: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14399.834411: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14399.834412: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14402.180013: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14402.180019: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14402.180028: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [004] 14402.180029: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14404.525627: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14404.525634: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14404.525641: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14404.525642: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14406.871238: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14406.871245: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14406.871254: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [004] 14406.871256: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [006] 14409.216849: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [006] 14409.216855: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [006] 14409.216862: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [006] 14409.216863: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [006] 14411.562475: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [006] 14411.562481: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [006] 14411.562489: kvm_apic_accept_irq:  apicid 1 vec 177 (LowPrio|level)
            kvm-1266  [006] 14411.562491: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)
            kvm-1266  [004] 14413.908074: kvm_set_irq:          gsi 9 level 1 source 0
            kvm-1266  [004] 14413.908080: kvm_pic_set_irq:      chip 1 pin 1 (edge|masked)
            kvm-1266  [004] 14413.908088: kvm_apic_accept_irq:  apicid 0 vec 177 (LowPrio|level)
            kvm-1266  [004] 14413.908089: kvm_ioapic_set_irq:   pin 9 dst 3 vec=177 (LowPrio|logical|level)

Thanks,
Zhang Haoyu

Quoted 17 lines

virt/kvm/ioapic.c +2 -0

index 25e16a6..8f4e211 100644
@@ -305,6 +305,7 @@ static int ioapic_service(struct kvm_ioapic *ioapic, int irq, bool line_status)
       return ret;
 }
 
+static int irq_status[256];
 int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                      int level, bool line_status)
 {

@@ -312,10 +313,13 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
       u32 mask = 1 << irq;
       union kvm_ioapic_redirect_entry entry;
       int ret, irq_level;
+       int old_irq;

Quoted 1 line

Quoted 1 line

I'm not sure which version of kvm the patch is against, anyway, all the modification should be moved to ioapic_set_irq() against the upstream kvm, I test the patch w/ win2k8 guest and w/o the Haoyu's comand line and it can fix the bug mentioned by Haoyu, in addition, press any key can reproduce the bug on my side instead of the small set which Haoyu mentioned.

Regards,
Wanpeng Li

Quoted 22 lines

Quoted 22 lines

       BUG_ON(irq < 0 || irq >= IOAPIC_NUM_PINS);

       spin_lock(&ioapic->lock);

virt/kvm/ioapic.c +6 -0

+       old_irq = irq_status[irq];
+       irq_status[irq] = level;
       old_irr = ioapic->irr;
       irq_level = __kvm_irq_line_state(&ioapic->irq_states[irq],
                                        irq_source_id, level);

@@ -332,6 +336,10 @@ int kvm_ioapic_set_irq(struct kvm_ioapic *ioapic, int irq, int irq_source_id,
                       goto out;
               }
               ioapic->irr |= mask;
+               if (edge && old_irq == irq_status[irq] && old_irq) {
+                       ret = 0;
+                       goto out;
+               }
               if ((edge && old_irr != ioapic->irr) ||
                   (!edge && !entry.fields.remote_irr))
                       ret = ioapic_service(ioapic, irq, line_status);

Best regards,
Yang