Hang (due to HW?) in qi_submit_sync()
From: Roland Dreier <[email protected]>
Hi, we're running kernel 3.10.59 (pretty recent long-term kernel) on a 2-socket Xeon E5 v3 (Haswell) system. We're using vfio to access some PCI devices from userspace, and occasionally when we kill a process, we see the system hang in qi_submit_sync().
Based on a very old patch from Intel <https://lkml.org/lkml/2009/5/20/341>,
we added code to the dmar driver:
int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu ) {
//...
/*
* update the HW tail register indicating the presence of
* new descriptors.
*/
writel(qi->free_head << DMAR_IQ_SHIFT, iommu->reg + DMAR_IQT_REG);
start_time = get_cycles();
while (qi->desc_status[wait_index] != QI_DONE) {
/*
* We will leave the interrupts disabled, to prevent interrupt
* context to queue another cmd while a cmd is already submitted
* and waiting for completion on this cpu. This is to avoid
* a deadlock where the interrupt context can wait indefinitely
* for free slots in the queue.
*/
rc = qi_check_fault(iommu, index);
if (rc)
break;
raw_spin_unlock(&qi->q_lock);// We added this -->
if (get_cycles() - start_time > DMAR_OPERATION_TIMEOUT) {
printk(KERN_EMERG "desc_status[%d] = %d.\n",
wait_index, qi->desc_status[wait_index]);
/* line 888: */ BUG();
}// <-- to here
cpu_relax(); raw_spin_lock(&qi->q_lock); }
and indeed when the system hangs, we see for example
desc_status[69] = 1.
------------[ cut here ]------------
kernel BUG at drivers/iommu/dmar.c:888!
CPU: 8 PID: 12211 Comm: foed Tainted: P O 3.10.59+ #201412290537+4e4984e.platinum
task: ffff88275ac643e0 ti: ffff8825d329a000 task.ti: ffff8825d329a000
RIP: 0010:[<ffffffff81529737>] [<ffffffff81529737>] qi_submit_sync+0x3f7/0x490
RSP: 0018:ffff8825d329ba10 EFLAGS: 00010092
RAX: 0000000000000014 RBX: 0000000000000044 RCX: ffff881fffb0ec00
RDX: 0000000000000000 RSI: ffff881fffb0d048 RDI: 0000000000000046
RBP: ffff8825d329ba78 R08: ffffffffffffffff R09: 000000000001a4a1
R10: 0000000000000051 R11: 00000000000000e4 R12: 00007068faa64fc8
R13: ffff881fff40c780 R14: 0000000000000114 R15: ffff883ffec01a00
FS: 00007f3c86ffb700(0000) GS:ffff881fffb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f996d3f1ba0 CR3: 00000026222f0000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8825d329ba88 0000000000000450 0000000000000440 ffff881ff3215000
00000044d329bb18 0000000000000086 0000000000000044 ffff882500000045
ffff881ff12b1600 0000000000000000 0000000000000246 ffff881ff278e858
Call Trace:
[<ffffffff8152f6b5>] free_irte+0xc5/0x100
[<ffffffff81530834>] free_remapped_irq+0x44/0x60
[<ffffffff81027b23>] destroy_irq+0x33/0xd0
[<ffffffff81027ede>] native_teardown_msi_irq+0xe/0x10
[<ffffffff812a6a70>] default_teardown_msi_irqs+0x60/0x80
[<ffffffff812a64d9>] free_msi_irqs+0x99/0x150
[<ffffffff812a749d>] pci_disable_msix+0x3d/0x60
[<ffffffffa0078748>] vfio_msi_disable+0xc8/0xe0 [vfio_pci]
[<ffffffffa0078f86>] vfio_pci_set_msi_trigger+0x2a6/0x2d0 [vfio_pci]
[<ffffffffa007941c>] vfio_pci_set_irqs_ioctl+0x8c/0xa0 [vfio_pci]
[<ffffffffa00773b0>] vfio_pci_release+0x70/0x150 [vfio_pci]
[<ffffffffa006dcbc>] vfio_device_fops_release+0x1c/0x40 [vfio]
[<ffffffff8114d7db>] __fput+0xdb/0x220
[<ffffffff8114d92e>] ____fput+0xe/0x10
[<ffffffff810614ac>] task_work_run+0xbc/0xe0
[<ffffffff81043d0e>] do_exit+0x3ce/0xe50
[<ffffffff8104557f>] do_group_exit+0x3f/0xa0
[<ffffffff81054769>] get_signal_to_deliver+0x1a9/0x5b0
[<ffffffff810023f8>] do_signal+0x48/0x5e0as far as I can understand the driver, this is a "shouldn't happen, your hardware is broken" occurrence. However I haven't been able to find any relevant looking sightings for our CPU.
Does anyone from Intel (or elsewhere) have any suggestions on how to chase this further?
Thanks!
Roland
A message that is not in the archive (sent off-list or not yet received): 1420505840-30096-1-git-send-email-roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Quoted 92 lines · Roland Dreier
From: Roland Dreier <[email protected]>
Hi, we're running kernel 3.10.59 (pretty recent long-term kernel) on a 2-socket Xeon E5 v3 (Haswell) system. We're using vfio to access some PCI devices from userspace, and occasionally when we kill a process, we see the system hang in
qi_submit_sync().Based on a very old patch from Intel <https://lkml.org/lkml/2009/5/20/341>,
we added code to the dmar driver:int
qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu ){//...
/* * update the HW tail register indicating the presence of * new descriptors. */ writel(qi->free_head << DMAR_IQ_SHIFT, iommu->reg + DMAR_IQT_REG); start_time = get_cycles(); while (qi->desc_status[wait_index] != QI_DONE) { /* * We will leave the interrupts disabled, to prevent interrupt * context to queue another cmd while a cmd is already submitted * and waiting for completion on this cpu. This is to avoid * a deadlock where the interrupt context can wait indefinitely * for free slots in the queue. */ rc = qi_check_fault(iommu, index); if (rc) break; raw_spin_unlock(&qi->q_lock);// We added this -->
if (get_cycles() - start_time > DMAR_OPERATION_TIMEOUT) { printk(KERN_EMERG "desc_status[%d] = %d.\n", wait_index, qi->desc_status[wait_index]); /* line 888: */ BUG(); }// <-- to here
cpu_relax(); raw_spin_lock(&qi->q_lock); }and indeed when the system hangs, we see for example
desc_status[69] = 1. ------------[ cut here ]------------ kernel BUG at drivers/iommu/dmar.c:888! CPU: 8 PID: 12211 Comm: foed Tainted: P O 3.10.59+ #201412290537+4e4984e.platinum task: ffff88275ac643e0 ti: ffff8825d329a000 task.ti: ffff8825d329a000 RIP: 0010:[<ffffffff81529737>] [<ffffffff81529737>] qi_submit_sync+0x3f7/0x490 RSP: 0018:ffff8825d329ba10 EFLAGS: 00010092 RAX: 0000000000000014 RBX: 0000000000000044 RCX: ffff881fffb0ec00 RDX: 0000000000000000 RSI: ffff881fffb0d048 RDI: 0000000000000046 RBP: ffff8825d329ba78 R08: ffffffffffffffff R09: 000000000001a4a1 R10: 0000000000000051 R11: 00000000000000e4 R12: 00007068faa64fc8 R13: ffff881fff40c780 R14: 0000000000000114 R15: ffff883ffec01a00 FS: 00007f3c86ffb700(0000) GS:ffff881fffb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f996d3f1ba0 CR3: 00000026222f0000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff8825d329ba88 0000000000000450 0000000000000440 ffff881ff3215000 00000044d329bb18 0000000000000086 0000000000000044 ffff882500000045 ffff881ff12b1600 0000000000000000 0000000000000246 ffff881ff278e858 Call Trace: [<ffffffff8152f6b5>] free_irte+0xc5/0x100 [<ffffffff81530834>] free_remapped_irq+0x44/0x60 [<ffffffff81027b23>] destroy_irq+0x33/0xd0 [<ffffffff81027ede>] native_teardown_msi_irq+0xe/0x10 [<ffffffff812a6a70>] default_teardown_msi_irqs+0x60/0x80 [<ffffffff812a64d9>] free_msi_irqs+0x99/0x150 [<ffffffff812a749d>] pci_disable_msix+0x3d/0x60 [<ffffffffa0078748>] vfio_msi_disable+0xc8/0xe0 [vfio_pci] [<ffffffffa0078f86>] vfio_pci_set_msi_trigger+0x2a6/0x2d0 [vfio_pci] [<ffffffffa007941c>] vfio_pci_set_irqs_ioctl+0x8c/0xa0 [vfio_pci] [<ffffffffa00773b0>] vfio_pci_release+0x70/0x150 [vfio_pci] [<ffffffffa006dcbc>] vfio_device_fops_release+0x1c/0x40 [vfio] [<ffffffff8114d7db>] __fput+0xdb/0x220 [<ffffffff8114d92e>] ____fput+0xe/0x10 [<ffffffff810614ac>] task_work_run+0xbc/0xe0 [<ffffffff81043d0e>] do_exit+0x3ce/0xe50 [<ffffffff8104557f>] do_group_exit+0x3f/0xa0 [<ffffffff81054769>] get_signal_to_deliver+0x1a9/0x5b0 [<ffffffff810023f8>] do_signal+0x48/0x5e0
Quoted 6 lines
as far as I can understand the driver, this is a "shouldn't happen, your hardware is broken" occurrence. However I haven't been able to find any relevant looking sightings for our CPU.
Does anyone from Intel (or elsewhere) have any suggestions on how to chase this further?
Try disabling CONFIG_NET_DMA
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Alex Williamson
Try disabling
CONFIG_NET_DMA
We already have that disabled (well, in 3.10 it depends on BROKEN, and we don't have BROKEN enabled :).
However I'm curious why you suggest that. Because some of the devices we're accessing via vfio are in fact Intel DMA engines (we blacklist ioatdma and use the DMA devices directly from userspace). Is there some known interaction between the Intel DMA engines and interrupt remapping?
Thanks!
Roland
A message that is not in the archive (sent off-list or not yet received): CAG4TOxPOgTOKYZa7q9Of8XzHV_wAadtJmXFC0bmyN2Qds7T9RA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 6 lines · Roland Dreier
Quoted 1 line · Alex Williamson
Try disabling
CONFIG_NET_DMAWe already have that disabled (well, in 3.10 it depends on BROKEN, and we don't have BROKEN enabled :).
Only since v3.10.26 is it marked BROKEN.
Quoted 5 lines
However I'm curious why you suggest that. Because some of the devices we're accessing via vfio are in fact Intel DMA engines (we blacklist ioatdma and use the DMA devices directly from userspace). Is there some known interaction between the Intel DMA engines and interrupt remapping?
I suggest it because I spent several weeks isolating why we hit the same lockup in qi_submit_sync() as you're seeing after we backported a number of iommu patches. In our case, finding that it's toggled by CONFIG_NET_DMA, which has since been marked broken and removed is a sufficient solution. I agree though that regardless of what terrible things NET_DMA was doing, we seem to be hitting a "broken hardware" condition, potentially invoked by the DMA engine.
What I observed was that it occurs when flushing an irte entry, the queued invalidation queue is working prior to this flush, but the wait descriptor value is never written to the status address, the queue head never advances past that wait descriptor once it gets wedged, and the status register never indicates any sort of error. Section 6.5.6 of the VT-d spec on interrupt draining has an interesting statement:
Interrupt draining is performed on Interrupt Entry Cache (IEC) invalidation requests. For IEC invalidations submitted through the queued invalidation interface, interrupt draining must be completed before the next Invalidation Wait Descriptor is completed by hardware.
Given the circumstances of the hang, that certainly makes me suspect that queued invalidation is failing to complete interrupt draining and hardware is therefore unable to advance past the subsequent wait descriptor or complete the wait descriptor because of this requirement. I wouldn't be surprised if there's an errata hidden in there somewhere. Thanks,
Alex
PS - Thanks for using vfio :)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
So we've managed to find one magic setup and workload that reproduces this reliably, and I've done a bit more debugging that leads me to believe we probably need Intel's help to really get to the bottom of this.
One question though: you mentioned that you saw this behavior until you turned off CONFIG_NET_DMA. What platform was that on? We see this on dual socket Xeon E5 v3 (Haswell EP / Grantley), and I don't really have any other setup I can try. Did you see this on other platforms (Ivy Bridge / Romley maybe)?
Anyway, I added the debugging patch at the end of this mail to our kernel to dump some status when the driver detects a hung queue. Below we see some example output. Things I notice:
- IQH == IQT, in other words the QI hardware thinks it has picked up and validated all the descriptors submitted by software.
- The queue has successfully processed many operations (although everything that we can see succeeded is type 2h, ie "IOTLB invalidate" as opposed to the type 4h "Interrupt Entry Cache invalidate" that we're stuck on.
- There haven't been any faults and FSTS is clear.
- The hardware hasn't executed the "Invalidation wait" descriptor.
Beyond that I'm not sure how to make much more progress without insight into the hardware -- it looks like the driver is doing everything right. Anyone from Intel have any thoughts?
Here's the actual output from the kernel log:
dmar: QI stuck waiting for index 252 / wait_index 253 IQH/T 00000fe0/00000fe0 FSTS 00000000
dmar: [ 0] stat 0 00000000000100f2 000001500027f000 dmar: [ 1] stat 0 0000000200000025 0000001fff412004 dmar: [ 2] stat 0 00000000000100f2 0000015000280000 dmar: [ 3] stat 0 0000000200000025 0000001fff41200c dmar: [ 4] stat 0 00000000000100f2 0000015000281000 dmar: [ 5] stat 0 0000000200000025 0000001fff412014 dmar: [ 6] stat 0 00000000000100f2 0000015000282000 dmar: [ 7] stat 0 0000000200000025 0000001fff41201c dmar: [ 8] stat 0 00000000000100f2 0000015000283000 dmar: [ 9] stat 0 0000000200000025 0000001fff412024 dmar: [ 10] stat 0 00000000000100f2 0000015000284000 dmar: [ 11] stat 0 0000000200000025 0000001fff41202c dmar: [ 12] stat 0 00000000000100f2 0000015000285000 dmar: [ 13] stat 0 0000000200000025 0000001fff412034 dmar: [ 14] stat 0 00000000000100f2 0000015000286000 dmar: [ 15] stat 0 0000000200000025 0000001fff41203c dmar: [ 16] stat 0 00000000000100f2 0000015000287000 dmar: [ 17] stat 0 0000000200000025 0000001fff412044 dmar: [ 18] stat 0 00000000000100f2 0000015000288000 dmar: [ 19] stat 0 0000000200000025 0000001fff41204c dmar: [ 20] stat 0 00000000000100f2 0000015000289000 dmar: [ 21] stat 0 0000000200000025 0000001fff412054 dmar: [ 22] stat 0 00000000000100f2 000001500028a000 dmar: [ 23] stat 0 0000000200000025 0000001fff41205c dmar: [ 24] stat 0 00000000000100f2 000001500026a000 dmar: [ 25] stat 0 0000000200000025 0000001fff412064 dmar: [ 26] stat 0 00000000000100f2 000001500026b000 dmar: [ 27] stat 0 0000000200000025 0000001fff41206c dmar: [ 28] stat 0 00000000000100f2 000001500026c000 dmar: [ 29] stat 0 0000000200000025 0000001fff412074 dmar: [ 30] stat 0 00000000000100f2 000001500026d000 dmar: [ 31] stat 0 0000000200000025 0000001fff41207c dmar: [ 32] stat 0 00000000000100f2 000001500026e000 dmar: [ 33] stat 0 0000000200000025 0000001fff412084 dmar: [ 34] stat 0 00000000000100f2 000001500026f000 dmar: [ 35] stat 0 0000000200000025 0000001fff41208c dmar: [ 36] stat 0 00000000000100f2 0000015000270000 dmar: [ 37] stat 0 0000000200000025 0000001fff412094 dmar: [ 38] stat 0 00000000000100f2 0000015000271000 dmar: [ 39] stat 0 0000000200000025 0000001fff41209c dmar: [ 40] stat 0 00000000000100f2 0000015000272000 dmar: [ 41] stat 0 0000000200000025 0000001fff4120a4 dmar: [ 42] stat 0 00000000000100f2 0000015000273000 dmar: [ 43] stat 0 0000000200000025 0000001fff4120ac dmar: [ 44] stat 0 00000000000100f2 0000015000274000 dmar: [ 45] stat 0 0000000200000025 0000001fff4120b4 dmar: [ 46] stat 0 00000000000100f2 0000015000275000 dmar: [ 47] stat 0 0000000200000025 0000001fff4120bc dmar: [ 48] stat 0 00000000000100f2 0000015000276000 dmar: [ 49] stat 0 0000000200000025 0000001fff4120c4 dmar: [ 50] stat 0 00000000000100f2 0000015000277000 dmar: [ 51] stat 0 0000000200000025 0000001fff4120cc dmar: [ 52] stat 0 00000000000100f2 0000015000278000 dmar: [ 53] stat 0 0000000200000025 0000001fff4120d4 dmar: [ 54] stat 0 00000000000100f2 0000015000279000 dmar: [ 55] stat 0 0000000200000025 0000001fff4120dc dmar: [ 56] stat 0 00000000000100f2 0000015000259000 dmar: [ 57] stat 0 0000000200000025 0000001fff4120e4 dmar: [ 58] stat 0 00000000000100f2 000001500025a000 dmar: [ 59] stat 0 0000000200000025 0000001fff4120ec dmar: [ 60] stat 0 00000000000100f2 000001500025b000 dmar: [ 61] stat 0 0000000200000025 0000001fff4120f4 dmar: [ 62] stat 0 00000000000100f2 000001500025c000 dmar: [ 63] stat 0 0000000200000025 0000001fff4120fc dmar: [ 64] stat 0 00000000000100f2 000001500025d000 dmar: [ 65] stat 0 0000000200000025 0000001fff412104 dmar: [ 66] stat 0 00000000000100f2 000001500025e000 dmar: [ 67] stat 0 0000000200000025 0000001fff41210c dmar: [ 68] stat 0 00000000000100f2 000001500025f000 dmar: [ 69] stat 0 0000000200000025 0000001fff412114 dmar: [ 70] stat 0 00000000000100f2 0000015000260000 dmar: [ 71] stat 0 0000000200000025 0000001fff41211c dmar: [ 72] stat 0 00000000000100f2 0000015000261000 dmar: [ 73] stat 0 0000000200000025 0000001fff412124 dmar: [ 74] stat 0 00000000000100f2 0000015000262000 dmar: [ 75] stat 0 0000000200000025 0000001fff41212c dmar: [ 76] stat 0 00000000000100f2 0000015000263000 dmar: [ 77] stat 0 0000000200000025 0000001fff412134 dmar: [ 78] stat 0 00000000000100f2 0000015000264000 dmar: [ 79] stat 0 0000000200000025 0000001fff41213c dmar: [ 80] stat 0 00000000000100f2 0000015000265000 dmar: [ 81] stat 0 0000000200000025 0000001fff412144 dmar: [ 82] stat 0 00000000000100f2 0000015000266000 dmar: [ 83] stat 0 0000000200000025 0000001fff41214c dmar: [ 84] stat 0 00000000000100f2 0000015000267000 dmar: [ 85] stat 0 0000000200000025 0000001fff412154 dmar: [ 86] stat 0 00000000000100f2 0000015000268000 dmar: [ 87] stat 0 0000000200000025 0000001fff41215c dmar: [ 88] stat 0 00000000000100f2 0000015000248000 dmar: [ 89] stat 0 0000000200000025 0000001fff412164 dmar: [ 90] stat 0 00000000000100f2 0000015000249000 dmar: [ 91] stat 0 0000000200000025 0000001fff41216c dmar: [ 92] stat 0 00000000000100f2 000001500024a000 dmar: [ 93] stat 0 0000000200000025 0000001fff412174 dmar: [ 94] stat 0 00000000000100f2 000001500024b000 dmar: [ 95] stat 0 0000000200000025 0000001fff41217c dmar: [ 96] stat 0 00000000000100f2 000001500024c000 dmar: [ 97] stat 0 0000000200000025 0000001fff412184 dmar: [ 98] stat 0 00000000000100f2 000001500024d000 dmar: [ 99] stat 0 0000000200000025 0000001fff41218c dmar: [100] stat 0 00000000000100f2 000001500024e000 dmar: [101] stat 0 0000000200000025 0000001fff412194 dmar: [102] stat 0 00000000000100f2 000001500024f000 dmar: [103] stat 0 0000000200000025 0000001fff41219c dmar: [104] stat 0 00000000000100f2 0000015000250000 dmar: [105] stat 0 0000000200000025 0000001fff4121a4 dmar: [106] stat 0 00000000000100f2 0000015000251000 dmar: [107] stat 0 0000000200000025 0000001fff4121ac dmar: [108] stat 0 00000000000100f2 0000015000252000 dmar: [109] stat 0 0000000200000025 0000001fff4121b4 dmar: [110] stat 0 00000000000100f2 0000015000253000 dmar: [111] stat 0 0000000200000025 0000001fff4121bc dmar: [112] stat 0 00000000000100f2 0000015000254000 dmar: [113] stat 0 0000000200000025 0000001fff4121c4 dmar: [114] stat 0 00000000000100f2 0000015000255000 dmar: [115] stat 0 0000000200000025 0000001fff4121cc dmar: [116] stat 0 00000000000100f2 0000015000256000 dmar: [117] stat 0 0000000200000025 0000001fff4121d4 dmar: [118] stat 0 00000000000100f2 0000015000257000 dmar: [119] stat 0 0000000200000025 0000001fff4121dc dmar: [120] stat 0 00000000000100f2 0000015000237000 dmar: [121] stat 0 0000000200000025 0000001fff4121e4 dmar: [122] stat 0 00000000000100f2 0000015000238000 dmar: [123] stat 0 0000000200000025 0000001fff4121ec dmar: [124] stat 0 00000000000100f2 0000015000239000 dmar: [125] stat 0 0000000200000025 0000001fff4121f4 dmar: [126] stat 0 00000000000100f2 000001500023a000 dmar: [127] stat 0 0000000200000025 0000001fff4121fc dmar: [128] stat 0 00000000000100f2 000001500023b000 dmar: [129] stat 0 0000000200000025 0000001fff412204 dmar: [130] stat 0 00000000000100f2 000001500023c000 dmar: [131] stat 0 0000000200000025 0000001fff41220c dmar: [132] stat 0 00000000000100f2 000001500023d000 dmar: [133] stat 0 0000000200000025 0000001fff412214 dmar: [134] stat 0 00000000000100f2 000001500023e000 dmar: [135] stat 0 0000000200000025 0000001fff41221c dmar: [136] stat 0 00000000000100f2 000001500023f000 dmar: [137] stat 0 0000000200000025 0000001fff412224 dmar: [138] stat 0 00000000000100f2 0000015000240000 dmar: [139] stat 0 0000000200000025 0000001fff41222c dmar: [140] stat 0 00000000000100f2 0000015000241000 dmar: [141] stat 0 0000000200000025 0000001fff412234 dmar: [142] stat 0 00000000000100f2 0000015000242000 dmar: [143] stat 0 0000000200000025 0000001fff41223c dmar: [144] stat 0 00000000000100f2 0000015000243000 dmar: [145] stat 0 0000000200000025 0000001fff412244 dmar: [146] stat 0 00000000000100f2 0000015000244000 dmar: [147] stat 0 0000000200000025 0000001fff41224c dmar: [148] stat 0 00000000000100f2 0000015000245000 dmar: [149] stat 0 0000000200000025 0000001fff412254 dmar: [150] stat 0 00000000000100f2 0000015000246000 dmar: [151] stat 0 0000000200000025 0000001fff41225c dmar: [152] stat 0 00000000000100f2 0000015000226000 dmar: [153] stat 0 0000000200000025 0000001fff412264 dmar: [154] stat 0 00000000000100f2 0000015000227000 dmar: [155] stat 0 0000000200000025 0000001fff41226c dmar: [156] stat 0 00000000000100f2 0000015000228000 dmar: [157] stat 0 0000000200000025 0000001fff412274 dmar: [158] stat 0 00000000000100f2 0000015000229000 dmar: [159] stat 0 0000000200000025 0000001fff41227c dmar: [160] stat 0 00000000000100f2 000001500022a000 dmar: [161] stat 0 0000000200000025 0000001fff412284 dmar: [162] stat 0 00000000000100f2 000001500022b000 dmar: [163] stat 0 0000000200000025 0000001fff41228c dmar: [164] stat 0 00000000000100f2 000001500022c000 dmar: [165] stat 0 0000000200000025 0000001fff412294 dmar: [166] stat 0 00000000000100f2 000001500022d000 dmar: [167] stat 0 0000000200000025 0000001fff41229c dmar: [168] stat 0 00000000000100f2 000001500022e000 dmar: [169] stat 0 0000000200000025 0000001fff4122a4 dmar: [170] stat 0 00000000000100f2 000001500022f000 dmar: [171] stat 0 0000000200000025 0000001fff4122ac dmar: [172] stat 0 00000000000100f2 0000015000230000 dmar: [173] stat 0 0000000200000025 0000001fff4122b4 dmar: [174] stat 0 00000000000100f2 0000015000231000 dmar: [175] stat 0 0000000200000025 0000001fff4122bc dmar: [176] stat 0 00000000000100f2 0000015000232000 dmar: [177] stat 0 0000000200000025 0000001fff4122c4 dmar: [178] stat 0 00000000000100f2 0000015000233000 dmar: [179] stat 0 0000000200000025 0000001fff4122cc dmar: [180] stat 0 00000000000100f2 0000015000234000 dmar: [181] stat 0 0000000200000025 0000001fff4122d4 dmar: [182] stat 0 00000000000100f2 0000015000235000 dmar: [183] stat 0 0000000200000025 0000001fff4122dc dmar: [184] stat 0 00000000000100f2 0000015000215000 dmar: [185] stat 0 0000000200000025 0000001fff4122e4 dmar: [186] stat 0 00000000000100f2 0000015000216000 dmar: [187] stat 0 0000000200000025 0000001fff4122ec dmar: [188] stat 0 00000000000100f2 0000015000217000 dmar: [189] stat 0 0000000200000025 0000001fff4122f4 dmar: [190] stat 0 00000000000100f2 0000015000218000 dmar: [191] stat 0 0000000200000025 0000001fff4122fc dmar: [192] stat 0 00000000000100f2 0000015000219000 dmar: [193] stat 0 0000000200000025 0000001fff412304 dmar: [194] stat 0 00000000000100f2 000001500021a000 dmar: [195] stat 0 0000000200000025 0000001fff41230c dmar: [196] stat 0 00000000000100f2 000001500021b000 dmar: [197] stat 0 0000000200000025 0000001fff412314 dmar: [198] stat 0 00000000000100f2 000001500021c000 dmar: [199] stat 0 0000000200000025 0000001fff41231c dmar: [200] stat 0 00000000000100f2 000001500021d000 dmar: [201] stat 0 0000000200000025 0000001fff412324 dmar: [202] stat 0 00000000000100f2 000001500021e000 dmar: [203] stat 0 0000000200000025 0000001fff41232c dmar: [204] stat 0 00000000000100f2 000001500021f000 dmar: [205] stat 0 0000000200000025 0000001fff412334 dmar: [206] stat 0 00000000000100f2 0000015000220000 dmar: [207] stat 0 0000000200000025 0000001fff41233c dmar: [208] stat 0 00000000000100f2 0000015000221000 dmar: [209] stat 0 0000000200000025 0000001fff412344 dmar: [210] stat 0 00000000000100f2 0000015000222000 dmar: [211] stat 0 0000000200000025 0000001fff41234c dmar: [212] stat 0 00000000000100f2 0000015000223000 dmar: [213] stat 0 0000000200000025 0000001fff412354 dmar: [214] stat 0 00000000000100f2 0000015000224000 dmar: [215] stat 0 0000000200000025 0000001fff41235c dmar: [216] stat 0 00000000000100f2 0000015000204000 dmar: [217] stat 0 0000000200000025 0000001fff412364 dmar: [218] stat 0 00000000000100f2 0000015000205000 dmar: [219] stat 0 0000000200000025 0000001fff41236c dmar: [220] stat 0 00000000000100f2 0000015000206000 dmar: [221] stat 0 0000000200000025 0000001fff412374 dmar: [222] stat 0 00000000000100f2 0000015000207000 dmar: [223] stat 0 0000000200000025 0000001fff41237c dmar: [224] stat 0 00000000000100f2 0000015000208000 dmar: [225] stat 0 0000000200000025 0000001fff412384 dmar: [226] stat 0 00000000000100f2 0000015000209000 dmar: [227] stat 0 0000000200000025 0000001fff41238c dmar: [228] stat 0 00000000000100f2 000001500020a000 dmar: [229] stat 0 0000000200000025 0000001fff412394 dmar: [230] stat 0 00000000000100f2 000001500020b000 dmar: [231] stat 0 0000000200000025 0000001fff41239c dmar: [232] stat 0 00000000000100f2 000001500020c000 dmar: [233] stat 0 0000000200000025 0000001fff4123a4 dmar: [234] stat 0 00000000000100f2 000001500020d000 dmar: [235] stat 0 0000000200000025 0000001fff4123ac dmar: [236] stat 0 00000000000100f2 000001500020e000 dmar: [237] stat 0 0000000200000025 0000001fff4123b4 dmar: [238] stat 0 00000000000100f2 000001500020f000 dmar: [239] stat 0 0000000200000025 0000001fff4123bc dmar: [240] stat 0 00000000000100f2 0000015000210000 dmar: [241] stat 0 0000000200000025 0000001fff4123c4 dmar: [242] stat 0 00000000000100f2 0000015000211000 dmar: [243] stat 0 0000000200000025 0000001fff4123cc dmar: [244] stat 0 00000000000100f2 0000015000212000 dmar: [245] stat 0 0000000200000025 0000001fff4123d4 dmar: [246] stat 0 00000000000100f2 0000015000213000 dmar: [247] stat 0 0000000200000025 0000001fff4123dc dmar: [248] stat 0 00000000000100f2 0000015000201000 dmar: [249] stat 0 0000000200000025 0000001fff4123e4 dmar: [250] stat 0 00000000000100f2 0000015000202000 dmar: [251] stat 0 0000000200000025 0000001fff4123ec dmar: [252] stat 1 0000001410000014 0000000000000000 dmar: [253] stat 1 0000000200000025 0000001fff4123f4 dmar: [254] stat 0 00000000000100f2 000001500027e000 dmar: [255] stat 0 0000000200000025 0000001fff4123fc ------------[ cut here ]------------ kernel BUG at drivers/iommu/dmar.c:901! invalid opcode: 0000 [#1] SMP CPU: 1 PID: 5310 Comm: foed Tainted: G O 3.10.59+ #201501080140+8825998.platinum task: ffff883fe5f743e0 ti: ffff883feb696000 task.ti: ffff883feb696000 RIP: 0010:[<ffffffff81529737>] [<ffffffff81529737>] qi_submit_sync+0x477/0x510 RSP: 0018:ffff883feb697a10 EFLAGS: 00010046 RAX: 0000000000000038 RBX: 0000000000000100 RCX: 00000000000000dd RDX: 00000000000000e2 RSI: 0000000000000046 RDI: 0000000000000046 RBP: ffff883feb697a78 R08: ffffffffffffffff R09: 0000000000003b72 R10: 0000000000000075 R11: 00000000000000d1 R12: ffff883ff2dda008 R13: 0000000000000400 R14: ffff881fff40c6c0 R15: ffff883ffec019c0 FS: 00007fb876bea700(0000) GS:ffff881fffa20000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f42963c9d50 CR3: 0000000003e0c000 CR4: 00000000001407e0 wait_for_sess_cmds: Completed cmd ffff882706266f70 tag 0x117514 tr_state 0x91 t_state 5 flags 0x4010b tas 0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff883feb697a88 0000000000000fd0 0000000000000fc0 ffff883ff2dd9000 0000000000000086 00000000000000fc 000000fc000000fd 000b415115c12cd0 ffff883fec30c800 0000000000000000 0000000000000246 ffff883fef949858 Call Trace: [<ffffffff8152f6b5>] free_irte+0xc5/0x100 [<ffffffff81530834>] free_remapped_irq+0x44/0x60 [<ffffffff81027b23>] destroy_irq+0x33/0xd0 [<ffffffff81027ede>] native_teardown_msi_irq+0xe/0x10 [<ffffffff812a6a70>] default_teardown_msi_irqs+0x60/0x80 [<ffffffff812a64d9>] free_msi_irqs+0x99/0x150 [<ffffffff812a749d>] pci_disable_msix+0x3d/0x60 [<ffffffffa006b748>] vfio_msi_disable+0xc8/0xe0 [vfio_pci] [<ffffffffa006bf86>] vfio_pci_set_msi_trigger+0x2a6/0x2d0 [vfio_pci] [<ffffffffa006c41c>] vfio_pci_set_irqs_ioctl+0x8c/0xa0 [vfio_pci] [<ffffffffa006a3b0>] vfio_pci_release+0x70/0x150 [vfio_pci] [<ffffffffa0060cbc>] vfio_device_fops_release+0x1c/0x40 [vfio] [<ffffffff8114d7db>] __fput+0xdb/0x220 [<ffffffff8114d92e>] ____fput+0xe/0x10 [<ffffffff810614ac>] task_work_run+0xbc/0xe0 [<ffffffff81043d0e>] do_exit+0x3ce/0xe50 [<ffffffff8104557f>] do_group_exit+0x3f/0xa0 [<ffffffff81054769>] get_signal_to_deliver+0x1a9/0x5b0 [<ffffffff810023f8>] do_signal+0x48/0x5e0 [<ffffffff81056778>] ? k_getrusage+0x368/0x3d0 [<ffffffff810595f0>] ? getrusage+0x30/0x40 [<ffffffff81009449>] ? read_tsc+0x9/0x20 [<ffffffff810029f5>] do_notify_resume+0x65/0x80 [<ffffffff81648ab8>] int_signal+0x12/0x17 Code: 89 de 4d 8b 04 24 48 c7 c7 b0 12 bd 81 83 c3 01 49 83 c4 10 42 8b
--------------------
And our debug patch (on top of 3.10.59):
drivers/iommu/dmar.c +31 -1
@@ -781,6 +782,8 @@ static int qi_check_fault(struct intel_iommu *iommu, int index)* No new descriptors are fetched until the ITE is cleared. */ if (fault &DMA_FSTS_ITE ) { +trace_printk ("DMA_FSTS_ITE set (%08x)\n" , fault); + head =readl (iommu->reg +DMAR_IQH_REG ); head = ((head >>DMAR_IQ_SHIFT ) -1 +QI_LENGTH ) %QI_LENGTH ; head |=1 ;
@@ -799,8 +802,10 @@ static int qi_check_fault(struct intel_iommu *iommu, int index)return -EAGAIN ; } -if (fault &DMA_FSTS_ICE ) +if (fault &DMA_FSTS_ICE ) { +trace_printk ("DMA_FSTS_ICE set (%08x)\n" , fault);writel (DMA_FSTS_ICE , iommu->reg +DMAR_FSTS_REG ); + }return 0 ; }
@@ -816,6 +821,7 @@ int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu)struct qi_desc *hw, wait_desc;int wait_index, index;unsigned long flags; +cycles_t start_time;if (!qi)return 0 ;
@@ -826,8 +832,16 @@ restart: rc =0 ;raw_spin_lock_irqsave (&qi->q_lock, flags); + start_time =get_cycles ();while (qi->free_cnt <3 ) {raw_spin_unlock_irqrestore (&qi->q_lock, flags); + +if (get_cycles () - start_time >DMAR_OPERATION_TIMEOUT ) { +printk (KERN_EMERG "Only %d free entries in invalidation queue.\n" , + qi->free_cnt); +BUG (); + } +cpu_relax ();raw_spin_lock_irqsave (&qi->q_lock, flags); }
@@ -857,6 +871,7 @@ restart:*/ writel (qi->free_head <<DMAR_IQ_SHIFT , iommu->reg +DMAR_IQT_REG ); + start_time =get_cycles ();while (qi->desc_status[wait_index] !=QI_DONE ) {/* * We will leave the interrupts disabled, to prevent interrupt
@@ -870,6 +885,22 @@ restart:break ;raw_spin_unlock (&qi->q_lock); + +if (get_cycles () - start_time >DMAR_OPERATION_TIMEOUT ) { +int i; + +pr_emerg ("QI stuck waiting for index %d / wait_index %d IQH/T %08x/%08x FSTS %08x\n" , + index, wait_index, +readl (iommu->reg +DMAR_IQH_REG ), +readl (iommu->reg +DMAR_IQT_REG ), +readl (iommu->reg +DMAR_FSTS_REG )); +for (i =0 ; i <QI_LENGTH ; ++i) +pr_emerg (" [%3d] stat %x %016llx %016llx\n" , + i, qi->desc_status[i], hw[i].low, hw[i].high); + +BUG (); + } +cpu_relax ();raw_spin_lock (&qi->q_lock); }
A message that is not in the archive (sent off-list or not yet received): 1420756610-20918-1-git-send-email-roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Quoted 6 lines · Roland Dreier
Hi,
So we've managed to find one magic setup and workload that reproduces this reliably, and I've done a bit more debugging that leads me to believe we probably need Intel's help to really get to the bottom of this.
Can you share your test case? My test case was also using vfio, repeatedly booting a pair of VMs with assigned GPUs, but I haven't seen any sign of the issue without NET_DMA. I agree that it seems like the hardware is getting somewhere that it shouldn't regardless of the software, but are you able to reproduce on a more recent kernel?
I've also seen a report of the same problem without vfio by manipulating IRQ affinity, but it was not a reliable test case.
Quoted 5 lines
One question though: you mentioned that you saw this behavior until you turned off
CONFIG_NET_DMA. What platform was that on? We see this on dual socket Xeon E5 v3 (Haswell EP / Grantley), and I don't really have any other setup I can try. Did you see this on other platforms (Ivy Bridge / Romley maybe)?
I saw it on a dual socket Xeon E5 v2 (Ivy Bridge EP / Patsburg), and the other report I mention above was the same, different systems though.
Quoted 4 lines
Anyway, I added the debugging patch at the end of this mail to our kernel to dump some status when the driver detects a hung queue. Below we see some example output. Things I notice:
Quoted 8 lines
- IQH == IQT, in other words the QI hardware thinks it has picked up and validated all the descriptors submitted by software.
- The queue has successfully processed many operations (although everything that we can see succeeded is type 2h, ie "IOTLB invalidate" as opposed to the type 4h "Interrupt Entry Cache invalidate" that we're stuck on.
- There haven't been any faults and FSTS is clear.
- The hardware hasn't executed the "Invalidation wait" descriptor.
This is all consistent with my observations as well.
Quoted 3 lines
Beyond that I'm not sure how to make much more progress without insight into the hardware -- it looks like the driver is doing everything right. Anyone from Intel have any thoughts?
Agreed, the hardware appears to be getting wedged without and sort of fault indication or recovery mechanism. Thanks,
Alex