Skip to contentSkip to conversation

fix iommu interrupt setup: setup_vector() may fail

PATCH In review Simon Horman 1 participant 2 messages Feb 16, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fix iommu interrupt setup: setup_vector() may fail

Since changeset 19175 (Cleanup IOMMU interrupt setup) I am experiencing a boot failure on my HP dc7800 running in x86_64 mode. The most recent version that I have tested is 19210 (x86: Clean up vpt-align patch.), which is after 19207 (iommu: Fix interrupt setup.) which Shimura-san created to fix some of the problems caused by 19175.

After investigating this problem I found that the cause is that for some vectors setup_vector() fails (returning -EBUSY).

setup_vector() is called by request_irq_vector(). And request_irq_vector() is called by iommu_set_interrupt(). Which is in turn called by init_vtd_hw().

And 19175 essentailly adds error checking code which now checks for errors returned by iommu_set_interrupt() and thus indirectly the error returned by setup_vector().

Unfortunately as things stands, the way that init_vtd_hw() handles such errors causes my dc7800 not to boot.

I am not entirely sure of the correctness of my proposed change, but by calling iommu_set_interrupt() before iommu_set_root_entry() in init_vtd_hw() my system boots correctly.

In particularl I'm not sure:
a) if it is always valid to call iommu_set_interrupt() before iommu_set_root_entry()
b) if this would adversely affect a situation where iommu_set_root_entry() fails.
- This seems to be ok, I tested replacing the body of

     iommu_set_root_entry() with return -EINVAL;

---

(XEN) Xen version 3.4-unstable ([email protected]) (gcc version 4.0.2) Mon Feb 16 14:00:20 EST 2009 (XEN) Latest ChangeSet: Fri Feb 13 09:48:56 2009 +0000 19210:09a6fa059b37
(XEN) Command line: com1=115200,8n1,0x1248,0 console=com1 loglvl=all guest_loglvl=all vtd=1 iommu=1 (XEN) Video information:

(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected

(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:

(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e8000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000defafe00 (usable)
(XEN)  00000000defafe00 - 00000000defb1ea0 (ACPI NVS)
(XEN)  00000000defb1ea0 - 00000000e0000000 (reserved)
(XEN)  00000000f4000000 - 00000000f8000000 (reserved)
(XEN)  00000000fec00000 - 00000000fed40000 (reserved)
(XEN)  00000000fed45000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000011c000000 (usable)

(XEN) System RAM: 4015MB (4111672kB)
(XEN) ACPI: RSDP 000E5C10, 0014 (r0 COMPAQ)
(XEN) ACPI: RSDT DEFC1E40, 0044 (r1 HPQOEM SLIC-BPC 20080318 0)
(XEN) ACPI: FACP DEFC1EE8, 0074 (r1 COMPAQ BEARLAKE 1 0)
(XEN) ACPI: DSDT DEFC2427, A370 (r1 COMPAQ DSDT_PRJ 1 MSFT 100000E)
(XEN) ACPI: FACS DEFC1E00, 0040
(XEN) ACPI: APIC DEFC1F5C, 0084 (r1 COMPAQ BEARLAKE 1 0)
(XEN) ACPI: ASF! DEFC1FE0, 0063 (r32 COMPAQ BEARLAKE 1 0)

(XEN) ACPI: MCFG DEFC2043, 003C (r1 COMPAQ BEARLAKE        1             0)
(XEN) ACPI: TCPA DEFC207F, 0032 (r1 COMPAQ BEARLAKE        1             0)
(XEN) ACPI: SLIC DEFC20B1, 0176 (r1 HPQOEM SLIC-BPC        1             0)
(XEN) ACPI: HPET DEFC2227, 0038 (r1 COMPAQ BEARLAKE        1             0)
(XEN) ACPI: DMAR DEFC225F, 0198 (r1 COMPAQ BEARLAKE        1             0)

(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-000000011c000000 (XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f9bf0
(XEN) DMI 2.5 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0xf808
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[f804,460], pm1x_evt[f800,0]
(XEN) ACPI: wakeup_vec[defc1e0c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) (XEN) Processor #0 7:7 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) (XEN) Processor #1 7:7 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x00] disabled) (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x00] disabled) (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) (XEN) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [VT-D]dmar.c:460: Host address width 36
(XEN) [VT-D]dmar.c:469: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fed90000
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1b.0 (XEN) [VT-D]dmar.c:469: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fed91000
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:469: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fed92000
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:3.0
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:3.2
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:3.3
(XEN) [VT-D]dmar.c:469: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fed93000
(XEN) [VT-D]dmar.c:346: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.7 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.7 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.0 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.1 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.2 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.0 (XEN) [VT-D]dmar.c:473: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.1 (XEN) Intel VT-d has been enabled
(XEN) Using ACPI (MADT) for SMP configuration information (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Initializing CPU#0
(XEN) Detected 2826.324 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:

(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap

(XEN) HVM: VMX enabled
(XEN) MCE: Intel newly family MC Init
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU0 has no CMCI support
(XEN) CPU0: Intel(R) Core(TM)2 Duo CPU E8300 @ 2.83GHz stepping 06
(XEN) Booting processor 1/1 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) MCE: Intel newly family MC Init
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU1 has no CMCI support
(XEN) CPU1: Intel(R) Core(TM)2 Duo CPU E8300 @ 2.83GHz stepping 06
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) [VT-D]iommu.c:1707: Queued Invalidation hardware not found
(XEN) [VT-D]iommu.c:1707: Queued Invalidation hardware not found
(XEN) [VT-D]iommu.c:1707: Queued Invalidation hardware not found
(XEN) [VT-D]iommu.c:1707: Queued Invalidation hardware not found
(XEN) [VT-D]iommu.c:1715: Interrupt Remapping hardware not found
(XEN) [VT-D]iommu.c:1715: Interrupt Remapping hardware not found
(XEN) [VT-D]iommu.c:1715: Interrupt Remapping hardware not found
(XEN) [VT-D]iommu.c:1715: Interrupt Remapping hardware not found
(XEN) Intel VT-d snoop control disabled
(XEN) I/O virtualisation enabled
(XEN) I/O virtualisation for PV guests disabled
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Init_mcheck_timer
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:0.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:3.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:3.2
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:3.3
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:19.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:1246:d32767 domain_context_mapping:PCIe: bdf = 0:1b.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1f.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1f.2
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1f.5
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1253:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:618: iommu_enable_translation: iommu->reg = ffff828bfff57000
(XEN) [VT-D]iommu.c:618: iommu_enable_translation: iommu->reg = ffff828bfff56000
(XEN) [VT-D]iommu.c:618: iommu_enable_translation: iommu->reg = ffff828bfff55000
(XEN) [VT-D]iommu.c:618: iommu_enable_translation: iommu->reg = ffff828bfff54000
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_init: not an ELF binary
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) [VT-D]iommu.c:721: iommu_page_fault: iommu->reg = ffff828bfff54000
(XEN) [VT-D]iommu.c:690: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:693: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:675: iommu_fault:DMA Read: 0:1d.2 addr defce000 REASON 6 iommu->reg = ffff828bfff54000 (XEN) print_vtd_entries: iommu = ffff83011bfd4310 bdf = 0:1d:2 gmfn = defce

(XEN)     root_entry = ffff83011bfcb000
(XEN)     root_entry[0] = 118453001
(XEN)     context = ffff830118453000
(XEN)     context[ea] = 101_11bfb1001
(XEN)     l3 = ffff83011bfb1000
(XEN)     l3_index = 3
(XEN)     l3[3] = 11862d003
(XEN)     l2 = ffff83011862d000
(XEN)     l2_index = f7
(XEN)     l2[f7] = 118535003
(XEN)     l1 = ffff830118535000
(XEN)     l1_index = 1ce
(XEN)     l1[1ce] = 0
(XEN)     l1[1ce] not present
xen/drivers/passthrough/vtd/iommu.c +6 -6
Index: xen-unstable.hg/xen/drivers/passthrough/vtd/iommu.c
===================================================================
@@ -1674,12 +1674,6 @@ static int init_vtd_hw(void)
     for_each_drhd_unit ( drhd )
     {
         iommu = drhd->iommu;
-        ret = iommu_set_root_entry(iommu);
-        if ( ret )
-        {
-            gdprintk(XENLOG_ERR VTDPREFIX, "IOMMU: set root entry failed\n");
-            return -EIO;
-        }
 
         vector = iommu_set_interrupt(iommu);
         if ( vector < 0 )
@@ -1687,6 +1681,12 @@ static int init_vtd_hw(void)
             gdprintk(XENLOG_ERR VTDPREFIX, "IOMMU: interrupt setup failed\n");
             return vector;
         }
+        ret = iommu_set_root_entry(iommu);
+        if ( ret )
+        {
+            gdprintk(XENLOG_ERR VTDPREFIX, "IOMMU: set root entry failed\n");
+            return -EIO;
+        }
         dma_msi_data_init(iommu, vector);
         dma_msi_addr_init(iommu, cpu_physical_id(first_cpu(cpu_online_map)));
         iommu->vector = vector;
#1 lore raw
Quoted 5 lines · Simon Horman

Since changeset 19175 (Cleanup IOMMU interrupt setup) I am experiencing a boot failure on my HP dc7800 running in x86_64 mode. The most recent version that I have tested is 19210 (x86: Clean up vpt-align patch.), which is after 19207 (iommu: Fix interrupt setup.) which Shimura-san created to fix some of the problems caused by 19175.

Despite my long-winded explanation, there seems to be an error in my testing and this patch not work. I hope to have a revised version available shortly.

Signature · 3 lines
Simon Horman
  VA Linux Systems Japan K.K., Sydney, Australia Satellite Office
  H: www.vergenet.net/~horms/             W: www.valinux.co.jp/en