IOMMU Updates for v3.7-rc1

Hi Linus,

The following changes since commit a0d271cbfed1dd50278c6b06bead3d00ba0a88f9:

Linux 3.6 (2012-09-30 16:47:46 -0700)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu.git tags/iommu-updates-v3.7-rc1

for you to fetch changes up to 009487258399cb4f431992919fa0f386d1b74ceb:

Merge branches 'dma-debug', 'iommu/fixes', 'arm/tegra', 'arm/exynos', 'x86/amd', 'x86/vt-d' and 'x86/amd-irq-remapping' into next (2012-10-02 14:40:03 +0200)

----------------------------------------------------------------

IOMMU Updates for Linux v3.7-rc1

This time the IOMMU updates contain a bunch of fixes and cleanups to various IOMMU drivers and the DMA debug code. New features are the code for IRQ remapping support with the AMD IOMMU (preperation for that was already merged in the last release) and a debugfs interface to export some statistics in the NVidia Tegra IOMMU driver.

----------------------------------------------------------------
Alex Williamson (1):
intel-iommu: Default to non-coherent for domains unattached to iommus

Borislav Petkov (1):
iommu/amd: Fix features reporting

Dan Carpenter (1):
iommu/amd: Fix possible use after free in get_irq_table()

Frank Arnold (1):
iommu/amd: Fix some typos

Hiroshi Doyu (2):
iommu/tegra: smmu: debugfs for TLB/PTC statistics iommu/tegra: smmu: Use debugfs_create_dir for directory

Joerg Roedel (18):
iommu/amd: Keep track of HPET and IOAPIC device ids
iommu/amd: Add slab-cache for irq remapping tables
iommu/amd: Allocate data structures to keep track of irq remapping tables
iommu/amd: Check if IOAPIC information is correct
iommu/amd: Split device table initialization into irq and dma part
iommu/amd: Make sure IOMMU is not considered to translate itself
iommu/amd: Add IRTE invalidation routine
iommu/amd: Add routines to manage irq remapping tables
iommu/amd: Add IOAPIC remapping routines
iommu/amd: Implement MSI routines for interrupt remapping
iommu/amd: Add call-back routine for HPET MSI
iommu/amd: Add initialization routines for AMD interrupt remapping
iommu/amd: Make sure irq remapping still works on dma init failure
iommu/irq: Use amd_iommu_irq_ops if supported
iommu/amd: Print message to system log when irq remapping is enabled
iommu/amd: Report irq remapping through IOMMU-API
iommu/amd: Remove obsolete comment line
Merge branches 'dma-debug', 'iommu/fixes', 'arm/tegra', 'arm/exynos', 'x86/amd', 'x86/vt-d' and 'x86/amd-irq-remapping' into next

Shuah Khan (1):
dma-debug: Remove local BUS_NOTIFY_UNBOUND_DRIVER define

Stephen Warren (1):
dma: tegra: move smmu.h into SMMU driver

Wei Yongjun (2):
iommu/amd: remove duplicated include from amd_iommu_init.c iommu/exynos: use list_del_init instead of list_del/INIT_LIST_HEAD

 arch/arm/mach-tegra/include/mach/smmu.h |   63 ----
 drivers/iommu/Kconfig                   |    2 +-
 drivers/iommu/amd_iommu.c               |  514 ++++++++++++++++++++++++++++++-
 drivers/iommu/amd_iommu_init.c          |  253 +++++++++++++--
 drivers/iommu/amd_iommu_proto.h         |    8 +
 drivers/iommu/amd_iommu_types.h         |   59 +++-
 drivers/iommu/exynos-iommu.c            |    3 +-
 drivers/iommu/intel-iommu.c             |    4 +-
 drivers/iommu/irq_remapping.c           |    5 +
 drivers/iommu/irq_remapping.h           |    6 +
 drivers/iommu/tegra-smmu.c              |  261 +++++++++++++++-
 lib/dma-debug.c                         |    5 -
 12 files changed, 1071 insertions(+), 112 deletions(-)
 delete mode 100644 arch/arm/mach-tegra/include/mach/smmu.h

Please pull.

Regards,

Joerg

I'm getting a kernel panic here when I try to boot from master (9aadf84) in in your IOMMU tree. I'll try booting the tagged commit in case it's the merge that caused the problem but I'm not particularly hopeful.

I have got interrupt remapping support enabled and the IOMMU enabled, the chipset is an RD890 (890FX).

I think the relevant bit of the kernel output is here:

ACPI: Core revision 20120711
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1

Kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC
Pid: 1, comm: swapper/0 Not tainted 3.6.0+ #2
Call Trace:
 [<ffffffff8163f617>] ? panic+0xc5/0x1d0
 [<ffffffff8163f76e>] ? printk+0x4c/0x51
 [<ffffffff81c96b52>] ? setup_IO_APIC+0x325/0x710
 [<ffffffff81c947b5>] ? native_smp_prepare_cpus+0x2b9/0x334
 [<ffffffff81c87b53>] ? kernel_init+0x59/0x190
 [<ffffffff81643b84>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81c87afa>] ? start_kernel+0x2c7/0x2c7
 [<ffffffff81643b80>] ? gs_change+0x13/0x13

There is a patch to fix a similar problem (32ab31e) for the Intel IRQ remapping code but I don't know if this is likely to be the same issue or not.

I've posted the full kernel log, kernel config and lspci output in case they are useful:
http://adoakley.name/amd-iommu/dmesg.txt
http://adoakley.name/amd-iommu/kernel_config.txt
http://adoakley.name/amd-iommu/lspci.txt

Thanks

Andrew Oakley

Hi Andrew,

Can you please boot with 'amd_iommu_dump nointremap' on your kernel command line and post dmesg? I guess your ACPI table is broken in some way.

The problem seen on VT-d exists on AMD too, but I have a workaround for that in the code already. This must be something else...

Thanks,

Joerg

http://adoakley.name/amd-iommu/dmesg_amd_iommu_dump.txt

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
AMD-Vi:        mmio-addr: 00000000fec30000
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:00.2
AMD-Vi:   DEV_SELECT			 devid: 00:02.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 01:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 01:00.1
AMD-Vi:   DEV_SELECT			 devid: 00:04.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 02:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:05.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:06.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:07.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 05:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:09.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 06:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:0b.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 07:00.0 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:11.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:12.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:12.2
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:13.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:13.2
AMD-Vi:   DEV_SELECT			 devid: 00:14.0 flags: d7
AMD-Vi:   DEV_SELECT			 devid: 00:14.2 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:14.3 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:14.4 flags: 00
AMD-Vi:   DEV_ALIAS_RANGE		 devid: 08:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END		 devid: 08:1f.7
AMD-Vi:   DEV_SELECT			 devid: 00:14.5 flags: 00
AMD-Vi:   DEV_SELECT			 devid: 00:15.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 09:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 09:1f.7
AMD-Vi:   DEV_SELECT_RANGE_START	 devid: 00:16.0 flags: 00
AMD-Vi:   DEV_RANGE_END		 devid: 00:16.2
AMD-Vi:   DEV_SPECIAL(HPET[0])		devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(IOAPIC[2])		devid: 00:00.1

Andrew Oakley

Okay, got it. This is the problem. The device id seems to be wrong. The BIOS should configure your IOAPIC as device id 00:14.0 and not 00:00.1. The current checking code does not check for this particular situation. But the attached diff should change that. Can you please test it and reports if the machine boots for you? Interrupt remapping will be disabled automatically, though.

index 18b0d99..81837b0 100644
@@ -1599,21 +1599,46 @@ static void __init free_on_init_error(void)
 #endif
 }
 
+/* SB IOAPIC is always on this device in AMD systems */
+#define IOAPIC_SB_DEVID		((0x00 << 8) | PCI_DEVFN(0x14, 0))
+
 static bool __init check_ioapic_information(void)
 {
+	bool ret, has_sb_ioapic;
 	int idx;
 
-	for (idx = 0; idx < nr_ioapics; idx++) {
-		int id = mpc_ioapic_id(idx);
+	has_sb_ioapic = false;
+	ret           = false;
 
-		if (get_ioapic_devid(id) < 0) {
-			pr_err(FW_BUG "AMD-Vi: IO-APIC[%d] not in IVRS table\n", id);
-			pr_err("AMD-Vi: Disabling interrupt remapping due to BIOS Bug\n");
-			return false;
+	for (idx = 0; idx < nr_ioapics; idx++) {
+		int devid, id = mpc_ioapic_id(idx);
+
+		devid = get_ioapic_devid(id);
+		if (devid < 0) {
+			pr_err(FW_BUG "AMD-Vi: IOAPIC[%d] not in IVRS table\n", id);
+			ret = false;
+		} else if (devid == IOAPIC_SB_DEVID) {
+			has_sb_ioapic = true;
+			ret           = true;
 		}
 	}
 
-	return true;
+	if (!has_sb_ioapic) {
+		/*
+		 * We expect the SB IOAPIC to be listed in the IVRS
+		 * table. The system timer is connected to the SB IOAPIC
+		 * and if we don't have it in the list the system will
+		 * panic at boot time.  This situation usually happens
+		 * when the BIOS is buggy and provides us the wrong
+		 * device id for the IOAPIC in the system.
+		 */
+		pr_err(FW_BUG "AMD-Vi: No southbridge IOAPIC found in IVRS table\n");
+	}
+
+	if (!ret)
+		pr_err("AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)\n");
+
+	return ret;
 }
 
 static void __init free_dma_resources(void)

Yes, that detects the problem and disables the interrupt remapping (the kernel boots).

If I modify the devid for the IOAPIC in init_iommu_from_acpi then it also seems to work. I haven't managed to get vfio/qemu working yet but it seems to be unrelated ("Invalid ROM contents", same thing happens when I try to dump the rom with stable kernels).

Thanks

Andrew Oakley

I've now managed to get a network card passed through and working properly in a guest. The bug has been reported to the vendor so hopefully it will get fixed in a BIOS update.

I'm not sure if the fix I've got is safe or if it's really a good idea to enable interrupt remapping with a broken BIOS.

Thanks

Andrew Oakley

From: Andrew Oakley <[email protected]>

The IOAPIC always has the same device id but is sometimes reported in the IVRS table with a different id. This patch allows interrupt remapping to work on devices with this bug.

---
 drivers/iommu/amd_iommu_init.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

index 18b0d99..ca266b9 100644
@@ -751,6 +751,9 @@ static void __init set_device_exclusion_range(u16 devid, struct ivmd_header *m)
 	}
 }
 
+/* SB IOAPIC is always on this device in AMD systems */
+#define IOAPIC_SB_DEVID		((0x00 << 8) | PCI_DEVFN(0x14, 0))
+
 /*
  * Takes a pointer to an AMD IOMMU entry in the ACPI table and
  * initializes the hardware and our data structures with it.

@@ -928,6 +931,12 @@ static int __init init_iommu_from_acpi(struct amd_iommu *iommu,
 				    PCI_SLOT(devid),
 				    PCI_FUNC(devid));
 
+			if (type == IVHD_SPECIAL_IOAPIC &&
+			    devid != IOAPIC_SB_DEVID) {
+				pr_err(FW_BUG "AMD-Vi: wrong IOAPIC devid\n");
+				devid = IOAPIC_SB_DEVID;
+			}
+
 			set_dev_entry_from_acpi(iommu, devid, e->flags, 0);
 			ret = add_special_device(type, handle, devid);
 			if (ret)

1.7.8.6

Sadly it is not. Your patch works for your particular case but is not generally applicable. There are indeed systems with 2 or more IOAPICs and they all use different device-ids.

So, typically, there is one IOAPIC per SB, like in your case. But on RD890 the chipset NB also includes an IOAPIC (which uses device-id 00:00.1). This IOAPIC exists on your system too but is disabled by the BIOS because the board does not use it.

So, in the end, device-id 00:00.1 might also be correct and the IVRS entry could refer to the NB IOAPIC. We don't have any reliable indication which IOAPIC is described in the IVRS table.

Cheer,

Joerg

AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632