IOMMU fixes for 2.6.34-rc3
Subsystems (3)
Hi Ingo,
The following changes since commit 2eaa9cfdf33b8d7fb7aff27792192e0019ae8fc6:
Linus Torvalds (1):
Linux 2.6.34-rc3
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux-2.6-iommu.git iommu/fixes
Chris Wright (5):
x86/amd-iommu: Pt mode fix for domain_destroy x86/amd-iommu: enable iommu before attaching devices x86/amd-iommu: warn when issuing command to uninitialized cmd buffer Revert "x86: disable IOMMUs on kernel crash" x86/amd-iommu: use for_each_pci_dev
Dan Carpenter (1):
dma-debug: Cleanup for copy-loop in filter_write()
Joerg Roedel (6):
x86/amd-iommu: Protect IOMMU-API map/unmap path x86/amd-iommu: Report errors in acpi parsing functions upstream x86/amd-iommu: Use helper function to destroy domain x86/amd-iommu: Remove obsolete parameter documentation Merge branch 'amd-iommu/fixes' into iommu/fixes x86/gart: Disable GART explicitly before initialization
Julia Lawall (1):
x86/amd-iommu: Remove double NULL check in check_device
Documentation/kernel-parameters.txt | 5 --- arch/x86/include/asm/amd_iommu_types.h | 3 ++ arch/x86/kernel/amd_iommu.c | 20 +++++++++---- arch/x86/kernel/amd_iommu_init.c | 48 ++++++++++++++++++++++---------- arch/x86/kernel/aperture_64.c | 15 +++++++++- arch/x86/kernel/crash.c | 6 ---- arch/x86/kernel/pci-gart_64.c | 3 ++ lib/dma-debug.c | 2 +- 8 files changed, 68 insertions(+), 34 deletions(-)
This pull request includes fixes that did not made it upstream during the merge window, fixes for recently discovered problems with kdump and amd-iommu enabled and a small documentation update. Please pull.
Thanks,
Joerg
PATCH 1/12 x86/amd-iommu: Remove double NULL check in check_device
From: Julia Lawall <[email protected]>
dev was tested just above, so drop the second test.
--- arch/x86/kernel/amd_iommu.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/amd_iommu.c +1 -1
@@ -118,7 +118,7 @@ static bool check_device(struct device *dev)return false;/* No device or no PCI device */ -if (!dev || dev->bus != &pci_bus_type) +if (dev->bus != &pci_bus_type)return false; devid =get_device_id (dev);
Signature · 1 line
1.7.0.4
PATCH 2/12 x86/amd-iommu: Protect IOMMU-API map/unmap path
In reply to Joerg RoedelThis patch introduces a mutex to lock page table updates in the IOMMU-API path. We can't use the spin_lock here because this patch might sleep.
--- arch/x86/include/asm/amd_iommu_types.h | 2 ++ arch/x86/kernel/amd_iommu.c | 9 +++++++++ 2 files changed, 11 insertions(+), 0 deletions(-)
arch/x86/include/asm/amd_iommu_types.h +2 -0
@@ -21,6 +21,7 @@#define _ASM_X86_AMD_IOMMU_TYPES_H#include <linux/types.h> +#include <linux/mutex.h> #include <linux/list.h> #include <linux/spinlock.h>
@@ -237,6 +238,7 @@ struct protection_domain {struct list_head list;/* for list of all protection domains */ struct list_head dev_list;/* List of all devices in this domain */ spinlock_t lock;/* mostly used to lock the page table*/ +struct mutex api_lock;/* protect page tables in the iommu-api path */ u16 id;/* the domain id written to the device table */ int mode;/* paging mode (0-6 levels) */ u64 *pt_root;/* page table root pointer */
arch/x86/kernel/amd_iommu.c +9 -0
@@ -2327,6 +2327,7 @@ static struct protection_domain *protection_domain_alloc(void)return NULL ;spin_lock_init (&domain->lock); +mutex_init (&domain->api_lock); domain->id =domain_id_alloc ();if (!domain->id) goto out_err;
@@ -2456,6 +2457,8 @@ static int amd_iommu_map_range(struct iommu_domain *dom, iova &=PAGE_MASK ; paddr &=PAGE_MASK ; +mutex_lock (&domain->api_lock); +for (i =0 ; i < npages; ++i) { ret =iommu_map_page (domain, iova, paddr, prot, PM_MAP_4k);if (ret)
@@ -2465,6 +2468,8 @@ static int amd_iommu_map_range(struct iommu_domain *dom, paddr +=PAGE_SIZE ; } +mutex_unlock (&domain->api_lock); +return 0 ; }
@@ -2477,12 +2482,16 @@ static void amd_iommu_unmap_range(struct iommu_domain *dom, iova &=PAGE_MASK ; +mutex_lock (&domain->api_lock); +for (i =0 ; i < npages; ++i) {iommu_unmap_page (domain, iova, PM_MAP_4k); iova +=PAGE_SIZE ; }iommu_flush_tlb_pde (domain); + +mutex_unlock (&domain->api_lock); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom,
Signature · 1 line
1.7.0.4
PATCH 3/12 x86/amd-iommu: Pt mode fix for domain_destroy
In reply to Joerg RoedelFrom: Chris Wright <[email protected]>
After a guest is shutdown, assigned devices are not properly returned to the pt domain. This can leave the device using stale cached IOMMU data, and result in a non-functional device after it's re-bound to the host driver. For example, I see this upon rebinding:
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0000 address=0x000000007e2a8000 flags=0x0050]
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0000 address=0x000000007e2a8040 flags=0x0050]
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0000 address=0x000000007e2a8080 flags=0x0050]
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0000 address=0x000000007e2a80c0 flags=0x0050] 0000:02:00.0: eth2: Detected Hardware Unit Hang:
...
The amd_iommu_destroy_domain() function calls do_detach() which doesn't reattach the pt domain to the device. Use __detach_device() instead.
--- arch/x86/kernel/amd_iommu.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/amd_iommu.c +1 -1
@@ -2298,7 +2298,7 @@ static void cleanup_domain(struct protection_domain *domain)list_for_each_entry_safe (dev_data, next, &domain->dev_list, list) {struct device *dev = dev_data->dev; -do_detach (dev); +__detach_device (dev);atomic_set (&dev_data->bind,0 ); }
Signature · 1 line
1.7.0.4
PATCH 4/12 x86/amd-iommu: Report errors in acpi parsing functions upstream
In reply to Joerg RoedelSince acpi_table_parse ignores the return values of the parsing function this patch introduces a workaround and reports these errors upstream via a global variable.
--- arch/x86/kernel/amd_iommu_init.c | 38 +++++++++++++++++++++++++++----------- 1 files changed, 27 insertions(+), 11 deletions(-)
arch/x86/kernel/amd_iommu_init.c +27 -11
@@ -138,9 +138,9 @@ int amd_iommus_present;bool amd_iommu_np_cache__read_mostly ;/* -* Set to true if ACPI table parsing and hardware intialization went properly +* The ACPI table parsing functions set this variable on an error */ -static bool amd_iommu_initialized; +static int __initdata amd_iommu_init_err;/* * List of protection domains - used during resume
@@ -391,9 +391,11 @@ static int __init find_last_devid_acpi(struct acpi_table_header *table)*/ for (i =0 ; i < table->length; ++i) checksum += p[i]; -if (checksum !=0 ) +if (checksum !=0 ) {/* ACPI table corrupt */ -return -ENODEV ; + amd_iommu_init_err = -ENODEV ; +return 0 ; + } p +=IVRS_HEADER_LENGTH ;
@@ -920,11 +922,16 @@ static int __init init_iommu_all(struct acpi_table_header *table) h->mmio_phys); iommu =kzalloc (sizeof (struct amd_iommu ),GFP_KERNEL ); -if (iommu ==NULL ) -return -ENOMEM ; +if (iommu ==NULL ) { + amd_iommu_init_err = -ENOMEM ; +return 0 ; + } + ret =init_iommu_one (iommu, h); -if (ret) -return ret; +if (ret) { + amd_iommu_init_err = ret; +return 0 ; + }break ;default :break ;
@@ -934,8 +941,6 @@ static int __init init_iommu_all(struct acpi_table_header *table) }WARN_ON (p != end); - amd_iommu_initialized = true; -return 0 ; }
@@ -1211,6 +1216,10 @@ static int __init amd_iommu_init(void)if (acpi_table_parse ("IVRS" , find_last_devid_acpi) !=0 )return -ENODEV ; + ret = amd_iommu_init_err; +if (ret) + goto out; + dev_table_size =tbl_size (DEV_TABLE_ENTRY_SIZE ); alias_table_size =tbl_size (ALIAS_TABLE_ENTRY_SIZE ); rlookup_table_size =tbl_size (RLOOKUP_TABLE_ENTRY_SIZE );
@@ -1270,12 +1279,19 @@ static int __init amd_iommu_init(void)if (acpi_table_parse ("IVRS" , init_iommu_all) !=0 ) goto free; -if (!amd_iommu_initialized) +if (amd_iommu_init_err) { + ret = amd_iommu_init_err; goto free; + }if (acpi_table_parse ("IVRS" , init_memory_definitions) !=0 ) goto free; +if (amd_iommu_init_err) { + ret = amd_iommu_init_err; + goto free; + } + ret =sysdev_class_register (&amd_iommu_sysdev_class);if (ret) goto free;
Signature · 1 line
1.7.0.4
PATCH 5/12 x86/amd-iommu: Use helper function to destroy domain
In reply to Joerg RoedelIn the amd_iommu_domain_destroy the protection_domain_free function is partly reimplemented. The 'partly' is the bug here because the domain is not deleted from the domain list. This results in use-after-free errors and data-corruption. Fix it by just using protection_domain_free instead.
--- arch/x86/kernel/amd_iommu.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)
arch/x86/kernel/amd_iommu.c +1 -3
@@ -2380,9 +2380,7 @@ static void amd_iommu_domain_destroy(struct iommu_domain *dom)free_pagetable (domain); -domain_id_free (domain->id); - -kfree (domain); +protection_domain_free (domain); dom->priv =NULL ; }
Signature · 1 line
1.7.0.4
PATCH 6/12 x86/amd-iommu: enable iommu before attaching devices
In reply to Joerg RoedelFrom: Chris Wright <[email protected]>
Hit another kdump problem as reported by Neil Horman. When initializaing the IOMMU, we attach devices to their domains before the IOMMU is fully (re)initialized. Attaching a device will issue some important invalidations. In the context of the newly kexec'd kdump kernel, the IOMMU may have stale cached data from the original kernel. Because we do the attach too early, the invalidation commands are placed in the new command buffer before the IOMMU is updated w/ that buffer. This leaves the stale entries in the kdump context and can renders device unusable. Simply enable the IOMMU before we do the attach.
--- arch/x86/kernel/amd_iommu_init.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kernel/amd_iommu_init.c +3 -2
@@ -1304,6 +1304,8 @@ static int __init amd_iommu_init(void)if (ret) goto free; +enable_iommus (); +if (iommu_pass_through) ret =amd_iommu_init_passthrough ();else
@@ -1316,8 +1318,6 @@ static int __init amd_iommu_init(void)amd_iommu_init_notifier (); -enable_iommus (); -if (iommu_pass_through) goto out;
@@ -1331,6 +1331,7 @@ out:return ret; free: +disable_iommus ();amd_iommu_uninit_devices ();
Signature · 1 line
1.7.0.4
PATCH 7/12 x86/amd-iommu: warn when issuing command to uninitialized cmd buffer
In reply to Joerg RoedelFrom: Chris Wright <[email protected]>
To catch future potential issues we can add a warning whenever we issue a command before the command buffer is fully initialized.
--- arch/x86/include/asm/amd_iommu_types.h | 1 + arch/x86/kernel/amd_iommu.c | 1 + arch/x86/kernel/amd_iommu_init.c | 5 +++-- 3 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/include/asm/amd_iommu_types.h +1 -0
@@ -141,6 +141,7 @@/* constants to configure the command buffer */ #define CMD_BUFFER_SIZE 8192 +#define CMD_BUFFER_UNINITIALIZED 1#define CMD_BUFFER_ENTRIES 512#define MMIO_CMD_SIZE_SHIFT 56#define MMIO_CMD_SIZE_512 (0x9ULL << MMIO_CMD_SIZE_SHIFT)
arch/x86/kernel/amd_iommu.c +1 -0
@@ -392,6 +392,7 @@ static int __iommu_queue_command(struct amd_iommu *iommu, struct iommu_cmd *cmd)u32 tail, head;u8 *target; +WARN_ON (iommu->cmd_buf_size &CMD_BUFFER_UNINITIALIZED ); tail =readl (iommu->mmio_base +MMIO_CMD_TAIL_OFFSET ); target = iommu->cmd_buf + tail;memcpy_toio (target, cmd,sizeof (*cmd));
arch/x86/kernel/amd_iommu_init.c +3 -2
@@ -438,7 +438,7 @@ static u8 * __init alloc_command_buffer(struct amd_iommu *iommu)if (cmd_buf ==NULL )return NULL ; - iommu->cmd_buf_size =CMD_BUFFER_SIZE ; + iommu->cmd_buf_size =CMD_BUFFER_SIZE |CMD_BUFFER_UNINITIALIZED ;return cmd_buf; }
@@ -474,12 +474,13 @@ static void iommu_enable_command_buffer(struct amd_iommu *iommu) &entry, sizeof(entry));amd_iommu_reset_cmd_buffer (iommu); + iommu->cmd_buf_size &= ~(CMD_BUFFER_UNINITIALIZED ); }static void __init free_command_buffer (struct amd_iommu *iommu) {free_pages ((unsigned long )iommu->cmd_buf, -get_order (iommu->cmd_buf_size)); +get_order (iommu->cmd_buf_size & ~(CMD_BUFFER_UNINITIALIZED ))); }/* allocates the memory where the IOMMU will log its events to */
Signature · 1 line
1.7.0.4
PATCH 8/12 Revert "x86: disable IOMMUs on kernel crash"
In reply to Joerg RoedelFrom: Chris Wright <[email protected]>
This effectively reverts commit 61d047be99757fd9b0af900d7abce9a13a337488.
Disabling the IOMMU can potetially allow DMA transactions to complete without being translated. Leave it enabled, and allow crash kernel to do the IOMMU reinitialization properly.
--- arch/x86/kernel/crash.c | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-)
arch/x86/kernel/crash.c +0 -6
@@ -27,7 +27,6 @@#include <asm/cpu.h> #include <asm/reboot.h> #include <asm/virtext.h> -#include <asm/x86_init.h> #if defined(CONFIG_SMP ) && defined(CONFIG_X86_LOCAL_APIC )
@@ -103,10 +102,5 @@ void native_machine_crash_shutdown(struct pt_regs *regs)#ifdef CONFIG_HPET_TIMER hpet_disable ();#endif - -#ifdef CONFIG_X86_64 - x86_platform.iommu_shutdown (); -#endif -crash_save_cpu (regs,safe_smp_processor_id ()); }
Signature · 1 line
1.7.0.4
PATCH 9/12 x86/amd-iommu: use for_each_pci_dev
In reply to Joerg RoedelFrom: Chris Wright <[email protected]>
Replace open coded version with for_each_pci_dev
--- arch/x86/kernel/amd_iommu.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/amd_iommu.c +1 -1
@@ -2187,7 +2187,7 @@ static void prealloc_protection_domains(void)struct dma_ops_domain *dma_dom;u16 devid; -while ((dev =pci_get_device (PCI_ANY_ID ,PCI_ANY_ID , dev)) !=NULL ) { +for_each_pci_dev (dev) {/* Do we handle this device? */ if (!check_device (&dev->dev))
Signature · 1 line
1.7.0.4
PATCH 10/12 x86/amd-iommu: Remove obsolete parameter documentation
In reply to Joerg RoedelSupport for the share and fullflush parameters was removed. Remove the documentation about them too.
--- Documentation/kernel-parameters.txt | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-)
Documentation/kernel-parameters.txt +0 -5
@@ -323,11 +323,6 @@ and is between 256 and 4096 characters. It is defined in the file amd_iommu= [HW,X86-84] Pass parameters to the AMD IOMMU driver in the system. Possible values are: - isolate - enable device isolation (each device, as far - as possible, will get its own protection - domain) [default] - share - put every device behind one IOMMU into the - same protection domain fullflush - enable flushing of IO/TLB entries when they are unmapped. Otherwise they are flushed before they will be reused, which
Signature · 1 line
1.7.0.4
PATCH 11/12 dma-debug: Cleanup for copy-loop in filter_write()
In reply to Joerg RoedelFrom: Dan Carpenter <[email protected]>
Earlier in this function we set the last byte of "buf" to NULL so we always hit the break statement and "i" is never equal to NAME_MAX_LEN. This patch doesn't change how the driver works but it silences a Smatch warning and it makes it clearer that we don't write past the end of the array.
--- lib/dma-debug.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
lib/dma-debug.c +1 -1
@@ -570,7 +570,7 @@ static ssize_t filter_write(struct file *file, const char __user *userbuf,* Now parse out the first token and use it as the name for the * driver to filter for. */ -for (i =0 ; i <NAME_MAX_LEN ; ++i) { +for (i =0 ; i <NAME_MAX_LEN -1 ; ++i) { current_driver_name[i] = buf[i];if (isspace (buf[i]) || buf[i] ==' ' || buf[i] ==0 )break ;
Signature · 1 line
1.7.0.4
PATCH 12/12 x86/gart: Disable GART explicitly before initialization
In reply to Joerg RoedelIf we boot into a crash-kernel the gart might still be enabled and its caches might be dirty. This can result in undefined behavior later. Fix it by explicitly disabling the gart hardware before initialization and flushing the caches after enablement.
--- arch/x86/kernel/aperture_64.c | 15 ++++++++++++++- arch/x86/kernel/pci-gart_64.c | 3 +++ 2 files changed, 17 insertions(+), 1 deletions(-)
arch/x86/kernel/aperture_64.c +14 -1
@@ -393,6 +393,7 @@ void __init gart_iommu_hole_init(void)for (i =0 ; i <ARRAY_SIZE (bus_dev_ranges); i++) {int bus;int dev_base, dev_limit; +u32 ctl; bus = bus_dev_ranges[i].bus; dev_base = bus_dev_ranges[i].dev_base;
@@ -406,7 +407,19 @@ void __init gart_iommu_hole_init(void) gart_iommu_aperture =1 ; x86_init.iommu.iommu_init = gart_iommu_init; - aper_order = (read_pci_config (bus, slot,3 ,AMD64_GARTAPERTURECTL ) >>1 ) &7 ; + ctl =read_pci_config (bus, slot,3 , +AMD64_GARTAPERTURECTL ); + +/* +* Before we do anything else disable the GART. It may +* still be enabled if we boot into a crash-kernel here. +* Reconfiguring the GART while it is enabled could have +* unknown side-effects. +*/ + ctl &= ~GARTEN ; +write_pci_config (bus, slot,3 ,AMD64_GARTAPERTURECTL , ctl); + + aper_order = (ctl >>1 ) &7 ; aper_size = (32 *1024 *1024 ) << aper_order; aper_base =read_pci_config (bus, slot,3 ,AMD64_GARTAPERTUREBASE ) &0x7fff ; aper_base <<=25 ;
arch/x86/kernel/pci-gart_64.c +3 -0
@@ -564,6 +564,9 @@ static void enable_gart_translations(void)enable_gart_translation (dev,__pa (agp_gatt_table)); } + +/* Flush the GART-TLB to remove stale entries */ +k8_flush_garts (); }/*
Signature · 1 line
1.7.0.4
Quoted 41 lines · Joerg Roedel
Hi Ingo,
The following changes since commit
2eaa9cfdf33b8d7fb7aff27792192e0019ae8fc6:
Linus Torvalds (1):Linux 2.6.34-rc3are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/joro/linux-2.6-iommu.gitiommu/fixesChris Wright (5):
x86/amd-iommu: Pt mode fix fordomain_destroyx86/amd-iommu: enable iommu before attaching devices x86/amd-iommu: warn when issuing command to uninitialized cmd buffer Revert "x86: disable IOMMUs on kernel crash" x86/amd-iommu: usefor_each_pci_devDan Carpenter (1):
dma-debug: Cleanup for copy-loop infilter_write()Joerg Roedel (6):
x86/amd-iommu: Protect IOMMU-API map/unmap path x86/amd-iommu: Report errors in acpi parsing functions upstream x86/amd-iommu: Use helper function to destroy domain x86/amd-iommu: Remove obsolete parameter documentation Merge branch 'amd-iommu/fixes' into iommu/fixes x86/gart: Disable GART explicitly before initializationJulia Lawall (1):
x86/amd-iommu: Remove double NULL check incheck_deviceDocumentation/kernel-parameters.txt | 5 --- arch/x86/include/asm/amd_iommu_types.h | 3 ++ arch/x86/kernel/amd_iommu.c | 20 +++++++++---- arch/x86/kernel/amd_iommu_init.c | 48 ++++++++++++++++++++++---------- arch/x86/kernel/aperture_64.c | 15 +++++++++- arch/x86/kernel/crash.c | 6 ---- arch/x86/kernel/pci-gart_64.c | 3 ++ lib/dma-debug.c | 2 +- 8 files changed, 68 insertions(+), 34 deletions(-)
Quoted 3 lines
This pull request includes fixes that did not made it upstream during the merge window, fixes for recently discovered problems with kdump and amd-iommu enabled and a small documentation update. Please pull.
Pulled, thanks a lot Joerg!
Ingo