IOMMU updates for v2.6.30
Subsystems (6)
Linus,
Please pull the latest iommu-for-linus git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git iommu-for-linus
Highlights:
- dma_mapping_ops cleanups and cross-arch unification
- CONFIG_DMA_API_DEBUG debugging helper (default disabled), to debug various buggy uses of DMA APIs, even on systems without a hardware IOMMU.
Known risks:
- CONFIG_DMA_API_DEBUG=y might emit warnings about DMA API bugs/problems it finds. Common ones were fixed (they were almost always driver bugs), but exposure to rare and old hardware was relatively low. There is at most a single warning per bootup. Fedora rawhide has been carrying (and enabling) this debug facility for some time.
- No known regressions.
Thanks,
Ingo
------------------>
David Woodhouse (2):
dma-debug: add function to dump dma mappings
dma-debug: print stacktrace of mapping path on unmap error
FUJITA Tomonori (31):
add map/unmap_single_attr and map/unmap_sg_attr to struct dma_mapping_ops add dma_mapping_ops for SBA IOMMU
add dma_mapping_ops for SWIOTLB and SBA IOMMU
add dma_mapping_ops for intel-iommu
add dma_mapping_ops for SGI Altix
add dma_mapping_ops for SWIOTLB
set up dma_ops appropriately
convert the DMA API to use dma_ops
remove dma operations in struct ia64_machine_vector
make sn DMA mapping functions static
add dma_get_ops to struct ia64_machine_vector
remove hwsw_dma_ops
make sba DMA mapping functions static
x86: add map_page and unmap_page to struct dma_mapping_ops x86, swiotlb: add map_page and unmap_page
x86, gart: add map_page and unmap_page
x86, calgary: add map_page and unmap_page
x86, AMD IOMMU: add map_page and unmap_page
intel-iommu: add map_page and unmap_page
x86, pci-nommu: add map_page
x86: remove map_single and unmap_single in struct dma_mapping_ops
generic: add common struct for dma map operations
x86, ia64: convert to use generic dma_map_ops struct
x86, ia64: remove duplicated swiotlb code
convert to use generic dma_map_ops struct, cleanup
IA64: fix compile error on IA64_DIG_VTD
IA64: fix swiotlb alloc_coherent for non DMA_64BIT_MASK devices
IA64: fix VT-d dma_mapping_error
intel-iommu: make dma mapping functions static
AMD IOMMU: remove unnecessary ifdef
dma-debug: warn of unmapping an invalid dma address
Hannes Eder (1):
IOMMU-API: use ANSI style function declaration for 'iommu_found'
Ian Campbell (3):
swiotlb: range_needs_mapping should take a physical address.
swiotlb: do not use sg_virt()
x86: rename arch/x86/kernel/pci-swiotlb_64.c => pci-swiotlb.c
Ingo Molnar (1):
dma-debug: fix dma_debug_add_bus() definition for !CONFIG_DMA_API_DEBUG
Joerg Roedel (20):
dma-debug: add Kconfig entry
dma-debug: add header file and core data structures
dma-debug: add hash functions for dma_debug_entries
dma-debug: add allocator code
dma-debug: add initialization code
dma-debug: add kernel command line parameters
dma-debug: add debugfs interface
dma-debug: add core checking functions
dma-debug: add checking for map/unmap_page/single
dma-debug: add add checking for map/unmap_sg
dma-debug: add checking for [alloc|free]_coherent
dma-debug: add checks for sync_single_*
dma-debug: add checks for sync_single_range_*
dma-debug: add checks for sync_single_sg_*
dma-debug: x86 architecture bindings
dma-debug: Documentation update
dma-debug: add checks for kernel text and rodata
dma-debug: add a check dma memory leaks dma-debug/x86: register pci bus for dma-debug leak detection
dma-debug: make memory range checks more consistent
Tony Luck (1):
ia64: fix section mismatch swiotlb_dma_init -> swiotlb_init
Yasunori Goto (1):
IA64: fix swiotlb alloc_coherent for non DMA_64BIT_MASK devices, fix
Documentation/DMA-API.txt | 106 +++
Documentation/kernel-parameters.txt | 10 +
arch/Kconfig | 2 +
arch/ia64/dig/Makefile | 4 +-
arch/ia64/dig/dig_vtd_iommu.c | 59 --
arch/ia64/hp/common/hwsw_iommu.c | 165 +---
arch/ia64/hp/common/sba_iommu.c | 79 ++-
arch/ia64/include/asm/dma-mapping.h | 194 +++--
arch/ia64/include/asm/machvec.h | 102 +--
arch/ia64/include/asm/machvec_dig_vtd.h | 20 -
arch/ia64/include/asm/machvec_hpzx1.h | 23 +-
arch/ia64/include/asm/machvec_hpzx1_swiotlb.h | 27 +-
arch/ia64/include/asm/machvec_sn2.h | 27 +-
arch/ia64/kernel/Makefile | 4 +-
arch/ia64/kernel/dma-mapping.c | 13 +
arch/ia64/kernel/machvec.c | 8 +-
arch/ia64/kernel/pci-dma.c | 46 +-
arch/ia64/kernel/pci-swiotlb.c | 30 +-
arch/ia64/sn/pci/pci_dma.c | 99 ++-
arch/x86/Kconfig | 1 +
arch/x86/include/asm/device.h | 2 +-
arch/x86/include/asm/dma-mapping.h | 188 ++--
arch/x86/include/asm/iommu.h | 2 +-
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/amd_iommu.c | 26 +-
arch/x86/kernel/pci-calgary_64.c | 38 +-
arch/x86/kernel/pci-dma.c | 14 +-
arch/x86/kernel/pci-gart_64.c | 34 +-
arch/x86/kernel/pci-nommu.c | 21 +-
.../x86/kernel/{pci-swiotlb_64.c => pci-swiotlb.c} | 19 +-
drivers/base/iommu.c | 2 +-
drivers/pci/intel-iommu.c | 50 +-
include/linux/dma-debug.h | 174 ++++
include/linux/dma-mapping.h | 48 +
include/linux/intel-iommu.h | 7 -
include/linux/swiotlb.h | 41 +-
lib/Kconfig.debug | 11 +
lib/Makefile | 2 +
lib/dma-debug.c | 955 ++++++++++++++++++++
lib/swiotlb.c | 88 +-
40 files changed, 1899 insertions(+), 844 deletions(-)
delete mode 100644 arch/ia64/dig/dig_vtd_iommu.c
create mode 100644 arch/ia64/kernel/dma-mapping.c
rename arch/x86/kernel/{pci-swiotlb_64.c => pci-swiotlb.c} (81%)
create mode 100644 include/linux/dma-debug.h
create mode 100644 lib/dma-debug.cDocumentation/DMA-API.txt +106 -0
@@ -609,3 +609,109 @@ size is the size (and should be a page-sized multiple). The return value will be either a pointer to the processor virtual address of the memory, or an error (via PTR_ERR()) if any part of the region is occupied. + +Part III - Debug drivers use of the DMA-API +------------------------------------------- + +The DMA-API as described above as some constraints. DMA addresses must be +released with the corresponding function with the same size for example. With +the advent of hardware IOMMUs it becomes more and more important that drivers +do not violate those constraints. In the worst case such a violation can +result in data corruption up to destroyed filesystems. + +To debug drivers and find bugs in the usage of the DMA-API checking code can +be compiled into the kernel which will tell the developer about those +violations. If your architecture supports it you can select the "Enable +debugging of DMA-API usage" option in your kernel configuration. Enabling this +option has a performance impact. Do not enable it in production kernels. + +If you boot the resulting kernel will contain code which does some bookkeeping +about what DMA memory was allocated for which device. If this code detects an +error it prints a warning message with some details into your kernel log. An +example warning message may look like this: + +------------[ cut here ]------------ +WARNING: at /data2/repos/linux-2.6-iommu/lib/dma-debug.c:448 + check_unmap+0x203/0x490() +Hardware name: +forcedeth 0000:00:08.0: DMA-API: device driver frees DMA memory with wrong + function [device address=0x00000000640444be] [size=66 bytes] [mapped as +single] [unmapped as page] +Modules linked in: nfsd exportfs bridge stp llc r8169 +Pid: 0, comm: swapper Tainted: G W 2.6.28-dmatest-09289-g8bb99c0 #1 +Call Trace: + <IRQ> [<ffffffff80240b22>] warn_slowpath+0xf2/0x130 + [<ffffffff80647b70>] _spin_unlock+0x10/0x30 + [<ffffffff80537e75>] usb_hcd_link_urb_to_ep+0x75/0xc0 + [<ffffffff80647c22>] _spin_unlock_irqrestore+0x12/0x40 + [<ffffffff8055347f>] ohci_urb_enqueue+0x19f/0x7c0 + [<ffffffff80252f96>] queue_work+0x56/0x60 + [<ffffffff80237e10>] enqueue_task_fair+0x20/0x50 + [<ffffffff80539279>] usb_hcd_submit_urb+0x379/0xbc0 + [<ffffffff803b78c3>] cpumask_next_and+0x23/0x40 + [<ffffffff80235177>] find_busiest_group+0x207/0x8a0 + [<ffffffff8064784f>] _spin_lock_irqsave+0x1f/0x50 + [<ffffffff803c7ea3>] check_unmap+0x203/0x490 + [<ffffffff803c8259>] debug_dma_unmap_page+0x49/0x50 + [<ffffffff80485f26>] nv_tx_done_optimized+0xc6/0x2c0 + [<ffffffff80486c13>] nv_nic_irq_optimized+0x73/0x2b0 + [<ffffffff8026df84>] handle_IRQ_event+0x34/0x70 + [<ffffffff8026ffe9>] handle_edge_irq+0xc9/0x150 + [<ffffffff8020e3ab>] do_IRQ+0xcb/0x1c0 + [<ffffffff8020c093>] ret_from_intr+0x0/0xa + <EOI> <4>---[ end trace f6435a98e2a38c0e ]--- + +The driver developer can find the driver and the device including a stacktrace +of the DMA-API call which caused this warning. + +Per default only the first error will result in a warning message. All other +errors will only silently counted. This limitation exist to prevent the code +from flooding your kernel log. To support debugging a device driver this can +be disabled via debugfs. See the debugfs interface documentation below for +details. + +The debugfs directory for the DMA-API debugging code is called dma-api/. In +this directory the following files can currently be found: + + dma-api/all_errors This file contains a numeric value. If this + value is not equal to zero the debugging code + will print a warning for every error it finds + into the kernel log. Be carefull with this + option. It can easily flood your logs. + + dma-api/disabled This read-only file contains the character 'Y' + if the debugging code is disabled. This can + happen when it runs out of memory or if it was + disabled at boot time + + dma-api/error_count This file is read-only and shows the total + numbers of errors found. + + dma-api/num_errors The number in this file shows how many + warnings will be printed to the kernel log + before it stops. This number is initialized to + one at system boot and be set by writing into + this file + + dma-api/min_free_entries + This read-only file can be read to get the + minimum number of free dma_debug_entries the + allocator has ever seen. If this value goes + down to zero the code will disable itself + because it is not longer reliable. + + dma-api/num_free_entries + The current number of free dma_debug_entries + in the allocator. + +If you have this code compiled into your kernel it will be enabled by default. +If you want to boot without the bookkeeping anyway you can provide +'dma_debug=off' as a boot parameter. This will disable DMA-API debugging. +Notice that you can not enable it again at runtime. You have to reboot to do +so. + +When the code disables itself at runtime this is most likely because it ran +out of dma_debug_entries. These entries are preallocated at boot. The number +of preallocated entries is defined per architecture. If it is too low for you +boot with 'dma_debug_entries=<your_desired_number>' to overwrite the +architectural default.
Documentation/kernel-parameters.txt +10 -0
@@ -492,6 +492,16 @@ and is between 256 and 4096 characters. It is defined in the file Range: 0 - 8192 Default: 64 + dma_debug=off If the kernel is compiled with DMA_API_DEBUG support + this option disables the debugging code at boot. + + dma_debug_entries=<number> + This option allows to tune the number of preallocated + entries for DMA-API debugging code. One entry is + required per DMA-API allocation. Use this if the + DMA-API debugging code disables itself because the + architectural default is too low. + hpet= [X86-32,HPET] option to control HPET usage Format: { enable (default) | disable | force | verbose }
arch/Kconfig +2 -0
@@ -106,3 +106,5 @@ config HAVE_CLK The <linux/clk.h> calls support software clock gating and thus are a key power management tool on many systems. +config HAVE_DMA_API_DEBUG + bool
arch/ia64/dig/Makefile +2 -2
@@ -7,8 +7,8 @@ obj-y := setup.o ifeq ($(CONFIG_DMAR), y) -obj-$(CONFIG_IA64_GENERIC) += machvec.o machvec_vtd.o dig_vtd_iommu.o +obj-$(CONFIG_IA64_GENERIC) += machvec.o machvec_vtd.o else obj-$(CONFIG_IA64_GENERIC) += machvec.o endif -obj-$(CONFIG_IA64_DIG_VTD) += dig_vtd_iommu.o +
arch/ia64/dig/dig_vtd_iommu.c deleted +0 -59
59 lines not on this page · raw message
arch/ia64/hp/common/hwsw_iommu.c +14 -151
188 lines not on this page · raw message
arch/ia64/hp/common/sba_iommu.c +56 -23
179 lines not on this page · raw message
arch/ia64/include/asm/dma-mapping.h +110 -84
227 lines not on this page · raw message
arch/ia64/include/asm/machvec.h +10 -92
155 lines not on this page · raw message
arch/ia64/include/asm/machvec_dig_vtd.h +0 -20
@@ -2,14 +2,6 @@#define _ASM_IA64_MACHVEC_DIG_VTD_hextern ia64_mv_setup_t dig_setup; -extern ia64_mv_dma_alloc_coherent vtd_alloc_coherent; -extern ia64_mv_dma_free_coherent vtd_free_coherent; -extern ia64_mv_dma_map_single_attrs vtd_map_single_attrs; -extern ia64_mv_dma_unmap_single_attrs vtd_unmap_single_attrs; -extern ia64_mv_dma_map_sg_attrs vtd_map_sg_attrs; -extern ia64_mv_dma_unmap_sg_attrs vtd_unmap_sg_attrs; -extern ia64_mv_dma_supported iommu_dma_supported; -extern ia64_mv_dma_mapping_error vtd_dma_mapping_error;extern ia64_mv_dma_init pci_iommu_alloc;/*
@@ -22,17 +14,5 @@ extern ia64_mv_dma_init pci_iommu_alloc;#define platform_name "dig_vtd"#define platform_setup dig_setup#define platform_dma_init pci_iommu_alloc -#define platform_dma_alloc_coherent vtd_alloc_coherent -#define platform_dma_free_coherent vtd_free_coherent -#define platform_dma_map_single_attrs vtd_map_single_attrs -#define platform_dma_unmap_single_attrs vtd_unmap_single_attrs -#define platform_dma_map_sg_attrs vtd_map_sg_attrs -#define platform_dma_unmap_sg_attrs vtd_unmap_sg_attrs -#define platform_dma_sync_single_for_cpu machvec_dma_sync_single -#define platform_dma_sync_sg_for_cpu machvec_dma_sync_sg -#define platform_dma_sync_single_for_device machvec_dma_sync_single -#define platform_dma_sync_sg_for_device machvec_dma_sync_sg -#define platform_dma_supported iommu_dma_supported -#define platform_dma_mapping_error vtd_dma_mapping_error#endif /* _ASM_IA64_MACHVEC_DIG_VTD_h */
arch/ia64/include/asm/machvec_hpzx1.h +2 -21
@@ -2,14 +2,7 @@#define _ASM_IA64_MACHVEC_HPZX1_hextern ia64_mv_setup_t dig_setup; -extern ia64_mv_dma_alloc_coherent sba_alloc_coherent; -extern ia64_mv_dma_free_coherent sba_free_coherent; -extern ia64_mv_dma_map_single_attrs sba_map_single_attrs; -extern ia64_mv_dma_unmap_single_attrs sba_unmap_single_attrs; -extern ia64_mv_dma_map_sg_attrs sba_map_sg_attrs; -extern ia64_mv_dma_unmap_sg_attrs sba_unmap_sg_attrs; -extern ia64_mv_dma_supported sba_dma_supported; -extern ia64_mv_dma_mapping_error sba_dma_mapping_error; +extern ia64_mv_dma_init sba_dma_init;/* * This stuff has dual use!
@@ -20,18 +13,6 @@ extern ia64_mv_dma_mapping_error sba_dma_mapping_error;*/ #define platform_name "hpzx1"#define platform_setup dig_setup -#define platform_dma_init machvec_noop -#define platform_dma_alloc_coherent sba_alloc_coherent -#define platform_dma_free_coherent sba_free_coherent -#define platform_dma_map_single_attrs sba_map_single_attrs -#define platform_dma_unmap_single_attrs sba_unmap_single_attrs -#define platform_dma_map_sg_attrs sba_map_sg_attrs -#define platform_dma_unmap_sg_attrs sba_unmap_sg_attrs -#define platform_dma_sync_single_for_cpu machvec_dma_sync_single -#define platform_dma_sync_sg_for_cpu machvec_dma_sync_sg -#define platform_dma_sync_single_for_device machvec_dma_sync_single -#define platform_dma_sync_sg_for_device machvec_dma_sync_sg -#define platform_dma_supported sba_dma_supported -#define platform_dma_mapping_error sba_dma_mapping_error +#define platform_dma_init sba_dma_init#endif /* _ASM_IA64_MACHVEC_HPZX1_h */
arch/ia64/include/asm/machvec_hpzx1_swiotlb.h +2 -25
@@ -2,18 +2,7 @@#define _ASM_IA64_MACHVEC_HPZX1_SWIOTLB_hextern ia64_mv_setup_t dig_setup; -extern ia64_mv_dma_alloc_coherent hwsw_alloc_coherent; -extern ia64_mv_dma_free_coherent hwsw_free_coherent; -extern ia64_mv_dma_map_single_attrs hwsw_map_single_attrs; -extern ia64_mv_dma_unmap_single_attrs hwsw_unmap_single_attrs; -extern ia64_mv_dma_map_sg_attrs hwsw_map_sg_attrs; -extern ia64_mv_dma_unmap_sg_attrs hwsw_unmap_sg_attrs; -extern ia64_mv_dma_supported hwsw_dma_supported; -extern ia64_mv_dma_mapping_error hwsw_dma_mapping_error; -extern ia64_mv_dma_sync_single_for_cpu hwsw_sync_single_for_cpu; -extern ia64_mv_dma_sync_sg_for_cpu hwsw_sync_sg_for_cpu; -extern ia64_mv_dma_sync_single_for_device hwsw_sync_single_for_device; -extern ia64_mv_dma_sync_sg_for_device hwsw_sync_sg_for_device; +extern ia64_mv_dma_get_ops hwsw_dma_get_ops;/* * This stuff has dual use!
@@ -23,20 +12,8 @@ extern ia64_mv_dma_sync_sg_for_device hwsw_sync_sg_for_device;* the macros are used directly. */ #define platform_name "hpzx1_swiotlb" -#define platform_setup dig_setup#define platform_dma_init machvec_noop -#define platform_dma_alloc_coherent hwsw_alloc_coherent -#define platform_dma_free_coherent hwsw_free_coherent -#define platform_dma_map_single_attrs hwsw_map_single_attrs -#define platform_dma_unmap_single_attrs hwsw_unmap_single_attrs -#define platform_dma_map_sg_attrs hwsw_map_sg_attrs -#define platform_dma_unmap_sg_attrs hwsw_unmap_sg_attrs -#define platform_dma_supported hwsw_dma_supported -#define platform_dma_mapping_error hwsw_dma_mapping_error -#define platform_dma_sync_single_for_cpu hwsw_sync_single_for_cpu -#define platform_dma_sync_sg_for_cpu hwsw_sync_sg_for_cpu -#define platform_dma_sync_single_for_device hwsw_sync_single_for_device -#define platform_dma_sync_sg_for_device hwsw_sync_sg_for_device +#define platform_dma_get_ops hwsw_dma_get_ops#endif /* _ASM_IA64_MACHVEC_HPZX1_SWIOTLB_h */
arch/ia64/include/asm/machvec_sn2.h +2 -25
@@ -55,19 +55,8 @@ extern ia64_mv_readb_t __sn_readb_relaxed;extern ia64_mv_readw_t __sn_readw_relaxed;extern ia64_mv_readl_t __sn_readl_relaxed;extern ia64_mv_readq_t __sn_readq_relaxed; -extern ia64_mv_dma_alloc_coherent sn_dma_alloc_coherent; -extern ia64_mv_dma_free_coherent sn_dma_free_coherent; -extern ia64_mv_dma_map_single_attrs sn_dma_map_single_attrs; -extern ia64_mv_dma_unmap_single_attrs sn_dma_unmap_single_attrs; -extern ia64_mv_dma_map_sg_attrs sn_dma_map_sg_attrs; -extern ia64_mv_dma_unmap_sg_attrs sn_dma_unmap_sg_attrs; -extern ia64_mv_dma_sync_single_for_cpu sn_dma_sync_single_for_cpu; -extern ia64_mv_dma_sync_sg_for_cpu sn_dma_sync_sg_for_cpu; -extern ia64_mv_dma_sync_single_for_device sn_dma_sync_single_for_device; -extern ia64_mv_dma_sync_sg_for_device sn_dma_sync_sg_for_device; -extern ia64_mv_dma_mapping_error sn_dma_mapping_error; -extern ia64_mv_dma_supported sn_dma_supported;extern ia64_mv_dma_get_required_mask sn_dma_get_required_mask; +extern ia64_mv_dma_init sn_dma_init;extern ia64_mv_migrate_t sn_migrate;extern ia64_mv_kernel_launch_event_t sn_kernel_launch_event;extern ia64_mv_setup_msi_irq_t sn_setup_msi_irq;
@@ -111,20 +100,8 @@ extern ia64_mv_pci_fixup_bus_t sn_pci_fixup_bus;#define platform_pci_get_legacy_mem sn_pci_get_legacy_mem#define platform_pci_legacy_read sn_pci_legacy_read#define platform_pci_legacy_write sn_pci_legacy_write -#define platform_dma_init machvec_noop -#define platform_dma_alloc_coherent sn_dma_alloc_coherent -#define platform_dma_free_coherent sn_dma_free_coherent -#define platform_dma_map_single_attrs sn_dma_map_single_attrs -#define platform_dma_unmap_single_attrs sn_dma_unmap_single_attrs -#define platform_dma_map_sg_attrs sn_dma_map_sg_attrs -#define platform_dma_unmap_sg_attrs sn_dma_unmap_sg_attrs -#define platform_dma_sync_single_for_cpu sn_dma_sync_single_for_cpu -#define platform_dma_sync_sg_for_cpu sn_dma_sync_sg_for_cpu -#define platform_dma_sync_single_for_device sn_dma_sync_single_for_device -#define platform_dma_sync_sg_for_device sn_dma_sync_sg_for_device -#define platform_dma_mapping_error sn_dma_mapping_error -#define platform_dma_supported sn_dma_supported#define platform_dma_get_required_mask sn_dma_get_required_mask +#define platform_dma_init sn_dma_init#define platform_migrate sn_migrate#define platform_kernel_launch_event sn_kernel_launch_event#ifdef CONFIG_PCI_MSI
arch/ia64/kernel/Makefile +1 -3
@@ -7,7 +7,7 @@ extra-y := head.o init_task.o vmlinux.lds obj-y := acpi.o entry.o efi.o efi_stub.o gate-data.o fsys.o ia64_ksyms.o irq.o irq_ia64.o \ irq_lsapic.o ivt.o machvec.o pal.o patch.o process.o perfmon.o ptrace.o sal.o \ salinfo.o setup.o signal.o sys_ia64.o time.o traps.o unaligned.o \ - unwind.o mca.o mca_asm.o topology.o + unwind.o mca.o mca_asm.o topology.o dma-mapping.o obj-$(CONFIG_IA64_BRL_EMU) += brl_emu.o obj-$(CONFIG_IA64_GENERIC) += acpi-ext.o
@@ -43,9 +43,7 @@ ifneq ($(CONFIG_IA64_ESI),) obj-y += esi_stub.o # must be in kernel proper endif obj-$(CONFIG_DMAR) += pci-dma.o -ifeq ($(CONFIG_DMAR), y) obj-$(CONFIG_SWIOTLB) += pci-swiotlb.o -endif # The gate DSO image is built using a special linker script. targets += gate.so gate-syms.o
arch/ia64/kernel/dma-mapping.c new file +13 -0
@@ -0,0 +1,13 @@ +#include <linux/dma-mapping.h> + +/* Set this to 1 if there is a HW IOMMU in the system */ +int iommu_detected__read_mostly ; + +struct dma_map_ops *dma_ops; +EXPORT_SYMBOL (dma_ops); + +struct dma_map_ops *dma_get_ops (struct device *dev) +{ +return dma_ops; +} +EXPORT_SYMBOL (dma_get_ops);
arch/ia64/kernel/machvec.c +5 -3
@@ -1,5 +1,5 @@#include <linux/module.h> - +#include <linux/dma-mapping.h> #include <asm/machvec.h> #include <asm/system.h>
@@ -75,14 +75,16 @@ machvec_timer_interrupt (int irq, void *dev_id)EXPORT_SYMBOL (machvec_timer_interrupt);void -machvec_dma_sync_single (struct device *hwdev,dma_addr_t dma_handle,size_t size,int dir) +machvec_dma_sync_single (struct device *hwdev,dma_addr_t dma_handle,size_t size, +enum dma_data_direction dir) {mb (); }EXPORT_SYMBOL (machvec_dma_sync_single);void -machvec_dma_sync_sg (struct device *hwdev,struct scatterlist *sg,int n,int dir) +machvec_dma_sync_sg (struct device *hwdev,struct scatterlist *sg,int n, +enum dma_data_direction dir) {mb (); }
arch/ia64/kernel/pci-dma.c +25 -21
71 lines not on this page · raw message
arch/ia64/kernel/pci-swiotlb.c +22 -8
@@ -13,23 +13,37 @@int swiotlb__read_mostly ;EXPORT_SYMBOL (swiotlb); -struct dma_mapping_ops swiotlb_dma_ops = { - .mapping_error = swiotlb_dma_mapping_error, - .alloc_coherent = swiotlb_alloc_coherent, +static void *ia64_swiotlb_alloc_coherent (struct device *dev,size_t size, +dma_addr_t *dma_handle,gfp_t gfp) +{ +if (dev->coherent_dma_mask !=DMA_64BIT_MASK ) + gfp |=GFP_DMA ; +return swiotlb_alloc_coherent (dev, size, dma_handle, gfp); +} + +struct dma_map_ops swiotlb_dma_ops = { + .alloc_coherent = ia64_swiotlb_alloc_coherent, .free_coherent = swiotlb_free_coherent, - .map_single = swiotlb_map_single, - .unmap_single = swiotlb_unmap_single, + .map_page = swiotlb_map_page, + .unmap_page = swiotlb_unmap_page, + .map_sg = swiotlb_map_sg_attrs, + .unmap_sg = swiotlb_unmap_sg_attrs, .sync_single_for_cpu = swiotlb_sync_single_for_cpu, .sync_single_for_device = swiotlb_sync_single_for_device, .sync_single_range_for_cpu = swiotlb_sync_single_range_for_cpu, .sync_single_range_for_device = swiotlb_sync_single_range_for_device, .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu, .sync_sg_for_device = swiotlb_sync_sg_for_device, - .map_sg = swiotlb_map_sg, - .unmap_sg = swiotlb_unmap_sg, - .dma_supported_op = swiotlb_dma_supported, + .dma_supported = swiotlb_dma_supported, + .mapping_error = swiotlb_dma_mapping_error, }; +void __init swiotlb_dma_init (void ) +{ + dma_ops = &swiotlb_dma_ops; +swiotlb_init (); +} +void __init pci_swiotlb_init (void ) {if (!iommu_detected) {
arch/ia64/sn/pci/pci_dma.c +55 -44
228 lines not on this page · raw message
arch/x86/Kconfig +1 -0
@@ -40,6 +40,7 @@ config X86 select HAVE_GENERIC_DMA_COHERENT if X86_32 select HAVE_EFFICIENT_UNALIGNED_ACCESS select USER_STACKTRACE_SUPPORT + select HAVE_DMA_API_DEBUG select HAVE_KERNEL_GZIP select HAVE_KERNEL_BZIP2 select HAVE_KERNEL_LZMA
arch/x86/include/asm/device.h +1 -1
@@ -6,7 +6,7 @@ struct dev_archdata {void *acpi_handle;#endif #ifdef CONFIG_X86_64 -struct dma_mapping_ops *dma_ops; +struct dma_map_ops *dma_ops;#endif #ifdef CONFIG_DMAR void *iommu;/* hook for IOMMU specific extension */
arch/x86/include/asm/dma-mapping.h +93 -95
317 lines not on this page · raw message
arch/x86/include/asm/iommu.h +1 -1
@@ -3,7 +3,7 @@extern void pci_iommu_shutdown (void );extern void no_iommu_init (void ); -extern struct dma_mapping_ops nommu_dma_ops; +extern struct dma_map_ops nommu_dma_ops;extern int force_iommu, no_iommu;extern int iommu_detected;
arch/x86/kernel/Makefile +1 -1
@@ -106,7 +106,7 @@ obj-$(CONFIG_MICROCODE) += microcode.o obj-$(CONFIG_X86_CHECK_BIOS_CORRUPTION) += check.o -obj-$(CONFIG_SWIOTLB) += pci-swiotlb_64.o # NB rename without _64 +obj-$(CONFIG_SWIOTLB) += pci-swiotlb.o ### # 64 bit specific files
arch/x86/kernel/amd_iommu.c +15 -11
@@ -22,10 +22,9 @@#include <linux/bitops.h> #include <linux/debugfs.h> #include <linux/scatterlist.h> +#include <linux/dma-mapping.h> #include <linux/iommu-helper.h> -#ifdef CONFIG_IOMMU_API #include <linux/iommu.h> -#endif #include <asm/proto.h> #include <asm/iommu.h> #include <asm/gart.h>
@@ -1297,8 +1296,10 @@ static void __unmap_single(struct amd_iommu *iommu,/* * The exported map_single function for dma_ops. */ -static dma_addr_t map_single (struct device *dev,phys_addr_t paddr, -size_t size,int dir) +static dma_addr_t map_page (struct device *dev,struct page *page, +unsigned long offset,size_t size, +enum dma_data_direction dir, +struct dma_attrs *attrs) {unsigned long flags;struct amd_iommu *iommu;
@@ -1306,6 +1307,7 @@ static dma_addr_t map_single(struct device *dev, phys_addr_t paddr,u16 devid;dma_addr_t addr;u64 dma_mask; +phys_addr_t paddr =page_to_phys (page) + offset;INC_STATS_COUNTER (cnt_map_single);
@@ -1340,8 +1342,8 @@ out:/* * The exported unmap_single function for dma_ops. */ -static void unmap_single (struct device *dev,dma_addr_t dma_addr, -size_t size,int dir) +static void unmap_page (struct device *dev,dma_addr_t dma_addr,size_t size, +enum dma_data_direction dir,struct dma_attrs *attrs) {unsigned long flags;struct amd_iommu *iommu;
@@ -1390,7 +1392,8 @@ static int map_sg_no_iommu(struct device *dev, struct scatterlist *sglist,* lists). */ static int map_sg (struct device *dev,struct scatterlist *sglist, -int nelems,int dir) +int nelems,enum dma_data_direction dir, +struct dma_attrs *attrs) {unsigned long flags;struct amd_iommu *iommu;
@@ -1457,7 +1460,8 @@ unmap:* lists). */ static void unmap_sg (struct device *dev,struct scatterlist *sglist, -int nelems,int dir) +int nelems,enum dma_data_direction dir, +struct dma_attrs *attrs) {unsigned long flags;struct amd_iommu *iommu;
@@ -1644,11 +1648,11 @@ static void prealloc_protection_domains(void) } } -static struct dma_mapping_ops amd_iommu_dma_ops = { +static struct dma_map_ops amd_iommu_dma_ops = { .alloc_coherent = alloc_coherent, .free_coherent = free_coherent, - .map_single = map_single, - .unmap_single = unmap_single, + .map_page = map_page, + .unmap_page = unmap_page, .map_sg = map_sg, .unmap_sg = unmap_sg, .dma_supported = amd_iommu_dma_supported,
arch/x86/kernel/pci-calgary_64.c +21 -17
@@ -380,8 +380,9 @@ static inline struct iommu_table *find_iommu_table(struct device *dev)return tbl; } -static void calgary_unmap_sg (struct device *dev, -struct scatterlist *sglist,int nelems,int direction) +static void calgary_unmap_sg (struct device *dev,struct scatterlist *sglist, +int nelems,enum dma_data_direction dir, +struct dma_attrs *attrs) {struct iommu_table *tbl =find_iommu_table (dev);struct scatterlist *s;
@@ -404,7 +405,8 @@ static void calgary_unmap_sg(struct device *dev, }static int calgary_map_sg (struct device *dev,struct scatterlist *sg, -int nelems,int direction) +int nelems,enum dma_data_direction dir, +struct dma_attrs *attrs) {struct iommu_table *tbl =find_iommu_table (dev);struct scatterlist *s;
@@ -429,15 +431,14 @@ static int calgary_map_sg(struct device *dev, struct scatterlist *sg, s->dma_address = (entry <<PAGE_SHIFT ) | s->offset;/* insert into HW table */ -tce_build (tbl, entry, npages, vaddr &PAGE_MASK , - direction); +tce_build (tbl, entry, npages, vaddr &PAGE_MASK , dir); s->dma_length = s->length; }return nelems; error: -calgary_unmap_sg (dev, sg, nelems, direction); +calgary_unmap_sg (dev, sg, nelems, dir,NULL );for_each_sg (sg, s, nelems, i) { sg->dma_address = bad_dma_address; sg->dma_length =0 ;
@@ -445,10 +446,12 @@ error:return 0 ; } -static dma_addr_t calgary_map_single (struct device *dev,phys_addr_t paddr, -size_t size,int direction) +static dma_addr_t calgary_map_page (struct device *dev,struct page *page, +unsigned long offset,size_t size, +enum dma_data_direction dir, +struct dma_attrs *attrs) { -void *vaddr =phys_to_virt (paddr); +void *vaddr =page_address (page) + offset;unsigned long uaddr;unsigned int npages;struct iommu_table *tbl =find_iommu_table (dev);
@@ -456,17 +459,18 @@ static dma_addr_t calgary_map_single(struct device *dev, phys_addr_t paddr, uaddr = (unsigned long )vaddr; npages =iommu_num_pages (uaddr, size,PAGE_SIZE ); -return iommu_alloc (dev, tbl, vaddr, npages, direction); +return iommu_alloc (dev, tbl, vaddr, npages, dir); } -static void calgary_unmap_single (struct device *dev,dma_addr_t dma_handle, -size_t size,int direction) +static void calgary_unmap_page (struct device *dev,dma_addr_t dma_addr, +size_t size,enum dma_data_direction dir, +struct dma_attrs *attrs) {struct iommu_table *tbl =find_iommu_table (dev);unsigned int npages; - npages =iommu_num_pages (dma_handle, size,PAGE_SIZE ); -iommu_free (tbl, dma_handle, npages); + npages =iommu_num_pages (dma_addr, size,PAGE_SIZE ); +iommu_free (tbl, dma_addr, npages); } static void* calgary_alloc_coherent(struct device *dev, size_t size,
@@ -515,13 +519,13 @@ static void calgary_free_coherent(struct device *dev, size_t size,free_pages ((unsigned long )vaddr,get_order (size)); } -static struct dma_mapping_ops calgary_dma_ops = { +static struct dma_map_ops calgary_dma_ops = { .alloc_coherent = calgary_alloc_coherent, .free_coherent = calgary_free_coherent, - .map_single = calgary_map_single, - .unmap_single = calgary_unmap_single, .map_sg = calgary_map_sg, .unmap_sg = calgary_unmap_sg, + .map_page = calgary_map_page, + .unmap_page = calgary_unmap_page, };static inline void __iomem *busno_to_bbar (unsigned char num)
arch/x86/kernel/pci-dma.c +12 -2
@@ -1,4 +1,5 @@#include <linux/dma-mapping.h> +#include <linux/dma-debug.h> #include <linux/dmar.h> #include <linux/bootmem.h> #include <linux/pci.h>
@@ -12,7 +13,7 @@static int forbid_dac__read_mostly ; -struct dma_mapping_ops *dma_ops; +struct dma_map_ops *dma_ops;EXPORT_SYMBOL (dma_ops);static int iommu_sac_force__read_mostly ;
@@ -44,6 +45,9 @@ struct device x86_dma_fallback_dev = { };EXPORT_SYMBOL (x86_dma_fallback_dev); +/* Number of entries preallocated for DMA-API debugging */ +#define PREALLOC_DMA_DEBUG_ENTRIES 32768 +int dma_set_mask (struct device *dev,u64 mask) {if (!dev->dma_mask || !dma_supported (dev, mask))
@@ -224,7 +228,7 @@ early_param("iommu", iommu_setup);int dma_supported (struct device *dev,u64 mask) { -struct dma_mapping_ops *ops =get_dma_ops (dev); +struct dma_map_ops *ops =get_dma_ops (dev);#ifdef CONFIG_PCI if (mask >0xffffffff && forbid_dac >0 ) {
@@ -265,6 +269,12 @@ EXPORT_SYMBOL(dma_supported);static int __init pci_iommu_init (void ) { +dma_debug_init (PREALLOC_DMA_DEBUG_ENTRIES ); + +#ifdef CONFIG_PCI +dma_debug_add_bus (&pci_bus_type); +#endif +calgary_iommu_init ();intel_iommu_init ();
arch/x86/kernel/pci-gart_64.c +19 -15
@@ -255,10 +255,13 @@ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem, }/* Map a single area into the IOMMU */ -static dma_addr_t -gart_map_single (struct device *dev,phys_addr_t paddr,size_t size,int dir) +static dma_addr_t gart_map_page (struct device *dev,struct page *page, +unsigned long offset,size_t size, +enum dma_data_direction dir, +struct dma_attrs *attrs) {unsigned long bus; +phys_addr_t paddr =page_to_phys (page) + offset;if (!dev) dev = &x86_dma_fallback_dev;
@@ -275,8 +278,9 @@ gart_map_single(struct device *dev, phys_addr_t paddr, size_t size, int dir)/* * Free a DMA mapping. */ -static void gart_unmap_single (struct device *dev,dma_addr_t dma_addr, -size_t size,int direction) +static void gart_unmap_page (struct device *dev,dma_addr_t dma_addr, +size_t size,enum dma_data_direction dir, +struct dma_attrs *attrs) {unsigned long iommu_page;int npages;
@@ -298,8 +302,8 @@ static void gart_unmap_single(struct device *dev, dma_addr_t dma_addr,/* * Wrapper for pci_unmap_single working with scatterlists. */ -static void -gart_unmap_sg (struct device *dev,struct scatterlist *sg,int nents,int dir) +static void gart_unmap_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir,struct dma_attrs *attrs) {struct scatterlist *s;int i;
@@ -307,7 +311,7 @@ gart_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, int dir)for_each_sg (sg, s, nents, i) {if (!s->dma_length || !s->length)break ; -gart_unmap_single (dev, s->dma_address, s->dma_length, dir); +gart_unmap_page (dev, s->dma_address, s->dma_length, dir,NULL ); } }
@@ -329,7 +333,7 @@ static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg, addr =dma_map_area (dev, addr, s->length, dir,0 );if (addr == bad_dma_address) {if (i >0 ) -gart_unmap_sg (dev, sg, i, dir); +gart_unmap_sg (dev, sg, i, dir,NULL ); nents =0 ; sg[0 ].dma_length =0 ;break ;
@@ -400,8 +404,8 @@ dma_map_cont(struct device *dev, struct scatterlist *start, int nelems,* DMA map all entries in a scatterlist. * Merge chunks that have page aligned sizes into a continuous mapping. */ -static int -gart_map_sg (struct device *dev,struct scatterlist *sg,int nents,int dir) +static int gart_map_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir,struct dma_attrs *attrs) {struct scatterlist *s, *ps, *start_sg, *sgmap;int need =0 , nextneed, i, out, start;
@@ -468,7 +472,7 @@ gart_map_sg(struct device *dev, struct scatterlist *sg, int nents, int dir) error:flush_gart (); -gart_unmap_sg (dev, sg, out, dir); +gart_unmap_sg (dev, sg, out, dir,NULL );/* When it was forced or merged try again in a dumb way */ if (force_iommu || iommu_merge) {
@@ -521,7 +525,7 @@ static void gart_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_addr) { -gart_unmap_single (dev, dma_addr, size, DMA_BIDIRECTIONAL); +gart_unmap_page (dev, dma_addr, size, DMA_BIDIRECTIONAL, NULL);free_pages ((unsigned long )vaddr,get_order (size)); }
@@ -707,11 +711,11 @@ static __init int init_k8_gatt(struct agp_kern_info *info)return -1 ; } -static struct dma_mapping_ops gart_dma_ops = { - .map_single = gart_map_single, - .unmap_single = gart_unmap_single, +static struct dma_map_ops gart_dma_ops = { .map_sg = gart_map_sg, .unmap_sg = gart_unmap_sg, + .map_page = gart_map_page, + .unmap_page = gart_unmap_page, .alloc_coherent = gart_alloc_coherent, .free_coherent = gart_free_coherent, };
arch/x86/kernel/pci-nommu.c +11 -10
@@ -25,19 +25,19 @@ check_addr(char *name, struct device *hwdev, dma_addr_t bus, size_t size)return 1 ; } -static dma_addr_t -nommu_map_single (struct device *hwdev,phys_addr_t paddr,size_t size, -int direction) +static dma_addr_t nommu_map_page (struct device *dev,struct page *page, +unsigned long offset,size_t size, +enum dma_data_direction dir, +struct dma_attrs *attrs) { -dma_addr_t bus = paddr; +dma_addr_t bus =page_to_phys (page) + offset;WARN_ON (size ==0 ); -if (!check_addr ("map_single" , hwdev, bus, size)) -return bad_dma_address; +if (!check_addr ("map_single" , dev, bus, size)) +return bad_dma_address;flush_write_buffers ();return bus; } -/* Map a set of buffers described by scatterlist in streaming * mode for DMA. This is the scatter-gather version of the * above pci_map_single interface. Here the scatter gather list
@@ -54,7 +54,8 @@ nommu_map_single(struct device *hwdev, phys_addr_t paddr, size_t size,* the same here. */ static int nommu_map_sg (struct device *hwdev,struct scatterlist *sg, -int nents,int direction) +int nents,enum dma_data_direction dir, +struct dma_attrs *attrs) {struct scatterlist *s;int i;
@@ -78,11 +79,11 @@ static void nommu_free_coherent(struct device *dev, size_t size, void *vaddr,free_pages ((unsigned long )vaddr,get_order (size)); } -struct dma_mapping_ops nommu_dma_ops = { +struct dma_map_ops nommu_dma_ops = { .alloc_coherent = dma_generic_alloc_coherent, .free_coherent = nommu_free_coherent, - .map_single = nommu_map_single, .map_sg = nommu_map_sg, + .map_page = nommu_map_page, .is_phys =1 , };
arch/x86/kernel/pci-swiotlb.c renamed from arch/x86/kernel/pci-swiotlb_64.c +6 -13
@@ -33,18 +33,11 @@ phys_addr_t swiotlb_bus_to_phys(dma_addr_t baddr)return baddr; } -int __weak swiotlb_arch_range_needs_mapping (void *ptr,size_t size) +int __weak swiotlb_arch_range_needs_mapping (phys_addr_t paddr,size_t size) {return 0 ; } -static dma_addr_t -swiotlb_map_single_phys (struct device *hwdev,phys_addr_t paddr,size_t size, -int direction) -{ -return swiotlb_map_single (hwdev,phys_to_virt (paddr), size, direction); -} -static void *x86_swiotlb_alloc_coherent (struct device *hwdev,size_t size,dma_addr_t *dma_handle,gfp_t flags) {
@@ -57,20 +50,20 @@ static void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size,return swiotlb_alloc_coherent (hwdev, size, dma_handle, flags); } -struct dma_mapping_ops swiotlb_dma_ops = { +struct dma_map_ops swiotlb_dma_ops = { .mapping_error = swiotlb_dma_mapping_error, .alloc_coherent = x86_swiotlb_alloc_coherent, .free_coherent = swiotlb_free_coherent, - .map_single = swiotlb_map_single_phys, - .unmap_single = swiotlb_unmap_single, .sync_single_for_cpu = swiotlb_sync_single_for_cpu, .sync_single_for_device = swiotlb_sync_single_for_device, .sync_single_range_for_cpu = swiotlb_sync_single_range_for_cpu, .sync_single_range_for_device = swiotlb_sync_single_range_for_device, .sync_sg_for_cpu = swiotlb_sync_sg_for_cpu, .sync_sg_for_device = swiotlb_sync_sg_for_device, - .map_sg = swiotlb_map_sg, - .unmap_sg = swiotlb_unmap_sg, + .map_sg = swiotlb_map_sg_attrs, + .unmap_sg = swiotlb_unmap_sg_attrs, + .map_page = swiotlb_map_page, + .unmap_page = swiotlb_unmap_page, .dma_supported =NULL , };
drivers/base/iommu.c +1 -1
@@ -31,7 +31,7 @@ void register_iommu(struct iommu_ops *ops) iommu_ops = ops; } -bool iommu_found () +bool iommu_found (void ) {return iommu_ops !=NULL ; }
drivers/pci/intel-iommu.c +33 -17
97 lines not on this page · raw message
include/linux/dma-debug.h new file +174 -0
174 lines not on this page · raw message
include/linux/dma-mapping.h +48 -0
60 lines not on this page · raw message
include/linux/intel-iommu.h +0 -7
@@ -331,11 +331,4 @@ extern int qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,extern int qi_submit_sync (struct qi_desc *desc,struct intel_iommu *iommu); -extern void *intel_alloc_coherent (struct device *,size_t ,dma_addr_t *,gfp_t ); -extern void intel_free_coherent (struct device *,size_t ,void *,dma_addr_t ); -extern dma_addr_t intel_map_single (struct device *,phys_addr_t ,size_t ,int ); -extern void intel_unmap_single (struct device *,dma_addr_t ,size_t ,int ); -extern int intel_map_sg (struct device *,struct scatterlist *,int ,int ); -extern void intel_unmap_sg (struct device *,struct scatterlist *,int ,int ); -#endif
include/linux/swiotlb.h +18 -23
81 lines not on this page · raw message
lib/Kconfig.debug +11 -0
@@ -912,6 +912,17 @@ config DYNAMIC_DEBUG See Documentation/dynamic-debug-howto.txt for additional information. +config DMA_API_DEBUG + bool "Enable debugging of DMA-API usage" + depends on HAVE_DMA_API_DEBUG + help + Enable this option to debug the use of the DMA API by device drivers. + With this option you will be able to detect common bugs in device + drivers like double-freeing of DMA mappings or freeing mappings that + were never allocated. + This option causes a performance degredation. Use only if you want + to debug device drivers. If unsure, say N. + source "samples/Kconfig" source "lib/Kconfig.kgdb"
lib/Makefile +2 -0
@@ -90,6 +90,8 @@ obj-$(CONFIG_DYNAMIC_DEBUG) += dynamic_debug.o obj-$(CONFIG_NLATTR) += nlattr.o +obj-$(CONFIG_DMA_API_DEBUG) += dma-debug.o + hostprogs-y := gen_crc32table clean-files := crc32table.h
lib/dma-debug.c new file +955 -0
315 lines not on this page · raw message
lib/swiotlb.c +39 -49
88 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.