Android ION for drivers/staging
Subsystems (2)
cover Android ION for drivers/staging
This patchset is to submit the Android ION code to drivers/staging.
ION is a constraint aware memory allocator used by Android in order to allocate memory that can be properly shared between a number of devices that possibly have different memory access constraints.
You can read some background on ION as well as some problematic aspects of ION in my lwn summary here:
https://lwn.net/Articles/565469/
Recently, Colin Cross has put in quite a bit of effort to resolve a number of the issues identified in that article as well as other.
While a number of issues remain, including negative sentiments from upstream maintainers about the interface, ION provides functionality (constraints based allocation), which is not currently available upstream.
As discussed at Linux Plumbers this year (https://lwn.net/Articles/569704/)
it was proposed to merge the ION code into staging. This will allow for better visibility to the upstream community, allowing further review and understanding of the code, but it will also aid in the design of dma-buf constraint-aware memory allocation helpers, allowing us to share the backing heap allocation implementations.
In order to preserve authorship history, I've taken all the ION changes in AOSP's common.git android-3.10 branch up to commit 8f1c42284159ebd6 (Improve ION config descriptions). The only modifications to the patches that I've made is pushing down Colin's patch to migrate ION to staging (ION was originally implemented in drivers/gpu/ion) so all the patches apply to staging from the start.
I've also had to add a single patch (the last one) which moves ION to use the new scan/count shrinker api introduced in 3.12.
Other then this one change, the only diff from AOSP's 3.10 branch are corrected path names in the header comment. ie:
- * drivers/gpu/ion/ion.c
+ * drivers/staging/android/ion/ion.c
This is a fairly large patchset, and is also available here:
http://git.linaro.org/git/people/john.stultz/android-dev.git dev/ion-staging
Please let me know if you have any thoughts or objections.
thanks
-john
Arve Hjønnevåg (2):
gpu: ion: __dma_page_cpu_to_dev -> arm_dma_ops.sync_single_for_device
hack
gpu: ion: Remove __GFP_NO_KSWAPD
Benjamin Gaignard (4):
gpu: ion: fix carveout ops
gpu: ion: fix compilation warning
gpu: ion: fix ion_platform_data definition
gpu: ion: add CMA heap
Colin Cross (36):
gpu: ion: ion_carveout_heap: fix for 3.4
ion: chunk_heap: fix leak in allocated counter
ion: add free list size to heap debug files
ion: convert map_kernel to return ERR_PTR
ion: remove IS_ERR_OR_NULL
ion: replace userspace handle cookies with idr
ion: index client->handles rbtree by buffer
ion: don't use id 0 for handle cookie
gpu: ion: delete ion_system_mapper.c
ion: move userspace api into uapi/ion.h
ion: hold reference to handle after ion_uhandle_get
ion: fix crash when alloc len is -1
ion: fix dma APIs
ion: convert sg_dma_len(sg) to sg->length
ion: check invalid values in ion_system_heap
ion: add test device for unit tests to interact with dma_bufs
ion: update idr to avoid deprecated apis
ion: don't use __arm_ioremap to map pages
ion: don't use phys_to_page or __phys_to_pfn
ion: fix printk warnings
gpu: ion: remove unnecessary function from system heap
ion: clean up ioctls
ion: drop dependency on ARM
ion: add alignment check to carveout heap
ion: optimize ion_heap_buffer_zero
ion: free low memory from page pools first
ion: check return value from remap_pfn_range
ion: use vm_insert_pfn for faulted pages
ion: remove ion_heap_alloc_pages
ion: allow cached mappings of chunk and system heap buffers
ion: use alloc_pages in system contig heap
ion: fix sparse warnings
ion: carveout heap: zero buffers on free, fix memory leak
ion: add helper to zero contiguous region of pages
ion: add alignment check to chunk heap
ion: fix bugs in cma heap
Dima Zavin (1):
gpu: ion: do not ask for compound pages in system heap
Greg Hackmann (2):
gpu: ion: fill in buffer->{dev,size} before mapping new buffers
gpu: ion: fix page offset in dma_buf_kmap()
JP Abgrall (1):
gpu: ion: fix kfree/list_del order
Johan Mossberg (2):
gpu: ion: Export ion_client_create
gpu: ion: Add support for sharing buffers with dma buf kernel handles
John Stultz (6):
ion: Add Kconfig dependency to ARM
ion: Fix compat support to use proper compat ioctl numbers
ion: Fix two small issues in system_heap allocation
ion: Cleanup whitespace issues and other checkpatch problems
ion: Improve ION config description
ion: Update system heap shrinker to use the new count/scan interface
KyongHo Cho (1):
gpu: ion: several bugfixes and enhancements of ION
Laura Abbott (3):
gpu: ion: Loop on the handle count when destroying
gpu: ion: Map only the vma size given
gpu: ion: Don't call ion_buffer_put on error path
Mitchel Humpherys (1):
gpu: ion: fix use-after-free in ion_heap_freelist_drain
Olav Haugan (3):
gpu: ion: Add missing argument to WARN call
gpu: ion: Add EXPORT_SYMBOL to functions
gpu: ion: IOCTL return success when error occurs
Rebecca Schultz Zavin (50):
gpu: ion: Add ION Memory Manager
ion: Switch map/unmap dma api to sg_tables
ion: Add reserve function to ion
ion: Switch ion to use dma-buf
gpu: ion: Use alloc_pages instead of vmalloc from the system heap
gpu: ion: support begin/end and kmap/kunmap dma_buf ops
gpu: ion: Allocate the sg_table at creation time rather than
dynamically
gpu: ion: Get an sg_table from an ion handle
gpu: ion: Set the dma_address of the sg list at alloc time
gpu: ion: ion_system_heap: Change allocations to GFP_HIGHUSER
gpu: ion: Add cache maintenance to ion.
gpu: ion: Modify the system heap to try to allocate large/huge pages
gpu: ion: Add explicit sync ioctl
gpu: ion: Only map as much of the vma as the user requested
gpu: ion: Switch to using kmalloc rather than kmap during allocation
gpu: ion: Fix race between ion_import and ion_free
gpu: ion: Fix bug in ion_free
gpu: ion: Add debug information for orphaned handles
gpu: ion: Fix memory leak of dirty bits
gpu: ion: Add support for cached mappings that don't fault
gpu: ion: optimize system heap for non fault buffers
gpu: ion: Stop trying to allocate from an order on first failure
gpu: ion: ion_system_heap: Fix bug preventing compilation
gpu: ion: use vmalloc to allocate page array to map kernel
gpu: ion: Add ion_page_pool.
gpu: ion: Use the ion_page_pool from the system heap
gpu: ion: Modify gfp flags in ion_system_heap
gpu: ion: Fix several issues with page pool
gpu: ion: Fix lockdep issue in ion_page_pool
gpu: ion: Switch to using a single shrink function
gpu: ion: Refactor locking
gpu: ion: Clear GFP_WAIT flag on high order allocations
gpu: ion: Don't flush allocatoins that come from the page pools
gpu: ion: Fix bug in ion_system_heap map_user
gpu: ion: Fix bug in zeroing pages in system heap
gpu: ion: Modify reserve function for carveouts with no start address
gpu: ion: Fix bug where MAP ioctl was no longer supported
gpu: ion: Switch heap rbtree to a prio list
gpu: ion: Refactor common mapping functions out of system heap
gpu: ion: Add chunk heap
gpu: ion: Clarify variable names and comments around heap ids v types
gpu: ion: Remove heapmask from client
gpu: ion: Modify zeroing code so it only allocates address space once
gpu: ion: Refactor the code to zero buffers
gpu: ion: Only flush buffers in the chunk heap if they were used
cached
gpu: ion: Make ion_free asynchronous
gpu: ion: ion_chunk_heap: Zero chunk heap memory at creation time
gpu: ion: Fix bug in ion shrinker
gpu: ion: Also shrink memory cached in the deferred free list
gpu: ion: Fix performance issue in faulting code
Rom Lemarchand (3):
ion: add new ion_user_handle_t type for the user-space token
ion: change ion_user_handle_t definition to int
ion: add compat_ioctl
drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 + drivers/staging/android/ion/Kconfig | 24 + drivers/staging/android/ion/Makefile | 7 + drivers/staging/android/ion/compat_ion.c | 177 +++ drivers/staging/android/ion/compat_ion.h | 30 + drivers/staging/android/ion/ion.c | 1549 +++++++++++++++++++++++ drivers/staging/android/ion/ion.h | 204 +++ drivers/staging/android/ion/ion_carveout_heap.c | 194 +++ drivers/staging/android/ion/ion_chunk_heap.c | 195 +++ drivers/staging/android/ion/ion_cma_heap.c | 218 ++++ drivers/staging/android/ion/ion_heap.c | 314 +++++ drivers/staging/android/ion/ion_page_pool.c | 195 +++ drivers/staging/android/ion/ion_priv.h | 360 ++++++ drivers/staging/android/ion/ion_system_heap.c | 483 +++++++ drivers/staging/android/ion/ion_test.c | 281 ++++ drivers/staging/android/ion/tegra/Makefile | 1 + drivers/staging/android/ion/tegra/tegra_ion.c | 96 ++ drivers/staging/android/uapi/ion.h | 196 +++ drivers/staging/android/uapi/ion_test.h | 71 ++ 20 files changed, 4599 insertions(+) create mode 100644 drivers/staging/android/ion/Kconfig create mode 100644 drivers/staging/android/ion/Makefile create mode 100644 drivers/staging/android/ion/compat_ion.c create mode 100644 drivers/staging/android/ion/compat_ion.h create mode 100644 drivers/staging/android/ion/ion.c create mode 100644 drivers/staging/android/ion/ion.h create mode 100644 drivers/staging/android/ion/ion_carveout_heap.c create mode 100644 drivers/staging/android/ion/ion_chunk_heap.c create mode 100644 drivers/staging/android/ion/ion_cma_heap.c create mode 100644 drivers/staging/android/ion/ion_heap.c create mode 100644 drivers/staging/android/ion/ion_page_pool.c create mode 100644 drivers/staging/android/ion/ion_priv.h create mode 100644 drivers/staging/android/ion/ion_system_heap.c create mode 100644 drivers/staging/android/ion/ion_test.c create mode 100644 drivers/staging/android/ion/tegra/Makefile create mode 100644 drivers/staging/android/ion/tegra/tegra_ion.c create mode 100644 drivers/staging/android/uapi/ion.h create mode 100644 drivers/staging/android/uapi/ion_test.h
Signature · 1 line
1.8.3.2
PATCH 1/115 gpu: ion: Add ION Memory Manager
From: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 + drivers/staging/android/ion/Kconfig | 12 + drivers/staging/android/ion/Makefile | 2 + drivers/staging/android/ion/ion.c | 1187 +++++++++++++++++++++++ drivers/staging/android/ion/ion.h | 344 +++++++ drivers/staging/android/ion/ion_carveout_heap.c | 162 ++++ drivers/staging/android/ion/ion_heap.c | 72 ++ drivers/staging/android/ion/ion_priv.h | 185 ++++ drivers/staging/android/ion/ion_system_heap.c | 198 ++++ drivers/staging/android/ion/ion_system_mapper.c | 114 +++ drivers/staging/android/ion/tegra/Makefile | 1 + drivers/staging/android/ion/tegra/tegra_ion.c | 96 ++ 13 files changed, 2377 insertions(+) create mode 100644 drivers/staging/android/ion/Kconfig create mode 100644 drivers/staging/android/ion/Makefile create mode 100644 drivers/staging/android/ion/ion.c create mode 100644 drivers/staging/android/ion/ion.h create mode 100644 drivers/staging/android/ion/ion_carveout_heap.c create mode 100644 drivers/staging/android/ion/ion_heap.c create mode 100644 drivers/staging/android/ion/ion_priv.h create mode 100644 drivers/staging/android/ion/ion_system_heap.c create mode 100644 drivers/staging/android/ion/ion_system_mapper.c create mode 100644 drivers/staging/android/ion/tegra/Makefile create mode 100644 drivers/staging/android/ion/tegra/tegra_ion.c
drivers/staging/android/Kconfig +2 -0
@@ -100,6 +100,8 @@ config SW_SYNC_USER *WARNING* improper use of this can result in deadlocking kernel drivers from userspace. +source "drivers/staging/android/ion/Kconfig" + endif # if ANDROID endmenu
drivers/staging/android/Makefile +2 -0
@@ -1,5 +1,7 @@ ccflags-y += -I$(src) # needed for trace events +obj-y += ion/ + obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o obj-$(CONFIG_ASHMEM) += ashmem.o obj-$(CONFIG_ANDROID_LOGGER) += logger.o
drivers/staging/android/ion/Kconfig new file +12 -0
@@ -0,0 +1,12 @@ +menuconfig ION + tristate "Ion Memory Manager" + select GENERIC_ALLOCATOR + help + Chose this option to enable the ION Memory Manager. + +config ION_TEGRA + tristate "Ion for Tegra" + depends on ARCH_TEGRA && ION + help + Choose this option if you wish to use ion on an nVidia Tegra. +
drivers/staging/android/ion/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +obj-$(CONFIG_ION) += ion.o ion_heap.o ion_system_heap.o ion_carveout_heap.o +obj-$(CONFIG_ION_TEGRA) += tegra/
drivers/staging/android/ion/ion.c new file +1187 -0
1,187 lines not on this page · raw message
drivers/staging/android/ion/ion.h new file +344 -0
344 lines not on this page · raw message
drivers/staging/android/ion/ion_carveout_heap.c new file +162 -0
162 lines not on this page · raw message
drivers/staging/android/ion/ion_heap.c new file +72 -0
72 lines not on this page · raw message
drivers/staging/android/ion/ion_priv.h new file +185 -0
185 lines not on this page · raw message
drivers/staging/android/ion/ion_system_heap.c new file +198 -0
198 lines not on this page · raw message
drivers/staging/android/ion/ion_system_mapper.c new file +114 -0
114 lines not on this page · raw message
drivers/staging/android/ion/tegra/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-y += tegra_ion.o
drivers/staging/android/ion/tegra/tegra_ion.c new file +96 -0
96 lines not on this page · raw message
Signature · 1 line
1.8.3.2
Greg KH
maintainerRe: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 8 lines · John Stultz
From: Rebecca Schultz Zavin <[email protected]>
Signed-off-by Rebecca Schultz Zavin <[email protected]>[jstultz: Squished in Colin Cross' move to staging change]Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 +
This patch breaks the build if this option is enabled.
I suggest only adding the driver to the build _after_ it can be built, for obvious reasons :)
Other than this "minor" thing, I have no objection to these patches at all. Can you redo this patch to fix this up, and maybe send another patch adding it to the build and let me know at what point in the series it should be applied?
Also, I only received 92 of these patches, not all 115, are they stuck somewhere in a mail queue?
thanks,
greg k-h
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 9 lines · Greg KH
Quoted 8 lines · John Stultz
From: Rebecca Schultz Zavin <[email protected]>
Signed-off-by Rebecca Schultz Zavin <[email protected]>[jstultz: Squished in Colin Cross' move to staging change]Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 +
Quoted 4 lines
This patch breaks the build if this option is enabled.
I suggest only adding the driver to the build _after_ it can be built, for obvious reasons :)
Right. Until recently it really only builds on ARM. There's a patch which adds that dependency (and then later removes it once other fixes land), but its later in the patch series.
Apologies.
Quoted 4 lines
Other than this "minor" thing, I have no objection to these patches at all. Can you redo this patch to fix this up, and maybe send another patch adding it to the build and let me know at what point in the series it should be applied?
I'll squish that config dependency down to the first patch and go through verifying it builds.
Quoted 2 lines
Also, I only received 92 of these patches, not all 115, are they stuck somewhere in a mail queue?
Maybe? I hope! I've never played GregKH and sent such a large patchset before!
thanks
-john
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 10 lines · John Stultz
Quoted 9 lines · Greg KH
Quoted 8 lines · John Stultz
From: Rebecca Schultz Zavin <[email protected]>
Signed-off-by Rebecca Schultz Zavin <[email protected]>[jstultz: Squished in Colin Cross' move to staging change]Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 +
Quoted 7 lines
Quoted 4 lines
This patch breaks the build if this option is enabled.
I suggest only adding the driver to the build _after_ it can be built, for obvious reasons :)
Right. Until recently it really only builds on ARM. There's a patch which adds that dependency (and then later removes it once other fixes land), but its later in the patch series.
Ah. Forgot about the shrinker interface changes (which I fix at the very end).
So yea, I'll disable the config in the first patch ("depends on BROKEN" good enough? or would you prefer "depends on BROKEN && !BROKEN" to be really sure? ;), and then yank that line in the last.
thanks
-john
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
In reply to Greg KHQuoted 11 lines · Greg KH
Quoted 8 lines · John Stultz
From: Rebecca Schultz Zavin <[email protected]>
Signed-off-by Rebecca Schultz Zavin <[email protected]>[jstultz: Squished in Colin Cross' move to staging change]Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/Kconfig | 2 + drivers/staging/android/Makefile | 2 +This patch breaks the build if this option is enabled.
Quoted 7 lines
I suggest only adding the driver to the build _after_ it can be built, for obvious reasons :)
Other than this "minor" thing, I have no objection to these patches at all. Can you redo this patch to fix this up, and maybe send another patch adding it to the build and let me know at what point in the series it should be applied?
Ok. Two patches attached. I ended up just yanking the build directory since disabling the config option caused trivial collisions in the series.
0001-ion-Disable-ION-as-it-doesn-t-build.patch - This can be folded in after the first patch.
0117-ion-Reenable-the-build.patch - This can be applied to the end of the series, or folded into the last patch "ion: Update system heap shrinker to use the new count/scan interface"
Let me know if you'd rather this be done some other way. Also let me know if you've still not seen the other patches in the series.
thanks
-john
Quoted 1 line
From
eba11fdcb50d53906f1b159969448803178c173fMon Sep 17 00:00:00 2001
From: John Stultz <[email protected]>
Date: Fri, 13 Dec 2013 16:56:53 -0800
Subject: [PATCH] ion: Disable ION, as it doesn't build
Since the ion patches are forward ported from 3.10, they won't build. So disable it from the build for now and we'll re-enable it after things are building again.
--- drivers/staging/android/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/staging/android/Makefile +2 -1
@@ -1,6 +1,7 @@ ccflags-y += -I$(src) # needed for trace events -obj-y += ion/ +# ION doesn't build just yet, so disable it from the build +#obj-y += ion/ obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o obj-$(CONFIG_ASHMEM) += ashmem.o
Signature · 1 line
1.8.3.2
Quoted 1 line
From
60a715a53406a68c399dcf706bdb33c83e9c42ffMon Sep 17 00:00:00 2001
From: John Stultz <[email protected]>
Date: Fri, 13 Dec 2013 17:12:18 -0800
Subject: [PATCH] ion: Reenable the build
Now that ION builds, reenable it in the build.
--- drivers/staging/android/Makefile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/staging/android/Makefile +1 -2
@@ -1,7 +1,6 @@ ccflags-y += -I$(src) # needed for trace events -# ION doesn't build just yet, so disable it from the build -#obj-y += ion/ +obj-y += ion/ obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o obj-$(CONFIG_ASHMEM) += ashmem.o
Signature · 1 line
1.8.3.2
Greg KH
maintainerRe: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 3 lines · John Stultz
0117-ion-Reenable-the-build.patch - This can be applied to the end of the series, or folded into the last patch "ion: Update system heap shrinker to use the new count/scan interface"
Was there a 0116 patch somewhere?
Everything works fine until this one (i.e. the driver doesn't build), but after applying this patch, the build breaks with a ton of undefined symbols at link time, here's a sample of some of them:
ERROR: "ion_carveout_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_carveout_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_contig_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_cma_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_buffer_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_heap_map_user" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_unmap_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_map_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_pages_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_pages_sync_for_device" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_buffer_cached" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_buffer_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
So I'm not going to apply this patch, but I have the 115 previous patches (I took your modified 001 patch that you sent off-list to me.)
Care to fix up the symbol issues in a new series of patches and then enabling the driver to build?
thanks,
greg k-h
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 5 lines · Greg KH
Quoted 3 lines · John Stultz
0117-ion-Reenable-the-build.patch - This can be applied to the end of the series, or folded into the last patch "ion: Update system heap shrinker to use the new count/scan interface"
Was there a 0116 patch somewhere?
No.. the disable build patch was inserted between patch 1 and 2, so after the renumbering the re-enable patch was 117.
When I selected the disable patch to send I renamed it 0001 just to make it more clear which one it would apply to.
Sorry for the confusion.
Quoted 20 lines
Everything works fine until this one (i.e. the driver doesn't build), but after applying this patch, the build breaks with a ton of undefined symbols at link time, here's a sample of some of them:
ERROR: "
ion_carveout_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_carveout_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_contig_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_cma_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_buffer_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_heap_map_user" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_unmap_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_map_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_pages_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_pages_sync_for_device" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_buffer_cached" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_buffer_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
Quoted 2 lines
So I'm not going to apply this patch, but I have the 115 previous patches (I took your modified 001 patch that you sent off-list to me.)
? Hrmm. I'm not seeing this (I wouldn't have sent the patches if I did!).
If you have the chance, could you send me your config?
Quoted 2 lines
Care to fix up the symbol issues in a new series of patches and then enabling the driver to build?
Will do.
thanks
-john
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 21 lines · John Stultz
Quoted 20 lines · Greg KH
Everything works fine until this one (i.e. the driver doesn't build), but after applying this patch, the build breaks with a ton of undefined symbols at link time, here's a sample of some of them:
ERROR: "
ion_carveout_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_carveout_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_system_contig_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_cma_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_chunk_heap_create" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_buffer_destroy" [drivers/staging/android/ion/ion_heap.ko] undefined!
ERROR: "ion_heap_map_user" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_unmap_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_map_kernel" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_pages_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_pages_sync_for_device" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_buffer_cached" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
ERROR: "ion_heap_buffer_zero" [drivers/staging/android/ion/ion_chunk_heap.ko] undefined!
Quoted 3 lines
Quoted 2 lines
So I'm not going to apply this patch, but I have the 115 previous patches (I took your modified 001 patch that you sent off-list to me.)
? Hrmm. I'm not seeing this (I wouldn't have sent the patches if I did!).
Ah.. its a problem with make modules. I see it now. Will have a fix for you shortly.
thanks
-john
PATCH 1/2 ion: Don't allow building ION as a module.
In reply to Greg KHION doesn't export the proper symbols for it to be a module. This causes build issues when ION is configured as a module.
Since Andorid kernels rarely use modules (I think recent policy requires no modules?), go ahead and set the ION config to a bool from the tristate option.
If folks decide ION as a module is important, we will have to go through and export the various needed symbols.
--- drivers/staging/android/ion/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/Kconfig +1 -1
@@ -1,5 +1,5 @@ menuconfig ION - tristate "Ion Memory Manager" + bool "Ion Memory Manager" select GENERIC_ALLOCATOR select DMA_SHARED_BUFFER ---help---
Signature · 1 line
1.8.3.2
PATCH 2/2 ion: Reenable the build
Now that ION builds, reenable it in the build.
--- drivers/staging/android/Makefile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/staging/android/Makefile +1 -2
@@ -1,7 +1,6 @@ ccflags-y += -I$(src) # needed for trace events -# ION doesn't build just yet, so disable it from the build -#obj-y += ion/ +obj-y += ion/ obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o obj-$(CONFIG_ASHMEM) += ashmem.o
Signature · 1 line
1.8.3.2
Greg KH
maintainerRe: [PATCH 1/2] ion: Don't allow building ION as a module.
In reply to John StultzQuoted 20 lines · John Stultz
ION doesn't export the proper symbols for it to be a module. This causes build issues when ION is configured as a module.
Since Andorid kernels rarely use modules (I think recent policy requires no modules?), go ahead and set the ION config to a bool from the tristate option.
If folks decide ION as a module is important, we will have to go through and export the various needed symbols.
Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/ion/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/staging/android/ion/Kconfig +0
-0
drivers/staging/android/ion/Kconfig line 2
Quoted 3 lines
drivers/staging/android/ion/Kconfig +1
-1@@ -1,5 +1,5 @@ menuconfig ION - tristate "Ion Memory Manager" + bool "Ion Memory Manager"
Or you can do:
depends on m
and leave it as a "tristate", then when switching it off, I think you can then be prompted as 'm' again, but I could be wrong about that.
So should ION ever be able to be built as a module in the future?
thanks,
greg k-h
Re: [PATCH 1/2] ion: Don't allow building ION as a module.
Quoted 29 lines · Greg KH
Quoted 20 lines · John Stultz
ION doesn't export the proper symbols for it to be a module. This causes build issues when ION is configured as a module.
Since Andorid kernels rarely use modules (I think recent policy requires no modules?), go ahead and set the ION config to a bool from the tristate option.
If folks decide ION as a module is important, we will have to go through and export the various needed symbols.
Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/ion/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/staging/android/ion/Kconfig +0
-0drivers/staging/android/ion/Kconfig line 2
Quoted 3 lines
drivers/staging/android/ion/Kconfig +1
-1@@ -1,5 +1,5 @@ menuconfig ION - tristate "Ion Memory Manager" + bool "Ion Memory Manager"Or you can do:
depends on m
and leave it as a "tristate", then when switching it off, I think you can then be prompted as 'm' again, but I could be wrong about that.
Quoted 1 line
So should ION ever be able to be built as a module in the future?
My understanding is that Android is moving away from using modules, but others could comment more authoritatively. So I don't think this is an issue. But as I said in the patch, if folks do want to have ION as a module, we'll have to export all the various symbols that ION users need.
thanks
-john
Re: [PATCH 1/2] ion: Don't allow building ION as a module.
Quoted 30 lines · John Stultz
Quoted 29 lines · Greg KH
Quoted 20 lines · John Stultz
ION doesn't export the proper symbols for it to be a module. This causes build issues when ION is configured as a module.
Since Andorid kernels rarely use modules (I think recent policy requires no modules?), go ahead and set the ION config to a bool from the tristate option.
If folks decide ION as a module is important, we will have to go through and export the various needed symbols.
Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/ion/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/staging/android/ion/Kconfig +0
-0drivers/staging/android/ion/Kconfig line 2
Quoted 3 lines
drivers/staging/android/ion/Kconfig +1
-1@@ -1,5 +1,5 @@ menuconfig ION - tristate "Ion Memory Manager" + bool "Ion Memory Manager"Or you can do:
depends on m
and leave it as a "tristate", then when switching it off, I think you can then be prompted as 'm' again, but I could be wrong about that.
Quoted 5 lines
Quoted 1 line
So should ION ever be able to be built as a module in the future?
My understanding is that Android is moving away from using modules, but others could comment more authoritatively. So I don't think this is an issue. But as I said in the patch, if folks do want to have ION as a module, we'll have to export all the various symbols that ION users need.
There's no reason ION couldn't be built as a module, it just doesn't make much sense given how central it is to the Android devices it is used on. It will need some TODOs fixed in the removal paths, and probably some refcounting to prevent removing the module while buffers are allocated.
As for modules in general on Android, they are rarely used on ARM Android devices and were considered a security risk. I don't think anybody would have a problem re-enabling them if CONFIG_MODULE_SIG_FORCE was also enabled.
Greg KH
maintainerRe: [PATCH 001/115] gpu: ion: Add ION Memory Manager
In reply to John Stultzdrivers/staging/android/ion/ion.c line 1171
Quoted 3 lines · John Stultz
drivers/staging/android/ion/ion.c new file +3
-0+ idev->debug_root =debugfs_create_dir ("ion" ,NULL ); +if (IS_ERR_OR_NULL (idev->debug_root)) +pr_err ("ion: failed to create debug files.\n" );
There's no need to check for this at all, you just printed out an error message if CONFIG_DEBUGFS is not enabled, which isn't all that nice.
Something to clean up later, no big deal now.
thanks,
greg k-h
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 9 lines · Greg KH
drivers/staging/android/ion/ion.c line 1171
Quoted 3 lines · John Stultz
drivers/staging/android/ion/ion.c new file +3
-0+ idev->debug_root =debugfs_create_dir ("ion" ,NULL ); +if (IS_ERR_OR_NULL (idev->debug_root)) +pr_err ("ion: failed to create debug files.\n" );There's no need to check for this at all, you just printed out an error message if
CONFIG_DEBUGFSis not enabled, which isn't all that nice.Something to clean up later, no big deal now.
Patch 75/115 (ion: remove IS_ERR_OR_NULL) replaces this chunk with:
idev->debug_root = debugfs_create_dir("ion", NULL);
if (!idev->debug_root)
pr_err("ion: failed to create debug files.\n");which won't print an error if debugfs is disabled.
Greg KH
maintainerRe: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 9 lines · Colin Cross
Quoted 8 lines · Greg KH
drivers/staging/android/ion/ion.c line 1171
Quoted 3 lines · John Stultz
drivers/staging/android/ion/ion.c new file +3
-0+ idev->debug_root =debugfs_create_dir ("ion" ,NULL ); +if (IS_ERR_OR_NULL (idev->debug_root)) +pr_err ("ion: failed to create debug files.\n" );There's no need to check for this at all, you just printed out an error message if
CONFIG_DEBUGFSis not enabled, which isn't all that nice.
Quoted 7 lines
Quoted 1 line
Something to clean up later, no big deal now.
Patch 75/115 (ion: remove
IS_ERR_OR_NULL) replaces this chunk with:idev->debug_root = debugfs_create_dir("ion", NULL); if (!idev->debug_root) pr_err("ion: failed to create debug files.\n");which won't print an error if debugfs is disabled.
If debugfs is disabled, debugfs_create_dir() returns ERR_PTR(-ENODEV).
So are you sure you tested this? :)
Re: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 12 lines · Greg KH
Quoted 11 lines · Colin Cross
Quoted 9 lines · Greg KH
drivers/staging/android/ion/ion.c line 1171
Quoted 3 lines · John Stultz
drivers/staging/android/ion/ion.c new file +3
-0+ idev->debug_root =debugfs_create_dir ("ion" ,NULL ); +if (IS_ERR_OR_NULL (idev->debug_root)) +pr_err ("ion: failed to create debug files.\n" );There's no need to check for this at all, you just printed out an error message if
CONFIG_DEBUGFSis not enabled, which isn't all that nice.Something to clean up later, no big deal now.
Quoted 7 lines
Quoted 5 lines
Patch 75/115 (ion: remove
IS_ERR_OR_NULL) replaces this chunk with:idev->debug_root = debugfs_create_dir("ion", NULL); if (!idev->debug_root) pr_err("ion: failed to create debug files.\n");which won't print an error if debugfs is disabled.
If debugfs is disabled,
debugfs_create_dir()returnsERR_PTR(-ENODEV).
ERR_PTR(-ENODEV) is not NULL, so this won't print an error. I thought that was the point of returning ERR_PTR when disabled and NULL when enabled.
Quoted 1 line
So are you sure you tested this? :)
I have not tested this, but it seems correct to me.
Greg KH
maintainerRe: [PATCH 001/115] gpu: ion: Add ION Memory Manager
Quoted 19 lines · Colin Cross
Quoted 18 lines · Greg KH
Quoted 16 lines · Colin Cross
Quoted 9 lines · Greg KH
drivers/staging/android/ion/ion.c line 1171
Quoted 3 lines · John Stultz
drivers/staging/android/ion/ion.c new file +3
-0+ idev->debug_root =debugfs_create_dir ("ion" ,NULL ); +if (IS_ERR_OR_NULL (idev->debug_root)) +pr_err ("ion: failed to create debug files.\n" );There's no need to check for this at all, you just printed out an error message if
CONFIG_DEBUGFSis not enabled, which isn't all that nice.Something to clean up later, no big deal now.
Patch 75/115 (ion: remove
IS_ERR_OR_NULL) replaces this chunk with:idev->debug_root = debugfs_create_dir("ion", NULL); if (!idev->debug_root) pr_err("ion: failed to create debug files.\n");which won't print an error if debugfs is disabled.
Quoted 5 lines
Quoted 1 line
If debugfs is disabled,
debugfs_create_dir()returnsERR_PTR(-ENODEV).
ERR_PTR(-ENODEV)is not NULL, so this won't print an error. I thought that was the point of returningERR_PTRwhen disabled and NULL when enabled.
{sigh}
Yes, you are right, I blame the jetlag, sorry.
greg k-h
PATCH 2/115 gpu: ion: ion_carveout_heap: fix for 3.4
In reply to John StultzFrom: Colin Cross <[email protected]>
__arch_ioremap is no longer available, use __arm_ioremap instead.
--- drivers/staging/android/ion/ion_carveout_heap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_carveout_heap.c +2 -2
@@ -99,14 +99,14 @@ void ion_carveout_heap_unmap_dma(struct ion_heap *heap,void *ion_carveout_heap_map_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { -return __arch_ioremap (buffer->priv_phys, buffer->size, +return __arm_ioremap (buffer->priv_phys, buffer->size,MT_MEMORY_NONCACHED ); }void ion_carveout_heap_unmap_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { -__arch_iounmap (buffer->vaddr); +__arm_iounmap (buffer->vaddr); buffer->vaddr =NULL ;return ; }
Signature · 1 line
1.8.3.2
PATCH 3/115 ion: Switch map/unmap dma api to sg_tables
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Switch these api's from scatterlists to sg_tables
--- drivers/staging/android/ion/ion.c | 16 +++---- drivers/staging/android/ion/ion.h | 6 +-- drivers/staging/android/ion/ion_priv.h | 18 ++------ drivers/staging/android/ion/ion_system_heap.c | 66 ++++++++++++++++----------- 4 files changed, 53 insertions(+), 53 deletions(-)
drivers/staging/android/ion/ion.c +8 -8
@@ -445,11 +445,11 @@ void *ion_map_kernel(struct ion_client *client, struct ion_handle *handle)return vaddr; } -struct scatterlist *ion_map_dma (struct ion_client *client, +struct sg_table *ion_map_dma (struct ion_client *client,struct ion_handle *handle) {struct ion_buffer *buffer; -struct scatterlist *sglist; +struct sg_table *table;mutex_lock (&client->lock);if (!ion_handle_validate (client, handle)) {
@@ -469,16 +469,16 @@ struct scatterlist *ion_map_dma(struct ion_client *client,return ERR_PTR (-ENODEV ); }if (_ion_map (&buffer->dmap_cnt, &handle->dmap_cnt)) { - sglist = buffer->heap->ops->map_dma (buffer->heap, buffer); -if (IS_ERR_OR_NULL (sglist)) + table = buffer->heap->ops->map_dma (buffer->heap, buffer); +if (IS_ERR_OR_NULL (table))_ion_unmap (&buffer->dmap_cnt, &handle->dmap_cnt); - buffer->sglist = sglist; + buffer->sg_table = table; }else { - sglist = buffer->sglist; + table = buffer->sg_table; }mutex_unlock (&buffer->lock);mutex_unlock (&client->lock); -return sglist; +return table; }void ion_unmap_kernel (struct ion_client *client,struct ion_handle *handle)
@@ -505,7 +505,7 @@ void ion_unmap_dma(struct ion_client *client, struct ion_handle *handle)mutex_lock (&buffer->lock);if (_ion_unmap (&buffer->dmap_cnt, &handle->dmap_cnt)) { buffer->heap->ops->unmap_dma (buffer->heap, buffer); - buffer->sglist =NULL ; + buffer->sg_table =NULL ; }mutex_unlock (&buffer->lock);mutex_unlock (&client->lock);
drivers/staging/android/ion/ion.h +3 -3
@@ -169,10 +169,10 @@ void ion_unmap_kernel(struct ion_client *client, struct ion_handle *handle);* @client: the client * @handle: handle to map * -* Return an sglist describing the given handle +* Return an sg_table describing the given handle */ -struct scatterlist *ion_map_dma (struct ion_client *client, -struct ion_handle *handle); +struct sg_table *ion_map_dma (struct ion_client *client, +struct ion_handle *handle);/** * ion_unmap_dma() - destroy a dma mapping for a handle
drivers/staging/android/ion/ion_priv.h +3 -15
@@ -24,18 +24,6 @@#include "ion.h" -struct ion_mapping ; - -struct ion_dma_mapping { -struct kref ref; -struct scatterlist *sglist; -}; - -struct ion_kernel_mapping { -struct kref ref; -void *vaddr; -}; -struct ion_buffer *ion_handle_buffer (struct ion_handle *handle);/**
@@ -54,7 +42,7 @@ struct ion_buffer *ion_handle_buffer(struct ion_handle *handle);* @kmap_cnt: number of times the buffer is mapped to the kernel * @vaddr: the kenrel mapping if kmap_cnt is not zero * @dmap_cnt: number of times the buffer is mapped for dma -* @sglist: the scatterlist for the buffer is dmap_cnt is not zero +* @sg_table: the sg table for the buffer if dmap_cnt is not zero */ struct ion_buffer {struct kref ref;
@@ -71,7 +59,7 @@ struct ion_buffer {int kmap_cnt;void *vaddr;int dmap_cnt; -struct scatterlist *sglist; +struct sg_table *sg_table; };/**
@@ -93,7 +81,7 @@ struct ion_heap_ops {void (*free) (struct ion_buffer *buffer);int (*phys) (struct ion_heap *heap,struct ion_buffer *buffer,ion_phys_addr_t *addr,size_t *len); -struct scatterlist *(*map_dma) (struct ion_heap *heap, +struct sg_table *(*map_dma) (struct ion_heap *heap,struct ion_buffer *buffer);void (*unmap_dma) (struct ion_heap *heap,struct ion_buffer *buffer);void * (*map_kernel) (struct ion_heap *heap,struct ion_buffer *buffer);
drivers/staging/android/ion/ion_system_heap.c +39 -27
96 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 4/115 ion: Add reserve function to ion
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Rather than requiring each platform call memblock_remove or reserve from the board file, add this to ion
--- drivers/staging/android/ion/ion.c | 17 +++++++++++++++++ drivers/staging/android/ion/ion.h | 11 +++++++++++ 2 files changed, 28 insertions(+)
drivers/staging/android/ion/ion.c +17 -0
@@ -19,6 +19,7 @@#include <linux/fs.h> #include <linux/anon_inodes.h> #include <linux/list.h> +#include <linux/memblock.h> #include <linux/miscdevice.h> #include <linux/export.h> #include <linux/mm.h>
@@ -1185,3 +1186,19 @@ void ion_device_destroy(struct ion_device *dev)/* XXX need to free the heaps and clients ? */ kfree (dev); } + +void __init ion_reserve (struct ion_platform_data *data) +{ +int i, ret; + +for (i =0 ; i < data->nr; i++) { +if (data->heaps[i].size ==0 ) +continue ; + ret =memblock_reserve (data->heaps[i].base, + data->heaps[i].size); +if (ret) +pr_err ("memblock reserve of %x@%lx failed\n" , + data->heaps[i].size, + data->heaps[i].base); + } +}
drivers/staging/android/ion/ion.h +11 -0
@@ -87,6 +87,17 @@ struct ion_platform_data { };/** +* ion_reserve() - reserve memory for ion heaps if applicable +* @data: platform data specifying starting physical address and +* size +* +* Calls memblock reserve to set aside memory for heaps that are +* located at specific memory addresses or of specfic sizes not +* managed by the kernel +*/ +void ion_reserve (struct ion_platform_data *data); + +/** * ion_client_create() - allocate a client and returns it * @dev: the global ion device * @heap_mask: mask of heaps this client can allocate from
Signature · 1 line
1.8.3.2
PATCH 5/115 gpu: ion: several bugfixes and enhancements of ION
In reply to John StultzFrom: KyongHo Cho <[email protected]>
1. Verifying if the size of memory allocation in ion_alloc() is aligned by PAGE_SIZE at least. If it is not, this change makes the size to be aligned by PAGE_SIZE.
2. Unmaps all mappings to the kernel and DMA address spaces when destroying ion_buffer in ion_buffer_destroy(). This prevents leaks in those virtual address spaces.
3. Makes the return value of ion_alloc() to be explicit Linux error code when it fails to allocate a buffer.
4. Makes ion_alloc() implementation simpler. Removes 'goto' statement and relavant call to ion_buffer_put().
5. Checks if the task is valid before calling put_task_struct() due to failure on creating a ion client in ion_client_create().
6. Returns error when buffer allocation requested by userspace is failed.
--- drivers/staging/android/ion/ion.c | 52 +++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 13 deletions(-)
drivers/staging/android/ion/ion.c +39 -13
@@ -166,6 +166,12 @@ static void ion_buffer_destroy(struct kref *kref) struct ion_buffer *buffer = container_of(kref, struct ion_buffer, ref);struct ion_device *dev = buffer->dev; +if (WARN_ON (buffer->kmap_cnt >0 )) + buffer->heap->ops->unmap_kernel (buffer->heap, buffer); + +if (WARN_ON (buffer->dmap_cnt >0 )) + buffer->heap->ops->unmap_dma (buffer->heap, buffer); + buffer->heap->ops->free (buffer);mutex_lock (&dev->lock);rb_erase (&buffer->node, &dev->buffers);
@@ -296,6 +302,11 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,* request of the caller allocate from it. Repeat until allocate has * succeeded or all heaps have been tried */ +if (WARN_ON (!len)) +return ERR_PTR (-EINVAL ); + + len =PAGE_ALIGN (len); +mutex_lock (&dev->lock);for (n =rb_first (&dev->heaps); n !=NULL ; n =rb_next (n)) { struct ion_heap *heap = rb_entry(n, struct ion_heap, node);
@@ -311,27 +322,26 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len, }mutex_unlock (&dev->lock); -if (IS_ERR_OR_NULL (buffer)) +if (buffer ==NULL ) +return ERR_PTR (-ENODEV ); + +if (IS_ERR (buffer))return ERR_PTR (PTR_ERR (buffer)); handle =ion_handle_create (client, buffer); -if (IS_ERR_OR_NULL (handle)) - goto end; -/* * ion_buffer_create will create a buffer with a ref_cnt of 1, * and ion_handle_create will take a second reference, drop one here */ ion_buffer_put (buffer); -mutex_lock (&client->lock); -ion_handle_add (client, handle); -mutex_unlock (&client->lock); -return handle; +if (!IS_ERR (handle)) { +mutex_lock (&client->lock); +ion_handle_add (client, handle); +mutex_unlock (&client->lock); + } -end: -ion_buffer_put (buffer);return handle; }
@@ -680,7 +690,8 @@ struct ion_client *ion_client_create(struct ion_device *dev, client =kzalloc (sizeof (struct ion_client ),GFP_KERNEL );if (!client) { -put_task_struct (current->group_leader); +if (task) +put_task_struct (current->group_leader);return ERR_PTR (-ENOMEM ); }
@@ -798,6 +809,15 @@ static void ion_vma_open(struct vm_area_struct *vma) vma->vm_private_data =NULL ;return ; } + +if (!ion_handle_validate (client, handle)) { +ion_client_put (client); + vma->vm_private_data =NULL ; +return ; + } + +ion_handle_get (handle); + pr_debug("%s: %d client_cnt %d handle_cnt %d alloc_cnt %d\n", __func__, __LINE__, atomic_read(&client->ref.refcount),
@@ -919,7 +939,7 @@ static int ion_ioctl_share(struct file *parent, struct ion_client *client,struct file *file;if (fd <0 ) -return -ENFILE ; +return fd; file =anon_inode_getfile ("ion_share_fd" , &ion_share_fops, handle->buffer,O_RDWR );
@@ -948,8 +968,14 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)return -EFAULT ; data.handle =ion_alloc (client, data.len, data.align, data.flags); -if (copy_to_user ((void __user *)arg, &data,sizeof (data))) + +if (IS_ERR (data.handle)) +return PTR_ERR (data.handle); + +if (copy_to_user ((void __user *)arg, &data,sizeof (data))) { +ion_free (client, data.handle);return -EFAULT ; + }break ; }case ION_IOC_FREE :
Signature · 1 line
1.8.3.2
PATCH 6/115 ion: Switch ion to use dma-buf
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Ion now uses dma-buf file descriptors to share buffers with userspace. Ion becomes a dma-buf exporter and any driver that can import dma-bufs can now import ion file descriptors.
--- drivers/staging/android/ion/Kconfig | 1 + drivers/staging/android/ion/ion.c | 648 +++++++++++++----------------------- drivers/staging/android/ion/ion.h | 58 +--- 3 files changed, 236 insertions(+), 471 deletions(-)
drivers/staging/android/ion/Kconfig +1 -0
@@ -1,6 +1,7 @@ menuconfig ION tristate "Ion Memory Manager" select GENERIC_ALLOCATOR + select DMA_SHARED_BUFFER help Chose this option to enable the ION Memory Manager.
drivers/staging/android/ion/ion.c +226 -422
872 lines not on this page · raw message
drivers/staging/android/ion/ion.h +9 -49
72 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 7/115 gpu: ion: Use alloc_pages instead of vmalloc from the system heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
With this change the ion_system_heap will only use kernel address space when the memory is mapped into the kernel (rare case).
--- drivers/staging/android/ion/ion_system_heap.c | 94 +++++++++++++++++---------- 1 file changed, 61 insertions(+), 33 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +61 -33
@@ -27,74 +27,102 @@ static int ion_system_heap_allocate(struct ion_heap *heap, unsigned long size, unsigned long align, unsigned long flags) { - buffer->priv_virt = vmalloc_user(size); -if (!buffer->priv_virt) -return -ENOMEM ; -return 0 ; -} - -void ion_system_heap_free (struct ion_buffer *buffer) -{ -vfree (buffer->priv_virt); -} - -struct sg_table *ion_system_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) -{ struct sg_table *table;struct scatterlist *sg; -int i; -int npages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; -void *vaddr = buffer->priv_virt; -int ret; +int i, j; +int npages =PAGE_ALIGN (size) /PAGE_SIZE ; - table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); + table =kmalloc (sizeof (struct sg_table ),GFP_KERNEL );if (!table) -return ERR_PTR (-ENOMEM ); - ret =sg_alloc_table (table, npages,GFP_KERNEL ); -if (ret) +return -ENOMEM ; + i =sg_alloc_table (table, npages,GFP_KERNEL ); +if (i) goto err0;for_each_sg (table->sgl, sg, table->nents, i) {struct page *page; - page =vmalloc_to_page (vaddr); -if (!page) { - ret = -ENOMEM ; + page =alloc_page (GFP_KERNEL ); +if (!page) goto err1; - }sg_set_page (sg, page,PAGE_SIZE ,0 ); - vaddr +=PAGE_SIZE ; } -return table; + buffer->priv_virt = table; +return 0 ; err1: +for_each_sg (table->sgl, sg, i, j) +__free_page (sg_page (sg));sg_free_table (table); err0:kfree (table); -return ERR_PTR (ret); +return -ENOMEM ; } -void ion_system_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +void ion_system_heap_free (struct ion_buffer *buffer) { +int i; +struct scatterlist *sg; +struct sg_table *table = buffer->priv_virt; + +for_each_sg (table->sgl, sg, table->nents, i) +__free_page (sg_page (sg));if (buffer->sg_table)sg_free_table (buffer->sg_table);kfree (buffer->sg_table); } +struct sg_table *ion_system_heap_map_dma (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +return buffer->priv_virt; +} + +void ion_system_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +return ; +} +void *ion_system_heap_map_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { -return buffer->priv_virt; +struct scatterlist *sg; +int i; +void *vaddr; +struct sg_table *table = buffer->priv_virt; +struct page **pages =kmalloc (sizeof (struct page *) * table->nents, +GFP_KERNEL ); + +for_each_sg (table->sgl, sg, table->nents, i) + pages[i] =sg_page (sg); + vaddr =vmap (pages, table->nents,VM_MAP ,PAGE_KERNEL ); +kfree (pages); + +return vaddr; }void ion_system_heap_unmap_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { +vunmap (buffer->vaddr); }int ion_system_heap_map_user (struct ion_heap *heap,struct ion_buffer *buffer,struct vm_area_struct *vma) { -return remap_vmalloc_range (vma, buffer->priv_virt, vma->vm_pgoff); +struct sg_table *table = buffer->priv_virt; +unsigned long addr = vma->vm_start; +unsigned long offset = vma->vm_pgoff; +struct scatterlist *sg; +int i; + +for_each_sg (table->sgl, sg, table->nents, i) { +if (offset) { + offset--; +continue ; + } +vm_insert_page (vma, addr,sg_page (sg)); + addr +=PAGE_SIZE ; + } +return 0 ; } static struct ion_heap_ops vmalloc_ops = {
Signature · 1 line
1.8.3.2
PATCH 8/115 gpu: ion: support begin/end and kmap/kunmap dma_buf ops
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
These ops were added in the 3.4 kernel. This patch adds support for them to ion. Previous ion_map/unmap_kernel api is also retained in addition to this new api.
--- drivers/staging/android/ion/ion.c | 88 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 26 deletions(-)
drivers/staging/android/ion/ion.c +62 -26
@@ -391,6 +391,22 @@ int ion_phys(struct ion_client *client, struct ion_handle *handle,return ret; } +static void *ion_buffer_kmap_get (struct ion_buffer *buffer) +{ +void *vaddr; + +if (buffer->kmap_cnt) { + buffer->kmap_cnt++; +return buffer->vaddr; + } + vaddr = buffer->heap->ops->map_kernel (buffer->heap, buffer); +if (IS_ERR_OR_NULL (vaddr)) +return vaddr; + buffer->vaddr = vaddr; + buffer->kmap_cnt++; +return vaddr; +} +static void *ion_handle_kmap_get (struct ion_handle *handle) {struct ion_buffer *buffer = handle->buffer;
@@ -399,34 +415,30 @@ static void *ion_handle_kmap_get(struct ion_handle *handle)if (handle->kmap_cnt) { handle->kmap_cnt++;return buffer->vaddr; - }else if (buffer->kmap_cnt) { - handle->kmap_cnt++; - buffer->kmap_cnt++; -return buffer->vaddr; } - - vaddr = buffer->heap->ops->map_kernel (buffer->heap, buffer); - buffer->vaddr = vaddr; -if (IS_ERR_OR_NULL (vaddr)) { - buffer->vaddr =NULL ; + vaddr =ion_buffer_kmap_get (buffer); +if (IS_ERR_OR_NULL (vaddr))return vaddr; - } handle->kmap_cnt++; - buffer->kmap_cnt++;return vaddr; } +static void ion_buffer_kmap_put (struct ion_buffer *buffer) +{ + buffer->kmap_cnt--; +if (!buffer->kmap_cnt) { + buffer->heap->ops->unmap_kernel (buffer->heap, buffer); + buffer->vaddr =NULL ; + } +} +static void ion_handle_kmap_put (struct ion_handle *handle) {struct ion_buffer *buffer = handle->buffer; handle->kmap_cnt--;if (!handle->kmap_cnt) - buffer->kmap_cnt--; -if (!buffer->kmap_cnt) { - buffer->heap->ops->unmap_kernel (buffer->heap, buffer); - buffer->vaddr =NULL ; - } +ion_buffer_kmap_put (buffer); }void *ion_map_kernel (struct ion_client *client,struct ion_handle *handle)
@@ -675,7 +687,8 @@ static void ion_dma_buf_release(struct dma_buf *dmabuf)static void *ion_dma_buf_kmap (struct dma_buf *dmabuf,unsigned long offset) { -return NULL ; +struct ion_buffer *buffer = dmabuf->priv; +return buffer->vaddr + offset; } static void ion_dma_buf_kunmap(struct dma_buf *dmabuf, unsigned long offset,
@@ -684,26 +697,49 @@ static void ion_dma_buf_kunmap(struct dma_buf *dmabuf, unsigned long offset,return ; } -static void *ion_dma_buf_kmap_atomic (struct dma_buf *dmabuf, -unsigned long offset) +static int ion_dma_buf_begin_cpu_access (struct dma_buf *dmabuf,size_t start, +size_t len, +enum dma_data_direction direction) { -return NULL ; +struct ion_buffer *buffer = dmabuf->priv; +void *vaddr; + +if (!buffer->heap->ops->map_kernel) { +pr_err ("%s: map kernel is not implemented by this heap.\n" , + __func__); +return -ENODEV ; + } + +mutex_lock (&buffer->lock); + vaddr =ion_buffer_kmap_get (buffer); +mutex_unlock (&buffer->lock); +if (IS_ERR (vaddr)) +return PTR_ERR (vaddr); +if (!vaddr) +return -ENOMEM ; +return 0 ; } -static void ion_dma_buf_kunmap_atomic (struct dma_buf *dmabuf, -unsigned long offset,void *ptr) +static void ion_dma_buf_end_cpu_access (struct dma_buf *dmabuf,size_t start, +size_t len, +enum dma_data_direction direction) { -return ; -} +struct ion_buffer *buffer = dmabuf->priv; +mutex_lock (&buffer->lock); +ion_buffer_kmap_put (buffer); +mutex_unlock (&buffer->lock); +}struct dma_buf_ops dma_buf_ops = { .map_dma_buf = ion_map_dma_buf, .unmap_dma_buf = ion_unmap_dma_buf, .mmap = ion_mmap, .release = ion_dma_buf_release, - .kmap_atomic = ion_dma_buf_kmap_atomic, - .kunmap_atomic = ion_dma_buf_kunmap_atomic, + .begin_cpu_access = ion_dma_buf_begin_cpu_access, + .end_cpu_access = ion_dma_buf_end_cpu_access, + .kmap_atomic = ion_dma_buf_kmap, + .kunmap_atomic = ion_dma_buf_kunmap, .kmap = ion_dma_buf_kmap, .kunmap = ion_dma_buf_kunmap, };
Signature · 1 line
1.8.3.2
PATCH 9/115 gpu: ion: Allocate the sg_table at creation time rather than dynamically
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Rather than calling map_dma on the allocations dynamically, this patch switches to creating the sg_table at the time the buffer is created. This is necessary because in future updates the sg_table will be used for cache maintenance.
--- drivers/staging/android/ion/ion.c | 79 ++++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 38 deletions(-)
drivers/staging/android/ion/ion.c +41 -38
@@ -135,6 +135,7 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, unsigned long flags) {struct ion_buffer *buffer; +struct sg_table *table;int ret; buffer =kzalloc (sizeof (struct ion_buffer ),GFP_KERNEL );
@@ -149,6 +150,15 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,kfree (buffer);return ERR_PTR (ret); } + + table = buffer->heap->ops->map_dma (buffer->heap, buffer); +if (IS_ERR_OR_NULL (table)) { + heap->ops->free (buffer); +kfree (buffer); +return ERR_PTR (PTR_ERR (table)); + } + buffer->sg_table = table; + buffer->dev = dev; buffer->size = len;mutex_init (&buffer->lock);
@@ -164,9 +174,7 @@ static void ion_buffer_destroy(struct kref *kref)if (WARN_ON (buffer->kmap_cnt >0 )) buffer->heap->ops->unmap_kernel (buffer->heap, buffer); -if (WARN_ON (buffer->dmap_cnt >0 )) - buffer->heap->ops->unmap_dma (buffer->heap, buffer); - + buffer->heap->ops->unmap_dma (buffer->heap, buffer); buffer->heap->ops->free (buffer);mutex_lock (&dev->lock);rb_erase (&buffer->node, &dev->buffers);
@@ -346,6 +354,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,mutex_unlock (&client->lock); } +return handle; }
@@ -607,53 +616,42 @@ void ion_client_destroy(struct ion_client *client)kfree (client); } -static struct sg_table *ion_map_dma_buf (struct dma_buf_attachment *attachment, -enum dma_data_direction direction) +struct sg_table *ion_map_dma (struct ion_client *client, +struct ion_handle *handle) { -struct dma_buf *dmabuf = attachment->dmabuf; -struct ion_buffer *buffer = dmabuf->priv; +struct ion_buffer *buffer;struct sg_table *table; -mutex_lock (&buffer->lock); - -if (!buffer->heap->ops->map_dma) { -pr_err ("%s: map_dma is not implemented by this heap.\n" , +mutex_lock (&client->lock); +if (!ion_handle_validate (client, handle)) { +pr_err ("%s: invalid handle passed to map_dma.\n" , __func__); -mutex_unlock (&buffer->lock); -return ERR_PTR (-ENODEV ); - } -/* if an sg list already exists for this buffer just return it */ -if (buffer->dmap_cnt) { - table = buffer->sg_table; - goto end; +mutex_unlock (&client->lock); +return ERR_PTR (-EINVAL ); } - -/* otherwise call into the heap to create one */ - table = buffer->heap->ops->map_dma (buffer->heap, buffer); -if (IS_ERR_OR_NULL (table)) - goto err; - buffer->sg_table = table; -end: - buffer->dmap_cnt++; -err: -mutex_unlock (&buffer->lock); + buffer = handle->buffer; + table = buffer->sg_table; +mutex_unlock (&client->lock);return table; } -static void ion_unmap_dma_buf (struct dma_buf_attachment *attachment, -struct sg_table *table, -enum dma_data_direction direction) +void ion_unmap_dma (struct ion_client *client,struct ion_handle *handle) +{ +} + +static struct sg_table *ion_map_dma_buf (struct dma_buf_attachment *attachment, +enum dma_data_direction direction) {struct dma_buf *dmabuf = attachment->dmabuf;struct ion_buffer *buffer = dmabuf->priv; -mutex_lock (&buffer->lock); - buffer->dmap_cnt--; -if (!buffer->dmap_cnt) { - buffer->heap->ops->unmap_dma (buffer->heap, buffer); - buffer->sg_table =NULL ; - } -mutex_unlock (&buffer->lock); +return buffer->sg_table; +} + +static void ion_unmap_dma_buf (struct dma_buf_attachment *attachment, +struct sg_table *table, +enum dma_data_direction direction) +{ }static int ion_mmap (struct dma_buf *dmabuf,struct vm_area_struct *vma)
@@ -987,6 +985,11 @@ void ion_device_add_heap(struct ion_device *dev, struct ion_heap *heap)struct rb_node *parent =NULL ;struct ion_heap *entry; +if (!heap->ops->allocate || !heap->ops->free || !heap->ops->map_dma || + !heap->ops->unmap_dma) +pr_err ("%s: can not add heap with invalid ops struct.\n" , + __func__); + heap->dev = dev;mutex_lock (&dev->lock);while (*p) {
Signature · 1 line
1.8.3.2
PATCH 10/115 gpu: ion: Get an sg_table from an ion handle
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch adds an interface to return and sg_table given a valid ion handle.
--- drivers/staging/android/ion/ion.c | 8 ++------ drivers/staging/android/ion/ion.h | 13 ++++++++++++- 2 files changed, 14 insertions(+), 7 deletions(-)
drivers/staging/android/ion/ion.c +2 -6
@@ -616,8 +616,8 @@ void ion_client_destroy(struct ion_client *client)kfree (client); } -struct sg_table *ion_map_dma (struct ion_client *client, -struct ion_handle *handle) +struct sg_table *ion_sg_table (struct ion_client *client, +struct ion_handle *handle) {struct ion_buffer *buffer;struct sg_table *table;
@@ -635,10 +635,6 @@ struct sg_table *ion_map_dma(struct ion_client *client,return table; } -void ion_unmap_dma (struct ion_client *client,struct ion_handle *handle) -{ -} -static struct sg_table *ion_map_dma_buf (struct dma_buf_attachment *attachment,enum dma_data_direction direction) {
drivers/staging/android/ion/ion.h +12 -1
@@ -149,7 +149,7 @@ void ion_free(struct ion_client *client, struct ion_handle *handle);* This function queries the heap for a particular handle to get the * handle's physical address. It't output is only correct if * a heap returns physically contiguous memory -- in other cases -* this api should not be implemented -- ion_map_dma should be used +* this api should not be implemented -- ion_sg_table should be used * instead. Returns -EINVAL if the handle is invalid. This has * no implications on the reference counting of the handle -- * the returned value may not be valid if the caller is not
@@ -159,6 +159,17 @@ int ion_phys(struct ion_client *client, struct ion_handle *handle, ion_phys_addr_t *addr, size_t *len);/** +* ion_map_dma - return an sg_table describing a handle +* @client: the client +* @handle: the handle +* +* This function returns the sg_table describing +* a particular ion handle. +*/ +struct sg_table *ion_sg_table (struct ion_client *client, +struct ion_handle *handle); + +/** * ion_map_kernel - create mapping for the given handle * @client: the client * @handle: handle to map
Signature · 1 line
1.8.3.2
PATCH 11/115 gpu: ion: fill in buffer->{dev,size} before mapping new buffers
In reply to John StultzFrom: Greg Hackmann <[email protected]>
At least one map_dma() implementation (EXYNOS_CONTIG) assumes the fields are filled in
--- drivers/staging/android/ion/ion.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion.c +3 -2
@@ -151,6 +151,9 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,return ERR_PTR (ret); } + buffer->dev = dev; + buffer->size = len; + table = buffer->heap->ops->map_dma (buffer->heap, buffer);if (IS_ERR_OR_NULL (table)) { heap->ops->free (buffer);
@@ -159,8 +162,6 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, } buffer->sg_table = table; - buffer->dev = dev; - buffer->size = len;mutex_init (&buffer->lock);ion_buffer_add (dev, buffer);return buffer;
Signature · 1 line
1.8.3.2
PATCH 12/115 gpu: ion: Set the dma_address of the sg list at alloc time
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch sets the dma_address field of the sglist representing an allocation at allocation time. This technically breaks the dma api which states that these addresses should be set when a particular device takes ownership of a buffer via the dma_map apis. In the case of our systems the only dma address space is physical addresses. Additionally, we can not afford the overhead of calling dma_map_sg from this location as it implies a cache invalidate that is not necessary if the memory was previously mapped cached. Instead, the expectation is that memory being returned from the heaps is ready for dma in that if any cached mappings of that memory exist they have been invalidated.
--- drivers/staging/android/ion/ion.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +12 -1
@@ -136,7 +136,8 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, {struct ion_buffer *buffer;struct sg_table *table; -int ret; +struct scatterlist *sg; +int i, ret; buffer =kzalloc (sizeof (struct ion_buffer ),GFP_KERNEL );if (!buffer)
@@ -163,6 +164,16 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, buffer->sg_table = table;mutex_init (&buffer->lock); +/* this will set up dma addresses for the sglist -- it is not +technically correct as per the dma api -- a specific +device isn't really taking ownership here. However, in practice on +our systems the only dma_address space is physical addresses. +Additionally, we can't afford the overhead of invalidating every +allocation via dma_map_sg. The implicit contract here is that +memory comming from the heaps is ready for dma, ie if it has a +cached mapping that mapping has been invalidated */ +for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) +sg_dma_address (sg) =sg_phys (sg);ion_buffer_add (dev, buffer);return buffer; }
Signature · 1 line
1.8.3.2
PATCH 13/115 gpu: ion: ion_system_heap: Change allocations to GFP_HIGHUSER
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Previously the ion_system_heap was using GFP_KERNEL, forcing all allocations to be in lowmem. This quickly causes us to run out of lowmem.
--- drivers/staging/android/ion/ion_system_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -40,7 +40,7 @@ static int ion_system_heap_allocate(struct ion_heap *heap, goto err0;for_each_sg (table->sgl, sg, table->nents, i) {struct page *page; - page =alloc_page (GFP_KERNEL ); + page =alloc_page (GFP_HIGHUSER | __GFP_ZERO);if (!page) goto err1;sg_set_page (sg, page,PAGE_SIZE ,0 );
Signature · 1 line
1.8.3.2
PATCH 14/115 gpu: ion: Loop on the handle count when destroying
In reply to John StultzFrom: Laura Abbott <[email protected]>
When destroying a handle, all kernel mappings to that handle should be destroyed. Other handles may still have references and valid mappings to the buffer underneath which should not be destroyed. Loop on the handle reference count, not the buffer reference count to get rid of all kernel mappings for the handle.
--- drivers/staging/android/ion/ion.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +1 -1
@@ -232,7 +232,7 @@ static void ion_handle_destroy(struct kref *kref)mutex_lock (&client->lock);mutex_lock (&buffer->lock); -while (buffer->kmap_cnt) +while (handle->kmap_cnt)ion_handle_kmap_put (handle);mutex_unlock (&buffer->lock);
Signature · 1 line
1.8.3.2
PATCH 15/115 gpu: ion: Map only the vma size given
In reply to John StultzFrom: Laura Abbott <[email protected]>
When mapping carveout buffers into userspace, only map the size of the vma given, not the full size of the buffer since clients may map less than the buffer size.
--- drivers/staging/android/ion/ion_carveout_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_carveout_heap.c +1 -1
@@ -116,7 +116,7 @@ int ion_carveout_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer, {return remap_pfn_range (vma, vma->vm_start,__phys_to_pfn (buffer->priv_phys) + vma->vm_pgoff, - buffer->size, + vma->vm_end - vma->vm_start,pgprot_noncached (vma->vm_page_prot)); }
Signature · 1 line
1.8.3.2
PATCH 16/115 gpu: ion: Add cache maintenance to ion.
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch adds cache maintenance operations to ion. As per mailing list discussions regarding dma_buf, cache operations are done implicitly. At buffer allocaiton time the user can select whether he'd like mappings (both kernel and user) to be cached. When cached mappings are selected, no mappings will be created for a buffer at mmap time. Instead pages will be faulted in one at a time so we can track which pages require flushing before dma. When the buffers are mapped for dma (via the dma_buf apis) any pages which were touched will be synced for device.
--- drivers/staging/android/ion/ion.c | 162 ++++++++++++++++++++++-- drivers/staging/android/ion/ion.h | 19 ++- drivers/staging/android/ion/ion_carveout_heap.c | 26 +++- drivers/staging/android/ion/ion_priv.h | 2 + drivers/staging/android/ion/ion_system_heap.c | 20 ++- 5 files changed, 210 insertions(+), 19 deletions(-)
drivers/staging/android/ion/ion.c +152 -10
@@ -34,7 +34,6 @@#include "ion.h" #include "ion_priv.h" -#define DEBUG /** * struct ion_device - the metadata of the ion device node
@@ -127,6 +126,8 @@ static void ion_buffer_add(struct ion_device *dev,rb_insert_color (&buffer->node, &dev->buffers); } +static int ion_buffer_alloc_dirty (struct ion_buffer *buffer); +/* this function should only be called while dev->lock is held */ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, struct ion_device *dev,
@@ -154,15 +155,38 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, buffer->dev = dev; buffer->size = len; + buffer->flags = flags; - table = buffer->heap->ops->map_dma (buffer->heap, buffer); + table = heap->ops->map_dma (heap, buffer);if (IS_ERR_OR_NULL (table)) { heap->ops->free (buffer);kfree (buffer);return ERR_PTR (PTR_ERR (table)); } buffer->sg_table = table; +if (buffer->flags &ION_FLAG_CACHED ) +for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, + i) { +if (sg_dma_len (sg) ==PAGE_SIZE ) +continue ; +pr_err ("%s: cached mappings must have pagewise " +"sg_lists\n" , __func__); + heap->ops->unmap_dma (heap, buffer); +kfree (buffer); +return ERR_PTR (-EINVAL ); + } + ret =ion_buffer_alloc_dirty (buffer); +if (ret) { + heap->ops->unmap_dma (heap, buffer); + heap->ops->free (buffer); +kfree (buffer); +return ERR_PTR (ret); + } + + buffer->dev = dev; + buffer->size = len; +INIT_LIST_HEAD (&buffer->vmas);mutex_init (&buffer->lock);/* this will set up dma addresses for the sglist -- it is not technically correct as per the dma api -- a specific
@@ -313,13 +337,16 @@ static void ion_handle_add(struct ion_client *client, struct ion_handle *handle) }struct ion_handle *ion_alloc (struct ion_client *client,size_t len, -size_t align,unsigned int flags) +size_t align,unsigned int heap_mask, +unsigned int flags) {struct rb_node *n;struct ion_handle *handle;struct ion_device *dev = client->dev;struct ion_buffer *buffer =NULL ; +pr_debug ("%s: len %d align %d heap_mask %u flags %x\n" , __func__, len, + align, heap_mask, flags);/* * traverse the list of heaps available in this system in priority * order. If the heap type is supported by the client, and matches the
@@ -338,7 +365,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,if (!((1 << heap->type) & client->heap_mask))continue ;/* if the caller didn't specify this heap type */ -if (!((1 << heap->id) & flags)) +if (!((1 << heap->id) & heap_mask))continue ; buffer =ion_buffer_create (heap, dev, len, align, flags);if (!IS_ERR_OR_NULL (buffer))
@@ -647,12 +674,18 @@ struct sg_table *ion_sg_table(struct ion_client *client,return table; } +static void ion_buffer_sync_for_device (struct ion_buffer *buffer, +struct device *dev, +enum dma_data_direction direction); +static struct sg_table *ion_map_dma_buf (struct dma_buf_attachment *attachment,enum dma_data_direction direction) {struct dma_buf *dmabuf = attachment->dmabuf;struct ion_buffer *buffer = dmabuf->priv; +if (buffer->flags &ION_FLAG_CACHED ) +ion_buffer_sync_for_device (buffer, attachment->dev, direction);return buffer->sg_table; }
@@ -662,10 +695,112 @@ static void ion_unmap_dma_buf(struct dma_buf_attachment *attachment, { } +static int ion_buffer_alloc_dirty (struct ion_buffer *buffer) +{ +unsigned long pages = buffer->sg_table->nents; +unsigned long length = (pages +BITS_PER_LONG -1 )/BITS_PER_LONG ; + + buffer->dirty =kzalloc (length *sizeof (unsigned long ),GFP_KERNEL ); +if (!buffer->dirty) +return -ENOMEM ; +return 0 ; +} + +struct ion_vma_list { +struct list_head list; +struct vm_area_struct *vma; +}; + +static void ion_buffer_sync_for_device (struct ion_buffer *buffer, +struct device *dev, +enum dma_data_direction dir) +{ +struct scatterlist *sg; +int i; +struct ion_vma_list *vma_list; + +pr_debug ("%s: syncing for device %s\n" , __func__, + dev ?dev_name (dev) :"null" ); +mutex_lock (&buffer->lock); +for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) { +if (!test_bit (i, buffer->dirty)) +continue ; +dma_sync_sg_for_device (dev, sg,1 , dir); +clear_bit (i, buffer->dirty); + } +list_for_each_entry (vma_list, &buffer->vmas, list) { +struct vm_area_struct *vma = vma_list->vma; + +zap_page_range (vma, vma->vm_start, vma->vm_end - vma->vm_start, +NULL ); + } +mutex_unlock (&buffer->lock); +} + +int ion_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) +{ +struct ion_buffer *buffer = vma->vm_private_data; +struct scatterlist *sg; +int i; + +mutex_lock (&buffer->lock); +set_bit (vmf->pgoff, buffer->dirty); + +for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) { +if (i != vmf->pgoff) +continue ; +dma_sync_sg_for_cpu (NULL , sg,1 ,DMA_BIDIRECTIONAL ); +vm_insert_page (vma, (unsigned long )vmf->virtual_address, +sg_page (sg)); +break ; + } +mutex_unlock (&buffer->lock); +return VM_FAULT_NOPAGE ; +} + +static void ion_vm_open (struct vm_area_struct *vma) +{ +struct ion_buffer *buffer = vma->vm_private_data; +struct ion_vma_list *vma_list; + + vma_list =kmalloc (sizeof (struct ion_vma_list ),GFP_KERNEL ); +if (!vma_list) +return ; + vma_list->vma = vma; +mutex_lock (&buffer->lock); +list_add (&vma_list->list, &buffer->vmas); +mutex_unlock (&buffer->lock); +pr_debug ("%s: adding %p\n" , __func__, vma); +} + +static void ion_vm_close (struct vm_area_struct *vma) +{ +struct ion_buffer *buffer = vma->vm_private_data; +struct ion_vma_list *vma_list, *tmp; + +pr_debug ("%s\n" , __func__); +mutex_lock (&buffer->lock); +list_for_each_entry_safe (vma_list, tmp, &buffer->vmas, list) { +if (vma_list->vma != vma) +continue ; +list_del (&vma_list->list); +kfree (vma_list); +pr_debug ("%s: deleting %p\n" , __func__, vma); +break ; + } +mutex_unlock (&buffer->lock); +} + +struct vm_operations_struct ion_vma_ops = { + .open = ion_vm_open, + .close = ion_vm_close, + .fault = ion_vm_fault, +}; +static int ion_mmap (struct dma_buf *dmabuf,struct vm_area_struct *vma) {struct ion_buffer *buffer = dmabuf->priv; -int ret; +int ret =0 ;if (!buffer->heap->ops->map_user) { pr_err("%s: this heap does not define a method for mapping "
@@ -673,10 +808,17 @@ static int ion_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)return -EINVAL ; } -mutex_lock (&buffer->lock); -/* now map it to userspace */ - ret = buffer->heap->ops->map_user (buffer->heap, buffer, vma); -mutex_unlock (&buffer->lock); +if (buffer->flags &ION_FLAG_CACHED ) { + vma->vm_private_data = buffer; + vma->vm_ops = &ion_vma_ops; +ion_vm_open (vma); + }else { + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +mutex_lock (&buffer->lock); +/* now map it to userspace */ + ret = buffer->heap->ops->map_user (buffer->heap, buffer, vma); +mutex_unlock (&buffer->lock); + }if (ret) pr_err("%s: failure mapping buffer to userspace\n",
@@ -828,7 +970,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; data.handle =ion_alloc (client, data.len, data.align, - data.flags); + data.heap_mask, data.flags);if (IS_ERR (data.handle))return PTR_ERR (data.handle);
drivers/staging/android/ion/ion.h +17 -2
@@ -42,6 +42,15 @@ enum ion_heap_type {#define ION_HEAP_SYSTEM_CONTIG_MASK (1 << ION_HEAP_TYPE_SYSTEM_CONTIG)#define ION_HEAP_CARVEOUT_MASK (1 << ION_HEAP_TYPE_CARVEOUT) +/** +* heap flags - the lower 16 bits are used by core ion, the upper 16 +* bits are reserved for use by the heaps themselves. +*/ +#define ION_FLAG_CACHED 1/* mappings of this buffer should be +cached, ion will do cache +maintenance when the buffer is +mapped for dma */ +#ifdef __KERNEL__struct ion_device ;struct ion_heap ;
@@ -121,14 +130,18 @@ void ion_client_destroy(struct ion_client *client);* @len: size of the allocation * @align: requested allocation alignment, lots of hardware blocks have * alignment requirements of some kind -* @flags: mask of heaps to allocate from, if multiple bits are set +* @heap_mask: mask of heaps to allocate from, if multiple bits are set * heaps will be tried in order from lowest to highest order bit +* @flags: heap flags, the low 16 bits are consumed by ion, the high 16 +* bits are passed on to the respective heap and can be heap +* custom * * Allocate memory in one of the heaps provided in heap mask and return * an opaque handle to it. */ struct ion_handle *ion_alloc (struct ion_client *client,size_t len, -size_t align,unsigned int flags); +size_t align,unsigned int heap_mask, +unsigned int flags);/** * ion_free - free a handle
@@ -218,6 +231,7 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd);* struct ion_allocation_data - metadata passed from userspace for allocations * @len: size of the allocation * @align: required alignment of the allocation +* @heap_mask: mask of heaps to allocate from * @flags: flags passed to heap * @handle: pointer that will be populated with a cookie to use to refer * to this allocation
@@ -227,6 +241,7 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd);struct ion_allocation_data {size_t len;size_t align; +unsigned int heap_mask;unsigned int flags;struct ion_handle *handle; };
drivers/staging/android/ion/ion_carveout_heap.c +22 -4
@@ -84,23 +84,41 @@ static void ion_carveout_heap_free(struct ion_buffer *buffer) buffer->priv_phys =ION_CARVEOUT_ALLOCATE_FAIL ; } -struct scatterlist *ion_carveout_heap_map_dma (struct ion_heap *heap, +struct sg_table *ion_carveout_heap_map_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -return ERR_PTR (-EINVAL ); +struct sg_table *table; +int ret; + + table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); +if (!table) +return ERR_PTR (-ENOMEM ); + ret =sg_alloc_table (table,1 ,GFP_KERNEL ); +if (ret) { +kfree (table); +return ERR_PTR (ret); + } +sg_set_page (table->sgl,phys_to_page (buffer->priv_phys), buffer->size, +0 ); +return table; }void ion_carveout_heap_unmap_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -return ; +sg_free_table (buffer->sg_table); }void *ion_carveout_heap_map_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { +int mtype =MT_MEMORY_NONCACHED ; + +if (buffer->flags &ION_FLAG_CACHED ) + mtype =MT_MEMORY ; +return __arm_ioremap (buffer->priv_phys, buffer->size, -MT_MEMORY_NONCACHED ); + mtype); } void ion_carveout_heap_unmap_kernel(struct ion_heap *heap,
drivers/staging/android/ion/ion_priv.h +2 -0
@@ -60,6 +60,8 @@ struct ion_buffer {void *vaddr;int dmap_cnt;struct sg_table *sg_table; +unsigned long *dirty; +struct list_head vmas; };/**
drivers/staging/android/ion/ion_system_heap.c +17 -3
@@ -87,13 +87,20 @@ void *ion_system_heap_map_kernel(struct ion_heap *heap,struct scatterlist *sg;int i;void *vaddr; +pgprot_t pgprot;struct sg_table *table = buffer->priv_virt;struct page **pages =kmalloc (sizeof (struct page *) * table->nents,GFP_KERNEL );for_each_sg (table->sgl, sg, table->nents, i) pages[i] =sg_page (sg); - vaddr =vmap (pages, table->nents,VM_MAP ,PAGE_KERNEL ); + +if (buffer->flags &ION_FLAG_CACHED ) + pgprot =PAGE_KERNEL ; +else + pgprot =pgprot_writecombine (PAGE_KERNEL ); + + vaddr =vmap (pages, table->nents,VM_MAP , pgprot);kfree (pages);return vaddr;
@@ -179,7 +186,7 @@ static int ion_system_contig_heap_phys(struct ion_heap *heap, }struct sg_table *ion_system_contig_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +struct ion_buffer *buffer) {struct sg_table *table;int ret;
@@ -197,6 +204,13 @@ struct sg_table *ion_system_contig_heap_map_dma(struct ion_heap *heap,return table; } +void ion_system_contig_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +sg_free_table (buffer->sg_table); +kfree (buffer->sg_table); +} +int ion_system_contig_heap_map_user (struct ion_heap *heap,struct ion_buffer *buffer,struct vm_area_struct *vma)
@@ -213,7 +227,7 @@ static struct ion_heap_ops kmalloc_ops = { .free = ion_system_contig_heap_free, .phys = ion_system_contig_heap_phys, .map_dma = ion_system_contig_heap_map_dma, - .unmap_dma = ion_system_heap_unmap_dma, + .unmap_dma = ion_system_contig_heap_unmap_dma, .map_kernel = ion_system_heap_map_kernel, .unmap_kernel = ion_system_heap_unmap_kernel, .map_user = ion_system_contig_heap_map_user,
Signature · 1 line
1.8.3.2
PATCH 17/115 gpu: ion: Modify the system heap to try to allocate large/huge pages
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
On some systems there is a performance benefit to reducing tlb pressure by minimizing the number of chunks in an allocation.
--- drivers/staging/android/ion/ion_system_heap.c | 116 ++++++++++++++++++++------ 1 file changed, 90 insertions(+), 26 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +90 -26
@@ -14,7 +14,10 @@* */ +#include <asm/page.h> +#include <linux/dma-mapping.h> #include <linux/err.h> +#include <linux/highmem.h> #include <linux/mm.h> #include <linux/scatterlist.h> #include <linux/slab.h>
@@ -22,6 +25,35 @@#include "ion.h" #include "ion_priv.h" +struct page_info { +struct page *page; +unsigned long order; +struct list_head list; +}; + +static struct page_info *alloc_largest_available (unsigned long size) +{ +static unsigned int orders[] = {8 ,4 ,0 }; +struct page *page; +struct page_info *info; +int i; + +for (i =0 ; i <ARRAY_SIZE (orders); i++) { +if (size < (1 << orders[i]) *PAGE_SIZE ) +continue ; + page =alloc_pages (GFP_HIGHUSER | __GFP_ZERO | __GFP_COMP | + __GFP_NOWARN | __GFP_NORETRY, orders[i]); +if (!page) +continue ; +split_page (page, orders[i]); + info =kmap (page); + info->page = page; + info->order = orders[i]; +return info; + } +return NULL ; +} + static int ion_system_heap_allocate(struct ion_heap *heap, struct ion_buffer *buffer, unsigned long size, unsigned long align,
@@ -29,30 +61,54 @@ static int ion_system_heap_allocate(struct ion_heap *heap, {struct sg_table *table;struct scatterlist *sg; -int i, j; -int npages =PAGE_ALIGN (size) /PAGE_SIZE ; +int ret; +struct list_head pages; +struct page_info *info, *tmp_info; +int i; +long size_remaining =PAGE_ALIGN (size); + +INIT_LIST_HEAD (&pages); +while (size_remaining >0 ) { + info =alloc_largest_available (size_remaining); +if (!info) + goto err; +list_add_tail (&info->list, &pages); + size_remaining -= (1 << info->order) *PAGE_SIZE ; + } table =kmalloc (sizeof (struct sg_table ),GFP_KERNEL );if (!table) -return -ENOMEM ; - i =sg_alloc_table (table, npages,GFP_KERNEL ); -if (i) - goto err0; -for_each_sg (table->sgl, sg, table->nents, i) { -struct page *page; - page =alloc_page (GFP_HIGHUSER | __GFP_ZERO); -if (!page) - goto err1; -sg_set_page (sg, page,PAGE_SIZE ,0 ); + goto err; + + ret =sg_alloc_table (table,PAGE_ALIGN (size) /PAGE_SIZE ,GFP_KERNEL ); +if (ret) + goto err1; + + sg = table->sgl; +list_for_each_entry_safe (info, tmp_info, &pages, list) { +struct page *page = info->page; +for (i =0 ; i < (1 << info->order); i++) { +sg_set_page (sg, page + i,PAGE_SIZE ,0 ); + sg =sg_next (sg); + } +list_del (&info->list); +memset (info,0 ,sizeof (struct page_info )); +kunmap (page); } + +dma_sync_sg_for_device (NULL , table->sgl, table->nents, +DMA_BIDIRECTIONAL ); + buffer->priv_virt = table;return 0 ; err1: -for_each_sg (table->sgl, sg, i, j) -__free_page (sg_page (sg)); -sg_free_table (table); -err0:kfree (table); +err: +list_for_each_entry (info, &pages, list) { +for (i =0 ; i < (1 << info->order); i++) +__free_page (info->page + i); +kunmap (info->page); + }return -ENOMEM ; }
@@ -63,7 +119,7 @@ void ion_system_heap_free(struct ion_buffer *buffer)struct sg_table *table = buffer->priv_virt;for_each_sg (table->sgl, sg, table->nents, i) -__free_page (sg_page (sg)); +__free_pages (sg_page (sg),get_order (sg_dma_len (sg)));if (buffer->sg_table)sg_free_table (buffer->sg_table);kfree (buffer->sg_table);
@@ -85,22 +141,29 @@ void *ion_system_heap_map_kernel(struct ion_heap *heap, struct ion_buffer *buffer) {struct scatterlist *sg; -int i; +int i, j;void *vaddr;pgprot_t pgprot;struct sg_table *table = buffer->priv_virt; -struct page **pages =kmalloc (sizeof (struct page *) * table->nents, -GFP_KERNEL ); - -for_each_sg (table->sgl, sg, table->nents, i) - pages[i] =sg_page (sg); +int npages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; +struct page **pages =kzalloc (sizeof (struct page *) * npages, +GFP_KERNEL ); +struct page **tmp = pages;if (buffer->flags &ION_FLAG_CACHED ) pgprot =PAGE_KERNEL ;else pgprot =pgprot_writecombine (PAGE_KERNEL ); - vaddr =vmap (pages, table->nents,VM_MAP , pgprot); +for_each_sg (table->sgl, sg, table->nents, i) { +int npages_this_entry =PAGE_ALIGN (sg_dma_len (sg)) /PAGE_SIZE ; +struct page *page =sg_page (sg); +BUG_ON (i >= npages); +for (j =0 ; j < npages_this_entry; j++) { + *(tmp++) = page++; + } + } + vaddr =vmap (pages, npages,VM_MAP , pgprot);kfree (pages);return vaddr;
@@ -126,8 +189,9 @@ int ion_system_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer, offset--;continue ; } -vm_insert_page (vma, addr,sg_page (sg)); - addr +=PAGE_SIZE ; +remap_pfn_range (vma, addr,page_to_pfn (sg_page (sg)), +sg_dma_len (sg), vma->vm_page_prot); + addr +=sg_dma_len (sg); }return 0 ; }
Signature · 1 line
1.8.3.2
PATCH 18/115 gpu: ion: Add explicit sync ioctl
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This is deprecated in favor of using the dma_buf api which will automatically sync a buffer to memory when it is mapped to a device. However, that functionality is not ready, so this patch adds the ability to sync a buffer explicitly.
--- drivers/staging/android/ion/ion.c | 38 ++++++++++++++++++++++++++++++++++++-- drivers/staging/android/ion/ion.h | 12 +++++++++++- 2 files changed, 47 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion.c +36 -2
@@ -684,8 +684,7 @@ static struct sg_table *ion_map_dma_buf(struct dma_buf_attachment *attachment,struct dma_buf *dmabuf = attachment->dmabuf;struct ion_buffer *buffer = dmabuf->priv; -if (buffer->flags &ION_FLAG_CACHED ) -ion_buffer_sync_for_device (buffer, attachment->dev, direction); +ion_buffer_sync_for_device (buffer, attachment->dev, direction);return buffer->sg_table; }
@@ -721,6 +720,10 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,pr_debug ("%s: syncing for device %s\n" , __func__, dev ?dev_name (dev) :"null" ); + +if (!(buffer->flags &ION_FLAG_CACHED )) +return ; +mutex_lock (&buffer->lock);for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) {if (!test_bit (i, buffer->dirty))
@@ -958,6 +961,28 @@ end:return handle; } +static int ion_sync_for_device (struct ion_client *client,int fd) +{ +struct dma_buf *dmabuf; +struct ion_buffer *buffer; + + dmabuf =dma_buf_get (fd); +if (IS_ERR_OR_NULL (dmabuf)) +return PTR_ERR (dmabuf); + +/* if this memory came from ion */ +if (dmabuf->ops != &dma_buf_ops) { +pr_err ("%s: can not sync dmabuf from another exporter\n" , + __func__); +dma_buf_put (dmabuf); +return -EINVAL ; + } + buffer = dmabuf->priv; +ion_buffer_sync_for_device (buffer,NULL ,DMA_BIDIRECTIONAL ); +dma_buf_put (dmabuf); +return 0 ; +} +static long ion_ioctl (struct file *filp,unsigned int cmd,unsigned long arg) {struct ion_client *client = filp->private_data;
@@ -1022,6 +1047,15 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)return -EFAULT ;break ; } +case ION_IOC_SYNC : + { +struct ion_fd_data data; +if (copy_from_user (&data, (void __user *)arg, +sizeof (struct ion_fd_data ))) +return -EFAULT ; +ion_sync_for_device (client, data.fd); +break ; + }case ION_IOC_CUSTOM : {struct ion_device *dev = client->dev;
drivers/staging/android/ion/ion.h +11 -1
@@ -328,7 +328,17 @@ struct ion_custom_data {* descriptor obtained from ION_IOC_SHARE and returns the struct with the handle * filed set to the corresponding opaque handle. */ -#define ION_IOC_IMPORT _IOWR(ION_IOC_MAGIC, 5, int) +#define ION_IOC_IMPORT _IOWR(ION_IOC_MAGIC, 5, struct ion_fd_data) + +/** +* DOC: ION_IOC_SYNC - syncs a shared file descriptors to memory +* +* Deprecated in favor of using the dma_buf api's correctly (syncing +* will happend automatically when the buffer is mapped to a device). +* If necessary should be used after touching a cached buffer from the cpu, +* this will make the buffer in memory coherent. +*/ +#define ION_IOC_SYNC _IOWR(ION_IOC_MAGIC, 7, struct ion_fd_data)/** * DOC: ION_IOC_CUSTOM - call architecture specific ion ioctl
Signature · 1 line
1.8.3.2
PATCH 19/115 gpu: ion: do not ask for compound pages in system heap
In reply to John StultzFrom: Dima Zavin <[email protected]>
--- drivers/staging/android/ion/ion_system_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -41,7 +41,7 @@ static struct page_info *alloc_largest_available(unsigned long size)for (i =0 ; i <ARRAY_SIZE (orders); i++) {if (size < (1 << orders[i]) *PAGE_SIZE )continue ; - page =alloc_pages (GFP_HIGHUSER | __GFP_ZERO | __GFP_COMP | + page =alloc_pages (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY, orders[i]);if (!page)continue ;
Signature · 1 line
1.8.3.2
PATCH 20/115 gpu: ion: Add missing argument to WARN call
In reply to John StultzFrom: Olav Haugan <[email protected]>
The condition argument to the WARN call in ion_free and ion_share_dma_buf are missing. Add the argument to allow correct printing of warning message.
--- drivers/staging/android/ion/ion.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion.c +2 -2
@@ -408,7 +408,7 @@ void ion_free(struct ion_client *client, struct ion_handle *handle)mutex_unlock (&client->lock);if (!valid_handle) { -WARN ("%s: invalid handle passed to free.\n" , __func__); +WARN (1 ,"%s: invalid handle passed to free.\n" , __func__);return ; }ion_handle_put (handle);
@@ -906,7 +906,7 @@ int ion_share_dma_buf(struct ion_client *client, struct ion_handle *handle) valid_handle =ion_handle_validate (client, handle);mutex_unlock (&client->lock);if (!valid_handle) { -WARN ("%s: invalid handle passed to share.\n" , __func__); +WARN (1 ,"%s: invalid handle passed to share.\n" , __func__);return -EINVAL ; }
Signature · 1 line
1.8.3.2
PATCH 21/115 gpu: ion: Add EXPORT_SYMBOL to functions
In reply to John StultzFrom: Olav Haugan <[email protected]>
Several functions in the ion interface is missing EXPORT_SYMBOL. This is needed to allow clients to use these functions from kernel modules.
Add EXPORT_SYMBOL to functions that are supposed to be exposed.
--- drivers/staging/android/ion/ion.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/staging/android/ion/ion.c +9 -0
@@ -396,6 +396,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,return handle; } +EXPORT_SYMBOL (ion_alloc);void ion_free (struct ion_client *client,struct ion_handle *handle) {
@@ -413,6 +414,7 @@ void ion_free(struct ion_client *client, struct ion_handle *handle) }ion_handle_put (handle); } +EXPORT_SYMBOL (ion_free);int ion_phys (struct ion_client *client,struct ion_handle *handle,ion_phys_addr_t *addr,size_t *len)
@@ -438,6 +440,7 @@ int ion_phys(struct ion_client *client, struct ion_handle *handle, ret = buffer->heap->ops->phys (buffer->heap, buffer, addr, len);return ret; } +EXPORT_SYMBOL (ion_phys);static void *ion_buffer_kmap_get (struct ion_buffer *buffer) {
@@ -517,6 +520,7 @@ void *ion_map_kernel(struct ion_client *client, struct ion_handle *handle)mutex_unlock (&client->lock);return vaddr; } +EXPORT_SYMBOL (ion_map_kernel);void ion_unmap_kernel (struct ion_client *client,struct ion_handle *handle) {
@@ -529,6 +533,7 @@ void ion_unmap_kernel(struct ion_client *client, struct ion_handle *handle)mutex_unlock (&buffer->lock);mutex_unlock (&client->lock); } +EXPORT_SYMBOL (ion_unmap_kernel);static int ion_debug_client_show (struct seq_file *s,void *unused) {
@@ -654,6 +659,7 @@ void ion_client_destroy(struct ion_client *client)kfree (client); } +EXPORT_SYMBOL (ion_client_destroy);struct sg_table *ion_sg_table (struct ion_client *client,struct ion_handle *handle)
@@ -673,6 +679,7 @@ struct sg_table *ion_sg_table(struct ion_client *client,mutex_unlock (&client->lock);return table; } +EXPORT_SYMBOL (ion_sg_table); static void ion_buffer_sync_for_device(struct ion_buffer *buffer, struct device *dev,
@@ -924,6 +931,7 @@ int ion_share_dma_buf(struct ion_client *client, struct ion_handle *handle) }return fd; } +EXPORT_SYMBOL (ion_share_dma_buf);struct ion_handle *ion_import_dma_buf (struct ion_client *client,int fd) {
@@ -960,6 +968,7 @@ end:dma_buf_put (dmabuf);return handle; } +EXPORT_SYMBOL (ion_import_dma_buf);static int ion_sync_for_device (struct ion_client *client,int fd) {
Signature · 1 line
1.8.3.2
PATCH 22/115 gpu: ion: IOCTL return success when error occurs
In reply to John StultzFrom: Olav Haugan <[email protected]>
ION_IOC_MAP, ION_IOC_SHARE, and ION_IOC_IMPORT may return success when an error occurs.
Add correct error handling to ION_IOC_MAP, ION_IOC_SHARE, and ION_IOC_IMPORT.
--- drivers/staging/android/ion/ion.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +8 -1
@@ -1040,20 +1040,27 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) data.fd =ion_share_dma_buf (client, data.handle);if (copy_to_user ((void __user *)arg, &data,sizeof (data)))return -EFAULT ; +if (data.fd <0 ) +return data.fd;break ; }case ION_IOC_IMPORT : {struct ion_fd_data data; +int ret =0 ;if (copy_from_user (&data, (void __user *)arg,sizeof (struct ion_fd_data )))return -EFAULT ; data.handle =ion_import_dma_buf (client, data.fd); -if (IS_ERR (data.handle)) +if (IS_ERR (data.handle)) { + ret =PTR_ERR (data.handle); data.handle =NULL ; + }if (copy_to_user ((void __user *)arg, &data,sizeof (struct ion_fd_data )))return -EFAULT ; +if (ret <0 ) +return ret;break ; }case ION_IOC_SYNC :
Signature · 1 line
1.8.3.2
PATCH 23/115 gpu: ion: Don't call ion_buffer_put on error path
In reply to John StultzFrom: Laura Abbott <[email protected]>
If dma_buf_fd fails, the dma_buf needs to be cleaned up by calling dma_buf_put. dma_buf_put will call ion_dma_buf_release which in turn calls ion_buffer_put to clean up the buffer reference. Calling ion_buffer_put after dma_buf_put drops the reference count by one more which is incorrect. Fix this by getting rid of the extra ion_buffer_put call.
--- drivers/staging/android/ion/ion.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion.c +2 -3
@@ -925,10 +925,9 @@ int ion_share_dma_buf(struct ion_client *client, struct ion_handle *handle)return PTR_ERR (dmabuf); } fd =dma_buf_fd (dmabuf,O_CLOEXEC ); -if (fd <0 ) { +if (fd <0 )dma_buf_put (dmabuf); -ion_buffer_put (buffer); - } +return fd; }EXPORT_SYMBOL (ion_share_dma_buf);
Signature · 1 line
1.8.3.2
PATCH 24/115 gpu: ion: Only map as much of the vma as the user requested
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion_system_heap.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/staging/android/ion/ion_system_heap.c +2 -0
@@ -192,6 +192,8 @@ int ion_system_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer,remap_pfn_range (vma, addr,page_to_pfn (sg_page (sg)),sg_dma_len (sg), vma->vm_page_prot); addr +=sg_dma_len (sg); +if (addr >= vma->vm_end) +return 0 ; }return 0 ; }
Signature · 1 line
1.8.3.2
PATCH 25/115 gpu: ion: Switch to using kmalloc rather than kmap during allocation
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Previously, metadata was stored in the allocated pages themselves during allocation. However the system can only have a limited number of kmapped pages. A very large allocation might exceed this limit.
--- drivers/staging/android/ion/ion_system_heap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +3 -3
@@ -46,7 +46,7 @@ static struct page_info *alloc_largest_available(unsigned long size)if (!page)continue ;split_page (page, orders[i]); - info =kmap (page); + info =kmalloc (sizeof (struct page_info *),GFP_KERNEL ); info->page = page; info->order = orders[i];return info;
@@ -93,7 +93,7 @@ static int ion_system_heap_allocate(struct ion_heap *heap, }list_del (&info->list);memset (info,0 ,sizeof (struct page_info )); -kunmap (page); +kfree (info); } dma_sync_sg_for_device(NULL, table->sgl, table->nents,
@@ -107,7 +107,7 @@ err:list_for_each_entry (info, &pages, list) {for (i =0 ; i < (1 << info->order); i++)__free_page (info->page + i); -kunmap (info->page); +kfree (info); }return -ENOMEM ; }
Signature · 1 line
1.8.3.2
PATCH 26/115 gpu: ion: fix page offset in dma_buf_kmap()
In reply to John StultzFrom: Greg Hackmann <[email protected]>
--- drivers/staging/android/ion/ion.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +1 -1
@@ -846,7 +846,7 @@ static void ion_dma_buf_release(struct dma_buf *dmabuf)static void *ion_dma_buf_kmap (struct dma_buf *dmabuf,unsigned long offset) {struct ion_buffer *buffer = dmabuf->priv; -return buffer->vaddr + offset; +return buffer->vaddr + offset *PAGE_SIZE ; } static void ion_dma_buf_kunmap(struct dma_buf *dmabuf, unsigned long offset,
Signature · 1 line
1.8.3.2
PATCH 27/115 gpu: ion: Fix race between ion_import and ion_free
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
If preemted during ion_free after the refcount is updated but before the handle can be removed from the rb_tree, import might find that handle in the tree and try to reuse it when execution returns to free, the handle will be cleaned up leaving the caller of import with a corrupt handle. This patch modifies the locking to protect agains this race.
--- drivers/staging/android/ion/ion.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/staging/android/ion/ion.c +1 -4
@@ -253,8 +253,6 @@ static void ion_handle_destroy(struct kref *kref)struct ion_client *client = handle->client;struct ion_buffer *buffer = handle->buffer; -mutex_lock (&client->lock); -mutex_lock (&buffer->lock);while (handle->kmap_cnt)ion_handle_kmap_put (handle);
@@ -262,7 +260,6 @@ static void ion_handle_destroy(struct kref *kref)if (!RB_EMPTY_NODE (&handle->node))rb_erase (&handle->node, &client->handles); -mutex_unlock (&client->lock);ion_buffer_put (buffer);kfree (handle);
@@ -406,13 +403,13 @@ void ion_free(struct ion_client *client, struct ion_handle *handle)mutex_lock (&client->lock); valid_handle =ion_handle_validate (client, handle); -mutex_unlock (&client->lock);if (!valid_handle) {WARN (1 ,"%s: invalid handle passed to free.\n" , __func__);return ; }ion_handle_put (handle); +mutex_unlock (&client->lock); }EXPORT_SYMBOL (ion_free);
Signature · 1 line
1.8.3.2
PATCH 28/115 gpu: ion: Fix bug in ion_free
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion.c | 1 + 1 file changed, 1 insertion(+)
drivers/staging/android/ion/ion.c +1 -0
@@ -406,6 +406,7 @@ void ion_free(struct ion_client *client, struct ion_handle *handle)if (!valid_handle) {WARN (1 ,"%s: invalid handle passed to free.\n" , __func__); +mutex_unlock (&client->lock);return ; }ion_handle_put (handle);
Signature · 1 line
1.8.3.2
PATCH 29/115 gpu: ion: Add debug information for orphaned handles
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
It is possible for a buffer to exist only as a dma_buf file descriptor without it being held in any handles. When this occurs it is impossible to track where the buffer is in the system (without traversing every process in the system and inspecting its file table). When buffers are orphaned like this, copy the task comm and pid of the last client to hold them into the buffer so we have a debugging hint as to where this buffer came from. In practice this will probalby be the process that allocated the buffer.
--- drivers/staging/android/ion/ion.c | 58 ++++++++++++++++++++++++++++++++++ drivers/staging/android/ion/ion_priv.h | 14 ++++++++ 2 files changed, 72 insertions(+)
drivers/staging/android/ion/ion.c +58 -0
@@ -228,6 +228,37 @@ static int ion_buffer_put(struct ion_buffer *buffer)return kref_put (&buffer->ref, ion_buffer_destroy); } +static void ion_buffer_add_to_handle (struct ion_buffer *buffer) +{ +mutex_lock (&buffer->dev->lock); + buffer->handle_count++; +mutex_unlock (&buffer->dev->lock); +} + +static void ion_buffer_remove_from_handle (struct ion_buffer *buffer) +{ +/* +* when a buffer is removed from a handle, if it is not in +* any other handles, copy the taskcomm and the pid of the +* process it's being removed from into the buffer. At this +* point there will be no way to track what processes this buffer is +* being used by, it only exists as a dma_buf file descriptor. +* The taskcomm and pid can provide a debug hint as to where this fd +* is in the system +*/ +mutex_lock (&buffer->dev->lock); + buffer->handle_count--; +BUG_ON (buffer->handle_count <0 ); +if (!buffer->handle_count) { +struct task_struct *task; + + task = current->group_leader; +get_task_comm (buffer->task_comm, task); + buffer->pid =task_pid_nr (task); + } +mutex_unlock (&buffer->dev->lock); +} +static struct ion_handle *ion_handle_create (struct ion_client *client,struct ion_buffer *buffer) {
@@ -240,6 +271,7 @@ static struct ion_handle *ion_handle_create(struct ion_client *client,RB_CLEAR_NODE (&handle->node); handle->client = client;ion_buffer_get (buffer); +ion_buffer_add_to_handle (buffer); handle->buffer = buffer;return handle;
@@ -261,7 +293,9 @@ static void ion_handle_destroy(struct kref *kref)if (!RB_EMPTY_NODE (&handle->node))rb_erase (&handle->node, &client->handles); +ion_buffer_remove_from_handle (buffer);ion_buffer_put (buffer); +kfree (handle); }
@@ -1141,8 +1175,11 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)struct ion_heap *heap = s->private;struct ion_device *dev = heap->dev;struct rb_node *n; +size_t total_size =0 ; +size_t total_orphaned_size =0 ;seq_printf (s,"%16.s %16.s %16.s\n" ,"client" ,"pid" ,"size" ); +seq_printf (s,"----------------------------------------------------\n" );for (n =rb_first (&dev->clients); n; n =rb_next (n)) { struct ion_client *client = rb_entry(n, struct ion_client,
@@ -1161,6 +1198,27 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused) client->pid, size); } } +seq_printf (s,"----------------------------------------------------\n" ); +seq_printf (s,"orphaned allocations (info is from last known client):" +"\n" ); +mutex_lock (&dev->lock); +for (n =rb_first (&dev->buffers); n; n =rb_next (n)) { + struct ion_buffer *buffer = rb_entry(n, struct ion_buffer, + node); +if (buffer->heap->type == heap->type) + total_size += buffer->size; +if (!buffer->handle_count) { +seq_printf (s,"%16.s %16u %16u\n" , buffer->task_comm, + buffer->pid, buffer->size); + total_orphaned_size += buffer->size; + } + } +mutex_unlock (&dev->lock); +seq_printf (s,"----------------------------------------------------\n" ); +seq_printf (s,"%16.s %16u\n" ,"total orphaned" , + total_orphaned_size); +seq_printf (s,"%16.s %16u\n" ,"total " , total_size); +return 0 ; }
drivers/staging/android/ion/ion_priv.h +14 -0
@@ -21,6 +21,7 @@#include <linux/mm_types.h> #include <linux/mutex.h> #include <linux/rbtree.h> +#include <linux/sched.h> #include "ion.h"
@@ -43,6 +44,15 @@ struct ion_buffer *ion_handle_buffer(struct ion_handle *handle);* @vaddr: the kenrel mapping if kmap_cnt is not zero * @dmap_cnt: number of times the buffer is mapped for dma * @sg_table: the sg table for the buffer if dmap_cnt is not zero +* @dirty: bitmask representing which pages of this buffer have +* been dirtied by the cpu and need cache maintenance +* before dma +* @vmas: list of vma's mapping this buffer +* @handle_count: count of handles referencing this buffer +* @task_comm: taskcomm of last client to reference this buffer in a +* handle, used for debugging +* @pid: pid of last client to reference this buffer in a +* handle, used for debugging */ struct ion_buffer {struct kref ref;
@@ -62,6 +72,10 @@ struct ion_buffer {struct sg_table *sg_table;unsigned long *dirty;struct list_head vmas; +/* used to track orphaned buffers */ +int handle_count; +char task_comm[TASK_COMM_LEN ]; +pid_t pid; };/**
Signature · 1 line
1.8.3.2
PATCH 30/115 gpu: ion: Fix memory leak of dirty bits
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
drivers/staging/android/ion/ion.c +14 -11
@@ -164,24 +164,20 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,return ERR_PTR (PTR_ERR (table)); } buffer->sg_table = table; -if (buffer->flags &ION_FLAG_CACHED ) +if (buffer->flags &ION_FLAG_CACHED ) {for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) {if (sg_dma_len (sg) ==PAGE_SIZE )continue ;pr_err ("%s: cached mappings must have pagewise " "sg_lists\n" , __func__); - heap->ops->unmap_dma (heap, buffer); -kfree (buffer); -return ERR_PTR (-EINVAL ); + ret = -EINVAL ; + goto err; } - ret =ion_buffer_alloc_dirty (buffer); -if (ret) { - heap->ops->unmap_dma (heap, buffer); - heap->ops->free (buffer); -kfree (buffer); -return ERR_PTR (ret); + ret =ion_buffer_alloc_dirty (buffer); +if (ret) + goto err; } buffer->dev = dev;
@@ -200,6 +196,12 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,sg_dma_address (sg) =sg_phys (sg);ion_buffer_add (dev, buffer);return buffer; + +err: + heap->ops->unmap_dma (heap, buffer); + heap->ops->free (buffer); +kfree (buffer); +return ERR_PTR (ret); }static void ion_buffer_destroy (struct kref *kref)
@@ -209,12 +211,13 @@ static void ion_buffer_destroy(struct kref *kref)if (WARN_ON (buffer->kmap_cnt >0 )) buffer->heap->ops->unmap_kernel (buffer->heap, buffer); - buffer->heap->ops->unmap_dma (buffer->heap, buffer); buffer->heap->ops->free (buffer);mutex_lock (&dev->lock);rb_erase (&buffer->node, &dev->buffers);mutex_unlock (&dev->lock); +if (buffer->flags &ION_FLAG_CACHED ) +kfree (buffer->dirty);kfree (buffer); }
Signature · 1 line
1.8.3.2
PATCH 31/115 gpu: ion: Add support for cached mappings that don't fault
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
We have found that faulting in the mappings for cached allocations has a significant performance impact and is only a benefit if only a small part of the buffer is touched by the cpu (an uncommon case for software rendering). This patch introduces a ION_FLAG_CACHED_NEEDS_SYNC which determines whether a mapping should be created by faulting or at mmap time. If this flag is set, userspace must manage the caches explictly using the SYNC ioctl.
--- drivers/staging/android/ion/ion.c | 28 ++++++++++++++++++---------- drivers/staging/android/ion/ion.h | 3 +++ 2 files changed, 21 insertions(+), 10 deletions(-)
drivers/staging/android/ion/ion.c +18 -10
@@ -164,7 +164,8 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,return ERR_PTR (PTR_ERR (table)); } buffer->sg_table = table; -if (buffer->flags &ION_FLAG_CACHED ) { +if (buffer->flags &ION_FLAG_CACHED && + !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )) {for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) {if (sg_dma_len (sg) ==PAGE_SIZE )
@@ -763,7 +764,8 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,pr_debug ("%s: syncing for device %s\n" , __func__, dev ?dev_name (dev) :"null" ); -if (!(buffer->flags &ION_FLAG_CACHED )) +if (!(buffer->flags &ION_FLAG_CACHED ) || + (buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC ))return ;mutex_lock (&buffer->lock);
@@ -853,18 +855,22 @@ static int ion_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)return -EINVAL ; } -if (buffer->flags &ION_FLAG_CACHED ) { +if (buffer->flags &ION_FLAG_CACHED && + !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )) { vma->vm_private_data = buffer; vma->vm_ops = &ion_vma_ops;ion_vm_open (vma); - }else { - vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); -mutex_lock (&buffer->lock); -/* now map it to userspace */ - ret = buffer->heap->ops->map_user (buffer->heap, buffer, vma); -mutex_unlock (&buffer->lock); +return 0 ; } +if (!(buffer->flags &ION_FLAG_CACHED )) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); + +mutex_lock (&buffer->lock); +/* now map it to userspace */ + ret = buffer->heap->ops->map_user (buffer->heap, buffer, vma); +mutex_unlock (&buffer->lock); +if (ret)pr_err ("%s: failure mapping buffer to userspace\n" , __func__);
@@ -1021,7 +1027,9 @@ static int ion_sync_for_device(struct ion_client *client, int fd)return -EINVAL ; } buffer = dmabuf->priv; -ion_buffer_sync_for_device (buffer,NULL ,DMA_BIDIRECTIONAL ); + +dma_sync_sg_for_device (NULL , buffer->sg_table->sgl, + buffer->sg_table->nents,DMA_BIDIRECTIONAL );dma_buf_put (dmabuf);return 0 ; }
drivers/staging/android/ion/ion.h +3 -0
@@ -50,6 +50,9 @@ enum ion_heap_type {cached, ion will do cache maintenance when the buffer is mapped for dma */ +#define ION_FLAG_CACHED_NEEDS_SYNC 2/* mappings of this buffer will created +at mmap time, if this is set +caches must be managed manually */ #ifdef __KERNEL__struct ion_device ;
Signature · 1 line
1.8.3.2
PATCH 32/115 gpu: ion: optimize system heap for non fault buffers
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
If a buffer's user mappings are not going to be faulted in it need not be allocated page wise. We can optimize this common case by allocating an sglist of larger chunks rather than creating an entry for each page in the allocation.
--- drivers/staging/android/ion/ion.c | 21 ++++++++------ drivers/staging/android/ion/ion_priv.h | 9 ++++++ drivers/staging/android/ion/ion_system_heap.c | 40 ++++++++++++++++++++------- 3 files changed, 51 insertions(+), 19 deletions(-)
drivers/staging/android/ion/ion.c +12 -9
@@ -100,6 +100,12 @@ struct ion_handle {unsigned int kmap_cnt; }; +bool ion_buffer_fault_user_mappings (struct ion_buffer *buffer) +{ +return ((buffer->flags &ION_FLAG_CACHED ) && + !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )); +} +/* this function should only be called while dev->lock is held */ static void ion_buffer_add (struct ion_device *dev,struct ion_buffer *buffer)
@@ -145,6 +151,7 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,return ERR_PTR (-ENOMEM ); buffer->heap = heap; + buffer->flags = flags;kref_init (&buffer->ref); ret = heap->ops->allocate (heap, buffer, len, align, flags);
@@ -155,7 +162,6 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, buffer->dev = dev; buffer->size = len; - buffer->flags = flags; table = heap->ops->map_dma (heap, buffer);if (IS_ERR_OR_NULL (table)) {
@@ -164,14 +170,13 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,return ERR_PTR (PTR_ERR (table)); } buffer->sg_table = table; -if (buffer->flags &ION_FLAG_CACHED && - !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )) { +if (ion_buffer_fault_user_mappings (buffer)) {for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) {if (sg_dma_len (sg) ==PAGE_SIZE )continue ; -pr_err ("%s: cached mappings must have pagewise " -"sg_lists\n" , __func__); +pr_err ("%s: cached mappings that will be faulted in " +"must have pagewise sg_lists\n" , __func__); ret = -EINVAL ; goto err; }
@@ -764,8 +769,7 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,pr_debug ("%s: syncing for device %s\n" , __func__, dev ?dev_name (dev) :"null" ); -if (!(buffer->flags &ION_FLAG_CACHED ) || - (buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )) +if (!ion_buffer_fault_user_mappings (buffer))return ;mutex_lock (&buffer->lock);
@@ -855,8 +859,7 @@ static int ion_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)return -EINVAL ; } -if (buffer->flags &ION_FLAG_CACHED && - !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )) { +if (ion_buffer_fault_user_mappings (buffer)) { vma->vm_private_data = buffer; vma->vm_ops = &ion_vma_ops;ion_vm_open (vma);
drivers/staging/android/ion/ion_priv.h +9 -0
@@ -132,6 +132,15 @@ struct ion_heap { };/** +* ion_buffer_fault_user_mappings - fault in user mappings of this buffer +* @buffer: buffer +* +* indicates whether userspace mappings of this buffer will be faulted +* in, this can affect how buffers are allocated from the heap. +*/ +bool ion_buffer_fault_user_mappings (struct ion_buffer *buffer); + +/** * ion_device_create - allocates and returns an ion device * @custom_ioctl: arch specific ioctl function if applicable *
drivers/staging/android/ion/ion_system_heap.c +30 -10
@@ -31,7 +31,8 @@ struct page_info {struct list_head list; }; -static struct page_info *alloc_largest_available (unsigned long size) +static struct page_info *alloc_largest_available (unsigned long size, +bool split_pages) {static unsigned int orders[] = {8 ,4 ,0 };struct page *page;
@@ -45,7 +46,8 @@ static struct page_info *alloc_largest_available(unsigned long size) __GFP_NOWARN | __GFP_NORETRY, orders[i]);if (!page)continue ; -split_page (page, orders[i]); +if (split_pages) +split_page (page, orders[i]); info =kmalloc (sizeof (struct page_info *),GFP_KERNEL ); info->page = page; info->order = orders[i];
@@ -64,35 +66,49 @@ static int ion_system_heap_allocate(struct ion_heap *heap,int ret;struct list_head pages;struct page_info *info, *tmp_info; -int i; +int i =0 ;long size_remaining =PAGE_ALIGN (size); +bool split_pages =ion_buffer_fault_user_mappings (buffer); +INIT_LIST_HEAD (&pages);while (size_remaining >0 ) { - info =alloc_largest_available (size_remaining); + info =alloc_largest_available (size_remaining, split_pages);if (!info) goto err;list_add_tail (&info->list, &pages); size_remaining -= (1 << info->order) *PAGE_SIZE ; + i++; } table =kmalloc (sizeof (struct sg_table ),GFP_KERNEL );if (!table) goto err; - ret =sg_alloc_table (table,PAGE_ALIGN (size) /PAGE_SIZE ,GFP_KERNEL ); +if (split_pages) + ret =sg_alloc_table (table,PAGE_ALIGN (size) /PAGE_SIZE , +GFP_KERNEL ); +else + ret =sg_alloc_table (table, i,GFP_KERNEL ); +if (ret) goto err1; sg = table->sgl;list_for_each_entry_safe (info, tmp_info, &pages, list) {struct page *page = info->page; -for (i =0 ; i < (1 << info->order); i++) { -sg_set_page (sg, page + i,PAGE_SIZE ,0 ); + +if (split_pages) { +for (i =0 ; i < (1 << info->order); i++) { +sg_set_page (sg, page + i,PAGE_SIZE ,0 ); + sg =sg_next (sg); + } + }else { +sg_set_page (sg, page, (1 << info->order) *PAGE_SIZE , +0 ); sg =sg_next (sg); }list_del (&info->list); -memset (info,0 ,sizeof (struct page_info ));kfree (info); }
@@ -105,8 +121,12 @@ err1:kfree (table); err:list_for_each_entry (info, &pages, list) { -for (i =0 ; i < (1 << info->order); i++) -__free_page (info->page + i); +if (split_pages) +for (i =0 ; i < (1 << info->order); i++) +__free_page (info->page + i); +else +__free_pages (info->page, info->order); +kfree (info); }return -ENOMEM ;
Signature · 1 line
1.8.3.2
PATCH 33/115 gpu: ion: Stop trying to allocate from an order on first failure
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
With this patch the system heap will only try to allocate from each order as long as allocations succeed. If it failes to obtain a higher order allocation, it doesn't retry that order.
--- drivers/staging/android/ion/ion_system_heap.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +9 -2
@@ -32,7 +32,8 @@ struct page_info { };static struct page_info *alloc_largest_available (unsigned long size, -bool split_pages) +bool split_pages, +unsigned int max_order) {static unsigned int orders[] = {8 ,4 ,0 };struct page *page;
@@ -42,6 +43,8 @@ static struct page_info *alloc_largest_available(unsigned long size,for (i =0 ; i <ARRAY_SIZE (orders); i++) {if (size < (1 << orders[i]) *PAGE_SIZE )continue ; +if (max_order < orders[i]) +continue ; page =alloc_pages (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY, orders[i]);if (!page)
@@ -71,13 +74,17 @@ static int ion_system_heap_allocate(struct ion_heap *heap,bool split_pages =ion_buffer_fault_user_mappings (buffer); +unsigned int max_order = orders[0 ]; +INIT_LIST_HEAD (&pages);while (size_remaining >0 ) { - info =alloc_largest_available (size_remaining, split_pages); + info =alloc_largest_available (size_remaining, split_pages, + max_order);if (!info) goto err;list_add_tail (&info->list, &pages); size_remaining -= (1 << info->order) *PAGE_SIZE ; + max_order = info->order; i++; }
Signature · 1 line
1.8.3.2
PATCH 34/115 gpu: ion: ion_system_heap: Fix bug preventing compilation
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion_system_heap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion_system_heap.c +2 -1
@@ -31,11 +31,12 @@ struct page_info {struct list_head list; }; +static unsigned int orders[] = {8 ,4 ,0 }; +static struct page_info *alloc_largest_available (unsigned long size,bool split_pages,unsigned int max_order) { -static unsigned int orders[] = {8 ,4 ,0 };struct page *page;struct page_info *info;int i;
Signature · 1 line
1.8.3.2
PATCH 35/115 gpu: ion: use vmalloc to allocate page array to map kernel
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
When ion_map_kernel is execute the system must allocate an array large enough to hold a pointer to each page in the buffer. If the buffer is very large and the system memory has become very fragmented, there may not be sufficient high order allocations available from kmalloc. Use vmalloc instead.
--- drivers/staging/android/ion/ion_system_heap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +5 -3
@@ -174,10 +174,12 @@ void *ion_system_heap_map_kernel(struct ion_heap *heap,pgprot_t pgprot;struct sg_table *table = buffer->priv_virt;int npages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; -struct page **pages =kzalloc (sizeof (struct page *) * npages, -GFP_KERNEL ); +struct page **pages =vmalloc (sizeof (struct page *) * npages);struct page **tmp = pages; +if (!pages) +return 0 ; +if (buffer->flags &ION_FLAG_CACHED ) pgprot =PAGE_KERNEL ;else
@@ -192,7 +194,7 @@ void *ion_system_heap_map_kernel(struct ion_heap *heap, } } vaddr =vmap (pages, npages,VM_MAP , pgprot); -kfree (pages); +vfree (pages);return vaddr; }
Signature · 1 line
1.8.3.2
PATCH 36/115 gpu: ion: Add ion_page_pool.
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch adds a new utility heaps can use to manage memory. In the past we have found it can be very expensive to manage the caches when allocating memory, but it is imposible to know whether a previous user of a given memory allocation had a cached mapping. This patch adds the ability to store a pool of pages that were previously used uncached so that cache maintenance only need be done when growing this pool. The pool also contains a shrinker so memory from the pool can be recovered in low memory conditions.
--- drivers/staging/android/ion/Makefile | 3 +- drivers/staging/android/ion/ion_page_pool.c | 163 ++++++++++++++++++++++++++++ drivers/staging/android/ion/ion_priv.h | 44 ++++++++ 3 files changed, 209 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/android/ion/ion_page_pool.c
drivers/staging/android/ion/Makefile +2 -1
@@ -1,2 +1,3 @@ -obj-$(CONFIG_ION) += ion.o ion_heap.o ion_system_heap.o ion_carveout_heap.o +obj-$(CONFIG_ION) += ion.o ion_heap.o ion_page_pool.o ion_system_heap.o \ + ion_carveout_heap.o obj-$(CONFIG_ION_TEGRA) += tegra/
drivers/staging/android/ion/ion_page_pool.c new file +163 -0
163 lines not on this page · raw message
drivers/staging/android/ion/ion_priv.h +44 -0
54 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 37/115 gpu: ion: Use the ion_page_pool from the system heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
With this change the system heap will use pagepools to avoid having to invalidate memory when it is allocated, a significant performance improvement on some systems.
--- drivers/staging/android/ion/ion.c | 18 ++- drivers/staging/android/ion/ion_priv.h | 11 ++ drivers/staging/android/ion/ion_system_heap.c | 191 +++++++++++++++++++++----- 3 files changed, 180 insertions(+), 40 deletions(-)
drivers/staging/android/ion/ion.c +14 -4
@@ -106,6 +106,11 @@ bool ion_buffer_fault_user_mappings(struct ion_buffer *buffer) !(buffer->flags & ION_FLAG_CACHED_NEEDS_SYNC)); } +bool ion_buffer_cached (struct ion_buffer *buffer) +{ +return !!(buffer->flags &ION_FLAG_CACHED ); +} +/* this function should only be called while dev->lock is held */ static void ion_buffer_add (struct ion_device *dev,struct ion_buffer *buffer)
@@ -1219,11 +1224,12 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)for (n =rb_first (&dev->buffers); n; n =rb_next (n)) { struct ion_buffer *buffer = rb_entry(n, struct ion_buffer, node); -if (buffer->heap->type == heap->type) - total_size += buffer->size; +if (buffer->heap->type != heap->type) +continue ; + total_size += buffer->size;if (!buffer->handle_count) { -seq_printf (s,"%16.s %16u %16u\n" , buffer->task_comm, - buffer->pid, buffer->size); +seq_printf (s,"%16.s %16u %16u %d %d\n" , buffer->task_comm, + buffer->pid, buffer->size, buffer->kmap_cnt, buffer->ref); total_orphaned_size += buffer->size; } }
@@ -1232,6 +1238,10 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)seq_printf (s,"%16.s %16u\n" ,"total orphaned" , total_orphaned_size);seq_printf (s,"%16.s %16u\n" ,"total " , total_size); +seq_printf (s,"----------------------------------------------------\n" ); + +if (heap->debug_show) + heap->debug_show (heap, s, unused);return 0 ; }
drivers/staging/android/ion/ion_priv.h +11 -0
@@ -118,6 +118,8 @@ struct ion_heap_ops {* allocating. These are specified by platform data and * MUST be unique * @name: used for debugging +* @debug_show: called when heap debug file is read to add any +* heap specific debug info to output * * Represents a pool of memory from which buffers can be made. In some * systems the only heap is regular system memory allocated via vmalloc.
@@ -131,9 +133,18 @@ struct ion_heap {struct ion_heap_ops *ops;int id;const char *name; +int (*debug_show)(struct ion_heap *heap,struct seq_file *,void *); };/** +* ion_buffer_cached - this ion buffer is cached +* @buffer: buffer +* +* indicates whether this ion buffer is cached +*/ +bool ion_buffer_cached (struct ion_buffer *buffer); + +/** * ion_buffer_fault_user_mappings - fault in user mappings of this buffer * @buffer: buffer *
drivers/staging/android/ion/ion_system_heap.c +155 -36
277 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 38/115 gpu: ion: Modify gfp flags in ion_system_heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
When allocations larger than order 4 are made, use _GFP_NORETRY and __GFP_NO_KSWAPD so kswapd doesn't get kicked off to reclaim these larger chunks. For smaller allocaitons, these are unnecessary, as the system should be able to reclaim these.
--- drivers/staging/android/ion/ion_system_heap.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +18 -7
@@ -26,6 +26,11 @@#include "ion.h" #include "ion_priv.h" +static unsigned int high_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | + __GFP_NOWARN | __GFP_NORETRY | + __GFP_NO_KSWAPD); +static unsigned int low_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | + __GFP_NOWARN);static const unsigned int orders[] = {8 ,4 ,0 };static const int num_orders =ARRAY_SIZE (orders);static int order_to_index (unsigned int order)
@@ -63,11 +68,15 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,struct ion_page_pool *pool = heap->pools[order_to_index (order)];struct page *page; -if (!cached) +if (!cached) { page =ion_page_pool_alloc (pool); -else - page =alloc_pages (GFP_HIGHUSER | __GFP_ZERO | - __GFP_NOWARN | __GFP_NORETRY, order); + }else { +gfp_t gfp_flags = low_order_gfp_flags; + +if (order >4 ) + gfp_flags = high_order_gfp_flags; + page =alloc_pages (gfp_flags, order); + }if (!page)return 0 ;if (split_pages)
@@ -342,9 +351,11 @@ struct ion_heap *ion_system_heap_create(struct ion_platform_heap *unused) goto err_alloc_pools;for (i =0 ; i < num_orders; i++) {struct ion_page_pool *pool; - pool =ion_page_pool_create (GFP_HIGHUSER | __GFP_ZERO | - __GFP_NOWARN | __GFP_NORETRY, - orders[i]); +gfp_t gfp_flags = low_order_gfp_flags; + +if (orders[i] >4 ) + gfp_flags = high_order_gfp_flags; + pool =ion_page_pool_create (gfp_flags, orders[i]);if (!pool) goto err_create_pool; heap->pools[i] = pool;
Signature · 1 line
1.8.3.2
PATCH 39/115 gpu: ion: Fix several issues with page pool
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Split out low and high mem pages so they are correctly reported when the shrinker is called.
Fix potential deadlock caused by holding the page pool lock while allocationg and also needing that lock from the shrink function
--- drivers/staging/android/ion/ion_page_pool.c | 77 ++++++++++++++++++--------- drivers/staging/android/ion/ion_priv.h | 12 +++-- drivers/staging/android/ion/ion_system_heap.c | 9 ++-- 3 files changed, 65 insertions(+), 33 deletions(-)
drivers/staging/android/ion/ion_page_pool.c +51 -26
@@ -54,24 +54,36 @@ static int ion_page_pool_add(struct ion_page_pool *pool, struct page *page)if (!item)return -ENOMEM ; item->page = page; -list_add_tail (&item->list, &pool->items); - pool->count++; +if (PageHighMem (page)) { +list_add_tail (&item->list, &pool->high_items); + pool->high_count++; + }else { +list_add_tail (&item->list, &pool->low_items); + pool->low_count++; + }return 0 ; } -static struct page *ion_page_pool_remove (struct ion_page_pool *pool) +static struct page *ion_page_pool_remove (struct ion_page_pool *pool,bool high) {struct ion_page_pool_item *item;struct page *page; -BUG_ON (!pool->count); -BUG_ON (list_empty (&pool->items)); +if (high) { +BUG_ON (!pool->high_count); + item =list_first_entry (&pool->high_items, + struct ion_page_pool_item, list); + pool->high_count--; + }else { +BUG_ON (!pool->low_count); + item =list_first_entry (&pool->low_items, + struct ion_page_pool_item, list); + pool->low_count--; + } - item = list_first_entry(&pool->items, struct ion_page_pool_item, list);list_del (&item->list); page = item->page;kfree (item); - pool->count--;return page; }
@@ -82,12 +94,15 @@ void *ion_page_pool_alloc(struct ion_page_pool *pool)BUG_ON (!pool);mutex_lock (&pool->mutex); -if (pool->count) - page =ion_page_pool_remove (pool); -else - page =ion_page_pool_alloc_pages (pool); +if (pool->high_count) + page =ion_page_pool_remove (pool, true); +else if (pool->low_count) + page =ion_page_pool_remove (pool, false);mutex_unlock (&pool->mutex); +if (!page) + page =ion_page_pool_alloc_pages (pool); +return page; }
@@ -97,9 +112,9 @@ void ion_page_pool_free(struct ion_page_pool *pool, struct page* page)mutex_lock (&pool->mutex); ret =ion_page_pool_add (pool, page); +mutex_unlock (&pool->mutex);if (ret)ion_page_pool_free_pages (pool, page); -mutex_unlock (&pool->mutex); } static int ion_page_pool_shrink(struct shrinker *shrinker,
@@ -110,30 +125,38 @@ static int ion_page_pool_shrink(struct shrinker *shrinker, shrinker);int nr_freed =0 ;int i; +bool high; + +if (sc->gfp_mask & __GFP_HIGHMEM) + high = true;if (sc->nr_to_scan ==0 ) -return pool->count * (1 << pool->order); +return high ? (pool->high_count + pool->low_count) * + (1 << pool->order) : + pool->low_count * (1 << pool->order); -mutex_lock (&pool->mutex); -for (i =0 ; i < sc->nr_to_scan && pool->count; i++) { -struct ion_page_pool_item *item; +for (i =0 ; i < sc->nr_to_scan; i++) {struct page *page; - item = list_first_entry(&pool->items, struct ion_page_pool_item, list); - page = item->page; -if (PageHighMem (page) && !(sc->gfp_mask & __GFP_HIGHMEM)) { -list_move_tail (&item->list, &pool->items); -continue ; +mutex_lock (&pool->mutex); +if (high && pool->high_count) { + page =ion_page_pool_remove (pool, true); + }else if (pool->low_count) { + page =ion_page_pool_remove (pool, false); + }else { +mutex_unlock (&pool->mutex); +break ; } -BUG_ON (page !=ion_page_pool_remove (pool)); +mutex_unlock (&pool->mutex);ion_page_pool_free_pages (pool, page); nr_freed += (1 << pool->order); }pr_info ("%s: shrunk page_pool of order %d by %d pages\n" , __func__, pool->order, nr_freed); -mutex_unlock (&pool->mutex); -return pool->count * (1 << pool->order); +return high ? (pool->high_count + pool->low_count) * + (1 << pool->order) : + pool->low_count * (1 << pool->order); }struct ion_page_pool *ion_page_pool_create (gfp_t gfp_mask,unsigned int order)
@@ -142,8 +165,10 @@ struct ion_page_pool *ion_page_pool_create(gfp_t gfp_mask, unsigned int order) GFP_KERNEL);if (!pool)return NULL ; - pool->count =0 ; -INIT_LIST_HEAD (&pool->items); + pool->high_count =0 ; + pool->low_count =0 ; +INIT_LIST_HEAD (&pool->low_items); +INIT_LIST_HEAD (&pool->high_items); pool->shrinker.shrink = ion_page_pool_shrink; pool->shrinker.seeks =DEFAULT_SEEKS *16 ; pool->shrinker.batch =0 ;
drivers/staging/android/ion/ion_priv.h +8 -4
@@ -217,8 +217,10 @@ void ion_carveout_free(struct ion_heap *heap, ion_phys_addr_t addr,/** * struct ion_page_pool - pagepool struct -* @count: number of items in the pool -* @items: list of items +* @high_count: number of highmem items in the pool +* @low_count: number of lowmem items in the pool +* @high_items: list of highmem items +* @low_items: list of lowmem items * @shrinker: a shrinker for the items * @mutex: lock protecting this struct and especially the count * item list
@@ -235,8 +237,10 @@ void ion_carveout_free(struct ion_heap *heap, ion_phys_addr_t addr,* on many systems */ struct ion_page_pool { -int count; -struct list_head items; +int high_count; +int low_count; +struct list_head high_items; +struct list_head low_items;struct shrinker shrinker;struct mutex mutex;void *(*alloc)(struct ion_page_pool *pool);
drivers/staging/android/ion/ion_system_heap.c +6 -3
@@ -328,9 +328,12 @@ static int ion_system_heap_debug_show(struct ion_heap *heap, struct seq_file *s,int i;for (i =0 ; i < num_orders; i++) {struct ion_page_pool *pool = sys_heap->pools[i]; -seq_printf (s,"%d order %u pages in pool = %lu total\n" , - pool->count, pool->order, - (1 << pool->order) *PAGE_SIZE * pool->count); +seq_printf (s,"%d order %u highmem pages in pool = %lu total\n" , + pool->high_count, pool->order, + (1 << pool->order) *PAGE_SIZE * pool->high_count); +seq_printf (s,"%d order %u lowmem pages in pool = %lu total\n" , + pool->low_count, pool->order, + (1 << pool->order) *PAGE_SIZE * pool->low_count); }return 0 ; }
Signature · 1 line
1.8.3.2
PATCH 40/115 gpu: ion: Fix lockdep issue in ion_page_pool
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Currently the mutex is held while kmalloc is called, under a low memory condition this might trigger the shrinker which also takes this mutex. Refactor so the mutex is not held during allocation.
--- drivers/staging/android/ion/ion_page_pool.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_page_pool.c +3 -2
@@ -53,6 +53,8 @@ static int ion_page_pool_add(struct ion_page_pool *pool, struct page *page) item =kmalloc (sizeof (struct ion_page_pool_item ),GFP_KERNEL );if (!item)return -ENOMEM ; + +mutex_lock (&pool->mutex); item->page = page;if (PageHighMem (page)) {list_add_tail (&item->list, &pool->high_items);
@@ -61,6 +63,7 @@ static int ion_page_pool_add(struct ion_page_pool *pool, struct page *page)list_add_tail (&item->list, &pool->low_items); pool->low_count++; } +mutex_unlock (&pool->mutex);return 0 ; }
@@ -110,9 +113,7 @@ void ion_page_pool_free(struct ion_page_pool *pool, struct page* page) {int ret; -mutex_lock (&pool->mutex); ret =ion_page_pool_add (pool, page); -mutex_unlock (&pool->mutex);if (ret)ion_page_pool_free_pages (pool, page); }
Signature · 1 line
1.8.3.2
PATCH 41/115 gpu: ion: Switch to using a single shrink function
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
The single shrink function will free lower order pages first. This enables compaction to work properly.
--- drivers/staging/android/ion/ion_page_pool.c | 150 ++++++++++++++++++++++------ drivers/staging/android/ion/ion_priv.h | 3 +- 2 files changed, 123 insertions(+), 30 deletions(-)
drivers/staging/android/ion/ion_page_pool.c +121 -29
@@ -14,13 +14,21 @@* */ +#include <linux/debugfs.h> #include <linux/dma-mapping.h> #include <linux/err.h> +#include <linux/fs.h> #include <linux/list.h> +#include <linux/module.h> #include <linux/slab.h> #include <linux/shrinker.h> #include "ion_priv.h" +/* #define DEBUG_PAGE_POOL_SHRINKER */ + +static struct plist_head pools =PLIST_HEAD_INIT (pools); +static struct shrinker shrinker; +struct ion_page_pool_item {struct page *page;struct list_head list;
@@ -118,46 +126,110 @@ void ion_page_pool_free(struct ion_page_pool *pool, struct page* page)ion_page_pool_free_pages (pool, page); } +#ifdef DEBUG_PAGE_POOL_SHRINKER +static int debug_drop_pools_set (void *data,u64 val) +{ +struct shrink_control sc; +int objs; + + sc.gfp_mask =-1 ; + sc.nr_to_scan =0 ; + +if (!val) +return 0 ; + + objs = shrinker.shrink (&shrinker, &sc); + sc.nr_to_scan = objs; + + shrinker.shrink (&shrinker, &sc); +return 0 ; +} + +static int debug_drop_pools_get (void *data,u64 *val) +{ +struct shrink_control sc; +int objs; + + sc.gfp_mask =-1 ; + sc.nr_to_scan =0 ; + + objs = shrinker.shrink (&shrinker, &sc); + *val = objs; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (debug_drop_pools_fops, debug_drop_pools_get, + debug_drop_pools_set,"%llu\n" ); + +static int debug_grow_pools_set (void *data,u64 val) +{ +struct ion_page_pool *pool; +struct page *page; + +plist_for_each_entry (pool, &pools, list) { +if (val != pool->list.prio) +continue ; + page =ion_page_pool_alloc_pages (pool); +if (page) +ion_page_pool_add (pool, page); + } + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (debug_grow_pools_fops, debug_drop_pools_get, + debug_grow_pools_set,"%llu\n" ); +#endif + +static int ion_page_pool_total (bool high) +{ +struct ion_page_pool *pool; +int total =0 ; + +plist_for_each_entry (pool, &pools, list) { + total += high ? (pool->high_count + pool->low_count) * + (1 << pool->order) : + pool->low_count * (1 << pool->order); + } +return total; +} +static int ion_page_pool_shrink (struct shrinker *shrinker,struct shrink_control *sc) { -struct ion_page_pool *pool =container_of (shrinker, - struct ion_page_pool, - shrinker); +struct ion_page_pool *pool;int nr_freed =0 ;int i;bool high; +int nr_to_scan = sc->nr_to_scan;if (sc->gfp_mask & __GFP_HIGHMEM) high = true; -if (sc->nr_to_scan ==0 ) -return high ? (pool->high_count + pool->low_count) * - (1 << pool->order) : - pool->low_count * (1 << pool->order); +if (nr_to_scan ==0 ) +return ion_page_pool_total (high); -for (i =0 ; i < sc->nr_to_scan; i++) { -struct page *page; +plist_for_each_entry (pool, &pools, list) { +for (i =0 ; i < nr_to_scan; i++) { +struct page *page; -mutex_lock (&pool->mutex); -if (high && pool->high_count) { - page =ion_page_pool_remove (pool, true); - }else if (pool->low_count) { - page =ion_page_pool_remove (pool, false); - }else { +mutex_lock (&pool->mutex); +if (high && pool->high_count) { + page =ion_page_pool_remove (pool, true); + }else if (pool->low_count) { + page =ion_page_pool_remove (pool, false); + }else { +mutex_unlock (&pool->mutex); +break ; + }mutex_unlock (&pool->mutex); -break ; +ion_page_pool_free_pages (pool, page); + nr_freed += (1 << pool->order); } -mutex_unlock (&pool->mutex); -ion_page_pool_free_pages (pool, page); - nr_freed += (1 << pool->order); + nr_to_scan -= i; } -pr_info ("%s: shrunk page_pool of order %d by %d pages\n" , __func__, - pool->order, nr_freed); -return high ? (pool->high_count + pool->low_count) * - (1 << pool->order) : - pool->low_count * (1 << pool->order); +return ion_page_pool_total (high); }struct ion_page_pool *ion_page_pool_create (gfp_t gfp_mask,unsigned int order)
@@ -170,20 +242,40 @@ struct ion_page_pool *ion_page_pool_create(gfp_t gfp_mask, unsigned int order) pool->low_count =0 ;INIT_LIST_HEAD (&pool->low_items);INIT_LIST_HEAD (&pool->high_items); - pool->shrinker.shrink = ion_page_pool_shrink; - pool->shrinker.seeks =DEFAULT_SEEKS *16 ; - pool->shrinker.batch =0 ; -register_shrinker (&pool->shrinker); pool->gfp_mask = gfp_mask; pool->order = order;mutex_init (&pool->mutex); +plist_node_init (&pool->list, order); +plist_add (&pool->list, &pools);return pool; }void ion_page_pool_destroy (struct ion_page_pool *pool) { -unregister_shrinker (&pool->shrinker); +plist_del (&pool->list, &pools);kfree (pool); } +static int __init ion_page_pool_init (void ) +{ + shrinker.shrink = ion_page_pool_shrink; + shrinker.seeks =DEFAULT_SEEKS ; + shrinker.batch =0 ; +register_shrinker (&shrinker); +#ifdef DEBUG_PAGE_POOL_SHRINKER +debugfs_create_file ("ion_pools_shrink" ,0644 ,NULL ,NULL , + &debug_drop_pools_fops); +debugfs_create_file ("ion_pools_grow" ,0644 ,NULL ,NULL , + &debug_grow_pools_fops); +#endif +return 0 ; +} + +static void __exit ion_page_pool_exit(void) +{ +unregister_shrinker (&shrinker); +} + +module_init (ion_page_pool_init); +module_exit (ion_page_pool_exit);
drivers/staging/android/ion/ion_priv.h +2 -1
@@ -230,6 +230,7 @@ void ion_carveout_free(struct ion_heap *heap, ion_phys_addr_t addr,* when the shrinker fires * @gfp_mask: gfp_mask to use from alloc * @order: order of pages in the pool +* @list: plist node for list of pools * * Allows you to keep a pool of pre allocated pages to use from your heap. * Keeping a pool of pages that is ready for dma, ie any cached mapping have
@@ -241,12 +242,12 @@ struct ion_page_pool {int low_count;struct list_head high_items;struct list_head low_items; -struct shrinker shrinker;struct mutex mutex;void *(*alloc)(struct ion_page_pool *pool);void (*free)(struct ion_page_pool *pool,struct page *page);gfp_t gfp_mask;unsigned int order; +struct plist_node list; };struct ion_page_pool *ion_page_pool_create (gfp_t gfp_mask,unsigned int order);
Signature · 1 line
1.8.3.2
PATCH 42/115 gpu: ion: Refactor locking
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Removes contention for lock between allocate and free by reducing the length of time the lock is held for. Split out a seperate lock to protect the list of heaps and replace it with a rwsem since the list will most likely only be updated during initialization.
--- drivers/staging/android/ion/ion.c | 45 ++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 20 deletions(-)
drivers/staging/android/ion/ion.c +25 -20
@@ -38,15 +38,17 @@/** * struct ion_device - the metadata of the ion device node * @dev: the actual misc device -* @buffers: an rb tree of all the existing buffers -* @lock: lock protecting the buffers & heaps trees +* @buffers: an rb tree of all the existing buffers +* @buffer_lock: lock protecting the tree of buffers +* @lock: rwsem protecting the tree of heaps and clients * @heaps: list of all the heaps in the system * @user_clients: list of all the clients created from userspace */ struct ion_device {struct miscdevice dev;struct rb_root buffers; -struct mutex lock; +struct mutex buffer_lock; +struct rw_semaphore lock;struct rb_root heaps;long (*custom_ioctl) (struct ion_client *client,unsigned int cmd,unsigned long arg);
@@ -205,7 +207,9 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,cached mapping that mapping has been invalidated */ for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i)sg_dma_address (sg) =sg_phys (sg); +mutex_lock (&dev->buffer_lock);ion_buffer_add (dev, buffer); +mutex_unlock (&dev->buffer_lock);return buffer; err:
@@ -224,9 +228,9 @@ static void ion_buffer_destroy(struct kref *kref) buffer->heap->ops->unmap_kernel (buffer->heap, buffer); buffer->heap->ops->unmap_dma (buffer->heap, buffer); buffer->heap->ops->free (buffer); -mutex_lock (&dev->lock); +mutex_lock (&dev->buffer_lock);rb_erase (&buffer->node, &dev->buffers); -mutex_unlock (&dev->lock); +mutex_unlock (&dev->buffer_lock);if (buffer->flags &ION_FLAG_CACHED )kfree (buffer->dirty);kfree (buffer);
@@ -244,9 +248,9 @@ static int ion_buffer_put(struct ion_buffer *buffer)static void ion_buffer_add_to_handle (struct ion_buffer *buffer) { -mutex_lock (&buffer->dev->lock); +mutex_lock (&buffer->lock); buffer->handle_count++; -mutex_unlock (&buffer->dev->lock); +mutex_unlock (&buffer->lock); }static void ion_buffer_remove_from_handle (struct ion_buffer *buffer)
@@ -260,7 +264,7 @@ static void ion_buffer_remove_from_handle(struct ion_buffer *buffer)* The taskcomm and pid can provide a debug hint as to where this fd * is in the system */ -mutex_lock (&buffer->dev->lock); +mutex_lock (&buffer->lock); buffer->handle_count--;BUG_ON (buffer->handle_count <0 );if (!buffer->handle_count) {
@@ -270,7 +274,7 @@ static void ion_buffer_remove_from_handle(struct ion_buffer *buffer)get_task_comm (buffer->task_comm, task); buffer->pid =task_pid_nr (task); } -mutex_unlock (&buffer->dev->lock); +mutex_unlock (&buffer->lock); } static struct ion_handle *ion_handle_create(struct ion_client *client,
@@ -403,7 +407,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len, len =PAGE_ALIGN (len); -mutex_lock (&dev->lock); +down_read (&dev->lock);for (n =rb_first (&dev->heaps); n !=NULL ; n =rb_next (n)) { struct ion_heap *heap = rb_entry(n, struct ion_heap, node);/* if the client doesn't support this heap type */
@@ -416,7 +420,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,if (!IS_ERR_OR_NULL (buffer))break ; } -mutex_unlock (&dev->lock); +up_read (&dev->lock);if (buffer ==NULL )return ERR_PTR (-ENODEV );
@@ -662,7 +666,7 @@ struct ion_client *ion_client_create(struct ion_device *dev, client->task = task; client->pid = pid; -mutex_lock (&dev->lock); +down_write (&dev->lock); p = &dev->clients.rb_node;while (*p) { parent = *p;
@@ -680,7 +684,7 @@ struct ion_client *ion_client_create(struct ion_device *dev, client->debug_root =debugfs_create_file (debug_name,0664 , dev->debug_root, client, &debug_client_fops); -mutex_unlock (&dev->lock); +up_write (&dev->lock);return client; }
@@ -696,12 +700,12 @@ void ion_client_destroy(struct ion_client *client) node);ion_handle_destroy (&handle->ref); } -mutex_lock (&dev->lock); +down_write (&dev->lock);if (client->task)put_task_struct (client->task);rb_erase (&client->node, &dev->clients);debugfs_remove_recursive (client->debug_root); -mutex_unlock (&dev->lock); +up_write (&dev->lock);kfree (client); }
@@ -1220,7 +1224,7 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)seq_printf (s,"----------------------------------------------------\n" );seq_printf (s,"orphaned allocations (info is from last known client):" "\n" ); -mutex_lock (&dev->lock); +mutex_lock (&dev->buffer_lock);for (n =rb_first (&dev->buffers); n; n =rb_next (n)) { struct ion_buffer *buffer = rb_entry(n, struct ion_buffer, node);
@@ -1233,7 +1237,7 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused) total_orphaned_size += buffer->size; } } -mutex_unlock (&dev->lock); +mutex_unlock (&dev->buffer_lock);seq_printf (s,"----------------------------------------------------\n" );seq_printf (s,"%16.s %16u\n" ,"total orphaned" , total_orphaned_size);
@@ -1270,7 +1274,7 @@ void ion_device_add_heap(struct ion_device *dev, struct ion_heap *heap) __func__); heap->dev = dev; -mutex_lock (&dev->lock); +down_write (&dev->lock);while (*p) { parent = *p; entry = rb_entry(parent, struct ion_heap, node);
@@ -1291,7 +1295,7 @@ void ion_device_add_heap(struct ion_device *dev, struct ion_heap *heap)debugfs_create_file (heap->name,0664 , dev->debug_root, heap, &debug_heap_fops); end: -mutex_unlock (&dev->lock); +up_write (&dev->lock); } struct ion_device *ion_device_create(long (*custom_ioctl)
@@ -1322,7 +1326,8 @@ struct ion_device *ion_device_create(long (*custom_ioctl) idev->custom_ioctl = custom_ioctl; idev->buffers =RB_ROOT ; -mutex_init (&idev->lock); +mutex_init (&idev->buffer_lock); +init_rwsem (&idev->lock); idev->heaps =RB_ROOT ; idev->clients =RB_ROOT ;return idev;
Signature · 1 line
1.8.3.2
PATCH 43/115 gpu: ion: Clear GFP_WAIT flag on high order allocations
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This will prevent the kernel from kicking off compaction when higher order allocations are made. Instead we will get these high order allocations only if they are readily available.
--- drivers/staging/android/ion/ion_system_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -28,7 +28,7 @@static unsigned int high_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY | - __GFP_NO_KSWAPD); + __GFP_NO_KSWAPD) & ~__GFP_WAIT;static unsigned int low_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN);static const unsigned int orders[] = {8 ,4 ,0 };
Signature · 1 line
1.8.3.2
PATCH 44/115 gpu: ion: Don't flush allocatoins that come from the page pools
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion_system_heap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +5 -3
@@ -76,9 +76,14 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,if (order >4 ) gfp_flags = high_order_gfp_flags; page =alloc_pages (gfp_flags, order); +if (!page) +return 0 ; +__dma_page_cpu_to_dev (page,0 ,PAGE_SIZE << order, +DMA_BIDIRECTIONAL ); }if (!page)return 0 ; +if (split_pages)split_page (page, order);return page;
@@ -201,9 +206,6 @@ static int ion_system_heap_allocate(struct ion_heap *heap,kfree (info); } -dma_sync_sg_for_device (NULL , table->sgl, table->nents, -DMA_BIDIRECTIONAL ); - buffer->priv_virt = table;return 0 ; err1:
Signature · 1 line
1.8.3.2
PATCH 45/115 gpu: ion: Fix bug in ion_system_heap map_user
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
When the requested mmap length was not an integer number of chunks or the buffer, or if an offset was provided, a bug would cause extra or incorrect pages of the buffer to be mapped.
--- drivers/staging/android/ion/ion_system_heap.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +15 -6
@@ -292,18 +292,27 @@ int ion_system_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer, {struct sg_table *table = buffer->priv_virt;unsigned long addr = vma->vm_start; -unsigned long offset = vma->vm_pgoff; +unsigned long offset = vma->vm_pgoff *PAGE_SIZE ;struct scatterlist *sg;int i;for_each_sg (table->sgl, sg, table->nents, i) { -if (offset) { - offset--; +struct page *page =sg_page (sg); +unsigned long remainder = vma->vm_end - addr; +unsigned long len =sg_dma_len (sg); + +if (offset >=sg_dma_len (sg)) { + offset -=sg_dma_len (sg);continue ; + }else if (offset) { + page += offset /PAGE_SIZE ; + len =sg_dma_len (sg) - offset; + offset =0 ; } -remap_pfn_range (vma, addr,page_to_pfn (sg_page (sg)), -sg_dma_len (sg), vma->vm_page_prot); - addr +=sg_dma_len (sg); + len =min (len, remainder); +remap_pfn_range (vma, addr,page_to_pfn (page), len, + vma->vm_page_prot); + addr += len;if (addr >= vma->vm_end)return 0 ; }
Signature · 1 line
1.8.3.2
PATCH 46/115 gpu: ion: Fix bug in zeroing pages in system heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Pages are zeroed for security purposes when returned to the ion heap. There was a bug in this code preventing this from happening.
--- drivers/staging/android/ion/ion_system_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -103,7 +103,7 @@ static void free_buffer_page(struct ion_system_heap *heap,security. This uses vmap as we want to set the pgprot so the writes to occur to noncached mappings, as the pool's purpose is to keep the pages out of the cache */ -for (i =0 ; i < order /PAGE_SIZE ; i++) { +for (i =0 ; i < (1 << order); i++) {struct page *sub_page = page + i;void *addr =vmap (&sub_page,1 ,VM_MAP ,pgprot_writecombine (PAGE_KERNEL ));
Signature · 1 line
1.8.3.2
PATCH 47/115 gpu: ion: fix carveout ops
In reply to John StultzFrom: Benjamin Gaignard <[email protected]>
when using carveout heap ion_buffer_create function failed because map_dma and unmap_dma operations aren't set by carveout heap.
--- drivers/staging/android/ion/ion_carveout_heap.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/staging/android/ion/ion_carveout_heap.c +2 -0
@@ -142,6 +142,8 @@ static struct ion_heap_ops carveout_heap_ops = { .allocate = ion_carveout_heap_allocate, .free = ion_carveout_heap_free, .phys = ion_carveout_heap_phys, + .map_dma = ion_carveout_heap_map_dma, + .unmap_dma = ion_carveout_heap_unmap_dma, .map_user = ion_carveout_heap_map_user, .map_kernel = ion_carveout_heap_map_kernel, .unmap_kernel = ion_carveout_heap_unmap_kernel,
Signature · 1 line
1.8.3.2
PATCH 48/115 gpu: ion: fix compilation warning
In reply to John StultzFrom: Benjamin Gaignard <[email protected]>
use atomic_read to get the refcount value to avoid compilation warning
--- drivers/staging/android/ion/ion.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +2 -1
@@ -1233,7 +1233,8 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused) total_size += buffer->size;if (!buffer->handle_count) {seq_printf (s,"%16.s %16u %16u %d %d\n" , buffer->task_comm, - buffer->pid, buffer->size, buffer->kmap_cnt, buffer->ref); + buffer->pid, buffer->size, buffer->kmap_cnt, +atomic_read (&buffer->ref.refcount)); total_orphaned_size += buffer->size; } }
Signature · 1 line
1.8.3.2
PATCH 49/115 gpu: ion: Modify reserve function for carveouts with no start address
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch allows you to specify a heap that requires carveout memory but that doesn't specify a start address. Memblock_alloc will be called to find a location for these heaps.
--- drivers/staging/android/ion/ion.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-)
drivers/staging/android/ion/ion.c +26 -7
@@ -1343,16 +1343,35 @@ void ion_device_destroy(struct ion_device *dev)void __init ion_reserve (struct ion_platform_data *data) { -int i, ret; +int i;for (i =0 ; i < data->nr; i++) {if (data->heaps[i].size ==0 )continue ; - ret =memblock_reserve (data->heaps[i].base, - data->heaps[i].size); -if (ret) -pr_err ("memblock reserve of %x@%lx failed\n" , - data->heaps[i].size, - data->heaps[i].base); + +if (data->heaps[i].base ==0 ) { +phys_addr_t paddr; + paddr =memblock_alloc_base (data->heaps[i].size, + data->heaps[i].align, +MEMBLOCK_ALLOC_ANYWHERE ); +if (!paddr) { +pr_err ("%s: error allocating memblock for " +"heap %d\n" , + __func__, i); +continue ; + } + data->heaps[i].base = paddr; + }else { +int ret =memblock_reserve (data->heaps[i].base, + data->heaps[i].size); +if (ret) +pr_err ("memblock reserve of %x@%lx failed\n" , + data->heaps[i].size, + data->heaps[i].base); + } +pr_info ("%s: %s reserved base %lx size %d\n" , __func__, + data->heaps[i].name, + data->heaps[i].base, + data->heaps[i].size); } }
Signature · 1 line
1.8.3.2
PATCH 50/115 gpu: ion: Fix bug where MAP ioctl was no longer supported
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion.c | 1 + 1 file changed, 1 insertion(+)
drivers/staging/android/ion/ion.c +1 -0
@@ -1086,6 +1086,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)break ; }case ION_IOC_SHARE : +case ION_IOC_MAP : {struct ion_fd_data data;
Signature · 1 line
1.8.3.2
PATCH 51/115 gpu: ion: Switch heap rbtree to a prio list
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Switches the rbtree tree of heaps for a plist. This significantly simplifies the code and the list is small and is modified only at first boot so the rbtree is unnecessary. This also switches the traversal of the heap list to traverse from highest to lowest id's. This allows allocations to pass a heap mask that falls back on the system heap -- typically id 0, which is the common case.
--- drivers/staging/android/ion/ion.c | 35 ++++++++-------------------------- drivers/staging/android/ion/ion_priv.h | 4 ++-- 2 files changed, 10 insertions(+), 29 deletions(-)
drivers/staging/android/ion/ion.c +8 -27
@@ -49,7 +49,7 @@ struct ion_device {struct rb_root buffers;struct mutex buffer_lock;struct rw_semaphore lock; -struct rb_root heaps; +struct plist_head heaps;long (*custom_ioctl) (struct ion_client *client,unsigned int cmd,unsigned long arg);struct rb_root clients;
@@ -389,10 +389,10 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len, size_t align, unsigned int heap_mask, unsigned int flags) { - struct rb_node *n; struct ion_handle *handle;struct ion_device *dev = client->dev;struct ion_buffer *buffer =NULL ; +struct ion_heap *heap;pr_debug ("%s: len %d align %d heap_mask %u flags %x\n" , __func__, len, align, heap_mask, flags);
@@ -408,8 +408,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len, len =PAGE_ALIGN (len);down_read (&dev->lock); -for (n =rb_first (&dev->heaps); n !=NULL ; n =rb_next (n)) { - struct ion_heap *heap = rb_entry(n, struct ion_heap, node); +plist_for_each_entry (heap, &dev->heaps, node) {/* if the client doesn't support this heap type */ if (!((1 << heap->type) & client->heap_mask))continue ;
@@ -1266,10 +1265,6 @@ static const struct file_operations debug_heap_fops = {void ion_device_add_heap (struct ion_device *dev,struct ion_heap *heap) { -struct rb_node **p = &dev->heaps.rb_node; -struct rb_node *parent =NULL ; -struct ion_heap *entry; -if (!heap->ops->allocate || !heap->ops->free || !heap->ops->map_dma || !heap->ops->unmap_dma) pr_err("%s: can not add heap with invalid ops struct.\n",
@@ -1277,26 +1272,12 @@ void ion_device_add_heap(struct ion_device *dev, struct ion_heap *heap) heap->dev = dev;down_write (&dev->lock); -while (*p) { - parent = *p; - entry = rb_entry(parent, struct ion_heap, node); - -if (heap->id < entry->id) { - p = &(*p)->rb_left; - }else if (heap->id > entry->id ) { - p = &(*p)->rb_right; - }else { -pr_err ("%s: can not insert multiple heaps with " -"id %d\n" , __func__, heap->id); - goto end; - } - } - -rb_link_node (&heap->node, parent, p); -rb_insert_color (&heap->node, &dev->heaps); +/* use negative heap->id to reverse the priority -- when traversing +the list later attempt higher id numbers first */ +plist_node_init (&heap->node, -heap->id); +plist_add (&heap->node, &dev->heaps);debugfs_create_file (heap->name,0664 , dev->debug_root, heap, &debug_heap_fops); -end:up_write (&dev->lock); }
@@ -1330,7 +1311,7 @@ struct ion_device *ion_device_create(long (*custom_ioctl) idev->buffers =RB_ROOT ;mutex_init (&idev->buffer_lock);init_rwsem (&idev->lock); - idev->heaps =RB_ROOT ; +plist_head_init (&idev->heaps); idev->clients =RB_ROOT ;return idev; }
drivers/staging/android/ion/ion_priv.h +2 -2
@@ -127,11 +127,11 @@ struct ion_heap_ops {* that are allocated from a specially reserved heap. */ struct ion_heap { -struct rb_node node; +struct plist_node node;struct ion_device *dev;enum ion_heap_type type;struct ion_heap_ops *ops; -int id; +unsigned int id;const char *name;int (*debug_show)(struct ion_heap *heap,struct seq_file *,void *); };
Signature · 1 line
1.8.3.2
PATCH 52/115 gpu: ion: Refactor common mapping functions out of system heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
The system heap contained several general purpose functions to map buffers to the kernel and userspace. This patch refactors those into ion_heap.c so they can be used by other heaps.
--- drivers/staging/android/ion/ion_heap.c | 75 ++++++++++++++++++++++++ drivers/staging/android/ion/ion_priv.h | 10 ++++ drivers/staging/android/ion/ion_system_heap.c | 84 ++------------------------- 3 files changed, 91 insertions(+), 78 deletions(-)
drivers/staging/android/ion/ion_heap.c +75 -0
@@ -15,9 +15,84 @@*/ #include <linux/err.h> +#include <linux/mm.h> +#include <linux/scatterlist.h> +#include <linux/vmalloc.h> #include "ion.h" #include "ion_priv.h" +void *ion_heap_map_kernel (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +struct scatterlist *sg; +int i, j; +void *vaddr; +pgprot_t pgprot; +struct sg_table *table = buffer->sg_table; +int npages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; +struct page **pages =vmalloc (sizeof (struct page *) * npages); +struct page **tmp = pages; + +if (!pages) +return 0 ; + +if (buffer->flags &ION_FLAG_CACHED ) + pgprot =PAGE_KERNEL ; +else + pgprot =pgprot_writecombine (PAGE_KERNEL ); + +for_each_sg (table->sgl, sg, table->nents, i) { +int npages_this_entry =PAGE_ALIGN (sg_dma_len (sg)) /PAGE_SIZE ; +struct page *page =sg_page (sg); +BUG_ON (i >= npages); +for (j =0 ; j < npages_this_entry; j++) { + *(tmp++) = page++; + } + } + vaddr =vmap (pages, npages,VM_MAP , pgprot); +vfree (pages); + +return vaddr; +} + +void ion_heap_unmap_kernel (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +vunmap (buffer->vaddr); +} + +int ion_heap_map_user (struct ion_heap *heap,struct ion_buffer *buffer, +struct vm_area_struct *vma) +{ +struct sg_table *table = buffer->sg_table; +unsigned long addr = vma->vm_start; +unsigned long offset = vma->vm_pgoff *PAGE_SIZE ; +struct scatterlist *sg; +int i; + +for_each_sg (table->sgl, sg, table->nents, i) { +struct page *page =sg_page (sg); +unsigned long remainder = vma->vm_end - addr; +unsigned long len =sg_dma_len (sg); + +if (offset >=sg_dma_len (sg)) { + offset -=sg_dma_len (sg); +continue ; + }else if (offset) { + page += offset /PAGE_SIZE ; + len =sg_dma_len (sg) - offset; + offset =0 ; + } + len =min (len, remainder); +remap_pfn_range (vma, addr,page_to_pfn (page), len, + vma->vm_page_prot); + addr += len; +if (addr >= vma->vm_end) +return 0 ; + } +return 0 ; +} +struct ion_heap *ion_heap_create (struct ion_platform_heap *heap_data) {struct ion_heap *heap =NULL ;
drivers/staging/android/ion/ion_priv.h +10 -0
@@ -178,6 +178,16 @@ void ion_device_destroy(struct ion_device *dev);void ion_device_add_heap (struct ion_device *dev,struct ion_heap *heap);/** +* some helpers for common operations on buffers using the sg_table +* and vaddr fields +*/ +void *ion_heap_map_kernel (struct ion_heap *,struct ion_buffer *); +void ion_heap_unmap_kernel (struct ion_heap *,struct ion_buffer *); +int ion_heap_map_user (struct ion_heap *,struct ion_buffer *, +struct vm_area_struct *); + + +/** * functions for creating and destroying the built in ion heaps. * architectures can add their own custom architecture specific * heaps as appropriate.
drivers/staging/android/ion/ion_system_heap.c +6 -78
107 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 53/115 gpu: ion: Add chunk heap
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
This patch adds support for a chunk heap that allows for buffers that are made up of a list of fixed size chunks taken from a carveout. Chunk sizes are configured when the heaps are created by passing the chunk size in the priv field of the heap platform data.
--- drivers/staging/android/ion/Makefile | 2 +- drivers/staging/android/ion/ion.h | 5 + drivers/staging/android/ion/ion_chunk_heap.c | 174 +++++++++++++++++++++++++++ drivers/staging/android/ion/ion_heap.c | 6 + drivers/staging/android/ion/ion_priv.h | 4 +- 5 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 drivers/staging/android/ion/ion_chunk_heap.c
drivers/staging/android/ion/Makefile +1 -1
@@ -1,3 +1,3 @@ obj-$(CONFIG_ION) += ion.o ion_heap.o ion_page_pool.o ion_system_heap.o \ - ion_carveout_heap.o + ion_carveout_heap.o ion_chunk_heap.o obj-$(CONFIG_ION_TEGRA) += tegra/
drivers/staging/android/ion/ion.h +5 -0
@@ -33,6 +33,7 @@ enum ion_heap_type {ION_HEAP_TYPE_SYSTEM ,ION_HEAP_TYPE_SYSTEM_CONTIG ,ION_HEAP_TYPE_CARVEOUT , +ION_HEAP_TYPE_CHUNK ,ION_HEAP_TYPE_CUSTOM ,/* must be last so device specific heaps always are at the end of this enum */ ION_NUM_HEAPS,
@@ -75,6 +76,8 @@ struct ion_buffer;* @name: used for debug purposes * @base: base address of heap in physical memory if applicable * @size: size of the heap in bytes if applicable +* @align: required alignment in physical memory if applicable +* @priv: private info passed from the board file * * Provided by the board file. */
@@ -84,6 +87,8 @@ struct ion_platform_heap {const char *name;ion_phys_addr_t base;size_t size; +ion_phys_addr_t align; +void *priv; };/**
drivers/staging/android/ion/ion_chunk_heap.c new file +174 -0
174 lines not on this page · raw message
drivers/staging/android/ion/ion_heap.c +6 -0
@@ -107,6 +107,9 @@ struct ion_heap *ion_heap_create(struct ion_platform_heap *heap_data)case ION_HEAP_TYPE_CARVEOUT : heap =ion_carveout_heap_create (heap_data);break ; +case ION_HEAP_TYPE_CHUNK : + heap =ion_chunk_heap_create (heap_data); +break ;default :pr_err ("%s: Invalid heap type %d\n" , __func__, heap_data->type);
@@ -140,6 +143,9 @@ void ion_heap_destroy(struct ion_heap *heap)case ION_HEAP_TYPE_CARVEOUT :ion_carveout_heap_destroy (heap);break ; +case ION_HEAP_TYPE_CHUNK : +ion_chunk_heap_destroy (heap); +break ;default :pr_err ("%s: Invalid heap type %d\n" , __func__, heap->type);
drivers/staging/android/ion/ion_priv.h +3 -1
@@ -195,7 +195,6 @@ int ion_heap_map_user(struct ion_heap *, struct ion_buffer *,struct ion_heap *ion_heap_create (struct ion_platform_heap *);void ion_heap_destroy (struct ion_heap *); -struct ion_heap *ion_system_heap_create (struct ion_platform_heap *);void ion_system_heap_destroy (struct ion_heap *);
@@ -204,6 +203,9 @@ void ion_system_contig_heap_destroy(struct ion_heap *);struct ion_heap *ion_carveout_heap_create (struct ion_platform_heap *);void ion_carveout_heap_destroy (struct ion_heap *); + +struct ion_heap *ion_chunk_heap_create (struct ion_platform_heap *); +void ion_chunk_heap_destroy (struct ion_heap *);/** * kernel api to allocate/free from carveout -- used when carveout is * used to back an architecture specific custom heap
Signature · 1 line
1.8.3.2
PATCH 54/115 gpu: ion: Clarify variable names and comments around heap ids v types
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
There is some confusion between when to use the heap type and when the id. This patch clarifies this by using clearer variable names and describing the intention in the comments. Also fixes the client debug code to print heaps by id instead of type.
--- drivers/staging/android/ion/ion.c | 37 ++++++++++++++-------------- drivers/staging/android/ion/ion.h | 51 +++++++++++++++++++++------------------ 2 files changed, 47 insertions(+), 41 deletions(-)
drivers/staging/android/ion/ion.c +19 -18
@@ -1,4 +1,5 @@/* +* drivers/staging/android/ion/ion.c * * Copyright (C) 2011 Google, Inc.
@@ -62,7 +63,7 @@ struct ion_device {* @dev: backpointer to ion device * @handles: an rb tree of all the handles in this client * @lock: lock protecting the tree of handles -* @heap_mask: mask of all supported heaps +* @heap_type_mask: mask of all supported heap types * @name: used for debugging * @task: used for debugging *
@@ -75,7 +76,7 @@ struct ion_client {struct ion_device *dev;struct rb_root handles;struct mutex lock; -unsigned int heap_mask; +unsigned int heap_type_mask;const char *name;struct task_struct *task;pid_t pid;
@@ -386,7 +387,7 @@ static void ion_handle_add(struct ion_client *client, struct ion_handle *handle) }struct ion_handle *ion_alloc (struct ion_client *client,size_t len, -size_t align,unsigned int heap_mask, +size_t align,unsigned int heap_id_mask,unsigned int flags) {struct ion_handle *handle;
@@ -394,8 +395,8 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,struct ion_buffer *buffer =NULL ;struct ion_heap *heap; -pr_debug ("%s: len %d align %d heap_mask %u flags %x\n" , __func__, len, - align, heap_mask, flags); +pr_debug ("%s: len %d align %d heap_id_mask %u flags %x\n" , __func__, + len, align, heap_id_mask, flags);/* * traverse the list of heaps available in this system in priority * order. If the heap type is supported by the client, and matches the
@@ -410,10 +411,10 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,down_read (&dev->lock);plist_for_each_entry (heap, &dev->heaps, node) {/* if the client doesn't support this heap type */ -if (!((1 << heap->type) & client->heap_mask)) +if (!((1 << heap->type) & client->heap_type_mask))continue ; -/* if the caller didn't specify this heap type */ -if (!((1 << heap->id) & heap_mask)) +/* if the caller didn't specify this heap id */ +if (!((1 << heap->id) & heap_id_mask))continue ; buffer =ion_buffer_create (heap, dev, len, align, flags);if (!IS_ERR_OR_NULL (buffer))
@@ -588,24 +589,24 @@ static int ion_debug_client_show(struct seq_file *s, void *unused) {struct ion_client *client = s->private;struct rb_node *n; -size_t sizes[ION_NUM_HEAPS ] = {0 }; -const char *names[ION_NUM_HEAPS ] = {0 }; +size_t sizes[ION_NUM_HEAP_IDS ] = {0 }; +const char *names[ION_NUM_HEAP_IDS ] = {0 };int i;mutex_lock (&client->lock);for (n =rb_first (&client->handles); n; n =rb_next (n)) { struct ion_handle *handle = rb_entry(n, struct ion_handle, node); -enum ion_heap_type type = handle->buffer->heap->type; +unsigned int id = handle->buffer->heap->id; -if (!names[type]) - names[type] = handle->buffer->heap->name; - sizes[type] += handle->buffer->size; +if (!names[id]) + names[id] = handle->buffer->heap->name; + sizes[id] += handle->buffer->size; }mutex_unlock (&client->lock);seq_printf (s,"%16.16s: %16.16s\n" ,"heap_name" ,"size_in_bytes" ); -for (i =0 ; i <ION_NUM_HEAPS ; i++) { +for (i =0 ; i <ION_NUM_HEAP_IDS ; i++) {if (!names[i])continue ;seq_printf (s,"%16.16s: %16u\n" , names[i], sizes[i]);
@@ -626,7 +627,7 @@ static const struct file_operations debug_client_fops = { };struct ion_client *ion_client_create (struct ion_device *dev, -unsigned int heap_mask, +unsigned int heap_type_mask,const char *name) {struct ion_client *client;
@@ -661,7 +662,7 @@ struct ion_client *ion_client_create(struct ion_device *dev, client->handles =RB_ROOT ;mutex_init (&client->lock); client->name = name; - client->heap_mask = heap_mask; + client->heap_type_mask = heap_type_mask; client->task = task; client->pid = pid;
@@ -1057,7 +1058,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; data.handle =ion_alloc (client, data.len, data.align, - data.heap_mask, data.flags); + data.heap_id_mask, data.flags);if (IS_ERR (data.handle))return PTR_ERR (data.handle);
drivers/staging/android/ion/ion.h +28 -23
94 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 55/115 gpu: ion: Export ion_client_create
In reply to John StultzFrom: Johan Mossberg <[email protected]>
Will enable modules to allocate memory with ion.
--- drivers/staging/android/ion/ion.c | 1 + 1 file changed, 1 insertion(+)
drivers/staging/android/ion/ion.c +1 -0
@@ -688,6 +688,7 @@ struct ion_client *ion_client_create(struct ion_device *dev,return client; } +EXPORT_SYMBOL (ion_client_create);void ion_client_destroy (struct ion_client *client) {
Signature · 1 line
1.8.3.2
PATCH 56/115 gpu: ion: Remove heapmask from client
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
The heapmask in the client generally wasn't being used. This patch removes it.
--- drivers/staging/android/ion/ion.c | 17 +++++------------ drivers/staging/android/ion/ion.h | 1 - 2 files changed, 5 insertions(+), 13 deletions(-)
drivers/staging/android/ion/ion.c +5 -12
@@ -63,7 +63,6 @@ struct ion_device {* @dev: backpointer to ion device * @handles: an rb tree of all the handles in this client * @lock: lock protecting the tree of handles -* @heap_type_mask: mask of all supported heap types * @name: used for debugging * @task: used for debugging *
@@ -76,7 +75,6 @@ struct ion_client {struct ion_device *dev;struct rb_root handles;struct mutex lock; -unsigned int heap_type_mask;const char *name;struct task_struct *task;pid_t pid;
@@ -410,9 +408,6 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,down_read (&dev->lock);plist_for_each_entry (heap, &dev->heaps, node) { -/* if the client doesn't support this heap type */ -if (!((1 << heap->type) & client->heap_type_mask)) -continue ;/* if the caller didn't specify this heap id */ if (!((1 << heap->id) & heap_id_mask))continue ;
@@ -627,7 +622,6 @@ static const struct file_operations debug_client_fops = { };struct ion_client *ion_client_create (struct ion_device *dev, -unsigned int heap_type_mask,const char *name) {struct ion_client *client;
@@ -662,7 +656,6 @@ struct ion_client *ion_client_create(struct ion_device *dev, client->handles =RB_ROOT ;mutex_init (&client->lock); client->name = name; - client->heap_type_mask = heap_type_mask; client->task = task; client->pid = pid;
@@ -1162,7 +1155,7 @@ static int ion_open(struct inode *inode, struct file *file)struct ion_client *client;pr_debug ("%s: %d\n" , __func__, __LINE__); - client =ion_client_create (dev,-1 ,"user" ); + client =ion_client_create (dev,"user" );if (IS_ERR_OR_NULL (client))return PTR_ERR (client); file->private_data = client;
@@ -1178,7 +1171,7 @@ static const struct file_operations ion_fops = { };static size_t ion_debug_heap_total (struct ion_client *client, -enum ion_heap_type type) +unsigned int id) {size_t size =0 ;struct rb_node *n;
@@ -1188,7 +1181,7 @@ static size_t ion_debug_heap_total(struct ion_client *client,struct ion_handle *handle =rb_entry (n, struct ion_handle, node); -if (handle->buffer->heap->type == type) +if (handle->buffer->heap->id == id) size += handle->buffer->size; }mutex_unlock (&client->lock);
@@ -1209,7 +1202,7 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)for (n =rb_first (&dev->clients); n; n =rb_next (n)) { struct ion_client *client = rb_entry(n, struct ion_client, node); -size_t size =ion_debug_heap_total (client, heap->type); +size_t size =ion_debug_heap_total (client, heap->id);if (!size)continue ;if (client->task) {
@@ -1230,7 +1223,7 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)for (n =rb_first (&dev->buffers); n; n =rb_next (n)) { struct ion_buffer *buffer = rb_entry(n, struct ion_buffer, node); -if (buffer->heap->type != heap->type) +if (buffer->heap->id != heap->id)continue ; total_size += buffer->size;if (!buffer->handle_count) {
drivers/staging/android/ion/ion.h +0 -1
@@ -124,7 +124,6 @@ void ion_reserve(struct ion_platform_data *data);* @name: used for debugging */ struct ion_client *ion_client_create (struct ion_device *dev, -unsigned int heap_type_mask,const char *name);/**
Signature · 1 line
1.8.3.2
PATCH 57/115 gpu: ion: Modify zeroing code so it only allocates address space once
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
vmap/vunmap spend a significant amount of time allocating the address space to map into. Rather than allocating address space for each page, instead allocate once for the entire allocation and then just map and unmap each page into that address space.
--- drivers/staging/android/ion/ion_system_heap.c | 28 ++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +21 -7
@@ -91,7 +91,7 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,static void free_buffer_page (struct ion_system_heap *heap,struct ion_buffer *buffer,struct page *page, -unsigned int order) +unsigned int order,struct vm_struct *vm_struct) {bool cached =ion_buffer_cached (buffer);bool split_pages =ion_buffer_fault_user_mappings (buffer);
@@ -105,10 +105,13 @@ static void free_buffer_page(struct ion_system_heap *heap,purpose is to keep the pages out of the cache */ for (i =0 ; i < (1 << order); i++) {struct page *sub_page = page + i; -void *addr =vmap (&sub_page,1 ,VM_MAP , -pgprot_writecombine (PAGE_KERNEL )); -memset (addr,0 ,PAGE_SIZE ); -vunmap (addr); +struct page **pages = &sub_page; +map_vm_area (vm_struct, +pgprot_writecombine (PAGE_KERNEL ), + &pages); +memset (vm_struct->addr,0 ,PAGE_SIZE ); +unmap_kernel_range ((unsigned long )vm_struct->addr, +PAGE_SIZE ); }ion_page_pool_free (pool, page); }else if (split_pages) {
@@ -164,6 +167,8 @@ static int ion_system_heap_allocate(struct ion_heap *heap,long size_remaining =PAGE_ALIGN (size);unsigned int max_order = orders[0 ];bool split_pages =ion_buffer_fault_user_mappings (buffer); +struct vm_struct *vm_struct; +pte_t *ptes;INIT_LIST_HEAD (&pages);while (size_remaining >0 ) {
@@ -211,10 +216,13 @@ static int ion_system_heap_allocate(struct ion_heap *heap, err1:kfree (table); err: + vm_struct =get_vm_area (PAGE_SIZE , &ptes);list_for_each_entry (info, &pages, list) { -free_buffer_page (sys_heap, buffer, info->page, info->order); +free_buffer_page (sys_heap, buffer, info->page, info->order, + vm_struct);kfree (info); } +free_vm_area (vm_struct);return -ENOMEM ; }
@@ -227,10 +235,16 @@ void ion_system_heap_free(struct ion_buffer *buffer)struct sg_table *table = buffer->sg_table;struct scatterlist *sg;LIST_HEAD (pages); +struct vm_struct *vm_struct; +pte_t *ptes;int i; + vm_struct =get_vm_area (PAGE_SIZE , &ptes); +for_each_sg (table->sgl, sg, table->nents, i) -free_buffer_page (sys_heap, buffer,sg_page (sg),get_order (sg_dma_len (sg))); +free_buffer_page (sys_heap, buffer,sg_page (sg), +get_order (sg_dma_len (sg)), vm_struct); +free_vm_area (vm_struct);sg_free_table (table);kfree (table); }
Signature · 1 line
1.8.3.2
PATCH 58/115 gpu: ion: Refactor the code to zero buffers
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Refactor the code in the system heap used to map and zero the buffers into a seperate utility so it can be called from other heaps. Use it from the chunk heap.
--- drivers/staging/android/ion/ion_chunk_heap.c | 2 ++ drivers/staging/android/ion/ion_heap.c | 37 +++++++++++++++++++++++++++ drivers/staging/android/ion/ion_priv.h | 1 + drivers/staging/android/ion/ion_system_heap.c | 34 ++++++------------------ 4 files changed, 48 insertions(+), 26 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +2 -0
@@ -101,6 +101,8 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)struct scatterlist *sg;int i; +ion_heap_buffer_zero (buffer); +for_each_sg (table->sgl, sg, table->nents, i) {__dma_page_cpu_to_dev (sg_page (sg),0 ,sg_dma_len (sg),DMA_BIDIRECTIONAL );
drivers/staging/android/ion/ion_heap.c +37 -0
@@ -93,6 +93,43 @@ int ion_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer,return 0 ; } +int ion_heap_buffer_zero (struct ion_buffer *buffer) +{ +struct sg_table *table = buffer->sg_table; +pgprot_t pgprot; +struct scatterlist *sg; +struct vm_struct *vm_struct; +int i, j, ret =0 ; + +if (buffer->flags &ION_FLAG_CACHED ) + pgprot =PAGE_KERNEL ; +else + pgprot =pgprot_writecombine (PAGE_KERNEL ); + + vm_struct =get_vm_area (PAGE_SIZE ,VM_ALLOC ); +if (!vm_struct) +return -ENOMEM ; + +for_each_sg (table->sgl, sg, table->nents, i) { +struct page *page =sg_page (sg); +unsigned long len =sg_dma_len (sg); + +for (j =0 ; j < len /PAGE_SIZE ; j++) { +struct page *sub_page = page + j; +struct page **pages = &sub_page; + ret =map_vm_area (vm_struct, pgprot, &pages); +if (ret) + goto end; +memset (vm_struct->addr,0 ,PAGE_SIZE ); +unmap_kernel_range ((unsigned long )vm_struct->addr, +PAGE_SIZE ); + } + } +end: +free_vm_area (vm_struct); +return ret; +} +struct ion_heap *ion_heap_create (struct ion_platform_heap *heap_data) {struct ion_heap *heap =NULL ;
drivers/staging/android/ion/ion_priv.h +1 -0
@@ -185,6 +185,7 @@ void *ion_heap_map_kernel(struct ion_heap *, struct ion_buffer *);void ion_heap_unmap_kernel (struct ion_heap *,struct ion_buffer *);int ion_heap_map_user (struct ion_heap *,struct ion_buffer *,struct vm_area_struct *); +int ion_heap_buffer_zero (struct ion_buffer *buffer);/**
drivers/staging/android/ion/ion_system_heap.c +8 -26
@@ -91,7 +91,7 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,static void free_buffer_page (struct ion_system_heap *heap,struct ion_buffer *buffer,struct page *page, -unsigned int order,struct vm_struct *vm_struct) +unsigned int order) {bool cached =ion_buffer_cached (buffer);bool split_pages =ion_buffer_fault_user_mappings (buffer);
@@ -99,20 +99,6 @@ static void free_buffer_page(struct ion_system_heap *heap,if (!cached) {struct ion_page_pool *pool = heap->pools[order_to_index (order)]; -/* zero the pages before returning them to the pool for -security. This uses vmap as we want to set the pgprot so -the writes to occur to noncached mappings, as the pool's -purpose is to keep the pages out of the cache */ -for (i =0 ; i < (1 << order); i++) { -struct page *sub_page = page + i; -struct page **pages = &sub_page; -map_vm_area (vm_struct, -pgprot_writecombine (PAGE_KERNEL ), - &pages); -memset (vm_struct->addr,0 ,PAGE_SIZE ); -unmap_kernel_range ((unsigned long )vm_struct->addr, -PAGE_SIZE ); - }ion_page_pool_free (pool, page); }else if (split_pages) {for (i =0 ; i < (1 << order); i++)
@@ -167,8 +153,6 @@ static int ion_system_heap_allocate(struct ion_heap *heap,long size_remaining =PAGE_ALIGN (size);unsigned int max_order = orders[0 ];bool split_pages =ion_buffer_fault_user_mappings (buffer); -struct vm_struct *vm_struct; -pte_t *ptes;INIT_LIST_HEAD (&pages);while (size_remaining >0 ) {
@@ -216,13 +200,10 @@ static int ion_system_heap_allocate(struct ion_heap *heap, err1:kfree (table); err: - vm_struct =get_vm_area (PAGE_SIZE , &ptes);list_for_each_entry (info, &pages, list) { -free_buffer_page (sys_heap, buffer, info->page, info->order, - vm_struct); +free_buffer_page (sys_heap, buffer, info->page, info->order);kfree (info); } -free_vm_area (vm_struct);return -ENOMEM ; }
@@ -233,18 +214,19 @@ void ion_system_heap_free(struct ion_buffer *buffer) struct ion_system_heap, heap);struct sg_table *table = buffer->sg_table; +bool cached =ion_buffer_cached (buffer);struct scatterlist *sg;LIST_HEAD (pages); -struct vm_struct *vm_struct; -pte_t *ptes;int i; - vm_struct =get_vm_area (PAGE_SIZE , &ptes); +/* uncached pages come from the page pools, zero them before returning +for security purposes (other allocations are zerod at alloc time */ +if (!cached) +ion_heap_buffer_zero (buffer);for_each_sg (table->sgl, sg, table->nents, i)free_buffer_page (sys_heap, buffer,sg_page (sg), -get_order (sg_dma_len (sg)), vm_struct); -free_vm_area (vm_struct); +get_order (sg_dma_len (sg)));sg_free_table (table);kfree (table); }
Signature · 1 line
1.8.3.2
PATCH 59/115 gpu: ion: Only flush buffers in the chunk heap if they were used cached
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
--- drivers/staging/android/ion/ion_chunk_heap.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +3 -2
@@ -104,8 +104,9 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)ion_heap_buffer_zero (buffer);for_each_sg (table->sgl, sg, table->nents, i) { -__dma_page_cpu_to_dev (sg_page (sg),0 ,sg_dma_len (sg), -DMA_BIDIRECTIONAL ); +if (ion_buffer_cached (buffer)) +__dma_page_cpu_to_dev (sg_page (sg),0 ,sg_dma_len (sg), +DMA_BIDIRECTIONAL );gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)),sg_dma_len (sg)); }
Signature · 1 line
1.8.3.2
PATCH 60/115 gpu: ion: Add support for sharing buffers with dma buf kernel handles
In reply to John StultzFrom: Johan Mossberg <[email protected]>
Currently ion can only share buffers with dma buf fd's. Fd's can not be used inside the kernel as they are process specific so support for sharing buffers with dma buf kernel handles is needed to support kernel only use cases. An example use case could be a GPU driver using ion that wants to share its output buffers with a 3d party display controller driver supporting dma buf.
--- drivers/staging/android/ion/ion.c | 26 ++++++++++++++++++++------ drivers/staging/android/ion/ion.h | 12 ++++++++++-- 2 files changed, 30 insertions(+), 8 deletions(-)
drivers/staging/android/ion/ion.c +20 -6
@@ -949,19 +949,19 @@ struct dma_buf_ops dma_buf_ops = { .kunmap = ion_dma_buf_kunmap, }; -int ion_share_dma_buf (struct ion_client *client,struct ion_handle *handle) +struct dma_buf *ion_share_dma_buf (struct ion_client *client, +struct ion_handle *handle) {struct ion_buffer *buffer;struct dma_buf *dmabuf;bool valid_handle; -int fd;mutex_lock (&client->lock); valid_handle =ion_handle_validate (client, handle);mutex_unlock (&client->lock);if (!valid_handle) {WARN (1 ,"%s: invalid handle passed to share.\n" , __func__); -return -EINVAL ; +return ERR_PTR (-EINVAL ); } buffer = handle->buffer;
@@ -969,15 +969,29 @@ int ion_share_dma_buf(struct ion_client *client, struct ion_handle *handle) dmabuf =dma_buf_export (buffer, &dma_buf_ops, buffer->size,O_RDWR );if (IS_ERR (dmabuf)) {ion_buffer_put (buffer); -return PTR_ERR (dmabuf); +return dmabuf; } + +return dmabuf; +} +EXPORT_SYMBOL (ion_share_dma_buf); + +int ion_share_dma_buf_fd (struct ion_client *client,struct ion_handle *handle) +{ +struct dma_buf *dmabuf; +int fd; + + dmabuf =ion_share_dma_buf (client, handle); +if (IS_ERR (dmabuf)) +return PTR_ERR (dmabuf); + fd =dma_buf_fd (dmabuf,O_CLOEXEC );if (fd <0 )dma_buf_put (dmabuf);return fd; } -EXPORT_SYMBOL (ion_share_dma_buf); +EXPORT_SYMBOL (ion_share_dma_buf_fd);struct ion_handle *ion_import_dma_buf (struct ion_client *client,int fd) {
@@ -1086,7 +1100,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; - data.fd =ion_share_dma_buf (client, data.handle); + data.fd =ion_share_dma_buf_fd (client, data.handle);if (copy_to_user ((void __user *)arg, &data,sizeof (data)))return -EFAULT ;if (data.fd <0 )
drivers/staging/android/ion/ion.h +10 -2
@@ -212,11 +212,19 @@ void *ion_map_kernel(struct ion_client *client, struct ion_handle *handle);void ion_unmap_kernel (struct ion_client *client,struct ion_handle *handle);/** -* ion_share_dma_buf() - given an ion client, create a dma-buf fd +* ion_share_dma_buf() - share buffer as dma-buf * @client: the client * @handle: the handle */ -int ion_share_dma_buf (struct ion_client *client,struct ion_handle *handle); +struct dma_buf *ion_share_dma_buf (struct ion_client *client, +struct ion_handle *handle); + +/** +* ion_share_dma_buf_fd() - given an ion client, create a dma-buf fd +* @client: the client +* @handle: the handle +*/ +int ion_share_dma_buf_fd (struct ion_client *client,struct ion_handle *handle);/** * ion_import_dma_buf() - given an dma-buf fd from the ion exporter get handle
Signature · 1 line
1.8.3.2
PATCH 61/115 gpu: ion: Make ion_free asynchronous
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Add the ability for a heap to free buffers asynchrounously. Freed buffers are placed on a free list and freed from a low priority background thread. If allocations from a particular heap fail, the free list is drained. This patch also enable asynchronous frees from the chunk heap.
--- drivers/staging/android/ion/ion.c | 112 +++++++++++++++++++++++--- drivers/staging/android/ion/ion.h | 2 +- drivers/staging/android/ion/ion_chunk_heap.c | 3 +- drivers/staging/android/ion/ion_priv.h | 20 ++++- drivers/staging/android/ion/ion_system_heap.c | 1 + 5 files changed, 126 insertions(+), 12 deletions(-)
drivers/staging/android/ion/ion.c +103 -9
@@ -17,8 +17,10 @@#include <linux/device.h> #include <linux/file.h> +#include <linux/freezer.h> #include <linux/fs.h> #include <linux/anon_inodes.h> +#include <linux/kthread.h> #include <linux/list.h> #include <linux/memblock.h> #include <linux/miscdevice.h>
@@ -26,6 +28,7 @@#include <linux/mm.h> #include <linux/mm_types.h> #include <linux/rbtree.h> +#include <linux/rtmutex.h> #include <linux/sched.h> #include <linux/slab.h> #include <linux/seq_file.h>
@@ -140,6 +143,7 @@ static void ion_buffer_add(struct ion_device *dev,static int ion_buffer_alloc_dirty (struct ion_buffer *buffer); +static bool ion_heap_drain_freelist (struct ion_heap *heap);/* this function should only be called while dev->lock is held */ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, struct ion_device *dev,
@@ -161,9 +165,16 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,kref_init (&buffer->ref); ret = heap->ops->allocate (heap, buffer, len, align, flags); +if (ret) { -kfree (buffer); -return ERR_PTR (ret); +if (!(heap->flags &ION_HEAP_FLAG_DEFER_FREE )) + goto err2; + +ion_heap_drain_freelist (heap); + ret = heap->ops->allocate (heap, buffer, len, align, + flags); +if (ret) + goto err2; } buffer->dev = dev;
@@ -214,27 +225,42 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, err: heap->ops->unmap_dma (heap, buffer); heap->ops->free (buffer); +err2:kfree (buffer);return ERR_PTR (ret); } -static void ion_buffer_destroy (struct kref *kref) +static void _ion_buffer_destroy (struct ion_buffer *buffer) { - struct ion_buffer *buffer = container_of(kref, struct ion_buffer, ref); -struct ion_device *dev = buffer->dev; -if (WARN_ON (buffer->kmap_cnt >0 )) buffer->heap->ops->unmap_kernel (buffer->heap, buffer); buffer->heap->ops->unmap_dma (buffer->heap, buffer); buffer->heap->ops->free (buffer); -mutex_lock (&dev->buffer_lock); -rb_erase (&buffer->node, &dev->buffers); -mutex_unlock (&dev->buffer_lock);if (buffer->flags &ION_FLAG_CACHED )kfree (buffer->dirty);kfree (buffer); } +static void ion_buffer_destroy (struct kref *kref) +{ + struct ion_buffer *buffer = container_of(kref, struct ion_buffer, ref); +struct ion_heap *heap = buffer->heap; +struct ion_device *dev = buffer->dev; + +mutex_lock (&dev->buffer_lock); +rb_erase (&buffer->node, &dev->buffers); +mutex_unlock (&dev->buffer_lock); + +if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) { +rt_mutex_lock (&heap->lock); +list_add (&buffer->list, &heap->free_list); +rt_mutex_unlock (&heap->lock); +wake_up (&heap->waitqueue); +return ; + } +_ion_buffer_destroy (buffer); +} +static void ion_buffer_get (struct ion_buffer *buffer) {kref_get (&buffer->ref);
@@ -1272,13 +1298,81 @@ static const struct file_operations debug_heap_fops = { .release = single_release, }; +static size_t ion_heap_free_list_is_empty (struct ion_heap *heap) +{ +bool is_empty; + +rt_mutex_lock (&heap->lock); + is_empty =list_empty (&heap->free_list); +rt_mutex_unlock (&heap->lock); + +return is_empty; +} + +static int ion_heap_deferred_free (void *data) +{ +struct ion_heap *heap = data; + +while (true) { +struct ion_buffer *buffer; + +wait_event_freezable (heap->waitqueue, + !ion_heap_free_list_is_empty (heap)); + +rt_mutex_lock (&heap->lock); +if (list_empty (&heap->free_list)) { +rt_mutex_unlock (&heap->lock); +continue ; + } + buffer = list_first_entry(&heap->free_list, struct ion_buffer, + list); +list_del (&buffer->list); +rt_mutex_unlock (&heap->lock); +_ion_buffer_destroy (buffer); + } + +return 0 ; +} + +static bool ion_heap_drain_freelist (struct ion_heap *heap) +{ +struct ion_buffer *buffer, *tmp; + +if (ion_heap_free_list_is_empty (heap)) +return false; +rt_mutex_lock (&heap->lock); +list_for_each_entry_safe (buffer, tmp, &heap->free_list, list) { +_ion_buffer_destroy (buffer); +list_del (&buffer->list); + } +BUG_ON (!list_empty (&heap->free_list)); +rt_mutex_unlock (&heap->lock); + + +return true; +} +void ion_device_add_heap (struct ion_device *dev,struct ion_heap *heap) { +struct sched_param param = { .sched_priority =0 }; +if (!heap->ops->allocate || !heap->ops->free || !heap->ops->map_dma || !heap->ops->unmap_dma)pr_err ("%s: can not add heap with invalid ops struct.\n" , __func__); +if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) { +INIT_LIST_HEAD (&heap->free_list); +rt_mutex_init (&heap->lock); +init_waitqueue_head (&heap->waitqueue); + heap->task =kthread_run (ion_heap_deferred_free, heap, +"%s" , heap->name); +sched_setscheduler (heap->task,SCHED_IDLE , ¶m); +if (IS_ERR (heap->task)) +pr_err ("%s: creating thread for deferred free failed\n" , + __func__); + } + heap->dev = dev;down_write (&dev->lock);/* use negative heap->id to reverse the priority -- when traversing
drivers/staging/android/ion/ion.h +1 -1
@@ -46,7 +46,7 @@ enum ion_heap_type {#define ION_NUM_HEAP_IDS sizeof(unsigned int) * 8/** -* heap flags - the lower 16 bits are used by core ion, the upper 16 +* allocation flags - the lower 16 bits are used by core ion, the upper 16 * bits are reserved for use by the heaps themselves. */ #define ION_FLAG_CACHED 1/* mappings of this buffer should be
drivers/staging/android/ion/ion_chunk_heap.c +2 -1
@@ -160,7 +160,8 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data)gen_pool_add (chunk_heap->pool, chunk_heap->base, heap_data->size,-1 ); chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ; -pr_info ("%s: base %lu size %ld align %ld\n" , __func__, chunk_heap->base, + chunk_heap->heap.flags =ION_HEAP_FLAG_DEFER_FREE ; +pr_info ("%s: base %lu size %u align %ld\n" , __func__, chunk_heap->base, heap_data->size, heap_data->align);return &chunk_heap->heap;
drivers/staging/android/ion/ion_priv.h +19 -1
@@ -58,7 +58,10 @@ struct ion_buffer *ion_handle_buffer(struct ion_handle *handle);*/ struct ion_buffer {struct kref ref; -struct rb_node node; +union { +struct rb_node node; +struct list_head list; + };struct ion_device *dev;struct ion_heap *heap;unsigned long flags;
@@ -109,15 +112,25 @@ struct ion_heap_ops { };/** +* heap flags - flags between the heaps and core ion code +*/ +#define ION_HEAP_FLAG_DEFER_FREE (1 << 0) + +/** * struct ion_heap - represents a heap in the system * @node: rb node to put the heap on the device's tree of heaps * @dev: back pointer to the ion_device * @type: type of heap * @ops: ops struct as above +* @flags: flags * @id: id of heap, also indicates priority of this heap when * allocating. These are specified by platform data and * MUST be unique * @name: used for debugging +* @free_list: free list head if deferred free is used +* @lock: protects the free list +* @waitqueue: queue to wait on from deferred free thread +* @task: task struct of deferred free thread * @debug_show: called when heap debug file is read to add any * heap specific debug info to output *
@@ -131,8 +144,13 @@ struct ion_heap {struct ion_device *dev;enum ion_heap_type type;struct ion_heap_ops *ops; +unsigned long flags;unsigned int id;const char *name; +struct list_head free_list; +struct rt_mutex lock; +wait_queue_head_t waitqueue; +struct task_struct *task;int (*debug_show)(struct ion_heap *heap,struct seq_file *,void *); };
drivers/staging/android/ion/ion_system_heap.c +1 -0
@@ -283,6 +283,7 @@ struct ion_heap *ion_system_heap_create(struct ion_platform_heap *unused)return ERR_PTR (-ENOMEM ); heap->heap.ops = &system_heap_ops; heap->heap.type =ION_HEAP_TYPE_SYSTEM ; + heap->heap.flags =ION_HEAP_FLAG_DEFER_FREE ; heap->pools =kzalloc (sizeof (struct ion_page_pool *) * num_orders,GFP_KERNEL );if (!heap->pools)
Signature · 1 line
1.8.3.2
PATCH 62/115 gpu: ion: fix kfree/list_del order
In reply to John StultzFrom: JP Abgrall <[email protected]>
With CONFIG_SLUB_DEBUG_ON it would panic duringion_alloc()
ion_buffer_create()
io_heap_drain_freelist()
--- drivers/staging/android/ion/ion.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +1 -1
@@ -1342,8 +1342,8 @@ static bool ion_heap_drain_freelist(struct ion_heap *heap)return false;rt_mutex_lock (&heap->lock);list_for_each_entry_safe (buffer, tmp, &heap->free_list, list) { -_ion_buffer_destroy (buffer);list_del (&buffer->list); +_ion_buffer_destroy (buffer); }BUG_ON (!list_empty (&heap->free_list));rt_mutex_unlock (&heap->lock);
Signature · 1 line
1.8.3.2
PATCH 63/115 gpu: ion: ion_chunk_heap: Zero chunk heap memory at creation time
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Allocations from the ion heap need to be zeroed to protect userspace from seeing memory belonging to other processes. First allocations from this heap were not zero'd allowing users to see memory from other processes on a warm reset.
--- drivers/staging/android/ion/ion_chunk_heap.c | 34 ++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +32 -2
@@ -140,6 +140,10 @@ static struct ion_heap_ops chunk_heap_ops = {struct ion_heap *ion_chunk_heap_create (struct ion_platform_heap *heap_data) {struct ion_chunk_heap *chunk_heap; +struct vm_struct *vm_struct; +pgprot_t pgprot =pgprot_writecombine (PAGE_KERNEL ); +int i, ret; + chunk_heap =kzalloc (sizeof (struct ion_chunk_heap ),GFP_KERNEL );if (!chunk_heap)
@@ -149,12 +153,30 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) chunk_heap->pool =gen_pool_create (get_order (chunk_heap->chunk_size) +PAGE_SHIFT ,-1 );if (!chunk_heap->pool) { -kfree (chunk_heap); -return ERR_PTR (-ENOMEM ); + ret = -ENOMEM ; + goto error_gen_pool_create; } chunk_heap->base = heap_data->base; chunk_heap->size = heap_data->size; chunk_heap->allocated =0 ; + + vm_struct =get_vm_area (PAGE_SIZE ,VM_ALLOC ); +if (!vm_struct) { + ret = -ENOMEM ; + goto error; + } +for (i =0 ; i < chunk_heap->size; i +=PAGE_SIZE ) { +struct page *page =phys_to_page (chunk_heap->base + i); +struct page **pages = &page; + + ret =map_vm_area (vm_struct, pgprot, &pages); +if (ret) + goto error_map_vm_area; +memset (vm_struct->addr,0 ,PAGE_SIZE ); +unmap_kernel_range ((unsigned long )vm_struct->addr,PAGE_SIZE ); + } +free_vm_area (vm_struct); +__dma_page_cpu_to_dev (phys_to_page (heap_data->base),0 , heap_data->size,DMA_BIDIRECTIONAL );gen_pool_add (chunk_heap->pool, chunk_heap->base, heap_data->size,-1 );
@@ -165,6 +187,14 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) heap_data->size, heap_data->align);return &chunk_heap->heap; + +error_map_vm_area: +free_vm_area (vm_struct); +error: +gen_pool_destroy (chunk_heap->pool); +error_gen_pool_create: +kfree (chunk_heap); +return ERR_PTR (ret); }void ion_chunk_heap_destroy (struct ion_heap *heap)
Signature · 1 line
1.8.3.2
PATCH 64/115 gpu: ion: Fix bug in ion shrinker
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
The high variable was sometimes used uninitialized
--- drivers/staging/android/ion/ion_page_pool.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/staging/android/ion/ion_page_pool.c +1 -2
@@ -203,8 +203,7 @@ static int ion_page_pool_shrink(struct shrinker *shrinker,bool high;int nr_to_scan = sc->nr_to_scan; -if (sc->gfp_mask & __GFP_HIGHMEM) - high = true; + high = sc->gfp_mask & __GFP_HIGHMEM;if (nr_to_scan ==0 )return ion_page_pool_total (high);
Signature · 1 line
1.8.3.2
PATCH 65/115 gpu: ion: Also shrink memory cached in the deferred free list
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
When the system is low on memory, we want to shrink any cached system memory ion is holding. Previously we were shrinking memory in the page pools, but not in the deferred free list. This patch makes it possible to shrink both. It also moves the shrinker code into the heaps so they can correctly manage any caches they might contain.
--- drivers/staging/android/ion/ion.c | 115 +++++++++-------------- drivers/staging/android/ion/ion_heap.c | 107 ++++++++++++++++++++++ drivers/staging/android/ion/ion_page_pool.c | 126 +++++--------------------- drivers/staging/android/ion/ion_priv.h | 57 +++++++++++- drivers/staging/android/ion/ion_system_heap.c | 49 ++++++++++ 5 files changed, 278 insertions(+), 176 deletions(-)
drivers/staging/android/ion/ion.c +45 -70
@@ -28,8 +28,6 @@#include <linux/mm.h> #include <linux/mm_types.h> #include <linux/rbtree.h> -#include <linux/rtmutex.h> -#include <linux/sched.h> #include <linux/slab.h> #include <linux/seq_file.h> #include <linux/uaccess.h>
@@ -143,7 +141,6 @@ static void ion_buffer_add(struct ion_device *dev,static int ion_buffer_alloc_dirty (struct ion_buffer *buffer); -static bool ion_heap_drain_freelist (struct ion_heap *heap);/* this function should only be called while dev->lock is held */ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, struct ion_device *dev,
@@ -170,7 +167,7 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,if (!(heap->flags &ION_HEAP_FLAG_DEFER_FREE )) goto err2; -ion_heap_drain_freelist (heap); +ion_heap_freelist_drain (heap,0 ); ret = heap->ops->allocate (heap, buffer, len, align, flags);if (ret)
@@ -230,7 +227,7 @@ err2:return ERR_PTR (ret); } -static void _ion_buffer_destroy (struct ion_buffer *buffer) +void ion_buffer_destroy (struct ion_buffer *buffer) {if (WARN_ON (buffer->kmap_cnt >0 )) buffer->heap->ops->unmap_kernel (buffer->heap, buffer);
@@ -241,7 +238,7 @@ static void _ion_buffer_destroy(struct ion_buffer *buffer)kfree (buffer); } -static void ion_buffer_destroy (struct kref *kref) +static void _ion_buffer_destroy (struct kref *kref) { struct ion_buffer *buffer = container_of(kref, struct ion_buffer, ref);struct ion_heap *heap = buffer->heap;
@@ -251,14 +248,10 @@ static void ion_buffer_destroy(struct kref *kref)rb_erase (&buffer->node, &dev->buffers);mutex_unlock (&dev->buffer_lock); -if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) { -rt_mutex_lock (&heap->lock); -list_add (&buffer->list, &heap->free_list); -rt_mutex_unlock (&heap->lock); -wake_up (&heap->waitqueue); -return ; - } -_ion_buffer_destroy (buffer); +if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) +ion_heap_freelist_add (heap, buffer); +else +ion_buffer_destroy (buffer); }static void ion_buffer_get (struct ion_buffer *buffer)
@@ -268,7 +261,7 @@ static void ion_buffer_get(struct ion_buffer *buffer)static int ion_buffer_put (struct ion_buffer *buffer) { -return kref_put (&buffer->ref, ion_buffer_destroy); +return kref_put (&buffer->ref, _ion_buffer_destroy); }static void ion_buffer_add_to_handle (struct ion_buffer *buffer)
@@ -1298,80 +1291,53 @@ static const struct file_operations debug_heap_fops = { .release = single_release, }; -static size_t ion_heap_free_list_is_empty (struct ion_heap *heap) +#ifdef DEBUG_HEAP_SHRINKER +static int debug_shrink_set (void *data,u64 val) { -bool is_empty; +struct ion_heap *heap = data; +struct shrink_control sc; +int objs; -rt_mutex_lock (&heap->lock); - is_empty =list_empty (&heap->free_list); -rt_mutex_unlock (&heap->lock); + sc.gfp_mask =-1 ; + sc.nr_to_scan =0 ; -return is_empty; -} - -static int ion_heap_deferred_free (void *data) -{ -struct ion_heap *heap = data; +if (!val) +return 0 ; -while (true) { -struct ion_buffer *buffer; + objs = heap->shrinker.shrink (&heap->shrinker, &sc); + sc.nr_to_scan = objs; -wait_event_freezable (heap->waitqueue, - !ion_heap_free_list_is_empty (heap)); - -rt_mutex_lock (&heap->lock); -if (list_empty (&heap->free_list)) { -rt_mutex_unlock (&heap->lock); -continue ; - } - buffer = list_first_entry(&heap->free_list, struct ion_buffer, - list); -list_del (&buffer->list); -rt_mutex_unlock (&heap->lock); -_ion_buffer_destroy (buffer); - } - -return 0 ; + heap->shrinker.shrink (&heap->shrinker, &sc); +return 0 ; } -static bool ion_heap_drain_freelist (struct ion_heap *heap) +static int debug_shrink_get (void *data,u64 *val) { -struct ion_buffer *buffer, *tmp; - -if (ion_heap_free_list_is_empty (heap)) -return false; -rt_mutex_lock (&heap->lock); -list_for_each_entry_safe (buffer, tmp, &heap->free_list, list) { -list_del (&buffer->list); -_ion_buffer_destroy (buffer); - } -BUG_ON (!list_empty (&heap->free_list)); -rt_mutex_unlock (&heap->lock); +struct ion_heap *heap = data; +struct shrink_control sc; +int objs; + sc.gfp_mask =-1 ; + sc.nr_to_scan =0 ; -return true; + objs = heap->shrinker.shrink (&heap->shrinker, &sc); + *val = objs; +return 0 ; } +DEFINE_SIMPLE_ATTRIBUTE (debug_shrink_fops, debug_shrink_get, + debug_shrink_set,"%llu\n" ); +#endif +void ion_device_add_heap (struct ion_device *dev,struct ion_heap *heap) { -struct sched_param param = { .sched_priority =0 }; -if (!heap->ops->allocate || !heap->ops->free || !heap->ops->map_dma || !heap->ops->unmap_dma)pr_err ("%s: can not add heap with invalid ops struct.\n" , __func__); -if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) { -INIT_LIST_HEAD (&heap->free_list); -rt_mutex_init (&heap->lock); -init_waitqueue_head (&heap->waitqueue); - heap->task =kthread_run (ion_heap_deferred_free, heap, -"%s" , heap->name); -sched_setscheduler (heap->task,SCHED_IDLE , ¶m); -if (IS_ERR (heap->task)) -pr_err ("%s: creating thread for deferred free failed\n" , - __func__); - } +if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) +ion_heap_init_deferred_free (heap); heap->dev = dev;down_write (&dev->lock);
@@ -1381,6 +1347,15 @@ void ion_device_add_heap(struct ion_device *dev, struct ion_heap *heap)plist_add (&heap->node, &dev->heaps);debugfs_create_file (heap->name,0664 , dev->debug_root, heap, &debug_heap_fops); +#ifdef DEBUG_HEAP_SHRINKER +if (heap->shrinker.shrink) { +char debug_name[64 ]; + +snprintf (debug_name,64 ,"%s_shrink" , heap->name); +debugfs_create_file (debug_name,0644 , dev->debug_root, heap, + &debug_shrink_fops); + } +#endif up_write (&dev->lock); }
drivers/staging/android/ion/ion_heap.c +107 -0
120 lines not on this page · raw message
drivers/staging/android/ion/ion_page_pool.c +22 -104
180 lines not on this page · raw message
drivers/staging/android/ion/ion_priv.h +55 -2
93 lines not on this page · raw message
drivers/staging/android/ion/ion_system_heap.c +49 -0
61 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 66/115 gpu: ion: __dma_page_cpu_to_dev -> arm_dma_ops.sync_single_for_device hack
In reply to John StultzFrom: Arve Hjønnevåg <[email protected]>
--- drivers/staging/android/ion/ion_chunk_heap.c | 10 ++++++---- drivers/staging/android/ion/ion_page_pool.c | 6 ++++-- drivers/staging/android/ion/ion_system_heap.c | 5 +++-- 3 files changed, 13 insertions(+), 8 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +6 -4
@@ -105,8 +105,9 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)for_each_sg (table->sgl, sg, table->nents, i) {if (ion_buffer_cached (buffer)) -__dma_page_cpu_to_dev (sg_page (sg),0 ,sg_dma_len (sg), -DMA_BIDIRECTIONAL ); + arm_dma_ops.sync_single_for_device (NULL , +pfn_to_dma (NULL ,page_to_pfn (sg_page (sg))), +sg_dma_len (sg),DMA_BIDIRECTIONAL );gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)),sg_dma_len (sg)); }
@@ -177,8 +178,9 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) }free_vm_area (vm_struct); -__dma_page_cpu_to_dev (phys_to_page (heap_data->base),0 , heap_data->size, -DMA_BIDIRECTIONAL ); + arm_dma_ops.sync_single_for_device (NULL , +pfn_to_dma (NULL ,page_to_pfn (phys_to_page (heap_data->base))), + heap_data->size,DMA_BIDIRECTIONAL );gen_pool_add (chunk_heap->pool, chunk_heap->base, heap_data->size,-1 ); chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ;
drivers/staging/android/ion/ion_page_pool.c +4 -2
@@ -37,8 +37,10 @@ static void *ion_page_pool_alloc_pages(struct ion_page_pool *pool)/* this is only being used to flush the page for dma, this api is not really suitable for calling from a driver but no better way to flush a page for dma exist at this time */ -__dma_page_cpu_to_dev (page,0 ,PAGE_SIZE << pool->order, -DMA_BIDIRECTIONAL ); + arm_dma_ops.sync_single_for_device (NULL , +pfn_to_dma (NULL ,page_to_pfn (page)), +PAGE_SIZE << pool->order, +DMA_BIDIRECTIONAL );return page; }
drivers/staging/android/ion/ion_system_heap.c +3 -2
@@ -78,8 +78,9 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap, page =alloc_pages (gfp_flags, order);if (!page)return 0 ; -__dma_page_cpu_to_dev (page,0 ,PAGE_SIZE << order, -DMA_BIDIRECTIONAL ); + arm_dma_ops.sync_single_for_device (NULL , +pfn_to_dma (NULL ,page_to_pfn (page)), +PAGE_SIZE << order,DMA_BIDIRECTIONAL ); }if (!page)return 0 ;
Signature · 1 line
1.8.3.2
PATCH 67/115 gpu: ion: Remove __GFP_NO_KSWAPD
In reply to John StultzFrom: Arve Hjønnevåg <[email protected]>
It no longer exists.
--- drivers/staging/android/ion/ion_system_heap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +2 -2
@@ -27,8 +27,8 @@#include "ion_priv.h" static unsigned int high_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | - __GFP_NOWARN | __GFP_NORETRY | - __GFP_NO_KSWAPD) & ~__GFP_WAIT; + __GFP_NOWARN | __GFP_NORETRY) & + ~__GFP_WAIT;static unsigned int low_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN);static const unsigned int orders[] = {8 ,4 ,0 };
Signature · 1 line
1.8.3.2
PATCH 68/115 ion: Add Kconfig dependency to ARM
In reply to John StultzThe ion code has some very specific arm-isms which keeps it from building on other architectures. These should probably be resolved, but in the mean time, add a dependency on CONFIG_ARM to avoid build failures.
v2: Fix earlier flub, sending out an early untested version of the patch.
--- drivers/staging/android/ion/Kconfig | 1 + 1 file changed, 1 insertion(+)
drivers/staging/android/ion/Kconfig +1 -0
@@ -1,5 +1,6 @@ menuconfig ION tristate "Ion Memory Manager" + depends on ARM select GENERIC_ALLOCATOR select DMA_SHARED_BUFFER help
Signature · 1 line
1.8.3.2
PATCH 69/115 gpu: ion: fix ion_platform_data definition
In reply to John StultzFrom: Benjamin Gaignard <[email protected]>
fix ion_platform_heap to make is use an usual way in board configuration file.
--- drivers/staging/android/ion/ion.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion.h +1 -1
@@ -103,7 +103,7 @@ struct ion_platform_heap {*/ struct ion_platform_data {int nr; -struct ion_platform_heap heaps[]; +struct ion_platform_heap *heaps; };/**
Signature · 1 line
1.8.3.2
PATCH 70/115 gpu: ion: add CMA heap
In reply to John StultzFrom: Benjamin Gaignard <[email protected]>
New heap type ION_HEAP_TYPE_DMA where allocation is done with dma_alloc_coherent API. device coherent_dma_mask must be set to DMA_BIT_MASK(32). ion_platform_heap private field is used to retrieve the device linked to CMA, if NULL the default CMA area is used.ion_cma_get_sgtable is a copy of dma_common_get_sgtable function which should be in kernel 3.5
--- drivers/staging/android/ion/Makefile | 2 +- drivers/staging/android/ion/ion.h | 7 +- drivers/staging/android/ion/ion_cma_heap.c | 233 +++++++++++++++++++++++++++++ drivers/staging/android/ion/ion_heap.c | 6 + drivers/staging/android/ion/ion_priv.h | 3 + 5 files changed, 249 insertions(+), 2 deletions(-) create mode 100644 drivers/staging/android/ion/ion_cma_heap.c
drivers/staging/android/ion/Makefile +1 -1
@@ -1,3 +1,3 @@ obj-$(CONFIG_ION) += ion.o ion_heap.o ion_page_pool.o ion_system_heap.o \ - ion_carveout_heap.o ion_chunk_heap.o + ion_carveout_heap.o ion_chunk_heap.o ion_cma_heap.o obj-$(CONFIG_ION_TEGRA) += tegra/
drivers/staging/android/ion/ion.h +6 -1
@@ -27,13 +27,17 @@ struct ion_handle;* @ION_HEAP_TYPE_CARVEOUT: memory allocated from a prereserved * carveout heap, allocations are physically * contiguous -* @ION_HEAP_END: helper for iterating over heaps +* @ION_HEAP_TYPE_DMA: memory allocated via DMA API +* @ION_NUM_HEAPS: helper for iterating over heaps, a bit mask +* is used to identify the heaps, so only 32 +* total heap types are supported */ enum ion_heap_type {ION_HEAP_TYPE_SYSTEM ,ION_HEAP_TYPE_SYSTEM_CONTIG ,ION_HEAP_TYPE_CARVEOUT ,ION_HEAP_TYPE_CHUNK , +ION_HEAP_TYPE_DMA ,ION_HEAP_TYPE_CUSTOM ,/* must be last so device specific heaps always are at the end of this enum */ ION_NUM_HEAPS ,
@@ -42,6 +46,7 @@ enum ion_heap_type {#define ION_HEAP_SYSTEM_MASK (1 << ION_HEAP_TYPE_SYSTEM)#define ION_HEAP_SYSTEM_CONTIG_MASK (1 << ION_HEAP_TYPE_SYSTEM_CONTIG)#define ION_HEAP_CARVEOUT_MASK (1 << ION_HEAP_TYPE_CARVEOUT) +#define ION_HEAP_TYPE_DMA_MASK (1 << ION_HEAP_TYPE_DMA)#define ION_NUM_HEAP_IDS sizeof(unsigned int) * 8
drivers/staging/android/ion/ion_cma_heap.c new file +233 -0
233 lines not on this page · raw message
drivers/staging/android/ion/ion_heap.c +6 -0
@@ -254,6 +254,9 @@ struct ion_heap *ion_heap_create(struct ion_platform_heap *heap_data)case ION_HEAP_TYPE_CHUNK : heap =ion_chunk_heap_create (heap_data);break ; +case ION_HEAP_TYPE_DMA : + heap =ion_cma_heap_create (heap_data); +break ;default :pr_err ("%s: Invalid heap type %d\n" , __func__, heap_data->type);
@@ -290,6 +293,9 @@ void ion_heap_destroy(struct ion_heap *heap)case ION_HEAP_TYPE_CHUNK :ion_chunk_heap_destroy (heap);break ; +case ION_HEAP_TYPE_DMA : +ion_cma_heap_destroy (heap); +break ;default :pr_err ("%s: Invalid heap type %d\n" , __func__, heap->type);
drivers/staging/android/ion/ion_priv.h +3 -0
@@ -270,6 +270,9 @@ void ion_carveout_heap_destroy(struct ion_heap *);struct ion_heap *ion_chunk_heap_create (struct ion_platform_heap *);void ion_chunk_heap_destroy (struct ion_heap *); +struct ion_heap *ion_cma_heap_create (struct ion_platform_heap *); +void ion_cma_heap_destroy (struct ion_heap *); +/** * kernel api to allocate/free from carveout -- used when carveout is * used to back an architecture specific custom heap
Signature · 1 line
1.8.3.2
PATCH 71/115 gpu: ion: Fix performance issue in faulting code
In reply to John StultzFrom: Rebecca Schultz Zavin <[email protected]>
Previously the code to fault ion buffers in one page at a time had a performance problem caused by the requirement to traverse the sg list looking for the right page to load in (a result of the fact that the items in the list may not be of uniform size). To fix the problem, for buffers that will be faulted in, also keep a flat array of all the pages in the buffer to use from the fault handler. To recover some of the additional memory footprint this creates per buffer, dirty bits used to indicate which pages have been faulted in to the cpu are now stored in the low bit of each page struct pointer in the page array.
--- drivers/staging/android/ion/ion.c | 105 +++++++++++++++----------- drivers/staging/android/ion/ion_cma_heap.c | 35 +-------- drivers/staging/android/ion/ion_heap.c | 18 ++++- drivers/staging/android/ion/ion_priv.h | 20 ++++- drivers/staging/android/ion/ion_system_heap.c | 25 +----- 5 files changed, 99 insertions(+), 104 deletions(-)
drivers/staging/android/ion/ion.c +60 -45
@@ -31,6 +31,7 @@#include <linux/slab.h> #include <linux/seq_file.h> #include <linux/uaccess.h> +#include <linux/vmalloc.h> #include <linux/debugfs.h> #include <linux/dma-buf.h>
@@ -104,13 +105,33 @@ struct ion_handle {bool ion_buffer_fault_user_mappings (struct ion_buffer *buffer) { -return ((buffer->flags &ION_FLAG_CACHED ) && - !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )); +return ((buffer->flags &ION_FLAG_CACHED ) && + !(buffer->flags &ION_FLAG_CACHED_NEEDS_SYNC )); }bool ion_buffer_cached (struct ion_buffer *buffer) { -return !!(buffer->flags &ION_FLAG_CACHED ); +return !!(buffer->flags &ION_FLAG_CACHED ); +} + +static inline struct page *ion_buffer_page (struct page *page) +{ +return (struct page *)((unsigned long )page & ~(1UL )); +} + +static inline bool ion_buffer_page_is_dirty (struct page *page) +{ +return !!((unsigned long )page &1UL ); +} + +static inline void ion_buffer_page_dirty (struct page **page) +{ + *page = (struct page *)((unsigned long )(*page) |1UL ); +} + +static inline void ion_buffer_page_clean (struct page **page) +{ + *page = (struct page *)((unsigned long )(*page) & ~(1UL )); }/* this function should only be called while dev->lock is held */
@@ -139,8 +160,6 @@ static void ion_buffer_add(struct ion_device *dev,rb_insert_color (&buffer->node, &dev->buffers); } -static int ion_buffer_alloc_dirty (struct ion_buffer *buffer); -/* this function should only be called while dev->lock is held */ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, struct ion_device *dev,
@@ -185,17 +204,23 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, } buffer->sg_table = table;if (ion_buffer_fault_user_mappings (buffer)) { -for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, - i) { -if (sg_dma_len (sg) ==PAGE_SIZE ) -continue ; -pr_err ("%s: cached mappings that will be faulted in " -"must have pagewise sg_lists\n" , __func__); - ret = -EINVAL ; - goto err; +int num_pages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; +struct scatterlist *sg; +int i, j, k =0 ; + + buffer->pages =vmalloc (sizeof (struct page *) * num_pages); +if (!buffer->pages) { + ret = -ENOMEM ; + goto err1; + } + +for_each_sg (table->sgl, sg, table->nents, i) { +struct page *page =sg_page (sg); + +for (j =0 ; j <sg_dma_len (sg) /PAGE_SIZE ; j++) + buffer->pages[k++] = page++; } - ret =ion_buffer_alloc_dirty (buffer);if (ret) goto err; }
@@ -222,6 +247,9 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, err: heap->ops->unmap_dma (heap, buffer); heap->ops->free (buffer); +err1: +if (buffer->pages) +vfree (buffer->pages); err2:kfree (buffer);return ERR_PTR (ret);
@@ -233,8 +261,8 @@ void ion_buffer_destroy(struct ion_buffer *buffer) buffer->heap->ops->unmap_kernel (buffer->heap, buffer); buffer->heap->ops->unmap_dma (buffer->heap, buffer); buffer->heap->ops->free (buffer); -if (buffer->flags &ION_FLAG_CACHED ) -kfree (buffer->dirty); +if (buffer->pages) +vfree (buffer->pages);kfree (buffer); }
@@ -764,17 +792,6 @@ static void ion_unmap_dma_buf(struct dma_buf_attachment *attachment, { } -static int ion_buffer_alloc_dirty (struct ion_buffer *buffer) -{ -unsigned long pages = buffer->sg_table->nents; -unsigned long length = (pages +BITS_PER_LONG -1 )/BITS_PER_LONG ; - - buffer->dirty =kzalloc (length *sizeof (unsigned long ),GFP_KERNEL ); -if (!buffer->dirty) -return -ENOMEM ; -return 0 ; -} -struct ion_vma_list {struct list_head list;struct vm_area_struct *vma;
@@ -784,9 +801,9 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer, struct device *dev, enum dma_data_direction dir) { -struct scatterlist *sg; -int i;struct ion_vma_list *vma_list; +int pages =PAGE_ALIGN (buffer->size) /PAGE_SIZE ; +int i;pr_debug ("%s: syncing for device %s\n" , __func__, dev ?dev_name (dev) :"null" );
@@ -795,11 +812,12 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,return ;mutex_lock (&buffer->lock); -for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) { -if (!test_bit (i, buffer->dirty)) -continue ; -dma_sync_sg_for_device (dev, sg,1 , dir); -clear_bit (i, buffer->dirty); +for (i =0 ; i < pages; i++) { +struct page *page = buffer->pages[i]; + +if (ion_buffer_page_is_dirty (page)) +__dma_page_cpu_to_dev (page,0 ,PAGE_SIZE , dir); +ion_buffer_page_clean (buffer->pages + i); }list_for_each_entry (vma_list, &buffer->vmas, list) {struct vm_area_struct *vma = vma_list->vma;
@@ -813,21 +831,18 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,int ion_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) {struct ion_buffer *buffer = vma->vm_private_data; -struct scatterlist *sg; -int i; +int ret;mutex_lock (&buffer->lock); -set_bit (vmf->pgoff, buffer->dirty); +ion_buffer_page_dirty (buffer->pages + vmf->pgoff); -for_each_sg (buffer->sg_table->sgl, sg, buffer->sg_table->nents, i) { -if (i != vmf->pgoff) -continue ; -dma_sync_sg_for_cpu (NULL , sg,1 ,DMA_BIDIRECTIONAL ); -vm_insert_page (vma, (unsigned long )vmf->virtual_address, -sg_page (sg)); -break ; - } +BUG_ON (!buffer->pages || !buffer->pages[vmf->pgoff]); + ret =vm_insert_page (vma, (unsigned long )vmf->virtual_address, +ion_buffer_page (buffer->pages[vmf->pgoff]));mutex_unlock (&buffer->lock); +if (ret) +return VM_FAULT_ERROR ; +return VM_FAULT_NOPAGE ; }
drivers/staging/android/ion/ion_cma_heap.c +3 -32
@@ -58,29 +58,6 @@ int ion_cma_get_sgtable(struct device *dev, struct sg_table *sgt,return 0 ; } -/* -* Create scatter-list for each page of the already allocated DMA buffer. -*/ -int ion_cma_get_sgtable_per_page (struct device *dev,struct sg_table *sgt, -void *cpu_addr,dma_addr_t handle,size_t size) -{ -struct page *page =virt_to_page (cpu_addr); -int ret, i; -struct scatterlist *sg; - - ret =sg_alloc_table (sgt,PAGE_ALIGN (size) /PAGE_SIZE ,GFP_KERNEL ); -if (unlikely (ret)) -return ret; - - sg = sgt->sgl; -for (i =0 ; i < (PAGE_ALIGN (size) /PAGE_SIZE ); i++) { - page =virt_to_page (cpu_addr + (i *PAGE_SIZE )); -sg_set_page (sg, page,PAGE_SIZE ,0 ); - sg =sg_next (sg); - } -return 0 ; -} -/* ION CMA heap operations functions */ static int ion_cma_allocate(struct ion_heap *heap, struct ion_buffer *buffer, unsigned long len, unsigned long align,
@@ -111,15 +88,9 @@ static int ion_cma_allocate(struct ion_heap *heap, struct ion_buffer *buffer, goto free_mem; } -if (ion_buffer_fault_user_mappings (buffer)) { -if (ion_cma_get_sgtable_per_page - (dev, info->table, info->cpu_addr, info->handle, len)) - goto free_table; - }else { -if (ion_cma_get_sgtable - (dev, info->table, info->cpu_addr, info->handle, len)) - goto free_table; - } +if (ion_cma_get_sgtable + (dev, info->table, info->cpu_addr, info->handle, len)) + goto free_table;/* keep this for memory release */ buffer->priv_virt = info;dev_dbg (dev,"Allocate buffer %p\n" , buffer);
drivers/staging/android/ion/ion_heap.c +16 -2
@@ -134,8 +134,22 @@ end:return ret; } -void ion_heap_free_page (struct ion_buffer *buffer,struct page *page, -unsigned int order) +struct page *ion_heap_alloc_pages (struct ion_buffer *buffer,gfp_t gfp_flags, +unsigned int order) +{ +struct page *page =alloc_pages (gfp_flags, order); + +if (!page) +return page; + +if (ion_buffer_fault_user_mappings (buffer)) +split_page (page, order); + +return page; +} + +void ion_heap_free_pages (struct ion_buffer *buffer,struct page *page, +unsigned int order) {int i;
drivers/staging/android/ion/ion_priv.h +16 -4
@@ -46,9 +46,8 @@ struct ion_buffer *ion_handle_buffer(struct ion_handle *handle);* @vaddr: the kenrel mapping if kmap_cnt is not zero * @dmap_cnt: number of times the buffer is mapped for dma * @sg_table: the sg table for the buffer if dmap_cnt is not zero -* @dirty: bitmask representing which pages of this buffer have -* been dirtied by the cpu and need cache maintenance -* before dma +* @pages: flat array of pages in the buffer -- used by fault +* handler and only valid for buffers that are faulted in * @vmas: list of vma's mapping this buffer * @handle_count: count of handles referencing this buffer * @task_comm: taskcomm of last client to reference this buffer in a
@@ -75,7 +74,7 @@ struct ion_buffer {void *vaddr;int dmap_cnt;struct sg_table *sg_table; -unsigned long *dirty; +struct page **pages;struct list_head vmas;/* used to track orphaned buffers */ int handle_count;
@@ -214,6 +213,19 @@ int ion_heap_map_user(struct ion_heap *, struct ion_buffer *,int ion_heap_buffer_zero (struct ion_buffer *buffer);/** +* ion_heap_alloc_pages - allocate pages from alloc_pages +* @buffer: the buffer to allocate for, used to extract the flags +* @gfp_flags: the gfp_t for the allocation +* @order: the order of the allocatoin +* +* This funciton allocations from alloc pages and also does any other +* necessary operations based on the buffer->flags. For buffers which +* will be faulted in the pages are split using split_page +*/ +struct page *ion_heap_alloc_pages (struct ion_buffer *buffer,gfp_t gfp_flags, +unsigned int order); + +/** * ion_heap_init_deferred_free -- initialize deferred free functionality * @heap: the heap *
drivers/staging/android/ion/ion_system_heap.c +4 -21
@@ -64,7 +64,6 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap, unsigned long order) {bool cached =ion_buffer_cached (buffer); -bool split_pages =ion_buffer_fault_user_mappings (buffer);struct ion_page_pool *pool = heap->pools[order_to_index (order)];struct page *page;
@@ -75,7 +74,7 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,if (order >4 ) gfp_flags = high_order_gfp_flags; - page =alloc_pages (gfp_flags, order); + page =ion_heap_alloc_pages (buffer, gfp_flags, order);if (!page)return 0 ; arm_dma_ops.sync_single_for_device(NULL,
@@ -85,8 +84,6 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,if (!page)return 0 ; -if (split_pages) -split_page (page, order);return page; }
@@ -153,7 +150,6 @@ static int ion_system_heap_allocate(struct ion_heap *heap,int i =0 ;long size_remaining =PAGE_ALIGN (size);unsigned int max_order = orders[0 ]; -bool split_pages =ion_buffer_fault_user_mappings (buffer);INIT_LIST_HEAD (&pages);while (size_remaining >0 ) {
@@ -170,28 +166,15 @@ static int ion_system_heap_allocate(struct ion_heap *heap,if (!table) goto err; -if (split_pages) - ret =sg_alloc_table (table,PAGE_ALIGN (size) /PAGE_SIZE , -GFP_KERNEL ); -else - ret =sg_alloc_table (table, i,GFP_KERNEL ); - + ret =sg_alloc_table (table, i,GFP_KERNEL );if (ret) goto err1; sg = table->sgl;list_for_each_entry_safe (info, tmp_info, &pages, list) {struct page *page = info->page; -if (split_pages) { -for (i =0 ; i < (1 << info->order); i++) { -sg_set_page (sg, page + i,PAGE_SIZE ,0 ); - sg =sg_next (sg); - } - }else { -sg_set_page (sg, page, (1 << info->order) *PAGE_SIZE , -0 ); - sg =sg_next (sg); - } +sg_set_page (sg, page, (1 << info->order) *PAGE_SIZE ,0 ); + sg =sg_next (sg);list_del (&info->list);kfree (info); }
Signature · 1 line
1.8.3.2
PATCH 72/115 ion: chunk_heap: fix leak in allocated counter
In reply to John StultzFrom: Colin Cross <[email protected]>
buffer->size is controlled by the outer ion layer, don't modify it inside the heap. Instead, compute the rounded up allocated size on demand.
--- drivers/staging/android/ion/ion_chunk_heap.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +9 -6
@@ -47,15 +47,15 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap,struct scatterlist *sg;int ret, i;unsigned long num_chunks; +unsigned long allocated_size;if (ion_buffer_fault_user_mappings (buffer))return -ENOMEM ; - num_chunks =ALIGN (size, chunk_heap->chunk_size) / - chunk_heap->chunk_size; - buffer->size = num_chunks * chunk_heap->chunk_size; + allocated_size =ALIGN (size, chunk_heap->chunk_size); + num_chunks = allocated_size / chunk_heap->chunk_size; -if (buffer->size > chunk_heap->size - chunk_heap->allocated) +if (allocated_size > chunk_heap->size - chunk_heap->allocated)return -ENOMEM ; table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL );
@@ -78,7 +78,7 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap, } buffer->priv_virt = table; - chunk_heap->allocated += buffer->size; + chunk_heap->allocated += allocated_size;return 0 ; err: sg = table->sgl;
@@ -100,6 +100,9 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)struct sg_table *table = buffer->priv_virt;struct scatterlist *sg;int i; +unsigned long allocated_size; + + allocated_size =ALIGN (buffer->size, chunk_heap->chunk_size);ion_heap_buffer_zero (buffer);
@@ -111,7 +114,7 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)),sg_dma_len (sg)); } - chunk_heap->allocated -= buffer->size; + chunk_heap->allocated -= allocated_size;sg_free_table (table);kfree (table); }
Signature · 1 line
1.8.3.2
PATCH 73/115 ion: add free list size to heap debug files
In reply to John StultzFrom: Colin Cross <[email protected]>
--- drivers/staging/android/ion/ion.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/staging/android/ion/ion.c +3 -0
@@ -1286,6 +1286,9 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)seq_printf (s,"%16.s %16u\n" ,"total orphaned" , total_orphaned_size);seq_printf (s,"%16.s %16u\n" ,"total " , total_size); +if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) +seq_printf (s,"%16.s %16u\n" ,"deferred free" , + heap->free_list_size);seq_printf (s,"----------------------------------------------------\n" );if (heap->debug_show)
Signature · 1 line
1.8.3.2
PATCH 74/115 ion: convert map_kernel to return ERR_PTR
In reply to John StultzFrom: Colin Cross <[email protected]>
ion is going to stop accepting NULL as an error value, use ERR_PTR.
--- drivers/staging/android/ion/ion_carveout_heap.c | 7 ++++++- drivers/staging/android/ion/ion_heap.c | 3 +++ 2 files changed, 9 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion_carveout_heap.c +6 -1
@@ -112,13 +112,18 @@ void ion_carveout_heap_unmap_dma(struct ion_heap *heap,void *ion_carveout_heap_map_kernel (struct ion_heap *heap,struct ion_buffer *buffer) { +void *ret;int mtype =MT_MEMORY_NONCACHED ;if (buffer->flags &ION_FLAG_CACHED ) mtype =MT_MEMORY ; -return __arm_ioremap (buffer->priv_phys, buffer->size, + ret =__arm_ioremap (buffer->priv_phys, buffer->size, mtype); +if (ret ==NULL ) +return ERR_PTR (-ENOMEM ); + +return ret; } void ion_carveout_heap_unmap_kernel(struct ion_heap *heap,
drivers/staging/android/ion/ion_heap.c +3 -0
@@ -56,6 +56,9 @@ void *ion_heap_map_kernel(struct ion_heap *heap, vaddr =vmap (pages, npages,VM_MAP , pgprot);vfree (pages); +if (vaddr ==NULL ) +return ERR_PTR (-ENOMEM ); +return vaddr; }
Signature · 1 line
1.8.3.2
PATCH 75/115 ion: remove IS_ERR_OR_NULL
In reply to John StultzFrom: Colin Cross <[email protected]>
IS_ERR_OR_NULL is often part of a bad pattern that can accidentally return 0 on error:
if (IS_ERR_OR_NULL(ptr))
return PTR_ERR(ptr);
It also usually means that the errors of a function are not well defined. Replace all uses in ion.c by ensure that the return type of any function in ion is an ERR_PTR.
Specify that the expected return value from map_kernel or map_dma heap ops is ERR_PTR, and warn if a heap returns NULL.
--- drivers/staging/android/ion/ion.c | 28 +++++++++++++++------------- drivers/staging/android/ion/ion_priv.h | 3 +++ 2 files changed, 18 insertions(+), 13 deletions(-)
drivers/staging/android/ion/ion.c +15 -13
@@ -197,7 +197,9 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap, buffer->size = len; table = heap->ops->map_dma (heap, buffer); -if (IS_ERR_OR_NULL (table)) { +if (WARN_ONCE (table ==NULL ,"heap->ops->map_dma should return ERR_PTR on error" )) + table =ERR_PTR (-EINVAL ); +if (IS_ERR (table)) { heap->ops->free (buffer);kfree (buffer);return ERR_PTR (PTR_ERR (table));
@@ -389,7 +391,7 @@ static struct ion_handle *ion_handle_lookup(struct ion_client *client,if (handle->buffer == buffer)return handle; } -return NULL ; +return ERR_PTR (-EINVAL ); }static bool ion_handle_validate (struct ion_client *client,struct ion_handle *handle)
@@ -459,7 +461,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,if (!((1 << heap->id) & heap_id_mask))continue ; buffer =ion_buffer_create (heap, dev, len, align, flags); -if (!IS_ERR_OR_NULL (buffer)) +if (!IS_ERR (buffer))break ; }up_read (&dev->lock);
@@ -543,7 +545,9 @@ static void *ion_buffer_kmap_get(struct ion_buffer *buffer)return buffer->vaddr; } vaddr = buffer->heap->ops->map_kernel (buffer->heap, buffer); -if (IS_ERR_OR_NULL (vaddr)) +if (WARN_ONCE (vaddr ==NULL ,"heap->ops->map_kernel should return ERR_PTR on error" )) +return ERR_PTR (-EINVAL ); +if (IS_ERR (vaddr))return vaddr; buffer->vaddr = vaddr; buffer->kmap_cnt++;
@@ -560,7 +564,7 @@ static void *ion_handle_kmap_get(struct ion_handle *handle)return buffer->vaddr; } vaddr =ion_buffer_kmap_get (buffer); -if (IS_ERR_OR_NULL (vaddr)) +if (IS_ERR (vaddr))return vaddr; handle->kmap_cnt++;return vaddr;
@@ -954,8 +958,6 @@ static int ion_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, size_t start,mutex_unlock (&buffer->lock);if (IS_ERR (vaddr))return PTR_ERR (vaddr); -if (!vaddr) -return -ENOMEM ;return 0 ; }
@@ -1034,7 +1036,7 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd)struct ion_handle *handle; dmabuf =dma_buf_get (fd); -if (IS_ERR_OR_NULL (dmabuf)) +if (IS_ERR (dmabuf))return ERR_PTR (PTR_ERR (dmabuf));/* if this memory came from ion */
@@ -1049,12 +1051,12 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd)mutex_lock (&client->lock);/* if a handle exists for this buffer just take a reference to it */ handle =ion_handle_lookup (client, buffer); -if (!IS_ERR_OR_NULL (handle)) { +if (!IS_ERR (handle)) {ion_handle_get (handle); goto end; } handle =ion_handle_create (client, buffer); -if (IS_ERR_OR_NULL (handle)) +if (IS_ERR (handle)) goto end;ion_handle_add (client, handle); end:
@@ -1070,7 +1072,7 @@ static int ion_sync_for_device(struct ion_client *client, int fd)struct ion_buffer *buffer; dmabuf =dma_buf_get (fd); -if (IS_ERR_OR_NULL (dmabuf)) +if (IS_ERR (dmabuf))return PTR_ERR (dmabuf);/* if this memory came from ion */
@@ -1204,7 +1206,7 @@ static int ion_open(struct inode *inode, struct file *file)pr_debug ("%s: %d\n" , __func__, __LINE__); client =ion_client_create (dev,"user" ); -if (IS_ERR_OR_NULL (client)) +if (IS_ERR (client))return PTR_ERR (client); file->private_data = client;
@@ -1400,7 +1402,7 @@ struct ion_device *ion_device_create(long (*custom_ioctl) } idev->debug_root =debugfs_create_dir ("ion" ,NULL ); -if (IS_ERR_OR_NULL (idev->debug_root)) +if (!idev->debug_root)pr_err ("ion: failed to create debug files.\n" ); idev->custom_ioctl = custom_ioctl;
drivers/staging/android/ion/ion_priv.h +3 -0
@@ -94,6 +94,9 @@ void ion_buffer_destroy(struct ion_buffer *buffer);* @map_kernel map memory to the kernel * @unmap_kernel unmap memory to the kernel * @map_user map memory to userspace +* +* allocate, phys, and map_user return 0 on success, -errno on error. +* map_dma and map_kernel return pointer on success, ERR_PTR on error. */ struct ion_heap_ops { int (*allocate) (struct ion_heap *heap,
Signature · 1 line
1.8.3.2
PATCH 76/115 ion: replace userspace handle cookies with idr
In reply to John StultzFrom: Colin Cross <[email protected]>
Userspace handles should not leak kernel virtual addresses to userspace. They have to be validated by looking them up in an rbtree anyways, so replace them with an idr and validate them by using idr_find to convert the id number to the struct ion_handle pointer.
--- drivers/staging/android/ion/ion.c | 107 +++++++++++++++++++++++++------------- 1 file changed, 71 insertions(+), 36 deletions(-)
drivers/staging/android/ion/ion.c +71 -36
@@ -34,6 +34,7 @@#include <linux/vmalloc.h> #include <linux/debugfs.h> #include <linux/dma-buf.h> +#include <linux/idr.h> #include "ion.h" #include "ion_priv.h"
@@ -64,6 +65,7 @@ struct ion_device {* @node: node in the tree of all clients * @dev: backpointer to ion device * @handles: an rb tree of all the handles in this client +* @idr: an idr space for allocating handle ids * @lock: lock protecting the tree of handles * @name: used for debugging * @task: used for debugging
@@ -76,6 +78,7 @@ struct ion_client {struct rb_node node;struct ion_device *dev;struct rb_root handles; +struct idr idr;struct mutex lock;const char *name;struct task_struct *task;
@@ -90,7 +93,7 @@ struct ion_client {* @buffer: pointer to the buffer * @node: node in the client's handle rbtree * @kmap_cnt: count of times this client has mapped to kernel -* @dmap_cnt: count of times this client has mapped for dma +* @id: client-unique id allocated by client->idr * * Modifications to node, map_cnt or mapping should be protected by the * lock in the client. Other fields are never changed after initialization.
@@ -101,6 +104,7 @@ struct ion_handle {struct ion_buffer *buffer;struct rb_node node;unsigned int kmap_cnt; +int id; };bool ion_buffer_fault_user_mappings (struct ion_buffer *buffer)
@@ -356,6 +360,7 @@ static void ion_handle_destroy(struct kref *kref)ion_handle_kmap_put (handle);mutex_unlock (&buffer->lock); +idr_remove (&client->idr, handle->id);if (!RB_EMPTY_NODE (&handle->node))rb_erase (&handle->node, &client->handles);
@@ -387,36 +392,42 @@ static struct ion_handle *ion_handle_lookup(struct ion_client *client,for (n =rb_first (&client->handles); n; n =rb_next (n)) { struct ion_handle *handle = rb_entry(n, struct ion_handle, - node); + node);if (handle->buffer == buffer)return handle; }return ERR_PTR (-EINVAL ); } -static bool ion_handle_validate (struct ion_client *client,struct ion_handle *handle) +static struct ion_handle *ion_uhandle_get (struct ion_client *client,int id) { -struct rb_node *n = client->handles.rb_node; +return idr_find (&client->idr, id); +} -while (n) { - struct ion_handle *handle_node = rb_entry(n, struct ion_handle, - node); -if (handle < handle_node) - n = n->rb_left; -else if (handle > handle_node) - n = n->rb_right; -else -return true; - } -return false; +static bool ion_handle_validate (struct ion_client *client,struct ion_handle *handle) +{ +return (ion_uhandle_get (client, handle->id) == handle); } -static void ion_handle_add (struct ion_client *client,struct ion_handle *handle) +static int ion_handle_add (struct ion_client *client,struct ion_handle *handle) { +int rc;struct rb_node **p = &client->handles.rb_node;struct rb_node *parent =NULL ;struct ion_handle *entry; +do { +int id; + rc =idr_pre_get (&client->idr,GFP_KERNEL ); +if (!rc) +return -ENOMEM ; + rc =idr_get_new (&client->idr, handle, &id); + handle->id = id; + }while (rc == -EAGAIN ); + +if (rc <0 ) +return rc; +while (*p) { parent = *p; entry = rb_entry(parent, struct ion_handle, node);
@@ -431,6 +442,8 @@ static void ion_handle_add(struct ion_client *client, struct ion_handle *handle)rb_link_node (&handle->node, parent, p);rb_insert_color (&handle->node, &client->handles); + +return 0 ; } struct ion_handle *ion_alloc(struct ion_client *client, size_t len,
@@ -441,6 +454,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,struct ion_device *dev = client->dev;struct ion_buffer *buffer =NULL ;struct ion_heap *heap; +int ret;pr_debug ("%s: len %d align %d heap_id_mask %u flags %x\n" , __func__, len, align, heap_id_mask, flags);
@@ -480,12 +494,16 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,*/ ion_buffer_put (buffer); -if (!IS_ERR (handle)) { -mutex_lock (&client->lock); -ion_handle_add (client, handle); -mutex_unlock (&client->lock); - } +if (IS_ERR (handle)) +return handle; +mutex_lock (&client->lock); + ret =ion_handle_add (client, handle); +if (ret) { +ion_handle_put (handle); + handle =ERR_PTR (ret); + } +mutex_unlock (&client->lock);return handle; }
@@ -705,6 +723,7 @@ struct ion_client *ion_client_create(struct ion_device *dev, client->dev = dev; client->handles =RB_ROOT ; +idr_init (&client->idr);mutex_init (&client->lock); client->name = name; client->task = task;
@@ -745,6 +764,10 @@ void ion_client_destroy(struct ion_client *client) node);ion_handle_destroy (&handle->ref); } + +idr_remove_all (&client->idr); +idr_destroy (&client->idr); +down_write (&dev->lock);if (client->task)put_task_struct (client->task);
@@ -1034,6 +1057,7 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd)struct dma_buf *dmabuf;struct ion_buffer *buffer;struct ion_handle *handle; +int ret; dmabuf =dma_buf_get (fd);if (IS_ERR (dmabuf))
@@ -1058,7 +1082,11 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd) handle =ion_handle_create (client, buffer);if (IS_ERR (handle)) goto end; -ion_handle_add (client, handle); + ret =ion_handle_add (client, handle); +if (ret) { +ion_handle_put (handle); + handle =ERR_PTR (ret); + } end:mutex_unlock (&client->lock);dma_buf_put (dmabuf);
@@ -1098,17 +1126,20 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)case ION_IOC_ALLOC : {struct ion_allocation_data data; +struct ion_handle *handle;if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; - data.handle =ion_alloc (client, data.len, data.align, + handle =ion_alloc (client, data.len, data.align, data.heap_id_mask, data.flags); -if (IS_ERR (data.handle)) -return PTR_ERR (data.handle); +if (IS_ERR (handle)) +return PTR_ERR (handle); + + data.handle = (struct ion_handle *)handle->id;if (copy_to_user ((void __user *)arg, &data,sizeof (data))) { -ion_free (client, data.handle); +ion_free (client, handle);return -EFAULT ; }break ;
@@ -1116,27 +1147,29 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)case ION_IOC_FREE : {struct ion_handle_data data; -bool valid; +struct ion_handle *handle;if (copy_from_user (&data, (void __user *)arg,sizeof (struct ion_handle_data )))return -EFAULT ;mutex_lock (&client->lock); - valid =ion_handle_validate (client, data.handle); + handle =ion_uhandle_get (client, (int )data.handle);mutex_unlock (&client->lock); -if (!valid) +if (!handle)return -EINVAL ; -ion_free (client, data.handle); +ion_free (client, handle);break ; }case ION_IOC_SHARE :case ION_IOC_MAP : {struct ion_fd_data data; +struct ion_handle *handle;if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; - data.fd =ion_share_dma_buf_fd (client, data.handle); + handle =ion_uhandle_get (client, (int )data.handle); + data.fd =ion_share_dma_buf_fd (client, handle);if (copy_to_user ((void __user *)arg, &data,sizeof (data)))return -EFAULT ;if (data.fd <0 )
@@ -1146,15 +1179,17 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)case ION_IOC_IMPORT : {struct ion_fd_data data; +struct ion_handle *handle;int ret =0 ;if (copy_from_user (&data, (void __user *)arg,sizeof (struct ion_fd_data )))return -EFAULT ; - data.handle =ion_import_dma_buf (client, data.fd); -if (IS_ERR (data.handle)) { - ret =PTR_ERR (data.handle); - data.handle =NULL ; - } + handle =ion_import_dma_buf (client, data.fd); +if (IS_ERR (handle)) + ret =PTR_ERR (handle); +else + data.handle = (struct ion_handle *)handle->id; +if (copy_to_user ((void __user *)arg, &data,sizeof (struct ion_fd_data )))return -EFAULT ;
Signature · 1 line
1.8.3.2
PATCH 77/115 ion: index client->handles rbtree by buffer
In reply to John StultzFrom: Colin Cross <[email protected]>
The only remaining users of the client->handles rbtree are iterating through it like a list. Keep the rbtree, but change its index to be the buffer address instead of the handle address, which makes ion_handle_lookup a fast rbtree search.
--- drivers/staging/android/ion/ion.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-)
drivers/staging/android/ion/ion.c +12 -9
@@ -388,13 +388,16 @@ static int ion_handle_put(struct ion_handle *handle)static struct ion_handle *ion_handle_lookup (struct ion_client *client,struct ion_buffer *buffer) { -struct rb_node *n; - -for (n =rb_first (&client->handles); n; n =rb_next (n)) { - struct ion_handle *handle = rb_entry(n, struct ion_handle, - node); -if (handle->buffer == buffer) -return handle; +struct rb_node *n = client->handles.rb_node; + +while (n) { + struct ion_handle *entry = rb_entry(n, struct ion_handle, node); +if (buffer < entry->buffer) + n = n->rb_left; +else if (buffer > entry->buffer) + n = n->rb_right; +else +return entry; }return ERR_PTR (-EINVAL ); }
@@ -432,9 +435,9 @@ static int ion_handle_add(struct ion_client *client, struct ion_handle *handle) parent = *p; entry = rb_entry(parent, struct ion_handle, node); -if (handle < entry) +if (handle->buffer < entry->buffer) p = &(*p)->rb_left; -else if (handle > entry) +else if (handle->buffer > entry->buffer) p = &(*p)->rb_right;else WARN (1 ,"%s: buffer already found." , __func__);
Signature · 1 line
1.8.3.2
PATCH 78/115 ion: don't use id 0 for handle cookie
In reply to John StultzFrom: Colin Cross <[email protected]>
ion userspace clients think that the cookie is a pointer, so they use NULL to check if the handle has been initialized. Set the first id number to 1.
--- drivers/staging/android/ion/ion.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion.c +1 -1
@@ -424,7 +424,7 @@ static int ion_handle_add(struct ion_client *client, struct ion_handle *handle) rc =idr_pre_get (&client->idr,GFP_KERNEL );if (!rc)return -ENOMEM ; - rc =idr_get_new (&client->idr, handle, &id); + rc =idr_get_new_above (&client->idr, handle,1 , &id); handle->id = id; }while (rc == -EAGAIN );
Signature · 1 line
1.8.3.2
PATCH 79/115 ion: add new ion_user_handle_t type for the user-space token
In reply to John StultzFrom: Rom Lemarchand <[email protected]>
Declare new ion_user_handle_t type to contain the token returned to user-space. This allows a 2-step migration of the user-space code to a new kernel header first, then will allow us to change the definition of the ion_user_handle_type_t to int without breaking the API.
--- drivers/staging/android/ion/ion.c | 4 ++-- drivers/staging/android/ion/ion.h | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/staging/android/ion/ion.c +2 -2
@@ -1139,7 +1139,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (IS_ERR (handle))return PTR_ERR (handle); - data.handle = (struct ion_handle *)handle->id; + data.handle = (ion_user_handle_t )handle->id;if (copy_to_user ((void __user *)arg, &data,sizeof (data))) {ion_free (client, handle);
@@ -1191,7 +1191,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (IS_ERR (handle)) ret =PTR_ERR (handle);else - data.handle = (struct ion_handle *)handle->id; + data.handle = (ion_user_handle_t )handle->id;if (copy_to_user ((void __user *)arg, &data,sizeof (struct ion_fd_data )))
drivers/staging/android/ion/ion.h +5 -3
@@ -20,6 +20,8 @@#include <linux/types.h> struct ion_handle ; +typedef struct ion_handle *ion_user_handle_t ; +/** * enum ion_heap_types - list of all possible types of heaps * @ION_HEAP_TYPE_SYSTEM: memory allocated via vmalloc
@@ -268,7 +270,7 @@ struct ion_allocation_data {size_t align;unsigned int heap_id_mask;unsigned int flags; -struct ion_handle *handle; +ion_user_handle_t handle; };/**
@@ -282,7 +284,7 @@ struct ion_allocation_data {* provides the file descriptor and the kernel returns the handle. */ struct ion_fd_data { -struct ion_handle *handle; +ion_user_handle_t handle;int fd; };
@@ -291,7 +293,7 @@ struct ion_fd_data {* @handle: a handle */ struct ion_handle_data { -struct ion_handle *handle; +ion_user_handle_t handle; };/**
Signature · 1 line
1.8.3.2
PATCH 80/115 ion: change ion_user_handle_t definition to int
In reply to John StultzFrom: Rom Lemarchand <[email protected]>
Turn ion_user_handle_t to int. This change reflects the underlying type returned by the ion driver.
--- drivers/staging/android/ion/ion.c | 8 ++++---- drivers/staging/android/ion/ion.h | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-)
drivers/staging/android/ion/ion.c +4 -4
@@ -1139,7 +1139,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (IS_ERR (handle))return PTR_ERR (handle); - data.handle = (ion_user_handle_t )handle->id; + data.handle = handle->id;if (copy_to_user ((void __user *)arg, &data,sizeof (data))) {ion_free (client, handle);
@@ -1156,7 +1156,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) sizeof(struct ion_handle_data)))return -EFAULT ;mutex_lock (&client->lock); - handle =ion_uhandle_get (client, (int )data.handle); + handle =ion_uhandle_get (client, data.handle);mutex_unlock (&client->lock);if (!handle)return -EINVAL ;
@@ -1171,7 +1171,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; - handle =ion_uhandle_get (client, (int )data.handle); + handle =ion_uhandle_get (client, data.handle); data.fd =ion_share_dma_buf_fd (client, handle);if (copy_to_user ((void __user *)arg, &data,sizeof (data)))return -EFAULT ;
@@ -1191,7 +1191,7 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (IS_ERR (handle)) ret =PTR_ERR (handle);else - data.handle = (ion_user_handle_t )handle->id; + data.handle = handle->id;if (copy_to_user ((void __user *)arg, &data,sizeof (struct ion_fd_data )))
drivers/staging/android/ion/ion.h +2 -2
@@ -19,8 +19,7 @@#include <linux/types.h> -struct ion_handle ; -typedef struct ion_handle *ion_user_handle_t ; +typedef int ion_user_handle_t ;/** * enum ion_heap_types - list of all possible types of heaps
@@ -65,6 +64,7 @@ enum ion_heap_type {caches must be managed manually */ #ifdef __KERNEL__ +struct ion_handle ;struct ion_device ;struct ion_heap ;struct ion_mapper ;
Signature · 1 line
1.8.3.2
PATCH 81/115 ion: add compat_ioctl
In reply to John StultzFrom: Rom Lemarchand <[email protected]>
Add a compat_ioctl to the ion driver
--- drivers/staging/android/ion/Makefile | 3 + drivers/staging/android/ion/compat_ion.c | 172 +++++++++++++++++++++++++++++++ drivers/staging/android/ion/compat_ion.h | 30 ++++++ drivers/staging/android/ion/ion.c | 2 + 4 files changed, 207 insertions(+) create mode 100644 drivers/staging/android/ion/compat_ion.c create mode 100644 drivers/staging/android/ion/compat_ion.h
drivers/staging/android/ion/Makefile +3 -0
@@ -1,3 +1,6 @@ obj-$(CONFIG_ION) += ion.o ion_heap.o ion_page_pool.o ion_system_heap.o \ ion_carveout_heap.o ion_chunk_heap.o ion_cma_heap.o +ifdef CONFIG_COMPAT +obj-$(CONFIG_ION) += compat_ion.o +endif obj-$(CONFIG_ION_TEGRA) += tegra/
drivers/staging/android/ion/compat_ion.c new file +172 -0
172 lines not on this page · raw message
drivers/staging/android/ion/compat_ion.h new file +30 -0
@@ -0,0 +1,30 @@ +/* + +* drivers/staging/android/ion/compat_ion.h +* +* Copyright (C) 2013 Google, Inc. +* +* This software is licensed under the terms of the GNU General Public +* License version 2, as published by the Free Software Foundation, and +* may be copied, distributed, and modified under those terms. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +*/ + +#ifndef _LINUX_COMPAT_ION_H +#define _LINUX_COMPAT_ION_H + +#if IS_ENABLED (CONFIG_COMPAT ) + +long compat_ion_ioctl (struct file *filp,unsigned int cmd,unsigned long arg); + +#else + +#define compat_ion_ioctl NULL + +#endif /* CONFIG_COMPAT */ +#endif /* _LINUX_COMPAT_ION_H */
drivers/staging/android/ion/ion.c +2 -0
@@ -38,6 +38,7 @@#include "ion.h" #include "ion_priv.h" +#include "compat_ion.h" /** * struct ion_device - the metadata of the ion device node
@@ -1256,6 +1257,7 @@ static const struct file_operations ion_fops = { .open = ion_open, .release = ion_release, .unlocked_ioctl = ion_ioctl, + .compat_ioctl = compat_ion_ioctl, }; static size_t ion_debug_heap_total(struct ion_client *client,
Signature · 1 line
1.8.3.2
PATCH 82/115 gpu: ion: delete ion_system_mapper.c
In reply to John StultzFrom: Colin Cross <[email protected]>
The mapper abstraction layer was removed before the initial ion commit, but a stray ion_system_mapper.c file was left in. Delete it.
--- drivers/staging/android/ion/ion_system_mapper.c | 114 ------------------------ 1 file changed, 114 deletions(-) delete mode 100644 drivers/staging/android/ion/ion_system_mapper.c
drivers/staging/android/ion/ion_system_mapper.c deleted +0 -114
@@ -1,114 +0,0 @@ -/* -* drivers/staging/android/ion/ion_system_mapper.c -* -* Copyright (C) 2011 Google, Inc. -* -* This software is licensed under the terms of the GNU General Public -* License version 2, as published by the Free Software Foundation, and -* may be copied, distributed, and modified under those terms. -* -* This program is distributed in the hope that it will be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -*/ - -#include <linux/err.h> -#include <linux/memory.h> -#include <linux/mm.h> -#include <linux/slab.h> -#include <linux/vmalloc.h> -#include "ion.h" -#include "ion_priv.h" -/* -* This mapper is valid for any heap that allocates memory that already has -* a kernel mapping, this includes vmalloc'd memory, kmalloc'd memory, -* pages obtained via io_remap, etc. -*/ -static void *ion_kernel_mapper_map (struct ion_mapper *mapper, -struct ion_buffer *buffer, -struct ion_mapping **mapping) -{ -if (!((1 << buffer->heap->type) & mapper->heap_mask)) { -pr_err ("%s: attempting to map an unsupported heap\n" , __func__); -return ERR_PTR (-EINVAL ); - } -/* XXX REVISIT ME!!! */ - *((unsigned long *)mapping) = (unsigned long )buffer->priv; -return buffer->priv; -} - -static void ion_kernel_mapper_unmap (struct ion_mapper *mapper, -struct ion_buffer *buffer, -struct ion_mapping *mapping) -{ -if (!((1 << buffer->heap->type) & mapper->heap_mask)) -pr_err ("%s: attempting to unmap an unsupported heap\n" , - __func__); -} - -static void *ion_kernel_mapper_map_kernel (struct ion_mapper *mapper, -struct ion_buffer *buffer, -struct ion_mapping *mapping) -{ -if (!((1 << buffer->heap->type) & mapper->heap_mask)) { -pr_err ("%s: attempting to unmap an unsupported heap\n" , - __func__); -return ERR_PTR (-EINVAL ); - } -return buffer->priv; -} - -static int ion_kernel_mapper_map_user (struct ion_mapper *mapper, -struct ion_buffer *buffer, -struct vm_area_struct *vma, -struct ion_mapping *mapping) -{ -int ret; - -switch (buffer->heap->type) { -case ION_HEAP_KMALLOC : - { -unsigned long pfn =__phys_to_pfn (virt_to_phys (buffer->priv)); - ret =remap_pfn_range (vma, vma->vm_start, pfn + vma->vm_pgoff, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); -break ; - } -case ION_HEAP_VMALLOC : - ret =remap_vmalloc_range (vma, buffer->priv, vma->vm_pgoff); -break ; -default : -pr_err ("%s: attempting to map unsupported heap to userspace\n" , - __func__); -return -EINVAL ; - } - -return ret; -} - -static struct ion_mapper_ops ops = { - .map = ion_kernel_mapper_map, - .map_kernel = ion_kernel_mapper_map_kernel, - .map_user = ion_kernel_mapper_map_user, - .unmap = ion_kernel_mapper_unmap, -}; - -struct ion_mapper *ion_system_mapper_create (void ) -{ -struct ion_mapper *mapper; - mapper =kzalloc (sizeof (struct ion_mapper ),GFP_KERNEL ); -if (!mapper) -return ERR_PTR (-ENOMEM ); - mapper->type =ION_SYSTEM_MAPPER ; - mapper->ops = &ops; - mapper->heap_mask = (1 <<ION_HEAP_VMALLOC ) | (1 <<ION_HEAP_KMALLOC ); -return mapper; -} - -void ion_system_mapper_destroy (struct ion_mapper *mapper) -{ -kfree (mapper); -} -
Signature · 1 line
1.8.3.2
PATCH 83/115 ion: move userspace api into uapi/ion.h
In reply to John StultzFrom: Colin Cross <[email protected]>
Split the userspace api out of drivers/staging/android/ion/ion.h into drivers/staging/android/uapi/ion.h
--- drivers/staging/android/ion/compat_ion.c | 2 +- drivers/staging/android/ion/ion.h | 176 +-------------------------- drivers/staging/android/uapi/ion.h | 196 +++++++++++++++++++++++++++++++ 3 files changed, 198 insertions(+), 176 deletions(-) create mode 100644 drivers/staging/android/uapi/ion.h
drivers/staging/android/ion/compat_ion.c +1 -1
@@ -21,7 +21,7 @@#include "ion.h" #include "compat_ion.h" -/* See drivers/staging/android/ion/ion.h for the definition of these structs */ +/* See drivers/staging/android/uapi/ion.h for the definition of these structs */ struct compat_ion_allocation_data {compat_size_t len;compat_size_t align;
drivers/staging/android/ion/ion.h +1 -175
187 lines not on this page · raw message
drivers/staging/android/uapi/ion.h new file +196 -0
196 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 84/115 ion: Fix compat support to use proper compat ioctl numbers
In reply to John StultzThe compat support added to ion didn't provide compat ioctl numbers (who's value depends on the compat structure size). So 32bit applications would get an error when trying to make ioctl calls.
This patch adds the needed COMPAT_ macros and uses them in the compat_ion_ioctl, translating them to their non-compat cmd when calling the normal ioctl call.
--- drivers/staging/android/ion/compat_ion.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
drivers/staging/android/ion/compat_ion.c +12 -7
@@ -35,6 +35,12 @@ struct compat_ion_custom_data {compat_ulong_t arg; }; +#define COMPAT_ION_IOC_ALLOC _IOWR(ION_IOC_MAGIC, 0, \ + struct compat_ion_allocation_data) +#define COMPAT_ION_IOC_FREE _IOWR(ION_IOC_MAGIC, 1, struct ion_handle_data) +#define COMPAT_ION_IOC_CUSTOM _IOWR(ION_IOC_MAGIC, 6, \ + struct compat_ion_custom_data) +static int compat_get_ion_allocation_data (struct compat_ion_allocation_data __user *data32,struct ion_allocation_data __user *data)
@@ -105,7 +111,7 @@ long compat_ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)return -ENOTTY ;switch (cmd) { -case ION_IOC_ALLOC : +case COMPAT_ION_IOC_ALLOC : {struct compat_ion_allocation_data __user *data32;struct ion_allocation_data __user *data;
@@ -119,13 +125,12 @@ long compat_ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) err =compat_get_ion_allocation_data (data32, data);if (err)return err; - - ret = filp->f_op->unlocked_ioctl (filp, cmd, + ret = filp->f_op->unlocked_ioctl (filp,ION_IOC_ALLOC , (unsigned long )data); err =compat_put_ion_allocation_data (data32, data);return ret ? ret : err; } -case ION_IOC_FREE : +case COMPAT_ION_IOC_FREE : {struct compat_ion_allocation_data __user *data32;struct ion_allocation_data __user *data;
@@ -140,10 +145,10 @@ long compat_ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (err)return err; -return filp->f_op->unlocked_ioctl (filp, cmd, +return filp->f_op->unlocked_ioctl (filp,ION_IOC_FREE , (unsigned long )data); } -case ION_IOC_CUSTOM : { +case COMPAT_ION_IOC_CUSTOM : {struct compat_ion_custom_data __user *data32;struct ion_custom_data __user *data;int err;
@@ -157,7 +162,7 @@ long compat_ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (err)return err; -return filp->f_op->unlocked_ioctl (filp, cmd, +return filp->f_op->unlocked_ioctl (filp,ION_IOC_CUSTOM , (unsigned long )data); }case ION_IOC_SHARE :
Signature · 1 line
1.8.3.2
PATCH 85/115 ion: hold reference to handle after ion_uhandle_get
In reply to John StultzFrom: Colin Cross <[email protected]>
commit 1262ab1846cf76f7549c66ef709120dbfbe6d49f (ion: replace userspace handle cookies with idr) broke the locking in ion. The ION_IOC_FREE and ION_IOC_MAP ioctls were relying on ion_handle_validate to detect the case where a call raced with another ION_IOC_FREE which may have freed the struct ion_handle.
Rename ion_uhandle_get to ion_handle_get_by_id, and have it take the client lock and return with an extra reference to the handle. Make each caller put its reference once it is done with the handle.
Also modify users of ion_handle_validate to continue to hold the client lock after calling ion_handle_validate until they are done with the handle, and warn if ion_handle_validate is called without the client lock held.
--- drivers/staging/android/ion/ion.c | 56 ++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 15 deletions(-)
drivers/staging/android/ion/ion.c +41 -15
@@ -383,7 +383,14 @@ static void ion_handle_get(struct ion_handle *handle)static int ion_handle_put (struct ion_handle *handle) { -return kref_put (&handle->ref, ion_handle_destroy); +struct ion_client *client = handle->client; +int ret; + +mutex_lock (&client->lock); + ret =kref_put (&handle->ref, ion_handle_destroy); +mutex_unlock (&client->lock); + +return ret; } static struct ion_handle *ion_handle_lookup(struct ion_client *client,
@@ -403,14 +410,24 @@ static struct ion_handle *ion_handle_lookup(struct ion_client *client,return ERR_PTR (-EINVAL ); } -static struct ion_handle *ion_uhandle_get (struct ion_client *client,int id) +static struct ion_handle *ion_handle_get_by_id (struct ion_client *client, +int id) { -return idr_find (&client->idr, id); +struct ion_handle *handle; + +mutex_lock (&client->lock); + handle =idr_find (&client->idr, id); +if (handle) +ion_handle_get (handle); +mutex_unlock (&client->lock); + +return handle ? handle :ERR_PTR (-EINVAL ); }static bool ion_handle_validate (struct ion_client *client,struct ion_handle *handle) { -return (ion_uhandle_get (client, handle->id) == handle); +WARN_ON (!mutex_is_locked (&client->lock)); +return (idr_find (&client->idr, handle->id) == handle); }static int ion_handle_add (struct ion_client *client,struct ion_handle *handle)
@@ -503,11 +520,11 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,mutex_lock (&client->lock); ret =ion_handle_add (client, handle); +mutex_unlock (&client->lock);if (ret) {ion_handle_put (handle); handle =ERR_PTR (ret); } -mutex_unlock (&client->lock);return handle; }
@@ -527,8 +544,8 @@ void ion_free(struct ion_client *client, struct ion_handle *handle)mutex_unlock (&client->lock);return ; } -ion_handle_put (handle);mutex_unlock (&client->lock); +ion_handle_put (handle); }EXPORT_SYMBOL (ion_free);
@@ -1021,14 +1038,15 @@ struct dma_buf *ion_share_dma_buf(struct ion_client *client,mutex_lock (&client->lock); valid_handle =ion_handle_validate (client, handle); -mutex_unlock (&client->lock);if (!valid_handle) {WARN (1 ,"%s: invalid handle passed to share.\n" , __func__); +mutex_unlock (&client->lock);return ERR_PTR (-EINVAL ); } - buffer = handle->buffer;ion_buffer_get (buffer); +mutex_unlock (&client->lock); + dmabuf =dma_buf_export (buffer, &dma_buf_ops, buffer->size,O_RDWR );if (IS_ERR (dmabuf)) {ion_buffer_put (buffer);
@@ -1081,18 +1099,24 @@ struct ion_handle *ion_import_dma_buf(struct ion_client *client, int fd) handle =ion_handle_lookup (client, buffer);if (!IS_ERR (handle)) {ion_handle_get (handle); +mutex_unlock (&client->lock); goto end; } +mutex_unlock (&client->lock); + handle =ion_handle_create (client, buffer);if (IS_ERR (handle)) goto end; + +mutex_lock (&client->lock); ret =ion_handle_add (client, handle); +mutex_unlock (&client->lock);if (ret) {ion_handle_put (handle); handle =ERR_PTR (ret); } + end: -mutex_unlock (&client->lock);dma_buf_put (dmabuf);return handle; }
@@ -1156,12 +1180,11 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (struct ion_handle_data )))return -EFAULT ; -mutex_lock (&client->lock); - handle =ion_uhandle_get (client, data.handle); -mutex_unlock (&client->lock); -if (!handle) -return -EINVAL ; + handle =ion_handle_get_by_id (client, data.handle); +if (IS_ERR (handle)) +return PTR_ERR (handle);ion_free (client, handle); +ion_handle_put (handle);break ; }case ION_IOC_SHARE :
@@ -1172,8 +1195,11 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)if (copy_from_user (&data, (void __user *)arg,sizeof (data)))return -EFAULT ; - handle =ion_uhandle_get (client, data.handle); + handle =ion_handle_get_by_id (client, data.handle); +if (IS_ERR (handle)) +return PTR_ERR (handle); data.fd =ion_share_dma_buf_fd (client, handle); +ion_handle_put (handle);if (copy_to_user ((void __user *)arg, &data,sizeof (data)))return -EFAULT ;if (data.fd <0 )
Signature · 1 line
1.8.3.2
PATCH 86/115 ion: fix crash when alloc len is -1
In reply to John StultzFrom: Colin Cross <[email protected]>
If userspace passes a length between -4095 and -1 to allocate it will pass the len != 0 check, but when len is page aligned it will be 0. Check len after page aligning.
Drop the warning as well, userspace shouldn't be able to trigger a warning in the kernel.
--- drivers/staging/android/ion/ion.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion.c +3 -3
@@ -485,11 +485,11 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,* request of the caller allocate from it. Repeat until allocate has * succeeded or all heaps have been tried */ -if (WARN_ON (!len)) -return ERR_PTR (-EINVAL ); - len =PAGE_ALIGN (len); +if (!len) +return ERR_PTR (-EINVAL ); +down_read (&dev->lock);plist_for_each_entry (heap, &dev->heaps, node) {/* if the caller didn't specify this heap id */
Signature · 1 line
1.8.3.2
PATCH 87/115 ion: fix dma APIs
In reply to John StultzFrom: Colin Cross <[email protected]>
__dma_page_cpu_to_dev is a private ARM api that is not available on 3.10 and was never available on other architectures. We can get the same behavior by calling dma_sync_sg_for_device with a scatterlist containing a single page. It's still not quite a kosher use of the dma apis, we still conflate physical addresses with bus addresses, but it should at least compile on all platforms, and work on any platform that doesn't have a physical to bus address translation.
--- drivers/staging/android/ion/ion.c | 20 +++++++++++++++++++- drivers/staging/android/ion/ion_chunk_heap.c | 15 +++++++-------- drivers/staging/android/ion/ion_page_pool.c | 9 ++------- drivers/staging/android/ion/ion_priv.h | 12 ++++++++++++ drivers/staging/android/ion/ion_system_heap.c | 5 ++--- 5 files changed, 42 insertions(+), 19 deletions(-)
drivers/staging/android/ion/ion.c +19 -1
@@ -840,6 +840,22 @@ static void ion_unmap_dma_buf(struct dma_buf_attachment *attachment, { } +void ion_pages_sync_for_device (struct device *dev,struct page *page, +size_t size,enum dma_data_direction dir) +{ +struct scatterlist sg; + +sg_init_table (&sg,1 ); +sg_set_page (&sg, page, size,0 ); +/* +* This is not correct - sg_dma_address needs a dma_addr_t that is valid +* for the the targeted device, but this works on the currently targeted +* hardware. +*/ +sg_dma_address (&sg) =page_to_phys (page); +dma_sync_sg_for_device (dev, &sg,1 , dir); +} +struct ion_vma_list {struct list_head list;struct vm_area_struct *vma;
@@ -864,7 +880,9 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,struct page *page = buffer->pages[i];if (ion_buffer_page_is_dirty (page)) -__dma_page_cpu_to_dev (page,0 ,PAGE_SIZE , dir); +ion_pages_sync_for_device (dev,ion_buffer_page (page), +PAGE_SIZE , dir); +ion_buffer_page_clean (buffer->pages + i); }list_for_each_entry (vma_list, &buffer->vmas, list) {
drivers/staging/android/ion/ion_chunk_heap.c +7 -8
@@ -106,11 +106,11 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)ion_heap_buffer_zero (buffer); +if (ion_buffer_cached (buffer)) +dma_sync_sg_for_device (NULL , table->sgl, table->nents, +DMA_BIDIRECTIONAL ); +for_each_sg (table->sgl, sg, table->nents, i) { -if (ion_buffer_cached (buffer)) - arm_dma_ops.sync_single_for_device (NULL , -pfn_to_dma (NULL ,page_to_pfn (sg_page (sg))), -sg_dma_len (sg),DMA_BIDIRECTIONAL );gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)),sg_dma_len (sg)); }
@@ -148,7 +148,6 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data)pgprot_t pgprot =pgprot_writecombine (PAGE_KERNEL );int i, ret; - chunk_heap =kzalloc (sizeof (struct ion_chunk_heap ),GFP_KERNEL );if (!chunk_heap)return ERR_PTR (-ENOMEM );
@@ -181,9 +180,9 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) }free_vm_area (vm_struct); - arm_dma_ops.sync_single_for_device (NULL , -pfn_to_dma (NULL ,page_to_pfn (phys_to_page (heap_data->base))), - heap_data->size,DMA_BIDIRECTIONAL ); +ion_pages_sync_for_device (NULL ,pfn_to_page (PFN_DOWN (heap_data->base)), + heap_data->size,DMA_BIDIRECTIONAL ); +gen_pool_add (chunk_heap->pool, chunk_heap->base, heap_data->size,-1 ); chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ;
drivers/staging/android/ion/ion_page_pool.c +2 -7
@@ -34,13 +34,8 @@ static void *ion_page_pool_alloc_pages(struct ion_page_pool *pool)if (!page)return NULL ; -/* this is only being used to flush the page for dma, -this api is not really suitable for calling from a driver -but no better way to flush a page for dma exist at this time */ - arm_dma_ops.sync_single_for_device (NULL , -pfn_to_dma (NULL ,page_to_pfn (page)), -PAGE_SIZE << pool->order, -DMA_BIDIRECTIONAL ); +ion_pages_sync_for_device (NULL , page,PAGE_SIZE << pool->order, +DMA_BIDIRECTIONAL );return page; }
drivers/staging/android/ion/ion_priv.h +12 -0
@@ -17,6 +17,7 @@#ifndef _ION_PRIV_H#define _ION_PRIV_H +#include <linux/dma-direction.h> #include <linux/kref.h> #include <linux/mm_types.h> #include <linux/mutex.h>
@@ -357,4 +358,15 @@ void ion_page_pool_free(struct ion_page_pool *, struct page *);int ion_page_pool_shrink (struct ion_page_pool *pool,gfp_t gfp_mask,int nr_to_scan); +/** +* ion_pages_sync_for_device - cache flush pages for use with the specified +* device +* @dev: the device the pages will be used with +* @page: the first page to be flushed +* @size: size in bytes of region to be flushed +* @dir: direction of dma transfer +*/ +void ion_pages_sync_for_device (struct device *dev,struct page *page, +size_t size,enum dma_data_direction dir); +#endif /* _ION_PRIV_H */
drivers/staging/android/ion/ion_system_heap.c +2 -3
@@ -77,9 +77,8 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap, page =ion_heap_alloc_pages (buffer, gfp_flags, order);if (!page)return 0 ; - arm_dma_ops.sync_single_for_device (NULL , -pfn_to_dma (NULL ,page_to_pfn (page)), -PAGE_SIZE << order,DMA_BIDIRECTIONAL ); +ion_pages_sync_for_device (NULL , page,PAGE_SIZE << order, +DMA_BIDIRECTIONAL ); }if (!page)return 0 ;
Signature · 1 line
1.8.3.2
PATCH 88/115 ion: convert sg_dma_len(sg) to sg->length
In reply to John StultzFrom: Colin Cross <[email protected]>
ion is always dealing with the allocation and not the mapping, so it should always be using sg->length and not sg->dma_length.
--- drivers/staging/android/ion/ion.c | 2 +- drivers/staging/android/ion/ion_chunk_heap.c | 4 ++-- drivers/staging/android/ion/ion_heap.c | 12 ++++++------ drivers/staging/android/ion/ion_system_heap.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)
drivers/staging/android/ion/ion.c +1 -1
@@ -224,7 +224,7 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,for_each_sg (table->sgl, sg, table->nents, i) {struct page *page =sg_page (sg); -for (j =0 ; j <sg_dma_len (sg) /PAGE_SIZE ; j++) +for (j =0 ; j < sg->length /PAGE_SIZE ; j++) buffer->pages[k++] = page++; }
drivers/staging/android/ion/ion_chunk_heap.c +2 -2
@@ -84,7 +84,7 @@ err: sg = table->sgl;for (i -=1 ; i >=0 ; i--) {gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)), -sg_dma_len (sg)); + sg->length); sg =sg_next (sg); }sg_free_table (table);
@@ -112,7 +112,7 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)for_each_sg (table->sgl, sg, table->nents, i) {gen_pool_free (chunk_heap->pool,page_to_phys (sg_page (sg)), -sg_dma_len (sg)); + sg->length); } chunk_heap->allocated -= allocated_size;sg_free_table (table);
drivers/staging/android/ion/ion_heap.c +6 -6
@@ -46,7 +46,7 @@ void *ion_heap_map_kernel(struct ion_heap *heap, pgprot =pgprot_writecombine (PAGE_KERNEL );for_each_sg (table->sgl, sg, table->nents, i) { -int npages_this_entry =PAGE_ALIGN (sg_dma_len (sg)) /PAGE_SIZE ; +int npages_this_entry =PAGE_ALIGN (sg->length) /PAGE_SIZE ;struct page *page =sg_page (sg);BUG_ON (i >= npages);for (j =0 ; j < npages_this_entry; j++) {
@@ -80,14 +80,14 @@ int ion_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer,for_each_sg (table->sgl, sg, table->nents, i) {struct page *page =sg_page (sg);unsigned long remainder = vma->vm_end - addr; -unsigned long len =sg_dma_len (sg); +unsigned long len = sg->length; -if (offset >=sg_dma_len (sg)) { - offset -=sg_dma_len (sg); +if (offset >= sg->length) { + offset -= sg->length;continue ; }else if (offset) { page += offset /PAGE_SIZE ; - len =sg_dma_len (sg) - offset; + len = sg->length - offset; offset =0 ; } len =min (len, remainder);
@@ -119,7 +119,7 @@ int ion_heap_buffer_zero(struct ion_buffer *buffer)for_each_sg (table->sgl, sg, table->nents, i) {struct page *page =sg_page (sg); -unsigned long len =sg_dma_len (sg); +unsigned long len = sg->length;for (j =0 ; j < len /PAGE_SIZE ; j++) {struct page *sub_page = page + j;
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -209,7 +209,7 @@ void ion_system_heap_free(struct ion_buffer *buffer)for_each_sg (table->sgl, sg, table->nents, i)free_buffer_page (sys_heap, buffer,sg_page (sg), -get_order (sg_dma_len (sg))); +get_order (sg->length));sg_free_table (table);kfree (table); }
Signature · 1 line
1.8.3.2
PATCH 89/115 ion: check invalid values in ion_system_heap
In reply to John StultzFrom: Colin Cross <[email protected]>
ion_system_heap can only satisfy page alignment, and ion_system_contig_heap can only satisify alignment to the allocation size. Neither can support faulting user mappings because they use slab pages.
--- drivers/staging/android/ion/ion_system_heap.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
drivers/staging/android/ion/ion_system_heap.c +14 -0
@@ -150,6 +150,12 @@ static int ion_system_heap_allocate(struct ion_heap *heap,long size_remaining =PAGE_ALIGN (size);unsigned int max_order = orders[0 ]; +if (align >PAGE_SIZE ) +return -EINVAL ; + +if (ion_buffer_fault_user_mappings (buffer)) +return -EINVAL ; +INIT_LIST_HEAD (&pages);while (size_remaining >0 ) { info =alloc_largest_available (sys_heap, buffer, size_remaining, max_order);
@@ -362,6 +368,14 @@ static int ion_system_contig_heap_allocate(struct ion_heap *heap,unsigned long align, unsigned long flags) { + int order = get_order(len); + +if (align > (PAGE_SIZE << order)) +return -EINVAL ; + +if (ion_buffer_fault_user_mappings (buffer)) +return -EINVAL ; + buffer->priv_virt =kzalloc (len,GFP_KERNEL );if (!buffer->priv_virt)return -ENOMEM ;
Signature · 1 line
1.8.3.2
PATCH 90/115 ion: add test device for unit tests to interact with dma_bufs
In reply to John StultzFrom: Colin Cross <[email protected]>
Add a /dev/ion-test device that will be created if CONFIG_ION_TEST is set. The device accepts a dma_buf fd and allows reading and writing to the backing memory using DMA-like apis or kernel mapping apis. Can be used to test the dma_buf mapping ops, including the ion implementations, from userspace.
--- drivers/staging/android/ion/Kconfig | 7 + drivers/staging/android/ion/Makefile | 1 + drivers/staging/android/ion/ion_test.c | 281 ++++++++++++++++++++++++++++++++ drivers/staging/android/uapi/ion_test.h | 71 ++++++++ 4 files changed, 360 insertions(+) create mode 100644 drivers/staging/android/ion/ion_test.c create mode 100644 drivers/staging/android/uapi/ion_test.h
drivers/staging/android/ion/Kconfig +7 -0
@@ -6,6 +6,13 @@ menuconfig ION help Chose this option to enable the ION Memory Manager. +config ION_TEST + tristate "Ion Test Device" + depends on ION + help + Choose this option to create a device that can be used to test the + kernel and device side ION functions. + config ION_TEGRA tristate "Ion for Tegra" depends on ARCH_TEGRA && ION
drivers/staging/android/ion/Makefile +1 -0
@@ -1,5 +1,6 @@ obj-$(CONFIG_ION) += ion.o ion_heap.o ion_page_pool.o ion_system_heap.o \ ion_carveout_heap.o ion_chunk_heap.o ion_cma_heap.o +obj-$(CONFIG_ION_TEST) += ion_test.o ifdef CONFIG_COMPAT obj-$(CONFIG_ION) += compat_ion.o endif
drivers/staging/android/ion/ion_test.c new file +281 -0
281 lines not on this page · raw message
drivers/staging/android/uapi/ion_test.h new file +71 -0
71 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH 91/115 ion: update idr to avoid deprecated apis
In reply to John StultzFrom: Colin Cross <[email protected]>
Use idr_alloc instead if idr_pre_get/idr_get_new_above, and remove idr_remove_all.
--- drivers/staging/android/ion/ion.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/staging/android/ion/ion.c +5 -12
@@ -432,22 +432,16 @@ static bool ion_handle_validate(struct ion_client *client, struct ion_handle *hastatic int ion_handle_add (struct ion_client *client,struct ion_handle *handle) { -int rc; +int id;struct rb_node **p = &client->handles.rb_node;struct rb_node *parent =NULL ;struct ion_handle *entry; -do { -int id; - rc =idr_pre_get (&client->idr,GFP_KERNEL ); -if (!rc) -return -ENOMEM ; - rc =idr_get_new_above (&client->idr, handle,1 , &id); - handle->id = id; - }while (rc == -EAGAIN ); + id =idr_alloc (&client->idr, handle,1 ,0 ,GFP_KERNEL ); +if (id <0 ) +return id; -if (rc <0 ) -return rc; + handle->id = id;while (*p) { parent = *p;
@@ -786,7 +780,6 @@ void ion_client_destroy(struct ion_client *client)ion_handle_destroy (&handle->ref); } -idr_remove_all (&client->idr);idr_destroy (&client->idr);down_write (&dev->lock);
Signature · 1 line
1.8.3.2
PATCH 92/115 ion: don't use __arm_ioremap to map pages
In reply to John StultzFrom: Colin Cross <[email protected]>
ion_heap_map_kernel already implements mapping a scatterlist of pages into the kernel, and all heaps are required to have struct pages associated with them, so delete the functions that use __arm_ioremap and use ion_heap_map_kernel instead.
--- drivers/staging/android/ion/ion_carveout_heap.c | 42 ++----------------------- drivers/staging/android/ion/ion_chunk_heap.c | 2 -- 2 files changed, 3 insertions(+), 41 deletions(-)
drivers/staging/android/ion/ion_carveout_heap.c +3 -39
@@ -25,8 +25,6 @@#include "ion.h" #include "ion_priv.h" -#include <asm/mach/map.h> -struct ion_carveout_heap {struct ion_heap heap;struct gen_pool *pool;
@@ -109,49 +107,15 @@ void ion_carveout_heap_unmap_dma(struct ion_heap *heap,sg_free_table (buffer->sg_table); } -void *ion_carveout_heap_map_kernel (struct ion_heap *heap, -struct ion_buffer *buffer) -{ -void *ret; -int mtype =MT_MEMORY_NONCACHED ; - -if (buffer->flags &ION_FLAG_CACHED ) - mtype =MT_MEMORY ; - - ret =__arm_ioremap (buffer->priv_phys, buffer->size, - mtype); -if (ret ==NULL ) -return ERR_PTR (-ENOMEM ); - -return ret; -} - -void ion_carveout_heap_unmap_kernel (struct ion_heap *heap, -struct ion_buffer *buffer) -{ -__arm_iounmap (buffer->vaddr); - buffer->vaddr =NULL ; -return ; -} - -int ion_carveout_heap_map_user (struct ion_heap *heap,struct ion_buffer *buffer, -struct vm_area_struct *vma) -{ -return remap_pfn_range (vma, vma->vm_start, -__phys_to_pfn (buffer->priv_phys) + vma->vm_pgoff, - vma->vm_end - vma->vm_start, -pgprot_noncached (vma->vm_page_prot)); -} -static struct ion_heap_ops carveout_heap_ops = { .allocate = ion_carveout_heap_allocate, .free = ion_carveout_heap_free, .phys = ion_carveout_heap_phys, .map_dma = ion_carveout_heap_map_dma, .unmap_dma = ion_carveout_heap_unmap_dma, - .map_user = ion_carveout_heap_map_user, - .map_kernel = ion_carveout_heap_map_kernel, - .unmap_kernel = ion_carveout_heap_unmap_kernel, + .map_user = ion_heap_map_user, + .map_kernel = ion_heap_map_kernel, + .unmap_kernel = ion_heap_unmap_kernel, };struct ion_heap *ion_carveout_heap_create (struct ion_platform_heap *heap_data)
drivers/staging/android/ion/ion_chunk_heap.c +0 -2
@@ -25,8 +25,6 @@#include "ion.h" #include "ion_priv.h" -#include <asm/mach/map.h> -struct ion_chunk_heap {struct ion_heap heap;struct gen_pool *pool;
Signature · 1 line
1.8.3.2
PATCH 93/115 ion: don't use phys_to_page or __phys_to_pfn
In reply to John StultzFrom: Colin Cross <[email protected]>
phys_to_page and __phys_to_pfn don't exist on all platforms. Use a combination of pfn_to_page, PFN_DOWN, page_to_pfn, and virt_to_page to get the same results.
--- drivers/staging/android/ion/ion_carveout_heap.c | 4 ++-- drivers/staging/android/ion/ion_chunk_heap.c | 5 +++-- drivers/staging/android/ion/ion_system_heap.c | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-)
drivers/staging/android/ion/ion_carveout_heap.c +2 -2
@@ -96,8 +96,8 @@ struct sg_table *ion_carveout_heap_map_dma(struct ion_heap *heap,kfree (table);return ERR_PTR (ret); } -sg_set_page (table->sgl,phys_to_page (buffer->priv_phys), buffer->size, -0 ); +sg_set_page (table->sgl,pfn_to_page (PFN_DOWN (buffer->priv_phys)), + buffer->size,0 );return table; }
drivers/staging/android/ion/ion_chunk_heap.c +3 -2
@@ -71,7 +71,8 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap, chunk_heap->chunk_size);if (!paddr) goto err; -sg_set_page (sg,phys_to_page (paddr), chunk_heap->chunk_size,0 ); +sg_set_page (sg,pfn_to_page (PFN_DOWN (paddr)), + chunk_heap->chunk_size,0 ); sg =sg_next (sg); }
@@ -167,7 +168,7 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) goto error; }for (i =0 ; i < chunk_heap->size; i +=PAGE_SIZE ) { -struct page *page =phys_to_page (chunk_heap->base + i); +struct page *page =pfn_to_page (PFN_DOWN (chunk_heap->base + i));struct page **pages = &page; ret =map_vm_area (vm_struct, pgprot, &pages);
drivers/staging/android/ion/ion_system_heap.c +1 -1
@@ -426,7 +426,7 @@ int ion_system_contig_heap_map_user(struct ion_heap *heap,struct ion_buffer *buffer, struct vm_area_struct *vma) { - unsigned long pfn = __phys_to_pfn(virt_to_phys(buffer->priv_virt)); + unsigned long pfn = page_to_pfn(virt_to_page(buffer->priv_virt));return remap_pfn_range (vma, vma->vm_start, pfn + vma->vm_pgoff, vma->vm_end - vma->vm_start, vma->vm_page_prot);
Signature · 1 line
1.8.3.2
PATCH 94/115 ion: fix printk warnings
From: Colin Cross <[email protected]>
Use %z for size_t and %pa for dma_addr_t to avoid warnings in printks.
--- drivers/staging/android/ion/ion.c | 23 ++++++++++++----------- drivers/staging/android/ion/ion_chunk_heap.c | 2 +- drivers/staging/android/ion/ion_cma_heap.c | 4 ++-- drivers/staging/android/ion/ion_heap.c | 2 +- 4 files changed, 16 insertions(+), 15 deletions(-)
drivers/staging/android/ion/ion.c +12 -11
@@ -471,7 +471,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,struct ion_heap *heap;int ret; -pr_debug ("%s: len %d align %d heap_id_mask %u flags %x\n" , __func__, +pr_debug ("%s: len %zu align %zu heap_id_mask %u flags %x\n" , __func__, len, align, heap_id_mask, flags);/* * traverse the list of heaps available in this system in priority
@@ -688,7 +688,7 @@ static int ion_debug_client_show(struct seq_file *s, void *unused)for (i =0 ; i <ION_NUM_HEAP_IDS ; i++) {if (!names[i])continue ; -seq_printf (s,"%16.16s: %16u\n" , names[i], sizes[i]); +seq_printf (s,"%16.16s: %16zu\n" , names[i], sizes[i]); }return 0 ; }
@@ -1336,10 +1336,10 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)char task_comm[TASK_COMM_LEN ];get_task_comm (task_comm, client->task); -seq_printf (s,"%16.s %16u %16u\n" , task_comm, +seq_printf (s,"%16.s %16u %16zu\n" , task_comm, client->pid, size); }else { -seq_printf (s,"%16.s %16u %16u\n" , client->name, +seq_printf (s,"%16.s %16u %16zu\n" , client->name, client->pid, size); } }
@@ -1354,19 +1354,20 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)continue ; total_size += buffer->size;if (!buffer->handle_count) { -seq_printf (s,"%16.s %16u %16u %d %d\n" , buffer->task_comm, - buffer->pid, buffer->size, buffer->kmap_cnt, +seq_printf (s,"%16.s %16u %16zu %d %d\n" , + buffer->task_comm, buffer->pid, + buffer->size, buffer->kmap_cnt,atomic_read (&buffer->ref.refcount)); total_orphaned_size += buffer->size; } }mutex_unlock (&dev->buffer_lock);seq_printf (s,"----------------------------------------------------\n" ); -seq_printf (s,"%16.s %16u\n" ,"total orphaned" , +seq_printf (s,"%16.s %16zu\n" ,"total orphaned" , total_orphaned_size); -seq_printf (s,"%16.s %16u\n" ,"total " , total_size); +seq_printf (s,"%16.s %16zu\n" ,"total " , total_size);if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) -seq_printf (s,"%16.s %16u\n" ,"deferred free" , +seq_printf (s,"%16.s %16zu\n" ,"deferred free" , heap->free_list_size);seq_printf (s,"----------------------------------------------------\n" );
@@ -1522,11 +1523,11 @@ void __init ion_reserve(struct ion_platform_data *data)int ret =memblock_reserve (data->heaps[i].base, data->heaps[i].size);if (ret) -pr_err ("memblock reserve of %x@%lx failed\n" , +pr_err ("memblock reserve of %zx@%lx failed\n" , data->heaps[i].size, data->heaps[i].base); } -pr_info ("%s: %s reserved base %lx size %d\n" , __func__, +pr_info ("%s: %s reserved base %lx size %zu\n" , __func__, data->heaps[i].name, data->heaps[i].base, data->heaps[i].size);
drivers/staging/android/ion/ion_chunk_heap.c +1 -1
@@ -186,7 +186,7 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ; chunk_heap->heap.flags =ION_HEAP_FLAG_DEFER_FREE ; -pr_info ("%s: base %lu size %u align %ld\n" , __func__, chunk_heap->base, +pr_info ("%s: base %lu size %zu align %ld\n" , __func__, chunk_heap->base, heap_data->size, heap_data->align);return &chunk_heap->heap;
drivers/staging/android/ion/ion_cma_heap.c +2 -2
@@ -128,8 +128,8 @@ static int ion_cma_phys(struct ion_heap *heap, struct ion_buffer *buffer,struct device *dev = cma_heap->dev;struct ion_cma_buffer_info *info = buffer->priv_virt; -dev_dbg (dev,"Return buffer %p physical address 0x%x\n" , buffer, - info->handle); +dev_dbg (dev,"Return buffer %p physical address 0x%pa\n" , buffer, + &info->handle); *addr = info->handle; *len = buffer->size;
drivers/staging/android/ion/ion_heap.c +1 -1
@@ -281,7 +281,7 @@ struct ion_heap *ion_heap_create(struct ion_platform_heap *heap_data) }if (IS_ERR_OR_NULL (heap)) { -pr_err ("%s: error creating heap %s type %d base %lu size %u\n" , +pr_err ("%s: error creating heap %s type %d base %lu size %zu\n" , __func__, heap_data->name, heap_data->type, heap_data->base, heap_data->size);return ERR_PTR (-EINVAL );
Signature · 1 line
1.8.3.2
Re: [PATCH 094/115] ion: fix printk warnings
Quoted 3 lines · John Stultz
From: Colin Cross <[email protected]>
Use %z for
size_tand %pa fordma_addr_tto avoid warnings in printks.
dma_addr_t is not a phys_addr_t and should _not_ be emitted with %pa.
Until %pad is in -next, use %lx, (unsigned long long)dma_addr_t.
Quoted 125 lines
Signed-off-by Colin Cross <[email protected]>Signed-off-by John Stultz <[email protected]>--- drivers/staging/android/ion/ion.c | 23 ++++++++++++----------- drivers/staging/android/ion/ion_chunk_heap.c | 2 +- drivers/staging/android/ion/ion_cma_heap.c | 4 ++-- drivers/staging/android/ion/ion_heap.c | 2 +- 4 files changed, 16 insertions(+), 15 deletions(-)drivers/staging/android/ion/ion.c +12
-11@@ -471,7 +471,7 @@ struct ion_handle *ion_alloc(struct ion_client *client, size_t len,struct ion_heap *heap;int ret; -pr_debug ("%s: len %d align %d heap_id_mask %u flags %x\n" , __func__, +pr_debug ("%s: len %zu align %zu heap_id_mask %u flags %x\n" , __func__, len, align, heap_id_mask, flags);/* * traverse the list of heaps available in this system in priority @@ -688,7 +688,7 @@ static int ion_debug_client_show(struct seq_file *s, void *unused)for (i =0 ; i <ION_NUM_HEAP_IDS ; i++) {if (!names[i])continue ; -seq_printf (s,"%16.16s: %16u\n" , names[i], sizes[i]); +seq_printf (s,"%16.16s: %16zu\n" , names[i], sizes[i]); }return 0 ; }@@ -1336,10 +1336,10 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)char task_comm[TASK_COMM_LEN ];get_task_comm (task_comm, client->task); -seq_printf (s,"%16.s %16u %16u\n" , task_comm, +seq_printf (s,"%16.s %16u %16zu\n" , task_comm, client->pid, size); }else { -seq_printf (s,"%16.s %16u %16u\n" , client->name, +seq_printf (s,"%16.s %16u %16zu\n" , client->name, client->pid, size); } }@@ -1354,19 +1354,20 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)continue ; total_size += buffer->size;if (!buffer->handle_count) { -seq_printf (s,"%16.s %16u %16u %d %d\n" , buffer->task_comm, - buffer->pid, buffer->size, buffer->kmap_cnt, +seq_printf (s,"%16.s %16u %16zu %d %d\n" , + buffer->task_comm, buffer->pid, + buffer->size, buffer->kmap_cnt,atomic_read (&buffer->ref.refcount)); total_orphaned_size += buffer->size; } }mutex_unlock (&dev->buffer_lock);seq_printf (s,"----------------------------------------------------\n" ); -seq_printf (s,"%16.s %16u\n" ,"total orphaned" , +seq_printf (s,"%16.s %16zu\n" ,"total orphaned" , total_orphaned_size); -seq_printf (s,"%16.s %16u\n" ,"total " , total_size); +seq_printf (s,"%16.s %16zu\n" ,"total " , total_size);if (heap->flags &ION_HEAP_FLAG_DEFER_FREE ) -seq_printf (s,"%16.s %16u\n" ,"deferred free" , +seq_printf (s,"%16.s %16zu\n" ,"deferred free" , heap->free_list_size);seq_printf (s,"----------------------------------------------------\n" );@@ -1522,11 +1523,11 @@ void __init ion_reserve(struct ion_platform_data *data)int ret =memblock_reserve (data->heaps[i].base, data->heaps[i].size);if (ret) -pr_err ("memblock reserve of %x@%lx failed\n" , +pr_err ("memblock reserve of %zx@%lx failed\n" , data->heaps[i].size, data->heaps[i].base); } -pr_info ("%s: %s reserved base %lx size %d\n" , __func__, +pr_info ("%s: %s reserved base %lx size %zu\n" , __func__, data->heaps[i].name, data->heaps[i].base, data->heaps[i].size);drivers/staging/android/ion/ion_chunk_heap.c +1
-1@@ -186,7 +186,7 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ; chunk_heap->heap.flags =ION_HEAP_FLAG_DEFER_FREE ; -pr_info ("%s: base %lu size %u align %ld\n" , __func__, chunk_heap->base, +pr_info ("%s: base %lu size %zu align %ld\n" , __func__, chunk_heap->base, heap_data->size, heap_data->align);return &chunk_heap->heap;drivers/staging/android/ion/ion_cma_heap.c +2
-2@@ -128,8 +128,8 @@ static int ion_cma_phys(struct ion_heap *heap, struct ion_buffer *buffer,struct device *dev = cma_heap->dev;struct ion_cma_buffer_info *info = buffer->priv_virt; -dev_dbg (dev,"Return buffer %p physical address 0x%x\n" , buffer, - info->handle); +dev_dbg (dev,"Return buffer %p physical address 0x%pa\n" , buffer, + &info->handle); *addr = info->handle; *len = buffer->size;drivers/staging/android/ion/ion_heap.c +1
-1@@ -281,7 +281,7 @@ struct ion_heap *ion_heap_create(struct ion_platform_heap *heap_data) }if (IS_ERR_OR_NULL (heap)) { -pr_err ("%s: error creating heap %s type %d base %lu size %u\n" , +pr_err ("%s: error creating heap %s type %d base %lu size %zu\n" , __func__, heap_data->name, heap_data->type, heap_data->base, heap_data->size);return ERR_PTR (-EINVAL );
PATCH 95/115 gpu: ion: remove unnecessary function from system heap
In reply to John StultzFrom: Colin Cross <[email protected]>
ion_system_contig_heap buffers have an sglist, just call ion_heap_map_user to map it.
--- drivers/staging/android/ion/ion_system_heap.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +1 -12
@@ -422,17 +422,6 @@ void ion_system_contig_heap_unmap_dma(struct ion_heap *heap,kfree (buffer->sg_table); } -int ion_system_contig_heap_map_user (struct ion_heap *heap, -struct ion_buffer *buffer, -struct vm_area_struct *vma) -{ -unsigned long pfn =page_to_pfn (virt_to_page (buffer->priv_virt)); -return remap_pfn_range (vma, vma->vm_start, pfn + vma->vm_pgoff, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); - -} -static struct ion_heap_ops kmalloc_ops = { .allocate = ion_system_contig_heap_allocate, .free = ion_system_contig_heap_free,
@@ -441,7 +430,7 @@ static struct ion_heap_ops kmalloc_ops = { .unmap_dma = ion_system_contig_heap_unmap_dma, .map_kernel = ion_heap_map_kernel, .unmap_kernel = ion_heap_unmap_kernel, - .map_user = ion_system_contig_heap_map_user, + .map_user = ion_heap_map_user, };struct ion_heap *ion_system_contig_heap_create (struct ion_platform_heap *unused)
Signature · 1 line
1.8.3.2
PATCH 96/115 ion: clean up ioctls
In reply to John StultzFrom: Colin Cross <[email protected]>
Convert the ion ioctls to use _IOW instead of _IOWR where appropriate, and factor out the copy_from_user and copy_to_user based on the _IOC_DIR bits. For the existing incorrect ioctls, add a function to wrap _IOC_DIR to return the corrected value.
--- drivers/staging/android/ion/ion.c | 110 ++++++++++++++++++++------------------ 1 file changed, 59 insertions(+), 51 deletions(-)
drivers/staging/android/ion/ion.c +59 -51
@@ -1157,41 +1157,65 @@ static int ion_sync_for_device(struct ion_client *client, int fd)return 0 ; } +/* fix up the cases where the ioctl direction bits are incorrect */ +static unsigned int ion_ioctl_dir (unsigned int cmd) +{ +switch (cmd) { +case ION_IOC_SYNC : +case ION_IOC_FREE : +case ION_IOC_CUSTOM : +return _IOC_WRITE; +default : +return _IOC_DIR (cmd); + } +} +static long ion_ioctl (struct file *filp,unsigned int cmd,unsigned long arg) {struct ion_client *client = filp->private_data; +struct ion_device *dev = client->dev; +struct ion_handle *cleanup_handle =NULL ; +int ret =0 ; +unsigned int dir; + +union { +struct ion_fd_data fd; +struct ion_allocation_data allocation; +struct ion_handle_data handle; +struct ion_custom_data custom; + } data; + + dir =ion_ioctl_dir (cmd); + +if (_IOC_SIZE (cmd) >sizeof (data)) +return -EINVAL ; + +if (dir & _IOC_WRITE) +if (copy_from_user (&data, (void __user *)arg,_IOC_SIZE (cmd))) +return -EFAULT ;switch (cmd) {case ION_IOC_ALLOC : { -struct ion_allocation_data data;struct ion_handle *handle; -if (copy_from_user (&data, (void __user *)arg,sizeof (data))) -return -EFAULT ; - handle =ion_alloc (client, data.len, data.align, - data.heap_id_mask, data.flags); - + handle =ion_alloc (client, data.allocation.len, + data.allocation.align, + data.allocation.heap_id_mask, + data.allocation.flags);if (IS_ERR (handle))return PTR_ERR (handle); - data.handle = handle->id; + data.allocation.handle = handle->id; -if (copy_to_user ((void __user *)arg, &data,sizeof (data))) { -ion_free (client, handle); -return -EFAULT ; - } + cleanup_handle = handle;break ; }case ION_IOC_FREE : { -struct ion_handle_data data;struct ion_handle *handle; -if (copy_from_user (&data, (void __user *)arg, -sizeof (struct ion_handle_data ))) -return -EFAULT ; - handle =ion_handle_get_by_id (client, data.handle); + handle =ion_handle_get_by_id (client, data.handle.handle);if (IS_ERR (handle))return PTR_ERR (handle);ion_free (client, handle);
@@ -1201,68 +1225,52 @@ static long ion_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)case ION_IOC_SHARE :case ION_IOC_MAP : { -struct ion_fd_data data;struct ion_handle *handle; -if (copy_from_user (&data, (void __user *)arg,sizeof (data))) -return -EFAULT ; - handle =ion_handle_get_by_id (client, data.handle); + handle =ion_handle_get_by_id (client, data.handle.handle);if (IS_ERR (handle))return PTR_ERR (handle); - data.fd =ion_share_dma_buf_fd (client, handle); + data.fd.fd =ion_share_dma_buf_fd (client, handle);ion_handle_put (handle); -if (copy_to_user ((void __user *)arg, &data,sizeof (data))) -return -EFAULT ; -if (data.fd <0 ) -return data.fd; +if (data.fd.fd <0 ) + ret = data.fd.fd;break ; }case ION_IOC_IMPORT : { -struct ion_fd_data data;struct ion_handle *handle; -int ret =0 ; -if (copy_from_user (&data, (void __user *)arg, -sizeof (struct ion_fd_data ))) -return -EFAULT ; - handle =ion_import_dma_buf (client, data.fd); + handle =ion_import_dma_buf (client, data.fd.fd);if (IS_ERR (handle)) ret =PTR_ERR (handle);else - data.handle = handle->id; - -if (copy_to_user ((void __user *)arg, &data, -sizeof (struct ion_fd_data ))) -return -EFAULT ; -if (ret <0 ) -return ret; + data.handle.handle = handle->id;break ; }case ION_IOC_SYNC : { -struct ion_fd_data data; -if (copy_from_user (&data, (void __user *)arg, -sizeof (struct ion_fd_data ))) -return -EFAULT ; -ion_sync_for_device (client, data.fd); + ret =ion_sync_for_device (client, data.fd.fd);break ; }case ION_IOC_CUSTOM : { -struct ion_device *dev = client->dev; -struct ion_custom_data data; -if (!dev->custom_ioctl)return -ENOTTY ; -if (copy_from_user (&data, (void __user *)arg, -sizeof (struct ion_custom_data ))) -return -EFAULT ; -return dev->custom_ioctl (client, data.cmd, data.arg); + ret = dev->custom_ioctl (client, data.custom.cmd, + data.custom.arg); +break ; }default :return -ENOTTY ; } -return 0 ; + +if (dir & _IOC_READ) { +if (copy_to_user ((void __user *)arg, &data,_IOC_SIZE (cmd))) { +if (cleanup_handle) +ion_free (client, cleanup_handle); +return -EFAULT ; + } + } +return ret; }static int ion_release (struct inode *inode,struct file *file)
Signature · 1 line
1.8.3.2
PATCH 97/115 gpu: ion: fix use-after-free in ion_heap_freelist_drain
In reply to John StultzFrom: Mitchel Humpherys <[email protected]>
The `buffer' variable is being used after being freed. Fix this.
--- drivers/staging/android/ion/ion_heap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/android/ion/ion_heap.c +1 -1
@@ -200,9 +200,9 @@ size_t ion_heap_freelist_drain(struct ion_heap *heap, size_t size)if (total_drained >= size)break ;list_del (&buffer->list); -ion_buffer_destroy (buffer); heap->free_list_size -= buffer->size; total_drained += buffer->size; +ion_buffer_destroy (buffer); }rt_mutex_unlock (&heap->lock);
Signature · 1 line
1.8.3.2
PATCH 98/115 ion: Fix two small issues in system_heap allocation
In reply to John StultzIn testing ion system heap allocations, I ran across two issues:
1) Not k*z*allocing the sg table. This can cause trouble if we end up trying call sg_alloc_table() with too many entries, then sg_alloc_table() internally fails and tries to free what it thinks is internal table structure, which causes bad pointer traversals.
2) The second list_for_each_entry probably should be _safe, since I was seeing strange lock warnings and oopses on occasion. This seems to resolve it, but could use some extra checking.
--- drivers/staging/android/ion/ion_system_heap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +2 -3
@@ -166,8 +166,7 @@ static int ion_system_heap_allocate(struct ion_heap *heap, max_order = info->order; i++; } - - table =kmalloc (sizeof (struct sg_table ),GFP_KERNEL ); + table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL );if (!table) goto err;
@@ -189,7 +188,7 @@ static int ion_system_heap_allocate(struct ion_heap *heap, err1:kfree (table); err: -list_for_each_entry (info, &pages, list) { +list_for_each_entry_safe (info, tmp_info, &pages, list) {free_buffer_page (sys_heap, buffer, info->page, info->order);kfree (info); }
Signature · 1 line
1.8.3.2
PATCH 99/115 ion: drop dependency on ARM
In reply to John StultzFrom: Colin Cross <[email protected]>
Ion will compile and run on other platforms now, remove the dependency on ARM.
--- drivers/staging/android/ion/Kconfig | 1 - 1 file changed, 1 deletion(-)
drivers/staging/android/ion/Kconfig +0 -1
@@ -1,6 +1,5 @@ menuconfig ION tristate "Ion Memory Manager" - depends on ARM select GENERIC_ALLOCATOR select DMA_SHARED_BUFFER help
Signature · 1 line
1.8.3.2
PATCH 100/115 ion: add alignment check to carveout heap
In reply to John StultzFrom: Colin Cross <[email protected]>
--- drivers/staging/android/ion/ion_carveout_heap.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/staging/android/ion/ion_carveout_heap.c +3 -0
@@ -70,6 +70,9 @@ static int ion_carveout_heap_allocate(struct ion_heap *heap, unsigned long size, unsigned long align, unsigned long flags) { + if (align > PAGE_SIZE) + return -EINVAL; + buffer->priv_phys =ion_carveout_allocate (heap, size, align);return buffer->priv_phys ==ION_CARVEOUT_ALLOCATE_FAIL ? -ENOMEM :0 ; }
Signature · 1 line
1.8.3.2
PATCH 101/115 ion: optimize ion_heap_buffer_zero
In reply to John StultzFrom: Colin Cross <[email protected]>
ion_heap_buffer_zero can spend a long time in unmap_kernel_range if it has to broadcast a tlb flush to every cpu for every page. Modify it to batch pages into a larger region to clear using a single mapping. This may cause the mapping size to change if the buffer size is not a multiple of the mapping size, so switch to allocating the address space for each chunk. This allows us to use vm_map_ram to handle the allocation and mapping together.
The number of pages to zero using a single mapping is set to 32 to hit the fastpath in vm_map_ram.
--- drivers/staging/android/ion/ion_heap.c | 36 +++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 14 deletions(-)
drivers/staging/android/ion/ion_heap.c +22 -14
@@ -100,40 +100,48 @@ int ion_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer,return 0 ; } +static int ion_heap_clear_pages (struct page **pages,int num,pgprot_t pgprot) +{ +void *addr =vm_map_ram (pages, num,-1 , pgprot); +if (!addr) +return -ENOMEM ; +memset (addr,0 ,PAGE_SIZE * num); +vm_unmap_ram (addr, num); + +return 0 ; +} +int ion_heap_buffer_zero (struct ion_buffer *buffer) {struct sg_table *table = buffer->sg_table;pgprot_t pgprot;struct scatterlist *sg; -struct vm_struct *vm_struct;int i, j, ret =0 ; +struct page *pages[32 ]; +int k =0 ;if (buffer->flags &ION_FLAG_CACHED ) pgprot =PAGE_KERNEL ;else pgprot =pgprot_writecombine (PAGE_KERNEL ); - vm_struct =get_vm_area (PAGE_SIZE ,VM_ALLOC ); -if (!vm_struct) -return -ENOMEM ; -for_each_sg (table->sgl, sg, table->nents, i) {struct page *page =sg_page (sg);unsigned long len = sg->length;for (j =0 ; j < len /PAGE_SIZE ; j++) { -struct page *sub_page = page + j; -struct page **pages = &sub_page; - ret =map_vm_area (vm_struct, pgprot, &pages); -if (ret) - goto end; -memset (vm_struct->addr,0 ,PAGE_SIZE ); -unmap_kernel_range ((unsigned long )vm_struct->addr, -PAGE_SIZE ); + pages[k++] = page + j; +if (k ==ARRAY_SIZE (pages)) { + ret =ion_heap_clear_pages (pages, k, pgprot); +if (ret) + goto end; + k =0 ; + } } +if (k) + ret =ion_heap_clear_pages (pages, k, pgprot); } end: -free_vm_area (vm_struct);return ret; }
Signature · 1 line
1.8.3.2
PATCH 102/115 ion: free low memory from page pools first
In reply to John StultzFrom: Colin Cross <[email protected]>
When the shrinkers are called with GFP_HIGH free low memory first, it is more important to have free than high memory.
--- drivers/staging/android/ion/ion_page_pool.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion_page_pool.c +3 -3
@@ -143,10 +143,10 @@ int ion_page_pool_shrink(struct ion_page_pool *pool, gfp_t gfp_mask,struct page *page;mutex_lock (&pool->mutex); -if (high && pool->high_count) { - page =ion_page_pool_remove (pool, true); - }else if (pool->low_count) { +if (pool->low_count) { page =ion_page_pool_remove (pool, false); + }else if (high && pool->high_count) { + page =ion_page_pool_remove (pool, true); }else {mutex_unlock (&pool->mutex);break ;
Signature · 1 line
1.8.3.2
PATCH 103/115 ion: check return value from remap_pfn_range
In reply to John StultzFrom: Colin Cross <[email protected]>
Check the return value of remap_pfn_range and return an error if it fails.
--- drivers/staging/android/ion/ion_heap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion_heap.c +4 -1
@@ -76,6 +76,7 @@ int ion_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer,unsigned long offset = vma->vm_pgoff *PAGE_SIZE ;struct scatterlist *sg;int i; +int ret;for_each_sg (table->sgl, sg, table->nents, i) {struct page *page =sg_page (sg);
@@ -91,8 +92,10 @@ int ion_heap_map_user(struct ion_heap *heap, struct ion_buffer *buffer, offset =0 ; } len =min (len, remainder); -remap_pfn_range (vma, addr,page_to_pfn (page), len, + ret =remap_pfn_range (vma, addr,page_to_pfn (page), len, vma->vm_page_prot); +if (ret) +return ret; addr += len;if (addr >= vma->vm_end)return 0 ;
Signature · 1 line
1.8.3.2
PATCH 104/115 ion: use vm_insert_pfn for faulted pages
In reply to John StultzFrom: Colin Cross <[email protected]>
Most ion userspace mappings are created with remap_pfn_range. Use vm_insert_pfn instead of vm_insert_page to make faulted cached mappings look more like uncached mappings.
--- drivers/staging/android/ion/ion.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/staging/android/ion/ion.c +6 -3
@@ -890,14 +890,15 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,int ion_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) {struct ion_buffer *buffer = vma->vm_private_data; +unsigned long pfn;int ret;mutex_lock (&buffer->lock);ion_buffer_page_dirty (buffer->pages + vmf->pgoff); -BUG_ON (!buffer->pages || !buffer->pages[vmf->pgoff]); - ret =vm_insert_page (vma, (unsigned long )vmf->virtual_address, -ion_buffer_page (buffer->pages[vmf->pgoff])); + + pfn =page_to_pfn (ion_buffer_page (buffer->pages[vmf->pgoff])); + ret =vm_insert_pfn (vma, (unsigned long )vmf->virtual_address, pfn);mutex_unlock (&buffer->lock);if (ret)return VM_FAULT_ERROR ;
@@ -956,6 +957,8 @@ static int ion_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) }if (ion_buffer_fault_user_mappings (buffer)) { + vma->vm_flags |=VM_IO |VM_PFNMAP |VM_DONTEXPAND | +VM_DONTDUMP ; vma->vm_private_data = buffer; vma->vm_ops = &ion_vma_ops;ion_vm_open (vma);
Signature · 1 line
1.8.3.2
PATCH 105/115 ion: remove ion_heap_alloc_pages
In reply to John StultzFrom: Colin Cross <[email protected]>
Now that ion_vm_fault doesn't need a struct page with a nonzero refcount, there is no need allocate heap memory for cached pages using split_page. Remove the ion_heap_alloc_pages and ion_heap_free_pages helpers in favor of direct calls to alloc_pages and __free_pages, and remove the special handling in the system heap.
--- drivers/staging/android/ion/ion_heap.c | 27 --------------------------- drivers/staging/android/ion/ion_priv.h | 13 ------------- drivers/staging/android/ion/ion_system_heap.c | 7 +------ 3 files changed, 1 insertion(+), 46 deletions(-)
drivers/staging/android/ion/ion_heap.c +0 -27
@@ -148,33 +148,6 @@ end:return ret; } -struct page *ion_heap_alloc_pages (struct ion_buffer *buffer,gfp_t gfp_flags, -unsigned int order) -{ -struct page *page =alloc_pages (gfp_flags, order); - -if (!page) -return page; - -if (ion_buffer_fault_user_mappings (buffer)) -split_page (page, order); - -return page; -} - -void ion_heap_free_pages (struct ion_buffer *buffer,struct page *page, -unsigned int order) -{ -int i; - -if (!ion_buffer_fault_user_mappings (buffer)) { -__free_pages (page, order); -return ; - } -for (i =0 ; i < (1 << order); i++) -__free_page (page + i); -} -void ion_heap_freelist_add (struct ion_heap *heap,struct ion_buffer * buffer) {rt_mutex_lock (&heap->lock);
drivers/staging/android/ion/ion_priv.h +0 -13
@@ -217,19 +217,6 @@ int ion_heap_map_user(struct ion_heap *, struct ion_buffer *,int ion_heap_buffer_zero (struct ion_buffer *buffer);/** -* ion_heap_alloc_pages - allocate pages from alloc_pages -* @buffer: the buffer to allocate for, used to extract the flags -* @gfp_flags: the gfp_t for the allocation -* @order: the order of the allocatoin -* -* This funciton allocations from alloc pages and also does any other -* necessary operations based on the buffer->flags. For buffers which -* will be faulted in the pages are split using split_page -*/ -struct page *ion_heap_alloc_pages (struct ion_buffer *buffer,gfp_t gfp_flags, -unsigned int order); - -/** * ion_heap_init_deferred_free -- initialize deferred free functionality * @heap: the heap *
drivers/staging/android/ion/ion_system_heap.c +1 -6
@@ -74,7 +74,7 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap,if (order >4 ) gfp_flags = high_order_gfp_flags; - page =ion_heap_alloc_pages (buffer, gfp_flags, order); + page =alloc_pages (gfp_flags, order);if (!page)return 0 ; ion_pages_sync_for_device(NULL, page, PAGE_SIZE << order,
@@ -91,15 +91,10 @@ static void free_buffer_page(struct ion_system_heap *heap, unsigned int order) {bool cached =ion_buffer_cached (buffer); -bool split_pages =ion_buffer_fault_user_mappings (buffer); -int i;if (!cached) {struct ion_page_pool *pool = heap->pools[order_to_index (order)];ion_page_pool_free (pool, page); - }else if (split_pages) { -for (i =0 ; i < (1 << order); i++) -__free_page (page + i); }else {__free_pages (page, order); }
Signature · 1 line
1.8.3.2
PATCH 106/115 ion: allow cached mappings of chunk and system heap buffers
In reply to John StultzFrom: Colin Cross <[email protected]>
Now that ion_vm_fault uses vm_insert_pfn instead of vm_insert_page cached buffers can be supported in any heap. Remove the checks in the chunk and system heaps.
--- drivers/staging/android/ion/ion_chunk_heap.c | 3 --- drivers/staging/android/ion/ion_system_heap.c | 3 --- 2 files changed, 6 deletions(-)
drivers/staging/android/ion/ion_chunk_heap.c +0 -3
@@ -47,9 +47,6 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap,unsigned long num_chunks;unsigned long allocated_size; -if (ion_buffer_fault_user_mappings (buffer)) -return -ENOMEM ; - allocated_size =ALIGN (size, chunk_heap->chunk_size); num_chunks = allocated_size / chunk_heap->chunk_size;
drivers/staging/android/ion/ion_system_heap.c +0 -3
@@ -148,9 +148,6 @@ static int ion_system_heap_allocate(struct ion_heap *heap,if (align >PAGE_SIZE )return -EINVAL ; -if (ion_buffer_fault_user_mappings (buffer)) -return -EINVAL ; -INIT_LIST_HEAD (&pages);while (size_remaining >0 ) { info =alloc_largest_available (sys_heap, buffer, size_remaining, max_order);
Signature · 1 line
1.8.3.2
PATCH 107/115 ion: use alloc_pages in system contig heap
In reply to John StultzFrom: Colin Cross <[email protected]>
There is no reason to use kzalloc, just call alloc_pages directly. Change the GFP from GFP_KERNEL to include __GFP_HIGH, to allow it to return contiguous pages from highmem. virt_to_* functions aren't valid on highmem pages, so store the struct page * in an sg_table in buffer->priv_virt like most other heaps, and replace virt_to_* with page_to_*.
--- drivers/staging/android/ion/ion_system_heap.c | 71 ++++++++++++++++++--------- 1 file changed, 48 insertions(+), 23 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +48 -23
@@ -360,29 +360,69 @@ static int ion_system_contig_heap_allocate(struct ion_heap *heap, unsigned long flags) {int order =get_order (len); +struct page *page; +struct sg_table *table; +unsigned long i; +int ret;if (align > (PAGE_SIZE << order))return -EINVAL ; -if (ion_buffer_fault_user_mappings (buffer)) -return -EINVAL ; - - buffer->priv_virt =kzalloc (len,GFP_KERNEL ); -if (!buffer->priv_virt) + page =alloc_pages (low_order_gfp_flags, order); +if (!page)return -ENOMEM ; + +split_page (page, order); + + len =PAGE_ALIGN (len); +for (i = len >>PAGE_SHIFT ; i < (1 << order); i++) +__free_page (page + i); + + table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); +if (!table) { + ret = -ENOMEM ; + goto out; + } + + ret =sg_alloc_table (table,1 ,GFP_KERNEL ); +if (ret) + goto out; + +sg_set_page (table->sgl, page, len,0 ); + + buffer->priv_virt = table; + +ion_pages_sync_for_device (NULL , page, len,DMA_BIDIRECTIONAL ); +return 0 ; + +out: +for (i =0 ; i < len >>PAGE_SHIFT ; i++) +__free_page (page + i); +kfree (table); +return ret; }void ion_system_contig_heap_free (struct ion_buffer *buffer) { -kfree (buffer->priv_virt); +struct sg_table *table = buffer->priv_virt; +struct page *page =sg_page (table->sgl); +unsigned long pages =PAGE_ALIGN (buffer->size) >>PAGE_SHIFT ; +unsigned long i; + +for (i =0 ; i < pages; i++) +__free_page (page + i); +sg_free_table (table); +kfree (table); }static int ion_system_contig_heap_phys (struct ion_heap *heap,struct ion_buffer *buffer,ion_phys_addr_t *addr,size_t *len) { - *addr =virt_to_phys (buffer->priv_virt); +struct sg_table *table = buffer->priv_virt; +struct page *page =sg_page (table->sgl); + *addr =page_to_phys (page); *len = buffer->size;return 0 ; }
@@ -390,27 +430,12 @@ static int ion_system_contig_heap_phys(struct ion_heap *heap,struct sg_table *ion_system_contig_heap_map_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -struct sg_table *table; -int ret; - - table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); -if (!table) -return ERR_PTR (-ENOMEM ); - ret =sg_alloc_table (table,1 ,GFP_KERNEL ); -if (ret) { -kfree (table); -return ERR_PTR (ret); - } -sg_set_page (table->sgl,virt_to_page (buffer->priv_virt), buffer->size, -0 ); -return table; +return buffer->priv_virt; }void ion_system_contig_heap_unmap_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -sg_free_table (buffer->sg_table); -kfree (buffer->sg_table); }static struct ion_heap_ops kmalloc_ops = {
Signature · 1 line
1.8.3.2
PATCH 108/115 ion: fix sparse warnings
In reply to John StultzFrom: Colin Cross <[email protected]>
Fix sparse warnings in ion.
--- drivers/staging/android/ion/ion.c | 8 +++---- drivers/staging/android/ion/ion_carveout_heap.c | 8 +++---- drivers/staging/android/ion/ion_chunk_heap.c | 8 +++---- drivers/staging/android/ion/ion_cma_heap.c | 15 +++++++------ drivers/staging/android/ion/ion_heap.c | 4 ++-- drivers/staging/android/ion/ion_page_pool.c | 2 +- drivers/staging/android/ion/ion_system_heap.c | 30 ++++++++++++------------- 7 files changed, 37 insertions(+), 38 deletions(-)
drivers/staging/android/ion/ion.c +4 -4
@@ -669,7 +669,7 @@ static int ion_debug_client_show(struct seq_file *s, void *unused)struct ion_client *client = s->private;struct rb_node *n;size_t sizes[ION_NUM_HEAP_IDS ] = {0 }; -const char *names[ION_NUM_HEAP_IDS ] = {0 }; +const char *names[ION_NUM_HEAP_IDS ] = {NULL };int i;mutex_lock (&client->lock);
@@ -887,7 +887,7 @@ static void ion_buffer_sync_for_device(struct ion_buffer *buffer,mutex_unlock (&buffer->lock); } -int ion_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) +static int ion_vm_fault (struct vm_area_struct *vma,struct vm_fault *vmf) {struct ion_buffer *buffer = vma->vm_private_data;unsigned long pfn;
@@ -939,7 +939,7 @@ static void ion_vm_close(struct vm_area_struct *vma)mutex_unlock (&buffer->lock); } -struct vm_operations_struct ion_vma_ops = { +static struct vm_operations_struct ion_vma_ops = { .open = ion_vm_open, .close = ion_vm_close, .fault = ion_vm_fault,
@@ -1030,7 +1030,7 @@ static void ion_dma_buf_end_cpu_access(struct dma_buf *dmabuf, size_t start,mutex_unlock (&buffer->lock); } -struct dma_buf_ops dma_buf_ops = { +static struct dma_buf_ops dma_buf_ops = { .map_dma_buf = ion_map_dma_buf, .unmap_dma_buf = ion_unmap_dma_buf, .mmap = ion_mmap,
drivers/staging/android/ion/ion_carveout_heap.c +4 -4
@@ -85,8 +85,8 @@ static void ion_carveout_heap_free(struct ion_buffer *buffer) buffer->priv_phys =ION_CARVEOUT_ALLOCATE_FAIL ; } -struct sg_table *ion_carveout_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static struct sg_table *ion_carveout_heap_map_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {struct sg_table *table;int ret;
@@ -104,8 +104,8 @@ struct sg_table *ion_carveout_heap_map_dma(struct ion_heap *heap,return table; } -void ion_carveout_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static void ion_carveout_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {sg_free_table (buffer->sg_table); }
drivers/staging/android/ion/ion_chunk_heap.c +4 -4
@@ -115,14 +115,14 @@ static void ion_chunk_heap_free(struct ion_buffer *buffer)kfree (table); } -struct sg_table *ion_chunk_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static struct sg_table *ion_chunk_heap_map_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {return buffer->priv_virt; } -void ion_chunk_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static void ion_chunk_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {return ; }
drivers/staging/android/ion/ion_cma_heap.c +8 -7
@@ -44,8 +44,8 @@ struct ion_cma_buffer_info {* This function could be replaced by dma_common_get_sgtable * as soon as it will avalaible. */ -int ion_cma_get_sgtable (struct device *dev,struct sg_table *sgt, -void *cpu_addr,dma_addr_t handle,size_t size) +static int ion_cma_get_sgtable (struct device *dev,struct sg_table *sgt, +void *cpu_addr,dma_addr_t handle,size_t size) {struct page *page =virt_to_page (cpu_addr);int ret;
@@ -137,16 +137,16 @@ static int ion_cma_phys(struct ion_heap *heap, struct ion_buffer *buffer,return 0 ; } -struct sg_table *ion_cma_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static struct sg_table *ion_cma_heap_map_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {struct ion_cma_buffer_info *info = buffer->priv_virt;return info->table; } -void ion_cma_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static void ion_cma_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {return ; }
@@ -162,7 +162,8 @@ static int ion_cma_mmap(struct ion_heap *mapper, struct ion_buffer *buffer, buffer->size); } -void *ion_cma_map_kernel (struct ion_heap *heap,struct ion_buffer *buffer) +static void *ion_cma_map_kernel (struct ion_heap *heap, +struct ion_buffer *buffer) {struct ion_cma_buffer_info *info = buffer->priv_virt;/* kernel memory mapping has been done at allocation time */
drivers/staging/android/ion/ion_heap.c +2 -2
@@ -38,7 +38,7 @@ void *ion_heap_map_kernel(struct ion_heap *heap,struct page **tmp = pages;if (!pages) -return 0 ; +return NULL ;if (buffer->flags &ION_FLAG_CACHED ) pgprot =PAGE_KERNEL ;
@@ -193,7 +193,7 @@ size_t ion_heap_freelist_drain(struct ion_heap *heap, size_t size)return total_drained; } -int ion_heap_deferred_free (void *data) +static int ion_heap_deferred_free (void *data) {struct ion_heap *heap = data;
drivers/staging/android/ion/ion_page_pool.c +1 -1
@@ -134,7 +134,7 @@ int ion_page_pool_shrink(struct ion_page_pool *pool, gfp_t gfp_mask,int i;bool high; - high = gfp_mask & __GFP_HIGHMEM; + high = !!(gfp_mask & __GFP_HIGHMEM);if (nr_to_scan ==0 )return ion_page_pool_total (pool, high);
drivers/staging/android/ion/ion_system_heap.c +14 -16
@@ -26,11 +26,9 @@#include "ion.h" #include "ion_priv.h" -static unsigned int high_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | - __GFP_NOWARN | __GFP_NORETRY) & - ~__GFP_WAIT; -static unsigned int low_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | - __GFP_NOWARN); +static gfp_t high_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN | + __GFP_NORETRY) & ~__GFP_WAIT; +static gfp_t low_order_gfp_flags = (GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN);static const unsigned int orders[] = {8 ,4 ,0 };static const int num_orders =ARRAY_SIZE (orders);static int order_to_index (unsigned int order)
@@ -76,12 +74,12 @@ static struct page *alloc_buffer_page(struct ion_system_heap *heap, gfp_flags = high_order_gfp_flags; page =alloc_pages (gfp_flags, order);if (!page) -return 0 ; +return NULL ;ion_pages_sync_for_device (NULL , page,PAGE_SIZE << order,DMA_BIDIRECTIONAL ); }if (!page) -return 0 ; +return NULL ;return page; }
@@ -187,7 +185,7 @@ err:return -ENOMEM ; } -void ion_system_heap_free (struct ion_buffer *buffer) +static void ion_system_heap_free (struct ion_buffer *buffer) {struct ion_heap *heap = buffer->heap; struct ion_system_heap *sys_heap = container_of(heap,
@@ -211,14 +209,14 @@ void ion_system_heap_free(struct ion_buffer *buffer)kfree (table); } -struct sg_table *ion_system_heap_map_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static struct sg_table *ion_system_heap_map_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {return buffer->priv_virt; } -void ion_system_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static void ion_system_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) {return ; }
@@ -403,7 +401,7 @@ out:return ret; } -void ion_system_contig_heap_free (struct ion_buffer *buffer) +static void ion_system_contig_heap_free (struct ion_buffer *buffer) {struct sg_table *table = buffer->priv_virt;struct page *page =sg_page (table->sgl);
@@ -427,14 +425,14 @@ static int ion_system_contig_heap_phys(struct ion_heap *heap,return 0 ; } -struct sg_table *ion_system_contig_heap_map_dma (struct ion_heap *heap, +static struct sg_table *ion_system_contig_heap_map_dma (struct ion_heap *heap,struct ion_buffer *buffer) {return buffer->priv_virt; } -void ion_system_contig_heap_unmap_dma (struct ion_heap *heap, -struct ion_buffer *buffer) +static void ion_system_contig_heap_unmap_dma (struct ion_heap *heap, +struct ion_buffer *buffer) { }
Signature · 1 line
1.8.3.2
PATCH 109/115 ion: carveout heap: zero buffers on free, fix memory leak
In reply to John StultzFrom: Colin Cross <[email protected]>
The carveout heap wasn't zeroing its buffers after use. Create the sg_table during allocate instead of map_dma, to allow using the sg_table during free, and call ion_heap_buffer_zero during free. Also fixes a missing kfree when destroying the table.
--- drivers/staging/android/ion/ion_carveout_heap.c | 69 +++++++++++++++++-------- 1 file changed, 48 insertions(+), 21 deletions(-)
drivers/staging/android/ion/ion_carveout_heap.c +48 -21
@@ -14,7 +14,7 @@* */ #include <linux/spinlock.h> - +#include <linux/dma-mapping.h> #include <linux/err.h> #include <linux/genalloc.h> #include <linux/io.h>
@@ -60,7 +60,11 @@ static int ion_carveout_heap_phys(struct ion_heap *heap,struct ion_buffer *buffer, ion_phys_addr_t *addr, size_t *len) { - *addr = buffer->priv_phys; + struct sg_table *table = buffer->priv_virt; +struct page *page =sg_page (table->sgl); +ion_phys_addr_t paddr =PFN_PHYS (page_to_pfn (page)); + + *addr = paddr; *len = buffer->size;return 0 ; }
@@ -70,44 +74,66 @@ static int ion_carveout_heap_allocate(struct ion_heap *heap, unsigned long size, unsigned long align, unsigned long flags) { + struct sg_table *table; +ion_phys_addr_t paddr; +int ret; +if (align >PAGE_SIZE )return -EINVAL ; - buffer->priv_phys =ion_carveout_allocate (heap, size, align); -return buffer->priv_phys ==ION_CARVEOUT_ALLOCATE_FAIL ? -ENOMEM :0 ; + table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); +if (!table) +return -ENOMEM ; + ret =sg_alloc_table (table,1 ,GFP_KERNEL ); +if (ret) + goto err_free; + + paddr =ion_carveout_allocate (heap, size, align); +if (paddr ==ION_CARVEOUT_ALLOCATE_FAIL ) { + ret = -ENOMEM ; + goto err_free_table; + } + +sg_set_page (table->sgl,pfn_to_page (PFN_DOWN (paddr)), size,0 ); + buffer->priv_virt = table; + +return 0 ; + +err_free_table: +sg_free_table (table); +err_free: +kfree (table); +return ret; }static void ion_carveout_heap_free (struct ion_buffer *buffer) {struct ion_heap *heap = buffer->heap; +struct sg_table *table = buffer->priv_virt; +struct page *page =sg_page (table->sgl); +ion_phys_addr_t paddr =PFN_PHYS (page_to_pfn (page)); + +ion_heap_buffer_zero (buffer); -ion_carveout_free (heap, buffer->priv_phys, buffer->size); - buffer->priv_phys =ION_CARVEOUT_ALLOCATE_FAIL ; +if (ion_buffer_cached (buffer)) +dma_sync_sg_for_device (NULL , table->sgl, table->nents, +DMA_BIDIRECTIONAL ); + +ion_carveout_free (heap, paddr, buffer->size); +sg_free_table (table); +kfree (table); }static struct sg_table *ion_carveout_heap_map_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -struct sg_table *table; -int ret; - - table =kzalloc (sizeof (struct sg_table ),GFP_KERNEL ); -if (!table) -return ERR_PTR (-ENOMEM ); - ret =sg_alloc_table (table,1 ,GFP_KERNEL ); -if (ret) { -kfree (table); -return ERR_PTR (ret); - } -sg_set_page (table->sgl,pfn_to_page (PFN_DOWN (buffer->priv_phys)), - buffer->size,0 ); -return table; +return buffer->priv_virt; }static void ion_carveout_heap_unmap_dma (struct ion_heap *heap,struct ion_buffer *buffer) { -sg_free_table (buffer->sg_table); +return ; } static struct ion_heap_ops carveout_heap_ops = {
@@ -139,6 +165,7 @@ struct ion_heap *ion_carveout_heap_create(struct ion_platform_heap *heap_data) -1); carveout_heap->heap.ops = &carveout_heap_ops; carveout_heap->heap.type =ION_HEAP_TYPE_CARVEOUT ; + carveout_heap->heap.flags =ION_HEAP_FLAG_DEFER_FREE ;return &carveout_heap->heap; }
Signature · 1 line
1.8.3.2
PATCH 110/115 ion: add helper to zero contiguous region of pages
In reply to John StultzFrom: Colin Cross <[email protected]>
Add ion_heap_pages_zero for ion heaps to use to zero pages during initialization or allocation, when a struct ion_buffer may not be available. Use it from the chunk heap and carveout heaps.
--- drivers/staging/android/ion/ion_carveout_heap.c | 13 ++++++ drivers/staging/android/ion/ion_chunk_heap.c | 39 ++++++------------ drivers/staging/android/ion/ion_heap.c | 53 +++++++++++++++---------- drivers/staging/android/ion/ion_priv.h | 1 + 4 files changed, 58 insertions(+), 48 deletions(-)
drivers/staging/android/ion/ion_carveout_heap.c +13 -0
@@ -150,6 +150,19 @@ static struct ion_heap_ops carveout_heap_ops = {struct ion_heap *ion_carveout_heap_create (struct ion_platform_heap *heap_data) {struct ion_carveout_heap *carveout_heap; +int ret; + +struct page *page; +size_t size; + + page =pfn_to_page (PFN_DOWN (heap_data->base)); + size = heap_data->size; + +ion_pages_sync_for_device (NULL , page, size,DMA_BIDIRECTIONAL ); + + ret =ion_heap_pages_zero (page, size,pgprot_writecombine (PAGE_KERNEL )); +if (ret) +return ERR_PTR (ret); carveout_heap =kzalloc (sizeof (struct ion_carveout_heap ),GFP_KERNEL );if (!carveout_heap)
drivers/staging/android/ion/ion_chunk_heap.c +12 -27
@@ -140,9 +140,18 @@ static struct ion_heap_ops chunk_heap_ops = {struct ion_heap *ion_chunk_heap_create (struct ion_platform_heap *heap_data) {struct ion_chunk_heap *chunk_heap; -struct vm_struct *vm_struct; -pgprot_t pgprot =pgprot_writecombine (PAGE_KERNEL ); -int i, ret; +int ret; +struct page *page; +size_t size; + + page =pfn_to_page (PFN_DOWN (heap_data->base)); + size = heap_data->size; + +ion_pages_sync_for_device (NULL , page, size,DMA_BIDIRECTIONAL ); + + ret =ion_heap_pages_zero (page, size,pgprot_writecombine (PAGE_KERNEL )); +if (ret) +return ERR_PTR (ret); chunk_heap =kzalloc (sizeof (struct ion_chunk_heap ),GFP_KERNEL );if (!chunk_heap)
@@ -159,26 +168,6 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data) chunk_heap->size = heap_data->size; chunk_heap->allocated =0 ; - vm_struct =get_vm_area (PAGE_SIZE ,VM_ALLOC ); -if (!vm_struct) { - ret = -ENOMEM ; - goto error; - } -for (i =0 ; i < chunk_heap->size; i +=PAGE_SIZE ) { -struct page *page =pfn_to_page (PFN_DOWN (chunk_heap->base + i)); -struct page **pages = &page; - - ret =map_vm_area (vm_struct, pgprot, &pages); -if (ret) - goto error_map_vm_area; -memset (vm_struct->addr,0 ,PAGE_SIZE ); -unmap_kernel_range ((unsigned long )vm_struct->addr,PAGE_SIZE ); - } -free_vm_area (vm_struct); - -ion_pages_sync_for_device (NULL ,pfn_to_page (PFN_DOWN (heap_data->base)), - heap_data->size,DMA_BIDIRECTIONAL ); -gen_pool_add (chunk_heap->pool, chunk_heap->base, heap_data->size,-1 ); chunk_heap->heap.ops = &chunk_heap_ops; chunk_heap->heap.type =ION_HEAP_TYPE_CHUNK ;
@@ -188,10 +177,6 @@ struct ion_heap *ion_chunk_heap_create(struct ion_platform_heap *heap_data)return &chunk_heap->heap; -error_map_vm_area: -free_vm_area (vm_struct); -error: -gen_pool_destroy (chunk_heap->pool); error_gen_pool_create:kfree (chunk_heap);return ERR_PTR (ret);
drivers/staging/android/ion/ion_heap.c +32 -21
70 lines not on this page · raw message
drivers/staging/android/ion/ion_priv.h +1 -0
@@ -215,6 +215,7 @@ void ion_heap_unmap_kernel(struct ion_heap *, struct ion_buffer *);int ion_heap_map_user (struct ion_heap *,struct ion_buffer *,struct vm_area_struct *);int ion_heap_buffer_zero (struct ion_buffer *buffer); +int ion_heap_pages_zero (struct page *page,size_t size,pgprot_t pgprot);/** * ion_heap_init_deferred_free -- initialize deferred free functionality
Signature · 1 line
1.8.3.2
PATCH 111/115 ion: add alignment check to chunk heap
In reply to John StultzFrom: Colin Cross <[email protected]>
--- drivers/staging/android/ion/ion_chunk_heap.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/staging/android/ion/ion_chunk_heap.c +3 -0
@@ -47,6 +47,9 @@ static int ion_chunk_heap_allocate(struct ion_heap *heap,unsigned long num_chunks;unsigned long allocated_size; +if (align > chunk_heap->chunk_size) +return -EINVAL ; + allocated_size =ALIGN (size, chunk_heap->chunk_size); num_chunks = allocated_size / chunk_heap->chunk_size;
Signature · 1 line
1.8.3.2
PATCH 112/115 ion: fix bugs in cma heap
In reply to John StultzFrom: Colin Cross <[email protected]>
Implement ion_cma_unmap_kernel, ion will call it unconditionally. Use correct gfp flags when calling dma_alloc_coherent so it doesn't try to use atomic DMA memory.
Check for invalid alignment when allocating.
Reject cached allocations - the cpu address returned by dma_alloc_coherent is always going to be an uncached mapping, so map_kernel will not see data written by a cached userspace mapping.
--- drivers/staging/android/ion/ion_cma_heap.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
drivers/staging/android/ion/ion_cma_heap.c +14 -1
@@ -69,13 +69,20 @@ static int ion_cma_allocate(struct ion_heap *heap, struct ion_buffer *buffer,dev_dbg (dev,"Request buffer allocation len %ld\n" , len); +if (buffer->flags &ION_FLAG_CACHED ) +return -EINVAL ; + +if (align >PAGE_SIZE ) +return -EINVAL ; + info =kzalloc (sizeof (struct ion_cma_buffer_info ),GFP_KERNEL );if (!info) {dev_err (dev,"Can't allocate buffer info\n" );return ION_CMA_ALLOCATE_FAILED ; } - info->cpu_addr =dma_alloc_coherent (dev, len, &(info->handle),0 ); + info->cpu_addr =dma_alloc_coherent (dev, len, &(info->handle), +GFP_HIGHUSER | __GFP_ZERO);if (!info->cpu_addr) {dev_err (dev,"Fail to allocate buffer\n" );
@@ -170,6 +177,11 @@ static void *ion_cma_map_kernel(struct ion_heap *heap,return info->cpu_addr; } +static void ion_cma_unmap_kernel (struct ion_heap *heap, +struct ion_buffer *buffer) +{ +} +static struct ion_heap_ops ion_cma_ops = { .allocate = ion_cma_allocate, .free = ion_cma_free,
@@ -178,6 +190,7 @@ static struct ion_heap_ops ion_cma_ops = { .phys = ion_cma_phys, .map_user = ion_cma_mmap, .map_kernel = ion_cma_map_kernel, + .unmap_kernel = ion_cma_unmap_kernel, };struct ion_heap *ion_cma_heap_create (struct ion_platform_heap *data)
Signature · 1 line
1.8.3.2
PATCH 113/115 ion: Cleanup whitespace issues and other checkpatch problems
In reply to John StultzJust some simple cleanups to address whitespace issues and other issues found w/ checkpatch.
--- drivers/staging/android/ion/compat_ion.c | 4 +- drivers/staging/android/ion/ion.c | 53 ++++++++++++++------------- drivers/staging/android/ion/ion.h | 2 +- drivers/staging/android/ion/ion_chunk_heap.c | 1 - drivers/staging/android/ion/ion_heap.c | 5 +-- drivers/staging/android/ion/ion_page_pool.c | 2 +- drivers/staging/android/ion/ion_priv.h | 2 +- drivers/staging/android/ion/ion_system_heap.c | 3 +- 8 files changed, 37 insertions(+), 35 deletions(-)
drivers/staging/android/ion/compat_ion.c +2 -2
@@ -91,8 +91,8 @@ static int compat_get_ion_custom_data(struct compat_ion_custom_data __user *data32, struct ion_custom_data __user *data) { - compat_uint_t cmd; -compat_ulong_t arg; + compat_uint_t cmd; +compat_ulong_t arg;int err; err =get_user (cmd, &data32->cmd);
drivers/staging/android/ion/ion.c +28 -25
96 lines not on this page · raw message
drivers/staging/android/ion/ion.h +1 -1
@@ -38,7 +38,7 @@ struct ion_buffer;* struct ion_platform_heap - defines a heap in the given platform * @type: type of the heap from ion_heap_type enum * @id: unique identifier for heap. When allocating higher numbers -* will be allocated from first. At allocation these are passed +* will be allocated from first. At allocation these are passed * as a bit mask and therefore can not exceed ION_NUM_HEAP_IDS. * @name: used for debug purposes * @base: base address of heap in physical memory if applicable
drivers/staging/android/ion/ion_chunk_heap.c +0 -1
@@ -13,7 +13,6 @@* GNU General Public License for more details. * */ -//#include <linux/spinlock.h> #include <linux/dma-mapping.h> #include <linux/err.h> #include <linux/genalloc.h>
drivers/staging/android/ion/ion_heap.c +2 -3
@@ -49,9 +49,8 @@ void *ion_heap_map_kernel(struct ion_heap *heap,int npages_this_entry =PAGE_ALIGN (sg->length) /PAGE_SIZE ;struct page *page =sg_page (sg);BUG_ON (i >= npages); -for (j =0 ; j < npages_this_entry; j++) { +for (j =0 ; j < npages_this_entry; j++) *(tmp++) = page++; - } } vaddr =vmap (pages, npages,VM_MAP , pgprot);vfree (pages);
@@ -159,7 +158,7 @@ int ion_heap_pages_zero(struct page *page, size_t size, pgprot_t pgprot)return ion_heap_sglist_zero (&sg,1 , pgprot); } -void ion_heap_freelist_add (struct ion_heap *heap,struct ion_buffer * buffer) +void ion_heap_freelist_add (struct ion_heap *heap,struct ion_buffer *buffer) {rt_mutex_lock (&heap->lock);list_add (&buffer->list, &heap->free_list);
drivers/staging/android/ion/ion_page_pool.c +1 -1
@@ -108,7 +108,7 @@ void *ion_page_pool_alloc(struct ion_page_pool *pool)return page; } -void ion_page_pool_free (struct ion_page_pool *pool,struct page * page) +void ion_page_pool_free (struct ion_page_pool *pool,struct page *page) {int ret;
drivers/staging/android/ion/ion_priv.h +1 -1
@@ -230,7 +230,7 @@ int ion_heap_init_deferred_free(struct ion_heap *heap);/** * ion_heap_freelist_add - add a buffer to the deferred free list * @heap: the heap -* @buffer: the buffer +* @buffer: the buffer * * Adds an item to the deferred freelist. */
drivers/staging/android/ion/ion_system_heap.c +2 -1
@@ -148,7 +148,8 @@ static int ion_system_heap_allocate(struct ion_heap *heap,INIT_LIST_HEAD (&pages);while (size_remaining >0 ) { - info =alloc_largest_available (sys_heap, buffer, size_remaining, max_order); + info =alloc_largest_available (sys_heap, buffer, size_remaining, + max_order);if (!info) goto err;list_add_tail (&info->list, &pages);
Signature · 1 line
1.8.3.2
PATCH 114/115 ion: Improve ION config description
In reply to John StultzMostly just to quiet checkpatch warnings, be more verbose in describing the ION config option.
--- drivers/staging/android/ion/Kconfig | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/staging/android/ion/Kconfig +6 -2
@@ -2,8 +2,12 @@ menuconfig ION tristate "Ion Memory Manager" select GENERIC_ALLOCATOR select DMA_SHARED_BUFFER - help - Chose this option to enable the ION Memory Manager. + ---help--- + Chose this option to enable the ION Memory Manager, + used by Android to efficiently allocate buffers + from userspace that can be shared between drivers. + If you're not using Android its probably safe to + say N here. config ION_TEST tristate "Ion Test Device"
Signature · 1 line
1.8.3.2
PATCH 115/115 ion: Update system heap shrinker to use the new count/scan interface
In reply to John StultzUpdate the ION system heap shrinker to use the new count/scan interfaces that landed in 3.12
--- drivers/staging/android/ion/ion_system_heap.c | 40 +++++++++++++++++++-------- 1 file changed, 28 insertions(+), 12 deletions(-)
drivers/staging/android/ion/ion_system_heap.c +28 -12
@@ -232,15 +232,37 @@ static struct ion_heap_ops system_heap_ops = { .map_user = ion_heap_map_user, }; -static int ion_system_heap_shrink (struct shrinker *shrinker, -struct shrink_control *sc) { - +static unsigned long ion_system_heap_shrink_count (struct shrinker *shrinker, +struct shrink_control *sc) +{ struct ion_heap *heap = container_of(shrinker, struct ion_heap, shrinker);struct ion_system_heap *sys_heap =container_of (heap, struct ion_system_heap, heap);int nr_total =0 ; +int i; + +/* total number of items is whatever the page pools are holding +plus whatever's in the freelist */ +for (i =0 ; i < num_orders; i++) { +struct ion_page_pool *pool = sys_heap->pools[i]; + nr_total +=ion_page_pool_shrink (pool, sc->gfp_mask,0 ); + } + nr_total +=ion_heap_freelist_size (heap) /PAGE_SIZE ; +return nr_total; + +} + +static unsigned long ion_system_heap_shrink_scan (struct shrinker *shrinker, +struct shrink_control *sc) +{ + + struct ion_heap *heap = container_of(shrinker, struct ion_heap, + shrinker); +struct ion_system_heap *sys_heap =container_of (heap, + struct ion_system_heap, + heap);int nr_freed =0 ;int i;
@@ -265,14 +287,7 @@ static int ion_system_heap_shrink(struct shrinker *shrinker, } end: -/* total number of items is whatever the page pools are holding -plus whatever's in the freelist */ -for (i =0 ; i < num_orders; i++) { -struct ion_page_pool *pool = sys_heap->pools[i]; - nr_total +=ion_page_pool_shrink (pool, sc->gfp_mask,0 ); - } - nr_total +=ion_heap_freelist_size (heap) /PAGE_SIZE ; -return nr_total; +return nr_freed; }
@@ -323,7 +338,8 @@ struct ion_heap *ion_system_heap_create(struct ion_platform_heap *unused) heap->pools[i] = pool; } - heap->heap.shrinker.shrink = ion_system_heap_shrink; + heap->heap.shrinker.scan_objects = ion_system_heap_shrink_scan; + heap->heap.shrinker.count_objects = ion_system_heap_shrink_count; heap->heap.shrinker.seeks =DEFAULT_SEEKS ; heap->heap.shrinker.batch =0 ;register_shrinker (&heap->heap.shrinker);
Signature · 1 line
1.8.3.2