page allocator bug in 3.16?
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this x86_64 desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.
This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failed
This dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
Besides Mel's page allocator changes in 3.16, another suspect commit is:
commit b13b1d2d8692b437203de7a404c6b809d2cc4d99
Author: Shaohua Li <[email protected]> Date: Tue Apr 8 15:58:09 2014 +0800
x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB
Specifically, this statement:
It could cause incorrect page aging and the (mistaken) reclaim of hot pages, but the chance of that should be relatively low.
I'm wondering if this could cause worse-case behavior with TTM? I'm testing a revert of this on mainline 3.16-final now, with no results yet.
Thoughts?
Regards,
Peter Hurley
[1]
SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
CPU 4: hi: 0, btch: 1 usd: 0
CPU 5: hi: 0, btch: 1 usd: 0
CPU 6: hi: 0, btch: 1 usd: 0
CPU 7: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 18
CPU 1: hi: 186, btch: 31 usd: 82
CPU 2: hi: 186, btch: 31 usd: 46
CPU 3: hi: 186, btch: 31 usd: 30
CPU 4: hi: 186, btch: 31 usd: 18
CPU 5: hi: 186, btch: 31 usd: 43
CPU 6: hi: 186, btch: 31 usd: 157
CPU 7: hi: 186, btch: 31 usd: 26
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 25
CPU 1: hi: 186, btch: 31 usd: 33
CPU 2: hi: 186, btch: 31 usd: 28
CPU 3: hi: 186, btch: 31 usd: 46
CPU 4: hi: 186, btch: 31 usd: 23
CPU 5: hi: 186, btch: 31 usd: 8
CPU 6: hi: 186, btch: 31 usd: 112
CPU 7: hi: 186, btch: 31 usd: 18
active_anon:382833 inactive_anon:12103 isolated_anon:0
active_file:1156997 inactive_file:733988 isolated_file:0
unevictable:15 dirty:35833 writeback:0 unstable:0
free:129383 slab_reclaimable:95038 slab_unreclaimable:11095
mapped:81924 shmem:12509 pagetables:9039 bounce:0
free_cma:0
Node 0 DMA free:15860kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15960kB managed:15876kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2974 9980 9980
Node 0 DMA32 free:166712kB min:20108kB low:25132kB high:30160kB active_anon:475548kB inactive_anon:15204kB active_file:1368716kB inactive_file:865832kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3127336kB managed:3048188kB mlocked:0kB dirty:38228kB writeback:0kB mapped:94340kB shmem:15436kB slab_reclaimable:116424kB slab_unreclaimable:12756kB kernel_stack:2512kB pagetables:11532kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 7006 7006
Node 0 Normal free:334960kB min:47368kB low:59208kB high:71052kB active_anon:1055784kB inactive_anon:33208kB active_file:3259272kB inactive_file:2070120kB unevictable:60kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174484kB mlocked:60kB dirty:105104kB writeback:0kB mapped:233356kB shmem:34600kB slab_reclaimable:263728kB slab_unreclaimable:31608kB kernel_stack:7344kB pagetables:24624kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15860kB
Node 0 DMA32: 209*4kB (UEM) 394*8kB (UEM) 303*16kB (UEM) 60*32kB (UEM) 314*64kB (UEM) 117*128kB (UEM) 9*256kB (EM) 3*512kB (UEM) 2*1024kB (EM) 2*2048kB (UM) 27*4096kB (MR) = 166404kB
Node 0 Normal: 17*4kB (UE) 460*8kB (UEM) 747*16kB (UM) 130*32kB (UEM) 521*64kB (UM) 184*128kB (UEM) 70*256kB (UM) 22*512kB (UM) 11*1024kB (UM) 2*2048kB (EM) 52*4096kB (MR) = 334292kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
1903443 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 10996456kB
Total swap = 10996456kB
2620832 pages RAM
0 pages HighMem/MovableOnly
41387 pages reserved
0 pages hwpoisoned
Hey,
Quoted 17 lines · Peter Hurley
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
Quoted 7 lines
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failedThis dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
Besides Mel's page allocator changes in 3.16, another suspect commit is:
Maybe related, but I've been seeing page corruption in nouveau as well, with 3.15.9:
http://paste.debian.net/122800/
I think it might be an even older bug because I've been using nouveau on my desktop and it hasn't been stable for the past few releases. I'm also lazy with updating kernel, still do it from time to time.
The lookup and nvapeek warnings/crashes are not important btw, I was testing some nouveau things. The linker trap probably is. After the second BUG Xorg was no longer able to recover.
But this was after various suspend/resume cycles, although I suspect I've hit some corruption on radeon too (on a somewhat more recent kernel) when I fiddle with vgaswitcheroo, ending up with a real massive amount of spam there, etc.
Unfortunately I haven't been able to find out what caused it yet, nor am I sure what debug options I should set in the kernel to debug this.
~Maarten
Quoted 9 lines · On Thu, 25 Sep 2014 14:55:02 -0400 Peter Hurley <[email protected]>…
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.
Quoted 8 lines
This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 33 lines · Peter Hurley
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failedThis dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
Besides Mel's page allocator changes in 3.16, another suspect commit is:
commit
b13b1d2d8692b437203de7a404c6b809d2cc4d99Author: Shaohua Li <[email protected]> Date: Tue Apr 8 15:58:09 2014 +0800x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB
Specifically, this statement:
Quoted 7 lines
It could cause incorrect page aging and the (mistaken) reclaim of hot pages, but the chance of that should be relatively low.
I'm wondering if this could cause worse-case behavior with TTM? I'm testing a revert of this on mainline 3.16-final now, with no results yet.
Thoughts?
You may also be seeing this:
https://lkml.org/lkml/2014/8/8/445
Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Peter Hurley
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
Quoted 5 lines
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failedThis dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
It's swapping, which is crazy because there's 7+GB of file cache [1] which should be dropped before swapping.
The alloc_contig_range() failure precedes the swapping but not immediately (44 mins. earlier).
How I reproduce this is to simply do a full distro kernel build. Skipping the TLB flush is not the problem; the results below are from 3.16-final with that commit reverted.
The slowdown is really obvious because workspace switching redraw takes multiple seconds to complete (all-cpu perf record of that below [2])
Regards,
Peter Hurley
[1]
SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
CPU 4: hi: 0, btch: 1 usd: 0
CPU 5: hi: 0, btch: 1 usd: 0
CPU 6: hi: 0, btch: 1 usd: 0
CPU 7: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 71
CPU 1: hi: 186, btch: 31 usd: 166
CPU 2: hi: 186, btch: 31 usd: 183
CPU 3: hi: 186, btch: 31 usd: 109
CPU 4: hi: 186, btch: 31 usd: 106
CPU 5: hi: 186, btch: 31 usd: 161
CPU 6: hi: 186, btch: 31 usd: 120
CPU 7: hi: 186, btch: 31 usd: 54
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 159
CPU 1: hi: 186, btch: 31 usd: 66
CPU 2: hi: 186, btch: 31 usd: 178
CPU 3: hi: 186, btch: 31 usd: 173
CPU 4: hi: 186, btch: 31 usd: 91
CPU 5: hi: 186, btch: 31 usd: 57
CPU 6: hi: 186, btch: 31 usd: 58
CPU 7: hi: 186, btch: 31 usd: 158
active_anon:170368 inactive_anon:173964 isolated_anon:0
active_file:982209 inactive_file:973911 isolated_file:0
unevictable:15 dirty:15 writeback:1 unstable:0
free:96067 slab_reclaimable:107401 slab_unreclaimable:12572
mapped:58271 shmem:10857 pagetables:9898 bounce:0
free_cma:18
Node 0 DMA free:15860kB min:104kB low:128kB high:156kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15960kB managed:15876kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2974 9980 9980
Node 0 DMA32 free:117740kB min:20108kB low:25132kB high:30160kB active_anon:205232kB inactive_anon:196308kB active_file:1186764kB inactive_file:1173760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3127336kB managed:3048212kB mlocked:0kB dirty:24kB writeback:4kB mapped:71600kB shmem:8776kB slab_reclaimable:129132kB slab_unreclaimable:13468kB kernel_stack:2864kB pagetables:11536kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 7006 7006
Node 0 Normal free:250668kB min:47368kB low:59208kB high:71052kB active_anon:476240kB inactive_anon:499548kB active_file:2742072kB inactive_file:2721884kB unevictable:60kB isolated(anon):0kB isolated(file):0kB present:7340032kB managed:7174484kB mlocked:60kB dirty:36kB writeback:0kB mapped:161484kB shmem:34652kB slab_reclaimable:300472kB slab_unreclaimable:36804kB kernel_stack:7232kB pagetables:28056kB unstable:0kB bounce:0kB free_cma:72kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15860kB
Node 0 DMA32: 4099*4kB (UEM) 4372*8kB (UEM) 668*16kB (UEM) 294*32kB (UEM) 47*64kB (UEM) 24*128kB (UEM) 19*256kB (UM) 5*512kB (UM) 0*1024kB 6*2048kB (M) 5*4096kB (M) = 117740kB
Node 0 Normal: 22224*4kB (UEMC) 8120*8kB (UEMC) 1594*16kB (UEMC) 301*32kB (UEMC) 154*64kB (UMC) 106*128kB (UEMC) 86*256kB (UMC) 13*512kB (UEMC) 3*1024kB (M) 3*2048kB (M) 1*4096kB (R) = 254400kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
1967616 total pagecache pages
637 pages in swap cache
Swap cache stats: add 9593, delete 8956, find 2162/2929
Free swap = 10966116kB
Total swap = 10996456kB
2620832 pages RAM
0 pages HighMem/MovableOnly
41387 pages reserved
0 pages hwpoisoned
[2] 'perf record -a -g' while workspace switching
Samples: 27K of event 'cycles', Event count (approx.): 15793770075
+ 87.11% 0.00% Xorg [kernel.kallsyms] [k] tracesys + 86.81% 0.00% Xorg [unknown] [k] 0x00007fca2df34e77 + 86.77% 0.00% Xorg [kernel.kallsyms] [k] sys_ioctl + 86.77% 0.00% Xorg [kernel.kallsyms] [k] do_vfs_ioctl + 86.77% 0.00% Xorg [nouveau] [k] nouveau_drm_ioctl + 86.77% 0.00% Xorg [drm] [k] drm_ioctl + 86.66% 0.00% Xorg [nouveau] [k] nouveau_gem_ioctl_new + 86.50% 0.00% Xorg [nouveau] [k] nouveau_gem_new + 86.49% 0.00% Xorg [nouveau] [k] nouveau_bo_new + 86.48% 0.00% Xorg [ttm] [k] ttm_bo_init + 86.47% 0.00% Xorg [ttm] [k] ttm_bo_validate + 86.46% 0.00% Xorg [ttm] [k] ttm_bo_handle_move_mem + 86.45% 0.00% Xorg [ttm] [k] ttm_tt_bind + 86.45% 0.00% Xorg [nouveau] [k] nouveau_ttm_tt_populate + 86.45% 0.00% Xorg [ttm] [k] ttm_dma_populate + 86.43% 0.01% Xorg [ttm] [k] ttm_dma_pool_alloc_new_pages + 86.42% 0.00% Xorg [kernel.kallsyms] [k] x86_swiotlb_alloc_coherent + 86.37% 0.00% Xorg [kernel.kallsyms] [k] dma_generic_alloc_coherent + 86.19% 0.00% Xorg [unknown] [k] 0x0000000000c00000 + 85.82% 0.31% Xorg [kernel.kallsyms] [k] dma_alloc_from_contiguous + 84.21% 1.05% Xorg [kernel.kallsyms] [k] alloc_contig_range + 46.56% 46.56% Xorg [kernel.kallsyms] [k] move_freepages + 46.53% 0.29% Xorg [kernel.kallsyms] [k] move_freepages_block + 39.78% 0.13% Xorg [kernel.kallsyms] [k] start_isolate_page_range + 39.22% 0.40% Xorg [kernel.kallsyms] [k] set_migratetype_isolate + 27.26% 0.17% Xorg [kernel.kallsyms] [k] undo_isolate_page_range + 26.62% 0.33% Xorg [kernel.kallsyms] [k] unset_migratetype_isolate + 15.54% 7.93% Xorg [kernel.kallsyms] [k] drain_all_pages
Quoted 37 lines · Alex Deucher
Quoted 36 lines · Peter Hurley
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failedThis dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
Besides Mel's page allocator changes in 3.16, another suspect commit is:
commit
b13b1d2d8692b437203de7a404c6b809d2cc4d99Author: Shaohua Li <[email protected]> Date: Tue Apr 8 15:58:09 2014 +0800x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB
Specifically, this statement:
It could cause incorrect page aging and the (mistaken) reclaim of hot pages, but the chance of that should be relatively low.
Quoted 7 lines
Quoted 4 lines
I'm wondering if this could cause worse-case behavior with TTM? I'm testing a revert of this on mainline 3.16-final now, with no results yet.
Thoughts?
You may also be seeing this:
https://lkml.org/lkml/2014/8/8/445
Thanks Alex. That is indeed the problem.
Still reading the email thread to find out where the patches are that fix this. Although it doesn't make much sense to me that nouveau sets up a 1GB GART and then uses TTM which is trying to shove all the DMA through a 16MB CMA window (which turns out to be the base Ubuntu config).
Regards,
Peter Hurley
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards,
Peter Hurley
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Quoted 7 lines
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards,
Peter Hurley
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
/Thomas
Quoted 15 lines · On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]>…
Quoted 14 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Quoted 7 lines
Quoted 2 lines
Regards,
Peter HurleyThe problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 18 lines · Chuck Ebbert
Quoted 15 lines · On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]>…
Quoted 14 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Regards,
Peter HurleyThe problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
/Thomas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 21 lines · Thomas Hellstrom
Quoted 20 lines · Chuck Ebbert
Quoted 17 lines · On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]>…
Quoted 16 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards,
Peter Hurley
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
BR,
-R
Quoted 6 lines
/Thomas
_______________________________________________
dri-devel mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 26 lines · Rob Clark
Quoted 25 lines · Thomas Hellstrom
Quoted 24 lines · Chuck Ebbert
Quoted 21 lines · On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]>…
Quoted 9 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Quoted 7 lines
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards,
Peter HurleyThe problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
But can the DMA subsystem or more specifically dma_alloc_coherent() really guarantee such things? Isn't it better for such devices to use CMA directly?
/Thomas
Quoted 14 lines
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
BR,
-RQuoted 6 lines
/Thomas
_______________________________________________
dri-devel mailing list
[email protected]
https://urldefense.proofpoint.com/v1/url?u=http://lists.freedesktop.org/mailman/listinfo/dri-devel&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=l5Ago9ekmVFZ3c4M6eauqrJWGwjf6fTb%2BP3CxbBFkVM%3D%0A&m=Uz7JXDXYXp4RlLs7G6qxMQlhOOT0trW3l78xpKg6Ass%3D%0A&s=50d6b7b3bfd093c93a228437a3d4414e49b4de817657c49c35154a115a5c2188
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 6 lines · Rob Clark
Quoted 4 lines · Thomas Hellstrom
Quoted 3 lines · Chuck Ebbert
On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Peter Hurley
Quoted 6 lines · Chuck Ebbert
Quoted 5 lines
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards, Peter Hurley
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Quoted 5 lines
Quoted 3 lines
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
Quoted 6 lines
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
CMA has its uses on x86. For example, CMA is used to allocate 1GB huge pages.
There may also be people with devices that do not scatter-gather, and need a large physically contiguous buffer, though there should be relatively few of those on x86.
I suspect it makes most sense to do DMA allocations up to PAGE_ORDER through the normal allocator on x86, and only invoking CMA for larger allocations.
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJUJV6lAAoJEM553pKExN6DjrQH/1N19Hp5FrJr+a3GpQDh6ouc YSBChxe+wG1h3OmcFGAG69tOK9XPw0oaV77ohwLxnvSv6BQZyi2CUIJvUdgSaOOx 8XYnn8VIIlMn4IKYmraAhSWT/gm3FkyDW7tckEdLV0NsrKeUcavCRHcLXxh41OBw XJboZyS3XvwF+scAwjHpWPxby1Byi0lZJizTAzI3xdlyVaM5Lio1xLvOW2MHY7dR h/ai8mfAAdQvnaHsFLoypBM/xYJqaUVU8IyCzhOeO86dUMy2xhD4vm/f9vSLuOju 4VYf7POuziNo1q2vJ8YcrThsAjB0Oiu9B5nDar471G3l1xN1zHQVw/RAnpNF9Kk= =iZlM
-----END PGP SIGNATURE-----
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 31 lines · Thomas Hellstrom
Quoted 30 lines · Rob Clark
Quoted 29 lines · Thomas Hellstrom
Quoted 18 lines · Chuck Ebbert
Quoted 15 lines · On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]>…
Quoted 14 lines · Peter Hurley
Quoted 8 lines · Chuck Ebbert
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Regards,
Peter HurleyThe problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
Quoted 7 lines
Quoted 3 lines
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
But can the DMA subsystem or more specifically
dma_alloc_coherent()really guarantee such things? Isn't it better for such devices to use CMA directly?
Well, I was thinking more specifically about a use-case that was mentioned several times during the early CMA discussions, about video decoders/encoders which needed Y and UV split across memory banks to achieve sufficient bandwidth. I assume they must use CMA directly for this (since they'd need multiple pools per device), but not really 100% sure about that.
So perhaps, yeah, if you shunt order 0 allocations away from CMA at the DMA layer, maybe it is ok. If there actually is a valid use-case for CMA on sane hardware, then maybe this is the better way, and let the insane hw folks hack around it.
(plus, well, the use-case I was mentioning isn't really about order 0 allocations anyway)
BR,
-R
Quoted 18 lines
/Thomas
Quoted 14 lines
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
BR,
-RQuoted 6 lines
/Thomas
_______________________________________________
dri-devel mailing list
[email protected]
https://urldefense.proofpoint.com/v1/url?u=http://lists.freedesktop.org/mailman/listinfo/dri-devel&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=l5Ago9ekmVFZ3c4M6eauqrJWGwjf6fTb%2BP3CxbBFkVM%3D%0A&m=Uz7JXDXYXp4RlLs7G6qxMQlhOOT0trW3l78xpKg6Ass%3D%0A&s=50d6b7b3bfd093c93a228437a3d4414e49b4de817657c49c35154a115a5c2188
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Peter Hurley
Acked-by Arnd Bergmann[ +cc Leann Ogasawara, Marek Szyprowski, Kyungmin Park, Arnd Bergmann ]
Quoted 49 lines · Rik van Riel
Quoted 6 lines · Rob Clark
Quoted 4 lines · Thomas Hellstrom
Quoted 3 lines · Chuck Ebbert
On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Peter Hurley
Quoted 6 lines · Chuck Ebbert
Quoted 5 lines
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards, Peter Hurley
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Quoted 5 lines
Quoted 3 lines
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
Quoted 6 lines
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
CMA has its uses on x86. For example, CMA is used to allocate 1GB huge pages.
Quoted 7 lines
There may also be people with devices that do not scatter-gather, and need a large physically contiguous buffer, though there should be relatively few of those on x86.
I suspect it makes most sense to do DMA allocations up to
PAGE_ORDERthrough the normal allocator on x86, and only invoking CMA for larger allocations.
The code that uses CMA to satisfy DMA allocations on x86 is specific to the x86 arch and was added in 2011 as a means of _testing_ CMA in KVM:
commit 0a2b9a6ea93650b8a00f9fd5ee8fdd25671e2df6
Author: Marek Szyprowski <[email protected]> Date: Thu Dec 29 13:09:51 2011 +0100
X86: integrate CMA with DMA-mapping subsystem
This patch adds support for CMA to dma-mapping subsystem for x86 architecture that uses common pci-dma/pci-nommu implementation. This allows to test CMA on KVM/QEMU and a lot of common x86 boxes.
(no x86 maintainer acks?).
Unfortunately, this code is enabled whenever CMA is enabled, rather than as a separate test configuration.
So, while enabling CMA may have other purposes on x86, using it for x86 swiotlb and nommu dma allocations is not one of the them.
And Ubuntu should not be enabling CONFIG_DMA_CMA for their i386 and amd64 configurations, as this is trying to drive _all_ dma mapping allocations through a _very_ small window (which is killing GPU performance).
Regards,
Peter Hurley
Quoted 84 lines · Peter Hurley
[ +cc Leann Ogasawara, Marek Szyprowski, Kyungmin Park, Arnd Bergmann ]
Quoted 49 lines · Rik van Riel
Quoted 6 lines · Rob Clark
Quoted 4 lines · Thomas Hellstrom
Quoted 3 lines · Chuck Ebbert
On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Peter Hurley
Quoted 6 lines · Chuck Ebbert
Quoted 5 lines
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards, Peter Hurley
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Quoted 5 lines
Quoted 3 lines
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
Quoted 6 lines
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
CMA has its uses on x86. For example, CMA is used to allocate 1GB huge pages.
Quoted 7 lines
There may also be people with devices that do not scatter-gather, and need a large physically contiguous buffer, though there should be relatively few of those on x86.
I suspect it makes most sense to do DMA allocations up to
PAGE_ORDERthrough the normal allocator on x86, and only invoking CMA for larger allocations.The code that uses CMA to satisfy DMA allocations on x86 is specific to the x86 arch and was added in 2011 as a means of _testing_ CMA in KVM:
commit
0a2b9a6ea93650b8a00f9fd5ee8fdd25671e2df6Author: Marek Szyprowski <[email protected]> Date: Thu Dec 29 13:09:51 2011 +0100X86: integrate CMA with DMA-mapping subsystem
This patch adds support for CMA to dma-mapping subsystem for x86 architecture that uses common pci-dma/pci-nommu implementation. This allows to test CMA on KVM/QEMU and a lot of common x86 boxes.
Signed-off-by Marek Szyprowski <[email protected]>Signed-off-by Kyungmin Park <[email protected]>CC Michal Nazarewicz <[email protected]>Acked-by Arnd Bergmann <[email protected]>(no x86 maintainer acks?).
Unfortunately, this code is enabled whenever CMA is enabled, rather than as a separate test configuration.
Quoted 7 lines
So, while enabling CMA may have other purposes on x86, using it for x86 swiotlb and nommu dma allocations is not one of the them.
And Ubuntu should not be enabling
CONFIG_DMA_CMAfor their i386 and amd64 configurations, as this is trying to drive _all_ dma mapping allocations through a _very_ small window (which is killing GPU performance).
Thanks for the note Peter. We do have this disabled for our upcoming Ubuntu 14.10 release. It is however still enabled in the previous 14.04 release. We have been tracking this in
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1362261 but users able to reproduce performance impacts in 14.10 were unable to reproduce in 14.04 which is why we hadn't yet disabled it there.
Thanks,
Leann
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 93 lines · Leann Ogasawara
Quoted 84 lines · Peter Hurley
[ +cc Leann Ogasawara, Marek Szyprowski, Kyungmin Park, Arnd Bergmann ]
Quoted 49 lines · Rik van Riel
Quoted 6 lines · Rob Clark
Quoted 4 lines · Thomas Hellstrom
Quoted 3 lines · Chuck Ebbert
On Fri, 26 Sep 2014 09:15:57 +0200 Thomas Hellstrom <[email protected]> wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Peter Hurley
Quoted 6 lines · Chuck Ebbert
Quoted 5 lines
There are six ttm patches queued for 3.16.4:
drm-ttm-choose-a-pool-to-shrink-correctly-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-handling-of-ttm_pl_flag_topdown-v2.patch
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drm-ttm-fix-possible-division-by-0-in-ttm_dma_pool_shrink_scan.patch
drm-ttm-fix-possible-stack-overflow-by-recursive-shrinker-calls.patch
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drm-ttm-pass-gfp-flags-in-order-to-avoid-deadlock.patch drm-ttm-use-mutex_trylock-to-avoid-deadlock-inside-shrinker-functions.patch
Thanks for info, Chuck.
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Unfortunately, none of these fix TTM dma allocation doing CMA dma allocation, which is the root problem.
Regards, Peter Hurley
The problem is not really in TTM but in CMA, There was a guy offering to fix this in the CMA code but I guess he didn't probably because he didn't receive any feedback.
Quoted 5 lines
Quoted 3 lines
Yeah, the "solution" to this problem seems to be "don't enable CMA on x86". Maybe it should even be disabled in the config system.
Or, as previously suggested, don't use CMA for order 0 (single page) allocations....
Quoted 6 lines
On devices that actually need CMA pools to arrange for memory to be in certain ranges, I think you probably do want to have order 0 pages come from the CMA pool.
Seems like disabling CMA on x86 (where it should be unneeded) is the better way, IMO
CMA has its uses on x86. For example, CMA is used to allocate 1GB huge pages.
Quoted 7 lines
There may also be people with devices that do not scatter-gather, and need a large physically contiguous buffer, though there should be relatively few of those on x86.
I suspect it makes most sense to do DMA allocations up to
PAGE_ORDERthrough the normal allocator on x86, and only invoking CMA for larger allocations.The code that uses CMA to satisfy DMA allocations on x86 is specific to the x86 arch and was added in 2011 as a means of _testing_ CMA in KVM:
commit
0a2b9a6ea93650b8a00f9fd5ee8fdd25671e2df6Author: Marek Szyprowski <[email protected]> Date: Thu Dec 29 13:09:51 2011 +0100X86: integrate CMA with DMA-mapping subsystem
This patch adds support for CMA to dma-mapping subsystem for x86 architecture that uses common pci-dma/pci-nommu implementation. This allows to test CMA on KVM/QEMU and a lot of common x86 boxes.
Signed-off-by Marek Szyprowski <[email protected]>Signed-off-by Kyungmin Park <[email protected]>CC Michal Nazarewicz <[email protected]>Acked-by Arnd Bergmann <[email protected]>(no x86 maintainer acks?).
Unfortunately, this code is enabled whenever CMA is enabled, rather than as a separate test configuration.
Quoted 7 lines
So, while enabling CMA may have other purposes on x86, using it for x86 swiotlb and nommu dma allocations is not one of the them.
And Ubuntu should not be enabling
CONFIG_DMA_CMAfor their i386 and amd64 configurations, as this is trying to drive _all_ dma mapping allocations through a _very_ small window (which is killing GPU performance).
Quoted 6 lines
Thanks for the note Peter. We do have this disabled for our upcoming Ubuntu 14.10 release. It is however still enabled in the previous 14.04 release. We have been tracking this in
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1362261 but users able to reproduce performance impacts in 14.10 were unable to reproduce in 14.04 which is why we hadn't yet disabled it there.
Leann,
Thanks for that important clue.
The missing piece specific to 3.16+ is these patches which impact every iommu config:
Akinobu Mita (5):
x86: make dma_alloc_coherent() return zeroed memory if CMA is enabled
x86: enable DMA CMA with swiotlb
intel-iommu: integrate DMA CMA
memblock: introduce memblock_alloc_range()
cma: add placement specifier for "cma=" kernel parameter
These patches take the pre-existing nommu CMA test configuration and hook it up to all the x86 iommus, effectively reducing 10GB of DMA-able memory to 64MB, and hooks it all up to an allocator that's not nearly as effective as the page allocator.
All to enable DMA allocation below 4GB which is already supported with the GFP_DMA32 flag to dma_alloc_coherent().
Regards,
Peter Hurley
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Op 25-09-14 om 23:10 schreef Peter Hurley:
Quoted 46 lines · Alex Deucher
Quoted 37 lines · Peter Hurley
Quoted 36 lines
After several days uptime with a 3.16 kernel (generally running Thunderbird, emacs, kernel builds, several Chrome tabs on multiple desktop workspaces) I've been seeing some really extreme slowdowns.
Mostly the slowdowns are associated with gpu-related tasks, like opening new emacs windows, switching workspaces, laughing at internet gifs, etc. Because this
x86_64desktop is nouveau-based, I didn't pursue it right away -- 3.15 is the first time suspend has worked reliably.This week I started looking into what the slowdown was and discovered it's happening during dma allocation through swiotlb (the cpus can do intel iommu but I don't use it because it's not the default for most users).
I'm still working on a bisection but each step takes 8+ hours to validate and even then I'm no longer sure I still have the 'bad' commit in the bisection. [edit: yup, I started over]
I just discovered a smattering of these in my logs and only on 3.16-rc+ kernels:
Sep 25 07:57:59 thor kernel: [28786.001300] alloc_contig_range test_pages_isolated(2bf560, 2bf562) failedThis dual-Xeon box has 10GB and sysrq Show Memory isn't showing heavy fragmentation [1].
Besides Mel's page allocator changes in 3.16, another suspect commit is:
commit
b13b1d2d8692b437203de7a404c6b809d2cc4d99Author: Shaohua Li <[email protected]> Date: Tue Apr 8 15:58:09 2014 +0800x86/mm: In the PTE swapout page reclaim case clear the accessed bit instead of flushing the TLB
Specifically, this statement:
It could cause incorrect page aging and the (mistaken) reclaim of hot pages, but the chance of that should be relatively low.
Quoted 6 lines
Quoted 4 lines
I'm wondering if this could cause worse-case behavior with TTM? I'm testing a revert of this on mainline 3.16-final now, with no results yet.
Thoughts?
You may also be seeing this:
https://lkml.org/lkml/2014/8/8/445Thanks Alex. That is indeed the problem.
Quoted 10 lines
Still reading the email thread to find out where the patches are that fix this. Although it doesn't make much sense to me that nouveau sets up a 1GB GART and then uses TTM which is trying to shove all the DMA through a 16MB CMA window (which turns out to be the base Ubuntu config).
Regards,
Peter Hurley
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1362261
CMA's already disabled on x86 in most recent ubuntu kernels. :-)
~Maarten
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>