prevent OOM triggered by TTM
Subsystems (7)
cover prevent OOM triggered by TTM
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
the following patches is for prevent OOM triggered by TTM. the basic idea is when allocating TTM pages, check the free swap space firt. if it is less than the fixe limit, reject the allocation request. but there are two exceptions which should allow it regardless of zone memory account limit.
a. page fault
for ttm_mem_global_reserve if serving for page fault routine, because page fault routing already grabbed system memory so the allowance of this exception is harmless. Otherwise, it will trigger
OOM killer.
b. suspend
anyway, we should allow suspend success always.
at last, if bdev.no_retry is false (by defaut), keep the original behavior no any change.
Roger He (5):
drm/ttm: check if the free swap space is under limit 256MB drm/ttm: keep original behavior except with flag no_retry drm/ttm: use bit flag to replace allow_reserved_eviction in
ttm_operation_ctx
drm/ttm: add bit flag TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY drm/ttm: add input parameter allow_allo_anyway for ttm_bo_evict_mm
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 10 +++--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 +++-- drivers/gpu/drm/nouveau/nouveau_drm.c | 2 +- drivers/gpu/drm/qxl/qxl_object.c | 4 +-- drivers/gpu/drm/radeon/radeon_device.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.h | 3 +- drivers/gpu/drm/ttm/ttm_bo.c | 19 +++++++---- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++-- drivers/gpu/drm/ttm/ttm_memory.c | 51 ++++++++++++++++++++++++++--- drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 1 - drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 ++-- include/drm/ttm/ttm_bo_api.h | 14 ++++++-- include/drm/ttm/ttm_memory.h | 6 ++++ 18 files changed, 111 insertions(+), 43 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): 1517907874-21248-1-git-send-email-Hongbo.He-5C7GfCeVMHo@public.gmane.org
PATCH 1/5 drm/ttm: check if the free swap space is under limit 256MB
for avoid OOM. if free swap space is less than 256MB, reject the TTM page allocation. Otherwise, swap space will be full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
to cover two cases:
a. If total swap space > 256MB and at that time free swap
space is under 256MB but available system mem > its lower limit, allow TTM allocation;
b. if total swap space < 256 or no swap disk at all, check
the available system mem, if it is bigger than its threshold, allow TTM allocation.
--- drivers/gpu/drm/ttm/ttm_memory.c | 31 ++++++++++++++++++++++++++++++- include/drm/ttm/ttm_memory.h | 3 +++ 2 files changed, 33 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_memory.c +30 -1
@@ -36,6 +36,7 @@#include <linux/mm.h> #include <linux/module.h> #include <linux/slab.h> +#include <linux/swap.h> #define TTM_MEMORY_ALLOC_RETRIES 4
@@ -374,7 +375,8 @@ int ttm_mem_global_init(struct ttm_mem_global *glob) }si_meminfo (&si); - +/* keep consistent with each zone->emer_mem */ + glob->sys_mem_limit = si.totalram >>2 ; ret =ttm_mem_init_kernel_zone (glob, &si);if (unlikely (ret !=0 )) goto out_no_zone;
@@ -469,6 +471,30 @@ void ttm_mem_global_free(struct ttm_mem_global *glob, }EXPORT_SYMBOL (ttm_mem_global_free); + +#define FREE_SWAP_SPACE (256 << 8) +/* +* check if the free swap space is under limit 256MB. +* +* a. if free swap space is under 256MB but available system mem is +* bigger than its lower limit, allow TTM allocation; +* +* b. if no swap disk at all or less than 256MB at the beginning, check +* the available system mem, if bigger than its threshold, allow TTM +* allocation. +*/ +static bool +ttm_check_over_swaplimit (struct ttm_mem_global *glob) +{ +bool ret = false; + +if (get_nr_swap_pages () <FREE_SWAP_SPACE + &&si_mem_available () < glob->sys_mem_limit) + ret = true; + +return ret; +} +static int ttm_mem_global_reserve (struct ttm_mem_global *glob,struct ttm_mem_zone *single_zone,uint64_t amount,bool reserve)
@@ -478,6 +504,9 @@ static int ttm_mem_global_reserve(struct ttm_mem_global *glob,unsigned int i;struct ttm_mem_zone *zone; +if (ttm_check_over_swaplimit (glob)) +return ret; +spin_lock (&glob->lock);for (i =0 ; i < glob->num_zones; ++i) { zone = glob->zones[i];
include/drm/ttm/ttm_memory.h +3 -0
@@ -49,6 +49,8 @@* @work: The workqueue callback for the shrink queue. * @lock: Lock to protect the @shrink - and the memory accounting members, * that is, essentially the whole structure with some exceptions. +* @sys_mem_limit: if free ram is under this threshold and free swap space is +* less than its limit, reject any TTM allocation request * @zones: Array of pointers to accounting zones. * @num_zones: Number of populated entries in the @zones array. * @zone_kernel: Pointer to the kernel zone.
@@ -67,6 +69,7 @@ struct ttm_mem_global {struct workqueue_struct *swap_queue;struct work_struct work;spinlock_t lock; +uint64_t sys_mem_limit;struct ttm_mem_zone *zones[TTM_MEM_MAX_ZONES ];unsigned int num_zones;struct ttm_mem_zone *zone_kernel;
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 2/5 drm/ttm: keep original behavior except with flag no_retry
In reply to a message not in the archiveset the no_retry flag in struct ttm_mem_global and init it after ttm_mem_global_init
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 +++++--- drivers/gpu/drm/ttm/ttm_memory.c | 3 +++ include/drm/ttm/ttm_memory.h | 3 +++ 3 files changed, 11 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5 -3
@@ -75,6 +75,7 @@ static void amdgpu_ttm_mem_global_release(struct drm_global_reference *ref)static int amdgpu_ttm_global_init (struct amdgpu_device *adev) {struct drm_global_reference *global_ref; +struct ttm_mem_global *mem_glob;struct amdgpu_ring *ring;struct drm_sched_rq *rq;int r;
@@ -91,6 +92,8 @@ static int amdgpu_ttm_global_init(struct amdgpu_device *adev) "subsystem.\n"); goto error_mem; } + mem_glob = (struct ttm_mem_global *)global_ref->object; + mem_glob->no_retry = adev->mman.bdev.no_retry; adev->mman.bo_global_ref.mem_glob = adev->mman.mem_global_ref.object;
@@ -1371,6 +1374,8 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)int r;u64 vis_vram_limit; +/* We opt to avoid OOM on system pages allocations */ + adev->mman.bdev.no_retry = true; r =amdgpu_ttm_global_init (adev);if (r) {return r;
@@ -1388,9 +1393,6 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) } adev->mman.initialized = true; -/* We opt to avoid OOM on system pages allocations */ - adev->mman.bdev.no_retry = true; - r =ttm_bo_init_mm (&adev->mman.bdev,TTM_PL_VRAM , adev->gmc.real_vram_size >>PAGE_SHIFT );if (r) {
drivers/gpu/drm/ttm/ttm_memory.c +3 -0
@@ -488,6 +488,9 @@ ttm_check_over_swaplimit(struct ttm_mem_global *glob) {bool ret = false; +if (!glob->no_retry) +return ret; +if (get_nr_swap_pages () <FREE_SWAP_SPACE &&si_mem_available () < glob->sys_mem_limit) ret = true;
include/drm/ttm/ttm_memory.h +3 -0
@@ -56,6 +56,8 @@* @zone_kernel: Pointer to the kernel zone. * @zone_highmem: Pointer to the highmem zone if there is one. * @zone_dma32: Pointer to the dma32 zone if there is one. +* @no_retry: if free swap space is under the fixed limit, no retry for +* avoid OOM * * Note that this structure is not per device. It should be global for all * graphics devices.
@@ -78,6 +80,7 @@ struct ttm_mem_global {#else struct ttm_mem_zone *zone_dma32;#endif +bool no_retry; };extern int ttm_mem_global_init (struct ttm_mem_global *glob);
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 3/5 drm/ttm: use bit flag to replace allow_reserved_eviction in ttm_operation_ctx
In reply to a message not in the archivefor saving memory and more bit flag can be used in future
--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 4 ++-- drivers/gpu/drm/ttm/ttm_bo.c | 3 ++- include/drm/ttm/ttm_bo_api.h | 7 +++++-- 4 files changed, 11 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +2 -2
@@ -346,8 +346,8 @@ static int amdgpu_cs_bo_validate(struct amdgpu_cs_parser *p,struct ttm_operation_ctx ctx = { .interruptible = true, .no_wait_gpu = false, - .allow_reserved_eviction = false, - .resv = bo->tbo.resv + .resv = bo->tbo.resv, + .flags =0 };uint32_t domain;int r;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +2 -2
@@ -341,8 +341,8 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev,struct ttm_operation_ctx ctx = { .interruptible = !kernel, .no_wait_gpu = false, - .allow_reserved_eviction = true, - .resv = resv + .resv = resv, + .flags =TTM_OPT_FLAG_ALLOW_RES_EVICT };struct amdgpu_bo *bo;enum ttm_bo_type type;
drivers/gpu/drm/ttm/ttm_bo.c +2 -1
@@ -730,7 +730,8 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo, *locked = false;if (bo->resv == ctx->resv) {reservation_object_assert_held (bo->resv); -if (ctx->allow_reserved_eviction || !list_empty (&bo->ddestroy)) +if (ctx->flags &TTM_OPT_FLAG_ALLOW_RES_EVICT + || !list_empty (&bo->ddestroy)) ret = true; }else { *locked =reservation_object_trylock (bo->resv);
include/drm/ttm/ttm_bo_api.h +5 -2
@@ -263,8 +263,8 @@ struct ttm_bo_kmap_obj {* * @interruptible: Sleep interruptible if sleeping. * @no_wait_gpu: Return immediately if the GPU is busy. -* @allow_reserved_eviction: Allow eviction of reserved BOs. * @resv: Reservation object to allow reserved evictions with. +* @flags: Including the following flags * * Context for TTM operations like changing buffer placement or general memory * allocation.
@@ -272,11 +272,14 @@ struct ttm_bo_kmap_obj {struct ttm_operation_ctx {bool interruptible;bool no_wait_gpu; -bool allow_reserved_eviction;struct reservation_object *resv;uint64_t bytes_moved; +uint32_t flags; }; +/* Allow eviction of reserved BOs */ +#define TTM_OPT_FLAG_ALLOW_RES_EVICT 0x1 +/** * ttm_bo_reference - reference a struct ttm_buffer_object *
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 4/5 drm/ttm: add bit flag TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY
In reply to a message not in the archiveset TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY when we are servicing for page fault routine.
for ttm_mem_global_reserve if in page fault routine, allow the gtt pages reservation always. because page fault routing already grabbed system memory and the allowance of this exception is harmless. Otherwise, it will trigger OOM killer.
v2: keep original behavior except ttm bo with flag no_retry
--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++++-- drivers/gpu/drm/ttm/ttm_memory.c | 25 +++++++++++++++++-------- drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 1 - include/drm/ttm/ttm_bo_api.h | 4 +++- 4 files changed, 24 insertions(+), 12 deletions(-)
drivers/gpu/drm/ttm/ttm_bo_vm.c +4 -2
@@ -224,7 +224,7 @@ static int ttm_bo_vm_fault(struct vm_fault *vmf) cvma.vm_page_prot =ttm_io_prot (bo->mem.placement, cvma.vm_page_prot); }else { -struct ttm_operation_ctx ctx = { +struct ttm_operation_ctx ttm_opt_ctx = { .interruptible = false, .no_wait_gpu = false };
@@ -233,8 +233,10 @@ static int ttm_bo_vm_fault(struct vm_fault *vmf) cvma.vm_page_prot =ttm_io_prot (bo->mem.placement, cvma.vm_page_prot); +if (ttm->page_flags &TTM_PAGE_FLAG_NO_RETRY ) + ttm_opt_ctx.flags |=TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY ;/* Allocate all page at once, most common usage */ -if (ttm->bdev->driver->ttm_tt_populate (ttm, &ctx)) { +if (ttm->bdev->driver->ttm_tt_populate (ttm, &ttm_opt_ctx)) { ret =VM_FAULT_OOM ; goto out_io_unlock; }
drivers/gpu/drm/ttm/ttm_memory.c +17 -8
@@ -488,9 +488,6 @@ ttm_check_over_swaplimit(struct ttm_mem_global *glob) {bool ret = false; -if (!glob->no_retry) -return ret; -if (get_nr_swap_pages () <FREE_SWAP_SPACE &&si_mem_available () < glob->sys_mem_limit) ret = true;
@@ -500,17 +497,28 @@ ttm_check_over_swaplimit(struct ttm_mem_global *glob)static int ttm_mem_global_reserve (struct ttm_mem_global *glob,struct ttm_mem_zone *single_zone, -uint64_t amount,bool reserve) +uint64_t amount,bool reserve, +bool allow_alloc_anyway) {uint64_t limit;int ret = -ENOMEM ;unsigned int i;struct ttm_mem_zone *zone; -if (ttm_check_over_swaplimit (glob)) +if (glob->no_retry && !allow_alloc_anyway + &&ttm_check_over_swaplimit (glob))return ret;spin_lock (&glob->lock); +/* +* to cover two special cases: +* a. if serving page_fault allow reservation anyway since +* it already allocated system pages. Otherwise it will trigger OOM. +* b. if serving suspend, allow reservation anyway as well. +*/ +if (glob->no_retry && allow_alloc_anyway) + goto reserve_direct; +for (i =0 ; i < glob->num_zones; ++i) { zone = glob->zones[i];if (single_zone && zone != single_zone)
@@ -523,6 +531,7 @@ static int ttm_mem_global_reserve(struct ttm_mem_global *glob, goto out_unlock; } +reserve_direct:if (reserve) {for (i =0 ; i < glob->num_zones; ++i) { zone = glob->zones[i];
@@ -547,10 +556,10 @@ static int ttm_mem_global_alloc_zone(struct ttm_mem_global *glob, struct ttm_operation_ctx *ctx) {int count =TTM_MEMORY_ALLOC_RETRIES ; +bool alloc_anyway = ctx->flags &TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY ; -while (unlikely (ttm_mem_global_reserve (glob, - single_zone, - memory, true) +while (unlikely (ttm_mem_global_reserve (glob, single_zone, memory, + true, alloc_anyway) !=0 )) {if (ctx->no_wait_gpu)return -ENOMEM ;
drivers/gpu/drm/ttm/ttm_page_alloc_dma.c +0 -1
@@ -944,7 +944,6 @@ int ttm_dma_populate(struct ttm_dma_tt *ttm_dma, struct device *dev, i =0 ; type =ttm_to_type (ttm->page_flags, ttm->caching_state); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (ttm->page_flags &TTM_PAGE_FLAG_DMA32 ) goto skip_huge;
include/drm/ttm/ttm_bo_api.h +3 -1
@@ -278,7 +278,9 @@ struct ttm_operation_ctx { };/* Allow eviction of reserved BOs */ -#define TTM_OPT_FLAG_ALLOW_RES_EVICT 0x1 +#define TTM_OPT_FLAG_ALLOW_RES_EVICT 0x1 +/* when serving page fault, allow alloc anyway */ +#define TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY 0x2/** * ttm_bo_reference - reference a struct ttm_buffer_object
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 5/5 drm/ttm: add input parameter allow_allo_anyway for ttm_bo_evict_mm
In reply to Roger Heif true for it, allocate TTM pages regardless of zone global memory account limit.
that is for another special case: suspend.
doesn't care the zone global memory account limit for this case.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 6 ++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 2 +- drivers/gpu/drm/nouveau/nouveau_drm.c | 2 +- drivers/gpu/drm/qxl/qxl_object.c | 4 ++-- drivers/gpu/drm/radeon/radeon_device.c | 6 +++--- drivers/gpu/drm/radeon/radeon_object.c | 6 ++++-- drivers/gpu/drm/radeon/radeon_object.h | 3 ++- drivers/gpu/drm/ttm/ttm_bo.c | 16 ++++++++++------ drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 +++--- include/drm/ttm/ttm_bo_api.h | 5 ++++- 12 files changed, 37 insertions(+), 25 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +1 -1
@@ -763,7 +763,7 @@ static int amdgpu_debugfs_evict_vram(struct seq_file *m, void *data)struct drm_device *dev = node->minor->dev;struct amdgpu_device *adev = dev->dev_private; -seq_printf (m,"(%d)\n" ,amdgpu_bo_evict_vram (adev)); +seq_printf (m,"(%d)\n" ,amdgpu_bo_evict_vram (adev, true));return 0 ; }
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -2
@@ -2168,7 +2168,7 @@ int amdgpu_device_suspend(struct drm_device *dev, bool suspend, bool fbcon) } }/* evict vram memory */ -amdgpu_bo_evict_vram (adev); +amdgpu_bo_evict_vram (adev, true);amdgpu_fence_driver_suspend (adev);
@@ -2178,7 +2178,7 @@ int amdgpu_device_suspend(struct drm_device *dev, bool suspend, bool fbcon)* This second call to evict vram is to evict the gart page table * using the CPU. */ -amdgpu_bo_evict_vram (adev); +amdgpu_bo_evict_vram (adev, true);pci_save_state (dev->pdev);if (suspend) {
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +4 -2
@@ -803,14 +803,16 @@ int amdgpu_bo_unpin(struct amdgpu_bo *bo)return r; } -int amdgpu_bo_evict_vram (struct amdgpu_device *adev) +int amdgpu_bo_evict_vram (struct amdgpu_device *adev, +bool allow_alloc_anyway) {/* late 2.6.33 fix IGP hibernate - we need pm ops to do this correct */ if (0 && (adev->flags &AMD_IS_APU )) {/* Useless to evict on IGP chips */ return 0 ; } -return ttm_bo_evict_mm (&adev->mman.bdev,TTM_PL_VRAM ); +return ttm_bo_evict_mm (&adev->mman.bdev,TTM_PL_VRAM , + allow_alloc_anyway); }static const char *amdgpu_vram_names[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +1 -1
@@ -227,7 +227,7 @@ int amdgpu_bo_pin_restricted(struct amdgpu_bo *bo, u32 domain, u64 min_offset, u64 max_offset, u64 *gpu_addr);int amdgpu_bo_unpin (struct amdgpu_bo *bo); -int amdgpu_bo_evict_vram (struct amdgpu_device *adev); +int amdgpu_bo_evict_vram (struct amdgpu_device *adev,bool allow_alloc_anyway);int amdgpu_bo_init (struct amdgpu_device *adev);void amdgpu_bo_fini (struct amdgpu_device *adev); int amdgpu_bo_fbdev_mmap(struct amdgpu_bo *bo,
drivers/gpu/drm/nouveau/nouveau_drm.c +1 -1
@@ -702,7 +702,7 @@ nouveau_do_suspend(struct drm_device *dev, bool runtime) }NV_DEBUG (drm,"evicting buffers...\n" ); -ttm_bo_evict_mm (&drm->ttm.bdev,TTM_PL_VRAM ); +ttm_bo_evict_mm (&drm->ttm.bdev,TTM_PL_VRAM , true);NV_DEBUG (drm,"waiting for kernel channels to go idle...\n" );if (drm->cechan) {
drivers/gpu/drm/qxl/qxl_object.c +2 -2
@@ -350,10 +350,10 @@ int qxl_bo_check_id(struct qxl_device *qdev, struct qxl_bo *bo)int qxl_surf_evict (struct qxl_device *qdev) { -return ttm_bo_evict_mm (&qdev->mman.bdev,TTM_PL_PRIV ); +return ttm_bo_evict_mm (&qdev->mman.bdev,TTM_PL_PRIV , true); }int qxl_vram_evict (struct qxl_device *qdev) { -return ttm_bo_evict_mm (&qdev->mman.bdev,TTM_PL_VRAM ); +return ttm_bo_evict_mm (&qdev->mman.bdev,TTM_PL_VRAM , true); }
drivers/gpu/drm/radeon/radeon_device.c +3 -3
@@ -1522,7 +1522,7 @@ void radeon_device_fini(struct radeon_device *rdev)DRM_INFO ("radeon: finishing device.\n" ); rdev->shutdown = true;/* evict vram memory */ -radeon_bo_evict_vram (rdev); +radeon_bo_evict_vram (rdev, true);radeon_fini (rdev);if (!pci_is_thunderbolt_attached (rdev->pdev))vga_switcheroo_unregister_client (rdev->pdev);
@@ -1607,7 +1607,7 @@ int radeon_suspend_kms(struct drm_device *dev, bool suspend, } }/* evict vram memory */ -radeon_bo_evict_vram (rdev); +radeon_bo_evict_vram (rdev, true);/* wait for gpu to finish processing current batch */ for (i =0 ; i <RADEON_NUM_RINGS ; i++) {
@@ -1626,7 +1626,7 @@ int radeon_suspend_kms(struct drm_device *dev, bool suspend,* This second call to evict vram is to evict the gart page table * using the CPU. */ -radeon_bo_evict_vram (rdev); +radeon_bo_evict_vram (rdev, true);radeon_agp_suspend (rdev);
drivers/gpu/drm/radeon/radeon_object.c +4 -2
@@ -420,7 +420,8 @@ int radeon_bo_unpin(struct radeon_bo *bo)return r; } -int radeon_bo_evict_vram (struct radeon_device *rdev) +int +radeon_bo_evict_vram (struct radeon_device *rdev,bool allow_alloc_anyway) {/* late 2.6.33 fix IGP hibernate - we need pm ops to do this correct */ if (0 && (rdev->flags &RADEON_IS_IGP )) {
@@ -428,7 +429,8 @@ int radeon_bo_evict_vram(struct radeon_device *rdev)/* Useless to evict on IGP chips */ return 0 ; } -return ttm_bo_evict_mm (&rdev->mman.bdev,TTM_PL_VRAM ); +return ttm_bo_evict_mm (&rdev->mman.bdev,TTM_PL_VRAM , + allow_alloc_anyway); }void radeon_bo_force_delete (struct radeon_device *rdev)
drivers/gpu/drm/radeon/radeon_object.h +2 -1
@@ -136,7 +136,8 @@ extern int radeon_bo_pin(struct radeon_bo *bo, u32 domain, u64 *gpu_addr);extern int radeon_bo_pin_restricted (struct radeon_bo *bo,u32 domain,u64 max_offset,u64 *gpu_addr);extern int radeon_bo_unpin (struct radeon_bo *bo); -extern int radeon_bo_evict_vram (struct radeon_device *rdev); +extern int radeon_bo_evict_vram (struct radeon_device *rdev, +bool allow_alloc_anyway);extern void radeon_bo_force_delete (struct radeon_device *rdev);extern int radeon_bo_init (struct radeon_device *rdev);extern void radeon_bo_fini (struct radeon_device *rdev);
drivers/gpu/drm/ttm/ttm_bo.c +10 -6
@@ -1342,15 +1342,17 @@ int ttm_bo_create(struct ttm_bo_device *bdev,EXPORT_SYMBOL (ttm_bo_create);static int ttm_bo_force_list_clean (struct ttm_bo_device *bdev, -unsigned mem_type) +unsigned mem_type,bool allow_alloc_anyway) { -struct ttm_operation_ctx ctx = { false, false }; +struct ttm_operation_ctx ttm_opt_ctx = { false, false };struct ttm_mem_type_manager *man = &bdev->man[mem_type];struct ttm_bo_global *glob = bdev->glob;struct dma_fence *fence;int ret;unsigned i; +if (allow_alloc_anyway) + ttm_opt_ctx.flags =TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY ;/* * Can't use standard list traversal since we're unlocking. */
@@ -1359,7 +1361,8 @@ static int ttm_bo_force_list_clean(struct ttm_bo_device *bdev,for (i =0 ; i <TTM_MAX_BO_PRIORITY ; ++i) {while (!list_empty (&man->lru[i])) {spin_unlock (&glob->lru_lock); - ret =ttm_mem_evict_first (bdev, mem_type,NULL , &ctx); + ret =ttm_mem_evict_first (bdev, mem_type,NULL , + &ttm_opt_ctx);if (ret)return ret;spin_lock (&glob->lru_lock);
@@ -1403,7 +1406,7 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type) ret =0 ;if (mem_type >0 ) { - ret =ttm_bo_force_list_clean (bdev, mem_type); + ret =ttm_bo_force_list_clean (bdev, mem_type, true);if (ret) {pr_err ("Cleanup eviction failed\n" );return ret;
@@ -1419,7 +1422,8 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type) }EXPORT_SYMBOL (ttm_bo_clean_mm); -int ttm_bo_evict_mm (struct ttm_bo_device *bdev,unsigned mem_type) +int ttm_bo_evict_mm (struct ttm_bo_device *bdev,unsigned mem_type, +bool allow_allo_anyway) {struct ttm_mem_type_manager *man = &bdev->man[mem_type];
@@ -1433,7 +1437,7 @@ int ttm_bo_evict_mm(struct ttm_bo_device *bdev, unsigned mem_type)return 0 ; } -return ttm_bo_force_list_clean (bdev, mem_type); +return ttm_bo_force_list_clean (bdev, mem_type, allow_allo_anyway); }EXPORT_SYMBOL (ttm_bo_evict_mm);
drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +3 -3
@@ -430,7 +430,7 @@ static int vmw_request_device(struct vmw_private *dev_priv)if (dev_priv->cman)vmw_cmdbuf_remove_pool (dev_priv->cman);if (dev_priv->has_mob) { - (void )ttm_bo_evict_mm (&dev_priv->bdev,VMW_PL_MOB ); + (void )ttm_bo_evict_mm (&dev_priv->bdev,VMW_PL_MOB , true);vmw_otables_takedown (dev_priv); }if (dev_priv->cman)
@@ -463,7 +463,7 @@ static void vmw_release_device_early(struct vmw_private *dev_priv)vmw_cmdbuf_remove_pool (dev_priv->cman);if (dev_priv->has_mob) { -ttm_bo_evict_mm (&dev_priv->bdev,VMW_PL_MOB ); +ttm_bo_evict_mm (&dev_priv->bdev,VMW_PL_MOB , true);vmw_otables_takedown (dev_priv); } }
@@ -1342,7 +1342,7 @@ void vmw_svga_disable(struct vmw_private *dev_priv)if (dev_priv->bdev.man[TTM_PL_VRAM ].use_type) { dev_priv->bdev.man[TTM_PL_VRAM ].use_type = false;spin_unlock (&dev_priv->svga_lock); -if (ttm_bo_evict_mm (&dev_priv->bdev,TTM_PL_VRAM )) +if (ttm_bo_evict_mm (&dev_priv->bdev,TTM_PL_VRAM , true))DRM_ERROR ("Failed evicting VRAM buffers.\n" ); vmw_write(dev_priv, SVGA_REG_ENABLE, SVGA_REG_ENABLE_HIDE |
include/drm/ttm/ttm_bo_api.h +4 -1
@@ -636,6 +636,8 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);* * @bdev: Pointer to a ttm_bo_device struct. * @mem_type: The memory type. +* @allow_alloc_anyway: if true allow ttm pages allocation always +* regardless of zone memory account limit * * Evicts all buffers on the lru list of the memory type. * This is normally part of a VT switch or an
@@ -649,7 +651,8 @@ int ttm_bo_clean_mm(struct ttm_bo_device *bdev, unsigned mem_type);* -ERESTARTSYS: The call was interrupted by a signal while waiting to * evict a buffer. */ -int ttm_bo_evict_mm (struct ttm_bo_device *bdev,unsigned mem_type); +int ttm_bo_evict_mm (struct ttm_bo_device *bdev,unsigned mem_type, +bool allow_alloc_anyway);/** * ttm_kmap_obj_virtual
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Christian König
maintainerNice work, but a few comments.
First of all you need to reorder the patches. Adding the exceptions to the restrictions should come first, then the restriction itself. Otherwise we might break a setup in between the patches and that is bad for bisecting.
Then make all values configurable, e.g. take a closer look at ttm_memory.c. Just add attributes directly under the memory_accounting directory (see ttm_mem_global_init).
Additional to that you can't put device specific information (the no_retry flag) into ttm_mem_global, that is driver unspecific and won't work like this.
Move the new call out of ttm_mem_global_reserve() and into ttm_page_alloc.c or ttm_page_alloc_dma.c (but keep it in ttm_memory.c). ttm_mem_global_reserve() is called for each page allocated and si_mem_available() is a bit to heavy for that.
Maybe name TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY something like _FORCE_ALLOCATION or _ALLOW_OOM.
And please also try if a criteria like (si_mem_available() + get_nr_swap_pages()) < limit works as well. This way we would have only a single new limit.
Regards,
Christian.
Am 06.02.2018 um 10:04 schrieb Roger He:
Quoted 42 lines
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
the following patches is for prevent OOM triggered by TTM. the basic idea is when allocating TTM pages, check the free swap space firt. if it is less than the fixe limit, reject the allocation request. but there are two exceptions which should allow it regardless of zone memory account limit.
a. page fault
forttm_mem_global_reserveif serving for page fault routine, because page fault routing already grabbed system memory so the allowance of this exception is harmless. Otherwise, it will trigger
OOM killer.
b. suspend
anyway, we should allow suspend success always.at last, if bdev.no_retry is false (by defaut), keep the original behavior no any change.
Roger He (5):
drm/ttm: check if the free swap space is under limit 256MB drm/ttm: keep original behavior except with flagno_retrydrm/ttm: use bit flag to replaceallow_reserved_evictionin
ttm_operation_ctx
drm/ttm: add bit flagTTM_OPT_FLAG_ALLOW_ALLOC_ANYWAYdrm/ttm: add input parameterallow_allo_anywayforttm_bo_evict_mmdrivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 10 +++--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 +++-- drivers/gpu/drm/nouveau/nouveau_drm.c | 2 +- drivers/gpu/drm/qxl/qxl_object.c | 4 +-- drivers/gpu/drm/radeon/radeon_device.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.h | 3 +-
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c | 19 +++++++---- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++-- drivers/gpu/drm/ttm/ttm_memory.c | 51 ++++++++++++++++++++++++++--- drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 1 - drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 ++-- include/drm/ttm/ttm_bo_api.h | 14 ++++++-- include/drm/ttm/ttm_memory.h | 6 ++++ 18 files changed, 111 insertions(+), 43 deletions(-)
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Move the new call out of ttm_mem_global_reserve() and into ttm_page_alloc.c or ttm_page_alloc_dma.c (but keep it in ttm_memory.c). ttm_mem_global_reserve() is called for each page allocated and si_mem_available() is a bit to heavy for that.
Good idea! Agree with you completely, because initially I also concern that but no better way at that time. Going to improve the patches. Thanks!
-----Original Message-----
From: Koenig, Christian
Sent: Tuesday, February 06, 2018 5:52 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: [email protected]
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTM
Nice work, but a few comments.
First of all you need to reorder the patches. Adding the exceptions to the restrictions should come first, then the restriction itself. Otherwise we might break a setup in between the patches and that is bad for bisecting.
Then make all values configurable, e.g. take a closer look at ttm_memory.c. Just add attributes directly under the memory_accounting directory (see ttm_mem_global_init).
Additional to that you can't put device specific information (the no_retry flag) into ttm_mem_global, that is driver unspecific and won't work like this.
Move the new call out of ttm_mem_global_reserve() and into ttm_page_alloc.c or ttm_page_alloc_dma.c (but keep it in ttm_memory.c). ttm_mem_global_reserve() is called for each page allocated andsi_mem_available() is a bit to heavy for that.
Maybe name TTM_OPT_FLAG_ALLOW_ALLOC_ANYWAY something like _FORCE_ALLOCATION or _ALLOW_OOM.
And please also try if a criteria like (si_mem_available() + get_nr_swap_pages()) < limit works as well. This way we would have only a single new limit.
Regards,
Christian.
Am 06.02.2018 um 10:04 schrieb Roger He:
Quoted 42 lines
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
the following patches is for prevent OOM triggered by TTM. the basic idea is when allocating TTM pages, check the free swap space firt. if it is less than the fixe limit, reject the allocation request. but there are two exceptions which should allow it regardless of zone memory account limit.
a. page fault
forttm_mem_global_reserveif serving for page fault routine, because page fault routing already grabbed system memory so the allowance of this exception is harmless. Otherwise, it will trigger
OOM killer.
b. suspend
anyway, we should allow suspend success always.at last, if bdev.no_retry is false (by defaut), keep the original behavior no any change.
Roger He (5):
drm/ttm: check if the free swap space is under limit 256MB drm/ttm: keep original behavior except with flagno_retrydrm/ttm: use bit flag to replaceallow_reserved_evictionin
ttm_operation_ctx
drm/ttm: add bit flagTTM_OPT_FLAG_ALLOW_ALLOC_ANYWAYdrm/ttm: add input parameterallow_allo_anywayforttm_bo_evict_mmdrivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 10 +++--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 8 +++-- drivers/gpu/drm/nouveau/nouveau_drm.c | 2 +- drivers/gpu/drm/qxl/qxl_object.c | 4 +-- drivers/gpu/drm/radeon/radeon_device.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.c | 6 ++-- drivers/gpu/drm/radeon/radeon_object.h | 3 +-
Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c | 19 +++++++---- drivers/gpu/drm/ttm/ttm_bo_vm.c | 6 ++-- drivers/gpu/drm/ttm/ttm_memory.c | 51 ++++++++++++++++++++++++++--- drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 1 - drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 6 ++-- include/drm/ttm/ttm_bo_api.h | 14 ++++++-- include/drm/ttm/ttm_memory.h | 6 ++++ 18 files changed, 111 insertions(+), 43 deletions(-)
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Thanks,
Thomas
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct ttm_bo_device it already has member no_retry to indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,
I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk. In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive. So again we provide a flag no_retry in struct ttm_bo_device to let driver set according to its request.
Thanks
Roger(Hongbo.He)
-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTM
Hi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Thanks,
Thomas
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 1 line
AMD prefers return value of no memory rather than OOM for now.
Yeah, but Thomas is right that the default for this feature should be that it is turned off.
That's why I said that we should make the limit configurable.
Regards,
Christian.
Am 07.02.2018 um 09:25 schrieb He, Roger:
Quoted 40 lines
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct
ttm_bo_deviceit already has memberno_retryto indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,
I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk. In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive. So again we provide a flag
no_retryin structttm_bo_deviceto let driver set according to its request.Thanks
Roger(Hongbo.He)-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTMHi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Quoted 7 lines
Thanks,
Thomas_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): d617ad27-095d-f17d-d310-89caf21d12ca-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
Sure, will make it turn off as default and make the limit configurable.
Thanks
Roger(Hongbo.He)
-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 4:41 PM
To: He, Roger <[email protected]>; Thomas Hellstrom <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTM
Quoted 1 line
AMD prefers return value of no memory rather than OOM for now.
Yeah, but Thomas is right that the default for this feature should be that it is turned off.
That's why I said that we should make the limit configurable.
Regards,
Christian.
Am 07.02.2018 um 09:25 schrieb He, Roger:
Quoted 44 lines
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct
ttm_bo_deviceit already has memberno_retryto indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.One thing I looked at at one point was to have TTM do the swapping
itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk. In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive. So again we provide a flag
no_retryin structttm_bo_deviceto let driver set according to its request.Thanks
Roger(Hongbo.He)-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTMHi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Quoted 7 lines
Thanks,
Thomas_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB0127DA7619E7AE8C95D8FE8DFDFC0-3iK1xFAIwjq9imrIu4W8xGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]
Hi, Roger.
Quoted 5 lines · Roger
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct
ttm_bo_deviceit already has memberno_retryto indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.
Understood, but why is that? I mean just because TTM doesn't invoke the OOM killer, that doesn't mean that the process will, the next millisecond, page in a number of pages and invoke it? So this mechanism would be pretty susceptible to races?
Quoted 3 lines
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,
I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk.
But that is true for a page handed off to the swap-cache as well. It won't be immediately flushed to disc, only when the swap cache is shrunk.
Quoted 1 line
In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive.
That's true, but with VRAM, TTM overcommits swap space which may lead to ugly memory allocation failures at hibernate time.
Quoted 1 line
So again we provide a flag
no_retryin structttm_bo_deviceto let driver set according to its request.
Thanks,
Thomas
Quoted 24 lines
Thanks
Roger(Hongbo.He)-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTMHi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Quoted 7 lines
Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Thanks,
Thomas
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Christian König
maintainerQuoted 1 line
Understood, but why is that?
Well because customers requested it :)
What we try to do here is having a parameter which says when less than x megabytes of memory are left then fail the allocation.
This is basically to prevent buggy applications which try to allocate as much memory as possible until they receive an -ENOMEM from running into the OOM killer.
Quoted 1 line
That's true, but with VRAM, TTM overcommits swap space which may lead to ugly memory allocation failures at hibernate time.
Yeah, that is exactly the reason why I said that Roger should disable the limit during suspend swap out :)
Regards,
Christian.
Am 07.02.2018 um 14:17 schrieb Thomas Hellstrom:
Quoted 16 lines
Hi, Roger.
Quoted 5 lines · Roger
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct
ttm_bo_deviceit already has memberno_retryto indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.Understood, but why is that? I mean just because TTM doesn't invoke the OOM killer, that doesn't mean that the process will, the next millisecond, page in a number of pages and invoke it? So this mechanism would be pretty susceptible to races?
Quoted 3 lines
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,
I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk.
But that is true for a page handed off to the swap-cache as well. It won't be immediately flushed to disc, only when the swap cache is shrunk.
Quoted 42 lines
Quoted 1 line
In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive.
That's true, but with VRAM, TTM overcommits swap space which may lead to ugly memory allocation failures at hibernate time.
Quoted 1 line
So again we provide a flag
no_retryin structttm_bo_deviceto let driver set according to its request.Thanks,
ThomasQuoted 31 lines
Thanks
Roger(Hongbo.He)-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTMHi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Thanks,
Thomas
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
A message that is not in the archive (sent off-list or not yet received): 4b46ea00-ddb4-a316-02b8-247bd3812a14-5C7GfCeVMHo@public.gmane.org
Hi,
Quoted 6 lines · Christian König
Quoted 1 line
Understood, but why is that?
Well because customers requested it :)
What we try to do here is having a parameter which says when less than x megabytes of memory are left then fail the allocation.
This is basically to prevent buggy applications which try to allocate as much memory as possible until they receive an
-ENOMEMfrom running into the OOM killer.
OK. Understood.
Quoted 3 lines
Quoted 2 lines
That's true, but with VRAM, TTM overcommits swap space which may lead to ugly memory allocation failures at hibernate time.
Yeah, that is exactly the reason why I said that Roger should disable the limit during suspend swap out :)
Well that was really in the context of the swapping implementation rather in the context of this change so it was a little off-topic. Even if disabling this limit, TTM can overcommit. But looking at the swapping implementation is a different issue.
/Thomas
Quoted 63 lines
Regards,
Christian.Am 07.02.2018 um 14:17 schrieb Thomas Hellstrom:
Quoted 58 lines
Hi, Roger.
Quoted 5 lines · Roger
Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?
Now, in TTM struct
ttm_bo_deviceit already has memberno_retryto indicate your option. If you prefer no OOM triggered by TTM, set it as true. The default is false to keep original behavior. AMD prefers return value of no memory rather than OOM for now.Understood, but why is that? I mean just because TTM doesn't invoke the OOM killer, that doesn't mean that the process will, the next millisecond, page in a number of pages and invoke it? So this mechanism would be pretty susceptible to races?
Quoted 3 lines
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when,
I prefer current mechanism of swap out. At the beginning the swapped pages stay in system memory by shmem until OS move to status with high memory pressure, that has an obvious advantage. For example, if the BO is swapped out into shmem, but not really be flushed into swap disk. When validate it and swap in it at this moment, the overhead is small compared to swap in from disk.
But that is true for a page handed off to the swap-cache as well. It won't be immediately flushed to disc, only when the swap cache is shrunk.
Quoted 1 line
In addition, No need swap space reservation for TTM pages when allocation since swap disk is shared not only for TTM exclusive.
That's true, but with VRAM, TTM overcommits swap space which may lead to ugly memory allocation failures at hibernate time.
Quoted 1 line
So again we provide a flag
no_retryin structttm_bo_deviceto let driver set according to its request.Thanks,
ThomasQuoted 31 lines
Thanks
Roger(Hongbo.He)-----Original Message-----
From: Thomas Hellstrom [mailto:[email protected]]
Sent: Wednesday, February 07, 2018 2:43 PM
To: He, Roger <[email protected]>; [email protected]; [email protected]
Cc: Koenig, Christian <[email protected]>
Subject: Re: [PATCH 0/5] prevent OOM triggered by TTMHi, Roger,
Quoted 5 lines · Roger He
currently ttm code has no any allocation limit. So it allows pages allocatation unlimited until OOM. Because if swap space is full of swapped pages and then system memory will be filled up with ttm pages. and then any memory allocation request will trigger OOM.
I'm a bit curious, isn't this the way things are supposed to work on a linux system?
If all memory resources are used up, the OOM killer will kill the most memory hungry (perhaps rogue) process rather than processes being nice and try to find out themselves whether allocations will succeed? Why should TTM be different in that aspect? It would be good to know your reasoning WRT this?Admittedly, graphics process OOM memory accounting doesn't work very well, due to not all BOs not being CPU mapped, but it looks like there is recent work towards fixing this?
One thing I looked at at one point was to have TTM do the swapping itself instead of handing it off to the shmem system. That way we could pre-allocate swap entries for all swappable (BO) memory, making sure that we wouldn't run out of swap space when, for example, hibernating and that would also limit the pinned non-swappable memory (from TTM driver kernel allocations for example) to half the system memory resources.
Thanks,
Thomas
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx