Skip to contentSkip to conversation

amdgpu/amdkfd: Add support for non-4K page size systems

RFC PATCH Superseded Donet Tom 7 participants 44 messages Dec 12, 2025 – Jan 9

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
author
#23 lore raw

RFC PATCH 8/8 amdgpu: Fix MQD and control stack alignment for non-4K CPU page size systems

In reply to Donet Tom

For gfxV9, due to a hardware bug ("based on the comments in the code here [1]"), the control stack of a user-mode compute queue must be allocated immediately after the page boundary of its regular MQD buffer. To handle this, we allocate an enlarged MQD buffer where the first page is used as the MQD and the remaining pages store the control stack. Although these regions share the same BO, they require different memory
types: the MQD must be UC (uncached), while the control stack must be NC (non-coherent), matching the behavior when the control stack is allocated in user space.

This logic works correctly on systems where the CPU page size matches the GPU page size (4K). However, the current implementation aligns both the MQD and the control stack to the CPU PAGE_SIZE. On systems with a larger CPU page size, the entire first CPU page is marked UC—even though that page may contain multiple GPU pages. The GPU treats the second 4K GPU page inside that CPU page as part of the control stack, but it is incorrectly mapped as UC. This misalignment leads to queue preemption and eviction failures.

This patch fixes the issue by aligning both the MQD and control stack sizes to the GPU page size (4K). The first 4K page is correctly marked as UC for the MQD, and the remaining GPU pages are marked NC for the control stack. This ensures proper memory type assignment on systems with larger CPU page sizes and prevents incorrect behavior in queue preemption and eviction.

[1]: https://elixir.bootlin.com/linux/v6.18/source/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c#L118

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c      | 29 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h      |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 16 ++--------
 .../gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c   | 15 +++++-----
 4 files changed, 41 insertions(+), 21 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +29 -0
index b2033f8352f5..0e1c017d10dc 100644
@@ -368,6 +368,35 @@ void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset,
 	}
 	drm_dev_exit(idx);
 }
+void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset,
+		    int pages, dma_addr_t *dma_addr, uint64_t flags)
+{
+	uint64_t page_base;
+	unsigned int i, j, t;
+	int idx;
+	uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC);
+	void *dst;
+
+	if (!adev->gart.ptr)
+		return;
+
+	if (!drm_dev_enter(adev_to_drm(adev), &idx))
+		return;
+
+	t = offset / AMDGPU_GPU_PAGE_SIZE;
+	dst = adev->gart.ptr;
+	for (i = 0; i < pages; i++) {
+		page_base = dma_addr[i];
+		for (j = 0; j < AMDGPU_GPU_PAGES_IN_CPU_PAGE; j++, t++) {
+			if ((i == 0) && (j == 0))
+				amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, flags);
+			else
+				amdgpu_gmc_set_pte_pde(adev, dst, t, page_base, ctrl_flags);
+			page_base += AMDGPU_GPU_PAGE_SIZE;
+		}
+	}
+	drm_dev_exit(idx);
+}
 
 /**
  * amdgpu_gart_bind - bind pages into the gart page table
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h +2 -0
index 7cc980bf4725..1beef780936d 100644
@@ -62,6 +62,8 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, uint64_t offset,
 void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset,
 		     int pages, dma_addr_t *dma_addr, uint64_t flags,
 		     void *dst);
+void amdgpu_gart_map_gfx9_mqd(struct amdgpu_device *adev, uint64_t offset,
+		     int pages, dma_addr_t *dma_addr, uint64_t flags);
 void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset,
 		      int pages, dma_addr_t *dma_addr, uint64_t flags);
 void amdgpu_gart_invalidate_tlb(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +3 -13
index 9d568c16beb1..c2e0d2518345 100644
@@ -882,25 +882,15 @@ static void amdgpu_ttm_gart_bind_gfx9_mqd(struct amdgpu_device *adev,
 	int num_xcc = max(1U, adev->gfx.num_xcc_per_xcp);
 	uint64_t page_idx, pages_per_xcc;
 	int i;
-	uint64_t ctrl_flags = AMDGPU_PTE_MTYPE_VG10(flags, AMDGPU_MTYPE_NC);
 
 	pages_per_xcc = total_pages;
 	do_div(pages_per_xcc, num_xcc);
 
 	for (i = 0, page_idx = 0; i < num_xcc; i++, page_idx += pages_per_xcc) {
-		/* MQD page: use default flags */
-		amdgpu_gart_bind(adev,
+		amdgpu_gart_map_gfx9_mqd(adev,
 				gtt->offset + (page_idx << PAGE_SHIFT),
-				1, &gtt->ttm.dma_address[page_idx], flags);
-		/*
-		 * Ctrl pages - modify the memory type to NC (ctrl_flags) from
-		 * the second page of the BO onward.
-		 */
-		amdgpu_gart_bind(adev,
-				gtt->offset + ((page_idx + 1) << PAGE_SHIFT),
-				pages_per_xcc - 1,
-				&gtt->ttm.dma_address[page_idx + 1],
-				ctrl_flags);
+				pages_per_xcc, &gtt->ttm.dma_address[page_idx],
+				flags);
 	}
 }
 
drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c +7 -8
index f2dee320fada..e45e39cd65fe 100644
@@ -43,8 +43,8 @@ static uint64_t mqd_stride_v9(struct mqd_manager *mm,
 {
 	if (mm->dev->kfd->cwsr_enabled &&
 	    q->type == KFD_QUEUE_TYPE_COMPUTE)
-		return ALIGN(q->ctl_stack_size, PAGE_SIZE) +
-			ALIGN(sizeof(struct v9_mqd), PAGE_SIZE);
+		return ALIGN(ALIGN(q->ctl_stack_size, AMDGPU_GPU_PAGE_SIZE) +
+			ALIGN(sizeof(struct v9_mqd), AMDGPU_GPU_PAGE_SIZE), PAGE_SIZE);
 
 	return mm->mqd_size;
 }
@@ -136,13 +136,12 @@ static struct kfd_mem_obj *allocate_mqd(struct kfd_node *node,
 		if (!mqd_mem_obj)
 			return NULL;
 		retval = amdgpu_amdkfd_alloc_gtt_mem(node->adev,
-			(ALIGN(q->ctl_stack_size, PAGE_SIZE) +
-			ALIGN(sizeof(struct v9_mqd), PAGE_SIZE)) *
+			(ALIGN(ALIGN(q->ctl_stack_size, AMDGPU_GPU_PAGE_SIZE) +
+			ALIGN(sizeof(struct v9_mqd), AMDGPU_GPU_PAGE_SIZE), PAGE_SIZE)) *
 			NUM_XCC(node->xcc_mask),
 			&(mqd_mem_obj->gtt_mem),
 			&(mqd_mem_obj->gpu_addr),
 			(void *)&(mqd_mem_obj->cpu_ptr), true);
-
 		if (retval) {
 			kfree(mqd_mem_obj);
 			return NULL;
@@ -343,7 +342,7 @@ static int get_wave_state(struct mqd_manager *mm, void *mqd,
 	struct kfd_context_save_area_header header;
 
 	/* Control stack is located one page after MQD. */
-	void *mqd_ctl_stack = (void *)((uintptr_t)mqd + PAGE_SIZE);
+	void *mqd_ctl_stack = (void *)((uintptr_t)mqd + AMDGPU_GPU_PAGE_SIZE);
 
 	m = get_mqd(mqd);
 
@@ -380,7 +379,7 @@ static void checkpoint_mqd(struct mqd_manager *mm, void *mqd, void *mqd_dst, voi
 {
 	struct v9_mqd *m;
 	/* Control stack is located one page after MQD. */
-	void *ctl_stack = (void *)((uintptr_t)mqd + PAGE_SIZE);
+	void *ctl_stack = (void *)((uintptr_t)mqd + AMDGPU_GPU_PAGE_SIZE);
 
 	m = get_mqd(mqd);
 
@@ -426,7 +425,7 @@ static void restore_mqd(struct mqd_manager *mm, void **mqd,
 		*gart_addr = addr;
 
 	/* Control stack is located one page after MQD. */
-	ctl_stack = (void *)((uintptr_t)*mqd + PAGE_SIZE);
+	ctl_stack = (void *)((uintptr_t)*mqd + AMDGPU_GPU_PAGE_SIZE);
 	memcpy(ctl_stack, ctl_stack_src, ctl_stack_size);
 
 	m->cp_hqd_pq_doorbell_control =
Signature · 1 line
2.52.0
author
#9 lore raw

RFC PATCH 4/8 amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation. AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to 512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
 #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
 #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
 #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
 extern const struct attribute_group amdgpu_vram_mgr_attr_group;
Signature · 1 line
2.52.0

Donet Tom

author
#0 lore raw

cover amdgpu/amdkfd: Add support for non-4K page size systems

This patch series addresses few issues which we encountered while running rocr debug agent and rccl unit tests with AMD GPU on Power10 (ppc64le), using 64k system pagesize.

Note that we don't observe any of these issues while booting with 4k system pagesize on Power. So with the 64K system pagesize what we observed so far is, at few of the places, the conversion between gpu pfn to cpu pfn (or vice versa) may not be done correctly (due to different page size of AMD GPU (4K) v/s cpu pagesize (64K)) which causes issues like gpu page faults or gpu hang while running these tests.

Changes so far in this series:
=============================
1. For now, during kfd queue creation, this patch lifts the restriction on EOP buffer size to be same buffer object mapping size.

2. Fix SVM range map/unmap operations to convert CPU page numbers to GPU page numbers before calling amdgpu_vm_update_range(), which expects 4K GPU pages. Without this the rocr-debug-agent tests and rccl unit tests were failing.

3. Fix GART PTE allocation in migration code to account for multiple GPU pages per CPU page. The current code only allocates PTEs based on number of CPU pages, but GART may need one PTE per 4K GPU page.

4. Adjust AMDGPU_GTT_MAX_TRANSFER_SIZE to respect the SDMA engine's 4MB hardware limit regardless of CPU page size. The hardcoded 512 pages worked on 4K systems but seems to be exceeding the limit with 64K system page size.

5. In the current driver, MMIO remap is supported only when the system page size is 4K. Error messages have been added to indicate that MMIO remap is not supported on systems with a non-4K page size.

6. Fix amdgpu page fault handler (for xnack) to pass the corresponding system pfn (instead of gpu pfn) for restoring SVM range mapping.

7. Align ctl_stack_size and wg_data_size to GPU page size.

8. On systems where the CPU page size is larger than the GPU’s 4K page size, the MQD and control stack are aligned to the CPU PAGE_SIZE, causing multiple GPU pages to inherit the UC attribute incorrectly. This results in the control-stack area being mis-mapped and leads to queue preemption and eviction failures. Aligning both regions to the GPU page size ensures the MQD is mapped UC and the control stack NC, restoring correct behavior.

9. Apart from these 8 changes, we also needed this change [1]. Without this change
kernel simply crashes when running rocminfo command itself.
[1]: https://github.com/greenforce-project/chromeos-kernel-mirror/commit/2b33fad96c3129a2a53a42b9d90fb3b906145b98

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Donet Tom (7):
drm/amdkfd: Relax size checking during queue buffer get amdkfd/kfd_svm: Fix SVM map/unmap address conversion for non-4k page
sizes
amdkfd/kfd_migrate: Fix GART PTE for non-4K pagesize in
svm_migrate_gart_map()
amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page
size
drm/amdgpu: Handle GPU page faults correctly on non-4K page systems
amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead
of CPU page size
amdgpu: Fix MQD and control stack alignment for non-4K CPU page size
systems

Ritesh Harjani (IBM) (1):
amdkfd/kfd_chardev: Add error message for non-4k pagesize failures

 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c      | 29 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h      |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 16 ++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  6 ++--
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      | 10 +++++--
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c      |  2 +-
 .../gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c   | 15 +++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c        | 17 ++++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 30 ++++++++++++++-----
 10 files changed, 86 insertions(+), 43 deletions(-)
Signature · 1 line
2.52.0
author
#1 lore raw

RFC PATCH 1/8 drm/amdkfd: Relax size checking during queue buffer get

HW-supported EOP buffer sizes are 4K and 32K. On systems that do not use 4K pages, the minimum buffer object (BO) allocation size is PAGE_SIZE (for example, 64K). During queue buffer acquisition, the driver currently checks the allocated BO size against the supported EOP buffer size. Since the allocated BO is larger than the expected size, this check fails, preventing queue creation.

Relax the strict size validation and allow PAGE_SIZE-sized BOs to be used. Only the required 4K region of the buffer will be used as the EOP buffer and avoids queue creation failures on non-4K page systems.

---
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +6 -4
index f1e7583650c4..dc857450fa16 100644
@@ -199,6 +199,7 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, void __user *addr, struct amdgpu_
 	struct amdgpu_bo_va_mapping *mapping;
 	u64 user_addr;
 	u64 size;
+	u64 bo_size;
 
 	user_addr = (u64)addr >> AMDGPU_GPU_PAGE_SHIFT;
 	size = expected_size >> AMDGPU_GPU_PAGE_SHIFT;
@@ -207,11 +208,12 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, void __user *addr, struct amdgpu_
 	if (!mapping)
 		goto out_err;
 
-	if (user_addr != mapping->start ||
-	    (size != 0 && user_addr + size - 1 != mapping->last)) {
-		pr_debug("expected size 0x%llx not equal to mapping addr 0x%llx size 0x%llx\n",
+	bo_size = mapping->last - mapping->start + 1;
+
+	if (user_addr != mapping->start || (size != 0 && bo_size < size)) {
+		pr_debug("expected size 0x%llx grater than mapping addr 0x%llx size 0x%llx\n",
 			expected_size, mapping->start << AMDGPU_GPU_PAGE_SHIFT,
-			(mapping->last - mapping->start + 1) << AMDGPU_GPU_PAGE_SHIFT);
+			bo_size <<  AMDGPU_GPU_PAGE_SHIFT);
 		goto out_err;
 	}
 
Signature · 1 line
2.52.0
author
#4 lore raw

RFC PATCH 2/8 amdkfd/kfd_svm: Fix SVM map/unmap address conversion for non-4k page sizes

In reply to Donet Tom

SVM range size is tracked using the system page size. The range start and end are aligned to system page-sized PFNs, so the total SVM range size equals the total number of pages in the SVM range multiplied by the system page size.

The SVM range map/unmap functions pass these system page-sized PFN numbers to amdgpu_vm_update_range(), which expects PFNs based on the GPU page size (4K). On non-4K page systems, this mismatch causes only part of the SVM range to be mapped in the GPU page table, while the rest remains unmapped. If the GPU accesses an unmapped address within the same range, it results in a GPU page fault.

To fix this, the required conversion has been added in both svm_range_map_to_gpu() and svm_range_unmap_from_gpu(), ensuring that all pages in the SVM range are correctly mapped on non-4K systems.

---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 30 ++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +22 -8
index 74a1d3e1d52b..a2636f2d6c71 100644
@@ -1314,11 +1314,16 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 			 struct dma_fence **fence)
 {
 	uint64_t init_pte_value = 0;
+	uint64_t gpu_start, gpu_end;
 
-	pr_debug("[0x%llx 0x%llx]\n", start, last);
+	// Convert CPU page range to GPU page range
+	gpu_start = start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+	gpu_end = (last + 1) * AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, false, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
+	pr_debug("%s: CPU[0x%llx 0x%llx] -> GPU[0x%llx 0x%llx]\n", __func__,
+		 start, last, gpu_start, gpu_end);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, false, NULL, gpu_start,
+				      gpu_end, init_pte_value, 0, 0, NULL, NULL,
 				      fence);
 }
 
@@ -1398,9 +1403,13 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		 last_start, last_start + npages - 1, readonly);
 
 	for (i = offset; i < offset + npages; i++) {
+		uint64_t gpu_start;
+		uint64_t gpu_end;
+
 		last_domain = dma_addr[i] & SVM_RANGE_VRAM_DOMAIN;
 		dma_addr[i] &= ~SVM_RANGE_VRAM_DOMAIN;
 
+
 		/* Collect all pages in the same address range and memory domain
 		 * that can be mapped with a single call to update mapping.
 		 */
@@ -1415,17 +1424,22 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
 		if (readonly)
 			pte_flags &= ~AMDGPU_PTE_WRITEABLE;
 
-		pr_debug("svms 0x%p map [0x%lx 0x%llx] vram %d PTE 0x%llx\n",
-			 prange->svms, last_start, prange->start + i,
-			 (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
-			 pte_flags);
 
 		/* For dGPU mode, we use same vm_manager to allocate VRAM for
 		 * different memory partition based on fpfn/lpfn, we should use
 		 * same vm_manager.vram_base_offset regardless memory partition.
 		 */
+		gpu_start = last_start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+		gpu_end = (prange->start + i + 1) * AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
+
+		pr_debug("svms 0x%p map CPU[0x%lx 0x%llx] GPU[0x%llx 0x%llx] vram %d PTE 0x%llx\n",
+			 prange->svms, last_start, prange->start + i,
+			 gpu_start, gpu_end,
+			 (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
+			 pte_flags);
+
 		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, true,
-					   NULL, last_start, prange->start + i,
+					   NULL, gpu_start, gpu_end,
 					   pte_flags,
 					   (last_start - prange->start) << PAGE_SHIFT,
 					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
Signature · 1 line
2.52.0
author
#17 lore raw

RFC PATCH 6/8 drm/amdgpu: Handle GPU page faults correctly on non-4K page systems

In reply to Donet Tom

During a GPU page fault, the driver restores the SVM range and then maps it into the GPU page tables. The current implementation passes a GPU-page-size (4K-based) PFN to svm_range_restore_pages() to restore the range.

SVM ranges are tracked using system-page-size PFNs. On systems where the system page size is larger than 4K, using GPU-page-size PFNs to restore the range causes two problems:

Range lookup fails:
Because the restore function receives PFNs in GPU (4K) units, the SVM range lookup does not find the existing range. This will result in a duplicate SVM range being created.

VMA lookup failure:
The restore function also tries to locate the VMA for the faulting address. It converts the GPU-page-size PFN into an address using the system page size, which results in an incorrect address on non-4K page-size systems. As a result, the VMA lookup fails with the message: "address 0xxxx VMA is removed".

This patch passes the system-page-size PFN to svm_range_restore_pages() so that the SVM range is restored correctly on non-4K page systems.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +3 -3
index 676e24fb8864..6a11c9093e0c 100644
@@ -2958,14 +2958,14 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
 	if (!root)
 		return false;
 
-	addr /= AMDGPU_GPU_PAGE_SIZE;
-
 	if (is_compute_context && !svm_range_restore_pages(adev, pasid, vmid,
-	    node_id, addr, ts, write_fault)) {
+	    node_id, addr >> PAGE_SHIFT, ts, write_fault)) {
 		amdgpu_bo_unref(&root);
 		return true;
 	}
 
+	addr /= AMDGPU_GPU_PAGE_SIZE;
+
 	r = amdgpu_bo_reserve(root, true);
 	if (r)
 		goto error_unref;
Signature · 1 line
2.52.0
author
#18 lore raw

RFC PATCH 7/8 amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead of CPU page size

In reply to Donet Tom

The ctl_stack_size and wg_data_size values are used to compute the total context save/restore buffer size and the control stack size. These buffers are programmed into the GPU and are used to store the queue state during context save and restore.

Currently, both ctl_stack_size and wg_data_size are aligned to the CPU PAGE_SIZE. On systems with a non-4K CPU page size, this causes unnecessary memory waste because the GPU internally calculates and uses buffer sizes aligned to a fixed 4K GPU page size.

Since the control stack and context save/restore buffers are consumed by the GPU, their sizes should be aligned to the GPU page size (4K), not the CPU page size. This patch updates the alignment of ctl_stack_size and wg_data_size to prevent over-allocation on systems with larger CPU page sizes.

---
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +4 -3
index dc857450fa16..00ab941c3e86 100644
@@ -445,10 +445,11 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 		    min(cu_num * 40, props->array_count / props->simd_arrays_per_engine * 512)
 		    : cu_num * 32;
 
-	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props), PAGE_SIZE);
+	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props),
+				AMDGPU_GPU_PAGE_SIZE);
 	ctl_stack_size = wave_num * CNTL_STACK_BYTES_PER_WAVE(gfxv) + 8;
 	ctl_stack_size = ALIGN(SIZEOF_HSA_USER_CONTEXT_SAVE_AREA_HEADER + ctl_stack_size,
-			       PAGE_SIZE);
+			       AMDGPU_GPU_PAGE_SIZE);
 
 	if ((gfxv / 10000 * 10000) == 100000) {
 		/* HW design limits control stack size to 0x7000.
@@ -460,7 +461,7 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 
 	props->ctl_stack_size = ctl_stack_size;
 	props->debug_memory_size = ALIGN(wave_num * DEBUGGER_BYTES_PER_WAVE, DEBUGGER_BYTES_ALIGN);
-	props->cwsr_size = ctl_stack_size + wg_data_size;
+	props->cwsr_size = ALIGN(ctl_stack_size + wg_data_size, PAGE_SIZE);
 
 	if (gfxv == 80002)	/* GFX_VERSION_TONGA */
 		props->eop_buffer_size = 0x8000;
Signature · 1 line
2.52.0
author
#16 lore raw

RFC PATCH 5/8 amdkfd/kfd_chardev: Add error message for non-4k pagesize failures

In reply to Donet Tom

From: "Ritesh Harjani (IBM)" <[email protected]>

In the current driver, MMIO remap is supported only when the system page size is 4K. Error messages have been added to indicate that MMIO remap is not supported on systems with a non-4K page size.

Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

---
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +8 -2
index 0f0719528bcc..19632795c389 100644
@@ -1134,6 +1134,8 @@ static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
 		offset = dev->adev->rmmio_remap.bus_addr;
 		if (!offset || (PAGE_SIZE > 4096)) {
 			err = -ENOMEM;
+			pr_err("%s: Failed MMIO remap off:0x%llx PAGE_SIZE:%lu (requires 4K)\n",
+				__func__, offset, PAGE_SIZE);
 			goto err_unlock;
 		}
 	}
@@ -2317,7 +2319,8 @@ static int criu_restore_memory_of_gpu(struct kfd_process_device *pdd,
 		}
 		offset = pdd->dev->adev->rmmio_remap.bus_addr;
 		if (!offset || (PAGE_SIZE > 4096)) {
-			pr_err("amdgpu_amdkfd_get_mmio_remap_phys_addr failed\n");
+			pr_err("%s: amdgpu_amdkfd_get_mmio_remap_phys_addr failed off:0x%llx, page_size:%lu\n",
+				__func__, offset, PAGE_SIZE);
 			return -ENOMEM;
 		}
 	} else if (bo_bucket->alloc_flags & KFD_IOC_ALLOC_MEM_FLAGS_USERPTR) {
@@ -3367,8 +3370,11 @@ static int kfd_mmio_mmap(struct kfd_node *dev, struct kfd_process *process,
 	if (vma->vm_end - vma->vm_start != PAGE_SIZE)
 		return -EINVAL;
 
-	if (PAGE_SIZE > 4096)
+	if (PAGE_SIZE > 4096) {
+		pr_err("%s: MMIO mmap not supported with PAGE_SIZE=%lu (requires 4K)\n",
+			__func__, PAGE_SIZE);
 		return -EINVAL;
+	}
 
 	address = dev->adev->rmmio_remap.bus_addr;
 
Signature · 1 line
2.52.0
author
#7 lore raw

RFC PATCH 3/8 amdkfd/kfd_migrate: Fix GART PTE for non-4K pagesize in svm_migrate_gart_map()

In reply to Donet Tom

In svm_migrate_gart_map(), while migrating GART mapping, the number of bytes copied for the GART table only accounts for CPU pages. On non-4K systems, each CPU page can contain multiple GPU pages, and the GART requires one 8-byte PTE per GPU page. As a result, an incorrect size was passed to the DMA, causing only a partial update of the GART table.

Fix this function to work correctly on non-4K page-size systems by accounting for the number of GPU pages per CPU page when calculating the number of bytes to be copied.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1 -1
index 59a5a3fea65d..ea8377071c39 100644
@@ -62,7 +62,7 @@ svm_migrate_gart_map(struct amdgpu_ring *ring, u64 npages,
 	*gart_addr = adev->gmc.gart_start;
 
 	num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
-	num_bytes = npages * 8;
+	num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
 
 	r = amdgpu_job_alloc_with_ib(adev, &adev->mman.high_pr,
 				     AMDGPU_FENCE_OWNER_UNDEFINED,
Signature · 1 line
2.52.0
maintainer
#10 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Donet Tom
Quoted 2 lines · Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Regards,
Christian.

Quoted 20 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
 #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
 #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
 #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
 extern const struct attribute_group amdgpu_vram_mgr_attr_group;
maintainer
#24 lore raw
In reply to Donet Tom
Quoted 56 lines · Donet Tom

This patch series addresses few issues which we encountered while running rocr debug agent and rccl unit tests with AMD GPU on Power10 (ppc64le), using 64k system pagesize.

Note that we don't observe any of these issues while booting with 4k system pagesize on Power. So with the 64K system pagesize what we observed so far is, at few of the places, the conversion between gpu pfn to cpu pfn (or vice versa) may not be done correctly (due to different page size of AMD GPU (4K) v/s cpu pagesize (64K)) which causes issues like gpu page faults or gpu hang while running these tests.

Changes so far in this series:
=============================
1. For now, during kfd queue creation, this patch lifts the restriction on EOP buffer size to be same buffer object mapping size.

2. Fix SVM range map/unmap operations to convert CPU page numbers to GPU page numbers before calling amdgpu_vm_update_range(), which expects 4K GPU pages. Without this the rocr-debug-agent tests and rccl unit tests were failing.

3. Fix GART PTE allocation in migration code to account for multiple GPU pages per CPU page. The current code only allocates PTEs based on number of CPU pages, but GART may need one PTE per 4K GPU page.

4. Adjust AMDGPU_GTT_MAX_TRANSFER_SIZE to respect the SDMA engine's 4MB hardware limit regardless of CPU page size. The hardcoded 512 pages worked on 4K systems but seems to be exceeding the limit with 64K system page size.

5. In the current driver, MMIO remap is supported only when the system page size is 4K. Error messages have been added to indicate that MMIO remap is not supported on systems with a non-4K page size.

6. Fix amdgpu page fault handler (for xnack) to pass the corresponding system pfn (instead of gpu pfn) for restoring SVM range mapping.

7. Align ctl_stack_size and wg_data_size to GPU page size.

8. On systems where the CPU page size is larger than the GPU’s 4K page size, the MQD and control stack are aligned to the CPU PAGE_SIZE, causing multiple GPU pages to inherit the UC attribute incorrectly. This results in the control-stack area being mis-mapped and leads to queue preemption and eviction failures. Aligning both regions to the GPU page size ensures the MQD is mapped UC and the control stack NC, restoring correct behavior.

9. Apart from these 8 changes, we also needed this change [1]. Without this change
kernel simply crashes when running rocminfo command itself.
[1]: https://github.com/greenforce-project/chromeos-kernel-mirror/commit/2b33fad96c3129a2a53a42b9d90fb3b906145b98

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Quoted 5 lines

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

The ROCm team needs to answer that.

Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

Quoted 10 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

What we can do is to support graphics and MM, but that should already work out of the box.

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Regards,
Christian.

Quoted 30 lines

Donet Tom (7):
drm/amdkfd: Relax size checking during queue buffer get amdkfd/kfd_svm: Fix SVM map/unmap address conversion for non-4k page
sizes
amdkfd/kfd_migrate: Fix GART PTE for non-4K pagesize in
svm_migrate_gart_map()
amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page
size
drm/amdgpu: Handle GPU page faults correctly on non-4K page systems
amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead
of CPU page size
amdgpu: Fix MQD and control stack alignment for non-4K CPU page size
systems

Ritesh Harjani (IBM) (1):
amdkfd/kfd_chardev: Add error message for non-4k pagesize failures

 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c      | 29 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h      |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 16 ++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  6 ++--
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      | 10 +++++--
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c      |  2 +-
 .../gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c   | 15 +++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c        | 17 ++++++-----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          | 30 ++++++++++++++-----
 10 files changed, 86 insertions(+), 43 deletions(-)
maintainer
#19 lore raw

Re: [RFC PATCH v1 7/8] amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead of CPU page size

In reply to Donet Tom
Quoted 10 lines · Donet Tom

The ctl_stack_size and wg_data_size values are used to compute the total context save/restore buffer size and the control stack size. These buffers are programmed into the GPU and are used to store the queue state during context save and restore.

Currently, both ctl_stack_size and wg_data_size are aligned to the CPU PAGE_SIZE. On systems with a non-4K CPU page size, this causes unnecessary memory waste because the GPU internally calculates and uses buffer sizes aligned to a fixed 4K GPU page size.

Quoted 5 lines

Since the control stack and context save/restore buffers are consumed by the GPU, their sizes should be aligned to the GPU page size (4K), not the CPU page size. This patch updates the alignment of ctl_stack_size and wg_data_size to prevent over-allocation on systems with larger CPU page sizes.

As far as I know the problem is that the debugger needs to consume that stuff on the CPU side as well.

I need to double check that, but I think the alignment is correct as it is.

Regards,
Christian.

Quoted 33 lines
---
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +4 -3
index dc857450fa16..00ab941c3e86 100644
@@ -445,10 +445,11 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 		    min(cu_num * 40, props->array_count / props->simd_arrays_per_engine * 512)
 		    : cu_num * 32;
 
-	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props), PAGE_SIZE);
+	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props),
+				AMDGPU_GPU_PAGE_SIZE);
 	ctl_stack_size = wave_num * CNTL_STACK_BYTES_PER_WAVE(gfxv) + 8;
 	ctl_stack_size = ALIGN(SIZEOF_HSA_USER_CONTEXT_SAVE_AREA_HEADER + ctl_stack_size,
-			       PAGE_SIZE);
+			       AMDGPU_GPU_PAGE_SIZE);
 
 	if ((gfxv / 10000 * 10000) == 100000) {
 		/* HW design limits control stack size to 0x7000.
@@ -460,7 +461,7 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 
 	props->ctl_stack_size = ctl_stack_size;
 	props->debug_memory_size = ALIGN(wave_num * DEBUGGER_BYTES_PER_WAVE, DEBUGGER_BYTES_ALIGN);
-	props->cwsr_size = ctl_stack_size + wg_data_size;
+	props->cwsr_size = ALIGN(ctl_stack_size + wg_data_size, PAGE_SIZE);
 
 	if (gfxv == 80002)	/* GFX_VERSION_TONGA */
 		props->eop_buffer_size = 0x8000;
author
#11 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Christian König
Quoted 7 lines · Christian König
Quoted 2 lines · Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 7 lines
Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Hi Christian

Thank you for the reply.

In svm_migrate_copy_memory_gart(), the number of bytes to copy passed to amdgpu_copy_buffer() is based on the PMD size. On systems with a 4K page size, the PMD size is calculated correctly because AMDGPU_GTT_MAX_TRANSFER_SIZE is 512, and 512 × 4K = 2MB.

On systems with a 64K page size, however, the calculation becomes 512 × 64K = 32MB. As a result, amdgpu_copy_buffer() ends up copying data in 4MB chunks instead of PMD-sized chunks. To ensure consistent behavior across both 64K and 4K page-size systems, we adjusted the transfer size so that the maximum transfer remains 2MB, matching the PMD size.

The issue we observed was that the rocr-debug-agent test triggered SDMA hangs. This happened because an incorrect size was being passed when copying the GART mapping in svm_migrate_gart_map(). That problem was addressed in patch 3/8. While root-causing that issue, we also identified this inconsistency between 4K and 64K systems, So we felt that this change was needed to align the behavior with 4K system page sizes.

Quoted 23 lines

Regards,
Christian.

Quoted 19 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
  extern const struct attribute_group amdgpu_vram_mgr_attr_group;
author
#20 lore raw

Re: [RFC PATCH v1 7/8] amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead of CPU page size

In reply to Christian König
Quoted 11 lines · Christian König
Quoted 10 lines · Donet Tom

The ctl_stack_size and wg_data_size values are used to compute the total context save/restore buffer size and the control stack size. These buffers are programmed into the GPU and are used to store the queue state during context save and restore.

Currently, both ctl_stack_size and wg_data_size are aligned to the CPU PAGE_SIZE. On systems with a non-4K CPU page size, this causes unnecessary memory waste because the GPU internally calculates and uses buffer sizes aligned to a fixed 4K GPU page size.

Quoted 8 lines
Quoted 5 lines

Since the control stack and context save/restore buffers are consumed by the GPU, their sizes should be aligned to the GPU page size (4K), not the CPU page size. This patch updates the alignment of ctl_stack_size and wg_data_size to prevent over-allocation on systems with larger CPU page sizes.

As far as I know the problem is that the debugger needs to consume that stuff on the CPU side as well.

I need to double check that, but I think the alignment is correct as it is.

Thanks Christian

We were observing the following errors during RCCL unit tests when running on more than two GPUs, which eventually led to a GPU hang:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [  606.696820] amdgpu 0048:0f:00.0: amdgpu: amdgpu: Failed to evict process queues
[  606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source:  4
[  610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [  610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues

After applying the alignment change and the change in patch 8/8, we are not seeing this issue.

Quoted 36 lines

Regards,
Christian.

Quoted 32 lines
---
  drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +4 -3
index dc857450fa16..00ab941c3e86 100644
@@ -445,10 +445,11 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
  		    min(cu_num * 40, props->array_count / props->simd_arrays_per_engine * 512)
  		    : cu_num * 32;
 
-	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props), PAGE_SIZE);
+	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props),
+				AMDGPU_GPU_PAGE_SIZE);
  	ctl_stack_size = wave_num * CNTL_STACK_BYTES_PER_WAVE(gfxv) + 8;
  	ctl_stack_size = ALIGN(SIZEOF_HSA_USER_CONTEXT_SAVE_AREA_HEADER + ctl_stack_size,
-			       PAGE_SIZE);
+			       AMDGPU_GPU_PAGE_SIZE);
 
  	if ((gfxv / 10000 * 10000) == 100000) {
  		/* HW design limits control stack size to 0x7000.
@@ -460,7 +461,7 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 
  	props->ctl_stack_size = ctl_stack_size;
  	props->debug_memory_size = ALIGN(wave_num * DEBUGGER_BYTES_PER_WAVE, DEBUGGER_BYTES_ALIGN);
-	props->cwsr_size = ctl_stack_size + wg_data_size;
+	props->cwsr_size = ALIGN(ctl_stack_size + wg_data_size, PAGE_SIZE);
 
  	if (gfxv == 80002)	/* GFX_VERSION_TONGA */
  		props->eop_buffer_size = 0x8000;
maintainer
#26 lore raw
In reply to Ritesh Harjani (IBM)
Quoted 14 lines · Ritesh Harjani (IBM)
Quoted 13 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 5 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 4 lines
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 4 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 6 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 8 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 2 lines

-ritesh

#25 lore raw
In reply to Christian König
Quoted 57 lines · Christian König
Quoted 56 lines · Donet Tom

This patch series addresses few issues which we encountered while running rocr debug agent and rccl unit tests with AMD GPU on Power10 (ppc64le), using 64k system pagesize.

Note that we don't observe any of these issues while booting with 4k system pagesize on Power. So with the 64K system pagesize what we observed so far is, at few of the places, the conversion between gpu pfn to cpu pfn (or vice versa) may not be done correctly (due to different page size of AMD GPU (4K) v/s cpu pagesize (64K)) which causes issues like gpu page faults or gpu hang while running these tests.

Changes so far in this series:
=============================
1. For now, during kfd queue creation, this patch lifts the restriction on EOP buffer size to be same buffer object mapping size.

2. Fix SVM range map/unmap operations to convert CPU page numbers to GPU page numbers before calling amdgpu_vm_update_range(), which expects 4K GPU pages. Without this the rocr-debug-agent tests and rccl unit tests were failing.

3. Fix GART PTE allocation in migration code to account for multiple GPU pages per CPU page. The current code only allocates PTEs based on number of CPU pages, but GART may need one PTE per 4K GPU page.

4. Adjust AMDGPU_GTT_MAX_TRANSFER_SIZE to respect the SDMA engine's 4MB hardware limit regardless of CPU page size. The hardcoded 512 pages worked on 4K systems but seems to be exceeding the limit with 64K system page size.

5. In the current driver, MMIO remap is supported only when the system page size is 4K. Error messages have been added to indicate that MMIO remap is not supported on systems with a non-4K page size.

6. Fix amdgpu page fault handler (for xnack) to pass the corresponding system pfn (instead of gpu pfn) for restoring SVM range mapping.

7. Align ctl_stack_size and wg_data_size to GPU page size.

8. On systems where the CPU page size is larger than the GPU’s 4K page size, the MQD and control stack are aligned to the CPU PAGE_SIZE, causing multiple GPU pages to inherit the UC attribute incorrectly. This results in the control-stack area being mis-mapped and leads to queue preemption and eviction failures. Aligning both regions to the GPU page size ensures the MQD is mapped UC and the control stack NC, restoring correct behavior.

9. Apart from these 8 changes, we also needed this change [1]. Without this change
kernel simply crashes when running rocminfo command itself.
[1]: https://github.com/greenforce-project/chromeos-kernel-mirror/commit/2b33fad96c3129a2a53a42b9d90fb3b906145b98

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Quoted 8 lines
Quoted 5 lines

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

Quoted 5 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Quoted 7 lines
Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

Quoted 5 lines
Quoted 5 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 7 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?
- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

-ritesh

#27 lore raw
In reply to Christian König
Quoted 47 lines · Christian König
Quoted 14 lines · Ritesh Harjani (IBM)
Quoted 13 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 5 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets, but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps. That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

Alex

Quoted 69 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 4 lines
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 4 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 6 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 8 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 2 lines

-ritesh

maintainer
#28 lore raw
Quoted 57 lines · Alex Deucher
Quoted 47 lines · Christian König
Quoted 14 lines · Ritesh Harjani (IBM)
Quoted 13 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 5 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 72 lines

Alex

Quoted 69 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 4 lines
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 4 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 6 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 8 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 2 lines

-ritesh

author
#29 lore raw
Quoted 66 lines · Christian König
Quoted 45 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

Quoted 8 lines

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Sorry, Cristian — I may be misunderstanding this point, so I would appreciate some clarification.

If the CPU page size is 64K and the GPU page size is 4K, then from the GPU side the page table entries are created and mapped at 4K granularity, while on the CPU side the pages remain 64K. To map a single CPU page to the GPU, we therefore need to create multiple GPU page table entries for that CPU page.

We found that this was not being handled correctly in the SVM path and addressed it with the change in patch 2/8.

Given this, if the memory is allocated and mapped in GPU page-size (4K) granularity on the GPU side, could you please clarify how memory waste occurs in this scenario?

Thank you for your time and guidance.

Quoted 60 lines

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

#36 lore raw
In reply to Christian König
Quoted 68 lines · Christian König
Quoted 57 lines · Alex Deucher
Quoted 47 lines · Christian König
Quoted 14 lines · Ritesh Harjani (IBM)
Quoted 13 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 5 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

Quoted 1 line

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Alex

Quoted 97 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 72 lines

Alex

Quoted 69 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 4 lines
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 4 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 6 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 8 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 2 lines

-ritesh

maintainer
#30 lore raw
In reply to Donet Tom
Quoted 72 lines · Donet Tom
Quoted 71 lines · Christian König
Quoted 45 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
   Name:                    gfx90a
   Marketing Name:          AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1]  and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right.  There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them.  In this case we remap the HDP flush registers into one of those register pages.  This allows applications to flush the HDP write FIFO from either the CPU or another device.  This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it).  This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control.  This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

Quoted 8 lines
Quoted 3 lines

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Sorry, Cristian — I may be misunderstanding this point, so I would appreciate some clarification.

If the CPU page size is 64K and the GPU page size is 4K, then from the GPU side the page table entries are created and mapped at 4K granularity, while on the CPU side the pages remain 64K. To map a single CPU page to the GPU, we therefore need to create multiple GPU page table entries for that CPU page.

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

Regards,
Christian.

Quoted 68 lines

We found that this was not being handled correctly in the SVM path and addressed it with the change in patch 2/8.

Given this, if the memory is allocated and mapped in GPU page-size (4K) granularity on the GPU side, could you please clarify how memory waste occurs in this scenario?

Thank you for your time and guidance.

Quoted 60 lines

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

#2 lore raw

Re: [RFC PATCH v1 1/8] drm/amdkfd: Relax size checking during queue buffer get

In reply to Donet Tom
Quoted 32 lines · Donet Tom

HW-supported EOP buffer sizes are 4K and 32K. On systems that do not use 4K pages, the minimum buffer object (BO) allocation size is PAGE_SIZE (for example, 64K). During queue buffer acquisition, the driver currently checks the allocated BO size against the supported EOP buffer size. Since the allocated BO is larger than the expected size, this check fails, preventing queue creation.

Relax the strict size validation and allow PAGE_SIZE-sized BOs to be used. Only the required 4K region of the buffer will be used as the EOP buffer and avoids queue creation failures on non-4K page systems.

---
  drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +1 -0
index f1e7583650c4..dc857450fa16 100644
@@ -199,6 +199,7 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, void __user *addr, struct amdgpu_
  	struct amdgpu_bo_va_mapping *mapping;
  	u64 user_addr;
  	u64 size;
+	u64 bo_size;
 
  	user_addr = (u64)addr >> AMDGPU_GPU_PAGE_SHIFT;
  	size = expected_size >> AMDGPU_GPU_PAGE_SHIFT;
@@ -207,11 +208,12 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, void __user *addr, struct amdgpu_
  	if (!mapping)
  		goto out_err;
 

drivers/gpu/drm/amd/amdkfd/kfd_queue.c int kfd_queue_buffer_get(struct amdgpu_vm *vm, void __user *addr, struct amdgpu_ · line 216

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +5 -4
-	if (user_addr != mapping->start ||
-	    (size != 0 && user_addr + size - 1 != mapping->last)) {
-		pr_debug("expected size 0x%llx not equal to mapping addr 0x%llx size 0x%llx\n",
+	bo_size = mapping->last - mapping->start + 1;
+
+	if (user_addr != mapping->start || (size != 0 && bo_size < size)) {
+		pr_debug("expected size 0x%llx grater than mapping addr 0x%llx size 0x%llx\n",
  			expected_size, mapping->start << AMDGPU_GPU_PAGE_SHIFT,
-			(mapping->last - mapping->start + 1) << AMDGPU_GPU_PAGE_SHIFT);
+			bo_size <<  AMDGPU_GPU_PAGE_SHIFT);

This change works, but also relax the size validation for ring buffer size etc, this may have side effect, for example FW and user space should have the same ring buffer size.

Other buffers already use PAGE_SIZE as expected size or size aligned to PAGE_SIZE, maybe only relax the eop buffer size check

@@ -275,7 +275,7 @@ int kfd_queue_acquire_buffers(struct kfd_process_device *pdd, struct queue_prope

        /* EOP buffer is not required for all ASICs */
        if (properties->eop_ring_buffer_address) {
-               if (properties->eop_ring_buffer_size != topo_dev->node_props.eop_buffer_size) { +               if (properties->eop_ring_buffer_size < topo_dev->node_props.eop_buffer_size) { pr_debug("queue eop bo size 0x%x not equal to node eop buf size 0x%x\n", properties->eop_ring_buffer_size, topo_dev->node_props.eop_buffer_size);
@@ -284,7 +284,7 @@ int kfd_queue_acquire_buffers(struct kfd_process_device *pdd, struct queue_prope }
                err = kfd_queue_buffer_get(vm, (void *)properties->eop_ring_buffer_address, &properties->eop_buf_bo,
- properties->eop_ring_buffer_size);
+ ALIGN(properties->eop_ring_buffer_size, PAGE_SIZE));
                if (err)
                        goto out_err_unreserve;
        }

Regards,
Philip

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +0 -0
  		goto out_err;
  	}
 
Reviewed-by on v1 2/8
#5 lore raw

Re: [RFC PATCH v1 2/8] amdkfd/kfd_svm: Fix SVM map/unmap address conversion for non-4k page sizes

In reply to Donet Tom
Quoted 30 lines · Donet Tom

SVM range size is tracked using the system page size. The range start and end are aligned to system page-sized PFNs, so the total SVM range size equals the total number of pages in the SVM range multiplied by the system page size.

The SVM range map/unmap functions pass these system page-sized PFN numbers to amdgpu_vm_update_range(), which expects PFNs based on the GPU page size (4K). On non-4K page systems, this mismatch causes only part of the SVM range to be mapped in the GPU page table, while the rest remains unmapped. If the GPU accesses an unmapped address within the same range, it results in a GPU page fault.

To fix this, the required conversion has been added in both svm_range_map_to_gpu() and svm_range_unmap_from_gpu(), ensuring that all pages in the SVM range are correctly mapped on non-4K systems.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 30 ++++++++++++++++++++--------
  1 file changed, 22 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 74a1d3e1d52b..a2636f2d6c71 100644
@@ -1314,11 +1314,16 @@ svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm,
  			 struct dma_fence **fence)
  {
  	uint64_t init_pte_value = 0;

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, · line 1323

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -3
+	uint64_t gpu_start, gpu_end;
 
-	pr_debug("[0x%llx 0x%llx]\n", start, last);
+	// Convert CPU page range to GPU page range
+	gpu_start = start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+	gpu_end = (last + 1) * AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
 
-	return amdgpu_vm_update_range(adev, vm, false, true, true, false, NULL, start,
-				      last, init_pte_value, 0, 0, NULL, NULL,
+	pr_debug("%s: CPU[0x%llx 0x%llx] -> GPU[0x%llx 0x%llx]\n", __func__,

dynamic debug control can enable function name, linenum print with +pfl, don't add __func__.

Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +3 -0
+		 start, last, gpu_start, gpu_end);
+	return amdgpu_vm_update_range(adev, vm, false, true, true, false, NULL, gpu_start,
+				      gpu_end, init_pte_value, 0, 0, NULL, NULL,
  				      fence);
  }
 

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange, · line 1412

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +4 -0
@@ -1398,9 +1403,13 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
  		 last_start, last_start + npages - 1, readonly);
 
  	for (i = offset; i < offset + npages; i++) {
+		uint64_t gpu_start;
+		uint64_t gpu_end;
+
  		last_domain = dma_addr[i] & SVM_RANGE_VRAM_DOMAIN;
  		dma_addr[i] &= ~SVM_RANGE_VRAM_DOMAIN;
 
+

remove extra blank line.

Quoted 19 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +3 -4
  		/* Collect all pages in the same address range and memory domain
  		 * that can be mapped with a single call to update mapping.
  		 */
@@ -1415,17 +1424,22 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,
  		if (readonly)
  			pte_flags &= ~AMDGPU_PTE_WRITEABLE;
 
-		pr_debug("svms 0x%p map [0x%lx 0x%llx] vram %d PTE 0x%llx\n",
-			 prange->svms, last_start, prange->start + i,
-			 (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
-			 pte_flags);
 
  		/* For dGPU mode, we use same vm_manager to allocate VRAM for
  		 * different memory partition based on fpfn/lpfn, we should use
  		 * same vm_manager.vram_base_offset regardless memory partition.
  		 */
+		gpu_start = last_start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+		gpu_end = (prange->start + i + 1) * AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
+

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange, · line 1443

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +7 -1
+		pr_debug("svms 0x%p map CPU[0x%lx 0x%llx] GPU[0x%llx 0x%llx] vram %d PTE 0x%llx\n",
+			 prange->svms, last_start, prange->start + i,
+			 gpu_start, gpu_end,
+			 (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
+			 pte_flags);
+
  		r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, true,
-					   NULL, last_start, prange->start + i,
+					   NULL, gpu_start, gpu_end,
  					   pte_flags,

With those fixed, this looks good to me.

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
  					   (last_start - prange->start) << PAGE_SHIFT,
  					   bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,
Reviewed-by on v1 3/8
#8 lore raw

Re: [RFC PATCH v1 3/8] amdkfd/kfd_migrate: Fix GART PTE for non-4K pagesize in svm_migrate_gart_map()

In reply to Donet Tom
Quoted 6 lines · Donet Tom

In svm_migrate_gart_map(), while migrating GART mapping, the number of bytes copied for the GART table only accounts for CPU pages. On non-4K systems, each CPU page can contain multiple GPU pages, and the GART requires one 8-byte PTE per GPU page. As a result, an incorrect size was passed to the DMA, causing only a partial update of the GART table.

Quoted 6 lines

Fix this function to work correctly on non-4K page-size systems by accounting for the number of GPU pages per CPU page when calculating the number of bytes to be copied.

Quoted 17 lines
---
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1 -1
index 59a5a3fea65d..ea8377071c39 100644
@@ -62,7 +62,7 @@ svm_migrate_gart_map(struct amdgpu_ring *ring, u64 npages,
  	*gart_addr = adev->gmc.gart_start;
 
  	num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8);
-	num_bytes = npages * 8;
+	num_bytes = npages * 8 * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
 
  	r = amdgpu_job_alloc_with_ib(adev, &adev->mman.high_pr,
  				     AMDGPU_FENCE_OWNER_UNDEFINED,
author
#31 lore raw
In reply to Christian König
Quoted 77 lines · Christian König
Quoted 76 lines · Donet Tom
Quoted 74 lines · Christian König
Quoted 45 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
   Name:                    gfx90a
   Marketing Name:          AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1]  and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right.  There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them.  In this case we remap the HDP flush registers into one of those register pages.  This allows applications to flush the HDP write FIFO from either the CPU or another device.  This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it).  This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control.  This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Quoted 8 lines
Quoted 3 lines

Sorry, Cristian — I may be misunderstanding this point, so I would appreciate some clarification.

If the CPU page size is 64K and the GPU page size is 4K, then from the GPU side the page table entries are created and mapped at 4K granularity, while on the CPU side the pages remain 64K. To map a single CPU page to the GPU, we therefore need to create multiple GPU page table entries for that CPU page.

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

If possible, could you share the steps to change the hardware page size? I can try testing it on our system.

Quoted 70 lines

Regards,
Christian.

Quoted 66 lines

We found that this was not being handled correctly in the SVM path and addressed it with the change in patch 2/8.

Given this, if the memory is allocated and mapped in GPU page-size (4K) granularity on the GPU side, could you please clarify how memory waste occurs in this scenario?

Thank you for your time and guidance.

Quoted 59 lines

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

author
#6 lore raw

Re: [RFC PATCH v1 2/8] amdkfd/kfd_svm: Fix SVM map/unmap address conversion for non-4k page sizes

In reply to Philip Yang
Quoted 84 lines · Philip Yang
Quoted 30 lines · Donet Tom

SVM range size is tracked using the system page size. The range start and end are aligned to system page-sized PFNs, so the total SVM range size equals the total number of pages in the SVM range multiplied by the system page size.

The SVM range map/unmap functions pass these system page-sized PFN numbers to amdgpu_vm_update_range(), which expects PFNs based on the GPU page size (4K). On non-4K page systems, this mismatch causes only part of the SVM range to be mapped in the GPU page table, while the rest remains unmapped. If the GPU accesses an unmapped address within the same range, it results in a GPU page fault.

To fix this, the required conversion has been added in both svm_range_map_to_gpu() and svm_range_unmap_from_gpu(), ensuring that all pages in the SVM range are correctly mapped on non-4K systems.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 30 ++++++++++++++++++++--------

  1 file changed, 22 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 74a1d3e1d52b..a2636f2d6c71 100644
@@ -1314,11 +1314,16 @@ svm_range_unmap_from_gpu(struct amdgpu_device 
 *adev, struct amdgpu_vm *vm,
                struct dma_fence **fence)
   {

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_unmap_from_gpu(struct amdgpu_device · line 1325

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +5 -1
       uint64_t init_pte_value = 0;
+    uint64_t gpu_start, gpu_end;
   -    pr_debug("[0x%llx 0x%llx]\n", start, last);
+    // Convert CPU page range to GPU page range
+    gpu_start = start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+    gpu_end = (last + 1) * AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
   -    return amdgpu_vm_update_range(adev, vm, false, true, true, false, NULL, start,
-                      last, init_pte_value, 0, 0, NULL, NULL,
+    pr_debug("%s: CPU[0x%llx 0x%llx] -> GPU[0x%llx 0x%llx]\n", 
 __func__,

dynamic debug control can enable function name, linenum print with +pfl, don't add __func__.

Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +3 -0
+         start, last, gpu_start, gpu_end);
+    return amdgpu_vm_update_range(adev, vm, false, true, true, 
 false, NULL, gpu_start,
+                      gpu_end, init_pte_value, 0, 0, NULL, NULL,

                        fence);
  }
  @@ -1398,9 +1403,13 @@ svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange,

Quoted 8 lines

           last_start, last_start + npages - 1, readonly); for (i = offset; i < offset + npages; i++) { +        uint64_t gpu_start;
+        uint64_t gpu_end;
+
          last_domain = dma_addr[i] & SVM_RANGE_VRAM_DOMAIN; dma_addr[i] &= ~SVM_RANGE_VRAM_DOMAIN;
  +

remove extra blank line.

Quoted 23 lines

          /* Collect all pages in the same address range and memory domain
           * that can be mapped with a single call to update mapping. */

drivers/gpu/drm/amd/amdkfd/kfd_svm.c +6 -3
@@ -1415,17 +1424,22 @@ svm_range_map_to_gpu(struct 
 kfd_process_device *pdd, struct svm_range *prange,
           if (readonly)
               pte_flags &= ~AMDGPU_PTE_WRITEABLE;
   -        pr_debug("svms 0x%p map [0x%lx 0x%llx] vram %d PTE 0x%llx\n",
-             prange->svms, last_start, prange->start + i,
-             (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
-             pte_flags);
             /* For dGPU mode, we use same vm_manager to allocate VRAM for
            * different memory partition based on fpfn/lpfn, we should use
            * same vm_manager.vram_base_offset regardless memory partition.
            */
+        gpu_start = last_start * AMDGPU_GPU_PAGES_IN_CPU_PAGE;
+        gpu_end = (prange->start + i + 1) * 
 AMDGPU_GPU_PAGES_IN_CPU_PAGE - 1;
+
+        pr_debug("svms 0x%p map CPU[0x%lx 0x%llx] GPU[0x%llx 0x%llx] 
 vram %d PTE 0x%llx\n",
+             prange->svms, last_start, prange->start + i,
+             gpu_start, gpu_end,
Quoted 8 lines

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_map_to_gpu(struct kfd_process_device *pdd, struct svm_range *prange, · line 1443

Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +4 -1
+             (last_domain == SVM_RANGE_VRAM_DOMAIN) ? 1 : 0,
+             pte_flags);
+
           r = amdgpu_vm_update_range(adev, vm, false, false, flush_tlb, true,
-                       NULL, last_start, prange->start + i,
+                       NULL, gpu_start, gpu_end,
                          pte_flags,

With those fixed, this looks good to me.

Than you. I will fix them in next version.

Quoted 5 lines
Quoted 2 lines

                         (last_start - prange->start) << PAGE_SHIFT, bo_adev ? bo_adev->vm_manager.vram_base_offset : 0,

author
#3 lore raw

Re: [RFC PATCH v1 1/8] drm/amdkfd: Relax size checking during queue buffer get

In reply to Philip Yang
Quoted 68 lines · Philip Yang
Quoted 37 lines · Donet Tom

HW-supported EOP buffer sizes are 4K and 32K. On systems that do not use 4K pages, the minimum buffer object (BO) allocation size is PAGE_SIZE (for example, 64K). During queue buffer acquisition, the driver currently checks the allocated BO size against the supported EOP buffer size. Since the allocated BO is larger than the expected size, this check fails, preventing queue creation.

Relax the strict size validation and allow PAGE_SIZE-sized BOs to be used. Only the required 4K region of the buffer will be used as the EOP buffer and avoids queue creation failures on non-4K page systems.

---
  drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 10 ++++++----

  1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +0 -0
index f1e7583650c4..dc857450fa16 100644
@@ -199,6 +199,7 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, 

void __user *addr, struct amdgpu_
      struct amdgpu_bo_va_mapping *mapping; u64 user_addr;
      u64 size;
+    u64 bo_size;
        user_addr = (u64)addr >> AMDGPU_GPU_PAGE_SHIFT; size = expected_size >> AMDGPU_GPU_PAGE_SHIFT;

drivers/gpu/drm/amd/amdkfd/kfd_queue.c +0 -0
@@ -207,11 +208,12 @@ int kfd_queue_buffer_get(struct amdgpu_vm *vm, 

void __user *addr, struct amdgpu_
      if (!mapping)
          goto out_err;
  -    if (user_addr != mapping->start ||
-        (size != 0 && user_addr + size - 1 != mapping->last)) { -        pr_debug("expected size 0x%llx not equal to mapping addr 0x%llx size 0x%llx\n",
+    bo_size = mapping->last - mapping->start + 1;

Quoted 8 lines

+
+    if (user_addr != mapping->start || (size != 0 && bo_size < size)) { +        pr_debug("expected size 0x%llx grater than mapping addr 0x%llx size 0x%llx\n",
              expected_size, mapping->start << AMDGPU_GPU_PAGE_SHIFT, -            (mapping->last - mapping->start + 1) << AMDGPU_GPU_PAGE_SHIFT);
+            bo_size <<  AMDGPU_GPU_PAGE_SHIFT);

This change works, but also relax the size validation for ring buffer size etc, this may have side effect, for example FW and user space should have the same ring buffer size.

Other buffers already use PAGE_SIZE as expected size or size aligned to PAGE_SIZE, maybe only relax the eop buffer size check

(quoted diff) +0 -0
@@ -275,7 +275,7 @@ int kfd_queue_acquire_buffers(struct 

kfd_process_device *pdd, struct queue_prope

        /* EOP buffer is not required for all ASICs */
        if (properties->eop_ring_buffer_address) {
-               if (properties->eop_ring_buffer_size != topo_dev->node_props.eop_buffer_size) {
+               if (properties->eop_ring_buffer_size < topo_dev->node_props.eop_buffer_size) {
                        pr_debug("queue eop bo size 0x%x not equal to node eop buf size 0x%x\n", properties->eop_ring_buffer_size,
topo_dev->node_props.eop_buffer_size);

(quoted diff) +0 -0
@@ -284,7 +284,7 @@ int kfd_queue_acquire_buffers(struct 

kfd_process_device *pdd, struct queue_prope

Quoted 8 lines

                }
                err = kfd_queue_buffer_get(vm, (void *)properties->eop_ring_buffer_address, &properties->eop_buf_bo,
- properties->eop_ring_buffer_size);
+ ALIGN(properties->eop_ring_buffer_size, PAGE_SIZE));
                if (err)
                        goto out_err_unreserve;
        }

Thank you. I will make this change in next version.

Quoted 6 lines

Regards,
Philip

Quoted 2 lines

          goto out_err;
      }

author
#37 lore raw
In reply to Alex Deucher
Quoted 56 lines · Alex Deucher
Quoted 54 lines · Christian König
Quoted 53 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

Quoted 6 lines
Quoted 1 line

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [  606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4
[  610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [  610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

Quoted 80 lines

Alex

Quoted 78 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

#38 lore raw
Quoted 84 lines · Donet Tom
Quoted 56 lines · Alex Deucher
Quoted 54 lines · Christian König
Quoted 53 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

Quoted 6 lines
Quoted 1 line

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4
[  610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Alex

Quoted 82 lines
Quoted 82 lines

Alex

Quoted 78 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

maintainer
#32 lore raw
In reply to Donet Tom
Quoted 8 lines · Donet Tom
Quoted 5 lines

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

If possible, could you share the steps to change the hardware page size? I can try testing it on our system.

Just typing that down from the front of my head, so don't nail me for 100% correctness.

Modern HW, e.g. gfx9/Vega and newer including all MI* products, has a maximum of 48bits of address space.

Those 48bits are divided on multiple page directories (PD) and a leave page table (PT).

IIRC vm_block_size module parameter controls the size of the PDs. If you set that to 13 instead of the default 9 you should already get 64k PDs instead of 4k PDs. But take that with a grain of salt I think we haven't tested that parameter in the last 10 years or so.

Then each page directory entry on level 0 (PDE0) has a field called block fragment size (see AMDGPU_PDE_BFS for MI products). This controls to how much memory each page table entry (PTE) finally points to.

So putting it all together you should be able to have a configuration with two levels PDs, each covering 13 bits of address space and 64k in size, plus a PT covering 18bits of address space and 2M in size where each PTE points to a 64k block.

Here are the relevant bits from function amdgpu_vm_adjust_size():
...

        tmp = roundup_pow_of_two(adev->vm_manager.max_pfn);
        if (amdgpu_vm_block_size != -1)
                tmp >>= amdgpu_vm_block_size - 9;
        tmp = DIV_ROUND_UP(fls64(tmp) - 1, 9) - 1;
        adev->vm_manager.num_level = min_t(unsigned int, max_level, tmp);
        switch (adev->vm_manager.num_level) {
        case 3:
                adev->vm_manager.root_level = AMDGPU_VM_PDB2;
                break;
        case 2:
                adev->vm_manager.root_level = AMDGPU_VM_PDB1;
                break;
        case 1:
                adev->vm_manager.root_level = AMDGPU_VM_PDB0;
                break;
        default:
                dev_err(adev->dev, "VMPT only supports 2~4+1 levels\n");
        }
        /* block size depends on vm size and hw setup*/
        if (amdgpu_vm_block_size != -1)
                adev->vm_manager.block_size =
                        min((unsigned)amdgpu_vm_block_size, max_bits
                            - AMDGPU_GPU_PAGE_SHIFT
                            - 9 * adev->vm_manager.num_level);
        else if (adev->vm_manager.num_level > 1)
                adev->vm_manager.block_size = 9;
        else
                adev->vm_manager.block_size = amdgpu_vm_get_block_size(tmp);

        if (amdgpu_vm_fragment_size == -1)
                adev->vm_manager.fragment_size = fragment_size_default;
        else
                adev->vm_manager.fragment_size = amdgpu_vm_fragment_size;

...

But again, that is probably tons of work since the AMDGPU_PAGE_SIZE macro needs to change as well and I'm not sure if the FW doesn't internally assume that we have 4k pages somewhere.

Regards,
Christian.

author
#39 lore raw
In reply to Alex Deucher
Quoted 83 lines · Alex Deucher
Quoted 82 lines · Donet Tom
Quoted 56 lines · Alex Deucher
Quoted 54 lines · Christian König
Quoted 53 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

Quoted 6 lines
Quoted 1 line

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4
[  610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues
Quoted 8 lines
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Thanks Alex

I am now suspecting that the queue preemption issue may be related to the unavailability of MMIO remapping. I am not very familiar with this area.

Could you please point me to the relevant code path where the PM4 packet is issued to flush the HDP cache?

I am consistently able to reproduce this issue on my system when using more than three GPUs if patches 7/8 and 8/8 are not applied. In your opinion, is there anything that can be done to speed up the HDP flush or to avoid this situation altogether?

Quoted 84 lines

Alex

Quoted 81 lines
Quoted 81 lines

Alex

Quoted 78 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

author
#33 lore raw
In reply to Christian König
Quoted 47 lines · Christian König
Quoted 7 lines · Donet Tom
Quoted 5 lines

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

If possible, could you share the steps to change the hardware page size? I can try testing it on our system.

Just typing that down from the front of my head, so don't nail me for 100% correctness.

Modern HW, e.g. gfx9/Vega and newer including all MI* products, has a maximum of 48bits of address space.

Those 48bits are divided on multiple page directories (PD) and a leave page table (PT).

IIRC vm_block_size module parameter controls the size of the PDs. If you set that to 13 instead of the default 9 you should already get 64k PDs instead of 4k PDs. But take that with a grain of salt I think we haven't tested that parameter in the last 10 years or so.

Then each page directory entry on level 0 (PDE0) has a field called block fragment size (see AMDGPU_PDE_BFS for MI products). This controls to how much memory each page table entry (PTE) finally points to.

So putting it all together you should be able to have a configuration with two levels PDs, each covering 13 bits of address space and 64k in size, plus a PT covering 18bits of address space and 2M in size where each PTE points to a 64k block.

Here are the relevant bits from function amdgpu_vm_adjust_size():
...

         tmp = roundup_pow_of_two(adev->vm_manager.max_pfn);
         if (amdgpu_vm_block_size != -1)
                 tmp >>= amdgpu_vm_block_size - 9;
         tmp = DIV_ROUND_UP(fls64(tmp) - 1, 9) - 1;
         adev->vm_manager.num_level = min_t(unsigned int, max_level, tmp);
         switch (adev->vm_manager.num_level) {
         case 3:
                 adev->vm_manager.root_level = AMDGPU_VM_PDB2;
                 break;
         case 2:
                 adev->vm_manager.root_level = AMDGPU_VM_PDB1;
                 break;
         case 1:
                 adev->vm_manager.root_level = AMDGPU_VM_PDB0;
                 break;
         default:
                 dev_err(adev->dev, "VMPT only supports 2~4+1 levels\n");
         }
         /* block size depends on vm size and hw setup*/
         if (amdgpu_vm_block_size != -1)
                 adev->vm_manager.block_size =
                         min((unsigned)amdgpu_vm_block_size, max_bits
                             - AMDGPU_GPU_PAGE_SHIFT
                             - 9 * adev->vm_manager.num_level);
         else if (adev->vm_manager.num_level > 1)
Quoted 8 lines
                 adev->vm_manager.block_size = 9;
         else
                 adev->vm_manager.block_size = amdgpu_vm_get_block_size(tmp);

         if (amdgpu_vm_fragment_size == -1)
                 adev->vm_manager.fragment_size = fragment_size_default;
         else
                 adev->vm_manager.fragment_size = amdgpu_vm_fragment_size;

Thanks Christian

I will try it.

Quoted 6 lines

...

But again, that is probably tons of work since the AMDGPU_PAGE_SIZE macro needs to change as well and I'm not sure if the FW doesn't internally assume that we have 4k pages somewhere.

Regards,
Christian.

author
#34 lore raw
In reply to Christian König
Quoted 75 lines · Christian König
Quoted 74 lines · Donet Tom
Quoted 73 lines · Christian König
Quoted 45 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
   Name:                    gfx90a
   Marketing Name:          AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1]  and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 8 lines

Right.  There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them.  In this case we remap the HDP flush registers into one of those register pages.  This allows applications to flush the HDP write FIFO from either the CPU or another device.  This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it).  This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control.  This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Sorry, Cristian — I may be misunderstanding this point, so I would appreciate some clarification.

If the CPU page size is 64K and the GPU page size is 4K, then from the GPU side the page table entries are created and mapped at 4K granularity, while on the CPU side the pages remain 64K. To map a single CPU page to the GPU, we therefore need to create multiple GPU page table entries for that CPU page.

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

Thank you, Christian, for the clarification.

When you say swapping to system memory, does that mean SVM migration to DRAM?

From my understanding of the code, SVM pages are tracked in system page–size PFNs, which on our system is 64 KB. With a 64 KB base page size, buffer objects (BOs) are allocated in 64 KB–aligned chunks, both in VRAM and GTT, while the GPU page-table mappings are still created using 4 KB pages.

During SVM migration from VRAM to system memory, I observed that an entire 64 KB page is migrated. Similarly, when XNACK is enabled, if the GPU accesses a 4 KB page, my understanding is that the entire 64 KB page is migrated.

If my understanding is correct, allocating 4 KB memory on a 64 KB page–size system results in a 64 KB BO allocation, meaning that around 60 KB is effectively wasted. Are you referring to this kind of over-allocation potentially leading to OOM situations under memory pressure?

Since I am still getting familiar with the AMDGPU codebase, could you please point me to the locations where special handling is implemented to avoid OOM conditions during swapping or migration?

Quoted 72 lines

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

Regards,
Christian.

Quoted 66 lines

We found that this was not being handled correctly in the SVM path and addressed it with the change in patch 2/8.

Given this, if the memory is allocated and mapped in GPU page-size (4K) granularity on the GPU side, could you please clarify how memory waste occurs in this scenario?

Thank you for your time and guidance.

Quoted 59 lines

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

maintainer
#35 lore raw
Quoted 7 lines · Donet Tom
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Sorry, Cristian — I may be misunderstanding this point, so I would appreciate some clarification.

If the CPU page size is 64K and the GPU page size is 4K, then from the GPU side the page table entries are created and mapped at 4K granularity, while on the CPU side the pages remain 64K. To map a single CPU page to the GPU, we therefore need to create multiple GPU page table entries for that CPU page.

Quoted 8 lines
Quoted 3 lines

The GPU page tables are 4k in size no matter what the CPU page size is and there is some special handling so that we can allocate them even under memory pressure. Background is that you sometimes need to split up higher order pages (1G, 2M) into lower order pages (2M, 4k) to be able to swap things to system memory for example and for that you need some an extra layer of page tables.

The problem is now that those 4k pages are rounded up to your CPU page size, resulting in both wasting quite some memory as well as messing up the special handling to not run into OOM situations when swapping things to system memory....

Thank you, Christian, for the clarification.

When you say swapping to system memory, does that mean SVM migration to DRAM?

Yes and no. It's mostly the normal BO based swapping of TTM. SVM is still an experimental and extremely rarely used feature.

Quoted 6 lines

From my understanding of the code, SVM pages are tracked in system page–size PFNs, which on our system is 64 KB. With a 64 KB base page size, buffer objects (BOs) are allocated in 64 KB–aligned chunks, both in VRAM and GTT, while the GPU page-table mappings are still created using 4 KB pages.

During SVM migration from VRAM to system memory, I observed that an entire 64 KB page is migrated. Similarly, when XNACK is enabled, if the GPU accesses a 4 KB page, my understanding is that the entire 64 KB page is migrated.

If my understanding is correct, allocating 4 KB memory on a 64 KB page–size system results in a 64 KB BO allocation, meaning that around 60 KB is effectively wasted. Are you referring to this kind of over-allocation potentially leading to OOM situations under memory pressure?

Correct, yes.

Quoted 1 line

Since I am still getting familiar with the AMDGPU codebase, could you please point me to the locations where special handling is implemented to avoid OOM conditions during swapping or migration?

See AMDGPU_VM_RESERVED_VRAM.

Regards,
Christian.

Quoted 74 lines
Quoted 74 lines

What we could potentially do is to switch to 64k pages on the GPU as well (the HW is flexible enough to be re-configurable), but that is tons of changes and probably not easily testable.

Regards,
Christian.

Quoted 66 lines

We found that this was not being handled correctly in the SVM path and addressed it with the change in patch 2/8.

Given this, if the memory is allocated and mapped in GPU page-size (4K) granularity on the GPU side, could you please clarify how memory waste occurs in this scenario?

Thank you for your time and guidance.

Quoted 59 lines

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

#40 lore raw
In reply to Donet Tom
Quoted 96 lines · Donet Tom
Quoted 83 lines · Alex Deucher
Quoted 82 lines · Donet Tom
Quoted 56 lines · Alex Deucher
Quoted 54 lines · Christian König
Quoted 53 lines · Alex Deucher
Quoted 35 lines · Christian König
Quoted 13 lines · Ritesh Harjani (IBM)
Quoted 12 lines · Christian König
Quoted 12 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize.
AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2] to test these changes. Is there anything else that you would suggest us to run to shake out any other page size related issues w.r.t the kernel driver?

Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation and will not be back before the end of the year.

I can check on Monday if some people are still around which could answer a couple of questions, but in general don't expect a quick response.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop ring buffer size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of all the events and actions the CP should execute when shaders and cache flushes finish.

The size depends on the HW generation and configuration of the GPU etc..., but don't ask me for details how that is calculated.

The point is that the size is completely unrelated to the CPU, so using PAGE_SIZE is clearly incorrect.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

3. Patch 5/8: also have a query w.r.t the error paths when system page size > 4K. Do we need to lift this restriction and add MMIO remap support for systems with non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW limitation then?

The problem is that the CPU must map some of the registers/resources of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

Quoted 6 lines
Quoted 1 line

As far as I know this is only done for graphics shader submissions to the classic CS interface, but not for compute dispatches through ROCm queues.

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4
[  610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for
queue with doorbell_id: 80030008
[  610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues
[  610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues
Quoted 8 lines
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Quoted 7 lines

Thanks Alex

I am now suspecting that the queue preemption issue may be related to the unavailability of MMIO remapping. I am not very familiar with this area.

Could you please point me to the relevant code path where the PM4 packet is issued to flush the HDP cache?

+ David who is more familiar with the ROCm runtime.

PM4 has a packet called HDP_FLUSH which flushes the HDP. For AQL, it's handled by one of the flags I think. Most things in ROCm use AQL.

@David Yat Sin Can you point to how HDP flushes are handled in the ROCm runtime?

Alex

Quoted 92 lines

I am consistently able to reproduce this issue on my system when using more than three GPUs if patches 7/8 and 8/8 are not applied. In your opinion, is there anything that can be done to speed up the HDP flush or to avoid this situation altogether?

Quoted 84 lines

Alex

Quoted 81 lines
Quoted 81 lines

Alex

Quoted 78 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for 100% correctness. It does work most of the time without it, but you then risk data corruption.

Apart from making the flush/invalidate an IOCTL I think we could also just use a global dummy page in VRAM.

If you make two 32bit writes which are apart from each other and then a read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

The only tricky part is that you need to get the HW barriers with the doorbell write right.....

Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global aggregated one needs to be 4k as well, or is it ok to over allocate there?

Thinking more about it there is also a major problem with page tables. Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Christian.

Quoted 57 lines

Alex

Quoted 55 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1] ROCr debug agent tests: https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests: https://github.com/ROCm/rccl/tree/develop/test

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Well this is generally nice to have, but there are unfortunately some HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

That's a bummer :(
- Do we have some HW documentation around what are these limitations around non-4K pagesize? Any links to such please?

You already mentioned MMIO remap which obviously has that problem, but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

This can all be worked around by delegating those MMIO accesses into the kernel, but that means tons of extra IOCTL overhead.

Especially the cache flushes which are necessary to avoid corruption are really bad for performance in such an approach.

Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such restrictions?

Not that I know off any.

Quoted 3 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.

- Maybe we should also document, what will work and what won't work due to these HW limitations.

Well pretty much everything, I need to double check how ROCm does HDP flushing/invalidating when the MMIO remap isn't available.

Could be that there is already a fallback path and that's the reason why this approach actually works at all.

Quoted 5 lines
Quoted 1 line

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 5 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the tests passing.

Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds can be implemented for those issues.

Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all people are available any more.

Regards,
Christian.

Quoted 1 line

-ritesh

#41 lore raw

[AMD Official Use Only - AMD Internal Distribution Only]

HDP flush is done in ROCm using these 3 methods:

1. For AQL packets, this is done by setting the system-scope acquire and release fences in the packet header. For example, it is set here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_kernel.cpp#L878

And when the headers are defined here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L85

2. Via a SDMA packet. This is done before doing a memory copy:
The function is called here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L484
And the packet (POLL_REGMEM) is generated here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L1154

3. By writing to a MMIO remapped address:
The address is stored in rocclr here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocdevice.cpp#L607

And the flush is triggered by writing a 1, e.g here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L3831

Regards,
David

Quoted 271 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Wednesday, December 17, 2025 9:23 AM
To: Donet Tom <[email protected]>; Yat Sin, David <[email protected]>
Cc: Koenig, Christian <[email protected]>; Ritesh Harjani (IBM) <[email protected]>; [email protected]; Kuehling, Felix <[email protected]>; Deucher, Alexander <[email protected]>; Russell, Kent <[email protected]>; Vaidyanathan Srinivasan <[email protected]>; Mukesh Kumar Chaurasiya <[email protected]>
Subject: Re: [RFC PATCH v1 0/8] amdgpu/amdkfd: Add support for non-4K page size systems

Quoted 4 lines · Donet Tom
Quoted 1 line · Alex Deucher

On Tue, Dec 16, 2025 at 8:55 AM Donet Tom <[email protected]>

Quoted 16 lines · wrote:
Quoted 16 lines
Quoted 16 lines · Alex Deucher
Quoted 14 lines
Quoted 12 lines · Christian König
Quoted 11 lines · Alex Deucher
Quoted 9 lines · Christian König
Quoted 8 lines · Ritesh Harjani (IBM)
Quoted 7 lines · Christian König
Quoted 7 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize. AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2]

to test

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

these changes. Is there anything else that you would suggest us

to run to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shake out any other page size related issues w.r.t the kernel

driver?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation

and will not be back before the end of the year.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I can check on Monday if some people are still around which could

answer a couple of questions, but in general don't expect a quick response.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop

ring buffer

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of

all the events and actions the CP should execute when shaders and cache flushes finish.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The size depends on the HW generation and configuration of the GPU

etc..., but don't ask me for details how that is calculated.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The point is that the size is completely unrelated to the CPU, so using

PAGE_SIZE is clearly incorrect.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

3. Patch 5/8: also have a query w.r.t the error paths when system

page size > 4K.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Do we need to lift this restriction and add MMIO remap support

for systems with

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW

limitation then?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The problem is that the CPU must map some of the registers/resources

of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the

classic CS interface, but not for compute dispatches through ROCm queues.

Quoted 34 lines
Quoted 24 lines
Quoted 24 lines
Quoted 5 lines

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following
errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed

for queue with doorbell_id: 80030008 [ 606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4 [ 610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [ 610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues

Quoted 8 lines
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Quoted 7 lines

Thanks Alex

I am now suspecting that the queue preemption issue may be related to the unavailability of MMIO remapping. I am not very familiar with this area.

Could you please point me to the relevant code path where the PM4 packet is issued to flush the HDP cache?

+ David who is more familiar with the ROCm runtime.

PM4 has a packet called HDP_FLUSH which flushes the HDP. For AQL, it's handled by one of the flags I think. Most things in ROCm use AQL.

@David Yat Sin Can you point to how HDP flushes are handled in the ROCm runtime?

Alex

Quoted 19 lines

I am consistently able to reproduce this issue on my system when using more than three GPUs if patches 7/8 and 8/8 are not applied. In your opinion, is there anything that can be done to speed up the HDP flush or to avoid this situation altogether?

Quoted 11 lines

Alex

Quoted 8 lines
Quoted 8 lines

Alex

Quoted 5 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for

100% correctness. It does work most of the time without it, but you then risk data corruption.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Apart from making the flush/invalidate an IOCTL I think we could also just

use a global dummy page in VRAM.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

If you make two 32bit writes which are apart from each other and then a

read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The only tricky part is that you need to get the HW barriers with the doorbell

write right.....

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global

aggregated one needs to be 4k as well, or is it ok to over allocate there?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Thinking more about it there is also a major problem with page tables.

Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Christian.

Quoted 8 lines

Alex

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

[1] ROCr debug agent tests:
https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests:
https://github.com/ROCm/rccl/tree/develop/test

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K

pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some

HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That's a bummer :(
- Do we have some HW documentation around what are these

limitations around non-4K pagesize? Any links to such please?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

You already mentioned MMIO remap which obviously has that problem,

but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This can all be worked around by delegating those MMIO accesses into

the kernel, but that means tons of extra IOCTL overhead.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Especially the cache flushes which are necessary to avoid corruption

are really bad for performance in such an approach.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

- Are there any latest AMD GPU versions which maybe lifts such

restrictions?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Not that I know off any.

Quoted 1 line
Quoted 1 line

What we can do is to support graphics and MM, but that should

already work out of the box.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

- Maybe we should also document, what will work and what won't work

due to these HW limitations.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well pretty much everything, I need to double check how ROCm does

HDP flushing/invalidating when the MMIO remap isn't available.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Could be that there is already a fallback path and that's the reason why

this approach actually works at all.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k

pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the

tests passing.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I need to talk with Alex and the ROCm team about it if workarounds

can be implemented for those issues.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all

people are available any more.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Regards,
Christian.

Quoted 1 line

-ritesh

author
#21 lore raw

Re: [RFC PATCH v1 7/8] amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead of CPU page size

In reply to Christian König
Quoted 11 lines · Christian König
Quoted 10 lines · Donet Tom

The ctl_stack_size and wg_data_size values are used to compute the total context save/restore buffer size and the control stack size. These buffers are programmed into the GPU and are used to store the queue state during context save and restore.

Currently, both ctl_stack_size and wg_data_size are aligned to the CPU PAGE_SIZE. On systems with a non-4K CPU page size, this causes unnecessary memory waste because the GPU internally calculates and uses buffer sizes aligned to a fixed 4K GPU page size.

Quoted 6 lines
Quoted 5 lines

Since the control stack and context save/restore buffers are consumed by the GPU, their sizes should be aligned to the GPU page size (4K), not the CPU page size. This patch updates the alignment of ctl_stack_size and wg_data_size to prevent over-allocation on systems with larger CPU page sizes.

As far as I know the problem is that the debugger needs to consume that stuff on the CPU side as well.

Thank you for your help.

As mentioned earlier, we were observing some queue preemption and GPU hang issues. To address this, we introduced a patch, and after applying the 7/8 and 8/8 patches, those issues have not been seen anymore

While debugging the GPU hang issue, I made some additional observations.

On my system, I booted a kernel with a 4 KB system page size and modified both the ROCR runtime and the GPU driver to set the control stack size to 64 KB. Even on a 4 KB page-size system, using a 64 KB control stack size reliably reproduces the queue preemption failure when running RCCL unit tests on 8 GPUs. This suggests that the issue is not related to the system page size, but rather to the control stack size being exactly 64 KB.

When the control stack size is set to 64 KB ± 4 KB, the tests pass on both 4 KB and 64 KB system page-size configurations.

For gfxv9, is there any documented hardware limitation on the control stack size? Specifically, is it valid to use a control stack size of exactly 64 KB?

Quoted 2 lines

I need to double check that, but I think the alignment is correct as it is.

The control stack is part of the context save-restore buffer, and we configure it on the GPU as shown below:

m->cp_hqd_ctx_save_base_addr_lo = lower_32_bits(q->ctx_save_restore_area_address); m->cp_hqd_ctx_save_base_addr_hi = upper_32_bits(q->ctx_save_restore_area_address); m->cp_hqd_ctx_save_size = q->ctx_save_restore_area_size; m->cp_hqd_cntl_stack_size = q->ctl_stack_size; m->cp_hqd_cntl_stack_offset = q->ctl_stack_size; m->cp_hqd_wg_state_offset = q->ctl_stack_size;

The control stack occupies the region from cp_hqd_cntl_stack_offset down to 0 within the ctx save restore area, and the remaining space is used for WG state. This buffer is fully managed by the GPU during preemption and restore operations. The control stack size is calculated based on hardware configuration (CU count and wave count). For example, on gfxv9, the size is typically around 32 KB. If we align this size to the system page size (e.g., 64 KB), two issues arise:

1. Unnecessary memory overhead.
2. Potential queue preemption issues.

On the CPU side, we copy the control stack contents to other buffers for processing. Since the control stack size is derived from hardware configuration, aligning it to the GPU page size seems more appropriate. Aligning to the system page size would waste memory without adding value. Using GPU page size alignment ensures consistency with hardware and avoids unnecessary overhead.

Would you agree that aligning the control stack size to the GPU page size is the right approach? Or do you see any concerns with this method?

Quoted 36 lines

Regards,
Christian.

Quoted 32 lines
---
  drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +4 -3
index dc857450fa16..00ab941c3e86 100644
@@ -445,10 +445,11 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
  		    min(cu_num * 40, props->array_count / props->simd_arrays_per_engine * 512)
  		    : cu_num * 32;
 
-	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props), PAGE_SIZE);
+	wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props),
+				AMDGPU_GPU_PAGE_SIZE);
  	ctl_stack_size = wave_num * CNTL_STACK_BYTES_PER_WAVE(gfxv) + 8;
  	ctl_stack_size = ALIGN(SIZEOF_HSA_USER_CONTEXT_SAVE_AREA_HEADER + ctl_stack_size,
-			       PAGE_SIZE);
+			       AMDGPU_GPU_PAGE_SIZE);
 
  	if ((gfxv / 10000 * 10000) == 100000) {
  		/* HW design limits control stack size to 0x7000.
@@ -460,7 +461,7 @@ void kfd_queue_ctx_save_restore_size(struct kfd_topology_device *dev)
 
  	props->ctl_stack_size = ctl_stack_size;
  	props->debug_memory_size = ALIGN(wave_num * DEBUGGER_BYTES_PER_WAVE, DEBUGGER_BYTES_ALIGN);
-	props->cwsr_size = ctl_stack_size + wg_data_size;
+	props->cwsr_size = ALIGN(ctl_stack_size + wg_data_size, PAGE_SIZE);
 
  	if (gfxv == 80002)	/* GFX_VERSION_TONGA */
  		props->eop_buffer_size = 0x8000;
author
#42 lore raw
In reply to Yat Sin, David
Quoted 19 lines · David

[AMD Official Use Only - AMD Internal Distribution Only]

HDP flush is done in ROCm using these 3 methods:

1. For AQL packets, this is done by setting the system-scope acquire and release fences in the packet header. For example, it is set here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_kernel.cpp#L878

And when the headers are defined here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L85

2. Via a SDMA packet. This is done before doing a memory copy:
The function is called here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L484
And the packet (POLL_REGMEM) is generated here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L1154

Quoted 6 lines

3. By writing to a MMIO remapped address:
The address is stored in rocclr here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocdevice.cpp#L607

And the flush is triggered by writing a 1, e.g here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L3831

Thank you David.

Quoted 248 lines

Regards,
David

Quoted 242 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Wednesday, December 17, 2025 9:23 AM
To: Donet Tom <[email protected]>; Yat Sin, David <[email protected]>
Cc: Koenig, Christian <[email protected]>; Ritesh Harjani (IBM) <[email protected]>; [email protected]; Kuehling, Felix <[email protected]>; Deucher, Alexander <[email protected]>; Russell, Kent <[email protected]>; Vaidyanathan Srinivasan <[email protected]>; Mukesh Kumar Chaurasiya <[email protected]>
Subject: Re: [RFC PATCH v1 0/8] amdgpu/amdkfd: Add support for non-4K page size systems

Quoted 3 lines · Donet Tom
Quoted 1 line · Alex Deucher

On Tue, Dec 16, 2025 at 8:55 AM Donet Tom <[email protected]>

Quoted 15 lines · wrote:
Quoted 15 lines
Quoted 15 lines · Alex Deucher
Quoted 14 lines
Quoted 12 lines · Christian König
Quoted 11 lines · Alex Deucher
Quoted 9 lines · Christian König
Quoted 8 lines · Ritesh Harjani (IBM)
Quoted 7 lines · Christian König
Quoted 7 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize. AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2]

to test

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

these changes. Is there anything else that you would suggest us

to run to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shake out any other page size related issues w.r.t the kernel

driver?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation

and will not be back before the end of the year.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I can check on Monday if some people are still around which could

answer a couple of questions, but in general don't expect a quick response.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop

ring buffer

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of

all the events and actions the CP should execute when shaders and cache flushes finish.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The size depends on the HW generation and configuration of the GPU

etc..., but don't ask me for details how that is calculated.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The point is that the size is completely unrelated to the CPU, so using

PAGE_SIZE is clearly incorrect.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

3. Patch 5/8: also have a query w.r.t the error paths when system

page size > 4K.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Do we need to lift this restriction and add MMIO remap support

for systems with

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW

limitation then?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The problem is that the CPU must map some of the registers/resources

of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the

classic CS interface, but not for compute dispatches through ROCm queues.

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
Quoted 5 lines

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following
errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed

for queue with doorbell_id: 80030008 [ 606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4 [ 610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [ 610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues

Quoted 8 lines
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Quoted 7 lines

Thanks Alex

I am now suspecting that the queue preemption issue may be related to the unavailability of MMIO remapping. I am not very familiar with this area.

Could you please point me to the relevant code path where the PM4 packet is issued to flush the HDP cache?

+ David who is more familiar with the ROCm runtime.

PM4 has a packet called HDP_FLUSH which flushes the HDP. For AQL, it's handled by one of the flags I think. Most things in ROCm use AQL.

@David Yat Sin Can you point to how HDP flushes are handled in the ROCm runtime?

Alex

Quoted 16 lines

I am consistently able to reproduce this issue on my system when using more than three GPUs if patches 7/8 and 8/8 are not applied. In your opinion, is there anything that can be done to speed up the HDP flush or to avoid this situation altogether?

Quoted 9 lines

Alex

Quoted 7 lines
Quoted 7 lines

Alex

Quoted 5 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for

100% correctness. It does work most of the time without it, but you then risk data corruption.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Apart from making the flush/invalidate an IOCTL I think we could also just

use a global dummy page in VRAM.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If you make two 32bit writes which are apart from each other and then a

read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The only tricky part is that you need to get the HW barriers with the doorbell

write right.....

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global

aggregated one needs to be 4k as well, or is it ok to over allocate there?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Thinking more about it there is also a major problem with page tables.

Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Christian.

Quoted 8 lines

Alex

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

[1] ROCr debug agent tests:
https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests:
https://github.com/ROCm/rccl/tree/develop/test

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K

pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some

HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That's a bummer :(
- Do we have some HW documentation around what are these

limitations around non-4K pagesize? Any links to such please?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

You already mentioned MMIO remap which obviously has that problem,

but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This can all be worked around by delegating those MMIO accesses into

the kernel, but that means tons of extra IOCTL overhead.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Especially the cache flushes which are necessary to avoid corruption

are really bad for performance in such an approach.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such

restrictions?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Not that I know off any.

Quoted 1 line
Quoted 1 line

What we can do is to support graphics and MM, but that should

already work out of the box.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Maybe we should also document, what will work and what won't work

due to these HW limitations.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well pretty much everything, I need to double check how ROCm does

HDP flushing/invalidating when the MMIO remap isn't available.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Could be that there is already a fallback path and that's the reason why

this approach actually works at all.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k

pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the

tests passing.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I need to talk with Alex and the ROCm team about it if workarounds

can be implemented for those issues.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all

people are available any more.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Regards,
Christian.

Quoted 1 line

-ritesh

author
#12 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Christian König
Quoted 7 lines · Christian König
Quoted 2 lines · Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 7 lines
Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Hi Christian, Felix, Alex and philip

Instead of hardcoding the AMDGPU_GTT_MAX_TRANSFER_SIZE value to 512, what do you think about doing something like the change below? This should work across all architectures and page sizes, so AMDGPU_GTT_MAX_TRANSFER_SIZE will always correspond to the PMD size on all architectures and with all page sizes.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +0 -0
index 0be2728aa872..c594ed7dff18 100644
@@ -37,7 +37,7 @@

 #define AMDGPU_PL_MMIO_REMAP   (TTM_PL_PRIV + 5)
 #define __AMDGPU_PL_NUM        (TTM_PL_PRIV + 6)

-#define AMDGPU_GTT_MAX_TRANSFER_SIZE   512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE   1 << (PMD_SHIFT - PAGE_SHIFT) #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS

Could you please provide your thoughts on above? Is it looking ok to you?

If this looks good - here is what we were thinking:

Patches 1-4 are required to fix initial non-4k pagesize support to AMD GPU. And since these patches are looking in good shape (since Philip has already reviewed [1-3])- We thought it will be good to split the patch series into two. I will send a v2 of Part-1 with patches [1-4] (will also address the review comments in v2 for Patch-1 & 2 from Philip) and for the rest of the patches [5-8] Part-2, we can continue the discussion till other things are sorted. That will also allow us to get these initial fixes in Part-1 ready before the 6.20 merge window.

Thoughts?

Quoted 23 lines

Regards,
Christian.

Quoted 19 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
  extern const struct attribute_group amdgpu_vram_mgr_attr_group;
author
#43 lore raw
In reply to Yat Sin, David
Quoted 19 lines · David

[AMD Official Use Only - AMD Internal Distribution Only]

HDP flush is done in ROCm using these 3 methods:

1. For AQL packets, this is done by setting the system-scope acquire and release fences in the packet header. For example, it is set here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_kernel.cpp#L878

And when the headers are defined here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L85

2. Via a SDMA packet. This is done before doing a memory copy:
The function is called here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L484
And the packet (POLL_REGMEM) is generated here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_blit_sdma.cpp#L1154

Quoted 6 lines

3. By writing to a MMIO remapped address:
The address is stored in rocclr here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocdevice.cpp#L607

And the flush is triggered by writing a 1, e.g here:
https://github.com/ROCm/rocm-systems/blob/develop/projects/clr/rocclr/device/rocm/rocvirtual.cpp#L3831

Hi David, Alex and Christian

Thank you for pointing me to the code where the HDP flush is handled.

From my understanding of the code, if MMIO remap is supported, the HDP flush is performed via MMIO remap. If MMIO remap is not supported, a workaround is used:
"write all arguments → sfence → write the last byte → mfence → read the last byte"

This sequence ensures that the HDP flush is completed before the GPU reads the data.

With this understanding, I think patch 5/8 (amdkfd/kfd_chardev: Add error message for non-4K page size failures) may not be needed, as it prints an error when MMIO remap is not supported, which is an expected case.

Should we drop patch 5/8, or do you think it would be better to change this to a pr_warn or pr_info message when MMIO remap is not supported?

Quoted 248 lines

Regards,
David

Quoted 242 lines

-----Original Message-----
From: Alex Deucher<[email protected]>
Sent: Wednesday, December 17, 2025 9:23 AM
To: Donet Tom<[email protected]>; Yat Sin, David <[email protected]>
Cc: Koenig, Christian<[email protected]>; Ritesh Harjani (IBM) <[email protected]>;[email protected]; Kuehling, Felix <[email protected]>; Deucher, Alexander <[email protected]>; Russell, Kent<[email protected]>; Vaidyanathan Srinivasan<[email protected]>; Mukesh Kumar Chaurasiya <[email protected]>
Subject: Re: [RFC PATCH v1 0/8] amdgpu/amdkfd: Add support for non-4K page size systems

Quoted 3 lines · Donet Tom
Quoted 1 line · Alex Deucher

On Tue, Dec 16, 2025 at 8:55 AM Donet Tom<[email protected]>

Quoted 15 lines · wrote:
Quoted 15 lines
Quoted 15 lines · Alex Deucher
Quoted 14 lines
Quoted 12 lines · Christian König
Quoted 11 lines · Alex Deucher
Quoted 9 lines · Christian König
Quoted 8 lines · Ritesh Harjani (IBM)
Quoted 7 lines · Christian König
Quoted 7 lines

Setup details:
============
System details: Power10 LPAR using 64K pagesize. AMD GPU:
Name: gfx90a
Marketing Name: AMD Instinct MI210

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Queries:
=======
1. We currently ran rocr-debug agent tests [1] and rccl unit tests [2]

to test

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

these changes. Is there anything else that you would suggest us

to run to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

shake out any other page size related issues w.r.t the kernel

driver?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

The ROCm team needs to answer that.

Is there any separate mailing list or list of people whom we can cc then?

With Felix on CC you already got the right person, but he's on vacation

and will not be back before the end of the year.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I can check on Monday if some people are still around which could

answer a couple of questions, but in general don't expect a quick response.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

2. Patch 1/8: We have a querry regarding eop buffer size Is this eop

ring buffer

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

size HW dependent? Should it be made PAGE_SIZE?

Yes and no.

If you could more elaborate on this please? I am assuming you would anyway respond with more context / details on Patch-1 itself. If yes, that would be great!

Well, in general the EOP (End of Pipe) buffer contains in a ring buffer of

all the events and actions the CP should execute when shaders and cache flushes finish.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The size depends on the HW generation and configuration of the GPU

etc..., but don't ask me for details how that is calculated.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The point is that the size is completely unrelated to the CPU, so using

PAGE_SIZE is clearly incorrect.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

3. Patch 5/8: also have a query w.r.t the error paths when system

page size > 4K.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Do we need to lift this restriction and add MMIO remap support

for systems with

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

non-4K page sizes?

The problem is the HW can't do this.

We aren't that familiar with the HW / SW stack here. Wanted to understand what functionality will be unsupported due to this HW

limitation then?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The problem is that the CPU must map some of the registers/resources

of the GPU into the address space of the application and you run into security issues when you map more than 4k at a time.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Right. There are some 4K pages with the MMIO register BAR which are empty and registers can be remapped into them. In this case we remap the HDP flush registers into one of those register pages. This allows applications to flush the HDP write FIFO from either the CPU or another device. This is needed to flush data written by the CPU or another device to the VRAM BAR out to VRAM (i.e., so the GPU can see it). This is flushed internally as part of the shader dispatch packets,

As far as I know this is only done for graphics shader submissions to the

classic CS interface, but not for compute dispatches through ROCm queues.

Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
Quoted 5 lines

There is an explicit PM4 packet to flush the HDP cache for userqs and for AQL the flush is handled via one of the flags in the dispatch packet. The MMIO remap is needed for more fine grained use cases where you might have the CPU or another device operating in a gang like scenario with the GPU.

Thank you, Alex.

We were encountering an issue while running the RCCL unit tests. With 2 GPUs, all tests passed successfully; however, when running with more than 2 GPUs, the tests began to fail at random points with the following
errors:

[  598.576821] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed

for queue with doorbell_id: 80030008 [ 606.696820] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 606.696826] amdgpu 0048:0f:00.0: amdgpu: GPU reset begin!. Source: 4 [ 610.696852] amdgpu 0048:0f:00.0: amdgpu: Queue preemption failed for queue with doorbell_id: 80030008 [ 610.696869] amdgpu 0048:0f:00.0: amdgpu: Failed to evict process queues [ 610.696942] amdgpu 0048:0f:00.0: amdgpu: Failed to restore process queues

Quoted 8 lines
Quoted 5 lines

After applying patches 7/8 and 8/8, we are no longer seeing this issue.

One question I have is: we only started observing this problem when the number of GPUs increased. Could this be related to MMIO remapping not being available?

It could be. E.g., if the CPU or a GPU writes data to VRAM on another GPU, you will need to flush the HDP to make sure that data hits VRAM before the GPU attached to the VRAM can see it.

Quoted 7 lines

Thanks Alex

I am now suspecting that the queue preemption issue may be related to the unavailability of MMIO remapping. I am not very familiar with this area.

Could you please point me to the relevant code path where the PM4 packet is issued to flush the HDP cache?

+ David who is more familiar with the ROCm runtime.

PM4 has a packet called HDP_FLUSH which flushes the HDP. For AQL, it's handled by one of the flags I think. Most things in ROCm use AQL.

@David Yat Sin Can you point to how HDP flushes are handled in the ROCm runtime?

Alex

Quoted 16 lines

I am consistently able to reproduce this issue on my system when using more than three GPUs if patches 7/8 and 8/8 are not applied. In your opinion, is there anything that can be done to speed up the HDP flush or to avoid this situation altogether?

Quoted 9 lines

Alex

Quoted 7 lines
Quoted 7 lines

Alex

Quoted 5 lines

That's the reason why ROCm needs the remapped MMIO register BAR.

Quoted 2 lines

but there are certain cases where an application may want more control. This is probably not a showstopper for most ROCm apps.

Well the problem is that you absolutely need the HDP flush/invalidation for

100% correctness. It does work most of the time without it, but you then risk data corruption.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Apart from making the flush/invalidate an IOCTL I think we could also just

use a global dummy page in VRAM.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

If you make two 32bit writes which are apart from each other and then a

read back a 32bit value from VRAM that should invalidate the HDP as well. It's less efficient than the MMIO BAR remap but still much better than going though an IOCTL.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The only tricky part is that you need to get the HW barriers with the doorbell

write right.....

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

That said, the region is only 4K so if you allow applications to map a larger region they would get access to GPU register pages which they shouldn't have access to.

But don't we also have problems with the doorbell? E.g. the global

aggregated one needs to be 4k as well, or is it ok to over allocate there?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Thinking more about it there is also a major problem with page tables.

Those are 4k by default on modern systems as well and while over allocating them to 64k is possible that not only wastes some VRAM but can also result in OOM situations because we can't allocate the necessary page tables to switch from 2MiB to 4k pages in some cases.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Christian.

Quoted 8 lines

Alex

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

[1] ROCr debug agent tests:
https://github.com/ROCm/rocr_debug_agent
[2] RCCL tests:
https://github.com/ROCm/rccl/tree/develop/test

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Please note that the changes in this series are on a best effort basis from our end. Therefore, requesting the amd-gfx community (who have deeper knowledge of the HW & SW stack) to kindly help with the review and provide feedback / comments on these patches. The idea here is, to also have non-4K

pagesize (e.g. 64K) well supported with amd gpu kernel driver.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well this is generally nice to have, but there are unfortunately some

HW limitations which makes ROCm pretty much unusable on non 4k page size systems.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

That's a bummer :(
- Do we have some HW documentation around what are these

limitations around non-4K pagesize? Any links to such please?

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

You already mentioned MMIO remap which obviously has that problem,

but if I'm not completely mistaken the PCIe doorbell BAR and some global seq counter resources will also cause problems here.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This can all be worked around by delegating those MMIO accesses into

the kernel, but that means tons of extra IOCTL overhead.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Especially the cache flushes which are necessary to avoid corruption

are really bad for performance in such an approach.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Are there any latest AMD GPU versions which maybe lifts such

restrictions?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Not that I know off any.

Quoted 1 line
Quoted 1 line

What we can do is to support graphics and MM, but that should

already work out of the box.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

- Maybe we should also document, what will work and what won't work

due to these HW limitations.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Well pretty much everything, I need to double check how ROCm does

HDP flushing/invalidating when the MMIO remap isn't available.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Could be that there is already a fallback path and that's the reason why

this approach actually works at all.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

What we can do is to support graphics and MM, but that should already work out of the box.>

So these patches helped us resolve most of the issues like SDMA hangs and GPU kernel page faults which we saw with rocr and rccl tests with 64K pagesize. Meaning, we didn't see this working out of box perhaps due to 64K pagesize.

Yeah, but this is all for ROCm and not the graphics side.

To be honest I'm not sure how ROCm even works when you have 64k

pages at the moment. I would expect much more issue lurking in the kernel driver.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

AFAIU, some of these patches may require re-work based on reviews, but at least with these changes, we were able to see all the

tests passing.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I need to talk with Alex and the ROCm team about it if workarounds

can be implemented for those issues.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Thanks a lot! That would be super helpful!

Quoted 3 lines

Regards,
Christian.

Thanks again for the quick response on the patch series.

You are welcome, but since it's so near to the end of the year not all

people are available any more.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Regards,
Christian.

Quoted 1 line

-ritesh

author
#22 lore raw

Re: [RFC PATCH v1 7/8] amdgpu: Align ctl_stack_size and wg_data_size to GPU page size instead of CPU page size

In reply to Donet Tom
Quoted 25 lines · Donet Tom
Quoted 11 lines · Christian König
Quoted 10 lines · Donet Tom

The ctl_stack_size and wg_data_size values are used to compute the total context save/restore buffer size and the control stack size. These buffers are programmed into the GPU and are used to store the queue state during context save and restore.

Currently, both ctl_stack_size and wg_data_size are aligned to the CPU PAGE_SIZE. On systems with a non-4K CPU page size, this causes unnecessary memory waste because the GPU internally calculates and uses buffer sizes aligned to a fixed 4K GPU page size.

Quoted 6 lines
Quoted 5 lines

Since the control stack and context save/restore buffers are consumed by the GPU, their sizes should be aligned to the GPU page size (4K), not the CPU page size. This patch updates the alignment of ctl_stack_size and wg_data_size to prevent over-allocation on systems with larger CPU page sizes.

As far as I know the problem is that the debugger needs to consume that stuff on the CPU side as well.

Thank you for your help.

As mentioned earlier, we were observing some queue preemption and GPU hang issues. To address this, we introduced a patch, and after applying the 7/8 and 8/8 patches, those issues have not been seen anymore

Quoted 7 lines

While debugging the GPU hang issue, I made some additional observations.

On my system, I booted a kernel with a 4 KB system page size and modified both the ROCR runtime and the GPU driver to set the control stack size to 64 KB. Even on a 4 KB page-size system, using a 64 KB control stack size reliably reproduces the queue preemption failure when running RCCL unit tests on 8 GPUs. This suggests that the issue is not related to the system page size, but rather to the control stack size being exactly 64 KB.

When the control stack size is set to 64 KB ± 4 KB, the tests pass on both 4 KB and 64 KB system page-size configurations.

For gfxv9, is there any documented hardware limitation on the control stack size? Specifically, is it valid to use a control stack size of exactly 64 KB?

I have one more question based on my understanding of the code. The control stack size depends on the number of CUs and waves. For GFXv9, what is the maximum possible control stack size? Can it reach 64K?

For GFX10, I’ve seen that the control stack size must be less than or equal to 0x7000. Is there a similar limitation for GFXv9?

I’m asking because, with both 4K and 64K page sizes, I’m seeing queue preemption failures on GFXv9 when the control stack size is set to 64K.

Quoted 62 lines
Quoted 4 lines

I need to double check that, but I think the alignment is correct as it is.

The control stack is part of the context save-restore buffer, and we configure it on the GPU as shown below:

m->cp_hqd_ctx_save_base_addr_lo = lower_32_bits(q->ctx_save_restore_area_address); m->cp_hqd_ctx_save_base_addr_hi = upper_32_bits(q->ctx_save_restore_area_address); m->cp_hqd_ctx_save_size = q->ctx_save_restore_area_size; m->cp_hqd_cntl_stack_size = q->ctl_stack_size; m->cp_hqd_cntl_stack_offset = q->ctl_stack_size; m->cp_hqd_wg_state_offset = q->ctl_stack_size;

The control stack occupies the region from cp_hqd_cntl_stack_offset down to 0 within the ctx save restore area, and the remaining space is used for WG state. This buffer is fully managed by the GPU during preemption and restore operations. The control stack size is calculated based on hardware configuration (CU count and wave count). For example, on gfxv9, the size is typically around 32 KB. If we align this size to the system page size (e.g., 64 KB), two issues arise:

1. Unnecessary memory overhead.
2. Potential queue preemption issues.

On the CPU side, we copy the control stack contents to other buffers for processing. Since the control stack size is derived from hardware configuration, aligning it to the GPU page size seems more appropriate. Aligning to the system page size would waste memory without adding value. Using GPU page size alignment ensures consistency with hardware and avoids unnecessary overhead.

Would you agree that aligning the control stack size to the GPU page size is the right approach? Or do you see any concerns with this method?

Quoted 36 lines

Regards,
Christian.

Quoted 32 lines

---
  drivers/gpu/drm/amd/amdkfd/kfd_queue.c | 7 ++++---

  1 file changed, 4 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_queue.c +3 -1
index dc857450fa16..00ab941c3e86 100644
@@ -445,10 +445,11 @@ void kfd_queue_ctx_save_restore_size(struct 
 kfd_topology_device *dev)
               min(cu_num * 40, props->array_count / props->simd_arrays_per_engine * 512)
               : cu_num * 32;
   -    wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, props), PAGE_SIZE);
+    wg_data_size = ALIGN(cu_num * WG_CONTEXT_DATA_SIZE_PER_CU(gfxv, 
 props),
+                AMDGPU_GPU_PAGE_SIZE);
       ctl_stack_size = wave_num * CNTL_STACK_BYTES_PER_WAVE(gfxv) + 8;
       ctl_stack_size = ALIGN(SIZEOF_HSA_USER_CONTEXT_SAVE_AREA_HEADER + ctl_stack_size,
-                   PAGE_SIZE);
+                   AMDGPU_GPU_PAGE_SIZE);
         if ((gfxv / 10000 * 10000) == 100000) {

          /* HW design limits control stack size to 0x7000.

drivers/gpu/drm/amd/amdkfd/kfd_queue.c +1 -1
@@ -460,7 +461,7 @@ void kfd_queue_ctx_save_restore_size(struct 
 kfd_topology_device *dev)
         props->ctl_stack_size = ctl_stack_size;
       props->debug_memory_size = ALIGN(wave_num * DEBUGGER_BYTES_PER_WAVE, DEBUGGER_BYTES_ALIGN);
-    props->cwsr_size = ctl_stack_size + wg_data_size;
+    props->cwsr_size = ALIGN(ctl_stack_size + wg_data_size, 
 PAGE_SIZE);

        if (gfxv == 80002)    /* GFX_VERSION_TONGA */ props->eop_buffer_size = 0x8000;

maintainer
#13 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Donet Tom
Quoted 39 lines · Donet Tom
Quoted 7 lines · Christian König
Quoted 2 lines · Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 7 lines
Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Hi Christian, Felix, Alex and philip

Instead of hardcoding the AMDGPU_GTT_MAX_TRANSFER_SIZE value to 512, what do you think about doing something like the change below? This should work across all architectures and page sizes, so AMDGPU_GTT_MAX_TRANSFER_SIZE will always correspond to the PMD size on all architectures and with all page sizes.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +0 -0
index 0be2728aa872..c594ed7dff18 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP   (TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM        (TTM_PL_PRIV + 6)

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE   512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE   1 << (PMD_SHIFT - PAGE_SHIFT)

 #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS

Quoted 1 line

Could you please provide your thoughts on above? Is it looking ok to you?

It's at least reasonable. My only concern is that we have patches in the pipeline to remove that define and make it independent of the PMD size.

@Pierre-Eric how far along are we with that?

Quoted 11 lines

If this looks good - here is what we were thinking:

Patches 1-4 are required to fix initial non-4k pagesize support to AMD GPU. And since these patches are looking in good shape (since Philip has already reviewed [1-3])- We thought it will be good to split the patch series into two. I will send a v2 of Part-1 with patches [1-4] (will also address the review comments in v2 for Patch-1 & 2 from Philip) and for the rest of the patches [5-8] Part-2, we can continue the discussion till other things are sorted. That will also allow us to get these initial fixes in Part-1 ready before the 6.20 merge window.

Quoted 1 line

Thoughts?

Sounds reasonable to me.

Regards,
Christian.

Quoted 24 lines
Quoted 24 lines

Regards,
Christian.

Quoted 19 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
 #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
 #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
 #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
 extern const struct attribute_group amdgpu_vram_mgr_attr_group;
#14 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

Hi,

Le 08/01/2026 à 13:31, Christian König a écrit :

Quoted 40 lines · Donet Tom
Quoted 39 lines · Christian König
Quoted 7 lines · Donet Tom
Quoted 2 lines

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 7 lines
Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Hi Christian, Felix, Alex and philip

Instead of hardcoding the AMDGPU_GTT_MAX_TRANSFER_SIZE value to 512, what do you think about doing something like the change below? This should work across all architectures and page sizes, so AMDGPU_GTT_MAX_TRANSFER_SIZE will always correspond to the PMD size on all architectures and with all page sizes.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..c594ed7dff18 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP   (TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM        (TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE   512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE   1 << (PMD_SHIFT - PAGE_SHIFT)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS
 
Quoted 5 lines
Quoted 1 line

Could you please provide your thoughts on above? Is it looking ok to you?

It's at least reasonable. My only concern is that we have patches in the pipeline to remove that define and make it independent of the PMD size.

@Pierre-Eric how far along are we with that?

My patchset is dropping AMDGPU_GTT_NUM_TRANSFER_WINDOWS and doubling AMDGPU_GTT_MAX_TRANSFER_SIZE so it's not negatively affected by Tom's patches.

Regards,
Pierre-Eric

Quoted 43 lines
Quoted 12 lines

If this looks good - here is what we were thinking:

Patches 1-4 are required to fix initial non-4k pagesize support to AMD GPU. And since these patches are looking in good shape (since Philip has already reviewed [1-3])- We thought it will be good to split the patch series into two. I will send a v2 of Part-1 with patches [1-4] (will also address the review comments in v2 for Patch-1 & 2 from Philip) and for the rest of the patches [5-8] Part-2, we can continue the discussion till other things are sorted. That will also allow us to get these initial fixes in Part-1 ready before the 6.20 merge window.

Quoted 1 line

Thoughts?

Sounds reasonable to me.

Regards,
Christian.

Quoted 24 lines
Quoted 24 lines

Regards,
Christian.

Quoted 19 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
  extern const struct attribute_group amdgpu_vram_mgr_attr_group;
author
#15 lore raw

Re: [RFC PATCH v1 4/8] amdgpu/amdgpu_ttm: Fix AMDGPU_GTT_MAX_TRANSFER_SIZE for non-4K page size

In reply to Christian König
Quoted 53 lines · Christian König
Quoted 37 lines · Donet Tom
Quoted 7 lines · Christian König
Quoted 2 lines · Donet Tom

The SDMA engine has a hardware limitation of 4 MB maximum transfer size per operation.

That is not correct. This is only true on ancient HW.

What problems are you seeing here?

Quoted 7 lines
Quoted 6 lines

AMDGPU_GTT_MAX_TRANSFER_SIZE was hardcoded to
512 pages, which worked correctly on systems with 4K pages but fails on systems with larger page sizes.

This patch divides the max transfer size / AMDGPU_GPU_PAGES_IN_CPU_PAGE to match with non-4K page size systems.

That is actually a bad idea. The value was meant to match the PMD size.

Hi Christian, Felix, Alex and philip

Instead of hardcoding the AMDGPU_GTT_MAX_TRANSFER_SIZE value to 512, what do you think about doing something like the change below? This should work across all architectures and page sizes, so AMDGPU_GTT_MAX_TRANSFER_SIZE will always correspond to the PMD size on all architectures and with all page sizes.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..c594ed7dff18 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP   (TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM        (TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE   512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE   1 << (PMD_SHIFT - PAGE_SHIFT)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS
 
Quoted 1 line

Could you please provide your thoughts on above? Is it looking ok to you?

It's at least reasonable. My only concern is that we have patches in the pipeline to remove that define and make it independent of the PMD size.

@Pierre-Eric how far along are we with that?

Quoted 10 lines

If this looks good - here is what we were thinking:

Patches 1-4 are required to fix initial non-4k pagesize support to AMD GPU. And since these patches are looking in good shape (since Philip has already reviewed [1-3])- We thought it will be good to split the patch series into two. I will send a v2 of Part-1 with patches [1-4] (will also address the review comments in v2 for Patch-1 & 2 from Philip) and for the rest of the patches [5-8] Part-2, we can continue the discussion till other things are sorted. That will also allow us to get these initial fixes in Part-1 ready before the 6.20 merge window.

Quoted 2 lines
Quoted 1 line

Thoughts?

Sounds reasonable to me.

Thanks Christian.

Quoted 27 lines

Regards,
Christian.

Quoted 23 lines
Quoted 23 lines

Regards,
Christian.

Quoted 19 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 -1
index 0be2728aa872..9d038feb25b0 100644
@@ -37,7 +37,7 @@
  #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
  #define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
 
-#define AMDGPU_GTT_MAX_TRANSFER_SIZE	512
+#define AMDGPU_GTT_MAX_TRANSFER_SIZE	(512 / AMDGPU_GPU_PAGES_IN_CPU_PAGE)
  #define AMDGPU_GTT_NUM_TRANSFER_WINDOWS	2
 
  extern const struct attribute_group amdgpu_vram_mgr_attr_group;