Skip to contentSkip to conversation

MES support

PATCH New Alex Deucher 1 participant 74 messages Apr 29, 2022

Subsystems (2)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

cover MES support

This patch set enables support for MES (Micro Engine Scheduler). This is similar the the HWS (HardWare Scheduler) on the MEC currently used for KFD. It's a scheduling microcontroller used for scheduling engine queues to hardware slots. This adds the basic infrastructure to enable MES in amdgpu.

Jack Xiao (70):
drm/amdgpu: define MQD abstract layer for hw ip drm/amdgpu: add helper function to initialize mqd from ring v4 drm/amdgpu: add the per-context meta data v3
drm/amdgpu: add mes ctx data in amdgpu_ring
drm/amdgpu: define ring structure to access rptr/wptr/fence drm/amdgpu: use ring structure to access rptr/wptr v2 drm/amdgpu: initialize/finalize the ring for mes queue drm/amdgpu: assign the cpu/gpu address of fence from ring drm/amdgpu/gfx10: implement mqd functions of gfx/compute eng v2 drm/amdgpu/gfx10: use per ctx CSA for ce metadata drm/amdgpu/gfx10: use per ctx CSA for de metadata drm/amdgpu/gfx10: associate mes queue id with fence v2 drm/amdgpu/gfx10: inherit vmid from mqd
drm/amdgpu/gfx10: use INVALIDATE_TLBS to invalidate TLBs v2 drm/amdgpu/gmc10: skip emitting pasid mapping packet drm/amdgpu: use the whole doorbell space for mes drm/amdgpu: update mes process/gang/queue definitions drm/amdgpu: add mes_kiq module parameter v2
drm/amdgpu: allocate doorbell index for mes kiq drm/amdgpu/mes: extend mes framework to support multiple mes pipes drm/amdgpu/gfx10: add mes queue fence handling drm/amdgpu/gfx10: add mes support for gfx ib test drm/amdgpu: don't use kiq to flush gpu tlb if mes enabled drm/amdgpu/sdma: use per-ctx sdma csa address for mes sdma queue drm/amdgpu/sdma5.2: initialize sdma mqd drm/amdgpu/sdma5.2: associate mes queue id with fence drm/amdgpu/sdma5.2: add mes queue fence handling drm/amdgpu/sdma5.2: add mes support for sdma ring test drm/amdgpu/sdma5.2: add mes support for sdma ib test drm/amdgpu/sdma5: initialize sdma mqd
drm/amdgpu/sdma5: associate mes queue id with fence drm/amdgpu/sdma5: add mes queue fence handling drm/amdgpu/sdma5: add mes support for sdma ring test drm/amdgpu/sdma5: add mes support for sdma ib test drm/amdgpu/mes: add mes kiq callback
drm/amdgpu: add mes kiq frontdoor loading support drm/amdgpu: enable mes kiq N-1 test on sienna cichlid drm/amdgpu/mes: manage mes doorbell allocation
drm/amdgpu: add mes queue id mask v2
drm/amdgpu/mes: initialize/finalize common mes structure v2 drm/amdgpu/mes: relocate status_fence slot allocation drm/amdgpu/mes10.1: call general mes initialization drm/amdgpu/mes10.1: add delay after mes engine enable drm/amdgpu/mes10.1: implement the suspend/resume routine drm/amdgpu/mes: implement creating mes process v2 drm/amdgpu/mes: implement destroying mes process drm/amdgpu/mes: implement adding mes gang
drm/amdgpu/mes: implement removing mes gang drm/amdgpu/mes: implement suspending all gangs drm/amdgpu/mes: implement resuming all gangs drm/amdgpu/mes: initialize mqd from queue properties drm/amdgpu/mes: implement adding mes queue
drm/amdgpu/mes: implement removing mes queue drm/amdgpu/mes: add helper function to convert ring to queue property drm/amdgpu/mes: add helper function to get the ctx meta data offset drm/amdgpu/mes: use ring for kernel queue submission drm/amdgpu/mes: implement removing mes ring drm/amdgpu/mes: add helper functions to alloc/free ctx metadata drm/amdgpu: skip kfd routines when mes enabled
drm/amdgpu: skip some checking for mes queue ib submission drm/amdgpu: skip kiq ib tests if mes enabled
drm/amdgpu: skip gds switch for mes queue
drm/amdgpu: kiq takes charge of all queues
drm/amdgpu/mes: map ctx metadata for mes self test drm/amdgpu/mes: create gang and queues for mes self test drm/amdgpu/mes: add ring/ib test for mes self test drm/amdgpu/mes: implement mes self test drm/amdgpu/mes10.1: add mes self test in late init drm/amdgpu/mes: fix vm csa update issue
drm/amdgpu/mes: disable mes sdma queue test

Likun Gao (1):
drm/amdgpu: add mes kiq PSP GFX FW type

Mukul Joshi (2):
drm/amdgpu: Enable KFD with MES enabled drm/amdgpu/mes: Update the doorbell function signatures

 drivers/gpu/drm/amd/amdgpu/Makefile          |    1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h          |   24 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c   |   42 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h |    6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c      |   10 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c    |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c      |    3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c       |    8 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c      | 1138 ++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h      |  168 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h  |  121 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c      |    6 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c     |  193 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h     |   22 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c     |   24 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c    |   10 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c       |    3 +-
 drivers/gpu/drm/amd/amdgpu/cik_sdma.c        |    8 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c       |  383 +++---
 drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c        |    8 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c        |   16 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c        |   20 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c        |   25 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c       |    6 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c       |    4 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c       |    4 +-
 drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c       |    4 +-
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c       |  461 ++++---
 drivers/gpu/drm/amd/amdgpu/nv.c              |    3 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c       |    8 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c       |   16 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c       |   28 +-
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c       |  169 ++-
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c       |  171 ++-
 drivers/gpu/drm/amd/amdgpu/si_dma.c          |    4 +-
 drivers/gpu/drm/amd/amdgpu/soc21.c           |    3 +-
 drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c        |    6 +-
 drivers/gpu/drm/amd/amdgpu/vce_v4_0.c        |    6 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c        |   12 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c        |   12 +-
 drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c        |   12 +-
 41 files changed, 2619 insertions(+), 553 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h
Signature · 1 line
2.35.1
author maintainer
#1 lore raw

PATCH 1/73 drm/amdgpu: define MQD abstract layer for hw ip

From: Jack Xiao <[email protected]>

Define MQD abstract layer for hw ip, for the passing mqd configuration not only from ring but more sources, like user queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +21 -0
index 1a598e3247ca..2eed9479e854 100644
@@ -720,6 +720,26 @@ struct ip_discovery_top;
 					  (rid == 0x01) || \
 					  (rid == 0x10))))
 
+struct amdgpu_mqd_prop {
+	uint64_t mqd_gpu_addr;
+	uint64_t hqd_base_gpu_addr;
+	uint64_t rptr_gpu_addr;
+	uint64_t wptr_gpu_addr;
+	uint32_t queue_size;
+	bool use_doorbell;
+	uint32_t doorbell_index;
+	uint64_t eop_gpu_addr;
+	uint32_t hqd_pipe_priority;
+	uint32_t hqd_queue_priority;
+	bool hqd_active;
+};
+
+struct amdgpu_mqd {
+	unsigned mqd_size;
+	int (*init_mqd)(struct amdgpu_device *adev, void *mqd,
+			struct amdgpu_mqd_prop *p);
+};
+
 #define AMDGPU_RESET_MAGIC_NUM 64
 #define AMDGPU_MAX_DF_PERFMONS 4
 #define AMDGPU_PRODUCT_NAME_LEN 64
@@ -920,6 +940,7 @@ struct amdgpu_device {
 	/* mes */
 	bool                            enable_mes;
 	struct amdgpu_mes               mes;
+	struct amdgpu_mqd               mqds[AMDGPU_HW_IP_NUM];
 
 	/* df */
 	struct amdgpu_df                df;
Signature · 1 line
2.35.1
author maintainer
#2 lore raw

PATCH 2/73 drm/amdgpu: add helper function to initialize mqd from ring v4

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add the helper function to initialize mqd from ring configuration.

v2: use if/else pair instead of ?/: pair
v3: use simpler way to judge hqd_active
v4: fix parameters to amdgpu_gfx_is_high_priority_compute_queue

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 48 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h |  2 +
 2 files changed, 50 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +48 -0
index 7f33ae87cb41..773954318216 100644
@@ -458,3 +458,51 @@ int amdgpu_ring_test_helper(struct amdgpu_ring *ring)
 	ring->sched.ready = !r;
 	return r;
 }
+
+static void amdgpu_ring_to_mqd_prop(struct amdgpu_ring *ring,
+				    struct amdgpu_mqd_prop *prop)
+{
+	struct amdgpu_device *adev = ring->adev;
+
+	memset(prop, 0, sizeof(*prop));
+
+	prop->mqd_gpu_addr = ring->mqd_gpu_addr;
+	prop->hqd_base_gpu_addr = ring->gpu_addr;
+	prop->rptr_gpu_addr = ring->rptr_gpu_addr;
+	prop->wptr_gpu_addr = ring->wptr_gpu_addr;
+	prop->queue_size = ring->ring_size;
+	prop->eop_gpu_addr = ring->eop_gpu_addr;
+	prop->use_doorbell = ring->use_doorbell;
+	prop->doorbell_index = ring->doorbell_index;
+
+	/* map_queues packet doesn't need activate the queue,
+	 * so only kiq need set this field.
+	 */
+	prop->hqd_active = ring->funcs->type == AMDGPU_RING_TYPE_KIQ;
+
+	if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
+		if (amdgpu_gfx_is_high_priority_compute_queue(adev, ring)) {
+			prop->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_HIGH;
+			prop->hqd_queue_priority =
+				AMDGPU_GFX_QUEUE_PRIORITY_MAXIMUM;
+		}
+	}
+}
+
+int amdgpu_ring_init_mqd(struct amdgpu_ring *ring)
+{
+	struct amdgpu_device *adev = ring->adev;
+	struct amdgpu_mqd *mqd_mgr;
+	struct amdgpu_mqd_prop prop;
+
+	amdgpu_ring_to_mqd_prop(ring, &prop);
+
+	ring->wptr = 0;
+
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		mqd_mgr = &adev->mqds[AMDGPU_HW_IP_COMPUTE];
+	else
+		mqd_mgr = &adev->mqds[ring->funcs->type];
+
+	return mqd_mgr->init_mqd(adev, ring->mqd_ptr, &prop);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +2 -0
index 317d80209e95..20dfe5a19a81 100644
@@ -369,6 +369,8 @@ int amdgpu_ring_test_helper(struct amdgpu_ring *ring);
 void amdgpu_debugfs_ring_init(struct amdgpu_device *adev,
 			      struct amdgpu_ring *ring);
 
+int amdgpu_ring_init_mqd(struct amdgpu_ring *ring);
+
 static inline u32 amdgpu_ib_get_value(struct amdgpu_ib *ib, int idx)
 {
 	return ib->ptr[idx];
Signature · 1 line
2.35.1
author maintainer
#3 lore raw

PATCH 3/73 drm/amdgpu: add the per-context meta data v3

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

The per-context meta data is a per-context data structure associated with a mes-managed hardware ring, which includes MCBP CSA, ring buffer and etc.

v2: fix typo
v3: a. use structure instead of typedef
b. move amdgpu_mes_ctx_get_offs_* to amdgpu_ring.h c. use __aligned to make alignement

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h         |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h | 118 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h    |   9 ++
 3 files changed, 128 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 2eed9479e854..24bce7e691a8 100644
@@ -91,6 +91,7 @@
 #include "amdgpu_dm.h"
 #include "amdgpu_virt.h"
 #include "amdgpu_csa.h"
+#include "amdgpu_mes_ctx.h"
 #include "amdgpu_gart.h"
 #include "amdgpu_debugfs.h"
 #include "amdgpu_job.h"
drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h new file +118 -0

118 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +9 -0
index 20dfe5a19a81..112c2b0ef0b1 100644
@@ -364,6 +364,15 @@ static inline void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
 	ring->count_dw -= count_dw;
 }
 
+#define amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset)			\
+	(ring->is_mes_queue && ring->mes_ctx ?				\
+	 (ring->mes_ctx->meta_data_gpu_addr + offset) : 0)
+
+#define amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset)			\
+	(ring->is_mes_queue && ring->mes_ctx ?				\
+	 (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
+	 NULL)
+
 int amdgpu_ring_test_helper(struct amdgpu_ring *ring);
 
 void amdgpu_debugfs_ring_init(struct amdgpu_device *adev,
Signature · 1 line
2.35.1
author maintainer
#4 lore raw

PATCH 4/73 drm/amdgpu: add mes ctx data in amdgpu_ring

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add mes context data structure in amdgpu_ring.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +5 -0
index 112c2b0ef0b1..317a66bcd258 100644
@@ -267,6 +267,11 @@ struct amdgpu_ring {
 	int			hw_prio;
 	unsigned 		num_hw_submission;
 	atomic_t		*sched_score;
+
+	/* used for mes */
+	bool			is_mes_queue;
+	uint32_t		hw_queue_id;
+	struct amdgpu_mes_ctx_data *mes_ctx;
 };
 
 #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
Signature · 1 line
2.35.1
author maintainer
#5 lore raw

PATCH 5/73 drm/amdgpu: define ring structure to access rptr/wptr/fence

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Define ring structure to access the cpu/gpu address of rptr/wptr/fence instead of dynamic calculation.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +6 -0
index 317a66bcd258..7d89a52091c0 100644
@@ -230,6 +230,8 @@ struct amdgpu_ring {
 	struct amdgpu_bo	*ring_obj;
 	volatile uint32_t	*ring;
 	unsigned		rptr_offs;
+	u64			rptr_gpu_addr;
+	volatile u32		*rptr_cpu_addr;
 	u64			wptr;
 	u64			wptr_old;
 	unsigned		ring_size;
@@ -250,7 +252,11 @@ struct amdgpu_ring {
 	bool			use_doorbell;
 	bool			use_pollmem;
 	unsigned		wptr_offs;
+	u64			wptr_gpu_addr;
+	volatile u32		*wptr_cpu_addr;
 	unsigned		fence_offs;
+	u64			fence_gpu_addr;
+	volatile u32		*fence_cpu_addr;
 	uint64_t		current_ctx;
 	char			name[16];
 	u32                     trail_seq;
Signature · 1 line
2.35.1
author maintainer
#6 lore raw

PATCH 6/73 drm/amdgpu: use ring structure to access rptr/wptr v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Use ring structure to access the cpu/gpu address of rptr/wptr.

v2: merge gfx10/sdma5/sdma5.2 patches

---
 drivers/gpu/drm/amd/amdgpu/cik_sdma.c  |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 37 +++++++++++++-------------
 drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c  |  8 +++---
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c  | 16 +++++------
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c  | 20 +++++++-------
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c  | 25 +++++++++--------
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c |  4 +--
 drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c |  4 +--
 drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c |  4 +--
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 11 ++++----
 drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c |  8 +++---
 drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c | 16 +++++------
 drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c | 28 ++++++++-----------
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 16 +++++------
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 16 +++++------
 drivers/gpu/drm/amd/amdgpu/si_dma.c    |  4 +--
 drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c  |  6 ++---
 drivers/gpu/drm/amd/amdgpu/vce_v4_0.c  |  6 ++---
 drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c  | 12 ++++-----
 drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c  | 12 ++++-----
 drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c  | 12 ++++-----
 21 files changed, 128 insertions(+), 145 deletions(-)
drivers/gpu/drm/amd/amdgpu/cik_sdma.c +3 -5
index 6c01199e9112..5647f13b98d4 100644
@@ -164,7 +164,7 @@ static uint64_t cik_sdma_ring_get_rptr(struct amdgpu_ring *ring)
 {
 	u32 rptr;
 
-	rptr = ring->adev->wb.wb[ring->rptr_offs];
+	rptr = *ring->rptr_cpu_addr;
 
 	return (rptr & 0x3fffc) >> 2;
 }
@@ -436,12 +436,10 @@ static int cik_sdma_gfx_resume(struct amdgpu_device *adev)
 	struct amdgpu_ring *ring;
 	u32 rb_cntl, ib_cntl;
 	u32 rb_bufsz;
-	u32 wb_offset;
 	int i, j, r;
 
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
-		wb_offset = (ring->rptr_offs * 4);
 
 		mutex_lock(&adev->srbm_mutex);
 		for (j = 0; j < 16; j++) {
@@ -477,9 +475,9 @@ static int cik_sdma_gfx_resume(struct amdgpu_device *adev)
 
 		/* set the wb address whether it's enabled or not */
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i],
-		       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+		       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i],
-		       ((adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC));
+		       ((ring->rptr_gpu_addr) & 0xFFFFFFFC));
 
 		rb_cntl |= SDMA0_GFX_RB_CNTL__RPTR_WRITEBACK_ENABLE_MASK;
 
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +19 -18
index 407074f958f4..f2dd53f2af61 100644
@@ -3519,9 +3519,8 @@ static void gfx10_kiq_set_resources(struct amdgpu_ring *kiq_ring, uint64_t queue
 static void gfx10_kiq_map_queues(struct amdgpu_ring *kiq_ring,
 				 struct amdgpu_ring *ring)
 {
-	struct amdgpu_device *adev = kiq_ring->adev;
 	uint64_t mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
-	uint64_t wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	uint64_t wptr_addr = ring->wptr_gpu_addr;
 	uint32_t eng_sel = ring->funcs->type == AMDGPU_RING_TYPE_GFX ? 4 : 0;
 
 	amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
@@ -6344,12 +6343,12 @@ static int gfx_v10_0_cp_gfx_resume(struct amdgpu_device *adev)
 	WREG32_SOC15(GC, 0, mmCP_RB0_WPTR_HI, upper_32_bits(ring->wptr));
 
 	/* set the wb address wether it's enabled or not */
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32_SOC15(GC, 0, mmCP_RB0_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32_SOC15(GC, 0, mmCP_RB0_RPTR_ADDR_HI, upper_32_bits(rptr_addr) &
 		     CP_RB_RPTR_ADDR_HI__RB_RPTR_ADDR_HI_MASK);
 
-	wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wptr_gpu_addr = ring->wptr_gpu_addr;
 	WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO,
 		     lower_32_bits(wptr_gpu_addr));
 	WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI,
@@ -6382,11 +6381,11 @@ static int gfx_v10_0_cp_gfx_resume(struct amdgpu_device *adev)
 		WREG32_SOC15(GC, 0, mmCP_RB1_WPTR, lower_32_bits(ring->wptr));
 		WREG32_SOC15(GC, 0, mmCP_RB1_WPTR_HI, upper_32_bits(ring->wptr));
 		/* Set the wb address wether it's enabled or not */
-		rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+		rptr_addr = ring->rptr_gpu_addr;
 		WREG32_SOC15(GC, 0, mmCP_RB1_RPTR_ADDR, lower_32_bits(rptr_addr));
 		WREG32_SOC15(GC, 0, mmCP_RB1_RPTR_ADDR_HI, upper_32_bits(rptr_addr) &
 			     CP_RB1_RPTR_ADDR_HI__RB_RPTR_ADDR_HI_MASK);
-		wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+		wptr_gpu_addr = ring->wptr_gpu_addr;
 		WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO,
 			     lower_32_bits(wptr_gpu_addr));
 		WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI,
@@ -6610,13 +6609,13 @@ static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_gfx_hqd_base_hi = upper_32_bits(hqd_gpu_addr);
 
 	/* set up hqd_rptr_addr/_hi, similar as CP_RB_RPTR */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_gfx_hqd_rptr_addr = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_gfx_hqd_rptr_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* set up rb_wptr_poll addr */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_rb_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_rb_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
@@ -6730,7 +6729,7 @@ static int gfx_v10_0_gfx_init_queue(struct amdgpu_ring *ring)
 			memcpy(mqd, adev->gfx.me.mqd_backup[mqd_idx], sizeof(*mqd));
 		/* reset the ring */
 		ring->wptr = 0;
-		adev->wb.wb[ring->wptr_offs] = 0;
+		*ring->wptr_cpu_addr = 0;
 		amdgpu_ring_clear_ring(ring);
 #ifdef BRING_UP_DEBUG
 		mutex_lock(&adev->srbm_mutex);
@@ -6904,13 +6903,13 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_control = tmp;
 
 	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
@@ -7130,7 +7129,7 @@ static int gfx_v10_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		/* reset ring buffer */
 		ring->wptr = 0;
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], 0);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, 0);
 		amdgpu_ring_clear_ring(ring);
 	} else {
 		amdgpu_ring_clear_ring(ring);
@@ -8496,7 +8495,8 @@ static void gfx_v10_0_get_clockgating_state(void *handle, u64 *flags)
 
 static u64 gfx_v10_0_ring_get_rptr_gfx(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs]; /* gfx10 is 32bit rptr*/
+	/* gfx10 is 32bit rptr*/
+	return *(uint32_t *)ring->rptr_cpu_addr;
 }
 
 static u64 gfx_v10_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
@@ -8506,7 +8506,7 @@ static u64 gfx_v10_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell) {
-		wptr = atomic64_read((atomic64_t *)&adev->wb.wb[ring->wptr_offs]);
+		wptr = atomic64_read((atomic64_t *)ring->wptr_cpu_addr);
 	} else {
 		wptr = RREG32_SOC15(GC, 0, mmCP_RB0_WPTR);
 		wptr += (u64)RREG32_SOC15(GC, 0, mmCP_RB0_WPTR_HI) << 32;
@@ -8521,7 +8521,7 @@ static void gfx_v10_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], ring->wptr);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, ring->wptr);
 		WDOORBELL64(ring->doorbell_index, ring->wptr);
 	} else {
 		WREG32_SOC15(GC, 0, mmCP_RB0_WPTR, lower_32_bits(ring->wptr));
@@ -8531,7 +8531,8 @@ static void gfx_v10_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
 
 static u64 gfx_v10_0_ring_get_rptr_compute(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs]; /* gfx10 hardware is 32bit rptr */
+	/* gfx10 hardware is 32bit rptr */
+	return *(uint32_t *)ring->rptr_cpu_addr;
 }
 
 static u64 gfx_v10_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
@@ -8540,7 +8541,7 @@ static u64 gfx_v10_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell)
-		wptr = atomic64_read((atomic64_t *)&ring->adev->wb.wb[ring->wptr_offs]);
+		wptr = atomic64_read((atomic64_t *)ring->wptr_cpu_addr);
 	else
 		BUG();
 	return wptr;
@@ -8552,7 +8553,7 @@ static void gfx_v10_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell) {
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], ring->wptr);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, ring->wptr);
 		WDOORBELL64(ring->doorbell_index, ring->wptr);
 	} else {
 		BUG(); /* only DOORBELL method supported on gfx10 now */
drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c +4 -4
index 6a8dadea40f9..29a91b320d4f 100644
@@ -2117,7 +2117,7 @@ static int gfx_v6_0_cp_gfx_resume(struct amdgpu_device *adev)
 	WREG32(mmCP_RB0_WPTR, ring->wptr);
 
 	/* set the wb address whether it's enabled or not */
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32(mmCP_RB0_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32(mmCP_RB0_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & 0xFF);
 
@@ -2139,7 +2139,7 @@ static int gfx_v6_0_cp_gfx_resume(struct amdgpu_device *adev)
 
 static u64 gfx_v6_0_ring_get_rptr(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs];
+	return *ring->rptr_cpu_addr;
 }
 
 static u64 gfx_v6_0_ring_get_wptr(struct amdgpu_ring *ring)
@@ -2203,7 +2203,7 @@ static int gfx_v6_0_cp_compute_resume(struct amdgpu_device *adev)
 	ring->wptr = 0;
 	WREG32(mmCP_RB1_WPTR, ring->wptr);
 
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32(mmCP_RB1_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32(mmCP_RB1_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & 0xFF);
 
@@ -2222,7 +2222,7 @@ static int gfx_v6_0_cp_compute_resume(struct amdgpu_device *adev)
 	WREG32(mmCP_RB2_CNTL, tmp | CP_RB2_CNTL__RB_RPTR_WR_ENA_MASK);
 	ring->wptr = 0;
 	WREG32(mmCP_RB2_WPTR, ring->wptr);
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32(mmCP_RB2_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32(mmCP_RB2_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & 0xFF);
 
drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +8 -8
index d17a6f399347..ac3f2dbba726 100644
@@ -2630,8 +2630,8 @@ static int gfx_v7_0_cp_gfx_resume(struct amdgpu_device *adev)
 	ring->wptr = 0;
 	WREG32(mmCP_RB0_WPTR, lower_32_bits(ring->wptr));
 
-	/* set the wb address whether it's enabled or not */
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	/* set the wb address wether it's enabled or not */
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32(mmCP_RB0_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32(mmCP_RB0_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & 0xFF);
 
@@ -2656,7 +2656,7 @@ static int gfx_v7_0_cp_gfx_resume(struct amdgpu_device *adev)
 
 static u64 gfx_v7_0_ring_get_rptr(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs];
+	return *ring->rptr_cpu_addr;
 }
 
 static u64 gfx_v7_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
@@ -2677,7 +2677,7 @@ static void gfx_v7_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
 static u64 gfx_v7_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
 {
 	/* XXX check if swapping is necessary on BE */
-	return ring->adev->wb.wb[ring->wptr_offs];
+	return *ring->wptr_cpu_addr;
 }
 
 static void gfx_v7_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
@@ -2685,7 +2685,7 @@ static void gfx_v7_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	/* XXX check if swapping is necessary on BE */
-	adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+	*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 	WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 }
 
@@ -2981,12 +2981,12 @@ static void gfx_v7_0_mqd_init(struct amdgpu_device *adev,
 		CP_HQD_PQ_CONTROL__KMD_QUEUE_MASK; /* assuming kernel queue control */
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
-	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	/* set the wb address wether it's enabled or not */
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +10 -10
index 25dc729d0ec2..e4e779a19c20 100644
@@ -4306,11 +4306,11 @@ static int gfx_v8_0_cp_gfx_resume(struct amdgpu_device *adev)
 	WREG32(mmCP_RB0_WPTR, lower_32_bits(ring->wptr));
 
 	/* set the wb address wether it's enabled or not */
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32(mmCP_RB0_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32(mmCP_RB0_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & 0xFF);
 
-	wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wptr_gpu_addr = ring->wptr_gpu_addr;
 	WREG32(mmCP_RB_WPTR_POLL_ADDR_LO, lower_32_bits(wptr_gpu_addr));
 	WREG32(mmCP_RB_WPTR_POLL_ADDR_HI, upper_32_bits(wptr_gpu_addr));
 	mdelay(1);
@@ -4393,7 +4393,7 @@ static int gfx_v8_0_kiq_kcq_enable(struct amdgpu_device *adev)
 	for (i = 0; i < adev->gfx.num_compute_rings; i++) {
 		struct amdgpu_ring *ring = &adev->gfx.compute_ring[i];
 		uint64_t mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
-		uint64_t wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+		uint64_t wptr_addr = ring->wptr_gpu_addr;
 
 		/* map queues */
 		amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
@@ -4517,13 +4517,13 @@ static int gfx_v8_0_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_control = tmp;
 
 	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
@@ -6051,7 +6051,7 @@ static int gfx_v8_0_set_clockgating_state(void *handle,
 
 static u64 gfx_v8_0_ring_get_rptr(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs];
+	return *ring->rptr_cpu_addr;
 }
 
 static u64 gfx_v8_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
@@ -6060,7 +6060,7 @@ static u64 gfx_v8_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell)
 		/* XXX check if swapping is necessary on BE */
-		return ring->adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32(mmCP_RB0_WPTR);
 }
@@ -6071,7 +6071,7 @@ static void gfx_v8_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32(mmCP_RB0_WPTR, lower_32_bits(ring->wptr));
@@ -6271,7 +6271,7 @@ static void gfx_v8_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
 
 static u64 gfx_v8_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->wptr_offs];
+	return *ring->wptr_cpu_addr;
 }
 
 static void gfx_v8_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
@@ -6279,7 +6279,7 @@ static void gfx_v8_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	/* XXX check if swapping is necessary on BE */
-	adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+	*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 	WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 }
 
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +12 -13
index d58fd83524ac..06182b7e4351 100644
@@ -780,9 +780,8 @@ static void gfx_v9_0_kiq_set_resources(struct amdgpu_ring *kiq_ring,
 static void gfx_v9_0_kiq_map_queues(struct amdgpu_ring *kiq_ring,
 				 struct amdgpu_ring *ring)
 {
-	struct amdgpu_device *adev = kiq_ring->adev;
 	uint64_t mqd_addr = amdgpu_bo_gpu_offset(ring->mqd_obj);
-	uint64_t wptr_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	uint64_t wptr_addr = ring->wptr_gpu_addr;
 	uint32_t eng_sel = ring->funcs->type == AMDGPU_RING_TYPE_GFX ? 4 : 0;
 
 	amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_MAP_QUEUES, 5));
@@ -3326,11 +3325,11 @@ static int gfx_v9_0_cp_gfx_resume(struct amdgpu_device *adev)
 	WREG32_SOC15(GC, 0, mmCP_RB0_WPTR_HI, upper_32_bits(ring->wptr));
 
 	/* set the wb address wether it's enabled or not */
-	rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	rptr_addr = ring->rptr_gpu_addr;
 	WREG32_SOC15(GC, 0, mmCP_RB0_RPTR_ADDR, lower_32_bits(rptr_addr));
 	WREG32_SOC15(GC, 0, mmCP_RB0_RPTR_ADDR_HI, upper_32_bits(rptr_addr) & CP_RB_RPTR_ADDR_HI__RB_RPTR_ADDR_HI_MASK);
 
-	wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wptr_gpu_addr = ring->wptr_gpu_addr;
 	WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO, lower_32_bits(wptr_gpu_addr));
 	WREG32_SOC15(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI, upper_32_bits(wptr_gpu_addr));
 
@@ -3542,13 +3541,13 @@ static int gfx_v9_0_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_control = tmp;
 
 	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
@@ -3830,7 +3829,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		/* reset ring buffer */
 		ring->wptr = 0;
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], 0);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, 0);
 		amdgpu_ring_clear_ring(ring);
 	} else {
 		amdgpu_ring_clear_ring(ring);
@@ -5279,7 +5278,7 @@ static void gfx_v9_0_get_clockgating_state(void *handle, u64 *flags)
 
 static u64 gfx_v9_0_ring_get_rptr_gfx(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs]; /* gfx9 is 32bit rptr*/
+	return *ring->rptr_cpu_addr; /* gfx9 is 32bit rptr*/
 }
 
 static u64 gfx_v9_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
@@ -5289,7 +5288,7 @@ static u64 gfx_v9_0_ring_get_wptr_gfx(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell) {
-		wptr = atomic64_read((atomic64_t *)&adev->wb.wb[ring->wptr_offs]);
+		wptr = atomic64_read((atomic64_t *)ring->wptr_cpu_addr);
 	} else {
 		wptr = RREG32_SOC15(GC, 0, mmCP_RB0_WPTR);
 		wptr += (u64)RREG32_SOC15(GC, 0, mmCP_RB0_WPTR_HI) << 32;
@@ -5304,7 +5303,7 @@ static void gfx_v9_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], ring->wptr);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, ring->wptr);
 		WDOORBELL64(ring->doorbell_index, ring->wptr);
 	} else {
 		WREG32_SOC15(GC, 0, mmCP_RB0_WPTR, lower_32_bits(ring->wptr));
@@ -5469,7 +5468,7 @@ static void gfx_v9_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
 
 static u64 gfx_v9_0_ring_get_rptr_compute(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs]; /* gfx9 hardware is 32bit rptr */
+	return *ring->rptr_cpu_addr; /* gfx9 hardware is 32bit rptr */
 }
 
 static u64 gfx_v9_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
@@ -5478,7 +5477,7 @@ static u64 gfx_v9_0_ring_get_wptr_compute(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell)
-		wptr = atomic64_read((atomic64_t *)&ring->adev->wb.wb[ring->wptr_offs]);
+		wptr = atomic64_read((atomic64_t *)ring->wptr_cpu_addr);
 	else
 		BUG();
 	return wptr;
@@ -5490,7 +5489,7 @@ static void gfx_v9_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
 
 	/* XXX check if swapping is necessary on BE */
 	if (ring->use_doorbell) {
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs], ring->wptr);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr, ring->wptr);
 		WDOORBELL64(ring->doorbell_index, ring->wptr);
 	} else{
 		BUG(); /* only DOORBELL method supported on gfx9 now */
drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c +2 -2
index 299de1d131d8..d2722adabd1b 100644
@@ -407,7 +407,7 @@ static uint64_t jpeg_v2_0_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(JPEG, 0, mmUVD_JRBC_RB_WPTR);
 }
@@ -424,7 +424,7 @@ static void jpeg_v2_0_dec_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(JPEG, 0, mmUVD_JRBC_RB_WPTR, lower_32_bits(ring->wptr));
drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c +2 -2
index 8c3227d0b8b4..c2bf036a7330 100644
@@ -402,7 +402,7 @@ static uint64_t jpeg_v2_5_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(JPEG, ring->me, mmUVD_JRBC_RB_WPTR);
 }
@@ -419,7 +419,7 @@ static void jpeg_v2_5_dec_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(JPEG, ring->me, mmUVD_JRBC_RB_WPTR, lower_32_bits(ring->wptr));
drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c +2 -2
index 41a00851b6c5..a1b751d9ac06 100644
@@ -427,7 +427,7 @@ static uint64_t jpeg_v3_0_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(JPEG, 0, mmUVD_JRBC_RB_WPTR);
 }
@@ -444,7 +444,7 @@ static void jpeg_v3_0_dec_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(JPEG, 0, mmUVD_JRBC_RB_WPTR, lower_32_bits(ring->wptr));
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +5 -6
index a7ec4ac89da5..0819ffe8e759 100644
@@ -46,7 +46,7 @@ static void mes_v10_1_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		atomic64_set((atomic64_t *)&adev->wb.wb[ring->wptr_offs],
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
 			     ring->wptr);
 		WDOORBELL64(ring->doorbell_index, ring->wptr);
 	} else {
@@ -56,7 +56,7 @@ static void mes_v10_1_ring_set_wptr(struct amdgpu_ring *ring)
 
 static u64 mes_v10_1_ring_get_rptr(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs];
+	return *ring->rptr_cpu_addr;
 }
 
 static u64 mes_v10_1_ring_get_wptr(struct amdgpu_ring *ring)
@@ -64,8 +64,7 @@ static u64 mes_v10_1_ring_get_wptr(struct amdgpu_ring *ring)
 	u64 wptr;
 
 	if (ring->use_doorbell)
-		wptr = atomic64_read((atomic64_t *)
-				     &ring->adev->wb.wb[ring->wptr_offs]);
+		wptr = atomic64_read((atomic64_t *)ring->wptr_cpu_addr);
 	else
 		BUG();
 	return wptr;
@@ -673,13 +672,13 @@ static int mes_v10_1_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_control = tmp;
 
 	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+	wb_gpu_addr = ring->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wb_gpu_addr = ring->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffff8;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
drivers/gpu/drm/amd/amdgpu/sdma_v2_4.c +3 -5
index 84b57b06b20c..6bdffdc1c0b9 100644
@@ -194,7 +194,7 @@ static int sdma_v2_4_init_microcode(struct amdgpu_device *adev)
 static uint64_t sdma_v2_4_ring_get_rptr(struct amdgpu_ring *ring)
 {
 	/* XXX check if swapping is necessary on BE */
-	return ring->adev->wb.wb[ring->rptr_offs] >> 2;
+	return *ring->rptr_cpu_addr >> 2;
 }
 
 /**
@@ -414,12 +414,10 @@ static int sdma_v2_4_gfx_resume(struct amdgpu_device *adev)
 	struct amdgpu_ring *ring;
 	u32 rb_cntl, ib_cntl;
 	u32 rb_bufsz;
-	u32 wb_offset;
 	int i, j, r;
 
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
-		wb_offset = (ring->rptr_offs * 4);
 
 		mutex_lock(&adev->srbm_mutex);
 		for (j = 0; j < 16; j++) {
@@ -455,9 +453,9 @@ static int sdma_v2_4_gfx_resume(struct amdgpu_device *adev)
 
 		/* set the wb address whether it's enabled or not */
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i],
-		       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+		       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i],
-		       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+		       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RPTR_WRITEBACK_ENABLE, 1);
 
drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c +7 -9
index 8af5c94d526a..2584fa3cb13e 100644
@@ -350,7 +350,7 @@ static int sdma_v3_0_init_microcode(struct amdgpu_device *adev)
 static uint64_t sdma_v3_0_ring_get_rptr(struct amdgpu_ring *ring)
 {
 	/* XXX check if swapping is necessary on BE */
-	return ring->adev->wb.wb[ring->rptr_offs] >> 2;
+	return *ring->rptr_cpu_addr >> 2;
 }
 
 /**
@@ -367,7 +367,7 @@ static uint64_t sdma_v3_0_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell || ring->use_pollmem) {
 		/* XXX check if swapping is necessary on BE */
-		wptr = ring->adev->wb.wb[ring->wptr_offs] >> 2;
+		wptr = *ring->wptr_cpu_addr >> 2;
 	} else {
 		wptr = RREG32(mmSDMA0_GFX_RB_WPTR + sdma_offsets[ring->me]) >> 2;
 	}
@@ -387,12 +387,12 @@ static void sdma_v3_0_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		u32 *wb = (u32 *)&adev->wb.wb[ring->wptr_offs];
+		u32 *wb = (u32 *)ring->wptr_cpu_addr;
 		/* XXX check if swapping is necessary on BE */
 		WRITE_ONCE(*wb, ring->wptr << 2);
 		WDOORBELL32(ring->doorbell_index, ring->wptr << 2);
 	} else if (ring->use_pollmem) {
-		u32 *wb = (u32 *)&adev->wb.wb[ring->wptr_offs];
+		u32 *wb = (u32 *)ring->wptr_cpu_addr;
 
 		WRITE_ONCE(*wb, ring->wptr << 2);
 	} else {
@@ -649,7 +649,6 @@ static int sdma_v3_0_gfx_resume(struct amdgpu_device *adev)
 	struct amdgpu_ring *ring;
 	u32 rb_cntl, ib_cntl, wptr_poll_cntl;
 	u32 rb_bufsz;
-	u32 wb_offset;
 	u32 doorbell;
 	u64 wptr_gpu_addr;
 	int i, j, r;
@@ -657,7 +656,6 @@ static int sdma_v3_0_gfx_resume(struct amdgpu_device *adev)
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
 		amdgpu_ring_clear_ring(ring);
-		wb_offset = (ring->rptr_offs * 4);
 
 		mutex_lock(&adev->srbm_mutex);
 		for (j = 0; j < 16; j++) {
@@ -694,9 +692,9 @@ static int sdma_v3_0_gfx_resume(struct amdgpu_device *adev)
 
 		/* set the wb address whether it's enabled or not */
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i],
-		       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+		       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 		WREG32(mmSDMA0_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i],
-		       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+		       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RPTR_WRITEBACK_ENABLE, 1);
 
@@ -715,7 +713,7 @@ static int sdma_v3_0_gfx_resume(struct amdgpu_device *adev)
 		WREG32(mmSDMA0_GFX_DOORBELL + sdma_offsets[i], doorbell);
 
 		/* setup the wptr shadow polling */
-		wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+		wptr_gpu_addr = ring->wptr_gpu_addr;
 
 		WREG32(mmSDMA0_GFX_RB_WPTR_POLL_ADDR_LO + sdma_offsets[i],
 		       lower_32_bits(wptr_gpu_addr));
drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c +11 -17
index 80de85847712..65181efba50e 100644
@@ -722,7 +722,7 @@ static uint64_t sdma_v4_0_ring_get_rptr(struct amdgpu_ring *ring)
 	u64 *rptr;
 
 	/* XXX check if swapping is necessary on BE */
-	rptr = ((u64 *)&ring->adev->wb.wb[ring->rptr_offs]);
+	rptr = ((u64 *)ring->rptr_cpu_addr);
 
 	DRM_DEBUG("rptr before shift == 0x%016llx\n", *rptr);
 	return ((*rptr) >> 2);
@@ -742,7 +742,7 @@ static uint64_t sdma_v4_0_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		wptr = READ_ONCE(*((u64 *)&adev->wb.wb[ring->wptr_offs]));
+		wptr = READ_ONCE(*((u64 *)ring->wptr_cpu_addr));
 		DRM_DEBUG("wptr/doorbell before shift == 0x%016llx\n", wptr);
 	} else {
 		wptr = RREG32_SDMA(ring->me, mmSDMA0_GFX_RB_WPTR_HI);
@@ -768,7 +768,7 @@ static void sdma_v4_0_ring_set_wptr(struct amdgpu_ring *ring)
 
 	DRM_DEBUG("Setting write pointer\n");
 	if (ring->use_doorbell) {
-		u64 *wb = (u64 *)&adev->wb.wb[ring->wptr_offs];
+		u64 *wb = (u64 *)ring->wptr_cpu_addr;
 
 		DRM_DEBUG("Using doorbell -- "
 				"wptr_offs == 0x%08x "
@@ -811,7 +811,7 @@ static uint64_t sdma_v4_0_page_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		wptr = READ_ONCE(*((u64 *)&adev->wb.wb[ring->wptr_offs]));
+		wptr = READ_ONCE(*((u64 *)ring->wptr_cpu_addr));
 	} else {
 		wptr = RREG32_SDMA(ring->me, mmSDMA0_PAGE_RB_WPTR_HI);
 		wptr = wptr << 32;
@@ -833,7 +833,7 @@ static void sdma_v4_0_page_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		u64 *wb = (u64 *)&adev->wb.wb[ring->wptr_offs];
+		u64 *wb = (u64 *)ring->wptr_cpu_addr;
 
 		/* XXX check if swapping is necessary on BE */
 		WRITE_ONCE(*wb, (ring->wptr << 2));
@@ -1174,13 +1174,10 @@ static void sdma_v4_0_gfx_resume(struct amdgpu_device *adev, unsigned int i)
 {
 	struct amdgpu_ring *ring = &adev->sdma.instance[i].ring;
 	u32 rb_cntl, ib_cntl, wptr_poll_cntl;
-	u32 wb_offset;
 	u32 doorbell;
 	u32 doorbell_offset;
 	u64 wptr_gpu_addr;
 
-	wb_offset = (ring->rptr_offs * 4);
-
 	rb_cntl = RREG32_SDMA(i, mmSDMA0_GFX_RB_CNTL);
 	rb_cntl = sdma_v4_0_rb_cntl(ring, rb_cntl);
 	WREG32_SDMA(i, mmSDMA0_GFX_RB_CNTL, rb_cntl);
@@ -1193,9 +1190,9 @@ static void sdma_v4_0_gfx_resume(struct amdgpu_device *adev, unsigned int i)
 
 	/* set the wb address whether it's enabled or not */
 	WREG32_SDMA(i, mmSDMA0_GFX_RB_RPTR_ADDR_HI,
-	       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+	       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 	WREG32_SDMA(i, mmSDMA0_GFX_RB_RPTR_ADDR_LO,
-	       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+	       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 	rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL,
 				RPTR_WRITEBACK_ENABLE, 1);
@@ -1225,7 +1222,7 @@ static void sdma_v4_0_gfx_resume(struct amdgpu_device *adev, unsigned int i)
 	WREG32_SDMA(i, mmSDMA0_GFX_MINOR_PTR_UPDATE, 0);
 
 	/* setup the wptr shadow polling */
-	wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wptr_gpu_addr = ring->wptr_gpu_addr;
 	WREG32_SDMA(i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_LO,
 		    lower_32_bits(wptr_gpu_addr));
 	WREG32_SDMA(i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_HI,
@@ -1264,13 +1261,10 @@ static void sdma_v4_0_page_resume(struct amdgpu_device *adev, unsigned int i)
 {
 	struct amdgpu_ring *ring = &adev->sdma.instance[i].page;
 	u32 rb_cntl, ib_cntl, wptr_poll_cntl;
-	u32 wb_offset;
 	u32 doorbell;
 	u32 doorbell_offset;
 	u64 wptr_gpu_addr;
 
-	wb_offset = (ring->rptr_offs * 4);
-
 	rb_cntl = RREG32_SDMA(i, mmSDMA0_PAGE_RB_CNTL);
 	rb_cntl = sdma_v4_0_rb_cntl(ring, rb_cntl);
 	WREG32_SDMA(i, mmSDMA0_PAGE_RB_CNTL, rb_cntl);
@@ -1283,9 +1277,9 @@ static void sdma_v4_0_page_resume(struct amdgpu_device *adev, unsigned int i)
 
 	/* set the wb address whether it's enabled or not */
 	WREG32_SDMA(i, mmSDMA0_PAGE_RB_RPTR_ADDR_HI,
-	       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+	       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 	WREG32_SDMA(i, mmSDMA0_PAGE_RB_RPTR_ADDR_LO,
-	       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+	       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 	rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_PAGE_RB_CNTL,
 				RPTR_WRITEBACK_ENABLE, 1);
@@ -1316,7 +1310,7 @@ static void sdma_v4_0_page_resume(struct amdgpu_device *adev, unsigned int i)
 	WREG32_SDMA(i, mmSDMA0_PAGE_MINOR_PTR_UPDATE, 0);
 
 	/* setup the wptr shadow polling */
-	wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+	wptr_gpu_addr = ring->wptr_gpu_addr;
 	WREG32_SDMA(i, mmSDMA0_PAGE_RB_WPTR_POLL_ADDR_LO,
 		    lower_32_bits(wptr_gpu_addr));
 	WREG32_SDMA(i, mmSDMA0_PAGE_RB_WPTR_POLL_ADDR_HI,
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +7 -9
index d3939c5f531d..ff359e7f1eb8 100644
@@ -347,7 +347,7 @@ static uint64_t sdma_v5_0_ring_get_rptr(struct amdgpu_ring *ring)
 	u64 *rptr;
 
 	/* XXX check if swapping is necessary on BE */
-	rptr = ((u64 *)&ring->adev->wb.wb[ring->rptr_offs]);
+	rptr = (u64 *)ring->rptr_cpu_addr;
 
 	DRM_DEBUG("rptr before shift == 0x%016llx\n", *rptr);
 	return ((*rptr) >> 2);
@@ -367,7 +367,7 @@ static uint64_t sdma_v5_0_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		wptr = READ_ONCE(*((u64 *)&adev->wb.wb[ring->wptr_offs]));
+		wptr = READ_ONCE(*((u64 *)ring->wptr_cpu_addr));
 		DRM_DEBUG("wptr/doorbell before shift == 0x%016llx\n", wptr);
 	} else {
 		wptr = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, ring->me, mmSDMA0_GFX_RB_WPTR_HI));
@@ -400,8 +400,8 @@ static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring)
 				lower_32_bits(ring->wptr << 2),
 				upper_32_bits(ring->wptr << 2));
 		/* XXX check if swapping is necessary on BE */
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr << 2);
-		adev->wb.wb[ring->wptr_offs + 1] = upper_32_bits(ring->wptr << 2);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
+			     ring->wptr << 2);
 		DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
 				ring->doorbell_index, ring->wptr << 2);
 		WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
@@ -708,7 +708,6 @@ static int sdma_v5_0_gfx_resume(struct amdgpu_device *adev)
 	struct amdgpu_ring *ring;
 	u32 rb_cntl, ib_cntl;
 	u32 rb_bufsz;
-	u32 wb_offset;
 	u32 doorbell;
 	u32 doorbell_offset;
 	u32 temp;
@@ -718,7 +717,6 @@ static int sdma_v5_0_gfx_resume(struct amdgpu_device *adev)
 
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
-		wb_offset = (ring->rptr_offs * 4);
 
 		if (!amdgpu_sriov_vf(adev))
 			WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_SEM_WAIT_FAIL_TIMER_CNTL), 0);
@@ -741,7 +739,7 @@ static int sdma_v5_0_gfx_resume(struct amdgpu_device *adev)
 		WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_HI), 0);
 
 		/* setup the wptr shadow polling */
-		wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+		wptr_gpu_addr = ring->wptr_gpu_addr;
 		WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_LO),
 		       lower_32_bits(wptr_gpu_addr));
 		WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_HI),
@@ -756,9 +754,9 @@ static int sdma_v5_0_gfx_resume(struct amdgpu_device *adev)
 
 		/* set the wb address whether it's enabled or not */
 		WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_RPTR_ADDR_HI),
-		       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+		       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 		WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_RPTR_ADDR_LO),
-		       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+		       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RPTR_WRITEBACK_ENABLE, 1);
 
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +7 -9
index 8298926f8502..bf2cf95cbf8f 100644
@@ -248,7 +248,7 @@ static uint64_t sdma_v5_2_ring_get_rptr(struct amdgpu_ring *ring)
 	u64 *rptr;
 
 	/* XXX check if swapping is necessary on BE */
-	rptr = ((u64 *)&ring->adev->wb.wb[ring->rptr_offs]);
+	rptr = (u64 *)ring->rptr_cpu_addr;
 
 	DRM_DEBUG("rptr before shift == 0x%016llx\n", *rptr);
 	return ((*rptr) >> 2);
@@ -268,7 +268,7 @@ static uint64_t sdma_v5_2_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		wptr = READ_ONCE(*((u64 *)&adev->wb.wb[ring->wptr_offs]));
+		wptr = READ_ONCE(*((u64 *)ring->wptr_cpu_addr));
 		DRM_DEBUG("wptr/doorbell before shift == 0x%016llx\n", wptr);
 	} else {
 		wptr = RREG32(sdma_v5_2_get_reg_offset(adev, ring->me, mmSDMA0_GFX_RB_WPTR_HI));
@@ -301,8 +301,8 @@ static void sdma_v5_2_ring_set_wptr(struct amdgpu_ring *ring)
 				lower_32_bits(ring->wptr << 2),
 				upper_32_bits(ring->wptr << 2));
 		/* XXX check if swapping is necessary on BE */
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr << 2);
-		adev->wb.wb[ring->wptr_offs + 1] = upper_32_bits(ring->wptr << 2);
+		atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
+			     ring->wptr << 2);
 		DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
 				ring->doorbell_index, ring->wptr << 2);
 		WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
@@ -609,7 +609,6 @@ static int sdma_v5_2_gfx_resume(struct amdgpu_device *adev)
 	struct amdgpu_ring *ring;
 	u32 rb_cntl, ib_cntl;
 	u32 rb_bufsz;
-	u32 wb_offset;
 	u32 doorbell;
 	u32 doorbell_offset;
 	u32 temp;
@@ -619,7 +618,6 @@ static int sdma_v5_2_gfx_resume(struct amdgpu_device *adev)
 
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
-		wb_offset = (ring->rptr_offs * 4);
 
 		if (!amdgpu_sriov_vf(adev))
 			WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_SEM_WAIT_FAIL_TIMER_CNTL), 0);
@@ -642,7 +640,7 @@ static int sdma_v5_2_gfx_resume(struct amdgpu_device *adev)
 		WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_HI), 0);
 
 		/* setup the wptr shadow polling */
-		wptr_gpu_addr = adev->wb.gpu_addr + (ring->wptr_offs * 4);
+		wptr_gpu_addr = ring->wptr_gpu_addr;
 		WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_LO),
 		       lower_32_bits(wptr_gpu_addr));
 		WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_WPTR_POLL_ADDR_HI),
@@ -657,9 +655,9 @@ static int sdma_v5_2_gfx_resume(struct amdgpu_device *adev)
 
 		/* set the wb address whether it's enabled or not */
 		WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_RPTR_ADDR_HI),
-		       upper_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFF);
+		       upper_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFF);
 		WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_RPTR_ADDR_LO),
-		       lower_32_bits(adev->wb.gpu_addr + wb_offset) & 0xFFFFFFFC);
+		       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RPTR_WRITEBACK_ENABLE, 1);
 
drivers/gpu/drm/amd/amdgpu/si_dma.c +2 -2
index 2f95235bbfb3..f675111ace20 100644
@@ -40,7 +40,7 @@ static void si_dma_set_irq_funcs(struct amdgpu_device *adev);
 
 static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring)
 {
-	return ring->adev->wb.wb[ring->rptr_offs>>2];
+	return *ring->rptr_cpu_addr;
 }
 
 static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring)
@@ -153,7 +153,7 @@ static int si_dma_start(struct amdgpu_device *adev)
 		WREG32(DMA_RB_RPTR + sdma_offsets[i], 0);
 		WREG32(DMA_RB_WPTR + sdma_offsets[i], 0);
 
-		rptr_addr = adev->wb.gpu_addr + (ring->rptr_offs * 4);
+		rptr_addr = ring->rptr_gpu_addr;
 
 		WREG32(DMA_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr));
 		WREG32(DMA_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF);
drivers/gpu/drm/amd/amdgpu/uvd_v7_0.c +3 -3
index 2f15b8e0f7d7..e668b3baa8c6 100644
@@ -118,7 +118,7 @@ static uint64_t uvd_v7_0_enc_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 
 	if (ring == &adev->uvd.inst[ring->me].ring_enc[0])
 		return RREG32_SOC15(UVD, ring->me, mmUVD_RB_WPTR);
@@ -153,7 +153,7 @@ static void uvd_v7_0_enc_ring_set_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		return;
 	}
@@ -754,7 +754,7 @@ static int uvd_v7_0_mmsch_start(struct amdgpu_device *adev,
 		if (adev->uvd.harvest_config & (1 << i))
 			continue;
 		WDOORBELL32(adev->uvd.inst[i].ring_enc[0].doorbell_index, 0);
-		adev->wb.wb[adev->uvd.inst[i].ring_enc[0].wptr_offs] = 0;
+		*adev->uvd.inst[i].ring_enc[0].wptr_cpu_addr = 0;
 		adev->uvd.inst[i].ring_enc[0].wptr = 0;
 		adev->uvd.inst[i].ring_enc[0].wptr_old = 0;
 	}
drivers/gpu/drm/amd/amdgpu/vce_v4_0.c +3 -3
index d1fc4e0b8265..66cd3d11aa4b 100644
@@ -83,7 +83,7 @@ static uint64_t vce_v4_0_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 
 	if (ring->me == 0)
 		return RREG32(SOC15_REG_OFFSET(VCE, 0, mmVCE_RB_WPTR));
@@ -106,7 +106,7 @@ static void vce_v4_0_ring_set_wptr(struct amdgpu_ring *ring)
 
 	if (ring->use_doorbell) {
 		/* XXX check if swapping is necessary on BE */
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		return;
 	}
@@ -177,7 +177,7 @@ static int vce_v4_0_mmsch_start(struct amdgpu_device *adev,
 	WREG32(SOC15_REG_OFFSET(VCE, 0, mmVCE_MMSCH_VF_MAILBOX_RESP), 0);
 
 	WDOORBELL32(adev->vce.ring[0].doorbell_index, 0);
-	adev->wb.wb[adev->vce.ring[0].wptr_offs] = 0;
+	*adev->vce.ring[0].wptr_cpu_addr = 0;
 	adev->vce.ring[0].wptr = 0;
 	adev->vce.ring[0].wptr_old = 0;
 
drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c +6 -6
index 7a7f35e83dd5..8421044d5629 100644
@@ -1336,7 +1336,7 @@ static uint64_t vcn_v2_0_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR);
 }
@@ -1357,7 +1357,7 @@ static void vcn_v2_0_dec_ring_set_wptr(struct amdgpu_ring *ring)
 			lower_32_bits(ring->wptr) | 0x80000000);
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(UVD, 0, mmUVD_RBC_RB_WPTR, lower_32_bits(ring->wptr));
@@ -1565,12 +1565,12 @@ static uint64_t vcn_v2_0_enc_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst->ring_enc[0]) {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(UVD, 0, mmUVD_RB_WPTR);
 	} else {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(UVD, 0, mmUVD_RB_WPTR2);
 	}
@@ -1589,14 +1589,14 @@ static void vcn_v2_0_enc_ring_set_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst->ring_enc[0]) {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR, lower_32_bits(ring->wptr));
 		}
 	} else {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(UVD, 0, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr));
drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c +6 -6
index 17d44be58877..9352d07539b9 100644
@@ -1491,7 +1491,7 @@ static uint64_t vcn_v2_5_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(VCN, ring->me, mmUVD_RBC_RB_WPTR);
 }
@@ -1508,7 +1508,7 @@ static void vcn_v2_5_dec_ring_set_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(VCN, ring->me, mmUVD_RBC_RB_WPTR, lower_32_bits(ring->wptr));
@@ -1607,12 +1607,12 @@ static uint64_t vcn_v2_5_enc_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst[ring->me].ring_enc[0]) {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR);
 	} else {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR2);
 	}
@@ -1631,14 +1631,14 @@ static void vcn_v2_5_enc_ring_set_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst[ring->me].ring_enc[0]) {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR, lower_32_bits(ring->wptr));
 		}
 	} else {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr));
drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c +6 -6
index cb5f0a12333f..19cdad38d134 100644
@@ -1695,7 +1695,7 @@ static uint64_t vcn_v3_0_dec_ring_get_wptr(struct amdgpu_ring *ring)
 	struct amdgpu_device *adev = ring->adev;
 
 	if (ring->use_doorbell)
-		return adev->wb.wb[ring->wptr_offs];
+		return *ring->wptr_cpu_addr;
 	else
 		return RREG32_SOC15(VCN, ring->me, mmUVD_RBC_RB_WPTR);
 }
@@ -1721,7 +1721,7 @@ static void vcn_v3_0_dec_ring_set_wptr(struct amdgpu_ring *ring)
 	}
 
 	if (ring->use_doorbell) {
-		adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+		*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 		WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 	} else {
 		WREG32_SOC15(VCN, ring->me, mmUVD_RBC_RB_WPTR, lower_32_bits(ring->wptr));
@@ -2012,12 +2012,12 @@ static uint64_t vcn_v3_0_enc_ring_get_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst[ring->me].ring_enc[0]) {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR);
 	} else {
 		if (ring->use_doorbell)
-			return adev->wb.wb[ring->wptr_offs];
+			return *ring->wptr_cpu_addr;
 		else
 			return RREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR2);
 	}
@@ -2036,14 +2036,14 @@ static void vcn_v3_0_enc_ring_set_wptr(struct amdgpu_ring *ring)
 
 	if (ring == &adev->vcn.inst[ring->me].ring_enc[0]) {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR, lower_32_bits(ring->wptr));
 		}
 	} else {
 		if (ring->use_doorbell) {
-			adev->wb.wb[ring->wptr_offs] = lower_32_bits(ring->wptr);
+			*ring->wptr_cpu_addr = lower_32_bits(ring->wptr);
 			WDOORBELL32(ring->doorbell_index, lower_32_bits(ring->wptr));
 		} else {
 			WREG32_SOC15(VCN, ring->me, mmUVD_RB_WPTR2, lower_32_bits(ring->wptr));
Signature · 1 line
2.35.1
author maintainer
#7 lore raw

PATCH 7/73 drm/amdgpu: initialize/finalize the ring for mes queue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Iniailize/finalize the ring for mes queue which submits the command stream to the mes-managed hardware queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 145 ++++++++++++++++-------
 1 file changed, 104 insertions(+), 41 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +104 -41
index 773954318216..13db99d653bd 100644
@@ -149,6 +149,16 @@ void amdgpu_ring_undo(struct amdgpu_ring *ring)
 		ring->funcs->end_use(ring);
 }
 
+#define amdgpu_ring_get_gpu_addr(ring, offset)				\
+	(ring->is_mes_queue ?						\
+	 (ring->mes_ctx->meta_data_gpu_addr + offset) :			\
+	 (ring->adev->wb.gpu_addr + offset * 4))
+
+#define amdgpu_ring_get_cpu_addr(ring, offset)				\
+	(ring->is_mes_queue ?						\
+	 (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
+	 (&ring->adev->wb.wb[offset]))
+
 /**
  * amdgpu_ring_init - init driver ring struct.
  *
@@ -189,51 +199,88 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 			return -EINVAL;
 
 		ring->adev = adev;
-		ring->idx = adev->num_rings++;
-		adev->rings[ring->idx] = ring;
 		ring->num_hw_submission = sched_hw_submission;
 		ring->sched_score = sched_score;
 		ring->vmid_wait = dma_fence_get_stub();
+
+		if (!ring->is_mes_queue) {
+			ring->idx = adev->num_rings++;
+			adev->rings[ring->idx] = ring;
+		}
+
 		r = amdgpu_fence_driver_init_ring(ring);
 		if (r)
 			return r;
 	}
 
-	r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
-	if (r) {
-		dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
-		return r;
-	}
+	if (ring->is_mes_queue) {
+		ring->rptr_offs = amdgpu_mes_ctx_get_offs(ring,
+				AMDGPU_MES_CTX_RPTR_OFFS);
+		ring->wptr_offs = amdgpu_mes_ctx_get_offs(ring,
+				AMDGPU_MES_CTX_WPTR_OFFS);
+		ring->fence_offs = amdgpu_mes_ctx_get_offs(ring,
+				AMDGPU_MES_CTX_FENCE_OFFS);
+		ring->trail_fence_offs = amdgpu_mes_ctx_get_offs(ring,
+				AMDGPU_MES_CTX_TRAIL_FENCE_OFFS);
+		ring->cond_exe_offs = amdgpu_mes_ctx_get_offs(ring,
+				AMDGPU_MES_CTX_COND_EXE_OFFS);
+	} else {
+		r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
+		if (r) {
+			dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
+			return r;
+		}
 
-	r = amdgpu_device_wb_get(adev, &ring->wptr_offs);
-	if (r) {
-		dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);
-		return r;
-	}
+		r = amdgpu_device_wb_get(adev, &ring->wptr_offs);
+		if (r) {
+			dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);
+			return r;
+		}
 
-	r = amdgpu_device_wb_get(adev, &ring->fence_offs);
-	if (r) {
-		dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);
-		return r;
-	}
+		r = amdgpu_device_wb_get(adev, &ring->fence_offs);
+		if (r) {
+			dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);
+			return r;
+		}
 
-	r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);
-	if (r) {
-		dev_err(adev->dev,
-			"(%d) ring trail_fence_offs wb alloc failed\n", r);
-		return r;
+		r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);
+		if (r) {
+			dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r);
+			return r;
+		}
+
+		r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);
+		if (r) {
+			dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);
+			return r;
+		}
 	}
+
+	ring->fence_gpu_addr =
+		amdgpu_ring_get_gpu_addr(ring, ring->fence_offs);
+	ring->fence_cpu_addr =
+		amdgpu_ring_get_cpu_addr(ring, ring->fence_offs);
+
+	ring->rptr_gpu_addr =
+		amdgpu_ring_get_gpu_addr(ring, ring->rptr_offs);
+	ring->rptr_cpu_addr =
+		amdgpu_ring_get_cpu_addr(ring, ring->rptr_offs);
+
+	ring->wptr_gpu_addr =
+		amdgpu_ring_get_gpu_addr(ring, ring->wptr_offs);
+	ring->wptr_cpu_addr =
+		amdgpu_ring_get_cpu_addr(ring, ring->wptr_offs);
+
 	ring->trail_fence_gpu_addr =
-		adev->wb.gpu_addr + (ring->trail_fence_offs * 4);
-	ring->trail_fence_cpu_addr = &adev->wb.wb[ring->trail_fence_offs];
+		amdgpu_ring_get_gpu_addr(ring, ring->trail_fence_offs);
+	ring->trail_fence_cpu_addr =
+		amdgpu_ring_get_cpu_addr(ring, ring->trail_fence_offs);
+
+	ring->cond_exe_gpu_addr =
+		amdgpu_ring_get_gpu_addr(ring, ring->cond_exe_offs);
+	ring->cond_exe_cpu_addr =
+		amdgpu_ring_get_cpu_addr(ring, ring->cond_exe_offs);
 
-	r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);
-	if (r) {
-		dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);
-		return r;
-	}
-	ring->cond_exe_gpu_addr = adev->wb.gpu_addr + (ring->cond_exe_offs * 4);
-	ring->cond_exe_cpu_addr = &adev->wb.wb[ring->cond_exe_offs];
 	/* always set cond_exec_polling to CONTINUE */
 	*ring->cond_exe_cpu_addr = 1;
 
@@ -248,8 +295,20 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 	ring->buf_mask = (ring->ring_size / 4) - 1;
 	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?
 		0xffffffffffffffff : ring->buf_mask;
+
 	/* Allocate ring buffer */
-	if (ring->ring_obj == NULL) {
+	if (ring->is_mes_queue) {
+		int offset = 0;
+
+		BUG_ON(ring->ring_size > PAGE_SIZE*4);
+
+		offset = amdgpu_mes_ctx_get_offs(ring,
+					 AMDGPU_MES_CTX_RING_OFFS);
+		ring->gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		ring->ring = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+		amdgpu_ring_clear_ring(ring);
+
+	} else if (ring->ring_obj == NULL) {
 		r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
 					    AMDGPU_GEM_DOMAIN_GTT,
 					    &ring->ring_obj,
@@ -286,26 +345,30 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring)
 {
 
 	/* Not to finish a ring which is not initialized */
-	if (!(ring->adev) || !(ring->adev->rings[ring->idx]))
+	if (!(ring->adev) ||
+	    (!ring->is_mes_queue && !(ring->adev->rings[ring->idx])))
 		return;
 
 	ring->sched.ready = false;
 
-	amdgpu_device_wb_free(ring->adev, ring->rptr_offs);
-	amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
+	if (!ring->is_mes_queue) {
+		amdgpu_device_wb_free(ring->adev, ring->rptr_offs);
+		amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
 
-	amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);
-	amdgpu_device_wb_free(ring->adev, ring->fence_offs);
+		amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);
+		amdgpu_device_wb_free(ring->adev, ring->fence_offs);
 
-	amdgpu_bo_free_kernel(&ring->ring_obj,
-			      &ring->gpu_addr,
-			      (void **)&ring->ring);
+		amdgpu_bo_free_kernel(&ring->ring_obj,
+				      &ring->gpu_addr,
+				      (void **)&ring->ring);
+	}
 
 	dma_fence_put(ring->vmid_wait);
 	ring->vmid_wait = NULL;
 	ring->me = 0;
 
-	ring->adev->rings[ring->idx] = NULL;
+	if (!ring->is_mes_queue)
+		ring->adev->rings[ring->idx] = NULL;
 }
 
 /**
Signature · 1 line
2.35.1
author maintainer
#8 lore raw

PATCH 8/73 drm/amdgpu: assign the cpu/gpu address of fence from ring

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

assign the cpu/gpu address of fence for the normal or mes ring from ring structure.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +2 -2
index 5d13ed376ab4..d16c8c1f72db 100644
@@ -422,8 +422,8 @@ int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
 	uint64_t index;
 
 	if (ring->funcs->type != AMDGPU_RING_TYPE_UVD) {
-		ring->fence_drv.cpu_addr = &adev->wb.wb[ring->fence_offs];
-		ring->fence_drv.gpu_addr = adev->wb.gpu_addr + (ring->fence_offs * 4);
+		ring->fence_drv.cpu_addr = ring->fence_cpu_addr;
+		ring->fence_drv.gpu_addr = ring->fence_gpu_addr;
 	} else {
 		/* put fence directly behind firmware */
 		index = ALIGN(adev->uvd.fw->size, 8);
Signature · 1 line
2.35.1
author maintainer
#9 lore raw

PATCH 9/73 drm/amdgpu/gfx10: implement mqd functions of gfx/compute eng v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Refine the existing gfx/compute mqd functions, and add them to engine mqd layer.

v2: rebase fix.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 111 +++++++++++++------------
 1 file changed, 56 insertions(+), 55 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +56 -55
index f2dd53f2af61..cc70594d7e4c 100644
@@ -3485,6 +3485,7 @@ static void gfx_v10_0_set_ring_funcs(struct amdgpu_device *adev);
 static void gfx_v10_0_set_irq_funcs(struct amdgpu_device *adev);
 static void gfx_v10_0_set_gds_init(struct amdgpu_device *adev);
 static void gfx_v10_0_set_rlc_funcs(struct amdgpu_device *adev);
+static void gfx_v10_0_set_mqd_funcs(struct amdgpu_device *adev);
 static int gfx_v10_0_get_cu_info(struct amdgpu_device *adev,
 				 struct amdgpu_cu_info *cu_info);
 static uint64_t gfx_v10_0_get_gpu_clock_counter(struct amdgpu_device *adev);
@@ -6564,10 +6565,10 @@ static void gfx_v10_0_kiq_setting(struct amdgpu_ring *ring)
 	}
 }
 
-static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
+static int gfx_v10_0_gfx_mqd_init(struct amdgpu_device *adev, void *m,
+				  struct amdgpu_mqd_prop *prop)
 {
-	struct amdgpu_device *adev = ring->adev;
-	struct v10_gfx_mqd *mqd = ring->mqd_ptr;
+	struct v10_gfx_mqd *mqd = m;
 	uint64_t hqd_gpu_addr, wb_gpu_addr;
 	uint32_t tmp;
 	uint32_t rb_bufsz;
@@ -6577,8 +6578,8 @@ static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_gfx_hqd_wptr_hi = 0;
 
 	/* set the pointer to the MQD */
-	mqd->cp_mqd_base_addr = ring->mqd_gpu_addr & 0xfffffffc;
-	mqd->cp_mqd_base_addr_hi = upper_32_bits(ring->mqd_gpu_addr);
+	mqd->cp_mqd_base_addr = prop->mqd_gpu_addr & 0xfffffffc;
+	mqd->cp_mqd_base_addr_hi = upper_32_bits(prop->mqd_gpu_addr);
 
 	/* set up mqd control */
 	tmp = RREG32_SOC15(GC, 0, mmCP_GFX_MQD_CONTROL);
@@ -6604,23 +6605,23 @@ static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_gfx_hqd_quantum = tmp;
 
 	/* set up gfx hqd base. this is similar as CP_RB_BASE */
-	hqd_gpu_addr = ring->gpu_addr >> 8;
+	hqd_gpu_addr = prop->hqd_base_gpu_addr >> 8;
 	mqd->cp_gfx_hqd_base = hqd_gpu_addr;
 	mqd->cp_gfx_hqd_base_hi = upper_32_bits(hqd_gpu_addr);
 
 	/* set up hqd_rptr_addr/_hi, similar as CP_RB_RPTR */
-	wb_gpu_addr = ring->rptr_gpu_addr;
+	wb_gpu_addr = prop->rptr_gpu_addr;
 	mqd->cp_gfx_hqd_rptr_addr = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_gfx_hqd_rptr_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* set up rb_wptr_poll addr */
-	wb_gpu_addr = ring->wptr_gpu_addr;
+	wb_gpu_addr = prop->wptr_gpu_addr;
 	mqd->cp_rb_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_rb_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* set up the gfx_hqd_control, similar as CP_RB0_CNTL */
-	rb_bufsz = order_base_2(ring->ring_size / 4) - 1;
+	rb_bufsz = order_base_2(prop->queue_size / 4) - 1;
 	tmp = RREG32_SOC15(GC, 0, mmCP_GFX_HQD_CNTL);
 	tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, RB_BUFSZ, rb_bufsz);
 	tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, RB_BLKSZ, rb_bufsz - 2);
@@ -6631,9 +6632,9 @@ static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
 
 	/* set up cp_doorbell_control */
 	tmp = RREG32_SOC15(GC, 0, mmCP_RB_DOORBELL_CONTROL);
-	if (ring->use_doorbell) {
+	if (prop->use_doorbell) {
 		tmp = REG_SET_FIELD(tmp, CP_RB_DOORBELL_CONTROL,
-				    DOORBELL_OFFSET, ring->doorbell_index);
+				    DOORBELL_OFFSET, prop->doorbell_index);
 		tmp = REG_SET_FIELD(tmp, CP_RB_DOORBELL_CONTROL,
 				    DOORBELL_EN, 1);
 	} else
@@ -6641,13 +6642,7 @@ static int gfx_v10_0_gfx_mqd_init(struct amdgpu_ring *ring)
 				    DOORBELL_EN, 0);
 	mqd->cp_rb_doorbell_control = tmp;
 
-	/*if there are 2 gfx rings, set the lower doorbell range of the first ring,
-	 *otherwise the range of the second ring will override the first ring */
-	if (ring->doorbell_index == adev->doorbell_index.gfx_ring0 << 1)
-		gfx_v10_0_cp_gfx_set_doorbell(adev, ring);
-
 	/* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
-	ring->wptr = 0;
 	mqd->cp_gfx_hqd_rptr = RREG32_SOC15(GC, 0, mmCP_GFX_HQD_RPTR);
 
 	/* active the queue */
@@ -6715,7 +6710,16 @@ static int gfx_v10_0_gfx_init_queue(struct amdgpu_ring *ring)
 		memset((void *)mqd, 0, sizeof(*mqd));
 		mutex_lock(&adev->srbm_mutex);
 		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
-		gfx_v10_0_gfx_mqd_init(ring);
+		amdgpu_ring_init_mqd(ring);
+
+		/*
+		 * if there are 2 gfx rings, set the lower doorbell
+		 * range of the first ring, otherwise the range of
+		 * the second ring will override the first ring
+		 */
+		if (ring->doorbell_index == adev->doorbell_index.gfx_ring0 << 1)
+			gfx_v10_0_cp_gfx_set_doorbell(adev, ring);
+
 #ifdef BRING_UP_DEBUG
 		gfx_v10_0_gfx_queue_init_register(ring);
 #endif
@@ -6808,23 +6812,10 @@ static int gfx_v10_0_cp_async_gfx_ring_resume(struct amdgpu_device *adev)
 	return r;
 }
 
-static void gfx_v10_0_compute_mqd_set_priority(struct amdgpu_ring *ring, struct v10_compute_mqd *mqd)
+static int gfx_v10_0_compute_mqd_init(struct amdgpu_device *adev, void *m,
+				      struct amdgpu_mqd_prop *prop)
 {
-	struct amdgpu_device *adev = ring->adev;
-
-	if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
-		if (amdgpu_gfx_is_high_priority_compute_queue(adev, ring)) {
-			mqd->cp_hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_HIGH;
-			mqd->cp_hqd_queue_priority =
-				AMDGPU_GFX_QUEUE_PRIORITY_MAXIMUM;
-		}
-	}
-}
-
-static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
-{
-	struct amdgpu_device *adev = ring->adev;
-	struct v10_compute_mqd *mqd = ring->mqd_ptr;
+	struct v10_compute_mqd *mqd = m;
 	uint64_t hqd_gpu_addr, wb_gpu_addr, eop_base_addr;
 	uint32_t tmp;
 
@@ -6836,7 +6827,7 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->compute_static_thread_mgmt_se3 = 0xffffffff;
 	mqd->compute_misc_reserved = 0x00000003;
 
-	eop_base_addr = ring->eop_gpu_addr >> 8;
+	eop_base_addr = prop->eop_gpu_addr >> 8;
 	mqd->cp_hqd_eop_base_addr_lo = eop_base_addr;
 	mqd->cp_hqd_eop_base_addr_hi = upper_32_bits(eop_base_addr);
 
@@ -6850,9 +6841,9 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	/* enable doorbell? */
 	tmp = RREG32_SOC15(GC, 0, mmCP_HQD_PQ_DOORBELL_CONTROL);
 
-	if (ring->use_doorbell) {
+	if (prop->use_doorbell) {
 		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
-				    DOORBELL_OFFSET, ring->doorbell_index);
+				    DOORBELL_OFFSET, prop->doorbell_index);
 		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
 				    DOORBELL_EN, 1);
 		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
@@ -6867,15 +6858,14 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_doorbell_control = tmp;
 
 	/* disable the queue if it's active */
-	ring->wptr = 0;
 	mqd->cp_hqd_dequeue_request = 0;
 	mqd->cp_hqd_pq_rptr = 0;
 	mqd->cp_hqd_pq_wptr_lo = 0;
 	mqd->cp_hqd_pq_wptr_hi = 0;
 
 	/* set the pointer to the MQD */
-	mqd->cp_mqd_base_addr_lo = ring->mqd_gpu_addr & 0xfffffffc;
-	mqd->cp_mqd_base_addr_hi = upper_32_bits(ring->mqd_gpu_addr);
+	mqd->cp_mqd_base_addr_lo = prop->mqd_gpu_addr & 0xfffffffc;
+	mqd->cp_mqd_base_addr_hi = upper_32_bits(prop->mqd_gpu_addr);
 
 	/* set MQD vmid to 0 */
 	tmp = RREG32_SOC15(GC, 0, mmCP_MQD_CONTROL);
@@ -6883,14 +6873,14 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_mqd_control = tmp;
 
 	/* set the pointer to the HQD, this is similar CP_RB0_BASE/_HI */
-	hqd_gpu_addr = ring->gpu_addr >> 8;
+	hqd_gpu_addr = prop->hqd_base_gpu_addr >> 8;
 	mqd->cp_hqd_pq_base_lo = hqd_gpu_addr;
 	mqd->cp_hqd_pq_base_hi = upper_32_bits(hqd_gpu_addr);
 
 	/* set up the HQD, this is similar to CP_RB0_CNTL */
 	tmp = RREG32_SOC15(GC, 0, mmCP_HQD_PQ_CONTROL);
 	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, QUEUE_SIZE,
-			    (order_base_2(ring->ring_size / 4) - 1));
+			    (order_base_2(prop->queue_size / 4) - 1));
 	tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, RPTR_BLOCK_SIZE,
 			    ((order_base_2(AMDGPU_GPU_PAGE_SIZE / 4) - 1) << 8));
 #ifdef __BIG_ENDIAN
@@ -6903,22 +6893,22 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_control = tmp;
 
 	/* set the wb address whether it's enabled or not */
-	wb_gpu_addr = ring->rptr_gpu_addr;
+	wb_gpu_addr = prop->rptr_gpu_addr;
 	mqd->cp_hqd_pq_rptr_report_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_rptr_report_addr_hi =
 		upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	/* only used if CP_PQ_WPTR_POLL_CNTL.CP_PQ_WPTR_POLL_CNTL__EN_MASK=1 */
-	wb_gpu_addr = ring->wptr_gpu_addr;
+	wb_gpu_addr = prop->wptr_gpu_addr;
 	mqd->cp_hqd_pq_wptr_poll_addr_lo = wb_gpu_addr & 0xfffffffc;
 	mqd->cp_hqd_pq_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr) & 0xffff;
 
 	tmp = 0;
 	/* enable the doorbell if requested */
-	if (ring->use_doorbell) {
+	if (prop->use_doorbell) {
 		tmp = RREG32_SOC15(GC, 0, mmCP_HQD_PQ_DOORBELL_CONTROL);
 		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
-				DOORBELL_OFFSET, ring->doorbell_index);
+				DOORBELL_OFFSET, prop->doorbell_index);
 
 		tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_DOORBELL_CONTROL,
 				    DOORBELL_EN, 1);
@@ -6931,7 +6921,6 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_pq_doorbell_control = tmp;
 
 	/* reset read and write pointers, similar to CP_RB0_WPTR/_RPTR */
-	ring->wptr = 0;
 	mqd->cp_hqd_pq_rptr = RREG32_SOC15(GC, 0, mmCP_HQD_PQ_RPTR);
 
 	/* set the vmid for the queue */
@@ -6947,13 +6936,10 @@ static int gfx_v10_0_compute_mqd_init(struct amdgpu_ring *ring)
 	mqd->cp_hqd_ib_control = tmp;
 
 	/* set static priority for a compute queue/ring */
-	gfx_v10_0_compute_mqd_set_priority(ring, mqd);
+	mqd->cp_hqd_pipe_priority = prop->hqd_pipe_priority;
+	mqd->cp_hqd_queue_priority = prop->hqd_queue_priority;
 
-	/* map_queues packet doesn't need activate the queue,
-	 * so only kiq need set this field.
-	 */
-	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
-		mqd->cp_hqd_active = 1;
+	mqd->cp_hqd_active = prop->hqd_active;
 
 	return 0;
 }
@@ -7094,7 +7080,7 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)
 		memset((void *)mqd, 0, sizeof(*mqd));
 		mutex_lock(&adev->srbm_mutex);
 		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
-		gfx_v10_0_compute_mqd_init(ring);
+		amdgpu_ring_init_mqd(ring);
 		gfx_v10_0_kiq_init_register(ring);
 		nv_grbm_select(adev, 0, 0, 0, 0);
 		mutex_unlock(&adev->srbm_mutex);
@@ -7116,7 +7102,7 @@ static int gfx_v10_0_kcq_init_queue(struct amdgpu_ring *ring)
 		memset((void *)mqd, 0, sizeof(*mqd));
 		mutex_lock(&adev->srbm_mutex);
 		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
-		gfx_v10_0_compute_mqd_init(ring);
+		amdgpu_ring_init_mqd(ring);
 		nv_grbm_select(adev, 0, 0, 0, 0);
 		mutex_unlock(&adev->srbm_mutex);
 
@@ -7799,6 +7785,7 @@ static int gfx_v10_0_early_init(void *handle)
 	gfx_v10_0_set_irq_funcs(adev);
 	gfx_v10_0_set_gds_init(adev);
 	gfx_v10_0_set_rlc_funcs(adev);
+	gfx_v10_0_set_mqd_funcs(adev);
 
 	/* init rlcg reg access ctrl */
 	gfx_v10_0_init_rlcg_reg_access_ctrl(adev);
@@ -9581,6 +9568,20 @@ static void gfx_v10_0_set_gds_init(struct amdgpu_device *adev)
 	adev->gds.oa_size = 16;
 }
 
+static void gfx_v10_0_set_mqd_funcs(struct amdgpu_device *adev)
+{
+	/* set gfx eng mqd */
+	adev->mqds[AMDGPU_HW_IP_GFX].mqd_size =
+		sizeof(struct v10_gfx_mqd);
+	adev->mqds[AMDGPU_HW_IP_GFX].init_mqd =
+		gfx_v10_0_gfx_mqd_init;
+	/* set compute eng mqd */
+	adev->mqds[AMDGPU_HW_IP_COMPUTE].mqd_size =
+		sizeof(struct v10_compute_mqd);
+	adev->mqds[AMDGPU_HW_IP_COMPUTE].init_mqd =
+		gfx_v10_0_compute_mqd_init;
+}
+
 static void gfx_v10_0_set_user_wgp_inactive_bitmap_per_sh(struct amdgpu_device *adev,
 							  u32 bitmap)
 {
Signature · 1 line
2.35.1
author maintainer
#10 lore raw

PATCH 10/73 drm/amdgpu/gfx10: use per ctx CSA for ce metadata

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

As MES requires per context preemption, use per context CSA address for CE metadata to correctly enable context MCBP preemption.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 28 +++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +19 -9
index cc70594d7e4c..56a7153474c6 100644
@@ -8849,26 +8849,36 @@ static void gfx_v10_0_ring_emit_ce_meta(struct amdgpu_ring *ring, bool resume)
 {
 	struct amdgpu_device *adev = ring->adev;
 	struct v10_ce_ib_state ce_payload = {0};
-	uint64_t csa_addr;
+	uint64_t offset, ce_payload_gpu_addr;
+	void *ce_payload_cpu_addr;
 	int cnt;
 
 	cnt = (sizeof(ce_payload) >> 2) + 4 - 2;
-	csa_addr = amdgpu_csa_vaddr(ring->adev);
+
+	if (ring->is_mes_queue) {
+		offset = offsetof(struct amdgpu_mes_ctx_meta_data,
+				  gfx[0].gfx_meta_data) +
+			offsetof(struct v10_gfx_meta_data, ce_payload);
+		ce_payload_gpu_addr =
+			amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		ce_payload_cpu_addr =
+			amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+	} else {
+		offset = offsetof(struct v10_gfx_meta_data, ce_payload);
+		ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
+		ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
+	}
 
 	amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
 	amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) |
 				 WRITE_DATA_DST_SEL(8) |
 				 WR_CONFIRM) |
 				 WRITE_DATA_CACHE_POLICY(0));
-	amdgpu_ring_write(ring, lower_32_bits(csa_addr +
-			      offsetof(struct v10_gfx_meta_data, ce_payload)));
-	amdgpu_ring_write(ring, upper_32_bits(csa_addr +
-			      offsetof(struct v10_gfx_meta_data, ce_payload)));
+	amdgpu_ring_write(ring, lower_32_bits(ce_payload_gpu_addr));
+	amdgpu_ring_write(ring, upper_32_bits(ce_payload_gpu_addr));
 
 	if (resume)
-		amdgpu_ring_write_multiple(ring, adev->virt.csa_cpu_addr +
-					   offsetof(struct v10_gfx_meta_data,
-						    ce_payload),
+		amdgpu_ring_write_multiple(ring, ce_payload_cpu_addr,
 					   sizeof(ce_payload) >> 2);
 	else
 		amdgpu_ring_write_multiple(ring, (void *)&ce_payload,
Signature · 1 line
2.35.1
author maintainer
#11 lore raw

PATCH 11/73 drm/amdgpu/gfx10: use per ctx CSA for de metadata

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

As MES requires per context preemption, use per context CSA address for DE metadata to correctly enable context MCBP preemption.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 39 ++++++++++++++++++--------
 1 file changed, 28 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +28 -11
index 56a7153474c6..d06807355f5f 100644
@@ -8889,12 +8889,33 @@ static void gfx_v10_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume)
 {
 	struct amdgpu_device *adev = ring->adev;
 	struct v10_de_ib_state de_payload = {0};
-	uint64_t csa_addr, gds_addr;
+	uint64_t offset, gds_addr, de_payload_gpu_addr;
+	void *de_payload_cpu_addr;
 	int cnt;
 
-	csa_addr = amdgpu_csa_vaddr(ring->adev);
-	gds_addr = ALIGN(csa_addr + AMDGPU_CSA_SIZE - adev->gds.gds_size,
-			 PAGE_SIZE);
+	if (ring->is_mes_queue) {
+		offset = offsetof(struct amdgpu_mes_ctx_meta_data,
+				  gfx[0].gfx_meta_data) +
+			offsetof(struct v10_gfx_meta_data, de_payload);
+		de_payload_gpu_addr =
+			amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		de_payload_cpu_addr =
+			amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+
+		offset = offsetof(struct amdgpu_mes_ctx_meta_data,
+				  gfx[0].gds_backup) +
+			offsetof(struct v10_gfx_meta_data, de_payload);
+		gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+	} else {
+		offset = offsetof(struct v10_gfx_meta_data, de_payload);
+		de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
+		de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
+
+		gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
+				 AMDGPU_CSA_SIZE - adev->gds.gds_size,
+				 PAGE_SIZE);
+	}
+
 	de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);
 	de_payload.gds_backup_addrhi = upper_32_bits(gds_addr);
 
@@ -8904,15 +8925,11 @@ static void gfx_v10_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume)
 				 WRITE_DATA_DST_SEL(8) |
 				 WR_CONFIRM) |
 				 WRITE_DATA_CACHE_POLICY(0));
-	amdgpu_ring_write(ring, lower_32_bits(csa_addr +
-			      offsetof(struct v10_gfx_meta_data, de_payload)));
-	amdgpu_ring_write(ring, upper_32_bits(csa_addr +
-			      offsetof(struct v10_gfx_meta_data, de_payload)));
+	amdgpu_ring_write(ring, lower_32_bits(de_payload_gpu_addr));
+	amdgpu_ring_write(ring, upper_32_bits(de_payload_gpu_addr));
 
 	if (resume)
-		amdgpu_ring_write_multiple(ring, adev->virt.csa_cpu_addr +
-					   offsetof(struct v10_gfx_meta_data,
-						    de_payload),
+		amdgpu_ring_write_multiple(ring, de_payload_cpu_addr,
 					   sizeof(de_payload) >> 2);
 	else
 		amdgpu_ring_write_multiple(ring, (void *)&de_payload,
Signature · 1 line
2.35.1
author maintainer
#12 lore raw

PATCH 12/73 drm/amdgpu/gfx10: associate mes queue id with fence v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Associate mes queue id with fence, so that EOP trap handler can look up which queue has issued the fence.

v2: move mes queue flag to amdgpu_mes_ctx.h

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h | 2 ++
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c      | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h +2 -0
index f3e1ba1a889f..544f1aa86edf 100644
@@ -115,4 +115,6 @@ struct amdgpu_mes_ctx_data {
 #define AMDGPU_FENCE_MES_QUEUE_FLAG     0x1000000u
 #define AMDGPU_FENCE_MES_QUEUE_ID_MASK  (AMDGPU_FENCE_MES_QUEUE_FLAG - 1)
 
+#define AMDGPU_FENCE_MES_QUEUE_FLAG     0x1000000u
+
 #endif
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 -1
index d06807355f5f..e6e601296097 100644
@@ -8678,7 +8678,8 @@ static void gfx_v10_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr,
 	amdgpu_ring_write(ring, upper_32_bits(addr));
 	amdgpu_ring_write(ring, lower_32_bits(seq));
 	amdgpu_ring_write(ring, upper_32_bits(seq));
-	amdgpu_ring_write(ring, 0);
+	amdgpu_ring_write(ring, ring->is_mes_queue ?
+			 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0);
 }
 
 static void gfx_v10_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
Signature · 1 line
2.35.1
author maintainer
#13 lore raw

PATCH 13/73 drm/amdgpu/gfx10: inherit vmid from mqd

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For MES manages vmid assignment, let vmid inherit from mqd instead of ib packet setting.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +8 -0
index e6e601296097..0d91632f563d 100644
@@ -8602,6 +8602,10 @@ static void gfx_v10_0_ring_emit_ib_gfx(struct amdgpu_ring *ring,
 				    (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false);
 	}
 
+	if (ring->is_mes_queue)
+		/* inherit vmid from mqd */
+		control |= 0x400000;
+
 	amdgpu_ring_write(ring, header);
 	BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
 	amdgpu_ring_write(ring,
@@ -8621,6 +8625,10 @@ static void gfx_v10_0_ring_emit_ib_compute(struct amdgpu_ring *ring,
 	unsigned vmid = AMDGPU_JOB_GET_VMID(job);
 	u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);
 
+	if (ring->is_mes_queue)
+		/* inherit vmid from mqd */
+		control |= 0x40000000;
+
 	/* Currently, there is a high possibility to get wave ID mismatch
 	 * between ME and GDS, leading to a hw deadlock, because ME generates
 	 * different wave IDs than the GDS expects. This situation happens
Signature · 1 line
2.35.1
author maintainer
#14 lore raw

PATCH 14/73 drm/amdgpu/gfx10: use INVALIDATE_TLBS to invalidate TLBs v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For MES queue VM flush, use INVALIDATE_TLBS to invalidate TLBs. This packet can let CP firmware to determine the current vmid and inv eng to invalidate.

v2: unify invalidate_tlbs functions

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 27 +++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +20 -7
index 0d91632f563d..2ab5259c7305 100644
@@ -3503,6 +3503,9 @@ static void gfx_v10_0_ring_emit_frame_cntl(struct amdgpu_ring *ring, bool start,
 static u32 gfx_v10_3_get_disabled_sa(struct amdgpu_device *adev);
 static void gfx_v10_3_program_pbb_mode(struct amdgpu_device *adev);
 static void gfx_v10_3_set_power_brake_sequence(struct amdgpu_device *adev);
+static void gfx_v10_0_ring_invalidate_tlbs(struct amdgpu_ring *ring,
+					   uint16_t pasid, uint32_t flush_type,
+					   bool all_hub, uint8_t dst_sel);
 
 static void gfx10_kiq_set_resources(struct amdgpu_ring *kiq_ring, uint64_t queue_mask)
 {
@@ -3595,12 +3598,7 @@ static void gfx10_kiq_invalidate_tlbs(struct amdgpu_ring *kiq_ring,
 				uint16_t pasid, uint32_t flush_type,
 				bool all_hub)
 {
-	amdgpu_ring_write(kiq_ring, PACKET3(PACKET3_INVALIDATE_TLBS, 0));
-	amdgpu_ring_write(kiq_ring,
-			PACKET3_INVALIDATE_TLBS_DST_SEL(1) |
-			PACKET3_INVALIDATE_TLBS_ALL_HUB(all_hub) |
-			PACKET3_INVALIDATE_TLBS_PASID(pasid) |
-			PACKET3_INVALIDATE_TLBS_FLUSH_TYPE(flush_type));
+	gfx_v10_0_ring_invalidate_tlbs(kiq_ring, pasid, flush_type, all_hub, 1);
 }
 
 static const struct kiq_pm4_funcs gfx_v10_0_kiq_pm4_funcs = {
@@ -8700,10 +8698,25 @@ static void gfx_v10_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
 			       upper_32_bits(addr), seq, 0xffffffff, 4);
 }
 
+static void gfx_v10_0_ring_invalidate_tlbs(struct amdgpu_ring *ring,
+				   uint16_t pasid, uint32_t flush_type,
+				   bool all_hub, uint8_t dst_sel)
+{
+	amdgpu_ring_write(ring, PACKET3(PACKET3_INVALIDATE_TLBS, 0));
+	amdgpu_ring_write(ring,
+			  PACKET3_INVALIDATE_TLBS_DST_SEL(dst_sel) |
+			  PACKET3_INVALIDATE_TLBS_ALL_HUB(all_hub) |
+			  PACKET3_INVALIDATE_TLBS_PASID(pasid) |
+			  PACKET3_INVALIDATE_TLBS_FLUSH_TYPE(flush_type));
+}
+
 static void gfx_v10_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
 					 unsigned vmid, uint64_t pd_addr)
 {
-	amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
+	if (ring->is_mes_queue)
+		gfx_v10_0_ring_invalidate_tlbs(ring, 0, 0, false, 0);
+	else
+		amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
 
 	/* compute doesn't have PFP */
 	if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
Signature · 1 line
2.35.1
author maintainer
#15 lore raw

PATCH 15/73 drm/amdgpu/gmc10: skip emitting pasid mapping packet

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For MES FW manages IH_VMID_x_LUT updating, skip emitting pasid mapping packet.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +4 -0
index 487c33937a87..9b4a035a5bf1 100644
@@ -517,6 +517,10 @@ static void gmc_v10_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned vmid
 	struct amdgpu_device *adev = ring->adev;
 	uint32_t reg;
 
+	/* MES fw manages IH_VMID_x_LUT updating */
+	if (ring->is_mes_queue)
+		return;
+
 	if (ring->funcs->vmhub == AMDGPU_GFXHUB_0)
 		reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid;
 	else
Signature · 1 line
2.35.1
author maintainer
#16 lore raw

PATCH 16/73 drm/amdgpu: use the whole doorbell space for mes

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Use the whole doorbell space for mes. Each queue in one process occupies one doorbell slot to ring the queue submitting.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 32 +++++++++++++---------
 1 file changed, 19 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +19 -13
index f33e3c341f8f..b9844249d464 100644
@@ -1044,19 +1044,25 @@ static int amdgpu_device_doorbell_init(struct amdgpu_device *adev)
 	adev->doorbell.base = pci_resource_start(adev->pdev, 2);
 	adev->doorbell.size = pci_resource_len(adev->pdev, 2);
 
-	adev->doorbell.num_doorbells = min_t(u32, adev->doorbell.size / sizeof(u32),
-					     adev->doorbell_index.max_assignment+1);
-	if (adev->doorbell.num_doorbells == 0)
-		return -EINVAL;
-
-	/* For Vega, reserve and map two pages on doorbell BAR since SDMA
-	 * paging queue doorbell use the second page. The
-	 * AMDGPU_DOORBELL64_MAX_ASSIGNMENT definition assumes all the
-	 * doorbells are in the first page. So with paging queue enabled,
-	 * the max num_doorbells should + 1 page (0x400 in dword)
-	 */
-	if (adev->asic_type >= CHIP_VEGA10)
-		adev->doorbell.num_doorbells += 0x400;
+	if (adev->enable_mes) {
+		adev->doorbell.num_doorbells =
+			adev->doorbell.size / sizeof(u32);
+	} else {
+		adev->doorbell.num_doorbells =
+			min_t(u32, adev->doorbell.size / sizeof(u32),
+			      adev->doorbell_index.max_assignment+1);
+		if (adev->doorbell.num_doorbells == 0)
+			return -EINVAL;
+
+		/* For Vega, reserve and map two pages on doorbell BAR since SDMA
+		 * paging queue doorbell use the second page. The
+		 * AMDGPU_DOORBELL64_MAX_ASSIGNMENT definition assumes all the
+		 * doorbells are in the first page. So with paging queue enabled,
+		 * the max num_doorbells should + 1 page (0x400 in dword)
+		 */
+		if (adev->asic_type >= CHIP_VEGA10)
+			adev->doorbell.num_doorbells += 0x400;
+	}
 
 	adev->doorbell.ptr = ioremap(adev->doorbell.base,
 				     adev->doorbell.num_doorbells *
Signature · 1 line
2.35.1
author maintainer
#17 lore raw

PATCH 17/73 drm/amdgpu: update mes process/gang/queue definitions

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Update the definitions of MES process/gang/queue.

v2: add missing includes

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 58 +++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +58 -0
index 7334982ea702..52483d7ce843 100644
@@ -24,6 +24,10 @@
 #ifndef __AMDGPU_MES_H__
 #define __AMDGPU_MES_H__
 
+#include "amdgpu_irq.h"
+#include "kgd_kfd_interface.h"
+#include "amdgpu_gfx.h"
+
 #define AMDGPU_MES_MAX_COMPUTE_PIPES        8
 #define AMDGPU_MES_MAX_GFX_PIPES            2
 #define AMDGPU_MES_MAX_SDMA_PIPES           2
@@ -37,11 +41,23 @@ enum amdgpu_mes_priority_level {
 	AMDGPU_MES_PRIORITY_NUM_LEVELS
 };
 
+#define AMDGPU_MES_PROC_CTX_SIZE 0x1000 /* one page area */
+#define AMDGPU_MES_GANG_CTX_SIZE 0x1000 /* one page area */
+
 struct amdgpu_mes_funcs;
 
 struct amdgpu_mes {
 	struct amdgpu_device            *adev;
 
+	struct mutex                    mutex;
+
+	struct idr                      pasid_idr;
+	struct idr                      gang_id_idr;
+	struct idr                      queue_id_idr;
+	struct ida                      doorbell_ida;
+
+	spinlock_t                      queue_id_lock;
+
 	uint32_t                        total_max_queue;
 	uint32_t                        doorbell_id_offset;
 	uint32_t                        max_doorbell_slices;
@@ -90,6 +106,48 @@ struct amdgpu_mes {
 	const struct amdgpu_mes_funcs   *funcs;
 };
 
+struct amdgpu_mes_process {
+	int			pasid;
+	struct			amdgpu_vm *vm;
+	uint64_t		pd_gpu_addr;
+	struct amdgpu_bo 	*proc_ctx_bo;
+	uint64_t 		proc_ctx_gpu_addr;
+	void 			*proc_ctx_cpu_ptr;
+	uint64_t 		process_quantum;
+	struct 			list_head gang_list;
+	uint32_t 		doorbell_index;
+	unsigned long 		*doorbell_bitmap;
+	struct mutex		doorbell_lock;
+};
+
+struct amdgpu_mes_gang {
+	int 				gang_id;
+	int 				priority;
+	int 				inprocess_gang_priority;
+	int 				global_priority_level;
+	struct list_head 		list;
+	struct amdgpu_mes_process 	*process;
+	struct amdgpu_bo 		*gang_ctx_bo;
+	uint64_t 			gang_ctx_gpu_addr;
+	void 				*gang_ctx_cpu_ptr;
+	uint64_t 			gang_quantum;
+	struct list_head 		queue_list;
+};
+
+struct amdgpu_mes_queue {
+	struct list_head 		list;
+	struct amdgpu_mes_gang 		*gang;
+	int 				queue_id;
+	uint64_t 			doorbell_off;
+	struct amdgpu_bo		*mqd_obj;
+	void				*mqd_cpu_ptr;
+	uint64_t 			mqd_gpu_addr;
+	uint64_t 			wptr_gpu_addr;
+	int 				queue_type;
+	int 				paging;
+	struct amdgpu_ring 		*ring;
+};
+
 struct mes_add_queue_input {
 	uint32_t	process_id;
 	uint64_t	page_table_base_addr;
Signature · 1 line
2.35.1
author maintainer
#18 lore raw

PATCH 18/73 drm/amdgpu: add mes_kiq module parameter v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

mes_kiq parameter is used to enable mes kiq pipe. This module parameter is unneccessary or enabled by default in final version.

v2: reword commit message.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  9 +++++++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 10 ++++++++++
 3 files changed, 19 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -0
index 24bce7e691a8..4264abc5604d 100644
@@ -208,6 +208,7 @@ extern int amdgpu_async_gfx_ring;
 extern int amdgpu_mcbp;
 extern int amdgpu_discovery;
 extern int amdgpu_mes;
+extern int amdgpu_mes_kiq;
 extern int amdgpu_noretry;
 extern int amdgpu_force_asic_type;
 extern int amdgpu_smartshift_bias;
@@ -940,6 +941,7 @@ struct amdgpu_device {
 
 	/* mes */
 	bool                            enable_mes;
+	bool                            enable_mes_kiq;
 	struct amdgpu_mes               mes;
 	struct amdgpu_mqd               mqds[AMDGPU_HW_IP_NUM];
 
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +7 -2
index b9844249d464..b2366d0d3047 100644
@@ -3669,8 +3669,13 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	if (amdgpu_mcbp)
 		DRM_INFO("MCBP is enabled\n");
 
-	if (amdgpu_mes && adev->asic_type >= CHIP_NAVI10)
-		adev->enable_mes = true;
+	if (adev->asic_type >= CHIP_NAVI10) {
+		if (amdgpu_mes || amdgpu_mes_kiq)
+			adev->enable_mes = true;
+
+		if (amdgpu_mes_kiq)
+			adev->enable_mes_kiq = true;
+	}
 
 	/*
 	 * Reset domain needs to be present early, before XGMI hive discovered
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +10 -0
index 80690cbac89f..9e72b3ec5d4d 100644
@@ -171,6 +171,7 @@ int amdgpu_async_gfx_ring = 1;
 int amdgpu_mcbp;
 int amdgpu_discovery = -1;
 int amdgpu_mes;
+int amdgpu_mes_kiq;
 int amdgpu_noretry = -1;
 int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = -1; /* auto */
@@ -636,6 +637,15 @@ MODULE_PARM_DESC(mes,
 	"Enable Micro Engine Scheduler (0 = disabled (default), 1 = enabled)");
 module_param_named(mes, amdgpu_mes, int, 0444);
 
+/**
+ * DOC: mes_kiq (int)
+ * Enable Micro Engine Scheduler KIQ. This is a new engine pipe for kiq.
+ * (0 = disabled (default), 1 = enabled)
+ */
+MODULE_PARM_DESC(mes_kiq,
+	"Enable Micro Engine Scheduler KIQ (0 = disabled (default), 1 = enabled)");
+module_param_named(mes_kiq, amdgpu_mes_kiq, int, 0444);
+
 /**
  * DOC: noretry (int)
  * Disable XNACK retry in the SQ by default on GFXv9 hardware. On ASICs that
Signature · 1 line
2.35.1
author maintainer
#19 lore raw

PATCH 19/73 drm/amdgpu: allocate doorbell index for mes kiq

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Allocate a doorbell index for mes kiq queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h | 6 ++++--
 drivers/gpu/drm/amd/amdgpu/nv.c              | 3 ++-
 drivers/gpu/drm/amd/amdgpu/soc21.c           | 3 ++-
 3 files changed, 8 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h +4 -2
index 89e6ad30396f..2d9485e67125 100644
@@ -53,7 +53,8 @@ struct amdgpu_doorbell_index {
 	uint32_t gfx_ring0;
 	uint32_t gfx_ring1;
 	uint32_t sdma_engine[8];
-	uint32_t mes_ring;
+	uint32_t mes_ring0;
+	uint32_t mes_ring1;
 	uint32_t ih;
 	union {
 		struct {
@@ -178,7 +179,8 @@ typedef enum _AMDGPU_NAVI10_DOORBELL_ASSIGNMENT
 	AMDGPU_NAVI10_DOORBELL_USERQUEUE_END		= 0x08A,
 	AMDGPU_NAVI10_DOORBELL_GFX_RING0		= 0x08B,
 	AMDGPU_NAVI10_DOORBELL_GFX_RING1		= 0x08C,
-	AMDGPU_NAVI10_DOORBELL_MES_RING		        = 0x090,
+	AMDGPU_NAVI10_DOORBELL_MES_RING0	        = 0x090,
+	AMDGPU_NAVI10_DOORBELL_MES_RING1		= 0x091,
 	/* SDMA:256~335*/
 	AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE0		= 0x100,
 	AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE1		= 0x10A,
drivers/gpu/drm/amd/amdgpu/nv.c +2 -1
index 0a7946c59a42..8cf1a7f8a632 100644
@@ -607,7 +607,8 @@ static void nv_init_doorbell_index(struct amdgpu_device *adev)
 	adev->doorbell_index.userqueue_end = AMDGPU_NAVI10_DOORBELL_USERQUEUE_END;
 	adev->doorbell_index.gfx_ring0 = AMDGPU_NAVI10_DOORBELL_GFX_RING0;
 	adev->doorbell_index.gfx_ring1 = AMDGPU_NAVI10_DOORBELL_GFX_RING1;
-	adev->doorbell_index.mes_ring = AMDGPU_NAVI10_DOORBELL_MES_RING;
+	adev->doorbell_index.mes_ring0 = AMDGPU_NAVI10_DOORBELL_MES_RING0;
+	adev->doorbell_index.mes_ring1 = AMDGPU_NAVI10_DOORBELL_MES_RING1;
 	adev->doorbell_index.sdma_engine[0] = AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE0;
 	adev->doorbell_index.sdma_engine[1] = AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE1;
 	adev->doorbell_index.sdma_engine[2] = AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE2;
drivers/gpu/drm/amd/amdgpu/soc21.c +2 -1
index c3069f5e299a..68985a59a6a6 100644
@@ -409,7 +409,8 @@ static void soc21_init_doorbell_index(struct amdgpu_device *adev)
 	adev->doorbell_index.userqueue_end = AMDGPU_NAVI10_DOORBELL_USERQUEUE_END;
 	adev->doorbell_index.gfx_ring0 = AMDGPU_NAVI10_DOORBELL_GFX_RING0;
 	adev->doorbell_index.gfx_ring1 = AMDGPU_NAVI10_DOORBELL_GFX_RING1;
-	adev->doorbell_index.mes_ring = AMDGPU_NAVI10_DOORBELL_MES_RING;
+	adev->doorbell_index.mes_ring0 = AMDGPU_NAVI10_DOORBELL_MES_RING0;
+	adev->doorbell_index.mes_ring1 = AMDGPU_NAVI10_DOORBELL_MES_RING1;
 	adev->doorbell_index.sdma_engine[0] = AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE0;
 	adev->doorbell_index.sdma_engine[1] = AMDGPU_NAVI10_DOORBELL_sDMA_ENGINE1;
 	adev->doorbell_index.ih = AMDGPU_NAVI10_DOORBELL_IH;
Signature · 1 line
2.35.1
author maintainer
#20 lore raw

PATCH 20/73 drm/amdgpu/mes: extend mes framework to support multiple mes pipes

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add support for multiple mes pipes, so that reuse the existing code to initialize more mes pipe and queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  35 ++--
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c  | 215 ++++++++++++++----------
 2 files changed, 149 insertions(+), 101 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +21 -14
index 52483d7ce843..91b020842eb0 100644
@@ -46,6 +46,12 @@ enum amdgpu_mes_priority_level {
 
 struct amdgpu_mes_funcs;
 
+enum admgpu_mes_pipe {
+	AMDGPU_MES_SCHED_PIPE = 0,
+	AMDGPU_MES_KIQ_PIPE,
+	AMDGPU_MAX_MES_PIPES = 2,
+};
+
 struct amdgpu_mes {
 	struct amdgpu_device            *adev;
 
@@ -67,27 +73,28 @@ struct amdgpu_mes {
 
 	struct amdgpu_ring              ring;
 
-	const struct firmware           *fw;
+	const struct firmware           *fw[AMDGPU_MAX_MES_PIPES];
 
 	/* mes ucode */
-	struct amdgpu_bo		*ucode_fw_obj;
-	uint64_t			ucode_fw_gpu_addr;
-	uint32_t			*ucode_fw_ptr;
-	uint32_t                        ucode_fw_version;
-	uint64_t                        uc_start_addr;
+	struct amdgpu_bo		*ucode_fw_obj[AMDGPU_MAX_MES_PIPES];
+	uint64_t			ucode_fw_gpu_addr[AMDGPU_MAX_MES_PIPES];
+	uint32_t			*ucode_fw_ptr[AMDGPU_MAX_MES_PIPES];
+	uint32_t                        ucode_fw_version[AMDGPU_MAX_MES_PIPES];
+	uint64_t                        uc_start_addr[AMDGPU_MAX_MES_PIPES];
 
 	/* mes ucode data */
-	struct amdgpu_bo		*data_fw_obj;
-	uint64_t			data_fw_gpu_addr;
-	uint32_t			*data_fw_ptr;
-	uint32_t                        data_fw_version;
-	uint64_t                        data_start_addr;
+	struct amdgpu_bo		*data_fw_obj[AMDGPU_MAX_MES_PIPES];
+	uint64_t			data_fw_gpu_addr[AMDGPU_MAX_MES_PIPES];
+	uint32_t			*data_fw_ptr[AMDGPU_MAX_MES_PIPES];
+	uint32_t                        data_fw_version[AMDGPU_MAX_MES_PIPES];
+	uint64_t                        data_start_addr[AMDGPU_MAX_MES_PIPES];
 
 	/* eop gpu obj */
-	struct amdgpu_bo		*eop_gpu_obj;
-	uint64_t                        eop_gpu_addr;
+	struct amdgpu_bo		*eop_gpu_obj[AMDGPU_MAX_MES_PIPES];
+	uint64_t                        eop_gpu_addr[AMDGPU_MAX_MES_PIPES];
 
-	void                            *mqd_backup;
+	void                            *mqd_backup[AMDGPU_MAX_MES_PIPES];
+	struct amdgpu_irq_src	        irq[AMDGPU_MAX_MES_PIPES];
 
 	uint32_t                        vmid_mask_gfxhub;
 	uint32_t                        vmid_mask_mmhub;
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +128 -87

436 lines not on this page · raw message

Signature · 1 line
2.35.1
author maintainer
#21 lore raw

PATCH 21/73 drm/amdgpu/gfx10: add mes queue fence handling

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

From IH ring buffer, look up the coresponding kernel queue and process.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 60 +++++++++++++++++---------
 1 file changed, 40 insertions(+), 20 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +40 -20
index 2ab5259c7305..0e009bd69a9b 100644
@@ -9188,31 +9188,51 @@ static int gfx_v10_0_eop_irq(struct amdgpu_device *adev,
 	int i;
 	u8 me_id, pipe_id, queue_id;
 	struct amdgpu_ring *ring;
+	uint32_t mes_queue_id = entry->src_data[0];
 
 	DRM_DEBUG("IH: CP EOP\n");
-	me_id = (entry->ring_id & 0x0c) >> 2;
-	pipe_id = (entry->ring_id & 0x03) >> 0;
-	queue_id = (entry->ring_id & 0x70) >> 4;
 
-	switch (me_id) {
-	case 0:
-		if (pipe_id == 0)
-			amdgpu_fence_process(&adev->gfx.gfx_ring[0]);
-		else
-			amdgpu_fence_process(&adev->gfx.gfx_ring[1]);
-		break;
-	case 1:
-	case 2:
-		for (i = 0; i < adev->gfx.num_compute_rings; i++) {
-			ring = &adev->gfx.compute_ring[i];
-			/* Per-queue interrupt is supported for MEC starting from VI.
-			  * The interrupt can only be enabled/disabled per pipe instead of per queue.
-			  */
-			if ((ring->me == me_id) && (ring->pipe == pipe_id) && (ring->queue == queue_id))
-				amdgpu_fence_process(ring);
+	if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {
+		struct amdgpu_mes_queue *queue;
+
+		mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;
+
+		spin_lock(&adev->mes.queue_id_lock);
+		queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);
+		if (queue) {
+			DRM_DEBUG("process mes queue id = %d\n", mes_queue_id);
+			amdgpu_fence_process(queue->ring);
+		}
+		spin_unlock(&adev->mes.queue_id_lock);
+	} else {
+		me_id = (entry->ring_id & 0x0c) >> 2;
+		pipe_id = (entry->ring_id & 0x03) >> 0;
+		queue_id = (entry->ring_id & 0x70) >> 4;
+
+		switch (me_id) {
+		case 0:
+			if (pipe_id == 0)
+				amdgpu_fence_process(&adev->gfx.gfx_ring[0]);
+			else
+				amdgpu_fence_process(&adev->gfx.gfx_ring[1]);
+			break;
+		case 1:
+		case 2:
+			for (i = 0; i < adev->gfx.num_compute_rings; i++) {
+				ring = &adev->gfx.compute_ring[i];
+				/* Per-queue interrupt is supported for MEC starting from VI.
+				 * The interrupt can only be enabled/disabled per pipe instead
+				 * of per queue.
+				 */
+				if ((ring->me == me_id) &&
+				    (ring->pipe == pipe_id) &&
+				    (ring->queue == queue_id))
+					amdgpu_fence_process(ring);
+			}
+			break;
 		}
-		break;
 	}
+
 	return 0;
 }
 
Signature · 1 line
2.35.1
author maintainer
#22 lore raw

PATCH 22/73 drm/amdgpu/gfx10: add mes support for gfx ib test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add mes support for gfx ib test.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 46 ++++++++++++++++++--------
 1 file changed, 33 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +33 -13
index 0e009bd69a9b..1208d01cc936 100644
@@ -3818,19 +3818,39 @@ static int gfx_v10_0_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 	struct dma_fence *f = NULL;
 	unsigned index;
 	uint64_t gpu_addr;
-	uint32_t tmp;
+	volatile uint32_t *cpu_ptr;
 	long r;
 
-	r = amdgpu_device_wb_get(adev, &index);
-	if (r)
-		return r;
-
-	gpu_addr = adev->wb.gpu_addr + (index * 4);
-	adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
 	memset(&ib, 0, sizeof(ib));
-	r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);
-	if (r)
-		goto err1;
+
+	if (ring->is_mes_queue) {
+		uint32_t padding, offset;
+
+		offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
+		padding = amdgpu_mes_ctx_get_offs(ring,
+						  AMDGPU_MES_CTX_PADDING_OFFS);
+
+		ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+
+		gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding);
+		cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding);
+		*cpu_ptr = cpu_to_le32(0xCAFEDEAD);
+	} else {
+		r = amdgpu_device_wb_get(adev, &index);
+		if (r)
+			return r;
+
+		gpu_addr = adev->wb.gpu_addr + (index * 4);
+		adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
+		cpu_ptr = &adev->wb.wb[index];
+
+		r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);
+		if (r) {
+			DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
+			goto err1;
+		}
+	}
 
 	ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);
 	ib.ptr[1] = WRITE_DATA_DST_SEL(5) | WR_CONFIRM;
@@ -3851,13 +3871,13 @@ static int gfx_v10_0_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 		goto err2;
 	}
 
-	tmp = adev->wb.wb[index];
-	if (tmp == 0xDEADBEEF)
+	if (le32_to_cpu(*cpu_ptr) == 0xDEADBEEF)
 		r = 0;
 	else
 		r = -EINVAL;
 err2:
-	amdgpu_ib_free(adev, &ib, NULL);
+	if (!ring->is_mes_queue)
+		amdgpu_ib_free(adev, &ib, NULL);
 	dma_fence_put(f);
 err1:
 	amdgpu_device_wb_free(adev, index);
Signature · 1 line
2.35.1
author maintainer
#23 lore raw

PATCH 23/73 drm/amdgpu: don't use kiq to flush gpu tlb if mes enabled

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

If MES is enabled, don't use kiq to flush gpu tlb, for it would result in conflicting with mes fw.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +1 -1
index 9b4a035a5bf1..b8c79789e1e4 100644
@@ -328,7 +328,7 @@ static void gmc_v10_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,
 	/* For SRIOV run time, driver shouldn't access the register through MMIO
 	 * Directly use kiq to do the vm invalidation instead
 	 */
-	if (adev->gfx.kiq.ring.sched.ready &&
+	if (adev->gfx.kiq.ring.sched.ready && !adev->enable_mes &&
 	    (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev)) &&
 	    down_read_trylock(&adev->reset_domain->sem)) {
 		struct amdgpu_vmhub *hub = &adev->vmhub[vmhub];
Signature · 1 line
2.35.1
author maintainer
#24 lore raw

PATCH 24/73 drm/amdgpu/sdma: use per-ctx sdma csa address for mes sdma queue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Use per context sdma csa address for mes sdma queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c +16 -8
index e1835fd4b237..8e221a1ba937 100644
@@ -74,14 +74,22 @@ uint64_t amdgpu_sdma_get_csa_mc_addr(struct amdgpu_ring *ring,
 	if (amdgpu_sriov_vf(adev) || vmid == 0 || !amdgpu_mcbp)
 		return 0;
 
-	r = amdgpu_sdma_get_index_from_ring(ring, &index);
-
-	if (r || index > 31)
-		csa_mc_addr = 0;
-	else
-		csa_mc_addr = amdgpu_csa_vaddr(adev) +
-			AMDGPU_CSA_SDMA_OFFSET +
-			index * AMDGPU_CSA_SDMA_SIZE;
+	if (ring->is_mes_queue) {
+		uint32_t offset = 0;
+
+		offset = offsetof(struct amdgpu_mes_ctx_meta_data,
+				  sdma[ring->idx].sdma_meta_data);
+		csa_mc_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+	} else {
+		r = amdgpu_sdma_get_index_from_ring(ring, &index);
+
+		if (r || index > 31)
+			csa_mc_addr = 0;
+		else
+			csa_mc_addr = amdgpu_csa_vaddr(adev) +
+				AMDGPU_CSA_SDMA_OFFSET +
+				index * AMDGPU_CSA_SDMA_SIZE;
+	}
 
 	return csa_mc_addr;
 }
Signature · 1 line
2.35.1
author maintainer
#25 lore raw

PATCH 25/73 drm/amdgpu/sdma5.2: initialize sdma mqd

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Initialize sdma mqd according to ring settings.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 44 ++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +44 -0
index bf2cf95cbf8f..f67801c5a6c1 100644
@@ -903,6 +903,49 @@ static int sdma_v5_2_start(struct amdgpu_device *adev)
 	return r;
 }
 
+static int sdma_v5_2_mqd_init(struct amdgpu_device *adev, void *mqd,
+			      struct amdgpu_mqd_prop *prop)
+{
+	struct v10_sdma_mqd *m = mqd;
+	uint64_t wb_gpu_addr;
+
+	m->sdmax_rlcx_rb_cntl =
+		order_base_2(prop->queue_size / 4) << SDMA0_RLC0_RB_CNTL__RB_SIZE__SHIFT |
+		1 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_ENABLE__SHIFT |
+		6 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_TIMER__SHIFT |
+		1 << SDMA0_RLC0_RB_CNTL__RB_PRIV__SHIFT;
+
+	m->sdmax_rlcx_rb_base = lower_32_bits(prop->hqd_base_gpu_addr >> 8);
+	m->sdmax_rlcx_rb_base_hi = upper_32_bits(prop->hqd_base_gpu_addr >> 8);
+
+	m->sdmax_rlcx_rb_wptr_poll_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, 0,
+						  mmSDMA0_GFX_RB_WPTR_POLL_CNTL));
+
+	wb_gpu_addr = prop->wptr_gpu_addr;
+	m->sdmax_rlcx_rb_wptr_poll_addr_lo = lower_32_bits(wb_gpu_addr);
+	m->sdmax_rlcx_rb_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr);
+
+	wb_gpu_addr = prop->rptr_gpu_addr;
+	m->sdmax_rlcx_rb_rptr_addr_lo = lower_32_bits(wb_gpu_addr);
+	m->sdmax_rlcx_rb_rptr_addr_hi = upper_32_bits(wb_gpu_addr);
+
+	m->sdmax_rlcx_ib_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, 0,
+							mmSDMA0_GFX_IB_CNTL));
+
+	m->sdmax_rlcx_doorbell_offset =
+		prop->doorbell_index << SDMA0_RLC0_DOORBELL_OFFSET__OFFSET__SHIFT;
+
+	m->sdmax_rlcx_doorbell = REG_SET_FIELD(0, SDMA0_RLC0_DOORBELL, ENABLE, 1);
+
+	return 0;
+}
+
+static void sdma_v5_2_set_mqd_funcs(struct amdgpu_device *adev)
+{
+	adev->mqds[AMDGPU_HW_IP_DMA].mqd_size = sizeof(struct v10_sdma_mqd);
+	adev->mqds[AMDGPU_HW_IP_DMA].init_mqd = sdma_v5_2_mqd_init;
+}
+
 /**
  * sdma_v5_2_ring_test_ring - simple async dma engine test
  *
@@ -1233,6 +1276,7 @@ static int sdma_v5_2_early_init(void *handle)
 	sdma_v5_2_set_buffer_funcs(adev);
 	sdma_v5_2_set_vm_pte_funcs(adev);
 	sdma_v5_2_set_irq_funcs(adev);
+	sdma_v5_2_set_mqd_funcs(adev);
 
 	return 0;
 }
Signature · 1 line
2.35.1
author maintainer
#26 lore raw

PATCH 26/73 drm/amdgpu/sdma5.2: associate mes queue id with fence

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Associate mes queue id with fence, so that EOP trap handler can look up which queue issues the fence.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +4 -2
index f67801c5a6c1..0b7de18df5f4 100644
@@ -460,10 +460,12 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se
 		amdgpu_ring_write(ring, upper_32_bits(seq));
 	}
 
-	if (flags & AMDGPU_FENCE_FLAG_INT) {
+	if ((flags & AMDGPU_FENCE_FLAG_INT)) {
+		uint32_t ctx = ring->is_mes_queue ?
+			(ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
 		/* generate an interrupt */
 		amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
-		amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
+		amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
 	}
 }
 
Signature · 1 line
2.35.1
author maintainer
#27 lore raw

PATCH 27/73 drm/amdgpu/sdma5.2: add mes queue fence handling

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

From IH ring buffer, look up the coresponding kernel queue and process.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +18 -0
index 0b7de18df5f4..9f246ab942f9 100644
@@ -1512,7 +1512,25 @@ static int sdma_v5_2_process_trap_irq(struct amdgpu_device *adev,
 				      struct amdgpu_irq_src *source,
 				      struct amdgpu_iv_entry *entry)
 {
+	uint32_t mes_queue_id = entry->src_data[0];
+
 	DRM_DEBUG("IH: SDMA trap\n");
+
+	if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {
+		struct amdgpu_mes_queue *queue;
+
+		mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;
+
+		spin_lock(&adev->mes.queue_id_lock);
+		queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);
+		if (queue) {
+			DRM_DEBUG("process smda queue id = %d\n", mes_queue_id);
+			amdgpu_fence_process(queue->ring);
+		}
+		spin_unlock(&adev->mes.queue_id_lock);
+		return 0;
+	}
+
 	switch (entry->client_id) {
 	case SOC15_IH_CLIENTID_SDMA0:
 		switch (entry->ring_id) {
Signature · 1 line
2.35.1
author maintainer
#28 lore raw

PATCH 28/73 drm/amdgpu/sdma5.2: add mes support for sdma ring test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES support for sdma ring test.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 37 ++++++++++++++++++--------
 1 file changed, 26 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +26 -11
index 9f246ab942f9..7c9c70382591 100644
@@ -965,18 +965,29 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring)
 	int r;
 	u32 tmp;
 	u64 gpu_addr;
+	volatile uint32_t *cpu_ptr = NULL;
 
-	r = amdgpu_device_wb_get(adev, &index);
-	if (r) {
-		dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
-		return r;
-	}
-
-	gpu_addr = adev->wb.gpu_addr + (index * 4);
 	tmp = 0xCAFEDEAD;
-	adev->wb.wb[index] = cpu_to_le32(tmp);
 
-	r = amdgpu_ring_alloc(ring, 5);
+	if (ring->is_mes_queue) {
+		uint32_t offset = 0;
+		offset = amdgpu_mes_ctx_get_offs(ring,
+					 AMDGPU_MES_CTX_PADDING_OFFS);
+		gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+		*cpu_ptr = tmp;
+	} else {
+		r = amdgpu_device_wb_get(adev, &index);
+		if (r) {
+			dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
+			return r;
+		}
+
+		gpu_addr = adev->wb.gpu_addr + (index * 4);
+		adev->wb.wb[index] = cpu_to_le32(tmp);
+	}
+
+	r = amdgpu_ring_alloc(ring, 20);
 	if (r) {
 		DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
 		amdgpu_device_wb_free(adev, index);
@@ -992,7 +1003,10 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring)
 	amdgpu_ring_commit(ring);
 
 	for (i = 0; i < adev->usec_timeout; i++) {
-		tmp = le32_to_cpu(adev->wb.wb[index]);
+		if (ring->is_mes_queue)
+			tmp = le32_to_cpu(*cpu_ptr);
+		else
+			tmp = le32_to_cpu(adev->wb.wb[index]);
 		if (tmp == 0xDEADBEEF)
 			break;
 		if (amdgpu_emu_mode == 1)
@@ -1004,7 +1018,8 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring)
 	if (i >= adev->usec_timeout)
 		r = -ETIMEDOUT;
 
-	amdgpu_device_wb_free(adev, index);
+	if (!ring->is_mes_queue)
+		amdgpu_device_wb_free(adev, index);
 
 	return r;
 }
Signature · 1 line
2.35.1
author maintainer
#29 lore raw

PATCH 29/73 drm/amdgpu/sdma5.2: add mes support for sdma ib test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES support for sdma ib test.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c | 50 ++++++++++++++++++--------
 1 file changed, 36 insertions(+), 14 deletions(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +36 -14
index 7c9c70382591..83c6ccaaa9e4 100644
@@ -1042,21 +1042,37 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 	long r;
 	u32 tmp = 0;
 	u64 gpu_addr;
+	volatile uint32_t *cpu_ptr = NULL;
 
-	r = amdgpu_device_wb_get(adev, &index);
-	if (r) {
-		dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
-		return r;
-	}
-
-	gpu_addr = adev->wb.gpu_addr + (index * 4);
 	tmp = 0xCAFEDEAD;
-	adev->wb.wb[index] = cpu_to_le32(tmp);
 	memset(&ib, 0, sizeof(ib));
-	r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
-	if (r) {
-		DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
-		goto err0;
+
+	if (ring->is_mes_queue) {
+		uint32_t offset = 0;
+		offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
+		ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+
+		offset = amdgpu_mes_ctx_get_offs(ring,
+					 AMDGPU_MES_CTX_PADDING_OFFS);
+		gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+		*cpu_ptr = tmp;
+	} else {
+		r = amdgpu_device_wb_get(adev, &index);
+		if (r) {
+			dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
+			return r;
+		}
+
+		gpu_addr = adev->wb.gpu_addr + (index * 4);
+		adev->wb.wb[index] = cpu_to_le32(tmp);
+
+		r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
+		if (r) {
+			DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
+			goto err0;
+		}
 	}
 
 	ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1083,7 +1099,12 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 		DRM_ERROR("amdgpu: fence wait failed (%ld).\n", r);
 		goto err1;
 	}
-	tmp = le32_to_cpu(adev->wb.wb[index]);
+
+	if (ring->is_mes_queue)
+		tmp = le32_to_cpu(*cpu_ptr);
+	else
+		tmp = le32_to_cpu(adev->wb.wb[index]);
+
 	if (tmp == 0xDEADBEEF)
 		r = 0;
 	else
@@ -1093,7 +1114,8 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 	amdgpu_ib_free(adev, &ib, NULL);
 	dma_fence_put(f);
 err0:
-	amdgpu_device_wb_free(adev, index);
+	if (!ring->is_mes_queue)
+		amdgpu_device_wb_free(adev, index);
 	return r;
 }
 
Signature · 1 line
2.35.1
author maintainer
#30 lore raw

PATCH 30/73 drm/amdgpu/sdma5: initialize sdma mqd

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Initialize sdma mqd according to ring settings.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 44 ++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +44 -0
index ff359e7f1eb8..30d12c9df911 100644
@@ -959,6 +959,49 @@ static int sdma_v5_0_start(struct amdgpu_device *adev)
 	return r;
 }
 
+static int sdma_v5_0_mqd_init(struct amdgpu_device *adev, void *mqd,
+			      struct amdgpu_mqd_prop *prop)
+{
+	struct v10_sdma_mqd *m = mqd;
+	uint64_t wb_gpu_addr;
+
+	m->sdmax_rlcx_rb_cntl =
+		order_base_2(prop->queue_size / 4) << SDMA0_RLC0_RB_CNTL__RB_SIZE__SHIFT |
+		1 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_ENABLE__SHIFT |
+		6 << SDMA0_RLC0_RB_CNTL__RPTR_WRITEBACK_TIMER__SHIFT |
+		1 << SDMA0_RLC0_RB_CNTL__RB_PRIV__SHIFT;
+
+	m->sdmax_rlcx_rb_base = lower_32_bits(prop->hqd_base_gpu_addr >> 8);
+	m->sdmax_rlcx_rb_base_hi = upper_32_bits(prop->hqd_base_gpu_addr >> 8);
+
+	m->sdmax_rlcx_rb_wptr_poll_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, 0,
+						  mmSDMA0_GFX_RB_WPTR_POLL_CNTL));
+
+	wb_gpu_addr = prop->wptr_gpu_addr;
+	m->sdmax_rlcx_rb_wptr_poll_addr_lo = lower_32_bits(wb_gpu_addr);
+	m->sdmax_rlcx_rb_wptr_poll_addr_hi = upper_32_bits(wb_gpu_addr);
+
+	wb_gpu_addr = prop->rptr_gpu_addr;
+	m->sdmax_rlcx_rb_rptr_addr_lo = lower_32_bits(wb_gpu_addr);
+	m->sdmax_rlcx_rb_rptr_addr_hi = upper_32_bits(wb_gpu_addr);
+
+	m->sdmax_rlcx_ib_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, 0,
+							mmSDMA0_GFX_IB_CNTL));
+
+	m->sdmax_rlcx_doorbell_offset =
+		prop->doorbell_index << SDMA0_RLC0_DOORBELL_OFFSET__OFFSET__SHIFT;
+
+	m->sdmax_rlcx_doorbell = REG_SET_FIELD(0, SDMA0_RLC0_DOORBELL, ENABLE, 1);
+
+	return 0;
+}
+
+static void sdma_v5_0_set_mqd_funcs(struct amdgpu_device *adev)
+{
+	adev->mqds[AMDGPU_HW_IP_DMA].mqd_size = sizeof(struct v10_sdma_mqd);
+	adev->mqds[AMDGPU_HW_IP_DMA].init_mqd = sdma_v5_0_mqd_init;
+}
+
 /**
  * sdma_v5_0_ring_test_ring - simple async dma engine test
  *
@@ -1289,6 +1332,7 @@ static int sdma_v5_0_early_init(void *handle)
 	sdma_v5_0_set_buffer_funcs(adev);
 	sdma_v5_0_set_vm_pte_funcs(adev);
 	sdma_v5_0_set_irq_funcs(adev);
+	sdma_v5_0_set_mqd_funcs(adev);
 
 	return 0;
 }
Signature · 1 line
2.35.1
author maintainer
#31 lore raw

PATCH 31/73 drm/amdgpu/sdma5: associate mes queue id with fence

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Associate mes queue id with fence, so that EOP trap handler can look up which queue issues the fence.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +3 -1
index 30d12c9df911..b73e45597031 100644
@@ -562,9 +562,11 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se
 	}
 
 	if (flags & AMDGPU_FENCE_FLAG_INT) {
+		uint32_t ctx = ring->is_mes_queue ?
+			(ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
 		/* generate an interrupt */
 		amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
-		amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
+		amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
 	}
 }
 
Signature · 1 line
2.35.1
author maintainer
#32 lore raw

PATCH 32/73 drm/amdgpu/sdma5: add mes queue fence handling

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

From IH ring buffer look up the coresponding kernel queue and process.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +18 -0
index b73e45597031..564adc7b010c 100644
@@ -1555,7 +1555,25 @@ static int sdma_v5_0_process_trap_irq(struct amdgpu_device *adev,
 				      struct amdgpu_irq_src *source,
 				      struct amdgpu_iv_entry *entry)
 {
+	uint32_t mes_queue_id = entry->src_data[0];
+
 	DRM_DEBUG("IH: SDMA trap\n");
+
+	if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {
+		struct amdgpu_mes_queue *queue;
+
+		mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;
+
+		spin_lock(&adev->mes.queue_id_lock);
+		queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);
+		if (queue) {
+			DRM_DEBUG("process smda queue id = %d\n", mes_queue_id);
+			amdgpu_fence_process(queue->ring);
+		}
+		spin_unlock(&adev->mes.queue_id_lock);
+		return 0;
+	}
+
 	switch (entry->client_id) {
 	case SOC15_IH_CLIENTID_SDMA0:
 		switch (entry->ring_id) {
Signature · 1 line
2.35.1
author maintainer
#33 lore raw

PATCH 33/73 drm/amdgpu/sdma5: add mes support for sdma ring test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES support for sdma ring test.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 37 ++++++++++++++++++--------
 1 file changed, 26 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +26 -11
index 564adc7b010c..1f0b19e18494 100644
@@ -1021,18 +1021,29 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring)
 	int r;
 	u32 tmp;
 	u64 gpu_addr;
+	volatile uint32_t *cpu_ptr = NULL;
 
-	r = amdgpu_device_wb_get(adev, &index);
-	if (r) {
-		dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
-		return r;
-	}
-
-	gpu_addr = adev->wb.gpu_addr + (index * 4);
 	tmp = 0xCAFEDEAD;
-	adev->wb.wb[index] = cpu_to_le32(tmp);
 
-	r = amdgpu_ring_alloc(ring, 5);
+	if (ring->is_mes_queue) {
+		uint32_t offset = 0;
+		offset = amdgpu_mes_ctx_get_offs(ring,
+					 AMDGPU_MES_CTX_PADDING_OFFS);
+		gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+		*cpu_ptr = tmp;
+	} else {
+		r = amdgpu_device_wb_get(adev, &index);
+		if (r) {
+			dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
+			return r;
+		}
+
+		gpu_addr = adev->wb.gpu_addr + (index * 4);
+		adev->wb.wb[index] = cpu_to_le32(tmp);
+	}
+
+	r = amdgpu_ring_alloc(ring, 20);
 	if (r) {
 		DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
 		amdgpu_device_wb_free(adev, index);
@@ -1048,7 +1059,10 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring)
 	amdgpu_ring_commit(ring);
 
 	for (i = 0; i < adev->usec_timeout; i++) {
-		tmp = le32_to_cpu(adev->wb.wb[index]);
+		if (ring->is_mes_queue)
+			tmp = le32_to_cpu(*cpu_ptr);
+		else
+			tmp = le32_to_cpu(adev->wb.wb[index]);
 		if (tmp == 0xDEADBEEF)
 			break;
 		if (amdgpu_emu_mode == 1)
@@ -1060,7 +1074,8 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring)
 	if (i >= adev->usec_timeout)
 		r = -ETIMEDOUT;
 
-	amdgpu_device_wb_free(adev, index);
+	if (!ring->is_mes_queue)
+		amdgpu_device_wb_free(adev, index);
 
 	return r;
 }
Signature · 1 line
2.35.1
author maintainer
#34 lore raw

PATCH 34/73 drm/amdgpu/sdma5: add mes support for sdma ib test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES support for sdma ib test.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c | 50 ++++++++++++++++++--------
 1 file changed, 36 insertions(+), 14 deletions(-)
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +36 -14
index 1f0b19e18494..1f9021f896a1 100644
@@ -1098,22 +1098,38 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 	long r;
 	u32 tmp = 0;
 	u64 gpu_addr;
+	volatile uint32_t *cpu_ptr = NULL;
 
-	r = amdgpu_device_wb_get(adev, &index);
-	if (r) {
-		dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
-		return r;
-	}
-
-	gpu_addr = adev->wb.gpu_addr + (index * 4);
 	tmp = 0xCAFEDEAD;
-	adev->wb.wb[index] = cpu_to_le32(tmp);
 	memset(&ib, 0, sizeof(ib));
-	r = amdgpu_ib_get(adev, NULL, 256,
+
+	if (ring->is_mes_queue) {
+		uint32_t offset = 0;
+		offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
+		ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+
+		offset = amdgpu_mes_ctx_get_offs(ring,
+					 AMDGPU_MES_CTX_PADDING_OFFS);
+		gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+		cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
+		*cpu_ptr = tmp;
+	} else {
+		r = amdgpu_device_wb_get(adev, &index);
+		if (r) {
+			dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
+			return r;
+		}
+
+		gpu_addr = adev->wb.gpu_addr + (index * 4);
+		adev->wb.wb[index] = cpu_to_le32(tmp);
+
+		r = amdgpu_ib_get(adev, NULL, 256,
 					AMDGPU_IB_POOL_DIRECT, &ib);
-	if (r) {
-		DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
-		goto err0;
+		if (r) {
+			DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
+			goto err0;
+		}
 	}
 
 	ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1140,7 +1156,12 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 		DRM_ERROR("amdgpu: fence wait failed (%ld).\n", r);
 		goto err1;
 	}
-	tmp = le32_to_cpu(adev->wb.wb[index]);
+
+	if (ring->is_mes_queue)
+		tmp = le32_to_cpu(*cpu_ptr);
+	else
+		tmp = le32_to_cpu(adev->wb.wb[index]);
+
 	if (tmp == 0xDEADBEEF)
 		r = 0;
 	else
@@ -1150,7 +1171,8 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout)
 	amdgpu_ib_free(adev, &ib, NULL);
 	dma_fence_put(f);
 err0:
-	amdgpu_device_wb_free(adev, index);
+	if (!ring->is_mes_queue)
+		amdgpu_device_wb_free(adev, index);
 	return r;
 }
 
Signature · 1 line
2.35.1
author maintainer
#35 lore raw

PATCH 35/73 drm/amdgpu: add mes kiq PSP GFX FW type

In reply to Alex Deucher

From: Likun Gao <[email protected]>

Add MES KIQ PSP GFX FW type and the convert type.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +6 -0
index aabb208bebde..d0fb14ef645c 100644
@@ -2190,6 +2190,12 @@ static int psp_get_fw_type(struct amdgpu_firmware_info *ucode,
 	case AMDGPU_UCODE_ID_CP_MES_DATA:
 		*type = GFX_FW_TYPE_MES_STACK;
 		break;
+	case AMDGPU_UCODE_ID_CP_MES1:
+		*type = GFX_FW_TYPE_CP_MES_KIQ;
+		break;
+	case AMDGPU_UCODE_ID_CP_MES1_DATA:
+		*type = GFX_FW_TYPE_MES_KIQ_STACK;
+		break;
 	case AMDGPU_UCODE_ID_CP_CE:
 		*type = GFX_FW_TYPE_CP_CE;
 		break;
Signature · 1 line
2.35.1
author maintainer
#36 lore raw

PATCH 36/73 drm/amdgpu/mes: add mes kiq callback

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Needed to properly initialize mes kiq.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +5 -0
index 91b020842eb0..117c95acfd48 100644
@@ -109,6 +109,9 @@ struct amdgpu_mes {
 	uint64_t			query_status_fence_gpu_addr;
 	uint64_t			*query_status_fence_ptr;
 
+	/* initialize kiq pipe */
+	int                             (*kiq_hw_init)(struct amdgpu_device *adev);
+
 	/* ip specific functions */
 	const struct amdgpu_mes_funcs   *funcs;
 };
@@ -204,4 +207,6 @@ struct amdgpu_mes_funcs {
 			   struct mes_resume_gang_input *input);
 };
 
+#define amdgpu_mes_kiq_hw_init(adev) (adev)->mes.kiq_hw_init((adev))
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#37 lore raw

PATCH 37/73 drm/amdgpu: add mes kiq frontdoor loading support

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add mes kiq frontdoor loading support.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +10 -0
index b7d575c7bcdc..a67f41465337 100644
@@ -711,6 +711,16 @@ static int amdgpu_ucode_init_single_fw(struct amdgpu_device *adev,
 			ucode_addr = (u8 *)ucode->fw->data +
 				le32_to_cpu(mes_hdr->mes_ucode_data_offset_bytes);
 			break;
+		case AMDGPU_UCODE_ID_CP_MES1:
+			ucode->ucode_size = le32_to_cpu(mes_hdr->mes_ucode_size_bytes);
+			ucode_addr = (u8 *)ucode->fw->data +
+				le32_to_cpu(mes_hdr->mes_ucode_offset_bytes);
+			break;
+		case AMDGPU_UCODE_ID_CP_MES1_DATA:
+			ucode->ucode_size = le32_to_cpu(mes_hdr->mes_ucode_data_size_bytes);
+			ucode_addr = (u8 *)ucode->fw->data +
+				le32_to_cpu(mes_hdr->mes_ucode_data_offset_bytes);
+			break;
 		case AMDGPU_UCODE_ID_DMCU_ERAM:
 			ucode->ucode_size = le32_to_cpu(header->ucode_size_bytes) -
 				le32_to_cpu(dmcu_hdr->intv_size_bytes);
Signature · 1 line
2.35.1
author maintainer
#38 lore raw

PATCH 38/73 drm/amdgpu: enable mes kiq N-1 test on sienna cichlid

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Enable kiq support on gfx10.3, enable mes kiq (n-1) test on sienna cichlid, so that mes kiq can be tested on sienna cichlid. The patch can be dropped once mes kiq is functional.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c |  32 ++--
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 202 ++++++++++++++++++++-----
 2 files changed, 184 insertions(+), 50 deletions(-)
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +20 -12
index 1208d01cc936..9042e0b480ce 100644
@@ -4897,16 +4897,18 @@ static int gfx_v10_0_sw_init(void *handle)
 		}
 	}
 
-	r = amdgpu_gfx_kiq_init(adev, GFX10_MEC_HPD_SIZE);
-	if (r) {
-		DRM_ERROR("Failed to init KIQ BOs!\n");
-		return r;
-	}
+	if (!adev->enable_mes_kiq) {
+		r = amdgpu_gfx_kiq_init(adev, GFX10_MEC_HPD_SIZE);
+		if (r) {
+			DRM_ERROR("Failed to init KIQ BOs!\n");
+			return r;
+		}
 
-	kiq = &adev->gfx.kiq;
-	r = amdgpu_gfx_kiq_init_ring(adev, &kiq->ring, &kiq->irq);
-	if (r)
-		return r;
+		kiq = &adev->gfx.kiq;
+		r = amdgpu_gfx_kiq_init_ring(adev, &kiq->ring, &kiq->irq);
+		if (r)
+			return r;
+	}
 
 	r = amdgpu_gfx_mqd_sw_init(adev, sizeof(struct v10_compute_mqd));
 	if (r)
@@ -4958,8 +4960,11 @@ static int gfx_v10_0_sw_fini(void *handle)
 		amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
 
 	amdgpu_gfx_mqd_sw_fini(adev);
-	amdgpu_gfx_kiq_free_ring(&adev->gfx.kiq.ring);
-	amdgpu_gfx_kiq_fini(adev);
+
+	if (!adev->enable_mes_kiq) {
+		amdgpu_gfx_kiq_free_ring(&adev->gfx.kiq.ring);
+		amdgpu_gfx_kiq_fini(adev);
+	}
 
 	gfx_v10_0_pfp_fini(adev);
 	gfx_v10_0_ce_fini(adev);
@@ -7213,7 +7218,10 @@ static int gfx_v10_0_cp_resume(struct amdgpu_device *adev)
 			return r;
 	}
 
-	r = gfx_v10_0_kiq_resume(adev);
+	if (adev->enable_mes_kiq && adev->mes.kiq_hw_init)
+		r = amdgpu_mes_kiq_hw_init(adev);
+	else
+		r = gfx_v10_0_kiq_resume(adev);
 	if (r)
 		return r;
 
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +164 -38

337 lines not on this page · raw message

Signature · 1 line
2.35.1
author maintainer
#39 lore raw

PATCH 39/73 drm/amdgpu/mes: manage mes doorbell allocation

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

It is used to manage the doorbell allocation of mes processes and queues. Driver calls into process doorbell allocation to get the slice doorbell for the process, then the doorbell for a queue is allocated from the process doorbell slice.

---
 drivers/gpu/drm/amd/amdgpu/Makefile     |   1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 133 ++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
drivers/gpu/drm/amd/amdgpu/Makefile +1 -0
index 7a1b13fabebb..803e7f5dc458 100644
@@ -144,6 +144,7 @@ amdgpu-y += \
 
 # add MES block
 amdgpu-y += \
+	amdgpu_mes.o \
 	mes_v10_1.o
 
 # add UVD block
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c new file +133 -0

133 lines not on this page · raw message

Signature · 1 line
2.35.1
author maintainer
#40 lore raw

PATCH 40/73 drm/amdgpu: add mes queue id mask v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES queue id mask.

v2: move queue id mask to amdgpu_mes_ctx.h

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes_ctx.h +1 -0
index 544f1aa86edf..c000f656aae5 100644
@@ -116,5 +116,6 @@ struct amdgpu_mes_ctx_data {
 #define AMDGPU_FENCE_MES_QUEUE_ID_MASK  (AMDGPU_FENCE_MES_QUEUE_FLAG - 1)
 
 #define AMDGPU_FENCE_MES_QUEUE_FLAG     0x1000000u
+#define AMDGPU_FENCE_MES_QUEUE_ID_MASK  (AMDGPU_FENCE_MES_QUEUE_FLAG - 1)
 
 #endif
Signature · 1 line
2.35.1
author maintainer
#41 lore raw

PATCH 41/73 drm/amdgpu/mes: initialize/finalize common mes structure v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Initialize/finalize common mes structure.

v2: add mutex_init for adev->mes.mutex

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 72 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  4 ++
 2 files changed, 76 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +72 -0
index 1c591cb45fd9..90c400564540 100644
@@ -131,3 +131,75 @@ static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev)
 	DRM_INFO("max_doorbell_slices=%ld\n", doorbell_process_limit);
 	return 0;
 }
+
+int amdgpu_mes_init(struct amdgpu_device *adev)
+{
+	int i, r;
+
+	adev->mes.adev = adev;
+
+	idr_init(&adev->mes.pasid_idr);
+	idr_init(&adev->mes.gang_id_idr);
+	idr_init(&adev->mes.queue_id_idr);
+	ida_init(&adev->mes.doorbell_ida);
+	spin_lock_init(&adev->mes.queue_id_lock);
+	mutex_init(&adev->mes.mutex);
+
+	adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;
+	adev->mes.vmid_mask_mmhub = 0xffffff00;
+	adev->mes.vmid_mask_gfxhub = 0xffffff00;
+
+	for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {
+		/* use only 1st MEC pipes */
+		if (i >= 4)
+			continue;
+		adev->mes.compute_hqd_mask[i] = 0xc;
+	}
+
+	for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)
+		adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffffffe;
+
+	for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++)
+		adev->mes.sdma_hqd_mask[i] = i ? 0 : 0x3fc;
+
+	for (i = 0; i < AMDGPU_MES_PRIORITY_NUM_LEVELS; i++)
+		adev->mes.agreegated_doorbells[i] = 0xffffffff;
+
+	r = amdgpu_device_wb_get(adev, &adev->mes.sch_ctx_offs);
+	if (r) {
+		dev_err(adev->dev,
+			"(%d) ring trail_fence_offs wb alloc failed\n", r);
+		goto error_ids;
+	}
+	adev->mes.sch_ctx_gpu_addr =
+		adev->wb.gpu_addr + (adev->mes.sch_ctx_offs * 4);
+	adev->mes.sch_ctx_ptr =
+		(uint64_t *)&adev->wb.wb[adev->mes.sch_ctx_offs];
+
+	r = amdgpu_mes_doorbell_init(adev);
+	if (r)
+		goto error;
+
+	return 0;
+
+error:
+	amdgpu_device_wb_free(adev, adev->mes.sch_ctx_offs);
+error_ids:
+	idr_destroy(&adev->mes.pasid_idr);
+	idr_destroy(&adev->mes.gang_id_idr);
+	idr_destroy(&adev->mes.queue_id_idr);
+	ida_destroy(&adev->mes.doorbell_ida);
+	mutex_destroy(&adev->mes.mutex);
+	return r;
+}
+
+void amdgpu_mes_fini(struct amdgpu_device *adev)
+{
+	amdgpu_device_wb_free(adev, adev->mes.sch_ctx_offs);
+
+	idr_destroy(&adev->mes.pasid_idr);
+	idr_destroy(&adev->mes.gang_id_idr);
+	idr_destroy(&adev->mes.queue_id_idr);
+	ida_destroy(&adev->mes.doorbell_ida);
+	mutex_destroy(&adev->mes.mutex);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +4 -0
index 117c95acfd48..e64b2114c7ba 100644
@@ -207,6 +207,10 @@ struct amdgpu_mes_funcs {
 			   struct mes_resume_gang_input *input);
 };
 
+
 #define amdgpu_mes_kiq_hw_init(adev) (adev)->mes.kiq_hw_init((adev))
 
+int amdgpu_mes_init(struct amdgpu_device *adev);
+void amdgpu_mes_fini(struct amdgpu_device *adev);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#42 lore raw

PATCH 42/73 drm/amdgpu/mes: relocate status_fence slot allocation

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Move the status_fence slot allocation from ip specific function to general mes function.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 11 +++++++++
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c  | 33 -------------------------
 2 files changed, 11 insertions(+), 33 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +11 -0
index 90c400564540..a988c232b4a3 100644
@@ -176,6 +176,17 @@ int amdgpu_mes_init(struct amdgpu_device *adev)
 	adev->mes.sch_ctx_ptr =
 		(uint64_t *)&adev->wb.wb[adev->mes.sch_ctx_offs];
 
+	r = amdgpu_device_wb_get(adev, &adev->mes.query_status_fence_offs);
+	if (r) {
+		dev_err(adev->dev,
+			"(%d) query_status_fence_offs wb alloc failed\n", r);
+		return r;
+	}
+	adev->mes.query_status_fence_gpu_addr =
+		adev->wb.gpu_addr + (adev->mes.query_status_fence_offs * 4);
+	adev->mes.query_status_fence_ptr =
+		(uint64_t *)&adev->wb.wb[adev->mes.query_status_fence_offs];
+
 	r = amdgpu_mes_doorbell_init(adev);
 	if (r)
 		goto error;
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +0 -33
index fecf3f26bf7c..d77242e0360e 100644
@@ -606,35 +606,6 @@ static int mes_v10_1_allocate_eop_buf(struct amdgpu_device *adev,
 	return 0;
 }
 
-static int mes_v10_1_allocate_mem_slots(struct amdgpu_device *adev)
-{
-	int r;
-
-	r = amdgpu_device_wb_get(adev, &adev->mes.sch_ctx_offs);
-	if (r) {
-		dev_err(adev->dev,
-			"(%d) mes sch_ctx_offs wb alloc failed\n", r);
-		return r;
-	}
-	adev->mes.sch_ctx_gpu_addr =
-		adev->wb.gpu_addr + (adev->mes.sch_ctx_offs * 4);
-	adev->mes.sch_ctx_ptr =
-		(uint64_t *)&adev->wb.wb[adev->mes.sch_ctx_offs];
-
-	r = amdgpu_device_wb_get(adev, &adev->mes.query_status_fence_offs);
-	if (r) {
-		dev_err(adev->dev,
-			"(%d) query_status_fence_offs wb alloc failed\n", r);
-		return r;
-	}
-	adev->mes.query_status_fence_gpu_addr =
-		adev->wb.gpu_addr + (adev->mes.query_status_fence_offs * 4);
-	adev->mes.query_status_fence_ptr =
-		(uint64_t *)&adev->wb.wb[adev->mes.query_status_fence_offs];
-
-	return 0;
-}
-
 static int mes_v10_1_mqd_init(struct amdgpu_ring *ring)
 {
 	struct amdgpu_device *adev = ring->adev;
@@ -991,10 +962,6 @@ static int mes_v10_1_sw_init(void *handle)
 	if (r)
 		return r;
 
-	r = mes_v10_1_allocate_mem_slots(adev);
-	if (r)
-		return r;
-
 	return 0;
 }
 
Signature · 1 line
2.35.1
author maintainer
#43 lore raw

PATCH 43/73 drm/amdgpu/mes10.1: call general mes initialization

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Call general mes initialization/finalization.

---
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +5 -0
index d77242e0360e..94812164998a 100644
@@ -935,6 +935,10 @@ static int mes_v10_1_sw_init(void *handle)
 	adev->mes.funcs = &mes_v10_1_funcs;
 	adev->mes.kiq_hw_init = &mes_v10_1_kiq_hw_init;
 
+	r = amdgpu_mes_init(adev);
+	if (r)
+		return r;
+
 	for (pipe = 0; pipe < AMDGPU_MAX_MES_PIPES; pipe++) {
 		if (!adev->enable_mes_kiq && pipe == AMDGPU_MES_KIQ_PIPE)
 			continue;
@@ -994,6 +998,7 @@ static int mes_v10_1_sw_fini(void *handle)
 	amdgpu_ring_fini(&adev->gfx.kiq.ring);
 	amdgpu_ring_fini(&adev->mes.ring);
 
+	amdgpu_mes_fini(adev);
 	return 0;
 }
 
Signature · 1 line
2.35.1
author maintainer
#44 lore raw

PATCH 44/73 drm/amdgpu/mes10.1: add delay after mes engine enable

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add delay after mes engine enable, for it needs more time to complete engine initialising.

---
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +1 -0
index 94812164998a..d4e64c5a3215 100644
@@ -473,6 +473,7 @@ static void mes_v10_1_enable(struct amdgpu_device *adev, bool enable)
 		data = REG_SET_FIELD(data, CP_MES_CNTL, MES_PIPE1_ACTIVE,
 				     adev->enable_mes_kiq ? 1 : 0);
 		WREG32_SOC15(GC, 0, mmCP_MES_CNTL, data);
+		udelay(50);
 	} else {
 		data = RREG32_SOC15(GC, 0, mmCP_MES_CNTL);
 		data = REG_SET_FIELD(data, CP_MES_CNTL, MES_PIPE0_ACTIVE, 0);
Signature · 1 line
2.35.1
author maintainer
#45 lore raw

PATCH 45/73 drm/amdgpu/mes10.1: implement the suspend/resume routine

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Implement the suspend/resume routine of mes.

---
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +16 -2
index d4e64c5a3215..d468cb5a8854 100644
@@ -1120,12 +1120,26 @@ static int mes_v10_1_hw_fini(void *handle)
 
 static int mes_v10_1_suspend(void *handle)
 {
-	return 0;
+	int r;
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+	r = amdgpu_mes_suspend(adev);
+	if (r)
+		return r;
+
+	return mes_v10_1_hw_fini(adev);
 }
 
 static int mes_v10_1_resume(void *handle)
 {
-	return 0;
+	int r;
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+	r = mes_v10_1_hw_init(adev);
+	if (r)
+		return r;
+
+	return amdgpu_mes_resume(adev);
 }
 
 static const struct amd_ip_funcs mes_v10_1_ip_funcs = {
Signature · 1 line
2.35.1
author maintainer
#46 lore raw

PATCH 46/73 drm/amdgpu/mes: implement creating mes process v2

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Create a mes process which contains process-related resources, like vm, doorbell bitmap, process ctx bo and etc.

v2: move the simple variable to the end

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 77 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  3 +
 2 files changed, 80 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +77 -0
index a988c232b4a3..55005a594be1 100644
@@ -214,3 +214,80 @@ void amdgpu_mes_fini(struct amdgpu_device *adev)
 	ida_destroy(&adev->mes.doorbell_ida);
 	mutex_destroy(&adev->mes.mutex);
 }
+
+int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
+			      struct amdgpu_vm *vm)
+{
+	struct amdgpu_mes_process *process;
+	int r;
+
+	mutex_lock(&adev->mes.mutex);
+
+	/* allocate the mes process buffer */
+	process = kzalloc(sizeof(struct amdgpu_mes_process), GFP_KERNEL);
+	if (!process) {
+		DRM_ERROR("no more memory to create mes process\n");
+		mutex_unlock(&adev->mes.mutex);
+		return -ENOMEM;
+	}
+
+	process->doorbell_bitmap =
+		kzalloc(DIV_ROUND_UP(AMDGPU_MES_MAX_NUM_OF_QUEUES_PER_PROCESS,
+				     BITS_PER_BYTE), GFP_KERNEL);
+	if (!process->doorbell_bitmap) {
+		DRM_ERROR("failed to allocate doorbell bitmap\n");
+		kfree(process);
+		mutex_unlock(&adev->mes.mutex);
+		return -ENOMEM;
+	}
+
+	/* add the mes process to idr list */
+	r = idr_alloc(&adev->mes.pasid_idr, process, pasid, pasid + 1,
+		      GFP_KERNEL);
+	if (r < 0) {
+		DRM_ERROR("failed to lock pasid=%d\n", pasid);
+		goto clean_up_memory;
+	}
+
+	/* allocate the process context bo and map it */
+	r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_PROC_CTX_SIZE, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_GTT,
+				    &process->proc_ctx_bo,
+				    &process->proc_ctx_gpu_addr,
+				    &process->proc_ctx_cpu_ptr);
+	if (r) {
+		DRM_ERROR("failed to allocate process context bo\n");
+		goto clean_up_pasid;
+	}
+	memset(process->proc_ctx_cpu_ptr, 0, AMDGPU_MES_PROC_CTX_SIZE);
+
+	/* allocate the starting doorbell index of the process */
+	r = amdgpu_mes_alloc_process_doorbells(adev, process);
+	if (r < 0) {
+		DRM_ERROR("failed to allocate doorbell for process\n");
+		goto clean_up_ctx;
+	}
+
+	DRM_DEBUG("process doorbell index = %d\n", process->doorbell_index);
+
+	INIT_LIST_HEAD(&process->gang_list);
+	process->vm = vm;
+	process->pasid = pasid;
+	process->process_quantum = adev->mes.default_process_quantum;
+	process->pd_gpu_addr = amdgpu_bo_gpu_offset(vm->root.bo);
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+
+clean_up_ctx:
+	amdgpu_bo_free_kernel(&process->proc_ctx_bo,
+			      &process->proc_ctx_gpu_addr,
+			      &process->proc_ctx_cpu_ptr);
+clean_up_pasid:
+	idr_remove(&adev->mes.pasid_idr, pasid);
+clean_up_memory:
+	kfree(process->doorbell_bitmap);
+	kfree(process);
+	mutex_unlock(&adev->mes.mutex);
+	return r;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3 -0
index e64b2114c7ba..010a9727cbec 100644
@@ -213,4 +213,7 @@ struct amdgpu_mes_funcs {
 int amdgpu_mes_init(struct amdgpu_device *adev);
 void amdgpu_mes_fini(struct amdgpu_device *adev);
 
+int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
+			      struct amdgpu_vm *vm);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#47 lore raw

PATCH 47/73 drm/amdgpu/mes: implement destroying mes process

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Destroy the mes process, which free resources of the process.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 58 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  1 +
 2 files changed, 59 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +58 -0
index 55005a594be1..05e27636ce20 100644
@@ -291,3 +291,61 @@ int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
 	mutex_unlock(&adev->mes.mutex);
 	return r;
 }
+
+void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid)
+{
+	struct amdgpu_mes_process *process;
+	struct amdgpu_mes_gang *gang, *tmp1;
+	struct amdgpu_mes_queue *queue, *tmp2;
+	struct mes_remove_queue_input queue_input;
+	unsigned long flags;
+	int r;
+
+	mutex_lock(&adev->mes.mutex);
+
+	process = idr_find(&adev->mes.pasid_idr, pasid);
+	if (!process) {
+		DRM_WARN("pasid %d doesn't exist\n", pasid);
+		mutex_unlock(&adev->mes.mutex);
+		return;
+	}
+
+	/* free all gangs in the process */
+	list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) {
+		/* free all queues in the gang */
+		list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) {
+			spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
+			idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
+			spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+
+			queue_input.doorbell_offset = queue->doorbell_off;
+			queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
+
+			r = adev->mes.funcs->remove_hw_queue(&adev->mes,
+							     &queue_input);
+			if (r)
+				DRM_WARN("failed to remove hardware queue\n");
+
+			list_del(&queue->list);
+			kfree(queue);
+		}
+
+		idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
+		amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
+				      &gang->gang_ctx_gpu_addr,
+				      &gang->gang_ctx_cpu_ptr);
+		list_del(&gang->list);
+		kfree(gang);
+	}
+
+	amdgpu_mes_free_process_doorbells(adev, process);
+
+	idr_remove(&adev->mes.pasid_idr, pasid);
+	amdgpu_bo_free_kernel(&process->proc_ctx_bo,
+			      &process->proc_ctx_gpu_addr,
+			      &process->proc_ctx_cpu_ptr);
+	kfree(process->doorbell_bitmap);
+	kfree(process);
+
+	mutex_unlock(&adev->mes.mutex);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +1 -0
index 010a9727cbec..fa2f47e4cd5a 100644
@@ -215,5 +215,6 @@ void amdgpu_mes_fini(struct amdgpu_device *adev);
 
 int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
 			      struct amdgpu_vm *vm);
+void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid);
 
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#48 lore raw

PATCH 48/73 drm/amdgpu/mes: implement adding mes gang

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Gang is a group of the same type queue, which is the scheduling unit of mes hardware scheduler.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 67 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 12 +++++
 2 files changed, 79 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +67 -0
index 05e27636ce20..74385e4b45c4 100644
@@ -349,3 +349,70 @@ void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid)
 
 	mutex_unlock(&adev->mes.mutex);
 }
+
+int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
+			struct amdgpu_mes_gang_properties *gprops,
+			int *gang_id)
+{
+	struct amdgpu_mes_process *process;
+	struct amdgpu_mes_gang *gang;
+	int r;
+
+	mutex_lock(&adev->mes.mutex);
+
+	process = idr_find(&adev->mes.pasid_idr, pasid);
+	if (!process) {
+		DRM_ERROR("pasid %d doesn't exist\n", pasid);
+		mutex_unlock(&adev->mes.mutex);
+		return -EINVAL;
+	}
+
+	/* allocate the mes gang buffer */
+	gang = kzalloc(sizeof(struct amdgpu_mes_gang), GFP_KERNEL);
+	if (!gang) {
+		mutex_unlock(&adev->mes.mutex);
+		return -ENOMEM;
+	}
+
+	/* add the mes gang to idr list */
+	r = idr_alloc(&adev->mes.gang_id_idr, gang, 1, 0,
+		      GFP_KERNEL);
+	if (r < 0) {
+		kfree(gang);
+		mutex_unlock(&adev->mes.mutex);
+		return r;
+	}
+
+	gang->gang_id = r;
+	*gang_id = r;
+
+	/* allocate the gang context bo and map it to cpu space */
+	r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_GANG_CTX_SIZE, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_GTT,
+				    &gang->gang_ctx_bo,
+				    &gang->gang_ctx_gpu_addr,
+				    &gang->gang_ctx_cpu_ptr);
+	if (r) {
+		DRM_ERROR("failed to allocate process context bo\n");
+		goto clean_up;
+	}
+	memset(gang->gang_ctx_cpu_ptr, 0, AMDGPU_MES_GANG_CTX_SIZE);
+
+	INIT_LIST_HEAD(&gang->queue_list);
+	gang->process = process;
+	gang->priority = gprops->priority;
+	gang->gang_quantum = gprops->gang_quantum ?
+		gprops->gang_quantum : adev->mes.default_gang_quantum;
+	gang->global_priority_level = gprops->global_priority_level;
+	gang->inprocess_gang_priority = gprops->inprocess_gang_priority;
+	list_add_tail(&gang->list, &process->gang_list);
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+
+clean_up:
+	idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
+	kfree(gang);
+	mutex_unlock(&adev->mes.mutex);
+	return r;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +12 -0
index fa2f47e4cd5a..3109bd1db6bc 100644
@@ -158,6 +158,14 @@ struct amdgpu_mes_queue {
 	struct amdgpu_ring 		*ring;
 };
 
+struct amdgpu_mes_gang_properties {
+	uint32_t 	priority;
+	uint32_t 	gang_quantum;
+	uint32_t 	inprocess_gang_priority;
+	uint32_t 	priority_level;
+	int 		global_priority_level;
+};
+
 struct mes_add_queue_input {
 	uint32_t	process_id;
 	uint64_t	page_table_base_addr;
@@ -217,4 +225,8 @@ int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
 			      struct amdgpu_vm *vm);
 void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid);
 
+int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
+			struct amdgpu_mes_gang_properties *gprops,
+			int *gang_id);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#49 lore raw

PATCH 49/73 drm/amdgpu/mes: implement removing mes gang

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Free the mes gang and its resources.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 30 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  1 +
 2 files changed, 31 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +30 -0
index 74385e4b45c4..07ddf7bf6a3b 100644
@@ -416,3 +416,33 @@ int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
 	mutex_unlock(&adev->mes.mutex);
 	return r;
 }
+
+int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id)
+{
+	struct amdgpu_mes_gang *gang;
+
+	mutex_lock(&adev->mes.mutex);
+
+	gang = idr_find(&adev->mes.gang_id_idr, gang_id);
+	if (!gang) {
+		DRM_ERROR("gang id %d doesn't exist\n", gang_id);
+		mutex_unlock(&adev->mes.mutex);
+		return -EINVAL;
+	}
+
+	if (!list_empty(&gang->queue_list)) {
+		DRM_ERROR("queue list is not empty\n");
+		mutex_unlock(&adev->mes.mutex);
+		return -EBUSY;
+	}
+
+	idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
+	amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
+			      &gang->gang_ctx_gpu_addr,
+			      &gang->gang_ctx_cpu_ptr);
+	list_del(&gang->list);
+	kfree(gang);
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +1 -0
index 3109bd1db6bc..f401a0a3eebd 100644
@@ -228,5 +228,6 @@ void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid);
 int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
 			struct amdgpu_mes_gang_properties *gprops,
 			int *gang_id);
+int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
 
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#50 lore raw

PATCH 50/73 drm/amdgpu/mes: implement suspending all gangs

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Implement suspending all gangs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 25 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  2 ++
 2 files changed, 27 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +25 -0
index 07ddf7bf6a3b..e64f2a4b5a3b 100644
@@ -446,3 +446,28 @@ int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id)
 	mutex_unlock(&adev->mes.mutex);
 	return 0;
 }
+
+int amdgpu_mes_suspend(struct amdgpu_device *adev)
+{
+	struct idr *idp;
+	struct amdgpu_mes_process *process;
+	struct amdgpu_mes_gang *gang;
+	struct mes_suspend_gang_input input;
+	int r, pasid;
+
+	mutex_lock(&adev->mes.mutex);
+
+	idp = &adev->mes.pasid_idr;
+
+	idr_for_each_entry(idp, process, pasid) {
+		list_for_each_entry(gang, &process->gang_list, list) {
+			r = adev->mes.funcs->suspend_gang(&adev->mes, &input);
+			if (r)
+				DRM_ERROR("failed to suspend pasid %d gangid %d",
+					 pasid, gang->gang_id);
+		}
+	}
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +2 -0
index f401a0a3eebd..667fc9f9b21b 100644
@@ -230,4 +230,6 @@ int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
 			int *gang_id);
 int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
 
+int amdgpu_mes_suspend(struct amdgpu_device *adev);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#51 lore raw

PATCH 51/73 drm/amdgpu/mes: implement resuming all gangs

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Implement resuming all gangs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 25 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  1 +
 2 files changed, 26 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +25 -0
index e64f2a4b5a3b..b58af81f04a3 100644
@@ -471,3 +471,28 @@ int amdgpu_mes_suspend(struct amdgpu_device *adev)
 	mutex_unlock(&adev->mes.mutex);
 	return 0;
 }
+
+int amdgpu_mes_resume(struct amdgpu_device *adev)
+{
+	struct idr *idp;
+	struct amdgpu_mes_process *process;
+	struct amdgpu_mes_gang *gang;
+	struct mes_resume_gang_input input;
+	int r, pasid;
+
+	mutex_lock(&adev->mes.mutex);
+
+	idp = &adev->mes.pasid_idr;
+
+	idr_for_each_entry(idp, process, pasid) {
+		list_for_each_entry(gang, &process->gang_list, list) {
+			r = adev->mes.funcs->resume_gang(&adev->mes, &input);
+			if (r)
+				DRM_ERROR("failed to resume pasid %d gangid %d",
+					 pasid, gang->gang_id);
+		}
+	}
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +1 -0
index 667fc9f9b21b..43d3a689732a 100644
@@ -231,5 +231,6 @@ int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
 int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
 
 int amdgpu_mes_suspend(struct amdgpu_device *adev);
+int amdgpu_mes_resume(struct amdgpu_device *adev);
 
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#52 lore raw

PATCH 52/73 drm/amdgpu/mes: initialize mqd from queue properties

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add helper function to initialize mqd from queue properties.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 54 +++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +54 -0
index b58af81f04a3..2cd2fa76b5c8 100644
@@ -496,3 +496,57 @@ int amdgpu_mes_resume(struct amdgpu_device *adev)
 	mutex_unlock(&adev->mes.mutex);
 	return 0;
 }
+
+static int amdgpu_mes_queue_init_mqd(struct amdgpu_device *adev,
+				     struct amdgpu_mes_queue *q,
+				     struct amdgpu_mes_queue_properties *p)
+{
+	struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type];
+	u32 mqd_size = mqd_mgr->mqd_size;
+	struct amdgpu_mqd_prop mqd_prop = {0};
+	int r;
+
+	r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_GTT,
+				    &q->mqd_obj,
+				    &q->mqd_gpu_addr, &q->mqd_cpu_ptr);
+	if (r) {
+		dev_warn(adev->dev, "failed to create queue mqd bo (%d)", r);
+		return r;
+	}
+	memset(q->mqd_cpu_ptr, 0, mqd_size);
+
+	mqd_prop.mqd_gpu_addr = q->mqd_gpu_addr;
+	mqd_prop.hqd_base_gpu_addr = p->hqd_base_gpu_addr;
+	mqd_prop.rptr_gpu_addr = p->rptr_gpu_addr;
+	mqd_prop.wptr_gpu_addr = p->wptr_gpu_addr;
+	mqd_prop.queue_size = p->queue_size;
+	mqd_prop.use_doorbell = true;
+	mqd_prop.doorbell_index = p->doorbell_off;
+	mqd_prop.eop_gpu_addr = p->eop_gpu_addr;
+	mqd_prop.hqd_pipe_priority = p->hqd_pipe_priority;
+	mqd_prop.hqd_queue_priority = p->hqd_queue_priority;
+	mqd_prop.hqd_active = false;
+
+	r = amdgpu_bo_reserve(q->mqd_obj, false);
+	if (unlikely(r != 0))
+		goto clean_up;
+
+	mqd_mgr->init_mqd(adev, q->mqd_cpu_ptr, &mqd_prop);
+
+	amdgpu_bo_unreserve(q->mqd_obj);
+	return 0;
+
+clean_up:
+	amdgpu_bo_free_kernel(&q->mqd_obj,
+			      &q->mqd_gpu_addr,
+			      &q->mqd_cpu_ptr);
+	return r;
+}
+
+static void amdgpu_mes_queue_free_mqd(struct amdgpu_mes_queue *q)
+{
+	amdgpu_bo_free_kernel(&q->mqd_obj,
+			      &q->mqd_gpu_addr,
+			      &q->mqd_cpu_ptr);
+}
Signature · 1 line
2.35.1
author maintainer
#53 lore raw

PATCH 53/73 drm/amdgpu/mes: implement adding mes queue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Allocate related resources for the queue and add it to mes for scheduling.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 105 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  19 +++++
 2 files changed, 124 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +105 -0
index 2cd2fa76b5c8..9f059c32c6c2 100644
@@ -550,3 +550,108 @@ static void amdgpu_mes_queue_free_mqd(struct amdgpu_mes_queue *q)
 			      &q->mqd_gpu_addr,
 			      &q->mqd_cpu_ptr);
 }
+
+int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
+			    struct amdgpu_mes_queue_properties *qprops,
+			    int *queue_id)
+{
+	struct amdgpu_mes_queue *queue;
+	struct amdgpu_mes_gang *gang;
+	struct mes_add_queue_input queue_input;
+	unsigned long flags;
+	int r;
+
+	mutex_lock(&adev->mes.mutex);
+
+	gang = idr_find(&adev->mes.gang_id_idr, gang_id);
+	if (!gang) {
+		DRM_ERROR("gang id %d doesn't exist\n", gang_id);
+		mutex_unlock(&adev->mes.mutex);
+		return -EINVAL;
+	}
+
+	/* allocate the mes queue buffer */
+	queue = kzalloc(sizeof(struct amdgpu_mes_queue), GFP_KERNEL);
+	if (!queue) {
+		mutex_unlock(&adev->mes.mutex);
+		return -ENOMEM;
+	}
+
+	/* add the mes gang to idr list */
+	spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
+	r = idr_alloc(&adev->mes.queue_id_idr, queue, 1, 0,
+		      GFP_ATOMIC);
+	if (r < 0) {
+		spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+		goto clean_up_memory;
+	}
+	spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+	*queue_id = queue->queue_id = r;
+
+	/* allocate a doorbell index for the queue */
+	r = amdgpu_mes_queue_doorbell_get(adev, gang->process,
+					  qprops->queue_type,
+					  &qprops->doorbell_off);
+	if (r)
+		goto clean_up_queue_id;
+
+	/* initialize the queue mqd */
+	r = amdgpu_mes_queue_init_mqd(adev, queue, qprops);
+	if (r)
+		goto clean_up_doorbell;
+
+	/* add hw queue to mes */
+	queue_input.process_id = gang->process->pasid;
+	queue_input.page_table_base_addr = gang->process->pd_gpu_addr;
+	queue_input.process_va_start = 0;
+	queue_input.process_va_end =
+		(adev->vm_manager.max_pfn - 1) << AMDGPU_GPU_PAGE_SHIFT;
+	queue_input.process_quantum = gang->process->process_quantum;
+	queue_input.process_context_addr = gang->process->proc_ctx_gpu_addr;
+	queue_input.gang_quantum = gang->gang_quantum;
+	queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
+	queue_input.inprocess_gang_priority = gang->inprocess_gang_priority;
+	queue_input.gang_global_priority_level = gang->global_priority_level;
+	queue_input.doorbell_offset = qprops->doorbell_off;
+	queue_input.mqd_addr = queue->mqd_gpu_addr;
+	queue_input.wptr_addr = qprops->wptr_gpu_addr;
+	queue_input.queue_type = qprops->queue_type;
+	queue_input.paging = qprops->paging;
+
+	r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input);
+	if (r) {
+		DRM_ERROR("failed to add hardware queue to MES, doorbell=0x%llx\n",
+			  qprops->doorbell_off);
+		goto clean_up_mqd;
+	}
+
+	DRM_DEBUG("MES hw queue was added, pasid=%d, gang id=%d, "
+		  "queue type=%d, doorbell=0x%llx\n",
+		  gang->process->pasid, gang_id, qprops->queue_type,
+		  qprops->doorbell_off);
+
+	queue->ring = qprops->ring;
+	queue->doorbell_off = qprops->doorbell_off;
+	queue->wptr_gpu_addr = qprops->wptr_gpu_addr;
+	queue->queue_type = qprops->queue_type;
+	queue->paging = qprops->paging;
+	queue->gang = gang;
+	list_add_tail(&queue->list, &gang->queue_list);
+
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+
+clean_up_mqd:
+	amdgpu_mes_queue_free_mqd(queue);
+clean_up_doorbell:
+	amdgpu_mes_queue_doorbell_free(adev, gang->process,
+				       qprops->doorbell_off);
+clean_up_queue_id:
+	spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
+	idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
+	spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+clean_up_memory:
+	kfree(queue);
+	mutex_unlock(&adev->mes.mutex);
+	return r;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +19 -0
index 43d3a689732a..ec727c2109bc 100644
@@ -158,6 +158,21 @@ struct amdgpu_mes_queue {
 	struct amdgpu_ring 		*ring;
 };
 
+struct amdgpu_mes_queue_properties {
+	int 			queue_type;
+	uint64_t                hqd_base_gpu_addr;
+	uint64_t                rptr_gpu_addr;
+	uint64_t                wptr_gpu_addr;
+	uint32_t                queue_size;
+	uint64_t                eop_gpu_addr;
+	uint32_t                hqd_pipe_priority;
+	uint32_t                hqd_queue_priority;
+	bool 			paging;
+	struct amdgpu_ring 	*ring;
+	/* out */
+	uint64_t       		doorbell_off;
+};
+
 struct amdgpu_mes_gang_properties {
 	uint32_t 	priority;
 	uint32_t 	gang_quantum;
@@ -233,4 +248,8 @@ int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
 int amdgpu_mes_suspend(struct amdgpu_device *adev);
 int amdgpu_mes_resume(struct amdgpu_device *adev);
 
+int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
+			    struct amdgpu_mes_queue_properties *qprops,
+			    int *queue_id);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#54 lore raw

PATCH 54/73 drm/amdgpu/mes: implement removing mes queue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Remove the MES queue from MES scheduling and free its resources.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 45 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  1 +
 2 files changed, 46 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +45 -0
index 9f059c32c6c2..df0e542bd687 100644
@@ -655,3 +655,48 @@ int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
 	mutex_unlock(&adev->mes.mutex);
 	return r;
 }
+
+int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id)
+{
+	unsigned long flags;
+	struct amdgpu_mes_queue *queue;
+	struct amdgpu_mes_gang *gang;
+	struct mes_remove_queue_input queue_input;
+	int r;
+
+	mutex_lock(&adev->mes.mutex);
+
+	/* remove the mes gang from idr list */
+	spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
+
+	queue = idr_find(&adev->mes.queue_id_idr, queue_id);
+	if (!queue) {
+		spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+		mutex_unlock(&adev->mes.mutex);
+		DRM_ERROR("queue id %d doesn't exist\n", queue_id);
+		return -EINVAL;
+	}
+
+	idr_remove(&adev->mes.queue_id_idr, queue_id);
+	spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
+
+	DRM_DEBUG("try to remove queue, doorbell off = 0x%llx\n",
+		  queue->doorbell_off);
+
+	gang = queue->gang;
+	queue_input.doorbell_offset = queue->doorbell_off;
+	queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
+
+	r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
+	if (r)
+		DRM_ERROR("failed to remove hardware queue, queue id = %d\n",
+			  queue_id);
+
+	amdgpu_mes_queue_free_mqd(queue);
+	list_del(&queue->list);
+	amdgpu_mes_queue_doorbell_free(adev, gang->process,
+				       queue->doorbell_off);
+	kfree(queue);
+	mutex_unlock(&adev->mes.mutex);
+	return 0;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +1 -0
index ec727c2109bc..bf90863852a7 100644
@@ -251,5 +251,6 @@ int amdgpu_mes_resume(struct amdgpu_device *adev);
 int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
 			    struct amdgpu_mes_queue_properties *qprops,
 			    int *queue_id);
+int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id);
 
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#55 lore raw

PATCH 55/73 drm/amdgpu/mes: add helper function to convert ring to queue property

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add the helper function to convert ring to queue property.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +17 -0
index df0e542bd687..8cb74d0d0a1f 100644
@@ -700,3 +700,20 @@ int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id)
 	mutex_unlock(&adev->mes.mutex);
 	return 0;
 }
+
+static void
+amdgpu_mes_ring_to_queue_props(struct amdgpu_device *adev,
+			       struct amdgpu_ring *ring,
+			       struct amdgpu_mes_queue_properties *props)
+{
+	props->queue_type = ring->funcs->type;
+	props->hqd_base_gpu_addr = ring->gpu_addr;
+	props->rptr_gpu_addr = ring->rptr_gpu_addr;
+	props->wptr_gpu_addr = ring->wptr_gpu_addr;
+	props->queue_size = ring->ring_size;
+	props->eop_gpu_addr = ring->eop_gpu_addr;
+	props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
+	props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
+	props->paging = false;
+	props->ring = ring;
+}
Signature · 1 line
2.35.1
author maintainer
#56 lore raw

PATCH 56/73 drm/amdgpu/mes: add helper function to get the ctx meta data offset

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add the helper function to get the corresponding ctx meta data offset.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 36 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  3 ++-
 2 files changed, 38 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +36 -0
index 8cb74d0d0a1f..4e99adcfbb0e 100644
@@ -717,3 +717,39 @@ amdgpu_mes_ring_to_queue_props(struct amdgpu_device *adev,
 	props->paging = false;
 	props->ring = ring;
 }
+
+#define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng)			\
+do {									\
+       if (id_offs < AMDGPU_MES_CTX_MAX_OFFS)				\
+		return offsetof(struct amdgpu_mes_ctx_meta_data,	\
+				_eng[ring->idx].slots[id_offs]);        \
+       else if (id_offs == AMDGPU_MES_CTX_RING_OFFS)			\
+		return offsetof(struct amdgpu_mes_ctx_meta_data,        \
+				_eng[ring->idx].ring);                  \
+       else if (id_offs == AMDGPU_MES_CTX_IB_OFFS)			\
+		return offsetof(struct amdgpu_mes_ctx_meta_data,        \
+				_eng[ring->idx].ib);                    \
+       else if (id_offs == AMDGPU_MES_CTX_PADDING_OFFS)			\
+		return offsetof(struct amdgpu_mes_ctx_meta_data,        \
+				_eng[ring->idx].padding);               \
+} while(0)
+
+int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs)
+{
+	switch (ring->funcs->type) {
+	case AMDGPU_RING_TYPE_GFX:
+		DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(gfx);
+		break;
+	case AMDGPU_RING_TYPE_COMPUTE:
+		DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(compute);
+		break;
+	case AMDGPU_RING_TYPE_SDMA:
+		DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(sdma);
+		break;
+	default:
+		break;
+	}
+
+	WARN_ON(1);
+	return -EINVAL;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +2 -1
index bf90863852a7..36684416f277 100644
@@ -230,9 +230,10 @@ struct amdgpu_mes_funcs {
 			   struct mes_resume_gang_input *input);
 };
 
-
 #define amdgpu_mes_kiq_hw_init(adev) (adev)->mes.kiq_hw_init((adev))
 
+int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs);
+
 int amdgpu_mes_init(struct amdgpu_device *adev);
 void amdgpu_mes_fini(struct amdgpu_device *adev);
 
Signature · 1 line
2.35.1
author maintainer
#57 lore raw

PATCH 57/73 drm/amdgpu/mes: use ring for kernel queue submission

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Use ring as the front end for kernel queue submission.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 93 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  5 ++
 2 files changed, 98 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +93 -0
index 4e99adcfbb0e..827391fcb2a3 100644
@@ -753,3 +753,96 @@ int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs)
 	WARN_ON(1);
 	return -EINVAL;
 }
+
+int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
+			int queue_type, int idx,
+			struct amdgpu_mes_ctx_data *ctx_data,
+			struct amdgpu_ring **out)
+{
+	struct amdgpu_ring *ring;
+	struct amdgpu_mes_gang *gang;
+	struct amdgpu_mes_queue_properties qprops = {0};
+	int r, queue_id, pasid;
+
+	mutex_lock(&adev->mes.mutex);
+	gang = idr_find(&adev->mes.gang_id_idr, gang_id);
+	if (!gang) {
+		DRM_ERROR("gang id %d doesn't exist\n", gang_id);
+		mutex_unlock(&adev->mes.mutex);
+		return -EINVAL;
+	}
+	pasid = gang->process->pasid;
+
+	ring = kzalloc(sizeof(struct amdgpu_ring), GFP_KERNEL);
+	if (!ring) {
+		mutex_unlock(&adev->mes.mutex);
+		return -ENOMEM;
+	}
+
+	ring->ring_obj = NULL;
+	ring->use_doorbell = true;
+	ring->is_mes_queue = true;
+	ring->mes_ctx = ctx_data;
+	ring->idx = idx;
+	ring->no_scheduler = true;
+
+	if (queue_type == AMDGPU_RING_TYPE_COMPUTE) {
+		int offset = offsetof(struct amdgpu_mes_ctx_meta_data,
+				      compute[ring->idx].mec_hpd);
+		ring->eop_gpu_addr =
+			amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
+	}
+
+	switch (queue_type) {
+	case AMDGPU_RING_TYPE_GFX:
+		ring->funcs = adev->gfx.gfx_ring[0].funcs;
+		break;
+	case AMDGPU_RING_TYPE_COMPUTE:
+		ring->funcs = adev->gfx.compute_ring[0].funcs;
+		break;
+	case AMDGPU_RING_TYPE_SDMA:
+		ring->funcs = adev->sdma.instance[0].ring.funcs;
+		break;
+	default:
+		BUG();
+	}
+
+	r = amdgpu_ring_init(adev, ring, 1024, NULL, 0,
+			     AMDGPU_RING_PRIO_DEFAULT, NULL);
+	if (r)
+		goto clean_up_memory;
+
+	amdgpu_mes_ring_to_queue_props(adev, ring, &qprops);
+
+	dma_fence_wait(gang->process->vm->last_update, false);
+	dma_fence_wait(ctx_data->meta_data_va->last_pt_update, false);
+	mutex_unlock(&adev->mes.mutex);
+
+	r = amdgpu_mes_add_hw_queue(adev, gang_id, &qprops, &queue_id);
+	if (r)
+		goto clean_up_ring;
+
+	ring->hw_queue_id = queue_id;
+	ring->doorbell_index = qprops.doorbell_off;
+
+	if (queue_type == AMDGPU_RING_TYPE_GFX)
+		sprintf(ring->name, "gfx_%d.%d.%d", pasid, gang_id, queue_id);
+	else if (queue_type == AMDGPU_RING_TYPE_COMPUTE)
+		sprintf(ring->name, "compute_%d.%d.%d", pasid, gang_id,
+			queue_id);
+	else if (queue_type == AMDGPU_RING_TYPE_SDMA)
+		sprintf(ring->name, "sdma_%d.%d.%d", pasid, gang_id,
+			queue_id);
+	else
+		BUG();
+
+	*out = ring;
+	return 0;
+
+clean_up_ring:
+	amdgpu_ring_fini(ring);
+clean_up_memory:
+	kfree(ring);
+	mutex_unlock(&adev->mes.mutex);
+	return r;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +5 -0
index 36684416f277..1fe5c869f37e 100644
@@ -254,4 +254,9 @@ int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
 			    int *queue_id);
 int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id);
 
+int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
+			int queue_type, int idx,
+			struct amdgpu_mes_ctx_data *ctx_data,
+			struct amdgpu_ring **out);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#58 lore raw

PATCH 58/73 drm/amdgpu/mes: implement removing mes ring

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Remove the mes ring and its resources.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 11 +++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  2 ++
 2 files changed, 13 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +11 -0
index 827391fcb2a3..fa43a7e3c9ab 100644
@@ -846,3 +846,14 @@ int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
 	mutex_unlock(&adev->mes.mutex);
 	return r;
 }
+
+void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
+			    struct amdgpu_ring *ring)
+{
+	if (!ring)
+		return;
+
+	amdgpu_mes_remove_hw_queue(adev, ring->hw_queue_id);
+	amdgpu_ring_fini(ring);
+	kfree(ring);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +2 -0
index 1fe5c869f37e..37232b396b06 100644
@@ -258,5 +258,7 @@ int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
 			int queue_type, int idx,
 			struct amdgpu_mes_ctx_data *ctx_data,
 			struct amdgpu_ring **out);
+void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
+			    struct amdgpu_ring *ring);
 
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#59 lore raw

PATCH 59/73 drm/amdgpu/mes: add helper functions to alloc/free ctx metadata

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add the helper functions to allocate/free context metadata.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 25 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  4 ++++
 2 files changed, 29 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +25 -0
index fa43a7e3c9ab..6c01581e3a7b 100644
@@ -857,3 +857,28 @@ void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
 	amdgpu_ring_fini(ring);
 	kfree(ring);
 }
+
+int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
+				   struct amdgpu_mes_ctx_data *ctx_data)
+{
+	int r;
+
+	r = amdgpu_bo_create_kernel(adev,
+			    sizeof(struct amdgpu_mes_ctx_meta_data),
+			    PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
+			    &ctx_data->meta_data_obj, NULL,
+			    &ctx_data->meta_data_ptr);
+	if (!ctx_data->meta_data_obj)
+		return -ENOMEM;
+
+	memset(ctx_data->meta_data_ptr, 0,
+	       sizeof(struct amdgpu_mes_ctx_meta_data));
+
+	return 0;
+}
+
+void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data)
+{
+	if (ctx_data->meta_data_obj)
+		amdgpu_bo_free_kernel(&ctx_data->meta_data_obj, NULL, NULL);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +4 -0
index 37232b396b06..50d490e69cb7 100644
@@ -261,4 +261,8 @@ int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
 void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
 			    struct amdgpu_ring *ring);
 
+int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
+				   struct amdgpu_mes_ctx_data *ctx_data);
+void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#60 lore raw

PATCH 60/73 drm/amdgpu: skip kfd routines when mes enabled

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For kfd hasn't supported mes, skip kfd routines.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 25 ++++++++++++++--------
 1 file changed, 16 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +16 -9
index b2366d0d3047..728e7be54a59 100644
@@ -2171,7 +2171,8 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 		adev->has_pr3 = parent ? pci_pr3_present(parent) : false;
 	}
 
-	amdgpu_amdkfd_device_probe(adev);
+	if (!adev->enable_mes)
+		amdgpu_amdkfd_device_probe(adev);
 
 	adev->pm.pp_feature = amdgpu_pp_feature_mask;
 	if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS)
@@ -2498,7 +2499,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 		goto init_failed;
 
 	/* Don't init kfd if whole hive need to be reset during init */
-	if (!adev->gmc.xgmi.pending_reset)
+	if (!adev->gmc.xgmi.pending_reset &&
+	    !adev->enable_mes)
 		amdgpu_amdkfd_device_init(adev);
 
 	amdgpu_fru_get_product_info(adev);
@@ -2861,7 +2863,8 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev)
 	if (adev->gmc.xgmi.num_physical_nodes > 1)
 		amdgpu_xgmi_remove_device(adev);
 
-	amdgpu_amdkfd_device_fini_sw(adev);
+	if (!adev->enable_mes)
+		amdgpu_amdkfd_device_fini_sw(adev);
 
 	for (i = adev->num_ip_blocks - 1; i >= 0; i--) {
 		if (!adev->ip_blocks[i].status.sw)
@@ -4122,7 +4125,7 @@ int amdgpu_device_suspend(struct drm_device *dev, bool fbcon)
 
 	amdgpu_device_ip_suspend_phase1(adev);
 
-	if (!adev->in_s0ix)
+	if (!adev->in_s0ix && !adev->enable_mes)
 		amdgpu_amdkfd_suspend(adev, adev->in_runpm);
 
 	amdgpu_device_evict_resources(adev);
@@ -4176,7 +4179,7 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
 	queue_delayed_work(system_wq, &adev->delayed_init_work,
 			   msecs_to_jiffies(AMDGPU_RESUME_MS));
 
-	if (!adev->in_s0ix) {
+	if (!adev->in_s0ix && !adev->enable_mes) {
 		r = amdgpu_amdkfd_resume(adev, adev->in_runpm);
 		if (r)
 			return r;
@@ -4459,7 +4462,8 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev,
 	int retry_limit = 0;
 
 retry:
-	amdgpu_amdkfd_pre_reset(adev);
+	if (!adev->enable_mes)
+		amdgpu_amdkfd_pre_reset(adev);
 
 	amdgpu_amdkfd_pre_reset(adev);
 
@@ -4497,7 +4501,9 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev,
 	if (!r) {
 		amdgpu_irq_gpu_reset_resume_helper(adev);
 		r = amdgpu_ib_ring_tests(adev);
-		amdgpu_amdkfd_post_reset(adev);
+
+		if (!adev->enable_mes)
+			amdgpu_amdkfd_post_reset(adev);
 	}
 
 error:
@@ -5142,7 +5148,7 @@ int amdgpu_device_gpu_recover_imp(struct amdgpu_device *adev,
 
 		cancel_delayed_work_sync(&tmp_adev->delayed_init_work);
 
-		if (!amdgpu_sriov_vf(tmp_adev))
+		if (!amdgpu_sriov_vf(tmp_adev) && !adev->enable_mes)
 			amdgpu_amdkfd_pre_reset(tmp_adev);
 
 		/*
@@ -5265,7 +5271,8 @@ int amdgpu_device_gpu_recover_imp(struct amdgpu_device *adev,
 skip_sched_resume:
 	list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
 		/* unlock kfd: SRIOV would do it separately */
-		if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev))
+		if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev) &&
+		    !adev->enable_mes)
 			amdgpu_amdkfd_post_reset(tmp_adev);
 
 		/* kfd_post_reset will do nothing if kfd device is not initialized,
Signature · 1 line
2.35.1
author maintainer
#61 lore raw

PATCH 61/73 drm/amdgpu: Enable KFD with MES enabled

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Enable KFD initialization with MES enabled.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 24 ++++++++--------------
 1 file changed, 9 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -15
index 728e7be54a59..e582f1044c0f 100644
@@ -2171,8 +2171,7 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 		adev->has_pr3 = parent ? pci_pr3_present(parent) : false;
 	}
 
-	if (!adev->enable_mes)
-		amdgpu_amdkfd_device_probe(adev);
+	amdgpu_amdkfd_device_probe(adev);
 
 	adev->pm.pp_feature = amdgpu_pp_feature_mask;
 	if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS)
@@ -2499,8 +2498,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 		goto init_failed;
 
 	/* Don't init kfd if whole hive need to be reset during init */
-	if (!adev->gmc.xgmi.pending_reset &&
-	    !adev->enable_mes)
+	if (!adev->gmc.xgmi.pending_reset)
 		amdgpu_amdkfd_device_init(adev);
 
 	amdgpu_fru_get_product_info(adev);
@@ -2863,8 +2861,7 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev)
 	if (adev->gmc.xgmi.num_physical_nodes > 1)
 		amdgpu_xgmi_remove_device(adev);
 
-	if (!adev->enable_mes)
-		amdgpu_amdkfd_device_fini_sw(adev);
+	amdgpu_amdkfd_device_fini_sw(adev);
 
 	for (i = adev->num_ip_blocks - 1; i >= 0; i--) {
 		if (!adev->ip_blocks[i].status.sw)
@@ -4125,7 +4122,7 @@ int amdgpu_device_suspend(struct drm_device *dev, bool fbcon)
 
 	amdgpu_device_ip_suspend_phase1(adev);
 
-	if (!adev->in_s0ix && !adev->enable_mes)
+	if (!adev->in_s0ix)
 		amdgpu_amdkfd_suspend(adev, adev->in_runpm);
 
 	amdgpu_device_evict_resources(adev);
@@ -4179,7 +4176,7 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
 	queue_delayed_work(system_wq, &adev->delayed_init_work,
 			   msecs_to_jiffies(AMDGPU_RESUME_MS));
 
-	if (!adev->in_s0ix && !adev->enable_mes) {
+	if (!adev->in_s0ix) {
 		r = amdgpu_amdkfd_resume(adev, adev->in_runpm);
 		if (r)
 			return r;
@@ -4462,8 +4459,7 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev,
 	int retry_limit = 0;
 
 retry:
-	if (!adev->enable_mes)
-		amdgpu_amdkfd_pre_reset(adev);
+	amdgpu_amdkfd_pre_reset(adev);
 
 	amdgpu_amdkfd_pre_reset(adev);
 
@@ -4502,8 +4498,7 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev,
 		amdgpu_irq_gpu_reset_resume_helper(adev);
 		r = amdgpu_ib_ring_tests(adev);
 
-		if (!adev->enable_mes)
-			amdgpu_amdkfd_post_reset(adev);
+		amdgpu_amdkfd_post_reset(adev);
 	}
 
 error:
@@ -5148,7 +5143,7 @@ int amdgpu_device_gpu_recover_imp(struct amdgpu_device *adev,
 
 		cancel_delayed_work_sync(&tmp_adev->delayed_init_work);
 
-		if (!amdgpu_sriov_vf(tmp_adev) && !adev->enable_mes)
+		if (!amdgpu_sriov_vf(tmp_adev))
 			amdgpu_amdkfd_pre_reset(tmp_adev);
 
 		/*
@@ -5271,8 +5266,7 @@ int amdgpu_device_gpu_recover_imp(struct amdgpu_device *adev,
 skip_sched_resume:
 	list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
 		/* unlock kfd: SRIOV would do it separately */
-		if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev) &&
-		    !adev->enable_mes)
+		if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev))
 			amdgpu_amdkfd_post_reset(tmp_adev);
 
 		/* kfd_post_reset will do nothing if kfd device is not initialized,
Signature · 1 line
2.35.1
author maintainer
#62 lore raw

PATCH 62/73 drm/amdgpu: skip some checking for mes queue ib submission

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Skip some checking for mes queue ib submission.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +2 -2
index d583766ea392..d8354453cc29 100644
@@ -155,12 +155,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned num_ibs,
 		fence_ctx = 0;
 	}
 
-	if (!ring->sched.ready) {
+	if (!ring->sched.ready && !ring->is_mes_queue) {
 		dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name);
 		return -EINVAL;
 	}
 
-	if (vm && !job->vmid) {
+	if (vm && !job->vmid && !ring->is_mes_queue) {
 		dev_err(adev->dev, "VM IB without ID\n");
 		return -EINVAL;
 	}
Signature · 1 line
2.35.1
author maintainer
#63 lore raw

PATCH 63/73 drm/amdgpu: skip kiq ib tests if mes enabled

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For kiq conflicts with mes, skip kiq ib tests.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +4 -0
index d8354453cc29..258cffe3c06a 100644
@@ -390,6 +390,10 @@ int amdgpu_ib_ring_tests(struct amdgpu_device *adev)
 		if (!ring->sched.ready || !ring->funcs->test_ib)
 			continue;
 
+		if (adev->enable_mes &&
+		    ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+			continue;
+
 		/* MM engine need more time */
 		if (ring->funcs->type == AMDGPU_RING_TYPE_UVD ||
 			ring->funcs->type == AMDGPU_RING_TYPE_VCE ||
Signature · 1 line
2.35.1
author maintainer
#64 lore raw

PATCH 64/73 drm/amdgpu: skip gds switch for mes queue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

For mes manages gds allocation, skip gds switch.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -1
index 4736174f5e4d..7276b03ef970 100644
@@ -639,7 +639,8 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job,
 	}
 	dma_fence_put(fence);
 
-	if (ring->funcs->emit_gds_switch && gds_switch_needed) {
+	if (!ring->is_mes_queue && ring->funcs->emit_gds_switch &&
+	    gds_switch_needed) {
 		id->gds_base = job->gds_base;
 		id->gds_size = job->gds_size;
 		id->gws_base = job->gws_base;
Signature · 1 line
2.35.1
author maintainer
#65 lore raw

PATCH 65/73 drm/amdgpu: kiq takes charge of all queues

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

To make kgq/kcq and mes queue co-exist, kiq needs take charge of all queues.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +3 -0
index 28a736c507bb..40df1e04d682 100644
@@ -535,6 +535,9 @@ int amdgpu_gfx_enable_kcq(struct amdgpu_device *adev)
 		return r;
 	}
 
+	if (adev->enable_mes)
+		queue_mask = ~0ULL;
+
 	kiq->pmf->kiq_set_resources(kiq_ring, queue_mask);
 	for (i = 0; i < adev->gfx.num_compute_rings; i++)
 		kiq->pmf->kiq_map_queues(kiq_ring, &adev->gfx.compute_ring[i]);
Signature · 1 line
2.35.1
author maintainer
#66 lore raw

PATCH 66/73 drm/amdgpu/mes: map ctx metadata for mes self test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Map ctx metadata for mes self test.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 37 +++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +37 -0
index 6c01581e3a7b..b440b36dd98a 100644
@@ -882,3 +882,40 @@ void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data)
 	if (ctx_data->meta_data_obj)
 		amdgpu_bo_free_kernel(&ctx_data->meta_data_obj, NULL, NULL);
 }
+
+static int amdgpu_mes_test_map_ctx_meta_data(struct amdgpu_device *adev,
+				     struct amdgpu_vm *vm,
+				     struct amdgpu_mes_ctx_data *ctx_data)
+{
+	struct amdgpu_bo_va *meta_data_va = NULL;
+	uint64_t meta_data_addr = AMDGPU_VA_RESERVED_SIZE;
+	int r;
+
+	r = amdgpu_map_static_csa(adev, vm, ctx_data->meta_data_obj,
+				  &meta_data_va, meta_data_addr,
+				  sizeof(struct amdgpu_mes_ctx_meta_data));
+	if (r)
+		return r;
+
+	r = amdgpu_vm_bo_update(adev, meta_data_va, false);
+	if (r)
+		goto error;
+
+	r = amdgpu_vm_update_pdes(adev, vm, false);
+	if (r)
+		goto error;
+
+	dma_fence_wait(vm->last_update, false);
+	dma_fence_wait(meta_data_va->last_pt_update, false);
+
+	ctx_data->meta_data_gpu_addr = meta_data_addr;
+	ctx_data->meta_data_va = meta_data_va;
+
+	return 0;
+
+error:
+	BUG_ON(amdgpu_bo_reserve(ctx_data->meta_data_obj, true));
+	amdgpu_vm_bo_rmv(adev, meta_data_va);
+	amdgpu_bo_unreserve(ctx_data->meta_data_obj);
+	return r;
+}
Signature · 1 line
2.35.1
author maintainer
#67 lore raw

PATCH 67/73 drm/amdgpu/mes: create gang and queues for mes self test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Create gang and queues for mes self test.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 39 +++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +39 -0
index b440b36dd98a..027f3aae6025 100644
@@ -919,3 +919,42 @@ static int amdgpu_mes_test_map_ctx_meta_data(struct amdgpu_device *adev,
 	amdgpu_bo_unreserve(ctx_data->meta_data_obj);
 	return r;
 }
+
+static int amdgpu_mes_test_create_gang_and_queues(struct amdgpu_device *adev,
+					  int pasid, int *gang_id,
+					  int queue_type, int num_queue,
+					  struct amdgpu_ring **added_rings,
+					  struct amdgpu_mes_ctx_data *ctx_data)
+{
+	struct amdgpu_ring *ring;
+	struct amdgpu_mes_gang_properties gprops = {0};
+	int r, j;
+
+	/* create a gang for the process */
+	gprops.priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
+	gprops.gang_quantum = adev->mes.default_gang_quantum;
+	gprops.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
+	gprops.priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
+	gprops.global_priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
+
+	r = amdgpu_mes_add_gang(adev, pasid, &gprops, gang_id);
+	if (r) {
+		DRM_ERROR("failed to add gang\n");
+		return r;
+	}
+
+	/* create queues for the gang */
+	for (j = 0; j < num_queue; j++) {
+		r = amdgpu_mes_add_ring(adev, *gang_id, queue_type, j,
+					ctx_data, &ring);
+		if (r) {
+			DRM_ERROR("failed to add ring\n");
+			break;
+		}
+
+		DRM_INFO("ring %s was added\n", ring->name);
+		added_rings[j] = ring;
+	}
+
+	return 0;
+}
Signature · 1 line
2.35.1
author maintainer
#68 lore raw

PATCH 68/73 drm/amdgpu/mes: add ring/ib test for mes self test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Run the ring test and ib test for mes self test.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +32 -0
index 027f3aae6025..e2b1da08ab64 100644
@@ -958,3 +958,35 @@ static int amdgpu_mes_test_create_gang_and_queues(struct amdgpu_device *adev,
 
 	return 0;
 }
+
+static int amdgpu_mes_test_queues(struct amdgpu_ring **added_rings)
+{
+	struct amdgpu_ring *ring;
+	int i, r;
+
+	for (i = 0; i < AMDGPU_MES_CTX_MAX_RINGS; i++) {
+		ring = added_rings[i];
+		if (!ring)
+			continue;
+
+		r = amdgpu_ring_test_ring(ring);
+		if (r) {
+			DRM_DEV_ERROR(ring->adev->dev,
+				      "ring %s test failed (%d)\n",
+				      ring->name, r);
+			return r;
+		} else
+			DRM_INFO("ring %s test pass\n", ring->name);
+
+		r = amdgpu_ring_test_ib(ring, 1000 * 10);
+		if (r) {
+			DRM_DEV_ERROR(ring->adev->dev,
+				      "ring %s ib test failed (%d)\n",
+				      ring->name, r);
+			return r;
+		} else
+			DRM_INFO("ring %s ib test pass\n", ring->name);
+	}
+
+	return 0;
+}
Signature · 1 line
2.35.1
author maintainer
#69 lore raw

PATCH 69/73 drm/amdgpu/mes: implement mes self test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add mes self test to verify its fundamental functionality by running ring test and ib test of mes kernel queue.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 97 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  2 +
 2 files changed, 99 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +97 -0
index e2b1da08ab64..c9516b3aa6d9 100644
@@ -990,3 +990,100 @@ static int amdgpu_mes_test_queues(struct amdgpu_ring **added_rings)
 
 	return 0;
 }
+
+int amdgpu_mes_self_test(struct amdgpu_device *adev)
+{
+	struct amdgpu_vm *vm = NULL;
+	struct amdgpu_mes_ctx_data ctx_data = {0};
+	struct amdgpu_ring *added_rings[AMDGPU_MES_CTX_MAX_RINGS] = { NULL };
+	int gang_ids[3] = {0};
+	int queue_types[][2] = { { AMDGPU_RING_TYPE_GFX,
+				   AMDGPU_MES_CTX_MAX_GFX_RINGS},
+				 { AMDGPU_RING_TYPE_COMPUTE,
+				   AMDGPU_MES_CTX_MAX_COMPUTE_RINGS},
+				 { AMDGPU_RING_TYPE_SDMA,
+				   AMDGPU_MES_CTX_MAX_SDMA_RINGS } };
+	int i, r, pasid, k = 0;
+
+	pasid = amdgpu_pasid_alloc(16);
+	if (pasid < 0) {
+		dev_warn(adev->dev, "No more PASIDs available!");
+		pasid = 0;
+	}
+
+	vm = kzalloc(sizeof(*vm), GFP_KERNEL);
+	if (!vm) {
+		r = -ENOMEM;
+		goto error_pasid;
+	}
+
+	r = amdgpu_vm_init(adev, vm);
+	if (r) {
+		DRM_ERROR("failed to initialize vm\n");
+		goto error_pasid;
+	}
+
+	r = amdgpu_mes_ctx_alloc_meta_data(adev, &ctx_data);
+	if (r) {
+		DRM_ERROR("failed to alloc ctx meta data\n");
+		goto error_pasid;
+	}
+
+	r = amdgpu_mes_test_map_ctx_meta_data(adev, vm, &ctx_data);
+	if (r) {
+		DRM_ERROR("failed to map ctx meta data\n");
+		goto error_vm;
+	}
+
+	r = amdgpu_mes_create_process(adev, pasid, vm);
+	if (r) {
+		DRM_ERROR("failed to create MES process\n");
+		goto error_vm;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(queue_types); i++) {
+		r = amdgpu_mes_test_create_gang_and_queues(adev, pasid,
+							   &gang_ids[i],
+							   queue_types[i][0],
+							   queue_types[i][1],
+							   &added_rings[k],
+							   &ctx_data);
+		if (r)
+			goto error_queues;
+
+		k += queue_types[i][1];
+	}
+
+	/* start ring test and ib test for MES queues */
+	amdgpu_mes_test_queues(added_rings);
+
+error_queues:
+	/* remove all queues */
+	for (i = 0; i < ARRAY_SIZE(added_rings); i++) {
+		if (!added_rings[i])
+			continue;
+		amdgpu_mes_remove_ring(adev, added_rings[i]);
+	}
+
+	for (i = 0; i < ARRAY_SIZE(gang_ids); i++) {
+		if (!gang_ids[i])
+			continue;
+		amdgpu_mes_remove_gang(adev, gang_ids[i]);
+	}
+
+	amdgpu_mes_destroy_process(adev, pasid);
+
+error_vm:
+	BUG_ON(amdgpu_bo_reserve(ctx_data.meta_data_obj, true));
+	amdgpu_vm_bo_rmv(adev, ctx_data.meta_data_va);
+	amdgpu_bo_unreserve(ctx_data.meta_data_obj);
+	amdgpu_vm_fini(adev, vm);
+
+error_pasid:
+	if (pasid)
+		amdgpu_pasid_free(pasid);
+
+	amdgpu_mes_ctx_free_meta_data(&ctx_data);
+	kfree(vm);
+	return 0;
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +2 -0
index 50d490e69cb7..5c9e7932c7a9 100644
@@ -265,4 +265,6 @@ int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
 				   struct amdgpu_mes_ctx_data *ctx_data);
 void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data);
 
+int amdgpu_mes_self_test(struct amdgpu_device *adev);
+
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1
author maintainer
#70 lore raw

PATCH 70/73 drm/amdgpu/mes10.1: add mes self test in late init

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Add MES self test in late init.

---
 drivers/gpu/drm/amd/amdgpu/mes_v10_1.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/gpu/drm/amd/amdgpu/mes_v10_1.c +10 -0
index d468cb5a8854..622aa17b18e7 100644
@@ -1142,8 +1142,18 @@ static int mes_v10_1_resume(void *handle)
 	return amdgpu_mes_resume(adev);
 }
 
+static int mes_v10_0_late_init(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+	amdgpu_mes_self_test(adev);
+
+	return 0;
+}
+
 static const struct amd_ip_funcs mes_v10_1_ip_funcs = {
 	.name = "mes_v10_1",
+	.late_init = mes_v10_0_late_init,
 	.sw_init = mes_v10_1_sw_init,
 	.sw_fini = mes_v10_1_sw_fini,
 	.hw_init = mes_v10_1_hw_init,
Signature · 1 line
2.35.1
author maintainer
#71 lore raw

PATCH 71/73 drm/amdgpu/mes: fix vm csa update issue

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Need reserve VM buffers before update VM csa.

v2: rebase fixes

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 81 ++++++++++++++++++-------
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  3 +
 2 files changed, 62 insertions(+), 22 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +59 -22
index c9516b3aa6d9..51a6f309ef22 100644
@@ -883,40 +883,76 @@ void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data)
 		amdgpu_bo_free_kernel(&ctx_data->meta_data_obj, NULL, NULL);
 }
 
-static int amdgpu_mes_test_map_ctx_meta_data(struct amdgpu_device *adev,
-				     struct amdgpu_vm *vm,
-				     struct amdgpu_mes_ctx_data *ctx_data)
+int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
+				 struct amdgpu_vm *vm,
+				 struct amdgpu_mes_ctx_data *ctx_data)
 {
-	struct amdgpu_bo_va *meta_data_va = NULL;
-	uint64_t meta_data_addr = AMDGPU_VA_RESERVED_SIZE;
+	struct amdgpu_bo_va *bo_va;
+	struct ww_acquire_ctx ticket;
+	struct list_head list;
+	struct amdgpu_bo_list_entry pd;
+	struct ttm_validate_buffer csa_tv;
+	struct amdgpu_sync sync;
 	int r;
 
-	r = amdgpu_map_static_csa(adev, vm, ctx_data->meta_data_obj,
-				  &meta_data_va, meta_data_addr,
-				  sizeof(struct amdgpu_mes_ctx_meta_data));
-	if (r)
+	amdgpu_sync_create(&sync);
+	INIT_LIST_HEAD(&list);
+	INIT_LIST_HEAD(&csa_tv.head);
+
+	csa_tv.bo = &ctx_data->meta_data_obj->tbo;
+	csa_tv.num_shared = 1;
+
+	list_add(&csa_tv.head, &list);
+	amdgpu_vm_get_pd_bo(vm, &list, &pd);
+
+	r = ttm_eu_reserve_buffers(&ticket, &list, true, NULL);
+	if (r) {
+		DRM_ERROR("failed to reserve meta data BO: err=%d\n", r);
 		return r;
+	}
 
-	r = amdgpu_vm_bo_update(adev, meta_data_va, false);
-	if (r)
+	bo_va = amdgpu_vm_bo_add(adev, vm, ctx_data->meta_data_obj);
+	if (!bo_va) {
+		ttm_eu_backoff_reservation(&ticket, &list);
+		DRM_ERROR("failed to create bo_va for meta data BO\n");
+		return -ENOMEM;
+	}
+
+	r = amdgpu_vm_bo_map(adev, bo_va, ctx_data->meta_data_gpu_addr, 0,
+			     sizeof(struct amdgpu_mes_ctx_meta_data),
+			     AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
+			     AMDGPU_PTE_EXECUTABLE);
+
+	if (r) {
+		DRM_ERROR("failed to do bo_map on meta data, err=%d\n", r);
 		goto error;
+	}
 
-	r = amdgpu_vm_update_pdes(adev, vm, false);
-	if (r)
+	r = amdgpu_vm_bo_update(adev, bo_va, false);
+	if (r) {
+		DRM_ERROR("failed to do vm_bo_update on meta data\n");
 		goto error;
+	}
+	amdgpu_sync_fence(&sync, bo_va->last_pt_update);
 
-	dma_fence_wait(vm->last_update, false);
-	dma_fence_wait(meta_data_va->last_pt_update, false);
+	r = amdgpu_vm_update_pdes(adev, vm, false);
+	if (r) {
+		DRM_ERROR("failed to update pdes on meta data\n");
+		goto error;
+	}
+	amdgpu_sync_fence(&sync, vm->last_update);
 
-	ctx_data->meta_data_gpu_addr = meta_data_addr;
-	ctx_data->meta_data_va = meta_data_va;
+	amdgpu_sync_wait(&sync, false);
+	ttm_eu_backoff_reservation(&ticket, &list);
 
+	amdgpu_sync_free(&sync);
+	ctx_data->meta_data_va = bo_va;
 	return 0;
 
 error:
-	BUG_ON(amdgpu_bo_reserve(ctx_data->meta_data_obj, true));
-	amdgpu_vm_bo_rmv(adev, meta_data_va);
-	amdgpu_bo_unreserve(ctx_data->meta_data_obj);
+	amdgpu_vm_bo_del(adev, bo_va);
+	ttm_eu_backoff_reservation(&ticket, &list);
+	amdgpu_sync_free(&sync);
 	return r;
 }
 
@@ -1029,7 +1065,8 @@ int amdgpu_mes_self_test(struct amdgpu_device *adev)
 		goto error_pasid;
 	}
 
-	r = amdgpu_mes_test_map_ctx_meta_data(adev, vm, &ctx_data);
+	ctx_data.meta_data_gpu_addr = AMDGPU_VA_RESERVED_SIZE;
+	r = amdgpu_mes_ctx_map_meta_data(adev, vm, &ctx_data);
 	if (r) {
 		DRM_ERROR("failed to map ctx meta data\n");
 		goto error_vm;
@@ -1075,7 +1112,7 @@ int amdgpu_mes_self_test(struct amdgpu_device *adev)
 
 error_vm:
 	BUG_ON(amdgpu_bo_reserve(ctx_data.meta_data_obj, true));
-	amdgpu_vm_bo_rmv(adev, ctx_data.meta_data_va);
+	amdgpu_vm_bo_del(adev, ctx_data.meta_data_va);
 	amdgpu_bo_unreserve(ctx_data.meta_data_obj);
 	amdgpu_vm_fini(adev, vm);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3 -0
index 5c9e7932c7a9..a965ace0fd0e 100644
@@ -264,6 +264,9 @@ void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
 int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
 				   struct amdgpu_mes_ctx_data *ctx_data);
 void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data);
+int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
+				 struct amdgpu_vm *vm,
+				 struct amdgpu_mes_ctx_data *ctx_data);
 
 int amdgpu_mes_self_test(struct amdgpu_device *adev);
 
Signature · 1 line
2.35.1
author maintainer
#72 lore raw

PATCH 72/73 drm/amdgpu/mes: disable mes sdma queue test

In reply to Alex Deucher

From: Jack Xiao <[email protected]>

Disable mes sdma queue test on sienna cichlid+, for fw hasn't supported to map sdma queue. The test can be enabled if fw supports.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +5 -0
index 51a6f309ef22..e23c864aca11 100644
@@ -1079,6 +1079,11 @@ int amdgpu_mes_self_test(struct amdgpu_device *adev)
 	}
 
 	for (i = 0; i < ARRAY_SIZE(queue_types); i++) {
+		/* On sienna cichlid+, fw hasn't supported to map sdma queue. */
+		if (adev->asic_type >= CHIP_SIENNA_CICHLID &&
+		    i == AMDGPU_RING_TYPE_SDMA)
+			continue;
+
 		r = amdgpu_mes_test_create_gang_and_queues(adev, pasid,
 							   &gang_ids[i],
 							   queue_types[i][0],
Signature · 1 line
2.35.1
author maintainer
#73 lore raw

PATCH 73/73 drm/amdgpu/mes: Update the doorbell function signatures

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Update the function signatures for process doorbell allocations with MES enabled to make them more generic. KFD would need to access these functions to allocate/free doorbells when MES is enabled.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 37 +++++++++++++++----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h |  9 ++++++
 2 files changed, 31 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +22 -15
index e23c864aca11..5be30bf68b0c 100644
@@ -29,31 +29,40 @@
 #define AMDGPU_MES_MAX_NUM_OF_QUEUES_PER_PROCESS 1024
 #define AMDGPU_ONE_DOORBELL_SIZE 8
 
-static int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev)
+int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev)
 {
 	return roundup(AMDGPU_ONE_DOORBELL_SIZE *
 		       AMDGPU_MES_MAX_NUM_OF_QUEUES_PER_PROCESS,
 		       PAGE_SIZE);
 }
 
-static int amdgpu_mes_alloc_process_doorbells(struct amdgpu_device *adev,
-				      struct amdgpu_mes_process *process)
+int amdgpu_mes_alloc_process_doorbells(struct amdgpu_device *adev,
+				      unsigned int *doorbell_index)
 {
 	int r = ida_simple_get(&adev->mes.doorbell_ida, 2,
 			       adev->mes.max_doorbell_slices,
 			       GFP_KERNEL);
 	if (r > 0)
-		process->doorbell_index = r;
+		*doorbell_index = r;
 
 	return r;
 }
 
-static void amdgpu_mes_free_process_doorbells(struct amdgpu_device *adev,
-				      struct amdgpu_mes_process *process)
+void amdgpu_mes_free_process_doorbells(struct amdgpu_device *adev,
+				      unsigned int doorbell_index)
 {
-	if (process->doorbell_index)
-		ida_simple_remove(&adev->mes.doorbell_ida,
-				  process->doorbell_index);
+	if (doorbell_index)
+		ida_simple_remove(&adev->mes.doorbell_ida, doorbell_index);
+}
+
+unsigned int amdgpu_mes_get_doorbell_dw_offset_in_bar(
+					struct amdgpu_device *adev,
+					uint32_t doorbell_index,
+					unsigned int doorbell_id)
+{
+	return ((doorbell_index *
+		amdgpu_mes_doorbell_process_slice(adev)) / sizeof(u32) +
+		doorbell_id * 2);
 }
 
 static int amdgpu_mes_queue_doorbell_get(struct amdgpu_device *adev,
@@ -79,10 +88,8 @@ static int amdgpu_mes_queue_doorbell_get(struct amdgpu_device *adev,
 
 	set_bit(found, process->doorbell_bitmap);
 
-	*doorbell_index =
-		(process->doorbell_index *
-		 amdgpu_mes_doorbell_process_slice(adev)) / sizeof(u32) +
-		found * 2;
+	*doorbell_index = amdgpu_mes_get_doorbell_dw_offset_in_bar(adev,
+				process->doorbell_index, found);
 
 	return 0;
 }
@@ -262,7 +269,7 @@ int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
 	memset(process->proc_ctx_cpu_ptr, 0, AMDGPU_MES_PROC_CTX_SIZE);
 
 	/* allocate the starting doorbell index of the process */
-	r = amdgpu_mes_alloc_process_doorbells(adev, process);
+	r = amdgpu_mes_alloc_process_doorbells(adev, &process->doorbell_index);
 	if (r < 0) {
 		DRM_ERROR("failed to allocate doorbell for process\n");
 		goto clean_up_ctx;
@@ -338,7 +345,7 @@ void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid)
 		kfree(gang);
 	}
 
-	amdgpu_mes_free_process_doorbells(adev, process);
+	amdgpu_mes_free_process_doorbells(adev, process->doorbell_index);
 
 	idr_remove(&adev->mes.pasid_idr, pasid);
 	amdgpu_bo_free_kernel(&process->proc_ctx_bo,
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +9 -0
index a965ace0fd0e..ba039984e431 100644
@@ -270,4 +270,13 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
 
 int amdgpu_mes_self_test(struct amdgpu_device *adev);
 
+int amdgpu_mes_alloc_process_doorbells(struct amdgpu_device *adev,
+					unsigned int *doorbell_index);
+void amdgpu_mes_free_process_doorbells(struct amdgpu_device *adev,
+					unsigned int doorbell_index);
+unsigned int amdgpu_mes_get_doorbell_dw_offset_in_bar(
+					struct amdgpu_device *adev,
+					uint32_t doorbell_index,
+					unsigned int doorbell_id);
+int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev);
 #endif /* __AMDGPU_MES_H__ */
Signature · 1 line
2.35.1