Skip to contentSkip to conversation

drm/amdgpu: add CHIP_IP_DISCOVERY support for virtualization

PATCH In review Alex Deucher 2 participants 20 messages Aug 30, 2022 – Aug 31, 2022

Subsystems (3)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

PATCH 1/19 drm/amdgpu: add CHIP_IP_DISCOVERY support for virtualization

From: Horace Chen <[email protected]>

For further chips we will use CHIP_IP_DISCOVERY, so add this support for virtualization

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +2 -0
index 9be57389301b..0707e4338e65 100644
@@ -707,6 +707,7 @@ void amdgpu_detect_virtualization(struct amdgpu_device *adev)
 	case CHIP_SIENNA_CICHLID:
 	case CHIP_ARCTURUS:
 	case CHIP_ALDEBARAN:
+	case CHIP_IP_DISCOVERY:
 		reg = RREG32(mmRCC_IOV_FUNC_IDENTIFIER);
 		break;
 	default: /* other chip doesn't support SRIOV */
@@ -750,6 +751,7 @@ void amdgpu_detect_virtualization(struct amdgpu_device *adev)
 		case CHIP_NAVI10:
 		case CHIP_NAVI12:
 		case CHIP_SIENNA_CICHLID:
+		case CHIP_IP_DISCOVERY:
 			nv_set_virt_ops(adev);
 			/* try send GPU_INIT_DATA request to host */
 			amdgpu_virt_request_init_data(adev);
Signature · 1 line
2.37.1
author maintainer
#1 lore raw

PATCH 2/19 drm/amdgpu: add sriov nbio callback structure

From: Horace Chen <[email protected]>

[Why]
under SR-IOV, the nbio doorbell range will be defined by PF. So VF nbio doorbell range registers will be blocked. It will cause violation if VF access those registers directly.

[How]
create an nbio_v4_3_sriov_funcs for sriov nbio_v4_3 initialization to skip the setting for the doorbell range registers.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c |  5 ++-
 drivers/gpu/drm/amd/amdgpu/nbio_v4_3.c        | 44 +++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/nbio_v4_3.h        |  1 +
 3 files changed, 49 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +4 -1
index 5fd56ebc3f6c..854ca8d4505b 100644
@@ -2242,7 +2242,10 @@ int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev)
 		break;
 	case IP_VERSION(4, 3, 0):
 	case IP_VERSION(4, 3, 1):
-		adev->nbio.funcs = &nbio_v4_3_funcs;
+		if (amdgpu_sriov_vf(adev))
+			adev->nbio.funcs = &nbio_v4_3_sriov_funcs;
+		else
+			adev->nbio.funcs = &nbio_v4_3_funcs;
 		adev->nbio.hdp_flush_reg = &nbio_v4_3_hdp_flush_reg;
 		break;
 	case IP_VERSION(7, 7, 0):
drivers/gpu/drm/amd/amdgpu/nbio_v4_3.c +44 -0

47 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/nbio_v4_3.h +1 -0
index ade43661d7a9..711999ceedf4 100644
@@ -28,5 +28,6 @@
 
 extern const struct nbio_hdp_flush_reg nbio_v4_3_hdp_flush_reg;
 extern const struct amdgpu_nbio_funcs nbio_v4_3_funcs;
+extern const struct amdgpu_nbio_funcs nbio_v4_3_sriov_funcs;
 
 #endif
Signature · 1 line
2.37.1
author maintainer
#2 lore raw

PATCH 3/19 drm/amdgpu: add a compute pipe reset for RS64

In reply to Alex Deucher

From: Horace Chen <[email protected]>

[Why]
Under SR-IOV, we are not sure whether pipe status is good or not when doing initialization. The compute engine maybe fail to bringup if pipe status is bad.

[How]
Do an RS64 pipe reset for MEC before we do initialization. Also apply to bare-metal.

---
 drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +15 -0
index f2d126b110e8..fa718318568e 100644
@@ -2725,6 +2725,21 @@ static void gfx_v11_0_config_gfx_rs64(struct amdgpu_device *adev)
 					mec_hdr->ucode_start_addr_hi >> 2);
 	}
 	soc21_grbm_select(adev, 0, 0, 0, 0);
+
+	/* reset mec pipe */
+	tmp = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE0_RESET, 1);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE1_RESET, 1);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE2_RESET, 1);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE3_RESET, 1);
+	WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, tmp);
+
+	/* clear mec pipe reset */
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE0_RESET, 0);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE1_RESET, 0);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE2_RESET, 0);
+	tmp = REG_SET_FIELD(tmp, CP_MEC_RS64_CNTL, MEC_PIPE3_RESET, 0);
+	WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, tmp);
 }
 
 static int gfx_v11_0_wait_for_rlc_autoload_complete(struct amdgpu_device *adev)
Signature · 1 line
2.37.1
author maintainer
#3 lore raw

PATCH 4/19 drm/amdgpu: enable WPTR_POLL_ENABLE for sriov on sdma_v6_0

In reply to Alex Deucher

From: Horace Chen <[email protected]>

[Why]
Under SR-IOV, if VF is switched out then its doorbell will be disabled, SDMA rely on WPTR_POLL to get doorbells which was sent during VF switched-out time.

[How]
For SR-IOV, set SDMA WPTR_POLL_ENABLE to 1.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +4 -1
index 2bc1407e885e..fb31dc911cd7 100644
@@ -594,7 +594,10 @@ static int sdma_v6_0_gfx_resume(struct amdgpu_device *adev)
 		       lower_32_bits(ring->rptr_gpu_addr) & 0xFFFFFFFC);
 
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_QUEUE0_RB_CNTL, RPTR_WRITEBACK_ENABLE, 1);
-		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_QUEUE0_RB_CNTL, WPTR_POLL_ENABLE, 0);
+		if (amdgpu_sriov_vf(adev))
+			rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_QUEUE0_RB_CNTL, WPTR_POLL_ENABLE, 1);
+		else
+			rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_QUEUE0_RB_CNTL, WPTR_POLL_ENABLE, 0);
 		rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_QUEUE0_RB_CNTL, F32_WPTR_POLL_ENABLE, 1);
 
 		WREG32_SOC15_IP(GC, sdma_v6_0_get_reg_offset(adev, i, regSDMA0_QUEUE0_RB_BASE), ring->gpu_addr >> 8);
Signature · 1 line
2.37.1
author maintainer
#4 lore raw

PATCH 5/19 drm/amdgpu: refine virtualization psp fw skip check

In reply to Alex Deucher

From: Horace Chen <[email protected]>

SR-IOV may need to load different firmwares for different ASIC inside VF.
So create a new function in amdgpu_virt to check whether FW load needs to be skipped.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c  | 17 +++-----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 29 ++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  2 ++
 3 files changed, 34 insertions(+), 14 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +3 -14
index b007371df59c..d00a9fe542bf 100644
@@ -340,11 +340,12 @@ static int psp_init_sriov_microcode(struct psp_context *psp)
 		ret = psp_init_cap_microcode(psp, "aldebaran");
 		ret &= psp_init_ta_microcode(psp, "aldebaran");
 		break;
+	case IP_VERSION(13, 0, 0):
+		break;
 	default:
 		BUG();
 		break;
 	}
-
 	return ret;
 }
 
@@ -2412,19 +2413,7 @@ static bool fw_load_skip_check(struct psp_context *psp,
 		return true;
 
 	if (amdgpu_sriov_vf(psp->adev) &&
-	   (ucode->ucode_id == AMDGPU_UCODE_ID_SDMA0
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA1
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA2
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA3
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA4
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA5
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA6
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SDMA7
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_RLC_G
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM
-	    || ucode->ucode_id == AMDGPU_UCODE_ID_SMC))
+	    amdgpu_virt_fw_load_skip_check(psp->adev, ucode->ucode_id))
 		/*skip ucode loading in SRIOV VF */
 		return true;
 
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +29 -0
index 0707e4338e65..79f1e37caaec 100644
@@ -809,6 +809,35 @@ enum amdgpu_sriov_vf_mode amdgpu_virt_get_sriov_vf_mode(struct amdgpu_device *ad
 	return mode;
 }
 
+bool amdgpu_virt_fw_load_skip_check(struct amdgpu_device *adev, uint32_t ucode_id)
+{
+	/* this version doesn't support sriov autoload */
+	if (adev->ip_versions[MP0_HWIP][0] == IP_VERSION(13, 0, 0)) {
+		if (ucode_id == AMDGPU_UCODE_ID_VCN1 ||
+		    ucode_id == AMDGPU_UCODE_ID_VCN)
+			return false;
+		else
+			return true;
+	}
+
+	if (ucode_id == AMDGPU_UCODE_ID_SDMA0
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA1
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA2
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA3
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA4
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA5
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA6
+	    || ucode_id == AMDGPU_UCODE_ID_SDMA7
+	    || ucode_id == AMDGPU_UCODE_ID_RLC_G
+	    || ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_CNTL
+	    || ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_GPM_MEM
+	    || ucode_id == AMDGPU_UCODE_ID_RLC_RESTORE_LIST_SRM_MEM
+	    || ucode_id == AMDGPU_UCODE_ID_SMC)
+		return true;
+
+	return false;
+}
+
 void amdgpu_virt_update_sriov_video_codec(struct amdgpu_device *adev,
 			struct amdgpu_video_codec_info *encode, uint32_t encode_array_size,
 			struct amdgpu_video_codec_info *decode, uint32_t decode_array_size)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +2 -0
index 239f232f9c02..cd6fce05978f 100644
@@ -343,4 +343,6 @@ void amdgpu_sriov_wreg(struct amdgpu_device *adev,
 		       u32 acc_flags, u32 hwip);
 u32 amdgpu_sriov_rreg(struct amdgpu_device *adev,
 		      u32 offset, u32 acc_flags, u32 hwip);
+bool amdgpu_virt_fw_load_skip_check(struct amdgpu_device *adev,
+			uint32_t ucode_id);
 #endif
Signature · 1 line
2.37.1
author maintainer
#5 lore raw

PATCH 6/19 drm/amdgpu: sriov remove vcn_4_0 and jpeg_4_0

In reply to Alex Deucher

From: Horace Chen <[email protected]>

SRIOV needs to initialize mmsch instead of multimedia engines directly. So currently remove them for SR-IOV until the code and firmwares are ready.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +4 -2
index 854ca8d4505b..06b2d1830091 100644
@@ -1908,8 +1908,10 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev)
 		case IP_VERSION(4, 0, 0):
 		case IP_VERSION(4, 0, 2):
 		case IP_VERSION(4, 0, 4):
-			amdgpu_device_ip_block_add(adev, &vcn_v4_0_ip_block);
-			amdgpu_device_ip_block_add(adev, &jpeg_v4_0_ip_block);
+			if (!amdgpu_sriov_vf(adev)) {
+				amdgpu_device_ip_block_add(adev, &vcn_v4_0_ip_block);
+				amdgpu_device_ip_block_add(adev, &jpeg_v4_0_ip_block);
+			}
 			break;
 		default:
 			dev_err(adev->dev,
Signature · 1 line
2.37.1
author maintainer
#6 lore raw

PATCH 7/19 drm/amdgpu: Support PSP 13.0.10 on SR-IOV

In reply to Alex Deucher

From: Horace Chen <[email protected]>

Add support for PSP 13.0.10 for SR-IOV VF

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c  | 12 ++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 62 ++++++++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  3 ++
 3 files changed, 55 insertions(+), 22 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +10 -2
index d00a9fe542bf..bae4154455cf 100644
@@ -328,19 +328,27 @@ static int psp_init_sriov_microcode(struct psp_context *psp)
 
 	switch (adev->ip_versions[MP0_HWIP][0]) {
 	case IP_VERSION(9, 0, 0):
+		adev->virt.autoload_ucode_id = AMDGPU_UCODE_ID_CP_MEC2;
 		ret = psp_init_cap_microcode(psp, "vega10");
 		break;
 	case IP_VERSION(11, 0, 9):
+		adev->virt.autoload_ucode_id = AMDGPU_UCODE_ID_CP_MEC2;
 		ret = psp_init_cap_microcode(psp, "navi12");
 		break;
 	case IP_VERSION(11, 0, 7):
+		adev->virt.autoload_ucode_id = AMDGPU_UCODE_ID_CP_MEC2;
 		ret = psp_init_cap_microcode(psp, "sienna_cichlid");
 		break;
 	case IP_VERSION(13, 0, 2):
+		adev->virt.autoload_ucode_id = AMDGPU_UCODE_ID_CP_MEC2;
 		ret = psp_init_cap_microcode(psp, "aldebaran");
 		ret &= psp_init_ta_microcode(psp, "aldebaran");
 		break;
 	case IP_VERSION(13, 0, 0):
+		adev->virt.autoload_ucode_id = 0;
+		break;
+	case IP_VERSION(13, 0, 10):
+		adev->virt.autoload_ucode_id = AMDGPU_UCODE_ID_CP_MES1_DATA;
 		break;
 	default:
 		BUG();
@@ -768,6 +776,7 @@ static bool psp_skip_tmr(struct psp_context *psp)
 	case IP_VERSION(11, 0, 9):
 	case IP_VERSION(11, 0, 7):
 	case IP_VERSION(13, 0, 2):
+	case IP_VERSION(13, 0, 10):
 		return true;
 	default:
 		return false;
@@ -2414,7 +2423,6 @@ static bool fw_load_skip_check(struct psp_context *psp,
 
 	if (amdgpu_sriov_vf(psp->adev) &&
 	    amdgpu_virt_fw_load_skip_check(psp->adev, ucode->ucode_id))
-		/*skip ucode loading in SRIOV VF */
 		return true;
 
 	if (psp->autoload_supported &&
@@ -2488,7 +2496,7 @@ static int psp_load_non_psp_fw(struct psp_context *psp)
 
 		/* Start rlc autoload after psp recieved all the gfx firmware */
 		if (psp->autoload_supported && ucode->ucode_id == (amdgpu_sriov_vf(adev) ?
-		    AMDGPU_UCODE_ID_CP_MEC2 : AMDGPU_UCODE_ID_RLC_G)) {
+		    adev->virt.autoload_ucode_id : AMDGPU_UCODE_ID_RLC_G)) {
 			ret = psp_rlc_autoload_start(psp);
 			if (ret) {
 				DRM_ERROR("Failed to start rlc autoload\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +42 -20

80 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +3 -0
index cd6fce05978f..d94c31e68a14 100644
@@ -253,6 +253,9 @@ struct amdgpu_virt {
 	uint32_t decode_max_frame_pixels;
 	uint32_t encode_max_dimension_pixels;
 	uint32_t encode_max_frame_pixels;
+
+	/* the ucode id to signal the autoload */
+	uint32_t autoload_ucode_id;
 };
 
 struct amdgpu_video_codec_info;
Signature · 1 line
2.37.1
author maintainer
#7 lore raw

PATCH 8/19 drm/amdgpu: Use PSP program IH_RB_CNTL registers under SRIOV

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
With L1 Policy applied, IH_RB_CNTL/RING cannot be accessed by VF.

[How]
Use PSP program IH_RB_CNTL in VF.

---
 drivers/gpu/drm/amd/amdgpu/ih_v6_0.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/ih_v6_0.c +23 -3
index 085e613f3646..7cd79a3844b2 100644
@@ -105,7 +105,13 @@ force_update_wptr_for_self_int(struct amdgpu_device *adev,
 	ih_rb_cntl = REG_SET_FIELD(ih_rb_cntl, IH_RB_CNTL_RING1,
 				   RB_USED_INT_THRESHOLD, threshold);
 
-	WREG32_SOC15(OSSSYS, 0, regIH_RB_CNTL_RING1, ih_rb_cntl);
+	if (amdgpu_sriov_vf(adev) && amdgpu_sriov_reg_indirect_ih(adev)) {
+		if (psp_reg_program(&adev->psp, PSP_REG_IH_RB_CNTL_RING1, ih_rb_cntl))
+			return;
+	} else {
+		WREG32_SOC15(OSSSYS, 0, regIH_RB_CNTL_RING1, ih_rb_cntl);
+	}
+
 	WREG32_SOC15(OSSSYS, 0, regIH_CNTL2, ih_cntl);
 }
 
@@ -132,7 +138,13 @@ static int ih_v6_0_toggle_ring_interrupts(struct amdgpu_device *adev,
 	/* enable_intr field is only valid in ring0 */
 	if (ih == &adev->irq.ih)
 		tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, ENABLE_INTR, (enable ? 1 : 0));
-	WREG32(ih_regs->ih_rb_cntl, tmp);
+
+	if (amdgpu_sriov_vf(adev) && amdgpu_sriov_reg_indirect_ih(adev)) {
+		if (psp_reg_program(&adev->psp, ih_regs->psp_reg_id, tmp))
+			return -ETIMEDOUT;
+	} else {
+		WREG32(ih_regs->ih_rb_cntl, tmp);
+	}
 
 	if (enable) {
 		ih->enabled = true;
@@ -242,7 +254,15 @@ static int ih_v6_0_enable_ring(struct amdgpu_device *adev,
 		tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, WPTR_OVERFLOW_ENABLE, 0);
 		tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, RB_FULL_DRAIN_ENABLE, 1);
 	}
-	WREG32(ih_regs->ih_rb_cntl, tmp);
+
+	if (amdgpu_sriov_vf(adev) && amdgpu_sriov_reg_indirect_ih(adev)) {
+		if (psp_reg_program(&adev->psp, ih_regs->psp_reg_id, tmp)) {
+			DRM_ERROR("PSP program IH_RB_CNTL failed!\n");
+			return -ETIMEDOUT;
+		}
+	} else {
+		WREG32(ih_regs->ih_rb_cntl, tmp);
+	}
 
 	if (ih == &adev->irq.ih) {
 		/* set the ih ring 0 writeback address whether it's enabled or not */
Signature · 1 line
2.37.1
author maintainer
#8 lore raw

PATCH 9/19 drm/amdgpu: Skip the program of MMMC_VM_AGP_* in SRIOV on MMHUB v3_0_0

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
VF should not program these registers, the value were defined in the host.

[How]
Skip writing them in SRIOV environment and program them on host side.

---
 drivers/gpu/drm/amd/amdgpu/mmhub_v3_0.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/mmhub_v3_0.c +5 -5
index bc11b2de37ae..a1d26c4d80b8 100644
@@ -169,17 +169,17 @@ static void mmhub_v3_0_init_system_aperture_regs(struct amdgpu_device *adev)
 	uint64_t value;
 	uint32_t tmp;
 
-	/* Disable AGP. */
-	WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_BASE, 0);
-	WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_TOP, 0);
-	WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_BOT, 0x00FFFFFF);
-
 	if (!amdgpu_sriov_vf(adev)) {
 		/*
 		 * the new L1 policy will block SRIOV guest from writing
 		 * these regs, and they will be programed at host.
 		 * so skip programing these regs.
 		 */
+		/* Disable AGP. */
+		WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_BASE, 0);
+		WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_TOP, 0);
+		WREG32_SOC15(MMHUB, 0, regMMMC_VM_AGP_BOT, 0x00FFFFFF);
+
 		/* Program the system aperture low logical page number. */
 		WREG32_SOC15(MMHUB, 0, regMMMC_VM_SYSTEM_APERTURE_LOW_ADDR,
 			     adev->gmc.vram_start >> 18);
Signature · 1 line
2.37.1
author maintainer
#9 lore raw

PATCH 10/19 drm/amdgpu: skip "Issue additional private vm invalidation to MMHUB" on SRIOV

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
vm_l2_bank_select_reserved_cid2 is a PF_only register that cannot be programmed by VF. This feature is only support HDP using GPUVM page tables to access FB memory which should be disabled on SRIOV.

[How]
Disable the feature on VF.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +2 -1
index 497f52283a30..592729cb0f48 100644
@@ -234,7 +234,8 @@ static void gmc_v11_0_flush_vm_hub(struct amdgpu_device *adev, uint32_t vmid,
 
 	/* Issue additional private vm invalidation to MMHUB */
 	if ((vmhub != AMDGPU_GFXHUB_0) &&
-	    (hub->vm_l2_bank_select_reserved_cid2)) {
+	    (hub->vm_l2_bank_select_reserved_cid2) &&
+		!amdgpu_sriov_vf(adev)) {
 		inv_req = RREG32_NO_KIQ(hub->vm_l2_bank_select_reserved_cid2);
 		/* bit 25: RSERVED_CACHE_PRIVATE_INVALIDATION */
 		inv_req |= (1 << 25);
Signature · 1 line
2.37.1
author maintainer
#10 lore raw

PATCH 11/19 drm/amdgpu: Skip the VRAM base offset on SRIOV

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
As VF cannot read MMMC_VM_FB_OFFSET with L1 Policy(read 0xffffffff). It leads to driver get the incorrect vram base offset.

[How]
Since SR-IOV is dGPU only, skip reading this register and set the fb_offest to 0.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +4 -1
index 592729cb0f48..846ccb6cf07d 100644
@@ -649,7 +649,10 @@ static void gmc_v11_0_vram_gtt_location(struct amdgpu_device *adev,
 	amdgpu_gmc_gart_location(adev, mc);
 
 	/* base offset of vram pages */
-	adev->vm_manager.vram_base_offset = adev->mmhub.funcs->get_mc_fb_offset(adev);
+	if (amdgpu_sriov_vf(adev))
+		adev->vm_manager.vram_base_offset = 0;
+	else
+		adev->vm_manager.vram_base_offset = adev->mmhub.funcs->get_mc_fb_offset(adev);
 }
 
 /**
Signature · 1 line
2.37.1
author maintainer
#11 lore raw

PATCH 12/19 drm/amdgpu: Skip program SDMA0_SEM_WAIT_FAIL_TIMER_CNTL under SRIOV VF

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
As SDMA0_SEM_WAIT_FAIL_TIMER_CNTL is a PF-only register, L1 would block this register for VF access.

[How]
VF do not program it.

---
 drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +2 -1
index fb31dc911cd7..7ae572a08cb3 100644
@@ -560,7 +560,8 @@ static int sdma_v6_0_gfx_resume(struct amdgpu_device *adev)
 	for (i = 0; i < adev->sdma.num_instances; i++) {
 		ring = &adev->sdma.instance[i].ring;
 
-		WREG32_SOC15_IP(GC, sdma_v6_0_get_reg_offset(adev, i, regSDMA0_SEM_WAIT_FAIL_TIMER_CNTL), 0);
+		if (!amdgpu_sriov_vf(adev))
+			WREG32_SOC15_IP(GC, sdma_v6_0_get_reg_offset(adev, i, regSDMA0_SEM_WAIT_FAIL_TIMER_CNTL), 0);
 
 		/* Set ring buffer size in dwords */
 		rb_bufsz = order_base_2(ring->ring_size / 4);
Signature · 1 line
2.37.1
author maintainer
#12 lore raw

PATCH 13/19 drm/amdgpu: Use RLCG to program GRBM_GFX_CNTL during full access time

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
KIQ register init requires GRBM_GFX_CNTL to select KIQ.

[How]
As RLCG accessing registers will save the data of GRBM_GFX_CNTL and restore it. Use RLCG indirect accessing register method to select grbm instead of mmio directly access.

---
 drivers/gpu/drm/amd/amdgpu/soc21.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/soc21.c +1 -1
index 54bce19a52ec..47ef671543b3 100644
@@ -179,7 +179,7 @@ void soc21_grbm_select(struct amdgpu_device *adev,
 	grbm_gfx_cntl = REG_SET_FIELD(grbm_gfx_cntl, GRBM_GFX_CNTL, VMID, vmid);
 	grbm_gfx_cntl = REG_SET_FIELD(grbm_gfx_cntl, GRBM_GFX_CNTL, QUEUEID, queue);
 
-	WREG32(SOC15_REG_OFFSET(GC, 0, regGRBM_GFX_CNTL), grbm_gfx_cntl);
+	WREG32_SOC15(GC, 0, regGRBM_GFX_CNTL, grbm_gfx_cntl);
 }
 
 static void soc21_vga_set_state(struct amdgpu_device *adev, bool state)
Signature · 1 line
2.37.1
author maintainer
#13 lore raw

PATCH 14/19 drm/admgpu: Skip CG/PG on SOC21 under SRIOV VF

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
There is no CG(Clock Gating)/PG(Power Gating) requirement on SRIOV VF. For multi VF, VF should not enable any CG/PG features.
For one VF, PF will program CG/PG related registers.

[How]
Do not set any cg/pg flag bit at early init under sriov.

---
 drivers/gpu/drm/amd/amdgpu/soc21.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/soc21.c +4 -0
index 47ef671543b3..34a267265971 100644
@@ -582,6 +582,10 @@ static int soc21_common_early_init(void *handle)
 			AMD_PG_SUPPORT_JPEG |
 			AMD_PG_SUPPORT_ATHUB |
 			AMD_PG_SUPPORT_MMHUB;
+		if (amdgpu_sriov_vf(adev)) {
+			adev->cg_flags = 0;
+			adev->pg_flags = 0;
+		}
 		adev->external_rev_id = adev->rev_id + 0x1; // TODO: need update
 		break;
 	case IP_VERSION(11, 0, 2):
Signature · 1 line
2.37.1
author maintainer
#14 lore raw

PATCH 15/19 drm/amd: Skip smu_v13 register irq on SRIOV VF

In reply to Alex Deucher

From: Yifan Zha <[email protected]>

[Why]
VF do not need to handle SMU IRQ state.
L1 Policy will block VF access THM_THERMAL_INT_CTRL and MP1_SMN_IH_SW_INT/CNTL.

[How]
Skip smu_v13 init register_irq_handler under SRIOV VF. And add irq_src check in enable/disable thermal alert to avoid thermal alert enable/disable fail.

---
 drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c +9 -0
index 4fee391f4cfa..bebcd668a766 100644
@@ -1108,6 +1108,9 @@ int smu_v13_0_enable_thermal_alert(struct smu_context *smu)
 {
 	int ret = 0;
 
+	if (!smu->irq_source.num_types)
+		return 0;
+
 	ret = amdgpu_irq_get(smu->adev, &smu->irq_source, 0);
 	if (ret)
 		return ret;
@@ -1117,6 +1120,9 @@ int smu_v13_0_enable_thermal_alert(struct smu_context *smu)
 
 int smu_v13_0_disable_thermal_alert(struct smu_context *smu)
 {
+	if (!smu->irq_source.num_types)
+		return 0;
+
 	return amdgpu_irq_put(smu->adev, &smu->irq_source, 0);
 }
 
@@ -1488,6 +1494,9 @@ int smu_v13_0_register_irq_handler(struct smu_context *smu)
 	struct amdgpu_irq_src *irq_src = &smu->irq_source;
 	int ret = 0;
 
+	if (amdgpu_sriov_vf(adev))
+		return 0;
+
 	irq_src->num_types = 1;
 	irq_src->funcs = &smu_v13_0_irq_funcs;
 
Signature · 1 line
2.37.1
author maintainer
#15 lore raw

PATCH 16/19 drm/amdgpu/vcn: Disable CG/PG for SRIOV

In reply to Alex Deucher

From: Jane Jian <[email protected]>

For sriov, CG and MG are controlled from hypervisor side, no need to manage them again in ip init

---
 drivers/gpu/drm/amd/amdgpu/soc21.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/soc21.c +5 -0
index 34a267265971..a26c5723c46e 100644
@@ -637,6 +637,11 @@ static int soc21_common_early_init(void *handle)
 		adev->pg_flags = AMD_PG_SUPPORT_VCN |
 			AMD_PG_SUPPORT_VCN_DPG |
 			AMD_PG_SUPPORT_JPEG;
+		if (amdgpu_sriov_vf(adev)) {
+			/* hypervisor control CG and PG enablement */
+			adev->cg_flags = 0;
+			adev->pg_flags = 0;
+		}
 		adev->external_rev_id = adev->rev_id + 0x20;
 		break;
 	default:
Signature · 1 line
2.37.1
author maintainer
#16 lore raw

PATCH 17/19 drm/amdgpu/vcn: Add vcn/vcn1 in white list to load its firmware under sriov

In reply to Alex Deucher

From: Jane Jian <[email protected]>

Previously since vcn0/vcn1 are not enabled, loading firmware is skipped. Now add firmware loading back since vcn0/vcn1 has already been enabled on sriov

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +3 -1
index a7de2c2b0cae..e4af40b9a8aa 100644
@@ -835,7 +835,9 @@ bool amdgpu_virt_fw_load_skip_check(struct amdgpu_device *adev, uint32_t ucode_i
 		|| ucode_id == AMDGPU_UCODE_ID_CP_MES
 		|| ucode_id == AMDGPU_UCODE_ID_CP_MES_DATA
 		|| ucode_id == AMDGPU_UCODE_ID_CP_MES1
-		|| ucode_id == AMDGPU_UCODE_ID_CP_MES1_DATA)
+		|| ucode_id == AMDGPU_UCODE_ID_CP_MES1_DATA
+		|| ucode_id == AMDGPU_UCODE_ID_VCN1
+		|| ucode_id == AMDGPU_UCODE_ID_VCN)
 			return false;
 		else
 			return true;
Signature · 1 line
2.37.1
author maintainer
#17 lore raw

PATCH 18/19 drm/amdgpu/vcn: Add sriov VCN v4_0 unified queue support

In reply to Alex Deucher

From: Jane Jian <[email protected]>

Enable unified queue support for sriov, abandon all previous multi-queue settings

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h       |  13 +
 drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c         | 297 +++++++++++++++++-
 3 files changed, 300 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +2 -3
index 06b2d1830091..9fa2a5ceb77d 100644
@@ -1908,10 +1908,9 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev)
 		case IP_VERSION(4, 0, 0):
 		case IP_VERSION(4, 0, 2):
 		case IP_VERSION(4, 0, 4):
-			if (!amdgpu_sriov_vf(adev)) {
-				amdgpu_device_ip_block_add(adev, &vcn_v4_0_ip_block);
+			amdgpu_device_ip_block_add(adev, &vcn_v4_0_ip_block);
+			if (!amdgpu_sriov_vf(adev))
 				amdgpu_device_ip_block_add(adev, &jpeg_v4_0_ip_block);
-			}
 			break;
 		default:
 			dev_err(adev->dev,
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h +13 -0
index 60c608144480..80b7a6cfd026 100644
@@ -161,6 +161,7 @@
 #define AMDGPU_VCN_SW_RING_FLAG		(1 << 9)
 #define AMDGPU_VCN_FW_LOGGING_FLAG	(1 << 10)
 #define AMDGPU_VCN_SMU_VERSION_INFO_FLAG (1 << 11)
+#define AMDGPU_VCN_VF_RB_SETUP_FLAG (1 << 12)
 
 #define AMDGPU_VCN_IB_FLAG_DECODE_BUFFER	0x00000001
 #define AMDGPU_VCN_CMD_FLAG_MSG_BUFFER		0x00000001
@@ -317,12 +318,24 @@ struct amdgpu_fw_shared {
 	struct amdgpu_fw_shared_smu_interface_info smu_interface_info;
 };
 
+struct amdgpu_fw_shared_rb_setup {
+	uint32_t is_rb_enabled_flags;
+	uint32_t rb_addr_lo;
+	uint32_t rb_addr_hi;
+	uint32_t  rb_size;
+	uint32_t  rb4_addr_lo;
+	uint32_t  rb4_addr_hi;
+	uint32_t  rb4_size;
+	uint32_t  reserved[6];
+};
+
 struct amdgpu_vcn4_fw_shared {
 	uint32_t present_flag_0;
 	uint8_t pad[12];
 	struct amdgpu_fw_shared_unified_queue_struct sq;
 	uint8_t pad1[8];
 	struct amdgpu_fw_shared_fw_logging fw_log;
+	struct amdgpu_fw_shared_rb_setup rb_setup;
 };
 
 struct amdgpu_vcn_fwlog {
drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c +285 -12

392 lines not on this page · raw message

Signature · 1 line
2.37.1
author maintainer
#18 lore raw

PATCH 19/19 drm/amdgpu/vcn: Add MMSCH v4_0 support for sriov

In reply to Alex Deucher

From: Jane Jian <[email protected]>

These structures are basically ported from MMSCH v3_0, besides, added RB and RB4 enablement flag to support unified queue

---
 drivers/gpu/drm/amd/amdgpu/mmsch_v4_0.h | 140 ++++++++++++++++++++++++
 1 file changed, 140 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/mmsch_v4_0.h
drivers/gpu/drm/amd/amdgpu/mmsch_v4_0.h new file +140 -0
new file mode 100644
index 000000000000..f772bb499f3e
@@ -0,0 +1,140 @@
+/*
+ * Copyright 2022 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __MMSCH_V4_0_H__
+#define __MMSCH_V4_0_H__
+
+#include "amdgpu_vcn.h"
+
+#define MMSCH_VERSION_MAJOR	4
+#define MMSCH_VERSION_MINOR	0
+#define MMSCH_VERSION	(MMSCH_VERSION_MAJOR << 16 | MMSCH_VERSION_MINOR)
+
+#define RB_ENABLED (1 << 0)
+#define RB4_ENABLED (1 << 1)
+#define MMSCH_DOORBELL_OFFSET 0x8
+
+#define MMSCH_VF_ENGINE_STATUS__PASS 0x1
+
+#define MMSCH_VF_MAILBOX_RESP__OK 0x1
+#define MMSCH_VF_MAILBOX_RESP__INCOMPLETE 0x2
+
+enum mmsch_v4_0_command_type {
+	MMSCH_COMMAND__DIRECT_REG_WRITE = 0,
+	MMSCH_COMMAND__DIRECT_REG_POLLING = 2,
+	MMSCH_COMMAND__DIRECT_REG_READ_MODIFY_WRITE = 3,
+	MMSCH_COMMAND__INDIRECT_REG_WRITE = 8,
+	MMSCH_COMMAND__END = 0xf
+};
+
+struct mmsch_v4_0_table_info {
+	uint32_t init_status;
+	uint32_t table_offset;
+	uint32_t table_size;
+};
+
+struct mmsch_v4_0_init_header {
+	uint32_t version;
+	uint32_t total_size;
+	struct mmsch_v4_0_table_info inst[AMDGPU_MAX_VCN_INSTANCES];
+	struct mmsch_v4_0_table_info jpegdec;
+};
+
+struct mmsch_v4_0_cmd_direct_reg_header {
+	uint32_t reg_offset   : 28;
+	uint32_t command_type : 4;
+};
+
+struct mmsch_v4_0_cmd_indirect_reg_header {
+	uint32_t reg_offset    : 20;
+	uint32_t reg_idx_space : 8;
+	uint32_t command_type  : 4;
+};
+
+struct mmsch_v4_0_cmd_direct_write {
+	struct mmsch_v4_0_cmd_direct_reg_header cmd_header;
+	uint32_t reg_value;
+};
+
+struct mmsch_v4_0_cmd_direct_read_modify_write {
+	struct mmsch_v4_0_cmd_direct_reg_header cmd_header;
+	uint32_t write_data;
+	uint32_t mask_value;
+};
+
+struct mmsch_v4_0_cmd_direct_polling {
+	struct mmsch_v4_0_cmd_direct_reg_header cmd_header;
+	uint32_t mask_value;
+	uint32_t wait_value;
+};
+
+struct mmsch_v4_0_cmd_end {
+	struct mmsch_v4_0_cmd_direct_reg_header cmd_header;
+};
+
+struct mmsch_v4_0_cmd_indirect_write {
+	struct mmsch_v4_0_cmd_indirect_reg_header cmd_header;
+	uint32_t reg_value;
+};
+
+#define MMSCH_V4_0_INSERT_DIRECT_RD_MOD_WT(reg, mask, data) { \
+	size = sizeof(struct mmsch_v4_0_cmd_direct_read_modify_write); \
+	size_dw = size / 4; \
+	direct_rd_mod_wt.cmd_header.reg_offset = reg; \
+	direct_rd_mod_wt.mask_value = mask; \
+	direct_rd_mod_wt.write_data = data; \
+	memcpy((void *)table_loc, &direct_rd_mod_wt, size); \
+	table_loc += size_dw; \
+	table_size += size_dw; \
+}
+
+#define MMSCH_V4_0_INSERT_DIRECT_WT(reg, value) { \
+	size = sizeof(struct mmsch_v4_0_cmd_direct_write); \
+	size_dw = size / 4; \
+	direct_wt.cmd_header.reg_offset = reg; \
+	direct_wt.reg_value = value; \
+	memcpy((void *)table_loc, &direct_wt, size); \
+	table_loc += size_dw; \
+	table_size += size_dw; \
+}
+
+#define MMSCH_V4_0_INSERT_DIRECT_POLL(reg, mask, wait) { \
+	size = sizeof(struct mmsch_v4_0_cmd_direct_polling); \
+	size_dw = size / 4; \
+	direct_poll.cmd_header.reg_offset = reg; \
+	direct_poll.mask_value = mask; \
+	direct_poll.wait_value = wait; \
+	memcpy((void *)table_loc, &direct_poll, size); \
+	table_loc += size_dw; \
+	table_size += size_dw; \
+}
+
+#define MMSCH_V4_0_INSERT_END() { \
+	size = sizeof(struct mmsch_v4_0_cmd_end); \
+	size_dw = size / 4; \
+	memcpy((void *)table_loc, &end, size); \
+	table_loc += size_dw; \
+	table_size += size_dw; \
+}
+
+#endif
Signature · 1 line
2.37.1
Acked-by on v1 1/19
#19 lore raw
In reply to Alex Deucher

Am 30.08.22 um 20:39 schrieb Alex Deucher:

Quoted 8 lines

From: Horace Chen <[email protected]>

For further chips we will use CHIP_IP_DISCOVERY, so add this support for virtualization

Quoted 24 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +2 -0
index 9be57389301b..0707e4338e65 100644
@@ -707,6 +707,7 @@ void amdgpu_detect_virtualization(struct amdgpu_device *adev)
  	case CHIP_SIENNA_CICHLID:
  	case CHIP_ARCTURUS:
  	case CHIP_ALDEBARAN:
+	case CHIP_IP_DISCOVERY:
  		reg = RREG32(mmRCC_IOV_FUNC_IDENTIFIER);
  		break;
  	default: /* other chip doesn't support SRIOV */
@@ -750,6 +751,7 @@ void amdgpu_detect_virtualization(struct amdgpu_device *adev)
  		case CHIP_NAVI10:
  		case CHIP_NAVI12:
  		case CHIP_SIENNA_CICHLID:
+		case CHIP_IP_DISCOVERY:
  			nv_set_virt_ops(adev);
  			/* try send GPU_INIT_DATA request to host */
  			amdgpu_virt_request_init_data(adev);