Skip to contentSkip to conversation

drm/amdgpu: improve the amdgpu device init progress in sriov mode

PATCH In review chong li 5 participants 18 messages Jul 8 – Jul 10

Subsystems (2)
lore
Share & tools
lore

chong li

author
#0 lore raw

PATCH 1/1 drm/amdgpu: improve the amdgpu device init progress in sriov mode

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
 2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
 #include <linux/apple-gmux.h>
 #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_client_event.h>
 #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 {
 	struct amdgpu_ip_block *ip_block;
 	struct pci_dev *parent;
-	bool total, skip_bios;
+	bool total, skip_bios, early_full_gpu_access = false;
 	uint32_t bios_flags;
 	int i, r;
 
 	amdgpu_device_enable_virtual_display(adev);
 
 	if (amdgpu_sriov_vf(adev)) {
-		r = amdgpu_virt_request_full_gpu(adev, true);
-		if (r)
-			return r;
+		early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+		if (early_full_gpu_access) {
+			r = amdgpu_virt_request_full_gpu(adev, true);
+			if (r)
+				return r;
+		}
 
 		r = amdgpu_virt_init_critical_region(adev);
 		if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 	if (!total)
 		return -ENODEV;
 
+	if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+		r = amdgpu_virt_request_full_gpu(adev, true);
+		if (r)
+			return r;
+	}
+
 	if (adev->gmc.xgmi.supported)
 		amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
 	} else if (req == IDH_REQ_GPU_INIT_DATA){
 		/* Dummy REQ_GPU_INIT_DATA handling */
 		r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-		/* version set to 0 since dummy */
-		adev->virt.req_init_data_ver = 0;	
+		/* Version is set to 1 since GPU_CRIT_REGION_V1 */
+		adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
 	}
 
 	return 0;
Signature · 12 lines
2.48.1

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver
sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. 
The amdgpu driver works normally.

I rewrote the background. Is anything still unclear?
Quoted 149 lines · Christian König
Quoted 36 lines · chong li
Quoted 6 lines

v2:
Some legacy ASICs do not support amdgpu_virt_request_init_data(). On platform ESXi, the PF can access VF VRAM only after the VF enters full gpu access mode.
Only move the full access request later when the required conditions are met

This is adjusting the driver to the hypervisor and not the hypervisor to the driver.

If the existing init order doesn't work on ESXi then that is an ESXi problem and needs to be fixed there.

So still absolutely clear NAK.

Quoted 22 lines

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization

background:

After the amdgpu driver sends GPU_INIT_DATA to the host SR-IOV driver, the host dumps the IP discovery/VBIOS/PF-VF exchange data into the VF FB. The VF can then read these data blocks without requesting full GPU access. This is the purpose of the GPU_INIT_DATA event in the SR-IOV init flow.

However, during ip_early_init, TTM is not initialized yet. The current amdgpu driver therefore falls back to reading these data through MM_INDEX/MM_DATA. That path requires register access, which still needs full GPU access.

As a result, even though amdgpu sends GPU_INIT_DATA and the host prepares the init data early, the full GPU access window is not reduced because the guest still needs full GPU access to copy the data.

Quoted 7 lines
Quoted 4 lines

To fix this, use amdgpu_device_read_fb_via_bar0() to copy the init data from VF FB before TTM is ready.
For this early copy, the guest no longer needs full GPU access so the full GPU access request can be moved later.

You are just explaining the chicken and egg problem here instead of giving an actual justification.

I guess there is no other technical reason to disallow accessing FB using visible BAR range. The only reason of going through HDP indirect path could be because the offsets could be outside of visible range. In SRIOV case, if host driver guarantees discovery offsets in visible FB range, there is no reason to disallow that.

Thanks,
Lijo

Quoted 98 lines

Regards,
Christian.

Quoted 94 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 32 +++++++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 +--
  2 files changed, 30 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +27 -4
index 610d82b79de3..ec353f4dd0d2 100644
@@ -38,6 +38,10 @@
  #include <linux/apple-gmux.h>
  #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
  #include <drm/drm_atomic_helper.h>
  #include <drm/drm_client_event.h>
  #include <drm/drm_crtc_helper.h>
@@ -1973,6 +1977,17 @@ static struct pci_dev *amdgpu_device_find_parent(struct amdgpu_device *adev)
  	return parent;
  }
 
+static bool amdgpu_device_delay_full_gpu_access(struct amdgpu_device 
+*adev) { #ifdef CONFIG_X86
+	return adev->virt.req_init_data_ver > 0 &&
+		(hypervisor_is_type(X86_HYPER_VMWARE) ||
+		hypervisor_is_type(X86_HYPER_MS_HYPERV));
+#else
+	return false;
+#endif
+}
+
  /**
   * amdgpu_device_ip_early_init - run early init for hardware IPs
   *
@@ -1987,16 +2002,19 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
  {
  	struct amdgpu_ip_block *ip_block;
  	struct pci_dev *parent;
-	bool total, skip_bios;
+	bool total, skip_bios, delay_full_gpu_access = false;
  	uint32_t bios_flags;
  	int i, r;
 
  	amdgpu_device_enable_virtual_display(adev);
 
  	if (amdgpu_sriov_vf(adev)) {
-		r = amdgpu_virt_request_full_gpu(adev, true);
-		if (r)
-			return r;
+		delay_full_gpu_access = amdgpu_device_delay_full_gpu_access(adev);
+		if (!delay_full_gpu_access) {
+			r = amdgpu_virt_request_full_gpu(adev, true);
+			if (r)
+				return r;
+		}
 
  		r = amdgpu_virt_init_critical_region(adev);
  		if (r)
@@ -2159,6 +2177,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
  	if (!total)
  		return -ENODEV;
 
+	if (amdgpu_sriov_vf(adev) && delay_full_gpu_access) {
+		r = amdgpu_virt_request_full_gpu(adev, true);
+		if (r)
+			return r;
+	}
+
  	if (adev->gmc.xgmi.supported)
  		amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
  	} else if (req == IDH_REQ_GPU_INIT_DATA){
  		/* Dummy REQ_GPU_INIT_DATA handling */
  		r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-		/* version set to 0 since dummy */
-		adev->virt.req_init_data_ver = 0;	
+		/* Version is set to 1 since GPU_CRIT_REGION_V1 */
+		adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
  	}
 
  	return 0;
maintainer
#1 lore raw
Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 75 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
 2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
 #include <linux/apple-gmux.h>
 #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_client_event.h>
 #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 {
 	struct amdgpu_ip_block *ip_block;
 	struct pci_dev *parent;
-	bool total, skip_bios;
+	bool total, skip_bios, early_full_gpu_access = false;
 	uint32_t bios_flags;
 	int i, r;
 
 	amdgpu_device_enable_virtual_display(adev);
 
 	if (amdgpu_sriov_vf(adev)) {
-		r = amdgpu_virt_request_full_gpu(adev, true);
-		if (r)
-			return r;
+		early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+		if (early_full_gpu_access) {
+			r = amdgpu_virt_request_full_gpu(adev, true);
+			if (r)
+				return r;
+		}
 
 		r = amdgpu_virt_init_critical_region(adev);
 		if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
 	if (!total)
 		return -ENODEV;
 
+	if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+		r = amdgpu_virt_request_full_gpu(adev, true);
+		if (r)
+			return r;
+	}
+
 	if (adev->gmc.xgmi.supported)
 		amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
 	} else if (req == IDH_REQ_GPU_INIT_DATA){
 		/* Dummy REQ_GPU_INIT_DATA handling */
 		r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-		/* version set to 0 since dummy */
-		adev->virt.req_init_data_ver = 0;	
+		/* Version is set to 1 since GPU_CRIT_REGION_V1 */
+		adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
 	}
 
 	return 0;
#2 lore raw

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake,
and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

I rewrote the background. Is anything still unclear?

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 77 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
 2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
 #include <linux/apple-gmux.h>
 #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_client_event.h>
 #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct
 amdgpu_device *adev)  {
      struct amdgpu_ip_block *ip_block;
      struct pci_dev *parent;
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
      uint32_t bios_flags;
      int i, r;
 
      amdgpu_device_enable_virtual_display(adev);
 
      if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
              r = amdgpu_virt_init_critical_region(adev);
              if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
      if (!total)
              return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
      if (adev->gmc.xgmi.supported)
              amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
      } else if (req == IDH_REQ_GPU_INIT_DATA){
              /* Dummy REQ_GPU_INIT_DATA handling */
              r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
      }
 
      return 0;
maintainer
#3 lore raw
Quoted 15 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake,
and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Quoted 7 lines

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

Regards,
Christian.

Quoted 129 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 77 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
 2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
 #include <linux/apple-gmux.h>
 #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_client_event.h>
 #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct
 amdgpu_device *adev)  {
      struct amdgpu_ip_block *ip_block;
      struct pci_dev *parent;
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
      uint32_t bios_flags;
      int i, r;
 
      amdgpu_device_enable_virtual_display(adev);
 
      if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
              r = amdgpu_virt_init_critical_region(adev);
              if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
      if (!total)
              return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
      if (adev->gmc.xgmi.supported)
              amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
      } else if (req == IDH_REQ_GPU_INIT_DATA){
              /* Dummy REQ_GPU_INIT_DATA handling */
              r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
      }
 
      return 0;
#4 lore raw
Quoted 19 lines · Christian König
Quoted 18 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake,
and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

Quoted 9 lines
Quoted 4 lines

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

Thanks,
Lijo

Quoted 133 lines

Regards,
Christian.

Quoted 129 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 77 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++----
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
  2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
  #include <linux/apple-gmux.h>
  #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
  #include <drm/drm_atomic_helper.h>
  #include <drm/drm_client_event.h>
  #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct
 amdgpu_device *adev)  {
       struct amdgpu_ip_block *ip_block;
       struct pci_dev *parent;
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
       uint32_t bios_flags;
       int i, r;
 
       amdgpu_device_enable_virtual_display(adev);
 
       if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
               r = amdgpu_virt_init_critical_region(adev);
               if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
       if (!total)
               return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
       if (adev->gmc.xgmi.supported)
               amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
       } else if (req == IDH_REQ_GPU_INIT_DATA){
               /* Dummy REQ_GPU_INIT_DATA handling */
               r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
       }
 
       return 0;
maintainer Reviewed-by Hawking Zhang
#5 lore raw
Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake,
and I have explained the situation at the end of the patch email.

I have pasted the content below:

         Hi, Christian and Lijo.

         Sorry, I made a mistake.
         Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

         I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

         I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 138 lines

Thanks,
Lijo

Quoted 133 lines

Regards,
Christian.

Quoted 129 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 77 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++++++++++++++++---- drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--

  2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
   #include <linux/apple-gmux.h>
   #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
   #include <drm/drm_atomic_helper.h>
   #include <drm/drm_client_event.h>
   #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int amdgpu_device_ip_early_init(struct
 amdgpu_device *adev)  {
        struct amdgpu_ip_block *ip_block;
        struct pci_dev *parent;
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
        uint32_t bios_flags;
        int i, r;
 
        amdgpu_device_enable_virtual_display(adev);
 
        if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                r = amdgpu_virt_init_critical_region(adev);
                if (r)
@@ -2159,6 +2166,12 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev)
        if (!total)
                return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
        if (adev->gmc.xgmi.supported)
                amdgpu_xgmi_early_init(adev);
 
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct amdgpu_device *adev,
        } else if (req == IDH_REQ_GPU_INIT_DATA){
                /* Dummy REQ_GPU_INIT_DATA handling */
                r = xgpu_ai_poll_msg(adev, IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
        }
 
        return 0;
Reviewed-by Hawking Zhang
#6 lore raw

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

        static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
        {
                ...
                case CHIP_VEGA20:
                case CHIP_ARCTURUS:
                case CHIP_ALDEBARAN:
                        soc15_set_virt_ops(adev);
                        break;
                case CHIP_NAVI10:
                case CHIP_NAVI12:
                case CHIP_SIENNA_CICHLID:
                case CHIP_IP_DISCOVERY:
                        nv_set_virt_ops(adev);
                        /* try send GPU_INIT_DATA request to host */
                        amdgpu_virt_request_init_data(adev);
                        break;
                ...
        }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
  2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
  #include <linux/apple-gmux.h>
  #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
  #include <drm/drm_atomic_helper.h>
  #include <drm/drm_client_event.h>
  #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int

amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
       struct amdgpu_ip_block *ip_block;
       struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
       uint32_t bios_flags;
       int i, r;
 
       amdgpu_device_enable_virtual_display(adev);
 
       if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver
+== 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
               r = amdgpu_virt_init_critical_region(adev);
               if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
       if (!total)
               return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
       if (adev->gmc.xgmi.supported)
               amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct
 amdgpu_device *adev,
       } else if (req == IDH_REQ_GPU_INIT_DATA){
               /* Dummy REQ_GPU_INIT_DATA handling */
               r = xgpu_ai_poll_msg(adev,
 IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
       }
       return 0;
maintainer
#7 lore raw

Hi Chong,

Quoted 7 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

Quoted 6 lines

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 240 lines


        static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
        {
                ...
                case CHIP_VEGA20:
                case CHIP_ARCTURUS:
                case CHIP_ALDEBARAN:
                        soc15_set_virt_ops(adev);
                        break;
                case CHIP_NAVI10:
                case CHIP_NAVI12:
                case CHIP_SIENNA_CICHLID:
                case CHIP_IP_DISCOVERY:
                        nv_set_virt_ops(adev);
                        /* try send GPU_INIT_DATA request to host */
                        amdgpu_virt_request_init_data(adev);
                        break;
                ...
        }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
  2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
  #include <linux/apple-gmux.h>
  #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
  #include <drm/drm_atomic_helper.h>
  #include <drm/drm_client_event.h>
  #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int

amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
       struct amdgpu_ip_block *ip_block;
       struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
       uint32_t bios_flags;
       int i, r;
 
       amdgpu_device_enable_virtual_display(adev);
 
       if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver
+== 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
               r = amdgpu_virt_init_critical_region(adev);
               if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
       if (!total)
               return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
       if (adev->gmc.xgmi.supported)
               amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct
 amdgpu_device *adev,
       } else if (req == IDH_REQ_GPU_INIT_DATA){
               /* Dummy REQ_GPU_INIT_DATA handling */
               r = xgpu_ai_poll_msg(adev,
 IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
       }
       return 0;
#8 lore raw

AMD General

Quoted 37 lines

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:49 PM
To: Li, Chong(Alan) <[email protected]>; Lazar, Lijo <[email protected]>; amd- [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi Chong,

Quoted 5 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related

to the ASIC type, not the host platform:

Quoted 5 lines

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not

send amdgpu_virt_request_init_data() to the host.

Quoted 2 lines

For those ASICs, the host dumps the early init data only after the guest requests full

GPU access.

Quoted 1 line

Therefore, those ASICs still need to request full GPU access before the guest driver

can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

Quoted 6 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Hi Christian, that patch " drm/amdgpu: invoke req full access early enough " is not correct since FB access right has already been granted by the init data event. Maybe that patch was only intended for the initial SIENNA_CICHLID bring-up.

Emily Deng
Best Wishes

Quoted 253 lines
Quoted 36 lines


        static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
        {
                ...
                case CHIP_VEGA20:
                case CHIP_ARCTURUS:
                case CHIP_ALDEBARAN:
                        soc15_set_virt_ops(adev);
                        break;
                case CHIP_NAVI10:
                case CHIP_NAVI12:
                case CHIP_SIENNA_CICHLID:
                case CHIP_IP_DISCOVERY:
                        nv_set_virt_ops(adev);
                        /* try send GPU_INIT_DATA request to host */
                        amdgpu_virt_request_init_data(adev);
                        break;
                ...
        }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 10 lines
Quoted 9 lines · Lijo
Quoted 6 lines · Christian König
Quoted 5 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow

the same path.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I verified this patch with KVM in my local environment and with ESXi on a

borrowed server.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver

sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

Quoted 13 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev-

virt.req_init_data_ver == 0);".

Quoted 5 lines
Quoted 3 lines

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation

is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

Quoted 2 lines

No, this was intentionally changed in 2020 because of a new feature. See this

commit here:

Quoted 13 lines

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

Quoted 5 lines

So as far as I can see this change here will break older hypervisor versions and that

is absolutely clear reason to NAK it.

Quoted 36 lines

Regards,
Christian.

Quoted 32 lines

Thanks,
Lijo

Quoted 28 lines

Regards,
Christian.

Quoted 25 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 3 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not

sent.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an

ESXi specific change.

Quoted 94 lines
Quoted 93 lines
Quoted 92 lines
Quoted 91 lines

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 82 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
  2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
  #include <linux/apple-gmux.h>
  #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
  #include <drm/drm_atomic_helper.h>
  #include <drm/drm_client_event.h>
  #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@ static

int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
       struct amdgpu_ip_block *ip_block;
       struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
       uint32_t bios_flags;
       int i, r;
 
       amdgpu_device_enable_virtual_display(adev);
 
       if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
               r = amdgpu_virt_init_critical_region(adev);
               if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
       if (!total)
               return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
       if (adev->gmc.xgmi.supported)
               amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct
 amdgpu_device *adev,
       } else if (req == IDH_REQ_GPU_INIT_DATA){
               /* Dummy REQ_GPU_INIT_DATA handling */
               r = xgpu_ai_poll_msg(adev,
 IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
       }
       return 0;
#9 lore raw
In reply to Christian König
Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 248 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 240 lines


         static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
         {
                 ...
                 case CHIP_VEGA20:
                 case CHIP_ARCTURUS:
                 case CHIP_ALDEBARAN:
                         soc15_set_virt_ops(adev);
                         break;
                 case CHIP_NAVI10:
                 case CHIP_NAVI12:
                 case CHIP_SIENNA_CICHLID:
                 case CHIP_IP_DISCOVERY:
                         nv_set_virt_ops(adev);
                         /* try send GPU_INIT_DATA request to host */
                         amdgpu_virt_request_init_data(adev);
                         break;
                 ...
         }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
   drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
   2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
   #include <linux/apple-gmux.h>
   #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
   #include <drm/drm_atomic_helper.h>
   #include <drm/drm_client_event.h>
   #include <drm/drm_crtc_helper.h>
@@ -1987,16 +1991,19 @@ static int

amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
        struct amdgpu_ip_block *ip_block;
        struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
        uint32_t bios_flags;
        int i, r;
 
        amdgpu_device_enable_virtual_display(adev);
 
        if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access = (adev->virt.req_init_data_ver
+== 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                r = amdgpu_virt_init_critical_region(adev);
                if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
        if (!total)
                return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
        if (adev->gmc.xgmi.supported)
                amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int xgpu_ai_send_access_requests(struct
 amdgpu_device *adev,
        } else if (req == IDH_REQ_GPU_INIT_DATA){
                /* Dummy REQ_GPU_INIT_DATA handling */
                r = xgpu_ai_poll_msg(adev,
 IDH_REQ_GPU_INIT_DATA_READY);
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
        }
        return 0;
#10 lore raw

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

With the legacy/existing read path, VF_FB_EN must be enabled so the host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 252 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 244 lines


         static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
         {
                 ...
                 case CHIP_VEGA20:
                 case CHIP_ARCTURUS:
                 case CHIP_ALDEBARAN:
                         soc15_set_virt_ops(adev);
                         break;
                 case CHIP_NAVI10:
                 case CHIP_NAVI12:
                 case CHIP_SIENNA_CICHLID:
                 case CHIP_IP_DISCOVERY:
                         nv_set_virt_ops(adev);
                         /* try send GPU_INIT_DATA request to host */
                         amdgpu_virt_request_init_data(adev);
                         break;
                 ...
         }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
   drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
   2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
   #include <linux/apple-gmux.h>
   #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
   #include <drm/drm_atomic_helper.h>
   #include <drm/drm_client_event.h>
   #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
        struct amdgpu_ip_block *ip_block;
        struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
        uint32_t bios_flags;
        int i, r;
 
        amdgpu_device_enable_virtual_display(adev);
 
        if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                r = amdgpu_virt_init_critical_region(adev);
                if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
        if (!total)
                return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
        if (adev->gmc.xgmi.supported)
                amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

        } else if (req == IDH_REQ_GPU_INIT_DATA){
                /* Dummy REQ_GPU_INIT_DATA handling */
                r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
        }
 
        return 0;
#11 lore raw
Quoted 9 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

Quoted 8 lines

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

With the legacy/existing read path, VF_FB_EN must be enabled so the host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 301 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 252 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 244 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
         }
 
         return 0;
#12 lore raw

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 305 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 252 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 244 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
         }
 
         return 0;
#13 lore raw
Quoted 7 lines · Chong(Alan)

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Not just for current, is this also applicable for old host driver? Otherwise, this breaks on old host driver.

Thanks,
Lijo

Quoted 345 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 305 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 252 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 244 lines


           static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
           {
                   ...
                   case CHIP_VEGA20:
                   case CHIP_ARCTURUS:
                   case CHIP_ALDEBARAN:
                           soc15_set_virt_ops(adev);
                           break;
                   case CHIP_NAVI10:
                   case CHIP_NAVI12:
                   case CHIP_SIENNA_CICHLID:
                   case CHIP_IP_DISCOVERY:
                           nv_set_virt_ops(adev);
                           /* try send GPU_INIT_DATA request to host */
                           amdgpu_virt_request_init_data(adev);
                           break;
                   ...
           }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
     drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
     drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
     2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
     #include <linux/apple-gmux.h>
     #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
     #include <drm/drm_atomic_helper.h>
     #include <drm/drm_client_event.h>
     #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
          struct amdgpu_ip_block *ip_block;
          struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
          uint32_t bios_flags;
          int i, r;
 
          amdgpu_device_enable_virtual_display(adev);
 
          if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                  r = amdgpu_virt_init_critical_region(adev);
                  if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
          if (!total)
                  return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
          if (adev->gmc.xgmi.supported)
                  amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

          } else if (req == IDH_REQ_GPU_INIT_DATA){
                  /* Dummy REQ_GPU_INIT_DATA handling */
                  r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
          }
 
          return 0;
maintainer
#14 lore raw
In reply to Li, Chong(Alan)

Hi all,

I've discussed the issue with Alex today.

The current approach of keeping full GPU access while discovery has been intentionally added this way over 6 years ago after some back and forth discussion.

So that has been working reliable for quite a while and we don't see any justification to change that.

If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side.

This patch is rejected.

Regards,
Christian.

Quoted 352 lines · Chong(Alan)

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 305 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 252 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 244 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 149 lines

Thanks,
Lijo

Quoted 144 lines

Regards,
Christian.

Quoted 140 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 83 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +2 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver = GPU_CRIT_REGION_V1;
         }
 
         return 0;
#15 lore raw

AMD General

Hi, Christian and Alex.

I do not think this patch conflicts with the original design.
The design principle is the same: full GPU access should be limited to the phase where the guest actually needs exclusive access to hardware resources.

For the legacy ASIC(like VEGA20 or ALDEBARAN about 6 years ago), the guest does not send REQUEST_INIT_DATA. The host prepares the init data only as part of the REQUEST_GPU_INIT flow,
so requesting full GPU access before ip discovery is necessary and should be preserved.

For newer ASICs that support REQUEST_INIT_DATA, the ordering is different. The guest first requests the early init data, and the host replies INIT_DATA_READY after discovery/VBIOS/PF2VF data has been copied to the VF-visible FB region.

In this flow, the guest copies the early init data from the VF-visible FB region and completes IP early init before accessing hardware resources. It then requests full GPU access only for the hardware initialization phase.

So this patch is not changing the legacy design.
It keeps the old behavior when REQUEST_INIT_DATA is not supported, and only moves full GPU access later for the newer REQUEST_INIT_DATA flow.

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Friday, July 10, 2026 1:32 AM
To: Li, Chong(Alan) <[email protected]>; Lazar, Lijo <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi all,

I've discussed the issue with Alex today.

The current approach of keeping full GPU access while discovery has been intentionally added this way over 6 years ago after some back and forth discussion.

So that has been working reliable for quite a while and we don't see any justification to change that.

If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side.

This patch is rejected.

Regards,
Christian.

Quoted 357 lines · Chong(Alan)

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 306 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 253 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 245 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 150 lines

Thanks,
Lijo

Quoted 145 lines

Regards,
Christian.

Quoted 141 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 84 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +3 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver =
+ GPU_CRIT_REGION_V1;
         }
 
         return 0;
#16 lore raw

AMD General

Hi, Christian and Alex.

"If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side."

This patch is not intended to solve the issue in "ESXi hypervisor".

This patch aims to reduce the full GPU access time on newer ASICs and speed up the GPU initialization flow.

When multiple VFs are initialized, moving the full GPU access request later shortens the full-access window and reduces the time other VFs spend waiting for exclusive access.

This is also the reason the REQUEST_INIT_DATA flow was introduced, allow the guest to obtain the early init data without relying on full GPU access.

Thanks,
Chong.

-----Original Message-----
From: Li, Chong(Alan)
Sent: Friday, July 10, 2026 12:06 PM
To: Koenig, Christian <[email protected]>; Deucher, Alexander <[email protected]>; Lazar, Lijo <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: RE: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi, Christian and Alex.

I do not think this patch conflicts with the original design.
The design principle is the same: full GPU access should be limited to the phase where the guest actually needs exclusive access to hardware resources.

For the legacy ASIC(like VEGA20 or ALDEBARAN about 6 years ago), the guest does not send REQUEST_INIT_DATA.
The host prepares the init data only as part of the REQUEST_GPU_INIT flow, so requesting full GPU access before ip discovery is necessary and should be preserved.

For newer ASICs that support REQUEST_INIT_DATA, the ordering is different.
The guest first requests the early init data, and the host replies INIT_DATA_READY after discovery/VBIOS/PF2VF data has been copied to the VF-visible FB region.

In this flow, the guest copies the early init data from the VF-visible FB region and completes IP early init before accessing hardware resources. It then requests full GPU access only for the hardware initialization phase.

So this patch is not changing the legacy design.
It keeps the old behavior when REQUEST_INIT_DATA is not supported, and only moves full GPU access later for the newer REQUEST_INIT_DATA flow.

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Friday, July 10, 2026 1:32 AM
To: Li, Chong(Alan) <[email protected]>; Lazar, Lijo <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi all,

I've discussed the issue with Alex today.

The current approach of keeping full GPU access while discovery has been intentionally added this way over 6 years ago after some back and forth discussion.

So that has been working reliable for quite a while and we don't see any justification to change that.

If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side.

This patch is rejected.

Regards,
Christian.

Quoted 357 lines · Chong(Alan)

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 306 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 253 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 245 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 150 lines

Thanks,
Lijo

Quoted 145 lines

Regards,
Christian.

Quoted 141 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 84 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +3 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver =
+ GPU_CRIT_REGION_V1;
         }
 
         return 0;
maintainer
#17 lore raw

Hi Chong,

Quoted 8 lines · Chong(Alan)

AMD General

Hi, Christian and Alex.

"If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side."

Quoted 6 lines

This patch is not intended to solve the issue in "ESXi hypervisor".

This patch aims to reduce the full GPU access time on newer ASICs and speed up the GPU initialization flow.

When multiple VFs are initialized, moving the full GPU access request later shortens the full-access window and reduces the time other VFs spend waiting for exclusive access.

Well that is at least finally a full explanation for the change.

But it makes it clear that this is basically just an optimization and not functional required.

Quoted 2 lines

This is also the reason the REQUEST_INIT_DATA flow was introduced, allow the guest to obtain the early init data without relying on full GPU access.

Yeah and that change was never discussed with Alex or me on the design part.

I'm now systematically rejecting anything which hasn't been akwnoledged at least a couple of month pre-hand before the implementation.

So this patch here is rejected, end of discussion.

Regards,
Christian.

Quoted 423 lines

Thanks,
Chong.

-----Original Message-----
From: Li, Chong(Alan)
Sent: Friday, July 10, 2026 12:06 PM
To: Koenig, Christian <[email protected]>; Deucher, Alexander <[email protected]>; Lazar, Lijo <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: RE: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi, Christian and Alex.

I do not think this patch conflicts with the original design.
The design principle is the same: full GPU access should be limited to the phase where the guest actually needs exclusive access to hardware resources.

For the legacy ASIC(like VEGA20 or ALDEBARAN about 6 years ago), the guest does not send REQUEST_INIT_DATA.
The host prepares the init data only as part of the REQUEST_GPU_INIT flow, so requesting full GPU access before ip discovery is necessary and should be preserved.

For newer ASICs that support REQUEST_INIT_DATA, the ordering is different.
The guest first requests the early init data, and the host replies INIT_DATA_READY after discovery/VBIOS/PF2VF data has been copied to the VF-visible FB region.

In this flow, the guest copies the early init data from the VF-visible FB region and completes IP early init before accessing hardware resources. It then requests full GPU access only for the hardware initialization phase.

So this patch is not changing the legacy design.
It keeps the old behavior when REQUEST_INIT_DATA is not supported, and only moves full GPU access later for the newer REQUEST_INIT_DATA flow.

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Friday, July 10, 2026 1:32 AM
To: Li, Chong(Alan) <[email protected]>; Lazar, Lijo <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Hi all,

I've discussed the issue with Alex today.

The current approach of keeping full GPU access while discovery has been intentionally added this way over 6 years ago after some back and forth discussion.

So that has been working reliable for quite a while and we don't see any justification to change that.

If the ESXi hypervisor has a problem with this then that needs to be handled on the hypervisor side.

This patch is rejected.

Regards,
Christian.

Quoted 357 lines · Chong(Alan)

AMD General

Hi, Lijo.

Yes, the current GIM driver ensures the early init data is placed within the VF’s visible BAR0 aperture.

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Thursday, July 9, 2026 3:56 PM
To: Li, Chong(Alan) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 11 lines · Chong(Alan)

AMD General

Hi, Lijo.

" In this commit, the statement is different."
" It says FB access is enabled if init_data request is sent."

Yes, VF FB access is enabled if init_data request is sent.

However, without full GPU access, the guest cannot access the MM_INDEX/MM_DATA registers.

Quoted 8 lines

With the legacy/existing read path, VF_FB_EN must be enabled so the
host can dump early init data into the VF FB, and full GPU access is required so the guest can copy that data through the MM_INDEX/MM_DATA registers.

I submitted patch " [PATCH 1/2] drm/amdgpu: read FB through BAR0 when aperture is unavailable " , With the new amdgpu_device_read_fb_via_bar0() path, the guest can read the VF FB without full GPU access

Yes, I saw that one. One other question - does host driver guarantee that discovery data will always be copied to visible BAR aperture of VF? If not, it will still require mmindex/data regardless of FB access enablement.

Thanks,
Lijo

Quoted 306 lines

Thanks,
Chong.

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, July 8, 2026 7:18 PM
To: Koenig, Christian <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 12 lines · Christian König

Hi Chong,

Quoted 9 lines · Chong(Alan)

AMD General

Hi, Christian.

The code " early_full_gpu_access = (adev->virt.req_init_data_ver == 0);" is related to the ASIC type, not the host platform:

adev->virt.req_init_data_ver is initialized by amdgpu_virt_request_init_data().

Quoted 9 lines
Quoted 4 lines

Some legacy ASICs, such as CHIP_VEGA20 and CHIP_ALDEBARAN, do not send amdgpu_virt_request_init_data() to the host.

For those ASICs, the host dumps the early init data only after the guest requests full GPU access. Therefore, those ASICs still need to request full GPU access before the guest driver can read the early init data from the VF FB.

No, exactly that is not the case as far as I can see. See the commit I mentioned below.

According to that starting from SIENNA_CICHLID the full GPU access is mandatory for reading the discovery table.

It could also be because the legacy/existing way used mmindex/data, and that required full access including mmio register access.

In this commit, the statement is different.

"In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB"

It says FB access is enabled if init_data request is sent.

Thanks
Lijo

Quoted 253 lines

So I absolutely clearly have to reject your patch here.

As far as I can see this is a hack for ESXi and that is not going to fly.

Regards,
Christian.

Quoted 245 lines


          static bool amdgpu_virt_init_req_data(struct amdgpu_device *adev, u32 reg)
          {
                  ...
                  case CHIP_VEGA20:
                  case CHIP_ARCTURUS:
                  case CHIP_ALDEBARAN:
                          soc15_set_virt_ops(adev);
                          break;
                  case CHIP_NAVI10:
                  case CHIP_NAVI12:
                  case CHIP_SIENNA_CICHLID:
                  case CHIP_IP_DISCOVERY:
                          nv_set_virt_ops(adev);
                          /* try send GPU_INIT_DATA request to host */
                          amdgpu_virt_request_init_data(adev);
                          break;
                  ...
          }

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 6:24 PM
To: Lazar, Lijo <[email protected]>; Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 25 lines · Lijo
Quoted 22 lines · Christian König
Quoted 21 lines · Chong(Alan)

AMD General

Hi, Christian.

This version of the patch is not host-platform dependent; all host platforms follow the same path.

I verified this patch with KVM in my local environment and with ESXi on a borrowed server.

Distinguishing the host platform was a mistake, and I have explained the situation at the end of the patch email.

I have pasted the content below:

Hi, Christian and Lijo.

Sorry, I made a mistake.
Host access to the VF FB is not platform-dependent. After the host driver sets VF_FB_EN in response to GPU_INIT_DATA, the host can access the VF FB.

I borrowed an ESXi server and tested this patch there. The amdgpu driver works normally.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I rewrote the background. Is anything still unclear?

You still have this check here in the code: "early_full_gpu_access = (adev->virt.req_init_data_ver == 0);".

As far as I can see that is Hypervisor specific and a NO-GO.

I think this should be seen as a policy set by host driver on when a guest operation is allowed. Host driver could do it based on other conditions also, not specific to hypervisor environment.

No, this was intentionally changed in 2020 because of a new feature. See this commit here:

commit 00a979f3d69e0c275e88c741b854dbe0d5238ae0

Author: Wenhui Sheng <[email protected]>
Date:   Tue Jun 23 13:43:49 2020 +0800

drm/amdgpu: invoke req full access early enough

From SIENNA_CICHLID, HW introduce a new protection feature which can control the FB, doorbell and MMIO write access for VF, so guest driver should request full access before ip discovery, or we couldn't access ip discovery data in FB.

So as far as I can see this change here will break older hypervisor versions and that is absolutely clear reason to NAK it.

Regards,
Christian.

Quoted 150 lines

Thanks,
Lijo

Quoted 145 lines

Regards,
Christian.

Quoted 141 lines

Thanks,
Chong.

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Wednesday, July 8, 2026 5:05 PM
To: Li, Chong(Alan) <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>; Chang, HaiJun <[email protected]>; Skvortsov, Victor <[email protected]>; Lazar, Lijo <[email protected]>; Cursor <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: improve the amdgpu device init progress in sriov mode

Quoted 20 lines · chong li

v2:
Some legacy ASICs do not send amdgpu_virt_request_init_data(). Only keep the full GPU access request early when request_init_data is not sent.

v1:
Move the initialization of non-GPU resources out of the full GPU access region during AMDGPU device initialization.

Background:
In SR-IOV, the guest sends GPU_INIT_DATA, then the host enables VF_FB_EN and places early initialization data, such as IP discovery, VBIOS, and PF-VF exchange data, in the VF FB. The guest should then be able to read this data before requesting full GPU access.

Before this patch, the VF still requested full GPU access in amdgpu_device_ip_early_init(). At that point TTM is not initialized yet, so the normal VRAM aperture mapping is unavailable and the guest falls back to MM_INDEX/MM_DATA register access. That register path requires full GPU access.

Quoted 4 lines

Use the BAR0 framebuffer read path, amdgpu_device_read_fb_via_bar0(), for the early init-data copy instead of MM_INDEX/MM_DATA. This lets the driver delay the full GPU access request until after the early init data has been copied.

That looks like it goes into the right direction, but as far as I can see it is still an ESXi specific change.

So once more: We can't adjust the driver to the hypervisor!

Either this works on all hypervisors or I have to reject the change.

Regards,
Christian.

Quoted 84 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21
+++++++++++++++++----
    drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |  4 ++--
    2 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
index 610d82b79de3..ac66796e8634 100644
@@ -38,6 +38,10 @@
    #include <linux/apple-gmux.h>
    #include <linux/nospec.h>
 
+#ifdef CONFIG_X86
+#include <asm/hypervisor.h>
+#endif
+
    #include <drm/drm_atomic_helper.h>
    #include <drm/drm_client_event.h>
    #include <drm/drm_crtc_helper.h> @@ -1987,16 +1991,19 @@

static int amdgpu_device_ip_early_init(struct

amdgpu_device *adev)  {
         struct amdgpu_ip_block *ip_block;
         struct pci_dev *parent;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 -4
-     bool total, skip_bios;
+     bool total, skip_bios, early_full_gpu_access = false;
         uint32_t bios_flags;
         int i, r;
 
         amdgpu_device_enable_virtual_display(adev);
 
         if (amdgpu_sriov_vf(adev)) {
-             r = amdgpu_virt_request_full_gpu(adev, true);
-             if (r)
-                     return r;
+             early_full_gpu_access =
+(adev->virt.req_init_data_ver == 0);
+             if (early_full_gpu_access) {
+                     r = amdgpu_virt_request_full_gpu(adev, true);
+                     if (r)
+                             return r;
+             }
 
                 r = amdgpu_virt_init_critical_region(adev);
                 if (r)
@@ -2159,6 +2166,12 @@ static int
 amdgpu_device_ip_early_init(struct amdgpu_device *adev)
         if (!total)
                 return -ENODEV;
 
+     if (amdgpu_sriov_vf(adev) && !early_full_gpu_access) {
+             r = amdgpu_virt_request_full_gpu(adev, true);
+             if (r)
+                     return r;
+     }
+
         if (adev->gmc.xgmi.supported)
                 amdgpu_xgmi_early_init(adev);
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +0 -0
index 9a40107a0869..340703d89d6b 100644
@@ -185,8 +185,8 @@ static int

xgpu_ai_send_access_requests(struct
amdgpu_device *adev,

         } else if (req == IDH_REQ_GPU_INIT_DATA){
                 /* Dummy REQ_GPU_INIT_DATA handling */
                 r = xgpu_ai_poll_msg(adev,

IDH_REQ_GPU_INIT_DATA_READY);

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +3 -2
-             /* version set to 0 since dummy */
-             adev->virt.req_init_data_ver = 0;
+             /* Version is set to 1 since GPU_CRIT_REGION_V1 */
+             adev->virt.req_init_data_ver =
+ GPU_CRIT_REGION_V1;
         }
 
         return 0;