Skip to contentSkip to conversation

Add UALink infrastructure series 1

PATCH Superseded Alex Deucher 1 participant 96 messages Aug 31 Page 1 of 2

Subsystems (5)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

v2 cover Add UALink infrastructure series 1

This adds the core infrastructure for supporting UALink (Ultra Accelerator Link) connected scale up pods. I've split this into two series, one to add the core infrastructure, and one to expose the new IOCTL interface and add the documentation to avoid spamming the larger audience with the implemenation defails. This is the first series.

This implements UALoE (UALink over Ethernet). An overview of the complete solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html

Overview

Connected GPUs in a pod can directly access the remove memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Nework Physical Address) space. This address space is partitioned between the GPUs so that each GPU has it's own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM.

Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned for P2P access. Remote TLB shootdowns from the exporter are used when the exported memory is freed in order to remove access by remote GPUs.

To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining refrences that are lost when that happens.

User interface

Export Memory

To export memory, a handle must be created for an allocation that can be shared with another node in the pod. To do this the exporter calls the GEM UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL returns a unique handle which can be shared with the remote host.

Import Memory

To import the memory, the handle from the remote node must be converted from a unique handle to a local GEM object which represents the local reference to the NPA space on the importer. If the memory has already been imported, it just returns a new reference to the existing object. If not, the importer queries the exporter to get the NPA address. Once it has that the importer can create a dma-buf to represent the NPA space used by the allocation and that is returned to the application.

This and the series 2 patches can be found here:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads

V2:
- #ifdef the sysfs ualink configuration API. It will be configured via out of band tooling. This code will be dropped in the near future.
- fix 32 bit build

Alex Sierra (1):
drm/amdgpu: log remote memory MTYPE for GC 12.1.0

Felix Kuehling (17):
drm/amdgpu: Add sysfs API for UALink information drm/amdgpu: Add sysfs API for UALink physical pod setup drm/amdgpu: Add sysfs API for UALink virtual pod config drm/amdgpu: Add sysfs API for UALink station configuration drm/amdgpu: Implement PSP cmd UAL_GET_CONFIG drm/amdgpu: Query initial UALink config from PSP drm/amdgpu: Implement PSP cmd UAL_SET_PPOD_CONFIG drm/amdgpu: Set physical pod configuration to PSP drm/amdgpu: Implement PSP cmd UAL_SET_VPOD_CONFIG drm/amdgpu: Set virtual pod configuration to PSP drm/amdgpu: Implement PSP cmd UAL_SET_STATION_CONFIG drm/amdgpu: Set UALink station config to PSP drm/amdgpu: Implement PSP cmd UAL_SET_NPA_CONFIG drm/amdgpu: Enable/disable NPA address translation using PSP drm/amdgpu: Prevent double-free of drm_exec drm/amdgpu: Pin page tables in NPA VMs
drm/amdgpu: Initialize NPA PT/PDs to noretry

Gangliang Xie (1):
drm/amdgpu: add handler for nHT error

Harish Kasiviswanathan (2):
drm/amdgpu: Add helper function to check psp xgmi ta drm/amdgpu: Add ual_config_state to ual_get_config

Jinzhou Su (1):
drm/amdgpu: Fix NULL pointer issue during ualink init

Le Ma (1):
drm/amdgpu: extend PSP command polling sleep range

Lijo Lazar (15):
drm/amdgpu: Add psp ualink command interfaces drm/amdgpu: Fetch asp ualink interface version drm/amdgpu: Separate out ualink init sequences drm/amdgpu: Add ualink as separate ip block drm/admgpu: Seggregate ualink nht messaging drm/amdgpu: Assign accel state based on ASP config drm/amdgpu: Drop duplicate vpod check functions drm/amdgpu: Add support to send ASP completion drm/amdgpu: Add handlers for ualink notifications drm/amdgpu: Improve ualink state transitions drm/amdgpu: Use uniform logic for inband/sideband drm/amdgpu: Add name for ualink ip block drm/amdgpu: Move ualink ip version related changes drm/amdgpu: Add hw_fini for ualink drm/amdgpu: Expose ualink info under each xcp

Mukul Joshi (42):
drm/amdgpu: Add UALink manager core infrastructure drm/amdgpu: Add a new NPA Address space
drm/amdgpu: Add address allocator for NPA addresses drm/amdgpu: Initialize VM for NPA addr management drm/amdgpu: Rework VMID reservation logic
drm/amdgpu: Reserve VMID for NPA VM
drm/amdgpu: Use reserved VMID for NPA VM
drm/amdgpu: Enable UALink Manager when pod becomes active drm/amdgpu: Create a drm client for UALink NPA BOs drm/amdgpu: Control NPA DMA-buf importing
drm/amdgpu: Add ualink handle to BOs
drm/amdgpu: Implement UALink handle export
drm/amdgpu: Add connection state management
drm/amdgpu: Implement UALink handle import ioctl drm/amdgpu: Implement mechanism to revoke exported memory drm/amdgpu: lock UALink import invalidation via drm_exec drm/amdgpu: Cleanup exported UALink handles
drm/amdgpu: Cleanup imported UALink handles
drm/amdgpu: Handle connection reset
drm/amdgpu: Setup PTE mappings for NPA addresses drm/amdgpu: Add handling for remote interrupts drm/amdgpu: Send TLB shootdown on exported memory unmap drm/amdgpu: Handle local GPUs in UALink import drm/amdgpu: Add debugfs to drop UALink protocol messages drm/amdgpu: Temporarily disable sending remote TLB shootdowns drm/amdgpu: Temporarily Flush TLB on NPA mapping always drm/amdgpu: fix NPA-RELEASE race in UALink exporter cleanup drm/amdgpu: initialize UALink importer node list head drm/amdgpu: Fix initialization flags for UALink XAs drm/amdgpu: fix dma_buf leak in UALink exporter cleanup drm/amdgpu: Fix uninitialized fence in UALink NPA unmap drm/amdgpu: Use vm->last_update fence in UALink NPA unmap paths drm/amdgpu: always use MTYPE_UC for remote memory on GFX 12.1 drm/amdkfd: program compute MQD coherent_aql_mtype on GFX 12.1 drm/amdgpu: Fix GART and SDMA entity leak on vPod reconfiguration drm/amdgpu: Add UALink diagnostic logging for vpod commit/activation drm/amdgpu: Handle UALink vPod reconfiguration while ACTIVE drm/amdgpu: Cleanup UALink XA entries on manager stop drm/amdgpu: Handle concurrent UALINK handle import race drm/amdgpu: create UALink NPA import BO directly in the NPA domain drm/amdgpu: add mtype_remote module parameter
drm/amdgpu: Honor mtype overrides for NPA remote memory

Philip Yang (15):
drm/amdgpu: Fix UALink vPod double-activation drm/amdgpu: Add UALink remote state structures and API declarations drm/amdgpu: Add UALink NPA address layout helpers drm/amdgpu: Add UALink NPA address computation for ring buffers drm/amdgpu: Add UALink NPA VM mapping for ring buffers drm/amdgpu: Add UALink SDMA scheduler entities drm/amdgpu: Add UALink GART helpers for NPA address access drm/amdgpu: Add UALink ring buffer allocation and firmware init drm/amdgpu: Add UALink remote command packets and SDMA dispatch drm/amdgpu: Add UALink firmware writeback address configuration drm/amdgpu: Add UALink cross-GPU TLB shootdown and remote interrupt drm/amdgpu: Add UALink software init, teardown, and reset drm/amdgpu: Add UALink IH ring and enable interrupt drm/amdgpu: UALink use LSDMA to send remote interrupt command drm/amdgpu: Increase UALink soft ring size

 drivers/gpu/drm/amd/amdgpu/Makefile           |    3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |    5 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c   |   14 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |    2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c |   21 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    5 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |    7 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h       |    1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c       |   33 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h       |   27 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h        |    1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c        |    1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c       |   17 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h       |    7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |   11 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    5 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c       |  280 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h       |   27 +
 .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h    |    1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   17 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    | 6222 +++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h    |  425 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |   44 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c     |   29 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c       |    6 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h       |    7 +
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c        |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c        |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c        |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c        |   51 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c         |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c         |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c         |    2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c       |    2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c        |    4 +
 drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c        |    5 +
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c        |   26 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c        |   98 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.h        |    1 +
 drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c         |    6 +
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c         |    6 +
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c         |    6 +
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c         |    6 +
 drivers/gpu/drm/amd/amdgpu/ih_v7_0.c          |    8 +
 drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h       |  162 +
 drivers/gpu/drm/amd/amdgpu/psp_v15_0_8.c      |   48 +-
 drivers/gpu/drm/amd/amdgpu/soc_v1_0.c         |    6 -
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c      |  143 +
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h      |   30 +
 .../drm/amd/amdkfd/kfd_mqd_manager_v12_1.c    |    7 +
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c          |    8 +-
 drivers/gpu/drm/amd/include/amd_shared.h      |    2 +
 .../ivsrcid/mpnht/irqsrcs_mpnht_15_0.h        |   30 +
 drivers/gpu/drm/amd/include/v12_structs.h     |    2 +-
 include/drm/ttm/ttm_resource.h                |    2 +-
 include/uapi/drm/amdgpu_drm.h                 |    6 +
 58 files changed, 7834 insertions(+), 69 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
 create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h
 create mode 100644 drivers/gpu/drm/amd/include/ivsrcid/mpnht/irqsrcs_mpnht_15_0.h
Signature · 1 line
2.55.0
author maintainer
#1 lore raw

PATCH v2 1/95 drm/amdgpu: Add psp ualink command interfaces

From: Lijo Lazar <[email protected]>

Add asp ualink command/response structures

---
 drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h | 126 ++++++++++++++++++++++++
 1 file changed, 126 insertions(+)
drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h +126 -0
index ad618b78685f6..6bcc04109d4cb 100644
@@ -53,6 +53,13 @@ enum psp_gfx_crtl_cmd_id
     GFX_CTRL_CMD_ID_MAX             = 0x000F0000,   /* max command ID */
 };
 
+typedef enum
+{
+    GFX_CTRL_ADDR_TYPE_AUTO_DETECT = 0,
+    GFX_CTRL_ADDR_TYPE_SYS_PHY_ADDR,
+    GFX_CTRL_ADDR_TYPE_GPU_PHY_ADDR,
+    GFX_CTRL_ADDR_TYPE_GPU_VIRT_ADDR
+}  GFX_CTRL_ADDR_TYPE;
 
 /*-----------------------------------------------------------------------------
     NOTE:   All physical addresses used in this interface are actually
@@ -110,6 +117,13 @@ enum psp_gfx_cmd_id
     GFX_CMD_ID_PERF_HW            = 0x0000004C,   /* performance monitor */
     GFX_CMD_ID_FB_FW_RESERV_ADDR  = 0x00000050,  /* Query FW reservation addr */
     GFX_CMD_ID_FB_FW_RESERV_EXT_ADDR = 0x00000051,  /* Query FW reservation extended addr */
+    GFX_CMD_ID_UAL_GET_INTERFACE_VER = 0x00000053,  /* Get UAL interface version */
+    GFX_CMD_ID_UAL_GET_CONFIG        = 0x00000054,  /* Get UAL full config */
+    GFX_CMD_ID_UAL_SET_PPOD_CONFIG   = 0x00000055,  /* Set UAL config */
+    GFX_CMD_ID_UAL_SET_VPOD_CONFIG   = 0x00000056,  /* Set UAL vPod config */
+    GFX_CMD_ID_UAL_SET_STATION_CONFIG= 0x00000057,  /* Set UAL Station config */
+    GFX_CMD_ID_UAL_SET_NPA_CONFIG    = 0x00000059,  /* Set UAL NPA config and VMID */
+    GFX_CMD_ID_UAL_SEND_COMPLETION   = 0x0000005A,  /* Ack from driver after UAL config completion */
     GFX_CMD_ID_SET_MMHUB_ECO_SEC_LEVEL = 0x0000005D,  /* Set MMHUB ECO sec lvls on VCN block */
 };
 
@@ -382,6 +396,100 @@ struct psp_gfx_cmd_req_perf_hw {
 	uint32_t pref_format2;
 };
 
+#define PSP_GFX_MAX_LOCAL_GPUS_UAL_V1   16       /* max gpu per local node */
+#define PSP_GFX_UAL_MAX_STATIONS_V1     64       /* max UALink stations */
+#define PSP_GFX_UAL_MAX_ACC_BIT_MASK    32       /* max accelerator bit mask 32*32 */
+
+enum psp_gfx_ual_npa_address_mode
+{
+    PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_ALIASING = 0,
+    PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_IDENTIFICATION = 1,
+    PSP_GFX_UAL_NPA_ADDRESS_MODE_MAX
+};
+
+enum psp_gfx_ual_ports_per_station
+{
+    PSP_GFX_UAL_PPS_1 = 1,    // 1x 800Gbps
+    PSP_GFX_UAL_PPS_2 = 2,    // 2x 400Gbps
+    PSP_GFX_UAL_PPS_4 = 4     // 4x 200Gbps
+};
+
+enum psp_gfx_ual_link_type
+{
+    PSP_GFX_UALOE = 0,
+    PSP_GFX_UALLINK = 1,
+    PSP_GFX_UALMAX
+};
+
+struct psp_gfx_cmd_set_ppod_config_ual_v1
+{
+    uint32_t accelerator_id;     // Accelerator ID
+    uint8_t  ppod_id[16];        // Physical Pod ID
+    uint32_t ppod_size;          // number of gpu in physical pod
+    uint32_t bandwidth;          // Bandwidth
+    uint32_t latency;            // Latency
+    uint32_t local_accelerators[PSP_GFX_MAX_LOCAL_GPUS_UAL_V1]; // local accelerator list, sorted in socket-id order
+};
+
+struct psp_gfx_cmd_set_vpod_config_ual_v1
+{
+    uint32_t vpod_id;                       // vPod ID
+    uint32_t vpod_size;                     // number of gpu in virtual pod
+    uint32_t vpod_active_accelerators[PSP_GFX_UAL_MAX_ACC_BIT_MASK];  // accelerator ID list in each vPOD
+    enum psp_gfx_ual_npa_address_mode addr_mode; // NPA Addressing Mode
+};
+
+struct psp_gfx_cmd_station_config_ual_v1
+{
+    /* Number of valid stations in this configuration
+     * Only lane_en_bitmap[0..n_stations-1] will be processed.
+     */
+    uint8_t num_stations;
+    /* Station configuration flags
+     *
+     * Bit [3:0]: PortPerStation (PPS) - 1, 2, or 4
+     * Bit [7:4]: Reserved
+     */
+    uint8_t station_flag;
+    uint8_t reserved[2]; /* Future use / alignment padding */
+    /* Bitmap of enabled lanes for each station
+     * in logical station order.
+     */
+    uint8_t lane_en_bitmap[PSP_GFX_UAL_MAX_STATIONS_V1];
+};
+
+struct psp_gfx_cmd_get_config_ual_v1
+{
+    uint32_t ual_cfg_addr_hi;               // UAL Config Buffer Address Hi
+    uint32_t ual_cfg_addr_lo;               // UAL Config Buffer Address Lo
+    uint32_t ual_cfg_size;                  // UAL Config Buffer Size
+    GFX_CTRL_ADDR_TYPE addr_type;    // Buffer Address Type
+};
+
+struct psp_gfx_get_config_ual_v1 {
+    enum psp_gfx_ual_link_type link_type;
+    uint32_t accelerator_id;
+    uint8_t ppod_id[16];
+    uint32_t ppod_size;
+    uint32_t bandwidth;
+    uint32_t latency;
+    uint32_t vpod_id;
+    uint32_t vpod_size;
+    uint32_t vpod_active_accelerators[PSP_GFX_UAL_MAX_ACC_BIT_MASK];
+    enum psp_gfx_ual_npa_address_mode addr_mode;
+};
+
+struct psp_gfx_cmd_send_completion_ual_v1
+{
+    uint32_t cmd_id;
+    uint32_t status;
+};
+
+struct psp_gfx_cmd_set_npa_config_ual_v1 {
+    uint32_t vmid;
+    uint32_t enable_npa_translation;
+};
+
 /* All GFX ring buffer commands. */
 union psp_gfx_commands
 {
@@ -399,6 +507,12 @@ union psp_gfx_commands
     struct psp_gfx_cmd_config_sq_perfmon config_sq_perfmon;
     struct psp_gfx_cmd_fb_memory_part cmd_memory_part;
     struct psp_gfx_cmd_req_perf_hw     cmd_req_perf_hw;
+    struct psp_gfx_cmd_get_config_ual_v1    cmd_get_config_ual;
+    struct psp_gfx_cmd_set_ppod_config_ual_v1 cmd_set_ppod_config_ual;
+    struct psp_gfx_cmd_set_vpod_config_ual_v1 cmd_set_vpod_config_ual;
+    struct psp_gfx_cmd_station_config_ual_v1  cmd_set_station_config_ual;
+    struct psp_gfx_cmd_send_completion_ual_v1 cmd_send_completion_ual;
+    struct psp_gfx_cmd_set_npa_config_ual_v1  cmd_set_npa_config_ual;
 };
 
 struct psp_gfx_uresp_reserved
@@ -432,6 +546,16 @@ struct psp_gfx_uresp_perf_hw {
 	uint32_t pref_format2;
 };
 
+struct psp_gfx_uresp_get_intf_ver_ual
+{
+    uint32_t intf_ver; /* [31:16] major version, [15:0] minor version */
+};
+
+struct psp_gfx_uresp_get_config_ual_v1
+{
+    uint32_t resp_size;
+};
+
 /* Union of command-specific responses for GPCOM ring. */
 union psp_gfx_uresp {
 	struct psp_gfx_uresp_reserved		reserved;
@@ -439,6 +563,8 @@ union psp_gfx_uresp {
 	struct psp_gfx_uresp_fwar_db_info	fwar_db_info;
 	struct psp_gfx_uresp_fw_reserve_info	fw_reserve_info;
 	struct psp_gfx_uresp_perf_hw		perf_hw_info;
+	struct psp_gfx_uresp_get_config_ual_v1  get_config_ual;
+	struct psp_gfx_uresp_get_intf_ver_ual   get_intf_ver_ual;
 };
 
 /* Structure of GFX Response buffer.
Signature · 1 line
2.55.0
author maintainer
#2 lore raw

PATCH v2 2/95 drm/amdgpu: Fetch asp ualink interface version

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Add interface to get ASP ualink interface version.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 25 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  1 +
 2 files changed, 26 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +25 -0
index a0f6df5067e6e..5c82d1d808157 100644
@@ -827,6 +827,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "NPS_MODE_CHANGE";
 	case GFX_CMD_ID_PERF_HW:
 		return "PERF MONITORING HW";
+	case GFX_CMD_ID_UAL_GET_INTERFACE_VER:
+		return "UAL_GET_INTERFACE_VER";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1187,6 +1189,29 @@ static int psp_get_fw_reservation_info(struct psp_context *psp,
 	return 0;
 }
 
+int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_GET_INTERFACE_VER;
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && !cmd->resp.status) {
+		*intf_ver = cmd->resp.uresp.get_intf_ver_ual.intf_ver;
+	} else if (!ret) {
+		pr_debug("ual_get_if_ver: PSP status %x\n", cmd->resp.status);
+		ret = -EINVAL;
+	}
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +1 -0
index 0aef9ee83bd0c..5968595647fef 100644
@@ -705,5 +705,6 @@ int amdgpu_psp_irq_mgr_register(
 	struct amdgpu_psp_irq_mgr *mgr,
 	const struct amdgpu_psp_irq_handler *handlers, int count,
 	const struct amdgpu_psp_irq_handler *default_handler);
+int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
 #endif
Signature · 1 line
2.55.0
author maintainer
#3 lore raw

PATCH v2 3/95 drm/amdgpu: Add sysfs API for UALink information

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

<device>/ualink/info: Directory contains current UALink attributes
- link_type: UALoE of UALink
- accel_state: unconfigured, configured, ready, active, error
- accel_id: Accelerator ID
- ppod_id: Physical pod ID (UUID)
- ppod_size: Number of accelerators in physical pod
- bandwidth: Estimated total bandwidth between pairs of accelerators
- latency: Estimated latency between pairs of accelerators
- vpod_id: Virtual pod ID
- vpod_size: Number of accelerators in virtual pod
- vpod_active_accels: List of all accelerator IDs in vpod
- local_accels: List of local accelerator IDs in vpod
- addr_mode: source-aliasing or source-identification

---
 drivers/gpu/drm/amd/amdgpu/Makefile        |   3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 226 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  97 +++++++++
 drivers/gpu/drm/amd/amdgpu/soc_v1_0.c      |  12 ++
 5 files changed, 342 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h
drivers/gpu/drm/amd/amdgpu/Makefile +3 -0
index ffb725fb10842..bb33119608a06 100644
@@ -278,6 +278,9 @@ amdgpu-y += amdgpu_amdkfd.o
 # add gfx usermode queue
 amdgpu-y += amdgpu_userq.o
 
+# add UALink manager support
+amdgpu-y += amdgpu_ualink.o
+
 ifneq ($(CONFIG_HSA_AMD),)
 AMDKFD_PATH := ../amdkfd
 include $(FULL_AMD_PATH)/amdkfd/Makefile
drivers/gpu/drm/amd/amdgpu/amdgpu.h +4 -0
index 8a7c89afc88e1..f619d95a5ccfe 100644
@@ -116,6 +116,7 @@
 #include "amdgpu_mes.h"
 #include "amdgpu_sa.h"
 #include "amdgpu_acpi.h"
+#include "amdgpu_ualink.h"
 #if defined(CONFIG_DRM_AMD_ISP)
 #include "amdgpu_isp.h"
 #endif
@@ -799,6 +800,9 @@ struct amdgpu_device {
 	/* display related functionality */
 	struct amdgpu_display_manager dm;
 
+	/* UALink manager */
+	struct amdgpu_ualink_mgr	ualink;
+
 #if defined(CONFIG_DRM_AMD_ISP)
 	/* isp */
 	struct amdgpu_isp		isp;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c new file +226 -0

226 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h new file +97 -0

97 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/soc_v1_0.c +12 -0
index ac4a4de9c932a..d5f945617f488 100644
@@ -444,6 +444,17 @@ static int soc_v1_0_common_late_init(struct amdgpu_ip_block *ip_block)
 
 static int soc_v1_0_common_sw_init(struct amdgpu_ip_block *ip_block)
 {
+	struct amdgpu_device *adev = ip_block->adev;
+
+	return amdgpu_ualink_sysfs_init(adev);
+}
+
+static int soc_v1_0_common_sw_fini(struct amdgpu_ip_block *ip_block)
+{
+	struct amdgpu_device *adev = ip_block->adev;
+
+	amdgpu_ualink_sysfs_fini(adev);
+
 	return 0;
 }
 
@@ -500,6 +511,7 @@ static const struct amd_ip_funcs soc_v1_0_common_ip_funcs = {
 	.early_init = soc_v1_0_common_early_init,
 	.late_init = soc_v1_0_common_late_init,
 	.sw_init = soc_v1_0_common_sw_init,
+	.sw_fini = soc_v1_0_common_sw_fini,
 	.hw_init = soc_v1_0_common_hw_init,
 	.hw_fini = soc_v1_0_common_hw_fini,
 	.suspend = soc_v1_0_common_suspend,
Signature · 1 line
2.55.0
author maintainer
#4 lore raw

PATCH v2 4/95 drm/amdgpu: Add sysfs API for UALink physical pod setup

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

<device>/ualink/setup: Directory for staging physical pod setup attributes. Write "true" to commit to apply settings

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 226 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  11 +
 2 files changed, 237 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +226 -0
index 374242372eab7..92c449f2004d4 100644
@@ -37,6 +37,17 @@
  * device itself or a subdevice.
  ****************************************************************************/
 
+/* UALink configuration APIs are being deprecated before upstreaming and are
+ * not enabled by default. They are replaced by a unified in-band/side-band
+ * approach through the IFoE driver. Querying of the current configuration
+ * will continue to be supported in sysfs.
+ *
+ * Uncomment the following line to enable the deprecated configuration APIs
+ * for testing purposes only. The code to support these APIs will be removed
+ * in a future patch.
+ */
+/* #define UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS */
+
 #define UALINK_VALUE_SHOW(prefix, name, field, format)			\
 static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 					       struct kobj_attribute *attr,\
@@ -46,6 +57,21 @@ static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 									\
 	return sysfs_emit(buf, format"\n", info->field);		\
 }
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+#define UALINK_VALUE_STORE(prefix, name, field, type, base)		\
+static ssize_t ualink_##prefix##_##name##_store(struct kobject *kobj,	\
+						struct kobj_attribute *attr,\
+						const char *buf, size_t count)\
+{									\
+	struct amdgpu_ualink_##prefix *info = to_ualink_##prefix(kobj);	\
+	int r;								\
+									\
+	r = kstrto##type(buf, base, &info->field);			\
+	if (r < 0)							\
+		return r;						\
+	return count;							\
+}
+#endif
 
 static ssize_t show_idbits(unsigned long *bits, unsigned int nbits, char *buf)
 {
@@ -89,6 +115,67 @@ static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 	return show_idarray(info->field, info->size, buf);		\
 }
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+static ssize_t store_idarray(u32 *array, u32 *size, u32 max_id, u32 max_size,
+			     const char *buf, size_t count)
+{
+	char *dup_buf, *str, *tok;
+	unsigned int index = 0;
+	ssize_t r;
+	u32 id;
+
+	*size = 0;
+	if (!count)
+		return 0;
+
+	dup_buf = kstrndup(buf, count, GFP_KERNEL);
+	if (unlikely(!dup_buf))
+		return -ENOMEM;
+	str = dup_buf;
+	do {
+		str += strspn(str, " ");
+		tok = strsep(&str, ", ");
+		r = kstrtou32(tok, 10, &id);
+		if (r < 0)
+			goto err;
+		if (index >= max_size) {
+			r = -ENOSPC;
+			goto err;
+		}
+		if (id >= max_id) {
+			r = -ERANGE;
+			goto err;
+		}
+		array[index++] = id;
+	} while (str);
+	kfree(dup_buf);
+
+	*size = index;
+	return count;
+
+err:
+	kfree(dup_buf);
+	return r;
+}
+#define UALINK_IDARRAY_STORE(prefix, name, field, size, max_id)		\
+static ssize_t ualink_##prefix##_##name##_store(struct kobject *kobj,	\
+						struct kobj_attribute *attr,\
+						const char *buf, size_t count)\
+{									\
+	struct amdgpu_ualink_##prefix *info = to_ualink_##prefix(kobj);	\
+	u32 new_array[ARRAY_SIZE(info->field)];				\
+	u32 new_size;							\
+	ssize_t r = store_idarray(new_array, &new_size, max_id,		\
+				  ARRAY_SIZE(new_array), buf, count);	\
+									\
+	if (r >= 0) {							\
+		memcpy(info->field, new_array, sizeof(new_array));	\
+		info->size = new_size;					\
+	}								\
+	return r;							\
+}
+#endif
+
 #define UALINK_UUID_SHOW(prefix, name, field)				\
 static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 					       struct kobj_attribute *attr,\
@@ -98,6 +185,19 @@ static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 									\
 	return sysfs_emit(buf, "%pU\n", &info->field);			\
 }
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+#define UALINK_UUID_STORE(prefix, name, field)				\
+static ssize_t ualink_##prefix##_##name##_store(struct kobject *kobj,	\
+						struct kobj_attribute *attr,\
+						const char *buf, size_t count)\
+{									\
+	struct amdgpu_ualink_##prefix *info = to_ualink_##prefix(kobj);	\
+	int r;								\
+									\
+	r = uuid_parse(buf, &info->field);				\
+	return r ? r : count;						\
+}
+#endif
 
 static ssize_t show_enum(u32 x, const char * const values[], unsigned int n,
 			 char *buf)
@@ -180,8 +280,101 @@ static const struct kobj_type ualink_info_ktype = {
 	.sysfs_ops = &kobj_sysfs_ops
 };
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+UALINK_VALUE_SHOW(ppod_setup, accel_id,  ppod.accel_id,  "%u");
+UALINK_VALUE_SHOW(ppod_setup, bandwidth, ppod.bandwidth, "%u");
+UALINK_VALUE_SHOW(ppod_setup, latency,   ppod.latency,   "%u");
+UALINK_UUID_SHOW(ppod_setup, ppod_id, ppod.id);
+UALINK_VALUE_SHOW(ppod_setup, ppod_size, ppod.size,      "%u");
+UALINK_IDARRAY_SHOW(ppod_setup, local_accels, local_accels, n_local_accels);
+
+UALINK_VALUE_STORE(ppod_setup, accel_id,  ppod.accel_id,  u32, 10);
+UALINK_VALUE_STORE(ppod_setup, bandwidth, ppod.bandwidth, u32, 10);
+UALINK_VALUE_STORE(ppod_setup, latency,   ppod.latency,   u32, 10);
+UALINK_UUID_STORE(ppod_setup, ppod_id,   ppod.id);
+UALINK_VALUE_STORE(ppod_setup, ppod_size, ppod.size,      u32, 10);
+UALINK_IDARRAY_STORE(ppod_setup, local_accels, local_accels, n_local_accels,
+		     AMDGPU_UALINK_ACCEL_MAX);
+
+static enum amdgpu_ualink_accel_state
+check_ppod_state(struct amdgpu_device *adev,
+		 const struct amdgpu_ualink_ppod_setup *setup)
+{
+	if (setup->ppod.size <= 0 || setup->ppod.size > AMDGPU_UALINK_ACCEL_MAX) {
+		dev_dbg(adev->dev, "pPod size %u out of range [1..%u]\n",
+			setup->ppod.size, AMDGPU_UALINK_ACCEL_MAX);
+		return AMDGPU_UALINK_ACCEL_STATE_UNCONFIGURED;
+	}
+	if (setup->ppod.accel_id >= setup->ppod.size) {
+		dev_dbg(adev->dev, "Accelerator ID %u greater or equal pPod size %u\n",
+			setup->ppod.accel_id, setup->ppod.size);
+		return AMDGPU_UALINK_ACCEL_STATE_UNCONFIGURED;
+	}
+	return AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
+}
+
+static ssize_t ualink_ppod_setup_commit_store(struct kobject *kobj,
+					      struct kobj_attribute *attr,
+					      const char *buf, size_t count)
+{
+	struct amdgpu_ualink_ppod_setup *setup = to_ualink_ppod_setup(kobj);
+	struct amdgpu_ualink_info *info = to_ualink_info(kobj->parent);
+	struct device *dev = kobj_to_dev(info->kobj.parent);
+	struct drm_device *ddev = dev_get_drvdata(dev);
+	struct amdgpu_device *adev = drm_to_adev(ddev);
+
+	if (!sysfs_streq(buf, "true"))
+		return -EINVAL;
+
+	/* TODO: Send configuration to ASP */
+
+	info->ppod = setup->ppod;
+	info->accel_state = check_ppod_state(adev, setup);
+
+	/* TODO: If accel_state was ACTIVE, reset all connections */
+
+	return count;
+}
+
+#define UALINK_PPOD_SETUP_ATTR(name) __ATTR(name, 0600,			\
+		ualink_ppod_setup_##name##_show,			\
+		ualink_ppod_setup_##name##_store)
+static struct kobj_attribute ualink_ppod_setup_accel_id  = UALINK_PPOD_SETUP_ATTR(accel_id);
+static struct kobj_attribute ualink_ppod_setup_bandwidth = UALINK_PPOD_SETUP_ATTR(bandwidth);
+static struct kobj_attribute ualink_ppod_setup_latency   = UALINK_PPOD_SETUP_ATTR(latency);
+static struct kobj_attribute ualink_ppod_setup_ppod_id   = UALINK_PPOD_SETUP_ATTR(ppod_id);
+static struct kobj_attribute ualink_ppod_setup_ppod_size = UALINK_PPOD_SETUP_ATTR(ppod_size);
+static struct kobj_attribute ualink_ppod_setup_local_accels = UALINK_PPOD_SETUP_ATTR(local_accels);
+static struct kobj_attribute ualink_ppod_setup_commit = __ATTR(commit, 0200, NULL,
+							       ualink_ppod_setup_commit_store);
+
+static const struct attribute *ualink_ppod_setup_attrs[] = {
+	&ualink_ppod_setup_accel_id.attr,
+	&ualink_ppod_setup_bandwidth.attr,
+	&ualink_ppod_setup_latency.attr,
+	&ualink_ppod_setup_ppod_id.attr,
+	&ualink_ppod_setup_ppod_size.attr,
+	&ualink_ppod_setup_local_accels.attr,
+	&ualink_ppod_setup_commit.attr,
+	NULL
+};
+#endif
+
+static void ualink_ppod_setup_release(struct kobject *kobj)
+{
+	struct amdgpu_ualink_ppod_setup *setup = to_ualink_ppod_setup(kobj);
+
+	kfree(setup);
+}
+
+static const struct kobj_type ualink_ppod_setup_ktype = {
+	.release = ualink_ppod_setup_release,
+	.sysfs_ops = &kobj_sysfs_ops
+};
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 {
+	struct amdgpu_ualink_ppod_setup *ppod_setup = NULL;
 	struct amdgpu_ualink_info *info = NULL;
 	int r;
 
@@ -203,10 +396,34 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 	if (r)
 		goto err_del_info;
 
+	ppod_setup = kzalloc(sizeof(*ppod_setup), GFP_KERNEL);
+	if (!ppod_setup) {
+		r = -ENOMEM;
+		goto err_remove_info_files;
+	}
+	r = kobject_init_and_add(&ppod_setup->kobj, &ualink_ppod_setup_ktype,
+				 &info->kobj, "setup");
+	if (r)
+		goto err_put_ppod_setup;
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+	r = sysfs_create_files(&ppod_setup->kobj, ualink_ppod_setup_attrs);
+	if (r)
+		goto err_del_ppod_setup;
+#endif
+
+	adev->ualink.setup = ppod_setup;
 	adev->ualink.info = info;
 
 	return r;
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+err_del_ppod_setup:
+	kobject_del(&ppod_setup->kobj);
+#endif
+err_put_ppod_setup:
+	kobject_put(&ppod_setup->kobj);
+err_remove_info_files:
+	sysfs_remove_files(&info->kobj, ualink_info_attrs);
 err_del_info:
 	kobject_del(&info->kobj);
 err_put_info:
@@ -216,6 +433,15 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev)
 {
+	if (adev->ualink.setup) {
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+		sysfs_remove_files(&adev->ualink.setup->kobj,
+				   ualink_ppod_setup_attrs);
+#endif
+		kobject_del(&adev->ualink.setup->kobj);
+		kobject_put(&adev->ualink.setup->kobj);
+		adev->ualink.setup = NULL;
+	}
 	if (adev->ualink.info) {
 		sysfs_remove_files(&adev->ualink.info->kobj,
 				   ualink_info_attrs);
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +11 -0
index 9c492e7518d9f..1c1fb41e75140 100644
@@ -87,8 +87,19 @@ struct amdgpu_ualink_info {
 };
 #define to_ualink_info(ko) container_of(ko, struct amdgpu_ualink_info, kobj)
 
+/* UALink physical pod setup */
+struct amdgpu_ualink_ppod_setup {
+	struct kobject kobj;
+	struct amdgpu_ualink_ppod_info ppod;
+	/* Local accelerator array indexed by socket ID */
+	u32 n_local_accels;
+	u32 local_accels[AMDGPU_UALINK_LOCAL_ACCELS_MAX];
+};
+#define to_ualink_ppod_setup(ko) container_of(ko, struct amdgpu_ualink_ppod_setup, kobj)
+
 struct amdgpu_ualink_mgr {
 	struct amdgpu_ualink_info *info;
+	struct amdgpu_ualink_ppod_setup *setup;
 };
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#5 lore raw

PATCH v2 5/95 drm/amdgpu: Add sysfs API for UALink virtual pod config

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

<device>/ualink/config: Directory for staging virtual pod configuration attributes. Write "true" to commit to apply settings

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 403 ++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   8 +
 2 files changed, 410 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +402 -1
index 92c449f2004d4..d60cd997dd3c6 100644
@@ -94,6 +94,57 @@ static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 	return show_idbits(info->field, sizeof(info->field)*8, buf);	\
 }
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+static ssize_t store_idbits(unsigned long *bits, unsigned int nbits,
+			    const char *buf, size_t count)
+{
+	char *dup_buf, *str, *tok;
+	ssize_t r;
+	u32 id;
+
+	bitmap_zero(bits, nbits);
+	if (!count)
+		return 0;
+
+	dup_buf = kstrndup(buf, count, GFP_KERNEL);
+	if (unlikely(!dup_buf))
+		return -ENOMEM;
+	str = dup_buf;
+	do {
+		str += strspn(str, " ");
+		tok = strsep(&str, ", ");
+		r = kstrtou32(tok, 10, &id);
+		if (r < 0)
+			goto err;
+		if (id >= nbits) {
+			r = -ERANGE;
+			goto err;
+		}
+		set_bit(id, bits);
+	} while (str);
+	kfree(dup_buf);
+
+	return count;
+
+err:
+	kfree(dup_buf);
+	return r;
+}
+#define UALINK_IDBITS_STORE(prefix, name, field, nbits)			\
+static ssize_t ualink_##prefix##_##name##_store(struct kobject *kobj,	\
+						struct kobj_attribute *attr,\
+						const char *buf, size_t count)\
+{									\
+	struct amdgpu_ualink_##prefix *info = to_ualink_##prefix(kobj);	\
+	DECLARE_BITMAP(bits, nbits);					\
+	ssize_t r = store_idbits(bits, nbits, buf, count);		\
+									\
+	if (r >= 0)							\
+		bitmap_copy(info->field, bits, nbits);			\
+	return r;							\
+}
+#endif
+
 static ssize_t show_idarray(const u32 *array, u32 size, char *buf)
 {
 	ssize_t len = 0;
@@ -215,6 +266,32 @@ static ssize_t ualink_##prefix##_##name##_show(struct kobject *kobj,	\
 			 ARRAY_SIZE(ualink_##name##_values), buf);	\
 }
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+static ssize_t store_enum(u32 *x, const char * const values[], unsigned int n,
+			  const char *buf, size_t count)
+{
+	unsigned int i;
+
+	for (i = 0; i < n; i++) {
+		if (values[i] && sysfs_streq(buf, values[i])) {
+			*x = i;
+			return count;
+		}
+	}
+	return -EINVAL;
+}
+#define UALINK_ENUM_STORE(prefix, name, field)				\
+static ssize_t ualink_##prefix##_##name##_store(struct kobject *kobj,	\
+						struct kobj_attribute *attr,\
+						const char *buf, size_t count)\
+{									\
+	struct amdgpu_ualink_##prefix *info = to_ualink_##prefix(kobj);	\
+									\
+	return store_enum(&info->field, ualink_##name##_values,		\
+			 ARRAY_SIZE(ualink_##name##_values),		\
+			 buf, count);					\
+}
+#endif
 static const char * const ualink_link_type_values[] = {
 	"UALoE", "UALink"
 };
@@ -372,8 +449,299 @@ static const struct kobj_type ualink_ppod_setup_ktype = {
 	.sysfs_ops = &kobj_sysfs_ops
 };
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+UALINK_VALUE_SHOW(vpod_config, vpod_id,   vpod.id,   "%u");
+UALINK_VALUE_SHOW(vpod_config, vpod_size, vpod.size, "%u");
+UALINK_IDBITS_SHOW(vpod_config, vpod_active_accels, vpod.active_accel_bits);
+UALINK_ENUM_SHOW(vpod_config, addr_mode, vpod.addr_mode);
+
+UALINK_VALUE_STORE(vpod_config, vpod_id,   vpod.id,   u32, 10);
+UALINK_VALUE_STORE(vpod_config, vpod_size, vpod.size, u32, 10);
+UALINK_IDBITS_STORE(vpod_config, vpod_active_accels, vpod.active_accel_bits,
+		    AMDGPU_UALINK_ACCEL_MAX);
+UALINK_ENUM_STORE(vpod_config, addr_mode, vpod.addr_mode);
+
+static bool check_vpod_info(struct amdgpu_device *adev,
+			    const struct amdgpu_ualink_info *info)
+{
+	unsigned int weight;
+
+	if (info->accel_state < AMDGPU_UALINK_ACCEL_STATE_CONFIGURED) {
+		dev_dbg(adev->dev, "pPod is not yet configured\n");
+		return false;
+	}
+	if (info->vpod.id >= AMDGPU_UALINK_ACCEL_MAX) {
+		dev_dbg(adev->dev, "vPod ID %u out of range [0..%u]\n",
+			info->vpod.id, AMDGPU_UALINK_ACCEL_MAX - 1);
+		return false;
+	}
+	if (info->vpod.size == 0 || info->vpod.size > info->ppod.size) {
+		dev_dbg(adev->dev, "vPod size %u out of range [1..%u]\n",
+			info->vpod.size, info->ppod.size);
+		return false;
+	}
+	if (info->vpod.addr_mode >= AMDGPU_UALINK_ADDR_MODE_MAX) {
+		dev_dbg(adev->dev, "Invalid addr mode %u\n", info->vpod.id);
+		return false;
+	}
+	weight = bitmap_weight(info->vpod.active_accel_bits, AMDGPU_UALINK_ACCEL_MAX);
+	if (weight != info->vpod.size) {
+		dev_dbg(adev->dev, "vPod size doesn't match vpod_active_accels list: %u != %u\n",
+			info->vpod.size, weight);
+		return false;
+	}
+	if (!test_bit(info->ppod.accel_id, info->vpod.active_accel_bits)) {
+		dev_dbg(adev->dev, "Accelerator ID %u not listed in vpod_active_accels\n",
+			info->ppod.accel_id);
+		return false;
+	}
+
+	return true;
+}
+
+static struct amdgpu_device *find_peer_adev(unsigned int accel_id)
+{
+	unsigned int i;
+
+	for (i = 0; i < mgpu_info.num_gpu; i++) {
+		struct amdgpu_device *peer_adev = mgpu_info.gpu_ins[i].adev;
+
+		if (peer_adev->ualink.info &&
+		    peer_adev->ualink.info->ppod.accel_id == accel_id)
+			return peer_adev;
+	}
+
+	return NULL;
+}
+
+static bool check_local_vpod_integrity(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_info *info = adev->ualink.info;
+	struct amdgpu_ualink_info *peer_info;
+	struct amdgpu_device *peer_adev;
+	unsigned int accel_id;
+	unsigned int i;
+
+	if (!check_vpod_info(adev, info))
+		return false;
+
+	/* Check that all local accelerators listed in vpod_active_accels have
+	 * matching pod IDs
+	 */
+	for_each_set_bit(accel_id, info->vpod.active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+
+		if (accel_id == info->ppod.accel_id)
+			continue;
+
+		peer_adev = find_peer_adev(accel_id);
+		if (!peer_adev)
+			continue;
+		peer_info = peer_adev->ualink.info;
+
+		if (peer_info->vpod.id != info->vpod.id) {
+			dev_dbg(adev->dev, "Peer %u vpod_id doesn't match: %u != %u",
+				accel_id, peer_info->vpod.id, info->vpod.id);
+			return false;
+		}
+		if (!uuid_equal(&peer_info->ppod.id, &info->ppod.id)) {
+			dev_dbg(adev->dev, "Peer %u ppod_id doesn't match: %pU != %pU",
+				accel_id, &peer_info->ppod.id, &info->ppod.id);
+			return false;
+		}
+	}
+
+	/* Derive local accels from pod IDs of GPUs in mgpu_info */
+	info->n_local_accels = 0;
+	for (i = 0; i < mgpu_info.num_gpu &&
+		    info->n_local_accels < AMDGPU_UALINK_LOCAL_ACCELS_MAX;
+	     i++) {
+		peer_adev = mgpu_info.gpu_ins[i].adev;
+		peer_info = peer_adev->ualink.info;
+
+		if (peer_adev == adev ||
+		    (peer_info && peer_info->vpod.id == info->vpod.id &&
+		     uuid_equal(&peer_info->ppod.id, &info->ppod.id)))
+			info->local_accels[info->n_local_accels++] =
+				peer_info->ppod.accel_id;
+	}
+
+	/* Then check consistency of the vpod information on all those GPUs */
+	for (i = 0; i < info->n_local_accels; i++) {
+		unsigned int j;
+
+		for (j = i + 1; j < info->n_local_accels; j++) {
+			if (info->local_accels[j] == accel_id) {
+				dev_dbg(adev->dev,
+					"Accelerator ID %u is not unique among local GPUs\n",
+					accel_id);
+				return false;
+			}
+		}
+
+		accel_id = info->local_accels[i];
+
+		/* Skip this GPU, we are looking for our peers */
+		if (accel_id == info->ppod.accel_id)
+			continue;
+
+		peer_adev = find_peer_adev(accel_id);
+		if (WARN_ON(!peer_adev || !peer_adev->ualink.info))
+			/* info->local_accels we just built is corrupted? */
+			return false;
+		peer_info = peer_adev->ualink.info;
+
+		/* Check peer vpod info and consistency */
+		if (!check_vpod_info(peer_adev, peer_info))
+			return false;
+
+		if (peer_info->ppod.size != info->ppod.size) {
+			dev_dbg(adev->dev, "Peer %u ppod_size doesn't match: %u != %u\n",
+				accel_id, peer_info->ppod.size, info->ppod.size);
+			return false;
+		}
+		if (peer_info->vpod.size != info->vpod.size) {
+			dev_dbg(adev->dev, "Peer %u vpod_size doesn't match: %u != %u\n",
+				accel_id, peer_info->vpod.size, info->vpod.size);
+			return false;
+		}
+		if (peer_info->vpod.addr_mode != info->vpod.addr_mode) {
+			dev_dbg(adev->dev, "Peer %u addr_mode doesn't match: %u != %u\n",
+				accel_id, peer_info->vpod.addr_mode, info->vpod.addr_mode);
+			return false;
+		}
+		if (!bitmap_equal(peer_info->vpod.active_accel_bits,
+				  info->vpod.active_accel_bits, AMDGPU_UALINK_ACCEL_MAX)) {
+			dev_dbg(adev->dev, "Peer %u vpod_active_accels don't match\n",
+				accel_id);
+			return false;
+		}
+
+		/* Update peer's local accelerator array */
+		peer_info->n_local_accels = info->n_local_accels;
+		memcpy(peer_info->local_accels, info->local_accels,
+		       sizeof(info->local_accels));
+	}
+	return true;
+}
+
+static void activate_accelerator(struct amdgpu_device *adev)
+{
+	if (adev->ualink.info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_READY)
+		return;
+
+	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
+}
+
+static void deactivate_accelerator(struct amdgpu_device *adev)
+{
+	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_READY)
+		return;
+
+	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
+}
+
+static void activate_local_vpod(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_info *info = adev->ualink.info;
+	struct amdgpu_device *peer_adev;
+	unsigned int accel_id;
+	unsigned int i;
+
+	for (i = 0; i < info->n_local_accels; i++) {
+		accel_id = info->local_accels[i];
+
+		peer_adev = find_peer_adev(accel_id);
+		if (WARN_ON(!peer_adev || !peer_adev->ualink.info))
+			/* info->local_accels is corrupted? */
+			continue;
+
+		activate_accelerator(peer_adev);
+	}
+}
+
+static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
+					       struct kobj_attribute *attr,
+					       const char *buf, size_t count)
+{
+	struct amdgpu_ualink_vpod_config *config = to_ualink_vpod_config(kobj);
+	struct amdgpu_ualink_info *info = to_ualink_info(kobj->parent);
+	struct device *dev = kobj_to_dev(info->kobj.parent);
+	struct drm_device *ddev = dev_get_drvdata(dev);
+	struct amdgpu_device *adev = drm_to_adev(ddev);
+
+	if (!sysfs_streq(buf, "true"))
+		return -EINVAL;
+	if (info->accel_state < AMDGPU_UALINK_ACCEL_STATE_CONFIGURED) {
+		dev_dbg(adev->dev, "Ualink ppod is not yet configured\n");
+		return -EINVAL;
+	}
+
+	/* TODO: instead of just copying the info, call ASP vpod config API
+	 * and query ASP for the updated info
+	 */
+	info->vpod = config->vpod;
+
+	/* The integrity check makes sure each new GPU is consistent with the
+	 * other GPUs already in the vPod. All known local GPUs can become
+	 * "ready" at the same time.
+	 *
+	 * Misconfiguration of one GPU does not reduce the state of other GPUs
+	 * already in the vPod.
+	 */
+	mutex_lock(&mgpu_info.mutex);
+	if (check_local_vpod_integrity(adev))
+		activate_local_vpod(adev);
+	else if (info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_CONFIGURED)
+		deactivate_accelerator(adev);
+	mutex_unlock(&mgpu_info.mutex);
+
+	/* TODO: Update KFD topology for in-domain link */
+
+	/* TODO: If state was ACTIVE:
+	 * - If addr_mode changed, reset all connections, reset state to READY
+	 * - If accelerators were removed, reset those links, but keep state ACTIVE
+	 * - If accelerators were added, keep state ACTIVE
+	 */
+
+	return count;
+}
+
+#define UALINK_VPOD_CONFIG_ATTR(name) __ATTR(name, 0600,		\
+		ualink_vpod_config_##name##_show,			\
+		ualink_vpod_config_##name##_store)
+static struct kobj_attribute ualink_vpod_config_vpod_id   = UALINK_VPOD_CONFIG_ATTR(vpod_id);
+static struct kobj_attribute ualink_vpod_config_vpod_size = UALINK_VPOD_CONFIG_ATTR(vpod_size);
+static struct kobj_attribute ualink_vpod_config_vpod_active_accels =
+						UALINK_VPOD_CONFIG_ATTR(vpod_active_accels);
+static struct kobj_attribute ualink_vpod_config_addr_mode = UALINK_VPOD_CONFIG_ATTR(addr_mode);
+static struct kobj_attribute ualink_vpod_config_commit = __ATTR(commit, 0200, NULL,
+								ualink_vpod_config_commit_store);
+
+static const struct attribute *ualink_vpod_config_attrs[] = {
+	&ualink_vpod_config_vpod_id.attr,
+	&ualink_vpod_config_vpod_size.attr,
+	&ualink_vpod_config_vpod_active_accels.attr,
+	&ualink_vpod_config_addr_mode.attr,
+	&ualink_vpod_config_commit.attr,
+	NULL
+};
+#endif
+
+static void ualink_vpod_config_release(struct kobject *kobj)
+{
+	struct amdgpu_ualink_vpod_config *config = to_ualink_vpod_config(kobj);
+
+	kfree(config);
+}
+
+static const struct kobj_type ualink_vpod_config_ktype = {
+	.release = ualink_vpod_config_release,
+	.sysfs_ops = &kobj_sysfs_ops
+};
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 {
+	struct amdgpu_ualink_vpod_config *vpod_config = NULL;
 	struct amdgpu_ualink_ppod_setup *ppod_setup = NULL;
 	struct amdgpu_ualink_info *info = NULL;
 	int r;
@@ -411,15 +779,39 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 		goto err_del_ppod_setup;
 #endif
 
+	vpod_config = kzalloc(sizeof(*vpod_config), GFP_KERNEL);
+	if (!vpod_config) {
+		r = -ENOMEM;
+		goto err_remove_ppod_setup_files;
+	}
+	r = kobject_init_and_add(&vpod_config->kobj, &ualink_vpod_config_ktype,
+				 &info->kobj, "config");
+	if (r)
+		goto err_put_vpod_config;
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+	r = sysfs_create_files(&vpod_config->kobj, ualink_vpod_config_attrs);
+	if (r)
+		goto err_del_vpod_config;
+#endif
+
+	adev->ualink.config = vpod_config;
 	adev->ualink.setup = ppod_setup;
 	adev->ualink.info = info;
 
 	return r;
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+err_del_vpod_config:
+	kobject_del(&vpod_config->kobj);
+#endif
+err_put_vpod_config:
+	kobject_put(&vpod_config->kobj);
+err_remove_ppod_setup_files:
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+	sysfs_remove_files(&ppod_setup->kobj, ualink_ppod_setup_attrs);
 err_del_ppod_setup:
-	kobject_del(&ppod_setup->kobj);
 #endif
+	kobject_del(&ppod_setup->kobj);
 err_put_ppod_setup:
 	kobject_put(&ppod_setup->kobj);
 err_remove_info_files:
@@ -433,6 +825,15 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev)
 {
+	if (adev->ualink.config) {
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+		sysfs_remove_files(&adev->ualink.config->kobj,
+				   ualink_vpod_config_attrs);
+#endif
+		kobject_del(&adev->ualink.config->kobj);
+		kobject_put(&adev->ualink.config->kobj);
+		adev->ualink.config = NULL;
+	}
 	if (adev->ualink.setup) {
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
 		sysfs_remove_files(&adev->ualink.setup->kobj,
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +8 -0
index 1c1fb41e75140..01a36b058ea08 100644
@@ -97,9 +97,17 @@ struct amdgpu_ualink_ppod_setup {
 };
 #define to_ualink_ppod_setup(ko) container_of(ko, struct amdgpu_ualink_ppod_setup, kobj)
 
+/* UAlink virtual pod config */
+struct amdgpu_ualink_vpod_config {
+	struct kobject kobj;
+	struct amdgpu_ualink_vpod_info vpod;
+};
+#define to_ualink_vpod_config(ko) container_of(ko, struct amdgpu_ualink_vpod_config, kobj)
+
 struct amdgpu_ualink_mgr {
 	struct amdgpu_ualink_info *info;
 	struct amdgpu_ualink_ppod_setup *setup;
+	struct amdgpu_ualink_vpod_config *config;
 };
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#6 lore raw

PATCH v2 6/95 drm/amdgpu: Add sysfs API for UALink station configuration

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

<device>/ualink/stations: Directory for staging UALink station/port configuration attributes:
- station_flags (currently only sets the number of port per station)
- lane_en_bitmap (hexadecimal bitmaps of lanes enabled per station)
- write "true" to commit to apply settings

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 154 ++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  16 +++
 2 files changed, 169 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +153 -1
index d60cd997dd3c6..3bffb3cb21681 100644
@@ -739,8 +739,127 @@ static const struct kobj_type ualink_vpod_config_ktype = {
 	.sysfs_ops = &kobj_sysfs_ops
 };
 
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+UALINK_VALUE_SHOW(station_config, flags, flags, "0x%x");
+
+UALINK_VALUE_STORE(station_config, flags, flags, u8, 16);
+
+static size_t lane_bitmap_to_string(char *str,
+		const struct amdgpu_ualink_station_config *stations)
+{
+	size_t i;
+
+	for (i = 0; i < stations->n_stations; i++)
+		str[i] = stations->lane_en_bitmap[i] +
+			(stations->lane_en_bitmap[i] < 10 ? '0' : 'a' - 10);
+	str[i] = '\0';
+
+	return i;
+}
+static ssize_t
+ualink_station_config_lane_en_bitmap_show(struct kobject *kobj,
+					  struct kobj_attribute *attr,
+					  char *buf)
+{
+	struct amdgpu_ualink_station_config *stations = to_ualink_station_config(kobj);
+	char str[AMDGPU_UALINK_STATIONS_MAX + 1];
+
+	lane_bitmap_to_string(str, stations);
+	return sysfs_emit(buf, "%s\n", str);
+}
+static ssize_t
+ualink_station_config_lane_en_bitmap_store(struct kobject *kobj,
+					   struct kobj_attribute *attr,
+					   const char *buf, size_t count)
+{
+	struct amdgpu_ualink_station_config *stations = to_ualink_station_config(kobj);
+	u8 lane_en_bitmap[AMDGPU_UALINK_STATIONS_MAX];
+	u32 n_stations = 0;
+	bool end = false;
+	size_t i;
+
+	for (i = 0; i < count; i++) {
+		/* Accept any number of \n in the end */
+		if (buf[i] == '\n') {
+			end = true;
+			continue;
+		}
+		/* Don't accept any more data after \n */
+		if (end)
+			return -EINVAL;
+		/* Don't accept more data than the array size */
+		if (i >= sizeof(lane_en_bitmap))
+			return -ENOSPC;
+		/* Accept hex digits */
+		if (buf[i] >= '0' && buf[i] <= '9')
+			lane_en_bitmap[n_stations++] = buf[i] - '0';
+		else if (buf[i] >= 'a' && buf[i] <= 'f')
+			lane_en_bitmap[n_stations++] = buf[i] + 10 - 'a';
+		else if (buf[i] >= 'A' && buf[i] <= 'F')
+			lane_en_bitmap[n_stations++] = buf[i] + 10 - 'A';
+		else
+			return -EINVAL;
+	}
+
+	memcpy(stations->lane_en_bitmap, lane_en_bitmap, sizeof(lane_en_bitmap));
+	stations->n_stations = n_stations;
+	return i;
+}
+
+static ssize_t ualink_station_config_commit_store(struct kobject *kobj,
+						  struct kobj_attribute *attr,
+						  const char *buf, size_t count)
+{
+	struct amdgpu_ualink_station_config *stations = to_ualink_station_config(kobj);
+	struct amdgpu_ualink_info *info = to_ualink_info(kobj->parent);
+	struct device *dev = kobj_to_dev(info->kobj.parent);
+	struct drm_device *ddev = dev_get_drvdata(dev);
+	struct amdgpu_device *adev = drm_to_adev(ddev);
+
+	/* DF reconfiguration does not interact with accelerator state */
+
+	if (!sysfs_streq(buf, "true"))
+		return -EINVAL;
+
+	/* TODO: Send configuration to ASP */
+	(void)adev;
+	(void)stations;
+
+	return count;
+}
+
+#define UALINK_STATION_CONFIG_ATTR(name) __ATTR(name, 0600,		\
+		ualink_station_config_##name##_show,			\
+		ualink_station_config_##name##_store)
+static struct kobj_attribute ualink_station_config_flags  = UALINK_STATION_CONFIG_ATTR(flags);
+static struct kobj_attribute ualink_station_config_lane_en_bitmap =
+							UALINK_STATION_CONFIG_ATTR(lane_en_bitmap);
+static struct kobj_attribute ualink_station_config_commit =
+				__ATTR(commit, 0200, NULL, ualink_station_config_commit_store);
+
+static const struct attribute *ualink_station_config_attrs[] = {
+	&ualink_station_config_flags.attr,
+	&ualink_station_config_lane_en_bitmap.attr,
+	&ualink_station_config_commit.attr,
+	NULL
+};
+#endif
+
+static void ualink_station_config_release(struct kobject *kobj)
+{
+	struct amdgpu_ualink_station_config *stations = to_ualink_station_config(kobj);
+
+	kfree(stations);
+}
+
+static const struct kobj_type ualink_station_config_ktype = {
+	.release = ualink_station_config_release,
+	.sysfs_ops = &kobj_sysfs_ops
+};
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 {
+	struct amdgpu_ualink_station_config *stations = NULL;
 	struct amdgpu_ualink_vpod_config *vpod_config = NULL;
 	struct amdgpu_ualink_ppod_setup *ppod_setup = NULL;
 	struct amdgpu_ualink_info *info = NULL;
@@ -794,6 +913,22 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 		goto err_del_vpod_config;
 #endif
 
+	stations = kzalloc(sizeof(*stations), GFP_KERNEL);
+	if (!stations) {
+		r = -ENOMEM;
+		goto err_remove_vpod_config_files;
+	}
+	r = kobject_init_and_add(&stations->kobj, &ualink_station_config_ktype,
+				 &info->kobj, "stations");
+	if (r)
+		goto err_put_stations;
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+	r = sysfs_create_files(&stations->kobj, ualink_station_config_attrs);
+	if (r)
+		goto err_del_stations;
+#endif
+
+	adev->ualink.stations = stations;
 	adev->ualink.config = vpod_config;
 	adev->ualink.setup = ppod_setup;
 	adev->ualink.info = info;
@@ -801,9 +936,17 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 	return r;
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+err_del_stations:
+	kobject_del(&stations->kobj);
+#endif
+err_put_stations:
+	kobject_put(&stations->kobj);
+err_remove_vpod_config_files:
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+	sysfs_remove_files(&vpod_config->kobj, ualink_vpod_config_attrs);
 err_del_vpod_config:
-	kobject_del(&vpod_config->kobj);
 #endif
+	kobject_del(&vpod_config->kobj);
 err_put_vpod_config:
 	kobject_put(&vpod_config->kobj);
 err_remove_ppod_setup_files:
@@ -825,6 +968,15 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev)
 {
+	if (adev->ualink.stations) {
+#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+		sysfs_remove_files(&adev->ualink.stations->kobj,
+				   ualink_station_config_attrs);
+#endif
+		kobject_del(&adev->ualink.stations->kobj);
+		kobject_put(&adev->ualink.stations->kobj);
+		adev->ualink.stations = NULL;
+	}
 	if (adev->ualink.config) {
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
 		sysfs_remove_files(&adev->ualink.config->kobj,
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +16 -0
index 01a36b058ea08..2d685f6dd7e7d 100644
@@ -33,6 +33,7 @@ struct amdgpu_device;
 
 #define AMDGPU_UALINK_ACCEL_MAX 256
 #define AMDGPU_UALINK_LOCAL_ACCELS_MAX 8
+#define AMDGPU_UALINK_STATIONS_MAX 64
 
 enum amdgpu_ualink_type {
 	AMDGPU_UALINK_TYPE_UALOE = 0,
@@ -104,10 +105,25 @@ struct amdgpu_ualink_vpod_config {
 };
 #define to_ualink_vpod_config(ko) container_of(ko, struct amdgpu_ualink_vpod_config, kobj)
 
+/* UALink station configuration */
+struct amdgpu_ualink_station_config {
+	struct kobject kobj;
+	/* Station configuration flags
+	 * bits [3:0]: PortPerStation (PPS) - 1, 2, or 4
+	 * bits [7:4]: Reserved
+	 */
+	u8 flags;
+	u8 n_stations;
+	/* bitmap or enabled lanes for each station in logical station order */
+	u8 lane_en_bitmap[AMDGPU_UALINK_STATIONS_MAX];
+};
+#define to_ualink_station_config(ko) container_of(ko, struct amdgpu_ualink_station_config, kobj)
+
 struct amdgpu_ualink_mgr {
 	struct amdgpu_ualink_info *info;
 	struct amdgpu_ualink_ppod_setup *setup;
 	struct amdgpu_ualink_vpod_config *config;
+	struct amdgpu_ualink_station_config *stations;
 };
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#7 lore raw

PATCH v2 7/95 drm/amdgpu: Add UALink manager core infrastructure

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

This patch introduces AMDGPU UALink manager core infrastructure which will be responsible for sharing buffer objects across multiple GPUs, with each GPU running their own OS, in a rack scale setup. The core infrastructure of the ualink manager will include:
- Store BOs exported from a GPU.
- Store BOs imported by a GPU.
- Store connection state with different GPUs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 42 +++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h | 41 ++++++++++++++++++++-
 2 files changed, 81 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +41 -1
index 3bffb3cb21681..d91365399c767 100644
@@ -22,9 +22,10 @@
  *
  */
 
+#include <linux/xarray.h>
+#include "amdgpu.h"
 #include "amdgpu_ualink.h"
 #include "amdgpu_xgmi.h"
-#include "amdgpu.h"
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
@@ -1003,3 +1004,42 @@ void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev)
 		adev->ualink.info = NULL;
 	}
 }
+
+int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
+{
+	int i, r;
+
+	adev->ualink.npa_wq = alloc_workqueue("NPA WQ", WQ_UNBOUND, 0);
+	if (unlikely(!adev->ualink.npa_wq)) {
+		dev_err(adev->dev, "Failed to allocate NPA WQ\n");
+		return -ENOMEM;
+	}
+
+	xa_init_flags(&adev->ualink.exp_xa, XA_FLAGS_LOCK_BH);
+	xa_init_flags(&adev->ualink.imp_xa, XA_FLAGS_LOCK_BH);
+	xa_init_flags(&adev->ualink.handle_invalid_xa, XA_FLAGS_LOCK_BH);
+
+	for (i = 0; i < AMDGPU_UALINK_ACCEL_MAX; i++) {
+		init_completion(&adev->ualink.conn_state[i].hello_done);
+		mutex_init(&adev->ualink.conn_state[i].lock);
+		adev->ualink.conn_state[i].state = AMDGPU_UALINK_CONN_NOT_READY;
+		INIT_LIST_HEAD(&adev->ualink.exp_handles_list[i]);
+		INIT_LIST_HEAD(&adev->ualink.imp_handles_list[i]);
+	}
+
+	return 0;
+}
+
+void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
+{
+	int i;
+
+	xa_destroy(&adev->ualink.exp_xa);
+	xa_destroy(&adev->ualink.imp_xa);
+	xa_destroy(&adev->ualink.handle_invalid_xa);
+
+	for (i = 0; i < AMDGPU_UALINK_ACCEL_MAX; i++)
+		mutex_destroy(&adev->ualink.conn_state[i].lock);
+
+	destroy_workqueue(adev->ualink.npa_wq);
+}
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +40 -1

60 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#8 lore raw

PATCH v2 8/95 drm/amdgpu: Implement PSP cmd UAL_GET_CONFIG

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

To query UALink configuration information from PSP.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 58 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  6 +++
 2 files changed, 64 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +58 -0
index 5c82d1d808157..006ec1a241f48 100644
@@ -829,6 +829,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "PERF MONITORING HW";
 	case GFX_CMD_ID_UAL_GET_INTERFACE_VER:
 		return "UAL_GET_INTERFACE_VER";
+	case GFX_CMD_ID_UAL_GET_CONFIG:
+		return "UAL_GET_CONFIG";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1212,6 +1214,62 @@ int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver)
 	return ret;
 }
 
+int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
+		       struct amdgpu_ualink_info *info)
+{
+	struct psp_gfx_get_config_ual_v1 *ual_config;
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	ual_config = psp->cmd_ext_resp_mem;
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_GET_CONFIG;
+	cmd->cmd.cmd_get_config_ual.ual_cfg_addr_lo = lower_32_bits(psp->cmd_ext_resp_mc_addr);
+	cmd->cmd.cmd_get_config_ual.ual_cfg_addr_hi = upper_32_bits(psp->cmd_ext_resp_mc_addr);
+	cmd->cmd.cmd_get_config_ual.ual_cfg_size = sizeof(*ual_config);
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && !cmd->resp.status) {
+		WARN_ON(cmd->resp.uresp.get_config_ual.resp_size < sizeof(*ual_config));
+
+		info->link_type = (enum amdgpu_ualink_type)ual_config->link_type;
+
+		info->ppod.accel_id = ual_config->accelerator_id;
+		info->ppod.bandwidth = ual_config->bandwidth;
+		info->ppod.latency = ual_config->latency;
+		info->ppod.size = ual_config->ppod_size;
+		memcpy(&info->ppod.id, ual_config->ppod_id, sizeof(info->ppod.id));
+
+		info->vpod.id = ual_config->vpod_id;
+		info->vpod.size = ual_config->vpod_size;
+		info->vpod.addr_mode = ual_config->addr_mode;
+		bitmap_from_arr32(info->vpod.active_accel_bits,
+				  ual_config->vpod_active_accelerators,
+				  min(AMDGPU_UALINK_ACCEL_MAX, PSP_GFX_UAL_MAX_ACC_BIT_MASK*32));
+		/* Ensure no uninitialized data in the bitmap, even if these
+		 * constants change in the future.
+		 */
+		if (AMDGPU_UALINK_ACCEL_MAX > PSP_GFX_UAL_MAX_ACC_BIT_MASK*32)
+			bitmap_clear(info->vpod.active_accel_bits, PSP_GFX_UAL_MAX_ACC_BIT_MASK*32,
+				     AMDGPU_UALINK_ACCEL_MAX - PSP_GFX_UAL_MAX_ACC_BIT_MASK*32);
+	} else if (!ret) {
+		ret = -EINVAL;
+	}
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +6 -0
index 5968595647fef..66b57c871e1a5 100644
@@ -705,6 +705,12 @@ int amdgpu_psp_irq_mgr_register(
 	struct amdgpu_psp_irq_mgr *mgr,
 	const struct amdgpu_psp_irq_handler *handlers, int count,
 	const struct amdgpu_psp_irq_handler *default_handler);
+
+struct amdgpu_ualink_info;
+
 int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
+int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
+		       struct amdgpu_ualink_info *info);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#9 lore raw

PATCH v2 9/95 drm/amdgpu: Query initial UALink config from PSP

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Query the initial UALink configuration info from PSP at driver initialization time.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 30 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  3 +++
 drivers/gpu/drm/amd/amdgpu/soc_v1_0.c      |  5 ++++
 3 files changed, 38 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +30 -0
index d91365399c767..41d39f828dfec 100644
@@ -29,6 +29,36 @@
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
+int amdgpu_ualink_init(struct amdgpu_device *adev)
+{
+	int r;
+
+	r = psp_ual_get_interface_version(&adev->psp, &adev->ualink.psp_if_ver);
+	if (r) {
+		adev->ualink.psp_if_ver = 0xffffffff;
+		dev_err(adev->dev,
+			"UALink interface version detection failed: %d", r);
+		return r;
+	}
+	dev_info(adev->dev, "Found UALink interface version 0x%x\n",
+		 adev->ualink.psp_if_ver);
+
+	/* Query initial configuration from ASP */
+	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
+			       adev->ualink.info);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed to query initial UALink config: %d\n", r);
+		return r;
+	}
+
+	return 0;
+}
+
+void amdgpu_ualink_fini(struct amdgpu_device *adev)
+{
+	/* empty */
+}
 
 /****************************************************************************
  * UALink info and configuration in sysfs
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index 3882f29ccdae0..318e49ec0de4a 100644
@@ -133,6 +133,7 @@ struct amdgpu_ualink_connection {
 };
 
 struct amdgpu_ualink_mgr {
+	u32 psp_if_ver;
 	struct amdgpu_ualink_info *info;
 	struct amdgpu_ualink_ppod_setup *setup;
 	struct amdgpu_ualink_vpod_config *config;
@@ -166,6 +167,8 @@ struct amdgpu_ualink_mgr {
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
+int amdgpu_ualink_init(struct amdgpu_device *adev);
+void amdgpu_ualink_fini(struct amdgpu_device *adev);
 int amdgpu_ualink_manager_start(struct amdgpu_device *adev);
 void amdgpu_ualink_manager_stop(struct amdgpu_device *adev);
 #endif
drivers/gpu/drm/amd/amdgpu/soc_v1_0.c +5 -0
index d5f945617f488..84cdb09f731e5 100644
@@ -439,6 +439,9 @@ static int soc_v1_0_common_late_init(struct amdgpu_ip_block *ip_block)
 	 */
 	adev->nbio.funcs->enable_doorbell_selfring_aperture(adev, true);
 
+	/* Depends on PSP being initialized */
+	amdgpu_ualink_init(adev);
+
 	return 0;
 }
 
@@ -472,6 +475,8 @@ static int soc_v1_0_common_hw_fini(struct amdgpu_ip_block *ip_block)
 {
 	struct amdgpu_device *adev = ip_block->adev;
 
+	amdgpu_ualink_fini(adev);
+
 	adev->nbio.funcs->enable_doorbell_aperture(adev, false);
 	adev->nbio.funcs->enable_doorbell_selfring_aperture(adev, false);
 
Signature · 1 line
2.55.0
author maintainer
#10 lore raw

PATCH v2 10/95 drm/amdgpu: Implement PSP cmd UAL_SET_PPOD_CONFIG

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

To set the physical pod configuration to PSP.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 45 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  4 +++
 2 files changed, 49 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +45 -0
index 006ec1a241f48..1d78d8589f54a 100644
@@ -831,6 +831,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "UAL_GET_INTERFACE_VER";
 	case GFX_CMD_ID_UAL_GET_CONFIG:
 		return "UAL_GET_CONFIG";
+	case GFX_CMD_ID_UAL_SET_PPOD_CONFIG:
+		return "UAL_SET_PPOD_CONFIG";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1270,6 +1272,49 @@ int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
 	return ret;
 }
 
+int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
+			    const struct amdgpu_ualink_ppod_setup *setup)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_SET_PPOD_CONFIG;
+	cmd->cmd.cmd_set_ppod_config_ual.accelerator_id = setup->ppod.accel_id;
+	memcpy(cmd->cmd.cmd_set_ppod_config_ual.ppod_id, &setup->ppod.id,
+	       sizeof(cmd->cmd.cmd_set_ppod_config_ual.ppod_id));
+	cmd->cmd.cmd_set_ppod_config_ual.ppod_size = setup->ppod.size;
+	cmd->cmd.cmd_set_ppod_config_ual.bandwidth = setup->ppod.bandwidth;
+	cmd->cmd.cmd_set_ppod_config_ual.latency = setup->ppod.latency;
+
+	memcpy(cmd->cmd.cmd_set_ppod_config_ual.local_accelerators,
+	       setup->local_accels,
+	       min(sizeof(cmd->cmd.cmd_set_ppod_config_ual.local_accelerators),
+		   setup->n_local_accels * sizeof(u32)));
+	/* Fill the remainder of the array with invalid accelerator IDs */
+	if (sizeof(cmd->cmd.cmd_set_ppod_config_ual.local_accelerators) >
+	    setup->n_local_accels * sizeof(u32))
+		memset(&cmd->cmd.cmd_set_ppod_config_ual.local_accelerators[setup->n_local_accels],
+		       0xff, sizeof(cmd->cmd.cmd_set_ppod_config_ual.local_accelerators) -
+		       setup->n_local_accels * sizeof(u32));
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && cmd->resp.status)
+		ret = -EINVAL;
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +4 -0
index 66b57c871e1a5..b65ac4f825d75 100644
@@ -707,10 +707,14 @@ int amdgpu_psp_irq_mgr_register(
 	const struct amdgpu_psp_irq_handler *default_handler);
 
 struct amdgpu_ualink_info;
+struct amdgpu_ualink_ppod_setup;
 
 int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
 int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
 		       struct amdgpu_ualink_info *info);
 
+int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
+			    const struct amdgpu_ualink_ppod_setup *setup);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#11 lore raw

PATCH v2 11/95 drm/amdgpu: Set physical pod configuration to PSP

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Set the physical pod configuration to PSP in the commit store callback.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +8 -2
index 41d39f828dfec..6fd1238a50691 100644
@@ -430,13 +430,19 @@ static ssize_t ualink_ppod_setup_commit_store(struct kobject *kobj,
 	struct device *dev = kobj_to_dev(info->kobj.parent);
 	struct drm_device *ddev = dev_get_drvdata(dev);
 	struct amdgpu_device *adev = drm_to_adev(ddev);
+	int r;
 
 	if (!sysfs_streq(buf, "true"))
 		return -EINVAL;
 
-	/* TODO: Send configuration to ASP */
+	r = psp_ual_set_ppod_config(&adev->psp, adev->ualink.psp_if_ver,
+				    setup);
+	if (r)
+		return r;
+	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver, info);
+	if (r)
+		return r;
 
-	info->ppod = setup->ppod;
 	info->accel_state = check_ppod_state(adev, setup);
 
 	/* TODO: If accel_state was ACTIVE, reset all connections */
Signature · 1 line
2.55.0
author maintainer
#12 lore raw

PATCH v2 12/95 drm/amdgpu: Implement PSP cmd UAL_SET_VPOD_CONFIG

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

To set the virtual pod configuration to PSP.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 42 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  4 +++
 2 files changed, 46 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +42 -0
index 1d78d8589f54a..cbd85b27c6f6c 100644
@@ -833,6 +833,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "UAL_GET_CONFIG";
 	case GFX_CMD_ID_UAL_SET_PPOD_CONFIG:
 		return "UAL_SET_PPOD_CONFIG";
+	case GFX_CMD_ID_UAL_SET_VPOD_CONFIG:
+		return "UAL_SET_VPOD_CONFIG";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1315,6 +1317,46 @@ int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
 	return ret;
 }
 
+int psp_ual_set_vpod_config(struct psp_context *psp, uint32_t intf_ver,
+			    const struct amdgpu_ualink_vpod_config *config)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_SET_VPOD_CONFIG;
+	cmd->cmd.cmd_set_vpod_config_ual.addr_mode =
+		(enum psp_gfx_ual_npa_address_mode)config->vpod.addr_mode;
+	cmd->cmd.cmd_set_vpod_config_ual.vpod_id = config->vpod.id;
+	cmd->cmd.cmd_set_vpod_config_ual.vpod_size = config->vpod.size;
+
+	bitmap_to_arr32(cmd->cmd.cmd_set_vpod_config_ual.vpod_active_accelerators,
+			config->vpod.active_accel_bits,
+			min(AMDGPU_UALINK_ACCEL_MAX, PSP_GFX_UAL_MAX_ACC_BIT_MASK*32));
+	/* Clear any remaining accelerator bits */
+	if (PSP_GFX_UAL_MAX_ACC_BIT_MASK > DIV_ROUND_UP(AMDGPU_UALINK_ACCEL_MAX, 32))
+		memset(cmd->cmd.cmd_set_vpod_config_ual.vpod_active_accelerators +
+		       DIV_ROUND_UP(AMDGPU_UALINK_ACCEL_MAX, 32), 0,
+		       (PSP_GFX_UAL_MAX_ACC_BIT_MASK -
+			DIV_ROUND_UP(AMDGPU_UALINK_ACCEL_MAX, 32)) * sizeof(u32));
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && cmd->resp.status)
+		ret = -EINVAL;
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +4 -0
index b65ac4f825d75..d3bc3ba30599c 100644
@@ -708,6 +708,7 @@ int amdgpu_psp_irq_mgr_register(
 
 struct amdgpu_ualink_info;
 struct amdgpu_ualink_ppod_setup;
+struct amdgpu_ualink_vpod_config;
 
 int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
@@ -717,4 +718,7 @@ int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
 int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
 			    const struct amdgpu_ualink_ppod_setup *setup);
 
+int psp_ual_set_vpod_config(struct psp_context *psp, uint32_t intf_ver,
+			    const struct amdgpu_ualink_vpod_config *config);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#13 lore raw

PATCH v2 13/95 drm/amdgpu: Set virtual pod configuration to PSP

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Set the virtual pod configuration to PSP in the commit store callback.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +8 -4
index 6fd1238a50691..657323427f0ee 100644
@@ -705,6 +705,7 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	struct device *dev = kobj_to_dev(info->kobj.parent);
 	struct drm_device *ddev = dev_get_drvdata(dev);
 	struct amdgpu_device *adev = drm_to_adev(ddev);
+	int r;
 
 	if (!sysfs_streq(buf, "true"))
 		return -EINVAL;
@@ -713,10 +714,13 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 		return -EINVAL;
 	}
 
-	/* TODO: instead of just copying the info, call ASP vpod config API
-	 * and query ASP for the updated info
-	 */
-	info->vpod = config->vpod;
+	r = psp_ual_set_vpod_config(&adev->psp, adev->ualink.psp_if_ver,
+				    config);
+	if (r)
+		return r;
+	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver, info);
+	if (r)
+		return r;
 
 	/* The integrity check makes sure each new GPU is consistent with the
 	 * other GPUs already in the vPod. All known local GPUs can become
Signature · 1 line
2.55.0
author maintainer
#14 lore raw

PATCH v2 14/95 drm/amdgpu: Implement PSP cmd UAL_SET_STATION_CONFIG

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

To set the UALink station configuration to PSP.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 35 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  4 +++
 2 files changed, 39 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +35 -0
index cbd85b27c6f6c..47367ff8c3903 100644
@@ -835,6 +835,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "UAL_SET_PPOD_CONFIG";
 	case GFX_CMD_ID_UAL_SET_VPOD_CONFIG:
 		return "UAL_SET_VPOD_CONFIG";
+	case GFX_CMD_ID_UAL_SET_STATION_CONFIG:
+		return "UAL_SET_STATION_CONFIG";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1357,6 +1359,39 @@ int psp_ual_set_vpod_config(struct psp_context *psp, uint32_t intf_ver,
 	return ret;
 }
 
+int psp_ual_set_station_config(struct psp_context *psp, uint32_t intf_ver,
+			       const struct amdgpu_ualink_station_config *stations)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_SET_STATION_CONFIG;
+	cmd->cmd.cmd_set_station_config_ual.num_stations = stations->n_stations;
+	cmd->cmd.cmd_set_station_config_ual.station_flag = stations->flags;
+	memcpy(cmd->cmd.cmd_set_station_config_ual.lane_en_bitmap,
+	       stations->lane_en_bitmap,
+	       min(stations->n_stations,
+		   sizeof(cmd->cmd.cmd_set_station_config_ual.lane_en_bitmap)));
+	/* The remainder is already 0-initialized by acquire_psp_cmd_buf */
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && cmd->resp.status)
+		ret = -EINVAL;
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +4 -0
index d3bc3ba30599c..60ee3dc25113b 100644
@@ -709,6 +709,7 @@ int amdgpu_psp_irq_mgr_register(
 struct amdgpu_ualink_info;
 struct amdgpu_ualink_ppod_setup;
 struct amdgpu_ualink_vpod_config;
+struct amdgpu_ualink_station_config;
 
 int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
@@ -721,4 +722,7 @@ int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
 int psp_ual_set_vpod_config(struct psp_context *psp, uint32_t intf_ver,
 			    const struct amdgpu_ualink_vpod_config *config);
 
+int psp_ual_set_station_config(struct psp_context *psp, uint32_t intf_ver,
+			       const struct amdgpu_ualink_station_config *stations);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#15 lore raw

PATCH v2 15/95 drm/amdgpu: Set UALink station config to PSP

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Set the UALink station configuration to PSP in the commit store callback.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +7 -3
index 657323427f0ee..ce73657a363f5 100644
@@ -856,15 +856,19 @@ static ssize_t ualink_station_config_commit_store(struct kobject *kobj,
 	struct device *dev = kobj_to_dev(info->kobj.parent);
 	struct drm_device *ddev = dev_get_drvdata(dev);
 	struct amdgpu_device *adev = drm_to_adev(ddev);
+	int r;
 
 	/* DF reconfiguration does not interact with accelerator state */
 
 	if (!sysfs_streq(buf, "true"))
 		return -EINVAL;
 
-	/* TODO: Send configuration to ASP */
-	(void)adev;
-	(void)stations;
+	r = psp_ual_set_station_config(&adev->psp, adev->ualink.psp_if_ver, stations);
+	if (r)
+		return r;
+	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver, info);
+	if (r)
+		return r;
 
 	return count;
 }
Signature · 1 line
2.55.0
author maintainer
#16 lore raw

PATCH v2 16/95 drm/amdgpu: Implement PSP cmd UAL_SET_NPA_CONFIG

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

To set NPA translation configuration to PSP.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 30 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  3 +++
 2 files changed, 33 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +30 -0
index 47367ff8c3903..d68b1e5118732 100644
@@ -837,6 +837,8 @@ static const char *psp_gfx_cmd_name(enum psp_gfx_cmd_id cmd_id)
 		return "UAL_SET_VPOD_CONFIG";
 	case GFX_CMD_ID_UAL_SET_STATION_CONFIG:
 		return "UAL_SET_STATION_CONFIG";
+	case GFX_CMD_ID_UAL_SET_NPA_CONFIG:
+		return "UAL_SET_NPA_CONFIG";
 	default:
 		return "UNKNOWN CMD";
 	}
@@ -1392,6 +1394,34 @@ int psp_ual_set_station_config(struct psp_context *psp, uint32_t intf_ver,
 	return ret;
 }
 
+int psp_ual_set_npa_config(struct psp_context *psp, uint32_t intf_ver,
+			   unsigned int vmid, bool enable)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_SET_NPA_CONFIG;
+	cmd->cmd.cmd_set_npa_config_ual.vmid = vmid;
+	cmd->cmd.cmd_set_npa_config_ual.enable_npa_translation = enable;
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && cmd->resp.status)
+		ret = -EINVAL;
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +3 -0
index 60ee3dc25113b..ee37170c1c2b3 100644
@@ -725,4 +725,7 @@ int psp_ual_set_vpod_config(struct psp_context *psp, uint32_t intf_ver,
 int psp_ual_set_station_config(struct psp_context *psp, uint32_t intf_ver,
 			       const struct amdgpu_ualink_station_config *stations);
 
+int psp_ual_set_npa_config(struct psp_context *psp, uint32_t intf_ver,
+			   unsigned int vmid, bool enable);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#17 lore raw

PATCH v2 17/95 drm/amdgpu: Enable/disable NPA address translation using PSP

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Enable or disable NPA address translation using PSP during accelerator state transitions in and out of READY state.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +11 -1
index ce73657a363f5..428296e5cec6a 100644
@@ -663,10 +663,16 @@ static bool check_local_vpod_integrity(struct amdgpu_device *adev)
 
 static void activate_accelerator(struct amdgpu_device *adev)
 {
+	int r;
+
 	if (adev->ualink.info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_READY)
 		return;
 
-	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
+	/* Enable incoming NPA address translation with VMID15 */
+	r = psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
+				   15, true);
+	if (!r)
+		adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
 }
 
 static void deactivate_accelerator(struct amdgpu_device *adev)
@@ -674,6 +680,10 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_READY)
 		return;
 
+	/* Disable incoming NPA address translation with VMID15 */
+	psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
+			       15, false);
+	/* ignore return value */
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
 }
 
Signature · 1 line
2.55.0
author maintainer
#18 lore raw

PATCH v2 18/95 drm/amdgpu: Add helper function to check psp xgmi ta

In reply to Alex Deucher

From: Harish Kasiviswanathan <[email protected]>

Helper function to check if psp xgmi ta is supported

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 11 ++++++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  1 +
 2 files changed, 9 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +8 -3
index d68b1e5118732..6a66a90c8e5b3 100644
@@ -2257,14 +2257,19 @@ int psp_xgmi_terminate(struct psp_context *psp)
 	return ret;
 }
 
+bool psp_is_xgmi_ta_supported(struct psp_context *psp)
+{
+	return psp->ta_fw &&
+	       psp->xgmi_context.context.bin_desc.size_bytes &&
+	       psp->xgmi_context.context.bin_desc.start_addr;
+}
+
 int psp_xgmi_initialize(struct psp_context *psp, bool set_extended_data, bool load_ta)
 {
 	struct ta_xgmi_shared_memory *xgmi_cmd;
 	int ret;
 
-	if (!psp->ta_fw ||
-	    !psp->xgmi_context.context.bin_desc.size_bytes ||
-	    !psp->xgmi_context.context.bin_desc.start_addr)
+	if (!psp_is_xgmi_ta_supported(psp))
 		return -ENOENT;
 
 	if (!load_ta)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +1 -0
index ee37170c1c2b3..bb84f3a3dc54b 100644
@@ -628,6 +628,7 @@ int psp_ta_invoke(struct psp_context *psp,
 			struct ta_context *context);
 
 int psp_xgmi_initialize(struct psp_context *psp, bool set_extended_data, bool load_ta);
+bool psp_is_xgmi_ta_supported(struct psp_context *psp);
 int psp_xgmi_terminate(struct psp_context *psp);
 int psp_xgmi_invoke(struct psp_context *psp, uint32_t ta_cmd_id);
 int psp_xgmi_get_hive_id(struct psp_context *psp, uint64_t *hive_id);
Signature · 1 line
2.55.0
author maintainer
#19 lore raw

PATCH v2 19/95 drm/amdgpu: add handler for nHT error

In reply to Alex Deucher

From: Gangliang Xie <[email protected]>

add handler for nHT error

v2:
add EA ill op check

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h |  1 +
 drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c  | 44 +++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +1 -0
index 6722213db8f01..f5fb2e585e8ae 100644
@@ -478,6 +478,7 @@ struct amdgpu_gfx {
 	struct amdgpu_irq_src		sq_irq;
 	struct amdgpu_irq_src		rlc_gc_fed_irq;
 	struct amdgpu_irq_src		rlc_poison_irq;
+	struct amdgpu_irq_src		pmr_ea_irq;
 	struct sq_work			sq_work;
 
 	/* gfx status */
drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c +44 -0

68 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#20 lore raw

PATCH v2 20/95 drm/amdgpu: Add ual_config_state to ual_get_config

In reply to Alex Deucher

From: Harish Kasiviswanathan <[email protected]>

Add psp_gfx_ual_config_state

---
 drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h +12 -0
index 6bcc04109d4cb..77be21c3eada3 100644
@@ -421,6 +421,17 @@ enum psp_gfx_ual_link_type
     PSP_GFX_UALMAX
 };
 
+enum psp_gfx_ual_config_state
+{
+	UAL_CFG_IDLE        = 0,
+	UAL_CFG_PPOD        = 1,
+	UAL_CFG_VPOD        = 2,
+	UAL_CFG_PAUSE       = 3,
+	UAL_CFG_STATION     = 4,
+	UAL_CFG_COMPLETE    = 5,
+	UAL_CFG_INVALID     = 0xFF
+};
+
 struct psp_gfx_cmd_set_ppod_config_ual_v1
 {
     uint32_t accelerator_id;     // Accelerator ID
@@ -477,6 +488,7 @@ struct psp_gfx_get_config_ual_v1 {
     uint32_t vpod_size;
     uint32_t vpod_active_accelerators[PSP_GFX_UAL_MAX_ACC_BIT_MASK];
     enum psp_gfx_ual_npa_address_mode addr_mode;
+    enum psp_gfx_ual_config_state config_state;
 };
 
 struct psp_gfx_cmd_send_completion_ual_v1
Signature · 1 line
2.55.0
author maintainer
#21 lore raw

PATCH v2 21/95 drm/amdgpu: extend PSP command polling sleep range

In reply to Alex Deucher

From: Le Ma <[email protected]>

Increase the upper bound of the PSP command polling sleep range to allow more time for commands such as UAL_SET_STATION_CONFIG to complete before the next status check.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +1 -1
index 6a66a90c8e5b3..b8def8b7e1426 100644
@@ -895,7 +895,7 @@ psp_cmd_submit_buf(struct psp_context *psp,
 		ras_intr = amdgpu_ras_intr_triggered();
 		if (ras_intr)
 			break;
-		usleep_range(60, 100);
+		usleep_range(60, 150);
 		amdgpu_device_invalidate_hdp(psp->adev, NULL);
 	}
 
Signature · 1 line
2.55.0
author maintainer
#22 lore raw

PATCH v2 22/95 drm/amdgpu: Fix NULL pointer issue during ualink init

In reply to Alex Deucher

From: Jinzhou Su <[email protected]>

PSP may be disabled during driver initialization. So psp->cmd is not allocated, so any psp_*() call would deref NULL. Also skip if sysfs_init failed and ualink.info was never allocated.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +11 -0
index 428296e5cec6a..76aec9eaa59fe 100644
@@ -33,6 +33,14 @@ int amdgpu_ualink_init(struct amdgpu_device *adev)
 {
 	int r;
 
+	/* UALink relies on PSP services. If the PSP IP block is not present
+	 * just skip UALink initialization.
+	 */
+	if (!amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_PSP)) {
+		adev->ualink.psp_if_ver = 0xffffffff;
+		return 0;
+	}
+
 	r = psp_ual_get_interface_version(&adev->psp, &adev->ualink.psp_if_ver);
 	if (r) {
 		adev->ualink.psp_if_ver = 0xffffffff;
@@ -920,6 +928,9 @@ int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev)
 	struct amdgpu_ualink_info *info = NULL;
 	int r;
 
+	if (!amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_PSP))
+		return 0;
+
 	info = kzalloc(sizeof(*info), GFP_KERNEL);
 	if (!info)
 		return -ENOMEM;
Signature · 1 line
2.55.0
author maintainer
#23 lore raw

PATCH v2 23/95 drm/amdgpu: Add a new NPA Address space

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add a new address space for NPA address management. This is needed for sharing buffer objects across GPUs with each running their own OS. The NPA address space size can change across different HW generations so the size is initialized during early init of the driver boot up process.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 11 +++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c    | 17 ++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h    |  4 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  1 +
 drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c     |  5 +++++
 include/drm/ttm/ttm_resource.h             |  2 +-
 include/uapi/drm/amdgpu_drm.h              |  6 ++++++
 8 files changed, 45 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -0
index 5d9d137209b6f..fc6d3fa62d794 100644
@@ -204,6 +204,14 @@ void amdgpu_bo_placement_from_domain(struct amdgpu_bo *abo, u32 domain)
 		c++;
 	}
 
+	if (domain & AMDGPU_GEM_DOMAIN_NPA) {
+		places[c].fpfn = 0;
+		places[c].lpfn = 0;
+		places[c].mem_type = AMDGPU_PL_NPA;
+		places[c].flags = 0;
+		c++;
+	}
+
 	if (!c) {
 		places[c].fpfn = 0;
 		places[c].lpfn = 0;
@@ -1663,6 +1671,9 @@ u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m)
 			case AMDGPU_PL_MMIO_REMAP:
 				placement = "MMIO REMAP";
 				break;
+			case AMDGPU_PL_NPA:
+				placement = "NPA";
+				break;
 			case TTM_PL_SYSTEM:
 			default:
 				placement = "CPU";
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +2 -0
index ff11a09034997..b41767c7913ac 100644
@@ -169,6 +169,8 @@ static inline unsigned amdgpu_mem_type_to_domain(u32 mem_type)
 		return AMDGPU_GEM_DOMAIN_OA;
 	case AMDGPU_PL_DOORBELL:
 		return AMDGPU_GEM_DOMAIN_DOORBELL;
+	case AMDGPU_PL_NPA:
+		return AMDGPU_GEM_DOMAIN_NPA;
 	default:
 		break;
 	}
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +16 -1
index 7a46a26a5fd34..bf575b05a4124 100644
@@ -129,6 +129,7 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,
 	case AMDGPU_PL_OA:
 	case AMDGPU_PL_DOORBELL:
 	case AMDGPU_PL_MMIO_REMAP:
+	case AMDGPU_PL_NPA:
 		placement->num_placement = 0;
 		return;
 
@@ -580,11 +581,13 @@ static int amdgpu_bo_move(struct ttm_buffer_object *bo, bool evict,
 	    old_mem->mem_type == AMDGPU_PL_OA ||
 	    old_mem->mem_type == AMDGPU_PL_DOORBELL ||
 	    old_mem->mem_type == AMDGPU_PL_MMIO_REMAP ||
+	    old_mem->mem_type == AMDGPU_PL_NPA ||
 	    new_mem->mem_type == AMDGPU_PL_GDS ||
 	    new_mem->mem_type == AMDGPU_PL_GWS ||
 	    new_mem->mem_type == AMDGPU_PL_OA ||
 	    new_mem->mem_type == AMDGPU_PL_DOORBELL ||
-	    new_mem->mem_type == AMDGPU_PL_MMIO_REMAP) {
+	    new_mem->mem_type == AMDGPU_PL_MMIO_REMAP ||
+	    new_mem->mem_type == AMDGPU_PL_NPA) {
 		/* Nothing to save here */
 		amdgpu_bo_move_notify(bo, evict, new_mem);
 		ttm_bo_move_null(bo, new_mem);
@@ -2277,6 +2280,16 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
 		dev_err(adev->dev, "Failed initializing oa heap.\n");
 		return r;
 	}
+
+	if (adev->ualink.npa_size) {
+		r = amdgpu_ttm_init_on_chip(adev, AMDGPU_PL_NPA,
+					    adev->ualink.npa_size);
+		if (r) {
+			dev_err(adev->dev, "Failed initializing NPA heap.\n");
+			return r;
+		}
+	}
+
 	if (amdgpu_bo_create_kernel(adev, PAGE_SIZE, PAGE_SIZE,
 				AMDGPU_GEM_DOMAIN_GTT,
 				&adev->mman.sdma_access_bo, NULL,
@@ -2330,6 +2343,7 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev)
 	ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_OA);
 	ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_DOORBELL);
 	ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_MMIO_REMAP);
+	ttm_range_man_fini(&adev->mman.bdev, AMDGPU_PL_NPA);
 	ttm_device_fini(&adev->mman.bdev);
 	adev->mman.initialized = false;
 	dev_info(adev->dev, " ttm finalized\n");
@@ -2712,6 +2726,7 @@ int amdgpu_ttm_evict_resources(struct amdgpu_device *adev, int mem_type)
 	case AMDGPU_PL_GWS:
 	case AMDGPU_PL_GDS:
 	case AMDGPU_PL_OA:
+	case AMDGPU_PL_NPA:
 		man = ttm_manager_type(&adev->mman.bdev, mem_type);
 		break;
 	default:
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +3 -1
index af1e7fcc71756..adb6cfae50527 100644
@@ -37,7 +37,9 @@
 #define AMDGPU_PL_PREEMPT	(TTM_PL_PRIV + 3)
 #define AMDGPU_PL_DOORBELL	(TTM_PL_PRIV + 4)
 #define AMDGPU_PL_MMIO_REMAP	(TTM_PL_PRIV + 5)
-#define __AMDGPU_PL_NUM	(TTM_PL_PRIV + 6)
+#define AMDGPU_PL_NPA		(TTM_PL_PRIV + 6)
+#define __AMDGPU_PL_NUM		(TTM_PL_PRIV + 7)
+
 
 #define AMDGPU_GTT_MAX_TRANSFER_SIZE	(1ULL << 22)
 
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +1 -0
index 318e49ec0de4a..4b3e1f3af3333 100644
@@ -133,6 +133,7 @@ struct amdgpu_ualink_connection {
 };
 
 struct amdgpu_ualink_mgr {
+	u64 npa_size;
 	u32 psp_if_ver;
 	struct amdgpu_ualink_info *info;
 	struct amdgpu_ualink_ppod_setup *setup;
drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c +5 -0
index 0c52eb5e8e288..73fc18ec94a20 100644
@@ -3197,6 +3197,11 @@ static int gfx_v12_1_early_init(struct amdgpu_ip_block *ip_block)
 
 	amdgpu_init_rlc_reg_funcs(adev);
 
+	/* Set NPA address size here as its needed in amdgpu_ttm_init().
+	 * NPA address space is 52 bits or 2^40 pages long.
+	 */
+	adev->ualink.npa_size = 1ULL << 40;
+
 	return gfx_v12_1_init_microcode(adev);
 }
 
include/drm/ttm/ttm_resource.h +1 -1
index a5d386583fb6e..f5d1ad7a4efc7 100644
@@ -35,7 +35,7 @@
 #include <drm/ttm/ttm_kmap_iter.h>
 
 #define TTM_MAX_BO_PRIORITY	4U
-#define TTM_NUM_MEM_TYPES 9
+#define TTM_NUM_MEM_TYPES 10
 
 struct dentry;
 struct dmem_cgroup_device;
include/uapi/drm/amdgpu_drm.h +6 -0
index 0a113529bdcbc..d6d5402a1e789 100644
@@ -115,6 +115,12 @@ extern "C" {
 #define AMDGPU_GEM_DOMAIN_GWS		0x10
 #define AMDGPU_GEM_DOMAIN_OA		0x20
 #define AMDGPU_GEM_DOMAIN_DOORBELL	0x40
+#define AMDGPU_GEM_DOMAIN_MMIO_REMAP	0x80
+#define AMDGPU_GEM_DOMAIN_NPA		0x100
+
+/* User-mode is not allowed to allocate NPA space. As a result,
+ * we don't add AMDGPU_GEM_DOMAIN_NPA in the DOMAIN_MASK below.
+ */
 #define AMDGPU_GEM_DOMAIN_MASK		(AMDGPU_GEM_DOMAIN_CPU | \
 					 AMDGPU_GEM_DOMAIN_GTT | \
 					 AMDGPU_GEM_DOMAIN_VRAM | \
Signature · 1 line
2.55.0
author maintainer
#24 lore raw

PATCH v2 24/95 drm/amdgpu: Add address allocator for NPA addresses

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add a address allocator, which uses DRM MM, for allocating and deallocating NPA addresses for BOs exported/imported in a rack-scale setup.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 86 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h | 10 +++
 2 files changed, 96 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +86 -0
index 76aec9eaa59fe..4582299d96c4f 100644
@@ -23,6 +23,7 @@
  */
 
 #include <linux/xarray.h>
+#include <drm/drm_mm.h>
 #include "amdgpu.h"
 #include "amdgpu_ualink.h"
 #include "amdgpu_xgmi.h"
@@ -1070,6 +1071,87 @@ void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev)
 	}
 }
 
+static int amdgpu_ualink_npa_alloc_va(struct amdgpu_device *adev,
+			       struct drm_mm_node *mm_node,
+			       u64 va, u64 range_start,
+			       u64 range_end, int size)
+{
+	struct amdgpu_ualink_npa_mm *npa_mm = &adev->ualink.npa_mm;
+	u64 alignment = 0;
+	int ret;
+
+	mutex_lock(&npa_mm->mm_lock);
+	if (va) {
+		mm_node->start = va;
+		mm_node->size = size;
+		ret = drm_mm_reserve_node(&npa_mm->mm, mm_node);
+	} else {
+		if (!range_start && !range_end) {
+			range_start = npa_mm->va_start;
+			range_end = npa_mm->va_start + npa_mm->va_size;
+		}
+
+		/* If size is greater than 2MB (or 512 pages),
+		 * align it to 2MB (or 512 pages) granularity.
+		 */
+		if (size >= 0x200)
+			alignment = 0x200;
+
+		ret = drm_mm_insert_node_in_range(&npa_mm->mm, mm_node, size,
+						  alignment, 0, range_start,
+						  range_end, 0);
+	}
+	mutex_unlock(&npa_mm->mm_lock);
+
+	if (ret)
+		dev_err(adev->dev, "Failed to allocate NPA address\n");
+
+	return ret;
+}
+
+static void amdgpu_ualink_npa_free_va(struct amdgpu_device *adev,
+			       struct drm_mm_node *mm_node)
+{
+	struct amdgpu_ualink_npa_mm *npa_mm = &adev->ualink.npa_mm;
+
+	mutex_lock(&npa_mm->mm_lock);
+	drm_mm_remove_node(mm_node);
+	mutex_unlock(&npa_mm->mm_lock);
+}
+
+static void amdgpu_ualink_npa_mm_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_npa_mm *npa_mm = &adev->ualink.npa_mm;
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	u64 va_size;
+
+	mutex_init(&npa_mm->mm_lock);
+
+	if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT) {
+		/* 51 bit address space in Source-Identification mode.
+		 * 39 bits in page granularity.
+		 */
+		npa_mm->va_start = 0;
+		va_size = GENMASK_ULL(38, 0);
+	} else {
+		/* 41 bit address space in Source-Alias mode.
+		 * 29 bits in page granularity.
+		 */
+		/* First 8MB is reserved for Metadata NPAs */
+		npa_mm->va_start = 0x800;
+		va_size = GENMASK_ULL(28, 0);
+	}
+
+	npa_mm->va_size = va_size - npa_mm->va_start;
+	drm_mm_init(&npa_mm->mm, npa_mm->va_start, npa_mm->va_size);
+}
+
+static void amdgpu_ualink_npa_mm_fini(struct amdgpu_device *adev)
+{
+	mutex_destroy(&adev->ualink.npa_mm.mm_lock);
+	drm_mm_takedown(&adev->ualink.npa_mm.mm);
+}
+
 int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 {
 	int i, r;
@@ -1092,6 +1174,8 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 		INIT_LIST_HEAD(&adev->ualink.imp_handles_list[i]);
 	}
 
+	amdgpu_ualink_npa_mm_init(adev);
+
 	return 0;
 }
 
@@ -1099,6 +1183,8 @@ void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 {
 	int i;
 
+	amdgpu_ualink_npa_mm_fini(adev);
+
 	xa_destroy(&adev->ualink.exp_xa);
 	xa_destroy(&adev->ualink.imp_xa);
 	xa_destroy(&adev->ualink.handle_invalid_xa);
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +10 -0
index 4b3e1f3af3333..a056d89d163ba 100644
@@ -125,6 +125,13 @@ struct amdgpu_ualink_station_config {
 };
 #define to_ualink_station_config(ko) container_of(ko, struct amdgpu_ualink_station_config, kobj)
 
+struct amdgpu_ualink_npa_mm {
+	struct drm_mm			mm;
+	u64				va_start;
+	u64				va_size;
+	struct mutex			mm_lock;
+};
+
 struct amdgpu_ualink_connection {
 	struct completion hello_done;
 	struct mutex lock;
@@ -164,6 +171,9 @@ struct amdgpu_ualink_mgr {
 
 	/* WQ to manage revocation of exported memory. */
 	struct workqueue_struct *npa_wq;
+
+	/* Memory-manager for managing NPA address space. */
+	struct amdgpu_ualink_npa_mm npa_mm;
 };
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#25 lore raw

PATCH v2 25/95 drm/amdgpu: Initialize VM for NPA addr management

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Initialize a NPA VM for managing NPA addresses on the exporter GPU.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 18 +++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  3 +++
 2 files changed, 20 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +17 -1
index 4582299d96c4f..9d550c21f69aa 100644
@@ -1156,10 +1156,20 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 {
 	int i, r;
 
+	r = amdgpu_vm_init(adev, &adev->ualink.npa_vm, 0);
+	if (r)
+		goto out;
+
+	/* For using CPU for page table updates. */
+	r = amdgpu_vm_make_compute(adev, &adev->ualink.npa_vm);
+	if (r)
+		goto uninit_vm;
+
 	adev->ualink.npa_wq = alloc_workqueue("NPA WQ", WQ_UNBOUND, 0);
 	if (unlikely(!adev->ualink.npa_wq)) {
 		dev_err(adev->dev, "Failed to allocate NPA WQ\n");
-		return -ENOMEM;
+		r = -ENOMEM;
+		goto uninit_vm;
 	}
 
 	xa_init_flags(&adev->ualink.exp_xa, XA_FLAGS_LOCK_BH);
@@ -1177,6 +1187,11 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 	amdgpu_ualink_npa_mm_init(adev);
 
 	return 0;
+
+uninit_vm:
+	amdgpu_vm_fini(adev, &adev->ualink.npa_vm);
+out:
+	return r;
 }
 
 void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
@@ -1193,4 +1208,5 @@ void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 		mutex_destroy(&adev->ualink.conn_state[i].lock);
 
 	destroy_workqueue(adev->ualink.npa_wq);
+	amdgpu_vm_fini(adev, &adev->ualink.npa_vm);
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index a056d89d163ba..c32b4fd3c3220 100644
@@ -174,6 +174,9 @@ struct amdgpu_ualink_mgr {
 
 	/* Memory-manager for managing NPA address space. */
 	struct amdgpu_ualink_npa_mm npa_mm;
+
+	/* NPA-VM used on the exporter.*/
+	struct amdgpu_vm npa_vm;
 };
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#26 lore raw

PATCH v2 26/95 drm/amdgpu: Rework VMID reservation logic

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Rework the VMID reservation logic by passing in the mask of VMIDs (for both GFXHUB and MMHUB) available for it to use. The VMID masks for both GFXHUB and MMHUB are initialized in the respective ASIC specific files.
This also helps in reserving per-ASIC VMIDs. This is in preparation for reserving a VMID for NPA added in the next patch.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c | 33 ++++++++++++++-----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h | 27 +++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c |  2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c  |  4 +++
 drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c  |  5 ++++
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c  | 14 +++++++----
 drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c   |  6 +++++
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c   |  6 +++++
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c   |  6 +++++
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   |  6 +++++
 17 files changed, 95 insertions(+), 28 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c +19 -14
index 8a2d64f0ebc3f..8c32c69437645 100644
@@ -625,12 +625,27 @@ void amdgpu_vmid_reset_all(struct amdgpu_device *adev)
 	for (i = 0; i < AMDGPU_MAX_VMHUBS; ++i) {
 		struct amdgpu_vmid_mgr *id_mgr =
 			&adev->vm_manager.id_mgr[i];
-
-		for (j = 1; j < id_mgr->num_ids; ++j)
+		for_each_set_bit(j, id_mgr->vmid_mask, AMDGPU_NUM_VMID)
 			amdgpu_vmid_reset(adev, i, j);
 	}
 }
 
+void amdgpu_vmid_mgr_set_vmid_mask(struct amdgpu_device *adev,
+				   unsigned long vmid_mask, bool for_mmhub)
+{
+	unsigned int i;
+
+	BUILD_BUG_ON(AMDGPU_NUM_VMID > BITS_PER_LONG);
+
+	for (i = 0; i < AMDGPU_MAX_VMHUBS; i++) {
+		bool is_mmhub = AMDGPU_IS_MMHUB0(i) || AMDGPU_IS_MMHUB1(i);
+
+		if (is_mmhub == for_mmhub)
+			bitmap_copy(adev->vm_manager.id_mgr[i].vmid_mask,
+				    &vmid_mask, AMDGPU_NUM_VMID);
+	}
+}
+
 /**
  * amdgpu_vmid_mgr_init - init the VMID manager
  *
@@ -649,19 +664,8 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev)
 		mutex_init(&id_mgr->lock);
 		INIT_LIST_HEAD(&id_mgr->ids_lru);
 
-		/* for GC <10, SDMA uses MMHUB so use first_kfd_vmid for both GC and MM */
-		if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(10, 0, 0))
-			/* manage only VMIDs not used by KFD */
-			id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
-		else if (AMDGPU_IS_MMHUB0(i) ||
-			 AMDGPU_IS_MMHUB1(i))
-			id_mgr->num_ids = 16;
-		else
-			/* manage only VMIDs not used by KFD */
-			id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
-
 		/* skip over VMID 0, since it is the system VM */
-		for (j = 1; j < id_mgr->num_ids; ++j) {
+		for_each_set_bit(j, id_mgr->vmid_mask, AMDGPU_NUM_VMID) {
 			amdgpu_vmid_reset(adev, i, j);
 			amdgpu_sync_create(&id_mgr->ids[j].active);
 			list_add_tail(&id_mgr->ids[j].list, &id_mgr->ids_lru);
@@ -685,6 +689,7 @@ void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev)
 			&adev->vm_manager.id_mgr[i];
 
 		mutex_destroy(&id_mgr->lock);
+
 		for (j = 0; j < AMDGPU_NUM_VMID; ++j) {
 			struct amdgpu_vmid *id = &id_mgr->ids[j];
 
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h +26 -1
index 4b55d0d9703a3..46b2c21601263 100644
@@ -26,6 +26,8 @@
 #include <linux/types.h>
 #include <linux/mutex.h>
 #include <linux/list.h>
+#include <linux/bitops.h>
+#include <linux/find.h>
 #include <linux/dma-fence.h>
 
 #include "amdgpu_sync.h"
@@ -33,6 +35,27 @@
 /* maximum number of VMIDs */
 #define AMDGPU_NUM_VMID	16
 
+/**
+ * for_each_vmid_and_zero - iterate over VMID 0 plus every VMID set in a hub's
+ *                          vmid_mask
+ * @vmid:	loop cursor (int / unsigned int)
+ * @adev:	struct amdgpu_device *
+ * @hub:	vmhub index into adev->vm_manager.id_mgr[]
+ *
+ * Equivalent to OR-ing BIT(0) into id_mgr[hub].vmid_mask and walking the
+ * result with for_each_set_bit(). Used by the gfx_v*_constants_init() paths
+ * which must always touch VMID 0 (system) in addition to the regular VMIDs
+ * owned by the hub.
+ *
+ * Relies on AMDGPU_NUM_VMID (16) fitting in a single unsigned long, i.e.
+ * vmid_mask[] being a 1-element bitmap.
+ */
+#define for_each_vmid_and_zero(vmid, adev, hub)				\
+	for (unsigned long __vmid_mask =				\
+		(adev)->vm_manager.id_mgr[(hub)].vmid_mask[0] | BIT(0);	\
+	     __vmid_mask; __vmid_mask = 0)				\
+		for_each_set_bit((vmid), &__vmid_mask, AMDGPU_NUM_VMID)
+
 struct amdgpu_device;
 struct amdgpu_fpriv;
 struct amdgpu_vm;
@@ -65,10 +88,10 @@ struct amdgpu_vmid {
 
 struct amdgpu_vmid_mgr {
 	struct mutex		lock;
-	unsigned		num_ids;
 	struct list_head	ids_lru;
 	struct amdgpu_vmid	ids[AMDGPU_NUM_VMID];
 	bool			reserved_vmid;
+	DECLARE_BITMAP(vmid_mask, AMDGPU_NUM_VMID);
 };
 
 int amdgpu_pasid_alloc(unsigned int bits, struct amdgpu_fpriv *fpriv);
@@ -95,5 +118,7 @@ void amdgpu_vmid_reset_all(struct amdgpu_device *adev);
 
 void amdgpu_vmid_mgr_init(struct amdgpu_device *adev);
 void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev);
+void amdgpu_vmid_mgr_set_vmid_mask(struct amdgpu_device *adev,
+				   unsigned long vmid_mask, bool for_mmhub);
 
 #endif
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +1 -1
index 5033f85d31022..045b9c7d2a835 100644
@@ -5368,7 +5368,7 @@ static void gfx_v10_0_constants_init(struct amdgpu_device *adev)
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		nv_grbm_select(adev, 0, 0, 0, i);
 		/* CP and shaders */
 		WREG32_SOC15(GC, 0, mmSH_MEM_CONFIG, DEFAULT_SH_MEM_CONFIG);
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +1 -1
index d704ce328e6ec..e5dbd71cce763 100644
@@ -2217,7 +2217,7 @@ static void gfx_v11_0_constants_init(struct amdgpu_device *adev)
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		soc21_grbm_select(adev, 0, 0, 0, i);
 		/* CP and shaders */
 		WREG32_SOC15(GC, 0, regSH_MEM_CONFIG, DEFAULT_SH_MEM_CONFIG);
drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c +1 -1
index df8d36be613ac..6bb1c42efc5e8 100644
@@ -1856,7 +1856,7 @@ static void gfx_v12_0_constants_init(struct amdgpu_device *adev)
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		soc24_grbm_select(adev, 0, 0, 0, i);
 		/* CP and shaders */
 		WREG32_SOC15(GC, 0, regSH_MEM_CONFIG, DEFAULT_SH_MEM_CONFIG);
drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c +1 -1
index 73fc18ec94a20..d60cd64d444eb 100644
@@ -1732,7 +1732,7 @@ static void gfx_v12_1_xcc_constants_init(struct amdgpu_device *adev,
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		soc_v1_0_grbm_select(adev, 0, 0, 0, i, GET_INST(GC, xcc_id));
 		/* CP and shaders */
 		WREG32_SOC15(GC, GET_INST(GC, xcc_id),
drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +1 -1
index 755371b665746..549d1239e326b 100644
@@ -1938,7 +1938,7 @@ static void gfx_v7_0_constants_init(struct amdgpu_device *adev)
 	WREG32(mmSH_STATIC_MEM_CONFIG, sh_static_mem_cfg);
 
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[0].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, 0) {
 		if (i == 0)
 			sh_mem_base = 0;
 		else
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +1 -1
index 63288382bcf40..950874ef8d045 100644
@@ -3765,7 +3765,7 @@ static void gfx_v8_0_constants_init(struct amdgpu_device *adev)
 	WREG32(mmSH_STATIC_MEM_CONFIG, sh_static_mem_cfg);
 
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[0].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, 0) {
 		vi_srbm_select(adev, 0, 0, 0, i);
 		/* CP and shaders */
 		if (i == 0) {
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +1 -1
index 0dae8ff9c762a..3ba2584e48c6a 100644
@@ -2686,7 +2686,7 @@ static void gfx_v9_0_constants_init(struct amdgpu_device *adev)
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		soc15_grbm_select(adev, 0, 0, 0, i, 0);
 		/* CP and shaders */
 		if (i == 0) {
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +1 -1
index e1603563c49de..ce48da12edb29 100644
@@ -1241,7 +1241,7 @@ static void gfx_v9_4_3_xcc_constants_init(struct amdgpu_device *adev,
 	/* XXX SH_MEM regs */
 	/* where to put LDS, scratch, GPUVM in FSA64 space */
 	mutex_lock(&adev->srbm_mutex);
-	for (i = 0; i < adev->vm_manager.id_mgr[AMDGPU_GFXHUB(0)].num_ids; i++) {
+	for_each_vmid_and_zero(i, adev, AMDGPU_GFXHUB(0)) {
 		soc15_grbm_select(adev, 0, 0, 0, i, GET_INST(GC, xcc_id));
 		/* CP and shaders */
 		if (i == 0) {
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +4 -0
index 75a552782c979..9a8a7921ce982 100644
@@ -869,6 +869,10 @@ static int gmc_v10_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * amdkfd will use VMIDs 8-15
 	 */
 	adev->vm_manager.first_kfd_vmid = 8;
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev, GENMASK(AMDGPU_NUM_VMID - 1, 1), true);
 
 	amdgpu_vm_manager_init(adev);
 
drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +5 -0
index 7fb97be16b7c1..bee269a8877a5 100644
@@ -855,6 +855,11 @@ static int gmc_v11_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 */
 	adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
 
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev, GENMASK(AMDGPU_NUM_VMID - 1, 1), true);
+
 	amdgpu_vm_manager_init(adev);
 
 	r = amdgpu_gmc_ras_sw_init(adev);
drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c +9 -5
index 18511e7995ef9..3ae3ba42c6b0f 100644
@@ -957,13 +957,17 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * number of VMs
 	 * VMID 0 is reserved for System
 	 * amdgpu graphics/compute will use VMIDs 1-7
-	 * amdkfd will use VMIDs 8-15
+	 * amdkfd will use VMIDs 8-15.
+	 * On GFX 12.1, amdkfd will use VMIDs 3-15.
 	 */
 	adev->vm_manager.first_kfd_vmid =
-		 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 1, 0) ?
-		3 : 8;
-	adev->vm_manager.first_kfd_vmid =
-		adev->gfx.disable_kq ? 1 : (adev->vm_manager.first_kfd_vmid);
+		adev->gfx.disable_kq ? 1 :
+		(amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 1, 0) ?
+		3 : 8);
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev, GENMASK(AMDGPU_NUM_VMID - 1, 1), true);
 
 	amdgpu_vm_manager_init(adev);
 
drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c +6 -0
index ca7afb9b7d0ae..cdd11587b12ae 100644
@@ -864,6 +864,12 @@ static int gmc_v6_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * amdkfd will use VMIDs 8-15
 	 */
 	adev->vm_manager.first_kfd_vmid = 8;
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      true);
 	amdgpu_vm_manager_init(adev);
 
 	/* base offset of vram pages */
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +6 -0
index 1c1eccfe383fd..c9cc55e18893b 100644
@@ -1045,6 +1045,12 @@ static int gmc_v7_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * amdkfd will use VMIDs 8-15
 	 */
 	adev->vm_manager.first_kfd_vmid = 8;
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      true);
 	amdgpu_vm_manager_init(adev);
 
 	/* base offset of vram pages */
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +6 -0
index bf089595e5dac..8d8a0775878ff 100644
@@ -1122,6 +1122,12 @@ static int gmc_v8_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * amdkfd will use VMIDs 8-15
 	 */
 	adev->vm_manager.first_kfd_vmid = 8;
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      true);
 	amdgpu_vm_manager_init(adev);
 
 	/* base offset of vram pages */
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +6 -0
index b46b87291c512..633772fa71fda 100644
@@ -2038,6 +2038,12 @@ static int gmc_v9_0_sw_init(struct amdgpu_ip_block *ip_block)
 			3 :
 			8;
 
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      false);
+	amdgpu_vmid_mgr_set_vmid_mask(adev,
+				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
+				      true);
 	amdgpu_vm_manager_init(adev);
 
 	gmc_v9_0_save_registers(adev);
Signature · 1 line
2.55.0
author maintainer
#27 lore raw

PATCH v2 27/95 drm/amdgpu: Reserve VMID for NPA VM

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Reserve MMHUB VMID 15 to be used for NPA VM.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h |  1 +
 drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c | 22 ++++++++++++++++------
 2 files changed, 17 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +1 -0
index 3533d677b4a20..7a0ee19a12b18 100644
@@ -411,6 +411,7 @@ struct amdgpu_vm_manager {
 
 	/* Global registration of recent page fault information */
 	struct amdgpu_vm_fault_info	fault_info;
+	unsigned int				npa_vmid;
 };
 
 struct amdgpu_bo_va_mapping;
drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c +16 -6
index 3ae3ba42c6b0f..0cc73982ed6f9 100644
@@ -814,6 +814,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
 	int r, vram_width = 0, vram_type = 0, vram_vendor = 0, dma_addr_bits;
 	struct amdgpu_device *adev = ip_block->adev;
 	uint64_t pte_addr_mask = 0;
+	u32 mmhub_vmid_mask;
 	int i;
 
 	adev->mmhub.funcs->init(adev);
@@ -958,16 +959,25 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block)
 	 * VMID 0 is reserved for System
 	 * amdgpu graphics/compute will use VMIDs 1-7
 	 * amdkfd will use VMIDs 8-15.
-	 * On GFX 12.1, amdkfd will use VMIDs 3-15.
+	 * On GFX 12.1, amdkfd will use VMIDs 3-15, with VMID 15 reserved
+	 * for NPA and therefore excluded from the MMHUB VMID pool.
 	 */
-	adev->vm_manager.first_kfd_vmid =
-		adev->gfx.disable_kq ? 1 :
-		(amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 1, 0) ?
-		3 : 8);
+	mmhub_vmid_mask = GENMASK(AMDGPU_NUM_VMID - 1, 1);
+	if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 1, 0)) {
+		adev->vm_manager.first_kfd_vmid = 3;
+		adev->vm_manager.npa_vmid = 15;
+		mmhub_vmid_mask &= ~BIT(adev->vm_manager.npa_vmid);
+	} else {
+		adev->vm_manager.first_kfd_vmid = 8;
+	}
+
+	if (adev->gfx.disable_kq)
+		adev->vm_manager.first_kfd_vmid = 1;
+
+	amdgpu_vmid_mgr_set_vmid_mask(adev, mmhub_vmid_mask, true);
 	amdgpu_vmid_mgr_set_vmid_mask(adev,
 				      GENMASK(adev->vm_manager.first_kfd_vmid - 1, 1),
 				      false);
-	amdgpu_vmid_mgr_set_vmid_mask(adev, GENMASK(AMDGPU_NUM_VMID - 1, 1), true);
 
 	amdgpu_vm_manager_init(adev);
 
Signature · 1 line
2.55.0
author maintainer
#28 lore raw

PATCH v2 28/95 drm/amdgpu: Use reserved VMID for NPA VM

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Map NPA VM to use the reserved VMID.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +9 -4
index 9d550c21f69aa..89ba7cef02e81 100644
@@ -677,9 +677,9 @@ static void activate_accelerator(struct amdgpu_device *adev)
 	if (adev->ualink.info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_READY)
 		return;
 
-	/* Enable incoming NPA address translation with VMID15 */
+	/* Enable incoming NPA address translation with NPA VMID */
 	r = psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
-				   15, true);
+				   adev->vm_manager.npa_vmid, true);
 	if (!r)
 		adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
 }
@@ -689,9 +689,9 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_READY)
 		return;
 
-	/* Disable incoming NPA address translation with VMID15 */
+	/* Disable incoming NPA address translation with NPA VMID */
 	psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
-			       15, false);
+			       adev->vm_manager.npa_vmid, false);
 	/* ignore return value */
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
 }
@@ -1172,6 +1172,10 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 		goto uninit_vm;
 	}
 
+	/* Map this VM to NPA VMID */
+	adev->mmhub.funcs->setup_vm_pt_regs(adev, adev->vm_manager.npa_vmid,
+			amdgpu_gmc_pd_addr(adev->ualink.npa_vm.root.bo));
+
 	xa_init_flags(&adev->ualink.exp_xa, XA_FLAGS_LOCK_BH);
 	xa_init_flags(&adev->ualink.imp_xa, XA_FLAGS_LOCK_BH);
 	xa_init_flags(&adev->ualink.handle_invalid_xa, XA_FLAGS_LOCK_BH);
@@ -1198,6 +1202,7 @@ void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 {
 	int i;
 
+	adev->mmhub.funcs->setup_vm_pt_regs(adev, adev->vm_manager.npa_vmid, 0);
 	amdgpu_ualink_npa_mm_fini(adev);
 
 	xa_destroy(&adev->ualink.exp_xa);
Signature · 1 line
2.55.0
author maintainer
#29 lore raw

PATCH v2 29/95 drm/amdgpu: Enable UALink Manager when pod becomes active

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Enable the UALink manager when the pod state transitions to active. This would enable exchanging of memory across GPUs in a rack-scale setup.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +14 -2
index 89ba7cef02e81..6329fc8e217e7 100644
@@ -680,8 +680,18 @@ static void activate_accelerator(struct amdgpu_device *adev)
 	/* Enable incoming NPA address translation with NPA VMID */
 	r = psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
 				   adev->vm_manager.npa_vmid, true);
-	if (!r)
-		adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
+	if (r) {
+		dev_err(adev->dev, "Failed to set NPA config\n");
+		return;
+	}
+
+	r = amdgpu_ualink_manager_start(adev);
+	if (r) {
+		dev_err(adev->dev, "Failed to start UALink manager\n");
+		return;
+	}
+
+	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
 }
 
 static void deactivate_accelerator(struct amdgpu_device *adev)
@@ -694,6 +704,8 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 			       adev->vm_manager.npa_vmid, false);
 	/* ignore return value */
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
+
+	amdgpu_ualink_manager_stop(adev);
 }
 
 static void activate_local_vpod(struct amdgpu_device *adev)
Signature · 1 line
2.55.0
author maintainer
#30 lore raw

PATCH v2 30/95 drm/amdgpu: Fix UALink vPod double-activation

In reply to Alex Deucher

From: Philip Yang <[email protected]>

ualink_ppod_setup_commit_store() updated accel_state via check_ppod_state() but never called deactivate_local_vpod() when the accelerator was already in READY state. A subsequent vpod config commit would then call activate_local_vpod() again, resulting in activate_accelerator() calling amdgpu_ualink_manager_start() a second time on top of already-initialized NPA VM, causing a NULL pointer dereference in rb_insert_color when drm_sched_entity nodes already in the SDMA scheduler FIFO rb-tree were double-initialized.

Fix: call deactivate_accelerator() in ualink_ppod_setup_commit_store() guarded by accel_state >= READY, under mgpu_info.mutex to serialize with activate_local_vpod() in ualink_vpod_config_commit_store().

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +13 -0
index 6329fc8e217e7..c0478087b95d4 100644
@@ -398,6 +398,8 @@ static const struct kobj_type ualink_info_ktype = {
 };
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
+static void deactivate_accelerator(struct amdgpu_device *adev);
+
 UALINK_VALUE_SHOW(ppod_setup, accel_id,  ppod.accel_id,  "%u");
 UALINK_VALUE_SHOW(ppod_setup, bandwidth, ppod.bandwidth, "%u");
 UALINK_VALUE_SHOW(ppod_setup, latency,   ppod.latency,   "%u");
@@ -452,7 +454,18 @@ static ssize_t ualink_ppod_setup_commit_store(struct kobject *kobj,
 	if (r)
 		return r;
 
+	/*
+	 * Hold mgpu_info.mutex to serialize with activate_local_vpod() in
+	 * vpod_config_commit_store which also holds this lock. Without it,
+	 * deactivate_accelerator() can race with activate_accelerator(),
+	 * causing concurrent vm_fini / vm_init on the same NPA VM.
+	 */
+	mutex_lock(&mgpu_info.mutex);
+	if (info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_READY)
+		deactivate_accelerator(adev);
+
 	info->accel_state = check_ppod_state(adev, setup);
+	mutex_unlock(&mgpu_info.mutex);
 
 	/* TODO: If accel_state was ACTIVE, reset all connections */
 
Signature · 1 line
2.55.0
author maintainer
#31 lore raw

PATCH v2 31/95 drm/amdgpu: Add UALink remote state structures and API declarations

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Extend the UALink manager with data structures for cross-GPU interrupt and TLB shootdown: per-ring state, per-peer channel pairs, and the top-level remote container.

Declare the public init, teardown, and reset APIs.

Define ring buffer capacity, firmware command and status codes, SMN mailbox register addresses, and the metadata and writeback buffer layouts shared between driver and MPNHT firmware.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 140 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  12 ++
 2 files changed, 152 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +140 -0
index c0478087b95d4..d08f45d07fae9 100644
@@ -1240,3 +1240,143 @@ void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 	destroy_workqueue(adev->ualink.npa_wq);
 	amdgpu_vm_fini(adev, &adev->ualink.npa_vm);
 }
+
+/*
+ * UALink remote interrupt and shootdown
+ */
+
+/* UALINK ring buffer size, same for both remote shootdown and interrupt ring */
+#define AMDGPU_UALINK_RB_SIZE	4096
+
+#define AMDGPU_UALINK_METADATA_HEADER	0x4E485446
+
+/* UALINK F/W commands */
+#define AMDGPU_UALINK_FW_CMD_LOAD_METADATA	0x1
+#define AMDGPU_UALINK_FW_CMD_HALT_OPERATION	0x2
+
+/* UALINK F/W status */
+#define AMDGPU_UALINK_FW_STATUS_PREINIT	0xA0
+#define AMDGPU_UALINK_FW_STATUS_READY	0xA1
+#define AMDGPU_UALINK_FW_STATUS_HALT	0xA2
+#define AMDGPU_UALINK_FW_STATUS_ERROR	0xA3
+#define AMDGPU_UALINK_FW_STATUS_FATAL	0xF0
+
+/* UALINK mailbox registers via SMN, copy of MP1 */
+/* send command to nht f/w */
+#define mmMPNHT_SMN_C2PMSG_22_ALT_2	0xAE10958
+/* additional data */
+#define mmMPNHT_SMN_C2PMSG_23_ALT_2	0xAE1095C
+/* metadata address low */
+#define mmMPNHT_SMN_C2PMSG_24_ALT_2	0xAE10960
+/* metadata address high */
+#define mmMPNHT_SMN_C2PMSG_25_ALT_2	0xAE10964
+/* f/w status */
+#define mmMPNHT_SMN_C2PMSG_26_ALT_2	0xAE10968
+
+struct amdgpu_ualink_metadata {
+	u32 header;
+
+	/*
+	 * "ring entries" as unit. So the ring-size-in-bytes could be calculated
+	 * as entry-size * 2^RBsize. This would make the minimum size a single
+	 * entry and the maximum size 32786 entries.
+	 *
+	 * u32 rb_size:4;
+	 * u32 reserved0:4;
+	 * u32 vmid:4;
+	 * u32 reserved1:20;
+	 */
+	u32 rb_size;
+
+	/* ring buffer base address for remote interrupt */
+	u64 ri_rb;
+
+	/* tlb invalidate ring buffer base address for remote shootdown */
+	u64 tlb_inv_rb;
+
+	/* remote interrupt Tail pointer, write pointer address */
+	u64 tailptr_ri;
+
+	/* TLB invalidate ring buffer's Tail pointer, write pointer address */
+	u64 tailptr_tlb_inv;
+};
+
+struct amdgpu_ualink_wb {
+	/* last finished command seq number */
+	u32 data;
+
+	/* command complete error code */
+	u32 status;
+
+	/* ring rptr updated by FW */
+	u64 rptr;
+};
+
+/*
+ * For remote interrupt and shootdown
+ */
+struct amdgpu_ualink_ring {
+	u32 rb_size;
+
+	/* writeback data */
+	u32 seq;
+
+	/* local copy */
+	u64 wptr, rptr;
+
+	/* NPA gart mapping for SDMA */
+	u64 rb_npa_gart;
+	u64 wptr_npa_gart;
+	u64 doorbell_npa_gart;
+
+	/* gart mapping node */
+	struct drm_mm_node mm_node_rb;
+	struct drm_mm_node mm_node_wptr;
+	struct drm_mm_node mm_node_doorbell;
+
+	/* true if fw write back address updated successfully */
+	bool ready;
+};
+
+struct amdgpu_ualink_peer {
+	/* remote interrupt and shootdown uses same SDMA entity */
+	struct mutex lock;
+
+	/* SDMA engine to send remote command via NPA */
+	struct drm_sched_entity entity;
+
+	/* to select different DXS ports, cycles through different values */
+	u32 dxs_port;
+
+	struct amdgpu_ualink_ring interrupt;
+	struct amdgpu_ualink_ring shootdown;
+};
+
+struct amdgpu_ualink_remote {
+	/* ualink metadata passed to MPNHT FW */
+	struct amdgpu_bo		*metadata_bo;
+	u64				metadata_gpu_addr;
+	void				*metadata_cpu_addr;
+
+	/* ualink ring, tlb ring buffer, wptr */
+	struct amdgpu_bo		*ring_bo;
+	u64				rb_gpu_addr;
+	void				*rb_cpu_addr;
+
+	/* ualink rptr, wb data, statuss for address alias mode */
+	struct amdgpu_bo		*rptr_bo;
+	u64				rptr_gpu_addr;
+	void				*rptr_cpu_addr;
+	u64				rptr_npa;
+
+	/* address alias mode alloc npa address for shared wb */
+	struct drm_mm_node		rptr_mm_node;
+
+	/* active accelator id bitmap of the pod */
+	unsigned long			*active_accel_bits;
+	u32				num_accel;
+
+	/* remote GPUs ring buffer, read, write pointer local copy and gart mapping */
+	struct amdgpu_ualink_peer	peer[AMDGPU_UALINK_ACCEL_MAX];
+};
+
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +12 -0
index c32b4fd3c3220..a75aac48604cb 100644
@@ -139,6 +139,8 @@ struct amdgpu_ualink_connection {
 	enum amdgpu_ualink_conn_state state;
 };
 
+struct amdgpu_ualink_remote;
+
 struct amdgpu_ualink_mgr {
 	u64 npa_size;
 	u32 psp_if_ver;
@@ -147,6 +149,12 @@ struct amdgpu_ualink_mgr {
 	struct amdgpu_ualink_vpod_config *config;
 	struct amdgpu_ualink_station_config *stations;
 
+	/* For remote interrupt and shootdown */
+	struct amdgpu_ualink_remote *remote;
+
+	/* handle irq from ualink client of remote GPUs */
+	struct amdgpu_irq_src irq;
+
 	/* Xarray to store info about exported BOs */
 	struct xarray	exp_xa;
 
@@ -179,6 +187,10 @@ struct amdgpu_ualink_mgr {
 	struct amdgpu_vm npa_vm;
 };
 
+int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev);
+int amdgpu_ualink_sw_init(struct amdgpu_device *adev);
+void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
 int amdgpu_ualink_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#32 lore raw

PATCH v2 32/95 drm/amdgpu: Add UALink NPA address layout helpers

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Add accessors for the local accelerator ID, addressing mode, and active peer bitmap, plus a TLB flush helper for the NPA VM. Provide per-ring size and byte offset helpers for both source-identification and source-alias addressing modes.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 101 +++++++++++++++++++++
 1 file changed, 101 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +101 -0
index d08f45d07fae9..ae3ae41c8f360 100644
@@ -1380,3 +1380,104 @@ struct amdgpu_ualink_remote {
 	struct amdgpu_ualink_peer	peer[AMDGPU_UALINK_ACCEL_MAX];
 };
 
+static inline struct amdgpu_ualink_remote *to_remote(struct amdgpu_device *adev)
+{
+	return adev->ualink.remote;
+}
+
+static inline u32 ualink_accel_id(struct amdgpu_device *adev)
+{
+	return adev->ualink.info->ppod.accel_id;
+}
+
+static inline enum amdgpu_ualink_addr_mode ualink_addr_mode(struct amdgpu_device *adev)
+{
+	return adev->ualink.info->vpod.addr_mode;
+}
+
+/*
+ * address mode and NPA address for ring wptr, rptr, wb status, wb data
+ *
+ * source identification address mode
+ *
+ * we need to export different tail pointers for different remote GPU accel_id,
+ * hence we need to allocate a whole page for each remote GPU separately.
+ *
+ * rptr, wb data, status share the whole page with wptr.
+ *
+ *     1 page reserved NPA address for 1 active remote GPU, max 255 pages
+ *     starting address: wptr NPA address + accel_id * 4K
+ *     offset:
+ *         remote interrupt ring wptr 0
+ *         remote shootdown ring wptr 8
+ *         remote interrupt wb status 16
+ *         remote interrupt wb data   20
+ *         remote interrupt ring rptr 24
+ *         remote shootdown wb status 32
+ *         remote shootdown wb data   36
+ *         remote shootdown ring rptr 40
+ *
+ * source aliasing address mode
+ *
+ * wptr for different remote GPU can share same reserved 1 page NPA address page.
+ *
+ * to save reserved NPA address space, rptr, wb data, status for different
+ * remote GPU share same allocated 2 pages NPA address.
+ *
+ *     1 page reserved NPA address for wptr of all active remote GPUs
+ *     offset:
+ *         remote interrupt ring wptr accel_id * 16
+ *         remote shootdown ring wptr accel_id * 16 + 8
+ *
+ *     2 page allocated NPA address for rptr, writeback of all active remote GPUs
+ *     starting address: at accel_id * 32 for each remote GPU
+ *     offset:
+ *         remote interrupt wb status 0
+ *         remote interrupt wb data   4
+ *         remote interrupt ring rptr 8
+ *         remote shootdown wb status 16
+ *         remote shootdown wb data   20
+ *         remote shootdown ring rptr 24
+ */
+static inline u32 ualink_wptr_size(struct amdgpu_device *adev)
+{
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT)
+		return AMDGPU_GPU_PAGE_SIZE;
+	else
+		return 0;	/* wptr uses the ring buffer for itself */
+}
+
+static inline u32 ualink_wb_size(struct amdgpu_device *adev)
+{
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT)
+		return 0;	/* share same page with wptr */
+	else
+		return 2 * sizeof(struct amdgpu_ualink_wb);
+}
+
+static inline u32 ualink_wptr_offset(struct amdgpu_device *adev, u32 accel_id)
+{
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT)
+		return 0;	/* one separate page per GPU */
+	else
+		return accel_id * 2 * sizeof(u64);	/* two pointers per GPU */
+}
+
+static inline u32 ualink_tlb_wptr_offset(struct amdgpu_device *adev, u32 accel_id)
+{
+	return ualink_wptr_offset(adev, accel_id) + sizeof(u64);
+}
+
+static inline u32 ualink_wb_offset(struct amdgpu_device *adev, u32 accel_id)
+{
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT)
+		return 2 * sizeof(u64);	/* after two wptr pointers */
+	else
+		return accel_id * ualink_wb_size(adev);
+}
+
+static inline u32 ualink_tlb_wb_offset(struct amdgpu_device *adev, u32 accel_id)
+{
+	return ualink_wb_offset(adev, accel_id) + sizeof(struct amdgpu_ualink_wb);
+}
+
Signature · 1 line
2.55.0
author maintainer
#33 lore raw

PATCH v2 33/95 drm/amdgpu: Add UALink NPA address computation for ring buffers

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Add a ring buffer type enum and address formula for the reserved NPA space between source and destination GPUs, covering both addressing modes. Add a GART-adjusted variant so the local SDMA engine can address remote GPU buffers.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 97 ++++++++++++++++++++++
 1 file changed, 97 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +97 -0
index ae3ae41c8f360..c4edefcef2141 100644
@@ -1273,6 +1273,9 @@ void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 /* f/w status */
 #define mmMPNHT_SMN_C2PMSG_26_ALT_2	0xAE10968
 
+/* 2MB NPA start address for 2MB page mapping */
+#define AMDGPU_UALINK_SOURCE_ALIAS_NPA_OFFSET SZ_2M
+
 struct amdgpu_ualink_metadata {
 	u32 header;
 
@@ -1481,3 +1484,97 @@ static inline u32 ualink_tlb_wb_offset(struct amdgpu_device *adev, u32 accel_id)
 	return ualink_wb_offset(adev, accel_id) + sizeof(struct amdgpu_ualink_wb);
 }
 
+/*
+ * Reserved NPA space for remote shootdown and interrupt ring buffer,
+ * write pointers, read pointers and writeback buffers
+ *
+ * NPA ring type: 0 = shootdown, 1 = interrupt, 2 = tailptr (wptr)
+ */
+enum ring_buffer_type {
+	RB_TYPE_TLB_INV			= 0,
+	RB_TYPE_REMOTE_INTERRUPT	= 1,
+	RB_TYPE_TAILPTR			= 2
+};
+
+/**
+ * amdgpu_ualink_npa_addr - Get reserved NPA address for ring buffer
+ * @adev: amdgpu device pointer
+ * @type: Ring buffer type (TLB_INV, REMOTE_INTERRUPT, or TAILPTR)
+ * @src_accel_id: Source accelerator ID
+ * @dst_accel_id: Destination accelerator ID
+ *
+ * Return: NPA address for the specified ring buffer type and GPUs
+ */
+static u64 amdgpu_ualink_npa_addr(struct amdgpu_device *adev, u32 type,
+				  u32 src_accel_id, u32 dst_accel_id)
+{
+	u32 addr_mode = ualink_addr_mode(adev);
+	u64 npa;
+
+	WARN_ON_ONCE(src_accel_id >= AMDGPU_UALINK_ACCEL_MAX ||
+		     dst_accel_id >= AMDGPU_UALINK_ACCEL_MAX ||
+		     type > RB_TYPE_TAILPTR ||
+		     (addr_mode != AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT &&
+		     addr_mode != AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS));
+
+	switch (addr_mode) {
+	case AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT:
+		npa = (u64)src_accel_id << 41 | type << 12;
+		break;
+
+	case AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS:
+	default:
+		if (type == RB_TYPE_TAILPTR)
+			npa = (u64)dst_accel_id << 13;
+		else
+			npa = (u64)src_accel_id << 13 | type << 12;
+
+		/*
+		 * In order to avoid address conflicts between source-identification
+		 * and source-aliasing mode, add 2MB to the buffer addresses in
+		 * source-aliasing mode. This way in a misconfigured cluster, a GPU
+		 * configured in the wrong address mode will access unmapped NPA addresses
+		 * rather than the wrong buffer mappings.
+		 */
+		npa += AMDGPU_UALINK_SOURCE_ALIAS_NPA_OFFSET;
+		break;
+	}
+
+	dev_dbg(adev->dev, "addr mode %d from accel %d to accel %d type %d NPA 0x%llx\n",
+		addr_mode, src_accel_id, dst_accel_id, type, npa);
+
+	return npa;
+}
+
+/**
+ * amdgpu_ualink_gart_npa_addr - Get GART-mapped NPA address for ring buffer
+ * @adev: amdgpu device pointer
+ * @type: Ring buffer type (TLB_INV, REMOTE_INTERRUPT, or TAILPTR)
+ * @src_accel_id: Source accelerator ID
+ * @dst_accel_id: Destination accelerator ID
+ *
+ * Computes the NPA address suitable for GART mapping by adding dst_accid_id to
+ * NPA address to access remote by SDMA.
+
+ * clears the source GPU ID bits from the NPA address which is set for
+ * address-identification mode.
+ *
+ * Return: NPA address adjusted for GART mapping
+ */
+
+#define AMDGPU_UALINK_GART_NPA_ADDR_GPUID_SHIFT		41
+#define AMDGPU_UALINK_GART_NPA_ADDR_GPUID_MASK		GENMASK_ULL(50, 41)
+
+static u64 amdgpu_ualink_gart_npa_addr(struct amdgpu_device *adev, u32 type,
+					u32 src_accel_id, u32 dst_accel_id)
+{
+	u64 npa;
+
+	npa = amdgpu_ualink_npa_addr(adev, type, src_accel_id, dst_accel_id);
+
+	/* wiping out source accelerator id */
+	npa &= ~AMDGPU_UALINK_GART_NPA_ADDR_GPUID_MASK;
+
+	return npa | ((u64)dst_accel_id << AMDGPU_UALINK_GART_NPA_ADDR_GPUID_SHIFT);
+}
+
Signature · 1 line
2.55.0
author maintainer
#34 lore raw

PATCH v2 34/95 drm/amdgpu: Add UALink NPA VM mapping for ring buffers

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Populate the NPA VM so remote GPUs can access local ring buffers and write pointer pages at their reserved NPA addresses. Teardown waits for DMA fences and issues a heavyweight TLB flush.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 429 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   3 +
 2 files changed, 432 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +429 -0
index c4edefcef2141..51e0e2336d052 100644
@@ -1484,6 +1484,92 @@ static inline u32 ualink_tlb_wb_offset(struct amdgpu_device *adev, u32 accel_id)
 	return ualink_wb_offset(adev, accel_id) + sizeof(struct amdgpu_ualink_wb);
 }
 
+static void amdgpu_ualink_flush_tlb(struct amdgpu_device *adev, u32 flush_type)
+{
+	uint64_t tlb_seq = amdgpu_vm_tlb_seq(&adev->ualink.npa_vm);
+	u32 bit;
+
+	if (atomic64_xchg(&adev->ualink.last_flushed_tlb_seq, tlb_seq) == tlb_seq)
+		return;
+
+	bit = AMDGPU_MMHUB0_START;
+
+	for_each_set_bit_from(bit, adev->vmhubs_mask, AMDGPU_MAX_VMHUBS)
+		amdgpu_gmc_flush_gpu_tlb(adev, adev->vm_manager.npa_vmid,
+					bit, flush_type);
+}
+
+/**
+ * amdgpu_ualink_npa_vm_map_range - Map a range in the NPA VM
+ * @adev: amdgpu device pointer
+ * @bo: buffer object backing the mapping
+ * @pte_flags: page table entry flags
+ * @offset: offset into the buffer object in bytes
+ * @size_in_pages: size of the range to map in pages
+ * @npa_in_pages: NPA target address in pages
+ *
+ * Maps a buffer object range into the NPA VM page table at the specified
+ * NPA address.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_npa_vm_map_range(struct amdgpu_device *adev, struct amdgpu_bo *bo,
+				   u64 pte_flags, u64 offset, u64 size_in_pages,
+				   u64 npa_in_pages)
+{
+	struct amdgpu_vm *npa_vm = &adev->ualink.npa_vm;
+	int r;
+
+	dev_dbg(adev->dev, "offset 0x%llx size 0x%llx flags 0x%llx npa 0x%llx\n",
+		offset, size_in_pages << AMDGPU_GPU_PAGE_SHIFT, pte_flags,
+		npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
+
+	r = amdgpu_vm_update_range(adev, npa_vm, false, false, true,
+				   false, NULL, npa_in_pages,
+				   npa_in_pages + size_in_pages - 1,
+				   pte_flags, offset, adev->vm_manager.vram_base_offset,
+				   bo->tbo.resource, NULL, &npa_vm->last_update);
+	if (r)
+		dev_dbg(adev->dev, "failed %d to map npa 0x%llx to NPA VM\n", r,
+			npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
+	return r;
+}
+
+/**
+ * amdgpu_npa_vm_unmap_range - Unmap a range from the NPA VM
+ * @adev: amdgpu device pointer
+ * @bo: buffer object backing the mapping
+ * @pte_flags: page table entry flags
+ * @offset: offset into the buffer object in bytes
+ * @size_in_pages: size of the range to unmap in pages
+ * @npa_in_pages: NPA target address in pages
+ *
+ * Removes a previously established mapping from the NPA VM page table.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_npa_vm_unmap_range(struct amdgpu_device *adev, struct amdgpu_bo *bo,
+				     u64 pte_flags, u64 offset, u64 size_in_pages,
+				     u64 npa_in_pages)
+{
+	struct amdgpu_vm *npa_vm = &adev->ualink.npa_vm;
+	int r;
+
+	dev_dbg(adev->dev, "offset 0x%llx size 0x%llx flags 0x%llx npa 0x%llx\n",
+		offset, size_in_pages << AMDGPU_GPU_PAGE_SHIFT, pte_flags,
+		npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
+
+	r = amdgpu_vm_update_range(adev, npa_vm, false, false, true,
+				   false, NULL, npa_in_pages,
+				   npa_in_pages + size_in_pages - 1,
+				   pte_flags, offset, 0, bo->tbo.resource, NULL,
+				   &npa_vm->last_update);
+	if (r)
+		dev_dbg(adev->dev, "failed %d to unmap npa 0x%llx from NPA VM\n", r,
+			npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
+	return r;
+}
+
 /*
  * Reserved NPA space for remote shootdown and interrupt ring buffer,
  * write pointers, read pointers and writeback buffers
@@ -1578,3 +1664,346 @@ static u64 amdgpu_ualink_gart_npa_addr(struct amdgpu_device *adev, u32 type,
 	return npa | ((u64)dst_accel_id << AMDGPU_UALINK_GART_NPA_ADDR_GPUID_SHIFT);
 }
 
+/**
+ * amdgpu_ualink_reserve_npa_vm_and_bos - Reserve the NPA VM page directory and BOs
+ * @adev: amdgpu device pointer
+ * @bos: array of buffer objects to reserve
+ * @n_bos: number of entries in @bos
+ * @exec: drm_exec context to initialize and use for locking
+ * @interruptible: true to use the interruptible dma_resv_lock
+ *
+ * Initializes @exec and uses it to lock all BOs in @bos together with the
+ * NPA VM page directory, retrying on contention. On failure, @exec is
+ * finalized and the error code is returned.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
+static int amdgpu_ualink_reserve_npa_vm_and_bos(struct amdgpu_device *adev,
+						struct amdgpu_bo *bos[], u32 n_bos,
+						struct drm_exec *exec,
+						bool interruptible)
+{
+	u32 flags = DRM_EXEC_IGNORE_DUPLICATES;
+	int i, r = 0;
+
+	if (interruptible)
+		flags |= DRM_EXEC_INTERRUPTIBLE_WAIT;
+
+	dev_dbg(adev->dev, "reserve NPA vm and %d bos\n", n_bos);
+
+	drm_exec_init(exec, flags, 0);
+
+	drm_exec_until_all_locked(exec) {
+		for (i = 0; i < n_bos; i++) {
+			r = drm_exec_lock_obj(exec, &bos[i]->tbo.base);
+			drm_exec_retry_on_contention(exec);
+			if (unlikely(r))
+				goto out;
+		}
+
+		r = amdgpu_vm_lock_pd(&adev->ualink.npa_vm, exec, 0);
+		drm_exec_retry_on_contention(exec);
+		if (unlikely(r))
+			goto out;
+	}
+
+out:
+	if (r)
+		drm_exec_fini(exec);
+	return r;
+
+}
+
+/**
+ * amdgpu_ualink_unreserve_npa_vm_and_bos - Release the NPA VM page directory and BOs
+ * @adev: amdgpu device pointer
+ * @exec: drm_exec context previously initialized by
+ *        amdgpu_ualink_reserve_npa_vm_and_bos()
+ *
+ * Finalizes @exec, releasing all locks on the NPA VM page directory and
+ * the associated BOs acquired during reservation.
+ */
+static void amdgpu_ualink_unreserve_npa_vm_and_bos(struct amdgpu_device *adev,
+						   struct drm_exec *exec)
+{
+	dev_dbg(adev->dev, "unreserve NPA vm and bos\n");
+	drm_exec_fini(exec);
+}
+
+/**
+ * amdgpu_ualink_metadata_npa_unmapping - Tear down NPA address mappings
+ * @adev: amdgpu device pointer
+ *
+ * Unmaps all NPA address mappings from the NPA VM for ring buffers,
+ * write pointers, and read pointers. Waits for outstanding DMA fences
+ * and flushes the TLB.
+ */
+static void amdgpu_ualink_metadata_npa_unmapping(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	u64 timeout = msecs_to_jiffies(2000);
+	u32 dst_accel_id = ualink_accel_id(adev);
+	struct amdgpu_bo *bos[2];
+	u32 n_bos;
+	struct dma_fence *fence;
+	struct drm_exec exec;
+	u32 rptr_size, rptr_size_in_pages;
+	u32 rb_size, rb_size_in_pages;
+	u64 pte_flags = adev->gmc.noretry_flags;
+	u64 npa;
+	int r;
+
+	if (!remote->ring_bo)
+		return;
+	if (!remote->active_accel_bits)
+		return;
+
+	rb_size = AMDGPU_UALINK_RB_SIZE;
+	rb_size_in_pages = rb_size >> AMDGPU_GPU_PAGE_SHIFT;
+
+	bos[0] = remote->ring_bo;
+	n_bos = 1;
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS) {
+		bos[1] = remote->rptr_bo;
+		n_bos = 2;
+	}
+
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, n_bos, &exec, false);
+	if (unlikely(r))
+		return;
+
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS) {
+		/* rptr npa mapping, up to allocated 2 pages npa address */
+		rptr_size = ualink_wb_size(adev);
+		rptr_size = AMDGPU_GPU_PAGE_ALIGN(rptr_size * remote->num_accel);
+		rptr_size_in_pages = rptr_size >> AMDGPU_GPU_PAGE_SHIFT;
+
+		npa = remote->rptr_npa;
+
+		amdgpu_ualink_npa_vm_unmap_range(adev, remote->rptr_bo,
+						 pte_flags, 0, rptr_size_in_pages,
+						 npa >> AMDGPU_GPU_PAGE_SHIFT);
+		amdgpu_ualink_npa_free_va(adev, &remote->rptr_mm_node);
+
+		/*
+		 * unmap wptr, remote interrupt, shootdown ring.
+		 * wptr page is part of the single 2MB mapping for remote GPUs
+		 * interrupt and shootdown ring buffer
+		 */
+		npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TLB_INV,
+					     0, dst_accel_id);
+		amdgpu_ualink_npa_vm_unmap_range(adev, remote->ring_bo, pte_flags,
+						 0,
+						 2 * rb_size_in_pages * AMDGPU_UALINK_ACCEL_MAX,
+						 npa >> AMDGPU_GPU_PAGE_SHIFT);
+	} else {
+		u32 wptr_offset = 2 * rb_size * remote->num_accel;
+		u32 idx = 0;
+		u32 accel_id;
+
+		/* source identification mode */
+		for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+			if (accel_id == dst_accel_id)
+				continue;
+
+			/* remote interrupt ring npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_REMOTE_INTERRUPT,
+						     accel_id, dst_accel_id);
+			amdgpu_ualink_npa_vm_unmap_range(adev, remote->ring_bo,
+							 pte_flags, idx * 2 * rb_size,
+							 rb_size_in_pages,
+							 npa >> AMDGPU_GPU_PAGE_SHIFT);
+
+			/* remote shootdown ring npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TLB_INV, accel_id,
+						     dst_accel_id);
+			amdgpu_ualink_npa_vm_unmap_range(adev, remote->ring_bo,
+							 pte_flags, (idx * 2 + 1) * rb_size,
+							 rb_size_in_pages,
+							 npa >> AMDGPU_GPU_PAGE_SHIFT);
+
+			/* wptr, rptr npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TAILPTR, accel_id,
+						      dst_accel_id);
+			amdgpu_ualink_npa_vm_unmap_range(adev, remote->ring_bo,
+						  pte_flags, wptr_offset + idx * PAGE_SIZE,
+						  1, npa >> AMDGPU_GPU_PAGE_SHIFT);
+			idx++;
+		}
+	}
+
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	if (r) {
+		dev_dbg(adev->dev, "failed %d to update directories\n", r);
+		goto out_unreserve;
+	}
+
+	fence = dma_fence_get(adev->ualink.npa_vm.last_update);
+	if (fence) {
+		r = dma_fence_wait_timeout(fence, true, timeout);
+		dma_fence_put(fence);
+		if (r <= 0)
+			dev_dbg(adev->dev, "failed %d to dma fence wait\n", r);
+	}
+
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+out_unreserve:
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+}
+
+/**
+ * amdgpu_ualink_metadata_npa_mapping - Setup NPA address mapping in VM
+ * @adev: amdgpu device pointer
+ *
+ * Maps NPA addresses to GPA for metadata ring buffers, write pointers,
+ * on NPA VMID 15.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_metadata_npa_mapping(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	u32 dst_accel_id = ualink_accel_id(adev);
+	u64 timeout = msecs_to_jiffies(2000);
+	struct amdgpu_bo *bos[2];
+	u32 n_bos;
+	struct dma_fence *fence;
+	struct drm_exec exec;
+	u32 rb_size, rb_size_in_pages;
+	u64 npa, npa_in_pages, pte_flags;
+	int r;
+
+	rb_size = AMDGPU_UALINK_RB_SIZE;
+	rb_size_in_pages = rb_size >> AMDGPU_GPU_PAGE_SHIFT;
+
+	bos[0] = remote->ring_bo;
+	n_bos = 1;
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS) {
+		bos[1] = remote->rptr_bo;
+		n_bos = 2;
+	}
+
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, n_bos, &exec, false);
+	if (unlikely(r))
+		return r;
+
+	pte_flags = amdgpu_ttm_tt_pte_flags(adev, remote->ring_bo->tbo.ttm,
+					    remote->ring_bo->tbo.resource);
+	dev_dbg(adev->dev, "init pte_flags 0x%llx\n", pte_flags);
+
+	amdgpu_gmc_get_vm_pte(adev, &adev->ualink.npa_vm, remote->ring_bo,
+			      AMDGPU_VM_MTYPE_DEFAULT, &pte_flags);
+	dev_dbg(adev->dev, "after get coherent pte_flags 0x%llx\n", pte_flags);
+
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS) {
+		u32 rptr_size, rptr_size_in_pages;
+
+		/* rptr npa mapping, up to allocated 2 pages npa address */
+		rptr_size = ualink_wb_size(adev);
+		rptr_size = AMDGPU_GPU_PAGE_ALIGN(rptr_size * remote->num_accel);
+		rptr_size_in_pages = rptr_size >> AMDGPU_GPU_PAGE_SHIFT;
+
+		r = amdgpu_ualink_npa_alloc_va(adev, &remote->rptr_mm_node,
+					       0, 0, 0,
+					       rptr_size_in_pages);
+		if (r)
+			goto out;
+
+		npa_in_pages = remote->rptr_mm_node.start;
+
+		dev_dbg(adev->dev, "source aliasing rptr alloc 0x%llx and map to npa vm\n",
+			npa_in_pages << AMDGPU_GPU_PAGE_SHIFT);
+
+		r = amdgpu_ualink_npa_vm_map_range(adev, remote->rptr_bo, pte_flags, 0,
+						   rptr_size_in_pages,
+						   npa_in_pages);
+		if (r)
+			goto error_npa_mapping;
+
+		remote->rptr_npa = npa_in_pages << AMDGPU_GPU_PAGE_SHIFT;
+
+		/* wptr, remote interrupt, shootdown ring, single big 2MB mapping */
+		npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TLB_INV,
+					     0, dst_accel_id);
+		r = amdgpu_ualink_npa_vm_map_range(adev, remote->ring_bo, pte_flags,
+						   0,
+						   2 * rb_size_in_pages * AMDGPU_UALINK_ACCEL_MAX,
+						   npa >> AMDGPU_GPU_PAGE_SHIFT);
+		if (r)
+			goto error_npa_mapping;
+
+	} else {
+		u32 wptr_offset = 2 * rb_size * remote->num_accel;
+		u32 accel_id, idx = 0;
+
+		/* Source identification mode */
+		for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+			if (accel_id == dst_accel_id)
+				continue;
+
+			/* remote interrupt ring npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_REMOTE_INTERRUPT,
+						      accel_id, dst_accel_id);
+			r = amdgpu_ualink_npa_vm_map_range(adev, remote->ring_bo, pte_flags,
+							   idx * 2 * rb_size,
+							   rb_size_in_pages,
+							   npa >> AMDGPU_GPU_PAGE_SHIFT);
+			if (r)
+				goto error_npa_mapping;
+
+			/* remote shootdown ring npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TLB_INV, accel_id,
+						     dst_accel_id);
+			r = amdgpu_ualink_npa_vm_map_range(adev, remote->ring_bo, pte_flags,
+							   (idx * 2 + 1) * rb_size,
+							   rb_size_in_pages,
+							   npa >> AMDGPU_GPU_PAGE_SHIFT);
+			if (r)
+				goto error_npa_mapping;
+
+			/* wptr, rptr npa mapping */
+			npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TAILPTR, accel_id,
+						     dst_accel_id);
+			r = amdgpu_ualink_npa_vm_map_range(adev, remote->ring_bo,
+							   pte_flags,
+							   wptr_offset + idx * PAGE_SIZE,
+							   1, npa >> AMDGPU_GPU_PAGE_SHIFT);
+			if (r)
+				goto error_npa_mapping;
+
+			idx++;
+		}
+	}
+
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	if (r) {
+		dev_dbg(adev->dev, "failed %d to update directories\n", r);
+		goto error_npa_mapping;
+	}
+
+	/* TODO: only wait the last fence, then flush TLB */
+	fence = dma_fence_get(adev->ualink.npa_vm.last_update);
+	if (fence) {
+		r = dma_fence_wait_timeout(fence, true, timeout);
+		dma_fence_put(fence);
+		if (r <= 0)
+			dev_dbg(adev->dev, "failed %d to dma fence wait\n", r);
+	}
+
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+	return 0;
+
+error_npa_mapping:
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+		amdgpu_ualink_npa_free_va(adev, &remote->rptr_mm_node);
+	if (r)
+		dev_dbg(adev->dev, "failed %d to map NPA vm\n", r);
+
+out:
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+
+	return r;
+}
+
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index a75aac48604cb..3fde1b80c3e92 100644
@@ -185,6 +185,9 @@ struct amdgpu_ualink_mgr {
 
 	/* NPA-VM used on the exporter.*/
 	struct amdgpu_vm npa_vm;
+
+	/* Sequence number to track the need for TLB flushes */
+	atomic64_t last_flushed_tlb_seq;
 };
 
 int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#35 lore raw

PATCH v2 35/95 drm/amdgpu: Add UALink SDMA scheduler entities

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Allocate one GPU scheduler entity per SDMA instance for remote command dispatch. Commands to the same peer are serialized through one entity; different peers can be targeted concurrently by different SDMA engines.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 53 ++++++++++++++++++++++
 1 file changed, 53 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +53 -0
index 51e0e2336d052..2cb805dd6cbf5 100644
@@ -2007,3 +2007,56 @@ static int amdgpu_ualink_metadata_npa_mapping(struct amdgpu_device *adev)
 	return r;
 }
 
+/**
+ * amdgpu_ualink_sdma_entities_init - Initialize SDMA job scheduler entities
+ * @adev: amdgpu device pointer
+ *
+ * Sets up SDMA job scheduler entities for remote interrupt and shootdown.
+ * Each entity uses one SDMA scheduler/ring.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_sdma_entities_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct drm_gpu_scheduler *sched;
+	u32 accel_id;
+	int i = 0, r;
+
+	dev_dbg(adev->dev, "enter\n");
+
+	for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+		sched = &adev->sdma.instance[accel_id % adev->sdma.num_instances].ring.sched;
+
+		r = drm_sched_entity_init(&remote->peer[accel_id].entity, DRM_SCHED_PRIORITY_HIGH,
+					  &sched, 1, NULL);
+		if (r)
+			goto out_free;
+		i++;
+	}
+
+	dev_dbg(adev->dev, "exit\n");
+	return 0;
+
+out_free:
+	for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+		if (i--)
+			drm_sched_entity_destroy(&remote->peer[accel_id].entity);
+		else
+			break;
+	}
+
+	return r;
+}
+
+static void amdgpu_ualink_sdma_entities_fini(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	u32 accel_id;
+
+	dev_dbg(adev->dev, "enter\n");
+	for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX)
+		drm_sched_entity_destroy(&remote->peer[accel_id].entity);
+	dev_dbg(adev->dev, "exit\n");
+}
+
Signature · 1 line
2.55.0
author maintainer
#36 lore raw

PATCH v2 36/95 drm/amdgpu: Add UALink GART helpers for NPA address access

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Add helpers to insert and remove NPA addresses in the GART page table without a TTM buffer object, allowing the local SDMA engine to reach remote GPU ring buffers, write pointer pages, and doorbell addresses.

v2: fix 32 bit build (Alex)

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 75 ++++++++++++++++++++++
 1 file changed, 75 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +75 -0
index 2cb805dd6cbf5..d7010cde11bb4 100644
@@ -2060,3 +2060,78 @@ static void amdgpu_ualink_sdma_entities_fini(struct amdgpu_device *adev)
 	dev_dbg(adev->dev, "exit\n");
 }
 
+/**
+ * amdgpu_ualink_gart_map - Allocate GART entry and map NPA address
+ * @adev: amdgpu device pointer
+ * @npages: Number of pages to map
+ * @npa: NPA address to map
+ * @mm_node: DRM memory manager node for GART allocation
+ * @pte_flags: the GART mapping flags
+ *
+ * Allocates GART entries and sets up NPA address mapping without TTM BO.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_gart_map(struct amdgpu_device *adev, u64 npages,
+				   u64 npa, struct drm_mm_node *mm_node,
+				   u64 pte_flags)
+{
+	struct ttm_resource_manager *man =
+				ttm_manager_type(&adev->mman.bdev, TTM_PL_TT);
+	struct amdgpu_gtt_mgr *mgr =
+				container_of(man, struct amdgpu_gtt_mgr, manager);
+	dma_addr_t *dma_addr;
+	int i, r;
+
+	dev_dbg(adev->dev, "npa 0x%llx npages 0x%llx\n", npa, npages);
+
+	if (npages > 1) {
+		dma_addr = kmalloc_array(npages, sizeof(*dma_addr), GFP_KERNEL);
+		if (!dma_addr)
+			return -ENOMEM;
+
+		for (i = 0; i < npages; i++)
+			dma_addr[i] = npa + i * AMDGPU_GPU_PAGE_SIZE;
+	} else {
+		dma_addr = (dma_addr_t *)&npa;
+	}
+
+	r = amdgpu_gtt_mgr_alloc_entries(mgr, mm_node, npages, DRM_MM_INSERT_BEST);
+	if (r)
+		goto out;
+	amdgpu_gart_bind(adev, mm_node->start << AMDGPU_GPU_PAGE_SHIFT, npages, dma_addr,
+			 pte_flags);
+
+out:
+	dev_dbg(adev->dev, "npa 0x%llx mapped to 0x%llx npages 0x%llx r=%d\n",
+		npa, mm_node->start << AMDGPU_GPU_PAGE_SHIFT, npages, r);
+
+	if (npages > 1)
+		kfree(dma_addr);
+	return r;
+}
+
+/**
+ * amdgpu_ualink_gart_unmap - Unmap and free GART entry
+ * @adev: amdgpu device pointer
+ * @npages: Number of pages to unmap
+ * @mm_node: DRM memory manager node for GART allocation
+ *
+ * Unbinds GART mapping and frees allocated entries.
+ */
+static void amdgpu_ualink_gart_unmap(struct amdgpu_device *adev, u64 npages,
+				      struct drm_mm_node *mm_node)
+{
+	struct ttm_resource_manager *man = ttm_manager_type(&adev->mman.bdev, TTM_PL_TT);
+	struct amdgpu_gtt_mgr *mgr =
+				container_of(man, struct amdgpu_gtt_mgr, manager);
+
+	dev_dbg(adev->dev, "0x%llx npages 0x%llx\n", mm_node->start << AMDGPU_GPU_PAGE_SHIFT,
+		npages);
+
+	if (!drm_mm_node_allocated(mm_node))
+		return;
+	amdgpu_gart_unbind(adev, mm_node->start << AMDGPU_GPU_PAGE_SHIFT, npages);
+	amdgpu_gtt_mgr_free_entries(mgr, mm_node);
+}
+
Signature · 1 line
2.55.0
author maintainer
#37 lore raw

PATCH v2 37/95 drm/amdgpu: Add UALink ring buffer allocation and firmware init

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Allocate pinned memory for firmware metadata, ring buffers, and write and read pointer pages. Populate per-GPU metadata entries and send them to MPNHT firmware via SMN mailbox, then poll for a ready acknowledgement.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 214 +++++++++++++++++++++
 1 file changed, 214 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +214 -0
index d7010cde11bb4..c8b01e9a2eabe 100644
@@ -1499,6 +1499,25 @@ static void amdgpu_ualink_flush_tlb(struct amdgpu_device *adev, u32 flush_type)
 					bit, flush_type);
 }
 
+static inline u32 amdgpu_ualink_mailbox_read(struct amdgpu_device *adev,
+					     u32 mailbox_reg)
+{
+	u32 value;
+
+	value = RREG32_PCIE(mailbox_reg);
+	dev_dbg_ratelimited(adev->dev, "ualink read mailbox 0x%x return value 0x%x\n",
+			    mailbox_reg, value);
+	return value;
+}
+
+static inline void amdgpu_ualink_mailbox_write(struct amdgpu_device *adev,
+					       u32 mailbox_reg, u32 value)
+{
+	dev_dbg(adev->dev, "ualink write mailbox 0x%x value 0x%x\n",
+		mailbox_reg, value);
+	WREG32_PCIE(mailbox_reg, value);
+}
+
 /**
  * amdgpu_ualink_npa_vm_map_range - Map a range in the NPA VM
  * @adev: amdgpu device pointer
@@ -2135,3 +2154,198 @@ static void amdgpu_ualink_gart_unmap(struct amdgpu_device *adev, u64 npages,
 	amdgpu_gtt_mgr_free_entries(mgr, mm_node);
 }
 
+/**
+ * amdgpu_ualink_metadata_fini - Clean up ualink metadata structures
+ * @adev: amdgpu device pointer
+ *
+ * Unmaps NPA addresses and frees all allocated buffers for metadata,
+ * ring buffers, and pointers.
+ */
+static void amdgpu_ualink_metadata_fini(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+
+	dev_dbg(adev->dev, "accel_id %u\n", ualink_accel_id(adev));
+	amdgpu_bo_free_kernel(&remote->rptr_bo, &remote->rptr_gpu_addr,
+				      &remote->rptr_cpu_addr);
+	amdgpu_bo_free_kernel(&remote->ring_bo, &remote->rb_gpu_addr,
+			      &remote->rb_cpu_addr);
+	amdgpu_bo_free_kernel(&remote->metadata_bo, &remote->metadata_gpu_addr,
+			      &remote->metadata_cpu_addr);
+}
+
+/**
+ * amdgpu_ualink_metadata_init - Initialize ualink metadata structures
+ * @adev: amdgpu device pointer
+ *
+ * Allocates and initializes metadata structures, ring buffers, and write/read
+ * pointers for multi-GPU ualink. Communicates with firmware to load metadata.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_metadata_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_ualink_metadata *metadata;
+	u32 size, rb_size, wptr_size, rptr_size, metadata_size;
+	u64 rb_gpu_addr, wptr_gpu_addr;
+	u32 status, accel_id;
+	int i, r;
+
+	remote->active_accel_bits = adev->ualink.info->vpod.active_accel_bits;
+	dev_dbg(adev->dev, "%d active accelerators config in vpod\n",
+		bitmap_weight(remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX));
+
+	/*
+	 * allocate metadata entries and ring buffer for all remote GPUs,
+	 * to get 2MB page ring buffer NPA mapping for remote access.
+	 */
+	remote->num_accel = AMDGPU_UALINK_ACCEL_MAX;
+
+	status = amdgpu_ualink_mailbox_read(adev, mmMPNHT_SMN_C2PMSG_26_ALT_2);
+	if (status != AMDGPU_UALINK_FW_STATUS_PREINIT &&
+	    status != AMDGPU_UALINK_FW_STATUS_HALT) {
+		dev_dbg(adev->dev, "fw status 0x%x not preinit or halt\n", status);
+		return -ENODEV;
+	}
+
+	dev_dbg(adev->dev, "accel_id %u addr_mode %d fw status 0x%x\n",
+		ualink_accel_id(adev), ualink_addr_mode(adev), status);
+
+	/* Alloc metadata structure for all GPUs */
+	metadata_size = sizeof(struct amdgpu_ualink_metadata) * AMDGPU_UALINK_ACCEL_MAX;
+	metadata_size = AMDGPU_GPU_PAGE_ALIGN(metadata_size);
+
+	dev_dbg(adev->dev, "metadata size 0x%x\n", metadata_size);
+
+	/* pinned system memory */
+	r = amdgpu_bo_create_kernel(adev, metadata_size, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_GTT,
+				    &remote->metadata_bo, &remote->metadata_gpu_addr,
+				    &remote->metadata_cpu_addr);
+	if (r)
+		goto out;
+
+	memset(remote->metadata_cpu_addr, 0, metadata_size);
+	metadata = remote->metadata_cpu_addr;
+
+	dev_dbg(adev->dev, "metadata gpu address 0x%llx\n", remote->metadata_gpu_addr);
+
+	/* Allocate ring buffers, wptr for remote interrupt and shootdown */
+	rb_size = AMDGPU_GPU_PAGE_ALIGN(2 * AMDGPU_UALINK_RB_SIZE);
+	wptr_size = ualink_wptr_size(adev);
+	size = (rb_size + wptr_size) * remote->num_accel;
+	size = AMDGPU_GPU_PAGE_ALIGN(size);
+
+	dev_dbg(adev->dev, "rb_size 0x%x wptr_size 0x%x total alloc size 0x%x\n",
+		rb_size, wptr_size, size);
+
+	/* pinned VRAM */
+	r = amdgpu_bo_create_kernel(adev, size, PAGE_SIZE,
+				    AMDGPU_GEM_DOMAIN_VRAM,
+				    &remote->ring_bo,
+				    &remote->rb_gpu_addr,
+				    &remote->rb_cpu_addr);
+	if (r)
+		goto out;
+
+	memset(remote->rb_cpu_addr, 0, size);
+
+	dev_dbg(adev->dev, "rb gpu addr 0x%llx cpu addr 0x%p vram_start 0x%llx vram_base 0x%llx\n",
+		remote->rb_gpu_addr, remote->rb_cpu_addr, adev->gmc.vram_start,
+		adev->vm_manager.vram_base_offset);
+
+	/*
+	 * address aliasing mode, shared wptr pagee left is not enough for wb data,
+	 * wb status and rptr, alloc another BO, and then alloc npa address and map
+	 * to npa vm, for remote to access.
+	 *
+	 * No gart mapping required for wb data, status and rptr because this is
+	 * updated by firmware.
+	 */
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS) {
+		rptr_size = AMDGPU_GPU_PAGE_ALIGN(ualink_wb_size(adev) * remote->num_accel);
+		dev_dbg(adev->dev, "source aliasing mode rptr_size 0x%x\n", rptr_size);
+
+		/* pinned VRAM */
+		r = amdgpu_bo_create_kernel(adev, rptr_size, PAGE_SIZE,
+					    AMDGPU_GEM_DOMAIN_VRAM,
+					    &remote->rptr_bo,
+					    &remote->rptr_gpu_addr,
+					    &remote->rptr_cpu_addr);
+		if (r)
+			goto out;
+
+		memset(remote->rptr_cpu_addr, 0, rptr_size);
+
+		dev_dbg(adev->dev, "source aliasing rptr gpu addr 0x%llx cpu addr 0x%p\n",
+			remote->rptr_gpu_addr, remote->rptr_cpu_addr);
+	}
+
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+		wptr_gpu_addr = remote->rb_gpu_addr + rb_size * ualink_accel_id(adev);
+
+	for_each_set_bit(accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+		dev_dbg(adev->dev, "init for accel_id %u\n", accel_id);
+
+		if (accel_id == ualink_accel_id(adev))
+			continue;
+
+		rb_gpu_addr = remote->rb_gpu_addr + rb_size * accel_id;
+
+		if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT) {
+			wptr_gpu_addr = remote->rb_gpu_addr + rb_size * remote->num_accel;
+			wptr_gpu_addr += wptr_size * accel_id;
+		}
+
+		metadata[accel_id].header = AMDGPU_UALINK_METADATA_HEADER;
+		metadata[accel_id].rb_size = fls(AMDGPU_UALINK_RB_SIZE / 64) - 1;
+
+		/*
+		 * remote shootdown type is 0, remote command type is 1
+		 * with 2MB ring buffer mapping, remote shootdown ring NPA is before interrupt
+		 */
+		metadata[accel_id].ri_rb = rb_gpu_addr + AMDGPU_UALINK_RB_SIZE;
+		metadata[accel_id].tlb_inv_rb = rb_gpu_addr;
+
+		metadata[accel_id].tailptr_ri = wptr_gpu_addr +
+						ualink_wptr_offset(adev, accel_id);
+		metadata[accel_id].tailptr_tlb_inv = wptr_gpu_addr +
+						     ualink_tlb_wptr_offset(adev, accel_id);
+
+		dev_dbg(adev->dev, "init from accel_id %u to accel_id %u, rb_size 0x%x\n",
+			accel_id, ualink_accel_id(adev), metadata[accel_id].rb_size);
+		dev_dbg(adev->dev, "rb 0x%llx tlb rb 0x%llx\n",
+			metadata[accel_id].ri_rb, metadata[accel_id].tlb_inv_rb);
+		dev_dbg(adev->dev, "rb wptr at 0x%llx tlb wptr at 0x%llx\n",
+			metadata[accel_id].tailptr_ri, metadata[accel_id].tailptr_tlb_inv);
+	}
+
+	amdgpu_ualink_mailbox_write(adev, mmMPNHT_SMN_C2PMSG_25_ALT_2,
+				    upper_32_bits(remote->metadata_gpu_addr));
+	amdgpu_ualink_mailbox_write(adev, mmMPNHT_SMN_C2PMSG_24_ALT_2,
+				    lower_32_bits(remote->metadata_gpu_addr));
+	amdgpu_ualink_mailbox_write(adev, mmMPNHT_SMN_C2PMSG_23_ALT_2,
+				    AMDGPU_UALINK_ACCEL_MAX << 8);
+	amdgpu_ualink_mailbox_write(adev, mmMPNHT_SMN_C2PMSG_22_ALT_2,
+				    AMDGPU_UALINK_FW_CMD_LOAD_METADATA);
+
+	for (i = 0; i < 2000; i++) {
+		status = amdgpu_ualink_mailbox_read(adev, mmMPNHT_SMN_C2PMSG_26_ALT_2);
+		if (status == AMDGPU_UALINK_FW_STATUS_READY)
+			break;
+		mdelay(1);
+	}
+	if (status != AMDGPU_UALINK_FW_STATUS_READY) {
+		dev_dbg(adev->dev, "f/w load metadata failed 0x%x\n", status);
+		r = -ETIME;
+	}
+
+out:
+	if (r)
+		amdgpu_ualink_metadata_fini(adev);
+
+	dev_dbg(adev->dev, "ret 0x%x\n", r);
+	return r;
+}
+
Signature · 1 line
2.55.0
author maintainer
#38 lore raw

PATCH v2 38/95 drm/amdgpu: Add UALink remote command packets and SDMA dispatch

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Add 16-dword packet layouts for TLB shootdown, remote interrupt, and writeback address update. The dispatch path issues three sequential SDMA copies (ring entry, write pointer, doorbell) and polls the firmware writeback buffer for the acknowledged sequence number.

v2: fix 32 bit build (Alex)

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 350 +++++++++++++++++++++
 1 file changed, 350 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +350 -0
index c8b01e9a2eabe..58bb506785c27 100644
@@ -27,6 +27,7 @@
 #include "amdgpu.h"
 #include "amdgpu_ualink.h"
 #include "amdgpu_xgmi.h"
+#include <linux/math.h>
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
@@ -2349,3 +2350,352 @@ static int amdgpu_ualink_metadata_init(struct amdgpu_device *adev)
 	return r;
 }
 
+/*
+ * nHT Firmware Error code
+ *
+ * Success				0x0
+ * FIFO overflow			0x1
+ * Invalid RB command			0x2
+ * Timeout				0x3
+ * Invalid metadata entry		0x4
+ * RB overflow				0x5
+ * Invalid wptr address			0x6
+ * Interrupt cookie send failure	0x7
+ * Invalid RB address			0x8
+ * Invalid writeback address		0x9
+ */
+static void amdgpu_ualink_remote_error(struct amdgpu_device *adev, u32 status)
+{
+	const char *fw_err_code_msg[] = {
+		"Unknown FW error status",
+		"FIFO overflow",		/* error code 1 */
+		"Invalid RB command",
+		"Timeout",
+		"Invalid metadata entry",
+		"RB overflow",
+		"Invalid wptr address",
+		"Interrupt cookie send failure",
+		"Invalid RB address",
+		"Invalid writeback address"	/* error code 9 */
+		};
+
+	if (status >= ARRAY_SIZE(fw_err_code_msg))
+		status = 0;
+
+	dev_err(adev->dev, "remote error %s\n", fw_err_code_msg[status]);
+	return;
+}
+
+/**
+ * amdgpu_ualink_remote_wait_timeout - Wait for remote operation completion
+ * @remote_accel_id: remote accelator id to wait for reply
+ * @adev: amdgpu device pointer
+ * @wb_cpu: CPU virtual address of writeback buffer
+ * @seq: Sequence number to wait for
+ *
+ * Polls the writeback buffer waiting for the sequence number to reach or
+ * exceed the expected value. The writeback buffer contains status and data
+ * fields updated by firmware to indicate completion and error conditions.
+ *
+ * Return: 0 on success
+ *         -ETIME on timeout
+ *         -ECOMM on firmware return error status
+ */
+static long amdgpu_ualink_remote_wait_timeout(struct amdgpu_device *adev,
+					      u32 remote_accel_id,
+					      struct amdgpu_ualink_wb *wb_cpu,
+					      u32 seq)
+{
+	/* 2 seconds timeout, long enough for FW to reply */
+	ktime_t timeout = ktime_add_us(ktime_get(), 2 * USEC_PER_SEC);
+	u32 status, data;
+	u64 rptr;
+
+	while (true) {
+		data = READ_ONCE(wb_cpu->data);
+		if (data >= seq) {
+			status = READ_ONCE(wb_cpu->status);
+			rptr = READ_ONCE(wb_cpu->rptr);
+			break;
+		}
+
+		if (ktime_after(ktime_get(), timeout)) {
+			dev_dbg(adev->dev, "remote %d wait timeout\n", remote_accel_id);
+			return -ETIME;
+		}
+
+		usleep_range(10, 50);
+	}
+
+	if (status) {
+		dev_dbg(adev->dev, "remote %u status 0x%x wb data 0x%x seq 0x%x rptr 0x%llx\n",
+			remote_accel_id, status, data, seq, rptr);
+
+		amdgpu_ualink_remote_error(adev, status);
+		return -ECOMM;
+	}
+
+	dev_dbg(adev->dev, "remote %u succeed seq 0x%x wb data 0x%x rptr 0x%llx\n",
+		remote_accel_id, seq, data, rptr);
+	return 0;
+}
+
+#define AMDGPU_UALINK_REMOTE_OP_TLB_INV		0x1
+#define AMDGPU_UALINK_REMOTE_OP_INT		0x2
+#define AMDGPU_UALINK_REMOTE_OP_UPDATE_WB	0x3
+
+static void amdgpu_ualink_emit_shootdown(u32 **cpu_addr_p, u32 wb_data,
+					 u32 dw0,  u32 dw1, u32 dw2, u32 dw3)
+{
+	u64 addr = (((u64)dw1 << 32) | (u64)dw2) << AMDGPU_GPU_PAGE_SHIFT;
+	u32 *cpu_addr = *cpu_addr_p;
+	u32 flush_type = dw0;
+	u32 size_in_pages = dw3;
+	u32 cmd;
+
+	cmd = AMDGPU_UALINK_REMOTE_OP_TLB_INV;
+	cmd |= 1 << 28;	/* headptr update */
+	cmd |= 1 << 29;	/* wb enable */
+
+	*cpu_addr++ = cmd;
+	*cpu_addr++ = wb_data;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0; /* metadata */
+	*cpu_addr++ = 0; /* metadata */
+
+	pr_debug("NPA addr 0x%llx npages 0x%x\n", addr, size_in_pages);
+
+	/* Always shootdown everything with full address size s-field coding */
+	addr = GENMASK_ULL(51, 11);
+
+	*cpu_addr++ = lower_32_bits(addr) | flush_type;
+	*cpu_addr++ = upper_32_bits(addr);
+
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr_p = cpu_addr;
+}
+
+static void amdgpu_ualink_emit_interrupt(u32 **cpu_addr_p, u32 wb_data,
+					 u32 dw0, u32 dw1, u32 dw2, u32 dw3)
+{
+	u32 *cpu_addr = *cpu_addr_p;
+	u32 cmd;
+
+	cmd = AMDGPU_UALINK_REMOTE_OP_INT;
+	cmd |= 1 << 28;	/* headptr_update */
+	cmd |= 1 << 29; /* wb enable */
+
+	*cpu_addr++ = cmd;
+	*cpu_addr++ = wb_data;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0; /* metadata */
+	*cpu_addr++ = 0; /* metadata */
+	/* updated by f/w, overwrite PASID with GPU ID */
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	/* IH ContextID 4 dwords */
+	*cpu_addr++ = dw0;
+	*cpu_addr++ = dw1;
+	*cpu_addr++ = dw2;
+	*cpu_addr++ = dw3;
+	*cpu_addr_p = cpu_addr;
+}
+
+static void amdgpu_ualink_emit_update_wb_addr(u32 **cpu_addr_p, u32 wb_data,
+					      u32 dw0, u32 dw1, u32 dw2, u32 dw3)
+{
+	u32 *cpu_addr = *cpu_addr_p;
+	u32 cmd;
+
+	cmd = AMDGPU_UALINK_REMOTE_OP_UPDATE_WB;
+	cmd |= 1 << 28;	/* headptr update */
+	cmd |= 1 << 29;	/* wb enable */
+
+	*cpu_addr++ = cmd;
+	*cpu_addr++ = wb_data;
+	*cpu_addr++ = dw1 & 0xFFFFFFFC;	/* lower32 wb_npa */
+	*cpu_addr++ = dw0 & 0xFFFFF;	/* upper32 wb_npa */
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0; /* metadata */
+	*cpu_addr++ = 0; /* metadata */
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr++ = 0;
+	*cpu_addr_p = cpu_addr;
+}
+
+/*
+ * Function proto to generate packet using above different emit_ functions
+ */
+typedef void (*ualink_emit_packet)(u32 **cpu_addr_p, u32 wb, u32 dw0,
+				   u32 dw1, u32 dw2, u32 dw3);
+
+static int amdgpu_ualink_send_command(struct amdgpu_device *adev,
+				      u32 remote_accel_id,
+				      struct amdgpu_ualink_ring *ring,
+				      struct amdgpu_ualink_wb *wb_cpu,
+				      ualink_emit_packet emit_func,
+				      u32 dw0, u32 dw1, u32 dw2, u32 dw3)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ring *sdma_ring;
+	struct dma_fence *fence;
+	struct amdgpu_job *job;
+	struct amdgpu_ib *ib;
+	u32 seq, ndw, ndw_copy_cmd, rem;
+	u64 doorbell_npa_gart;
+	u32 *src_cpu;
+	u64 src;
+	int r;
+
+	peer = &remote->peer[remote_accel_id];
+
+	/*
+	 * 3 sdma copy commands: write data to ring buffer, update wptr, ring doorbell
+	 *
+	 * insert 2 NOP commands to break SDMA back-to-back copy command overlap, to
+	 * ensure the ordering of 3 sdma copy commands execution.
+	 */
+
+	/* buffer after sdma commands, starting at 8 dwords boundary */
+	ndw_copy_cmd = ALIGN(3 * adev->mman.buffer_funcs->copy_num_dw + 2, 8);
+
+	 /* remote command 16 dwords, wptr 2 dwords, doorbell 1 dword */
+	ndw = ndw_copy_cmd + 16 + 2 + 1;
+	r = amdgpu_job_alloc_with_ib(adev, &peer->entity, AMDGPU_FENCE_OWNER_VM,
+				     ndw * 4, AMDGPU_IB_POOL_IMMEDIATE,
+				     AMDGPU_KERNEL_JOB_ID_TTM_COPY_BUFFER, &job);
+	if (r)
+		return r;
+
+	mutex_lock(&peer->lock);
+
+	ring->rptr = READ_ONCE(wb_cpu->rptr);
+
+	if (WARN_ON_ONCE(ring->rptr > ring->wptr)) {
+		dev_err(adev->dev, "accel_id %u ring overflow wptr 0x%llx rptr 0x%llx\n",
+			remote_accel_id, ring->wptr, ring->rptr);
+		r = -EFAULT;
+		goto unlock_free;
+	}
+
+	if ((ring->wptr + 1 - ring->rptr) >= ring->rb_size) {
+		dev_err(adev->dev, "accel_id %u command ring full wptr 0x%llx rptr 0x%llx\n",
+			remote_accel_id, ring->wptr, ring->rptr);
+		r = -ENOSPC;
+		goto unlock_free;
+	}
+
+	ib = &job->ibs[0];
+	src = ib->gpu_addr + ndw_copy_cmd * 4;
+	src_cpu = ib->ptr + ndw_copy_cmd;
+
+	seq = ++ring->seq;
+
+	div_u64_rem(ring->wptr, ring->rb_size, &rem);
+	dev_dbg(adev->dev, "src 0x%llx to npa gart rb 0x%llx wptr 0x%llx doorbell 0x%llx seq 0x%x\n",
+		src, ring->rb_npa_gart + rem * 64,
+		ring->wptr_npa_gart, ring->doorbell_npa_gart, seq);
+
+	dev_dbg(adev->dev, "ring wptr 0x%llx rptr 0x%llx\n", ring->wptr, ring->rptr);
+
+	/* remote command packet */
+	emit_func(&src_cpu, seq, dw0, dw1, dw2, dw3);
+
+	/* wptr value */
+	*src_cpu++ = lower_32_bits(ring->wptr + 1);
+	*src_cpu++ = upper_32_bits(ring->wptr + 1);
+
+	/* doorbell value */
+	*src_cpu++ = lower_32_bits(ring->wptr + 1);
+
+	/*
+	 * Add an offset to the doorbell address that cycles through different
+	 * values of bit 11:8 to use different DXS ports.
+	 */
+	peer->dxs_port = (peer->dxs_port + 1) & 0xF;
+	doorbell_npa_gart = ring->doorbell_npa_gart | (peer->dxs_port << 8);
+	dev_dbg(adev->dev, "dxs_port 0x%x, doorbell_npa_gart 0x%llx -> 0x%llx\n",
+		peer->dxs_port, ring->doorbell_npa_gart, doorbell_npa_gart);
+
+	sdma_ring = &adev->sdma.instance[0].ring;
+
+	/*
+	 * SDMA commands copy from ib to NPA, insert NOPs to ensure SDMA copy commands
+	 * execution doesn't overlap, doorbell update is after data and wptr update
+	 * finished.
+	 */
+	div_u64_rem(ring->wptr, ring->rb_size, &rem);
+	amdgpu_emit_copy_buffer(adev, ib, src,
+				ring->rb_npa_gart + rem * 64,
+				64, 0);
+
+	ib->ptr[ib->length_dw++] = sdma_ring->funcs->nop;
+
+	amdgpu_emit_copy_buffer(adev, ib, src + 64, ring->wptr_npa_gart, 8, 0);
+
+	ib->ptr[ib->length_dw++] = sdma_ring->funcs->nop;
+
+	amdgpu_emit_copy_buffer(adev, ib, src + 72, doorbell_npa_gart, 4, 0);
+
+	amdgpu_ring_pad_ib(sdma_ring, ib);
+	WARN_ON(ib->length_dw > ndw_copy_cmd);
+
+	fence = amdgpu_job_submit(job);
+	r = dma_fence_wait_timeout(fence, false, AMDGPU_FENCE_JIFFIES_TIMEOUT);
+	dma_fence_put(fence);
+	if (r <= 0) {
+		dev_dbg(adev->dev, "remote %u sdma fence wait return r %d\n",
+			remote_accel_id, r);
+
+		if (r == 0)
+			r = -ETIME;
+
+		mutex_unlock(&peer->lock);
+		return r;
+	}
+
+	/*
+	 * poll remote completion writeback data
+	 */
+	r = amdgpu_ualink_remote_wait_timeout(adev, remote_accel_id, wb_cpu, seq);
+
+	/* increase local copy ring wptr, only if FW not timeout */
+	if (r != -ETIME)
+		ring->wptr++;
+
+	/*
+	 * Release ring lock after the remote FW handle command completes to
+	 * prevent race conditions.
+	 */
+	mutex_unlock(&peer->lock);
+	return r;
+
+
+unlock_free:
+	mutex_unlock(&peer->lock);
+	amdgpu_job_free(job);
+	dev_dbg(adev->dev, "ret r = %d\n", r);
+	return r;
+}
+
Signature · 1 line
2.55.0
author maintainer
#39 lore raw

PATCH v2 39/95 drm/amdgpu: Add UALink firmware writeback address configuration

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Compute the writeback buffer addresses for a ring, send a configuration command to firmware, and mark the ring ready on acknowledgement. Configuration is deferred to first use.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 70 ++++++++++++++++++++++
 1 file changed, 70 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +70 -0
index 58bb506785c27..831e8e8e45fbf 100644
@@ -1338,6 +1338,9 @@ struct amdgpu_ualink_ring {
 	struct drm_mm_node mm_node_wptr;
 	struct drm_mm_node mm_node_doorbell;
 
+	/* seq, complete status write back cpu address */
+	struct amdgpu_ualink_wb *wb_cpu;
+
 	/* true if fw write back address updated successfully */
 	bool ready;
 };
@@ -2699,3 +2702,70 @@ static int amdgpu_ualink_send_command(struct amdgpu_device *adev,
 	return r;
 }
 
+static void amdgpu_ualink_get_wb_addr(struct amdgpu_device *adev,
+				      u32 remote_accel_id,
+				      struct amdgpu_ualink_wb **wb_cpu,
+				      u64 *wb_npa, u32 type)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	u32 accel_id = ualink_accel_id(adev);
+	u32 rb_size = AMDGPU_GPU_PAGE_ALIGN(2 * AMDGPU_UALINK_RB_SIZE);
+	u64 wb, npa;
+	u32 offset = 0;
+
+	if (ualink_addr_mode(adev) == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT) {
+		wb = (u64)remote->rb_cpu_addr + rb_size * remote->num_accel;
+		npa = amdgpu_ualink_npa_addr(adev, RB_TYPE_TAILPTR,
+					     remote_accel_id, accel_id);
+	} else {
+		wb = (u64)remote->rptr_cpu_addr;
+		npa = remote->rptr_npa;
+		npa |= (u64)accel_id << AMDGPU_UALINK_GART_NPA_ADDR_GPUID_SHIFT;
+	}
+
+	if (type == RB_TYPE_TLB_INV)
+		offset = ualink_tlb_wb_offset(adev, remote_accel_id);
+	else if (type == RB_TYPE_REMOTE_INTERRUPT)
+		offset = ualink_wb_offset(adev, remote_accel_id);
+
+	*wb_cpu = (struct amdgpu_ualink_wb *)(wb + offset);
+	if (wb_npa)
+		*wb_npa = npa + offset;
+
+	dev_dbg(adev->dev, "source %d remote %d wb npa 0x%llx\n", accel_id,
+		remote_accel_id, npa + offset);
+}
+
+static int amdgpu_ualink_update_wb_address(struct amdgpu_device *adev,
+					   u32 remote_accel_id, u32 ring_type)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ualink_ring *ring;
+	struct amdgpu_ualink_wb *wb_cpu;
+	u64 wb_npa;
+	int r;
+
+	peer = &remote->peer[remote_accel_id];
+	if (ring_type == RB_TYPE_REMOTE_INTERRUPT)
+		ring = &peer->interrupt;
+	else if (ring_type == RB_TYPE_TLB_INV)
+		ring = &peer->shootdown;
+	else
+		return -EINVAL;
+
+	amdgpu_ualink_get_wb_addr(adev, remote_accel_id, &wb_cpu, &wb_npa,
+				  ring_type);
+
+	r = amdgpu_ualink_send_command(adev, remote_accel_id, ring, wb_cpu,
+				       amdgpu_ualink_emit_update_wb_addr,
+				       upper_32_bits(wb_npa),
+				       lower_32_bits(wb_npa),
+				       0, 0);
+	if (!r) {
+		ring->wb_cpu = wb_cpu;
+		ring->ready = true;
+	}
+	return r;
+}
+
Signature · 1 line
2.55.0
author maintainer
#40 lore raw

PATCH v2 40/95 drm/amdgpu: Add UALink cross-GPU TLB shootdown and remote interrupt

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Expose the two public send operations for remote TLB invalidation and remote interrupt delivery. Set up per-peer GART entries covering ring buffers, write pointer pages, and doorbells for every active remote GPU.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 244 +++++++++++++++++++++
 1 file changed, 244 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +244 -0
index 831e8e8e45fbf..e749f304efb76 100644
@@ -2769,3 +2769,247 @@ static int amdgpu_ualink_update_wb_address(struct amdgpu_device *adev,
 	return r;
 }
 
+/**
+ * amdgpu_ualink_remote_shootdown - Send a remote TLB invalidation request
+ * @adev: amdgpu device pointer
+ * @remote_accel_id: accelerator ID of the remote GPU to shootdown
+ * @addr: page-aligned address to invalidate
+ * @size_in_pages: number of pages to invalidate, 0 to invalidate entire TLB
+ * @flush_type: type of TLB flush to perform
+ *
+ * Sends a remote TLB shootdown command to the specified peer GPU via the
+ * UALink ring buffer. The address and size are translated using the S-field
+ * encoding before being written to the ring.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_remote_shootdown(struct amdgpu_device *adev,
+				   u32 remote_accel_id, u64 addr,
+				   u32 size_in_pages, u32 flush_type)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ualink_ring *ring;
+	int r;
+
+	peer = &remote->peer[remote_accel_id];
+	ring = &peer->shootdown;
+	if (!ring->ready) {
+		dev_dbg(adev->dev, "accel_id %u ring not ready\n", remote_accel_id);
+		r = amdgpu_ualink_update_wb_address(adev, remote_accel_id,
+						    RB_TYPE_TLB_INV);
+		if (r)
+			return r;
+	}
+
+	r = amdgpu_ualink_send_command(adev, remote_accel_id, ring, ring->wb_cpu,
+				       amdgpu_ualink_emit_shootdown,
+				       flush_type, upper_32_bits(addr),
+				       lower_32_bits(addr), size_in_pages);
+	return r;
+}
+
+/**
+ * amdgpu_ualink_remote_interrupt - Send a remote interrupt to a peer GPU
+ * @adev: amdgpu device pointer
+ * @remote_accel_id: accelerator ID of the remote GPU to interrupt
+ * @dw0: IH context ID dword 0
+ * @dw1: IH context ID dword 1
+ * @dw2: IH context ID dword 2
+ * @dw3: IH context ID dword 3
+ *
+ * Sends a remote interrupt command to the specified peer GPU via the
+ * UALink ring buffer. The four context ID dwords are delivered to the
+ * remote GPU's interrupt handler.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_remote_interrupt(struct amdgpu_device *adev,
+				   u32 remote_accel_id, u32 dw0, u32 dw1,
+				   u32 dw2, u32 dw3)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ualink_ring *ring;
+	int r;
+
+	peer = &remote->peer[remote_accel_id];
+	ring = &peer->interrupt;
+	if (!ring->ready) {
+		dev_dbg(adev->dev, "accel_id %u ring not ready\n", remote_accel_id);
+		r = amdgpu_ualink_update_wb_address(adev, remote_accel_id,
+						    RB_TYPE_REMOTE_INTERRUPT);
+		if (r)
+			return r;
+	}
+
+	r = amdgpu_ualink_send_command(adev, remote_accel_id, ring, ring->wb_cpu,
+				       amdgpu_ualink_emit_interrupt,
+				       dw0, dw1, dw2, dw3);
+	return r;
+}
+
+/**
+ * amdgpu_ualink_peer_remote_init - Initialize remote peer GPU connections
+ * @adev: amdgpu device pointer
+ *
+ * Sets up ring buffers, GART mappings, and control structures for
+ * communicating with remote GPUs in the fabric.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+static int amdgpu_ualink_peer_remote_init(struct amdgpu_device *adev)
+{
+	u32 rb_pages = AMDGPU_UALINK_RB_SIZE >> AMDGPU_GPU_PAGE_SHIFT;
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	struct amdgpu_bo *bo = remote->ring_bo;
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ualink_ring *interrupt;
+	struct amdgpu_ualink_ring *shootdown;
+	u32 src_accel_id = ualink_accel_id(adev);
+	u32 dst_accel_id;
+	u64 npa, flags;
+	int r = 0;
+
+	/* NPA mapping PTE flags VSCT = 0011 */
+	flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, bo->tbo.resource);
+	flags |= AMDGPU_PTE_SNOOPED | AMDGPU_PTE_PRT_GFX12 | AMDGPU_PTE_BUS_ATOMICS;
+	flags &= ~AMDGPU_PTE_VALID;
+	/* PTE.X=0 turn off RPC checks for RBs, wptr and doorbell NPA address */
+	flags &= ~AMDGPU_PTE_EXECUTABLE;
+
+	dev_dbg(adev->dev, "src_accel_id %u gart mapping flags 0x%llx\n",
+		src_accel_id, flags);
+
+	for_each_set_bit(dst_accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+		if (dst_accel_id == src_accel_id)
+			continue;
+
+		dev_dbg(adev->dev, "peer_remote to dst_accel_id %u\n", dst_accel_id);
+
+		peer = &remote->peer[dst_accel_id];
+		mutex_init(&peer->lock);
+
+		interrupt = &peer->interrupt;
+		shootdown = &peer->shootdown;
+
+		/* ring buffer npa gart mapping */
+		npa = amdgpu_ualink_gart_npa_addr(adev, RB_TYPE_REMOTE_INTERRUPT,
+						   src_accel_id, dst_accel_id);
+		r = amdgpu_ualink_gart_map(adev, rb_pages, npa, &interrupt->mm_node_rb, flags);
+		if (r)
+			break;
+		interrupt->rb_npa_gart = adev->gmc.gart_start +
+					 (interrupt->mm_node_rb.start << AMDGPU_GPU_PAGE_SHIFT);
+
+		dev_dbg(adev->dev, "rb npa 0x%llx mapped on gart 0x%llx\n",
+			npa, interrupt->rb_npa_gart);
+
+		/* tlb invalidate ring buffer npa gart mapping */
+		npa = amdgpu_ualink_gart_npa_addr(adev, RB_TYPE_TLB_INV, src_accel_id,
+						  dst_accel_id);
+		r = amdgpu_ualink_gart_map(adev, rb_pages, npa, &shootdown->mm_node_rb, flags);
+		if (r)
+			break;
+		shootdown->rb_npa_gart = adev->gmc.gart_start +
+					 (shootdown->mm_node_rb.start << AMDGPU_GPU_PAGE_SHIFT);
+
+		dev_dbg(adev->dev, "tlb rb npa 0x%llx mapped on gart 0x%llx\n",
+			npa, shootdown->rb_npa_gart);
+
+		/* wptr npa gart mapping */
+		npa = amdgpu_ualink_gart_npa_addr(adev, RB_TYPE_TAILPTR, src_accel_id,
+						  dst_accel_id);
+		r = amdgpu_ualink_gart_map(adev, 1, npa, &interrupt->mm_node_wptr, flags);
+		if (r)
+			break;
+
+		interrupt->wptr_npa_gart = adev->gmc.gart_start +
+					   (interrupt->mm_node_wptr.start << AMDGPU_GPU_PAGE_SHIFT)
+					    + ualink_wptr_offset(adev, src_accel_id);
+		shootdown->wptr_npa_gart = adev->gmc.gart_start +
+					   (interrupt->mm_node_wptr.start << AMDGPU_GPU_PAGE_SHIFT)
+					    + ualink_tlb_wptr_offset(adev, src_accel_id);
+
+		dev_dbg(adev->dev, "rb wptr npa 0x%llx on gart 0x%llx, tlb wptr on gart 0x%llx\n",
+			npa, interrupt->wptr_npa_gart, shootdown->wptr_npa_gart);
+
+		/*
+		 * doorbell npa gart mapping, doorbell NPA address coding
+		 *
+		 * ReqAddr[51:40] = 0xFFF
+		 * ReqAddr[39:30] = GPU accel_id
+		 * ReqAddr[29:12] = 18'h00000
+		 * ReqAddr[11:8]  = Random value
+		 * ReqAddr[7:6]   = PMI Index
+		 *     0 - PMI0 Remote Interrupt
+		 *     1 - PMI1 Remote TLB Shootdown
+		 * ReqAddr[5:0]   = 6'h0
+		 */
+		npa = 0xFFFUL << 40 | (u64)dst_accel_id << 30;
+		r = amdgpu_ualink_gart_map(adev, 1, npa, &interrupt->mm_node_doorbell, flags);
+		if (r)
+			break;
+
+		interrupt->doorbell_npa_gart = adev->gmc.gart_start +
+			(interrupt->mm_node_doorbell.start << AMDGPU_GPU_PAGE_SHIFT);
+
+		dev_dbg(adev->dev, "rb doorbell npa 0x%llx mapped on gart 0x%llx\n",
+			npa, interrupt->doorbell_npa_gart);
+
+		shootdown->doorbell_npa_gart = adev->gmc.gart_start+
+			(interrupt->mm_node_doorbell.start << AMDGPU_GPU_PAGE_SHIFT) + (1 << 6);
+
+		dev_dbg(adev->dev, "tlb doorbell npa 0x%llx mapped on gart 0x%llx\n",
+			npa, shootdown->doorbell_npa_gart);
+
+		interrupt->rb_size = AMDGPU_UALINK_RB_SIZE / 64;
+		interrupt->wptr = 0;
+		interrupt->rptr = 0;
+		interrupt->seq = 0;
+
+		shootdown->rb_size = AMDGPU_UALINK_RB_SIZE / 64;
+		shootdown->wptr = 0;
+		shootdown->rptr = 0;
+		shootdown->seq = 0;
+	}
+
+	dev_dbg(adev->dev, "peer remote init done r=%d\n", r);
+	return r;
+}
+
+/**
+ * amdgpu_ualink_peer_remote_fini - Clean up remote peer GPU connections
+ * @adev: amdgpu device pointer
+ *
+ * Unmaps GART entries and frees resources for remote GPU communication.
+ */
+static void amdgpu_ualink_peer_remote_fini(struct amdgpu_device *adev)
+{
+	u32 rb_pages = AMDGPU_UALINK_RB_SIZE >> AMDGPU_GPU_PAGE_SHIFT;
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+	u32 src_accel_id = ualink_accel_id(adev);
+	u32 dst_accel_id;
+	struct amdgpu_ualink_peer *peer;
+	struct amdgpu_ualink_ring *ring;
+
+	dev_dbg(adev->dev, "src_accel_id %u\n", src_accel_id);
+
+	for_each_set_bit(dst_accel_id, remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
+		dev_dbg(adev->dev, "dst_accel_id %u\n", dst_accel_id);
+		if (dst_accel_id == src_accel_id)
+			continue;
+
+		peer = &remote->peer[dst_accel_id];
+		mutex_destroy(&peer->lock);
+
+		ring = &peer->interrupt;
+		amdgpu_ualink_gart_unmap(adev, rb_pages, &ring->mm_node_rb);
+		amdgpu_ualink_gart_unmap(adev, 1, &ring->mm_node_wptr);
+		amdgpu_ualink_gart_unmap(adev, 1, &ring->mm_node_doorbell);
+
+		ring = &peer->shootdown;
+		amdgpu_ualink_gart_unmap(adev, rb_pages, &ring->mm_node_rb);
+	}
+}
+
Signature · 1 line
2.55.0
author maintainer
#41 lore raw

PATCH v2 41/95 drm/amdgpu: Add UALink software init, teardown, and reset

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Sequence the setup steps into a single init path with rollback on failure, and a teardown that halts firmware first. Hook both paths into the virtual pod commit so resources rebuild on reconfiguration.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 96 ++++++++++++++++++++++
 1 file changed, 96 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +96 -0
index e749f304efb76..d1e8f51a88b2d 100644
@@ -705,6 +705,13 @@ static void activate_accelerator(struct amdgpu_device *adev)
 		return;
 	}
 
+	r = amdgpu_ualink_sw_init(adev);
+	if (r) {
+		dev_err(adev->dev, "Failed to init UALink sw r=%d\n", r);
+		amdgpu_ualink_manager_stop(adev);
+		return;
+	}
+
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
 }
 
@@ -719,6 +726,7 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 	/* ignore return value */
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_CONFIGURED;
 
+	amdgpu_ualink_sw_fini(adev);
 	amdgpu_ualink_manager_stop(adev);
 }
 
@@ -3013,3 +3021,91 @@ static void amdgpu_ualink_peer_remote_fini(struct amdgpu_device *adev)
 	}
 }
 
+/**
+ * amdgpu_ualink_sw_init - Initialize UALink software resources
+ * @adev: amdgpu device pointer
+ *
+ * Performs full UALink software initialization including metadata allocation,
+ * NPA address mapping, SDMA scheduler entity setup, and remote peer GPU
+ * connection initialization. On failure, all previously initialized resources
+ * are cleaned up.
+ *
+ * Return: 0 on success, negative error code on failure
+ */
+int amdgpu_ualink_sw_init(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote;
+	int r;
+
+	remote = kzalloc(sizeof(*remote), GFP_KERNEL);
+	if (!remote)
+		return -ENOMEM;
+	adev->ualink.remote = remote;
+
+	r = amdgpu_ualink_metadata_init(adev);
+	if (r)
+		goto out;
+
+	r = amdgpu_ualink_metadata_npa_mapping(adev);
+	if (r)
+		goto out_metadata_fini;
+
+	r = amdgpu_ualink_sdma_entities_init(adev);
+	if (r)
+		goto out_npa_unmap;
+
+	r = amdgpu_ualink_peer_remote_init(adev);
+	if (r)
+		goto out_sdma_entities_fini;
+
+	dev_dbg(adev->dev, "ualink sw init succeed\n");
+	return 0;
+
+out_sdma_entities_fini:
+	amdgpu_ualink_sdma_entities_fini(adev);
+out_npa_unmap:
+	amdgpu_ualink_metadata_npa_unmapping(adev);
+out_metadata_fini:
+	amdgpu_ualink_metadata_fini(adev);
+out:
+	kfree(adev->ualink.remote);
+	adev->ualink.remote = NULL;
+	dev_dbg(adev->dev, "ualink sw init failed %d\n", r);
+	return r;
+}
+
+/**
+ * amdgpu_ualink_sw_fini - Tear down UALink software resources
+ * @adev: amdgpu device pointer
+ *
+ * Cleans up all UALink software resources in reverse order of initialization:
+ * remote peer connections, SDMA entities, NPA mappings, and metadata.
+ */
+void amdgpu_ualink_sw_fini(struct amdgpu_device *adev)
+{
+	u32 status;
+	int i;
+
+	dev_dbg(adev->dev, "halt accel_id %u addr_mode %d\n", ualink_accel_id(adev),
+		ualink_addr_mode(adev));
+
+	amdgpu_ualink_mailbox_write(adev, mmMPNHT_SMN_C2PMSG_22_ALT_2,
+				    AMDGPU_UALINK_FW_CMD_HALT_OPERATION);
+
+	for (i = 0; i < 2000; i++) {
+		status = amdgpu_ualink_mailbox_read(adev, mmMPNHT_SMN_C2PMSG_26_ALT_2);
+		if (status == AMDGPU_UALINK_FW_STATUS_HALT)
+			break;
+		mdelay(1);
+	}
+	if (status != AMDGPU_UALINK_FW_STATUS_HALT)
+		dev_warn(adev->dev, "f/w halt failed status 0x%x\n", status);
+
+	amdgpu_ualink_peer_remote_fini(adev);
+	amdgpu_ualink_sdma_entities_fini(adev);
+	amdgpu_ualink_metadata_npa_unmapping(adev);
+	amdgpu_ualink_metadata_fini(adev);
+	kfree(adev->ualink.remote);
+	adev->ualink.remote = NULL;
+}
+
Signature · 1 line
2.55.0
author maintainer
#42 lore raw

PATCH v2 42/95 drm/amdgpu: Add UALink IH ring and enable interrupt

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Register the UALink IRQ source with the IH subsystem. Hardware IH entries are re-queued onto a dedicated soft ring and drained by a work queue handler. Add the soft ring, worker, and delegate helper to the IRQ layer.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    | 17 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h    |  7 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 82 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/ih_v7_0.c       |  8 +++
 4 files changed, 112 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +17 -0
index 91f1ca95ea543..dcf90a974326d 100644
@@ -264,6 +264,14 @@ static void amdgpu_irq_handle_ih_psp(struct work_struct *work)
 
 	amdgpu_ih_process(adev, &adev->irq.ih_psp);
 }
+	
+static void amdgpu_irq_handle_ih_ualink(struct work_struct *work)
+{
+	struct amdgpu_device *adev = container_of(work, struct amdgpu_device,
+						  irq.ih_ualink_work);
+
+	amdgpu_ih_process(adev, &adev->irq.ih_ualink);
+}
 
 /**
  * amdgpu_msi_ok - check whether MSI functionality is enabled
@@ -341,6 +349,7 @@ int amdgpu_irq_init(struct amdgpu_device *adev)
 	INIT_WORK(&adev->irq.ih2_work, amdgpu_irq_handle_ih2);
 	INIT_WORK(&adev->irq.ih_soft_work, amdgpu_irq_handle_ih_soft);
 	INIT_WORK(&adev->irq.ih_psp_work, amdgpu_irq_handle_ih_psp);
+	INIT_WORK(&adev->irq.ih_ualink_work, amdgpu_irq_handle_ih_ualink);
 
 	/* Use vector 0 for MSI-X. */
 	r = pci_irq_vector(adev->pdev, 0);
@@ -566,6 +575,14 @@ void amdgpu_irq_psp_delegate(struct amdgpu_device *adev,
 	schedule_work(&adev->irq.ih_psp_work);
 }
 
+void amdgpu_irq_ualink_delegate(struct amdgpu_device *adev,
+				struct amdgpu_iv_entry *entry,
+				unsigned int num_dw)
+{
+	amdgpu_ih_ring_write(adev, &adev->irq.ih_ualink, entry->iv_entry, num_dw);
+	schedule_work(&adev->irq.ih_ualink_work);
+}
+
 /**
  * amdgpu_irq_update - update hardware interrupt state
  *
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h +5 -2
index deee64b69885d..c88431471c303 100644
@@ -90,10 +90,10 @@ struct amdgpu_irq {
 	bool				msi_enabled; /* msi enabled */
 
 	/* interrupt rings */
-	struct amdgpu_ih_ring		ih, ih1, ih2, ih_soft, ih_psp;
+	struct amdgpu_ih_ring		ih, ih1, ih2, ih_soft, ih_psp, ih_ualink;
 	const struct amdgpu_ih_funcs    *ih_funcs;
 	struct work_struct		ih1_work, ih2_work, ih_soft_work,
-					ih_psp_work;
+					ih_psp_work, ih_ualink_work;
 	struct amdgpu_irq_src		self_irq;
 
 	/* gen irq stuff */
@@ -138,6 +138,9 @@ void amdgpu_irq_delegate(struct amdgpu_device *adev,
 void amdgpu_irq_psp_delegate(struct amdgpu_device *adev,
 			     struct amdgpu_iv_entry *entry,
 			     unsigned int num_dw);
+void amdgpu_irq_ualink_delegate(struct amdgpu_device *adev,
+				 struct amdgpu_iv_entry *entry,
+				 unsigned int num_dw);
 int amdgpu_irq_update(struct amdgpu_device *adev, struct amdgpu_irq_src *src,
 		      unsigned type);
 int amdgpu_irq_get(struct amdgpu_device *adev, struct amdgpu_irq_src *src,
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +82 -0

91 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/ih_v7_0.c +8 -0
index 58d5156a44dbc..c93ef937c4309 100644
@@ -424,6 +424,9 @@ static int ih_v7_0_irq_init(struct amdgpu_device *adev)
 	if (adev->irq.ih_psp.ring_size)
 		adev->irq.ih_psp.enabled = true;
 
+	if (adev->irq.ih_ualink.ring_size)
+		adev->irq.ih_ualink.enabled = true;
+
 	return 0;
 }
 
@@ -639,6 +642,11 @@ static int ih_v7_0_sw_init(struct amdgpu_ip_block *ip_block)
 	if (r)
 		return r;
 
+	dev_dbg(adev->dev, "ualink init ih_ualink\n");
+	r = amdgpu_ih_ring_init(adev, &adev->irq.ih_ualink, sw_ring_size, true);
+	if (r)
+		return r;
+
 	r = amdgpu_irq_init(adev);
 
 	return r;
Signature · 1 line
2.55.0
author maintainer
#43 lore raw

PATCH v2 43/95 drm/amdgpu: UALink use LSDMA to send remote interrupt command

In reply to Alex Deucher

From: Philip Yang <[email protected]>

Add a use_lsdma flag to amdgpu_ualink_remote and set it during SDMA entity initialization. When use_lsdma is set, use amdgpu_lsdma_copy_mem() instead of SDMA IBs to write the remote command ring data, wptr and doorbell to their respective NPA GART addresses.

This provides an alternative low-latency path for sending UALink remote interrupt commands via LSDMA, avoiding SDMA ring/IB overhead for these small NPA writes.

v2: fix 32 bit build (Alex)

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +20 -0
index 2d2b1b1145cd4..d424c71905ff2 100644
@@ -1400,6 +1400,9 @@ struct amdgpu_ualink_remote {
 
 	/* remote GPUs ring buffer, read, write pointer local copy and gart mapping */
 	struct amdgpu_ualink_peer	peer[AMDGPU_UALINK_ACCEL_MAX];
+
+	/* use lsdma write to NPA address for remote interrupt */
+	bool			       use_lsdma;
 };
 
 static inline struct amdgpu_ualink_remote *to_remote(struct amdgpu_device *adev)
@@ -2073,6 +2076,8 @@ static int amdgpu_ualink_sdma_entities_init(struct amdgpu_device *adev)
 		i++;
 	}
 
+	remote->use_lsdma = true;
+
 	dev_dbg(adev->dev, "exit\n");
 	return 0;
 
@@ -2656,6 +2661,20 @@ static int amdgpu_ualink_send_command(struct amdgpu_device *adev,
 	dev_dbg(adev->dev, "dxs_port 0x%x, doorbell_npa_gart 0x%llx -> 0x%llx\n",
 		peer->dxs_port, ring->doorbell_npa_gart, doorbell_npa_gart);
 
+	dev_dbg(adev->dev, "use %s\n", remote->use_lsdma ? "lsdma" : "sdma");
+
+	if (remote->use_lsdma) {
+		div_u64_rem(ring->wptr, ring->rb_size, &rem);
+		r = amdgpu_lsdma_copy_mem(adev, src,
+					  ring->rb_npa_gart + rem * 64,
+					  64);
+		r = amdgpu_lsdma_copy_mem(adev, src + 64, ring->wptr_npa_gart, 8);
+
+		r = amdgpu_lsdma_copy_mem(adev, src + 72, doorbell_npa_gart, 4);
+		amdgpu_job_free(job);
+		goto out_wait_complete;
+        }
+
 	sdma_ring = &adev->sdma.instance[0].ring;
 
 	/*
@@ -2693,6 +2712,7 @@ static int amdgpu_ualink_send_command(struct amdgpu_device *adev,
 		return r;
 	}
 
+out_wait_complete:
 	/*
 	 * poll remote completion writeback data
 	 */
Signature · 1 line
2.55.0
author maintainer
#44 lore raw

PATCH v2 44/95 drm/amdgpu: Create a drm client for UALink NPA BOs

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Create a drm client for storing the GEM handles corresponding to NPA BOs. We always want to keep one GEM handle alive for the NPA BOs during the lifetime of the BO.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 28 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  3 +++
 2 files changed, 31 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +28 -0
index d424c71905ff2..51078e83e3e1a 100644
@@ -31,6 +31,27 @@
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
+static const struct drm_client_funcs ualink_client_funcs = {
+	.unregister	= drm_client_release,
+};
+
+static int amdgpu_ualink_drm_client_create(struct amdgpu_device *adev)
+{
+	int ret;
+
+	ret = drm_client_init(&adev->ddev, &adev->ualink.client, "ualink",
+			      &ualink_client_funcs);
+	if (ret) {
+		dev_err(adev->dev, "Failed to init UALink DRM client: %d\n",
+			ret);
+		return ret;
+	}
+
+	drm_client_register(&adev->ualink.client);
+
+	return 0;
+}
+
 int amdgpu_ualink_init(struct amdgpu_device *adev)
 {
 	int r;
@@ -62,6 +83,13 @@ int amdgpu_ualink_init(struct amdgpu_device *adev)
 		return r;
 	}
 
+	r = amdgpu_ualink_drm_client_create(adev);
+	if (r) {
+		dev_err(adev->dev, "Failed to create UALink DRM client: %d\n",
+			r);
+		return r;
+	}
+
 	r = amdgpu_ualink_init_interrupt(adev);
 	if (r) {
 		dev_err(adev->dev,
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index 3fde1b80c3e92..cb971ac8bdd9e 100644
@@ -186,6 +186,9 @@ struct amdgpu_ualink_mgr {
 	/* NPA-VM used on the exporter.*/
 	struct amdgpu_vm npa_vm;
 
+	/* DRM client for UALink to manage GEM handles for NPA BOs */
+	struct drm_client_dev client;
+
 	/* Sequence number to track the need for TLB flushes */
 	atomic64_t last_flushed_tlb_seq;
 };
Signature · 1 line
2.55.0
author maintainer
#45 lore raw

PATCH v2 45/95 drm/amdgpu: Control NPA DMA-buf importing

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

NPA DMA-Bufs can only be imported on the same device as the original BO. In source-identification mode NPA addresses are importer-specific. Each GPU needs to import the UALink handle by themselves. Sharing the NPA addresses between devices is meaningless and potentially dangerous. Put controls to ensure this is not violated.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c +5 -0
index b33c300e26e2e..b4fbd8a407f15 100644
@@ -558,6 +558,11 @@ struct drm_gem_object *amdgpu_gem_prime_import(struct drm_device *dev,
 			 */
 			drm_gem_object_get(obj);
 			return obj;
+		} else {
+			struct ttm_resource *mem = gem_to_amdgpu_bo(obj)->tbo.resource;
+			/* NPA DMA-buf can only be imported on the same device. */
+			if (mem && mem->mem_type == AMDGPU_PL_NPA)
+				return ERR_PTR(-EPERM);
 		}
 	}
 
Signature · 1 line
2.55.0
author maintainer
#46 lore raw

PATCH v2 46/95 drm/amdgpu: Add ualink handle to BOs

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add a ualink handle to the amdgpu_bo struct. The ualink handle is a 128-bit unique identifier associated with a BO and is shared across the network to facilitate NPA based memory sharing across GPUs in a rack scale setup. We are only storing the low 64 bits of that handle in the BO struct as that is gauranteed to be unique.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +3 -0
index b41767c7913ac..2e6a8e6f577a7 100644
@@ -125,6 +125,9 @@ struct amdgpu_bo {
 	 * for memory accounting.
 	 */
 	int8_t				xcp_id;
+
+	/* UALink Handle low 64 bits*/
+	u64				ualink_handle_lo;
 };
 
 struct amdgpu_bo_user {
Signature · 1 line
2.55.0
author maintainer
#47 lore raw

PATCH v2 47/95 drm/amdgpu: Implement UALink handle export

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

This patch adds the implementation for exporting a ualink handle associated with a BO. If the BO doesn't have a ualink handle generated yet, we first allocate a unique ualink handle, add it to a lookup table and return the handle to user-space.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 131 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  67 +++++++++++
 2 files changed, 198 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +131 -0
index 51078e83e3e1a..fb698c15af909 100644
@@ -24,6 +24,7 @@
 
 #include <linux/xarray.h>
 #include <drm/drm_mm.h>
+#include <linux/random.h>
 #include "amdgpu.h"
 #include "amdgpu_ualink.h"
 #include "amdgpu_xgmi.h"
@@ -1221,6 +1222,136 @@ static void amdgpu_ualink_npa_mm_fini(struct amdgpu_device *adev)
 	drm_mm_takedown(&adev->ualink.npa_mm.mm);
 }
 
+/* The caller of this function is expected to hold the XA lock when calling
+ * this function.
+ */
+static void amdgpu_generate_ualink_handle(struct amdgpu_device *adev,
+				   struct amdgpu_ualink_handle *handle)
+{
+	bool unique;
+
+	do {
+		handle->handle_lo = get_random_u64();
+		/* Replace bottom 10 bits in handle_lo with accId */
+		handle->handle_lo &= ~AMDGPU_UALINK_HANDLE_ACCID_MASK;
+		handle->handle_lo |= adev->ualink.info->ppod.accel_id;
+
+		/* Don't generate/store a Handle with value 0. */
+		if (!handle->handle_lo)
+			continue;
+
+		/* Find if the handle already exists in the exporter xarray.
+		 * If it already exists, then regenerate the handle since we
+		 * want the handle to be unique.
+		 */
+		unique = !xa_load(&adev->ualink.exp_xa, handle->handle_lo);
+	} while (!unique);
+
+	handle->handle_hi = get_random_u64();
+	dev_dbg(adev->dev, "GENERATE-HANDLE: generated handle: %llx:%llx\n",
+		handle->handle_hi, handle->handle_lo);
+}
+
+static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
+{
+}
+
+int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
+				u32 gem_handle,
+				struct amdgpu_ualink_handle *handle_out)
+{
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	struct amdgpu_ualink_handle handle;
+	struct drm_gem_object *gobj;
+	struct amdgpu_device *adev;
+	struct amdgpu_bo *robj;
+	int r = 0, i;
+
+	gobj = drm_gem_object_lookup(filp, gem_handle);
+	if (!gobj)
+		return -ENOENT;
+
+	robj = gem_to_amdgpu_bo(gobj);
+	adev = amdgpu_ttm_adev(robj->tbo.bdev);
+
+	if (!(robj->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM)) {
+		dev_err(adev->dev, "Only VRAM BOs can be exported\n");
+		r = -EOPNOTSUPP;
+		goto out;
+	}
+
+	if (!robj->ualink_handle_lo) {
+		/* If no ualink handle generated for BO, then generate one and
+		 * add it to the exporter Xarray.
+		 */
+		exp_xa_node = kzalloc(sizeof(*exp_xa_node), GFP_KERNEL);
+		if (!exp_xa_node) {
+			dev_err(adev->dev, "Failed to allocate exp_xa_node\n");
+			r = -ENOMEM;
+			goto out;
+		}
+
+		amdgpu_bo_ref(robj);
+		exp_xa_node->bo = robj;
+		init_completion(&exp_xa_node->npa_done);
+		bitmap_zero(exp_xa_node->importers_bitmap,
+			    AMDGPU_UALINK_ACCEL_MAX);
+		bitmap_zero(exp_xa_node->npa_release_bitmap,
+			AMDGPU_UALINK_ACCEL_MAX);
+		kref_init(&exp_xa_node->refcount);
+		mutex_init(&exp_xa_node->node_lock);
+		INIT_WORK(&exp_xa_node->cleanup_work,
+			  amdgpu_ualink_exp_cleanup_worker);
+		for (i = 0; i < AMDGPU_UALINK_ACCEL_MAX; i++) {
+			INIT_LIST_HEAD(&exp_xa_node->importer_entries[i].list);
+			exp_xa_node->importer_entries[i].parent = exp_xa_node;
+		}
+		/* DMABuf handle for local import of fabric handles */
+		exp_xa_node->dmabuf = drm_gem_prime_handle_to_dmabuf(&adev->ddev, filp,
+						gem_handle, DRM_CLOEXEC | DRM_RDWR);
+		if (IS_ERR(exp_xa_node->dmabuf)) {
+			r = PTR_ERR(exp_xa_node->dmabuf);
+			dev_err(adev->dev, "Failed to generate DMABuf for the BO\n");
+			kfree(exp_xa_node);
+			goto out;
+		}
+
+		xa_lock(&adev->ualink.exp_xa);
+		amdgpu_generate_ualink_handle(adev, &handle);
+		exp_xa_node->handle = handle;
+		r = __xa_insert(&adev->ualink.exp_xa, handle.handle_lo,
+				exp_xa_node, GFP_KERNEL);
+		xa_unlock(&adev->ualink.exp_xa);
+		if (r) {
+			dev_err(adev->dev, "Failed to insert exp_xa_node into XA: %d\n", r);
+			dma_buf_put(exp_xa_node->dmabuf);
+			amdgpu_bo_unref(&robj);
+			kfree(exp_xa_node);
+			goto out;
+		}
+
+		robj->ualink_handle_lo = handle.handle_lo;
+		/* Return the generated handle back to the caller */
+		*handle_out = handle;
+	} else {
+		handle_out->handle_lo = robj->ualink_handle_lo;
+
+		/* Do a sanity check to ensure the handle exists in the XA */
+		xa_lock(&adev->ualink.exp_xa);
+		exp_xa_node = xa_load(&adev->ualink.exp_xa,
+				      robj->ualink_handle_lo);
+		xa_unlock(&adev->ualink.exp_xa);
+		WARN(!exp_xa_node, "Exp XA: Handle_Lo: %llx not found",
+		     robj->ualink_handle_lo);
+		if (exp_xa_node)
+			handle_out->handle_hi = exp_xa_node->handle.handle_hi;
+	}
+
+out:
+	drm_gem_object_put(gobj);
+	return r;
+}
+
 int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 {
 	int i, r;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +67 -0

89 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#48 lore raw

PATCH v2 48/95 drm/amdgpu: Add connection state management

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add functions for managing connection state between GPUs before import/export of ualink handles occurs. These functions handle the lifecycle of connections between GPUs,including handling GPU resets, in the NPA-based memory sharing model.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 227 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  13 +-
 2 files changed, 239 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +227 -0
index fb698c15af909..45f17ce514493 100644
@@ -32,6 +32,10 @@
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
+static int amdgpu_ualink_remote_interrupt(struct amdgpu_device *adev,
+				u32 remote_accel_id, u32 dw0, u32 dw1,
+				u32 dw2, u32 dw3);
+
 static const struct drm_client_funcs ualink_client_funcs = {
 	.unregister	= drm_client_release,
 };
@@ -1252,6 +1256,229 @@ static void amdgpu_generate_ualink_handle(struct amdgpu_device *adev,
 		handle->handle_hi, handle->handle_lo);
 }
 
+static int amdgpu_ualink_send_hello_ack_msg(struct amdgpu_device *adev,
+					    u32 remote_acc_id)
+{
+	dev_dbg(adev->dev, "SEND HELLO-ACK: HELLO-ACK message to remote AccId:%u\n",
+		remote_acc_id);
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id,
+		AMDGPU_UALINK_HELLO_ACK_MSG,
+		0, 0, 0);
+}
+
+static int amdgpu_ualink_send_hello_msg(struct amdgpu_device *adev,
+					u32 remote_acc_id)
+{
+	u32 dw0;
+
+	dw0 = AMDGPU_UALINK_HELLO_MSG;
+	dw0 |= (remote_acc_id << AMDGPU_UALINK_HELLO_MSG_RECV_ACCID_SHIFT);
+	dw0 |= (adev->ualink.info->ppod.accel_id <<
+		AMDGPU_UALINK_HELLO_MSG_SENDER_ACCID_SHIFT);
+
+	dev_dbg(adev->dev, "SEND HELLO: HELLO message to remote AccId:%u\n",
+		remote_acc_id);
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, 0,
+					      0, 0);
+}
+
+static u32 amdgpu_ualink_check_conn_ready(struct amdgpu_device *adev,
+					  u32 remote_acc_id, u32 gen_count)
+{
+	struct amdgpu_ualink_connection *conn_state;
+	u32 current_gen_count = 0;
+
+	conn_state = &adev->ualink.conn_state[remote_acc_id];
+
+	/* Check if the connection is established. */
+	mutex_lock(&conn_state->lock);
+	if ((conn_state->state == AMDGPU_UALINK_CONN_ESTABLISHED) &&
+	    (!gen_count || conn_state->generation_count == gen_count))
+		current_gen_count = conn_state->generation_count;
+	mutex_unlock(&conn_state->lock);
+
+	return current_gen_count;
+}
+
+static void amdgpu_ualink_process_hello_ack_msg(struct amdgpu_device *adev,
+					       u32 sender_acc_id)
+{
+	struct amdgpu_ualink_connection *conn_state;
+
+	if (sender_acc_id >= AMDGPU_UALINK_ACCEL_MAX) {
+		dev_err(adev->dev,
+			"HELLO-ACK: sender AccId out of range:%u\n",
+			sender_acc_id);
+		return;
+	}
+
+	conn_state = &adev->ualink.conn_state[sender_acc_id];
+
+	/* If we are in IN_PROGRESS state, then transition the connection
+	 * state to established and signal that the HELLO ACK is received.
+	 * Otherwise, ignore the HELLO ACK.
+	 */
+	mutex_lock(&conn_state->lock);
+
+	if (conn_state->state == AMDGPU_UALINK_CONN_IN_PROGRESS) {
+		conn_state->state = AMDGPU_UALINK_CONN_ESTABLISHED;
+		conn_state->generation_count++;
+		complete(&conn_state->hello_done);
+	} else {
+		dev_dbg(adev->dev,
+			"HELLO-ACK: already connected, ignoring from AccId:%u\n",
+			sender_acc_id);
+	}
+	mutex_unlock(&conn_state->lock);
+}
+
+static void amdgpu_ualink_process_hello_msg(struct amdgpu_device *adev,
+					   u32 receiver_acc_id,
+					   u32 sender_acc_id,
+					   u32 src_acc_id)
+{
+	struct amdgpu_ualink_connection *conn_state;
+	int r;
+
+	if (receiver_acc_id != adev->ualink.info->ppod.accel_id) {
+		dev_err(adev->dev,
+			"HELLO: receiver AccId mismatch got:%u self:%u\n",
+			receiver_acc_id, adev->ualink.info->ppod.accel_id);
+		return;
+	}
+
+	/* src_acc_id is the AccId received in IH cookie. Confirm it
+	 * matches with the sender AccId.
+	 */
+	if (sender_acc_id != src_acc_id) {
+		dev_err(adev->dev,
+			"HELLO: sender AccId mismatch sender:%u IH cookie:%u\n",
+			sender_acc_id, src_acc_id);
+		return;
+	}
+
+	if (sender_acc_id >= AMDGPU_UALINK_ACCEL_MAX) {
+		dev_err(adev->dev,
+			"HELLO: sender AccId out of range:%u\n",
+			sender_acc_id);
+		return;
+	}
+
+	conn_state = &adev->ualink.conn_state[sender_acc_id];
+
+	/* Check if connection is already established. If yes, then receiving HELLO msg
+	 * triggers a reset handling scenario.
+	 * If the connection is not ready, and we receive a HELLO msg, then
+	 * transition the state to PENDING.
+	 * Otherwise, leave it IN_PROGRESS.
+	 */
+	mutex_lock(&conn_state->lock);
+	if (conn_state->state != AMDGPU_UALINK_CONN_ESTABLISHED) {
+		if (conn_state->state == AMDGPU_UALINK_CONN_NOT_READY)
+			conn_state->state = AMDGPU_UALINK_CONN_PENDING;
+		/* otherwise, leave it IN_PROGRESS to signal the completion below */
+		mutex_unlock(&conn_state->lock);
+	} else {
+		/* Set the connection state back to In Progress and revoke
+		 * all exports and release all imports corresponding to the
+		 * sender GPU. Added in later patches.
+		 */
+		conn_state->state = AMDGPU_UALINK_CONN_PENDING;
+		mutex_unlock(&conn_state->lock);
+	}
+
+	r = amdgpu_ualink_send_hello_ack_msg(adev, sender_acc_id);
+	if (r)
+		dev_err(adev->dev, "HELLO-ACK: send failed to remote AccId:%u\n",
+			sender_acc_id);
+
+	mutex_lock(&conn_state->lock);
+	if (r) {
+		conn_state->state = AMDGPU_UALINK_CONN_NOT_READY;
+	} else {
+		/* If we are in IN_PROGRESS state and we receivied the HELLO message,
+		 * upon receiving the HELLO message, transition the state to ESTABLISHED
+		 * and signal the completion.
+		 */
+		if (conn_state->state == AMDGPU_UALINK_CONN_IN_PROGRESS)
+			complete(&conn_state->hello_done);
+		conn_state->state = AMDGPU_UALINK_CONN_ESTABLISHED;
+		conn_state->generation_count++;
+	}
+	mutex_unlock(&conn_state->lock);
+}
+
+static int amdgpu_ualink_setup_connection(struct amdgpu_device *adev,
+					  u32 remote_acc_id)
+{
+	struct amdgpu_ualink_connection *conn_state;
+	int r;
+
+	conn_state = &adev->ualink.conn_state[remote_acc_id];
+
+	/* Connection state management goes through different states.
+	 * The states are:
+	 * - NOT_READY: The connection is not ready.
+	 * - IN_PROGRESS: GPU sent HELLO message and is waiting for the HELLO_ACK.
+	 * - PENDING: GPU received HELLO message and is in the process of sending
+	 *            the HELLO_ACK.
+	 * - ESTABLISHED: The connection is established.
+	 */
+
+	/* Grab the lock and check if connection establishment was
+	 * already done by another thread.
+	 */
+	mutex_lock(&conn_state->lock);
+	if (conn_state->state == AMDGPU_UALINK_CONN_ESTABLISHED) {
+		r = 0;
+		goto out;
+	} else if (conn_state->state == AMDGPU_UALINK_CONN_IN_PROGRESS ||
+		   conn_state->state == AMDGPU_UALINK_CONN_PENDING) {
+		r = -EAGAIN;
+		goto out;
+	}
+
+	conn_state->state = AMDGPU_UALINK_CONN_IN_PROGRESS;
+	mutex_unlock(&conn_state->lock);
+
+	/* Send HELLO message */
+	r = amdgpu_ualink_send_hello_msg(adev, remote_acc_id);
+	if (r) {
+		dev_warn(adev->dev,
+			"HELLO: Send failed to remote AccId:%u\n",
+			remote_acc_id);
+		goto reset_state;
+	}
+
+	/* Wait for the HELLO_ACK to come back */
+	/* complete(conn_state->hello_done) should be called from the IRQ
+	 * handler when the HELLO_ACK is received.
+	 */
+	r = wait_for_completion_interruptible_timeout(&conn_state->hello_done,
+				msecs_to_jiffies(AMDGPU_UALINK_RESP_TIMEOUT));
+	if (r == -ERESTARTSYS) {
+		dev_err_ratelimited(adev->dev,
+			"HELLO-ACK: interrupted by signal\n");
+		goto reset_state;
+	} else if (r == 0) {
+		dev_warn(adev->dev,
+			"HELLO-ACK: Timeout from remote AccId:%u\n",
+			remote_acc_id);
+		r = -ETIMEDOUT;
+		goto reset_state;
+	}
+
+	return 0;
+
+reset_state:
+	mutex_lock(&conn_state->lock);
+	conn_state->state = AMDGPU_UALINK_CONN_NOT_READY;
+out:
+	mutex_unlock(&conn_state->lock);
+
+	return r;
+}
+
 static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 {
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +12 -1
index 4efb0ece4d113..2ea01874554f3 100644
@@ -35,12 +35,23 @@ struct amdgpu_device;
 #define AMDGPU_UALINK_LOCAL_ACCELS_MAX 8
 #define AMDGPU_UALINK_STATIONS_MAX 64
 
+#define AMDGPU_UALINK_RESP_TIMEOUT			5000 /* 5s timeout */
+
 #define AMDGPU_UALINK_HANDLE_ACCID_MASK			GENMASK_ULL(9, 0)
+#define AMDGPU_UALINK_HELLO_MSG_RECV_ACCID_SHIFT	10
+#define AMDGPU_UALINK_HELLO_MSG_SENDER_ACCID_SHIFT	20
+
+enum AMDGPU_UALINK_PROTOCOL_MESSAGES {
+	AMDGPU_UALINK_HELLO_MSG				= 1,
+	AMDGPU_UALINK_HELLO_ACK_MSG			= 2,
+	AMDGPU_UALINK_MAX_PROTOCOL_MSG
+};
 
 enum amdgpu_ualink_conn_state {
 	AMDGPU_UALINK_CONN_NOT_READY			= 0,
 	AMDGPU_UALINK_CONN_IN_PROGRESS			= 1,
-	AMDGPU_UALINK_CONN_ESTABLISHED			= 2
+	AMDGPU_UALINK_CONN_PENDING			= 2,
+	AMDGPU_UALINK_CONN_ESTABLISHED			= 3
 };
 
 enum amdgpu_ualink_type {
Signature · 1 line
2.55.0
author maintainer
#49 lore raw

PATCH v2 49/95 drm/amdgpu: Implement UALink handle import ioctl

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

The ualink handle import process involves NPA protocol message exchange between the exporting and the importing GPUs in the rack scale setup. The process is as follows:
1. On the importing GPU, check if connection is already setup with the exporting GPU. If not, then exchange HELLO/HELLO_ACK messages to setup the connection.
2. Once the connection is setup, then exchange NPA-REQ/NPA-RSP messages with the exporter GPU to get the NPA address and size of the BO associated with the ualink handle.
3. On the exporter GPU, upon receipt of NPA-REQ message, do basic validation to check the ualink handle is valid. If not, send a NPA-FAIL message back. If its valid, then using the address allocator, allocate a NPA address, map it into the NPA VM and send back the address and size in the NPA-RSP to the importing GPU.
4. On the importer GPU, upon receipt of NPA-RSP message, allocate a NPA BO at the NPA address received in the NPA-RSP message.
5. Finally, generate the corresponding DMABuf for the NPA BO and return the dmabuf to user-space.
6. We are using refcount to keep track of the importer/exporter xarray entries. The cleanup functions for these are added in the subsequent patches.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 900 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  73 ++
 2 files changed, 973 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +900 -0
index 45f17ce514493..ad993ab50b5c3 100644
@@ -35,6 +35,20 @@
 static int amdgpu_ualink_remote_interrupt(struct amdgpu_device *adev,
 				u32 remote_accel_id, u32 dw0, u32 dw1,
 				u32 dw2, u32 dw3);
+static void amdgpu_ualink_flush_tlb(struct amdgpu_device *adev,
+				    u32 flush_type);
+static int amdgpu_ualink_reserve_npa_vm_and_bos(struct amdgpu_device *adev,
+						struct amdgpu_bo *bos[], u32 n_bos,
+						struct drm_exec *exec,
+						bool interruptible);
+static void amdgpu_ualink_unreserve_npa_vm_and_bos(struct amdgpu_device *adev,
+						   struct drm_exec *exec);
+#define STRIP_NPA(addr)						\
+	(((u64)(addr) & ~AMDGPU_UALINK_NPA_ADDR_GPUID_MASK))
+
+#define GENERATE_NPA(addr, remote_acc_id)			\
+		((u64)(((u64)(addr)) |				\
+		 ((u64)(remote_acc_id) << AMDGPU_UALINK_NPA_ADDR_GPUID_SHIFT)))
 
 static const struct drm_client_funcs ualink_client_funcs = {
 	.unregister	= drm_client_release,
@@ -1256,6 +1270,222 @@ static void amdgpu_generate_ualink_handle(struct amdgpu_device *adev,
 		handle->handle_hi, handle->handle_lo);
 }
 
+static void amdgpu_ualink_cleanup_exp_xa_node(struct kref *ref)
+{
+}
+
+static void amdgpu_ualink_cleanup_imp_xa_node(struct kref *ref)
+{
+}
+
+static int amdgpu_ualink_exp_xa_entry_get(struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	return kref_get_unless_zero(&exp_xa_node->refcount);
+}
+
+static void amdgpu_ualink_exp_xa_entry_put(struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	kref_put(&exp_xa_node->refcount, amdgpu_ualink_cleanup_exp_xa_node);
+}
+
+static int amdgpu_ualink_imp_xa_entry_get(struct amdgpu_ualink_imp_xa_node *imp_xa_node)
+{
+	return kref_get_unless_zero(&imp_xa_node->refcount);
+}
+
+static void amdgpu_ualink_imp_xa_entry_put(struct amdgpu_ualink_imp_xa_node *imp_xa_node)
+{
+	kref_put(&imp_xa_node->refcount, amdgpu_ualink_cleanup_imp_xa_node);
+}
+
+static int amdgpu_ualink_send_npa_fail_msg(struct amdgpu_device *adev,
+				    u32 remote_acc_id,
+				    struct amdgpu_ualink_handle handle,
+				    u32 fail_reason)
+{
+	u32 dw0, dw1, dw2, dw3;
+
+	dw0 = lower_32_bits(handle.handle_lo);
+	dw0 &= ~AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	dw0 |= AMDGPU_UALINK_NPA_FAIL_MSG;
+
+	dw1 = upper_32_bits(handle.handle_lo);
+	dw2 = fail_reason & 0xFF;
+	dw3 = 0;
+
+	dev_dbg(adev->dev, "SEND NPA-FAIL: remote_acc_id %u handle 0x%llx:%llx dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+		remote_acc_id, handle.handle_hi, handle.handle_lo, dw0, dw1, dw2, dw3);
+
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, dw1,
+					      dw2, dw3);
+}
+
+static int amdgpu_ualink_send_npa_rsp_msg(struct amdgpu_device *adev,
+					  u32 remote_acc_id,
+					  struct amdgpu_ualink_handle handle,
+					  u32 npa_addr, u32 size)
+{
+	u32 dw0, dw1, dw2, dw3;
+
+	dw0 = lower_32_bits(handle.handle_lo);
+	dw0 &= ~AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	dw0 |= AMDGPU_UALINK_NPA_RSP_MSG;
+
+	dw1 = upper_32_bits(handle.handle_lo);
+	dw2 = size;
+	dw3 = npa_addr;
+
+	dev_dbg(adev->dev, "SEND NPA-RSP: remote_acc_id %u handle %llx:%llx dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+		remote_acc_id, handle.handle_hi, handle.handle_lo, dw0, dw1, dw2, dw3);
+
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, dw1,
+					      dw2, dw3);
+}
+
+static int amdgpu_ualink_send_npa_req_msg(struct amdgpu_device *adev,
+					  u32 remote_acc_id,
+					  struct amdgpu_ualink_handle handle)
+{
+	u32 dw0, dw1, dw2, dw3;
+
+	dw0 =  lower_32_bits(handle.handle_lo);
+	dw0 &= ~AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	dw0 |= AMDGPU_UALINK_NPA_REQ_MSG;
+
+	dw1 = upper_32_bits(handle.handle_lo);
+	dw2 = lower_32_bits(handle.handle_hi);
+	dw3 = upper_32_bits(handle.handle_hi);
+
+	dev_dbg(adev->dev, "SEND NPA-REQ: remote_acc_id %u handle 0x%llx:%llx dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+		remote_acc_id, handle.handle_hi, handle.handle_lo, dw0, dw1, dw2, dw3);
+
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, dw1,
+					      dw2, dw3);
+}
+
+static int amdgpu_ualink_send_tlb_shootdown(struct amdgpu_device *adev,
+					    u32 remote_acc_id)
+{
+	return 0;
+}
+
+static u64 amdgpu_ualink_get_export_pte_flags(struct amdgpu_device *adev,
+				       struct amdgpu_bo *bo,
+				       u64 mapping_flags)
+{
+	u64 pte_flags = adev->gmc.init_pte_flags;
+
+	pte_flags |= (AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE |
+		      AMDGPU_PTE_WRITEABLE);
+	mapping_flags |= AMDGPU_VM_MTYPE_DEFAULT;
+
+	amdgpu_gmc_get_vm_pte(adev, &adev->ualink.npa_vm, bo, mapping_flags,
+			      &pte_flags);
+
+	return pte_flags;
+}
+
+static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
+					struct amdgpu_bo *bo,
+					u64 npa_addr, u64 size)
+{
+	uint64_t pte_value = adev->gmc.noretry_flags;
+	struct amdgpu_bo *bos[] = { bo };
+	struct dma_fence *fence;
+	struct drm_exec exec;
+	int r;
+
+	amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+
+	r = amdgpu_vm_update_range(adev, &adev->ualink.npa_vm, false, false, true,
+				false, NULL, npa_addr, npa_addr + size - 1,
+				pte_value, 0, 0, NULL, NULL, &fence);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed to unmap NPA addr (%llx) from NPA VM\n", npa_addr);
+		goto out;
+	}
+
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed %d to update page directories during unmapping NPA: 0x%llx\n",
+			r, npa_addr);
+		goto out;
+	}
+
+	if (fence) {
+		r = dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+		fence = NULL;
+		if (r)
+			goto out;
+	}
+
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+out:
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+
+	return r;
+}
+
+static int amdgpu_ualink_map_npa_addr(struct amdgpu_device *adev, u64 npa_addr,
+				u64 size, struct amdgpu_bo *bo, u64 offset,
+				u64 pte_flags)
+{
+	struct amdgpu_vm *vm = &adev->ualink.npa_vm;
+	struct amdgpu_bo *bos[] = { bo };
+	struct dma_fence *fence = NULL;
+	struct drm_exec exec;
+	int r;
+
+	amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+
+	r = amdgpu_vm_update_range(adev, vm, false, false, true,
+				false, NULL, npa_addr, npa_addr + size - 1,
+				pte_flags, offset, adev->vm_manager.vram_base_offset,
+				bo->tbo.resource, NULL, &vm->last_update);
+	if (r) {
+		dev_warn(adev->dev,
+			"Failed to map NPA addr (%llx) into NPA VM\n", npa_addr);
+		amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+		goto out;
+	}
+
+	r = amdgpu_vm_update_pdes(adev, vm, false);
+	if (r) {
+		dev_err(adev->dev,
+			"failed %d to update page directories for NPA: 0x%llx\n",
+			r, npa_addr);
+		amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+		goto unmap_npa;
+	}
+
+	fence = dma_fence_get(vm->last_update);
+	if (fence) {
+		r = dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+		fence = NULL;
+		if (r) {
+			pr_debug("failed %d to dma fence wait\n", r);
+			amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+			goto unmap_npa;
+		}
+	}
+
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+
+	/* TLB flush may be needed after updated page directories */
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+
+	return 0;
+
+unmap_npa:
+	amdgpu_ualink_unmap_npa_addr(adev, bo, npa_addr, size);
+out:
+	return r;
+}
+
 static int amdgpu_ualink_send_hello_ack_msg(struct amdgpu_device *adev,
 					    u32 remote_acc_id)
 {
@@ -1483,6 +1713,676 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 {
 }
 
+static int amdgpu_ualink_map_npa_to_dmabuf(struct amdgpu_device *adev,
+				struct amdgpu_ualink_imp_xa_node *imp_xa_node)
+{
+	u64 alloc_flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS, npa_addr, size;
+	struct ttm_operation_ctx ctx = { false, false };
+	u32 initial_domain = AMDGPU_GEM_DOMAIN_CPU;
+	struct drm_gem_object *gobj = NULL;
+	struct dma_buf *dmabuf;
+	struct amdgpu_bo *bo;
+	u32 handle;
+	int r;
+
+	npa_addr = imp_xa_node->npa_addr;
+	size = imp_xa_node->size;
+
+	dev_dbg(adev->dev, "Create NPA BO addr 0x%llx size in pages 0x%llx\n",
+		npa_addr, size);
+
+	/* TODO: Check if this needs to be on a xcp_id basis */
+	r = amdgpu_gem_object_create(adev, size * AMDGPU_GPU_PAGE_SIZE, 1,
+				     initial_domain, alloc_flags,
+				     ttm_bo_type_device, NULL, &gobj, 0);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed to create NPA BO in CPU domain. ret %d\n", r);
+		return r;
+	}
+
+	bo = gem_to_amdgpu_bo(gobj);
+	amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_NPA);
+
+	bo->placements[0].fpfn = npa_addr;
+	bo->placements[0].lpfn = npa_addr + size;
+
+	r = amdgpu_bo_reserve(bo, false);
+	if (unlikely(r != 0)) {
+		dev_err(adev->dev, "Failed to reserve NPA BO, r: %d\n", r);
+		goto err_reserve_failed;
+	}
+
+	r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
+	amdgpu_bo_unreserve(bo);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed to validate BO in NPA domain, r: %d\n", r);
+		goto err_validate_failed;
+	}
+
+	r = drm_gem_handle_create(adev->ualink.client.file, gobj, &handle);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed to get handle for NPA GEM object, r: %d\n", r);
+		goto err_validate_failed;
+	}
+	drm_gem_object_put(gobj);
+
+	dmabuf = drm_gem_prime_handle_to_dmabuf(&adev->ddev, adev->ualink.client.file,
+						handle, DRM_CLOEXEC | DRM_RDWR);
+	if (IS_ERR(dmabuf)) {
+		r = PTR_ERR(dmabuf);
+		dev_err(adev->dev,
+			"Failed to generate DMABuf for NPA GEM object\n");
+		goto err_dmabuf_failed;
+	}
+
+	imp_xa_node->dmabuf = dmabuf;
+	imp_xa_node->gem_handle = handle;
+
+	return 0;
+
+err_dmabuf_failed:
+	drm_gem_handle_delete(adev->ualink.client.file, handle);
+	return r;
+err_validate_failed:
+err_reserve_failed:
+	drm_gem_object_put(gobj);
+
+	return r;
+}
+
+static void amdgpu_ualink_process_npa_fail_msg(struct amdgpu_device *adev,
+				       u32 remote_acc_id, u64 partial_handle,
+				       u32 fail_reason)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	int r = 0;
+
+	if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id, 0)) {
+		dev_warn(adev->dev,
+			"NPA-FAIL: no connection with remote AccId:%u\n",
+			remote_acc_id);
+		goto conn_setup;
+	}
+
+	xa_lock(&adev->ualink.imp_xa);
+	imp_xa_node = xa_load(&adev->ualink.imp_xa, partial_handle);
+	if (!imp_xa_node) {
+		xa_unlock(&adev->ualink.imp_xa);
+		dev_warn(adev->dev,
+			"NPA-FAIL: imp XA handle not found:%llx\n",
+			partial_handle);
+		return;
+	}
+
+	imp_xa_node->fail_reason = fail_reason;
+	/* Signal completion done to signal response received for NPA-REQ
+	 * message.
+	 * If the node is in NOT_READY state, then set the node state to
+	 * PENDING and signal the completion. If the node is not in NOT_READY
+	 * state, then it is an unsolicited NPA-FAIL message and we
+	 * log a debug message.
+	 */
+	if (READ_ONCE(imp_xa_node->node_state) == AMDGPU_UALINK_NODE_NOT_READY) {
+		WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_PENDING);
+		complete(&imp_xa_node->npa_done);
+	} else {
+		dev_dbg(adev->dev,
+			"NPA-FAIL: unsolicited for handle:%llx:%llx from AccId:%u\n",
+			imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo,
+			remote_acc_id);
+	}
+	xa_unlock(&adev->ualink.imp_xa);
+
+	return;
+
+conn_setup:
+	r = amdgpu_ualink_setup_connection(adev, remote_acc_id);
+	if (r)
+		dev_warn(adev->dev,
+			"NPA-FAIL: connection setup failed with remote AccId:%u\n",
+			remote_acc_id);
+}
+
+static void amdgpu_ualink_process_npa_rsp_msg(struct amdgpu_device *adev,
+				      u32 remote_acc_id, u64 partial_handle,
+				      u64 npa_addr, u64 size)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	int r = 0;
+
+	/* Check if the connection is established. If it is not, then start
+	 * connection setup.
+	 */
+	if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id, 0)) {
+		dev_warn(adev->dev,
+			"NPA-RSP: no connection with remote AccId:%u\n",
+			remote_acc_id);
+		goto conn_setup;
+	}
+
+	xa_lock(&adev->ualink.imp_xa);
+	imp_xa_node = xa_load(&adev->ualink.imp_xa, partial_handle);
+	if (!imp_xa_node) {
+		xa_unlock(&adev->ualink.imp_xa);
+		dev_warn(adev->dev,
+			"NPA-RSP: imp XA handle not found:%llx\n", partial_handle);
+		return;
+	}
+
+	/* NPA addr received in NPA-RSP is page aligned and without the remote
+	 * GPU-id in Bits 41-50. Assemble back the NPA address before storing
+	 * it.
+	 */
+	imp_xa_node->npa_addr = GENERATE_NPA(npa_addr, remote_acc_id);
+	/* Size is in number of GPU pages granularity. */
+	imp_xa_node->size = size;
+
+	/* Signal completion done to signal NPA_RSP received.
+	 * If the node is in NOT_READY state, then set the node state to
+	 * PENDING and signal the completion. If the node is not in NOT_READY
+	 * state, then it is an unsolicited NPA-RSP message and we
+	 * log a debug message.
+	 */
+	if (READ_ONCE(imp_xa_node->node_state) == AMDGPU_UALINK_NODE_NOT_READY) {
+		WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_PENDING);
+		complete(&imp_xa_node->npa_done);
+	} else {
+		dev_dbg(adev->dev,
+			"NPA-RSP: unsolicited for handle:%llx:%llx from AccId:%u\n",
+			imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo,
+			remote_acc_id);
+	}
+	xa_unlock(&adev->ualink.imp_xa);
+
+	return;
+
+conn_setup:
+	r = amdgpu_ualink_setup_connection(adev, remote_acc_id);
+	if (r)
+		dev_warn(adev->dev,
+			"NPA-RSP: connection setup failed with remote AccId:%u\n",
+			remote_acc_id);
+}
+
+static void amdgpu_ualink_process_npa_req_msg(struct amdgpu_device *adev,
+				      u32 remote_acc_id,
+				      struct amdgpu_ualink_handle handle)
+{
+	struct amdgpu_ualink_importer_entry *importer_entry, *npa_addr_entry;
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	u64 range_start, range_end, pte_flags;
+	struct drm_mm_node *mm_node = NULL;
+	int r = 0, fail_reason = 0;
+	bool send_npa_fail = true;
+	u64 npa_addr = 0, size;
+	struct amdgpu_bo *bo;
+	u32 gen_count;
+
+	/* Check if the connection is established. If it is not, then start
+	 * connection setup.
+	 */
+	gen_count = amdgpu_ualink_check_conn_ready(adev, remote_acc_id, 0);
+	if (!gen_count) {
+		dev_warn(adev->dev,
+			"NPA-REQ: no connection with remote AccId:%u\n",
+			remote_acc_id);
+		goto conn_setup;
+	}
+
+	/* Check entry exists in Exporter XA. If yes, increase the refcount
+	 * for the node.
+	 */
+	xa_lock(&adev->ualink.exp_xa);
+	exp_xa_node = xa_load(&adev->ualink.exp_xa, handle.handle_lo);
+	if (!exp_xa_node  || (handle.handle_hi != exp_xa_node->handle.handle_hi) ||
+	    !amdgpu_ualink_exp_xa_entry_get(exp_xa_node)) {
+		xa_unlock(&adev->ualink.exp_xa);
+		dev_warn(adev->dev,
+			"NPA-REQ: exp XA handle not found handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_INVALID_HANDLE;
+		goto handle_invalid_fail;
+	}
+	xa_unlock(&adev->ualink.exp_xa);
+
+	bo = exp_xa_node->bo;
+	size = amdgpu_bo_ngpu_pages(bo);
+
+	/* Pin the BO */
+	r = amdgpu_bo_reserve(bo, true);
+	if (unlikely(r)) {
+		dev_warn(adev->dev,
+			"NPA-REQ: BO reserve failed handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_ERROR;
+		goto bo_reserve_fail;
+	}
+	r = amdgpu_bo_pin(bo, AMDGPU_GEM_DOMAIN_VRAM);
+	amdgpu_bo_unreserve(bo);
+	if (r) {
+		dev_warn(adev->dev,
+			"NPA-REQ: BO pin failed handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_ERROR;
+		goto bo_pin_fail;
+	}
+
+	if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT) {
+		mutex_lock(&exp_xa_node->node_lock);
+		importer_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		npa_addr_entry = importer_entry;
+		mutex_unlock(&exp_xa_node->node_lock);
+		/* Check if NPA address is already allocated for this importer.
+		 * If yes, then send the NPA-FAIL message back to the remote GPU.
+		 */
+		if (importer_entry->npa_addr) {
+			fail_reason = AMDGPU_UALINK_NPA_FAIL_DUPLICATE;
+			goto npa_duplicate_fail;
+		}
+
+		range_start = ((u64)remote_acc_id << AMDGPU_UALINK_NPA_ADDR_GPUID_SHIFT) |
+				AMDGPU_UALINK_NPA_ADDR_RANGE_RESERVED;
+		range_end = range_start | AMDGPU_UALINK_NPA_ADDR_RANGE_MASK;
+	} else {
+		/* We store NPA-address in importer_entries[0] in
+		 * Source-Aliasing mode.
+		 */
+		mutex_lock(&exp_xa_node->node_lock);
+		npa_addr_entry = &exp_xa_node->importer_entries[0];
+		importer_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		/* Check if NPA address is already allocated for this importer.
+		 * If yes, then set the corresponding bit in the importers_bitmap,
+		 * set the generation count and send the NPA-RSP back to the remote GPU.
+		 */
+		if (npa_addr_entry->npa_addr) {
+			npa_addr = npa_addr_entry->npa_addr;
+			set_bit(remote_acc_id, exp_xa_node->importers_bitmap);
+			importer_entry->generation_count = gen_count;
+			mutex_unlock(&exp_xa_node->node_lock);
+			dev_dbg(adev->dev,
+				"NPA-REQ: NPA:%llx size:%llx handle:%llx:%llx\n",
+				npa_addr, size, handle.handle_hi, handle.handle_lo);
+
+			goto send_npa_rsp;
+		}
+		mutex_unlock(&exp_xa_node->node_lock);
+
+		range_start = 0;
+		range_end = 0;
+	}
+
+	mm_node = kzalloc(sizeof(*mm_node), GFP_KERNEL);
+	if (!mm_node) {
+		dev_warn(adev->dev,
+			"NPA-REQ: mm_node alloc failed handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_NOSPACE;
+		goto mem_alloc_fail;
+	}
+
+	/* Allocate NPA address */
+	r = amdgpu_ualink_npa_alloc_va(adev, mm_node, 0, range_start,
+					range_end, size);
+	if (r) {
+		dev_warn(adev->dev,
+			"NPA-REQ: NPA addr alloc failed handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_NOSPACE;
+		goto npa_alloc_fail;
+	}
+	npa_addr = mm_node->start;
+
+	pte_flags = amdgpu_ualink_get_export_pte_flags(adev, bo, 0);
+	dev_dbg(adev->dev,
+		"NPA-REQ: Allocated NPA:%llx size:%llx PTE:%llx handle:%llx:%llx\n",
+		npa_addr, size, pte_flags, handle.handle_hi, handle.handle_lo);
+
+	/* Map the NPA address into NPA VM*/
+	r = amdgpu_ualink_map_npa_addr(adev, npa_addr, size, bo, 0, pte_flags);
+	if (r) {
+		fail_reason = AMDGPU_UALINK_NPA_FAIL_ERROR;
+		dev_warn(adev->dev,
+			"NPA-REQ: NPA addr (%llx) map failed handle:%llx:%llx\n",
+			npa_addr, handle.handle_hi, handle.handle_lo);
+		goto map_npa_fail;
+	}
+
+	dev_dbg(adev->dev,
+		"NPA-REQ: Mapped NPA:%llx size:%llx pte:%llx handle:%llx:%llx\n",
+		npa_addr, size, pte_flags, handle.handle_hi, handle.handle_lo);
+send_npa_rsp:
+	/* Send NPA-RSP back to the remote GPU */
+	r = amdgpu_ualink_send_npa_rsp_msg(adev, remote_acc_id, handle,
+					   STRIP_NPA(npa_addr), size);
+	if (r) {
+		dev_warn(adev->dev,
+			"NPA-REQ: send NPA-RSP failed remote:%u handle:%llx:%llx\n",
+			remote_acc_id, handle.handle_hi, handle.handle_lo);
+		send_npa_fail = false;
+		goto send_npa_rsp_fail;
+	}
+
+	dev_dbg(adev->dev,
+		"NPA-REQ: Sent NPA-RSP with NPA:%llx size:%llx handle:%llx:%llx\n",
+		npa_addr, size, handle.handle_hi, handle.handle_lo);
+
+	/* If this is the first time we are setting the bit for this importer,
+	 * then store the NPA address, mm_node and generation count.
+	 */
+	mutex_lock(&exp_xa_node->node_lock);
+	if (!test_and_set_bit(remote_acc_id, exp_xa_node->importers_bitmap)) {
+		npa_addr_entry->npa_addr = npa_addr;
+		npa_addr_entry->mm_node = mm_node;
+		importer_entry->generation_count = gen_count;
+	}
+	mutex_unlock(&exp_xa_node->node_lock);
+
+	dev_dbg(adev->dev,
+		"NPA-REQ: BO pin_count:%d, importers:%d, handle:%llx:%llx\n",
+		bo->tbo.pin_count, bitmap_weight(exp_xa_node->importers_bitmap,
+		AMDGPU_UALINK_ACCEL_MAX), handle.handle_hi, handle.handle_lo);
+	WARN_ON(bo->tbo.pin_count < bitmap_weight(exp_xa_node->importers_bitmap,
+						   AMDGPU_UALINK_ACCEL_MAX));
+
+	/* Add this node to the exported handles list for the remote GPU,
+	 * but only if the node is still in exp_xa. If revoke already erased
+	 * it, skip the list_add to avoid a dangling list entry. The cleanup
+	 * worker is guaranteed to run after we drop our ref, so it will see
+	 * this importer in the bitmap and send NPA-REVOKE.
+	 */
+	xa_lock(&adev->ualink.exp_xa);
+	if (xa_load(&adev->ualink.exp_xa, exp_xa_node->handle.handle_lo) == exp_xa_node)
+		list_add(&importer_entry->list, &adev->ualink.exp_handles_list[remote_acc_id]);
+	xa_unlock(&adev->ualink.exp_xa);
+
+	amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+
+	return;
+
+send_npa_rsp_fail:
+	mutex_lock(&exp_xa_node->node_lock);
+	clear_bit(remote_acc_id, exp_xa_node->importers_bitmap);
+	mutex_unlock(&exp_xa_node->node_lock);
+	if (mm_node)
+		amdgpu_ualink_unmap_npa_addr(adev, bo, npa_addr, size);
+
+map_npa_fail:
+	if (mm_node)
+		amdgpu_ualink_npa_free_va(adev, mm_node);
+
+npa_alloc_fail:
+	kfree(mm_node);
+mem_alloc_fail:
+npa_duplicate_fail:
+	r = amdgpu_bo_reserve(bo, true);
+	if (likely(!r)) {
+		amdgpu_bo_unpin(bo);
+		amdgpu_bo_unreserve(bo);
+	} else {
+		dev_warn(adev->dev,
+			"NPA-REQ: BO reserve to unpin failed for handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+	}
+
+bo_pin_fail:
+bo_reserve_fail:
+	amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+
+handle_invalid_fail:
+	if (send_npa_fail) {
+		r = amdgpu_ualink_send_npa_fail_msg(adev, remote_acc_id,
+						    handle, fail_reason);
+		if (r)
+			dev_warn(adev->dev,
+				"NPA-REQ: send NPA-FAIL failed remote:%u handle:%llx:%llx\n",
+				remote_acc_id, handle.handle_hi, handle.handle_lo);
+	}
+	return;
+conn_setup:
+	r = amdgpu_ualink_setup_connection(adev, remote_acc_id);
+	if (r)
+		dev_warn(adev->dev,
+			"NPA-REQ: connection setup failed with remote AccId:%u\n",
+			remote_acc_id);
+}
+
+static int amdgpu_ualink_translate_npa_fail_reason(struct amdgpu_device *adev,
+						   u32 fail_reason)
+{
+	switch (fail_reason) {
+	case AMDGPU_UALINK_NPA_FAIL_NOSPACE:
+		return -ENOSPC;
+	case AMDGPU_UALINK_NPA_FAIL_INVALID_HANDLE:
+	case AMDGPU_UALINK_NPA_FAIL_DUPLICATE:
+	case AMDGPU_UALINK_NPA_FAIL_ERROR:
+		return -EINVAL;
+	default:
+		dev_err(adev->dev,
+			"IMPORT: invalid NPA-FAIL reason:%u\n",
+			fail_reason);
+		return -EINVAL;
+	}
+}
+
+static int amdgpu_ualink_do_import_handle(struct amdgpu_device *adev,
+				struct amdgpu_ualink_imp_xa_node *imp_xa_node,
+				u32 remote_acc_id)
+{
+	struct amdgpu_ualink_handle handle = imp_xa_node->handle;
+	int r;
+
+	/* First check if the connection is setup with the
+	 * remote GPU. If yes, then initiate the NPA protocol to
+	 * get the NPA address.
+	 * If not, then initiate the HELLO protocol to first setup
+	 * the connection and once the connection is setup, then
+	 * initiate the NPA protocol.
+	 */
+	r = amdgpu_ualink_setup_connection(adev, remote_acc_id);
+	if (r) {
+		if (r != -EAGAIN)
+			dev_warn(adev->dev,
+			"IMPORT: connection setup failed with remote AccId:%u\n",
+			remote_acc_id);
+		return r;
+	}
+
+	/* Send NPA_REQ message */
+	r = amdgpu_ualink_send_npa_req_msg(adev, remote_acc_id, handle);
+	if (r) {
+		dev_warn(adev->dev,
+			"IMPORT: NPA-REQ send failed to remote AccId:%u\n",
+			remote_acc_id);
+		return r;
+	}
+
+	/* Wait for the NPA_RSP to come back */
+	r = wait_for_completion_interruptible_timeout(&imp_xa_node->npa_done,
+				msecs_to_jiffies(AMDGPU_UALINK_RESP_TIMEOUT));
+	if (r == -ERESTARTSYS) {
+		dev_err_ratelimited(adev->dev,
+			"IMPORT: NPA-RSP wait interrupted by signal\n");
+		return r;
+	} else if (r == 0) {
+		dev_warn(adev->dev,
+			"IMPORT: NPA-RSP timeout from remote AccId:%u\n",
+			remote_acc_id);
+		return -ETIMEDOUT;
+	}
+
+	/* If the NPA addr/size isn't filled with valid values, then either
+	 * we got a NPA_FAIL or something bad happened. In either case, we
+	 * return the error back to user-space.
+	 */
+	if (imp_xa_node->fail_reason) {
+		dev_warn(adev->dev,
+			"IMPORT: NPA-REQ failed with fail_reason:%d handle:%llx:%llx\n",
+			imp_xa_node->fail_reason, handle.handle_hi, handle.handle_lo);
+		return amdgpu_ualink_translate_npa_fail_reason(adev,
+						imp_xa_node->fail_reason);
+	}
+
+	if (!imp_xa_node->npa_addr || !imp_xa_node->size) {
+		dev_warn(adev->dev,
+			"IMPORT: invalid npa:%llx or size:%llx\n",
+			imp_xa_node->npa_addr, imp_xa_node->size);
+		imp_xa_node->npa_addr = 0;
+		imp_xa_node->size = 0;
+		return -EINVAL;
+	}
+
+	r = amdgpu_ualink_map_npa_to_dmabuf(adev, imp_xa_node);
+	if (r) {
+		dev_warn(adev->dev,
+			"IMPORT: dmabuf creation failed npa:%llx size:%llx\n",
+			imp_xa_node->npa_addr, imp_xa_node->size);
+		imp_xa_node->npa_addr = 0;
+		imp_xa_node->size = 0;
+		return r;
+	}
+
+	/* Add this node to the imported handles list for the remote GPU */
+	xa_lock(&adev->ualink.imp_xa);
+	list_add(&imp_xa_node->list, &adev->ualink.imp_handles_list[remote_acc_id]);
+	xa_unlock(&adev->ualink.imp_xa);
+
+	return 0;
+}
+
+int amdgpu_ualink_import_handle(struct drm_device *dev,
+				const struct amdgpu_ualink_handle *ualink_handle,
+				int *fd_out)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	struct amdgpu_device *adev = drm_to_adev(dev);
+	struct amdgpu_ualink_handle handle = *ualink_handle;
+	u32 remote_acc_id, node_state;
+	int r = 0, fd;
+
+	remote_acc_id = (handle.handle_lo &
+			AMDGPU_UALINK_HANDLE_ACCID_MASK);
+
+	if (remote_acc_id >= AMDGPU_UALINK_ACCEL_MAX) {
+		dev_err(adev->dev,
+			"IMPORT: invalid remote AccId:%u\n", remote_acc_id);
+		return -EINVAL;
+	}
+
+	xa_lock(&adev->ualink.imp_xa);
+	imp_xa_node = xa_load(&adev->ualink.imp_xa, handle.handle_lo);
+
+	if (imp_xa_node) {
+		/* If node state is Not_ready/Pending, then some other
+		 * thread is already trying the NPA protocol for the same
+		 * ualink handle. Back off and let the thread finish.
+		 * If the node state is Teardown, then it means this
+		 * node is about to be removed. So let user-space know
+		 * that this handle is invalid.
+		 */
+		node_state = READ_ONCE(imp_xa_node->node_state);
+		if (node_state == AMDGPU_UALINK_NODE_NOT_READY ||
+		    node_state == AMDGPU_UALINK_NODE_PENDING) {
+			xa_unlock(&adev->ualink.imp_xa);
+			r = -EAGAIN;
+			goto out;
+		} else if (node_state == AMDGPU_UALINK_NODE_TEARDOWN) {
+			xa_unlock(&adev->ualink.imp_xa);
+			r = -EINVAL;
+			goto out;
+		}
+
+		/* Increase the refcount while we are processing the request */
+		r = amdgpu_ualink_imp_xa_entry_get(imp_xa_node) ? 0 : -EINVAL;
+		xa_unlock(&adev->ualink.imp_xa);
+
+		/* If the refcount has become 0 but the entry is not yet
+		 * removed from the Xarray, then return error to user-space.
+		 */
+		if (r)
+			goto out;
+	} else {
+		xa_unlock(&adev->ualink.imp_xa);
+		/* if the partial handle doesn't exist in the Importer xarray then
+		 * initiate the NPA protocol and generate the DMABuf corresponding
+		 * to the NPA address.
+		 * First store the entry in the Xarray.
+		 */
+		imp_xa_node = kzalloc(sizeof(*imp_xa_node), GFP_KERNEL);
+		if (!imp_xa_node) {
+			r = -ENOMEM;
+			goto out;
+		}
+		imp_xa_node->adev = adev;
+		imp_xa_node->node_state = AMDGPU_UALINK_NODE_NOT_READY;
+		imp_xa_node->handle = handle;
+		init_completion(&imp_xa_node->npa_done);
+		kref_init(&imp_xa_node->refcount);
+
+		/* Take an extra reference to store in the Xarray. The error
+		 * handling paths will drop both these references, while a
+		 * successful path will drop only one reference to the Xarray
+		 * entry.
+		 */
+		amdgpu_ualink_imp_xa_entry_get(imp_xa_node);
+
+		/* Check if another thread created a node for the same handle while
+		 * we were trying to create and initialize the node.
+		 */
+		r = xa_insert(&adev->ualink.imp_xa, handle.handle_lo,
+			      imp_xa_node, GFP_KERNEL);
+		if (r) {
+			kfree(imp_xa_node);
+			dev_err(adev->dev,
+				"IMPORT: XA insert failed for handle:%llx:%llx err:%d\n",
+				handle.handle_hi, handle.handle_lo, r);
+			goto out;
+		}
+
+		r = amdgpu_ualink_do_import_handle(adev, imp_xa_node, remote_acc_id);
+
+		/* If error is returned, then cleanup the xarray entry before returning
+		 * the error back to user-space
+		 */
+		if (r) {
+			amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
+			if (r != -EAGAIN)
+				dev_err(adev->dev,
+					"IMPORT: XA import failed for handle:%llx:%llx\n",
+					handle.handle_hi, handle.handle_lo);
+			goto cleanup;
+		} else {
+			WRITE_ONCE(imp_xa_node->node_state,
+				   AMDGPU_UALINK_NODE_READY);
+		}
+	}
+
+	/* dma_buf_fd consumes a reference and assigns it to the fd.
+	 * Therefore take an extra reference to be consumed. It will be
+	 * released when user mode closes the fd.
+	 */
+	get_dma_buf(imp_xa_node->dmabuf);
+
+	fd = dma_buf_fd(imp_xa_node->dmabuf, O_CLOEXEC | O_RDWR);
+	if (fd >= 0) {
+		*fd_out = fd;
+	} else {
+		dma_buf_put(imp_xa_node->dmabuf);
+		r = fd;
+		dev_err(adev->dev,
+			"IMPORT: dma-buf fd creation failed handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+	}
+
+cleanup:
+	amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
+out:
+	return r;
+}
+
 int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
 				u32 gem_handle,
 				struct amdgpu_ualink_handle *handle_out)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +73 -0

95 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#50 lore raw

PATCH v2 50/95 drm/amdgpu: Implement mechanism to revoke exported memory

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

This patch adds implementation to revoke exported memory. The exporter GPU will send a NPA-REVOKE message to inform the importer GPUs to stop using the exported memory. Once the importers stop using the exporter memory, they would send back a NPA-RELEASE message to inform the exporters to free the exporter memory.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 249 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   3 +
 2 files changed, 252 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +249 -0
index ad993ab50b5c3..9d21f08123eed 100644
@@ -1298,6 +1298,48 @@ static void amdgpu_ualink_imp_xa_entry_put(struct amdgpu_ualink_imp_xa_node *imp
 	kref_put(&imp_xa_node->refcount, amdgpu_ualink_cleanup_imp_xa_node);
 }
 
+static int amdgpu_ualink_send_npa_release_msg(struct amdgpu_device *adev,
+				    u32 remote_acc_id,
+				    struct amdgpu_ualink_handle handle)
+{
+	u32 dw0, dw1, dw2, dw3;
+
+	dw0 = lower_32_bits(handle.handle_lo);
+	dw0 &= ~AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	dw0 |= AMDGPU_UALINK_NPA_RELEASE_MSG;
+
+	dw1 = upper_32_bits(handle.handle_lo);
+	dw2 = lower_32_bits(handle.handle_hi);
+	dw3 = upper_32_bits(handle.handle_hi);
+
+	dev_dbg(adev->dev, "SEND NPA-RELEASE: remote_acc_id %u handle %llx:%llx dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+		remote_acc_id, handle.handle_hi, handle.handle_lo, dw0, dw1, dw2, dw3);
+
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, dw1,
+					      dw2, dw3);
+}
+
+static int amdgpu_ualink_send_npa_revoke_msg(struct amdgpu_device *adev,
+					u32 remote_acc_id,
+					struct amdgpu_ualink_handle handle)
+{
+	u32 dw0, dw1, dw2, dw3;
+
+	dw0 = lower_32_bits(handle.handle_lo);
+	dw0 &= ~AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	dw0 |= AMDGPU_UALINK_NPA_REVOKE_MSG;
+
+	dw1 = upper_32_bits(handle.handle_lo);
+	dw2 = lower_32_bits(handle.handle_hi);
+	dw3 = upper_32_bits(handle.handle_hi);
+
+	dev_dbg(adev->dev, "SEND NPA-REVOKE: remote_acc_id %u handle %llx:%llx dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+		remote_acc_id, handle.handle_hi, handle.handle_lo, dw0, dw1, dw2, dw3);
+
+	return amdgpu_ualink_remote_interrupt(adev, remote_acc_id, dw0, dw1,
+					      dw2, dw3);
+}
+
 static int amdgpu_ualink_send_npa_fail_msg(struct amdgpu_device *adev,
 				    u32 remote_acc_id,
 				    struct amdgpu_ualink_handle handle,
@@ -1713,6 +1755,78 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 {
 }
 
+/* This function is a copy of amdgpu_dma_buf_move_notify() function.
+ * amdgpu_dma_buf_move_notify is only called for import attachments.
+ * But NPA DMABufs don't use attachments because they are imported
+ * on the same device. So we need to invalidate the GPUVM mappings
+ * manually.
+ */
+static void amdgpu_ualink_invalidate_import_mappings(struct amdgpu_bo *bo)
+{
+	struct drm_gem_object *obj = &bo->tbo.base;
+	struct ww_acquire_ctx *ticket = dma_resv_locking_ctx(obj->resv);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+	struct ttm_operation_ctx ctx = { false, false };
+	struct ttm_placement placement = {};
+	struct amdgpu_vm_bo_base *bo_base;
+	int r;
+
+	amdgpu_bo_reserve(bo, false);
+
+	/* FIXME: This should be after the "if", but needs a fix to make sure
+	 * DMABuf imports are initialized in the right VM list.
+	 */
+	amdgpu_vm_bo_invalidate(bo, false);
+	if (!bo->tbo.resource || bo->tbo.resource->mem_type == TTM_PL_SYSTEM)
+		goto out;
+
+	r = ttm_bo_validate(&bo->tbo, &placement, &ctx);
+	if (r) {
+		dev_err(adev->dev, "Failed to invalidate NPA DMA-buf import (%d)\n",
+			r);
+		goto out;
+	}
+
+	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
+		struct amdgpu_vm *vm = bo_base->vm;
+		struct dma_resv *resv = vm->root.bo->tbo.base.resv;
+
+		if (ticket) {
+			/* When we get an error here it means that somebody
+			 * else is holding the VM lock and updating page tables
+			 * So we can just continue here.
+			 */
+			r = dma_resv_lock(resv, ticket);
+			if (r)
+				continue;
+
+		} else {
+			/* TODO: This is more problematic and we actually need
+			 * to allow page tables updates without holding the
+			 * lock.
+			 */
+			if (!dma_resv_trylock(resv))
+				continue;
+		}
+
+		/* Reserve fences for two SDMA page table updates */
+		r = dma_resv_reserve_fences(resv, 2);
+		if (!r)
+			r = amdgpu_vm_clear_freed(adev, vm, NULL);
+		if (!r)
+			r = amdgpu_vm_handle_moved(adev, vm, ticket);
+
+		if (r && r != -EBUSY)
+			dev_err(adev->dev, "Failed to invalidate VM page tables (%d))\n",
+				r);
+
+		dma_resv_unlock(resv);
+	}
+
+out:
+	amdgpu_bo_unreserve(bo);
+}
+
 static int amdgpu_ualink_map_npa_to_dmabuf(struct amdgpu_device *adev,
 				struct amdgpu_ualink_imp_xa_node *imp_xa_node)
 {
@@ -1793,6 +1907,141 @@ static int amdgpu_ualink_map_npa_to_dmabuf(struct amdgpu_device *adev,
 	return r;
 }
 
+void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo)
+{
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+	struct amdgpu_ualink_importer_entry *imp_entry;
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	u32 remote_acc_id;
+	int r;
+
+	if (!bo->ualink_handle_lo)
+		return;
+
+	/* Remove the entry from the Xarray. */
+	xa_lock(&adev->ualink.exp_xa);
+	exp_xa_node = __xa_erase(&adev->ualink.exp_xa,
+				 bo->ualink_handle_lo);
+	if (!exp_xa_node) {
+		xa_unlock(&adev->ualink.exp_xa);
+		dev_warn(adev->dev,
+			 "Exp XA: handle_lo:%llx not found\n",
+			 bo->ualink_handle_lo);
+		return;
+	}
+
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX) {
+		imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		list_del_init(&imp_entry->list);
+	}
+	xa_unlock(&adev->ualink.exp_xa);
+
+	/* Add it to the Handle_Invalid xarray */
+	r = xa_err(xa_store(&adev->ualink.handle_invalid_xa,
+			    bo->ualink_handle_lo,
+			    exp_xa_node, GFP_KERNEL));
+	if (r)
+		dev_err(adev->dev,
+			"Handle_Invalid XA store failed handle:%llx:%llx error:%d\n",
+			exp_xa_node->handle.handle_hi,
+			exp_xa_node->handle.handle_lo, r);
+
+	amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+}
+
+static void amdgpu_ualink_process_npa_release_msg(struct amdgpu_device *adev,
+						 u32 remote_acc_id,
+						 struct amdgpu_ualink_handle handle)
+{
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+
+	xa_lock(&adev->ualink.handle_invalid_xa);
+	exp_xa_node = xa_load(&adev->ualink.handle_invalid_xa, handle.handle_lo);
+	if (!exp_xa_node) {
+		dev_warn(adev->dev,
+			 "NPA-RELEASE: Handle (%llx:%llx) not found\n",
+			 handle.handle_hi, handle.handle_lo);
+		goto out;
+	}
+
+	/* Confirm that the complete handle matches. */
+	if (handle.handle_hi != exp_xa_node->handle.handle_hi) {
+		dev_warn(adev->dev,
+			 "NPA-RELEASE: handle_hi mismatch exp:%llx got:%llx:%llx remote:%u\n",
+			 exp_xa_node->handle.handle_hi,
+			 handle.handle_hi, handle.handle_lo, remote_acc_id);
+		goto out;
+	}
+
+	/* Test and clear the bit corresponding to the remote GPU id to signal
+	 * the arrival of NPA_RELEASE message from it.
+	 * If the corresponding bit wasn't set, then raise a warning and ignore
+	 * the NPA-Release message from the remote GPU.
+	 */
+	if (!test_and_clear_bit(remote_acc_id, exp_xa_node->npa_release_bitmap)) {
+		dev_warn(adev->dev,
+			 "NPA-RELEASE: unexpected from remote:%u handle:%llx:%llx\n",
+			 remote_acc_id, handle.handle_hi, handle.handle_lo);
+		goto out;
+	}
+
+	/* Signal completion if NPA_Release received from all importers */
+	if (bitmap_empty(exp_xa_node->npa_release_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX))
+		complete(&exp_xa_node->npa_done);
+
+out:
+	xa_unlock(&adev->ualink.handle_invalid_xa);
+}
+
+static void amdgpu_ualink_process_npa_revoke_msg(struct amdgpu_device *adev,
+						u32 remote_acc_id,
+						struct amdgpu_ualink_handle handle)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	struct amdgpu_bo *bo;
+	int r = 0;
+
+	/* Remove the entry from the Xarray. */
+	xa_lock(&adev->ualink.imp_xa);
+	imp_xa_node = xa_load(&adev->ualink.imp_xa, handle.handle_lo);
+	if (!imp_xa_node) {
+		xa_unlock(&adev->ualink.imp_xa);
+		dev_warn(adev->dev,
+			 "NPA-REVOKE: Handle (%llx:%llx) not found\n",
+			 handle.handle_hi, handle.handle_lo);
+		return;
+	}
+
+	/* Confirm that the complete handle matches. */
+	if (handle.handle_hi != imp_xa_node->handle.handle_hi) {
+		xa_unlock(&adev->ualink.imp_xa);
+		dev_warn(adev->dev,
+			 "NPA-REVOKE: handle_hi mismatch exp:%llx got:%llx:%llx remote:%u\n",
+			 imp_xa_node->handle.handle_hi,
+			 handle.handle_hi, handle.handle_lo, remote_acc_id);
+		return;
+	}
+
+	WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_TEARDOWN);
+	list_del_init(&imp_xa_node->list);
+	xa_unlock(&adev->ualink.imp_xa);
+
+	/* Invalidate the GPUVM mappings */
+	bo = gem_to_amdgpu_bo(imp_xa_node->dmabuf->priv);
+	amdgpu_ualink_invalidate_import_mappings(bo);
+
+	/* Drop the refcount for the node */
+	amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
+
+	r = amdgpu_ualink_send_npa_release_msg(adev, remote_acc_id, handle);
+	if (r)
+		dev_err(adev->dev,
+			"NPA-Release send failed remote:%u handle:%llx:%llx error:%d\n",
+			remote_acc_id, handle.handle_hi, handle.handle_lo, r);
+}
+
 static void amdgpu_ualink_process_npa_fail_msg(struct amdgpu_device *adev,
 				       u32 remote_acc_id, u64 partial_handle,
 				       u32 fail_reason)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index 4a977af9565b9..8aee16ad39e9a 100644
@@ -76,6 +76,8 @@ enum AMDGPU_UALINK_PROTOCOL_MESSAGES {
 	AMDGPU_UALINK_NPA_REQ_MSG			= 3,
 	AMDGPU_UALINK_NPA_RSP_MSG			= 4,
 	AMDGPU_UALINK_NPA_FAIL_MSG			= 5,
+	AMDGPU_UALINK_NPA_REVOKE_MSG			= 6,
+	AMDGPU_UALINK_NPA_RELEASE_MSG			= 7,
 	AMDGPU_UALINK_MAX_PROTOCOL_MSG
 };
 
@@ -354,4 +356,5 @@ int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
 int amdgpu_ualink_import_handle(struct drm_device *dev,
 				const struct amdgpu_ualink_handle *ualink_handle,
 				int *fd_out);
+void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo);
 #endif
Signature · 1 line
2.55.0
author maintainer
#51 lore raw

PATCH v2 51/95 drm/amdgpu: lock UALink import invalidation via drm_exec

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Convert amdgpu_ualink_invalidate_import_mappings() to lock the BO and every client VM root PD in a single drm_exec transaction instead of borrowing the caller's reservation ticket via dma_resv_locking_ctx().

drm_exec owns its ww_acquire_ctx (exec.ticket) for the lifetime of the transaction, so amdgpu_vm_handle_moved() now uses &exec.ticket.

Also, drop the explicit dma_resv_reserve_fences call as amdgpu_vm_lock_pd() already reserves 2 + num_fences on the same VM root PD.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 63 +++++++++++-----------
 1 file changed, 30 insertions(+), 33 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +30 -33
index 9d21f08123eed..82a362e44a3fe 100644
@@ -1763,68 +1763,65 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
  */
 static void amdgpu_ualink_invalidate_import_mappings(struct amdgpu_bo *bo)
 {
-	struct drm_gem_object *obj = &bo->tbo.base;
-	struct ww_acquire_ctx *ticket = dma_resv_locking_ctx(obj->resv);
 	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	struct ttm_operation_ctx ctx = { false, false };
 	struct ttm_placement placement = {};
 	struct amdgpu_vm_bo_base *bo_base;
+	struct drm_exec exec;
 	int r;
 
-	amdgpu_bo_reserve(bo, false);
+	/*
+	 * Lock the BO together with every client VM page directory it is
+	 * mapped into in a single drm_exec transaction.
+	 */
+	drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);
+	drm_exec_until_all_locked(&exec) {
+		r = drm_exec_lock_obj(&exec, &bo->tbo.base);
+		drm_exec_retry_on_contention(&exec);
+		if (unlikely(r))
+			goto fini;
+
+		for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
+			r = amdgpu_vm_lock_pd(bo_base->vm, &exec, 0);
+			drm_exec_retry_on_contention(&exec);
+			if (unlikely(r))
+				goto fini;
+		}
+	}
 
 	/* FIXME: This should be after the "if", but needs a fix to make sure
 	 * DMABuf imports are initialized in the right VM list.
 	 */
 	amdgpu_vm_bo_invalidate(bo, false);
 	if (!bo->tbo.resource || bo->tbo.resource->mem_type == TTM_PL_SYSTEM)
-		goto out;
+		goto fini;
 
 	r = ttm_bo_validate(&bo->tbo, &placement, &ctx);
 	if (r) {
 		dev_err(adev->dev, "Failed to invalidate NPA DMA-buf import (%d)\n",
 			r);
-		goto out;
+		goto fini;
 	}
 
 	for (bo_base = bo->vm_bo; bo_base; bo_base = bo_base->next) {
 		struct amdgpu_vm *vm = bo_base->vm;
-		struct dma_resv *resv = vm->root.bo->tbo.base.resv;
-
-		if (ticket) {
-			/* When we get an error here it means that somebody
-			 * else is holding the VM lock and updating page tables
-			 * So we can just continue here.
-			 */
-			r = dma_resv_lock(resv, ticket);
-			if (r)
-				continue;
-
-		} else {
-			/* TODO: This is more problematic and we actually need
-			 * to allow page tables updates without holding the
-			 * lock.
-			 */
-			if (!dma_resv_trylock(resv))
-				continue;
-		}
 
-		/* Reserve fences for two SDMA page table updates */
-		r = dma_resv_reserve_fences(resv, 2);
-		if (!r)
-			r = amdgpu_vm_clear_freed(adev, vm, NULL);
+		/*
+		 * Fences for the two SDMA page table updates were already
+		 * reserved by amdgpu_vm_lock_pd() above (it reserves 2 +
+		 * num_fences on the same VM root PD dma_resv).
+		 */
+		r = amdgpu_vm_clear_freed(adev, vm, NULL);
 		if (!r)
-			r = amdgpu_vm_handle_moved(adev, vm, ticket);
+			r = amdgpu_vm_handle_moved(adev, vm, &exec.ticket);
 
 		if (r && r != -EBUSY)
 			dev_err(adev->dev, "Failed to invalidate VM page tables (%d))\n",
 				r);
-
-		dma_resv_unlock(resv);
 	}
 
-out:
-	amdgpu_bo_unreserve(bo);
+fini:
+	drm_exec_fini(&exec);
 }
 
 static int amdgpu_ualink_map_npa_to_dmabuf(struct amdgpu_device *adev,
Signature · 1 line
2.55.0
author maintainer
#52 lore raw

PATCH v2 52/95 drm/amdgpu: Cleanup exported UALink handles

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add implementation for freeing up exporter GPU's UALink handles and the corresponding xarray entries. Before freeing, the exporter GPU would send the NPA-REVOKE message to all the importers and wait for the NPA-RELEASE messages to come back before freeing the xarray entry.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 343 +++++++++++++++++++++
 1 file changed, 343 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +343 -0
index 82a362e44a3fe..d87b66d1d51d3 100644
@@ -1272,6 +1272,13 @@ static void amdgpu_generate_ualink_handle(struct amdgpu_device *adev,
 
 static void amdgpu_ualink_cleanup_exp_xa_node(struct kref *ref)
 {
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	struct amdgpu_device *adev;
+
+	exp_xa_node = container_of(ref, struct amdgpu_ualink_exp_xa_node,
+				   refcount);
+	adev = amdgpu_ttm_adev(exp_xa_node->bo->tbo.bdev);
+	queue_work(adev->ualink.npa_wq, &exp_xa_node->cleanup_work);
 }
 
 static void amdgpu_ualink_cleanup_imp_xa_node(struct kref *ref)
@@ -1751,8 +1758,344 @@ static int amdgpu_ualink_setup_connection(struct amdgpu_device *adev,
 	return r;
 }
 
+/* Set PTE.X = 1 for all importer entries to retry RPCs. */
+static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
+				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	struct amdgpu_ualink_importer_entry *imp_entry;
+	struct amdgpu_vm *vm = &adev->ualink.npa_vm;
+	u64 pte_flags, npa_addr, size;
+	struct dma_fence *fence = NULL;
+	struct amdgpu_bo *bo;
+	struct drm_exec exec;
+	u32 remote_acc_id;
+	int r;
+
+	bo = exp_xa_node->bo;
+	size = amdgpu_bo_ngpu_pages(bo);
+
+	amdgpu_ualink_reserve_npa_vm_and_bos(adev, &bo, 1, &exec, false);
+
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX) {
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			imp_entry = &exp_xa_node->importer_entries[0];
+		else
+			imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		npa_addr = imp_entry->npa_addr;
+
+		/* If the connection state changed while we are freeing
+		 * the BO, then ignore this importer. We will unmap this
+		 * address eventually in amdgpu_ualink_unmap_all_npa_addr()
+		 * function. Also, we will clear the corresponding bit in
+		 * the importer_bitmap in the same function.
+		 */
+		if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id,
+					imp_entry->generation_count))
+			continue;
+
+		/* Set PTE.X = 1 */
+		pte_flags = amdgpu_ualink_get_export_pte_flags(adev, bo,
+					AMDGPU_VM_PAGE_EXECUTABLE);
+		dev_dbg(adev->dev,
+			"RETRY-RPC: setting PTE.X=1 for NPA:%llx remote:%u pte:0x%llx\n",
+			npa_addr, remote_acc_id, pte_flags);
+
+		r = amdgpu_vm_update_range(adev, vm, false, false, true,
+					false, NULL, npa_addr, npa_addr + size - 1,
+					pte_flags, 0, adev->vm_manager.vram_base_offset,
+					bo->tbo.resource, NULL, &vm->last_update);
+
+		if (r)
+			dev_warn(adev->dev,
+				"RETRY-RPC: PTE.X update failed for NPA:%llx remote:%u, r: %d\n",
+				npa_addr, remote_acc_id, r);
+
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			break;
+	}
+
+	r = amdgpu_vm_update_pdes(adev, vm, false);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed %d to update page directories during force retry rpcs\n",
+			r);
+		amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+		return;
+	}
+
+	fence = dma_fence_get(vm->last_update);
+	if (fence) {
+		r = dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+		fence = NULL;
+		if (r)
+			dev_dbg(adev->dev, "RETRY-RPC: dma fence wait failed err:%d\n", r);
+	}
+
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+}
+
+/* Unmap all NPA addresses associated with a BO (UALink handle). This function is used
+ * only in Source Identification mode.
+ */
+static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
+				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	struct amdgpu_ualink_importer_entry *imp_entry;
+	u64 pte_value = adev->gmc.noretry_flags;
+	struct dma_fence *fence = NULL;
+	struct drm_exec exec;
+	u64 npa_addr, size;
+	u32 remote_acc_id;
+	int r;
+
+	size = amdgpu_bo_ngpu_pages(exp_xa_node->bo);
+
+	amdgpu_ualink_reserve_npa_vm_and_bos(adev, &exp_xa_node->bo, 1,
+					     &exec, false);
+
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX) {
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			imp_entry = &exp_xa_node->importer_entries[0];
+		else
+			imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		npa_addr = imp_entry->npa_addr;
+
+		if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id,
+					imp_entry->generation_count)) {
+			clear_bit(remote_acc_id,
+				  exp_xa_node->importers_bitmap);
+			continue;
+		}
+
+		dev_dbg(adev->dev,
+			"UNMAP-NPA: Unmapping NPA:%llx, handle:%llx:%llx remote:%u pte:0x%llx\n",
+			npa_addr, exp_xa_node->handle.handle_hi, exp_xa_node->handle.handle_lo,
+			remote_acc_id, pte_value);
+
+		r = amdgpu_vm_update_range(adev, &adev->ualink.npa_vm, false,
+					   false, true, false, NULL, npa_addr,
+					   npa_addr + size - 1, pte_value, 0,
+					   0, NULL, NULL, &fence);
+
+		if (r)
+			dev_err(adev->dev,
+				"UNMAP-NPA: Unmap failed NPA:%llx, handle:%llx:%llx remote:%u\n",
+				npa_addr, exp_xa_node->handle.handle_hi,
+				exp_xa_node->handle.handle_lo, remote_acc_id);
+
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			break;
+	}
+
+	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	if (r) {
+		dev_err(adev->dev,
+			"Failed %d to update page directories during all NPA addresses unmapping\n",
+			r);
+		amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+		return;
+	}
+
+	if (fence) {
+		r = dma_fence_wait(fence, false);
+		dma_fence_put(fence);
+		fence = NULL;
+		if (r)
+			dev_err(adev->dev,
+				"UNMAP-NPA: dma fence wait failed\n");
+	}
+
+	amdgpu_ualink_unreserve_npa_vm_and_bos(adev, &exec);
+
+	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
+}
+
+static void amdgpu_ualink_free_all_npa_va(struct amdgpu_device *adev,
+				struct amdgpu_ualink_exp_xa_node *exp_xa_node,
+				unsigned long *importers_bitmap)
+{
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	struct drm_mm_node *mm_node;
+	u32 remote_acc_id;
+
+	for_each_set_bit(remote_acc_id, importers_bitmap, AMDGPU_UALINK_ACCEL_MAX) {
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			mm_node = exp_xa_node->importer_entries[0].mm_node;
+		else
+			mm_node = exp_xa_node->importer_entries[remote_acc_id].mm_node;
+
+		if (!mm_node) {
+			if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+				break;
+			continue;
+		}
+
+		dev_dbg(adev->dev,
+			"FREE-NPA: freeing NPA address:%llx, handle:%llx:%llx remote:%u\n",
+			mm_node->start, exp_xa_node->handle.handle_hi,
+			exp_xa_node->handle.handle_lo, remote_acc_id);
+
+		amdgpu_ualink_npa_free_va(adev, mm_node);
+		kfree(mm_node);
+
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			break;
+	}
+}
+
 static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 {
+	DECLARE_BITMAP(orig_importers_bitmap, AMDGPU_UALINK_ACCEL_MAX);
+	struct amdgpu_ualink_importer_entry *imp_entry;
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	struct amdgpu_ualink_handle handle;
+	struct amdgpu_device *adev;
+	struct amdgpu_bo *bo;
+	u32 remote_acc_id;
+	int r;
+
+	exp_xa_node = container_of(work, struct amdgpu_ualink_exp_xa_node,
+				   cleanup_work);
+	bo = exp_xa_node->bo;
+	adev = amdgpu_ttm_adev(bo->tbo.bdev);
+	handle = exp_xa_node->handle;
+
+	/* Revoking access to an exported memory follows the steps:
+	 * 1. Set PTE.X = 1 to retry for RPCs.
+	 * 2. Send Remote TLB Shootdowns to all importers.
+	 * 3. Unmap the NPA address from NPA VM.
+	 * 4. Drop the ref count for the BO.
+	 * 5. Send NPA_REVOKE to all importers.
+	 * 6. Wait for NPA_RELEASE from all importers.
+	 * 7. Free the NPA address once all responses are received.
+	 */
+	/* If there are no importers for this BO/handle */
+	if (bitmap_empty(exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX)) {
+		/* Drop the BO reference so it can be freed. */
+		amdgpu_bo_unref(&bo);
+		exp_xa_node->bo = NULL;
+		goto free_node;
+	}
+
+	dev_dbg(adev->dev,
+		"EXP-CLEANUP: handle:%llx:%llx importers bitmap: %*pbl\n",
+		handle.handle_hi, handle.handle_lo,
+		AMDGPU_UALINK_ACCEL_MAX, exp_xa_node->importers_bitmap);
+
+	bitmap_copy(orig_importers_bitmap, exp_xa_node->importers_bitmap,
+		    AMDGPU_UALINK_ACCEL_MAX);
+
+	/* Set PTE.X = 1 for NPA addresses from all importers*/
+	amdgpu_ualink_force_retry_rpcs(adev, exp_xa_node);
+
+	/* Send TLB-shootdown to all importer GPUs */
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX) {
+		dev_dbg(adev->dev,
+			"EXP-CLEANUP: Sending TLB-shootdown to remote:%u\n",
+			remote_acc_id);
+		r = amdgpu_ualink_send_tlb_shootdown(adev, remote_acc_id);
+		if (r)
+			dev_err(adev->dev,
+				"EXP-CLEANUP: TLB shootdown send failed to remote:%u\n",
+				remote_acc_id);
+	}
+
+	/* Unmap all NPA addresses for this BO from NPA VM */
+	amdgpu_ualink_unmap_all_npa_addr(adev, exp_xa_node);
+
+	dev_dbg(adev->dev, "EXP-CLEANUP: handle:%llx:%llx Unpin BO, pin_count:%u, importers:%u\n",
+		handle.handle_hi, handle.handle_lo, bo->tbo.pin_count,
+		bitmap_weight(orig_importers_bitmap, AMDGPU_UALINK_ACCEL_MAX));
+	WARN_ON(bo->tbo.pin_count < bitmap_weight(orig_importers_bitmap,
+						AMDGPU_UALINK_ACCEL_MAX));
+
+	/* Unpin the BO */
+	if (likely(!amdgpu_bo_reserve(bo, true))) {
+		bo->ualink_handle_lo = 0ULL;
+		for_each_set_bit(remote_acc_id, orig_importers_bitmap,
+				 AMDGPU_UALINK_ACCEL_MAX)
+			amdgpu_bo_unpin(bo);
+		amdgpu_bo_unreserve(bo);
+	} else {
+		dev_warn(adev->dev,
+			"EXP-CLEANUP: BO reserve to unpin failed for handle:%llx:%llx\n",
+			handle.handle_hi, handle.handle_lo);
+	}
+
+	/* Free the DMABuf */
+	dma_buf_put(exp_xa_node->dmabuf);
+	/* Drop the reference to the BO so it can be freed. */
+	amdgpu_bo_unref(&bo);
+	exp_xa_node->bo = NULL;
+
+	/* Send NPA-REVOKE to all importers which have imported this memory */
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+				 AMDGPU_UALINK_ACCEL_MAX) {
+		imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id,
+					imp_entry->generation_count)) {
+			clear_bit(remote_acc_id,
+				  exp_xa_node->importers_bitmap);
+			continue;
+		}
+
+		dev_dbg(adev->dev,
+			"EXP-CLEANUP: Sending NPA-REVOKE to remote:%u\n",
+			remote_acc_id);
+		set_bit(remote_acc_id, exp_xa_node->npa_release_bitmap);
+		r = amdgpu_ualink_send_npa_revoke_msg(adev, remote_acc_id, handle);
+		if (r) {
+			dev_err(adev->dev,
+				"EXP-CLEANUP: NPA-REVOKE send failed to remote:%u\n",
+				remote_acc_id);
+			clear_bit(remote_acc_id, exp_xa_node->npa_release_bitmap);
+		}
+	}
+
+	if (!bitmap_empty(exp_xa_node->importers_bitmap,
+			AMDGPU_UALINK_ACCEL_MAX)) {
+		dev_dbg(adev->dev,
+			"EXP-CLEANUP: handle:%llx:%llx NPA-RELEASE bitmap: %*pbl\n",
+			handle.handle_hi, handle.handle_lo,
+			AMDGPU_UALINK_ACCEL_MAX, exp_xa_node->npa_release_bitmap);
+
+		/* Wait for the NPA_RELEASE to come back from all importers */
+		r = wait_for_completion_timeout(&exp_xa_node->npa_done,
+					msecs_to_jiffies(AMDGPU_UALINK_RESP_TIMEOUT));
+
+		if (r == 0)
+			dev_warn(adev->dev,
+				"EXP-CLEANUP: NPA-RELEASE timeout for handle:%llx:%llx\n",
+				handle.handle_hi, handle.handle_lo);
+	}
+
+	/* Free the NPA addresses given to all the importers */
+	amdgpu_ualink_free_all_npa_va(adev, exp_xa_node,
+				      orig_importers_bitmap);
+
+	/* Warn about all importers that didn't respond back with
+	 * NPA-RELEASE message. This will trigger connection timeout
+	 * handling which is added later.
+	 */
+	for_each_set_bit(remote_acc_id, exp_xa_node->npa_release_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX)
+		dev_warn(adev->dev,
+			"EXP-CLEANUP: handle:%llx:%llx NPA-RELEASE timeout from remote:%u\n",
+			handle.handle_hi, handle.handle_lo, remote_acc_id);
+
+free_node:
+	xa_erase(&adev->ualink.handle_invalid_xa, handle.handle_lo);
+	mutex_destroy(&exp_xa_node->node_lock);
+	kfree(exp_xa_node);
 }
 
 /* This function is a copy of amdgpu_dma_buf_move_notify() function.
Signature · 1 line
2.55.0
author maintainer
#53 lore raw

PATCH v2 53/95 drm/amdgpu: Cleanup imported UALink handles

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add implementation for cleaning up ualink handles imported on the importer GPU and the corresponding xarray entries. This includes cleaning up the NPA BO dmabuf and freeing the NPA BO.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 31 ++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +31 -0
index d87b66d1d51d3..9ece8c71c0627 100644
@@ -1283,6 +1283,37 @@ static void amdgpu_ualink_cleanup_exp_xa_node(struct kref *ref)
 
 static void amdgpu_ualink_cleanup_imp_xa_node(struct kref *ref)
 {
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	struct amdgpu_device *adev;
+
+	imp_xa_node = container_of(ref, struct amdgpu_ualink_imp_xa_node,
+				   refcount);
+	adev = imp_xa_node->adev;
+
+	dev_dbg(adev->dev, "IMP-CLEANUP: Remove entry from XA for handle:%llx:%llx\n",
+		imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo);
+
+	/* Remove node from the Xarray */
+	if (!xa_erase(&adev->ualink.imp_xa, imp_xa_node->handle.handle_lo))
+		dev_err(adev->dev, "IMP-CLEANUP: Failed to find entry in XA for handle:%llx:%llx\n",
+			imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo);
+
+	if (imp_xa_node->npa_addr && imp_xa_node->size && imp_xa_node->dmabuf) {
+		dev_dbg(adev->dev,
+			"IMP-CLEANUP: dmabuf free for NPA:%llx handle:%llx:%llx, fc:%lu\n",
+			imp_xa_node->npa_addr, imp_xa_node->handle.handle_hi,
+			imp_xa_node->handle.handle_lo,
+			file_count(imp_xa_node->dmabuf->file));
+		dma_buf_put(imp_xa_node->dmabuf);
+		drm_gem_handle_delete(adev->ualink.client.file,
+				      imp_xa_node->gem_handle);
+
+	}
+
+	dev_dbg(adev->dev, "IMP-CLEANUP: Freeing XA entry for handle:%llx:%llx\n",
+		imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo);
+
+	kfree(imp_xa_node);
 }
 
 static int amdgpu_ualink_exp_xa_entry_get(struct amdgpu_ualink_exp_xa_node *exp_xa_node)
Signature · 1 line
2.55.0
author maintainer
#54 lore raw

PATCH v2 54/95 drm/amdgpu: Handle connection reset

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

This patch adds connection reset handling. There can be two case which signal connection reset:
1. Receiving a HELLO message from a remote GPU, when the connection state is already setup, signals the remote GPU underwent a reset.
2. If no response received for a NPA-REQ/NPA_REVOKE message.

In either of the two case, we cleanup all exported and imported ualink handles exchanged with the remote GPU.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 207 ++++++++++++++++++++-
 1 file changed, 202 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +202 -5
index 9ece8c71c0627..17d64c222c0d9 100644
@@ -43,6 +43,11 @@ static int amdgpu_ualink_reserve_npa_vm_and_bos(struct amdgpu_device *adev,
 						bool interruptible);
 static void amdgpu_ualink_unreserve_npa_vm_and_bos(struct amdgpu_device *adev,
 						   struct drm_exec *exec);
+static void amdgpu_ualink_handle_connection_reset(struct amdgpu_device *adev,
+						  u32 remote_accel_id, u32 state,
+						  u32 generation_count);
+static void amdgpu_ualink_invalidate_import_mappings(struct amdgpu_bo *bo);
+
 #define STRIP_NPA(addr)						\
 	(((u64)(addr) & ~AMDGPU_UALINK_NPA_ADDR_GPUID_MASK))
 
@@ -1648,6 +1653,7 @@ static void amdgpu_ualink_process_hello_msg(struct amdgpu_device *adev,
 					   u32 src_acc_id)
 {
 	struct amdgpu_ualink_connection *conn_state;
+	u32 generation_count;
 	int r;
 
 	if (receiver_acc_id != adev->ualink.info->ppod.accel_id) {
@@ -1691,10 +1697,15 @@ static void amdgpu_ualink_process_hello_msg(struct amdgpu_device *adev,
 	} else {
 		/* Set the connection state back to In Progress and revoke
 		 * all exports and release all imports corresponding to the
-		 * sender GPU. Added in later patches.
+		 * sender GPU.
 		 */
 		conn_state->state = AMDGPU_UALINK_CONN_PENDING;
+		generation_count = conn_state->generation_count;
 		mutex_unlock(&conn_state->lock);
+
+		amdgpu_ualink_handle_connection_reset(adev, sender_acc_id,
+						AMDGPU_UALINK_CONN_PENDING,
+						generation_count);
 	}
 
 	r = amdgpu_ualink_send_hello_ack_msg(adev, sender_acc_id);
@@ -1789,6 +1800,177 @@ static int amdgpu_ualink_setup_connection(struct amdgpu_device *adev,
 	return r;
 }
 
+static void amdgpu_ualink_cleanup_imp_xa_entries(struct amdgpu_device *adev,
+						 u32 remote_acc_id)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	struct list_head *imp_handles_list;
+	struct amdgpu_bo *bo;
+
+	dev_dbg(adev->dev,
+		"IMP-RESET: Cleaning up all XA entries for remote:%u\n",
+		remote_acc_id);
+
+	imp_handles_list = &adev->ualink.imp_handles_list[remote_acc_id];
+
+	xa_lock(&adev->ualink.imp_xa);
+	while (!list_empty(imp_handles_list)) {
+		imp_xa_node = list_first_entry(imp_handles_list,
+					struct amdgpu_ualink_imp_xa_node, list);
+		list_del_init(&imp_xa_node->list);
+		WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_TEARDOWN);
+		xa_unlock(&adev->ualink.imp_xa);
+
+		dev_dbg(adev->dev,
+			"IMP-RESET: remote:%u handle:%llx:%llx npa:%llx size:%llx\n",
+			remote_acc_id, imp_xa_node->handle.handle_hi,
+			imp_xa_node->handle.handle_lo,
+			imp_xa_node->npa_addr, imp_xa_node->size);
+
+		bo = gem_to_amdgpu_bo(imp_xa_node->dmabuf->priv);
+		/* Invalidate the imported mappings */
+		amdgpu_ualink_invalidate_import_mappings(bo);
+
+		/* Drop the refcount for the node */
+		amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
+		xa_lock(&adev->ualink.imp_xa);
+	}
+	xa_unlock(&adev->ualink.imp_xa);
+}
+
+static void amdgpu_ualink_cleanup_exp_xa_entries(struct amdgpu_device *adev,
+						 u32 remote_acc_id)
+{
+	struct amdgpu_ualink_importer_entry *importer_entry;
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	struct list_head *exp_handles_list;
+	struct drm_mm_node *mm_node;
+	u64 npa_addr, size;
+
+	/* Get the list head for the list containing all the handles
+	 * exported to this remote GPU.
+	 */
+	exp_handles_list = &adev->ualink.exp_handles_list[remote_acc_id];
+
+	dev_dbg(adev->dev,
+		"EXP-RESET: Cleaning up all XA entries for remote:%u\n",
+		remote_acc_id);
+
+	xa_lock(&adev->ualink.exp_xa);
+	while (!list_empty(exp_handles_list)) {
+		importer_entry = list_first_entry(exp_handles_list,
+					struct amdgpu_ualink_importer_entry, list);
+		list_del_init(&importer_entry->list);
+
+		exp_xa_node = importer_entry->parent;
+		if (!amdgpu_ualink_exp_xa_entry_get(exp_xa_node))
+			continue;
+
+		xa_unlock(&adev->ualink.exp_xa);
+		/* Clear the bit corresponding to this remote GPU in
+		 * the importer bitmap.
+		 */
+		if (!test_and_clear_bit(remote_acc_id,
+					exp_xa_node->importers_bitmap)) {
+			amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+			xa_lock(&adev->ualink.exp_xa);
+			continue;
+		}
+
+		mutex_lock(&exp_xa_node->node_lock);
+		size = amdgpu_bo_ngpu_pages(exp_xa_node->bo);
+		/* Unpin the BO */
+		if (likely(!amdgpu_bo_reserve(exp_xa_node->bo, true))) {
+			amdgpu_bo_unpin(exp_xa_node->bo);
+			amdgpu_bo_unreserve(exp_xa_node->bo);
+		} else {
+			dev_warn(adev->dev,
+				"EXP-RESET: BO reserve to unpin failed handle:%llx:%llx\n",
+				exp_xa_node->handle.handle_hi, exp_xa_node->handle.handle_lo);
+		}
+
+		dev_dbg(adev->dev,
+			"EXP-RESET: handle:%llx:%llx pin_count:%d, importers:%d\n",
+			exp_xa_node->handle.handle_hi, exp_xa_node->handle.handle_lo,
+			exp_xa_node->bo->tbo.pin_count,
+			bitmap_weight(exp_xa_node->importers_bitmap,
+			AMDGPU_UALINK_ACCEL_MAX));
+
+		WARN_ON(exp_xa_node->bo->tbo.pin_count <
+			bitmap_weight(exp_xa_node->importers_bitmap,
+			AMDGPU_UALINK_ACCEL_MAX));
+
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_IDENT) {
+			npa_addr = importer_entry->npa_addr;
+			mm_node = importer_entry->mm_node;
+			mutex_unlock(&exp_xa_node->node_lock);
+
+			dev_dbg(adev->dev,
+				"EXP-RESET: Unmap NPA:%llx size: %llx remote:%u handle:%llx:%llx\n",
+				npa_addr, size, remote_acc_id,
+				exp_xa_node->handle.handle_hi,
+				exp_xa_node->handle.handle_lo);
+			amdgpu_ualink_unmap_npa_addr(adev, exp_xa_node->bo,
+						     npa_addr, size);
+			amdgpu_ualink_npa_free_va(adev, mm_node);
+			kfree(mm_node);
+
+			/* Reset the NPA addr and mm_node */
+			mutex_lock(&exp_xa_node->node_lock);
+			importer_entry->npa_addr = 0;
+			importer_entry->mm_node = NULL;
+		} else if (bitmap_empty(exp_xa_node->importers_bitmap,
+				AMDGPU_UALINK_ACCEL_MAX)) {
+			/* In Source-Aliasing mode, if there are no importers
+			 * for this handle, then we can unmap and free the
+			 * NPA address.
+			 */
+			mm_node = exp_xa_node->importer_entries[0].mm_node;
+			npa_addr = exp_xa_node->importer_entries[0].npa_addr;
+			mutex_unlock(&exp_xa_node->node_lock);
+
+			dev_dbg(adev->dev,
+				"EXP-RESET: Unmap NPA:%llx size: %llx handle:%llx:%llx\n",
+				npa_addr, size, exp_xa_node->handle.handle_hi,
+				exp_xa_node->handle.handle_lo);
+			amdgpu_ualink_unmap_npa_addr(adev, exp_xa_node->bo,
+						     npa_addr, size);
+			amdgpu_ualink_npa_free_va(adev, mm_node);
+			kfree(mm_node);
+			mutex_lock(&exp_xa_node->node_lock);
+			exp_xa_node->importer_entries[0].npa_addr = 0;
+			exp_xa_node->importer_entries[0].mm_node = NULL;
+		}
+
+		mutex_unlock(&exp_xa_node->node_lock);
+		xa_lock(&adev->ualink.exp_xa);
+		amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+	}
+	xa_unlock(&adev->ualink.exp_xa);
+}
+
+static void amdgpu_ualink_handle_connection_reset(struct amdgpu_device *adev,
+						u32 remote_acc_id, u32 state,
+						u32 generation_count)
+{
+	struct amdgpu_ualink_connection *conn_state;
+
+	conn_state = &adev->ualink.conn_state[remote_acc_id];
+
+	mutex_lock(&conn_state->lock);
+	if ((conn_state->state == AMDGPU_UALINK_CONN_ESTABLISHED) &&
+	     (conn_state->generation_count == generation_count)) {
+		conn_state->state = state;
+		mutex_unlock(&conn_state->lock);
+
+		amdgpu_ualink_cleanup_imp_xa_entries(adev, remote_acc_id);
+		amdgpu_ualink_cleanup_exp_xa_entries(adev, remote_acc_id);
+	} else {
+		mutex_unlock(&conn_state->lock);
+	}
+}
+
 /* Set PTE.X = 1 for all importer entries to retry RPCs. */
 static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
 				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
@@ -2114,14 +2296,18 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 				      orig_importers_bitmap);
 
 	/* Warn about all importers that didn't respond back with
-	 * NPA-RELEASE message. This will trigger connection timeout
-	 * handling which is added later.
+	 * NPA-RELEASE message and trigger connection timeout handling.
 	 */
 	for_each_set_bit(remote_acc_id, exp_xa_node->npa_release_bitmap,
-			 AMDGPU_UALINK_ACCEL_MAX)
+			 AMDGPU_UALINK_ACCEL_MAX) {
 		dev_warn(adev->dev,
 			"EXP-CLEANUP: handle:%llx:%llx NPA-RELEASE timeout from remote:%u\n",
 			handle.handle_hi, handle.handle_lo, remote_acc_id);
+		imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+		amdgpu_ualink_handle_connection_reset(adev, remote_acc_id,
+						AMDGPU_UALINK_CONN_NOT_READY,
+						imp_entry->generation_count);
+	}
 
 free_node:
 	xa_erase(&adev->ualink.handle_invalid_xa, handle.handle_lo);
@@ -2793,6 +2979,7 @@ static int amdgpu_ualink_do_import_handle(struct amdgpu_device *adev,
 				u32 remote_acc_id)
 {
 	struct amdgpu_ualink_handle handle = imp_xa_node->handle;
+	u32 generation_count;
 	int r;
 
 	/* First check if the connection is setup with the
@@ -2831,7 +3018,8 @@ static int amdgpu_ualink_do_import_handle(struct amdgpu_device *adev,
 		dev_warn(adev->dev,
 			"IMPORT: NPA-RSP timeout from remote AccId:%u\n",
 			remote_acc_id);
-		return -ETIMEDOUT;
+		r = -ETIMEDOUT;
+		goto reset_conn;
 	}
 
 	/* If the NPA addr/size isn't filled with valid values, then either
@@ -2871,6 +3059,15 @@ static int amdgpu_ualink_do_import_handle(struct amdgpu_device *adev,
 	xa_unlock(&adev->ualink.imp_xa);
 
 	return 0;
+
+reset_conn:
+	dev_dbg(adev->dev,
+		"IMPORT: Resetting connection for remote:%u\n", remote_acc_id);
+	generation_count = amdgpu_ualink_check_conn_ready(adev, remote_acc_id, 0);
+	amdgpu_ualink_handle_connection_reset(adev, remote_acc_id,
+					      AMDGPU_UALINK_CONN_NOT_READY,
+					      generation_count);
+	return r;
 }
 
 int amdgpu_ualink_import_handle(struct drm_device *dev,
Signature · 1 line
2.55.0
author maintainer
#55 lore raw

PATCH v2 55/95 drm/amdgpu: Setup PTE mappings for NPA addresses

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Map NPA addresses into importer's application page tables using a unique pairing of PTE fields.

---
 .../gpu/drm/amd/amdgpu/amdgpu_res_cursor.h    |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        |  4 ++-
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c        | 30 +++++++++++++++++--
 3 files changed, 32 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_res_cursor.h +1 -0
index 8908d9e08a304..6650b1194c0f2 100644
@@ -92,6 +92,7 @@ static inline void amdgpu_res_first(struct ttm_resource *res,
 	case TTM_PL_TT:
 	case AMDGPU_PL_DOORBELL:
 	case AMDGPU_PL_MMIO_REMAP:
+	case AMDGPU_PL_NPA:
 		node = to_ttm_range_mgr_node(res)->mm_nodes;
 		while (start >= node->size << PAGE_SHIFT)
 			start -= node++->size << PAGE_SHIFT;
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +3 -1
index 4c8341b4fe1bc..fc928c2d8b21e 100644
@@ -1196,7 +1196,9 @@ int amdgpu_vm_update_range(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 		uint64_t tmp, num_entries, addr;
 
 		num_entries = cursor.size >> AMDGPU_GPU_PAGE_SHIFT;
-		if (pages_addr) {
+		if (res && res->mem_type == AMDGPU_PL_NPA) {
+			addr = cursor.start;
+		} else if (pages_addr) {
 			bool contiguous = true;
 
 			if (num_entries > AMDGPU_GPU_PAGES_IN_CPU_PAGE) {
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c +28 -2
index a8e32ee93735d..1468047687652 100644
@@ -600,6 +600,21 @@ static void gmc_v12_1_get_vm_pde(struct amdgpu_device *adev, int level,
 	}
 }
 
+static void gmc_v12_1_get_npa_flags(struct amdgpu_device *adev,
+				    uint64_t *flags)
+{
+	bool is_aid_a1 = (adev->rev_id & 0x10);
+	unsigned int mtype_remote;
+
+	mtype_remote = is_aid_a1 ? MTYPE_NC : MTYPE_UC;
+
+	*flags = AMDGPU_PTE_MTYPE_GFX12(*flags, mtype_remote);
+	/* VSCT = 0011 to identify NPA. Additionally PTE.B = 1 */
+	*flags |= AMDGPU_PTE_SNOOPED | AMDGPU_PTE_PRT_GFX12 |
+		   AMDGPU_PTE_BUS_ATOMICS;
+	*flags &= ~AMDGPU_PTE_VALID;
+}
+
 static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
 					  struct amdgpu_bo *bo,
 					  uint64_t *flags)
@@ -668,6 +683,8 @@ static void gmc_v12_1_get_vm_pte(struct amdgpu_device *adev,
 				 uint32_t vm_flags,
 				 uint64_t *flags)
 {
+	struct ttm_resource *mem;
+
 	if (vm_flags & AMDGPU_VM_PAGE_EXECUTABLE)
 		*flags |= AMDGPU_PTE_EXECUTABLE;
 	else
@@ -689,8 +706,17 @@ static void gmc_v12_1_get_vm_pte(struct amdgpu_device *adev,
 		break;
 	}
 
-	if ((*flags & AMDGPU_PTE_VALID) && bo)
-		gmc_v12_1_get_coherence_flags(adev, bo, flags);
+	if (bo) {
+		mem = bo->tbo.resource;
+		if (mem && mem->mem_type == AMDGPU_PL_NPA) {
+			dev_dbg(adev->dev,
+				"Setting PTE for NPA BO, mem->type: %d, mem->start: %lx, mem->size: %u, cur_flags: %llx\n",
+				mem->mem_type, mem->start, (u32)mem->size, *flags);
+			gmc_v12_1_get_npa_flags(adev, flags);
+		} else if (*flags & AMDGPU_PTE_VALID) {
+			gmc_v12_1_get_coherence_flags(adev, bo, flags);
+		}
+	}
 }
 
 static const struct amdgpu_gmc_funcs gmc_v12_1_gmc_funcs = {
Signature · 1 line
2.55.0
author maintainer
#56 lore raw

PATCH v2 56/95 drm/amdgpu: Add handling for remote interrupts

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add interrupt handling to process the NPA remote interrupts.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 114 ++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   3 +
 2 files changed, 113 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +110 -4
index 17d64c222c0d9..762713194bb6d 100644
@@ -5239,10 +5239,31 @@ void amdgpu_ualink_sw_fini(struct amdgpu_device *adev)
 	adev->ualink.remote = NULL;
 }
 
+int ualink_send_hello(struct amdgpu_device *adev, u32 remote_accel_id)
+{
+	return amdgpu_ualink_send_hello_msg(adev, remote_accel_id);
+}
+
+/*
+ * The low bits of handle_lo carry the message-header field on the wire.
+ * Strip those bits and splice in @acc_id to recover the
+ * fully-qualified handle_lo value.
+ */
+static inline u64 amdgpu_ualink_reassemble_handle(u64 handle_lo, u32 acc_id)
+{
+	return (handle_lo & ~AMDGPU_UALINK_MESSAGE_HEADER_MASK) |
+	       (acc_id & AMDGPU_UALINK_HANDLE_ACCID_MASK);
+}
+
 static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 				     struct amdgpu_irq_src *source,
 				     struct amdgpu_iv_entry *entry)
 {
+	u32 sender_acc_id, receiver_acc_id, msg_type, src_acc_id;
+	struct amdgpu_ualink_handle handle;
+	u32 size, npa_addr, fail_reason;
+	u32 dw0, dw1, dw2, dw3;
+	u32 local_acc_id;
 	int handled = 1;
 
 	dev_dbg(adev->dev, "%s client_id 0x%x src_id 0x%x ih\n",
@@ -5256,10 +5277,95 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 		return handled;
 	}
 
-	/*
-	 * Call amdgpu_ualink_interrupt handler
-	 * amdgpu_ualink_interrupt(adev, entry);
-	 */
+	/* ContextID 4 dwords */
+	src_acc_id = entry->pasid;
+	dw0 = entry->src_data[0];
+	dw1 = entry->src_data[1];
+	dw2 = entry->src_data[2];
+	dw3 = entry->src_data[3];
+
+	dev_dbg(adev->dev, "src accel_id %u context id 0x%x 0x%x 0x%x 0x%x\n",
+		src_acc_id, dw0, dw1, dw2, dw3);
+
+	msg_type = dw0 & AMDGPU_UALINK_MESSAGE_HEADER_MASK;
+	local_acc_id = adev->ualink.info->ppod.accel_id;
+	dev_dbg(adev->dev, "Got MSG: remote acc_id %u msg_type %u\n",
+		src_acc_id, msg_type);
+
+	switch (msg_type) {
+	case AMDGPU_UALINK_HELLO_MSG:
+		receiver_acc_id = (dw0 >> AMDGPU_UALINK_HELLO_MSG_RECV_ACCID_SHIFT) &
+					AMDGPU_UALINK_HELLO_MSG_ACCID_MASK;
+		sender_acc_id = (dw0 >> AMDGPU_UALINK_HELLO_MSG_SENDER_ACCID_SHIFT) &
+					AMDGPU_UALINK_HELLO_MSG_ACCID_MASK;
+		dev_dbg(adev->dev,
+			"Got HELLO MSG: src acc_id %u receiver_acc_id %u sender_acc_id %u\n",
+			src_acc_id, receiver_acc_id, sender_acc_id);
+		amdgpu_ualink_process_hello_msg(adev, receiver_acc_id, sender_acc_id,
+						src_acc_id);
+		break;
+	case AMDGPU_UALINK_HELLO_ACK_MSG:
+		dev_dbg(adev->dev,
+			"Got HELLO-ACK MSG: remote acc_id %u\n", src_acc_id);
+		amdgpu_ualink_process_hello_ack_msg(adev, src_acc_id);
+		break;
+	case AMDGPU_UALINK_NPA_REQ_MSG:
+		memcpy(&handle, entry->src_data, sizeof(struct amdgpu_ualink_handle));
+		handle.handle_lo = amdgpu_ualink_reassemble_handle(handle.handle_lo,
+								   local_acc_id);
+		dev_dbg(adev->dev,
+			"Got NPA-REQ MSG: remote acc_id %u handle %llx:%llx\n",
+			src_acc_id, handle.handle_hi, handle.handle_lo);
+		amdgpu_ualink_process_npa_req_msg(adev, src_acc_id, handle);
+		break;
+	case AMDGPU_UALINK_NPA_RSP_MSG:
+		memcpy(&handle.handle_lo, entry->src_data, sizeof(handle.handle_lo));
+		handle.handle_lo = amdgpu_ualink_reassemble_handle(handle.handle_lo,
+								   src_acc_id);
+		size = entry->src_data[2];
+		npa_addr = entry->src_data[3];
+		dev_dbg(adev->dev,
+			"Got NPA-RSP MSG: remote acc_id %u handle_lo %llx size 0x%x npa_addr 0x%x\n",
+			src_acc_id, handle.handle_lo, size, npa_addr);
+		amdgpu_ualink_process_npa_rsp_msg(adev, src_acc_id,
+						  handle.handle_lo,
+						  npa_addr, size);
+		break;
+	case AMDGPU_UALINK_NPA_FAIL_MSG:
+		memcpy(&handle.handle_lo, entry->src_data, sizeof(handle.handle_lo));
+		handle.handle_lo = amdgpu_ualink_reassemble_handle(handle.handle_lo,
+								   src_acc_id);
+		fail_reason = entry->src_data[2] &
+				AMDGPU_UALINK_NPA_FAIL_MSG_FAIL_REASON_MASK;
+		dev_dbg(adev->dev,
+			"Got NPA-FAIL MSG: remote acc_id %u handle_lo %llx fail_reason %u\n",
+			src_acc_id, handle.handle_lo, fail_reason);
+		amdgpu_ualink_process_npa_fail_msg(adev, src_acc_id,
+						   handle.handle_lo,
+						   fail_reason);
+		break;
+	case AMDGPU_UALINK_NPA_REVOKE_MSG:
+		memcpy(&handle, entry->src_data, sizeof(struct amdgpu_ualink_handle));
+		handle.handle_lo = amdgpu_ualink_reassemble_handle(handle.handle_lo,
+								   src_acc_id);
+		dev_dbg(adev->dev,
+			"Got NPA-REVOKE MSG: remote acc_id %u handle %llx:%llx\n",
+			src_acc_id, handle.handle_hi, handle.handle_lo);
+		amdgpu_ualink_process_npa_revoke_msg(adev, src_acc_id, handle);
+		break;
+	case AMDGPU_UALINK_NPA_RELEASE_MSG:
+		memcpy(&handle, entry->src_data, sizeof(struct amdgpu_ualink_handle));
+		handle.handle_lo = amdgpu_ualink_reassemble_handle(handle.handle_lo,
+								   local_acc_id);
+		dev_dbg(adev->dev,
+			"Got NPA-RELEASE MSG: remote acc_id %u handle %llx:%llx\n",
+			src_acc_id, handle.handle_hi, handle.handle_lo);
+		amdgpu_ualink_process_npa_release_msg(adev, src_acc_id, handle);
+		break;
+	default:
+		dev_err(adev->dev, "Unknown message type (%u)\n", msg_type);
+		break;
+	}
 
 	return handled;
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index 8aee16ad39e9a..b143a496b4dd9 100644
@@ -39,6 +39,7 @@ struct amdgpu_device;
 
 #define AMDGPU_UALINK_HANDLE_ACCID_MASK			GENMASK_ULL(9, 0)
 #define AMDGPU_UALINK_MESSAGE_HEADER_MASK		GENMASK_ULL(9, 0)
+#define AMDGPU_UALINK_HELLO_MSG_ACCID_MASK		GENMASK_U32(9, 0)
 #define AMDGPU_UALINK_HELLO_MSG_RECV_ACCID_SHIFT	10
 #define AMDGPU_UALINK_HELLO_MSG_SENDER_ACCID_SHIFT	20
 #define AMDGPU_UALINK_NPA_FAIL_MSG_FAIL_REASON_MASK	GENMASK_U32(7, 0)
@@ -344,6 +345,8 @@ int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev);
 int amdgpu_ualink_sw_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
 
+int ualink_send_hello(struct amdgpu_device *adev, u32 remote_accel_id);
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
 int amdgpu_ualink_init(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#57 lore raw

PATCH v2 57/95 drm/amdgpu: Send TLB shootdown on exported memory unmap

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Replace the stub implementation of TLB shootdown message with the actual implementation.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 66 ++++++++++++++++------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  3 +
 2 files changed, 52 insertions(+), 17 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +49 -17
index 762713194bb6d..4e312eb3441d1 100644
@@ -47,6 +47,9 @@ static void amdgpu_ualink_handle_connection_reset(struct amdgpu_device *adev,
 						  u32 remote_accel_id, u32 state,
 						  u32 generation_count);
 static void amdgpu_ualink_invalidate_import_mappings(struct amdgpu_bo *bo);
+static int amdgpu_ualink_remote_shootdown(struct amdgpu_device *adev,
+					  u32 remote_accel_id, u64 addr,
+					  u32 size_in_pages, u32 flush_type);
 
 #define STRIP_NPA(addr)						\
 	(((u64)(addr) & ~AMDGPU_UALINK_NPA_ADDR_GPUID_MASK))
@@ -1448,12 +1451,6 @@ static int amdgpu_ualink_send_npa_req_msg(struct amdgpu_device *adev,
 					      dw2, dw3);
 }
 
-static int amdgpu_ualink_send_tlb_shootdown(struct amdgpu_device *adev,
-					    u32 remote_acc_id)
-{
-	return 0;
-}
-
 static u64 amdgpu_ualink_get_export_pte_flags(struct amdgpu_device *adev,
 				       struct amdgpu_bo *bo,
 				       u64 mapping_flags)
@@ -2052,6 +2049,51 @@ static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
 	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
 }
 
+static void amdgpu_ualink_send_tlb_shootdown(struct amdgpu_device *adev,
+				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
+	u32 accel_id = adev->ualink.info->ppod.accel_id;
+	struct amdgpu_ualink_importer_entry *imp_entry;
+	u64 npa_addr, size;
+	u32 remote_acc_id;
+	int r;
+
+	size = amdgpu_bo_ngpu_pages(exp_xa_node->bo);
+
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+			 AMDGPU_UALINK_ACCEL_MAX) {
+		/*
+		 * Replace the GPU-id bits with local accel_id
+		 */
+		if (addr_mode == AMDGPU_UALINK_ADDR_MODE_SOURCE_ALIAS)
+			imp_entry = &exp_xa_node->importer_entries[0];
+		else
+			imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
+
+		npa_addr = STRIP_NPA(imp_entry->npa_addr);
+		npa_addr = GENERATE_NPA(npa_addr, accel_id);
+
+		if (!amdgpu_ualink_check_conn_ready(adev, remote_acc_id,
+					imp_entry->generation_count)) {
+			clear_bit(remote_acc_id,
+				  exp_xa_node->importers_bitmap);
+			continue;
+		}
+
+		dev_dbg(adev->dev,
+			"EXP-CLEANUP: Sending TLB-shootdown to remote:%u\n",
+			remote_acc_id);
+		r = amdgpu_ualink_remote_shootdown(adev, remote_acc_id,
+					npa_addr, size,
+					AMDGPU_UALINK_HEAVYWEIGHT_TLB_SHOOTDOWN);
+		if (r)
+			dev_err(adev->dev,
+				"EXP-CLEANUP: TLB shootdown send failed to remote:%u\n",
+				remote_acc_id);
+	}
+}
+
 /* Unmap all NPA addresses associated with a BO (UALink handle). This function is used
  * only in Source Identification mode.
  */
@@ -2210,17 +2252,7 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 	amdgpu_ualink_force_retry_rpcs(adev, exp_xa_node);
 
 	/* Send TLB-shootdown to all importer GPUs */
-	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
-			 AMDGPU_UALINK_ACCEL_MAX) {
-		dev_dbg(adev->dev,
-			"EXP-CLEANUP: Sending TLB-shootdown to remote:%u\n",
-			remote_acc_id);
-		r = amdgpu_ualink_send_tlb_shootdown(adev, remote_acc_id);
-		if (r)
-			dev_err(adev->dev,
-				"EXP-CLEANUP: TLB shootdown send failed to remote:%u\n",
-				remote_acc_id);
-	}
+	amdgpu_ualink_send_tlb_shootdown(adev, exp_xa_node);
 
 	/* Unmap all NPA addresses for this BO from NPA VM */
 	amdgpu_ualink_unmap_all_npa_addr(adev, exp_xa_node);
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index b143a496b4dd9..fa116ef44cb50 100644
@@ -82,6 +82,9 @@ enum AMDGPU_UALINK_PROTOCOL_MESSAGES {
 	AMDGPU_UALINK_MAX_PROTOCOL_MSG
 };
 
+#define AMDGPU_UALINK_LIGHTWEIGHT_TLB_SHOOTDOWN		0x1
+#define AMDGPU_UALINK_HEAVYWEIGHT_TLB_SHOOTDOWN		0x2
+
 enum amdgpu_ualink_conn_state {
 	AMDGPU_UALINK_CONN_NOT_READY			= 0,
 	AMDGPU_UALINK_CONN_IN_PROGRESS			= 1,
Signature · 1 line
2.55.0
author maintainer
#58 lore raw

PATCH v2 58/95 drm/amdgpu: Handle local GPUs in UALink import

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

When the remote GPU is part of the local_accels array (same physical pod), bypass the HELLO/NPA-REQ messaging protocol and directly look up the exported BO in the peer's exporter xarray. Export it as a DMA-buf and return the fd to user-space without creating an importer xarray entry.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 100 +++++++++++++++++----
 1 file changed, 85 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +85 -15
index 4e312eb3441d1..e21c48212f0a4 100644
@@ -28,6 +28,7 @@
 #include "amdgpu.h"
 #include "amdgpu_ualink.h"
 #include "amdgpu_xgmi.h"
+#include "amdgpu_dma_buf.h"
 #include <linux/math.h>
 #include <linux/sysfs.h>
 #include <linux/string.h>
@@ -571,6 +572,37 @@ static const struct kobj_type ualink_ppod_setup_ktype = {
 	.sysfs_ops = &kobj_sysfs_ops
 };
 
+static struct amdgpu_device *find_peer_adev(unsigned int accel_id)
+{
+	unsigned int i;
+
+	for (i = 0; i < mgpu_info.num_gpu; i++) {
+		struct amdgpu_device *peer_adev = mgpu_info.gpu_ins[i].adev;
+
+		if (peer_adev->ualink.info &&
+		    peer_adev->ualink.info->ppod.accel_id == accel_id)
+			return peer_adev;
+	}
+
+	return NULL;
+}
+
+static bool amdgpu_ualink_is_local_accel(struct amdgpu_device *adev,
+					 u32 accel_id)
+{
+	struct amdgpu_ualink_info *info = adev->ualink.info;
+	unsigned int i;
+
+	if (!info)
+		return false;
+
+	for (i = 0; i < info->n_local_accels; i++) {
+		if (info->local_accels[i] == accel_id)
+			return true;
+	}
+	return false;
+}
+
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
 UALINK_VALUE_SHOW(vpod_config, vpod_id,   vpod.id,   "%u");
 UALINK_VALUE_SHOW(vpod_config, vpod_size, vpod.size, "%u");
@@ -621,21 +653,6 @@ static bool check_vpod_info(struct amdgpu_device *adev,
 	return true;
 }
 
-static struct amdgpu_device *find_peer_adev(unsigned int accel_id)
-{
-	unsigned int i;
-
-	for (i = 0; i < mgpu_info.num_gpu; i++) {
-		struct amdgpu_device *peer_adev = mgpu_info.gpu_ins[i].adev;
-
-		if (peer_adev->ualink.info &&
-		    peer_adev->ualink.info->ppod.accel_id == accel_id)
-			return peer_adev;
-	}
-
-	return NULL;
-}
-
 static bool check_local_vpod_integrity(struct amdgpu_device *adev)
 {
 	struct amdgpu_ualink_info *info = adev->ualink.info;
@@ -3102,6 +3119,55 @@ static int amdgpu_ualink_do_import_handle(struct amdgpu_device *adev,
 	return r;
 }
 
+static int amdgpu_ualink_local_import(struct amdgpu_device *adev,
+				      u32 remote_acc_id,
+				      struct amdgpu_ualink_handle *handle,
+				      int *fd_out)
+{
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	struct amdgpu_device *peer_adev;
+	int fd;
+
+	mutex_lock(&mgpu_info.mutex);
+	peer_adev = find_peer_adev(remote_acc_id);
+	mutex_unlock(&mgpu_info.mutex);
+	if (!peer_adev) {
+		dev_err(adev->dev,
+			"IMPORT LOCAL: peer adev not found for AccId:%u\n",
+			remote_acc_id);
+		return -ENODEV;
+	}
+
+	xa_lock(&peer_adev->ualink.exp_xa);
+	exp_xa_node = xa_load(&peer_adev->ualink.exp_xa, handle->handle_lo);
+	if (!exp_xa_node ||
+	    exp_xa_node->handle.handle_hi != handle->handle_hi ||
+	    !amdgpu_ualink_exp_xa_entry_get(exp_xa_node)) {
+		xa_unlock(&peer_adev->ualink.exp_xa);
+		dev_err(adev->dev,
+			"IMPORT LOCAL: handle:%llx:%llx not found in peer exp_xa\n",
+			handle->handle_hi, handle->handle_lo);
+		return -EINVAL;
+	}
+	xa_unlock(&peer_adev->ualink.exp_xa);
+
+	get_dma_buf(exp_xa_node->dmabuf);
+	/* Get a new fd for the DMABuf */
+	fd = dma_buf_fd(exp_xa_node->dmabuf, O_CLOEXEC | O_RDWR);
+	if (fd < 0) {
+		dev_err(adev->dev,
+			"IMPORT LOCAL: dma-buf fd failed handle:%llx:%llx\n",
+			handle->handle_hi, handle->handle_lo);
+		dma_buf_put(exp_xa_node->dmabuf);
+		amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+		return fd;
+	}
+
+	amdgpu_ualink_exp_xa_entry_put(exp_xa_node);
+	*fd_out = fd;
+	return 0;
+}
+
 int amdgpu_ualink_import_handle(struct drm_device *dev,
 				const struct amdgpu_ualink_handle *ualink_handle,
 				int *fd_out)
@@ -3121,6 +3187,10 @@ int amdgpu_ualink_import_handle(struct drm_device *dev,
 		return -EINVAL;
 	}
 
+	if (amdgpu_ualink_is_local_accel(adev, remote_acc_id))
+		return amdgpu_ualink_local_import(adev, remote_acc_id,
+						  &handle, fd_out);
+
 	xa_lock(&adev->ualink.imp_xa);
 	imp_xa_node = xa_load(&adev->ualink.imp_xa, handle.handle_lo);
 
Signature · 1 line
2.55.0
author maintainer
#59 lore raw

PATCH v2 59/95 drm/amdgpu: Add debugfs to drop UALink protocol messages

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Introduce a debugfs-controlled bitmap, amdgpu_ualink_drop_msg_bitmap, where each bit corresponds to a value from
enum AMDGPU_UALINK_PROTOCOL_MESSAGES (e.g. NPA-REQ, NPA-RSP, NPA-FAIL, NPA-REVOKE, NPA-RELEASE).

When a bit is set, the next incoming message of that type is dropped in amdgpu_ualink_process_irq() and the bit is atomically cleared via test_and_clear_bit(), so subsequent incoming messages are processed normally. This allows exercising the UALink connection reset and recovery paths (NPA-RSP timeout on the importer, retransmit/teardown on the exporter, etc.) by injecting a single packet loss from userspace, e.g.:

    # drop one NPA-REQ
    echo 0x08 > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap
    # drop one NPA-RSP
    echo 0x10 > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap

The bitmap lives in struct amdgpu_ualink_mgr and is exposed via debugfs_create_ulong() next to the existing amdgpu_ualink_test entry.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 14 ++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c  | 13 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h  | 16 ++++++++++++++++
 3 files changed, 43 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +14 -0
index 132d054900b5b..d80fe17556dab 100644
@@ -2229,6 +2229,20 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev)
 	debugfs_create_file("amdgpu_benchmark", 0200, root, adev,
 			    &amdgpu_benchmark_fops);
 
+	/* Debug-only: bitmap of incoming UALink protocol messages to drop.
+	 * Each bit position corresponds to an enum AMDGPU_UALINK_PROTOCOL_MESSAGES
+	 * value. Setting a bit causes exactly one matching incoming packet to be
+	 * dropped, after which the bit auto-clears and traffic resumes. Used to
+	 * exercise the connection reset paths.
+	 *
+	 * Examples (drop one NPA-REQ):
+	 *   echo 0x8  > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap
+	 * (drop one NPA-RSP):
+	 *   echo 0x10 > /sys/kernel/debug/dri/0/amdgpu_ualink_drop_msg_bitmap
+	 */
+	debugfs_create_ulong("amdgpu_ualink_drop_msg_bitmap", 0600, root,
+			     &adev->ualink.drop_msg_bitmap);
+
 	adev->debugfs_vbios_blob.data = adev->bios;
 	adev->debugfs_vbios_blob.size = adev->bios_size;
 	debugfs_create_blob("amdgpu_vbios", 0444, root,
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +13 -0
index e21c48212f0a4..4e49ff0be6e35 100644
@@ -5394,6 +5394,19 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 	dev_dbg(adev->dev, "Got MSG: remote acc_id %u msg_type %u\n",
 		src_acc_id, msg_type);
 
+	/* Debug hook: if the bit corresponding to this msg_type is set in
+	 * drop_msg_bitmap, drop this single packet and clear the bit so that
+	 * any subsequent incoming messages are processed normally. Used to
+	 * exercise the connection reset/recovery paths via debugfs.
+	 */
+	if (msg_type < BITS_PER_LONG &&
+	    test_and_clear_bit(msg_type, &adev->ualink.drop_msg_bitmap)) {
+		dev_warn(adev->dev,
+			 "DROP MSG (debugfs): src acc_id %u msg_type %u dw[0-3] 0x%x 0x%x 0x%x 0x%x\n",
+			 src_acc_id, msg_type, dw0, dw1, dw2, dw3);
+		return handled;
+	}
+
 	switch (msg_type) {
 	case AMDGPU_UALINK_HELLO_MSG:
 		receiver_acc_id = (dw0 >> AMDGPU_UALINK_HELLO_MSG_RECV_ACCID_SHIFT) &
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +16 -0
index fa116ef44cb50..d4d4ca0199dc7 100644
@@ -342,6 +342,22 @@ struct amdgpu_ualink_mgr {
 
 	/* Sequence number to track the need for TLB flushes */
 	atomic64_t last_flushed_tlb_seq;
+
+	/* Debug-only: bitmap of incoming UALink protocol messages to drop.
+	 *
+	 * Each bit position corresponds to a value from
+	 * enum AMDGPU_UALINK_PROTOCOL_MESSAGES:
+	 *   BIT(AMDGPU_UALINK_NPA_REQ_MSG) drops one incoming NPA-REQ.
+	 *   BIT(AMDGPU_UALINK_NPA_RSP_MSG) drops one incoming NPA-RSP.
+	 *
+	 *   On reception of a message whose corresponding bit is set, the bit
+	 *   is atomically cleared and the message is silently dropped. This
+	 *   means at most one packet per set bit is dropped; any further
+	 *   incoming packets of the same type are processed normally. This is
+	 *   intended to exercise the connection reset / recovery paths from a
+	 *   debugfs handle.
+	 */
+	unsigned long drop_msg_bitmap;
 };
 
 int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev);
Signature · 1 line
2.55.0
author maintainer
#60 lore raw

PATCH v2 60/95 drm/amdgpu: Temporarily disable sending remote TLB shootdowns

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Remote TLB shootdown is not working currently and hangs the driver. As a result, temporarily prevent sending the Remote TLB shootdown messages to remote GPUs for now.
Additionally, also temporarily set PTE.X = 0 for imported memory.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 4 ++++
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c     | 1 +
 2 files changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +4 -0
index 4e49ff0be6e35..8d767a91172ca 100644
@@ -5032,6 +5032,10 @@ static int amdgpu_ualink_remote_shootdown(struct amdgpu_device *adev,
 	struct amdgpu_ualink_ring *ring;
 	int r;
 
+	dev_dbg(adev->dev, "skip remote shootdown to remote_accel_id %u\n",
+		remote_accel_id);
+	return 0;
+
 	peer = &remote->peer[remote_accel_id];
 	ring = &peer->shootdown;
 	if (!ring->ready) {
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c +1 -0
index 1468047687652..34d5767287d78 100644
@@ -613,6 +613,7 @@ static void gmc_v12_1_get_npa_flags(struct amdgpu_device *adev,
 	*flags |= AMDGPU_PTE_SNOOPED | AMDGPU_PTE_PRT_GFX12 |
 		   AMDGPU_PTE_BUS_ATOMICS;
 	*flags &= ~AMDGPU_PTE_VALID;
+	*flags &= ~AMDGPU_PTE_EXECUTABLE;
 }
 
 static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
Signature · 1 line
2.55.0
author maintainer
#61 lore raw

PATCH v2 61/95 drm/amdgpu: Temporarily Flush TLB on NPA mapping always

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

This is a temporary patch to always flush TLBs when mapping NPA addresses.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 4 ----
 1 file changed, 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +0 -4
index 8d767a91172ca..69cc35c24a2e8 100644
@@ -3713,12 +3713,8 @@ static inline u32 ualink_tlb_wb_offset(struct amdgpu_device *adev, u32 accel_id)
 
 static void amdgpu_ualink_flush_tlb(struct amdgpu_device *adev, u32 flush_type)
 {
-	uint64_t tlb_seq = amdgpu_vm_tlb_seq(&adev->ualink.npa_vm);
 	u32 bit;
 
-	if (atomic64_xchg(&adev->ualink.last_flushed_tlb_seq, tlb_seq) == tlb_seq)
-		return;
-
 	bit = AMDGPU_MMHUB0_START;
 
 	for_each_set_bit_from(bit, adev->vmhubs_mask, AMDGPU_MAX_VMHUBS)
Signature · 1 line
2.55.0
author maintainer
#62 lore raw

PATCH v2 62/95 drm/amdgpu: log remote memory MTYPE for GC 12.1.0

In reply to Alex Deucher

From: Alex Sierra <[email protected]>

MTYPE UC in A0 and MTYPE NC in A1 for remote memory

---
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c +3 -1
index 34d5767287d78..302ebf2f3ccd1 100644
@@ -649,7 +649,9 @@ static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
 		} else if (amdgpu_mtype_local == 2) {
 			DRM_INFO_ONCE("MTYPE_CC not supported, using %s for local memory\n", is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
 		} else {
-			DRM_INFO_ONCE("Using %s for local memory\n", is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
+			DRM_INFO_ONCE("Using %s for local memory and %s for remote memory\n",
+					is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC",
+					is_aid_a1 ? "MTYPE_NC" : "MTYPE_UC");
 		}
 
 		is_local = (is_vram && adev == bo_adev);
Signature · 1 line
2.55.0
author maintainer
#63 lore raw

PATCH v2 63/95 drm/amdgpu: Prevent double-free of drm_exec

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Always check the return value of amdgpu_ualink_reserve_npa_vm_and_bos. It's not expected to fail when used in non-interruptible mode. But if it fails anyway it calls drm_exec_fini. The caller must not call amdgpu_ualink_unreserve_npa_vm_and_bos in this case to avoid a double-free.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 26 +++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +21 -5
index 69cc35c24a2e8..c5fbc9776c3d0 100644
@@ -1494,7 +1494,11 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 	struct drm_exec exec;
 	int r;
 
-	amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+	if (unlikely(r)) {
+		dev_err(adev->dev, "Failed to reserve VM and BO in unmap_npa_addr\n");
+		return r;
+	}
 
 	r = amdgpu_vm_update_range(adev, &adev->ualink.npa_vm, false, false, true,
 				false, NULL, npa_addr, npa_addr + size - 1,
@@ -1538,7 +1542,11 @@ static int amdgpu_ualink_map_npa_addr(struct amdgpu_device *adev, u64 npa_addr,
 	struct drm_exec exec;
 	int r;
 
-	amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, bos, ARRAY_SIZE(bos), &exec, false);
+	if (unlikely(r)) {
+		dev_err(adev->dev, "Failed to reserve VM and BO in map_npa_addr\n");
+		return r;
+	}
 
 	r = amdgpu_vm_update_range(adev, vm, false, false, true,
 				false, NULL, npa_addr, npa_addr + size - 1,
@@ -2002,7 +2010,11 @@ static void amdgpu_ualink_force_retry_rpcs(struct amdgpu_device *adev,
 	bo = exp_xa_node->bo;
 	size = amdgpu_bo_ngpu_pages(bo);
 
-	amdgpu_ualink_reserve_npa_vm_and_bos(adev, &bo, 1, &exec, false);
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, &bo, 1, &exec, false);
+	if (unlikely(r)) {
+		dev_err(adev->dev, "Failed to reserve VM and BO in force_retry_rpcs\n");
+		return;
+	}
 
 	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
 			 AMDGPU_UALINK_ACCEL_MAX) {
@@ -2128,8 +2140,12 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 
 	size = amdgpu_bo_ngpu_pages(exp_xa_node->bo);
 
-	amdgpu_ualink_reserve_npa_vm_and_bos(adev, &exp_xa_node->bo, 1,
-					     &exec, false);
+	r = amdgpu_ualink_reserve_npa_vm_and_bos(adev, &exp_xa_node->bo, 1,
+						 &exec, false);
+	if (unlikely(r)) {
+		dev_err(adev->dev, "Failed to reserve VM and BO in unmap_all_npa_addr\n");
+		return;
+	}
 
 	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
 			 AMDGPU_UALINK_ACCEL_MAX) {
Signature · 1 line
2.55.0
author maintainer
#64 lore raw

PATCH v2 64/95 drm/amdgpu: fix NPA-RELEASE race in UALink exporter cleanup

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Processing of NPA-RELEASE messages can run concurrently with sending of NPA-REVOKEs to all the importers. This can lead to a race condition where the interrupt handler can signal npa_done as soon as npa_release_bitmap becomes empty even though the cleanup worker was still sending NPA-REVOKEs to all the importers. This would prevent the cleanup worker thread to wait for all NPA-RELEASE messages to come before freeing the NPA addresses.
Fix this by setting the importer_bitmap for all importers before sending any NPA-REVOKE messages.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +11 -3
index c5fbc9776c3d0..ea89187750126 100644
@@ -2315,7 +2315,7 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 	amdgpu_bo_unref(&bo);
 	exp_xa_node->bo = NULL;
 
-	/* Send NPA-REVOKE to all importers which have imported this memory */
+	/* Build the full npa_release_bitmap before sending any NPA-REVOKE. */
 	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
 				 AMDGPU_UALINK_ACCEL_MAX) {
 		imp_entry = &exp_xa_node->importer_entries[remote_acc_id];
@@ -2326,10 +2326,17 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 			continue;
 		}
 
+		set_bit(remote_acc_id, exp_xa_node->npa_release_bitmap);
+	}
+
+	/* Send NPA-REVOKE to all importers which have imported this memory.
+	 * On send failure clear the bit (no response will arrive).
+	 */
+	for_each_set_bit(remote_acc_id, exp_xa_node->importers_bitmap,
+				 AMDGPU_UALINK_ACCEL_MAX) {
 		dev_dbg(adev->dev,
 			"EXP-CLEANUP: Sending NPA-REVOKE to remote:%u\n",
 			remote_acc_id);
-		set_bit(remote_acc_id, exp_xa_node->npa_release_bitmap);
 		r = amdgpu_ualink_send_npa_revoke_msg(adev, remote_acc_id, handle);
 		if (r) {
 			dev_err(adev->dev,
@@ -2339,7 +2346,8 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 		}
 	}
 
-	if (!bitmap_empty(exp_xa_node->importers_bitmap,
+	/* Wait for the NPA_RELEASE to come back from all importers */
+	if (!bitmap_empty(exp_xa_node->npa_release_bitmap,
 			AMDGPU_UALINK_ACCEL_MAX)) {
 		dev_dbg(adev->dev,
 			"EXP-CLEANUP: handle:%llx:%llx NPA-RELEASE bitmap: %*pbl\n",
Signature · 1 line
2.55.0
author maintainer
#65 lore raw

PATCH v2 65/95 drm/amdgpu: initialize UALink importer node list head

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Initialize the list head at node creation so list_del_init() is safe on a node that has not yet been added to imp_handles_list.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +1 -0
index ea89187750126..451c3437f5bd5 100644
@@ -3263,6 +3263,7 @@ int amdgpu_ualink_import_handle(struct drm_device *dev,
 		imp_xa_node->node_state = AMDGPU_UALINK_NODE_NOT_READY;
 		imp_xa_node->handle = handle;
 		init_completion(&imp_xa_node->npa_done);
+		INIT_LIST_HEAD(&imp_xa_node->list);
 		kref_init(&imp_xa_node->refcount);
 
 		/* Take an extra reference to store in the Xarray. The error
Signature · 1 line
2.55.0
author maintainer
#66 lore raw

PATCH v2 66/95 drm/amdgpu: Fix initialization flags for UALink XAs

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

The UALink exp_xa, imp_xa and handle_invalid_xa xarrays are only ever accessed from process context: ioctls or workqueues. None of them run in softirq or hardirq context, so initializing the Xarray with BH flag is unnecessary.

Drop XA_FLAGS_LOCK_BH and use the default plain-spinlock locking (xa_init() and the plain xa_* accessors).

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +3 -3
index 451c3437f5bd5..be38e9ab47a7b 100644
@@ -3447,9 +3447,9 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 	adev->mmhub.funcs->setup_vm_pt_regs(adev, adev->vm_manager.npa_vmid,
 			amdgpu_gmc_pd_addr(adev->ualink.npa_vm.root.bo));
 
-	xa_init_flags(&adev->ualink.exp_xa, XA_FLAGS_LOCK_BH);
-	xa_init_flags(&adev->ualink.imp_xa, XA_FLAGS_LOCK_BH);
-	xa_init_flags(&adev->ualink.handle_invalid_xa, XA_FLAGS_LOCK_BH);
+	xa_init(&adev->ualink.exp_xa);
+	xa_init(&adev->ualink.imp_xa);
+	xa_init(&adev->ualink.handle_invalid_xa);
 
 	for (i = 0; i < AMDGPU_UALINK_ACCEL_MAX; i++) {
 		init_completion(&adev->ualink.conn_state[i].hello_done);
Signature · 1 line
2.55.0
author maintainer
#67 lore raw

PATCH v2 67/95 drm/amdgpu: fix dma_buf leak in UALink exporter cleanup

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

At export time the driver code always builds a dma_buf for the BO which is then shared for local imports.
During cleanup, if there are no importers, which is true for local imports, we never free the dma_buf thus causing the memory leak.

Fix by dropping the dma_buf reference in the no-importers branch as well.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +6 -0
index be38e9ab47a7b..84e556e0c4d19 100644
@@ -2267,6 +2267,12 @@ static void amdgpu_ualink_exp_cleanup_worker(struct work_struct *work)
 	/* If there are no importers for this BO/handle */
 	if (bitmap_empty(exp_xa_node->importers_bitmap,
 			 AMDGPU_UALINK_ACCEL_MAX)) {
+		/* Release the dma_buf created at export time (used for the
+		 * local-import shortcut). The has-importers path below drops
+		 * it too; this branch must not skip it or the dma_buf and the
+		 * BO it pins are leaked.
+		 */
+		dma_buf_put(exp_xa_node->dmabuf);
 		/* Drop the BO reference so it can be freed. */
 		amdgpu_bo_unref(&bo);
 		exp_xa_node->bo = NULL;
Signature · 1 line
2.55.0
author maintainer
#68 lore raw

PATCH v2 68/95 drm/amdgpu: Increase UALink soft ring size

In reply to Alex Deucher

From: Philip Yang <[email protected]>

The UALink soft ring overflows under bursts of interrupts from remote GPUs. Size it at 16x the default soft ring to absorb the burst.

While here, read the source accel_id and context dwords before the delegate path and log them, so the originating GPU is recorded for every interrupt.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h     |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 17 ++++++++---------
 drivers/gpu/drm/amd/amdgpu/ih_v7_0.c       |  2 +-
 3 files changed, 10 insertions(+), 10 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h +1 -0
index 11f834c97d937..8d74a89b40cf8 100644
@@ -30,6 +30,7 @@
 #define IH_RING_SIZE	(256 * 1024)
 #define IH_SW_RING_SIZE	(16 * 1024)	/* enough for 512 CAM entries */
 #define IH_PSP_RING_SIZE (SZ_4K)
+#define IH_UALINK_RING_SIZE    (256 * 1024)
 
 struct amdgpu_device;
 struct amdgpu_iv_entry;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +8 -9
index 84e556e0c4d19..d4a5304ee2d81 100644
@@ -5403,13 +5403,6 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 		entry->ih == &adev->irq.ih ? "ring" : "ualink soft ring",
 		entry->client_id, entry->src_id);
 
-	/* Copy IH entry into ualink soft ring. */
-	if (entry->ih == &adev->irq.ih) {
-		dev_dbg(adev->dev, "delegate to ualink irq soft ring\n");
-		amdgpu_irq_ualink_delegate(adev, entry, 8);
-		return handled;
-	}
-
 	/* ContextID 4 dwords */
 	src_acc_id = entry->pasid;
 	dw0 = entry->src_data[0];
@@ -5417,8 +5410,14 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 	dw2 = entry->src_data[2];
 	dw3 = entry->src_data[3];
 
-	dev_dbg(adev->dev, "src accel_id %u context id 0x%x 0x%x 0x%x 0x%x\n",
-		src_acc_id, dw0, dw1, dw2, dw3);
+	/* Copy IH entry into ualink soft ring. */
+	if (entry->ih == &adev->irq.ih) {
+		dev_dbg(adev->dev, "src accel_id %u context id 0x%x 0x%x 0x%x 0x%x\n",
+			src_acc_id, dw0, dw1, dw2, dw3);
+		dev_dbg(adev->dev, "delegate to ualink irq soft ring\n");
+		amdgpu_irq_ualink_delegate(adev, entry, 8);
+		return handled;
+	}
 
 	msg_type = dw0 & AMDGPU_UALINK_MESSAGE_HEADER_MASK;
 	local_acc_id = adev->ualink.info->ppod.accel_id;
drivers/gpu/drm/amd/amdgpu/ih_v7_0.c +1 -1
index c93ef937c4309..3745439b86b36 100644
@@ -643,7 +643,7 @@ static int ih_v7_0_sw_init(struct amdgpu_ip_block *ip_block)
 		return r;
 
 	dev_dbg(adev->dev, "ualink init ih_ualink\n");
-	r = amdgpu_ih_ring_init(adev, &adev->irq.ih_ualink, sw_ring_size, true);
+	r = amdgpu_ih_ring_init(adev, &adev->irq.ih_ualink, IH_UALINK_RING_SIZE, true);
 	if (r)
 		return r;
 
Signature · 1 line
2.55.0
author maintainer
#69 lore raw

PATCH v2 69/95 drm/amdgpu: Fix uninitialized fence in UALink NPA unmap

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Initialize fence to NULL in UALink NPA unmap to make sure we don't end with a garbage value which then gets incorrectly dereferenced inside amdgpu_vm_tlb_flush().

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +1 -1
index d4a5304ee2d81..85276db902b15 100644
@@ -1490,7 +1490,7 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 {
 	uint64_t pte_value = adev->gmc.noretry_flags;
 	struct amdgpu_bo *bos[] = { bo };
-	struct dma_fence *fence;
+	struct dma_fence *fence = NULL;
 	struct drm_exec exec;
 	int r;
 
Signature · 1 line
2.55.0
author maintainer
#70 lore raw

PATCH v2 70/95 drm/amdgpu: Use vm->last_update fence in UALink NPA unmap paths

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

amdgpu_ualink_map_npa_addr() already uses vm->last_update as the fence output for amdgpu_vm_update_range(), but the two unmap functions used a local fence variable instead. This is incorrect because amdgpu_vm_update_pdes() always commits into vm->last_update regardless of the fence passed to update_range(). As a result the PDE update job (submitted after the range update) would not wait before the TLB flush.

As a result, switch both amdgpu_ualink_unmap_npa_addr() and amdgpu_ualink_unmap_all_npa_addr() to use vm->last_update fence.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +14 -7
index 85276db902b15..5811218b35d33 100644
@@ -1488,6 +1488,7 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 					struct amdgpu_bo *bo,
 					u64 npa_addr, u64 size)
 {
+	struct amdgpu_vm *vm = &adev->ualink.npa_vm;
 	uint64_t pte_value = adev->gmc.noretry_flags;
 	struct amdgpu_bo *bos[] = { bo };
 	struct dma_fence *fence = NULL;
@@ -1500,16 +1501,16 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 		return r;
 	}
 
-	r = amdgpu_vm_update_range(adev, &adev->ualink.npa_vm, false, false, true,
+	r = amdgpu_vm_update_range(adev, vm, false, false, true,
 				false, NULL, npa_addr, npa_addr + size - 1,
-				pte_value, 0, 0, NULL, NULL, &fence);
+				pte_value, 0, 0, NULL, NULL, &vm->last_update);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed to unmap NPA addr (%llx) from NPA VM\n", npa_addr);
 		goto out;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	r = amdgpu_vm_update_pdes(adev, vm, false);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed %d to update page directories during unmapping NPA: 0x%llx\n",
@@ -1517,12 +1518,16 @@ static int amdgpu_ualink_unmap_npa_addr(struct amdgpu_device *adev,
 		goto out;
 	}
 
+	fence = dma_fence_get(vm->last_update);
 	if (fence) {
 		r = dma_fence_wait(fence, false);
 		dma_fence_put(fence);
 		fence = NULL;
-		if (r)
+		if (r) {
+			dev_dbg(adev->dev,
+				"UNMAP-NPA: dma fence wait failed, error: %d\n", r);
 			goto out;
+		}
 	}
 
 	amdgpu_ualink_flush_tlb(adev, TLB_FLUSH_HEAVYWEIGHT);
@@ -2129,6 +2134,7 @@ static void amdgpu_ualink_send_tlb_shootdown(struct amdgpu_device *adev,
 static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
 {
+	struct amdgpu_vm *vm = &adev->ualink.npa_vm;
 	u32 addr_mode = adev->ualink.info->vpod.addr_mode;
 	struct amdgpu_ualink_importer_entry *imp_entry;
 	u64 pte_value = adev->gmc.noretry_flags;
@@ -2167,10 +2173,10 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 			npa_addr, exp_xa_node->handle.handle_hi, exp_xa_node->handle.handle_lo,
 			remote_acc_id, pte_value);
 
-		r = amdgpu_vm_update_range(adev, &adev->ualink.npa_vm, false,
+		r = amdgpu_vm_update_range(adev, vm, false,
 					   false, true, false, NULL, npa_addr,
 					   npa_addr + size - 1, pte_value, 0,
-					   0, NULL, NULL, &fence);
+					   0, NULL, NULL, &vm->last_update);
 
 		if (r)
 			dev_err(adev->dev,
@@ -2182,7 +2188,7 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 			break;
 	}
 
-	r = amdgpu_vm_update_pdes(adev, &adev->ualink.npa_vm, false);
+	r = amdgpu_vm_update_pdes(adev, vm, false);
 	if (r) {
 		dev_err(adev->dev,
 			"Failed %d to update page directories during all NPA addresses unmapping\n",
@@ -2191,6 +2197,7 @@ static void amdgpu_ualink_unmap_all_npa_addr(struct amdgpu_device *adev,
 		return;
 	}
 
+	fence = dma_fence_get(vm->last_update);
 	if (fence) {
 		r = dma_fence_wait(fence, false);
 		dma_fence_put(fence);
Signature · 1 line
2.55.0
author maintainer
#71 lore raw

PATCH v2 71/95 drm/amdgpu: Pin page tables in NPA VMs

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

With latest POR, remote importers aren't guaranteed to support instruction replay-on-fault. Therefore, we cannot support unpinned exports or unpinned page-tables in the exporter mappings into NPA space.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c |  4 +--
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 40 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h     |  3 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c  | 25 +++++++++++++-
 4 files changed, 69 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +2 -2
index 5811218b35d33..0f57a7e2c3341 100644
@@ -3444,8 +3444,8 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 	if (r)
 		goto out;
 
-	/* For using CPU for page table updates. */
-	r = amdgpu_vm_make_compute(adev, &adev->ualink.npa_vm);
+	/* For pinning page tables and using CPU for page table updates. */
+	r = amdgpu_vm_make_npa(adev, &adev->ualink.npa_vm);
 	if (r)
 		goto uninit_vm;
 
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +40 -0
index fc928c2d8b21e..47cde54594693 100644
@@ -2752,6 +2752,46 @@ int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm)
 	return r;
 }
 
+/**
+ * amdgpu_vm_make_npa - Turn a GFX VM into an NPA VM
+ *
+ * @adev: amdgpu_device pointer
+ * @vm: requested vm
+ *
+ * This only works on GFX VMs that don't have any BOs added and no
+ * page tables allocated yet.
+ *
+ * Changes the following VM parameters:
+ * - use_cpu_for_update
+ * - pins page tables
+ * - initializes PTEs to no-retry encoding
+ *
+ * Reinitializes the page directory to reflect the changed ATS
+ * setting.
+ *
+ * Returns:
+ * 0 for success, -errno for errors.
+ */
+int amdgpu_vm_make_npa(struct amdgpu_device *adev, struct amdgpu_vm *vm)
+{
+	int r = amdgpu_vm_make_compute(adev, vm);
+
+	if (r)
+		return r;
+	vm->is_npa = true;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;
+	r = amdgpu_bo_pin(vm->root.bo, AMDGPU_GEM_DOMAIN_VRAM);
+	amdgpu_bo_unreserve(vm->root.bo);
+	if (r)
+		return r;
+
+	vm->is_npa = true;
+
+	return 0;
+}
+
 static int amdgpu_vm_stats_is_zero(struct amdgpu_vm *vm)
 {
 	for (int i = 0; i < __AMDGPU_PL_NUM; ++i) {
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +3 -0
index 7a0ee19a12b18..0f6634b749746 100644
@@ -369,6 +369,8 @@ struct amdgpu_vm {
 	struct ttm_lru_bulk_move lru_bulk_move;
 	/* Flag to indicate if VM is used for compute */
 	bool			is_compute_context;
+	/* Flag to indicate that page tables are for NPA mappings */
+	bool			is_npa;
 	/* Flag to indicate if VM needs a TLB fence (KFD or KGD) */
 	bool			need_tlb_fence;
 
@@ -429,6 +431,7 @@ void amdgpu_vm_manager_fini(struct amdgpu_device *adev);
 long amdgpu_vm_wait_idle(struct amdgpu_vm *vm, long timeout);
 int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, int32_t xcp_id);
 int amdgpu_vm_make_compute(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+int amdgpu_vm_make_npa(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 int amdgpu_vm_lock_pd(struct amdgpu_vm *vm, struct drm_exec *exec,
 		      unsigned int num_fences);
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c +24 -1
index b85b70a1a428d..b9831a5f85d8b 100644
@@ -445,6 +445,7 @@ int amdgpu_vm_pt_create(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 {
 	struct amdgpu_bo_param bp;
 	unsigned int num_entries;
+	int r;
 
 	memset(&bp, 0, sizeof(bp));
 
@@ -477,7 +478,24 @@ int amdgpu_vm_pt_create(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 	if (vm->root.bo)
 		bp.resv = vm->root.bo->tbo.base.resv;
 
-	return amdgpu_bo_create_vm(adev, &bp, vmbo);
+	r = amdgpu_bo_create_vm(adev, &bp, vmbo);
+	if (r)
+		return r;
+
+	/* Assumes that reservation is shared with the VM root and that the
+	 * reservation is locked
+	 */
+	if (vm->root.bo && vm->is_npa) {
+		struct amdgpu_bo *pt_bo = &(*vmbo)->bo;
+
+		r = amdgpu_bo_pin(pt_bo, AMDGPU_GEM_DOMAIN_VRAM);
+		if (r) {
+			amdgpu_bo_unref(&pt_bo);
+			return r;
+		}
+	}
+
+	return 0;
 }
 
 /**
@@ -527,6 +545,8 @@ static int amdgpu_vm_pt_alloc(struct amdgpu_device *adev,
 	return 0;
 
 error_free_pt:
+	if (vm->is_npa)
+		amdgpu_bo_unpin(pt_bo);
 	amdgpu_bo_unref(&pt_bo);
 	return r;
 }
@@ -541,6 +561,9 @@ static void amdgpu_vm_pt_free(struct amdgpu_vm_bo_base *entry)
 	if (!entry->bo)
 		return;
 
+	if (entry->vm->is_npa)
+		amdgpu_bo_unpin(entry->bo);
+
 	amdgpu_vm_update_stats(entry, entry->bo->tbo.resource, -1);
 	entry->bo->vm_bo = NULL;
 	ttm_bo_set_bulk_move(&entry->bo->tbo, NULL);
Signature · 1 line
2.55.0
author maintainer
#72 lore raw

PATCH v2 72/95 drm/amdgpu: Initialize NPA PT/PDs to noretry

In reply to Alex Deucher

From: Felix Kuehling <[email protected]>

Unmapped NPA addresses should result in no-retry faults on the importer side.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c +4 -0
index b9831a5f85d8b..1cdf2b854f261 100644
@@ -408,10 +408,14 @@ int amdgpu_vm_pt_clear(struct amdgpu_device *adev, struct amdgpu_vm *vm,
 
 	if (adev->asic_type >= CHIP_VEGA10) {
 		if (level != AMDGPU_VM_PTB) {
+			if (vm->is_npa)
+				flags = adev->gmc.noretry_flags;
 			/* Handle leaf PDEs as PTEs */
 			flags |= AMDGPU_PDE_PTE_FLAG(adev);
 			amdgpu_gmc_get_vm_pde(adev, level,
 					      &value, &flags);
+		} else if (vm->is_npa) {
+			flags = adev->gmc.noretry_flags;
 		} else {
 			/* Workaround for fault priority problem on GMC9 */
 			flags = AMDGPU_PTE_EXECUTABLE | adev->gmc.init_pte_flags;
Signature · 1 line
2.55.0
author maintainer
#73 lore raw

PATCH v2 73/95 drm/amdgpu: always use MTYPE_UC for remote memory on GFX 12.1

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

On GFX 12.1 remote (non-local / system) memory accesses should always be mapped as MTYPE_UC.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c | 11 +++++------
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c   |  4 ++--
 2 files changed, 7 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c +5 -6
index 302ebf2f3ccd1..fd24d66290f55 100644
@@ -638,20 +638,19 @@ static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
 		is_aid_a1 = (adev->rev_id & 0x10);
 
 		mtype_local = is_aid_a1 ? MTYPE_RW : MTYPE_NC;
-		mtype_remote = is_aid_a1 ? MTYPE_NC : MTYPE_UC;
+		/* Remote memory always uses MTYPE_UC on GFX 12.1. */
+		mtype_remote = MTYPE_UC;
 		if (amdgpu_mtype_local == 0) {
-			DRM_INFO_ONCE("Using MTYPE_RW for local memory and MTYPE_NC for remote memory\n");
+			DRM_INFO_ONCE("Using MTYPE_RW for local memory\n");
 			mtype_local = MTYPE_RW;
-			mtype_remote = MTYPE_NC;
 		} else if (amdgpu_mtype_local == 1) {
 			DRM_INFO_ONCE("Using MTYPE_NC for local memory\n");
 			mtype_local = MTYPE_NC;
 		} else if (amdgpu_mtype_local == 2) {
 			DRM_INFO_ONCE("MTYPE_CC not supported, using %s for local memory\n", is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
 		} else {
-			DRM_INFO_ONCE("Using %s for local memory and %s for remote memory\n",
-					is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC",
-					is_aid_a1 ? "MTYPE_NC" : "MTYPE_UC");
+			DRM_INFO_ONCE("Using %s for local memory and MTYPE_UC for remote memory\n",
+					is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
 		}
 
 		is_local = (is_vram && adev == bo_adev);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +2 -2
index fb5eca38e6a93..64fb38400bd1a 100644
@@ -1348,8 +1348,8 @@ svm_range_get_pte_flags(struct kfd_node *node, struct amdgpu_vm *vm,
 		mtype_local = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_RW :
 				amdgpu_mtype_local == 1 ? AMDGPU_VM_MTYPE_NC :
 				is_aid_a1 ? AMDGPU_VM_MTYPE_RW : AMDGPU_VM_MTYPE_NC;
-		mtype_remote = amdgpu_mtype_local == 0 ? AMDGPU_VM_MTYPE_NC :
-				is_aid_a1 ? AMDGPU_VM_MTYPE_NC : AMDGPU_VM_MTYPE_UC;
+		/* Remote memory always uses MTYPE_UC on GFX 12.1. */
+		mtype_remote = AMDGPU_VM_MTYPE_UC;
 		snoop = true;
 
 		if (is_local) /* local HBM  */ {
Signature · 1 line
2.55.0
author maintainer
#74 lore raw

PATCH v2 74/95 drm/amdkfd: program compute MQD coherent_aql_mtype on GFX 12.1

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Name the compute MQD dword at offset 509 (previously reserved_509) as coherent_aql_mtype in v12_1_compute_mqd and program it when initializing a compute queue MQD.

The field must be set to 0 whenever the driver maps local or remote memory as MTYPE_NC, and to 1 in all other cases. Add a shared helper gmc_v12_1_get_mtypes() as the single source of truth for the local and remote MTYPE decision (used by both the PTE coherence flags and the MQD programming), and gmc_v12_1_get_coherent_aql_mtype() which derives the 0/1 value from it.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c        | 63 +++++++++++++------
 drivers/gpu/drm/amd/amdgpu/gmc_v12_1.h        |  1 +
 .../drm/amd/amdkfd/kfd_mqd_manager_v12_1.c    |  7 +++
 drivers/gpu/drm/amd/include/v12_structs.h     |  2 +-
 4 files changed, 54 insertions(+), 19 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.c +45 -18
index fd24d66290f55..a3b7e240abaed 100644
@@ -616,6 +616,50 @@ static void gmc_v12_1_get_npa_flags(struct amdgpu_device *adev,
 	*flags &= ~AMDGPU_PTE_EXECUTABLE;
 }
 
+/*
+ * Resolve the MTYPEs used for local and remote memory accesses on GFX 12.1.
+ * Remote memory always uses MTYPE_UC; local memory depends on the AID stepping
+ * and the amdgpu_mtype_local module parameter.
+ */
+static void gmc_v12_1_get_mtypes(struct amdgpu_device *adev,
+				 unsigned int *mtype_local,
+				 unsigned int *mtype_remote)
+{
+	bool is_aid_a1 = (adev->rev_id & 0x10);
+
+	*mtype_local = is_aid_a1 ? MTYPE_RW : MTYPE_NC;
+	/* Remote memory always uses MTYPE_UC on GFX 12.1. */
+	*mtype_remote = MTYPE_UC;
+
+	if (amdgpu_mtype_local == 0) {
+		DRM_INFO_ONCE("Using MTYPE_RW for local memory\n");
+		*mtype_local = MTYPE_RW;
+	} else if (amdgpu_mtype_local == 1) {
+		DRM_INFO_ONCE("Using MTYPE_NC for local memory\n");
+		*mtype_local = MTYPE_NC;
+	} else if (amdgpu_mtype_local == 2) {
+		DRM_INFO_ONCE("MTYPE_CC not supported, using %s for local memory\n",
+			      is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
+	} else {
+		DRM_INFO_ONCE("Using %s for local memory and MTYPE_UC for remote memory\n",
+			      is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
+	}
+}
+
+/*
+ * The compute MQD coherent_aql_mtype field (offset 509) must be programmed to
+ * 0 whenever the driver maps local or remote memory as MTYPE_NC, and to 1 in
+ * all other cases.
+ */
+u32 gmc_v12_1_get_coherent_aql_mtype(struct amdgpu_device *adev)
+{
+	unsigned int mtype_local, mtype_remote;
+
+	gmc_v12_1_get_mtypes(adev, &mtype_local, &mtype_remote);
+
+	return (mtype_local == MTYPE_NC || mtype_remote == MTYPE_NC) ? 0 : 1;
+}
+
 static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
 					  struct amdgpu_bo *bo,
 					  uint64_t *flags)
@@ -631,27 +675,10 @@ static void gmc_v12_1_get_coherence_flags(struct amdgpu_device *adev,
 	unsigned int mtype, mtype_local, mtype_remote;
 	bool snoop = false;
 	bool is_local = false;
-	bool is_aid_a1;
 
 	switch (gc_ip_version) {
 	case IP_VERSION(12, 1, 0):
-		is_aid_a1 = (adev->rev_id & 0x10);
-
-		mtype_local = is_aid_a1 ? MTYPE_RW : MTYPE_NC;
-		/* Remote memory always uses MTYPE_UC on GFX 12.1. */
-		mtype_remote = MTYPE_UC;
-		if (amdgpu_mtype_local == 0) {
-			DRM_INFO_ONCE("Using MTYPE_RW for local memory\n");
-			mtype_local = MTYPE_RW;
-		} else if (amdgpu_mtype_local == 1) {
-			DRM_INFO_ONCE("Using MTYPE_NC for local memory\n");
-			mtype_local = MTYPE_NC;
-		} else if (amdgpu_mtype_local == 2) {
-			DRM_INFO_ONCE("MTYPE_CC not supported, using %s for local memory\n", is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
-		} else {
-			DRM_INFO_ONCE("Using %s for local memory and MTYPE_UC for remote memory\n",
-					is_aid_a1 ? "MTYPE_RW" : "MTYPE_NC");
-		}
+		gmc_v12_1_get_mtypes(adev, &mtype_local, &mtype_remote);
 
 		is_local = (is_vram && adev == bo_adev);
 		snoop = true;
drivers/gpu/drm/amd/amdgpu/gmc_v12_1.h +1 -0
index 22da84a1d5185..1c7aad7a1791d 100644
@@ -28,4 +28,5 @@ void gmc_v12_1_set_gmc_funcs(struct amdgpu_device *adev);
 void gmc_v12_1_set_irq_funcs(struct amdgpu_device *adev);
 void gmc_v12_1_init_vram_info(struct amdgpu_device *adev);
 void gmc_v12_1_init_nps_details(struct amdgpu_device *adev);
+u32 gmc_v12_1_get_coherent_aql_mtype(struct amdgpu_device *adev);
 #endif
drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v12_1.c +7 -0
index 13a9ce02f3918..92ec081842df6 100644
@@ -31,6 +31,7 @@
 #include "gc/gc_12_1_0_sh_mask.h"
 #include "amdgpu_amdkfd.h"
 #include "kfd_device_queue_manager.h"
+#include "gmc_v12_1.h"
 
 static void update_mqd(struct mqd_manager *mm, void *mqd,
 		       struct queue_properties *q,
@@ -235,6 +236,12 @@ static void init_mqd(struct mqd_manager *mm, void **mqd,
 		m->cp_hqd_wg_state_offset = q->ctl_stack_size;
 	}
 
+	/*
+	 * coherent_aql_mtype (offset 509): program 0 when the driver maps local
+	 * or remote memory as MTYPE_NC, and 1 in all other cases.
+	 */
+	m->coherent_aql_mtype = gmc_v12_1_get_coherent_aql_mtype(mm->dev->adev);
+
 	*mqd = m;
 	if (gart_addr)
 		*gart_addr = addr;
drivers/gpu/drm/amd/include/v12_structs.h +1 -1
index 28f4b01326811..7fa0c07b51c67 100644
@@ -1696,7 +1696,7 @@ struct v12_1_compute_mqd {
     uint32_t reserved_506; // offset: 506  (0x1FA)
     uint32_t reserved_507; // offset: 507  (0x1FB)
     uint32_t reserved_508; // offset: 508  (0x1FC)
-    uint32_t reserved_509; // offset: 509  (0x1FD)
+    uint32_t coherent_aql_mtype; // offset: 509  (0x1FD)
     uint32_t reserved_510; // offset: 510  (0x1FE)
     uint32_t reserved_511; // offset: 511  (0x1FF)
     uint32_t reserved_512; // offset: 512  (0x200)
Signature · 1 line
2.55.0
author maintainer
#75 lore raw

PATCH v2 75/95 drm/amdgpu: Separate out ualink init sequences

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Separate out sw_init and hw_init part of ualink. ualink init errors are ignored temporarily. Keep ualink sysfs init together with sysfs init sequence of other IPs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  10 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 329 +++++++++++----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  16 +-
 drivers/gpu/drm/amd/amdgpu/soc_v1_0.c      |  23 --
 4 files changed, 202 insertions(+), 176 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +10 -0
index 18cfa2dbcc2dc..02c4d6956d6e2 100644
@@ -2459,6 +2459,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 		}
 	}
 
+	amdgpu_ualink_mgr_sw_init(adev);
+
 	if (amdgpu_sriov_vf(adev))
 		amdgpu_virt_init_data_exchange(adev);
 
@@ -2485,6 +2487,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
+	amdgpu_ualink_mgr_hw_init(adev);
+
 	/*
 	 * retired pages will be loaded from eeprom and reserved here,
 	 * it should be called after amdgpu_device_ip_hw_init_phase2  since
@@ -2768,6 +2772,8 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
 		adev->ip_blocks[i].status.late_initialized = true;
 	}
 
+	amdgpu_ualink_mgr_late_init(adev);
+
 	r = amdgpu_ras_late_init(adev);
 	if (r) {
 		dev_err(adev->dev, "amdgpu_ras_late_init failed %d", r);
@@ -2960,6 +2966,8 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev)
 
 	amdgpu_amdkfd_device_fini_sw(adev);
 
+	amdgpu_ualink_mgr_sw_fini(adev);
+
 	for (i = adev->num_ip_blocks - 1; i >= 0; i--) {
 		if (!adev->ip_blocks[i].status.sw)
 			continue;
@@ -3729,6 +3737,7 @@ static int amdgpu_device_sys_interface_init(struct amdgpu_device *adev)
 	amdgpu_xcp_sysfs_init(adev);
 	amdgpu_uma_sysfs_init(adev);
 	amdgpu_ptl_sysfs_init(adev);
+	amdgpu_ualink_sysfs_init(adev);
 
 	return r;
 }
@@ -3750,6 +3759,7 @@ static void amdgpu_device_sys_interface_fini(struct amdgpu_device *adev)
 	amdgpu_xcp_sysfs_fini(adev);
 	amdgpu_uma_sysfs_fini(adev);
 	amdgpu_ptl_sysfs_fini(adev);
+	amdgpu_ualink_sysfs_fini(adev);
 }
 
 static bool
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +178 -151

430 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +14 -2
index d4d4ca0199dc7..90116ed532d16 100644
@@ -113,6 +113,13 @@ enum amdgpu_ualink_addr_mode {
 	AMDGPU_UALINK_ADDR_MODE_MAX
 };
 
+enum amdgpu_ualink_mgr_state {
+	AMDGPU_UALINK_INIT_NONE = 0,
+	AMDGPU_UALINK_INIT_HW,
+	AMDGPU_UALINK_INIT_COMPLETE,
+	AMDGPU_UALINK_INIT_ERROR
+};
+
 /* Physical pod info shared between query and setup API */
 struct amdgpu_ualink_ppod_info {
 	u32 accel_id;
@@ -299,6 +306,8 @@ struct amdgpu_ualink_mgr {
 	struct amdgpu_ualink_ppod_setup *setup;
 	struct amdgpu_ualink_vpod_config *config;
 	struct amdgpu_ualink_station_config *stations;
+	bool sysfs_init;
+	enum amdgpu_ualink_mgr_state mgr_state;
 
 	/* For remote interrupt and shootdown */
 	struct amdgpu_ualink_remote *remote;
@@ -366,10 +375,13 @@ void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
 
 int ualink_send_hello(struct amdgpu_device *adev, u32 remote_accel_id);
 
+int amdgpu_ualink_mgr_sw_init(struct amdgpu_device *adev);
+void amdgpu_ualink_mgr_sw_fini(struct amdgpu_device *adev);
+int amdgpu_ualink_mgr_hw_init(struct amdgpu_device *adev);
+int amdgpu_ualink_mgr_late_init(struct amdgpu_device *adev);
+
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
-int amdgpu_ualink_init(struct amdgpu_device *adev);
-void amdgpu_ualink_fini(struct amdgpu_device *adev);
 int amdgpu_ualink_manager_start(struct amdgpu_device *adev);
 void amdgpu_ualink_manager_stop(struct amdgpu_device *adev);
 int amdgpu_ualink_export_handle(struct drm_device *dev, struct drm_file *filp,
drivers/gpu/drm/amd/amdgpu/soc_v1_0.c +0 -23
index 84cdb09f731e5..757da0a30fdc4 100644
@@ -439,25 +439,6 @@ static int soc_v1_0_common_late_init(struct amdgpu_ip_block *ip_block)
 	 */
 	adev->nbio.funcs->enable_doorbell_selfring_aperture(adev, true);
 
-	/* Depends on PSP being initialized */
-	amdgpu_ualink_init(adev);
-
-	return 0;
-}
-
-static int soc_v1_0_common_sw_init(struct amdgpu_ip_block *ip_block)
-{
-	struct amdgpu_device *adev = ip_block->adev;
-
-	return amdgpu_ualink_sysfs_init(adev);
-}
-
-static int soc_v1_0_common_sw_fini(struct amdgpu_ip_block *ip_block)
-{
-	struct amdgpu_device *adev = ip_block->adev;
-
-	amdgpu_ualink_sysfs_fini(adev);
-
 	return 0;
 }
 
@@ -475,8 +456,6 @@ static int soc_v1_0_common_hw_fini(struct amdgpu_ip_block *ip_block)
 {
 	struct amdgpu_device *adev = ip_block->adev;
 
-	amdgpu_ualink_fini(adev);
-
 	adev->nbio.funcs->enable_doorbell_aperture(adev, false);
 	adev->nbio.funcs->enable_doorbell_selfring_aperture(adev, false);
 
@@ -515,8 +494,6 @@ static const struct amd_ip_funcs soc_v1_0_common_ip_funcs = {
 	.name = "soc_v1_0_common",
 	.early_init = soc_v1_0_common_early_init,
 	.late_init = soc_v1_0_common_late_init,
-	.sw_init = soc_v1_0_common_sw_init,
-	.sw_fini = soc_v1_0_common_sw_fini,
 	.hw_init = soc_v1_0_common_hw_init,
 	.hw_fini = soc_v1_0_common_hw_fini,
 	.suspend = soc_v1_0_common_suspend,
Signature · 1 line
2.55.0
author maintainer
#76 lore raw

PATCH v2 76/95 drm/amdgpu: Add ualink as separate ip block

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Manage ualink lifecycle as a separate IP block. For now, use GC version to add specific ualink ip block.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    |  8 ----
 drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c | 21 +++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    | 44 +++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h    |  6 +--
 drivers/gpu/drm/amd/include/amd_shared.h      |  2 +
 5 files changed, 50 insertions(+), 31 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -8
index 02c4d6956d6e2..44091ac12fa9c 100644
@@ -2459,8 +2459,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 		}
 	}
 
-	amdgpu_ualink_mgr_sw_init(adev);
-
 	if (amdgpu_sriov_vf(adev))
 		amdgpu_virt_init_data_exchange(adev);
 
@@ -2487,8 +2485,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	amdgpu_ualink_mgr_hw_init(adev);
-
 	/*
 	 * retired pages will be loaded from eeprom and reserved here,
 	 * it should be called after amdgpu_device_ip_hw_init_phase2  since
@@ -2772,8 +2768,6 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
 		adev->ip_blocks[i].status.late_initialized = true;
 	}
 
-	amdgpu_ualink_mgr_late_init(adev);
-
 	r = amdgpu_ras_late_init(adev);
 	if (r) {
 		dev_err(adev->dev, "amdgpu_ras_late_init failed %d", r);
@@ -2966,8 +2960,6 @@ static int amdgpu_device_ip_fini(struct amdgpu_device *adev)
 
 	amdgpu_amdkfd_device_fini_sw(adev);
 
-	amdgpu_ualink_mgr_sw_fini(adev);
-
 	for (i = adev->num_ip_blocks - 1; i >= 0; i--) {
 		if (!adev->ip_blocks[i].status.sw)
 			continue;
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +21 -0
index 81c3cdba4caf5..7710a8ed3d44d 100644
@@ -26,6 +26,7 @@
 
 #include "amdgpu.h"
 #include "amdgpu_discovery.h"
+#include "amdgpu_ualink.h"
 #include "soc15_hw_ip.h"
 #include "discovery.h"
 #include "amdgpu_ras.h"
@@ -3359,6 +3360,22 @@ static int amdgpu_discovery_set_isp_ip_blocks(struct amdgpu_device *adev)
 	return 0;
 }
 
+static int amdgpu_discovery_set_ualink_ip_blocks(struct amdgpu_device *adev)
+{
+	/* TODO: no dedicated IP discovery version for UALink yet; key off the
+	 * GC IP version for now.
+	 */
+	switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
+	case IP_VERSION(12, 1, 0):
+		amdgpu_device_ip_block_add(adev, &ualink_v1_0_ip_block);
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
 int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev)
 {
 	int r;
@@ -4008,6 +4025,10 @@ int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev)
 		return r;
 
 	r = amdgpu_discovery_set_isp_ip_blocks(adev);
+	if (r)
+		return r;
+
+	r = amdgpu_discovery_set_ualink_ip_blocks(adev);
 	if (r)
 		return r;
 	return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +25 -19

88 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +2 -4
index 90116ed532d16..64ebdf943aa72 100644
@@ -375,10 +375,6 @@ void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
 
 int ualink_send_hello(struct amdgpu_device *adev, u32 remote_accel_id);
 
-int amdgpu_ualink_mgr_sw_init(struct amdgpu_device *adev);
-void amdgpu_ualink_mgr_sw_fini(struct amdgpu_device *adev);
-int amdgpu_ualink_mgr_hw_init(struct amdgpu_device *adev);
-int amdgpu_ualink_mgr_late_init(struct amdgpu_device *adev);
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
@@ -391,4 +387,6 @@ int amdgpu_ualink_import_handle(struct drm_device *dev,
 				const struct amdgpu_ualink_handle *ualink_handle,
 				int *fd_out);
 void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo);
+
+extern const struct amdgpu_ip_block_version ualink_v1_0_ip_block;
 #endif
drivers/gpu/drm/amd/include/amd_shared.h +2 -0
index 7e29fe312ac69..e81ce9d790097 100644
@@ -90,6 +90,7 @@ enum amd_apu_flags {
 * @AMD_IP_BLOCK_TYPE_UMSCH_MM: User Mode Scheduler for Multimedia
 * @AMD_IP_BLOCK_TYPE_ISP: Image Signal Processor
 * @AMD_IP_BLOCK_TYPE_RAS: Reliability, Availability, Serviceability
+* @AMD_IP_BLOCK_TYPE_UALINK: Ultra Accelerator Link
 * @AMD_IP_BLOCK_TYPE_NUM: Total number of IP block types
 */
 enum amd_ip_block_type {
@@ -111,6 +112,7 @@ enum amd_ip_block_type {
 	AMD_IP_BLOCK_TYPE_UMSCH_MM,
 	AMD_IP_BLOCK_TYPE_ISP,
 	AMD_IP_BLOCK_TYPE_RAS,
+	AMD_IP_BLOCK_TYPE_UALINK,
 	AMD_IP_BLOCK_TYPE_NUM,
 };
 
Signature · 1 line
2.55.0
author maintainer
#77 lore raw

PATCH v2 77/95 drm/admgpu: Seggregate ualink nht messaging

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Message handshake with NHT firmware is IP specific. Separate that into IP specific file. Move ip block v1.0 definition also the version specific file.

---
 drivers/gpu/drm/amd/amdgpu/Makefile        |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 112 ++-----------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  20 ++++
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c   | 122 +++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h   |  30 +++++
 5 files changed, 185 insertions(+), 101 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c
 create mode 100644 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h
drivers/gpu/drm/amd/amdgpu/Makefile +1 -1
index bb33119608a06..7b08814a24da4 100644
@@ -279,7 +279,7 @@ amdgpu-y += amdgpu_amdkfd.o
 amdgpu-y += amdgpu_userq.o
 
 # add UALink manager support
-amdgpu-y += amdgpu_ualink.o
+amdgpu-y += amdgpu_ualink.o ualink_v1_0.o
 
 ifneq ($(CONFIG_HSA_AMD),)
 AMDKFD_PATH := ../amdkfd
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +12 -100

178 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +20 -0
index 64ebdf943aa72..d4e727c9f98a0 100644
@@ -35,6 +35,13 @@ struct amdgpu_device;
 #define AMDGPU_UALINK_LOCAL_ACCELS_MAX 8
 #define AMDGPU_UALINK_STATIONS_MAX 64
 
+/* nHT firmware status */
+#define AMDGPU_NHT_FW_ST_PREINIT	0xA0
+#define AMDGPU_NHT_FW_ST_READY	0xA1
+#define AMDGPU_NHT_FW_ST_HALT	0xA2
+#define AMDGPU_NHT_FW_ST_ERROR	0xA3
+#define AMDGPU_NHT_FW_ST_FATAL	0xF0
+
 #define AMDGPU_UALINK_RESP_TIMEOUT			5000 /* 5s timeout */
 
 #define AMDGPU_UALINK_HANDLE_ACCID_MASK			GENMASK_ULL(9, 0)
@@ -299,9 +306,17 @@ struct amdgpu_ualink_connection {
 
 struct amdgpu_ualink_remote;
 
+struct amdgpu_ualink_msg_ctl {
+	u32 (*check_status)(struct amdgpu_device *adev);
+	int (*send_metadata)(struct amdgpu_device *adev, u64 metadata_gpu_addr,
+			     u32 accel_data);
+	int (*send_halt)(struct amdgpu_device *adev);
+};
+
 struct amdgpu_ualink_mgr {
 	u64 npa_size;
 	u32 psp_if_ver;
+	const struct amdgpu_ualink_msg_ctl *msg_ctl;
 	struct amdgpu_ualink_info *info;
 	struct amdgpu_ualink_ppod_setup *setup;
 	struct amdgpu_ualink_vpod_config *config;
@@ -388,5 +403,10 @@ int amdgpu_ualink_import_handle(struct drm_device *dev,
 				int *fd_out);
 void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo);
 
+int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block);
+int ualink_ip_late_init(struct amdgpu_ip_block *ip_block);
+int ualink_ip_sw_init(struct amdgpu_ip_block *ip_block);
+int ualink_ip_sw_fini(struct amdgpu_ip_block *ip_block);
+
 extern const struct amdgpu_ip_block_version ualink_v1_0_ip_block;
 #endif
drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c new file +122 -0

123 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/ualink_v1_0.h new file +30 -0
new file mode 100644
index 0000000000000..d794b0e3e800e
@@ -0,0 +1,30 @@
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+#ifndef __UALINK_V1_0_H__
+#define __UALINK_V1_0_H__
+
+#include "amdgpu_ualink.h"
+
+extern const struct amdgpu_ualink_msg_ctl ualink_v1_0_msg_ctl;
+
+#endif
Signature · 1 line
2.55.0
author maintainer
#78 lore raw

PATCH v2 78/95 drm/amdgpu: Assign accel state based on ASP config

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Add explicit PPOD/VPOD states for clarity. Assign the state accordingly as received from ASP during initialization.

UALink accel state transitions:

UNCONFIGURED -> PPOD_CONFIGURED -> VPOD_CONFIGURED -> READY -> ACTIVE

- PPOD: sysfs ppod commit/ ASP {UAL_CFG_PPOD}
- VPOD: sysfs vpod commit/ ASP {UAL_CFG_VPOD} | {UAL_CFG_STATION}
- READY: local vpod integrity + activate_accelerator()
- ACTIVE: all local gpus belonging to one vpod are configured and ready.

deactivate_accelerator(): READY -> PPOD_CONFIGURED invalid vpod: -> ERROR

Query ualink info during late init phase. Add a local vpod integrity check before activating devices in the vpod. Move the devices to active state once integrity check passes.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c    |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h    |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 346 +++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   3 +-
 4 files changed, 293 insertions(+), 65 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +5 -1
index b8def8b7e1426..4fc7aed58bfc3 100644
@@ -1223,7 +1223,8 @@ int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver)
 }
 
 int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
-		       struct amdgpu_ualink_info *info)
+		       struct amdgpu_ualink_info *info,
+		       enum psp_gfx_ual_config_state *cfg_state)
 {
 	struct psp_gfx_get_config_ual_v1 *ual_config;
 	struct psp_gfx_cmd_resp *cmd;
@@ -1269,6 +1270,9 @@ int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
 		if (AMDGPU_UALINK_ACCEL_MAX > PSP_GFX_UAL_MAX_ACC_BIT_MASK*32)
 			bitmap_clear(info->vpod.active_accel_bits, PSP_GFX_UAL_MAX_ACC_BIT_MASK*32,
 				     AMDGPU_UALINK_ACCEL_MAX - PSP_GFX_UAL_MAX_ACC_BIT_MASK*32);
+
+		if (cfg_state)
+			*cfg_state = ual_config->config_state;
 	} else if (!ret) {
 		ret = -EINVAL;
 	}
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +2 -1
index bb84f3a3dc54b..2ba7ced36b7cc 100644
@@ -715,7 +715,8 @@ struct amdgpu_ualink_station_config;
 int psp_ual_get_interface_version(struct psp_context *psp, uint32_t *intf_ver);
 
 int psp_ual_query_info(struct psp_context *psp, uint32_t intf_ver,
-		       struct amdgpu_ualink_info *info);
+		       struct amdgpu_ualink_info *info,
+		       enum psp_gfx_ual_config_state *cfg_state);
 
 int psp_ual_set_ppod_config(struct psp_context *psp, uint32_t intf_ver,
 			    const struct amdgpu_ualink_ppod_setup *setup);
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +284 -62

475 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +2 -1
index d4e727c9f98a0..6f522bcafa442 100644
@@ -107,7 +107,8 @@ enum amdgpu_ualink_type {
 
 enum amdgpu_ualink_accel_state {
 	AMDGPU_UALINK_ACCEL_STATE_UNCONFIGURED = 0,
-	AMDGPU_UALINK_ACCEL_STATE_CONFIGURED,
+	AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED,
+	AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED,
 	AMDGPU_UALINK_ACCEL_STATE_READY,
 	AMDGPU_UALINK_ACCEL_STATE_ACTIVE,
 	AMDGPU_UALINK_ACCEL_STATE_ERROR,
Signature · 1 line
2.55.0
author maintainer
#79 lore raw

PATCH v2 79/95 drm/amdgpu: Drop duplicate vpod check functions

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Use __check_local_vpod_integrity and __check_vpod_info for vpod validation checks.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 127 +--------------------
 1 file changed, 4 insertions(+), 123 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +4 -123
index 3cae3f49bb31f..5efee75faddb2 100644
@@ -736,128 +736,6 @@ static bool amdgpu_ualink_is_local_accel(struct amdgpu_device *adev,
 	return false;
 }
 
-#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
-static bool check_vpod_info(struct amdgpu_device *adev,
-			    const struct amdgpu_ualink_info *info)
-{
-	if (info->accel_state < AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED) {
-		dev_dbg(adev->dev, "pPod is not yet configured\n");
-		return false;
-	}
-	return __check_vpod_info(adev, info);
-}
-
-static bool check_local_vpod_integrity(struct amdgpu_device *adev)
-{
-	struct amdgpu_ualink_info *info = adev->ualink.info;
-	struct amdgpu_ualink_info *peer_info;
-	struct amdgpu_device *peer_adev;
-	unsigned int accel_id;
-	unsigned int i;
-
-	if (!check_vpod_info(adev, info))
-		return false;
-
-	/* Check that all local accelerators listed in vpod_active_accels have
-	 * matching pod IDs
-	 */
-	for_each_set_bit(accel_id, info->vpod.active_accel_bits, AMDGPU_UALINK_ACCEL_MAX) {
-
-		if (accel_id == info->ppod.accel_id)
-			continue;
-
-		peer_adev = find_peer_adev(accel_id);
-		if (!peer_adev)
-			continue;
-		peer_info = peer_adev->ualink.info;
-
-		if (peer_info->vpod.id != info->vpod.id) {
-			dev_dbg(adev->dev, "Peer %u vpod_id doesn't match: %u != %u",
-				accel_id, peer_info->vpod.id, info->vpod.id);
-			return false;
-		}
-		if (!uuid_equal(&peer_info->ppod.id, &info->ppod.id)) {
-			dev_dbg(adev->dev, "Peer %u ppod_id doesn't match: %pU != %pU",
-				accel_id, &peer_info->ppod.id, &info->ppod.id);
-			return false;
-		}
-	}
-
-	/* Derive local accels from pod IDs of GPUs in mgpu_info */
-	info->n_local_accels = 0;
-	for (i = 0; i < mgpu_info.num_gpu &&
-		    info->n_local_accels < AMDGPU_UALINK_LOCAL_ACCELS_MAX;
-	     i++) {
-		peer_adev = mgpu_info.gpu_ins[i].adev;
-		peer_info = peer_adev->ualink.info;
-
-		if (peer_adev == adev ||
-		    (peer_info && peer_info->vpod.id == info->vpod.id &&
-		     uuid_equal(&peer_info->ppod.id, &info->ppod.id)))
-			info->local_accels[info->n_local_accels++] =
-				peer_info->ppod.accel_id;
-	}
-
-	/* Then check consistency of the vpod information on all those GPUs */
-	for (i = 0; i < info->n_local_accels; i++) {
-		unsigned int j;
-
-		for (j = i + 1; j < info->n_local_accels; j++) {
-			if (info->local_accels[j] == accel_id) {
-				dev_dbg(adev->dev,
-					"Accelerator ID %u is not unique among local GPUs\n",
-					accel_id);
-				return false;
-			}
-		}
-
-		accel_id = info->local_accels[i];
-
-		/* Skip this GPU, we are looking for our peers */
-		if (accel_id == info->ppod.accel_id)
-			continue;
-
-		peer_adev = find_peer_adev(accel_id);
-		if (WARN_ON(!peer_adev || !peer_adev->ualink.info))
-			/* info->local_accels we just built is corrupted? */
-			return false;
-		peer_info = peer_adev->ualink.info;
-
-		/* Check peer vpod info and consistency */
-		if (!check_vpod_info(peer_adev, peer_info))
-			return false;
-
-		if (peer_info->ppod.size != info->ppod.size) {
-			dev_dbg(adev->dev, "Peer %u ppod_size doesn't match: %u != %u\n",
-				accel_id, peer_info->ppod.size, info->ppod.size);
-			return false;
-		}
-		if (peer_info->vpod.size != info->vpod.size) {
-			dev_dbg(adev->dev, "Peer %u vpod_size doesn't match: %u != %u\n",
-				accel_id, peer_info->vpod.size, info->vpod.size);
-			return false;
-		}
-		if (peer_info->vpod.addr_mode != info->vpod.addr_mode) {
-			dev_dbg(adev->dev, "Peer %u addr_mode doesn't match: %u != %u\n",
-				accel_id, peer_info->vpod.addr_mode, info->vpod.addr_mode);
-			return false;
-		}
-		if (!bitmap_equal(peer_info->vpod.active_accel_bits,
-				  info->vpod.active_accel_bits, AMDGPU_UALINK_ACCEL_MAX)) {
-			dev_dbg(adev->dev, "Peer %u vpod_active_accels don't match\n",
-				accel_id);
-			return false;
-		}
-
-		/* Update peer's local accelerator array */
-		peer_info->n_local_accels = info->n_local_accels;
-		memcpy(peer_info->local_accels, info->local_accels,
-		       sizeof(info->local_accels));
-	}
-	return true;
-}
-#endif
-
 static void activate_accelerator(struct amdgpu_device *adev)
 {
 	int r;
@@ -1096,6 +974,8 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	if (r)
 		return r;
 
+	if (!__check_vpod_info(adev, info))
+		return -EINVAL;
 	/* The integrity check makes sure each new GPU is consistent with the
 	 * other GPUs already in the vPod. All known local GPUs can become
 	 * "ready" at the same time.
@@ -1104,7 +984,8 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	 * already in the vPod.
 	 */
 	mutex_lock(&mgpu_info.mutex);
-	if (check_local_vpod_integrity(adev))
+	r = __check_local_vpod_integrity(adev);
+	if (!r)
 		activate_local_vpod(adev);
 	else if (info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED)
 		deactivate_accelerator(adev);
Signature · 1 line
2.55.0
author maintainer
#80 lore raw

PATCH v2 80/95 drm/amdgpu: Add support to send ASP completion

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Add interface to send completion command to ASPs after processing ualink notifications.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 28 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h |  3 +++
 2 files changed, 31 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +28 -0
index 4fc7aed58bfc3..cb34148269160 100644
@@ -1426,6 +1426,34 @@ int psp_ual_set_npa_config(struct psp_context *psp, uint32_t intf_ver,
 	return ret;
 }
 
+int psp_ual_send_completion(struct psp_context *psp, uint32_t intf_ver,
+			    uint32_t cmd_id, uint32_t status)
+{
+	struct psp_gfx_cmd_resp *cmd;
+	int ret;
+
+	/* TBD check interface version 1.x */
+	if (intf_ver > 0x1ffff) {
+		pr_warn("PSP UAL interface version mismatch: 0x%x\n", intf_ver);
+		return -EOPNOTSUPP;
+	}
+
+	cmd = acquire_psp_cmd_buf(psp);
+
+	cmd->cmd_id = GFX_CMD_ID_UAL_SEND_COMPLETION;
+	cmd->cmd.cmd_send_completion_ual.cmd_id = cmd_id;
+	cmd->cmd.cmd_send_completion_ual.status = status;
+
+	ret = psp_cmd_submit_buf(psp, NULL, cmd, psp->fence_buf_mc_addr);
+
+	if (!ret && cmd->resp.status)
+		ret = -EINVAL;
+
+	release_psp_cmd_buf(psp);
+
+	return ret;
+}
+
 int psp_update_fw_reservation(struct psp_context *psp)
 {
 	int ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +3 -0
index 2ba7ced36b7cc..35839f42b8a08 100644
@@ -730,4 +730,7 @@ int psp_ual_set_station_config(struct psp_context *psp, uint32_t intf_ver,
 int psp_ual_set_npa_config(struct psp_context *psp, uint32_t intf_ver,
 			   unsigned int vmid, bool enable);
 
+int psp_ual_send_completion(struct psp_context *psp, uint32_t intf_ver,
+			    uint32_t cmd_id, uint32_t status);
+
 #endif
Signature · 1 line
2.55.0
author maintainer
#81 lore raw

PATCH v2 81/95 drm/amdgpu: Add handlers for ualink notifications

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Add handlers for ualink notifications from ASP. Pause/Resume notifications are not handled currently.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 64 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  3 +
 drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h    | 34 ++++++++++--
 drivers/gpu/drm/amd/amdgpu/psp_v15_0_8.c   | 48 +++++++++++++++-
 4 files changed, 143 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +64 -0
index 5efee75faddb2..de79b2ea1f99a 100644
@@ -204,6 +204,70 @@ static int amdgpu_ualink_query_info(struct amdgpu_device *adev)
 	return 0;
 }
 
+int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev)
+{
+	int r;
+	u32 status = 0;
+
+	/* TBD: Stop ASP interrupts if driver faced an issue */
+	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
+		dev_dbg(adev->dev,
+			"UALink not initialized, skipping config update\n");
+		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
+		goto out;
+	}
+
+	/*TBD: find the right value of status to be sent to ASP*/
+	r = amdgpu_ualink_query_info(adev);
+	if (r) {
+		dev_info(adev->dev, "UALink config update failed %d\n", r);
+		status = 1;
+	}
+
+out:
+	return psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
+				       PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID,
+				       status);
+}
+
+int amdgpu_ualink_pause_handler(struct amdgpu_device *adev)
+{
+	u32 status = 0;
+
+	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
+		dev_dbg(adev->dev,
+			"UALink not initialized, skipping pause update\n");
+		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
+		goto out;
+	}
+
+	dev_dbg(adev->dev, "UALink pause command is not handled\n");
+
+out:
+	return psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
+				       PSP_GFX_INT_CTXT_UAL_CMD_PAUSE_ID,
+				       status);
+}
+
+int amdgpu_ualink_resume_handler(struct amdgpu_device *adev)
+{
+	u32 status = 0;
+
+	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
+		dev_dbg(adev->dev,
+			"UALink not initialized, skipping pause update\n");
+		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
+		goto out;
+	}
+
+	dev_dbg(adev->dev, "UALink resume command is not handled\n");
+
+out:
+	return psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
+				       PSP_GFX_INT_CTXT_UAL_CMD_RESUME_ID,
+				       status);
+}
+
 int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block)
 {
 	struct amdgpu_device *adev = ip_block->adev;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index 6f522bcafa442..ce8b643811dfa 100644
@@ -391,6 +391,9 @@ void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
 
 int ualink_send_hello(struct amdgpu_device *adev, u32 remote_accel_id);
 
+int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev);
+int amdgpu_ualink_pause_handler(struct amdgpu_device *adev);
+int amdgpu_ualink_resume_handler(struct amdgpu_device *adev);
 
 int amdgpu_ualink_sysfs_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sysfs_fini(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/psp_gfx_if.h +29 -5
index 77be21c3eada3..0803cfc445dc7 100644
@@ -400,11 +400,35 @@ struct psp_gfx_cmd_req_perf_hw {
 #define PSP_GFX_UAL_MAX_STATIONS_V1     64       /* max UALink stations */
 #define PSP_GFX_UAL_MAX_ACC_BIT_MASK    32       /* max accelerator bit mask 32*32 */
 
-enum psp_gfx_ual_npa_address_mode
-{
-    PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_ALIASING = 0,
-    PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_IDENTIFICATION = 1,
-    PSP_GFX_UAL_NPA_ADDRESS_MODE_MAX
+/* Interrupt category identifiers (upper byte) - ASP interrupts to Driver */
+
+/* Commands from ASP */
+#define PSP_GFX_INT_CTXT_UAL_CAT_CMD 0x01000000
+/* Mask for Command bits */
+#define PSP_GFX_INT_CTXT_UAL_CAT_CMD_MASK 0x000000FF
+/* Notifications from ASP */
+#define PSP_GFX_INT_CTXT_UAL_CAT_NOTIFY 0x02000000
+/* Mask for category bits */
+#define PSP_GFX_INT_CTXT_UAL_CAT_MASK 0xFF000000
+
+#define PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID 0x00000001
+#define PSP_GFX_INT_CTXT_UAL_CMD_PAUSE_ID 0x00000002
+#define PSP_GFX_INT_CTXT_UAL_CMD_RESUME_ID 0x00000003
+
+/* Command interrupts from ASP (GIM must execute action) */
+#define PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE \
+	(PSP_GFX_INT_CTXT_UAL_CAT_CMD | PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID)
+
+#define PSP_GFX_INT_CTXT_UAL_CMD_PAUSE \
+	(PSP_GFX_INT_CTXT_UAL_CAT_CMD | PSP_GFX_INT_CTXT_UAL_CMD_PAUSE_ID)
+
+#define PSP_GFX_INT_CTXT_UAL_CMD_RESUME \
+	(PSP_GFX_INT_CTXT_UAL_CAT_CMD | PSP_GFX_INT_CTXT_UAL_CMD_RESUME_ID)
+
+enum psp_gfx_ual_npa_address_mode {
+	PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_ALIASING = 0,
+	PSP_GFX_UAL_NPA_ADDRESS_MODE_SOURCE_IDENTIFICATION = 1,
+	PSP_GFX_UAL_NPA_ADDRESS_MODE_MAX
 };
 
 enum psp_gfx_ual_ports_per_station
drivers/gpu/drm/amd/amdgpu/psp_v15_0_8.c +47 -1

60 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#82 lore raw

PATCH v2 82/95 drm/amdgpu: Improve ualink state transitions

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Keep state transitions under lock. Validate the ppod/vpod config or both based on the state passed by ASP. When a config update is received, if the GPU is already active on a vpod, local vpod gpu integrity check is skipped to keep minimal disruption. A gpu removed from the vpod will get the new vpod id as 0. A GPU is not expected to transition directly from a valid/nonzero vpod id to another valid vpod id. It needs to be removed from the existing vpod first.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 199 +++++++++++++++------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |   3 +
 2 files changed, 148 insertions(+), 54 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +145 -54
index de79b2ea1f99a..8604a87f48bfa 100644
@@ -34,7 +34,7 @@
 #include <linux/sysfs.h>
 #include <linux/string.h>
 
-static void amdgpu_ualink_activate_vpod(struct amdgpu_device *adev);
+static void deactivate_accelerator(struct amdgpu_device *adev);
 static int amdgpu_ualink_remote_interrupt(struct amdgpu_device *adev,
 				u32 remote_accel_id, u32 dw0, u32 dw1,
 				u32 dw2, u32 dw3);
@@ -53,6 +53,7 @@ static void amdgpu_ualink_invalidate_import_mappings(struct amdgpu_bo *bo);
 static int amdgpu_ualink_remote_shootdown(struct amdgpu_device *adev,
 					  u32 remote_accel_id, u64 addr,
 					  u32 size_in_pages, u32 flush_type);
+static void __amdgpu_ualink_activate_vpod_locked(struct amdgpu_device *adev);
 
 #define STRIP_NPA(addr)						\
 	(((u64)(addr) & ~AMDGPU_UALINK_NPA_ADDR_GPUID_MASK))
@@ -156,78 +157,166 @@ amdgpu_ualink_info_set_accel_state(struct amdgpu_device *adev,
 				   struct amdgpu_ualink_info *info,
 				   enum psp_gfx_ual_config_state cfg_state)
 {
+	enum amdgpu_ualink_accel_state cur = info->accel_state;
+	enum amdgpu_ualink_accel_state target;
+	bool ppod_validated;
+	bool vpod_validated;
+
 	if (!info)
 		return;
 
+	ppod_validated = cur >= AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED &&
+			 cur <= AMDGPU_UALINK_ACCEL_STATE_ACTIVE;
+	vpod_validated = cur >= AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED &&
+			 cur <= AMDGPU_UALINK_ACCEL_STATE_ACTIVE;
+
 	switch (cfg_state) {
 	case UAL_CFG_IDLE:
-		break;
+		return;
 	case UAL_CFG_PPOD:
-		if (!__check_ppod_info(adev, info)) {
-			info->accel_state =
-				AMDGPU_UALINK_ACCEL_STATE_UNCONFIGURED;
-			break;
-		}
-		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED;
+		target = AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED;
 		break;
 	case UAL_CFG_VPOD:
 	case UAL_CFG_STATION:
-		if (!__check_vpod_info(adev, info)) {
-			info->accel_state = AMDGPU_UALINK_ACCEL_STATE_ERROR;
-			dev_err(adev->dev,
-				"vpod configuration is invalid, setting to error state");
-			break;
-		}
-		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED;
+		target = AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED;
 		break;
 	case UAL_CFG_COMPLETE:
-		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
+		target = AMDGPU_UALINK_ACCEL_STATE_READY;
 		break;
 	default:
 		dev_dbg(adev->dev, "invalid configuration state %u", cfg_state);
-		break;
+		return;
 	}
-}
 
-static int amdgpu_ualink_query_info(struct amdgpu_device *adev)
-{
-	enum psp_gfx_ual_config_state cfg_state;
-	int r;
+	/* ppod stage: should be part of a ppod first */
+	if (!ppod_validated && !__check_ppod_info(adev, info)) {
+		if (target == AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED) {
+			info->accel_state =
+				AMDGPU_UALINK_ACCEL_STATE_UNCONFIGURED;
+		} else {
+			info->accel_state = AMDGPU_UALINK_ACCEL_STATE_ERROR;
+			dev_err(adev->dev,
+				"ppod configuration is invalid, setting to error state");
+		}
+		return;
+	}
+	if (target == AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED) {
+		if (!ppod_validated)
+			info->accel_state =
+				AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED;
+		return;
+	}
 
-	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
-			       adev->ualink.info, &cfg_state);
-	if (r)
-		return r;
+	/* A vpod_id of 0 means the GPU is not part of any vPod */
+	if (info->vpod.id == AMDGPU_UALINK_VPOD_ID_INVALID) {
+		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED;
+		return;
+	}
 
-	amdgpu_ualink_info_set_accel_state(adev, adev->ualink.info, cfg_state);
+	/* vpod stage: required to reach vpod_configured or ready */
+	if (!vpod_validated && !__check_vpod_info(adev, info)) {
+		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_ERROR;
+		dev_err(adev->dev,
+			"vpod configuration is invalid, setting to error state");
+		return;
+	}
+	if (target == AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED) {
+		if (!vpod_validated)
+			info->accel_state =
+				AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED;
+		return;
+	}
+
+	/* complete stage: advance to ready unless already ready/active */
+	if (cur < AMDGPU_UALINK_ACCEL_STATE_READY ||
+	    cur > AMDGPU_UALINK_ACCEL_STATE_ACTIVE)
+		info->accel_state = AMDGPU_UALINK_ACCEL_STATE_READY;
+}
 
+static int amdgpu_ualink_update_vpod_config(struct amdgpu_device *adev)
+{
+	/* TBD: Do updates/cleanup based on updated vpod configuration */
 	return 0;
 }
 
 int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev)
 {
-	int r;
-	u32 status = 0;
+	enum amdgpu_ualink_accel_state prev_state;
+	enum psp_gfx_ual_config_state cfg_state;
+	u32 prev_vpod_id;
+	int r, qerr;
 
 	/* TBD: Stop ASP interrupts if driver faced an issue */
 	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
+		u32 status;
+
 		dev_dbg(adev->dev,
 			"UALink not initialized, skipping config update\n");
 		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
-		goto out;
+		return psp_ual_send_completion(
+			&adev->psp, adev->ualink.psp_if_ver,
+			PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, status);
 	}
 
+	prev_state = adev->ualink.info->accel_state;
+	prev_vpod_id = adev->ualink.info->vpod.id;
+
+	qerr = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
+				  adev->ualink.info, &cfg_state);
+
 	/*TBD: find the right value of status to be sent to ASP*/
-	r = amdgpu_ualink_query_info(adev);
-	if (r) {
-		dev_info(adev->dev, "UALink config update failed %d\n", r);
-		status = 1;
+	r = psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
+				    PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, 0);
+	if (r || qerr)
+		goto err;
+
+	/* If the device is already active and its vpod_id is unchanged, the
+	 * update does not affect vpod membership. Skip the local vpod
+	 * integrity check and re-activation.
+	 */
+	if (prev_state == AMDGPU_UALINK_ACCEL_STATE_ACTIVE &&
+	    adev->ualink.info->vpod.id == prev_vpod_id) {
+		amdgpu_ualink_update_vpod_config(adev);
+		return 0;
 	}
 
-out:
-	return psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
-				       PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID,
-				       status);
+	/* A new vpod_id of 0 means this GPU was removed from the vPod. */
+	if (adev->ualink.info->vpod.id == AMDGPU_UALINK_VPOD_ID_INVALID) {
+		amdgpu_ualink_update_vpod_config(adev);
+		scoped_guard(mutex, &mgpu_info.mutex)
+			deactivate_accelerator(adev);
+		return 0;
+	}
+
+	/* GPU joining a new vpod should be with invalid vpod id*/
+	scoped_guard(mutex, &mgpu_info.mutex) {
+		if (prev_vpod_id == AMDGPU_UALINK_VPOD_ID_INVALID) {
+			amdgpu_ualink_info_set_accel_state(
+				adev, adev->ualink.info, cfg_state);
+			__amdgpu_ualink_activate_vpod_locked(adev);
+		} else {
+			/* GPU should first get removal which will set invalid vpod_id
+			* and then join a new vpod
+			*/
+			dev_err(adev->dev,
+				"Invalid vpod transition from %u to %u\n",
+				prev_vpod_id, adev->ualink.info->vpod.id);
+			goto err;
+		}
+	}
+
+	return 0;
+
+err:
+	scoped_guard(mutex, &mgpu_info.mutex) {
+		deactivate_accelerator(adev);
+		adev->ualink.info->accel_state =
+			AMDGPU_UALINK_ACCEL_STATE_ERROR;
+	}
+	dev_err(adev->dev,
+		"UALink config update failed, setting to error state");
+
+	return r;
 }
 
 int amdgpu_ualink_pause_handler(struct amdgpu_device *adev)
@@ -280,9 +369,6 @@ int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block)
 	r = psp_ual_get_interface_version(&adev->psp, &adev->ualink.psp_if_ver);
 	if (r) {
 		adev->ualink.psp_if_ver = 0xffffffff;
-		dev_info(adev->dev,
-			 "UALink disabled, PSP interface version detection failed: %d\n",
-			 r);
 		goto disable;
 	}
 	dev_info(adev->dev, "Found UALink interface version 0x%x\n",
@@ -299,16 +385,22 @@ int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block)
 int ualink_ip_late_init(struct amdgpu_ip_block *ip_block)
 {
 	struct amdgpu_device *adev = ip_block->adev;
+	enum psp_gfx_ual_config_state cfg_state;
 	int r;
 
 	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_HW)
 		return 0;
 
-	r = amdgpu_ualink_query_info(adev);
+	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
+			       adev->ualink.info, &cfg_state);
 	if (r)
 		return r;
 
-	amdgpu_ualink_activate_vpod(adev);
+	scoped_guard(mutex, &mgpu_info.mutex) {
+		amdgpu_ualink_info_set_accel_state(adev, adev->ualink.info,
+						   cfg_state);
+		__amdgpu_ualink_activate_vpod_locked(adev);
+	}
 
 	r = amdgpu_ualink_drm_client_create(adev);
 	if (r) {
@@ -661,8 +753,6 @@ static const struct kobj_type ualink_info_ktype = {
 };
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
-static void deactivate_accelerator(struct amdgpu_device *adev);
-
 UALINK_VALUE_SHOW(ppod_setup, accel_id,  ppod.accel_id,  "%u");
 UALINK_VALUE_SHOW(ppod_setup, bandwidth, ppod.bandwidth, "%u");
 UALINK_VALUE_SHOW(ppod_setup, latency,   ppod.latency,   "%u");
@@ -831,7 +921,6 @@ static void activate_accelerator(struct amdgpu_device *adev)
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_ACTIVE;
 }
 
-#ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
 static void deactivate_accelerator(struct amdgpu_device *adev)
 {
 	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_ACTIVE)
@@ -847,7 +936,6 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 	amdgpu_ualink_sw_fini(adev);
 	amdgpu_ualink_manager_stop(adev);
 }
-#endif
 
 static void activate_local_vpod(struct amdgpu_device *adev)
 {
@@ -982,22 +1070,24 @@ static int __check_local_vpod_integrity(struct amdgpu_device *adev)
 	return 0;
 }
 
-static void amdgpu_ualink_activate_vpod(struct amdgpu_device *adev)
+static void __amdgpu_ualink_activate_vpod_locked(struct amdgpu_device *adev)
 {
 	int ret;
 
-	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_READY)
+	if (adev->ualink.info->accel_state <
+		    AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED ||
+	    adev->ualink.info->accel_state ==
+		    AMDGPU_UALINK_ACCEL_STATE_ERROR)
 		return;
-	mutex_lock(&mgpu_info.mutex);
+
 	ret = __check_local_vpod_integrity(adev);
 	if (ret && ret != -EAGAIN) {
-		dev_err(adev->dev, "Local vpod integrity check failed: %d\n",
-			ret);
+		dev_err(adev->dev,
+			"Local vpod integrity check failed: %d\n", ret);
 		return;
 	}
 	if (!ret)
 		activate_local_vpod(adev);
-	mutex_unlock(&mgpu_info.mutex);
 }
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
@@ -1242,7 +1332,7 @@ int ualink_ip_sw_init(struct amdgpu_ip_block *ip_block)
 	info->ppod.accel_id = 0xffffffff;
 	info->ppod.bandwidth = 0xffffffff;
 	info->ppod.latency = 0xffffffff;
-	info->vpod.id = 0xffffffff;
+	info->vpod.id = AMDGPU_UALINK_VPOD_ID_INVALID;
 	info->vpod.addr_mode = AMDGPU_UALINK_ADDR_MODE_MAX;
 
 	/*
@@ -5684,3 +5774,4 @@ int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev)
 			      UALINK_IH_SOURCE_ID, &adev->ualink.irq);
 	return r;
 }
+
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -0
index ce8b643811dfa..e23aaf8b313e1 100644
@@ -35,6 +35,9 @@ struct amdgpu_device;
 #define AMDGPU_UALINK_LOCAL_ACCELS_MAX 8
 #define AMDGPU_UALINK_STATIONS_MAX 64
 
+/* A vpod_id of 0 is reserved and treated as invalid/no vPod */
+#define AMDGPU_UALINK_VPOD_ID_INVALID 0
+
 /* nHT firmware status */
 #define AMDGPU_NHT_FW_ST_PREINIT	0xA0
 #define AMDGPU_NHT_FW_ST_READY	0xA1
Signature · 1 line
2.55.0
author maintainer
#83 lore raw

PATCH v2 83/95 drm/amdgpu: Use uniform logic for inband/sideband

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

For updating the vpod configuration, use similar logic in both sideband and inband paths.

In inband path, force the vpod id as invalid when ppod configuration is done. Remove the redundant query call after station configuration as it station configuration doesn't affect vpod configuration.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 109 +++++++++++----------
 1 file changed, 57 insertions(+), 52 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +57 -52
index 8604a87f48bfa..73aa9e98202b5 100644
@@ -239,37 +239,10 @@ static int amdgpu_ualink_update_vpod_config(struct amdgpu_device *adev)
 	return 0;
 }
 
-int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev)
+static int amdgpu_ualink_update_accel_state(
+	struct amdgpu_device *adev, enum amdgpu_ualink_accel_state prev_state,
+	u32 prev_vpod_id, enum psp_gfx_ual_config_state cfg_state)
 {
-	enum amdgpu_ualink_accel_state prev_state;
-	enum psp_gfx_ual_config_state cfg_state;
-	u32 prev_vpod_id;
-	int r, qerr;
-
-	/* TBD: Stop ASP interrupts if driver faced an issue */
-	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
-		u32 status;
-
-		dev_dbg(adev->dev,
-			"UALink not initialized, skipping config update\n");
-		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
-		return psp_ual_send_completion(
-			&adev->psp, adev->ualink.psp_if_ver,
-			PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, status);
-	}
-
-	prev_state = adev->ualink.info->accel_state;
-	prev_vpod_id = adev->ualink.info->vpod.id;
-
-	qerr = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
-				  adev->ualink.info, &cfg_state);
-
-	/*TBD: find the right value of status to be sent to ASP*/
-	r = psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
-				    PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, 0);
-	if (r || qerr)
-		goto err;
-
 	/* If the device is already active and its vpod_id is unchanged, the
 	 * update does not affect vpod membership. Skip the local vpod
 	 * integrity check and re-activation.
@@ -301,11 +274,50 @@ int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev)
 			dev_err(adev->dev,
 				"Invalid vpod transition from %u to %u\n",
 				prev_vpod_id, adev->ualink.info->vpod.id);
-			goto err;
+			return -EINVAL;
 		}
 	}
 
 	return 0;
+}
+
+int amdgpu_ualink_config_update_handler(struct amdgpu_device *adev)
+{
+	enum amdgpu_ualink_accel_state prev_state;
+	enum psp_gfx_ual_config_state cfg_state;
+	u32 prev_vpod_id;
+	int r, qerr;
+
+	/* TBD: Stop ASP interrupts if driver faced an issue */
+	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE) {
+		u32 status;
+
+		dev_dbg(adev->dev,
+			"UALink not initialized, skipping config update\n");
+		status = !!(adev->ualink.mgr_state == AMDGPU_UALINK_INIT_ERROR);
+		return psp_ual_send_completion(
+			&adev->psp, adev->ualink.psp_if_ver,
+			PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, status);
+	}
+
+	prev_state = adev->ualink.info->accel_state;
+	prev_vpod_id = adev->ualink.info->vpod.id;
+
+	qerr = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver,
+				  adev->ualink.info, &cfg_state);
+
+	/*TBD: find the right value of status to be sent to ASP*/
+	r = psp_ual_send_completion(&adev->psp, adev->ualink.psp_if_ver,
+				    PSP_GFX_INT_CTXT_UAL_CMD_CFG_UPDATE_ID, 0);
+	if (r || qerr)
+		goto err;
+
+	r = amdgpu_ualink_update_accel_state(adev, prev_state, prev_vpod_id,
+					     cfg_state);
+	if (r)
+		goto err;
+
+	return 0;
 
 err:
 	scoped_guard(mutex, &mgpu_info.mutex) {
@@ -816,8 +828,9 @@ static ssize_t ualink_ppod_setup_commit_store(struct kobject *kobj,
 	mutex_lock(&mgpu_info.mutex);
 	if (info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_READY)
 		deactivate_accelerator(adev);
-
 	info->accel_state = check_ppod_state(adev, setup);
+	/* PPOD is expected to be configured first */
+	info->vpod.id = AMDGPU_UALINK_VPOD_ID_INVALID;
 	mutex_unlock(&mgpu_info.mutex);
 
 	/* TODO: If accel_state was ACTIVE, reset all connections */
@@ -1111,6 +1124,8 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	struct device *dev = kobj_to_dev(info->kobj.parent);
 	struct drm_device *ddev = dev_get_drvdata(dev);
 	struct amdgpu_device *adev = drm_to_adev(ddev);
+	enum amdgpu_ualink_accel_state prev_state;
+	u32 prev_vpod_id;
 	int r;
 
 	if (!sysfs_streq(buf, "true"))
@@ -1120,6 +1135,8 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 		return -EINVAL;
 	}
 
+	prev_state = info->accel_state;
+	prev_vpod_id = info->vpod.id;
 	r = psp_ual_set_vpod_config(&adev->psp, adev->ualink.psp_if_ver,
 				    config);
 	if (r)
@@ -1128,30 +1145,21 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	if (r)
 		return r;
 
-	if (!__check_vpod_info(adev, info))
+	if (info->vpod.id != AMDGPU_UALINK_VPOD_ID_INVALID &&
+	    !__check_vpod_info(adev, info))
 		return -EINVAL;
 	/* The integrity check makes sure each new GPU is consistent with the
 	 * other GPUs already in the vPod. All known local GPUs can become
 	 * "ready" at the same time.
 	 *
 	 * Misconfiguration of one GPU does not reduce the state of other GPUs
-	 * already in the vPod.
-	 */
-	mutex_lock(&mgpu_info.mutex);
-	r = __check_local_vpod_integrity(adev);
-	if (!r)
-		activate_local_vpod(adev);
-	else if (info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED)
-		deactivate_accelerator(adev);
-	mutex_unlock(&mgpu_info.mutex);
-
-	/* TODO: Update KFD topology for in-domain link */
-
-	/* TODO: If state was ACTIVE:
-	 * - If addr_mode changed, reset all connections, reset state to READY
-	 * - If accelerators were removed, reset those links, but keep state ACTIVE
-	 * - If accelerators were added, keep state ACTIVE
+	 * already in the vPod. GPU is intentionally not put to ERROR state if
+	 * misconfiguration occurs.
 	 */
+	r = amdgpu_ualink_update_accel_state(adev, prev_state, prev_vpod_id,
+					     UAL_CFG_VPOD);
+	if (r)
+		return r;
 
 	return count;
 }
@@ -1271,9 +1279,6 @@ static ssize_t ualink_station_config_commit_store(struct kobject *kobj,
 		return -EINVAL;
 
 	r = psp_ual_set_station_config(&adev->psp, adev->ualink.psp_if_ver, stations);
-	if (r)
-		return r;
-	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver, info, NULL);
 	if (r)
 		return r;
 
Signature · 1 line
2.55.0
author maintainer
#84 lore raw

PATCH v2 84/95 drm/amdgpu: Fix GART and SDMA entity leak on vPod reconfiguration

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

remote->active_accel_bits aliased info->vpod.active_accel_bits, which psp_ual_query_info() refreshes on every commit. Teardown (peer_remote_fini / gart unmap) then iterated the new set instead of the one sw_init mapped, orphaning drm_mm nodes in the shared GTT manager. Make active_accel_bits an owned snapshot copied at sw_init time. Drop the now-redundant not-initialised guard in metadata_npa_unmapping(), since the earlier remote->ring_bo NULL-check already covers that case.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +11 -5
index 73aa9e98202b5..2d9317db32027 100644
@@ -3887,8 +3887,14 @@ struct amdgpu_ualink_remote {
 	/* address alias mode alloc npa address for shared wb */
 	struct drm_mm_node		rptr_mm_node;
 
-	/* active accelator id bitmap of the pod */
-	unsigned long			*active_accel_bits;
+	/*
+	 * Owned snapshot of the vPod's active accelerator bitmap, taken at
+	 * sw_init time. Must NOT alias info->vpod.active_accel_bits, which
+	 * psp_ual_query_info() refreshes on every commit: aliasing it would let
+	 * teardown unmap a different set than setup mapped, leaking drm_mm nodes
+	 * in the shared GTT manager.
+	 */
+	DECLARE_BITMAP(active_accel_bits, AMDGPU_UALINK_ACCEL_MAX);
 	u32				num_accel;
 
 	/* remote GPUs ring buffer, read, write pointer local copy and gart mapping */
@@ -4266,8 +4272,6 @@ static void amdgpu_ualink_metadata_npa_unmapping(struct amdgpu_device *adev)
 
 	if (!remote->ring_bo)
 		return;
-	if (!remote->active_accel_bits)
-		return;
 
 	rb_size = AMDGPU_UALINK_RB_SIZE;
 	rb_size_in_pages = rb_size >> AMDGPU_GPU_PAGE_SHIFT;
@@ -4686,7 +4690,9 @@ static int amdgpu_ualink_metadata_init(struct amdgpu_device *adev)
 	u32 status, accel_id;
 	int r;
 
-	remote->active_accel_bits = adev->ualink.info->vpod.active_accel_bits;
+	bitmap_copy(remote->active_accel_bits,
+		    adev->ualink.info->vpod.active_accel_bits,
+		    AMDGPU_UALINK_ACCEL_MAX);
 	dev_dbg(adev->dev, "%d active accelerators config in vpod\n",
 		bitmap_weight(remote->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX));
 
Signature · 1 line
2.55.0
author maintainer
#85 lore raw

PATCH v2 85/95 drm/amdgpu: Add UALink diagnostic logging for vpod commit/activation

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

Add UALINK:-tagged dev_info/dev_warn logging along the vpod commit -> update_accel_state -> integrity -> activate path so failing and -EAGAIN peer-wait cases are visible without dynamic debug. Also fix a wrong print arg (vpod->id -> vpod->addr_mode) in the invalid addr mode message.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 156 +++++++++++++++------
 1 file changed, 114 insertions(+), 42 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +114 -42
index 2d9317db32027..7a84d8ddbf867 100644
@@ -126,26 +126,29 @@ static bool __check_vpod_info(struct amdgpu_device *adev,
 	unsigned int weight;
 
 	if (vpod->size == 0 || vpod->size > ppod->size) {
-		dev_dbg(adev->dev, "vPod size %u out of range [1..%u]\n",
-			vpod->size, ppod->size);
+		dev_warn(adev->dev,
+			 "UALINK: vPod size %u out of range [1..%u]\n",
+			 vpod->size, ppod->size);
 		return false;
 	}
 	if (vpod->addr_mode >= AMDGPU_UALINK_ADDR_MODE_MAX) {
-		dev_dbg(adev->dev, "Invalid addr mode %u\n", vpod->id);
+		dev_warn(adev->dev,
+			 "UALINK: invalid addr mode %u\n",
+			 vpod->addr_mode);
 		return false;
 	}
 	weight =
 		bitmap_weight(vpod->active_accel_bits, AMDGPU_UALINK_ACCEL_MAX);
 	if (weight != vpod->size) {
-		dev_dbg(adev->dev,
-			"vPod size doesn't match vpod_active_accels list: %u != %u\n",
-			vpod->size, weight);
+		dev_warn(adev->dev,
+			 "UALINK: vPod size doesn't match vpod_active_accels list: %u != %u\n",
+			 vpod->size, weight);
 		return false;
 	}
 	if (!test_bit(ppod->accel_id, vpod->active_accel_bits)) {
-		dev_dbg(adev->dev,
-			"Accelerator ID %u not listed in vpod_active_accels\n",
-			ppod->accel_id);
+		dev_warn(adev->dev,
+			 "UALINK: accelerator ID %u not listed in vpod_active_accels\n",
+			 ppod->accel_id);
 		return false;
 	}
 
@@ -243,18 +246,27 @@ static int amdgpu_ualink_update_accel_state(
 	struct amdgpu_device *adev, enum amdgpu_ualink_accel_state prev_state,
 	u32 prev_vpod_id, enum psp_gfx_ual_config_state cfg_state)
 {
+	dev_info(adev->dev,
+		 "UALINK: update_accel_state: prev_state=%d prev_vpod_id=%u new_vpod_id=%u cfg_state=%d\n",
+		 prev_state, prev_vpod_id, adev->ualink.info->vpod.id,
+		 cfg_state);
+
 	/* If the device is already active and its vpod_id is unchanged, the
 	 * update does not affect vpod membership. Skip the local vpod
 	 * integrity check and re-activation.
 	 */
 	if (prev_state == AMDGPU_UALINK_ACCEL_STATE_ACTIVE &&
 	    adev->ualink.info->vpod.id == prev_vpod_id) {
+		dev_info(adev->dev,
+			 "UALINK: update_accel_state: already ACTIVE, vpod_id unchanged\n");
 		amdgpu_ualink_update_vpod_config(adev);
 		return 0;
 	}
 
 	/* A new vpod_id of 0 means this GPU was removed from the vPod. */
 	if (adev->ualink.info->vpod.id == AMDGPU_UALINK_VPOD_ID_INVALID) {
+		dev_info(adev->dev,
+			 "UALINK: update_accel_state: vpod_id=0, removing accelerator from vPod\n");
 		amdgpu_ualink_update_vpod_config(adev);
 		scoped_guard(mutex, &mgpu_info.mutex)
 			deactivate_accelerator(adev);
@@ -264,6 +276,9 @@ static int amdgpu_ualink_update_accel_state(
 	/* GPU joining a new vpod should be with invalid vpod id*/
 	scoped_guard(mutex, &mgpu_info.mutex) {
 		if (prev_vpod_id == AMDGPU_UALINK_VPOD_ID_INVALID) {
+			dev_info(adev->dev,
+				 "UALINK: update_accel_state: joining vpod_id=%u\n",
+				 adev->ualink.info->vpod.id);
 			amdgpu_ualink_info_set_accel_state(
 				adev, adev->ualink.info, cfg_state);
 			__amdgpu_ualink_activate_vpod_locked(adev);
@@ -910,6 +925,11 @@ static void activate_accelerator(struct amdgpu_device *adev)
 	if (adev->ualink.info->accel_state >= AMDGPU_UALINK_ACCEL_STATE_ACTIVE)
 		return;
 
+	dev_info(adev->dev,
+		 "UALINK: activating accelerator accel_id=%u (accel_state=%d)\n",
+		 adev->ualink.info->ppod.accel_id,
+		 adev->ualink.info->accel_state);
+
 	/* Enable incoming NPA address translation with NPA VMID */
 	r = psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
 				   adev->vm_manager.npa_vmid, true);
@@ -932,6 +952,9 @@ static void activate_accelerator(struct amdgpu_device *adev)
 	}
 
 	adev->ualink.info->accel_state = AMDGPU_UALINK_ACCEL_STATE_ACTIVE;
+	dev_info(adev->dev,
+		 "UALINK: accelerator accel_id=%u is now ACTIVE\n",
+		 adev->ualink.info->ppod.accel_id);
 }
 
 static void deactivate_accelerator(struct amdgpu_device *adev)
@@ -939,6 +962,11 @@ static void deactivate_accelerator(struct amdgpu_device *adev)
 	if (adev->ualink.info->accel_state < AMDGPU_UALINK_ACCEL_STATE_ACTIVE)
 		return;
 
+	dev_info(adev->dev,
+		 "UALINK: deactivating accelerator accel_id=%u (accel_state=%d -> PPOD_CONFIGURED)\n",
+		 adev->ualink.info->ppod.accel_id,
+		 adev->ualink.info->accel_state);
+
 	/* Disable incoming NPA address translation with NPA VMID */
 	psp_ual_set_npa_config(&adev->psp, adev->ualink.psp_if_ver,
 			       adev->vm_manager.npa_vmid, false);
@@ -1001,66 +1029,75 @@ static int __check_local_vpod_integrity(struct amdgpu_device *adev)
 		peer_info = peer_adev->ualink.info;
 		/* peer device ppod not configured */
 		if (peer_info->accel_state <
-		    AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED)
+		    AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED) {
+			dev_info(adev->dev,
+				 "UALINK: integrity EAGAIN: peer accel_id=%u not ppod-configured (accel_state=%d)\n",
+				 peer_info->ppod.accel_id,
+				 peer_info->accel_state);
 			return -EAGAIN;
+		}
 
 		accel_id = peer_info->ppod.accel_id;
 		if (!test_bit(accel_id, info->vpod.active_accel_bits))
 			continue;
 		/* peer device vpod not configured */
 		if (peer_info->accel_state <
-		    AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED)
+		    AMDGPU_UALINK_ACCEL_STATE_VPOD_CONFIGURED) {
+			dev_info(adev->dev,
+				 "UALINK: integrity EAGAIN: vpod peer accel_id=%u not vpod-configured (accel_state=%d)\n",
+				 accel_id, peer_info->accel_state);
 			return -EAGAIN;
+		}
 
 		if (!uuid_equal(&peer_info->ppod.id, &info->ppod.id)) {
-			dev_dbg(adev->dev,
-				"Peer %u ppod_id doesn't match: %pU != %pU",
-				peer_info->ppod.accel_id, &peer_info->ppod.id,
-				&info->ppod.id);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u ppod_id doesn't match: %pU != %pU\n",
+				 peer_info->ppod.accel_id, &peer_info->ppod.id,
+				 &info->ppod.id);
 			return -EINVAL;
 		}
 
 		if (peer_info->ppod.size != info->ppod.size) {
-			dev_dbg(adev->dev,
-				"Peer %u ppod_size doesn't match: %u != %u\n",
-				accel_id, peer_info->ppod.size,
-				info->ppod.size);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u ppod_size doesn't match: %u != %u\n",
+				 accel_id, peer_info->ppod.size,
+				 info->ppod.size);
 			return -EINVAL;
 		}
 
 		if (peer_info->vpod.id != info->vpod.id) {
-			dev_dbg(adev->dev,
-				"Peer %u vpod_id doesn't match: %u != %u",
-				accel_id, peer_info->vpod.id, info->vpod.id);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u vpod_id doesn't match: %u != %u\n",
+				 accel_id, peer_info->vpod.id, info->vpod.id);
 			return -EINVAL;
 		}
 		if (peer_info->vpod.size != info->vpod.size) {
-			dev_dbg(adev->dev,
-				"Peer %u vpod_size doesn't match: %u != %u\n",
-				accel_id, peer_info->vpod.size,
-				info->vpod.size);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u vpod_size doesn't match: %u != %u\n",
+				 accel_id, peer_info->vpod.size,
+				 info->vpod.size);
 			return -EINVAL;
 		}
 		if (peer_info->vpod.addr_mode != info->vpod.addr_mode) {
-			dev_dbg(adev->dev,
-				"Peer %u addr_mode doesn't match: %u != %u\n",
-				accel_id, peer_info->vpod.addr_mode,
-				info->vpod.addr_mode);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u addr_mode doesn't match: %u != %u\n",
+				 accel_id, peer_info->vpod.addr_mode,
+				 info->vpod.addr_mode);
 			return -EINVAL;
 		}
 		if (!bitmap_equal(peer_info->vpod.active_accel_bits,
 				  info->vpod.active_accel_bits,
 				  AMDGPU_UALINK_ACCEL_MAX)) {
-			dev_dbg(adev->dev,
-				"Peer %u vpod_active_accels don't match\n",
-				accel_id);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: peer %u vpod_active_accels don't match\n",
+				 accel_id);
 			return -EINVAL;
 		}
 
 		if (__test_and_set_bit(accel_id, local_accel_ids)) {
-			dev_dbg(adev->dev,
-				"Duplicate accel_id %u among local vpod peers\n",
-				accel_id);
+			dev_warn(adev->dev,
+				 "UALINK: integrity fail: duplicate accel_id %u among local vpod peers\n",
+				 accel_id);
 			return -EINVAL;
 		}
 		local_accels[n_local_accels++] = accel_id;
@@ -1099,8 +1136,16 @@ static void __amdgpu_ualink_activate_vpod_locked(struct amdgpu_device *adev)
 			"Local vpod integrity check failed: %d\n", ret);
 		return;
 	}
-	if (!ret)
-		activate_local_vpod(adev);
+	if (ret == -EAGAIN) {
+		dev_info(adev->dev,
+			 "UALINK: activate deferred, waiting for local vpod peers to reach VPOD_CONFIGURED\n");
+		return;
+	}
+
+	dev_info(adev->dev,
+		 "UALINK: integrity OK, applying local vpod (%u local accels)\n",
+		 adev->ualink.info->n_local_accels);
+	activate_local_vpod(adev);
 }
 
 #ifdef UALINK_ENABLE_DEPRECATED_CONFIG_SYSFS
@@ -1130,8 +1175,16 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 
 	if (!sysfs_streq(buf, "true"))
 		return -EINVAL;
+
+	dev_dbg(adev->dev,
+		"UALINK: vpod-commit enter: accel_state=%d staged vpod_id=%u vpod_size=%u addr_mode=%u\n",
+		info->accel_state, config->vpod.id, config->vpod.size,
+		config->vpod.addr_mode);
+
 	if (info->accel_state < AMDGPU_UALINK_ACCEL_STATE_PPOD_CONFIGURED) {
-		dev_dbg(adev->dev, "Ualink ppod is not yet configured\n");
+		dev_warn(adev->dev,
+			 "UALINK: vpod-commit rejected, ppod not configured (accel_state=%d)\n",
+			 info->accel_state);
 		return -EINVAL;
 	}
 
@@ -1139,15 +1192,34 @@ static ssize_t ualink_vpod_config_commit_store(struct kobject *kobj,
 	prev_vpod_id = info->vpod.id;
 	r = psp_ual_set_vpod_config(&adev->psp, adev->ualink.psp_if_ver,
 				    config);
-	if (r)
+	if (r) {
+		dev_warn(adev->dev,
+			 "UALINK: vpod-commit psp_ual_set_vpod_config failed: %d\n",
+			 r);
 		return r;
+	}
+
 	r = psp_ual_query_info(&adev->psp, adev->ualink.psp_if_ver, info, NULL);
-	if (r)
+	if (r) {
+		dev_warn(adev->dev,
+			 "UALINK: vpod-commit psp_ual_query_info failed: %d\n",
+			 r);
 		return r;
+	}
+
+	dev_dbg(adev->dev,
+		"UALINK: vpod-commit fw read-back: vpod_id=%u vpod_size=%u accel_id=%u ppod.size=%u addr_mode=%u\n",
+		info->vpod.id, info->vpod.size, info->ppod.accel_id,
+		info->ppod.size, info->vpod.addr_mode);
 
 	if (info->vpod.id != AMDGPU_UALINK_VPOD_ID_INVALID &&
-	    !__check_vpod_info(adev, info))
+	    !__check_vpod_info(adev, info)) {
+		dev_err(adev->dev,
+			"UALINK: vpod-commit __check_vpod_info() failed: %d\n",
+			r);
 		return -EINVAL;
+	}
+
 	/* The integrity check makes sure each new GPU is consistent with the
 	 * other GPUs already in the vPod. All known local GPUs can become
 	 * "ready" at the same time.
Signature · 1 line
2.55.0
author maintainer
#86 lore raw

PATCH v2 86/95 drm/amdgpu: Handle UALink vPod reconfiguration while ACTIVE

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

An already-ACTIVE accelerator with an unchanged vpod_id skipped re-activation, so growing or shrinking a live vPod left staying GPUs with links/GART built for the old member set. Detect membership changes (owned snapshot vs firmware read-back) and bounce affected ACTIVE peers once the integrity check passes. Firmware requires a full metadata reload while halted, so a full deactivate/activate is needed rather than per-peer deltas.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 60 ++++++++++++++++++++--
 1 file changed, 57 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +57 -3
index 7a84d8ddbf867..7d2acf10123bc 100644
@@ -54,6 +54,7 @@ static int amdgpu_ualink_remote_shootdown(struct amdgpu_device *adev,
 					  u32 remote_accel_id, u64 addr,
 					  u32 size_in_pages, u32 flush_type);
 static void __amdgpu_ualink_activate_vpod_locked(struct amdgpu_device *adev);
+static bool amdgpu_ualink_vpod_membership_changed(struct amdgpu_device *adev);
 
 #define STRIP_NPA(addr)						\
 	(((u64)(addr) & ~AMDGPU_UALINK_NPA_ADDR_GPUID_MASK))
@@ -257,9 +258,24 @@ static int amdgpu_ualink_update_accel_state(
 	 */
 	if (prev_state == AMDGPU_UALINK_ACCEL_STATE_ACTIVE &&
 	    adev->ualink.info->vpod.id == prev_vpod_id) {
-		dev_info(adev->dev,
-			 "UALINK: update_accel_state: already ACTIVE, vpod_id unchanged\n");
 		amdgpu_ualink_update_vpod_config(adev);
+
+		/* Same vpod_id, but the member set may have changed (vPod grown
+		 * or shrunk while this GPU stayed ACTIVE). If so, run the local
+		 * vpod activation path: once every local peer has committed the
+		 * new config (integrity passes), it bounces the affected ACTIVE
+		 * peers to rebuild links/GART for the new member set.
+		 */
+		if (amdgpu_ualink_vpod_membership_changed(adev)) {
+			dev_info(adev->dev,
+				 "UALINK: update_accel_state: ACTIVE vpod_id=%u membership changed, reconfiguring\n",
+				 adev->ualink.info->vpod.id);
+			scoped_guard(mutex, &mgpu_info.mutex)
+				__amdgpu_ualink_activate_vpod_locked(adev);
+		} else {
+			dev_info(adev->dev,
+				 "UALINK: update_accel_state: already ACTIVE, vpod_id/membership unchanged\n");
+		}
 		return 0;
 	}
 
@@ -993,7 +1009,24 @@ static void activate_local_vpod(struct amdgpu_device *adev)
 			/* info->local_accels is corrupted? */
 			continue;
 
-		activate_accelerator(peer_adev);
+		/* Bring the peer up to match the current vPod membership.
+		 *
+		 * A peer that is not yet ACTIVE (or whose ACTIVE membership
+		 * changed on a grow/shrink) has its remote metadata, links and
+		 * GART mappings built for a stale member set (or none at all).
+		 * Firmware only accepts a full metadata reload while halted, so
+		 * surgical per-peer deltas are not possible: fully bounce the
+		 * accelerator. deactivate_accelerator() is a no-op when the peer
+		 * is not ACTIVE, so this handles first-time bring-up too.
+		 * Unchanged ACTIVE peers are left untouched.
+		 */
+		if (amdgpu_ualink_vpod_membership_changed(peer_adev)) {
+			dev_info(peer_adev->dev,
+				 "UALINK: (re)configuring vpod for accel_id=%u\n",
+				 peer_adev->ualink.info->ppod.accel_id);
+			deactivate_accelerator(peer_adev);
+			activate_accelerator(peer_adev);
+		}
 	}
 }
 
@@ -3981,6 +4014,27 @@ static inline struct amdgpu_ualink_remote *to_remote(struct amdgpu_device *adev)
 	return adev->ualink.remote;
 }
 
+/*
+ * Returns true if the vPod membership most recently reported by firmware
+ * (info->vpod.active_accel_bits, read back via psp_ual_query_info() on the
+ * committing GPU) differs from the owned snapshot captured when this
+ * accelerator was last activated (remote->active_accel_bits). Used to
+ * decide whether an accelerator must be (re)built to match the new member
+ * set. A NULL remote means the accelerator has never been activated (no
+ * snapshot yet), which also counts as "changed" so it gets brought up.
+ */
+static bool amdgpu_ualink_vpod_membership_changed(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_remote *remote = to_remote(adev);
+
+	if (!remote)
+		return true;
+
+	return !bitmap_equal(remote->active_accel_bits,
+			     adev->ualink.info->vpod.active_accel_bits,
+			     AMDGPU_UALINK_ACCEL_MAX);
+}
+
 static inline u32 ualink_accel_id(struct amdgpu_device *adev)
 {
 	return adev->ualink.info->ppod.accel_id;
Signature · 1 line
2.55.0
author maintainer
#87 lore raw

PATCH v2 87/95 drm/amdgpu: Add name for ualink ip block

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Add namestring for ualink

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c | 1 +
 1 file changed, 1 insertion(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c +1 -0
index 922f4b15619d5..a426327f21416 100644
@@ -198,6 +198,7 @@ static const char *const ip_block_names[] = {
 	[AMD_IP_BLOCK_TYPE_UMSCH_MM] = "umsch_mm",
 	[AMD_IP_BLOCK_TYPE_ISP] = "isp",
 	[AMD_IP_BLOCK_TYPE_RAS] = "ras",
+	[AMD_IP_BLOCK_TYPE_UALINK] = "ualink",
 };
 
 static const char *ip_block_name(struct amdgpu_device *adev,
Signature · 1 line
2.55.0
author maintainer
#88 lore raw

PATCH v2 88/95 drm/amdgpu: Cleanup UALink XA entries on manager stop

In reply to Alex Deucher

From: Mukul Joshi <[email protected]>

amdgpu_ualink_manager_stop() only called xa_destroy() on the exporter, importer and handle-invalid xarrays, which frees the internal tree but not the node objects. Entries still present at teardown thus leaked the nodes, left NPA addresses mapped and kept exported BOs pinned.

Free the entries in all three xarrays explicitly, without exchanging any NPA protocol messages (the F/W is already halted by amdgpu_ualink_sw_fini()): mark connections down so the drained workers and teardown take the message-free path, drain in-flight exporter cleanup work to empty handle_invalid_xa, then free the remaining importer and exporter entries before the NPA allocator is torn down.

Also factor the shared importer-node release into amdgpu_ualink_release_imp_xa_node(), drop/drain stale remote interrupts in the IRQ handler when the accelerator is not ACTIVE, and refuse outbound messages in amdgpu_ualink_send_command() when not ACTIVE so an in-flight cleanup worker cannot emit onto torn-down peer/SDMA state or block waiting for a response that will never arrive.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 153 +++++++++++++++++++--
 1 file changed, 141 insertions(+), 12 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +141 -12
index 7d2acf10123bc..d7d74b1ae7fc6 100644
@@ -2205,12 +2205,33 @@ static int amdgpu_ualink_setup_connection(struct amdgpu_device *adev,
 	return r;
 }
 
+/* Invalidate an importer node's mappings and drop its last ref (frees the
+ * node, dma-buf and GEM handle). Caller must have unlinked it from the
+ * per-remote list and must not hold the imp_xa lock.
+ */
+static void amdgpu_ualink_release_imp_xa_node(struct amdgpu_device *adev,
+				struct amdgpu_ualink_imp_xa_node *imp_xa_node)
+{
+	struct amdgpu_bo *bo;
+
+	dev_dbg(adev->dev,
+		"IMP-CLEANUP: handle:%llx:%llx npa:%llx size:%llx\n",
+		imp_xa_node->handle.handle_hi, imp_xa_node->handle.handle_lo,
+		imp_xa_node->npa_addr, imp_xa_node->size);
+
+	if (imp_xa_node->dmabuf) {
+		bo = gem_to_amdgpu_bo(imp_xa_node->dmabuf->priv);
+		amdgpu_ualink_invalidate_import_mappings(bo);
+	}
+
+	amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
+}
+
 static void amdgpu_ualink_cleanup_imp_xa_entries(struct amdgpu_device *adev,
 						 u32 remote_acc_id)
 {
 	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
 	struct list_head *imp_handles_list;
-	struct amdgpu_bo *bo;
 
 	dev_dbg(adev->dev,
 		"IMP-RESET: Cleaning up all XA entries for remote:%u\n",
@@ -2226,18 +2247,8 @@ static void amdgpu_ualink_cleanup_imp_xa_entries(struct amdgpu_device *adev,
 		WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_TEARDOWN);
 		xa_unlock(&adev->ualink.imp_xa);
 
-		dev_dbg(adev->dev,
-			"IMP-RESET: remote:%u handle:%llx:%llx npa:%llx size:%llx\n",
-			remote_acc_id, imp_xa_node->handle.handle_hi,
-			imp_xa_node->handle.handle_lo,
-			imp_xa_node->npa_addr, imp_xa_node->size);
-
-		bo = gem_to_amdgpu_bo(imp_xa_node->dmabuf->priv);
-		/* Invalidate the imported mappings */
-		amdgpu_ualink_invalidate_import_mappings(bo);
+		amdgpu_ualink_release_imp_xa_node(adev, imp_xa_node);
 
-		/* Drop the refcount for the node */
-		amdgpu_ualink_imp_xa_entry_put(imp_xa_node);
 		xa_lock(&adev->ualink.imp_xa);
 	}
 	xa_unlock(&adev->ualink.imp_xa);
@@ -3860,13 +3871,114 @@ int amdgpu_ualink_manager_start(struct amdgpu_device *adev)
 	return r;
 }
 
+/* Free every importer entry left in imp_xa at manager stop. */
+static void amdgpu_ualink_teardown_imp_xa_entries(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_imp_xa_node *imp_xa_node;
+	unsigned long index;
+
+	xa_for_each(&adev->ualink.imp_xa, index, imp_xa_node) {
+		list_del_init(&imp_xa_node->list);
+		WRITE_ONCE(imp_xa_node->node_state, AMDGPU_UALINK_NODE_TEARDOWN);
+		amdgpu_ualink_release_imp_xa_node(adev, imp_xa_node);
+	}
+}
+
+/* Free a single exporter node at manager stop. This is the message-free
+ * portion of amdgpu_ualink_exp_cleanup_worker().
+ */
+static void amdgpu_ualink_teardown_exp_xa_node(struct amdgpu_device *adev,
+				struct amdgpu_ualink_exp_xa_node *exp_xa_node)
+{
+	DECLARE_BITMAP(importers_bitmap, AMDGPU_UALINK_ACCEL_MAX);
+	struct amdgpu_bo *bo = exp_xa_node->bo;
+	u32 remote_acc_id;
+
+	bitmap_copy(importers_bitmap, exp_xa_node->importers_bitmap,
+		    AMDGPU_UALINK_ACCEL_MAX);
+
+	dev_dbg(adev->dev,
+		"EXP-STOP: handle:%llx:%llx importers bitmap: %*pbl\n",
+		exp_xa_node->handle.handle_hi, exp_xa_node->handle.handle_lo,
+		AMDGPU_UALINK_ACCEL_MAX, importers_bitmap);
+
+	if (!bitmap_empty(importers_bitmap, AMDGPU_UALINK_ACCEL_MAX)) {
+		amdgpu_ualink_unmap_all_npa_addr(adev, exp_xa_node);
+
+		/* Unlink importers and unpin the BO once per importer. */
+		if (likely(!amdgpu_bo_reserve(bo, true))) {
+			bo->ualink_handle_lo = 0ULL;
+			for_each_set_bit(remote_acc_id, importers_bitmap,
+					 AMDGPU_UALINK_ACCEL_MAX) {
+				list_del_init(&exp_xa_node->importer_entries[remote_acc_id].list);
+				amdgpu_bo_unpin(bo);
+			}
+			amdgpu_bo_unreserve(bo);
+		} else {
+			dev_warn(adev->dev,
+				"EXP-STOP: BO reserve to unpin failed handle:%llx:%llx\n",
+				exp_xa_node->handle.handle_hi,
+				exp_xa_node->handle.handle_lo);
+			for_each_set_bit(remote_acc_id, importers_bitmap,
+					 AMDGPU_UALINK_ACCEL_MAX)
+				list_del_init(&exp_xa_node->importer_entries[remote_acc_id].list);
+		}
+
+		amdgpu_ualink_free_all_npa_va(adev, exp_xa_node, importers_bitmap);
+	}
+
+	/* Release the export dma-buf and drop the BO ref. */
+	dma_buf_put(exp_xa_node->dmabuf);
+	amdgpu_bo_unref(&bo);
+	exp_xa_node->bo = NULL;
+
+	mutex_destroy(&exp_xa_node->node_lock);
+	kfree(exp_xa_node);
+}
+
+/* Free every exporter entry left in exp_xa at manager stop. These were
+ * exported but never revoked, so no cleanup worker was queued for them.
+ */
+static void amdgpu_ualink_teardown_exp_xa_entries(struct amdgpu_device *adev)
+{
+	struct amdgpu_ualink_exp_xa_node *exp_xa_node;
+	unsigned long index;
+
+	xa_for_each(&adev->ualink.exp_xa, index, exp_xa_node) {
+		xa_erase(&adev->ualink.exp_xa, index);
+		amdgpu_ualink_teardown_exp_xa_node(adev, exp_xa_node);
+	}
+}
+
 void amdgpu_ualink_manager_stop(struct amdgpu_device *adev)
 {
 	int i;
 
 	adev->mmhub.funcs->setup_vm_pt_regs(adev, adev->vm_manager.npa_vmid, 0);
+
+	/* Mark connections down so the drained workers and the teardown below
+	 * take the message-free path in amdgpu_ualink_check_conn_ready().
+	 */
+	for (i = 0; i < AMDGPU_UALINK_ACCEL_MAX; i++) {
+		mutex_lock(&adev->ualink.conn_state[i].lock);
+		adev->ualink.conn_state[i].state = AMDGPU_UALINK_CONN_NOT_READY;
+		mutex_unlock(&adev->ualink.conn_state[i].lock);
+	}
+
+	/* Drain in-flight exporter cleanup work so revoked nodes in
+	 * handle_invalid_xa free themselves and empty that xarray.
+	 */
+	drain_workqueue(adev->ualink.npa_wq);
+
+	/* Free live entries before the NPA allocator is torn down, so NPA
+	 * addresses can still be unmapped and freed.
+	 */
+	amdgpu_ualink_teardown_imp_xa_entries(adev);
+	amdgpu_ualink_teardown_exp_xa_entries(adev);
+
 	amdgpu_ualink_npa_mm_fini(adev);
 
+	/* All three xarrays are empty by now. */
 	xa_destroy(&adev->ualink.exp_xa);
 	xa_destroy(&adev->ualink.imp_xa);
 	xa_destroy(&adev->ualink.handle_invalid_xa);
@@ -5177,6 +5289,10 @@ static int amdgpu_ualink_send_command(struct amdgpu_device *adev,
 	u64 src;
 	int r;
 
+	/* Refuse sends unless active */
+	if (adev->ualink.info->accel_state != AMDGPU_UALINK_ACCEL_STATE_ACTIVE)
+		return -ESHUTDOWN;
+
 	peer = &remote->peer[remote_accel_id];
 
 	/*
@@ -5734,6 +5850,7 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 				     struct amdgpu_iv_entry *entry)
 {
 	u32 sender_acc_id, receiver_acc_id, msg_type, src_acc_id;
+	enum amdgpu_ualink_accel_state accel_state;
 	struct amdgpu_ualink_handle handle;
 	u32 size, npa_addr, fail_reason;
 	u32 dw0, dw1, dw2, dw3;
@@ -5746,6 +5863,18 @@ static int amdgpu_ualink_process_irq(struct amdgpu_device *adev,
 		return handled;
 	}
 
+	/* Only handle interrupts while ACTIVE. Otherwise (e.g. during teardown)
+	 * consume stale ring entries but drop them, so they don't touch state
+	 * being freed.
+	 */
+	accel_state = adev->ualink.info->accel_state;
+	if (accel_state != AMDGPU_UALINK_ACCEL_STATE_ACTIVE) {
+		dev_dbg(adev->dev,
+			"Dropping stale UALink interrupt, accel_state %d\n",
+			accel_state);
+		return handled;
+	}
+
 	dev_dbg(adev->dev, "%s client_id 0x%x src_id 0x%x ih\n",
 		entry->ih == &adev->irq.ih ? "ring" : "ualink soft ring",
 		entry->client_id, entry->src_id);
Signature · 1 line
2.55.0
author maintainer
#89 lore raw

PATCH v2 89/95 drm/amdgpu: Move ualink ip version related changes

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Move more IP version specific related things to ualink version file. Use soc v1.0 IH client id defintion and define mpnht interrupt source id in ivsrcid header.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c    | 28 +++++------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h    |  4 ++-
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c      | 24 +++++++++++++--
 .../ivsrcid/mpnht/irqsrcs_mpnht_15_0.h        | 30 +++++++++++++++++++
 4 files changed, 63 insertions(+), 23 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/include/ivsrcid/mpnht/irqsrcs_mpnht_15_0.h
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +8 -20
index d7d74b1ae7fc6..5957a3a2c85b1 100644
@@ -1419,8 +1419,6 @@ static const struct kobj_type ualink_station_config_ktype = {
 
 int ualink_ip_sw_init(struct amdgpu_ip_block *ip_block)
 {
-	int r;
-
 	struct amdgpu_ualink_station_config *stations;
 	struct amdgpu_ualink_vpod_config *vpod_config;
 	struct amdgpu_ualink_ppod_setup *ppod_setup;
@@ -1460,12 +1458,6 @@ int ualink_ip_sw_init(struct amdgpu_ip_block *ip_block)
 	adev->ualink.config = vpod_config;
 	adev->ualink.stations = stations;
 
-	r = amdgpu_ualink_init_interrupt(adev);
-	if (r) {
-		dev_err(adev->dev, "Failed to add UALink irq: %d\n", r);
-		return r;
-	}
-
 	return 0;
 }
 
@@ -6012,32 +6004,28 @@ static const struct amdgpu_irq_src_funcs ualink_irq_funcs = {
 	.process = amdgpu_ualink_process_irq,
 };
 
-/* TODO: if move to header file soc21_enum.h */
-#define UALINK_IH_CLIENT_ID 0x1C
-#define UALINK_IH_SOURCE_ID 0x0
-
 /**
- * amdgpu_ualink_init_interrupt - initialization of UALink IRQ
+ * amdgpu_ualink_init_interrupt - register the UALink IRQ source
  * @adev: amdgpu device pointer
+ * @client_id: IH client ID for the interrupt source
+ * @src_id: source ID for the interrupt source
  *
- * Registers the UALink interrupt source with the IH (Interrupt Handler)
- * subsystem during early device initialization. This sets up the IRQ
- * callback functions for handling remote interrupts from peer GPUs.
+ * Registers the UALink interrupt source with the IH subsystem.
  *
  * Return: 0 on success, negative error code on failure
  */
-int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev)
+int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev,
+				 unsigned int client_id, unsigned int src_id)
 {
 	int r;
 
 	dev_dbg(adev->dev, "init ualink irq client_id 0x%x src_id 0x%x\n",
-		UALINK_IH_CLIENT_ID, UALINK_IH_SOURCE_ID);
+		client_id, src_id);
 
 	adev->ualink.irq.num_types = 1;
 	adev->ualink.irq.funcs = &ualink_irq_funcs;
 
-	r = amdgpu_irq_add_id(adev, UALINK_IH_CLIENT_ID,
-			      UALINK_IH_SOURCE_ID, &adev->ualink.irq);
+	r = amdgpu_irq_add_id(adev, client_id, src_id, &adev->ualink.irq);
 	return r;
 }
 
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +3 -1
index e23aaf8b313e1..792b0fc9fed97 100644
@@ -388,7 +388,9 @@ struct amdgpu_ualink_mgr {
 	unsigned long drop_msg_bitmap;
 };
 
-int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev);
+int amdgpu_ualink_init_interrupt(struct amdgpu_device *adev,
+				 unsigned int client_id, unsigned int src_id);
+
 int amdgpu_ualink_sw_init(struct amdgpu_device *adev);
 void amdgpu_ualink_sw_fini(struct amdgpu_device *adev);
 
drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c +22 -2
index 0fe2c0fba4cf9..58cf98ac94add 100644
@@ -23,6 +23,7 @@
 
 #include <linux/delay.h>
 #include "amdgpu.h"
+#include "ivsrcid/mpnht/irqsrcs_mpnht_15_0.h"
 #include "amdgpu_ualink.h"
 #include "ualink_v1_0.h"
 
@@ -104,11 +105,30 @@ static int ualink_v1_0_early_init(struct amdgpu_ip_block *ip_block)
 	return 0;
 }
 
+static int ualink_v1_0_sw_init(struct amdgpu_ip_block *ip_block)
+{
+	struct amdgpu_device *adev = ip_block->adev;
+	int r;
+
+	r = ualink_ip_sw_init(ip_block);
+	if (r)
+		return r;
+
+	r = amdgpu_ualink_init_interrupt(adev, SOC_V1_0_IH_CLIENTID_nHT,
+					 MPNHT_15_0__SRCID__REMOTE_INTERRUPT);
+	if (r) {
+		dev_err(adev->dev, "Failed to add UALink irq: %d\n", r);
+		return r;
+	}
+
+	return 0;
+}
+
 static const struct amd_ip_funcs ualink_v1_0_ip_funcs = {
 	.name = "ualink",
 	.early_init = ualink_v1_0_early_init,
 	.late_init = ualink_ip_late_init,
-	.sw_init = ualink_ip_sw_init,
+	.sw_init = ualink_v1_0_sw_init,
 	.sw_fini = ualink_ip_sw_fini,
 	.hw_init = ualink_ip_hw_init,
 };
@@ -119,4 +139,4 @@ const struct amdgpu_ip_block_version ualink_v1_0_ip_block = {
 	.minor = 0,
 	.rev = 0,
 	.funcs = &ualink_v1_0_ip_funcs,
-};
\ No newline at end of file
+};
drivers/gpu/drm/amd/include/ivsrcid/mpnht/irqsrcs_mpnht_15_0.h new file +30 -0
new file mode 100644
index 0000000000000..c960353f107db
@@ -0,0 +1,30 @@
+/*
+ * Copyright 2026 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __IRQSRCS_MPNHT_15_0_H__
+#define __IRQSRCS_MPNHT_15_0_H__
+
+/* MPNHT remote interrupt */
+#define MPNHT_15_0__SRCID__REMOTE_INTERRUPT            0x00
+
+#endif
Signature · 1 line
2.55.0
author maintainer
#90 lore raw

PATCH v2 90/95 drm/amdgpu: Add hw_fini for ualink

In reply to Alex Deucher

From: Lijo Lazar <[email protected]>

Deactivate accelerator from vpod during hw_fini sequence.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 13 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  1 +
 drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c   |  1 +
 3 files changed, 15 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c +13 -0
index 5957a3a2c85b1..19c2df55581b3 100644
@@ -425,6 +425,19 @@ int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block)
 	return 0;
 }
 
+int ualink_ip_hw_fini(struct amdgpu_ip_block *ip_block)
+{
+	struct amdgpu_device *adev = ip_block->adev;
+
+	if (adev->ualink.mgr_state != AMDGPU_UALINK_INIT_COMPLETE)
+		return 0;
+
+	scoped_guard(mutex, &mgpu_info.mutex)
+		deactivate_accelerator(adev);
+
+	return 0;
+}
+
 int ualink_ip_late_init(struct amdgpu_ip_block *ip_block)
 {
 	struct amdgpu_device *adev = ip_block->adev;
drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +1 -0
index 792b0fc9fed97..8ec0bc80a0f7d 100644
@@ -413,6 +413,7 @@ int amdgpu_ualink_import_handle(struct drm_device *dev,
 void amdgpu_ualink_revoke_exported_memory(struct amdgpu_bo *bo);
 
 int ualink_ip_hw_init(struct amdgpu_ip_block *ip_block);
+int ualink_ip_hw_fini(struct amdgpu_ip_block *ip_block);
 int ualink_ip_late_init(struct amdgpu_ip_block *ip_block);
 int ualink_ip_sw_init(struct amdgpu_ip_block *ip_block);
 int ualink_ip_sw_fini(struct amdgpu_ip_block *ip_block);
drivers/gpu/drm/amd/amdgpu/ualink_v1_0.c +1 -0
index 58cf98ac94add..d440ed6b57336 100644
@@ -131,6 +131,7 @@ static const struct amd_ip_funcs ualink_v1_0_ip_funcs = {
 	.sw_init = ualink_v1_0_sw_init,
 	.sw_fini = ualink_ip_sw_fini,
 	.hw_init = ualink_ip_hw_init,
+	.hw_fini = ualink_ip_hw_fini,
 };
 
 const struct amdgpu_ip_block_version ualink_v1_0_ip_block = {
Signature · 1 line
2.55.0