Skip to contentSkip to conversation

net: knod: in-kernel network offload device

RFC PATCH In review Taehee Yoo 6 participants 32 messages Jul 19 – Jul 21

Subsystems (15)
lore
Share & tools
lore

Taehee Yoo

author
#0 lore raw

cover net: knod: in-kernel network offload device

knod (in-kernel network offload device) drives a GPU directly from the Linux kernel - with no userspace GPU runtime such as CUDA or ROCm, and no userspace component in the data path - to accelerate packet processing.

The kernel itself allocates the GPU's queues, JIT-compiles the per-packet program to GPU machine code, and dispatches it; the NIC DMAs received packets straight into GPU memory and the GPU returns a verdict. The GPU is programmed like any other in-kernel offload, not through a userspace framework, so existing XDP programs and IPsec SAs can be offloaded to it transparently.

The design and motivation were presented at Linux Plumbers Conference 2025:

https://lpc.events/event/19/contributions/2267/

Motivation
==========

Line-rate packet processing that does non-trivial per-packet work - an XDP program doing L4 load balancing, or IPsec crypto - is bound by the host CPU: each core handles one packet at a time, so scaling means spending more cores. A GPU is the opposite shape - thousands of lanes running the same small program over many packets at once (SIMT) - which happens to match the per-packet-program model of XDP.

knod moves that per-packet compute off the host CPU and onto a GPU. The NIC DMAs received packets directly into GPU memory, the GPU runs the program across a batch of packets in parallel, and only the result comes back: a verdict for every packet, plus the packet itself for the ones destined to the host. The CPU no longer pays the per-packet program cost, and throughput scales with GPU occupancy rather than core count.

Crucially this happens entirely inside the kernel. GPU packet processing today generally launches work from a userspace GPU runtime (CUDA and friends) and keeps that runtime in the data path; knod instead builds the GPU queues, compiles the program, and dispatches it from the kernel, so it plugs into existing offload paths (XDP, xfrm) with nothing to install or keep running in userspace.

Model
=====

knod binds two endpoints through a third object:

- a NIC-side netdev, registered by the NIC driver;
- an accelerator (the GPU), registered by a provider built into the

    GPU driver;

- an offload device, created on attach, that connects them and owns the per-queue data-path state.

The accelerator ops are feature-agnostic, so different per-packet programs plug in behind one interface. This series ships two features to show the framework is not tied to a single use case (the accel-type uAPI also reserves "dpu" for future non-GPU backends):

- BPF: an XDP program attached in offload mode is JIT-compiled from
eBPF to an AMD GCN shader and executed on the GPU.
- IPsec: RX ESP full-packet decrypt on the GPU (proof of concept, see below).

Data path
=========

For a NIC RX queue bound to an accelerator, a received packet flows as follows:

1. Attach reconfigures the NIC's page_pool so its pages come from GPU memory, exported as a dma-buf and plugged in through the devmem memory provider. The NIC therefore DMAs the packet straight into GPU memory - there is no host-side copy on RX.

2. Instead of building an skb and entering the stack, the NIC's NAPI pushes a small descriptor (memory ref + offset + length) onto a per-queue lock-free SPSC ring shared with the GPU.

3. A persistent GPU worker drains descriptors and runs the active feature's shader over a batch of packets in parallel - one workgroup per packet - then writes a per-packet verdict back onto the ring: PASS, DROP or TX.

4. Back on the NIC NAPI, each verdict is applied:
- DROP: the GPU page is recycled to the pool;
- TX: the packet is sent back out the NIC directly from GPU
memory (XDP_TX), with no host round-trip;
- PASS: the packet is copied out of GPU memory into a host delivery page by the GPU's DMA engine (async SDMA), wrapped as an skb, and handed to the normal stack.

So only PASS packets ever touch host memory, and even then the copy is done by the GPU, not the CPU.

The control plane is a generic netlink family ("knod") for attach/detach, accelerator/NIC inventory, and per-accelerator feature selection, with notifications on bind/unbind.

Code layout
===========

knod separates into a subsystem-neutral core and a GPU-specific provider; nothing in the core knows about GPUs. Roughly:

  - core + control plane + NIC drivers   net/, kernel/bpf/       (~4.4k)
  - GPU provider (queues, JIT, shaders)  drivers/gpu/drm/amd/    (~39k)

The provider is the bulk of the diff, but most of it is self-contained GPU code generation and hand-written shaders, not core logic. It lives under amdkfd because that is where AMD GPU compute lives; the load-bearing boundary is the accelerator interface, not amdkfd.

Usage
=====

The control plane is driven with the in-tree YNL CLI. Examples, run from the kernel tree (ifindex is the NIC's ifindex, e.g. from ip -j link):

  SPEC=Documentation/netlink/specs/knod.yaml
  CLI="tools/net/ynl/pyynl/cli.py --spec $SPEC"

  # list accelerators (id, type, supported/enabled features)
  $CLI --dump accel-get

  # attach a NIC to accelerator 0
  $CLI --do attach --json '{"nic-ifindex": <ifindex>, "accel-id": 0}'

  # enable a feature on accelerator 0 ("none" | "bpf" | "ipsec")
  $CLI --do accel-set --json '{"id": 0, "feature-ena": "bpf"}'

  # show current NIC <-> accelerator bindings
  $CLI --dump dev-get

  # detach the NIC
  $CLI --do detach --json '{"nic-ifindex": <ifindex>}'

Attach/detach also emit notifications on the "mgmt" multicast group (dev-add-ntf / dev-del-ntf), which the CLI can watch with --subscribe mgmt.

Hardware
========

The reference GPUs are GCN (Radeon RX Vega64) and RDNA2 (Radeon RX 6600 and Radeon PRO V620). GCN does not implement the 64-bit atomics the data path relies on, so active development targets RDNA2; all numbers below are measured on RDNA2.

Performance
===========

The BPF path is measured with kondor, a katran-based L4 load balancer running as the offloaded XDP program.

  Feature     GPU        Throughput   GPU power
  ---------   --------   ----------   -------------------
  BPF (XDP)   RX 6600     32 Mpps      41 W (100 W board)
  BPF (XDP)   V620        70 Mpps      80 W (300 W board)
  IPsec RX    RX 6600     80 Gbps     100 W
  IPsec RX    V620        80 Gbps     100 W

Host CPU utilization drops by roughly the cost of the XDP program or the IPsec crypto that no longer runs on the CPU, offset by a small knod bookkeeping overhead (SPSC ring management and packet delivery).

IPsec status
============

The IPsec feature is a functional proof of concept and should be read as "this is possible on the framework", not as a production-ready implementation. It is RX only (no TX offload) and supports both tunnel and transport mode.

Development tree
================

Work continues in a public tree:

https://github.com/TaeheeYoo/knod/tree/knod-7.2

Taehee Yoo (13):
net: knod: add uapi and core headers
net: devmem: extend memory provider for knod
net: core: add XDP_MODE_HW offload hook for knod
net: knod: add offload device core and control plane
bpf: offload: allow PERCPU_ARRAY maps for offloaded programs drm/amdkfd: prepare kfd core for the knod provider drm/amdkfd: add knod provider core
drm/amdkfd: add GPU instruction emitter and disassembler drm/amdkfd: add BPF-to-GPU JIT offload
net/mlx5e: add knod XDP offload support
bnxt_en: add knod XDP offload support
selftests: drivers/net: add knod tests
drm/amdkfd: add IPsec full-packet offload

 Documentation/netlink/specs/knod.yaml         |   176 +
 drivers/gpu/drm/amd/amdgpu/Makefile           |     1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h    |    13 +-
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    65 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |     3 +
 drivers/gpu/drm/amd/amdkfd/Kconfig            |    31 +
 drivers/gpu/drm/amd/amdkfd/Makefile           |     9 +
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      |   117 +-
 drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c     |    20 +-
 drivers/gpu/drm/amd/amdkfd/kfd_events.c       |   112 +-
 drivers/gpu/drm/amd/amdkfd/kfd_events.h       |     4 +
 drivers/gpu/drm/amd/amdkfd/kfd_hsa.h          |   451 +
 drivers/gpu/drm/amd/amdkfd/kfd_knod.c         |  2202 +++
 drivers/gpu/drm/amd/amdkfd/kfd_module.c       |     1 +
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h         |    34 +
 drivers/gpu/drm/amd/amdkfd/kfd_process.c      |    26 +-
 .../gpu/drm/amd/amdkfd/knod/aesgcm_shader.h   |   984 ++
 .../drm/amd/amdkfd/knod/ipsec_fused_gfx10.h   |  1796 +++
 .../drm/amd/amdkfd/knod/ipsec_fused_gfx9.h    |  1349 ++
 drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h    |   270 +
 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h |   173 +
 .../drm/amd/amdkfd/knod/knod_amdgpu_insn.h    |  6362 +++++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c    | 11554 ++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h    |   597 +
 .../gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h |  3978 ++++++
 .../gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h  |  4068 ++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c  |  4273 ++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h  |   596 +
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |   155 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |    12 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |   274 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h |    12 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |    10 +
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  |   351 +
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |    19 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |    38 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |   107 +-
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |    45 +
 include/net/devmem.h                          |    58 +
 include/net/knod.h                            |   467 +
 include/net/netmem.h                          |     9 +
 include/net/page_pool/memory_provider.h       |     4 +
 include/net/page_pool/types.h                 |    23 +-
 include/net/spsc_ring.h                       |   645 +
 include/uapi/linux/knod.h                     |    67 +
 kernel/bpf/offload.c                          |     3 +-
 net/Kconfig                                   |     2 +
 net/Makefile                                  |     1 +
 net/core/dev.c                                |     2 +-
 net/core/devmem.c                             |   103 +-
 net/core/devmem.h                             |     7 +-
 net/core/page_pool.c                          |    22 +-
 net/knod/Kconfig                              |     5 +
 net/knod/Makefile                             |     8 +
 net/knod/knod-nl-gen.c                        |   121 +
 net/knod/knod-nl-gen.h                        |    31 +
 net/knod/knod.h                               |    26 +
 net/knod/knod_core.c                          |  1231 ++
 net/knod/knod_nl.c                            |   406 +
 .../selftests/drivers/net/knod/Makefile       |    17 +
 .../testing/selftests/drivers/net/knod/config |     8 +
 .../selftests/drivers/net/knod/knod_attach.sh |   135 +
 .../drivers/net/knod/knod_xdp_ktime.sh        |   173 +
 .../drivers/net/knod/knod_xdp_loop.sh         |   129 +
 .../testing/selftests/drivers/net/knod/lib.sh |   181 +
 .../drivers/net/knod/xdp_ktime.bpf.c          |    32 +
 .../selftests/drivers/net/knod/xdp_loop.bpf.c |    42 +
 67 files changed, 44131 insertions(+), 115 deletions(-)
 create mode 100644 Documentation/netlink/specs/knod.yaml
 create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_hsa.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_knod.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h
 create mode 100644 include/net/devmem.h
 create mode 100644 include/net/knod.h
 create mode 100644 include/net/spsc_ring.h
 create mode 100644 include/uapi/linux/knod.h
 create mode 100644 net/knod/Kconfig
 create mode 100644 net/knod/Makefile
 create mode 100644 net/knod/knod-nl-gen.c
 create mode 100644 net/knod/knod-nl-gen.h
 create mode 100644 net/knod/knod.h
 create mode 100644 net/knod/knod_core.c
 create mode 100644 net/knod/knod_nl.c
 create mode 100644 tools/testing/selftests/drivers/net/knod/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/knod/config
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_attach.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_loop.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/lib.sh
 create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c
 create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c

base-commit: ce6b4d3216b63f902bb8e9695ee6c10c83415f65

Signature · 1 line
2.43.0
author
#1 lore raw

RFC PATCH 1/13 net: knod: add uapi and core headers

Add the uAPI and core headers for KNOD, an in-kernel network offload device that connects a NIC RX path to an accelerator (e.g. a GPU) for zero-copy packet processing.

- include/uapi/linux/knod.h: genetlink uAPI
- Documentation/netlink/specs/knod.yaml: netlink spec
- include/net/knod.h: core framework types and ops
- include/net/spsc_ring.h: lock-free SPSC ring for the data path

(cherry picked from commit d93829acbce14528fcf70e155a0594c0c6151cce)
---
 Documentation/netlink/specs/knod.yaml | 176 +++++++
 include/net/knod.h                    | 467 +++++++++++++++++++
 include/net/spsc_ring.h               | 645 ++++++++++++++++++++++++++
 include/uapi/linux/knod.h             |  67 +++
 4 files changed, 1355 insertions(+)
 create mode 100644 Documentation/netlink/specs/knod.yaml
 create mode 100644 include/net/knod.h
 create mode 100644 include/net/spsc_ring.h
 create mode 100644 include/uapi/linux/knod.h
Documentation/netlink/specs/knod.yaml new file +176 -0
new file mode 100644
index 000000000000..7c64df467634
@@ -0,0 +1,176 @@
+# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+---
+name: knod
+
+doc:
+  KNOD (KFD network offload) control plane.
+
+  Bind a NIC netdevice to a GPU/DPU offload accelerator and select which
+  offload feature (BPF, IPsec) the accelerator runs.
+
+definitions:
+  -
+    type: enum
+    name: feature
+    entries: [none, bpf, ipsec]
+  -
+    type: enum
+    name: accel-type
+    entries: [gpu, dpu]
+
+attribute-sets:
+  -
+    name: accel
+    attributes:
+      -
+        name: id
+        doc: Accelerator id.
+        type: u32
+        checks:
+          min: 1
+      -
+        name: name
+        doc: Accelerator name.
+        type: string
+      -
+        name: type
+        doc: Accelerator type.
+        type: u32
+        enum: accel-type
+      -
+        name: feature-cap
+        doc: Bitmask of offload features the accelerator supports.
+        type: u32
+        enum: feature
+        enum-as-flags: true
+      -
+        name: feature-ena
+        doc: Currently active offload feature on the accelerator.
+        type: u32
+        enum: feature
+  -
+    name: nic
+    attributes:
+      -
+        name: ifindex
+        doc: ifindex of the NIC netdevice registered with KNOD.
+        type: u32
+        checks:
+          min: 1
+      -
+        name: name
+        doc: Name of the NIC netdevice.
+        type: string
+  -
+    name: dev
+    attributes:
+      -
+        name: nic-ifindex
+        doc: ifindex of the bound NIC netdevice.
+        type: u32
+        checks:
+          min: 1
+      -
+        name: accel-id
+        doc: id of the bound accelerator.
+        type: u32
+        checks:
+          min: 1
+
+operations:
+  list:
+    -
+      name: accel-get
+      doc: Get / dump accelerators registered on the system.
+      attribute-set: accel
+      do:
+        request:
+          attributes:
+            - id
+        reply: &accel-all
+          attributes:
+            - id
+            - name
+            - type
+            - feature-cap
+            - feature-ena
+      dump:
+        reply: *accel-all
+    -
+      name: accel-set
+      doc: Select the active offload feature of an accelerator.
+      attribute-set: accel
+      flags: [admin-perm]
+      do:
+        request:
+          attributes:
+            - id
+            - feature-ena
+        reply:
+          attributes: []
+    -
+      name: nic-get
+      doc: Get / dump NICs registered with the KNOD framework.
+      attribute-set: nic
+      do:
+        request:
+          attributes:
+            - ifindex
+        reply: &nic-all
+          attributes:
+            - ifindex
+            - name
+      dump:
+        reply: *nic-all
+    -
+      name: attach
+      doc: Attach a NIC netdevice to an accelerator.
+      attribute-set: dev
+      flags: [admin-perm]
+      do:
+        request:
+          attributes:
+            - nic-ifindex
+            - accel-id
+        reply:
+          attributes: []
+    -
+      name: detach
+      doc: Detach a NIC netdevice from its accelerator.
+      attribute-set: dev
+      flags: [admin-perm]
+      do:
+        request:
+          attributes:
+            - nic-ifindex
+        reply:
+          attributes: []
+    -
+      name: dev-get
+      doc: Get / dump active NIC <-> accelerator bindings.
+      attribute-set: dev
+      do:
+        request:
+          attributes:
+            - nic-ifindex
+        reply: &dev-all
+          attributes:
+            - nic-ifindex
+            - accel-id
+      dump:
+        reply: *dev-all
+    -
+      name: dev-add-ntf
+      doc: Notification about a new NIC <-> accelerator binding.
+      notify: dev-get
+      mcgrp: mgmt
+    -
+      name: dev-del-ntf
+      doc: Notification about a removed NIC <-> accelerator binding.
+      notify: dev-get
+      mcgrp: mgmt
+
+mcast-groups:
+  list:
+    -
+      name: mgmt
include/net/knod.h new file +467 -0

467 lines not on this page · raw message

include/net/spsc_ring.h new file +645 -0

645 lines not on this page · raw message

include/uapi/linux/knod.h new file +67 -0

67 lines not on this page · raw message

Signature · 1 line
2.43.0
#2 lore raw

Re: [RFC PATCH net-next 01/13] net: knod: add uapi and core headers

Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider: - [High] Broken lock-free API: `spsc_push` and `spsc_pop` introduce data races. - [High] Use-After-Free in…

Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
- [High] Broken lock-free API: spsc_push and spsc_pop introduce data races.
- [High] Use-After-Free in knod_napi_kick() due to lifecycle mismatch with standard NAPI teardown.
- [High] NULL pointer dereference in inline helpers when no accelerator is attached.
- [High] spsc_acquire() lacks a Write Memory Barrier (wmb()) before publishing the advanced cursor, risking data corruption on WC memory.
- [Medium] The attach Netlink operation violates uAPI design guidelines by missing reply attributes.
- [Medium] Severe false sharing in struct spsc_ring due to unpadded producer and consumer cursors.
- [Low] Missing n parameter in for_each_xdev_safe macro.
--

--- Patch [1]: [RFC PATCH net-next 01/13] net: knod: add uapi and core headers --- commit 518fe6d810fd2f47140b3bb011b370cf924724ed
Author: Taehee Yoo <[email protected]>

net: knod: add uapi and core headers

Add the uAPI and core headers for KNOD, an in-kernel network offload device that connects a NIC RX path to an accelerator (e.g. a GPU) for zero-copy packet processing.

Quoted 5 lines
Documentation/netlink/specs/knod.yaml new file +0 -0
new file mode 100644
index 0000000000000..7c64df4676349

[ ... ]

Quoted 4 lines
Documentation/netlink/specs/knod.yaml new file +3 -0
@@ -0,0 +1,176 @@
+# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+---
+name: knod

[ ... ]

Documentation/netlink/specs/knod.yaml line 136

Quoted 12 lines
Documentation/netlink/specs/knod.yaml new file +12 -0
+    -
+      name: attach
+      doc: Attach a NIC netdevice to an accelerator.
+      attribute-set: dev
+      flags: [admin-perm]
+      do:
+        request:
+          attributes:
+            - nic-ifindex
+            - accel-id
+        reply:
+          attributes: []

[Severity: Medium]
UAPI design rules state that NEW/ADD operations must reply with a full message describing the new object. By leaving attributes empty, does this permanently freeze a broken ABI for userspace applications relying on generic netlink?

Quoted 4 lines
Documentation/netlink/specs/knod.yaml new file +4 -0
+    -
+      name: detach
+      doc: Detach a NIC netdevice from its accelerator.
+      attribute-set: dev

[ ... ]

Quoted 5 lines
include/net/knod.h new file +0 -0
new file mode 100644
index 0000000000000..50333b0e028b7

[ ... ]

Quoted 4 lines
include/net/knod.h new file +3 -0
@@ -0,0 +1,467 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __NET_KNOD_H
+#define __NET_KNOD_H

[ ... ]

Quoted 19 lines
include/net/knod.h new file +19 -0
+struct knod_work_priv {
+	struct dma_buf *dmabuf;
+	netmem_ref *netmems;
+	unsigned int *data_lens;
+	int *data_offs;
+	int cnt;
+	int index;
+	struct napi_struct *napi;
+	struct spsc_ring spsc_bds;
+	void *spsc_pool_priv;	/* accel driver priv for spsc pool memory */
+	u64 spsc_pool_gaddr;	/* device-visible address of spsc pool */
+	/* framework-owned delivery pool */
+	struct page_pool *pass_pool;
+	/* provider ctx (owner storage) */
+	struct page_pool_hostmem pass_hm;
+	/* d2h: SDMA-issued, awaiting drain */
+	struct spsc_ring pass_pending;
+} ____cacheline_aligned_in_smp;
+

include/net/knod.h line 123

Quoted 10 lines
include/net/knod.h new file +10 -0
+static inline void knod_napi_kick(struct knod_work_priv *wpriv)
+{
+	struct napi_struct *napi;
+
+	rcu_read_lock();
+	napi = READ_ONCE(wpriv->napi);
+	if (napi)
+		napi_schedule(napi);
+	rcu_read_unlock();
+}

[Severity: High]
Does this RCU lock actually protect the NAPI structure from being freed? Standard NIC drivers often dismantle and free their queue arrays synchronously after napi_disable(), without an RCU grace period. Could this lead to a use-after-free when calling napi_schedule()?

Quoted 4 lines
include/net/knod.h new file +4 -0
+
+#define KNOD_DEFAULT_PASS_SLOTS	64
+
+/*

[ ... ]

Quoted 4 lines
include/net/knod.h new file +4 -0
+struct knod_dev {
+	struct list_head list;
+	struct knod_netdev *knetdev;
+	struct knod_accel *accel;

[ ... ]

include/net/knod.h line 406

Quoted 12 lines
include/net/knod.h new file +12 -0
+};
+
+static inline bool knod_dev_active(struct knod_dev *knodev)
+{
+	return !!knodev->accel->xdp.xdp_hw.prog;
+}
+
+static inline struct bpf_prog *
+knod_dev_offloaded(struct knod_dev *knodev)
+{
+	return knodev->accel->xdp.bpf_offloaded;
+}

[Severity: High]
Can knodev->accel be NULL when a NIC is registered but detached from an accelerator? If so, will these inline helpers unconditionally dereference a NULL pointer and cause a kernel panic?

Quoted 3 lines
include/net/knod.h new file +3 -0
+
+static inline void knod_dev_offload(struct knod_dev *knodev,
+					   struct bpf_prog *bpf_offloaded)

[ ... ]

include/net/knod.h line 445

Quoted 6 lines
include/net/knod.h new file +6 -0
+extern struct list_head knod_accel_list;
+
+#define for_each_xdev(d)         \
+		list_for_each_entry(d, &knod_dev_list, list)
+#define for_each_xdev_safe(d)         \
+		list_for_each_entry_safe(d, n, &knod_dev_list, list)

[Severity: Low]
Is the n parameter missing from the macro signature? Any file using this macro without a pre-declared local variable n will likely fail to compile.

Quoted 2 lines
include/net/knod.h new file +2 -0
+#define for_each_nodev(d)         \
+		list_for_each_entry(d, &knod_netdev_list, list)

[ ... ]

Quoted 5 lines
include/net/spsc_ring.h new file +0 -0
new file mode 100644
index 0000000000000..1eccadc927d62

[ ... ]

Quoted 3 lines
include/net/spsc_ring.h new file +2 -0
@@ -0,0 +1,645 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*

[ ... ]

Quoted 5 lines
include/net/spsc_ring.h new file +5 -0
+struct spsc_ring {
+	void		**slots;	/* pointer-per-slot into pool */
+	unsigned int	mask;		/* capacity - 1 */
+
+	/* Producer side */

include/net/spsc_ring.h line 69

Quoted 10 lines
include/net/spsc_ring.h new file +10 -0
+	unsigned int	head;		/* next slot to publish */
+
+	/* Consumer side */
+	unsigned int	tail;		/* oldest unconsumed slot */
+	unsigned int	acquired;	/* sliding window read cursor
+					 *   tail <= acquired <= head
+					 */
+
+	/* Element pool */
+	struct page	*pool_page;	/* compound page backing elements */

[Severity: Medium]
Does placing the heavily contended producer cursor (head) and consumer cursors (tail, acquired) in the same cacheline without padding cause severe false sharing? This could significantly impact throughput due to continuous cacheline bouncing.

Quoted 5 lines
include/net/spsc_ring.h new file +5 -0
+	unsigned int	pool_order;	/* page order */
+	unsigned int	elem_stride;	/* cacheline-aligned element size */
+} ____cacheline_aligned_in_smp;
+
+/* ================================================================== */

[ ... ]

Quoted 8 lines
include/net/spsc_ring.h new file +8 -0
+static inline int spsc_acquire(struct spsc_ring *r, void **out,
+			       unsigned int max, unsigned int *cnt)
+{
+	/* acquire the producer's head; slots it published are now visible */
+	unsigned int head = smp_load_acquire(&r->head);
+	unsigned int acq = r->acquired;
+	unsigned int avail;
+	unsigned int i;

[ ... ]

include/net/spsc_ring.h line 443

Quoted 11 lines
include/net/spsc_ring.h new file +11 -0
+	if (out) {
+		for (i = 0; i < avail; i++)
+			out[i] = r->slots[(acq + i) & r->mask];
+	}
+
+	/* Publish the window to the releasing consumer (a different CPU than
+	 * this acquirer): pair with the smp_load_acquire() in spsc_release()
+	 * so it cannot observe the advanced cursor before the element stores
+	 * (e.g. an accel verdict) those slots now point at.
+	 */
+	smp_store_release(&r->acquired, acq + avail);

[Severity: High]
On architectures like x86, smp_store_release is only a compiler barrier and does not flush Write-Combining buffers. Is a wmb() needed here before publishing the advanced cursor, similar to spsc_produce_commit, to prevent a concurrent releaser from reading stale memory before the WC stores land?

Quoted 7 lines
include/net/spsc_ring.h new file +7 -0
+
+	if (cnt)
+		*cnt = avail;
+	return 0;
+}
+
+/**

[ ... ]

include/net/spsc_ring.h line 576

Quoted 11 lines
include/net/spsc_ring.h new file +11 -0
+static inline int spsc_push(struct spsc_ring *r, void **out)
+{
+	int ret;
+
+	ret = spsc_produce(r, out);
+	if (ret)
+		return ret;
+
+	spsc_produce_commit(r);
+	return 0;
+}

[Severity: High]
Doesn't spsc_produce_commit publish the slot to the consumer immediately? If the slot is published before the caller writes the data into the pointer returned in out, could a concurrent consumer read uninitialized data?

Quoted 18 lines
include/net/spsc_ring.h new file +18 -0
+
+/**
+ * spsc_pop - dequeue one element
+ * @r:   ring buffer
+ * @out: receives pointer to the consumed element
+ *
+ * The element pointer remains valid until the next spsc_push() or
+ * spsc_produce() reuses that slot.
+ *
+ * Returns 0 on success, -ENOENT if empty.
+ */
+static inline int spsc_pop(struct spsc_ring *r, void **out)
+{
+	/* acquire the producer's head; slots it published are now visible */
+	unsigned int head = smp_load_acquire(&r->head);
+	unsigned int tail = r->tail;
+
+	if (tail == head)

include/net/spsc_ring.h line 604

Quoted 10 lines
include/net/spsc_ring.h new file +10 -0
+		return -ENOENT;
+
+	*out = r->slots[tail & r->mask];
+
+	r->acquired = tail + 1;
+	/* release: hand the consumed slot back to the producer */
+	smp_store_release(&r->tail, tail + 1);
+
+	return 0;
+}

[Severity: High]
By advancing the tail cursor before returning the element pointer to the consumer, is the slot made available to the producer too early? A concurrent producer might overwrite the data while the consumer is actively reading it.

Quoted 3 lines
include/net/spsc_ring.h new file +3 -0
+
+/**
+ * spsc_pop_n - dequeue up to @n elements

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#3 lore raw

RFC PATCH 2/13 net: devmem: extend memory provider for knod

In reply to Taehee Yoo

Extend the devmem memory-provider path so a knod accelerator can back a NIC page_pool with accelerator-exported memory (dma-buf), letting the NIC DMA received packets directly into accelerator memory.

(cherry picked from commit d511a8cb3e229f8f5cf060985880d45bd384db87)
---
 include/net/devmem.h                    |  58 +++++++++++++
 include/net/netmem.h                    |   9 +++
 include/net/page_pool/memory_provider.h |   4 +
 include/net/page_pool/types.h           |  23 +++++-
 net/core/devmem.c                       | 103 +++++++++++++++++++-----
 net/core/devmem.h                       |   7 +-
 net/core/page_pool.c                    |  22 ++++-
 7 files changed, 198 insertions(+), 28 deletions(-)
 create mode 100644 include/net/devmem.h
include/net/devmem.h new file +58 -0
new file mode 100644
index 000000000000..f1c3895d7833
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Device memory TCP support
+ *
+ * Authors:	Mina Almasry <[email protected]>
+ *		Willem de Bruijn <[email protected]>
+ *		Kaiyuan Zhang <[email protected]>
+ *
+ */
+#ifndef _NET_DEVMEM_H
+#define _NET_DEVMEM_H
+
+#include <linux/dma-direction.h>
+#include <linux/err.h>
+#include <linux/types.h>
+
+struct device;
+struct dma_buf;
+struct dma_buf_attach_ops;
+struct net_device;
+struct net_devmem_dmabuf_binding;
+struct netlink_ext_ack;
+
+#if defined(CONFIG_NET_DEVMEM)
+struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+			    struct dma_buf *dmabuf,
+			    enum dma_data_direction direction,
+			    const struct dma_buf_attach_ops *importer_ops,
+			    struct netlink_ext_ack *extack);
+int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+					   struct net_devmem_dmabuf_binding *binding);
+void net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding);
+#else
+static inline struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+			    struct dma_buf *dmabuf,
+			    enum dma_data_direction direction,
+			    const struct dma_buf_attach_ops *importer_ops,
+			    struct netlink_ext_ack *extack)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline int
+net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+				       struct net_devmem_dmabuf_binding *binding)
+{
+	return -EOPNOTSUPP;
+}
+
+static inline void
+net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding)
+{
+}
+#endif
+
+#endif /* _NET_DEVMEM_H */
include/net/netmem.h +9 -0
index bccacd21b6c3..3ddfbd37500f 100644
@@ -127,6 +127,15 @@ static inline void net_iov_init(struct net_iov *niov,
 	niov->type = type;
 }
 
+/* Global page index within the dma-buf, accounting for multi-chunk
+ * scatter-gather layouts where each chunk owner's niovs start at 0.
+ */
+static inline unsigned int net_iov_binding_idx(const struct net_iov *niov)
+{
+	return (net_iov_owner(niov)->base_virtual >> PAGE_SHIFT) +
+	       net_iov_idx(niov);
+}
+
 /* netmem */
 
 /**
include/net/page_pool/memory_provider.h +4 -0
index 255ce4cfd975..4b58a9702fb7 100644
@@ -23,6 +23,10 @@ bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr);
 void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *niov);
 void net_mp_niov_clear_page_pool(struct net_iov *niov);
 
+void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netmem,
+				   dma_addr_t addr);
+void page_pool_clear_pp_info(netmem_ref netmem);
+
 int netif_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx,
 		      const struct pp_memory_provider_params *p,
 		      struct netlink_ext_ack *extack);
include/net/page_pool/types.h +19 -4
index 03da138722f5..3e866f249768 100644
@@ -31,8 +31,16 @@
  */
 #define PP_FLAG_ALLOW_UNREADABLE_NETMEM BIT(3)
 
+/* Driver-managed pool with a directly-supplied memory provider, not bound to a
+ * netdev rx queue. Setting this flag requires page_pool_params.mp_ops and
+ * .mp_priv to both be set.
+ */
+#define PP_FLAG_CUSTOM_MEMORY_PROVIDER	BIT(4)
+
 #define PP_FLAG_ALL		(PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \
-				 PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM)
+				 PP_FLAG_SYSTEM_POOL | \
+				 PP_FLAG_ALLOW_UNREADABLE_NETMEM | \
+				 PP_FLAG_CUSTOM_MEMORY_PROVIDER)
 
 /* Index limit to stay within PP_DMA_INDEX_BITS for DMA indices */
 #define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1)
@@ -54,11 +62,11 @@
  * would have to take a slower code path.
  */
 #if PAGE_SIZE >= SZ_64K
-#define PP_ALLOC_CACHE_REFILL	4
+#define PP_ALLOC_CACHE_REFILL	256
 #elif PAGE_SIZE >= SZ_16K
-#define PP_ALLOC_CACHE_REFILL	16
+#define PP_ALLOC_CACHE_REFILL	1024
 #else
-#define PP_ALLOC_CACHE_REFILL	64
+#define PP_ALLOC_CACHE_REFILL	4096
 #endif
 
 #define PP_ALLOC_CACHE_SIZE	(PP_ALLOC_CACHE_REFILL * 2)
@@ -67,6 +75,8 @@ struct pp_alloc_cache {
 	netmem_ref cache[PP_ALLOC_CACHE_SIZE];
 };
 
+struct memory_provider_ops;
+
 /**
  * struct page_pool_params - page pool parameters
  * @fast:	params accessed frequently on hotpath
@@ -83,6 +93,9 @@ struct pp_alloc_cache {
  * @queue_idx:	queue idx this page_pool is being created for.
  * @flags:	PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL,
  *		PP_FLAG_ALLOW_UNREADABLE_NETMEM.
+ * @mp_ops:	driver-supplied memory provider for a pool not bound to a
+ *		netdev rx queue (NULL to use rxq->mp_params instead)
+ * @mp_priv:	context passed to @mp_ops
  */
 struct page_pool_params {
 	struct_group_tagged(page_pool_params_fast, fast,
@@ -99,6 +112,8 @@ struct page_pool_params {
 		struct net_device *netdev;
 		unsigned int queue_idx;
 		unsigned int	flags;
+		const struct memory_provider_ops *mp_ops;
+		void *mp_priv;
 /* private: used by test code only */
 		void (*init_callback)(netmem_ref netmem, void *arg);
 		void *init_arg;
net/core/devmem.c +83 -20

157 lines not on this page · raw message

net/core/devmem.h +4 -3
index 3852a56036cb..6b57837ab454 100644
@@ -7,9 +7,10 @@
  *		Kaiyuan Zhang <[email protected]>
  *
  */
-#ifndef _NET_DEVMEM_H
-#define _NET_DEVMEM_H
+#ifndef _NET_CORE_DEVMEM_H
+#define _NET_CORE_DEVMEM_H
 
+#include <net/devmem.h>
 #include <net/netmem.h>
 #include <net/netdev_netlink.h>
 
@@ -240,4 +241,4 @@ net_devmem_iov_binding(const struct net_iov *niov)
 }
 #endif
 
-#endif /* _NET_DEVMEM_H */
+#endif /* _NET_CORE_DEVMEM_H */
net/core/page_pool.c +21 -1
index 21dc4a9c8714..fd7444943b9f 100644
@@ -272,7 +272,17 @@ static int page_pool_init(struct page_pool *pool,
 
 	xa_init_flags(&pool->dma_mapped, XA_FLAGS_ALLOC1);
 
-	if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) {
+	if (pool->slow.flags & PP_FLAG_CUSTOM_MEMORY_PROVIDER) {
+		/* Driver-managed pool with a directly-supplied memory
+		 * provider, not bound to a netdev rx queue.
+		 */
+		if (WARN_ON(!pool->slow.mp_ops || !pool->slow.mp_priv)) {
+			err = -EINVAL;
+			goto free_ptr_ring;
+		}
+		pool->mp_priv = pool->slow.mp_priv;
+		pool->mp_ops = pool->slow.mp_ops;
+	} else if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) {
 		netdev_assert_locked(pool->slow.netdev);
 		rxq = __netif_get_rx_queue(pool->slow.netdev,
 					   pool->slow.queue_idx);
@@ -725,6 +735,16 @@ void page_pool_clear_pp_info(netmem_ref netmem)
 	netmem_set_pp(netmem, NULL);
 }
 
+void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netmem,
+				   dma_addr_t addr)
+{
+	netmem_to_nmdesc(netmem)->pp_magic = 0;
+	netmem_to_nmdesc(netmem)->pp = NULL;
+	atomic_long_set(&netmem_to_nmdesc(netmem)->pp_ref_count, 0);
+	page_pool_set_pp_info(pool, netmem);
+	page_pool_set_dma_addr_netmem(netmem, addr);
+}
+
 static __always_inline void __page_pool_release_netmem_dma(struct page_pool *pool,
 							   netmem_ref netmem)
 {
Signature · 1 line
2.43.0
#4 lore raw

Re: [RFC PATCH net-next 02/13] net: devmem: extend memory provider for knod

Quoted 6 lines · Taehee Yoo

Extend the devmem memory-provider path so a knod accelerator can back a NIC page_pool with accelerator-exported memory (dma-buf), letting the NIC DMA received packets directly into accelerator memory.

Changes look fine in general but I do not understand in the design how the page_pool not bound to rx queue would be used. Maybe elaborate on the design in the commit message or cover letter in the next iteration.

Quoted 1 line

(cherry picked from commit d511a8cb3e229f8f5cf060985880d45bd384db87)

I guess remove unintended cherry-pick tag.

include/net/devmem.h

Quoted 16 lines
---
 include/net/devmem.h                    |  58 +++++++++++++
 include/net/netmem.h                    |   9 +++
 include/net/page_pool/memory_provider.h |   4 +
 include/net/page_pool/types.h           |  23 +++++-
 net/core/devmem.c                       | 103 +++++++++++++++++++-----
 net/core/devmem.h                       |   7 +-
 net/core/page_pool.c                    |  22 ++++-
 7 files changed, 198 insertions(+), 28 deletions(-)
 create mode 100644 include/net/devmem.h
include/net/devmem.h new file +0 -0
new file mode 100644
index 000000000000..f1c3895d7833

Try to keep only include/net/netmem.h and net/core/devmem.h rather than add an include/net/devmem.h

Quoted 120 lines
include/net/devmem.h new file +58 -0
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Device memory TCP support
+ *
+ * Authors:    Mina Almasry <[email protected]>
+ *             Willem de Bruijn <[email protected]>
+ *             Kaiyuan Zhang <[email protected]>
+ *
+ */
+#ifndef _NET_DEVMEM_H
+#define _NET_DEVMEM_H
+
+#include <linux/dma-direction.h>
+#include <linux/err.h>
+#include <linux/types.h>
+
+struct device;
+struct dma_buf;
+struct dma_buf_attach_ops;
+struct net_device;
+struct net_devmem_dmabuf_binding;
+struct netlink_ext_ack;
+
+#if defined(CONFIG_NET_DEVMEM)
+struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+                           struct dma_buf *dmabuf,
+                           enum dma_data_direction direction,
+                           const struct dma_buf_attach_ops *importer_ops,
+                           struct netlink_ext_ack *extack);
+int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                          struct net_devmem_dmabuf_binding *binding);
+void net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding);
+#else
+static inline struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+                           struct dma_buf *dmabuf,
+                           enum dma_data_direction direction,
+                           const struct dma_buf_attach_ops *importer_ops,
+                           struct netlink_ext_ack *extack)
+{
+       return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline int
+net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                      struct net_devmem_dmabuf_binding *binding)
+{
+       return -EOPNOTSUPP;
+}
+
+static inline void
+net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding)
+{
+}
+#endif
+
+#endif /* _NET_DEVMEM_H */
include/net/netmem.h +9 -0
index bccacd21b6c3..3ddfbd37500f 100644
@@ -127,6 +127,15 @@ static inline void net_iov_init(struct net_iov *niov,
        niov->type = type;
 }
 
+/* Global page index within the dma-buf, accounting for multi-chunk
+ * scatter-gather layouts where each chunk owner's niovs start at 0.
+ */
+static inline unsigned int net_iov_binding_idx(const struct net_iov *niov)
+{
+       return (net_iov_owner(niov)->base_virtual >> PAGE_SHIFT) +
+              net_iov_idx(niov);
+}
+
 /* netmem */
 
 /**
include/net/page_pool/memory_provider.h +4 -0
index 255ce4cfd975..4b58a9702fb7 100644
@@ -23,6 +23,10 @@ bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr);
 void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *niov);
 void net_mp_niov_clear_page_pool(struct net_iov *niov);
 
+void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netmem,
+                                  dma_addr_t addr);
+void page_pool_clear_pp_info(netmem_ref netmem);
+
 int netif_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx,
                      const struct pp_memory_provider_params *p,
                      struct netlink_ext_ack *extack);
include/net/page_pool/types.h +9 -1
index 03da138722f5..3e866f249768 100644
@@ -31,8 +31,16 @@
  */
 #define PP_FLAG_ALLOW_UNREADABLE_NETMEM BIT(3)
 
+/* Driver-managed pool with a directly-supplied memory provider, not bound to a
+ * netdev rx queue. Setting this flag requires page_pool_params.mp_ops and
+ * .mp_priv to both be set.
+ */
+#define PP_FLAG_CUSTOM_MEMORY_PROVIDER BIT(4)
+
 #define PP_FLAG_ALL            (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \
-                                PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM)
+                                PP_FLAG_SYSTEM_POOL | \
+                                PP_FLAG_ALLOW_UNREADABLE_NETMEM | \
+                                PP_FLAG_CUSTOM_MEMORY_PROVIDER)
 
 /* Index limit to stay within PP_DMA_INDEX_BITS for DMA indices */
 #define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1)
@@ -54,11 +62,11 @@
  * would have to take a slower code path.
  */
 #if PAGE_SIZE >= SZ_64K

include/net/page_pool/types.h line 71

Quoted 10 lines
include/net/page_pool/types.h +3 -3
-#define PP_ALLOC_CACHE_REFILL  4
+#define PP_ALLOC_CACHE_REFILL  256
 #elif PAGE_SIZE >= SZ_16K
-#define PP_ALLOC_CACHE_REFILL  16
+#define PP_ALLOC_CACHE_REFILL  1024
 #else
-#define PP_ALLOC_CACHE_REFILL  64
+#define PP_ALLOC_CACHE_REFILL  4096
 #endif
 

I'm guessing this is just a workaround/optimization. You need a proper change for this, maybe make it configurable arguments when creating a pp or something.

Quoted 66 lines
include/net/page_pool/types.h +7 -0
 #define PP_ALLOC_CACHE_SIZE    (PP_ALLOC_CACHE_REFILL * 2)
@@ -67,6 +75,8 @@ struct pp_alloc_cache {
        netmem_ref cache[PP_ALLOC_CACHE_SIZE];
 };
 
+struct memory_provider_ops;
+
 /**
  * struct page_pool_params - page pool parameters
  * @fast:      params accessed frequently on hotpath
@@ -83,6 +93,9 @@ struct pp_alloc_cache {
  * @queue_idx: queue idx this page_pool is being created for.
  * @flags:     PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL,
  *             PP_FLAG_ALLOW_UNREADABLE_NETMEM.
+ * @mp_ops:    driver-supplied memory provider for a pool not bound to a
+ *             netdev rx queue (NULL to use rxq->mp_params instead)
+ * @mp_priv:   context passed to @mp_ops
  */
 struct page_pool_params {
        struct_group_tagged(page_pool_params_fast, fast,
@@ -99,6 +112,8 @@ struct page_pool_params {
                struct net_device *netdev;
                unsigned int queue_idx;
                unsigned int    flags;
+               const struct memory_provider_ops *mp_ops;
+               void *mp_priv;
 /* private: used by test code only */
                void (*init_callback)(netmem_ref netmem, void *arg);
                void *init_arg;
net/core/devmem.c +11 -5
index 957d6b96216b..9e21cffc9643 100644
@@ -121,12 +121,9 @@ void net_devmem_free_dmabuf(struct net_iov *niov)
        gen_pool_free(binding->chunk_pool, dma_addr, PAGE_SIZE);
 }
 
-void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
+static void
+net_devmem_binding_unpublish(struct net_devmem_dmabuf_binding *binding)
 {
-       struct netdev_rx_queue *rxq;
-       unsigned long xa_idx;
-       unsigned int rxq_idx;
-
        xa_erase(&net_devmem_dmabuf_bindings, binding->id);
 
        /* Ensure no tx net_devmem_lookup_dmabuf() are in flight after the
@@ -136,6 +133,15 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
 
        if (binding->list.next)
                list_del(&binding->list);
+}
+
+void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
+{
+       struct netdev_rx_queue *rxq;
+       unsigned long xa_idx;
+       unsigned int rxq_idx;
+
+       net_devmem_binding_unpublish(binding);
 
        xa_for_each(&binding->bound_rxqs, xa_idx, rxq) {
                const struct pp_memory_provider_params mp_params = {
@@ -151,6 +157,47 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
        percpu_ref_kill(&binding->ref);

net/core/devmem.c void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) · line 167

Quoted 10 lines
net/core/devmem.c +8 -0
 }
 
+/* Bind/unbind variants for in-kernel offload importers that drive the rx
+ * queue lifecycle themselves. The mp_params are poked directly, without the
+ * tcp-data-split/XDP guards or the queue reconfigure that the netlink control
+ * plane applies through netif_mp_open_rxq()/netif_mp_close_rxq().
+ */
+int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                          struct net_devmem_dmabuf_binding *binding)
+{

The LLM says in this function you need to check if the queue is already bound to a devmem tcp memory provider.

Signature · 2 lines
Thanks,
Mina
author
#5 lore raw

Re: [RFC PATCH net-next 02/13] net: devmem: extend memory provider for knod

Quoted 1 line · Mina Almasry

Hi Mina,
Thanks a lot for your review!

Quoted 6 lines · Taehee Yoo
Quoted 5 lines

Extend the devmem memory-provider path so a knod accelerator can back a NIC page_pool with accelerator-exported memory (dma-buf), letting the NIC DMA received packets directly into accelerator memory.

Quoted 7 lines
Quoted 1 line

Changes look fine in general but I do not understand in the design how the page_pool not bound to rx queue would be used. Maybe elaborate on the design in the commit message or cover letter in the next iteration.

Sorry, this is not clear from the commit message or cover letter.

We added a custom provider type for page_pool that allocates GTT memory
- host memory that is also mapped to a GPU VA, which lets us copy between VRAM and host without a separate staging buffer. So when the XDP verdict is XDP_PASS, or on the IPsec path, this page_pool is used for the allocation. To configure the page_pool with our KNOD backend, we define mp_ops/priv in knod_core.c and set them directly as the page_pool's mp_ops/priv, rather than going through netdev_rx_queue.

As a result, each combined channel has two page_pools: a NETMEM page_pool created by the device driver to refill the RX ring through the normal netdev_rx_queue path, and one CUSTOM_PROVIDER page_pool created and managed by knod core for the XDP_PASS / IPsec allocations. The latter is created per channel by knod core, so it is not bound through netdev_rx_queue.

We will describe this distinction in the next revision.

Quoted 4 lines
Quoted 1 line

(cherry picked from commit d511a8cb3e229f8f5cf060985880d45bd384db87)

I guess remove unintended cherry-pick tag.

Yes, we will remove it.

Quoted 11 lines
Quoted 11 lines
---
 include/net/devmem.h                    |  58 +++++++++++++
 include/net/netmem.h                    |   9 +++
 include/net/page_pool/memory_provider.h |   4 +
 include/net/page_pool/types.h           |  23 +++++-
 net/core/devmem.c                       | 103 +++++++++++++++++++-----
 net/core/devmem.h                       |   7 +-
 net/core/page_pool.c                    |  22 ++++-
 7 files changed, 198 insertions(+), 28 deletions(-)
 create mode 100644 include/net/devmem.h
Quoted 9 lines

include/net/devmem.h

Quoted 5 lines
include/net/devmem.h new file +0 -0
new file mode 100644
index 000000000000..f1c3895d7833

Try to keep only include/net/netmem.h and net/core/devmem.h rather than add an include/net/devmem.h

Thanks. We will keep the declarations in include/net/netmem.h and

Quoted 126 lines
Quoted 126 lines
include/net/devmem.h new file +58 -0
@@ -0,0 +1,58 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Device memory TCP support
+ *
+ * Authors:    Mina Almasry <[email protected]>
+ *             Willem de Bruijn <[email protected]>
+ *             Kaiyuan Zhang <[email protected]>
+ *
+ */
+#ifndef _NET_DEVMEM_H
+#define _NET_DEVMEM_H
+
+#include <linux/dma-direction.h>
+#include <linux/err.h>
+#include <linux/types.h>
+
+struct device;
+struct dma_buf;
+struct dma_buf_attach_ops;
+struct net_device;
+struct net_devmem_dmabuf_binding;
+struct netlink_ext_ack;
+
+#if defined(CONFIG_NET_DEVMEM)
+struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+                           struct dma_buf *dmabuf,
+                           enum dma_data_direction direction,
+                           const struct dma_buf_attach_ops *importer_ops,
+                           struct netlink_ext_ack *extack);
+int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                          struct net_devmem_dmabuf_binding *binding);
+void net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding);
+#else
+static inline struct net_devmem_dmabuf_binding *
+__net_devmem_binding_create(struct net_device *dev, struct device *dma_dev,
+                           struct dma_buf *dmabuf,
+                           enum dma_data_direction direction,
+                           const struct dma_buf_attach_ops *importer_ops,
+                           struct netlink_ext_ack *extack)
+{
+       return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline int
+net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                      struct net_devmem_dmabuf_binding *binding)
+{
+       return -EOPNOTSUPP;
+}
+
+static inline void
+net_devmem_unbind_dmabuf_direct(struct net_devmem_dmabuf_binding *binding)
+{
+}
+#endif
+
+#endif /* _NET_DEVMEM_H */
include/net/netmem.h +9 -0
index bccacd21b6c3..3ddfbd37500f 100644
@@ -127,6 +127,15 @@ static inline void net_iov_init(struct net_iov *niov,
        niov->type = type;
 }
 
+/* Global page index within the dma-buf, accounting for multi-chunk
+ * scatter-gather layouts where each chunk owner's niovs start at 0.
+ */
+static inline unsigned int net_iov_binding_idx(const struct net_iov *niov)
+{
+       return (net_iov_owner(niov)->base_virtual >> PAGE_SHIFT) +
+              net_iov_idx(niov);
+}
+
 /* netmem */
 
 /**
include/net/page_pool/memory_provider.h +4 -0
index 255ce4cfd975..4b58a9702fb7 100644
@@ -23,6 +23,10 @@ bool net_mp_niov_set_dma_addr(struct net_iov *niov, dma_addr_t addr);
 void net_mp_niov_set_page_pool(struct page_pool *pool, struct net_iov *niov);
 void net_mp_niov_clear_page_pool(struct net_iov *niov);
 
+void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netmem,
+                                  dma_addr_t addr);
+void page_pool_clear_pp_info(netmem_ref netmem);
+
 int netif_mp_open_rxq(struct net_device *dev, unsigned int rxq_idx,
                      const struct pp_memory_provider_params *p,
                      struct netlink_ext_ack *extack);
include/net/page_pool/types.h +11 -3
index 03da138722f5..3e866f249768 100644
@@ -31,8 +31,16 @@
  */
 #define PP_FLAG_ALLOW_UNREADABLE_NETMEM BIT(3)
 
+/* Driver-managed pool with a directly-supplied memory provider, not bound to a
+ * netdev rx queue. Setting this flag requires page_pool_params.mp_ops and
+ * .mp_priv to both be set.
+ */
+#define PP_FLAG_CUSTOM_MEMORY_PROVIDER BIT(4)
+
 #define PP_FLAG_ALL            (PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV | \
-                                PP_FLAG_SYSTEM_POOL | PP_FLAG_ALLOW_UNREADABLE_NETMEM)
+                                PP_FLAG_SYSTEM_POOL | \
+                                PP_FLAG_ALLOW_UNREADABLE_NETMEM | \
+                                PP_FLAG_CUSTOM_MEMORY_PROVIDER)
 
 /* Index limit to stay within PP_DMA_INDEX_BITS for DMA indices */
 #define PP_DMA_INDEX_LIMIT XA_LIMIT(1, BIT(PP_DMA_INDEX_BITS) - 1)
@@ -54,11 +62,11 @@
  * would have to take a slower code path.
  */
 #if PAGE_SIZE >= SZ_64K
-#define PP_ALLOC_CACHE_REFILL  4
+#define PP_ALLOC_CACHE_REFILL  256
 #elif PAGE_SIZE >= SZ_16K
-#define PP_ALLOC_CACHE_REFILL  16
+#define PP_ALLOC_CACHE_REFILL  1024
 #else
Quoted 9 lines

include/net/page_pool/types.h line 71

Quoted 4 lines
include/net/page_pool/types.h +1 -1
-#define PP_ALLOC_CACHE_REFILL  64
+#define PP_ALLOC_CACHE_REFILL  4096
 #endif
 

I'm guessing this is just a workaround/optimization. You need a proper change for this, maybe make it configurable arguments when creating a pp or something.

Agreed. We will rework this as a per-pool setting instead of changing the global page_pool cache size.

Quoted 68 lines
Quoted 68 lines
include/net/page_pool/types.h +7 -0
 #define PP_ALLOC_CACHE_SIZE    (PP_ALLOC_CACHE_REFILL * 2)
@@ -67,6 +75,8 @@ struct pp_alloc_cache {
        netmem_ref cache[PP_ALLOC_CACHE_SIZE];
 };
 
+struct memory_provider_ops;
+
 /**
  * struct page_pool_params - page pool parameters
  * @fast:      params accessed frequently on hotpath
@@ -83,6 +93,9 @@ struct pp_alloc_cache {
  * @queue_idx: queue idx this page_pool is being created for.
  * @flags:     PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV, PP_FLAG_SYSTEM_POOL,
  *             PP_FLAG_ALLOW_UNREADABLE_NETMEM.
+ * @mp_ops:    driver-supplied memory provider for a pool not bound to a
+ *             netdev rx queue (NULL to use rxq->mp_params instead)
+ * @mp_priv:   context passed to @mp_ops
  */
 struct page_pool_params {
        struct_group_tagged(page_pool_params_fast, fast,
@@ -99,6 +112,8 @@ struct page_pool_params {
                struct net_device *netdev;
                unsigned int queue_idx;
                unsigned int    flags;
+               const struct memory_provider_ops *mp_ops;
+               void *mp_priv;
 /* private: used by test code only */
                void (*init_callback)(netmem_ref netmem, void *arg);
                void *init_arg;
net/core/devmem.c +11 -5
index 957d6b96216b..9e21cffc9643 100644
@@ -121,12 +121,9 @@ void net_devmem_free_dmabuf(struct net_iov *niov)
        gen_pool_free(binding->chunk_pool, dma_addr, PAGE_SIZE);
 }
 
-void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
+static void
+net_devmem_binding_unpublish(struct net_devmem_dmabuf_binding *binding)
 {
-       struct netdev_rx_queue *rxq;
-       unsigned long xa_idx;
-       unsigned int rxq_idx;
-
        xa_erase(&net_devmem_dmabuf_bindings, binding->id);
 
        /* Ensure no tx net_devmem_lookup_dmabuf() are in flight after the
@@ -136,6 +133,15 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
 
        if (binding->list.next)
                list_del(&binding->list);
+}
+
+void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
+{
+       struct netdev_rx_queue *rxq;
+       unsigned long xa_idx;
+       unsigned int rxq_idx;
+
+       net_devmem_binding_unpublish(binding);
 
        xa_for_each(&binding->bound_rxqs, xa_idx, rxq) {
                const struct pp_memory_provider_params mp_params = {
@@ -151,6 +157,47 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding)
        percpu_ref_kill(&binding->ref);
 }
 
Quoted 12 lines

net/core/devmem.c void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) · line 167

Quoted 8 lines
net/core/devmem.c +8 -0
+/* Bind/unbind variants for in-kernel offload importers that drive the rx
+ * queue lifecycle themselves. The mp_params are poked directly, without the
+ * tcp-data-split/XDP guards or the queue reconfigure that the netlink control
+ * plane applies through netif_mp_open_rxq()/netif_mp_close_rxq().
+ */
+int net_devmem_bind_dmabuf_to_queue_direct(struct net_device *dev, u32 rxq_idx,
+                                          struct net_devmem_dmabuf_binding *binding)
+{

The LLM says in this function you need to check if the queue is already bound to a devmem tcp memory provider.

Right. We will reject the bind if the RX queue already has a memory provider.

Thanks a lot!
Taehee Yoo

Quoted 3 lines

--
Thanks,
Mina

#6 lore raw

Re: [RFC PATCH net-next 02/13] net: devmem: extend memory provider for knod

In reply to Taehee Yoo
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The patch silently increases PP_ALLOC_CACHE_REFILL from 64 to up to 4096 (for 4K pages), severely…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The patch silently increases PP_ALLOC_CACHE_REFILL from 64 to up to 4096 (for 4K pages), severely bloating struct page_pool and causing order-5 page allocation failures.
- [High] Ignored error return value of page_pool_set_dma_addr_netmem can lead to silent DMA address truncation and memory corruption on 32-bit architectures with 64-bit DMA.
- [Medium] The is_kernel_rodata check in page_pool_init prevents loadable driver modules from utilizing the new PP_FLAG_CUSTOM_MEMORY_PROVIDER feature. --

--- Patch [2]: [RFC PATCH net-next 02/13] net: devmem: extend memory provider for knod --- commit 685f164a3b04db75b3dba6257fd2806218155a24
Author: Taehee Yoo <[email protected]>

net: devmem: extend memory provider for knod

This commit extends the devmem memory-provider path so a knod accelerator can back a NIC page_pool with accelerator-exported memory. This lets the NIC DMA received packets directly into accelerator memory.

Quoted 4 lines
include/net/page_pool/types.h +0 -0
index 03da138722f58..3e866f249768e 100644

[ ... ]

include/net/page_pool/types.h line 70

Quoted 13 lines
include/net/page_pool/types.h +3 -3
@@ -54,11 +62,11 @@
  * would have to take a slower code path.
  */
 #if PAGE_SIZE >= SZ_64K
-#define PP_ALLOC_CACHE_REFILL	4
+#define PP_ALLOC_CACHE_REFILL	256
 #elif PAGE_SIZE >= SZ_16K
-#define PP_ALLOC_CACHE_REFILL	16
+#define PP_ALLOC_CACHE_REFILL	1024
 #else
-#define PP_ALLOC_CACHE_REFILL	64
+#define PP_ALLOC_CACHE_REFILL	4096
 #endif

[Severity: High]
Will this increase to PP_ALLOC_CACHE_REFILL cause page_pool_create() to fail unpredictably under memory fragmentation?

Since PP_ALLOC_CACHE_SIZE is defined as PP_ALLOC_CACHE_REFILL * 2, the inline cache array inside struct page_pool grows to 8,192 entries for 4K pages. At 8 bytes per pointer, this array alone consumes ~65.5KB.

This forces an order-5 kzalloc_node allocation for the page_pool struct, which is highly prone to failure under fragmentation and drastically bloats the memory footprint per RX queue.

[ ... ]

Quoted 9 lines
net/core/page_pool.c +0 -1
index 21dc4a9c8714d..fd7444943b9f5 100644
@@ -272,7 +272,17 @@ static int page_pool_init(struct page_pool *pool,
 
 	xa_init_flags(&pool->dma_mapped, XA_FLAGS_ALLOC1);
 
-	if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) {

net/core/page_pool.c static int page_pool_init(struct page_pool *pool, · line 284

Quoted 10 lines
net/core/page_pool.c +10 -0
+	if (pool->slow.flags & PP_FLAG_CUSTOM_MEMORY_PROVIDER) {
+		/* Driver-managed pool with a directly-supplied memory
+		 * provider, not bound to a netdev rx queue.
+		 */
+		if (WARN_ON(!pool->slow.mp_ops || !pool->slow.mp_priv)) {
+			err = -EINVAL;
+			goto free_ptr_ring;
+		}
+		pool->mp_priv = pool->slow.mp_priv;
+		pool->mp_ops = pool->slow.mp_ops;

[Severity: Medium]
Will this new flag work for loadable driver modules?

When PP_FLAG_CUSTOM_MEMORY_PROVIDER is used, pool->mp_ops is populated here. Later in page_pool_init(), there is a preexisting check:

    if (WARN_ON(!is_kernel_rodata((unsigned long)pool->mp_ops))) {
        err = -EFAULT;
        goto free_ptr_ring;
    }

Because is_kernel_rodata() only checks against the core kernel's __start_rodata and __end_rodata boundaries, any memory provider ops struct allocated in a loadable module's .rodata section will fail this check. This renders the new feature unusable for modular drivers.

Quoted 4 lines
net/core/page_pool.c +1 -0
+	} else if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) {
 		netdev_assert_locked(pool->slow.netdev);
 		rxq = __netif_get_rx_queue(pool->slow.netdev,
 					   pool->slow.queue_idx);

[ ... ]

net/core/page_pool.c void page_pool_clear_pp_info(netmem_ref netmem) · line 745

Quoted 12 lines
net/core/page_pool.c +8 -0
@@ -725,6 +735,16 @@ void page_pool_clear_pp_info(netmem_ref netmem)
 	netmem_set_pp(netmem, NULL);
 }
 
+void page_pool_provider_set_netmem(struct page_pool *pool, netmem_ref netmem,
+				   dma_addr_t addr)
+{
+	netmem_to_nmdesc(netmem)->pp_magic = 0;
+	netmem_to_nmdesc(netmem)->pp = NULL;
+	atomic_long_set(&netmem_to_nmdesc(netmem)->pp_ref_count, 0);
+	page_pool_set_pp_info(pool, netmem);
+	page_pool_set_dma_addr_netmem(netmem, addr);

[Severity: High]
Does this silently ignore DMA address truncation?

page_pool_set_dma_addr_netmem() returns a boolean indicating whether the DMA address exceeded the capacity of the storage representation (such as on 32-bit architectures with 64-bit DMA support).

Because page_pool_provider_set_netmem() returns void and discards this boolean, memory providers will be unaware if their DMA address was truncated. This could lead to silent data corruption or IOMMU faults when the hardware uses the truncated address.

Quoted 5 lines
net/core/page_pool.c +2 -0
+}
+
 static __always_inline void __page_pool_release_netmem_dma(struct page_pool *pool,
 							   netmem_ref netmem)
 {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#7 lore raw

RFC PATCH 3/13 net: core: add XDP_MODE_HW offload hook for knod

In reply to Taehee Yoo

Route XDP program install/uninstall to the knod accelerator when a device is attached, so BPF programs run on the accelerator instead of the host.

(cherry picked from commit 9689730acf7d6e4d34cd677e4615be10856c709c)
---
 net/core/dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/dev.c +1 -1
index 7c21bc0a1e34..5ba1c969029b 100644
@@ -10334,7 +10334,7 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
 		return -EBUSY;
 	}
 
-	if (dev_get_min_mp_channel_count(dev)) {
+	if (dev_get_min_mp_channel_count(dev) && mode != XDP_MODE_HW) {
 		NL_SET_ERR_MSG(extack, "unable to install XDP to device using memory provider");
 		return -EBUSY;
 	}
Signature · 1 line
2.43.0
#8 lore raw

Re: [RFC PATCH net-next 03/13] net: core: add XDP_MODE_HW offload hook for knod

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Medium] The patch introduces an order-dependent state violation where attaching a memory provider and a hardware-offloaded XDP program succeeds or fails depending purely on the order of operations.

Pre-existing issues:
- [High] Blocking XDP uninstalls when a memory provider is active leads to a resource leak of the BPF program and a permanently active 'zombie' XDP program on slave devices. --

--- Patch [3]: [RFC PATCH net-next 03/13] net: core: add XDP_MODE_HW offload hook for knod --- commit 2719ee9dd236296c86e09fa87d5113250098ff78
Author: Taehee Yoo <[email protected]>

net: core: add XDP_MODE_HW offload hook for knod

Route XDP program install/uninstall to the knod accelerator when a device is attached, so BPF programs run on the accelerator instead of the host.

net/core/dev.c static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode, · line 10337

Quoted 10 lines
net/core/dev.c +1 -1
index 7c21bc0a1e349..5ba1c969029bc 100644
@@ -10334,7 +10334,7 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
 		return -EBUSY;
 	}
 
-	if (dev_get_min_mp_channel_count(dev)) {
+	if (dev_get_min_mp_channel_count(dev) && mode != XDP_MODE_HW) {

[Severity: High]
This is a pre-existing issue, but blocking XDP uninstalls when a memory provider is active appears to lead to a resource leak and an inconsistent state on slave devices.

If we attach an XDP program to a bonding device, it propagates to slave devices without updating their individual dev->xdp_state. If we then bind a memory provider to the slave device (which succeeds since its local prog count is 0), what happens when we try to destroy the bond device?

netif_xdp_propagate() will fail because of the memory provider:

net/core/dev.c:netif_xdp_propagate() {
    ...
	if (dev_get_min_mp_channel_count(dev)) {
		NL_SET_ERR_MSG(bpf->extack, "unable to propagate XDP...");
		return -EBUSY;
	}
    ...
}

In dev_xdp_uninstall(), this failure is ignored:

net/core/dev.c:dev_xdp_uninstall() {
    ...
		WARN_ON(dev_xdp_install(dev, mode, bpf_op, NULL, 0, NULL));

		/* auto-detach link from net device */
		link = dev_xdp_link(dev, mode);
		if (link)
			link->dev = NULL;
		else
			bpf_prog_put(prog);
    ...
}

Doesn't this result in the bond dropping its reference and clearing its state, while the slave driver never unloads the program because ndo_bpf is not called? Will this leave a zombie XDP program permanently executing on the slave device and leak the BPF program memory?

[Severity: Medium]
Does modifying this check create an order-dependent state regression?

If a memory provider is attached first, this modified check allows an XDP_MODE_HW program to be installed successfully.

However, if an XDP_MODE_HW program is installed first, a subsequent attempt to bind a memory provider fails in __netif_mp_open_rxq() because it unconditionally checks the program count:

net/core/netdev_rx_queue.c:__netif_mp_open_rxq() {
    ...
	if (dev_xdp_prog_count(dev)) {
		NL_SET_ERR_MSG(extack, "unable to custom memory provider...");
		return -EEXIST;
	}
    ...
}

Should __netif_mp_open_rxq() also check if the existing programs are hardware offloaded to ensure the API behaves consistently regardless of the operation order?

Quoted 3 lines
net/core/dev.c +0 -0
 		NL_SET_ERR_MSG(extack, "unable to install XDP to device using memory provider");
 		return -EBUSY;
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#9 lore raw

RFC PATCH 4/13 net: knod: add offload device core and control plane

In reply to Taehee Yoo

Add the KNOD core framework and its genetlink control plane.

The core maintains three object types - a NIC-side netdev, an accelerator, and a dynamically created offload device that binds them - and drives the RX data path over per-queue SPSC rings backed by a page_pool of accelerator memory.

The genetlink family ("knod") exposes attach/detach, device/nic/accel inventory, and per-accelerator feature selection as a stable uAPI.

(cherry picked from commit f17f82a39be852aa8c520aea7177ce0086f7e811)
---
 net/Kconfig            |    2 +
 net/Makefile           |    1 +
 net/knod/Kconfig       |    5 +
 net/knod/Makefile      |    8 +
 net/knod/knod-nl-gen.c |  121 ++++
 net/knod/knod-nl-gen.h |   31 +
 net/knod/knod.h        |   26 +
 net/knod/knod_core.c   | 1231 ++++++++++++++++++++++++++++++++++++++++
 net/knod/knod_nl.c     |  406 +++++++++++++
 9 files changed, 1831 insertions(+)
 create mode 100644 net/knod/Kconfig
 create mode 100644 net/knod/Makefile
 create mode 100644 net/knod/knod-nl-gen.c
 create mode 100644 net/knod/knod-nl-gen.h
 create mode 100644 net/knod/knod.h
 create mode 100644 net/knod/knod_core.c
 create mode 100644 net/knod/knod_nl.c
net/Kconfig +2 -0
index e38477393551..e7a2200e92cc 100644
@@ -72,6 +72,8 @@ config NET_DEVMEM
 	depends on DMA_SHARED_BUFFER
 	depends on PAGE_POOL
 
+source "net/knod/Kconfig"
+
 config NET_SHAPER
 	bool
 
net/Makefile +1 -0
index 5b2dd7f07a85..ea4ef5c5906d 100644
@@ -19,6 +19,7 @@ obj-$(CONFIG_TLS)		+= tls/
 obj-$(CONFIG_XFRM)		+= xfrm/
 obj-$(CONFIG_UNIX)		+= unix/
 obj-$(CONFIG_INET_PSP)		+= psp/
+obj-$(CONFIG_KNOD)	+= knod/
 obj-y				+= ipv6/
 obj-$(CONFIG_PACKET)		+= packet/
 obj-$(CONFIG_NET_KEY)		+= key/
net/knod/Kconfig new file +5 -0
new file mode 100644
index 000000000000..d5a6d76d33e3
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+config KNOD
+	def_bool y
+	depends on NET_DEVMEM
net/knod/Makefile new file +8 -0
new file mode 100644
index 000000000000..73abe5c99f8c
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Makefile for the KNOD (KFD network offload) control plane.
+#
+
+obj-$(CONFIG_KNOD) += knod.o
+
+knod-y := knod_core.o knod_nl.o knod-nl-gen.o
net/knod/knod-nl-gen.c new file +121 -0

121 lines not on this page · raw message

net/knod/knod-nl-gen.h new file +31 -0
new file mode 100644
index 000000000000..a12b6a174bed
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */
+/* Do not edit directly, auto-generated from: */
+/*	Documentation/netlink/specs/knod.yaml */
+/* YNL-GEN kernel header */
+/* To regenerate run: tools/net/ynl/ynl-regen.sh */
+
+#ifndef _LINUX_KNOD_GEN_H
+#define _LINUX_KNOD_GEN_H
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include <uapi/linux/knod.h>
+
+int knod_nl_accel_get_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_accel_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+int knod_nl_accel_set_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_nic_get_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_nic_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+int knod_nl_attach_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_detach_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_dev_get_doit(struct sk_buff *skb, struct genl_info *info);
+int knod_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+
+enum {
+	KNOD_NLGRP_MGMT,
+};
+
+extern struct genl_family knod_nl_family;
+
+#endif /* _LINUX_KNOD_GEN_H */
net/knod/knod.h new file +26 -0
new file mode 100644
index 000000000000..eaff0e82ca23
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright (c) 2021 Taehee Yoo <[email protected]>
+ * Copyright (c) 2021 Hoyeon Lee <[email protected]>
+ */
+
+#ifndef _NET_KNOD_KNOD_H
+#define _NET_KNOD_KNOD_H
+
+#include <net/knod.h>
+
+#include "knod-nl-gen.h"
+
+struct net_device;
+
+/* knod_core.c */
+struct knod_netdev *knod_netdev_lookup(struct net_device *dev);
+struct knod_dev *knod_dev_lookup(struct net_device *dev);
+struct knod_accel *knod_accel_lookup(int id);
+int knod_dev_attach(struct knod_netdev *knetdev,
+			   struct knod_accel *accel);
+int knod_dev_detach(struct knod_dev *knodev);
+
+/* knod_nl.c */
+void knod_nl_notify_dev(struct knod_dev *knodev, u32 cmd);
+
+#endif /* _NET_KNOD_KNOD_H */
net/knod/knod_core.c new file +1231 -0

1,231 lines not on this page · raw message

net/knod/knod_nl.c new file +406 -0

406 lines not on this page · raw message

Signature · 1 line
2.43.0
#10 lore raw

Re: [RFC PATCH net-next 04/13] net: knod: add offload device core and control plane

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider: - [High] Lockless traversal of knod_dev_list in IPsec datapath races with list mutations in…

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] Lockless traversal of knod_dev_list in IPsec datapath races with list mutations in knod_dev_detach, leading to kernel panic or Use-After-Free.
- [High] Lack of hot-unplug detachment and missing net_device refcounts lead to Use-After-Free if hardware is removed.
- [High] knod_pass_detach uses a timeout that forces premature freeing of page pool structures, causing Use-After-Free when delayed SKBs are eventually released.
- [High] Uninitialized variable 'i' is used in the free_spsc error cleanup path if memory allocation fails.
- [High] Hardware SDMA target pages are prematurely recycled if the hardware fence times out, leading to silent data corruption.
- [High] PP_FLAG_DMA_MAP is improperly set for a custom memory provider that supplies pre-mapped device memory.
- [High] Hardcoded sdma_idx = 0 causes hardware fence polling on the wrong queue.
- [Medium] Netlink operations do not reply with a full message for object creation, and notifications ignore NLM_F_ECHO requirements.
- [Medium] Direct modification of netdev->features bypasses networking core feature synchronization and notifications.
- [Medium] Double-attachment in knod_ipsec_attach corrupts the saved xfrmdev_ops pointer.
--

--- Patch [4]: [RFC PATCH net-next 04/13] net: knod: add offload device core and control plane --- commit e070e2966937b9789951030b969d12e24910bb8f
Author: Taehee Yoo <[email protected]>
Subject: net: knod: add offload device core and control plane

Add the KNOD core framework and its genetlink control plane. The core maintains three object types and drives the RX data path over per-queue SPSC rings. The genetlink family exposes attach/detach, inventory, and feature selection.

Quoted 4 lines
net/knod/knod_core.c new file +0 -0
index af50a69bca651..[snip]

[ ... ]

Quoted 7 lines
		desc = ptr;
		desc->netmem = dst;
		desc->src = src;
		desc->off = off;
		desc->len = len;
		desc->fence_val = fv;
		desc->sdma_idx = 0;

[Severity: High]
Does this hardcoded index cause hardware fence polling on the wrong queue?

By setting sdma_idx to 0 instead of the dynamically assigned napi_index, subsequent calls to d2h_fence() in knod_d2h_drain() and knod_pass_flush() will always poll queue 0. This could lead to premature delivery of incomplete DMA buffers or hang indefinitely on multi-queue configurations.

[ ... ]

Quoted 8 lines

static struct knod_dev *knod_ipsec_find_xdev(struct net_device *dev)

{
	struct knod_dev *knodev;

	list_for_each_entry(knodev, &knod_dev_list, list) {
		if (knodev->netdev == dev)
			return knodev;
	}

[Severity: High]
Can this lockless iteration of knod_dev_list cause a use-after-free or kernel panic?

This function is called from the IPsec transmit path via knod_ipsec_xdo_offload_ok(). Concurrently, knod_dev_detach() for other devices modifies knod_dev_list using list_del() and kfree() without RCU protection or locking, meaning this traversal could read poisoned list pointers or freed memory.

[ ... ]

Quoted 4 lines
	knodev->ipsec_orig_xfrmdev_ops = knodev->netdev->xfrmdev_ops;
	knodev->ipsec_added_hw_esp =
		!(knodev->netdev->features & NETIF_F_HW_ESP);
	knodev->netdev->xfrmdev_ops = &knod_ipsec_xfrmdev_ops;

[Severity: Medium]
Is there protection against double-attachment overwriting the original operations pointer?

If knod_ipsec_attach() is called twice, the previously installed proxy operations will be saved as the original xfrmdev_ops. During detach, this proxy will be restored instead of the true hardware operations, breaking the offload and potentially creating recursion loops.

Quoted 2 lines
	knodev->netdev->features |= NETIF_F_HW_ESP;
	knodev->netdev->hw_enc_features |= NETIF_F_HW_ESP;

[Severity: Medium]
Does directly modifying netdev->features bypass core network synchronization?

Dynamically modifying features should typically be done by updating hw_features or wanted_features and invoking netdev_update_features(). Modifying them directly misses the NETDEV_FEAT_CHANGE notification, leaving userspace tools like ethtool out of sync with the hardware state.

[ ... ]

Quoted 7 lines

static bool hostmem_pp_release_netmem(struct page_pool *pool, netmem_ref netmem)

{
	struct page_pool_hostmem *hm = pool->mp_priv;
	unsigned long addr = page_pool_get_dma_addr_netmem(netmem);

	page_pool_clear_pp_info(netmem);
	gen_pool_free(hm->genpool, addr, PAGE_SIZE);

[Severity: High]
Can this lead to a use-after-free when delayed SKBs are released?

In knod_pass_detach(), there is a timeout that forcefully frees the wpriv structure (which contains pass_hm) if the page pool hasn't drained within 5 seconds. When the network stack eventually frees those delayed SKBs, this callback will dereference the freed pool->mp_priv (hm) when calling gen_pool_free().

[ ... ]

Quoted 9 lines
	struct page_pool_params pp = {
		.order		= 0,
		.pool_size	= KNOD_PASS_SLOTS,
		.nid		= NUMA_NO_NODE,
		.dev		= knodev->netdev->dev.parent,
		.dma_dir	= DMA_FROM_DEVICE,
		.max_len	= PAGE_SIZE,
		.flags		= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV |
				  PP_FLAG_CUSTOM_MEMORY_PROVIDER,

[Severity: High]
Is PP_FLAG_DMA_MAP appropriate for a custom memory provider returning pre-mapped device memory?

Setting PP_FLAG_DMA_MAP instructs the page pool core to automatically unmap the page via dma_unmap_page_attrs() on release. Since hostmem_pp_init uses GTT device addresses instead of standard system RAM DMA mappings, this will result in invalid DMA API unmap operations.

[ ... ]

Quoted 8 lines
			if (desc->src) {
				struct page_pool *src_pp =
					netmem_get_pp(desc->src);

				page_pool_recycle_direct_netmem(src_pp,
								desc->src);
			}
			page_pool_put_full_netmem(pool, desc->netmem, false);

[Severity: High]
Could recycling these DMA target pages after a timeout cause silent data corruption?

If the polling loop in knod_pass_flush() times out, the pages are recycled back to the active pools immediately. Since the hardware operation was not definitively aborted, the accelerator might complete its delayed DMA write later, silently overwriting whatever new packet data has been placed in that page.

[ ... ]

Quoted 1 line
	knodev->netdev = knetdev->dev;

[Severity: High]
Should dev_hold() be called here to take a reference on the net_device?

If the device is hot-removed, the hardware driver invokes unregister_netdev() and frees the net_device. knod_netdev_unregister() only removes the object from the list without detaching it. This leaves a dangling pointer in knodev->netdev, which can cause a use-after-free when accessed by Netlink queries like knod_nl_dev_get_dumpit().

[ ... ]

Quoted 13 lines
		if (knodev->accel_ops->alloc_mem) {
			size_t total = pool_size * nqueues;
			u64 base_gaddr;
			void *base_pool;
			void *pool_priv;

			base_pool = knodev->accel_ops->alloc_mem(knodev, total,
					&base_gaddr, NULL, &pool_priv);
			if (!base_pool) {
				err = -ENOMEM;
				pr_err("%s: alloc_mem failed\n", __func__);
				goto free_spsc;
			}

[ ... ]

Quoted 3 lines

free_spsc:

	for (i--; i >= 0; i--)
		spsc_destroy(&knodev->wpriv[i].spsc_bds);

[Severity: High]
Will this loop use an uninitialized variable if alloc_mem() fails?

If alloc_mem() fails, the code jumps to free_spsc before 'i' is initialized. The loop then operates on an uninitialized stack value for 'i', leading to an out-of-bounds array access on knodev->wpriv[i] and passing an invalid pointer to spsc_destroy().

Quoted 4 lines
net/knod/knod_nl.c new file +0 -0
index 77799202abcf4..[snip]

[ ... ]

Quoted 2 lines
	genlmsg_multicast_netns(&knod_nl_family, net, ntf, 0,
				KNOD_NLGRP_MGMT, GFP_KERNEL);

[Severity: Medium]
Does this ignore the NLM_F_ECHO flag?

Notifications emitted in response to requests should pass the request information to genl_notify() to honor the NLM_F_ECHO flag. Using genlmsg_multicast_netns() drops the request context.

[ ... ]

Quoted 8 lines
	err = knod_dev_attach(knetdev, accel);
	if (!err)
		knod_nl_notify_dev(knetdev->knodev, KNOD_CMD_DEV_ADD_NTF);

unlock:

	mutex_unlock(&knod_lock);
	rtnl_unlock();
	return err;

[Severity: Medium]
Should this operation reply with a full message?

According to Netlink uAPI design rules, NEW/ADD operations must reply with a full message carrying identifying information about the newly created object, rather than just returning zero and ACKing the request.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author
#11 lore raw

RFC PATCH 5/13 bpf: offload: allow PERCPU_ARRAY maps for offloaded programs

In reply to Taehee Yoo

The knod BPF offload keeps hot per-CPU statistics in a PERCPU_ARRAY map that is mirrored into accelerator memory and aggregated back on the host. Permit this map type on the offload path.

(cherry picked from commit c447c9012940d48c1ec20a8ebac8d779e7d5a6fa)
---
 kernel/bpf/offload.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/bpf/offload.c +2 -1
index 0d6f5569588c..36e747a12cbe 100644
@@ -510,7 +510,8 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)
 	if (!capable(CAP_SYS_ADMIN))
 		return ERR_PTR(-EPERM);
 	if (attr->map_type != BPF_MAP_TYPE_ARRAY &&
-	    attr->map_type != BPF_MAP_TYPE_HASH)
+	    attr->map_type != BPF_MAP_TYPE_HASH &&
+	    attr->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
 		return ERR_PTR(-EINVAL);
 
 	offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE);
Signature · 1 line
2.43.0
#12 lore raw

Re: [RFC PATCH net-next 05/13] bpf: offload: allow PERCPU_ARRAY maps for offloaded programs

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Permitting BPF_MAP_TYPE_PERCPU_ARRAY on the offload path introduces a kernel heap buffer overflow (OOB write) and an uninitialized memory leak during map lookups, because the offload driver API lacks the flags parameter required to determine the value buffer size.
- [Medium] User-triggerable WARN_ON in netdevsim driver when attempting to allocate an offloaded PERCPU_ARRAY map.

Pre-existing issues:
- [High] NULL pointer dereference in prepare_seq_file for offloaded maps due to incomplete bpf_map_offload_ops. --

--- Patch [5]: [RFC PATCH net-next 05/13] bpf: offload: allow PERCPU_ARRAY maps for offloaded programs --- commit 48af783b3fca05c679c6224bf05a40782d3679ed
Author: Taehee Yoo <[email protected]>

bpf: offload: allow PERCPU_ARRAY maps for offloaded programs

The knod BPF offload keeps hot per-CPU statistics in a PERCPU_ARRAY map that is mirrored into accelerator memory and aggregated back on the host. Permit this map type on the offload path.

kernel/bpf/offload.c struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) · line 514

Quoted 11 lines
kernel/bpf/offload.c +2 -1
index 0d6f5569588c3..36e747a12cbe7 100644
@@ -510,7 +510,8 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)
 	if (!capable(CAP_SYS_ADMIN))
 		return ERR_PTR(-EPERM);
 	if (attr->map_type != BPF_MAP_TYPE_ARRAY &&
-	    attr->map_type != BPF_MAP_TYPE_HASH)
+	    attr->map_type != BPF_MAP_TYPE_HASH &&
+	    attr->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)

[Severity: Medium]
Does this change trigger a WARN_ON in the netdevsim driver?

Looking at nsim_bpf_map_alloc() in drivers/net/netdevsim/bpf.c, there is an explicit check that will panic if panic_on_warn is set:

drivers/net/netdevsim/bpf.c:nsim_bpf_map_alloc() {
    if (WARN_ON(offmap->map.map_type != BPF_MAP_TYPE_ARRAY &&
                offmap->map.map_type != BPF_MAP_TYPE_HASH))
        return -EINVAL;
}

Since BPF_MAP_TYPE_PERCPU_ARRAY is now allowed, could this assertion be hit when creating an offloaded PERCPU_ARRAY map on a netdevsim device?

[Severity: High]
Can this lead to a buffer overflow or uninitialized memory disclosure during map lookups?

When user-space calls bpf(BPF_MAP_LOOKUP_ELEM) on a PERCPU_ARRAY, map_lookup_elem() dynamically allocates the value buffer size based on whether BPF_F_CPU is passed:

kernel/bpf/syscall.c:map_lookup_elem() {
    ...
    value_size = bpf_map_value_size(map, attr->flags);
    err = -ENOMEM;
    value = kvmalloc(value_size, GFP_USER | __GFP_NOWARN);
    ...
}

However, bpf_map_offload_lookup_elem() drops the flags argument entirely when calling the driver's hook:

kernel/bpf/offload.c:bpf_map_offload_lookup_elem() {
    int bpf_map_offload_lookup_elem(struct bpf_map *map, void *key,
                                    void *value)
    {
        struct bpf_offloaded_map *offmap = map_to_offmap(map);
    ...
}

If a user specifies BPF_F_CPU, the allocated buffer is small. If the driver returns full per-CPU data, could it overwrite the buffer? Alternatively, if no flag is passed but the driver only returns one element, could the rest of the uninitialized buffer be copied to user-space?

[Severity: High]
This isn't a bug introduced by this patch, but does allowing PERCPU_ARRAY (as well as existing ARRAY and HASH maps) expose a NULL pointer dereference in the BPF iterator code?

When calling open() on a BPF iterator file descriptor for an offloaded map, prepare_seq_file() executes:

kernel/bpf/bpf_iter.c:prepare_seq_file() {
    const struct bpf_iter_seq_info *seq_info = __get_seq_info(link);
    ...
    total_priv_dsize = offsetof(struct bpf_iter_priv_data, target_private) +
                       seq_info->seq_priv_size;
    ...
}

Because bpf_map_offload_ops does not implement .iter_seq_info, __get_seq_info() returns NULL:

kernel/bpf/bpf_iter.c:__get_seq_info() {
    if (link->aux.map) {
        seq_info = link->aux.map->ops->iter_seq_info;
        if (seq_info)
            return seq_info;
    }
    return link->tinfo->reg_info->seq_info;
}

Will this result in a direct NULL pointer dereference when accessing seq_info->seq_priv_size since both sources of seq_info are NULL for offloaded maps?

Quoted 3 lines
kernel/bpf/offload.c +0 -0
 		return ERR_PTR(-EINVAL);
 
 	offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author
#13 lore raw

RFC PATCH 6/13 drm/amdkfd: prepare kfd core for the knod provider

In reply to Taehee Yoo

Expose the kfd internals the knod provider needs to build and run GPU queues on behalf of the kernel: kernel-side event waiting, GPUVM allocation for VRAM/GTT, doorbell access, and the HSA queue/packet layout (kfd_hsa.h). No functional change for existing user-mode queue users; the provider itself is added in a following patch.

(cherry picked from commit 7d2c7cbfbb2a4a80052b5793841f6a229271973d)
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h    |  13 +-
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  65 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c       |   3 +
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c      | 117 ++---
 drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c     |  20 +-
 drivers/gpu/drm/amd/amdkfd/kfd_events.c       | 112 ++++-
 drivers/gpu/drm/amd/amdkfd/kfd_events.h       |   4 +
 drivers/gpu/drm/amd/amdkfd/kfd_hsa.h          | 451 ++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_module.c       |   1 +
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h         |  34 ++
 drivers/gpu/drm/amd/amdkfd/kfd_process.c      |  26 +-
 11 files changed, 778 insertions(+), 68 deletions(-)
 create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_hsa.h
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +12 -1
index e443a7277299..cf906b32eacb 100644
@@ -328,6 +328,8 @@ int amdgpu_amdkfd_gpuvm_sync_memory(
 		struct amdgpu_device *adev, struct kgd_mem *mem, bool intr);
 int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem,
 					     void **kptr, uint64_t *size);
+int amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel(struct kgd_mem *mem,
+					      void **kptr, uint64_t *size);
 void amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(struct kgd_mem *mem);
 
 int amdgpu_amdkfd_map_gtt_bo_to_gart(struct amdgpu_bo *bo, struct amdgpu_bo **bo_gart);
@@ -446,7 +448,6 @@ bool kgd2kfd_vmfault_fast_path(struct amdgpu_device *adev, struct amdgpu_iv_entr
 			       bool retry_fault);
 void kgd2kfd_lock_kfd(void);
 void kgd2kfd_teardown_processes(struct amdgpu_device *adev);
-
 #else
 static inline int kgd2kfd_init(void)
 {
@@ -576,5 +577,15 @@ static inline void kgd2kfd_teardown_processes(struct amdgpu_device *adev)
 {
 }
 
+#endif
+
+#if defined(CONFIG_HSA_AMD_KNOD)
+int knod_init(struct amdgpu_device *adev);
+void knod_fini(struct amdgpu_device *adev);
+void knod_exit(void);
+#else
+static inline int knod_init(struct amdgpu_device *adev) { return 0; }
+static inline void knod_fini(struct amdgpu_device *adev) { }
+static inline void knod_exit(void) { }
 #endif
 #endif /* AMDGPU_AMDKFD_H_INCLUDED */
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +65 -0

71 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +3 -0
index 4c0c77eafbd1..aa974929d22a 100644
@@ -2480,6 +2480,8 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 		drm_client_setup(adev_to_drm(adev), format);
 	}
 
+	knod_init(adev);
+
 	ret = amdgpu_debugfs_init(adev);
 	if (ret)
 		DRM_ERROR("Creating debugfs files failed (%d).\n", ret);
@@ -2540,6 +2542,7 @@ amdgpu_pci_remove(struct pci_dev *pdev)
 	struct drm_device *dev = pci_get_drvdata(pdev);
 	struct amdgpu_device *adev = drm_to_adev(dev);
 
+	knod_fini(adev);
 	amdgpu_ras_eeprom_check_and_recover(adev);
 	amdgpu_xcp_dev_unplug(adev);
 	amdgpu_gmc_prepare_nps_mode_change(adev);
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +60 -57

246 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +19 -1
index fdcf7f2d1b5b..78b7e956f590 100644
@@ -145,10 +145,28 @@ int kfd_doorbell_mmap(struct kfd_node *dev, struct kfd_process *process,
 				vma->vm_page_prot);
 }
 
+void __iomem *kfd_kernel_doorbell_mmap(struct kfd_node *dev,
+				       struct kfd_process *process)
+{
+	phys_addr_t address;
+	struct kfd_process_device *pdd;
+
+	pdd = kfd_get_process_device_data(dev, process);
+	if (!pdd)
+		return NULL;
+
+	/* Calculate physical address of doorbell */
+	address = kfd_get_process_doorbells(pdd);
+	if (!address)
+		return NULL;
+
+	return ioremap(address, kfd_doorbell_process_slice(dev->kfd));
+}
+
 
 /* get kernel iomem pointer for a doorbell */
 void __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd,
-					unsigned int *doorbell_off)
+				      unsigned int *doorbell_off)
 {
 	u32 inx;
 
drivers/gpu/drm/amd/amdkfd/kfd_events.c +111 -1

124 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/kfd_events.h +4 -0
index 1dc21c13833b..aa6ead122084 100644
@@ -87,5 +87,9 @@ struct kfd_event {
 extern void kfd_signal_event_interrupt(u32 pasid, uint32_t partial_id,
 				       uint32_t valid_id_bits,
 				       bool signal_mailbox_updated);
+int kfd_wait_on_events_kernel(struct kfd_process *p,
+			      uint32_t num_events, void __user *data,
+			      bool all, uint32_t *user_timeout_ms,
+			      uint32_t *wait_result);
 
 #endif
drivers/gpu/drm/amd/amdkfd/kfd_hsa.h new file +451 -0

451 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/kfd_module.c +1 -0
index 33aa23450b3f..2a2405db5b9c 100644
@@ -77,6 +77,7 @@ static int kfd_init(void)
 
 static void kfd_exit(void)
 {
+	knod_exit();
 	kfd_cleanup_processes();
 	kfd_process_destroy_wq();
 	kfd_debugfs_fini();
drivers/gpu/drm/amd/amdkfd/kfd_priv.h +34 -0
index acd0e41e744c..4f5d4b0bfa89 100644
@@ -1069,6 +1069,7 @@ bool kfd_dev_is_large_bar(struct kfd_node *dev);
 struct kfd_process *create_process(const struct task_struct *thread, bool primary);
 int kfd_process_create_wq(void);
 void kfd_process_destroy_wq(void);
+void kfd_process_flush_wq(void);
 void kfd_cleanup_processes(void);
 struct kfd_process *kfd_create_process(struct task_struct *thread);
 int kfd_create_process_sysfs(struct kfd_process *process);
@@ -1536,6 +1537,9 @@ void kfd_signal_hw_exception_event(u32 pasid);
 int kfd_set_event(struct kfd_process *p, uint32_t event_id);
 int kfd_reset_event(struct kfd_process *p, uint32_t event_id);
 int kfd_kmap_event_page(struct kfd_process *p, uint64_t event_page_offset);
+int kfd_event_page_set(struct kfd_process *p, void *kernel_address,
+		       uint64_t size, uint64_t user_handle);
+
 
 int kfd_event_create(struct file *devkfd, struct kfd_process *p,
 		     uint32_t event_type, bool auto_reset, uint32_t node_id,
@@ -1641,3 +1645,33 @@ static inline void kfd_debugfs_remove_process(struct kfd_process *p) {}
 #endif
 
 #endif
+
+int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd,
+			    uint64_t gpu_va, uint32_t size,
+			    uint32_t flags, struct kgd_mem **mem, void **kptr);
+void kfd_process_free_gpuvm(struct kgd_mem *mem,
+			    struct kfd_process_device *pdd, void **kptr);
+int kfd_create_queue(struct kfd_process *p, struct queue_properties *q_properties,
+		     u32 gpu_id, u32 *queue_id_out, u64 *doorbell_offset_out);
+int kfd_ioctl_set_event(struct file *filp, struct kfd_process *p,
+			void *data);
+int kfd_ioctl_reset_event(struct file *filp, struct kfd_process *p,
+			  void *data);
+int kfd_ioctl_wait_events(struct file *filp, struct kfd_process *p, void *data);
+
+int kfd_ioctl_set_trap_handler(struct file *filep,
+			       struct kfd_process *p, void *data);
+int kfd_ioctl_set_scratch_backing_va(struct file *filep,
+				     struct kfd_process *p, void *data);
+
+
+int kfd_ioctl_acquire_vm(struct file *filep, struct kfd_process *p, void *data);
+int kfd_ioctl_map_memory_to_gpu(struct file *filep, struct kfd_process *p,
+				void *data);
+int kfd_ioctl_alloc_memory_of_gpu(struct file *filep, struct kfd_process *p,
+				  void *data);
+int kfd_ioctl_export_dmabuf(struct file *filep,
+			    struct kfd_process *p, void *data);
+
+void __iomem *kfd_kernel_doorbell_mmap(struct kfd_node *dev,
+				       struct kfd_process *process);
drivers/gpu/drm/amd/amdkfd/kfd_process.c +18 -8
index ca71fa726e32..3c9268241800 100644
@@ -715,8 +715,14 @@ void kfd_process_destroy_wq(void)
 	}
 }
 
-static void kfd_process_free_gpuvm(struct kgd_mem *mem,
-			struct kfd_process_device *pdd, void **kptr)
+void kfd_process_flush_wq(void)
+{
+	if (kfd_process_wq)
+		flush_workqueue(kfd_process_wq);
+}
+
+void kfd_process_free_gpuvm(struct kgd_mem *mem,
+			    struct kfd_process_device *pdd, void **kptr)
 {
 	struct kfd_node *dev = pdd->dev;
 
@@ -736,9 +742,9 @@ static void kfd_process_free_gpuvm(struct kgd_mem *mem,
  *	to avoid concurrency. Because of that exclusiveness, we do
  *	not need to take p->mutex.
  */
-static int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd,
-				   uint64_t gpu_va, uint32_t size,
-				   uint32_t flags, struct kgd_mem **mem, void **kptr)
+int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd,
+			    uint64_t gpu_va, uint32_t size,
+			    uint32_t flags, struct kgd_mem **mem, void **kptr)
 {
 	struct kfd_node *kdev = pdd->dev;
 	int err;
@@ -761,10 +767,14 @@ static int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd,
 	}
 
 	if (kptr) {
-		err = amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(
-				(struct kgd_mem *)*mem, kptr, NULL);
+		if (flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM)
+			err = amdgpu_amdkfd_gpuvm_map_vram_bo_to_kernel(
+					(struct kgd_mem *)*mem, kptr, NULL);
+		else
+			err = amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(
+					(struct kgd_mem *)*mem, kptr, NULL);
 		if (err) {
-			pr_debug("Map GTT BO to kernel failed\n");
+			pr_debug("Map BO to kernel failed\n");
 			goto sync_memory_failed;
 		}
 	}
Signature · 1 line
2.43.0
#14 lore raw

Re: [RFC PATCH net-next 06/13] drm/amdkfd: prepare kfd core for the knod provider

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [High] kfd_wait_on_events_kernel mixes user and kernel memory semantics, causing persistent failures and…

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [High] kfd_wait_on_events_kernel mixes user and kernel memory semantics, causing persistent failures and security risks.
- [High] kfd_ioctl_alloc_memory_of_gpu crashes if called from a kernel thread due to current->mm dereference.
- [High] kfd_ioctl_map_memory_to_gpu fails unconditionally for kernel callers due to memdup_array_user.
- [Medium] The return value of knod_init() is silently ignored during probe.
- [Medium] kfd_ioctl_export_dmabuf leaks a file descriptor when called from a kernel thread.
--

--- Patch [6]: [RFC PATCH net-next 06/13] drm/amdkfd: prepare kfd core for the knod provider --- commit c9a02222fa92c5ac2eb02dcfd71c15c6d2016bd7
Author: Taehee Yoo <[email protected]>
Subject: drm/amdkfd: prepare kfd core for the knod provider

Expose the kfd internals the knod provider needs to build and run GPU queues on behalf of the kernel: kernel-side event waiting, GPUVM allocation for VRAM/GTT, doorbell access, and the HSA queue/packet layout (kfd_hsa.h). No functional change for existing user-mode queue users; the provider itself is added in a following patch.

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c static int amdgpu_pci_probe(struct pci_dev *pdev, · line 2484

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +2 -0
index 4c0c77eafbd1c..aa974929d22a1 100644
@@ -2480,6 +2480,8 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 		drm_client_setup(adev_to_drm(adev), format);
 	}
 
+	knod_init(adev);
+

[Severity: Medium]
Is it intentional to discard the return value of knod_init()?

If this initialization fails or cannot allocate its resources, the error is not propagated. This might leave the system in an inconsistent state and lead to further issues when knod_fini() operates on uninitialized structures.

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
 	ret = amdgpu_debugfs_init(adev);
 	if (ret)
 		DRM_ERROR("Creating debugfs files failed (%d).\n", ret);

[ ... ]

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
index c7edebd2fd8a4..7879a6ffaa762 100644

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int kfd_ioctl_get_available_memory(struct file *filep, · line 1088

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -2
@@ -1078,8 +1081,8 @@ static int kfd_ioctl_get_available_memory(struct file *filep,
 	return 0;
 }
 
-static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
-					struct kfd_process *p, void *data)
+int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
+				  struct kfd_process *p, void *data)
 {
 	struct kfd_ioctl_alloc_memory_of_gpu_args *args = data;
 	struct kfd_process_device *pdd;

[Severity: High]
Could this cause a NULL pointer dereference when called from a kernel thread?

This function is now exported for kernel callers, but internally it has blocks that dereference current->mm:

svm_range_list_lock_and_flush_work(&p->svms, current->mm); mutex_lock(&p->svms.lock); mmap_write_unlock(current->mm);

Since kernel threads execute without a user address space, current->mm is NULL, which would cause a panic when accessed.

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int kfd_ioctl_free_memory_of_gpu(struct file *filep, · line 1289

Quoted 11 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -2
@@ -1279,8 +1282,8 @@ static int kfd_ioctl_free_memory_of_gpu(struct file *filep,
 	return ret;
 }
 
-static int kfd_ioctl_map_memory_to_gpu(struct file *filep,
-					struct kfd_process *p, void *data)
+int kfd_ioctl_map_memory_to_gpu(struct file *filep,
+				struct kfd_process *p, void *data)
 {
 	struct kfd_ioctl_map_memory_to_gpu_args *args = data;
 	struct kfd_process_device *pdd, *peer_pdd;

[Severity: High]
Will memdup_array_user() fail unconditionally when this API is invoked by a kernel caller?

kfd_ioctl_map_memory_to_gpu() copies the device_ids_array_ptr parameter using memdup_array_user(). Because memdup_array_user() internally relies on copy_from_user(), it will fail the access_ok() check and return -EFAULT if provided a kernel-allocated array of device IDs.

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int kfd_ioctl_import_dmabuf(struct file *filep, · line 1634

Quoted 11 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -2
@@ -1624,8 +1627,8 @@ static int kfd_ioctl_import_dmabuf(struct file *filep,
 	return r;
 }
 
-static int kfd_ioctl_export_dmabuf(struct file *filep,
-				   struct kfd_process *p, void *data)
+int kfd_ioctl_export_dmabuf(struct file *filep,
+			    struct kfd_process *p, void *data)
 {
 	struct kfd_ioctl_export_dmabuf_args *args = data;
 	struct kfd_process_device *pdd;

[Severity: Medium]
Does this code permanently leak the file descriptor when called from a kernel thread?

kfd_ioctl_export_dmabuf() calls dma_buf_fd(), which uses fd_install() to allocate an integer file descriptor in the current process's FD table. Because kernel threads execute with isolated FD tables and have no mechanism to pass integer FDs back to userspace or automatically clean them up, both the file descriptor and its underlying dmabuf reference might leak.

Quoted 37 lines
drivers/gpu/drm/amd/amdkfd/kfd_events.c +29 -0
index 81900b49d9d5b..91ef4c61a8bc3 100644
@@ -1068,6 +1068,116 @@ int kfd_wait_on_events(struct kfd_process *p,
 	return ret;
 }
 
+int kfd_wait_on_events_kernel(struct kfd_process *p,
+			      uint32_t num_events, void __user *data,
+			      bool all, uint32_t *user_timeout_ms,
+			      uint32_t *wait_result)
+{
+	struct kfd_event_data *events = (struct kfd_event_data *) data;
+	uint32_t i;
+	int ret = 0;
+
+	struct kfd_event_waiter *event_waiters = NULL;
+	long timeout = user_timeout_to_jiffies(*user_timeout_ms);
+
+	event_waiters = alloc_event_waiters(num_events);
+	if (!event_waiters) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	/* Use p->event_mutex here to protect against concurrent creation and
+	 * destruction of events while we initialize event_waiters.
+	 */
+	mutex_lock(&p->event_mutex);
+
+	for (i = 0; i < num_events; i++) {
+		struct kfd_event_data event_data;
+
+		memcpy(&event_data, &events[i], sizeof(struct kfd_event_data));
+		ret = init_event_waiter(p, &event_waiters[i], &event_data);
+		if (ret)

drivers/gpu/drm/amd/amdkfd/kfd_events.c int kfd_wait_on_events(struct kfd_process *p, · line 1109

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_events.c +10 -0
+			goto out_unlock;
+	}
+
+	/* Check condition once. */
+	*wait_result = test_event_condition(all, num_events, event_waiters);
+	if (*wait_result == KFD_IOC_WAIT_RESULT_COMPLETE) {
+		ret = copy_signaled_event_data(num_events,
+					       event_waiters, events);
+		goto out_unlock;
+	}

[Severity: High]
Can this mix of memory handling semantics cause failures for both user and kernel callers?

When evaluating the loop in kfd_wait_on_events_kernel(), there is a direct memcpy() from the data pointer. If a user pointer is inadvertently passed, this bypasses copy_from_user(), which can trigger a SMAP violation.

However, when waking up on the fast or slow path, it relies on copy_signaled_event_data() which internally uses copy_to_user(). If a kernel caller correctly provides a kernel pointer, copy_to_user() fails the access_ok() check and returns -EFAULT, causing the event data to be dropped.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author
#15 lore raw

RFC PATCH 7/13 drm/amdkfd: add knod provider core

In reply to Taehee Yoo

Add the knod accelerator provider, built into amdgpu. It allocates and drives GPU AQL/SDMA queues, manages GPU memory for the RX data path, and registers a knod accelerator that binds to a NIC via the knod core. Feature workers (BPF, IPsec) plug in on top through the accel ops.

(cherry picked from commit acaff825a0e87188906fd59f8a377c04d0b0ed2e)
---
 drivers/gpu/drm/amd/amdgpu/Makefile        |    1 +
 drivers/gpu/drm/amd/amdkfd/Kconfig         |   11 +
 drivers/gpu/drm/amd/amdkfd/Makefile        |    4 +
 drivers/gpu/drm/amd/amdkfd/kfd_knod.c      | 2202 ++++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h |  270 +++
 5 files changed, 2488 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_knod.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h
drivers/gpu/drm/amd/amdgpu/Makefile +1 -0
index ba80542ead9d..7a9826df9f37 100644
@@ -42,6 +42,7 @@ ccflags-y := -I$(FULL_AMD_PATH)/include/asic_reg \
 	-I$(FULL_AMD_DISPLAY_PATH)/dc \
 	-I$(FULL_AMD_DISPLAY_PATH)/amdgpu_dm \
 	-I$(FULL_AMD_PATH)/amdkfd \
+	-I$(FULL_AMD_PATH)/amdkfd/knod \
 	-I$(FULL_AMD_PATH)/ras/ras_mgr
 
 # Locally disable W=1 warnings enabled in drm subsystem Makefile
drivers/gpu/drm/amd/amdkfd/Kconfig +11 -0
index a5d7467c2f34..d93f1af749ff 100644
@@ -38,3 +38,14 @@ config HSA_AMD_P2P
 	  GPUs with large memory BARs that expose the entire VRAM in PCIe bus
 	  address space within the physical address limits of the GPUs.
 
+config HSA_AMD_KNOD
+	bool "KNOD GPU network offload core"
+	depends on HSA_AMD
+	default y
+	help
+	  Core framework for KNOD, GPU-accelerated zero-copy network packet
+	  processing on AMD GPUs.  It is built into amdgpu and cannot be a
+	  module because it relies on non-exported amdgpu/amdkfd internals.
+
+	  Say N to drop the KNOD core along with the BPF and IPsec offloads
+	  layered on top of it.  If unsure, say Y.
drivers/gpu/drm/amd/amdkfd/Makefile +4 -0
index 85fc67d521e5..1834faa54863 100644
@@ -71,3 +71,7 @@ ifneq ($(CONFIG_HSA_AMD_SVM),)
 AMDKFD_FILES += $(AMDKFD_PATH)/kfd_svm.o \
 		$(AMDKFD_PATH)/kfd_migrate.o
 endif
+
+ifneq ($(CONFIG_HSA_AMD_KNOD),)
+AMDKFD_FILES += $(AMDKFD_PATH)/kfd_knod.o
+endif
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +2202 -0

2,202 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +270 -0

270 lines not on this page · raw message

Signature · 1 line
2.43.0
#16 lore raw

Re: [RFC PATCH net-next 07/13] drm/amdkfd: add knod provider core

Thank you for your contribution! Sashiko AI review found 22 potential issue(s) to consider: - [Critical] Reliance on a user-space binary (/bin/sleep) to provide a process context for the GPU network…

Thank you for your contribution! Sashiko AI review found 22 potential issue(s) to consider:
- [Critical] Reliance on a user-space binary (/bin/sleep) to provide a process context for the GPU network offload core.
- [Critical] Vulnerability to PID recycling in stop_umh(), leading to potential arbitrary process termination.
- [High] Hardcoded VFS path lookup from kernel space to open DRM render nodes makes the core vulnerable to namespace spoofing.
- [High] AMDGPU ring buffer write pointers are defined as 32-bit signed integers instead of u64, causing sign-extension corruption.
- [High] Global accels array is overwritten when multiple AMD GPUs are initialized.
- [High] Array bounds overflow for queue structures due to NR_CPUS vs KNOD_MAX_QUEUE_CNT mismatch.
- [High] GPU memory allocated by __knod_alloc_mem is silently leaked upon free.
- [High] GPU virtual addresses are passed as __user pointers to KFD queue creation.
- [High] The KFD process and MMU notifier are leaked on initialization error paths in knod_alloc_ctx_init and knod_alloc_ctx.
- [High] Hardcoded access to pdds[0] causes index mismatches on multi-GPU systems.
- [High] SDMA queue GPU memory is not explicitly zeroed, causing the hardware engine to execute garbage commands.
- [High] knod_exit() tears down knod contexts without stopping the background worker thread, leading to a Use-After-Free.
- [High] Concurrent modification of a single SDMA ring buffer across multiple CPUs.
- [High] Modifying accel_ops.xdp_ops during per-device feature activation overrides features for all devices system-wide.
- [High] Unprotected concurrent modifications to the global ctx_list.
- [High] Data race between MMU notifier cleanup and KNOD's explicit queue teardown.
- [High] Concurrent feature lifecycle operations can lead to a data race on knod->worker, causing multiple threads to simultaneously stop the same kthread.
- [High] Hardware packets and descriptors are written to GPU memory using host native endianness instead of little-endian.
- [High] A memory barrier is missing before writing to the AQL doorbell, which can cause the GPU to observe a stale dispatch ID.
- [Medium] Lack of locking in knod_accel_xdp_register() allows it to race with knod_attach(), causing missed XDP initialization.
- [Medium] knod_init returns 0 (success) on allocation failure, causing silent failures during accelerator initialization.
- [Low] A typo in the endianness conditional macro for knod_aql_rsrc1 causes the struct to be empty on big-endian architectures. --

--- Patch [7]: [RFC PATCH net-next 07/13] drm/amdkfd: add knod provider core --- commit 3676145304477ec735999c96871ed634c55e8b47
Author: Taehee Yoo <[email protected]>

drm/amdkfd: add knod provider core

This patch introduces the knod accelerator provider into amdgpu. It allocates and drives GPU AQL/SDMA queues, manages GPU memory for the RX data path, and registers a knod accelerator that binds to a NIC via the knod core.

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +0 -0
@@ -0,0 +1,2202 @@

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 549

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +3 -0
+static int launch_and_get_pid(void)
+{
+	static const char * const argv[] = { "/bin/sleep", "2147483647", NULL };

[Severity: Critical]
Is it safe to rely on a user-space binary to provide the process context?

The core uses call_usermodehelper_exec() to launch /bin/sleep to bind the KFD GPU context. If the /bin/sleep process is killed by an administrator, the OOM killer, or exits unexpectedly, the KFD MMU notifier will asynchronously tear down the GPU resources locklessly.

Can this lead to a severe crash in the network driver?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 536

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +10 -0
+static void stop_umh(pid_t umh_pid)
+{
+	struct pid *p = find_get_pid(umh_pid);
+
+	if (!p)
+		return;
+
+	kill_pid(p, SIGKILL, 1);
+	put_pid(p);
+}

[Severity: Critical]
Is there a risk of terminating unrelated processes due to PID recycling?

When the KNOD context is released, stop_umh() looks up the user mode helper process by its numeric PID and unconditionally sends a SIGKILL.

Because knod_alloc_ctx() drops the safe struct pid reference immediately after creation, if the original helper process exited prematurely, its PID might have been reassigned. Could detaching the network device unintentionally kill an innocent process?

[ ... ]

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +3 -0
+struct knod_mem *__knod_alloc_mem(struct knod *knod, size_t size,
+				  int flags)
+{

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 151

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +4 -0
+	err = amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(kdev->adev, mem->gaddr,
+						      size, pdd->drm_priv,
+						      &mem->mem, NULL, flags,
+						      false);

[Severity: High]
Does this code leak GPU memory when the packets are freed?

Looking at __knod_alloc_mem(), it uses amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu() directly which bypasses the KFD tracking list.

However, knod_free_mem() uses kfd_process_free_gpuvm() which expects the memory to be in the process tracking list. If it is not found, will the GPU memory remain permanently leaked?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 298

Quoted 11 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +11 -0
+void knod_sdma_copy(struct knod *knod, u64 dst_offset,
+		    u64 src_offset, int idx, int size)
+{
+	struct knod_sdma *sdma = &knod->sdma[idx];
+	u32 ring_mask = (sdma->sdma->size / 4) - 1;
+	u64 *wptr = (u64 *)sdma->queue->kaddr + 1;
+	u32 *ptr = sdma->sdma->kaddr;
+
+	ptr[sdma->idx++ & ring_mask] = SDMA_PKT_HEADER_OP(SDMA_OP_COPY) |
+		SDMA_PKT_HEADER_SUB_OP(SDMA_SUBOP_COPY_LINEAR) |
+		SDMA_PKT_COPY_LINEAR_HEADER_TMZ((0));

[Severity: High]
Can this sequence cause data races and ring buffer corruption?

This function is called from NAPI context by multiple RX queues concurrently, but it increments the software write cursor (sdma->idx++) and writes to the ring buffer without locks or atomic operations.

[Severity: High]
Are these descriptors written to the GPU in the correct byte order?

AMDGPU hardware typically expects ring buffer packets to be formatted in little-endian. Writing multi-byte fields directly using native host endianness might cause the hardware to interpret garbage commands on big-endian systems. Should helpers like cpu_to_le32() be used here?

[ ... ]

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +2 -0
+static void knod_destroy_one_queue(struct knod *knod, int idx)
+{

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 639

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +5 -0
+	if (knod->aql_queue_created[idx]) {
+		int ret;
+
+		ret = knod_destroy_queue(knod->process,
+					 knod->aql_queue_id[idx]);

[Severity: High]
Does this manual queue teardown race with the lockless MMU notifier cleanup?

If the dummy /bin/sleep process is killed, mmu_notifier_release() will call pqm_uninit() which locklessly iterates over and frees pqm->queues.

If the KNOD framework concurrently initiates cleanup via knod_destroy_queue(), which locks p->mutex and deletes queues from the exact same list, could this lead to list corruption and double-freeing of process_queue_node structures?

[ ... ]

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +4 -0
+static int knod_alloc_one_queue(struct knod *knod, int idx,
+				struct kfd_topology_device *topo_dev,
+				struct kfd_process_device *pdd, void *ptr)
+{

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 792

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +3 -0
+	qp.write_ptr =
+		(void __user *)((u64)knod->kaql[idx].queue->gaddr + 0x38);
+	qp.read_ptr = (void __user *)((u64)knod->kaql[idx].queue->gaddr + 0x80);

[Severity: High]
Can casting a GPU virtual address to a user-space CPU address cause faults?

KFD queue management uses get_user() and put_user() on these pointers. Since the raw GPU virtual address has no CPU mapping in the user process, won't this trigger an -EFAULT page fault during queue creation or sync operations?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 977

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +10 -0
+static void knod_stop_worker(struct knod *knod)
+{
+	if (!knod->worker)
+		return;
+
+	if (knod->flush_fn)
+		knod->flush_fn(knod->worker_ctx);
+
+	kthread_stop(knod->worker);
+	put_task_struct(knod->worker);

[Severity: High]
Is there a data race here that could lead to a double-free of the task struct?

knod_stop_worker() checks and modifies knod->worker without holding any locks. If feature transitions and module unload invoke this concurrently, two threads could simultaneously pass the if (!knod->worker) check.

Both would then call kthread_stop() and put_task_struct() on the same task structure, potentially dropping the refcount to zero prematurely.

[ ... ]

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +4 -0
+static int knod_alloc_ctx_init(struct knod *knod, int id, void **doorbell,
+			       struct kfd_topology_device **out_topo_dev,
+			       struct kfd_process_device **out_pdd)
+{

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1035

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +10 -0
+	struct kfd_topology_device *topo_dev;
+	struct kfd_process_device *pdd;
+	struct file *drm_file;
+	size_t mem_size;
+	char path[64];
+	int err;
+
+	sprintf(path, "/dev/dri/renderD%d", id);
+
+	drm_file = filp_open(path, O_RDWR, 0);

[Severity: High]
Does this hardcoded VFS path lookup introduce a namespace spoofing risk?

The initialization path uses filp_open() from kernel space to obtain a struct file for the DRM device, which resolves in the user-space mount namespace.

If an unprivileged user in a user namespace bind-mounts a malicious file over /dev/dri/renderD0, could this lead to arbitrary kernel memory read/write when AMDGPU casts filp->private_data without validating the inode?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1052

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +1 -0
+	pdd = knod->process->pdds[0];

[Severity: High]
Will unconditionally accessing pdds[0] fail on multi-GPU systems?

This hardcodes access to the first AMD GPU, regardless of the renderD node passed to knod_alloc_ctx_init().

On a system with multiple GPUs, would this result in initializing the VM of GPU 0 with the DRM file of a different GPU?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1161

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +6 -0
+err_unref_process:
+	kfd_unref_process(knod->process);
+err_filp_close:
+	fput(knod->drm_file);
+	return err;
+}

[Severity: High]
Are the KFD process and MMU notifier leaked on this error path?

The setup code earlier takes two references to the KFD process, but kfd_unref_process() is only called once here.

Additionally, kfd_process_notifier_release_internal() is omitted. Will this leave the process permanently embedded in KFD's hash tables and mmu_notifier tracking?

[ ... ]

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +3 -0
+struct knod *knod_alloc_ctx(struct knod_dev *knodev, int queue_cnt, int id,
+			    int channels)
+{

[ ... ]

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +1 -0
+	for (idx = 0; idx < knod->sdma_cnt; idx++) {

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1311

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +6 -0
+		knod->sdma[idx].sdma = knod_alloc_mem(knod, PAGE_SIZE << 4,
+			KFD_IOC_ALLOC_MEM_FLAGS_GTT |
+			KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE |
+			KFD_IOC_ALLOC_MEM_FLAGS_COHERENT |
+			KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED |
+			KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE);

[Severity: High]
Can the hardware engine execute garbage commands here?

Unlike the AQL queues which use memset() to clear the memory, the SDMA queue memory (sdma, queue, and queue_signal) allocated here is not explicitly zeroed.

When this queue is submitted to the hardware, could the command processor read uninitialized read/write pointers and execute random garbage memory?

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1413

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +1 -0
+	list_add_tail(&knod->list, &ctx_list);

[Severity: High]
Is it safe to modify the global ctx_list without holding a lock?

knod_alloc_ctx() adds the newly allocated context to the global ctx_list via list_add_tail(), and knod_release_ctx() removes it via list_del().

Neither of these operations appears to be protected by spinlocks or mutexes, which could lead to list corruption if multiple interfaces are attached or detached concurrently.

[ ... ]

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +5 -0
+void knod_release_ctx(struct knod *knod)
+{
+	int idx;
+
+	list_del(&knod->list);

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +2 -0
+	kfree(knod);
+}

[Severity: High]
Does tearing down the context here leave the worker thread running?

Neither knod_exit() nor knod_release_ctx() calls knod_stop_worker(). If the module is unloaded while a knod context is still active, the active knod->worker kthread might be left running in the background and could immediately dereference the freed knod context, causing a Use-After-Free.

[ ... ]

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +2 -0
+static int knod_feature_activate(struct knod *knod)
+{

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1673

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +6 -0
+		/*
+		 * Publish xdp_ops, then phase B: register the offload dev so
+		 * user XDP progs/maps can bind (their install/free route
+		 * through accel_ops.xdp_ops->xdp_install).
+		 */
+		WRITE_ONCE(accel_ops.xdp_ops, registered_xdp_ops);

[Severity: High]
Will this override the features for all devices system-wide?

accel_ops is a single static global instance shared among all registered knod_accel contexts. Modifying accel_ops.xdp_ops during a per-device feature activation will immediately alter the operations for all other devices concurrently.

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 1965

Quoted 13 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +13 -0
+static u32 knod_accel_d2h_submit(struct knod_dev *knodev, u64 dst, int queue,
+				 u32 page_idx, u16 off, u32 len)
+{
+	struct knod *knod = knodev->accel->priv;
+	struct knod_sdma_copy_desc c;
+
+	/* The netmem page_pool dma_addr is the NIC address; SDMA needs
+	 * the GPU VM address, so derive the source from buf[queue].
+	 */
+	c.src = knod->buf[queue]->gaddr + ((u64)page_idx << PAGE_SHIFT) + off;
+	c.dst = dst;
+	c.len = len;
+	return knod_sdma_submit(knod, 0, &c, 1);

[Severity: High]
Does hardcoding the SDMA queue index to 0 cause concurrent modifications?

knod_accel_d2h_submit() passes 0 for the SDMA queue index to knod_sdma_submit().

Because this is called from NAPI context by multiple RX queues concurrently, it will result in data races and corrupted SDMA ring buffers across multiple CPUs.

[ ... ]

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 2057

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +10 -0
+void knod_accel_xdp_register(struct knod_accel_xdp_ops *xdp_ops)
+{
+	int i;
+
+	/*
+	 * Module load advertises the feature and sets up the permanent
+	 * per-attach state (bpf_offload_dev) on already-attached accels;
+	 * GPU compute resources wait for ->activate() on feature select.
+	 */
+	WRITE_ONCE(registered_xdp_ops, xdp_ops);

[Severity: Medium]
Is there a race condition with knod_attach() here due to a lack of locking?

knod_accel_xdp_register() writes to registered_xdp_ops and iterates over accels[i]->knodev without synchronization.

If a device is attaching simultaneously, knod_attach() might read registered_xdp_ops as NULL, while knod_accel_xdp_register() sees accel->knodev before it is fully initialized, potentially missing the initialization or crashing.

[ ... ]

Quoted 13 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +13 -0
+int knod_init(struct amdgpu_device *adev)
+{
+	int base_id = adev_to_drm(adev)->render->index * KNOD_MAX_AQL;
+	struct knod_accel *accel;
+	int i, n;
+
+	n = KNOD_MAX_AQL;
+
+	for (i = 0; i < n; i++) {
+		accel = kzalloc_obj(struct knod_accel, GFP_KERNEL);
+		if (!accel) {
+			pr_err("knod: Failed to allocate accel node %d\n", i);
+			goto err;

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 2155

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +10 -0
+		}
+
+		INIT_LIST_HEAD(&accel->list);
+		accel->type = KNOD_TYPE_GPU;
+		accel->accel_ops = &accel_ops;
+		accel->owner = THIS_MODULE;
+		accel->id = base_id + i;
+		snprintf(accel->name, sizeof(accel->name), "amdgpu-%d", i);
+		knod_accel_register(accel);
+		accels[i] = accel;

[Severity: High]
Does this unintentionally overwrite the global accels array for system GPUs?

The global accels array is populated based on index i (which loops 0 to n - 1, where n is KNOD_MAX_AQL which is 1). Since knod_init() is called per-device, it will unconditionally store the allocated context at accels[0].

On systems with multiple AMD GPUs, this overwrites accels[0], leaking the previous GPU's context and preventing its teardown later.

drivers/gpu/drm/amd/amdkfd/kfd_knod.c line 2168

Quoted 13 lines
drivers/gpu/drm/amd/amdkfd/kfd_knod.c new file +13 -0
+	}
+	nr_accels = n;
+
+	return 0;
+
+err:
+	while (i-- > 0) {
+		knod_accel_unregister(accels[i]);
+		kfree(accels[i]);
+		accels[i] = NULL;
+	}
+	return 0;
+}

[Severity: Medium]
Should the error path return a negative error code instead of 0?

If kzalloc_obj() fails, the err label cleans up the allocations but immediately returns 0. This masks the allocation failure from the upstream caller, leading to silent failures and likely null-pointer dereferences downstream.

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +0 -0
@@ -0,0 +1,270 @@

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h line 42

Quoted 8 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +8 -0
+union knod_aql_rsrc1 {
+	struct {
+#if defined(__LITTLE_ENDIAN)
+		unsigned int base_address_hi : 16;
+		unsigned int stride : 14;
+		unsigned int cache_swizzle : 1;
+		unsigned int swizzle_enable : 1;
+#elif defined(__LITTLE_ENDIAN)

[Severity: Low]
Is the second elif directive a typo for big-endian architectures?

The conditional macro uses elif defined(__LITTLE_ENDIAN) again instead of elif defined(__BIG_ENDIAN).

On a big-endian system, neither condition is met, leaving the struct entirely empty and causing compilation errors.

[ ... ]

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +2 -0
+struct knod_aql {
+	struct knod_mem *aql;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h line 63

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +3 -0
+	u64 *doorbell;
+	int idx;
+};

[Severity: High]
Can this signed 32-bit integer write pointer cause sign-extension corruption?

The AQL write pointer idx is declared as a signed int instead of a u64. After enough operations, this pointer will overflow to a negative value.

In knod_setup_header(), this negative value will undergo sign-extension to 64 bits when written to the hardware doorbell via writeq(), which breaks the hardware's monotonic expectation and violates AMDGPU subsystem rules.

[ ... ]

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +2 -0
+struct knod {
+	struct list_head list;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h line 137

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +1 -0
+	struct knod_aql kaql[NR_CPUS];

[Severity: High]
Could this array sizing cause a heap buffer overflow?

The array fields for queue tracking are sized using NR_CPUS. However, in knod_alloc_ctx(), the loop iterates up to queue_cnt, which can be up to KNOD_MAX_QUEUE_CNT (32).

On kernel configurations where NR_CPUS is less than 32, this loop will write out-of-bounds.

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h line 219

Quoted 13 lines
drivers/gpu/drm/amd/amdkfd/knod/kfd_knod.h new file +13 -0
+static inline void
+knod_setup_header(struct knod *knod,
+		  const struct knod_dispatch_params *p, int q_idx)
+{
+	struct amd_queue *amd_queue = (struct amd_queue *)knod->kaql[q_idx].amd_queue->kaddr;
+	int curr_idx = knod->kaql[q_idx].idx;
+	int next_idx = curr_idx + 1;
+	u64 *ptr = knod->kaql[q_idx].doorbell;
+
+	knod_setup_invalidate(knod, next_idx % knod->nr_aql_ring, q_idx);
+	knod_setup_dispatch(knod, curr_idx % knod->nr_aql_ring, p, q_idx);
+	WRITE_ONCE(amd_queue->write_dispatch_id, curr_idx);
+	writeq(curr_idx, ptr);

[Severity: High]
Is a memory barrier needed before writing to the AQL doorbell?

While WRITE_ONCE prevents compiler reordering, it does not prevent CPU reordering. The CPU can reorder the doorbell write so it becomes visible to the hardware before the write_dispatch_id store reaches memory.

Should there be a wmb() before the writeq() to ensure the GPU does not observe a stale dispatch ID?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author
#17 lore raw

RFC PATCH 8/13 drm/amdkfd: add GPU instruction emitter and disassembler

In reply to Taehee Yoo

Add the AMD GCN (gfx9/gfx10) instruction encoder used to build the GPU shaders that knod dispatches, plus a matching disassembler used for debugging the generated code.

(cherry picked from commit bbbe8531de11017f565a922b072d4aa5f99674fc)
---
 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h |  173 +
 .../drm/amd/amdkfd/knod/knod_amdgpu_insn.h    | 6362 +++++++++++++++++
 .../gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h | 3978 +++++++++++
 .../gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h  | 4068 +++++++++++
 4 files changed, 14581 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h
drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h new file +173 -0
new file mode 100644
index 000000000000..77d1c6afdd0a
@@ -0,0 +1,173 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright (c) 2021 Taehee Yoo <[email protected]>
+ * Copyright (c) 2021 Hoyeon Lee <[email protected]>
+ */
+
+#ifndef KFD_AMDGPU_H_INCLUDED
+#define KFD_AMDGPU_H_INCLUDED
+
+#define KNOD_AMDGPU_REG_PAIR(x)         ((x) / 2)
+
+enum amdgcn_param_type {
+	AMDGCN_PARAM_TYPE_SGPR,
+	AMDGCN_PARAM_TYPE_VCC_LO,
+	AMDGCN_PARAM_TYPE_VCC_HI,
+	AMDGCN_PARAM_TYPE_TTPM_BASE,
+	AMDGCN_PARAM_TYPE_M0,
+	AMDGCN_PARAM_TYPE_NULL,
+	AMDGCN_PARAM_TYPE_EXEC_LO,
+	AMDGCN_PARAM_TYPE_EXEC_HI,
+	AMDGCN_PARAM_TYPE_INTEGER_0,
+	AMDGCN_PARAM_TYPE_INTEGER_MINUS_1,
+	AMDGCN_PARAM_TYPE_SHARED_BASE,
+	AMDGCN_PARAM_TYPE_SHARED_LIMIT,
+	AMDGCN_PARAM_TYPE_PRIVATE_BASE,
+	AMDGCN_PARAM_TYPE_PRIVATE_LIMIT,
+	AMDGCN_PARAM_TYPE_POPS_EXITING_WAVE_ID,
+	AMDGCN_PARAM_TYPE_SDWA,
+	AMDGCN_PARAM_TYPE_DPP16,
+	AMDGCN_PARAM_TYPE_VCCZ,
+	AMDGCN_PARAM_TYPE_EXECZ,
+	AMDGCN_PARAM_TYPE_SCC,
+	AMDGCN_PARAM_TYPE_LITERAL_CONST,
+	AMDGCN_PARAM_TYPE_VGPR,
+	__AMDGCN_PARAM_TYPE_MAX,
+};
+
+struct amdgcn_param32 {
+	enum amdgcn_param_type type;
+	int v;
+};
+
+struct amdgcn_param64 {
+	struct amdgcn_param32 lo;
+	struct amdgcn_param32 hi;
+	u64 imm;
+};
+
+inline void knod_set(struct amdgcn_param32 *param,
+				  enum amdgcn_param_type type, int v)
+{
+	param->type = type;
+	param->v = v;
+}
+
+inline void knod_vset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_VGPR;
+	param->v = v;
+}
+
+inline void knod_vset64(struct amdgcn_param64 *param, int v)
+{
+	param->lo.type = AMDGCN_PARAM_TYPE_VGPR;
+	param->lo.v = v;
+	param->hi.type = AMDGCN_PARAM_TYPE_VGPR;
+	param->hi.v = v + 1;
+}
+
+inline void knod_sset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_SGPR;
+	param->v = v;
+}
+
+inline void knod_sset64(struct amdgcn_param64 *param, int v)
+{
+	param->lo.type = AMDGCN_PARAM_TYPE_SGPR;
+	param->lo.v = v;
+	param->hi.type = AMDGCN_PARAM_TYPE_SGPR;
+	param->hi.v = v + 1;
+}
+
+inline void knod_iset32(struct amdgcn_param32 *param, int v)
+{
+	if (v > 64 || v < -16) {
+		param->type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		param->v = v;
+	} else if (v >= 0) {
+		param->type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->v = v;
+	} else {
+		param->type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->v = ~v;
+	}
+}
+
+inline void knod_iset64(struct amdgcn_param64 *param,
+				     u64 v)
+{
+	int imm0 = v & ~0U;
+	int imm1 = v >> 32;
+
+	param->imm = v;
+
+	if (imm0 > 64 || imm0 < -16) {
+		param->lo.type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		param->lo.v = imm0;
+	} else if (imm0 >= 0) {
+		param->lo.type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->lo.v = imm0;
+	} else {
+		param->lo.type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->lo.v = ~imm0;
+	}
+
+	if (imm1 > 64 || imm1 < -16) {
+		param->hi.type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		WARN_ON_ONCE(1);
+		param->hi.v = imm1;
+	} else if (imm1 >= 0) {
+		param->hi.type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->hi.v = imm1;
+	} else {
+		param->hi.type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->hi.v = ~imm1;
+	}
+}
+
+inline void knod_lset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+	param->v = v;
+}
+
+inline void knod_vccset(struct amdgcn_param32 *param)
+{
+	param->type = AMDGCN_PARAM_TYPE_VCC_LO;
+	param->v = 0;
+}
+
+inline bool knod_param_is_literal(struct amdgcn_param32 param)
+{
+	return param.type == AMDGCN_PARAM_TYPE_LITERAL_CONST;
+}
+
+/* ======================================================================
+ * Param constructors - common to GFX9/GFX10 shader emitters.
+ *
+ * Usage: pass directly to emit_gfx{9,10}_* functions that take
+ *        struct amdgcn_param32 operands.
+ * ======================================================================
+ */
+
+#define P_S(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_SGPR, (n) })
+#define P_V(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VGPR, (n) })
+#define P_I(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_INTEGER_0, (n) })
+#define P_L(v)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_LITERAL_CONST, (v) })
+#define P_VCC	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VCC_LO, 0 })
+#define P_EXEC	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_EXEC_LO, 0 })
+
+/* ======================================================================
+ * Branch patching - operates directly on u32 *buf
+ * ======================================================================
+ */
+
+static inline void patch_branch(u32 *buf, int patch_pos, int target_pos)
+{
+	int offset = target_pos - (patch_pos + 1);
+
+	buf[patch_pos] = (buf[patch_pos] & 0xFFFF0000u) | (offset & 0xFFFF);
+}
+
+#endif
drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h new file +6362 -0

2,827 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h new file +3978 -0

3,978 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h new file +4068 -0

4,068 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.43.0

Part of the diff is not shown here. View the raw message.

#18 lore raw

Re: [RFC PATCH net-next 08/13] drm/amdkfd: add GPU instruction emitter and disassembler

Quoted 3 lines · Taehee Yoo

Add the AMD GCN (gfx9/gfx10) instruction encoder used to build the GPU shaders that knod dispatches, plus a matching disassembler used for debugging the generated code.

Is it really necessary to have a full-on compiler and disassembler in the kernel driver? This patch is massive and I'm wondering how much benefit it really provides. Is there really no way to move GPU compilation out of the kernel, one way or another? Could you get acceptable perf with a static shader that interprets BPF programs at runtime? Such a shader can be compiled beforehand and just embedded into the kernel - there's prior art there with the CWSR trap handler in amdkfd.

In case you really, really need to compile the BPF to native ISA, could you still have userspace take care of that in one way or another?

This would probably reduce the size of your patches to much more reasonable numbers as well.

Best,
Natalie

Quoted 14,619 lines
(cherry picked from commit bbbe8531de11017f565a922b072d4aa5f99674fc)
---
  drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h |  173 +
  .../drm/amd/amdkfd/knod/knod_amdgpu_insn.h    | 6362 +++++++++++++++++
  .../gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h | 3978 +++++++++++
  .../gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h  | 4068 +++++++++++
  4 files changed, 14581 insertions(+)

create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h

drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu.h new file +173 -0
new file mode 100644
index 000000000000..77d1c6afdd0a
@@ -0,0 +1,173 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright (c) 2021 Taehee Yoo <[email protected]>
+ * Copyright (c) 2021 Hoyeon Lee <[email protected]>
+ */
+
+#ifndef KFD_AMDGPU_H_INCLUDED
+#define KFD_AMDGPU_H_INCLUDED
+
+#define KNOD_AMDGPU_REG_PAIR(x)         ((x) / 2)
+
+enum amdgcn_param_type {
+	AMDGCN_PARAM_TYPE_SGPR,
+	AMDGCN_PARAM_TYPE_VCC_LO,
+	AMDGCN_PARAM_TYPE_VCC_HI,
+	AMDGCN_PARAM_TYPE_TTPM_BASE,
+	AMDGCN_PARAM_TYPE_M0,
+	AMDGCN_PARAM_TYPE_NULL,
+	AMDGCN_PARAM_TYPE_EXEC_LO,
+	AMDGCN_PARAM_TYPE_EXEC_HI,
+	AMDGCN_PARAM_TYPE_INTEGER_0,
+	AMDGCN_PARAM_TYPE_INTEGER_MINUS_1,
+	AMDGCN_PARAM_TYPE_SHARED_BASE,
+	AMDGCN_PARAM_TYPE_SHARED_LIMIT,
+	AMDGCN_PARAM_TYPE_PRIVATE_BASE,
+	AMDGCN_PARAM_TYPE_PRIVATE_LIMIT,
+	AMDGCN_PARAM_TYPE_POPS_EXITING_WAVE_ID,
+	AMDGCN_PARAM_TYPE_SDWA,
+	AMDGCN_PARAM_TYPE_DPP16,
+	AMDGCN_PARAM_TYPE_VCCZ,
+	AMDGCN_PARAM_TYPE_EXECZ,
+	AMDGCN_PARAM_TYPE_SCC,
+	AMDGCN_PARAM_TYPE_LITERAL_CONST,
+	AMDGCN_PARAM_TYPE_VGPR,
+	__AMDGCN_PARAM_TYPE_MAX,
+};
+
+struct amdgcn_param32 {
+	enum amdgcn_param_type type;
+	int v;
+};
+
+struct amdgcn_param64 {
+	struct amdgcn_param32 lo;
+	struct amdgcn_param32 hi;
+	u64 imm;
+};
+
+inline void knod_set(struct amdgcn_param32 *param,
+				  enum amdgcn_param_type type, int v)
+{
+	param->type = type;
+	param->v = v;
+}
+
+inline void knod_vset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_VGPR;
+	param->v = v;
+}
+
+inline void knod_vset64(struct amdgcn_param64 *param, int v)
+{
+	param->lo.type = AMDGCN_PARAM_TYPE_VGPR;
+	param->lo.v = v;
+	param->hi.type = AMDGCN_PARAM_TYPE_VGPR;
+	param->hi.v = v + 1;
+}
+
+inline void knod_sset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_SGPR;
+	param->v = v;
+}
+
+inline void knod_sset64(struct amdgcn_param64 *param, int v)
+{
+	param->lo.type = AMDGCN_PARAM_TYPE_SGPR;
+	param->lo.v = v;
+	param->hi.type = AMDGCN_PARAM_TYPE_SGPR;
+	param->hi.v = v + 1;
+}
+
+inline void knod_iset32(struct amdgcn_param32 *param, int v)
+{
+	if (v > 64 || v < -16) {
+		param->type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		param->v = v;
+	} else if (v >= 0) {
+		param->type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->v = v;
+	} else {
+		param->type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->v = ~v;
+	}
+}
+
+inline void knod_iset64(struct amdgcn_param64 *param,
+				     u64 v)
+{
+	int imm0 = v & ~0U;
+	int imm1 = v >> 32;
+
+	param->imm = v;
+
+	if (imm0 > 64 || imm0 < -16) {
+		param->lo.type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		param->lo.v = imm0;
+	} else if (imm0 >= 0) {
+		param->lo.type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->lo.v = imm0;
+	} else {
+		param->lo.type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->lo.v = ~imm0;
+	}
+
+	if (imm1 > 64 || imm1 < -16) {
+		param->hi.type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+		WARN_ON_ONCE(1);
+		param->hi.v = imm1;
+	} else if (imm1 >= 0) {
+		param->hi.type = AMDGCN_PARAM_TYPE_INTEGER_0;
+		param->hi.v = imm1;
+	} else {
+		param->hi.type = AMDGCN_PARAM_TYPE_INTEGER_MINUS_1;
+		param->hi.v = ~imm1;
+	}
+}
+
+inline void knod_lset32(struct amdgcn_param32 *param, int v)
+{
+	param->type = AMDGCN_PARAM_TYPE_LITERAL_CONST;
+	param->v = v;
+}
+
+inline void knod_vccset(struct amdgcn_param32 *param)
+{
+	param->type = AMDGCN_PARAM_TYPE_VCC_LO;
+	param->v = 0;
+}
+
+inline bool knod_param_is_literal(struct amdgcn_param32 param)
+{
+	return param.type == AMDGCN_PARAM_TYPE_LITERAL_CONST;
+}
+
+/* ======================================================================
+ * Param constructors - common to GFX9/GFX10 shader emitters.
+ *
+ * Usage: pass directly to emit_gfx{9,10}_* functions that take
+ *        struct amdgcn_param32 operands.
+ * ======================================================================
+ */
+
+#define P_S(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_SGPR, (n) })
+#define P_V(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VGPR, (n) })
+#define P_I(n)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_INTEGER_0, (n) })
+#define P_L(v)	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_LITERAL_CONST, (v) })
+#define P_VCC	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_VCC_LO, 0 })
+#define P_EXEC	((struct amdgcn_param32){ AMDGCN_PARAM_TYPE_EXEC_LO, 0 })
+
+/* ======================================================================
+ * Branch patching - operates directly on u32 *buf
+ * ======================================================================
+ */
+
+static inline void patch_branch(u32 *buf, int patch_pos, int target_pos)
+{
+	int offset = target_pos - (patch_pos + 1);
+
+	buf[patch_pos] = (buf[patch_pos] & 0xFFFF0000u) | (offset & 0xFFFF);
+}
+
+#endif
drivers/gpu/drm/amd/amdkfd/knod/knod_amdgpu_insn.h new file +6362 -0
new file mode 100644
index 000000000000..9703bf781ff5
@@ -0,0 +1,6362 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright (c) 2021 Taehee Yoo <[email protected]>
+ * Copyright (c) 2021 Hoyeon Lee <[email protected]>
+ */
+
+#ifndef KFD_AMDGPU_INSN_H_INCLUDED
+#define KFD_AMDGPU_INSN_H_INCLUDED
+
+#include "knod_amdgpu.h"
+#include "knod_gfx10_insn.h"
+#include "knod_gfx9_insn.h"
+
+enum amdgcn_insn_type {
+	AMDGCN_INSN_TYPE_SOP2,
+	AMDGCN_INSN_TYPE_SOPK,
+	AMDGCN_INSN_TYPE_SOP1,
+	AMDGCN_INSN_TYPE_SOPC,
+	AMDGCN_INSN_TYPE_SOPP,
+	AMDGCN_INSN_TYPE_SMEM,
+	AMDGCN_INSN_TYPE_VOP2,
+	AMDGCN_INSN_TYPE_VOP1,
+	AMDGCN_INSN_TYPE_VOPC,
+	AMDGCN_INSN_TYPE_VOP3A,
+	AMDGCN_INSN_TYPE_VOP3B,
+	AMDGCN_INSN_TYPE_VOP3P,
+	AMDGCN_INSN_TYPE_SDWA,
+	AMDGCN_INSN_TYPE_SDWAB,
+	AMDGCN_INSN_TYPE_DPP16,
+	AMDGCN_INSN_TYPE_DPP8,
+	AMDGCN_INSN_TYPE_VINTRP,
+	AMDGCN_INSN_TYPE_DS,
+	AMDGCN_INSN_TYPE_MTBUF,
+	AMDGCN_INSN_TYPE_MUBUF,
+	AMDGCN_INSN_TYPE_MIMG,
+	AMDGCN_INSN_TYPE_FLAT,
+	AMDGCN_INSN_TYPE_EXP,
+	__AMDGCN_INSN_TYPE_MAX,
+};
+
+static const char opnames_gfx10[__AMDGCN_INSN_TYPE_MAX][900][30] = {
+	[AMDGCN_INSN_TYPE_VOP3A] = {
+		[GFX10_V_FMA_LEGACY_F32] = "v_fma_legacy_f32",
+		[GFX10_V_MAD_I32_I24] = "V_MAD_I32_I24",
+		[GFX10_V_MAD_U32_U24] = "v_mad_u32_u24",
+		[GFX10_V_CUBEID_F32] = "v_cubeid_f32",
+		[GFX10_V_CUBESC_F32] = "v_cubesc_f32",
+		[GFX10_V_CUBETC_F32] = "v_cubetc_f32",
+		[GFX10_V_CUBEMA_F32] = "v_cubema_f32",
+		[GFX10_V_BFE_U32] = "v_bfe_u32",
+		[GFX10_V_BFE_I32] = "v_bfe_i32",
+		[GFX10_V_BFI_B32] = "v_bfi_b32",
+		[GFX10_V_FMA_F32] = "v_fma_f32",
+		[GFX10_V_FMA_F64] = "v_fma_f64",
+		[GFX10_V_LERP_U8] = "v_lerp_u8",
+		[GFX10_V_ALIGNBIT_B32] = "v_alignbit_b32",
+		[GFX10_V_ALIGNBYTE_B32] = "v_alignbyte_b32",
+		[GFX10_V_MULLIT_F32] = "v_mullit_f32",
+		[GFX10_V_MIN3_F32] = "v_min3_f32",
+		[GFX10_V_MIN3_I32] = "v_min3_i32",
+		[GFX10_V_MIN3_U32] = "v_min3_u32",
+		[GFX10_V_MAX3_F32] = "v_max3_f32",
+		[GFX10_V_MAX3_I32] = "v_max3_i32",
+		[GFX10_V_MAX3_U32] = "v_max3_u32",
+		[GFX10_V_MED3_F32] = "v_med3_f32",
+		[GFX10_V_MED3_I32] = "v_med3_i32",
+		[GFX10_V_MED3_U32] = "v_med3_u32",
+		[GFX10_V_SAD_U8] = "v_sad_u8",
+		[GFX10_V_SAD_HI_U8] = "v_sad_hi_u8",
+		[GFX10_V_SAD_U16] = "v_sad_u16",
+		[GFX10_V_SAD_U32] = "v_sad_u32",
+		[GFX10_V_CVT_PK_U8_F32] = "v_cvt_pk_u8_f32",
+		[GFX10_V_DIV_FIXUP_F32] = "v_div_fixup_f32",
+		[GFX10_V_DIV_FIXUP_F64] = "v_div_fixup_f64",
+		[GFX10_V_ADD_F64] = "v_add_f64",
+		[GFX10_V_MUL_F64] = "v_mul_f64",
+		[GFX10_V_MIN_F64] = "v_min_f64",
+		[GFX10_V_MAX_F64] = "v_max_f64",
+		[GFX10_V_LDEXP_F64] = "v_ldexp_f64",
+		[GFX10_V_MUL_LO_U32] = "v_mul_lo_u32",
+		[GFX10_V_MUL_HI_U32] = "v_mul_hi_u32",
+		[GFX10_V_MUL_HI_I32] = "v_mul_hi_i32",
+		[GFX10_V_DIV_FMAS_F32] = "v_div_fmas_f32",
+		[GFX10_V_DIV_FMAS_F64] = "v_div_fmas_f64",
+		[GFX10_V_MSAD_U8] = "v_msad_u8",
+		[GFX10_V_QSAD_PK_U16_U8] = "v_qsad_pk_u16_u8",
+		[GFX10_V_MQSAD_PK_U16_U8] = "v_mqsad_pk_u16_u8",
+		[GFX10_V_TRIG_PREOP_F64] = "v_trig_preop_f64",
+		[GFX10_V_MQSAD_U32_U8] = "v_mqsad_u32_u8",
+		[GFX10_V_XOR3_B32] = "v_xor3_b32",
+		[GFX10_V_LSHLREV_B64] = "v_lshlrev_b64",
+		[GFX10_V_LSHRREV_B64] = "v_lshrrev_b64",
+		[GFX10_V_ASHRREV_I64] = "v_ashrrev_i64",
+		[GFX10_V_ADD_NC_U16] = "v_add_nc_u16",
+		[GFX10_V_SUB_NC_U16] = "v_sub_nc_u16",
+		[GFX10_V_MUL_LO_U16] = "v_mul_lo_u16",
+		[GFX10_V_LSHRREV_B16] = "v_lshrrev_b16",
+		[GFX10_V_ASHRREV_I16] = "v_ashrrev_i16",
+		[GFX10_V_MAX_U16] = "v_max_u16",
+		[GFX10_V_MAX_I16] = "v_max_i16",
+		[GFX10_V_MIN_U16] = "v_min_u16",
+		[GFX10_V_MIN_I16] = "v_min_i16",
+		[GFX10_V_ADD_NC_I16] = "v_add_nc_i16",
+		[GFX10_V_SUB_NC_I16] = "v_sub_nc_i16",
+		[GFX10_V_PACK_B32_F16] = "v_pack_b32_f16",
+		[GFX10_V_CVT_PKNORM_I16_F16] = "v_cvt_pknorm_i16_f16",
+		[GFX10_V_CVT_PKNORM_U16_F16] = "v_cvt_pknorm_u16_f16",
+		[GFX10_V_LSHLREV_B16] = "v_lshlrev_b16",
+		[GFX10_V_MAD_U16] = "v_mad_u16",
+		[GFX10_V_INTERP_P1LL_F16] = "v_interp_p1ll_f16",
+		[GFX10_V_INTERP_P1LV_F16] = "v_interp_p1lv_f16",
+		[GFX10_V_PERM_B32] = "v_perm_b32",
+		[GFX10_V_XAD_U32] = "v_xad_u32",
+		[GFX10_V_LSHL_ADD_U32] = "v_lshl_add_u32",
+		[GFX10_V_ADD_LSHL_U32] = "v_add_lshl_u32",
+		[GFX10_V_FMA_F16] = "v_fma_f16",
+		[GFX10_V_MIN3_F16] = "v_min3_f16",
+		[GFX10_V_MIN3_I16] = "v_min3_i16",
+		[GFX10_V_MIN3_U16] = "v_min3_u16",
+		[GFX10_V_MAX3_F16] = "v_max3_f16",
+		[GFX10_V_MAX3_I16] = "v_max3_i16",
+		[GFX10_V_MAX3_U16] = "v_max3_u16",
+		[GFX10_V_MED3_F16] = "v_med3_f16",
+		[GFX10_V_MED3_I16] = "v_med3_i16",
+		[GFX10_V_MED3_U16] = "v_med3_u16",
+		[GFX10_V_INTERP_P2_F16] = "v_interp_p2_f16",
+		[GFX10_V_MAD_I16] = "v_mad_i16",
+		[GFX10_V_DIV_FIXUP_F16] = "v_div_fixup_f16",
+		[GFX10_V_READLANE_B32] = "v_readlane_b32",
+		[GFX10_V_WRITELANE_B32] = "v_writelane_b32",
+		[GFX10_V_LDEXP_F32] = "v_ldexp_f32",
+		[GFX10_V_BFM_B32] = "v_bfm_b32",
+		[GFX10_V_BCNT_U32_B32] = "v_bcnt_u32_b32",
+		[GFX10_V_MBCNT_LO_U32_B32] = "v_mbcnt_lo_u32_b32",
+		[GFX10_V_MBCNT_HI_U32_B32] = "v_mbcnt_hi_u32_b32",
+		[GFX10_V_CVT_PKNORM_I16_F32] = "v_cvt_pknorm_i16_f32",
+		[GFX10_V_CVT_PKNORM_U16_F32] = "v_cvt_pknorm_u16_f32",
+		[GFX10_V_CVT_PK_U16_U32] = "v_cvt_pk_u16_u32",
+		[GFX10_V_CVT_PK_I16_I32] = "v_cvt_pk_i16_i32",
+		[GFX10_V_ADD3_U32] = "v_add3_u32",
+		[GFX10_V_LSHL_OR_B32] = "v_lshl_or_b32",
+		[GFX10_V_AND_OR_B32] = "v_and_or_b32",
+		[GFX10_V_OR3_B32] = "v_or3_b32",
+		[GFX10_V_MAD_U32_U16] = "v_mad_u32_u16",
+		[GFX10_V_MAD_I32_I16] = "v_mad_i32_i16",
+		[GFX10_V_SUB_NC_I32] = "v_sub_nc_i32",
+		[GFX10_V_PERMLANE16_B32] = "v_permlane16_b32",
+		[GFX10_V_PERMLANEX16_B32] = "v_permlanex16_b32",
+		[GFX10_V_ADD_NC_I32] = "v_add_nc_i32",
+	},
+	[AMDGCN_INSN_TYPE_SOP2] = {
+		[GFX10_S_ADD_U32] = "s_add_u32",
+		[GFX10_S_SUB_U32] = "s_sub_u32",
+		[GFX10_S_ADD_I32] = "s_add_i32",
+		[GFX10_S_SUB_I32] = "s_sub_i32",
+		[GFX10_S_ADDC_U32] = "s_addc_u32",
+		[GFX10_S_SUBB_U32] = "s_subb_u32",
+		[GFX10_S_MIN_I32] = "s_min_i32",
+		[GFX10_S_MIN_U32] = "s_min_u32",
+		[GFX10_S_MAX_I32] = "s_max_i32",
+		[GFX10_S_MAX_U32] = "s_max_u32",
+		[GFX10_S_CSELECT_B32] = "s_cselect_b32",
+		[GFX10_S_CELECT_B64] = "s_celect_b64",
+		[GFX10_S_AND_B32] = "s_and_b32",
+		[GFX10_S_AND_B64] = "s_and_b64",
+		[GFX10_S_OR_B32] = "s_or_b32",
+		[GFX10_S_OR_B64] = "s_or_b64",
+		[GFX10_S_XOR_B32] = "s_xor_b32",
+		[GFX10_S_XOR_B64] = "s_xor_b64",
+		[GFX10_S_ANDN2_B32] = "s_andn2_b32",
+		[GFX10_S_ANDN2_B64] = "s_andn2_b64",
+		[GFX10_S_ORN2_B32] = "s_orn2_b32",
+		[GFX10_S_ORN2_B64] = "s_orn2_b64",
+		[GFX10_S_NAND_B32] = "s_nand_b32",
+		[GFX10_S_NAND_B64] = "s_nand_b64",
+		[GFX10_S_NOR_B32] = "s_nor_b32",
+		[GFX10_S_NOR_B64] = "s_nor_b64",
+		[GFX10_S_XNOR_B32] = "s_xnor_b32",
+		[GFX10_S_XNOR_B64] = "s_xnor_b64",
+		[GFX10_S_LSHL_B32] = "s_lshl_b32",
+		[GFX10_S_LSHL_B64] = "s_lshl_b64",
+		[GFX10_S_LSHR_B32] = "s_lshr_b32",
+		[GFX10_S_LSHR_B64] = "s_lshr_b64",
+		[GFX10_S_ASHR_I32] = "s_ashr_i32",
+		[GFX10_S_ASHR_I64] = "s_ashr_i64",
+		[GFX10_S_BFM_B32] = "s_bfm_b32",
+		[GFX10_S_BFM_B64] = "s_bfm_b64",
+		[GFX10_S_MUL_I32] = "s_mul_i32",
+		[GFX10_S_MUL_I64] = "s_mul_i64",
+		[GFX10_S_BFE_U32] = "s_bfe_u32",
+		[GFX10_S_BFE_I32] = "s_bfe_i32",
+		[GFX10_S_BFE_U64] = "s_bfe_u64",
+		[GFX10_S_ABSDIFF_I32] = "s_absdiff_i32",
+		[GFX10_S_LSHL1_ADD_U32] = "s_lshl1_add_u32",
+		[GFX10_S_LSHL2_ADD_U32] = "s_lshl2_add_u32",
+		[GFX10_S_LSHL3_ADD_U32] = "s_lshl3_add_u32",
+		[GFX10_S_LSHL4_ADD_U32] = "s_lshl4_add_u32",
+		[GFX10_S_PACK_LL_B32_B16] = "s_pack_ll_b32_b16",
+		[GFX10_S_PACK_LH_B32_B16] = "s_pack_lh_b32_b16",
+		[GFX10_S_MUL_HI_U32] = "s_mul_hi_u32",
+		[GFX10_S_MUL_HI_I32] = "s_mul_hi_i32",
+	},
+	[AMDGCN_INSN_TYPE_SOPK] = {
+		[GFX10_S_MOVK_I32] = "s_movk_i32",
+		[GFX10_S_VERSION] = "s_version",
+		[GFX10_S_CMOVK_I32] = "s_cmovk_i32",
+		[GFX10_S_CMPK_EQ_I32] = "s_cmpk_eq_i32",
+		[GFX10_S_CMPK_LG_I32] = "s_cmpk_lg_i32",
+		[GFX10_S_CMPK_GT_I32] = "s_cmpk_gt_i32",
+		[GFX10_S_CMPK_GE_I32] = "s_cmpk_ge_i32",
+		[GFX10_S_CMPK_LT_I32] = "s_cmpk_lt_i32",
+		[GFX10_S_CMPK_LE_I32] = "s_cmpk_le_i32",
+		[GFX10_S_CMPK_EQ_U32] = "s_cmpk_eq_u32",
+		[GFX10_S_CMPK_LG_U32] = "s_cmpk_lg_u32",
+		[GFX10_S_CMPK_GT_U32] = "s_cmpk_gt_u32",
+		[GFX10_S_CMPK_GE_U32] = "s_cmpk_ge_u32",
+		[GFX10_S_CMPK_LT_U32] = "s_cmpk_lt_u32",
+		[GFX10_S_CMPK_LE_U32] = "s_cmpk_le_u32",
+		[GFX10_S_ADDK_I32] = "s_addk_i32",
+		[GFX10_S_MULK_I32] = "s_mulk_i32",
+		[GFX10_S_GETREG_B32] = "s_getreg_b32",
+		[GFX10_S_SETREG_B32] = "s_setreg_b32",
+		[GFX10_S_SETREG_IMM32_B32] = "s_setreg_imm32_b32",
+		[GFX10_S_CALL_B64] = "s_call_b64",
+		[GFX10_S_WAITCNT_VSCNT] = "s_waitcnt_vscnt",
+		[GFX10_S_WAITCNT_VMCNT] = "s_waitcnt_vmcnt",
+		[GFX10_S_WAITCNT_EXPCNT] = "s_waitcnt_expcnt",
+		[GFX10_S_WAITCNT_LGKMCNT] = "s_waitcnt_lgkmcnt",
+		[GFX10_S_SUBVECTOR_LOOP_BEGIN] = "s_subvector_loop_begin",
+		[GFX10_S_SUBVECTOR_LOOP_END] = "s_subvector_loop_end",
+	},
+	[AMDGCN_INSN_TYPE_SOP1] = {
+		[GFX10_S_MOV_B32] = "s_mov_b32",
+		[GFX10_S_MOV_B64] = "s_mov_b64",
+		[GFX10_S_CMOV_B32] = "s_cmov_b32",
+		[GFX10_S_CMOV_B64] = "s_cmov_b64",
+		[GFX10_S_NOT_B32] = "s_not_b32",
+		[GFX10_S_NOT_B64] = "s_not_b64",
+		[GFX10_S_WQM_B32] = "s_wqm_b32",
+		[GFX10_S_WQM_B64] = "s_wqm_b64",
+		[GFX10_S_BREV_B32] = "s_brev_b32",
+		[GFX10_S_BREV_B64] = "s_brev_b64",
+		[GFX10_S_BCNT0_I32_B32] = "s_bcnt0_i32_b32",
+		[GFX10_S_BCNT0_I32_B64] = "s_bcnt0_i32_b64",
+		[GFX10_S_BCNT1_I32_B32] = "s_bcnt1_i32_b32",
+		[GFX10_S_BCNT1_I32_B64] = "s_bcnt1_i32_b64",
+		[GFX10_S_FF0_I32_B32] = "s_ff0_i32_b32",
+		[GFX10_S_FF0_I32_B64] = "s_ff0_i32_b64",
+		[GFX10_S_FF1_I32_B32] = "s_ff1_i32_b32",
+		[GFX10_S_FF1_I32_B64] = "s_ff1_i32_b64",
+		[GFX10_S_FLBIT_I32_B32] = "s_flbit_i32_b32",
+		[GFX10_S_FLBIT_I32_B64] = "s_flbit_i32_b64",
+		[GFX10_S_FLBIT_I32] = "s_flbit_i32",
+		[GFX10_S_FLBIT_I32_I64] = "s_flbit_i32_i64",
+		[GFX10_S_SEXT_I32_I8] = "s_sext_i32_i8",
+		[GFX10_S_SEXT_I32_I16] = "s_sext_i32_i16",
+		[GFX10_S_BITSET0_B32] = "s_bitset0_b32",
+		[GFX10_S_BITSET0_B64] = "s_bitset0_b64",
+		[GFX10_S_BITSET1_B32] = "s_bitset1_b32",
+		[GFX10_S_BITSET1_B64] = "s_bitset1_b64",
+		[GFX10_S_GETPC_B64] = "s_getpc_b64",
+		[GFX10_S_SETPC_B64] = "s_setpc_b64",
+		[GFX10_S_SWAPPC_B64] = "s_swappc_b64",
+		[GFX10_S_RFE_B64] = "s_rfe_b64",
+		[GFX10_S_AND_SAVEEXEC_B64] = "s_and_saveexec_b64",
+		[GFX10_S_XNOR_SAVEEXEC_B64] = "s_xnor_saveexec_b64",
+		[GFX10_S_QUADMASK_B32] = "s_quadmask_b32",
+		[GFX10_S_QUADMASK_B64] = "s_quadmask_b64",
+		[GFX10_S_MOVRELS_B32] = "s_movrels_b32",
+		[GFX10_S_MOVRELS_B64] = "s_movrels_b64",
+		[GFX10_S_MOVRELD_B32] = "s_movreld_b32",
+		[GFX10_S_MOVRELD_B64] = "s_movreld_b64",
+		[GFX10_S_ABS_I32] = "s_abs_i32",
+		[GFX10_S_ANDN1_SAVEEXEC_B64] = "s_andn1_saveexec_b64",
+		[GFX10_S_ORN1_SAVEEXEC_B64] = "s_orn1_saveexec_b64",
+		[GFX10_S_ANDN1_WREXEC_B64] = "s_andn1_wrexec_b64",
+		[GFX10_S_ANDN2_WREXEC_B64] = "s_andn2_wrexec_b64",
+		[GFX10_S_BITREPLICATE_B64_B32] = "s_bitreplicate_b64_b32",
+		[GFX10_S_AND_SAVEEXEC_B32] = "s_and_saveexec_b32",
+		[GFX10_S_OR_SAVEEXEC_B32] = "s_or_saveexec_b32",
+		[GFX10_S_XOR_SAVEEXEC_B32] = "s_xor_saveexec_b32",
+		[GFX10_S_ANDN2_SAVEEXEC_B32] = "s_andn2_saveexec_b32",
+		[GFX10_S_ORN2_SAVEEXEC_B32] = "s_orn2_saveexec_b32",
+		[GFX10_S_NAND_SAVEEXEC_B32] = "s_nand_saveexec_b32",
+		[GFX10_S_NOR_SAVEEXEC_B32] = "s_nor_saveexec_b32",
+		[GFX10_S_XNOR_SAVEEXEC_B32] = "s_xnor_saveexec_b32",
+		[GFX10_S_ANDN1_SAVEEXEC_B32] = "s_andn1_saveexec_b32",
+		[GFX10_S_ORN1_SAVEEXEC_B32] = "s_orn1_saveexec_b32",
+		[GFX10_S_ANDN1_WREXEC_B32] = "s_andn1_wrexec_b32",
+		[GFX10_S_ANDN2_WREXEC_B32] = "s_andn2_wrexec_b32",
+		[GFX10_S_MOVRELSD_2_B32] = "s_movrelsd_2_b32",
+	},
+	[AMDGCN_INSN_TYPE_SOPC] = {
+		[GFX10_S_CMP_EQ_I32] = "s_cmp_eq_i32",
+		[GFX10_S_CMP_LG_I32] = "s_cmp_lg_i32",
+		[GFX10_S_CMP_GT_I32] = "s_cmp_gt_i32",
+		[GFX10_S_CMP_GE_I32] = "s_cmp_ge_i32",
+		[GFX10_S_CMP_LT_I32] = "s_cmp_lt_i32",
+		[GFX10_S_CMP_LE_I32] = "s_cmp_le_i32",
+		[GFX10_S_CMP_EQ_U32] = "s_cmp_eq_u32",
+		[GFX10_S_CMP_LG_U32] = "s_cmp_lg_u32",
+		[GFX10_S_CMP_GT_U32] = "s_cmp_gt_u32",
+		[GFX10_S_CMP_GE_U32] = "s_cmp_ge_u32",
+		[GFX10_S_CMP_LT_U32] = "s_cmp_lt_u32",
+		[GFX10_S_CMP_LE_U32] = "s_cmp_le_u32",
+		[GFX10_S_BITCMP0_B32] = "s_bitcmp0_b32",
+		[GFX10_S_BITCMP1_B32] = "s_bitcmp1_b32",
+		[GFX10_S_BITCMP0_B64] = "s_bitcmp0_b64",
+		[GFX10_S_BITCMP1_B64] = "s_bitcmp1_b64",
+		[GFX10_S_CMP_EQ_U64] = "s_cmp_eq_u64",
+		[GFX10_S_CMP_LG_U64] = "s_cmp_lg_u64",
+	},
+	[AMDGCN_INSN_TYPE_SOPP] = {
+		[GFX10_S_NOP] = "s_nop",
+		[GFX10_S_ENDPGM] = "s_endpgm",
+		[GFX10_S_BRANCH] = "s_branch",
+		[GFX10_S_WAKEUP] = "s_wakeup",
+		[GFX10_S_CBRANCH_SCC0] = "s_cbranch_scc0",
+		[GFX10_S_CBRANCH_SCC1] = "s_cbranch_scc1",
+		[GFX10_S_CBRANCH_VCCZ] = "s_cbranch_vccz",
+		[GFX10_S_CBRANCH_VCCNZ] = "s_cbranch_vccnz",
+		[GFX10_S_CBRANCH_EXECZ] = "s_cbranch_execz",
+		[GFX10_S_CBRANCH_EXECNZ] = "s_cbranch_execnz",
+		[GFX10_S_BARRIER] = "s_barrier",
+		[GFX10_S_SETKILL] = "s_setkill",
+		[GFX10_S_WAITCNT] = "s_waitcnt",
+		[GFX10_S_SETHALT] = "s_sethalt",
+		[GFX10_S_SLEEP] = "s_sleep",
+		[GFX10_S_SETPRIO] = "s_setprio",
+		[GFX10_S_SENDMSG] = "s_sendmsg",
+		[GFX10_S_SENDMSGHALT] = "s_sendmsghalt",
+		[GFX10_S_TRAP] = "s_trap",
+		[GFX10_S_ICACHE_INV] = "s_icache_inv",
+		[GFX10_S_INCPERFLEVEL] = "s_incperflevel",
+		[GFX10_S_DECPERFLEVEL] = "s_decperflevel",
+		[GFX10_S_TTRACEDATA] = "s_ttracedata",
+		[GFX10_S_CBRANCH_CDBGSYS] = "s_cbranch_cdbgsys",
+		[GFX10_S_CBRANCH_CDBGUSER] = "s_cbranch_cdbguser",
+		[GFX10_S_CBRANCH_CDBGSYS_OR_USER] = "s_cbranch_cdbgsys_or_user",
+		[GFX10_S_CBRANCH_CDBGSYS_AND_USER] = "s_cbranch_cdbgsys_and_user",
+		[GFX10_S_ENDPGM_SAVED] = "s_endpgm_saved",
+		[GFX10_S_ENDPGM_ORDERED_PS_DONE] = "s_endpgm_ordered_ps_done",
+		[GFX10_S_CODE_END] = "s_code_end",
+		[GFX10_S_INST_PREFETCH] = "s_inst_prefetch",
+		[GFX10_S_CLAUSE] = "s_clause",
+		[GFX10_S_WAITCNT_DEPCTR] = "s_waitcnt_depctr",
+		[GFX10_S_ROUND_MODE] = "s_round_mode",
+		[GFX10_S_DENORM_MODE] = "s_denorm_mode",
+		[GFX10_S_TTRACEDATA_IMM] = "s_ttracedata_imm",
+	},
+	[AMDGCN_INSN_TYPE_SMEM] = {
+		[GFX10_S_LOAD_DWORD] = "s_load_dword",
+		[GFX10_S_LOAD_DWORDX2] = "s_load_dwordx2",
+		[GFX10_S_LOAD_DWORDX4] = "s_load_dwordx4",
+		[GFX10_S_LOAD_DWORDX8] = "s_load_dwordx8",
+		[GFX10_S_LOAD_DWORDX16] = "s_load_dwordx16",
+		[GFX10_S_BUFFER_LOAD_DWORD] = "s_buffer_load_dword",
+		[GFX10_S_BUFFER_LOAD_DWORDX2] = "s_buffer_load_dwordx2",
+		[GFX10_S_BUFFER_LOAD_DWORDX4] = "s_buffer_load_dwordx4",
+		[GFX10_S_BUFFER_LOAD_DWORDX8] = "s_buffer_load_dwordx8",
+		[GFX10_S_BUFFER_LOAD_DWORDX16] = "s_buffer_load_dwordx16",
+		[GFX10_S_GL1_INV] = "s_gl1_inv",
+		[GFX10_S_DCACHE_INV] = "s_dcache_inv",
+		[GFX10_S_MEMTIME] = "s_memtime",
+		[GFX10_S_MEMREALTIME] = "s_memrealtime",
+		[GFX10_S_ATC_PROBE] = "s_atc_probe",
+		[GFX10_S_ATC_PROBE_BUFFER] = "s_atc_probe_buffer",
+	},
+	[AMDGCN_INSN_TYPE_VOP2] = {
+		[GFX10_V_CNDMASK_B32] = "v_cndmask_b32",
+		[GFX10_V_DOT2C_F32_F16] = "v_dot2c_f32_f16",
+		[GFX10_V_ADD_F32] = "v_add_f32",
+		[GFX10_V_SUB_F32] = "v_sub_f32",
+		[GFX10_V_SUBREV_F32] = "v_subrev_f32",
+		[GFX10_V_FMAC_LEGACY_F32] = "v_fmac_legacy_f32",
+		[GFX10_V_MUL_LEGACY_F32] = "v_mul_legacy_f32",
+		[GFX10_V_MUL_F32] = "v_mul_f32",
+		[GFX10_V_MUL_I32_I24] = "v_mul_i32_i24",
+		[GFX10_V_MUL_HI_I32_I24] = "v_mul_hi_i32_i24",
+		[GFX10_V_MUL_U32_U24] = "v_mul_u32_u24",
+		[GFX10_V_MUL_HI_U32_U24] = "v_mul_hi_u32_u24",
+		[GFX10_V_DOT4C_I32_I8] = "v_dot4c_i32_i8",
+		[GFX10_V_MIN_F32] = "v_min_f32",
+		[GFX10_V_MAX_F32] = "v_max_f32",
+		[GFX10_V_MIN_I32] = "v_min_i32",
+		[GFX10_V_MAX_I32] = "v_max_i32",
+		[GFX10_V_MIN_U32] = "v_min_u32",
+		[GFX10_V_MAX_U32] = "v_max_u32",
+		[GFX10_V_LSHRREV_B32] = "v_lshrrev_b32",
+		[GFX10_V_ASHRREV_I32] = "v_ashrrev_i32",
+		[GFX10_V_LSHLREV_B32] = "v_lshlrev_b32",
+		[GFX10_V_AND_B32] = "v_and_b32",
+		[GFX10_V_OR_B32] = "v_or_b32",
+		[GFX10_V_XOR_B32] = "v_xor_b32",
+		[GFX10_V_XNOR_B32] = "v_xnor_b32",
+		[GFX10_V_ADD_NC_U32] = "v_add_nc_u32",
+		[GFX10_V_SUB_NC_U32] = "v_sub_nc_u32",
+		[GFX10_V_SUBREV_NC_U32] = "v_subrev_nc_u32",
+		[GFX10_V_ADD_CO_CI_U32] = "v_add_co_ci_u32",
+		[GFX10_V_SUB_CO_CI_U32] = "v_sub_co_ci_u32",
+		[GFX10_V_SUBREV_CO_CI_U32] = "v_subrev_co_ci_u32",
+		[GFX10_V_FMAC_F32] = "v_fmac_f32",
+		[GFX10_V_FMAMK_F32] = "v_fmamk_f32",
+		[GFX10_V_FMAAK_F32] = "v_fmaak_f32",
+		[GFX10_V_CVT_PKRTZ_F16_F32] = "v_cvt_pkrtz_f16_f32",
+		[GFX10_V_ADD_F16] = "v_add_f16",
+		[GFX10_V_SUB_F16] = "v_sub_f16",
+		[GFX10_V_SUBREV_F16] = "v_subrev_f16",
+		[GFX10_V_MUL_F16] = "v_mul_f16",
+		[GFX10_V_FMAC_F16] = "v_fmac_f16",
+		[GFX10_V_FMAMK_F16] = "v_fmamk_f16",
+		[GFX10_V_FMAAK_F16] = "v_fmaak_f16",
+		[GFX10_V_MAX_F16] = "v_max_f16",
+		[GFX10_V_MIN_F16] = "v_min_f16",
+		[GFX10_V_LDEXP_F16] = "v_ldexp_f16",
+		[GFX10_V_PK_FMAC_F16] = "v_pk_fmac_f16",
+	},
+	[AMDGCN_INSN_TYPE_VOP1] = {
+		[GFX10_V_NOP] = "v_nop",
+		[GFX10_V_MOV_B32] = "v_mov_b32",
+		[GFX10_V_READFIRSTLANE_B32] = "v_readfirstlane_b32",
+		[GFX10_V_CVT_I32_F64] = "v_cvt_i32_f64",
+		[GFX10_V_CVT_F64_I32] = "v_cvt_f64_i32",
+		[GFX10_V_CVT_F32_I32] = "v_cvt_f32_i32",
+		[GFX10_V_CVT_F32_U32] = "v_cvt_f32_u32",
+		[GFX10_V_CVT_U32_F32] = "v_cvt_u32_f32",
+		[GFX10_V_CVT_I32_F32] = "v_cvt_i32_f32",
+		[GFX10_V_CVT_F16_F32] = "v_cvt_f16_f32",
+		[GFX10_V_CVT_F32_F16] = "v_cvt_f32_f16",
+		[GFX10_V_CVT_RPI_I32_F32] = "v_cvt_rpi_i32_f32",
+		[GFX10_V_CVT_FLR_I32_F32] = "v_cvt_flr_i32_f32",
+		[GFX10_V_CVT_OFF_F32_I4] = "v_cvt_off_f32_i4",
+		[GFX10_V_CVT_F32_F64] = "v_cvt_f32_f64",
+		[GFX10_V_CVT_F64_F32] = "v_cvt_f64_f32",
+		[GFX10_V_CVT_F32_UBYTE0] = "v_cvt_f32_ubyte0",
+		[GFX10_V_CVT_F32_UBYTE1] = "v_cvt_f32_ubyte1",
+		[GFX10_V_CVT_F32_UBYTE2] = "v_cvt_f32_ubyte2",
+		[GFX10_V_CVT_F32_UBYTE3] = "v_cvt_f32_ubyte3",
+		[GFX10_V_CVT_U32_F64] = "v_cvt_u32_f64",
+		[GFX10_V_CVT_F64_U32] = "v_cvt_f64_u32",
+		[GFX10_V_TRUNC_F64] = "v_trunc_f64",
+		[GFX10_V_CEIL_F64] = "v_ceil_f64",
+		[GFX10_V_RNDNE_F64] = "v_rndne_f64",
+		[GFX10_V_FLOOR_F64] = "v_floor_f64",
+		[GFX10_V_PIPEFLUSH] = "v_pipeflush",
+		[GFX10_V_FRACT_F32] = "v_fract_f32",
+		[GFX10_V_TRUNC_F32] = "v_trunc_f32",
+		[GFX10_V_CEIL_F32] = "v_ceil_f32",
+		[GFX10_V_RNDNE_F32] = "v_rndne_f32",
+		[GFX10_V_FLOOR_F32] = "v_floor_f32",
+		[GFX10_V_EXP_F32] = "v_exp_f32",
+		[GFX10_V_LOG_F32] = "v_log_f32",
+		[GFX10_V_RCP_F32] = "v_rcp_f32",
+		[GFX10_V_RCP_IFLAG_F32] = "v_rcp_iflag_f32",
+		[GFX10_V_RSQ_F32] = "v_rsq_f32",
+		[GFX10_V_RCP_F64] = "v_rcp_f64",
+		[GFX10_V_RSQ_F64] = "v_rsq_f64",
+		[GFX10_V_SQRT_F32] = "v_sqrt_f32",
+		[GFX10_V_SQRT_F64] = "v_sqrt_f64",
+		[GFX10_V_SIN_F32] = "v_sin_f32",
+		[GFX10_V_COS_F32] = "v_cos_f32",
+		[GFX10_V_NOT_B32] = "v_not_b32",
+		[GFX10_V_BFREV_B32] = "v_bfrev_b32",
+		[GFX10_V_FFBH_U32] = "v_ffbh_u32",
+		[GFX10_V_FFBL_B32] = "v_ffbl_b32",
+		[GFX10_V_FFBH_I32] = "v_ffbh_i32",
+		[GFX10_V_FREXP_EXP_I32_F64] = "v_frexp_exp_i32_f64",
+		[GFX10_V_FREXP_MANT_F64] = "v_frexp_mant_f64",
+		[GFX10_V_FRACT_F64] = "v_fract_f64",
+		[GFX10_V_FREXP_EXP_I32_F32] = "v_frexp_exp_i32_f32",
+		[GFX10_V_FREXP_MANT_F32] = "v_frexp_mant_f32",
+		[GFX10_V_CLREXCP] = "v_clrexcp",
+		[GFX10_V_MOVRELD_B32] = "v_movreld_b32",
+		[GFX10_V_MOVRELS_B32] = "v_movrels_b32",
+		[GFX10_V_MOVRELSD_B32] = "v_movrelsd_b32",
+		[GFX10_V_MOVRELSD_2_B32] = "v_movrelsd_2_b32",
+		[GFX10_V_CVT_F16_U16] = "v_cvt_f16_u16",
+		[GFX10_V_CVT_F16_I16] = "v_cvt_f16_i16",
+		[GFX10_V_CVT_U16_F16] = "v_cvt_u16_f16",
+		[GFX10_V_CVT_I16_F16] = "v_cvt_i16_f16",
+		[GFX10_V_RCP_F16] = "v_rcp_f16",
+		[GFX10_V_SQRT_F16] = "v_sqrt_f16",
+		[GFX10_V_RSQ_F16] = "v_rsq_f16",
+		[GFX10_V_LOG_F16] = "v_log_f16",
+		[GFX10_V_EXP_F16] = "v_exp_f16",
+		[GFX10_V_FREXP_MANT_F16] = "v_frexp_mant_f16",
+		[GFX10_V_FREXP_EXP_I16_F16] = "v_frexp_exp_i16_f16",
+		[GFX10_V_FLOOR_F16] = "v_floor_f16",
+		[GFX10_V_CEIL_F16] = "v_ceil_f16",
+		[GFX10_V_TRUNC_F16] = "v_trunc_f16",
+		[GFX10_V_RNDNE_F16] = "v_rndne_f16",
+		[GFX10_V_FRACT_F16] = "v_fract_f16",
+		[GFX10_V_SIN_F16] = "v_sin_f16",
+		[GFX10_V_COS_F16] = "v_cos_f16",
+		[GFX10_V_SAT_PK_U8_I16] = "v_sat_pk_u8_i16",
+		[GFX10_V_CVT_NORM_I16_F16] = "v_cvt_norm_i16_f16",
+		[GFX10_V_CVT_NORM_U16_F16] = "v_cvt_norm_u16_f16",
+		[GFX10_V_SWAP_B32] = "v_swap_b32",
+		[GFX10_V_SWAPREL_B32] = "v_swaprel_b32",
+	},
+	[AMDGCN_INSN_TYPE_VOPC] = {
+		[GFX10_V_CMP_F_F32] = "v_cmp_f_f32",
+		[GFX10_V_CMP_LT_F32] = "v_cmp_lt_f32",
+		[GFX10_V_CMP_EQ_F32] = "v_cmp_eq_f32",
+		[GFX10_V_CMP_LE_F32] = "v_cmp_le_f32",
+		[GFX10_V_CMP_GT_F32] = "v_cmp_gt_f32",
+		[GFX10_V_CMP_LG_F32] = "v_cmp_lg_f32",
+		[GFX10_V_CMP_GE_F32] = "v_cmp_ge_f32",
+		[GFX10_V_CMP_O_F32] = "v_cmp_o_f32",
+		[GFX10_V_CMP_U_F32] = "v_cmp_u_f32",
+		[GFX10_V_CMP_NGE_F32] = "v_cmp_nge_f32",
+		[GFX10_V_CMP_NLG_F32] = "v_cmp_nlg_f32",
+		[GFX10_V_CMP_NGT_F32] = "v_cmp_ngt_f32",
+		[GFX10_V_CMP_NLE_F32] = "v_cmp_nle_f32",
+		[GFX10_V_CMP_NEQ_F32] = "v_cmp_neq_f32",
+		[GFX10_V_CMP_NLT_F32] = "v_cmp_nlt_f32",
+		[GFX10_V_CMP_TRU_F32] = "v_cmp_tru_f32",
+		[GFX10_V_CMPX_F_F32] = "v_cmpx_f_f32",
+		[GFX10_V_CMPX_LT_F32] = "v_cmpx_lt_f32",
+		[GFX10_V_CMPX_EQ_F32] = "v_cmpx_eq_f32",
+		[GFX10_V_CMPX_LE_F32] = "v_cmpx_le_f32",
+		[GFX10_V_CMPX_GT_F32] = "v_cmpx_gt_f32",
+		[GFX10_V_CMPX_LG_F32] = "v_cmpx_lg_f32",
+		[GFX10_V_CMPX_GE_F32] = "v_cmpx_ge_f32",
+		[GFX10_V_CMPX_O_F32] = "v_cmpx_o_f32",
+		[GFX10_V_CMPX_U_F32] = "v_cmpx_u_f32",
+		[GFX10_V_CMPX_NGE_F32] = "v_cmpx_nge_f32",
+		[GFX10_V_CMPX_NLG_F32] = "v_cmpx_nlg_f32",
+		[GFX10_V_CMPX_NGT_F32] = "v_cmpx_ngt_f32",
+		[GFX10_V_CMPX_NLE_F32] = "v_cmpx_nle_f32",
+		[GFX10_V_CMPX_NEQ_F32] = "v_cmpx_neq_f32",
+		[GFX10_V_CMPX_NLT_F32] = "v_cmpx_nlt_f32",
+		[GFX10_V_CMPX_TRU_F32] = "v_cmpx_tru_f32",
+		[GFX10_V_CMP_F_F64] = "v_cmp_f_f64",
+		[GFX10_V_CMP_LT_F64] = "v_cmp_lt_f64",
+		[GFX10_V_CMP_EQ_F64] = "v_cmp_eq_f64",
+		[GFX10_V_CMP_LE_F64] = "v_cmp_le_f64",
+		[GFX10_V_CMP_GT_F64] = "v_cmp_gt_f64",
+		[GFX10_V_CMP_LG_F64] = "v_cmp_lg_f64",
+		[GFX10_V_CMP_GE_F64] = "v_cmp_ge_f64",
+		[GFX10_V_CMP_O_F64] = "v_cmp_o_f64",
+		[GFX10_V_CMP_U_F64] = "v_cmp_u_f64",
+		[GFX10_V_CMP_NGE_F64] = "v_cmp_nge_f64",
+		[GFX10_V_CMP_NLG_F64] = "v_cmp_nlg_f64",
+		[GFX10_V_CMP_NGT_F64] = "v_cmp_ngt_f64",
+		[GFX10_V_CMP_NLE_F64] = "v_cmp_nle_f64",
+		[GFX10_V_CMP_NEQ_F64] = "v_cmp_neq_f64",
+		[GFX10_V_CMP_NLT_F64] = "v_cmp_nlt_f64",
+		[GFX10_V_CMP_TRU_F64] = "v_cmp_tru_f64",
+		[GFX10_V_CMPX_F_F64] = "v_cmpx_f_f64",
+		[GFX10_V_CMPX_LT_F64] = "v_cmpx_lt_f64",
+		[GFX10_V_CMPX_EQ_F64] = "v_cmpx_eq_f64",
+		[GFX10_V_CMPX_LE_F64] = "v_cmpx_le_f64",
+		[GFX10_V_CMPX_GT_F64] = "v_cmpx_gt_f64",
+		[GFX10_V_CMPX_LG_F64] = "v_cmpx_lg_f64",
+		[GFX10_V_CMPX_GE_F64] = "v_cmpx_ge_f64",
+		[GFX10_V_CMPX_O_F64] = "v_cmpx_o_f64",
+		[GFX10_V_CMPX_U_F64] = "v_cmpx_u_f64",
+		[GFX10_V_CMPX_NGE_F64] = "v_cmpx_nge_f64",
+		[GFX10_V_CMPX_NLG_F64] = "v_cmpx_nlg_f64",
+		[GFX10_V_CMPX_NGT_F64] = "v_cmpx_ngt_f64",
+		[GFX10_V_CMPX_NLE_F64] = "v_cmpx_nle_f64",
+		[GFX10_V_CMPX_NEQ_F64] = "v_cmpx_neq_f64",
+		[GFX10_V_CMPX_NLT_F64] = "v_cmpx_nlt_f64",
+		[GFX10_V_CMPX_TRU_F64] = "v_cmpx_tru_f64",
+		[GFX10_V_CMP_F_I32] = "v_cmp_f_i32",
+		[GFX10_V_CMP_LT_I32] = "v_cmp_lt_i32",
+		[GFX10_V_CMP_EQ_I32] = "v_cmp_eq_i32",
+		[GFX10_V_CMP_LE_I32] = "v_cmp_le_i32",
+		[GFX10_V_CMP_GT_I32] = "v_cmp_gt_i32",
+		[GFX10_V_CMP_NE_I32] = "v_cmp_ne_i32",
+		[GFX10_V_CMP_GE_I32] = "v_cmp_ge_i32",
+		[GFX10_V_CMP_T_I32] = "v_cmp_t_i32",
+		[GFX10_V_CMP_CLASS_F32] = "v_cmp_class_f32",
+		[GFX10_V_CMP_LT_I16] = "v_cmp_lt_i16",
+		[GFX10_V_CMP_EQ_I16] = "v_cmp_eq_i16",
+		[GFX10_V_CMP_LE_I16] = "v_cmp_le_i16",
+		[GFX10_V_CMP_GT_I16] = "v_cmp_gt_i16",
+		[GFX10_V_CMP_NE_I16] = "v_cmp_ne_i16",
+		[GFX10_V_CMP_GE_I16] = "v_cmp_ge_i16",
+		[GFX10_V_CMP_CLASS_F16] = "v_cmp_class_f16",
+		[GFX10_V_CMPX_F_I32] = "v_cmpx_f_i32",
+		[GFX10_V_CMPX_LT_I32] = "v_cmpx_lt_i32",
+		[GFX10_V_CMPX_EQ_I32] = "v_cmpx_eq_i32",
+		[GFX10_V_CMPX_LE_I32] = "v_cmpx_le_i32",
+		[GFX10_V_CMPX_GT_I32] = "v_cmpx_gt_i32",
+		[GFX10_V_CMPX_NE_I32] = "v_cmpx_ne_i32",
+		[GFX10_V_CMPX_GE_I32] = "v_cmpx_ge_i32",
+		[GFX10_V_CMPX_T_I32] = "v_cmpx_t_i32",
+		[GFX10_V_CMPX_CLASS_F32] = "v_cmpx_class_f32",
+		[GFX10_V_CMPX_LT_I16] = "v_cmpx_lt_i16",
+		[GFX10_V_CMPX_EQ_I16] = "v_cmpx_eq_i16",
+		[GFX10_V_CMPX_LE_I16] = "v_cmpx_le_i16",
+		[GFX10_V_CMPX_GT_I16] = "v_cmpx_gt_i16",
+		[GFX10_V_CMPX_NE_I16] = "v_cmpx_ne_i16",
+		[GFX10_V_CMPX_GE_I16] = "v_cmpx_ge_i16",
+		[GFX10_V_CMPX_CLASS_F16] = "v_cmpx_class_f16",
+		[GFX10_V_CMP_F_I64] = "v_cmp_f_i64",
+		[GFX10_V_CMP_LT_I64] = "v_cmp_lt_i64",
+		[GFX10_V_CMP_EQ_I64] = "v_cmp_eq_i64",
+		[GFX10_V_CMP_LE_I64] = "v_cmp_le_i64",
+		[GFX10_V_CMP_GT_I64] = "v_cmp_gt_i64",
+		[GFX10_V_CMP_NE_I64] = "v_cmp_ne_i64",
+		[GFX10_V_CMP_GE_I64] = "v_cmp_ge_i64",
+		[GFX10_V_CMP_T_I64] = "v_cmp_t_i64",
+		[GFX10_V_CMP_CLASS_F64] = "v_cmp_class_f64",
+		[GFX10_V_CMP_LT_U16] = "v_cmp_lt_u16",
+		[GFX10_V_CMP_EQ_U16] = "v_cmp_eq_u16",
+		[GFX10_V_CMP_LE_U16] = "v_cmp_le_u16",
+		[GFX10_V_CMP_GT_U16] = "v_cmp_gt_u16",
+		[GFX10_V_CMP_NE_U16] = "v_cmp_ne_u16",
+		[GFX10_V_CMP_GE_U16] = "v_cmp_ge_u16",
+		[GFX10_V_CMPX_F_I64] = "v_cmpx_f_i64",
+		[GFX10_V_CMPX_LT_I64] = "v_cmpx_lt_i64",
+		[GFX10_V_CMPX_EQ_I64] = "v_cmpx_eq_i64",
+		[GFX10_V_CMPX_LE_I64] = "v_cmpx_le_i64",
+		[GFX10_V_CMPX_GT_I64] = "v_cmpx_gt_i64",
+		[GFX10_V_CMPX_NE_I64] = "v_cmpx_ne_i64",
+		[GFX10_V_CMPX_GE_I64] = "v_cmpx_ge_i64",
+		[GFX10_V_CMPX_T_I64] = "v_cmpx_t_i64",
+		[GFX10_V_CMPX_CLASS_F64] = "v_cmpx_class_f64",
+		[GFX10_V_CMPX_LT_U16] = "v_cmpx_lt_u16",
+		[GFX10_V_CMPX_EQ_U16] = "v_cmpx_eq_u16",
+		[GFX10_V_CMPX_LE_U16] = "v_cmpx_le_u16",
+		[GFX10_V_CMPX_GT_U16] = "v_cmpx_gt_u16",
+		[GFX10_V_CMPX_NE_U16] = "v_cmpx_ne_u16",
+		[GFX10_V_CMPX_GE_U16] = "v_cmpx_ge_u16",
+		[GFX10_V_CMP_F_U32] = "v_cmp_f_u32",
+		[GFX10_V_CMP_LT_U32] = "v_cmp_lt_u32",
+		[GFX10_V_CMP_EQ_U32] = "v_cmp_eq_u32",
+		[GFX10_V_CMP_LE_U32] = "v_cmp_le_u32",
+		[GFX10_V_CMP_GT_U32] = "v_cmp_gt_u32",
+		[GFX10_V_CMP_NE_U32] = "v_cmp_ne_u32",
+		[GFX10_V_CMP_GE_U32] = "v_cmp_ge_u32",
+		[GFX10_V_CMP_T_U32] = "v_cmp_t_u32",
+		[GFX10_V_CMP_F_F16] = "v_cmp_f_f16",
+		[GFX10_V_CMP_LT_F16] = "v_cmp_lt_f16",
+		[GFX10_V_CMP_EQ_F16] = "v_cmp_eq_f16",
+		[GFX10_V_CMP_LE_F16] = "v_cmp_le_f16",
+		[GFX10_V_CMP_GT_F16] = "v_cmp_gt_f16",
+		[GFX10_V_CMP_LG_F16] = "v_cmp_lg_f16",
+		[GFX10_V_CMP_GE_F16] = "v_cmp_ge_f16",
+		[GFX10_V_CMP_O_F16] = "v_cmp_o_f16",
+		[GFX10_V_CMPX_F_U32] = "v_cmpx_f_u32",
+		[GFX10_V_CMPX_LT_U32] = "v_cmpx_lt_u32",
+		[GFX10_V_CMPX_EQ_U32] = "v_cmpx_eq_u32",
+		[GFX10_V_CMPX_LE_U32] = "v_cmpx_le_u32",
+		[GFX10_V_CMPX_GT_U32] = "v_cmpx_gt_u32",
+		[GFX10_V_CMPX_NE_U32] = "v_cmpx_ne_u32",
+		[GFX10_V_CMPX_GE_U32] = "v_cmpx_ge_u32",
+		[GFX10_V_CMPX_T_U32] = "v_cmpx_t_u32",
+		[GFX10_V_CMPX_F_F16] = "v_cmpx_f_f16",
+		[GFX10_V_CMPX_LT_F16] = "v_cmpx_lt_f16",
+		[GFX10_V_CMPX_EQ_F16] = "v_cmpx_eq_f16",
+		[GFX10_V_CMPX_LE_F16] = "v_cmpx_le_f16",
+		[GFX10_V_CMPX_GT_F16] = "v_cmpx_gt_f16",
+		[GFX10_V_CMPX_LG_F16] = "v_cmpx_lg_f16",
+		[GFX10_V_CMPX_GE_F16] = "v_cmpx_ge_f16",
+		[GFX10_V_CMPX_O_F16] = "v_cmpx_o_f16",
+		[GFX10_V_CMP_F_U64] = "v_cmp_f_u64",
+		[GFX10_V_CMP_LT_U64] = "v_cmp_lt_u64",
+		[GFX10_V_CMP_EQ_U64] = "v_cmp_eq_u64",
+		[GFX10_V_CMP_LE_U64] = "v_cmp_le_u64",
+		[GFX10_V_CMP_GT_U64] = "v_cmp_gt_u64",
+		[GFX10_V_CMP_NE_U64] = "v_cmp_ne_u64",
+		[GFX10_V_CMP_GE_U64] = "v_cmp_ge_u64",
+		[GFX10_V_CMP_T_U64] = "v_cmp_t_u64",
+		[GFX10_V_CMP_U_F16] = "v_cmp_u_f16",
+		[GFX10_V_CMP_NGE_F16] = "v_cmp_nge_f16",
+		[GFX10_V_CMP_NLG_F16] = "v_cmp_nlg_f16",
+		[GFX10_V_CMP_NGT_F16] = "v_cmp_ngt_f16",
+		[GFX10_V_CMP_NLE_F16] = "v_cmp_nle_f16",
+		[GFX10_V_CMP_NEQ_F16] = "v_cmp_neq_f16",
+		[GFX10_V_CMP_NLT_F16] = "v_cmp_nlt_f16",
+		[GFX10_V_CMP_TRU_F16] = "v_cmp_tru_f16",
+		[GFX10_V_CMPX_F_U64] = "v_cmpx_f_u64",
+		[GFX10_V_CMPX_LT_U64] = "v_cmpx_lt_u64",
+		[GFX10_V_CMPX_EQ_U64] = "v_cmpx_eq_u64",
+		[GFX10_V_CMPX_LE_U64] = "v_cmpx_le_u64",
+		[GFX10_V_CMPX_GT_U64] = "v_cmpx_gt_u64",
+		[GFX10_V_CMPX_NE_U64] = "v_cmpx_ne_u64",
+		[GFX10_V_CMPX_GE_U64] = "v_cmpx_ge_u64",
+		[GFX10_V_CMPX_T_U64] = "v_cmpx_t_u64",
+		[GFX10_V_CMPX_U_F16] = "v_cmpx_u_f16",
+		[GFX10_V_CMPX_NGE_F16] = "v_cmpx_nge_f16",
+		[GFX10_V_CMPX_NLG_F16] = "v_cmpx_nlg_f16",
+		[GFX10_V_CMPX_NGT_F16] = "v_cmpx_ngt_f16",
+		[GFX10_V_CMPX_NLE_F16] = "v_cmpx_nle_f16",
+		[GFX10_V_CMPX_NEQ_F16] = "v_cmpx_neq_f16",
+		[GFX10_V_CMPX_NLT_F16] = "v_cmpx_nlt_f16",
+		[GFX10_V_CMPX_TRU_F16] = "v_cmpx_tru_f16",
+	},
+	[AMDGCN_INSN_TYPE_VOP3B] = {
+		[GFX10_V_DIV_SCALE_F32] = "v_div_scale_f32",
+		[GFX10_V_DIV_SCALE_F64] = "v_div_scale_f64",
+		[GFX10_V_MAD_U64_U32] = "v_mad_u64_u32",
+		[GFX10_V_MAD_I64_I32] = "v_mad_i64_i32",
+		[GFX10_V_ADD_CO_U32] = "v_add_co_u32",
+		[GFX10_V_SUB_CO_U32] = "v_sub_co_u32",
+		[GFX10_V_SUBREV_CO_U32] = "v_subrev_co_u32",
+	},
+	[AMDGCN_INSN_TYPE_VOP3P] = {
+		[GFX10_V_PK_MAD_I16] = "v_pk_mad_i16",
+		[GFX10_V_PK_MUL_LO_U16] = "v_pk_mul_lo_u16",
+		[GFX10_V_PK_ADD_I16] = "v_pk_add_i16",
+		[GFX10_V_PK_SUB_I16] = "v_pk_sub_i16",
+		[GFX10_V_PK_LSHLREV_B16] = "v_pk_lshlrev_b16",
+		[GFX10_V_PK_LSHRREV_B16] = "v_pk_lshrrev_b16",
+		[GFX10_V_PK_ASHRREV_I16] = "v_pk_ashrrev_i16",
+		[GFX10_V_PK_MAX_I16] = "v_pk_max_i16",
+		[GFX10_V_PK_MIN_I16] = "v_pk_min_i16",
+		[GFX10_V_PK_MAD_U16] = "v_pk_mad_u16",
+		[GFX10_V_PK_ADD_U16] = "v_pk_add_u16",
+		[GFX10_V_PK_SUB_U16] = "v_pk_sub_u16",
+		[GFX10_V_PK_MAX_U16] = "v_pk_max_u16",
+		[GFX10_V_PK_MIN_U16] = "v_pk_min_u16",
+		[GFX10_V_PK_FMA_F16] = "v_pk_fma_f16",
+		[GFX10_V_PK_ADD_F16] = "v_pk_add_f16",
+		[GFX10_V_PK_MUL_F16] = "v_pk_mul_f16",
+		[GFX10_V_PK_MIN_F16] = "v_pk_min_f16",
+		[GFX10_V_PK_MAX_F16] = "v_pk_max_f16",
+		[GFX10_V_DOT2_F32_F16] = "v_dot2_f32_f16",
+		[GFX10_V_DOT2_I32_I16] = "v_dot2_i32_i16",
+		[GFX10_V_DOT2_U32_U16] = "v_dot2_u32_u16",
+		[GFX10_V_DOT4_I32_I8] = "v_dot4_i32_i8",
+		[GFX10_V_DOT4_U32_U8] = "v_dot4_u32_u8",
+		[GFX10_V_DOT8_I32_I4] = "v_dot8_i32_i4",
+		[GFX10_V_DOT8_U32_U4] = "v_dot8_u32_u4",
+		[GFX10_V_FMA_MIX_F32] = "v_fma_mix_f32",
+		[GFX10_V_FMA_MIXLO_F16] = "v_fma_mixlo_f16",
+		[GFX10_V_FMA_MIXHI_F16] = "v_fma_mixhi_f16",
+	},
+	[AMDGCN_INSN_TYPE_MUBUF] = {
+		[GFX10_BUFFER_LOAD_FORMAT_X] = "buffer_load_format_x",
+		[GFX10_BUFFER_LOAD_FORMAT_XY] = "buffer_load_format_xy",
+		[GFX10_BUFFER_LOAD_FORMAT_XYZ] = "buffer_load_format_xyz",
+		[GFX10_BUFFER_LOAD_FORMAT_XYZW] = "buffer_load_format_xyzw",
+		[GFX10_BUFFER_STORE_FORMAT_X] = "buffer_store_format_x",
+		[GFX10_BUFFER_STORE_FORMAT_XY] = "buffer_store_format_xy",
+		[GFX10_BUFFER_STORE_FORMAT_XYZ] = "buffer_store_format_xyz",
+		[GFX10_BUFFER_STORE_FORMAT_XYZW] = "buffer_store_format_xyzw",
+		[GFX10_BUFFER_LOAD_UBYTE] = "buffer_load_ubyte",
+		[GFX10_BUFFER_LOAD_SBYTE] = "buffer_load_sbyte",
+		[GFX10_BUFFER_LOAD_USHORT] = "buffer_load_ushort",
+		[GFX10_BUFFER_LOAD_SSHORT] = "buffer_load_sshort",
+		[GFX10_BUFFER_LOAD_DWORD] = "buffer_load_dword",
+		[GFX10_BUFFER_LOAD_DWORDX2] = "buffer_load_dwordx2",
+		[GFX10_BUFFER_LOAD_DWORDX4] = "buffer_load_dwordx4",
+		[GFX10_BUFFER_LOAD_DWORDX3] = "buffer_load_dwordx3",
+		[GFX10_BUFFER_STORE_BYTE] = "buffer_store_byte",
+		[GFX10_BUFFER_STORE_BYTE_D16_HI] = "buffer_store_byte_d16_hi",
+		[GFX10_BUFFER_STORE_SHORT] = "buffer_store_short",
+		[GFX10_BUFFER_STORE_SHORT_D16_HI] = "buffer_store_short_d16_hi",
+		[GFX10_BUFFER_STORE_DWORD] = "buffer_store_dword",
+		[GFX10_BUFFER_STORE_DWORDX2] = "buffer_store_dwordx2",
+		[GFX10_BUFFER_STORE_DWORDX4] = "buffer_store_dwordx4",
+		[GFX10_BUFFER_STORE_DWORDX3] = "buffer_store_dwordx3",
+		[GFX10_BUFFER_LOAD_UBYTE_D16] = "buffer_load_ubyte_d16",
+		[GFX10_BUFFER_LOAD_UBYTE_D16_HI] = "buffer_load_ubyte_d16_hi",
+		[GFX10_BUFFER_LOAD_SBYTE_D16] = "buffer_load_sbyte_d16",
+		[GFX10_BUFFER_LOAD_SBYTE_D16_HI] = "buffer_load_sbyte_d16_hi",
+		[GFX10_BUFFER_LOAD_SHORT_D16] = "buffer_load_short_d16",
+		[GFX10_BUFFER_LOAD_SHORT_D16_HI] = "buffer_load_short_d16_hi",
+		[GFX10_BUFFER_LOAD_FORMAT_D16_HI_X] = "buffer_load_format_d16_hi_x",
+		[GFX10_BUFFER_STORE_FORMAT_D16_HI_X] = "buffer_store_format_d16_hi_x",
+		[GFX10_BUFFER_ATOMIC_SWAP] = "buffer_atomic_swap",
+		[GFX10_BUFFER_ATOMIC_CMPSWAP] = "buffer_atomic_cmpswap",
+		[GFX10_BUFFER_ATOMIC_ADD] = "buffer_atomic_add",
+		[GFX10_BUFFER_ATOMIC_SUB] = "buffer_atomic_sub",
+		[GFX10_BUFFER_ATOMIC_CSUB] = "buffer_atomic_csub",
+		[GFX10_BUFFER_ATOMIC_SMIN] = "buffer_atomic_smin",
+		[GFX10_BUFFER_ATOMIC_UMIN] = "buffer_atomic_umin",
+		[GFX10_BUFFER_ATOMIC_SMAX] = "buffer_atomic_smax",
+		[GFX10_BUFFER_ATOMIC_UMAX] = "buffer_atomic_umax",
+		[GFX10_BUFFER_ATOMIC_AND] = "buffer_atomic_and",
+		[GFX10_BUFFER_ATOMIC_OR] = "buffer_atomic_or",
+		[GFX10_BUFFER_ATOMIC_XOR] = "buffer_atomic_xor",
+		[GFX10_BUFFER_ATOMIC_INC] = "buffer_atomic_inc",
+		[GFX10_BUFFER_ATOMIC_DEC] = "buffer_atomic_dec",
+		[GFX10_BUFFER_ATOMIC_FCMPSWAP] = "buffer_atomic_fcmpswap",
+		[GFX10_BUFFER_ATOMIC_FMIN] = "buffer_atomic_fmin",
+		[GFX10_BUFFER_ATOMIC_FMAX] = "buffer_atomic_fmax",
+		[GFX10_BUFFER_ATOMIC_SWAP_X2] = "buffer_atomic_swap_x2",
+		[GFX10_BUFFER_ATOMIC_CMPSWAP_X2] = "buffer_atomic_cmpswap_x2",
+		[GFX10_BUFFER_ATOMIC_ADD_X2] = "buffer_atomic_add_x2",
+		[GFX10_BUFFER_ATOMIC_SUB_X2] = "buffer_atomic_sub_x2",
+		[GFX10_BUFFER_ATOMIC_SMIN_X2] = "buffer_atomic_smin_x2",
+		[GFX10_BUFFER_ATOMIC_UMIN_X2] = "buffer_atomic_umin_x2",
+		[GFX10_BUFFER_ATOMIC_SMAX_X2] = "buffer_atomic_smax_x2",
+		[GFX10_BUFFER_ATOMIC_UMAX_X2] = "buffer_atomic_umax_x2",
+		[GFX10_BUFFER_ATOMIC_AND_X2] = "buffer_atomic_and_x2",
+		[GFX10_BUFFER_ATOMIC_OR_X2] = "buffer_atomic_or_x2",
+		[GFX10_BUFFER_ATOMIC_XOR_X2] = "buffer_atomic_xor_x2",
+		[GFX10_BUFFER_ATOMIC_INC_X2] = "buffer_atomic_inc_x2",
+		[GFX10_BUFFER_ATOMIC_DEC_X2] = "buffer_atomic_dec_x2",
+		[GFX10_BUFFER_ATOMIC_FCMPSWAP_X2] = "buffer_atomic_fcmpswap_x2",
+		[GFX10_BUFFER_ATOMIC_FMIN_X2] = "buffer_atomic_fmin_x2",
+		[GFX10_BUFFER_ATOMIC_FMAX_X2] = "buffer_atomic_fmax_x2",
+		[GFX10_BUFFER_GL0_INV] = "buffer_gl0_inv",
+		[GFX10_BUFFER_GL1_INV] = "buffer_gl1_inv",
+		[GFX10_BUFFER_LOAD_FORMAT_D16_X] = "buffer_load_format_d16_x",
+		[GFX10_BUFFER_LOAD_FORMAT_D16_XY] = "buffer_load_format_d16_xy",
+		[GFX10_BUFFER_LOAD_FORMAT_D16_XYZ] = "buffer_load_format_d16_xyz",
+		[GFX10_BUFFER_LOAD_FORMAT_D16_XYZW] = "buffer_load_format_d16_xyzw",
+		[GFX10_BUFFER_STORE_FORMAT_D16_X] = "buffer_store_format_d16_x",
+		[GFX10_BUFFER_STORE_FORMAT_D16_XY] = "buffer_store_format_d16_xy",
+		[GFX10_BUFFER_STORE_FORMAT_D16_XYZ] = "buffer_store_format_d16_xyz",
+		[GFX10_BUFFER_STORE_FORMAT_D16_XYZW] = "buffer_store_format_d16_xyzw",
+	},
+	[AMDGCN_INSN_TYPE_FLAT] = {
+		[GFX10_GLOBAL_LOAD_UBYTE] = "global_load_ubyte",
+		[GFX10_GLOBAL_LOAD_SBYTE] = "global_load_sbyte",
+		[GFX10_GLOBAL_LOAD_USHORT] = "global_load_ushort",
+		[GFX10_GLOBAL_LOAD_SSHORT] = "global_load_sshort",
+		[GFX10_GLOBAL_LOAD_DWORD] = "global_load_dword",
+		[GFX10_GLOBAL_LOAD_DWORDX2] = "global_load_dwordx2",
+		[GFX10_GLOBAL_LOAD_DWORDX4] = "global_load_dwordx4",
+		[GFX10_GLOBAL_LOAD_DWORDX3] = "global_load_dwordx3",
+		[GFX10_GLOBAL_LOAD_DWORD_ADDTID] = "global_load_dword_addtid",
+		[GFX10_GLOBAL_STORE_DWORD_ADDTID] = "global_store_dword_addtid",
+		[GFX10_GLOBAL_STORE_BYTE] = "global_store_byte",
+		[GFX10_GLOBAL_STORE_BYTE_D16_HI] = "global_store_byte_d16_hi",
+		[GFX10_GLOBAL_STORE_SHORT] = "global_store_short",
+		[GFX10_GLOBAL_STORE_SHORT_D16_HI] = "global_store_short_d16_hi",
+		[GFX10_GLOBAL_STORE_DWORD] = "global_store_dword",
+		[GFX10_GLOBAL_STORE_DWORDX2] = "global_store_dwordx2",
+		[GFX10_GLOBAL_STORE_DWORDX4] = "global_store_dwordx4",
+		[GFX10_GLOBAL_STORE_DWORDX3] = "global_store_dwordx3",
+		[GFX10_GLOBAL_LOAD_UBYTE_D16] = "global_load_ubyte_d16",
+		[GFX10_GLOBAL_LOAD_UBYTE_D16_HI] = "global_load_ubyte_d16_hi",
+		[GFX10_GLOBAL_LOAD_SBYTE_D16] = "global_load_sbyte_d16",
+		[GFX10_GLOBAL_LOAD_SBYTE_D16_HI] = "global_load_sbyte_d16_hi",
+		[GFX10_GLOBAL_LOAD_SHORT_D16] = "global_load_short_d16",
+		[GFX10_GLOBAL_LOAD_SHORT_D16_HI] = "global_load_short_d16_hi",
+		[GFX10_GLOBAL_ATOMIC_SWAP] = "global_atomic_swap",
+		[GFX10_GLOBAL_ATOMIC_CMPSWAP] = "global_atomic_cmpswap",
+		[GFX10_GLOBAL_ATOMIC_ADD] = "global_atomic_add",
+		[GFX10_GLOBAL_ATOMIC_SUB] = "global_atomic_sub",
+		[GFX10_GLOBAL_ATOMIC_CSUB] = "global_atomic_csub",
+		[GFX10_GLOBAL_ATOMIC_SMIN] = "global_atomic_smin",
+		[GFX10_GLOBAL_ATOMIC_UMIN] = "global_atomic_umin",
+		[GFX10_GLOBAL_ATOMIC_SMAX] = "global_atomic_smax",
+		[GFX10_GLOBAL_ATOMIC_UMAX] = "global_atomic_umax",
+		[GFX10_GLOBAL_ATOMIC_AND] = "global_atomic_and",
+		[GFX10_GLOBAL_ATOMIC_OR] = "global_atomic_or",
+		[GFX10_GLOBAL_ATOMIC_XOR] = "global_atomic_xor",
+		[GFX10_GLOBAL_ATOMIC_INC] = "global_atomic_inc",
+		[GFX10_GLOBAL_ATOMIC_DEC] = "global_atomic_dec",
+		[GFX10_GLOBAL_ATOMIC_FCMPSWAP] = "global_atomic_fcmpswap",
+		[GFX10_GLOBAL_ATOMIC_FMIN] = "global_atomic_fmin",
+		[GFX10_GLOBAL_ATOMIC_FMAX] = "global_atomic_fmax",
+		[GFX10_GLOBAL_ATOMIC_SWAP_X2] = "global_atomic_swap_x2",
+		[GFX10_GLOBAL_ATOMIC_CMPSWAP_X2] = "global_atomic_cmpswap_x2",
+		[GFX10_GLOBAL_ATOMIC_ADD_X2] = "global_atomic_add_x2",
+		[GFX10_GLOBAL_ATOMIC_SUB_X2] = "global_atomic_sub_x2",
+		[GFX10_GLOBAL_ATOMIC_SMIN_X2] = "global_atomic_smin_x2",
+		[GFX10_GLOBAL_ATOMIC_UMIN_X2] = "global_atomic_umin_x2",
+		[GFX10_GLOBAL_ATOMIC_SMAX_X2] = "global_atomic_smax_x2",
+		[GFX10_GLOBAL_ATOMIC_UMAX_X2] = "global_atomic_umax_x2",
+		[GFX10_GLOBAL_ATOMIC_AND_X2] = "global_atomic_and_x2",
+		[GFX10_GLOBAL_ATOMIC_OR_X2] = "global_atomic_or_x2",
+		[GFX10_GLOBAL_ATOMIC_XOR_X2] = "global_atomic_xor_x2",
+		[GFX10_GLOBAL_ATOMIC_INC_X2] = "global_atomic_inc_x2",
+		[GFX10_GLOBAL_ATOMIC_DEC_X2] = "global_atomic_dec_x2",
+		[GFX10_GLOBAL_ATOMIC_FCMPSWAP_X2] = "global_atomic_fcmpswap_x2",
+		[GFX10_GLOBAL_ATOMIC_FMIN_X2] = "global_atomic_fmin_x2",
+		[GFX10_GLOBAL_ATOMIC_FMAX_X2] = "global_atomic_fmax_x2",
+	},
+};
+
+static const char opnames_gfx9[__AMDGCN_INSN_TYPE_MAX][900][30] = {
+	[AMDGCN_INSN_TYPE_SOP2] = {
+		[GFX9_S_ADD_U32] = "s_add_u32",
+		[GFX9_S_SUB_U32] = "s_sub_u32",
+		[GFX9_S_ADD_I32] = "s_add_i32",
+		[GFX9_S_SUB_I32] = "s_sub_i32",
+		[GFX9_S_ADDC_U32] = "s_addc_u32",
+		[GFX9_S_SUBB_U32] = "s_subb_u32",
+		[GFX9_S_MIN_I32] = "s_min_i32",
+		[GFX9_S_MIN_U32] = "s_min_u32",
+		[GFX9_S_MAX_I32] = "s_max_i32",
+		[GFX9_S_MAX_U32] = "s_max_u32",
+		[GFX9_S_CSELECT_B32] = "s_cselect_b32",
+		[GFX9_S_CSELECT_B64] = "s_cselect_b64",
+		[GFX9_S_AND_B32] = "s_and_b32",
+		[GFX9_S_AND_B64] = "s_and_b64",
+		[GFX9_S_OR_B32] = "s_or_b32",
+		[GFX9_S_OR_B64] = "s_or_b64",
+		[GFX9_S_XOR_B32] = "s_xor_b32",
+		[GFX9_S_XOR_B64] = "s_xor_b64",
+		[GFX9_S_ANDN2_B32] = "s_andn2_b32",
+		[GFX9_S_ANDN2_B64] = "s_andn2_b64",
+		[GFX9_S_ORN2_B32] = "s_orn2_b32",
+		[GFX9_S_ORN2_B64] = "s_orn2_b64",
+		[GFX9_S_NAND_B32] = "s_nand_b32",
+		[GFX9_S_NAND_B64] = "s_nand_b64",
+		[GFX9_S_NOR_B32] = "s_nor_b32",
+		[GFX9_S_NOR_B64] = "s_nor_b64",
+		[GFX9_S_XNOR_B32] = "s_xnor_b32",
+		[GFX9_S_XNOR_B64] = "s_xnor_b64",
+		[GFX9_S_LSHL_B32] = "s_lshl_b32",
+		[GFX9_S_LSHL_B64] = "s_lshl_b64",
+		[GFX9_S_LSHR_B32] = "s_lshr_b32",
+		[GFX9_S_LSHR_B64] = "s_lshr_b64",
+		[GFX9_S_ASHR_I32] = "s_ashr_i32",
+		[GFX9_S_ASHR_I64] = "s_ashr_i64",
+		[GFX9_S_BFM_B32] = "s_bfm_b32",
+		[GFX9_S_BFM_B64] = "s_bfm_b64",
+		[GFX9_S_MUL_I32] = "s_mul_i32",
+		[GFX9_S_BFE_U32] = "s_bfe_u32",
+		[GFX9_S_BFE_I32] = "s_bfe_i32",
+		[GFX9_S_BFE_U64] = "s_bfe_u64",
+		[GFX9_S_BFE_I64] = "s_bfe_i64",
+		[GFX9_S_CBRANCH_G_FORK] = "s_cbranch_g_fork",
+		[GFX9_S_ABSDIFF_I32] = "s_absdiff_i32",
+		[GFX9_S_RFE_RESTORE_B64] = "s_rfe_restore_b64",
+		[GFX9_S_MUL_HI_U32] = "s_mul_hi_u32",
+		[GFX9_S_MUL_HI_I32] = "s_mul_hi_i32",
+		[GFX9_S_LSHL1_ADD_U32] = "s_lshl1_add_u32",
+		[GFX9_S_LSHL2_ADD_U32] = "s_lshl2_add_u32",
+		[GFX9_S_LSHL3_ADD_U32] = "s_lshl3_add_u32",
+		[GFX9_S_LSHL4_ADD_U32] = "s_lshl4_add_u32",
+		[GFX9_S_PACK_LL_B32_B16] = "s_pack_ll_b32_b16",
+		[GFX9_S_PACK_LH_B32_B16] = "s_pack_lh_b32_b16",
+		[GFX9_S_PACK_HH_B32_B16] = "s_pack_hh_b32_b16",
+	},
+	[AMDGCN_INSN_TYPE_SOPK] = {
+		[GFX9_S_MOVK_I32] = "s_movk_i32",
+		[GFX9_S_CMOVK_I32] = "s_cmovk_i32",
+		[GFX9_S_CMPK_EQ_I32] = "s_cmpk_eq_i32",
+		[GFX9_S_CMPK_LG_I32] = "s_cmpk_lg_i32",
+		[GFX9_S_CMPK_GT_I32] = "s_cmpk_gt_i32",
+		[GFX9_S_CMPK_GE_I32] = "s_cmpk_ge_i32",
+		[GFX9_S_CMPK_LT_I32] = "s_cmpk_lt_i32",
+		[GFX9_S_CMPK_LE_I32] = "s_cmpk_le_i32",
+		[GFX9_S_CMPK_EQ_U32] = "s_cmpk_eq_u32",
+		[GFX9_S_CMPK_LG_U32] = "s_cmpk_lg_u32",
+		[GFX9_S_CMPK_GT_U32] = "s_cmpk_gt_u32",
+		[GFX9_S_CMPK_GE_U32] = "s_cmpk_ge_u32",
+		[GFX9_S_CMPK_LT_U32] = "s_cmpk_lt_u32",
+		[GFX9_S_CMPK_LE_U32] = "s_cmpk_le_u32",
+		[GFX9_S_ADDK_I32] = "s_addk_i32",
+		[GFX9_S_MULK_I32] = "s_mulk_i32",
+		[GFX9_S_CBRANCH_I_FORK] = "s_cbranch_i_fork",
+		[GFX9_S_GETREG_B32] = "s_getreg_b32",
+		[GFX9_S_SETREG_B32] = "s_setreg_b32",
+		[GFX9_S_SETREG_IMM32_B32] = "s_setreg_imm32_b32",
+		[GFX9_S_CALL_B64] = "s_call_b64",
+	},
+	[AMDGCN_INSN_TYPE_SOP1] = {
+		[GFX9_S_MOV_B32] = "s_mov_b32",
+		[GFX9_S_MOV_B64] = "s_mov_b64",
+		[GFX9_S_CMOV_B32] = "s_cmov_b32",
+		[GFX9_S_CMOV_B64] = "s_cmov_b64",
+		[GFX9_S_NOT_B32] = "s_not_b32",
+		[GFX9_S_NOT_B64] = "s_not_b64",
+		[GFX9_S_WQM_B32] = "s_wqm_b32",
+		[GFX9_S_WQM_B64] = "s_wqm_b64",
+		[GFX9_S_BREV_B32] = "s_brev_b32",
+		[GFX9_S_BREV_B64] = "s_brev_b64",
+		[GFX9_S_BCNT0_I32_B32] = "s_bcnt0_i32_b32",
+		[GFX9_S_BCNT0_I32_B64] = "s_bcnt0_i32_b64",
+		[GFX9_S_BCNT1_I32_B32] = "s_bcnt1_i32_b32",
+		[GFX9_S_BCNT1_I32_B64] = "s_bcnt1_i32_b64",
+		[GFX9_S_FF0_I32_B32] = "s_ff0_i32_b32",
+		[GFX9_S_FF0_I32_B64] = "s_ff0_i32_b64",
+		[GFX9_S_FF1_I32_B32] = "s_ff1_i32_b32",
+		[GFX9_S_FF1_I32_B64] = "s_ff1_i32_b64",
+		[GFX9_S_FLBIT_I32_B32] = "s_flbit_i32_b32",
+		[GFX9_S_FLBIT_I32_B64] = "s_flbit_i32_b64",
+		[GFX9_S_FLBIT_I32] = "s_flbit_i32",
+		[GFX9_S_FLBIT_I32_I64] = "s_flbit_i32_i64",
+		[GFX9_S_SEXT_I32_I8] = "s_sext_i32_i8",
+		[GFX9_S_SEXT_I32_I16] = "s_sext_i32_i16",
+		[GFX9_S_BITSET0_B32] = "s_bitset0_b32",
+		[GFX9_S_BITSET0_B64] = "s_bitset0_b64",
+		[GFX9_S_BITSET1_B32] = "s_bitset1_b32",
+		[GFX9_S_BITSET1_B64] = "s_bitset1_b64",
+		[GFX9_S_GETPC_B64] = "s_getpc_b64",
+		[GFX9_S_SETPC_B64] = "s_setpc_b64",
+		[GFX9_S_SWAPPC_B64] = "s_swappc_b64",
+		[GFX9_S_RFE_B64] = "s_rfe_b64",
+		[GFX9_S_AND_SAVEEXEC_B64] = "s_and_saveexec_b64",
+		[GFX9_S_XNOR_SAVEEXEC_B64] = "s_xnor_saveexec_b64",
+		[GFX9_S_QUADMASK_B32] = "s_quadmask_b32",
+		[GFX9_S_QUADMASK_B64] = "s_quadmask_b64",
+		[GFX9_S_MOVRELS_B32] = "s_movrels_b32",
+		[GFX9_S_MOVRELS_B64] = "s_movrels_b64",
+		[GFX9_S_MOVRELD_B32] = "s_movreld_b32",
+		[GFX9_S_MOVRELD_B64] = "s_movreld_b64",
+		[GFX9_S_ABS_I32] = "s_abs_i32",
+		[GFX9_S_ANDN1_SAVEEXEC_B64] = "s_andn1_saveexec_b64",
+		[GFX9_S_ORN1_SAVEEXEC_B64] = "s_orn1_saveexec_b64",
+		[GFX9_S_ANDN1_WREXEC_B64] = "s_andn1_wrexec_b64",
+		[GFX9_S_ANDN2_WREXEC_B64] = "s_andn2_wrexec_b64",
+		[GFX9_S_BITREPLICATE_B64_B32] = "s_bitreplicate_b64_b32",
+		/* SOP1 opcodes missing from the table (Vega ISA 12.3). */
+		[GFX9_S_OR_SAVEEXEC_B64] = "s_or_saveexec_b64",
+		[GFX9_S_XOR_SAVEEXEC_B64] = "s_xor_saveexec_b64",
+		[GFX9_S_ANDN2_SAVEEXEC_B64] = "s_andn2_saveexec_b64",
+		[GFX9_S_ORN2_SAVEEXEC_B64] = "s_orn2_saveexec_b64",
+		[GFX9_S_NAND_SAVEEXEC_B64] = "s_nand_saveexec_b64",
+		[GFX9_S_NOR_SAVEEXEC_B64] = "s_nor_saveexec_b64",
+		[GFX9_S_SET_GPR_IDX_IDX] = "s_set_gpr_idx_idx",
+	},
+	[AMDGCN_INSN_TYPE_SOPC] = {
+		[GFX9_S_CMP_EQ_I32] = "s_cmp_eq_i32",
+		[GFX9_S_CMP_LG_I32] = "s_cmp_lg_i32",
+		[GFX9_S_CMP_GT_I32] = "s_cmp_gt_i32",
+		[GFX9_S_CMP_GE_I32] = "s_cmp_ge_i32",
+		[GFX9_S_CMP_LT_I32] = "s_cmp_lt_i32",
+		[GFX9_S_CMP_LE_I32] = "s_cmp_le_i32",
+		[GFX9_S_CMP_EQ_U32] = "s_cmp_eq_u32",
+		[GFX9_S_CMP_LG_U32] = "s_cmp_lg_u32",
+		[GFX9_S_CMP_GT_U32] = "s_cmp_gt_u32",
+		[GFX9_S_CMP_GE_U32] = "s_cmp_ge_u32",
+		[GFX9_S_CMP_LT_U32] = "s_cmp_lt_u32",
+		[GFX9_S_CMP_LE_U32] = "s_cmp_le_u32",
+		[GFX9_S_BITCMP0_B32] = "s_bitcmp0_b32",
+		[GFX9_S_BITCMP1_B32] = "s_bitcmp1_b32",
+		[GFX9_S_BITCMP0_B64] = "s_bitcmp0_b64",
+		[GFX9_S_BITCMP1_B64] = "s_bitcmp1_b64",
+		[GFX9_S_SETVSKIP] = "s_setvskip",
+		[GFX9_S_SET_GPR_IDX_ON] = "s_set_gpr_idx_on",
+		[GFX9_S_CMP_EQ_U64] = "s_cmp_eq_u64",
+		[GFX9_S_CMP_LG_U64] = "s_cmp_lg_u64",
+	},
+	[AMDGCN_INSN_TYPE_SOPP] = {
+		[GFX9_S_NOP] = "s_nop",
+		[GFX9_S_ENDPGM] = "s_endpgm",
+		[GFX9_S_BRANCH] = "s_branch",
+		[GFX9_S_WAKEUP] = "s_wakeup",
+		[GFX9_S_CBRANCH_SCC0] = "s_cbranch_scc0",
+		[GFX9_S_CBRANCH_SCC1] = "s_cbranch_scc1",
+		[GFX9_S_CBRANCH_VCCZ] = "s_cbranch_vccz",
+		[GFX9_S_CBRANCH_VCCNZ] = "s_cbranch_vccnz",
+		[GFX9_S_CBRANCH_EXECZ] = "s_cbranch_execz",
+		[GFX9_S_CBRANCH_EXECNZ] = "s_cbranch_execnz",
+		[GFX9_S_BARRIER] = "s_barrier",
+		[GFX9_S_SETKILL] = "s_setkill",
+		[GFX9_S_WAITCNT] = "s_waitcnt",
+		[GFX9_S_SETHALT] = "s_sethalt",
+		[GFX9_S_SLEEP] = "s_sleep",
+		[GFX9_S_SETPRIO] = "s_setprio",
+		[GFX9_S_SENDMSG] = "s_sendmsg",
+		[GFX9_S_SENDMSGHALT] = "s_sendmsghalt",
+		[GFX9_S_TRAP] = "s_trap",
+		[GFX9_S_ICACHE_INV] = "s_icache_inv",
+		[GFX9_S_INCPERFLEVEL] = "s_incperflevel",
+		[GFX9_S_DECPERFLEVEL] = "s_decperflevel",
+		[GFX9_S_TTRACEDATA] = "s_ttracedata",
+		[GFX9_S_CBRANCH_CDBGSYS] = "s_cbranch_cdbgsys",
+		[GFX9_S_CBRANCH_CDBGUSER] = "s_cbranch_cdbguser",
+		[GFX9_S_CBRANCH_CDBGSYS_OR_USER] = "s_cbranch_cdbgsys_or_user",
+		[GFX9_S_CBRANCH_CDBGSYS_AND_USER] = "s_cbranch_cdbgsys_and_user",
+		[GFX9_S_ENDPGM_SAVED] = "s_endpgm_saved",
+		[GFX9_S_SET_GPR_IDX_OFF] = "s_set_gpr_idx_off",
+		[GFX9_S_SET_GPR_IDX_MODE] = "s_set_gpr_idx_mode",
+		[GFX9_S_ENDPGM_ORDERED_PS_DONE] = "s_endpgm_ordered_ps_done",
+	},
+	[AMDGCN_INSN_TYPE_SMEM] = {
+		[GFX9_S_LOAD_DWORD] = "s_load_dword",
+		[GFX9_S_LOAD_DWORDX2] = "s_load_dwordx2",
+		[GFX9_S_LOAD_DWORDX4] = "s_load_dwordx4",
+		[GFX9_S_LOAD_DWORDX8] = "s_load_dwordx8",
+		[GFX9_S_LOAD_DWORDX16] = "s_load_dwordx16",
+		[GFX9_S_SCRATCH_LOAD_DWORD] = "s_scratch_load_dword",
+		[GFX9_S_SCRATCH_LOAD_DWORDX2] = "s_scratch_load_dwordx2",
+		[GFX9_S_SCRATCH_LOAD_DWORDX4] = "s_scratch_load_dwordx4",
+		[GFX9_S_BUFFER_LOAD_DWORD] = "s_buffer_load_dword",
+		[GFX9_S_BUFFER_LOAD_DWORDX2] = "s_buffer_load_dwordx2",
+		[GFX9_S_BUFFER_LOAD_DWORDX4] = "s_buffer_load_dwordx4",
+		[GFX9_S_BUFFER_LOAD_DWORDX8] = "s_buffer_load_dwordx8",
+		[GFX9_S_BUFFER_LOAD_DWORDX16] = "s_buffer_load_dwordx16",
+		[GFX9_S_STORE_DWORD] = "s_store_dword",
+		[GFX9_S_STORE_DWORDX2] = "s_store_dwordx2",
+		[GFX9_S_STORE_DWORDX4] = "s_store_dwordx4",
+		[GFX9_S_SCRATCH_STORE_DWORD] = "s_scratch_store_dword",
+		[GFX9_S_SCRATCH_STORE_DWORDX2] = "s_scratch_store_dwordx2",
+		[GFX9_S_SCRATCH_STORE_DWORDX4] = "s_scratch_store_dwordx4",
+		[GFX9_S_BUFFER_STORE_DWORD] = "s_buffer_store_dword",
+		[GFX9_S_BUFFER_STORE_DWORDX2] = "s_buffer_store_dwordx2",
+		[GFX9_S_BUFFER_STORE_DWORDX4] = "s_buffer_store_dwordx4",
+		[GFX9_S_DCACHE_INV] = "s_dcache_inv",
+		[GFX9_S_DCACHE_WB] = "s_dcache_wb",
+		[GFX9_S_DCACHE_INV_VOL] = "s_dcache_inv_vol",
+		[GFX9_S_DCACHE_WB_VOL] = "s_dcache_wb_vol",
+		[GFX9_S_MEMTIME] = "s_memtime",
+		[GFX9_S_MEMREALTIME] = "s_memrealtime",
+		[GFX9_S_ATC_PROBE] = "s_atc_probe",
+		[GFX9_S_ATC_PROBE_BUFFER] = "s_atc_probe_buffer",
+		[GFX9_S_DCACHE_DISCARD] = "s_dcache_discard",
+		[GFX9_S_DCACHE_DISCARD_X2] = "s_dcache_discard_x2",
+		[GFX9_S_BUFFER_ATOMIC_SWAP] = "s_buffer_atomic_swap",
+		[GFX9_S_BUFFER_ATOMIC_CMPSWAP] = "s_buffer_atomic_cmpswap",
+		[GFX9_S_BUFFER_ATOMIC_ADD] = "s_buffer_atomic_add",
+		[GFX9_S_BUFFER_ATOMIC_SUB] = "s_buffer_atomic_sub",
+		[GFX9_S_BUFFER_ATOMIC_SMIN] = "s_buffer_atomic_smin",
+		[GFX9_S_BUFFER_ATOMIC_UMIN] = "s_buffer_atomic_umin",
+		[GFX9_S_BUFFER_ATOMIC_SMAX] = "s_buffer_atomic_smax",
+		[GFX9_S_BUFFER_ATOMIC_UMAX] = "s_buffer_atomic_umax",
+		[GFX9_S_BUFFER_ATOMIC_AND] = "s_buffer_atomic_and",
+		[GFX9_S_BUFFER_ATOMIC_OR] = "s_buffer_atomic_or",
+		[GFX9_S_BUFFER_ATOMIC_XOR] = "s_buffer_atomic_xor",
+		[GFX9_S_BUFFER_ATOMIC_INC] = "s_buffer_atomic_inc",
+		[GFX9_S_BUFFER_ATOMIC_DEC] = "s_buffer_atomic_dec",
+		[GFX9_S_BUFFER_ATOMIC_SWAP_X2] = "s_buffer_atomic_swap_x2",
+		[GFX9_S_BUFFER_ATOMIC_CMPSWAP_X2] = "s_buffer_atomic_cmpswap_x2",
+		[GFX9_S_BUFFER_ATOMIC_ADD_X2] = "s_buffer_atomic_add_x2",
+		[GFX9_S_BUFFER_ATOMIC_SUB_X2] = "s_buffer_atomic_sub_x2",
+		[GFX9_S_BUFFER_ATOMIC_SMIN_X2] = "s_buffer_atomic_smin_x2",
+		[GFX9_S_BUFFER_ATOMIC_UMIN_X2] = "s_buffer_atomic_umin_x2",
+		[GFX9_S_BUFFER_ATOMIC_SMAX_X2] = "s_buffer_atomic_smax_x2",
+		[GFX9_S_BUFFER_ATOMIC_UMAX_X2] = "s_buffer_atomic_umax_x2",
+		[GFX9_S_BUFFER_ATOMIC_AND_X2] = "s_buffer_atomic_and_x2",
+		[GFX9_S_BUFFER_ATOMIC_OR_X2] = "s_buffer_atomic_or_x2",
+		[GFX9_S_BUFFER_ATOMIC_XOR_X2] = "s_buffer_atomic_xor_x2",
+		[GFX9_S_BUFFER_ATOMIC_INC_X2] = "s_buffer_atomic_inc_x2",
+		[GFX9_S_BUFFER_ATOMIC_DEC_X2] = "s_buffer_atomic_dec_x2",
+		[GFX9_S_ATOMIC_SWAP] = "s_atomic_swap",
+		[GFX9_S_ATOMIC_CMPSWAP] = "s_atomic_cmpswap",
+		[GFX9_S_ATOMIC_ADD] = "s_atomic_add",
+		[GFX9_S_ATOMIC_SUB] = "s_atomic_sub",
+		[GFX9_S_ATOMIC_SMIN] = "s_atomic_smin",
+		[GFX9_S_ATOMIC_UMIN] = "s_atomic_umin",
+		[GFX9_S_ATOMIC_SMAX] = "s_atomic_smax",
+		[GFX9_S_ATOMIC_UMAX] = "s_atomic_umax",
+		[GFX9_S_ATOMIC_AND] = "s_atomic_and",
+		[GFX9_S_ATOMIC_OR] = "s_atomic_or",
+		[GFX9_S_ATOMIC_XOR] = "s_atomic_xor",
+		[GFX9_S_ATOMIC_INC] = "s_atomic_inc",
+		[GFX9_S_ATOMIC_DEC] = "s_atomic_dec",
+		[GFX9_S_ATOMIC_SWAP_X2] = "s_atomic_swap_x2",
+		[GFX9_S_ATOMIC_CMPSWAP_X2] = "s_atomic_cmpswap_x2",
+		[GFX9_S_ATOMIC_ADD_X2] = "s_atomic_add_x2",
+		[GFX9_S_ATOMIC_SUB_X2] = "s_atomic_sub_x2",
+		[GFX9_S_ATOMIC_SMIN_X2] = "s_atomic_smin_x2",
+		[GFX9_S_ATOMIC_UMIN_X2] = "s_atomic_umin_x2",
+		[GFX9_S_ATOMIC_SMAX_X2] = "s_atomic_smax_x2",
+		[GFX9_S_ATOMIC_UMAX_X2] = "s_atomic_umax_x2",
+		[GFX9_S_ATOMIC_AND_X2] = "s_atomic_and_x2",
+		[GFX9_S_ATOMIC_OR_X2] = "s_atomic_or_x2",
+		[GFX9_S_ATOMIC_XOR_X2] = "s_atomic_xor_x2",
+		[GFX9_S_ATOMIC_INC_X2] = "s_atomic_inc_x2",
+		[GFX9_S_ATOMIC_DEC_X2] = "s_atomic_dec_x2",
+	},
+	[AMDGCN_INSN_TYPE_VOP2] = {
+		[GFX9_V_CNDMASK_B32] = "v_cndmask_b32",
+		[GFX9_V_ADD_F32] = "v_add_f32",
+		[GFX9_V_SUB_F32] = "v_sub_f32",
+		[GFX9_V_SUBREV_F32] = "v_subrev_f32",
+		[GFX9_V_MUL_LEGACY_F32] = "v_mul_legacy_f32",
+		[GFX9_V_MUL_F32] = "v_mul_f32",
+		[GFX9_V_MUL_I32_I24] = "v_mul_i32_i24",
+		[GFX9_V_MUL_HI_I32_I24] = "v_mul_hi_i32_i24",
+		[GFX9_V_MUL_U32_U24] = "v_mul_u32_u24",
+		[GFX9_V_MUL_HI_U32_U24] = "v_mul_hi_u32_u24",
+		[GFX9_V_MIN_F32] = "v_min_f32",
+		[GFX9_V_MAX_F32] = "v_max_f32",
+		[GFX9_V_MIN_I32] = "v_min_i32",
+		[GFX9_V_MAX_I32] = "v_max_i32",
+		[GFX9_V_MIN_U32] = "v_min_u32",
+		[GFX9_V_MAX_U32] = "v_max_u32",
+		[GFX9_V_LSHRREV_B32] = "v_lshrrev_b32",
+		[GFX9_V_ASHRREV_I32] = "v_ashrrev_i32",
+		[GFX9_V_LSHLREV_B32] = "v_lshlrev_b32",
+		[GFX9_V_AND_B32] = "v_and_b32",
+		[GFX9_V_OR_B32] = "v_or_b32",
+		[GFX9_V_XOR_B32] = "v_xor_b32",
+		[GFX9_V_MAC_F32] = "v_mac_f32",
+		[GFX9_V_MADMK_F32] = "v_madmk_f32",
+		[GFX9_V_MADAK_F32] = "v_madak_f32",
+		[GFX9_V_ADD_CO_U32] = "v_add_co_u32",
+		[GFX9_V_SUB_CO_U32] = "v_sub_co_u32",
+		[GFX9_V_SUBREV_CO_U32] = "v_subrev_co_u32",
+		[GFX9_V_ADDC_CO_U32] = "v_addc_co_u32",
+		[GFX9_V_SUBB_CO_U32] = "v_subb_co_u32",
+		[GFX9_V_SUBBREV_CO_U32] = "v_subbrev_co_u32",
+		[GFX9_V_ADD_F16] = "v_add_f16",
+		[GFX9_V_SUB_F16] = "v_sub_f16",
+		[GFX9_V_SUBREV_F16] = "v_subrev_f16",
+		[GFX9_V_MUL_F16] = "v_mul_f16",
+		[GFX9_V_MAC_F16] = "v_mac_f16",
+		[GFX9_V_MADMK_F16] = "v_madmk_f16",
+		[GFX9_V_MADAK_F16] = "v_madak_f16",
+		[GFX9_V_ADD_U16] = "v_add_u16",
+		[GFX9_V_SUB_U16] = "v_sub_u16",
+		[GFX9_V_SUBREV_U16] = "v_subrev_u16",
+		[GFX9_V_MUL_LO_U16] = "v_mul_lo_u16",
+		[GFX9_V_LSHLREV_B16] = "v_lshlrev_b16",
+		[GFX9_V_LSHRREV_B16] = "v_lshrrev_b16",
+		[GFX9_V_ASHRREV_I16] = "v_ashrrev_i16",
+		[GFX9_V_MAX_F16] = "v_max_f16",
+		[GFX9_V_MIN_F16] = "v_min_f16",
+		[GFX9_V_MAX_U16] = "v_max_u16",
+		[GFX9_V_MAX_I16] = "v_max_i16",
+		[GFX9_V_MIN_U16] = "v_min_u16",
+		[GFX9_V_MIN_I16] = "v_min_i16",
+		[GFX9_V_LDEXP_F16] = "v_ldexp_f16",
+		[GFX9_V_ADD_U32] = "v_add_u32",
+		[GFX9_V_SUB_U32] = "v_sub_u32",
+		[GFX9_V_SUBREV_U32] = "v_subrev_u32",
+	},
+	[AMDGCN_INSN_TYPE_VOP1] = {
+		[GFX9_V_NOP] = "v_nop",
+		[GFX9_V_MOV_B32] = "v_mov_b32",
+		[GFX9_V_READFIRSTLANE_B32] = "v_readfirstlane_b32",
+		[GFX9_V_CVT_I32_F64] = "v_cvt_i32_f64",
+		[GFX9_V_CVT_F64_I32] = "v_cvt_f64_i32",
+		[GFX9_V_CVT_F32_I32] = "v_cvt_f32_i32",
+		[GFX9_V_CVT_F32_U32] = "v_cvt_f32_u32",
+		[GFX9_V_CVT_U32_F32] = "v_cvt_u32_f32",
+		[GFX9_V_CVT_I32_F32] = "v_cvt_i32_f32",
+		[GFX9_V_CVT_F16_F32] = "v_cvt_f16_f32",
+		[GFX9_V_CVT_F32_F16] = "v_cvt_f32_f16",
+		[GFX9_V_CVT_RPI_I32_F32] = "v_cvt_rpi_i32_f32",
+		[GFX9_V_CVT_FLR_I32_F32] = "v_cvt_flr_i32_f32",
+		[GFX9_V_CVT_OFF_F32_I4] = "v_cvt_off_f32_i4",
+		[GFX9_V_CVT_F32_F64] = "v_cvt_f32_f64",
+		[GFX9_V_CVT_F64_F32] = "v_cvt_f64_f32",
+		[GFX9_V_CVT_F32_UBYTE0] = "v_cvt_f32_ubyte0",
+		[GFX9_V_CVT_F32_UBYTE1] = "v_cvt_f32_ubyte1",
+		[GFX9_V_CVT_F32_UBYTE2] = "v_cvt_f32_ubyte2",
+		[GFX9_V_CVT_F32_UBYTE3] = "v_cvt_f32_ubyte3",
+		[GFX9_V_CVT_U32_F64] = "v_cvt_u32_f64",
+		[GFX9_V_CVT_F64_U32] = "v_cvt_f64_u32",
+		[GFX9_V_TRUNC_F64] = "v_trunc_f64",
+		[GFX9_V_CEIL_F64] = "v_ceil_f64",
+		[GFX9_V_RNDNE_F64] = "v_rndne_f64",
+		[GFX9_V_FLOOR_F64] = "v_floor_f64",
+		[GFX9_V_FRACT_F32] = "v_fract_f32",
+		[GFX9_V_TRUNC_F32] = "v_trunc_f32",
+		[GFX9_V_CEIL_F32] = "v_ceil_f32",
+		[GFX9_V_RNDNE_F32] = "v_rndne_f32",
+		[GFX9_V_FLOOR_F32] = "v_floor_f32",
+		[GFX9_V_EXP_F32] = "v_exp_f32",
+		[GFX9_V_LOG_F32] = "v_log_f32",
+		[GFX9_V_RCP_F32] = "v_rcp_f32",
+		[GFX9_V_RCP_IFLAG_F32] = "v_rcp_iflag_f32",
+		[GFX9_V_RSQ_F32] = "v_rsq_f32",
+		[GFX9_V_RCP_F64] = "v_rcp_f64",
+		[GFX9_V_RSQ_F64] = "v_rsq_f64",
+		[GFX9_V_SQRT_F32] = "v_sqrt_f32",
+		[GFX9_V_SQRT_F64] = "v_sqrt_f64",
+		[GFX9_V_SIN_F32] = "v_sin_f32",
+		[GFX9_V_COS_F32] = "v_cos_f32",
+		[GFX9_V_NOT_B32] = "v_not_b32",
+		[GFX9_V_BFREV_B32] = "v_bfrev_b32",
+		[GFX9_V_FFBH_U32] = "v_ffbh_u32",
+		[GFX9_V_FFBL_B32] = "v_ffbl_b32",
+		[GFX9_V_FFBH_I32] = "v_ffbh_i32",
+		[GFX9_V_FREXP_EXP_I32_F64] = "v_frexp_exp_i32_f64",
+		[GFX9_V_FREXP_MANT_F64] = "v_frexp_mant_f64",
+		[GFX9_V_FRACT_F64] = "v_fract_f64",
+		[GFX9_V_FREXP_EXP_I32_F32] = "v_frexp_exp_i32_f32",
+		[GFX9_V_FREXP_MANT_F32] = "v_frexp_mant_f32",
+		[GFX9_V_CLREXCP] = "v_clrexcp",
+		[GFX9_V_SCREEN_PARTITION_4SE_B32] = "v_screen_partition_4se_b32",
+		[GFX9_V_CVT_F16_U16] = "v_cvt_f16_u16",
+		[GFX9_V_CVT_F16_I16] = "v_cvt_f16_i16",
+		[GFX9_V_CVT_U16_F16] = "v_cvt_u16_f16",
+		[GFX9_V_CVT_I16_F16] = "v_cvt_i16_f16",
+		[GFX9_V_RCP_F16] = "v_rcp_f16",
+		[GFX9_V_SQRT_F16] = "v_sqrt_f16",
+		[GFX9_V_RSQ_F16] = "v_rsq_f16",
+		[GFX9_V_LOG_F16] = "v_log_f16",
+		[GFX9_V_EXP_F16] = "v_exp_f16",
+		[GFX9_V_FREXP_MANT_F16] = "v_frexp_mant_f16",
+		[GFX9_V_FREXP_EXP_I16_F16] = "v_frexp_exp_i16_f16",
+		[GFX9_V_FLOOR_F16] = "v_floor_f16",
+		[GFX9_V_CEIL_F16] = "v_ceil_f16",
+		[GFX9_V_TRUNC_F16] = "v_trunc_f16",
+		[GFX9_V_RNDNE_F16] = "v_rndne_f16",
+		[GFX9_V_FRACT_F16] = "v_fract_f16",
+		[GFX9_V_SIN_F16] = "v_sin_f16",
+		[GFX9_V_COS_F16] = "v_cos_f16",
+		[GFX9_V_EXP_LEGACY_F32] = "v_exp_legacy_f32",
+		[GFX9_V_LOG_LEGACY_F32] = "v_log_legacy_f32",
+		[GFX9_V_CVT_NORM_I16_F16] = "v_cvt_norm_i16_f16",
+		[GFX9_V_CVT_NORM_U16_F16] = "v_cvt_norm_u16_f16",
+		[GFX9_V_SAT_PK_U8_I16] = "v_sat_pk_u8_i16",
+		[GFX9_V_SWAP_B32] = "v_swap_b32",
+	},
+	[AMDGCN_INSN_TYPE_VOPC] = {
+		[GFX9_V_CMP_CLASS_F32] = "v_cmp_class_f32",
+		[GFX9_V_CMPX_CLASS_F32] = "v_cmpx_class_f32",
+		[GFX9_V_CMP_CLASS_F64] = "v_cmp_class_f64",
+		[GFX9_V_CMPX_CLASS_F64] = "v_cmpx_class_f64",
+		[GFX9_V_CMP_CLASS_F16] = "v_cmp_class_f16",
+		[GFX9_V_CMPX_CLASS_F16] = "v_cmpx_class_f16",
+		[GFX9_V_CMP_F_F16] = "v_cmp_f_f16",
+		[GFX9_V_CMP_LT_F16] = "v_cmp_lt_f16",
+		[GFX9_V_CMP_EQ_F16] = "v_cmp_eq_f16",
+		[GFX9_V_CMP_LE_F16] = "v_cmp_le_f16",
+		[GFX9_V_CMP_GT_F16] = "v_cmp_gt_f16",
+		[GFX9_V_CMP_LG_F16] = "v_cmp_lg_f16",
+		[GFX9_V_CMP_GE_F16] = "v_cmp_ge_f16",
+		[GFX9_V_CMP_O_F16] = "v_cmp_o_f16",
+		[GFX9_V_CMP_U_F16] = "v_cmp_u_f16",
+		[GFX9_V_CMP_NGE_F16] = "v_cmp_nge_f16",
+		[GFX9_V_CMP_NLG_F16] = "v_cmp_nlg_f16",
+		[GFX9_V_CMP_NGT_F16] = "v_cmp_ngt_f16",
+		[GFX9_V_CMP_NLE_F16] = "v_cmp_nle_f16",
+		[GFX9_V_CMP_NEQ_F16] = "v_cmp_neq_f16",
+		[GFX9_V_CMP_NLT_F16] = "v_cmp_nlt_f16",
+		[GFX9_V_CMP_TRU_F16] = "v_cmp_tru_f16",
+		[GFX9_V_CMPX_F_F16] = "v_cmpx_f_f16",
+		[GFX9_V_CMPX_LT_F16] = "v_cmpx_lt_f16",
+		[GFX9_V_CMPX_EQ_F16] = "v_cmpx_eq_f16",
+		[GFX9_V_CMPX_LE_F16] = "v_cmpx_le_f16",
+		[GFX9_V_CMPX_GT_F16] = "v_cmpx_gt_f16",
+		[GFX9_V_CMPX_LG_F16] = "v_cmpx_lg_f16",
+		[GFX9_V_CMPX_GE_F16] = "v_cmpx_ge_f16",
+		[GFX9_V_CMPX_O_F16] = "v_cmpx_o_f16",
+		[GFX9_V_CMPX_U_F16] = "v_cmpx_u_f16",
+		[GFX9_V_CMPX_NGE_F16] = "v_cmpx_nge_f16",
+		[GFX9_V_CMPX_NLG_F16] = "v_cmpx_nlg_f16",
+		[GFX9_V_CMPX_NGT_F16] = "v_cmpx_ngt_f16",
+		[GFX9_V_CMPX_NLE_F16] = "v_cmpx_nle_f16",
+		[GFX9_V_CMPX_NEQ_F16] = "v_cmpx_neq_f16",
+		[GFX9_V_CMPX_NLT_F16] = "v_cmpx_nlt_f16",
+		[GFX9_V_CMPX_TRU_F16] = "v_cmpx_tru_f16",
+		[GFX9_V_CMP_F_F32] = "v_cmp_f_f32",
+		[GFX9_V_CMP_LT_F32] = "v_cmp_lt_f32",
+		[GFX9_V_CMP_EQ_F32] = "v_cmp_eq_f32",
+		[GFX9_V_CMP_LE_F32] = "v_cmp_le_f32",
+		[GFX9_V_CMP_GT_F32] = "v_cmp_gt_f32",
+		[GFX9_V_CMP_LG_F32] = "v_cmp_lg_f32",
+		[GFX9_V_CMP_GE_F32] = "v_cmp_ge_f32",
+		[GFX9_V_CMP_O_F32] = "v_cmp_o_f32",
+		[GFX9_V_CMP_U_F32] = "v_cmp_u_f32",
+		[GFX9_V_CMP_NGE_F32] = "v_cmp_nge_f32",
+		[GFX9_V_CMP_NLG_F32] = "v_cmp_nlg_f32",
+		[GFX9_V_CMP_NGT_F32] = "v_cmp_ngt_f32",
+		[GFX9_V_CMP_NLE_F32] = "v_cmp_nle_f32",
+		[GFX9_V_CMP_NEQ_F32] = "v_cmp_neq_f32",
+		[GFX9_V_CMP_NLT_F32] = "v_cmp_nlt_f32",
+		[GFX9_V_CMP_TRU_F32] = "v_cmp_tru_f32",
+		[GFX9_V_CMPX_F_F32] = "v_cmpx_f_f32",
+		[GFX9_V_CMPX_LT_F32] = "v_cmpx_lt_f32",
+		[GFX9_V_CMPX_EQ_F32] = "v_cmpx_eq_f32",
+		[GFX9_V_CMPX_LE_F32] = "v_cmpx_le_f32",
+		[GFX9_V_CMPX_GT_F32] = "v_cmpx_gt_f32",
+		[GFX9_V_CMPX_LG_F32] = "v_cmpx_lg_f32",
+		[GFX9_V_CMPX_GE_F32] = "v_cmpx_ge_f32",
+		[GFX9_V_CMPX_O_F32] = "v_cmpx_o_f32",
+		[GFX9_V_CMPX_U_F32] = "v_cmpx_u_f32",
+		[GFX9_V_CMPX_NGE_F32] = "v_cmpx_nge_f32",
+		[GFX9_V_CMPX_NLG_F32] = "v_cmpx_nlg_f32",
+		[GFX9_V_CMPX_NGT_F32] = "v_cmpx_ngt_f32",
+		[GFX9_V_CMPX_NLE_F32] = "v_cmpx_nle_f32",
+		[GFX9_V_CMPX_NEQ_F32] = "v_cmpx_neq_f32",
+		[GFX9_V_CMPX_NLT_F32] = "v_cmpx_nlt_f32",
+		[GFX9_V_CMPX_TRU_F32] = "v_cmpx_tru_f32",
+		[GFX9_V_CMP_F_F64] = "v_cmp_f_f64",
+		[GFX9_V_CMP_LT_F64] = "v_cmp_lt_f64",
+		[GFX9_V_CMP_EQ_F64] = "v_cmp_eq_f64",
+		[GFX9_V_CMP_LE_F64] = "v_cmp_le_f64",
+		[GFX9_V_CMP_GT_F64] = "v_cmp_gt_f64",
+		[GFX9_V_CMP_LG_F64] = "v_cmp_lg_f64",
+		[GFX9_V_CMP_GE_F64] = "v_cmp_ge_f64",
+		[GFX9_V_CMP_O_F64] = "v_cmp_o_f64",
+		[GFX9_V_CMP_U_F64] = "v_cmp_u_f64",
+		[GFX9_V_CMP_NGE_F64] = "v_cmp_nge_f64",
+		[GFX9_V_CMP_NLG_F64] = "v_cmp_nlg_f64",
+		[GFX9_V_CMP_NGT_F64] = "v_cmp_ngt_f64",
+		[GFX9_V_CMP_NLE_F64] = "v_cmp_nle_f64",
+		[GFX9_V_CMP_NEQ_F64] = "v_cmp_neq_f64",
+		[GFX9_V_CMP_NLT_F64] = "v_cmp_nlt_f64",
+		[GFX9_V_CMP_TRU_F64] = "v_cmp_tru_f64",
+		[GFX9_V_CMPX_F_F64] = "v_cmpx_f_f64",
+		[GFX9_V_CMPX_LT_F64] = "v_cmpx_lt_f64",
+		[GFX9_V_CMPX_EQ_F64] = "v_cmpx_eq_f64",
+		[GFX9_V_CMPX_LE_F64] = "v_cmpx_le_f64",
+		[GFX9_V_CMPX_GT_F64] = "v_cmpx_gt_f64",
+		[GFX9_V_CMPX_LG_F64] = "v_cmpx_lg_f64",
+		[GFX9_V_CMPX_GE_F64] = "v_cmpx_ge_f64",
+		[GFX9_V_CMPX_O_F64] = "v_cmpx_o_f64",
+		[GFX9_V_CMPX_U_F64] = "v_cmpx_u_f64",
+		[GFX9_V_CMPX_NGE_F64] = "v_cmpx_nge_f64",
+		[GFX9_V_CMPX_NLG_F64] = "v_cmpx_nlg_f64",
+		[GFX9_V_CMPX_NGT_F64] = "v_cmpx_ngt_f64",
+		[GFX9_V_CMPX_NLE_F64] = "v_cmpx_nle_f64",
+		[GFX9_V_CMPX_NEQ_F64] = "v_cmpx_neq_f64",
+		[GFX9_V_CMPX_NLT_F64] = "v_cmpx_nlt_f64",
+		[GFX9_V_CMPX_TRU_F64] = "v_cmpx_tru_f64",
+		[GFX9_V_CMP_F_I16] = "v_cmp_f_i16",
+		[GFX9_V_CMP_LT_I16] = "v_cmp_lt_i16",
+		[GFX9_V_CMP_EQ_I16] = "v_cmp_eq_i16",
+		[GFX9_V_CMP_LE_I16] = "v_cmp_le_i16",
+		[GFX9_V_CMP_GT_I16] = "v_cmp_gt_i16",
+		[GFX9_V_CMP_NE_I16] = "v_cmp_ne_i16",
+		[GFX9_V_CMP_GE_I16] = "v_cmp_ge_i16",
+		[GFX9_V_CMP_T_I16] = "v_cmp_t_i16",
+		[GFX9_V_CMP_F_U16] = "v_cmp_f_u16",
+		[GFX9_V_CMP_LT_U16] = "v_cmp_lt_u16",
+		[GFX9_V_CMP_EQ_U16] = "v_cmp_eq_u16",
+		[GFX9_V_CMP_LE_U16] = "v_cmp_le_u16",
+		[GFX9_V_CMP_GT_U16] = "v_cmp_gt_u16",
+		[GFX9_V_CMP_NE_U16] = "v_cmp_ne_u16",
+		[GFX9_V_CMP_GE_U16] = "v_cmp_ge_u16",
+		[GFX9_V_CMP_T_U16] = "v_cmp_t_u16",
+		[GFX9_V_CMPX_F_I16] = "v_cmpx_f_i16",
+		[GFX9_V_CMPX_LT_I16] = "v_cmpx_lt_i16",
+		[GFX9_V_CMPX_EQ_I16] = "v_cmpx_eq_i16",
+		[GFX9_V_CMPX_LE_I16] = "v_cmpx_le_i16",
+		[GFX9_V_CMPX_GT_I16] = "v_cmpx_gt_i16",
+		[GFX9_V_CMPX_NE_I16] = "v_cmpx_ne_i16",
+		[GFX9_V_CMPX_GE_I16] = "v_cmpx_ge_i16",
+		[GFX9_V_CMPX_T_I16] = "v_cmpx_t_i16",
+		[GFX9_V_CMPX_F_U16] = "v_cmpx_f_u16",
+		[GFX9_V_CMPX_LT_U16] = "v_cmpx_lt_u16",
+		[GFX9_V_CMPX_EQ_U16] = "v_cmpx_eq_u16",
+		[GFX9_V_CMPX_LE_U16] = "v_cmpx_le_u16",
+		[GFX9_V_CMPX_GT_U16] = "v_cmpx_gt_u16",
+		[GFX9_V_CMPX_NE_U16] = "v_cmpx_ne_u16",
+		[GFX9_V_CMPX_GE_U16] = "v_cmpx_ge_u16",
+		[GFX9_V_CMPX_T_U16] = "v_cmpx_t_u16",
+		[GFX9_V_CMP_F_I32] = "v_cmp_f_i32",
+		[GFX9_V_CMP_LT_I32] = "v_cmp_lt_i32",
+		[GFX9_V_CMP_EQ_I32] = "v_cmp_eq_i32",
+		[GFX9_V_CMP_LE_I32] = "v_cmp_le_i32",
+		[GFX9_V_CMP_GT_I32] = "v_cmp_gt_i32",
+		[GFX9_V_CMP_NE_I32] = "v_cmp_ne_i32",
+		[GFX9_V_CMP_GE_I32] = "v_cmp_ge_i32",
+		[GFX9_V_CMP_T_I32] = "v_cmp_t_i32",
+		[GFX9_V_CMP_F_U32] = "v_cmp_f_u32",
+		[GFX9_V_CMP_LT_U32] = "v_cmp_lt_u32",
+		[GFX9_V_CMP_EQ_U32] = "v_cmp_eq_u32",
+		[GFX9_V_CMP_LE_U32] = "v_cmp_le_u32",
+		[GFX9_V_CMP_GT_U32] = "v_cmp_gt_u32",
+		[GFX9_V_CMP_NE_U32] = "v_cmp_ne_u32",
+		[GFX9_V_CMP_GE_U32] = "v_cmp_ge_u32",
+		[GFX9_V_CMP_T_U32] = "v_cmp_t_u32",
+		[GFX9_V_CMPX_F_I32] = "v_cmpx_f_i32",
+		[GFX9_V_CMPX_LT_I32] = "v_cmpx_lt_i32",
+		[GFX9_V_CMPX_EQ_I32] = "v_cmpx_eq_i32",
+		[GFX9_V_CMPX_LE_I32] = "v_cmpx_le_i32",
+		[GFX9_V_CMPX_GT_I32] = "v_cmpx_gt_i32",
+		[GFX9_V_CMPX_NE_I32] = "v_cmpx_ne_i32",
+		[GFX9_V_CMPX_GE_I32] = "v_cmpx_ge_i32",
+		[GFX9_V_CMPX_T_I32] = "v_cmpx_t_i32",
+		[GFX9_V_CMPX_F_U32] = "v_cmpx_f_u32",
+		[GFX9_V_CMPX_LT_U32] = "v_cmpx_lt_u32",
+		[GFX9_V_CMPX_EQ_U32] = "v_cmpx_eq_u32",
+		[GFX9_V_CMPX_LE_U32] = "v_cmpx_le_u32",
+		[GFX9_V_CMPX_GT_U32] = "v_cmpx_gt_u32",
+		[GFX9_V_CMPX_NE_U32] = "v_cmpx_ne_u32",
+		[GFX9_V_CMPX_GE_U32] = "v_cmpx_ge_u32",
+		[GFX9_V_CMPX_T_U32] = "v_cmpx_t_u32",
+		[GFX9_V_CMP_F_I64] = "v_cmp_f_i64",
+		[GFX9_V_CMP_LT_I64] = "v_cmp_lt_i64",
+		[GFX9_V_CMP_EQ_I64] = "v_cmp_eq_i64",
+		[GFX9_V_CMP_LE_I64] = "v_cmp_le_i64",
+		[GFX9_V_CMP_GT_I64] = "v_cmp_gt_i64",
+		[GFX9_V_CMP_NE_I64] = "v_cmp_ne_i64",
+		[GFX9_V_CMP_GE_I64] = "v_cmp_ge_i64",
+		[GFX9_V_CMP_T_I64] = "v_cmp_t_i64",
+		[GFX9_V_CMP_F_U64] = "v_cmp_f_u64",
+		[GFX9_V_CMP_LT_U64] = "v_cmp_lt_u64",
+		[GFX9_V_CMP_EQ_U64] = "v_cmp_eq_u64",
+		[GFX9_V_CMP_LE_U64] = "v_cmp_le_u64",
+		[GFX9_V_CMP_GT_U64] = "v_cmp_gt_u64",
+		[GFX9_V_CMP_NE_U64] = "v_cmp_ne_u64",
+		[GFX9_V_CMP_GE_U64] = "v_cmp_ge_u64",
+		[GFX9_V_CMP_T_U64] = "v_cmp_t_u64",
+		[GFX9_V_CMPX_F_I64] = "v_cmpx_f_i64",
+		[GFX9_V_CMPX_LT_I64] = "v_cmpx_lt_i64",
+		[GFX9_V_CMPX_EQ_I64] = "v_cmpx_eq_i64",
+		[GFX9_V_CMPX_LE_I64] = "v_cmpx_le_i64",
+		[GFX9_V_CMPX_GT_I64] = "v_cmpx_gt_i64",
+		[GFX9_V_CMPX_NE_I64] = "v_cmpx_ne_i64",
+		[GFX9_V_CMPX_GE_I64] = "v_cmpx_ge_i64",
+		[GFX9_V_CMPX_T_I64] = "v_cmpx_t_i64",
+		[GFX9_V_CMPX_F_U64] = "v_cmpx_f_u64",
+		[GFX9_V_CMPX_LT_U64] = "v_cmpx_lt_u64",
+		[GFX9_V_CMPX_EQ_U64] = "v_cmpx_eq_u64",
+		[GFX9_V_CMPX_LE_U64] = "v_cmpx_le_u64",
+		[GFX9_V_CMPX_GT_U64] = "v_cmpx_gt_u64",
+		[GFX9_V_CMPX_NE_U64] = "v_cmpx_ne_u64",
+		[GFX9_V_CMPX_GE_U64] = "v_cmpx_ge_u64",
+		[GFX9_V_CMPX_T_U64] = "v_cmpx_t_u64",
+	},
+	[AMDGCN_INSN_TYPE_VOP3A] = {
+		[GFX9_V_MAD_LEGACY_F32] = "v_mad_legacy_f32",
+		[GFX9_V_MAD_F32] = "v_mad_f32",
+		[GFX9_V_MAD_I32_I24] = "v_mad_i32_i24",
+		[GFX9_V_MAD_U32_U24] = "v_mad_u32_u24",
+		[GFX9_V_CUBEID_F32] = "v_cubeid_f32",
+		[GFX9_V_CUBESC_F32] = "v_cubesc_f32",
+		[GFX9_V_CUBETC_F32] = "v_cubetc_f32",
+		[GFX9_V_CUBEMA_F32] = "v_cubema_f32",
+		[GFX9_V_BFE_U32] = "v_bfe_u32",
+		[GFX9_V_BFE_I32] = "v_bfe_i32",
+		[GFX9_V_BFI_B32] = "v_bfi_b32",
+		[GFX9_V_FMA_F32] = "v_fma_f32",
+		[GFX9_V_FMA_F64] = "v_fma_f64",
+		[GFX9_V_LERP_U8] = "v_lerp_u8",
+		[GFX9_V_ALIGNBIT_B32] = "v_alignbit_b32",
+		[GFX9_V_ALIGNBYTE_B32] = "v_alignbyte_b32",
+		[GFX9_V_MIN3_F32] = "v_min3_f32",
+		[GFX9_V_MIN3_I32] = "v_min3_i32",
+		[GFX9_V_MIN3_U32] = "v_min3_u32",
+		[GFX9_V_MAX3_F32] = "v_max3_f32",
+		[GFX9_V_MAX3_I32] = "v_max3_i32",
+		[GFX9_V_MAX3_U32] = "v_max3_u32",
+		[GFX9_V_MED3_F32] = "v_med3_f32",
+		[GFX9_V_MED3_I32] = "v_med3_i32",
+		[GFX9_V_MED3_U32] = "v_med3_u32",
+		[GFX9_V_SAD_U8] = "v_sad_u8",
+		[GFX9_V_SAD_HI_U8] = "v_sad_hi_u8",
+		[GFX9_V_SAD_U16] = "v_sad_u16",
+		[GFX9_V_SAD_U32] = "v_sad_u32",
+		[GFX9_V_CVT_PK_U8_F32] = "v_cvt_pk_u8_f32",
+		[GFX9_V_DIV_FIXUP_F32] = "v_div_fixup_f32",
+		[GFX9_V_DIV_FIXUP_F64] = "v_div_fixup_f64",
+		[GFX9_V_DIV_FMAS_F32] = "v_div_fmas_f32",
+		[GFX9_V_DIV_FMAS_F64] = "v_div_fmas_f64",
+		[GFX9_V_MSAD_U8] = "v_msad_u8",
+		[GFX9_V_QSAD_PK_U16_U8] = "v_qsad_pk_u16_u8",
+		[GFX9_V_MQSAD_PK_U16_U8] = "v_mqsad_pk_u16_u8",
+		[GFX9_V_MQSAD_U32_U8] = "v_mqsad_u32_u8",
+		[GFX9_V_MAD_LEGACY_F16] = "v_mad_legacy_f16",
+		[GFX9_V_MAD_LEGACY_U16] = "v_mad_legacy_u16",
+		[GFX9_V_MAD_LEGACY_I16] = "v_mad_legacy_i16",
+		[GFX9_V_PERM_B32] = "v_perm_b32",
+		[GFX9_V_FMA_LEGACY_F16] = "v_fma_legacy_f16",
+		[GFX9_V_DIV_FIXUP_LEGACY_F16] = "v_div_fixup_legacy_f16",
+		[GFX9_V_CVT_PKACCUM_U8_F32] = "v_cvt_pkaccum_u8_f32",
+		[GFX9_V_MAD_U32_U16] = "v_mad_u32_u16",
+		[GFX9_V_MAD_I32_I16] = "v_mad_i32_i16",
+		[GFX9_V_XAD_U32] = "v_xad_u32",
+		[GFX9_V_MIN3_F16] = "v_min3_f16",
+		[GFX9_V_MIN3_I16] = "v_min3_i16",
+		[GFX9_V_MIN3_U16] = "v_min3_u16",
+		[GFX9_V_MAX3_F16] = "v_max3_f16",
+		[GFX9_V_MAX3_I16] = "v_max3_i16",
+		[GFX9_V_MAX3_U16] = "v_max3_u16",
+		[GFX9_V_MED3_F16] = "v_med3_f16",
+		[GFX9_V_MED3_I16] = "v_med3_i16",
+		[GFX9_V_MED3_U16] = "v_med3_u16",
+		[GFX9_V_LSHL_ADD_U32] = "v_lshl_add_u32",
+		[GFX9_V_ADD_LSHL_U32] = "v_add_lshl_u32",
+		[GFX9_V_ADD3_U32] = "v_add3_u32",
+		[GFX9_V_LSHL_OR_B32] = "v_lshl_or_b32",
+		[GFX9_V_AND_OR_B32] = "v_and_or_b32",
+		[GFX9_V_OR3_B32] = "v_or3_b32",
+		[GFX9_V_MAD_F16] = "v_mad_f16",
+		[GFX9_V_MAD_U16] = "v_mad_u16",
+		[GFX9_V_MAD_I16] = "v_mad_i16",
+		[GFX9_V_FMA_F16] = "v_fma_f16",
+		[GFX9_V_DIV_FIXUP_F16] = "v_div_fixup_f16",
+		[GFX9_V_INTERP_P1LL_F16] = "v_interp_p1ll_f16",
+		[GFX9_V_INTERP_P1LV_F16] = "v_interp_p1lv_f16",
+		[GFX9_V_INTERP_P2_LEGACY_F16] = "v_interp_p2_legacy_f16",
+		[GFX9_V_INTERP_P2_F16] = "v_interp_p2_f16",
+		[GFX9_V_ADD_F64] = "v_add_f64",
+		[GFX9_V_MUL_F64] = "v_mul_f64",
+		[GFX9_V_MIN_F64] = "v_min_f64",
+		[GFX9_V_MAX_F64] = "v_max_f64",
+		[GFX9_V_LDEXP_F64] = "v_ldexp_f64",
+		[GFX9_V_MUL_LO_U32] = "v_mul_lo_u32",
+		[GFX9_V_MUL_HI_U32] = "v_mul_hi_u32",
+		[GFX9_V_MUL_HI_I32] = "v_mul_hi_i32",
+		[GFX9_V_LDEXP_F32] = "v_ldexp_f32",
+		[GFX9_V_READLANE_B32] = "v_readlane_b32",
+		[GFX9_V_WRITELANE_B32] = "v_writelane_b32",
+		[GFX9_V_BCNT_U32_B32] = "v_bcnt_u32_b32",
+		[GFX9_V_MBCNT_LO_U32_B32] = "v_mbcnt_lo_u32_b32",
+		[GFX9_V_MBCNT_HI_U32_B32] = "v_mbcnt_hi_u32_b32",
+		[GFX9_V_LSHLREV_B64] = "v_lshlrev_b64",
+		[GFX9_V_LSHRREV_B64] = "v_lshrrev_b64",
+		[GFX9_V_ASHRREV_I64] = "v_ashrrev_i64",
+		[GFX9_V_TRIG_PREOP_F64] = "v_trig_preop_f64",
+		[GFX9_V_BFM_B32] = "v_bfm_b32",
+		[GFX9_V_CVT_PKNORM_I16_F32] = "v_cvt_pknorm_i16_f32",
+		[GFX9_V_CVT_PKNORM_U16_F32] = "v_cvt_pknorm_u16_f32",
+		[GFX9_V_CVT_PKRTZ_F16_F32] = "v_cvt_pkrtz_f16_f32",
+		[GFX9_V_CVT_PK_U16_U32] = "v_cvt_pk_u16_u32",
+		[GFX9_V_CVT_PK_I16_I32] = "v_cvt_pk_i16_i32",
+		[GFX9_V_CVT_PKNORM_I16_F16] = "v_cvt_pknorm_i16_f16",
+		[GFX9_V_CVT_PKNORM_U16_F16] = "v_cvt_pknorm_u16_f16",
+		[GFX9_V_ADD_I32] = "v_add_i32",
+		[GFX9_V_SUB_I32] = "v_sub_i32",
+		[GFX9_V_ADD_I16] = "v_add_i16",
+		[GFX9_V_SUB_I16] = "v_sub_i16",
+		[GFX9_V_PACK_B32_F16] = "v_pack_b32_f16",
+	},
+	[AMDGCN_INSN_TYPE_VOP3B] = {
+		[GFX9_V_DIV_SCALE_F64] = "v_div_scale_f64",
+		[GFX9_V_MAD_U64_U32] = "v_mad_u64_u32",
+		[GFX9_V_MAD_I64_I32] = "v_mad_i64_i32",
+	},
+	[AMDGCN_INSN_TYPE_VOP3P] = {
+		[GFX9_V_PK_MAD_I16] = "v_pk_mad_i16",
+		[GFX9_V_PK_MUL_LO_U16] = "v_pk_mul_lo_u16",
+		[GFX9_V_PK_ADD_I16] = "v_pk_add_i16",
+		[GFX9_V_PK_SUB_I16] = "v_pk_sub_i16",
+		[GFX9_V_PK_LSHLREV_B16] = "v_pk_lshlrev_b16",
+		[GFX9_V_PK_LSHRREV_B16] = "v_pk_lshrrev_b16",
+		[GFX9_V_PK_ASHRREV_I16] = "v_pk_ashrrev_i16",
+		[GFX9_V_PK_MAX_I16] = "v_pk_max_i16",
+		[GFX9_V_PK_MIN_I16] = "v_pk_min_i16",
+		[GFX9_V_PK_MAD_U16] = "v_pk_mad_u16",
+		[GFX9_V_PK_ADD_U16] = "v_pk_add_u16",
+		[GFX9_V_PK_SUB_U16] = "v_pk_sub_u16",
+		[GFX9_V_PK_MAX_U16] = "v_pk_max_u16",
+		[GFX9_V_PK_MIN_U16] = "v_pk_min_u16",
+		[GFX9_V_PK_FMA_F16] = "v_pk_fma_f16",
+		[GFX9_V_PK_ADD_F16] = "v_pk_add_f16",
+		[GFX9_V_PK_MUL_F16] = "v_pk_mul_f16",
+		[GFX9_V_PK_MIN_F16] = "v_pk_min_f16",
+		[GFX9_V_PK_MAX_F16] = "v_pk_max_f16",
+		[GFX9_V_MAD_MIX_F32] = "v_mad_mix_f32",
+		[GFX9_V_MAD_MIXLO_F16] = "v_mad_mixlo_f16",
+		[GFX9_V_MAD_MIXHI_F16] = "v_mad_mixhi_f16",
+	},
+	[AMDGCN_INSN_TYPE_MUBUF] = {
+		[GFX9_BUFFER_LOAD_FORMAT_X] = "buffer_load_format_x",
+		[GFX9_BUFFER_LOAD_FORMAT_XY] = "buffer_load_format_xy",
+		[GFX9_BUFFER_LOAD_FORMAT_XYZ] = "buffer_load_format_xyz",
+		[GFX9_BUFFER_LOAD_FORMAT_XYZW] = "buffer_load_format_xyzw",
+		[GFX9_BUFFER_STORE_FORMAT_X] = "buffer_store_format_x",
+		[GFX9_BUFFER_STORE_FORMAT_XY] = "buffer_store_format_xy",
+		[GFX9_BUFFER_STORE_FORMAT_XYZ] = "buffer_store_format_xyz",
+		[GFX9_BUFFER_STORE_FORMAT_XYZW] = "buffer_store_format_xyzw",
+		[GFX9_BUFFER_LOAD_FORMAT_D16_X] = "buffer_load_format_d16_x",
+		[GFX9_BUFFER_LOAD_FORMAT_D16_XY] = "buffer_load_format_d16_xy",
+		[GFX9_BUFFER_LOAD_FORMAT_D16_XYZ] = "buffer_load_format_d16_xyz",
+		[GFX9_BUFFER_LOAD_FORMAT_D16_XYZW] = "buffer_load_format_d16_xyzw",
+		[GFX9_BUFFER_STORE_FORMAT_D16_X] = "buffer_store_format_d16_x",
+		[GFX9_BUFFER_STORE_FORMAT_D16_XY] = "buffer_store_format_d16_xy",
+		[GFX9_BUFFER_STORE_FORMAT_D16_XYZ] = "buffer_store_format_d16_xyz",
+		[GFX9_BUFFER_STORE_FORMAT_D16_XYZW] = "buffer_store_format_d16_xyzw",
+		[GFX9_BUFFER_LOAD_UBYTE] = "buffer_load_ubyte",
+		[GFX9_BUFFER_LOAD_SBYTE] = "buffer_load_sbyte",
+		[GFX9_BUFFER_LOAD_USHORT] = "buffer_load_ushort",
+		[GFX9_BUFFER_LOAD_SSHORT] = "buffer_load_sshort",
+		[GFX9_BUFFER_LOAD_DWORD] = "buffer_load_dword",
+		[GFX9_BUFFER_LOAD_DWORDX2] = "buffer_load_dwordx2",
+		[GFX9_BUFFER_LOAD_DWORDX3] = "buffer_load_dwordx3",
+		[GFX9_BUFFER_LOAD_DWORDX4] = "buffer_load_dwordx4",
+		[GFX9_BUFFER_STORE_BYTE] = "buffer_store_byte",
+		[GFX9_BUFFER_STORE_BYTE_D16_HI] = "buffer_store_byte_d16_hi",
+		[GFX9_BUFFER_STORE_SHORT] = "buffer_store_short",
+		[GFX9_BUFFER_STORE_SHORT_D16_HI] = "buffer_store_short_d16_hi",
+		[GFX9_BUFFER_STORE_DWORD] = "buffer_store_dword",
+		[GFX9_BUFFER_STORE_DWORDX2] = "buffer_store_dwordx2",
+		[GFX9_BUFFER_STORE_DWORDX3] = "buffer_store_dwordx3",
+		[GFX9_BUFFER_STORE_DWORDX4] = "buffer_store_dwordx4",
+		[GFX9_BUFFER_LOAD_UBYTE_D16] = "buffer_load_ubyte_d16",
+		[GFX9_BUFFER_LOAD_UBYTE_D16_HI] = "buffer_load_ubyte_d16_hi",
+		[GFX9_BUFFER_LOAD_SBYTE_D16] = "buffer_load_sbyte_d16",
+		[GFX9_BUFFER_LOAD_SBYTE_D16_HI] = "buffer_load_sbyte_d16_hi",
+		[GFX9_BUFFER_LOAD_SHORT_D16] = "buffer_load_short_d16",
+		[GFX9_BUFFER_LOAD_SHORT_D16_HI] = "buffer_load_short_d16_hi",
+		[GFX9_BUFFER_LOAD_FORMAT_D16_HI_X] = "buffer_load_format_d16_hi_x",
+		[GFX9_BUFFER_STORE_FORMAT_D16_HI_X] = "buffer_store_format_d16_hi_x",
+		[GFX9_BUFFER_STORE_LDS_DWORD] = "buffer_store_lds_dword",
+		[GFX9_BUFFER_WBINVL1] = "buffer_wbinvl1",
+		[GFX9_BUFFER_WBINVL1_VOL] = "buffer_wbinvl1_vol",
+		[GFX9_BUFFER_ATOMIC_SWAP] = "buffer_atomic_swap",
+		[GFX9_BUFFER_ATOMIC_CMPSWAP] = "buffer_atomic_cmpswap",
+		[GFX9_BUFFER_ATOMIC_ADD] = "buffer_atomic_add",
+		[GFX9_BUFFER_ATOMIC_SUB] = "buffer_atomic_sub",
+		[GFX9_BUFFER_ATOMIC_SMIN] = "buffer_atomic_smin",
+		[GFX9_BUFFER_ATOMIC_UMIN] = "buffer_atomic_umin",
+		[GFX9_BUFFER_ATOMIC_SMAX] = "buffer_atomic_smax",
+		[GFX9_BUFFER_ATOMIC_UMAX] = "buffer_atomic_umax",
+		[GFX9_BUFFER_ATOMIC_AND] = "buffer_atomic_and",
+		[GFX9_BUFFER_ATOMIC_OR] = "buffer_atomic_or",
+		[GFX9_BUFFER_ATOMIC_XOR] = "buffer_atomic_xor",
+		[GFX9_BUFFER_ATOMIC_INC] = "buffer_atomic_inc",
+		[GFX9_BUFFER_ATOMIC_DEC] = "buffer_atomic_dec",
+		[GFX9_BUFFER_ATOMIC_SWAP_X2] = "buffer_atomic_swap_x2",
+		[GFX9_BUFFER_ATOMIC_CMPSWAP_X2] = "buffer_atomic_cmpswap_x2",
+		[GFX9_BUFFER_ATOMIC_ADD_X2] = "buffer_atomic_add_x2",
+		[GFX9_BUFFER_ATOMIC_SUB_X2] = "buffer_atomic_sub_x2",
+		[GFX9_BUFFER_ATOMIC_SMIN_X2] = "buffer_atomic_smin_x2",
+		[GFX9_BUFFER_ATOMIC_UMIN_X2] = "buffer_atomic_umin_x2",
+		[GFX9_BUFFER_ATOMIC_SMAX_X2] = "buffer_atomic_smax_x2",
+		[GFX9_BUFFER_ATOMIC_UMAX_X2] = "buffer_atomic_umax_x2",
+		[GFX9_BUFFER_ATOMIC_AND_X2] = "buffer_atomic_and_x2",
+		[GFX9_BUFFER_ATOMIC_OR_X2] = "buffer_atomic_or_x2",
+		[GFX9_BUFFER_ATOMIC_XOR_X2] = "buffer_atomic_xor_x2",
+		[GFX9_BUFFER_ATOMIC_INC_X2] = "buffer_atomic_inc_x2",
+		[GFX9_BUFFER_ATOMIC_DEC_X2] = "buffer_atomic_dec_x2",
+	},
+	[AMDGCN_INSN_TYPE_FLAT] = {
+		[GFX9_GLOBAL_LOAD_UBYTE] = "global_load_ubyte",
+		[GFX9_GLOBAL_LOAD_SBYTE] = "global_load_sbyte",
+		[GFX9_GLOBAL_LOAD_USHORT] = "global_load_ushort",
+		[GFX9_GLOBAL_LOAD_SSHORT] = "global_load_sshort",
+		[GFX9_GLOBAL_LOAD_DWORD] = "global_load_dword",
+		[GFX9_GLOBAL_LOAD_DWORDX2] = "global_load_dwordx2",
+		[GFX9_GLOBAL_LOAD_DWORDX3] = "global_load_dwordx3",
+		[GFX9_GLOBAL_LOAD_DWORDX4] = "global_load_dwordx4",
+		[GFX9_GLOBAL_STORE_BYTE] = "global_store_byte",
+		[GFX9_GLOBAL_STORE_BYTE_D16_HI] = "global_store_byte_d16_hi",
+		[GFX9_GLOBAL_STORE_SHORT] = "global_store_short",
+		[GFX9_GLOBAL_STORE_SHORT_D16_HI] = "global_store_short_d16_hi",
+		[GFX9_GLOBAL_STORE_DWORD] = "global_store_dword",
+		[GFX9_GLOBAL_STORE_DWORDX2] = "global_store_dwordx2",
+		[GFX9_GLOBAL_STORE_DWORDX3] = "global_store_dwordx3",
+		[GFX9_GLOBAL_STORE_DWORDX4] = "global_store_dwordx4",
+		[GFX9_GLOBAL_LOAD_UBYTE_D16] = "global_load_ubyte_d16",
+		[GFX9_GLOBAL_LOAD_UBYTE_D16_HI] = "global_load_ubyte_d16_hi",
+		[GFX9_GLOBAL_LOAD_SBYTE_D16] = "global_load_sbyte_d16",
+		[GFX9_GLOBAL_LOAD_SBYTE_D16_HI] = "global_load_sbyte_d16_hi",
+		[GFX9_GLOBAL_LOAD_SHORT_D16] = "global_load_short_d16",
+		[GFX9_GLOBAL_LOAD_SHORT_D16_HI] = "global_load_short_d16_hi",
+		[GFX9_GLOBAL_ATOMIC_SWAP] = "global_atomic_swap",
+		[GFX9_GLOBAL_ATOMIC_CMPSWAP] = "global_atomic_cmpswap",
+		[GFX9_GLOBAL_ATOMIC_ADD] = "global_atomic_add",
+		[GFX9_GLOBAL_ATOMIC_SUB] = "global_atomic_sub",
+		[GFX9_GLOBAL_ATOMIC_SMIN] = "global_atomic_smin",
+		[GFX9_GLOBAL_ATOMIC_UMIN] = "global_atomic_umin",
+		[GFX9_GLOBAL_ATOMIC_SMAX] = "global_atomic_smax",
+		[GFX9_GLOBAL_ATOMIC_UMAX] = "global_atomic_umax",
+		[GFX9_GLOBAL_ATOMIC_AND] = "global_atomic_and",
+		[GFX9_GLOBAL_ATOMIC_OR] = "global_atomic_or",
+		[GFX9_GLOBAL_ATOMIC_XOR] = "global_atomic_xor",
+		[GFX9_GLOBAL_ATOMIC_INC] = "global_atomic_inc",
+		[GFX9_GLOBAL_ATOMIC_DEC] = "global_atomic_dec",
+		[GFX9_GLOBAL_ATOMIC_SWAP_X2] = "global_atomic_swap_x2",
+		[GFX9_GLOBAL_ATOMIC_CMPSWAP_X2] = "global_atomic_cmpswap_x2",
+		[GFX9_GLOBAL_ATOMIC_ADD_X2] = "global_atomic_add_x2",
+		[GFX9_GLOBAL_ATOMIC_SUB_X2] = "global_atomic_sub_x2",
+		[GFX9_GLOBAL_ATOMIC_SMIN_X2] = "global_atomic_smin_x2",
+		[GFX9_GLOBAL_ATOMIC_UMIN_X2] = "global_atomic_umin_x2",
+		[GFX9_GLOBAL_ATOMIC_SMAX_X2] = "global_atomic_smax_x2",
+		[GFX9_GLOBAL_ATOMIC_UMAX_X2] = "global_atomic_umax_x2",
+		[GFX9_GLOBAL_ATOMIC_AND_X2] = "global_atomic_and_x2",
+		[GFX9_GLOBAL_ATOMIC_OR_X2] = "global_atomic_or_x2",
+		[GFX9_GLOBAL_ATOMIC_XOR_X2] = "global_atomic_xor_x2",
+		[GFX9_GLOBAL_ATOMIC_INC_X2] = "global_atomic_inc_x2",
+		[GFX9_GLOBAL_ATOMIC_DEC_X2] = "global_atomic_dec_x2",
+	},
+	[AMDGCN_INSN_TYPE_DS] = {
+		/* DS opcode names (Vega ISA 12.13, opcodes 0-255). The DS
+		 * table was entirely unpopulated, so DS instructions
+		 * disassembled with a blank mnemonic. Enum values were
+		 * ISA-verified (one known value bug at DS_CONDXCHG32_RTN_B64
+		 * is tracked separately).
+		 */
+		[GFX9_DS_ADD_U32] = "ds_add_u32",
+		[GFX9_DS_SUB_U32] = "ds_sub_u32",
+		[GFX9_DS_RSUB_U32] = "ds_rsub_u32",
+		[GFX9_DS_INC_U32] = "ds_inc_u32",
+		[GFX9_DS_DEC_U32] = "ds_dec_u32",
+		[GFX9_DS_MIN_I32] = "ds_min_i32",
+		[GFX9_DS_MAX_I32] = "ds_max_i32",
+		[GFX9_DS_MIN_U32] = "ds_min_u32",
+		[GFX9_DS_MAX_U32] = "ds_max_u32",
+		[GFX9_DS_AND_B32] = "ds_and_b32",
+		[GFX9_DS_OR_B32] = "ds_or_b32",
+		[GFX9_DS_XOR_B32] = "ds_xor_b32",
+		[GFX9_DS_MSKOR_B32] = "ds_mskor_b32",
+		[GFX9_DS_WRITE_B32] = "ds_write_b32",
+		[GFX9_DS_WRITE2_B32] = "ds_write2_b32",
+		[GFX9_DS_WRITE2ST64_B32] = "ds_write2st64_b32",
+		[GFX9_DS_CMPST_B32] = "ds_cmpst_b32",
+		[GFX9_DS_CMPST_F32] = "ds_cmpst_f32",
+		[GFX9_DS_MIN_F32] = "ds_min_f32",
+		[GFX9_DS_MAX_F32] = "ds_max_f32",
+		[GFX9_DS_NOP] = "ds_nop",
+		[GFX9_DS_ADD_F32] = "ds_add_f32",
+		[GFX9_DS_WRITE_ADDTID_B32] = "ds_write_addtid_b32",
+		[GFX9_DS_WRITE_B8] = "ds_write_b8",
+		[GFX9_DS_WRITE_B16] = "ds_write_b16",
+		[GFX9_DS_ADD_RTN_U32] = "ds_add_rtn_u32",
+		[GFX9_DS_SUB_RTN_U32] = "ds_sub_rtn_u32",
+		[GFX9_DS_RSUB_RTN_U32] = "ds_rsub_rtn_u32",
+		[GFX9_DS_INC_RTN_U32] = "ds_inc_rtn_u32",
+		[GFX9_DS_DEC_RTN_U32] = "ds_dec_rtn_u32",
+		[GFX9_DS_MIN_RTN_I32] = "ds_min_rtn_i32",
+		[GFX9_DS_MAX_RTN_I32] = "ds_max_rtn_i32",
+		[GFX9_DS_MIN_RTN_U32] = "ds_min_rtn_u32",
+		[GFX9_DS_MAX_RTN_U32] = "ds_max_rtn_u32",
+		[GFX9_DS_AND_RTN_B32] = "ds_and_rtn_b32",
+		[GFX9_DS_OR_RTN_B32] = "ds_or_rtn_b32",
+		[GFX9_DS_XOR_RTN_B32] = "ds_xor_rtn_b32",
+		[GFX9_DS_MSKOR_RTN_B32] = "ds_mskor_rtn_b32",
+		[GFX9_DS_WRXCHG_RTN_B32] = "ds_wrxchg_rtn_b32",
+		[GFX9_DS_WRXCHG2_RTN_B32] = "ds_wrxchg2_rtn_b32",
+		[GFX9_DS_WRXCHG2ST64_RTN_B32] = "ds_wrxchg2st64_rtn_b32",
+		[GFX9_DS_CMPST_RTN_B32] = "ds_cmpst_rtn_b32",
+		[GFX9_DS_CMPST_RTN_F32] = "ds_cmpst_rtn_f32",
+		[GFX9_DS_MIN_RTN_F32] = "ds_min_rtn_f32",
+		[GFX9_DS_MAX_RTN_F32] = "ds_max_rtn_f32",
+		[GFX9_DS_WRAP_RTN_B32] = "ds_wrap_rtn_b32",
+		[GFX9_DS_ADD_RTN_F32] = "ds_add_rtn_f32",
+		[GFX9_DS_READ_B32] = "ds_read_b32",
+		[GFX9_DS_READ2_B32] = "ds_read2_b32",
+		[GFX9_DS_READ2ST64_B32] = "ds_read2st64_b32",
+		[GFX9_DS_READ_I8] = "ds_read_i8",
+		[GFX9_DS_READ_U8] = "ds_read_u8",
+		[GFX9_DS_READ_I16] = "ds_read_i16",
+		[GFX9_DS_READ_U16] = "ds_read_u16",
+		[GFX9_DS_SWIZZLE_B32] = "ds_swizzle_b32",
+		[GFX9_DS_PERMUTE_B32] = "ds_permute_b32",
+		[GFX9_DS_BPERMUTE_B32] = "ds_bpermute_b32",
+		[GFX9_DS_ADD_U64] = "ds_add_u64",
+		[GFX9_DS_SUB_U64] = "ds_sub_u64",
+		[GFX9_DS_RSUB_U64] = "ds_rsub_u64",
+		[GFX9_DS_INC_U64] = "ds_inc_u64",
+		[GFX9_DS_DEC_U64] = "ds_dec_u64",
+		[GFX9_DS_MIN_I64] = "ds_min_i64",
+		[GFX9_DS_MAX_I64] = "ds_max_i64",
+		[GFX9_DS_MIN_U64] = "ds_min_u64",
+		[GFX9_DS_MAX_U64] = "ds_max_u64",
+		[GFX9_DS_AND_B64] = "ds_and_b64",
+		[GFX9_DS_OR_B64] = "ds_or_b64",
+		[GFX9_DS_XOR_B64] = "ds_xor_b64",
+		[GFX9_DS_MSKOR_B64] = "ds_mskor_b64",
+		[GFX9_DS_WRITE_B64] = "ds_write_b64",
+		[GFX9_DS_WRITE2_B64] = "ds_write2_b64",
+		[GFX9_DS_WRITE2ST64_B64] = "ds_write2st64_b64",
+		[GFX9_DS_CMPST_B64] = "ds_cmpst_b64",
+		[GFX9_DS_CMPST_F64] = "ds_cmpst_f64",
+		[GFX9_DS_MIN_F64] = "ds_min_f64",
+		[GFX9_DS_MAX_F64] = "ds_max_f64",
+		[GFX9_DS_WRITE_B8_D16_HI] = "ds_write_b8_d16_hi",
+		[GFX9_DS_WRITE_B16_D16_HI] = "ds_write_b16_d16_hi",
+		[GFX9_DS_READ_U8_D16] = "ds_read_u8_d16",
+		[GFX9_DS_READ_U8_D16_HI] = "ds_read_u8_d16_hi",
+		[GFX9_DS_READ_I8_D16] = "ds_read_i8_d16",
+		[GFX9_DS_READ_I8_D16_HI] = "ds_read_i8_d16_hi",
+		[GFX9_DS_READ_U16_D16] = "ds_read_u16_d16",
+		[GFX9_DS_READ_U16_D16_HI] = "ds_read_u16_d16_hi",
+		[GFX9_DS_ADD_RTN_U64] = "ds_add_rtn_u64",
+		[GFX9_DS_SUB_RTN_U64] = "ds_sub_rtn_u64",
+		[GFX9_DS_RSUB_RTN_U64] = "ds_rsub_rtn_u64",
+		[GFX9_DS_INC_RTN_U64] = "ds_inc_rtn_u64",
+		[GFX9_DS_DEC_RTN_U64] = "ds_dec_rtn_u64",
+		[GFX9_DS_MIN_RTN_I64] = "ds_min_rtn_i64",
+		[GFX9_DS_MAX_RTN_I64] = "ds_max_rtn_i64",
+		[GFX9_DS_MIN_RTN_U64] = "ds_min_rtn_u64",
+		[GFX9_DS_MAX_RTN_U64] = "ds_max_rtn_u64",
+		[GFX9_DS_AND_RTN_B64] = "ds_and_rtn_b64",
+		[GFX9_DS_OR_RTN_B64] = "ds_or_rtn_b64",
+		[GFX9_DS_XOR_RTN_B64] = "ds_xor_rtn_b64",
+		[GFX9_DS_MSKOR_RTN_B64] = "ds_mskor_rtn_b64",
+		[GFX9_DS_WRXCHG_RTN_B64] = "ds_wrxchg_rtn_b64",
+		[GFX9_DS_WRXCHG2_RTN_B64] = "ds_wrxchg2_rtn_b64",
+		[GFX9_DS_WRXCHG2ST64_RTN_B64] = "ds_wrxchg2st64_rtn_b64",
+		[GFX9_DS_CMPST_RTN_B64] = "ds_cmpst_rtn_b64",
+		[GFX9_DS_CMPST_RTN_F64] = "ds_cmpst_rtn_f64",
+		[GFX9_DS_MIN_RTN_F64] = "ds_min_rtn_f64",
+		[GFX9_DS_MAX_RTN_F64] = "ds_max_rtn_f64",
+		[GFX9_DS_READ_B64] = "ds_read_b64",
+		[GFX9_DS_READ2_B64] = "ds_read2_b64",
+		[GFX9_DS_CONDXCHG32_RTN_B64] = "ds_condxchg32_rtn_b64",
+		[GFX9_DS_ADD_SRC2_U32] = "ds_add_src2_u32",
+		[GFX9_DS_SUB_SRC2_U32] = "ds_sub_src2_u32",
+		[GFX9_DS_RSUB_SRC2_U32] = "ds_rsub_src2_u32",
+		[GFX9_DS_INC_SRC2_U32] = "ds_inc_src2_u32",
+		[GFX9_DS_DEC_SRC2_U32] = "ds_dec_src2_u32",
+		[GFX9_DS_MIN_SRC2_I32] = "ds_min_src2_i32",
+		[GFX9_DS_MAX_SRC2_I32] = "ds_max_src2_i32",
+		[GFX9_DS_MIN_SRC2_U32] = "ds_min_src2_u32",
+		[GFX9_DS_MAX_SRC2_U32] = "ds_max_src2_u32",
+		[GFX9_DS_AND_SRC2_B32] = "ds_and_src2_b32",
+		[GFX9_DS_OR_SRC2_B32] = "ds_or_src2_b32",
+		[GFX9_DS_XOR_SRC2_B32] = "ds_xor_src2_b32",
+		[GFX9_DS_WRITE_SRC2_B32] = "ds_write_src2_b32",
+		[GFX9_DS_MIN_SRC2_F32] = "ds_min_src2_f32",
+		[GFX9_DS_MAX_SRC2_F32] = "ds_max_src2_f32",
+		[GFX9_DS_ADD_SRC2_F32] = "ds_add_src2_f32",
+		[GFX9_DS_GWS_SEMA_RELEASE_ALL] = "ds_gws_sema_release_all",
+		[GFX9_DS_ADD_SRC2_U64] = "ds_add_src2_u64",
+		[GFX9_DS_SUB_SRC2_U64] = "ds_sub_src2_u64",
+		[GFX9_DS_RSUB_SRC2_U64] = "ds_rsub_src2_u64",
+		[GFX9_DS_INC_SRC2_U64] = "ds_inc_src2_u64",
+		[GFX9_DS_DEC_SRC2_U64] = "ds_dec_src2_u64",
+		[GFX9_DS_MIN_SRC2_I64] = "ds_min_src2_i64",
+		[GFX9_DS_MAX_SRC2_I64] = "ds_max_src2_i64",
+		[GFX9_DS_MIN_SRC2_U64] = "ds_min_src2_u64",
+		[GFX9_DS_MAX_SRC2_U64] = "ds_max_src2_u64",
+		[GFX9_DS_AND_SRC2_B64] = "ds_and_src2_b64",
+		[GFX9_DS_OR_SRC2_B64] = "ds_or_src2_b64",
+		[GFX9_DS_XOR_SRC2_B64] = "ds_xor_src2_b64",
+		[GFX9_DS_WRITE_SRC2_B64] = "ds_write_src2_b64",
+		[GFX9_DS_MIN_SRC2_F64] = "ds_min_src2_f64",
+		[GFX9_DS_MAX_SRC2_F64] = "ds_max_src2_f64",
+		[GFX9_DS_WRITE_B96] = "ds_write_b96",
+		[GFX9_DS_WRITE_B128] = "ds_write_b128",
+		[GFX9_DS_READ_B96] = "ds_read_b96",
+		[GFX9_DS_READ_B128] = "ds_read_b128",
+	},
+};
+
+struct amdgcn_insn  {
+	union {
+		union amdgcn_gfx10_insn gfx10;
+		union amdgcn_gfx9_insn gfx9;
+	};
+	u32 size;
+	u32 idx;
+	enum amdgcn_insn_type type;
+};
+
+static inline void emit_s_load_dwordx2(int version, struct amdgcn_insn *insn,
+				struct amdgcn_param32 dst,
+				struct amdgcn_param32 src, int offset)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_s_load_dwordx2(&insn->gfx10, dst,
+						       src,
+						       offset);
+		insn->type = AMDGCN_INSN_TYPE_SMEM;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_s_load_dwordx2(&insn->gfx9, dst,
+						      src,
+						      offset);
+		insn->type = AMDGCN_INSN_TYPE_SMEM;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_s_load_dwordx2_soff(int version,
+				     struct amdgcn_insn *insn,
+				     struct amdgcn_param32 dst,
+				     struct amdgcn_param32 src,
+				     int offset, u8 soffset)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_s_load_dwordx2(&insn->gfx10, dst,
+						       src, offset);
+		insn->gfx10.smem.soffset = soffset;
+		insn->type = AMDGCN_INSN_TYPE_SMEM;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_s_load_dwordx2(&insn->gfx9, dst,
+						      src, offset);
+		insn->gfx9.smem.soe = 1;
+		insn->gfx9.smem.soffset = soffset;
+		insn->type = AMDGCN_INSN_TYPE_SMEM;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_s_lshl_b32(int version, struct amdgcn_insn *insn,
+			     struct amdgcn_param32 dst,
+			     struct amdgcn_param32 src0,
+			     struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_s_lshl_b32(&insn->gfx10, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_SOP2;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_s_lshl_b32(&insn->gfx9, dst,
+						   src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_SOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_bfe_i32(int version, struct amdgcn_insn *insn,
+			   struct amdgcn_param32 dst,
+			   struct amdgcn_param32 src0,
+			   struct amdgcn_param32 src1,
+			   struct amdgcn_param32 src2)
+{
+	WARN_ON(knod_param_is_literal(src0) ||
+		knod_param_is_literal(src1) ||
+		knod_param_is_literal(src2));
+	if (version == 10) {
+		insn->size = emit_gfx10_v_bfe_i32(&insn->gfx10,
+						  dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_bfe_i32(&insn->gfx9,
+						 dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_bfe_u32(int version, struct amdgcn_insn *insn,
+			   struct amdgcn_param32 dst,
+			   struct amdgcn_param32 src0,
+			   struct amdgcn_param32 src1,
+			   struct amdgcn_param32 src2)
+{
+	WARN_ON(knod_param_is_literal(src0) ||
+		knod_param_is_literal(src1) ||
+		knod_param_is_literal(src2));
+	if (version == 10) {
+		insn->size = emit_gfx10_v_bfe_u32(&insn->gfx10,
+						  dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_bfe_u32(&insn->gfx9,
+						 dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_bfi_b32(int version, struct amdgcn_insn *insn,
+			   struct amdgcn_param32 dst,
+			   struct amdgcn_param32 src0,
+			   struct amdgcn_param32 src1,
+			   struct amdgcn_param32 src2)
+{
+	WARN_ON(knod_param_is_literal(src0) ||
+		knod_param_is_literal(src1) ||
+		knod_param_is_literal(src2));
+	if (version == 10) {
+		insn->size = emit_gfx10_v_bfi_b32(&insn->gfx10,
+						  dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_bfi_b32(&insn->gfx9,
+						 dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshl_add_u32(int version, struct amdgcn_insn *insn,
+				struct amdgcn_param32 dst,
+				struct amdgcn_param32 src0,
+				struct amdgcn_param32 src1,
+				struct amdgcn_param32 src2)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_lshl_add_u32(&insn->gfx10,
+						       dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_lshl_add_u32(&insn->gfx9,
+						      dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshl_or_b32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1,
+			       struct amdgcn_param32 src2)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_lshl_or_b32(&insn->gfx10,
+						      dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_lshl_or_b32(&insn->gfx9,
+						     dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_alignbit_b32(int version, struct amdgcn_insn *insn,
+				struct amdgcn_param32 dst,
+				struct amdgcn_param32 src0,
+				struct amdgcn_param32 src1,
+				struct amdgcn_param32 src2)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_alignbit_b32(&insn->gfx10,
+						       dst, src0,
+						       src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_alignbit_b32(&insn->gfx9,
+						      dst, src0,
+						      src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_perm_b32(int version, struct amdgcn_insn *insn,
+			    struct amdgcn_param32 dst,
+			    struct amdgcn_param32 src0,
+			    struct amdgcn_param32 src1,
+			    struct amdgcn_param32 src2)
+{
+	WARN_ON(knod_param_is_literal(src0) ||
+		knod_param_is_literal(src1) ||
+		knod_param_is_literal(src2));
+	if (version == 10) {
+		insn->size = emit_gfx10_v_perm_b32(&insn->gfx10,
+						    dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_perm_b32(&insn->gfx9,
+						   dst, src0, src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mad_u64_u32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param64 dst,
+			       struct amdgcn_param32 dst2,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1,
+			       struct amdgcn_param64 src2)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mad_u64_u32(&insn->gfx10, dst,
+						      dst2, src0,
+						      src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3B;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mad_u64_u32(&insn->gfx9, dst,
+						     dst2, src0,
+						     src1, src2);
+		insn->type = AMDGCN_INSN_TYPE_VOP3B;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_s_mov_b32(int version, struct amdgcn_insn *insn,
+			   struct amdgcn_param32 dst, struct amdgcn_param32 src)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_s_mov_b32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_SOP1;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_s_mov_b32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_SOP1;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mov_b32_e32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mov_b32_e32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOP1;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mov_b32_e32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOP1;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_readfirstlane_b32(int version,
+				     struct amdgcn_insn *insn,
+				     u8 sdst, u8 vsrc)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_readfirstlane_b32(&insn->gfx10,
+							    sdst, vsrc);
+		insn->type = AMDGCN_INSN_TYPE_VOP1;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_readfirstlane_b32(&insn->gfx9,
+							   sdst, vsrc);
+		insn->type = AMDGCN_INSN_TYPE_VOP1;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_add_co_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_add_co_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3B;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_add_co_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_add_co_ci_u32_e32(int version,
+				     struct amdgcn_insn *insn,
+				     struct amdgcn_param32 dst,
+				     struct amdgcn_param32 src0,
+				     struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_add_co_ci_u32_e32(&insn->gfx10,
+							    dst, src0,
+							    src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_addc_co_u32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+/* No carry in/out */
+static inline void emit_v_add_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_add_nc_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_add_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+/* No carry in/out */
+static inline void emit_v_sub_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_sub_nc_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_sub_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_xor_b32_e32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_xor_b32_e32(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_xor_b32_e32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_or_b32_e32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_or_b32_e32(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_or_b32_e32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cndmask_b32_e32(int version, struct amdgcn_insn *insn,
+				   struct amdgcn_param32 dst,
+				   struct amdgcn_param32 src0,
+				   struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_cndmask_b32_e32(&insn->gfx10, dst,
+							   src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_cndmask_b32_e32(&insn->gfx9, dst,
+							  src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_and_b32_e32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_and_b32_e32(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_and_b32_e32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_sub_co_ci_u32_e32(int version,
+				     struct amdgcn_insn *insn,
+				     struct amdgcn_param32 dst,
+				     struct amdgcn_param32 src0,
+				     struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_sub_co_ci_u32_e32(&insn->gfx10,
+							    dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_subb_co_u32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_subrev_co_ci_u32_e32(int version,
+					struct amdgcn_insn *insn,
+					struct amdgcn_param32 dst,
+					struct amdgcn_param32 src0,
+					struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_subrev_co_ci_u32_e32(&insn->gfx10,
+							       dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_subbrev_co_u32(&insn->gfx9, dst,
+							src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_sub_co_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_sub_co_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3B;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_sub_co_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_subrev_co_u32(int version, struct amdgcn_insn *insn,
+				 struct amdgcn_param32 dst,
+				 struct amdgcn_param32 src0,
+				 struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_subrev_co_u32(&insn->gfx10, dst,
+							src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3B;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_subrev_co_u32(&insn->gfx9, dst,
+						       src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mul_lo_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mul_lo_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mul_lo_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mbcnt_lo_u32_b32(int version,
+				    struct amdgcn_insn *insn,
+				    struct amdgcn_param32 dst,
+				    struct amdgcn_param32 src0,
+				    struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mbcnt_lo_u32_b32(&insn->gfx10,
+							    dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mbcnt_lo_u32_b32(&insn->gfx9,
+							   dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mbcnt_hi_u32_b32(int version,
+				    struct amdgcn_insn *insn,
+				    struct amdgcn_param32 dst,
+				    struct amdgcn_param32 src0,
+				    struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mbcnt_hi_u32_b32(&insn->gfx10,
+							    dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mbcnt_hi_u32_b32(&insn->gfx9,
+							   dst, src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_mul_hi_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src0,
+			      struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_mul_hi_u32(&insn->gfx10, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_mul_hi_u32(&insn->gfx9, dst,
+						    src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshlrev_b64(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param64 dst,
+			       struct amdgcn_param64 src0,
+			       struct amdgcn_param64 src1)
+{
+	/* D.u64 = S1.u64 << S0.u[5:0]. */
+	if (version == 10) {
+		insn->size = emit_gfx10_v_lshlrev_b64(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_lshlrev_b64(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshrrev_b64(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param64 dst,
+			       struct amdgcn_param64 src0,
+			       struct amdgcn_param64 src1)
+{
+	WARN_ON(src1.lo.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_lshrrev_b64(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_lshrrev_b64(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_ashrrev_i64(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param64 dst,
+			       struct amdgcn_param64 src0,
+			       struct amdgcn_param64 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_ashrrev_i64(&insn->gfx10, dst, src0,
+						      src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_ashrrev_i64(&insn->gfx9, dst, src0,
+						     src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_ashrrev_i32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		insn->size = emit_gfx10_v_ashrrev_i32(&insn->gfx10, dst, src0,
+						      src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_ashrrev_i32(&insn->gfx9, dst, src0,
+						     src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP3A;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshlrev_b32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_lshlrev_b32(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_lshlrev_b32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_lshrrev_b32(int version, struct amdgcn_insn *insn,
+			       struct amdgcn_param32 dst,
+			       struct amdgcn_param32 src0,
+			       struct amdgcn_param32 src1)
+{
+	if (version == 10) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx10_v_lshrrev_b32(&insn->gfx10, dst,
+						      src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else if (version == 9) {
+		WARN_ON_ONCE(src1.type != AMDGCN_PARAM_TYPE_VGPR);
+		insn->size = emit_gfx9_v_lshrrev_b32(&insn->gfx9, dst,
+						     src0, src1);
+		insn->type = AMDGCN_INSN_TYPE_VOP2;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_eq_u64(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_eq_u64(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_eq_u64(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_eq_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_eq_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_eq_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_gt_u64(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param64 dst,
+			      struct amdgcn_param64 src)
+{
+	WARN_ON_ONCE(dst.lo.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_gt_u64(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_gt_u64(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_gt_i64(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param64 dst,
+			      struct amdgcn_param64 src)
+{
+	WARN_ON_ONCE(dst.lo.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_gt_i64(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_gt_i64(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_ge_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_ge_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_ge_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_gt_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_gt_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_gt_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_lt_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_lt_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_lt_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_le_u32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_le_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_le_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_gt_i32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_gt_i32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_gt_i32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_ge_i32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_ge_i32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_ge_i32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_lt_i32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_lt_i32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_lt_i32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_le_i32(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param32 dst,
+			      struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	WARN_ON(src.type != AMDGCN_PARAM_TYPE_VGPR);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmp_le_i32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmp_le_i32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+/* EXEC &= (src0 < src1), per-lane mask update */
+static inline void emit_v_cmpx_lt_u32(int version, struct amdgcn_insn *insn,
+				struct amdgcn_param32 dst,
+				struct amdgcn_param32 src)
+{
+	WARN_ON_ONCE(dst.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
+	if (version == 10) {
+		insn->size = emit_gfx10_v_cmpx_lt_u32(&insn->gfx10, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else if (version == 9) {
+		insn->size = emit_gfx9_v_cmpx_lt_u32(&insn->gfx9, dst, src);
+		insn->type = AMDGCN_INSN_TYPE_VOPC;
+	} else {
+		WARN_ON_ONCE(1);
+	}
+}
+
+static inline void emit_v_cmp_ge_u64(int version, struct amdgcn_insn *insn,
+			      struct amdgcn_param64 dst,
+			      struct amdgcn_param64 src)
+{
+	WARN_ON_ONCE(dst.lo.type == AMDGCN_PARAM_TYPE_LITERAL_CONST);
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

drivers/gpu/drm/amd/amdkfd/knod/knod_gfx10_insn.h new file +3978 -0
new file mode 100644
index 000000000000..f2699adb6c7d

Lines of this file are not shown. View the raw message.

drivers/gpu/drm/amd/amdkfd/knod/knod_gfx9_insn.h new file +4068 -0
new file mode 100644
index 000000000000..ea987188f982

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#19 lore raw

Re: [RFC PATCH net-next 08/13] drm/amdkfd: add GPU instruction emitter and disassembler

Quoted 13 lines · Natalie Vock
Quoted 3 lines · Taehee Yoo

Add the AMD GCN (gfx9/gfx10) instruction encoder used to build the GPU shaders that knod dispatches, plus a matching disassembler used for debugging the generated code.

Is it really necessary to have a full-on compiler and disassembler in the kernel driver? This patch is massive and I'm wondering how much benefit it really provides. Is there really no way to move GPU compilation out of the kernel, one way or another? Could you get acceptable perf with a static shader that interprets BPF programs at runtime? Such a shader can be compiled beforehand and just embedded into the kernel - there's prior art there with the CWSR trap handler in amdkfd.

Quoted 2 lines

In case you really, really need to compile the BPF to native ISA, could you still have userspace take care of that in one way or another?

There was a long and painful discussion about P4, and offloading it to hardware. The proponents of that wanted to do the compilation stage in user space to produce a binary blob, but it was hard to prove that the P4 passed to the kernel for software processing, and the binary blob passed to the hardware actually where the same. It opened up the path for closed source P4 where the kernel never got to see the actual P4 code. So it was not really offload, but kernel bypass.

So having a compiler in the kernel is probably the correct way to go, if you want to be friendly to open source.

The other option is to get the GPU to do the compilation itself, so you pass BPF byte codes to the GPU and it generates its own native code. I've no idea if that is possible, but clang can target OpenMP, so maybe it is possible to move this compiler into the GPU?

Andrew

#20 lore raw

Re: [RFC PATCH net-next 08/13] drm/amdkfd: add GPU instruction emitter and disassembler

Quoted 26 lines · Andrew Lunn
Quoted 13 lines · Natalie Vock
Quoted 3 lines · Taehee Yoo

Add the AMD GCN (gfx9/gfx10) instruction encoder used to build the GPU shaders that knod dispatches, plus a matching disassembler used for debugging the generated code.

Is it really necessary to have a full-on compiler and disassembler in the kernel driver? This patch is massive and I'm wondering how much benefit it really provides. Is there really no way to move GPU compilation out of the kernel, one way or another? Could you get acceptable perf with a static shader that interprets BPF programs at runtime? Such a shader can be compiled beforehand and just embedded into the kernel - there's prior art there with the CWSR trap handler in amdkfd.

Quoted 2 lines

In case you really, really need to compile the BPF to native ISA, could you still have userspace take care of that in one way or another?

There was a long and painful discussion about P4, and offloading it to hardware. The proponents of that wanted to do the compilation stage in user space to produce a binary blob, but it was hard to prove that the P4 passed to the kernel for software processing, and the binary blob passed to the hardware actually where the same. It opened up the path for closed source P4 where the kernel never got to see the actual P4 code. So it was not really offload, but kernel bypass.

Quoted 8 lines

So having a compiler in the kernel is probably the correct way to go, if you want to be friendly to open source.

The other option is to get the GPU to do the compilation itself, so you pass BPF byte codes to the GPU and it generates its own native code. I've no idea if that is possible, but clang can target OpenMP, so maybe it is possible to move this compiler into the GPU?

Hi Natalie and Andrew,
Thank you both for the review and suggestions.

We tried running a BPF interpreter on the GPU using OpenCL in an early prototype. The interpreter overhead dominated because each BPF instruction required many native GPU instructions, so the performance was not practical. Since this was only an early feasibility prototype, we do not have benchmark data suitable for a meaningful comparison.

Based on that result, we chose to compile BPF directly into native GPU instructions. Moving compilation to userspace would also raise the problem Andrew described: the kernel could no longer easily establish that the verified BPF program and the submitted GPU binary are equivalent. We have not tried compiling on the GPU itself, but that may be worth investigating separately.

The disassembler is only used for debugging the generated GPU instructions and is not required for execution. We can leave it out of the initial series and submit it separately if needed.

Regards,
Hoyeon Lee

Quoted 1 line

Andrew

author
#21 lore raw

RFC PATCH 9/13 drm/amdkfd: add BPF-to-GPU JIT offload

In reply to Taehee Yoo

Add the knod BPF feature: an XDP program attached in offload mode is JIT-compiled from eBPF to an AMD GCN shader and dispatched on the GPU against packets DMA'd into GPU memory, keeping the PASS/DROP/TX verdict path off the host CPU. Built as a separate module (knod_bpf).

(cherry picked from commit 132d51819ffced59b5890e14bd39cab2e9c12eb4)
---
 drivers/gpu/drm/amd/amdkfd/Kconfig         |    11 +
 drivers/gpu/drm/amd/amdkfd/Makefile        |     2 +
 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c | 11554 +++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h |   597 +
 4 files changed, 12164 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h
drivers/gpu/drm/amd/amdkfd/Kconfig +11 -0
index d93f1af749ff..708aa5fc051b 100644
@@ -49,3 +49,14 @@ config HSA_AMD_KNOD
 
 	  Say N to drop the KNOD core along with the BPF and IPsec offloads
 	  layered on top of it.  If unsure, say Y.
+
+config HSA_AMD_KNOD_BPF
+	tristate "KNOD BPF"
+	depends on HSA_AMD_KNOD
+	help
+	  GPU-accelerated XDP/BPF packet processing via KNOD.  BPF programs
+	  attached in XDP mode are JIT-compiled to AMD GCN shaders and run
+	  on the GPU against packets DMA'd directly into GPU memory, keeping
+	  the verdict path (PASS/DROP/TX) off the host CPU.
+
+	  If unsure, say N.
drivers/gpu/drm/amd/amdkfd/Makefile +2 -0
index 1834faa54863..4df3850e1466 100644
@@ -75,3 +75,5 @@ endif
 ifneq ($(CONFIG_HSA_AMD_KNOD),)
 AMDKFD_FILES += $(AMDKFD_PATH)/kfd_knod.o
 endif
+
+obj-$(CONFIG_HSA_AMD_KNOD_BPF) += $(AMDKFD_PATH)/knod/knod_bpf.o
drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.c new file +11554 -0

2,981 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/knod_bpf.h new file +597 -0

597 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.43.0

Part of the diff is not shown here. View the raw message.

author
#22 lore raw

RFC PATCH 10/13 net/mlx5e: add knod XDP offload support

In reply to Taehee Yoo

Let mlx5e act as a knod NIC: register with the knod core, feed received packets to the accelerator over the per-queue SPSC ring, and transmit accelerator verdicts (XDP_TX) back through the XDP SQ. Restricted to inline-none TX and single-page MTU.

(cherry picked from commit 218c3a122d6b53638b99441560b7a0da65f6d8c1)
---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  10 +
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 351 ++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  19 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  38 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 107 +++++-
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  45 +++
 6 files changed, 564 insertions(+), 6 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +10 -0
index d507289096c2..1401a24e1740 100644
@@ -48,6 +48,7 @@
 #include <net/udp_tunnel.h>
 #include <net/switchdev.h>
 #include <net/psp/types.h>
+#include <net/knod.h>
 #include <net/xdp.h>
 #include <linux/dim.h>
 #include <linux/bits.h>
@@ -568,6 +569,8 @@ struct mlx5e_icosq {
 
 struct mlx5e_frag_page {
 	netmem_ref netmem;
+	struct page_pool *pp;
+	u32 page_idx;
 	u16 frags;
 };
 
@@ -739,6 +742,11 @@ struct mlx5e_rq {
 
 	struct mlx5e_xdp_buff mxbuf;
 
+	struct knod_dev *knodev;
+	struct knod_netdev *knetdev;
+	u32                    knod_spsc_prod_head;
+	bool                   knod_spsc_prod_valid;
+
 	/* AF_XDP zero-copy */
 	struct xsk_buff_pool  *xsk_pool;
 
@@ -985,6 +993,8 @@ struct mlx5e_priv {
 	struct dentry             *dfs_root;
 	struct mlx5_devcom_comp_dev *devcom;
 	struct ethtool_fec_hist_range *fec_ranges;
+	struct knod_dev *knodev;
+	struct knod_netdev *knetdev;
 };
 
 static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv)
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +351 -0

380 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +18 -1
index 3c54f8962664..4de6babdafc5 100644
@@ -34,6 +34,7 @@
 
 #include <linux/indirect_call_wrapper.h>
 #include <net/xdp_sock.h>
+#include <net/knod.h>
 
 #include "en.h"
 #include "en/txrx.h"
@@ -64,6 +65,8 @@ enum mlx5e_xdp_xmit_mode {
 	 * page. The UMEM Completion Ring producer pointer has to be increased.
 	 */
 	MLX5E_XDP_XMIT_MODE_XSK,
+
+	MLX5E_XDP_XMIT_MODE_OFFLOAD,
 };
 
 /* xmit_mode entry is pushed to the fifo per packet, followed by multiple
@@ -78,6 +81,9 @@ enum mlx5e_xdp_xmit_mode {
  *
  * MLX5E_XDP_XMIT_MODE_XSK:
  *    frame.xsk_meta.
+ *
+ * MLX5E_XDP_XMIT_MODE_OFFLOAD:
+ *    offload.netmem.
  */
 #define MLX5E_XDP_FIFO_ENTRIES2DS_MAX_RATIO 4
 
@@ -93,6 +99,10 @@ union mlx5e_xdp_info {
 		struct page *page;
 	} page;
 	struct xsk_tx_metadata_compl xsk_meta;
+	struct {
+		netmem_ref netmem;
+		struct page_pool *pp;
+	} offload;
 };
 
 struct mlx5e_xsk_param;
@@ -100,14 +110,21 @@ int mlx5e_xdp_max_mtu(struct mlx5e_params *params,
 		      struct mlx5e_rq_opt_param *rqo);
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq,
 		      struct bpf_prog *prog, struct mlx5e_xdp_buff *mlctx);
+int mlx5e_rx_offload_act_handler(struct mlx5e_rq *rq, bool flush, int budget);
+void mlx5e_knod_spsc_flush(struct mlx5e_rq *rq);
 void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq);
 bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
 void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq);
 void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw);
+int mlx5e_knod_init(struct mlx5e_priv *priv);
+void mlx5e_knod_uninit(struct mlx5e_priv *priv);
 void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq);
 int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
 		   u32 flags);
-
+void mlx5e_rx_offload_start(struct mlx5e_priv *priv);
+void mlx5e_rx_offload_stop(struct mlx5e_priv *priv);
+void mlx5e_rx_offload_set_napi(struct mlx5e_priv *priv);
+void mlx5e_rx_offload_clear_napi(struct mlx5e_priv *priv);
 extern const struct xdp_metadata_ops mlx5e_xdp_metadata_ops;
 extern const struct xsk_tx_metadata_ops mlx5e_xsk_tx_metadata_ops;
 
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +35 -3
index aa8610cedaa8..cfc6ba6dc492 100644
@@ -918,6 +918,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
 	pool_size = 1 << params->log_rq_mtu_frames;
 
 	rq->mkey_be = cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey);
+	rq->knodev = rq->priv->knodev;
 
 	switch (rq->wq_type) {
 	case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
@@ -1022,7 +1023,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
 		pp_params.queue_idx = rq->ix;
 
 		/* Shampo header data split allow for unreadable netmem */
-		if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state))
+		if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state) || rq->knodev)
 			pp_params.flags |= PP_FLAG_ALLOW_UNREADABLE_NETMEM;
 
 		/* page_pool can be used even when there is no rq->xdp_prog,
@@ -2873,7 +2874,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 	c->netdev   = priv->netdev;
 	c->mkey_be  = cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey);
 	c->num_tc   = mlx5e_get_dcb_num_tc(params);
-	c->xdp      = !!params->xdp_prog;
+	c->xdp      = !!params->xdp_prog || !!priv->knodev;
 	c->stats    = &priv->channel_stats[ix]->ch;
 	c->aff_mask = irq_get_effective_affinity_mask(irq);
 	c->lag_port = mlx5e_enumerate_lag_port(mdev, ix);
@@ -3365,6 +3366,8 @@ void mlx5e_activate_priv_channels(struct mlx5e_priv *priv)
 
 	if (priv->rx_res)
 		mlx5e_rx_res_channels_activate(priv->rx_res, &priv->channels);
+
+	mlx5e_rx_offload_set_napi(priv);
 }
 
 static void mlx5e_cancel_tx_timeout_work(struct mlx5e_priv *priv)
@@ -3376,6 +3379,8 @@ static void mlx5e_cancel_tx_timeout_work(struct mlx5e_priv *priv)
 
 void mlx5e_deactivate_priv_channels(struct mlx5e_priv *priv)
 {
+	mlx5e_rx_offload_clear_napi(priv);
+
 	if (priv->rx_res)
 		mlx5e_rx_res_channels_deactivate(priv->rx_res);
 
@@ -3592,6 +3597,7 @@ int mlx5e_open(struct net_device *netdev)
 		mlx5e_modify_admin_state(priv->mdev, MLX5_PORT_UP);
 	mutex_unlock(&priv->state_lock);
 
+	mlx5e_rx_offload_start(priv);
 	return err;
 }
 
@@ -3623,6 +3629,7 @@ int mlx5e_close(struct net_device *netdev)
 	if (!netif_device_present(netdev))
 		return -ENODEV;
 
+	mlx5e_rx_offload_stop(priv);
 	mutex_lock(&priv->state_lock);
 	mlx5e_modify_admin_state(priv->mdev, MLX5_PORT_DOWN);
 	err = mlx5e_close_locked(netdev);
@@ -4475,7 +4482,7 @@ void mlx5e_set_xdp_feature(struct mlx5e_priv *priv)
 	    params->packet_merge.type == MLX5E_PACKET_MERGE_NONE)
 		val = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT |
 		      NETDEV_XDP_ACT_XSK_ZEROCOPY |
-		      NETDEV_XDP_ACT_RX_SG;
+		      NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_HW_OFFLOAD;
 
 	if (netdev->netdev_ops->ndo_xdp_xmit && params->xdp_prog)
 		val |= NETDEV_XDP_ACT_NDO_XMIT |
@@ -4738,6 +4745,14 @@ int mlx5e_change_mtu(struct net_device *netdev, int new_mtu,
 		goto out;
 	}
 
+	if (priv->knodev && new_mtu > mlx5e_xdp_max_mtu(&new_params, NULL)) {
+		netdev_warn(netdev,
+			    "MTU %d too big for single-page RX offload (max %d)\n",
+			    new_mtu, mlx5e_xdp_max_mtu(&new_params, NULL));
+		err = -EINVAL;
+		goto out;
+	}
+
 	if (priv->xsk.refcnt &&
 	    !mlx5e_xsk_validate_mtu(netdev, &priv->channels,
 				    &new_params, priv->mdev)) {
@@ -5261,15 +5276,30 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog)
 
 static int mlx5e_xdp(struct net_device *dev, struct netdev_bpf *xdp)
 {
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct knod_dev *knodev = priv->knodev;
+	int rc;
+
 	switch (xdp->command) {
 	case XDP_SETUP_PROG:
 		return mlx5e_xdp_set(dev, xdp->prog);
 	case XDP_SETUP_XSK_POOL:
 		return mlx5e_xsk_setup_pool(dev, xdp->xsk.pool,
 					    xdp->xsk.queue_id);
+	case XDP_SETUP_PROG_HW:
+	case BPF_OFFLOAD_MAP_ALLOC:
+	case BPF_OFFLOAD_MAP_FREE:
+		if (!knodev)
+			return -EOPNOTSUPP;
+
+		rc = knod_dev_xdp_install(knodev, xdp);
+		break;
+
 	default:
 		return -EINVAL;
 	}
+
+	return rc;
 }
 
 #ifdef CONFIG_MLX5_ESWITCH
@@ -6931,6 +6961,7 @@ static int _mlx5e_probe(struct auxiliary_device *adev)
 	mlx5e_dcbnl_init_app(priv);
 	mlx5_core_uplink_netdev_set(mdev, netdev);
 	mlx5e_params_print_info(mdev, &priv->channels.params);
+	mlx5e_knod_init(priv);
 	return 0;
 
 err_resume:
@@ -6982,6 +7013,7 @@ static void _mlx5e_remove(struct auxiliary_device *adev)
 	struct mlx5e_priv *priv = netdev_priv(netdev);
 	struct mlx5_core_dev *mdev = edev->mdev;
 
+	mlx5e_knod_uninit(priv);
 	mlx5_eswitch_safe_aux_devs_remove(mdev);
 	mlx5_core_uplink_netdev_set(mdev, NULL);
 
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +105 -2

153 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +45 -0

51 lines not on this page · raw message

Signature · 1 line
2.43.0
#23 lore raw

Re: [RFC PATCH net-next 10/13] net/mlx5e: add knod XDP offload support

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider: - [High] mlx5e_open() starts the offload worker unconditionally even if mlx5e_open_locked() fails.…

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] mlx5e_open() starts the offload worker unconditionally even if mlx5e_open_locked() fails.
- [High] mlx5e_knod_uninit() illegally frees priv->knodev which is owned and freed by the KNOD framework.
- [High] Unbounded iteration over channels causes out-of-bounds accesses on knodev->wpriv and mlx5e_knod_release_batch.
- [High] mlx5e_xmit_xdp_offload_buff() incorrectly casts standard netmem_ref pages to net_iov, reading garbage DMA addresses.
- [High] RX buffers mapped as DMA_FROM_DEVICE are transmitted on the TX queue without remapping, causing IOMMU faults.
- [High] Global static batch array is used concurrently by different netdev NAPI instances.
- [High] Missing channel restart during offload attach/detach leads to silent failure and Use-After-Free of rq->knodev.
- [High] mlx5e_rx_offload_stop races with active NAPI RX processing, leading to SPSC ring corruption and use-after-free of page pool pages.
- [High] A permanent failure during offload transmission (e.g., oversized MTU) causes the release loop to wedge forever, breaking the RX queue.
- [Low] The nic_ops structure is exported to the global linker namespace without the static keyword.
--

--- Patch [10]: [RFC PATCH net-next 10/13] net/mlx5e: add knod XDP offload support --- commit de6f2fbf7fa4e2b457c356167b75d55f4c90a073
Author: Taehee Yoo <[email protected]>

net/mlx5e: add knod XDP offload support

Let mlx5e act as a knod NIC: register with the knod core, feed received packets to the accelerator over the per-queue SPSC ring, and transmit accelerator verdicts (XDP_TX) back through the XDP SQ. Restricted to inline-none TX and single-page MTU.

Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +0 -0
index d8c7cb8837d74..82adfc9b7c1b6 100644

[ ... ]

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +5 -0
+static inline bool mlx5e_xmit_xdp_offload_buff(struct mlx5e_xdpsq *sq,
+					       struct mlx5e_rq *rq,
+					       struct spsc_bd *bd)
+{
+	struct mlx5e_xmit_data_frags xdptxdf = {};

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c int mlx5e_xdp_max_mtu(struct mlx5e_params *params, · line 85

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10 -0
+	struct mlx5e_xmit_data *xdptxd;
+
+	/* attach is restricted to inline-none NICs, so the WQE inlines no
+	 * header and xdptxd->data is never read (left NULL here).
+	 */
+	xdptxd = &xdptxdf.xd;
+	xdptxd->len = bd->len;
+	xdptxd->has_frags = 0;
+	xdptxd->dma_addr = netmem_to_net_iov(bd->netmem)->desc.dma_addr +
+			   bd->off;

[Severity: High]
Is this cast safe if the memory isn't a devmem/net_iov?

If the interface uses standard host memory, netmem_to_net_iov() will read garbage data for the DMA address instead of properly mapping it.

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c int mlx5e_xdp_max_mtu(struct mlx5e_params *params, · line 88

Quoted 3 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +3 -0
+
+	if (!mlx5e_xmit_xdp_frame(sq, xdptxd, 0, NULL))
+		return false;

[Severity: High]
Will this trigger an IOMMU access violation?

If rq->buff.map_dir defaults to DMA_FROM_DEVICE (when there's no software XDP program), this transmits the buffer without remapping it for device read access.

[ ... ]

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c bool mlx5e_xdp_handle(struct mlx5e_rq *rq, · line 423

Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +7 -0
+struct mlx5e_knod_release_batch {
+	struct spsc_bd *bds[NAPI_POLL_WEIGHT];
+	struct spsc_pass_bd pass[NAPI_POLL_WEIGHT];
+};
+
+static struct mlx5e_knod_release_batch
+mlx5e_knod_release_batch[KNOD_SPSC_MAX];

[Severity: High]
Is this static array prone to cross-device data races?

If multiple mlx5e network interfaces are active concurrently, their NAPI threads might run in parallel on different CPUs. Because this array is global and indexed only by rq->ix in mlx5e_rx_offload_release_pending(), they could overwrite each other's batches.

Additionally, since mlx5e supports creating one RX channel per CPU core, can rq->ix exceed KNOD_SPSC_MAX and cause an out-of-bounds access?

[ ... ]

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c bool mlx5e_xdp_handle(struct mlx5e_rq *rq, · line 468

Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +4 -0
+			case KNOD_TX:
+				if (!mlx5e_xmit_xdp_offload_buff(rq->xdpsq, rq,
+								 bds[i]))
+					goto stop_release;

[Severity: High]
Does this cause an infinite retry loop for permanent failures?

If mlx5e_xmit_xdp_offload_buff() fails because a packet exceeds the TX MTU limit, taking this goto stop_release leaves the buffer in the ring.

The next NAPI poll would then retry the exact same oversized packet indefinitely, stalling the queue.

[ ... ]

Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 -0
+static int mlx5e_rx_offload_xdp_attach(struct knod_dev *knodev)
+{

[ ... ]

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +6 -0
+	pr_debug("Attaching XDP offload to netdev %s\n", knodev->netdev->name);
+	WRITE_ONCE(priv->knodev, knodev);
+
+	return 0;
+}
+

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · line 1217

Quoted 10 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +10 -0
+static int mlx5e_rx_offload_xdp_detach(struct knod_dev *knodev)
+{
+	struct mlx5e_priv *priv = netdev_priv(knodev->netdev);
+
+	pr_debug("Detaching XDP offload from netdev %s\n",
+		 knodev->netdev->name);
+	WRITE_ONCE(priv->knodev, NULL);
+
+	return 0;
+}

[Severity: High]
Could this lead to use-after-free and silent data path failures if the interface is administratively UP?

By updating priv->knodev without restarting the active network channels (e.g., via mlx5e_safe_switch_params()), a hot-detach leaves the active RX queues holding stale pointers to a knodev structure that the framework is about to free.

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · line 1222

Quoted 5 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +5 -0
+
+struct knod_nic_ops nic_ops = {
+	.attach = mlx5e_rx_offload_xdp_attach,
+	.detach = mlx5e_rx_offload_xdp_detach,
+};

[Severity: Low]
Should this struct be marked static?

Exporting nic_ops to the global namespace could create a collision hazard if any other driver defines a variable with the same name.

Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 -0
+
+int mlx5e_knod_init(struct mlx5e_priv *priv)

[ ... ]

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · line 1253

Quoted 8 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +8 -0
+void mlx5e_knod_uninit(struct mlx5e_priv *priv)
+{
+	knod_netdev_unregister(priv->knetdev);
+	kfree(priv->knetdev);
+	priv->knetdev = NULL;
+	kfree(priv->knodev);
+	WRITE_ONCE(priv->knodev, NULL);
+}

[Severity: High]
Does this code free memory it doesn't own?

The knodev pointer is a framework-owned object passed via the attach() callback. Calling kfree(priv->knodev) here could lead to a double-free when the framework subsequently cleans up the device.

[ ... ]

Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 -0
+void mlx5e_rx_offload_stop(struct mlx5e_priv *priv)
+{

[ ... ]

Quoted 2 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +2 -0
+	synchronize_net();
+	for (i = 0; i < KNOD_SPSC_MAX; i++) {

[ ... ]

drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw) · line 1327

Quoted 6 lines
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +6 -0
+		spsc_rewind(&knodev->wpriv[i].spsc_bds);
+		while (!spsc_pop(&knodev->wpriv[i].spsc_bds, (void **)&bd))
+			page_pool_put_full_netmem(netmem_get_pp(bd->netmem),
+						  bd->netmem, true);
+	}
+}

[Severity: High]
Could this race with NAPI and corrupt the SPSC ring or page pool?

In mlx5e_close(), mlx5e_rx_offload_stop() is called before mlx5e_close_locked(), meaning NAPI is still actively processing.

Calling page_pool_put_full_netmem() with allow_direct = true concurrently with NAPI violates the page pool API locking constraints.

drivers/net/ethernet/mellanox/mlx5/core/en_main.c static int mlx5e_alloc_rq(struct mlx5e_params *params, · line 921

Quoted 9 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -0
index aa8610cedaa85..cfc6ba6dc492b 100644
@@ -918,6 +918,7 @@ static int mlx5e_alloc_rq(struct mlx5e_params *params,
 	pool_size = 1 << params->log_rq_mtu_frames;
 
 	rq->mkey_be = cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey);
+	rq->knodev = rq->priv->knodev;

[Severity: High]
Does rq->buff.map_dir need to be DMA_BIDIRECTIONAL when hardware offload is active?

In mlx5e_alloc_rq() just above this addition, rq->buff.map_dir is set to DMA_FROM_DEVICE if params->xdp_prog is false. This causes IOMMU faults later when the buffer is transmitted directly on the TX queue without being remapped.

[ ... ]

drivers/net/ethernet/mellanox/mlx5/core/en_main.c int mlx5e_open(struct net_device *netdev) · line 3602

Quoted 7 lines
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -0
@@ -3592,6 +3597,7 @@ int mlx5e_open(struct net_device *netdev)
 		mlx5e_modify_admin_state(priv->mdev, MLX5_PORT_UP);
 	mutex_unlock(&priv->state_lock);
 
+	mlx5e_rx_offload_start(priv);
 	return err;
 }

[Severity: High]
Should mlx5e_rx_offload_start() be skipped if mlx5e_open_locked() fails?

If err indicates failure, the netdev remains closed, so mlx5e_close() won't be called to clean up. This leaves the worker running indefinitely and can cause a use-after-free when it accesses the failed queues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author
#24 lore raw

RFC PATCH 11/13 bnxt_en: add knod XDP offload support

In reply to Taehee Yoo

Let bnxt_en act as a knod NIC: register with the knod core, feed received packets to the accelerator over the per-queue SPSC ring, and transmit accelerator verdicts back through the XDP TX ring. Restricted to page-mode MTU.

(cherry picked from commit c9ef892dd5859cc71b0f8a546d87032cf70bb442)
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 155 +++++++++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  12 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 274 +++++++++++++++++-
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h |  12 +-
 4 files changed, 442 insertions(+), 11 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +147 -8
index 7513618793da..6ddf4bafc3d4 100644
@@ -993,11 +993,36 @@ static inline u8 *__bnxt_alloc_rx_frag(struct bnxt *bp, dma_addr_t *mapping,
 	return page_address(page) + offset;
 }
 
+static int bnxt_alloc_rx_off_netmem(struct bnxt *bp,
+				    struct bnxt_rx_ring_info *rxr,
+				    u16 prod, gfp_t gfp)
+{
+	struct rx_bd *rxbd =
+		&rxr->rx_desc_ring[RX_RING(bp, prod)][RX_IDX(prod)];
+	struct bnxt_sw_rx_bd *rx_buf = &rxr->rx_buf_ring[RING_RX(bp, prod)];
+	unsigned int offset = 0;
+	dma_addr_t mapping;
+	netmem_ref netmem;
+
+	netmem = __bnxt_alloc_rx_netmem(bp, &mapping, rxr, &offset, gfp);
+	if (!netmem)
+		return -ENOMEM;
+
+	mapping += bp->rx_dma_offset;
+	rx_buf->data = (void *)netmem;
+	rx_buf->data_ptr = netmem_address(netmem) + bp->rx_offset;
+	rx_buf->mapping = mapping;
+
+	rxbd->rx_bd_haddr = cpu_to_le64(mapping);
+	return 0;
+}
+
 int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
 		       u16 prod, gfp_t gfp)
 {
 	struct rx_bd *rxbd = &rxr->rx_desc_ring[RX_RING(bp, prod)][RX_IDX(prod)];
 	struct bnxt_sw_rx_bd *rx_buf = &rxr->rx_buf_ring[RING_RX(bp, prod)];
+	unsigned int offset = 0;
 	dma_addr_t mapping;
 
 	if (BNXT_RX_PAGE_MODE(bp)) {
@@ -1012,6 +1037,16 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
 		rx_buf->data = page;
 		rx_buf->data_ptr = page_address(page) + offset + bp->rx_offset;
 		rx_buf->offset = offset;
+	} else if (BNXT_RX_OFFLOAD_MODE(bp)) {
+		netmem_ref netmem = __bnxt_alloc_rx_netmem(bp, &mapping, rxr,
+							   &offset, gfp);
+		if (!netmem)
+			return -ENOMEM;
+
+		mapping += bp->rx_dma_offset;
+		rx_buf->data = (void *)netmem;
+		rx_buf->data_ptr = netmem_address(netmem) + bp->rx_offset;
+		rx_buf->offset = offset;
 	} else {
 		u8 *data = __bnxt_alloc_rx_frag(bp, &mapping, rxr, gfp);
 
@@ -1250,6 +1285,25 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp,
 	return skb;
 }
 
+static struct sk_buff *bnxt_rx_offload_netmem(struct bnxt *bp,
+					      struct bnxt_rx_ring_info *rxr,
+					      u16 cons, void *data,
+					      u8 *data_ptr,
+					      dma_addr_t dma_addr,
+					      unsigned int offset_and_len)
+{
+	u16 prod = rxr->rx_prod;
+	int err;
+
+	err = bnxt_alloc_rx_off_netmem(bp, rxr, prod, GFP_ATOMIC);
+	if (unlikely(err)) {
+		bnxt_reuse_rx_data(rxr, cons, data);
+		return NULL;
+	}
+
+	return NULL;
+}
+
 static struct sk_buff *bnxt_rx_skb(struct bnxt *bp,
 				   struct bnxt_rx_ring_info *rxr, u16 cons,
 				   void *data, u8 *data_ptr,
@@ -2249,6 +2303,36 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
 	len = flags >> RX_CMP_LEN_SHIFT;
 	dma_addr = rx_buf->mapping;
 
+	if (BNXT_RX_OFFLOAD_MODE(bp)) {
+		if (bnxt_alloc_rx_off_netmem(bp, rxr, rxr->rx_prod,
+					     GFP_ATOMIC)) {
+			bnxt_reuse_rx_data(rxr, cons, data);
+			bnapi->cp_ring.sw_stats->rx.rx_buf_errors++;
+		} else {
+			struct knod_work_priv *wpriv;
+			struct spsc_bd *bd;
+
+			if (bnapi->index >= KNOD_SPSC_MAX) {
+				rc = 1;
+				goto next_rx;
+			}
+
+			wpriv = &bp->knodev->wpriv[bnapi->index];
+			if (spsc_produce(&wpriv->spsc_bds, (void **)&bd)) {
+				bnxt_reuse_rx_data(rxr, cons, data);
+			} else {
+				bd->netmem = (netmem_ref)data;
+				bd->len = len;
+				bd->off = bp->rx_offset;
+				bd->page_idx = net_iov_binding_idx(
+					netmem_to_net_iov((netmem_ref)data));
+				spsc_produce_commit(&wpriv->spsc_bds);
+			}
+		}
+		rc = 1;
+		goto next_rx;
+	}
+
 	if (bnxt_xdp_attached(bp, rxr)) {
 		bnxt_xdp.rxcmp = rxcmp;
 		bnxt_xdp.rxcmp1 = rxcmp1;
@@ -3272,6 +3356,11 @@ static int bnxt_poll(struct napi_struct *napi, int budget)
 		napi_complete(napi);
 		return 0;
 	}
+
+	if (BNXT_RX_OFFLOAD_MODE(bp))
+		work_done += bnxt_rx_offload_act_handler(bnapi,
+							 budget - work_done);
+
 	while (1) {
 		work_done += bnxt_poll_work(bp, cpr, budget - work_done);
 
@@ -3296,6 +3385,7 @@ static int bnxt_poll(struct napi_struct *napi, int budget)
 				  &dim_sample);
 		net_dim(&cpr->dim, &dim_sample);
 	}
+
 	return work_done;
 }
 
@@ -3363,6 +3453,11 @@ static int bnxt_poll_p5(struct napi_struct *napi, int budget)
 		cpr->has_more_work = 0;
 		work_done = __bnxt_poll_cqs(bp, bnapi, budget);
 	}
+
+	if (BNXT_RX_OFFLOAD_MODE(bp))
+		work_done += bnxt_rx_offload_act_handler(bnapi,
+							 budget - work_done);
+
 	while (1) {
 		u16 type;
 
@@ -3427,6 +3522,7 @@ static int bnxt_poll_p5(struct napi_struct *napi, int budget)
 				  &dim_sample);
 		net_dim(&cpr->dim, &dim_sample);
 	}
+
 	return work_done;
 }
 
@@ -3458,6 +3554,13 @@ static void bnxt_free_one_tx_ring_skbs(struct bnxt *bp,
 			tx_buf->xdpf = NULL;
 			i++;
 			continue;
+		} else if (tx_buf->action == BNXT_NETMEM_TX) {
+			page_pool_recycle_direct_netmem(
+				netmem_get_pp(tx_buf->netmem),
+				tx_buf->netmem);
+			tx_buf->action = 0;
+			tx_buf->netmem = 0;
+			continue;
 		}
 
 		skb = tx_buf->skb;
@@ -3551,6 +3654,10 @@ static void bnxt_free_one_rx_ring(struct bnxt *bp, struct bnxt_rx_ring_info *rxr
 		rx_buf->data = NULL;
 		if (BNXT_RX_PAGE_MODE(bp))
 			page_pool_recycle_direct(rxr->page_pool, data);
+		else if (BNXT_RX_OFFLOAD_MODE(bp))
+			page_pool_put_full_netmem(
+				netmem_get_pp((netmem_ref)data),
+				(netmem_ref)data, false);
 		else
 			page_pool_free_va(rxr->head_pool, data, true);
 	}
@@ -3905,7 +4012,7 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp,
 	struct page_pool *pool;
 
 	pp.pool_size = bnxt_rx_agg_ring_fill_level(bp, rxr) / agg_size_fac;
-	if (BNXT_RX_PAGE_MODE(bp))
+	if (BNXT_RX_PAGE_MODE(bp) || BNXT_RX_OFFLOAD_MODE(bp))
 		pp.pool_size += bp->rx_ring_size / rx_size_fac;
 
 	pp.order = get_order(rxr->rx_page_size);
@@ -3923,8 +4030,10 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp,
 		return PTR_ERR(pool);
 	rxr->page_pool = pool;
 
-	rxr->need_head_pool = page_pool_is_unreadable(pool);
-	rxr->need_head_pool |= !!pp.order;
+	if (!BNXT_RX_OFFLOAD_MODE(bp)) {
+		rxr->need_head_pool = page_pool_is_unreadable(pool);
+		rxr->need_head_pool |= !!pp.order;
+	}
 	if (bnxt_separate_head_pool(rxr)) {
 		pp.order = 0;
 		pp.max_len = PAGE_SIZE;
@@ -4673,6 +4782,9 @@ static int bnxt_init_rx_rings(struct bnxt *bp)
 	if (BNXT_RX_PAGE_MODE(bp)) {
 		bp->rx_offset = NET_IP_ALIGN + XDP_PACKET_HEADROOM;
 		bp->rx_dma_offset = XDP_PACKET_HEADROOM;
+	} else if (BNXT_RX_OFFLOAD_MODE(bp)) {
+		bp->rx_offset = NET_IP_ALIGN + XDP_PACKET_HEADROOM;
+		bp->rx_dma_offset = XDP_PACKET_HEADROOM;
 	} else {
 		bp->rx_offset = BNXT_RX_OFFSET;
 		bp->rx_dma_offset = BNXT_RX_DMA_OFFSET;
@@ -4961,12 +5073,14 @@ void bnxt_set_ring_params(struct bnxt *bp)
 /* Changing allocation mode of RX rings.
  * TODO: Update when extending xdp_rxq_info to support allocation modes.
  */
-static void __bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode)
+static void __bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode)
 {
 	struct net_device *dev = bp->dev;
 
-	if (page_mode) {
-		bp->flags &= ~(BNXT_FLAG_AGG_RINGS | BNXT_FLAG_NO_AGG_RINGS);
+	if (page_mode & BNXT_FLAG_RX_PAGE_MODE) {
+		bp->flags &= ~(BNXT_FLAG_AGG_RINGS |
+			       BNXT_FLAG_NO_AGG_RINGS |
+			       BNXT_FLAG_RX_OFFLOAD_MODE);
 		bp->flags |= BNXT_FLAG_RX_PAGE_MODE;
 
 		if (bp->xdp_prog->aux->xdp_has_frags)
@@ -4982,15 +5096,24 @@ static void __bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode)
 			bp->rx_skb_func = bnxt_rx_page_skb;
 		}
 		bp->rx_dir = DMA_BIDIRECTIONAL;
+	} else if (page_mode & BNXT_FLAG_RX_OFFLOAD_MODE) {
+		bp->flags &= ~(BNXT_FLAG_AGG_RINGS |
+			       BNXT_FLAG_RX_PAGE_MODE);
+		bp->flags |= (BNXT_FLAG_RX_OFFLOAD_MODE |
+			      BNXT_FLAG_NO_AGG_RINGS);
+		dev->max_mtu = min_t(u16, bp->max_mtu, BNXT_MAX_PAGE_MODE_MTU);
+		bp->rx_skb_func = bnxt_rx_offload_netmem;
+		bp->rx_dir = DMA_BIDIRECTIONAL;
 	} else {
 		dev->max_mtu = bp->max_mtu;
 		bp->flags &= ~BNXT_FLAG_RX_PAGE_MODE;
+		bp->flags &= ~BNXT_FLAG_RX_OFFLOAD_MODE;
 		bp->rx_dir = DMA_FROM_DEVICE;
 		bp->rx_skb_func = bnxt_rx_skb;
 	}
 }
 
-void bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode)
+void bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode)
 {
 	__bnxt_set_rx_skb_mode(bp, page_mode);
 
@@ -5663,6 +5786,12 @@ static int bnxt_alloc_mem(struct bnxt *bp, bool irq_re_init)
 					BNXT_RING_TO_TC(bp, txr->txq_index);
 				bnapi2->tx_ring[txr->tx_napi_idx] = txr;
 				bnapi2->tx_int = bnxt_tx_int;
+			} else if (BNXT_RX_OFFLOAD_MODE(bp)) {
+				bnapi2 = bp->bnapi[j];
+				bnapi2->flags |= BNXT_NAPI_FLAG_XDP;
+				bnapi2->tx_ring[0] = txr;
+				bnapi2->tx_int = bnxt_tx_int_xdp;
+				j++;
 			} else {
 				bnapi2 = bp->bnapi[j];
 				bnapi2->flags |= BNXT_NAPI_FLAG_XDP;
@@ -11958,6 +12087,8 @@ static void bnxt_del_napi(struct bnxt *bp)
 	if (!bp->bnapi)
 		return;
 
+	bnxt_rx_offload_clear_napi(bp);
+
 	for (i = 0; i < bp->rx_nr_rings; i++)
 		netif_queue_set_napi(bp->dev, i, NETDEV_QUEUE_TYPE_RX, NULL);
 	for (i = 0; i < bp->tx_nr_rings - bp->tx_nr_rings_xdp; i++)
@@ -11997,6 +12128,8 @@ static void bnxt_init_napi(struct bnxt *bp)
 		bnapi = bp->bnapi[cp_nr_rings];
 		netif_napi_add_locked(bp->dev, &bnapi->napi, bnxt_poll_nitroa0);
 	}
+
+	bnxt_rx_offload_set_napi(bp);
 }
 
 static void bnxt_disable_napi(struct bnxt *bp)
@@ -13356,6 +13489,7 @@ static int bnxt_open(struct net_device *dev)
 						   BNXT_RESTART_ULP_SP_EVENT);
 		}
 	}
+	bnxt_rx_offload_start(bp);
 
 	return rc;
 }
@@ -13443,6 +13577,7 @@ static int bnxt_close(struct net_device *dev)
 {
 	struct bnxt *bp = netdev_priv(dev);
 
+	bnxt_rx_offload_stop(bp);
 	bnxt_close_nic(bp, true, true);
 	bnxt_hwrm_shutdown_link(bp);
 	bnxt_hwrm_if_change(bp, false);
@@ -13643,6 +13778,8 @@ bnxt_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats)
 		stats->tx_errors = BNXT_GET_TX_PORT_STATS64(tx, tx_err);
 	}
 	clear_bit(BNXT_STATE_READ_STATS, &bp->state);
+	if (BNXT_RX_OFFLOAD_MODE(bp))
+		knod_dev_get_stats64(bp->knodev, stats);
 }
 
 static void bnxt_get_one_ring_drv_stats(struct bnxt *bp,
@@ -16518,6 +16655,7 @@ static void bnxt_remove_one(struct pci_dev *pdev)
 
 	bnxt_aux_devices_del(bp);
 
+	bnxt_knod_uninit(bp);
 	unregister_netdev(dev);
 	bnxt_ptp_clear(bp);
 
@@ -17080,7 +17218,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	}
 
 	dev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT |
-			    NETDEV_XDP_ACT_RX_SG;
+			    NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_HW_OFFLOAD;
 
 #ifdef CONFIG_BNXT_SRIOV
 	init_waitqueue_head(&bp->sriov_cfg_wait);
@@ -17198,6 +17336,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	bnxt_print_device_info(bp);
 
 	pci_save_state(pdev);
+	bnxt_knod_init(bp);
 
 	return 0;
 init_err_cleanup:
drivers/net/ethernet/broadcom/bnxt/bnxt.h +11 -1
index 6335dfc14c98..d99590db6efa 100644
@@ -29,6 +29,8 @@
 #include <net/devlink.h>
 #include <net/dst_metadata.h>
 #include <net/xdp.h>
+#include <net/knod.h>
+#include <net/spsc_ring.h>
 #include <linux/dim.h>
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/bnxt/ulp.h>
@@ -884,10 +886,14 @@ struct nqe_cn {
 #define BNXT_REDIRECT_EVENT	8
 #define BNXT_TX_CMP_EVENT	0x10
 
+#define BNXT_NETMEM_ACT		0xf0
+#define BNXT_NETMEM_TX		(XDP_TX + BNXT_NETMEM_ACT)
+
 struct bnxt_sw_tx_bd {
 	union {
 		struct sk_buff		*skb;
 		struct xdp_frame	*xdpf;
+		netmem_ref		netmem;
 	};
 	DEFINE_DMA_UNMAP_ADDR(mapping);
 	DEFINE_DMA_UNMAP_LEN(len);
@@ -2300,6 +2306,7 @@ struct bnxt {
 	#define BNXT_FLAG_TX_COAL_CMPL	0x8000000
 	#define BNXT_FLAG_PORT_STATS_EXT	0x10000000
 	#define BNXT_FLAG_HDS		0x20000000
+	#define BNXT_FLAG_RX_OFFLOAD_MODE	0x40000000
 	#define BNXT_FLAG_AGG_RINGS	(BNXT_FLAG_JUMBO | BNXT_FLAG_GRO | \
 					 BNXT_FLAG_LRO | BNXT_FLAG_HDS)
 
@@ -2324,6 +2331,7 @@ struct bnxt {
 				 (bp)->link_info.phy_state == BNXT_PHY_STATE_ENABLED)
 #define BNXT_CHIP_TYPE_NITRO_A0(bp) ((bp)->flags & BNXT_FLAG_CHIP_NITRO_A0)
 #define BNXT_RX_PAGE_MODE(bp)	((bp)->flags & BNXT_FLAG_RX_PAGE_MODE)
+#define BNXT_RX_OFFLOAD_MODE(bp)	((bp)->flags & BNXT_FLAG_RX_OFFLOAD_MODE)
 #define BNXT_SUPPORTS_TPA(bp)	(!BNXT_CHIP_TYPE_NITRO_A0(bp) &&	\
 				 (!((bp)->flags & BNXT_FLAG_CHIP_P5_PLUS) ||\
 				  (bp)->max_tpa_v2) && !is_kdump_kernel())
@@ -2748,6 +2756,8 @@ struct bnxt {
 #define BNXT_DUMP_LIVE_WITH_CTX_L1_CACHE	3
 
 	struct bpf_prog		*xdp_prog;
+	struct knod_netdev	*knetdev;
+	struct knod_dev	*knodev;
 
 	struct bnxt_ptp_cfg	*ptp_cfg;
 	u8			ptp_all_rx_tstamp;
@@ -2964,7 +2974,7 @@ u32 bnxt_fw_health_readl(struct bnxt *bp, int reg_idx);
 bool bnxt_bs_trace_avail(struct bnxt *bp, u16 type);
 void bnxt_set_tpa_flags(struct bnxt *bp);
 void bnxt_set_ring_params(struct bnxt *);
-void bnxt_set_rx_skb_mode(struct bnxt *bp, bool page_mode);
+void bnxt_set_rx_skb_mode(struct bnxt *bp, int page_mode);
 void bnxt_insert_usr_fltr(struct bnxt *bp, struct bnxt_filter_base *fltr);
 void bnxt_del_one_usr_fltr(struct bnxt *bp, struct bnxt_filter_base *fltr);
 int bnxt_hwrm_func_drv_rgtr(struct bnxt *bp, unsigned long *bmap,
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +273 -1

308 lines not on this page · raw message

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h +11 -1
index fb4f9143929f..7f2a850e4787 100644
@@ -24,9 +24,14 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp,
 				   dma_addr_t mapping, u32 len,
 				   struct xdp_buff *xdp);
 void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi *bnapi, int budget);
+void bnxt_tx_int_offload(struct bnxt *bp, struct bnxt_napi *bnapi, int budget);
 bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons,
 		 struct xdp_buff *xdp, struct page *page, u8 **data_ptr,
 		 unsigned int *len, u8 *event);
+void bnxt_rx_offload_stop(struct bnxt *bp);
+void bnxt_rx_offload_start(struct bnxt *bp);
+void bnxt_rx_offload_set_napi(struct bnxt *bp);
+void bnxt_rx_offload_clear_napi(struct bnxt *bp);
 int bnxt_xdp(struct net_device *dev, struct netdev_bpf *xdp);
 int bnxt_xdp_xmit(struct net_device *dev, int num_frames,
 		  struct xdp_frame **frames, u32 flags);
@@ -43,5 +48,10 @@ struct sk_buff *bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb,
 				   struct xdp_buff *xdp);
 int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash,
 		     enum xdp_rss_hash_type *rss_type);
-
+int bnxt_rx_offload_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
+			u16 cons, void *data, unsigned int len, int index,
+			u8 *event);
+int bnxt_rx_offload_act_handler(struct bnxt_napi *bnapi, int budget);
+int bnxt_knod_init(struct bnxt *bp);
+void bnxt_knod_uninit(struct bnxt *bp);
 #endif
Signature · 1 line
2.43.0
#25 lore raw

Re: [RFC PATCH net-next 11/13] bnxt_en: add knod XDP offload support

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider: - [High] Memory leaks in `bnxt_rx_pkt` when `BNXT_RX_OFFLOAD_MODE` is active. - [High] Incomplete error…

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] Memory leaks in bnxt_rx_pkt when BNXT_RX_OFFLOAD_MODE is active.
- [High] Incomplete error rollback during offload attach leads to NULL pointer dereferences.
- [High] Invalid free of knod core-owned memory in bnxt_knod_uninit().
- [High] Detaching the offload accelerator silently corrupts standard XDP state.
- [High] Type confusion via unconditional cast of netmem_ref to net_iov.
- [High] Prematurely setting bp->knodev to NULL during detach triggers multiple NULL pointer dereferences.
- [High] Ignoring the return value of bnxt_knod_init() causes a NULL pointer dereference on unload.
- [High] Head-of-line blocking and packet stalls in bnxt_rx_offload_act_handler due to restrictive limit based on TX availability.
- [High] Unbalanced lifecycle operations for knod_dev_start leading to operational failure or resource leaks.
- [High] Incorrect queue bounds check in bnxt_rx_offload_act_handler() leads to out-of-bounds access and stalls. --

--- Patch [11]: [RFC PATCH net-next 11/13] bnxt_en: add knod XDP offload support --- commit bc105761ac52d8c362d8a3b4da5f278514f84ca4
Author: Taehee Yoo <[email protected]>

bnxt_en: add knod XDP offload support

This commit introduces knod XDP offload support for the bnxt_en driver, allowing it to register as a knod NIC. It sets up the transmission of accelerator verdicts through the XDP TX ring and feeds received packets to the accelerator over the per-queue SPSC ring.

Quoted 4 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -0
index 7513618793daf..6ddf4bafc3d47 100644

[ ... ]

Quoted 6 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -0
@@ -2249,6 +2303,36 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
 	len = flags >> RX_CMP_LEN_SHIFT;
 	dma_addr = rx_buf->mapping;
 
+	if (BNXT_RX_OFFLOAD_MODE(bp)) {
+		if (bnxt_alloc_rx_off_netmem(bp, rxr, rxr->rx_prod,

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, · line 2317

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +10 -0
+					     GFP_ATOMIC)) {
+			bnxt_reuse_rx_data(rxr, cons, data);
+			bnapi->cp_ring.sw_stats->rx.rx_buf_errors++;
+		} else {
+			struct knod_work_priv *wpriv;
+			struct spsc_bd *bd;
+
+			if (bnapi->index >= KNOD_SPSC_MAX) {
+				rc = 1;
+				goto next_rx;

[Severity: High]
Does this path leak the current data buffer? It appears to jump to next_rx without freeing or reusing data.

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, · line 2322

Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +5 -0
+			}
+
+			wpriv = &bp->knodev->wpriv[bnapi->index];
+			if (spsc_produce(&wpriv->spsc_bds, (void **)&bd)) {
+				bnxt_reuse_rx_data(rxr, cons, data);

[Severity: High]
If the accelerator SPSC ring is full and spsc_produce() fails, doesn't calling bnxt_reuse_rx_data() here permanently leak the newly allocated replacement buffer by overwriting it with the old data?

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, · line 2328

Quoted 6 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +6 -0
+			} else {
+				bd->netmem = (netmem_ref)data;
+				bd->len = len;
+				bd->off = bp->rx_offset;
+				bd->page_idx = net_iov_binding_idx(
+					netmem_to_net_iov((netmem_ref)data));

[Severity: High]
Is it safe to unconditionally cast data to a net_iov here? The RX page pool allows standard pages, so if a standard page is used, could this cause type confusion and out-of-bounds reads?

Quoted 8 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +7 -0
+				spsc_produce_commit(&wpriv->spsc_bds);
+			}
+		}
+		rc = 1;
+		goto next_rx;
+	}
+
 	if (bnxt_xdp_attached(bp, rxr)) {

[ ... ]

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_open(struct net_device *dev) · line 13492

Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -0
@@ -13356,6 +13489,7 @@ static int bnxt_open(struct net_device *dev)
 						   BNXT_RESTART_ULP_SP_EVENT);
 		}
 	}
+	bnxt_rx_offload_start(bp);

[Severity: High]
Is it intended to unconditionally call bnxt_rx_offload_start() here? If the preceding __bnxt_open_nic() fails, this will start the accelerator while the NIC remains down, which could leak active workers.

Quoted 3 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -0
 	return rc;
 }

[ ... ]

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) · line 17339

Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -0
@@ -17198,6 +17336,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	bnxt_print_device_info(bp);
 
 	pci_save_state(pdev);
+	bnxt_knod_init(bp);

[Severity: High]
Could this leave the driver in a corrupted state if memory allocation fails? bnxt_knod_init() can return -ENOMEM, but the error is ignored here. If it fails, bp->knetdev remains NULL, which might lead to a NULL pointer dereference during module unload in bnxt_knod_uninit().

Quoted 7 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -0
 	return 0;
 init_err_cleanup:
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +0 -0
index 9e5009be8e989..a264a2cf8302a 100644

[ ... ]

Quoted 7 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +3 -0
@@ -441,19 +446,123 @@ static int bnxt_xdp_set(struct bnxt *bp, struct bpf_prog *prog)
 	return 0;
 }
 
+
+static int bnxt_xdp_offload_set(struct bnxt *bp, int enable)
+{

[ ... ]

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c static int bnxt_xdp_set(struct bnxt *bp, struct bpf_prog *prog) · line 486

Quoted 6 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +6 -0
+	if (enable) {
+		bnxt_set_rx_skb_mode(bp, BNXT_FLAG_RX_OFFLOAD_MODE);
+		xdp_features_set_redirect_target_locked(dev, true);
+	} else {
+		xdp_features_clear_redirect_target_locked(dev);
+		bnxt_set_rx_skb_mode(bp, false);

[Severity: High]
Does detaching the offload accelerator silently corrupt standard XDP state? If a standard XDP program is still active (bp->xdp_prog), explicitly clearing the BNXT_FLAG_RX_PAGE_MODE flag here without restoring the correct settings for the active standard XDP program could cause packet processing failures.

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c static int bnxt_xdp_set(struct bnxt *bp, struct bpf_prog *prog) · line 496

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +10 -0
+	}
+	bp->tx_nr_rings_xdp = tx_xdp;
+	bp->tx_nr_rings = bp->tx_nr_rings_per_tc * tc + tx_xdp;
+	tx_cp = bnxt_num_tx_to_cp(bp, bp->tx_nr_rings);
+	bp->cp_nr_rings = max_t(int, tx_cp, bp->rx_nr_rings);
+	bnxt_set_tpa_flags(bp);
+	bnxt_set_ring_params(bp);
+
+	if (netif_running(dev))
+		return bnxt_open_nic(bp, true, false);

[Severity: High]
When attaching offload to an already-running interface, is the accelerator ever started? This restarts the NIC via bnxt_open_nic(), but it seems to omit a call to bnxt_rx_offload_start(), which might leave the accelerator unable to process packets.

Quoted 3 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +3 -0
+
+	return 0;
+}

[ ... ]

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 659

Quoted 9 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +9 -0
+static int bnxt_rx_offload_xdp_attach(struct knod_dev *knodev)
+{
+	struct bnxt *bp = netdev_priv(knodev->netdev);
+	int rc;
+
+	WRITE_ONCE(bp->knodev, knodev);
+	rc = bnxt_xdp_offload_set(bp, true);
+	if (rc)
+		WRITE_ONCE(bp->knodev, NULL);

[Severity: High]
Could this error handling path lead to a NULL pointer dereference? If bnxt_xdp_offload_set() sets BNXT_FLAG_RX_OFFLOAD_MODE but subsequently fails, bp->knodev is set back to NULL without clearing the offload flag. A later stats read or interface up might see the flag and dereference the NULL knodev.

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 669

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +10 -0
+
+	return rc;
+}
+
+static int bnxt_rx_offload_xdp_detach(struct knod_dev *knodev)
+{
+	struct bnxt *bp = netdev_priv(knodev->netdev);
+
+	WRITE_ONCE(bp->knodev, NULL);
+	bnxt_xdp_offload_set(bp, false);

[Severity: High]
Is it safe to set bp->knodev to NULL before shutting down the interface? Since BNXT_RX_OFFLOAD_MODE is still active until bnxt_xdp_offload_set() finishes, concurrent NAPI polling in bnxt_rx_pkt() or teardown paths like bnxt_rx_offload_clear_napi() could observe the flag and dereference the NULL pointer.

Quoted 3 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +3 -0
+
+	return 0;
+}

[ ... ]

Quoted 7 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +7 -0
+int bnxt_rx_offload_act_handler(struct bnxt_napi *bnapi, int budget)
+{
+	struct bnxt_tx_ring_info *txr = bnapi->tx_ring[0];
+	struct bnxt_rx_ring_info *rxr = bnapi->rx_ring;
+	struct spsc_bd *bds[NAPI_POLL_WEIGHT];
+	u32 tx_avail, cnt, i, nxmit = 0;
+	struct knod_dev *knodev;

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 708

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +10 -0
+	struct knod_work_priv *wpriv;
+	struct napi_struct *napi;
+	struct bnxt *bp = bnapi->bp;
+	dma_addr_t mapping;
+
+	knodev = bp->knodev;
+	if (!knodev)
+		return 0;
+
+	if (bnapi->index >= bp->dev->real_num_tx_queues) {

[Severity: High]
Should this bounds check compare against KNOD_SPSC_MAX instead of the number of TX queues? If there are more RX queues than TX queues, valid RX NAPI instances might return early and abandon packets. Conversely, if there are many TX queues, this could fail to properly bound accesses to the wpriv array.

Quoted 4 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +4 -0
+		this_cpu_inc(knodev->stats->tx_dropped);
+		return 0;
+	}
+

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 722

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +10 -0
+	wpriv = &knodev->wpriv[bnapi->index];
+	napi = READ_ONCE(wpriv->napi);
+	if (!napi)
+		return 0;
+
+	tx_avail = bnxt_tx_avail(bp, txr);
+	cnt = min_t(u32, NAPI_POLL_WEIGHT, tx_avail);
+	cnt = min_t(u32, cnt, budget);
+	if (!cnt)
+		return 0;

[Severity: High]
Could this early return cause head-of-line blocking and permanent packet stalls? If the TX ring is full (tx_avail == 0), cnt becomes 0. Returning immediately skips knod_dev_xdp_drain_pass(), which could trap packets meant for the host stack indefinitely.

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 735

Quoted 13 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +13 -0
+
+	spsc_release(&wpriv->spsc_bds, (void **)bds, cnt, &cnt);
+	if (!cnt)
+		return 0;
+
+	for (i = 0; i < cnt; i++) {
+		switch (bds[i]->act) {
+		case KNOD_ACT_INFLIGHT:
+		case KNOD_IPSEC_INFLIGHT:
+			goto stop_release;
+		case XDP_TX:
+			mapping = netmem_to_net_iov(bds[i]->netmem)->desc.dma_addr +
+				  bds[i]->off;

[Severity: High]
Can this path be reached when netmem is a standard page rather than a DMABUF? If so, using netmem_to_net_iov() without verifying netmem_is_net_iov() could misinterpret standard page structure fields as DMA addresses, potentially causing memory corruption.

Quoted 2 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +2 -0
+			__bnxt_xmit_netmem(bp, txr, mapping, bds[i]->len,
+					   bds[i]->netmem);

[ ... ]

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c int bnxt_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, · line 800

Quoted 6 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +6 -0
+void bnxt_knod_uninit(struct bnxt *bp)
+{
+	knod_netdev_unregister(bp->knetdev);
+	kfree(bp->knetdev);
+	bp->knetdev = NULL;
+	kfree(bp->knodev);

[Severity: High]
Is it correct to call kfree(bp->knodev) here? bp->knodev points to a struct knod_dev allocated and managed by the knod core framework, so freeing it from the driver could result in heap corruption or a use-after-free during device unbind.

Quoted 2 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +2 -0
+	WRITE_ONCE(bp->knodev, NULL);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author
#26 lore raw

RFC PATCH 12/13 selftests: drivers/net: add knod tests

In reply to Taehee Yoo

Add attach/detach and XDP offload selftests for knod, exercising the genetlink control plane and the BPF data path over a knod-capable NIC.

(cherry picked from commit 320315bfd9a324ab029c2fe9eee52997592e5d0c)
---
 .../selftests/drivers/net/knod/Makefile       |  17 ++
 .../testing/selftests/drivers/net/knod/config |   7 +
 .../selftests/drivers/net/knod/knod_attach.sh | 135 +++++++++++++
 .../drivers/net/knod/knod_xdp_ktime.sh        | 173 +++++++++++++++++
 .../drivers/net/knod/knod_xdp_loop.sh         | 129 +++++++++++++
 .../testing/selftests/drivers/net/knod/lib.sh | 181 ++++++++++++++++++
 .../drivers/net/knod/xdp_ktime.bpf.c          |  32 ++++
 .../selftests/drivers/net/knod/xdp_loop.bpf.c |  42 ++++
 8 files changed, 716 insertions(+)
 create mode 100644 tools/testing/selftests/drivers/net/knod/Makefile
 create mode 100644 tools/testing/selftests/drivers/net/knod/config
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_attach.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/knod_xdp_loop.sh
 create mode 100755 tools/testing/selftests/drivers/net/knod/lib.sh
 create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c
 create mode 100644 tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c
tools/testing/selftests/drivers/net/knod/Makefile new file +17 -0
new file mode 100644
index 000000000000..08138529b55e
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: GPL-2.0
+
+TEST_PROGS := \
+	knod_xdp_ktime.sh \
+	knod_xdp_loop.sh \
+	knod_attach.sh \
+# end of TEST_PROGS
+
+TEST_FILES := \
+	lib.sh \
+# end of TEST_FILES
+
+TEST_GEN_FILES += $(patsubst %.c,%.o,$(wildcard *.bpf.c))
+
+include ../../../lib.mk
+
+include ../../../net/bpf.mk
tools/testing/selftests/drivers/net/knod/config new file +7 -0
new file mode 100644
index 000000000000..a60ece9da112
@@ -0,0 +1,7 @@
+CONFIG_BPF=y
+CONFIG_BPF_SYSCALL=y
+CONFIG_XDP_SOCKETS=y
+CONFIG_KNOD=m
+CONFIG_HSA_AMD=y
+CONFIG_HSA_AMD_KNOD_BPF=m
+CONFIG_DEBUG_FS=y
tools/testing/selftests/drivers/net/knod/knod_attach.sh new file +135 -0

135 lines not on this page · raw message

tools/testing/selftests/drivers/net/knod/knod_xdp_ktime.sh new file +173 -0

173 lines not on this page · raw message

tools/testing/selftests/drivers/net/knod/knod_xdp_loop.sh new file +129 -0

129 lines not on this page · raw message

tools/testing/selftests/drivers/net/knod/lib.sh new file +181 -0

181 lines not on this page · raw message

tools/testing/selftests/drivers/net/knod/xdp_ktime.bpf.c new file +32 -0
new file mode 100644
index 000000000000..5025e41cf3e8
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 2);
+	__type(key, __u32);
+	__type(value, __u64);
+} ktime_map SEC(".maps");
+
+SEC("xdp")
+int xdp_ktime_test(struct xdp_md *ctx)
+{
+	__u32 key_ts = 0;
+	__u32 key_cnt = 1;
+	__u64 ts = bpf_ktime_get_ns();
+	__u64 *cnt;
+	__u64 new_cnt;
+
+	bpf_map_update_elem(&ktime_map, &key_ts, &ts, BPF_ANY);
+
+	cnt = bpf_map_lookup_elem(&ktime_map, &key_cnt);
+	if (cnt) {
+		new_cnt = *cnt + 1;
+		bpf_map_update_elem(&ktime_map, &key_cnt, &new_cnt, BPF_ANY);
+	}
+
+	return XDP_PASS;
+}
+
+char LICENSE[] SEC("license") = "GPL";
tools/testing/selftests/drivers/net/knod/xdp_loop.bpf.c new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.43.0
#27 lore raw

Re: [RFC PATCH net-next 12/13] selftests: drivers/net: add knod tests

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] The tests hardcode relative paths to in-tree pyynl scripts and YAML specs, which breaks when…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The tests hardcode relative paths to in-tree pyynl scripts and YAML specs, which breaks when tests are run from an installed directory.
- [Medium] The map lookup script uses the GNU awk extension strtonum(), causing the test to unconditionally fail on systems using non-GNU awk (like mawk). --

--- Patch [12]: [RFC PATCH net-next 12/13] selftests: drivers/net: add knod tests --- commit 8c1df38ed71ba173e2fb3bc36bc8224d06cad0df
Author: Taehee Yoo <[email protected]>

selftests: drivers/net: add knod tests

Add attach/detach and XDP offload selftests for knod, exercising the genetlink control plane and the BPF data path over a knod-capable NIC.

Quoted 8 lines
tools/testing/selftests/drivers/net/knod/lib.sh new file +2 -0
new file mode 100755
index 0000000000000..d0d9b03f109dc
@@ -0,0 +1,181 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0

tools/testing/selftests/drivers/net/knod/lib.sh line 12

Quoted 10 lines
tools/testing/selftests/drivers/net/knod/lib.sh new file +10 -0
+#
+# lib.sh - KNOD XDP offload test utilities
+#
+# The KNOD control plane is the "knod" generic-netlink family; it is driven
+# here through the in-tree ynl CLI (tools/net/ynl/pyynl/cli.py) so the tests
+# need no dedicated user-space tool.
+
+KSRC=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../../.." && pwd)
+readonly KNOD_YNL="$KSRC/tools/net/ynl/pyynl/cli.py"
+readonly KNOD_SPEC="$KSRC/Documentation/netlink/specs/knod.yaml"

[Severity: Medium]
Will this hardcoded relative path cause the tests to fail when run from an installed directory?

When tests are installed (e.g., via make install) and run out-of-tree, these paths to cli.py and the YAML specs will likely be invalid since they are not copied to the install directory by default.

[ ... ]

Quoted 9 lines
tools/testing/selftests/drivers/net/knod/lib.sh new file +9 -0
+knod_map_lookup_u64() {
+	local map_id=$1
+	local key=$2
+	local hex
+
+	hex=$(bpftool map lookup id "$map_id" \
+	      key $key 0 0 0 2>/dev/null | \
+	      grep -o 'value:.*' | sed 's/value: //')
+	if [ -z "$hex" ]; then

tools/testing/selftests/drivers/net/knod/lib.sh line 180

Quoted 10 lines
tools/testing/selftests/drivers/net/knod/lib.sh new file +10 -0
+		echo 0
+		return
+	fi
+
+	printf '%d' "$(echo "$hex" | awk '{
+		v = 0;
+		for (i = 8; i >= 1; i--)
+			v = v * 256 + strtonum("0x" $i);
+		printf "0x%x", v;
+	}')"

[Severity: Medium]
Does the use of the strtonum() function break this script on systems using non-GNU awk?

The strtonum() function is a gawk-specific extension. On systems where the default awk is mawk or BusyBox awk, this may fail with an undefined function error, causing the extracted map value to evaluate to 0 and the test to falsely fail.

Quoted 1 line
tools/testing/selftests/drivers/net/knod/lib.sh new file +1 -0
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author
#28 lore raw

RFC PATCH 13/13 drm/amdkfd: add IPsec full-packet offload

In reply to Taehee Yoo

Add a second knod feature, alongside BPF, to show the accel ops are not tied to a single use case: GPU-offloaded IPsec ESP. xfrm SAs installed with XFRM_DEV_OFFLOAD_PACKET are handed to the GPU, which parses ESP, looks up the SA, checks anti-replay, does AES-GCM decrypt and writes the verdict back. Built as a separate module (knod_ipsec).

This is a functional proof of concept: it demonstrates that a full-packet crypto feature can run on top of the knod accelerator, not a hardened, production-ready implementation.

(cherry picked from commit 56ea4f90b43677e6944f1bc3ca4cf02e736ec340)
---
 drivers/gpu/drm/amd/amdkfd/Kconfig            |    9 +
 drivers/gpu/drm/amd/amdkfd/Makefile           |    3 +
 .../gpu/drm/amd/amdkfd/knod/aesgcm_shader.h   |  984 ++++
 .../drm/amd/amdkfd/knod/ipsec_fused_gfx10.h   | 1796 +++++++
 .../drm/amd/amdkfd/knod/ipsec_fused_gfx9.h    | 1349 ++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c  | 4273 +++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h  |  596 +++
 .../testing/selftests/drivers/net/knod/config |    1 +
 8 files changed, 9011 insertions(+)
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c
 create mode 100644 drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h
drivers/gpu/drm/amd/amdkfd/Kconfig +9 -0
index 708aa5fc051b..67efae53e395 100644
@@ -60,3 +60,12 @@ config HSA_AMD_KNOD_BPF
 	  the verdict path (PASS/DROP/TX) off the host CPU.
 
 	  If unsure, say N.
+
+config HSA_AMD_KNOD_IPSEC
+	tristate "KNOD IPsec (xfrm) full-packet offload"
+	depends on HSA_AMD_KNOD && XFRM_OFFLOAD && INET_ESP
+	help
+	  GPU-accelerated IPsec ESP full-packet offload via KNOD.
+	  Accepts xfrm SAs configured with XFRM_DEV_OFFLOAD_PACKET and
+	  performs ESP parse, SA lookup, anti-replay, AES-GCM decrypt and
+	  verdict writeback on the GPU.
drivers/gpu/drm/amd/amdkfd/Makefile +3 -0
index 4df3850e1466..c5d801ddabb7 100644
@@ -77,3 +77,6 @@ AMDKFD_FILES += $(AMDKFD_PATH)/kfd_knod.o
 endif
 
 obj-$(CONFIG_HSA_AMD_KNOD_BPF) += $(AMDKFD_PATH)/knod/knod_bpf.o
+
+
+obj-$(CONFIG_HSA_AMD_KNOD_IPSEC) += $(AMDKFD_PATH)/knod/knod_ipsec.o
drivers/gpu/drm/amd/amdkfd/knod/aesgcm_shader.h new file +984 -0

984 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h new file +1796 -0

1,796 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx9.h new file +1349 -0

202 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c new file +4273 -0

4,273 lines not on this page · raw message

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h new file +596 -0

596 lines not on this page · raw message

tools/testing/selftests/drivers/net/knod/config +1 -0
index a60ece9da112..837187205f89 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.43.0

Part of the diff is not shown here. View the raw message.

#29 lore raw

Re: [RFC PATCH net-next 13/13] drm/amdkfd: add IPsec full-packet offload

Thank you for your contribution! Sashiko AI review found 23 potential issue(s) to consider: - [Critical] The anti-replay sliding window is maintained per RX queue instead of globally per SA…

Thank you for your contribution! Sashiko AI review found 23 potential issue(s) to consider:
- [Critical] The anti-replay sliding window is maintained per RX queue instead of globally per SA, allowing attackers to bypass anti-replay protection.
- [Critical] An undersized packet causes an integer underflow in the GPU shader's ciphertext length calculation, leading to an out-of-bounds VRAM read that crashes the GPU (Remote Denial of Service).
- [High] The GPU shader does not include the high-order 32 bits of the Extended Sequence Number (seq_hi) in the AAD computation, causing all ESN packets to fail ICV verification.
- [High] The GPU shader is hardcoded to process exactly 254 ciphertext blocks, breaking Jumbo frame support and causing out-of-bounds VRAM reads on large packets.
- [High] Missing dma_wmb() in knod_ipsec_write_sa_entry causes a TOCTOU race condition between the CPU and the GPU shader, leading to GPU page faults.
- [High] If xa_insert fails during SA addition, the active GPU SA entry is not rolled back, leading to a GPU use-after-free.
- [High] Freeing SA VRAM allocations in xdo_state_free after an RCU grace period does not protect asynchronous GPU executions, causing a VRAM use-after-free.
- [High] The GPU verdict only returns the bare slot index without an SA version or SPI check. If a slot is freed and rapidly reallocated, an old decrypted packet will be incorrectly attributed to the new SA.
- [High] Premature KNOD_IPSEC_PASS assignment for bypassed/missed packets causes data corruption via early netmem recycling.
- [High] Unlocked access to slot->x in the dispatcher and NAPI hook races with xfrm_state teardown, leading to UAF.
- [High] Use-After-Free of ipsec_priv during device teardown due to missing RCU read lock in XFRM callbacks.
- [High] The in-kernel KAT selftest backs up, zeroes, and restores the live GPU SA table without locks, allowing concurrent traffic or configurations to permanently corrupt the table.
- [High] Unsafe allocation from page_pool in preemptible kthread context.
- [High] The SDMA engine writes up to 40 bytes past the end of the PAGE_SIZE delivery pool buffer for valid jumbo packets due to a flawed length check.
- [High] The CPU passes unmapped NIC DMA addresses from the page pool to the AMDGPU SDMA engine, which will cause a GPU VM or IOMMU fault.
- [High] Timeouts in knod_ipsec_fence_wait and knod_ipsec_dispatch_wait fail open, leading to processing of incomplete GPU/SDMA operations.
- [Medium] Use of this_cpu_inc() and this_cpu_ptr() in preemptible contexts causes smp_processor_id() warnings and stat corruption.
- [Medium] The RX shader checks the IP version but fails to verify that the protocol is ESP (IPPROTO_ESP / 50), allowing attackers to drop valid non-ESP traffic.
- [Medium] knod_ipsec_xdo_state_update_stats overwrites x->curlft instead of adding the hardware delta, destroying software-accumulated statistics.
- [Medium] Extremely large array allocation on the kernel stack in knod_ipsec_kat_h_one.
- [Medium] The self-test inadvertently unparks a dispatcher thread that may have been intentionally stopped by the control plane.
- [Medium] Missing RCU read lock around xa_load in knod_ipsec_xdo_state_update_stats.
- [Medium] Racy zeroing of remote per-CPU statistics.
--

--- Patch [13]: [RFC PATCH net-next 13/13] drm/amdkfd: add IPsec full-packet offload --- commit a531874de77887797921e4f4aa4a3418e089408c
Author: Taehee Yoo <[email protected]>
Subject: drm/amdkfd: add IPsec full-packet offload

This patch adds a KNOD feature for GPU-offloaded IPsec ESP, allowing XFRM_DEV_OFFLOAD_PACKET SAs to decrypt AES-GCM payloads directly on the GPU.

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf) · line 219

Quoted 8 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
@@ -216,8 +216,6 @@ static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf)
 	_E(emit_gfx10_s_and_b32_p, I10(buf, n), P_S(28), P_I(0xF), P_S(28));
 
 	/* Check IPv4 (version==4) */
 	_E(emit_gfx10_s_cmp_eq_u32, I10(buf, n), P_S(28), P_I(4));

[Severity: Medium]
Does the shader only check the IP version without verifying that the protocol is ESP (IPPROTO_ESP)? Could valid non-ESP traffic that accidentally matches an active SPI be dropped?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
 	br_ipv4 = _BR(emit_gfx10_s_cbranch_scc1, I10(buf, n), 0);
 

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf) · line 494

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
@@ -493,8 +491,6 @@ static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf)
 	_E(emit_gfx10_v_readfirstlane_b32, I10(buf, n), SR_CTEXT_LEN,
 	   VR_SAVE_PKTLEN);
 	_E(emit_gfx10_s_sub_u32_p, I10(buf, n), P_S(SR_CTEXT_LEN),
 	   P_S(SR_CTEXT_LEN), P_S(28));

[Severity: Critical]
If bd->len is smaller than the combined headers and ICV overhead, could this calculation underflow, producing a massive block count that leads to out-of-bounds VRAM reads and a GPU crash?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
 	/* s[SR_NBLOCKS_GCM] = (ctext_len + 15) >> 4 */
 	_E(emit_gfx10_s_add_u32, I10(buf, n), P_S(SR_NBLOCKS_GCM), P_I(15),

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf) · line 625

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
@@ -626,8 +622,6 @@ static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf)
 	 * here then free for reuse by the ctext section.
 	 */
 	_E(emit_gfx10_v_sub_nc_u32, I10(buf, n), P_V(VR_TMP),
 	   P_S(SR_TOTAL_GHASH_BLK), P_V(VR_TID));

[Severity: High]
With the workgroup size hardcoded to 256 threads, won't packets exceeding 254 ciphertext blocks (like MTU 9000 jumbo frames) fail decryption and cause out-of-bounds VRAM reads when accessing the H-power table?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
 	_E(emit_gfx10_v_add_nc_u32, I10(buf, n), P_V(VR_TMP),
 	   P_L(0xFFFFFFFF), P_V(VR_TMP));

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf) · line 647

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
@@ -649,8 +643,6 @@ static inline int kfd_ipsec_gen_fused_shader_gfx10(void *vbuf)
 	 */
 	_E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA0),
 	   P_V(VR_SAVE_SPI));
 	_E(emit_gfx10_v_mov_b32_e32, I10(buf, n), P_V(VR_DATA1),
 	   P_V(VR_SAVE_SEQ));

[Severity: High]
If the control plane accepts SAs with ESN enabled, but the shader only uses the 32-bit seq_lo when assembling the AAD, won't all valid ESN packets fail the ICV verification?

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h struct knod_ipsec_sa_slot { · line 268

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/knod/ipsec_fused_gfx10.h +0 -0
 	/* DATA2, DATA3 already 0 */
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h +0 -0
@@ -266,4 +266,3 @@ struct knod_ipsec_sa_slot {
 	 * queue - do not touch from control plane while SA is active.
 	 */
 	struct knod_ipsec_sa_window win[KNOD_SPSC_MAX];

[Severity: Critical]
Does maintaining the anti-replay sliding window per RX queue rather than globally per SA allow an attacker to bypass replay protection by altering outer headers to change the RSS hash?

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_write_sa_entry(struct knod_ipsec_priv *priv, · line 331

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.h +0 -0
 };
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -329,4 +329,3 @@ static void knod_ipsec_write_sa_entry(struct knod_ipsec_priv *priv,
 	e = (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr;
 	e += slot_idx;
 	memset(e, 0, sizeof(*e));

[Severity: High]
Could clearing the SA entry here without a dma_wmb() allow the GPU shader to match a valid spi but read a zeroed key_gpu_addr or htable_gpu_addr, causing a fatal GPU page fault?

Quoted 2 lines

if (!slot || !x) {

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_xdo_state_add(struct knod_dev *knodev, · line 425

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -424,4 +423,3 @@ static int knod_ipsec_xdo_state_add(struct knod_dev *knodev,
 
 	if (x->props.flags & XFRM_STATE_ESN)
 		flags |= 1u << 0;

[Severity: High]
Are ESN packets expected to be supported by the shader? If the control plane accepts SAs with ESN enabled, but the shader lacks full seq_hi support for the AAD, won't all valid ESN packets drop?

Quoted 2 lines

/* Store SPI in host byte order (little-endian on this platform).

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_xdo_state_add(struct knod_dev *knodev, · line 493

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -493,4 +491,3 @@ static int knod_ipsec_xdo_state_add(struct knod_dev *knodev,
 	err = xa_insert(&priv->spi_to_slot, spi, slot, GFP_KERNEL);
 	if (err)
 		goto err_replay;

[Severity: High]
If xa_insert() fails here, the error path frees the VRAM allocations but leaves the SA entry active in the GPU sa_table. Could a concurrent GPU dispatch dereference these freed VRAM allocations?

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_xdo_state_add(struct knod_dev *knodev,

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	x->xso.offload_handle = (unsigned long)slot;
 
 	mutex_unlock(&priv->slot_lock);
 
 	this_cpu_inc(priv->stats->sa_add);

[Severity: Medium]
Can using this_cpu_inc() in process contexts trigger smp_processor_id() warnings and corrupt statistics upon migration?

Quoted 2 lines

e_dbg = (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr + slot_idx;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_xdo_state_delete(struct knod_dev *knodev, · line 523

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -524,4 +521,3 @@ static void knod_ipsec_xdo_state_delete(struct knod_dev *knodev,
 	spi = be32_to_cpu(x->id.spi);
 
 	mutex_lock(&priv->slot_lock);

[Severity: High]
Since XFRM callbacks like knod_ipsec_xdo_state_delete() can be invoked from process context without rcu_read_lock(), could knod_ipsec_nod_exit() free priv concurrently, causing a Use-After-Free?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	slot = knod_ipsec_lookup_slot_by_spi(priv, spi);
 	if (!slot) {

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_xdo_state_free(struct knod_dev *knodev, · line 559

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -561,4 +557,3 @@ static void knod_ipsec_xdo_state_free(struct knod_dev *knodev,
 		knod_free_mem(priv->knod, slot->htable_mem);
 	if (slot->key_mem)
 		knod_free_mem(priv->knod, slot->key_mem);

[Severity: High]
Because the GPU dispatcher doesn't participate in the CPU RCU grace period, is it possible for an in-flight GPU dispatch to access key_mem or htable_mem while it is being freed here?

Quoted 2 lines

memset(slot, 0, sizeof(*slot));

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_xdo_state_update_stats(struct knod_dev *knodev, · line 593

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -597,4 +592,3 @@ static void knod_ipsec_xdo_state_update_stats(struct knod_dev *knodev,
 	spi = be32_to_cpu(x->id.spi);
 	slot = xa_load(&priv->spi_to_slot, spi);

[Severity: Medium]
Since this can be called from process context without holding the RCU read lock, is it safe to call xa_load() here?

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_xdo_state_update_stats(struct knod_dev *knodev,

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	if (!slot || !slot->active)
 		return;
 
 	/* Read GPU-side per-SA counters (atomically updated by shader). */
 	gs = (struct knod_ipsec_sa_gpu_stats *)
 		((u8 *)priv->sa_table->kaddr + KNOD_IPSEC_STATS_REGION_OFF +
 		 slot->slot_idx * KNOD_IPSEC_SA_STATS_SIZE);
 
 	x->curlft.packets = le64_to_cpu(READ_ONCE(gs->rx_packets));
 	x->curlft.bytes   = le64_to_cpu(READ_ONCE(gs->rx_bytes));

[Severity: Medium]
Does assigning the hardware counter directly to x->curlft.packets overwrite any software-accumulated statistics (e.g., from bypassed packets)? Should this add the hardware delta instead?

Quoted 2 lines

}

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_nod_exit(struct knod_dev *knodev) · line 880

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -885,4 +878,3 @@ static void knod_ipsec_nod_exit(struct knod_dev *knodev)
 	knodev->post_copy = NULL;
 	WRITE_ONCE(ipsec_priv, NULL);
 	synchronize_net();

[Severity: High]
Is it safe to assume all readers are finished here? As asked earlier, if XFRM callbacks run without rcu_read_lock(), wouldn't this synchronize_net() fail to wait for them, resulting in a Use-After-Free of priv below?

Quoted 2 lines

knod_ipsec_disp_destroy_all(priv);

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_fill_dispatch(struct knod *knod, · line 1281

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1287,4 +1279,3 @@ static void knod_ipsec_fill_dispatch(struct knod *knod,
 {
 	p->workgroup_size_x = 256;
 	p->grid_size_x = 256;

[Severity: High]
With the workgroup size hardcoded to 256 threads, won't packets exceeding 254 ciphertext blocks (like MTU 9000 jumbo frames) fail decryption?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	p->grid_size_y = max(work->nr_packets, 1);
 	p->private_segment_size = 0;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_fence_wait(struct knod_ipsec_work *w) · line 1399

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1406,4 +1397,3 @@ static void knod_ipsec_fence_wait(struct knod_ipsec_work *w)
 	while (!knod_ipsec_fence_passed(w)) {
 		if (--timeout <= 0)
 			break;

[Severity: High]
If the bounded spin loops in knod_ipsec_fence_wait() and knod_ipsec_dispatch_wait() time out, does the dispatcher unconditionally proceed and process incomplete GPU/SDMA operations?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		cpu_relax();
 	}

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp, · line 1505

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1513,4 +1503,3 @@ static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp,
 			n_sdma++;
 
 			bd->act = KNOD_IPSEC_PASS;

[Severity: High]
For bypassed or missed packets, does assigning KNOD_IPSEC_PASS immediately after submitting the asynchronous SDMA copy allow the NIC NAPI to recycle and overwrite the source buffer before the copy completes?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 			continue;
 		}

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp, · line 1525

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1535,4 +1524,3 @@ static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp,
 		inner_len = bd->len;
 		if (inner_len == 0 || inner_len > PAGE_SIZE) {

[Severity: High]
Since copies[0].dst is advanced by l3_hdr_len below, could an inner_len close to PAGE_SIZE cause the SDMA engine to write up to 40 bytes past the end of the PAGE_SIZE delivery pool buffer?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 			if (stats_on)
 				s->rx_drop_malformed++;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp, · line 1533

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1544,4 +1532,3 @@ static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp,
 
 		netmem = pool ? page_pool_dev_alloc_netmems(pool) : 0;

[Severity: High]
Because the dispatcher kthread runs with preemption enabled, can calling page_pool_dev_alloc_netmems() corrupt the page_pool unprotected allocation cache?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		if (!netmem) {
 			if (stats_on)

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp, · line 1542

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1553,4 +1540,3 @@ static void knod_ipsec_finish_rx_deliver(struct knod_ipsec_dispatcher *disp,
 		pkt_gaddr = le64_to_cpu(param->sub[i].pkt_addr);
 		out_gaddr = le64_to_cpu(param->sub[i].out_addr);
 		dst_base = page_pool_get_dma_addr_netmem(netmem);

[Severity: High]
Since these host pages are retrieved from the NIC page pool, are they mapped into the AMDGPU virtual address space or IOMMU domain? Could passing them to the SDMA engine trigger a GPU VM or IOMMU fault?

Quoted 2 lines

if (pkt_mode == XFRM_MODE_TRANSPORT) {

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb, · line 1773

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1785,4 +1771,3 @@ static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb,
 
 	slot = &priv->slots[sa_slot];
 	if (!slot->active) {

[Severity: High]
If an SA is rapidly deleted and its slot reallocated while its decrypted packets are still pending in the delivery ring, could this accept those packets under the new SA context since there is no SPI or version validation?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		if (stats_on)
 			s->rx_drop_no_sa++;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb, · line 1786

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1800,4 +1785,3 @@ static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb,
 	seq = ((u64)desc->feat.seq_hi << 32) | (u64)desc->feat.seq_lo;
 	if (!knod_ipsec_sa_window_check(&slot->win[queue_idx], seq)) {

[Severity: Critical]
Does maintaining the anti-replay sliding window per RX queue here rather than globally per SA allow an attacker to bypass replay protection by altering outer headers to change the RSS hash?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		if (stats_on)
 			s->rx_drop_replay++;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb, · line 1847

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1861,4 +1845,3 @@ static bool knod_ipsec_post_copy(struct knod_dev *knodev, struct sk_buff *skb,
 			return false;
 		}
 		xfrm_state_hold(slot->x);

[Severity: High]
Is it safe to access and take a reference on slot->x here without locks? If the SA is deleted concurrently, could this lead to a Use-After-Free when accessing the freed xfrm_state?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		sp->xvec[sp->len++] = slot->x;
 		sp->olen++;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_dispatch_wait(struct knod_ipsec_dispatcher *disp, · line 1967

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -1983,4 +1966,3 @@ static void knod_ipsec_dispatch_wait(struct knod_ipsec_dispatcher *disp,
 	while (READ_ONCE(signal->value) > work->sigval) {
 		if (--timeout <= 0) {

[Severity: High]
Does breaking early on this timeout cause the dispatcher to process incomplete GPU/SDMA operations?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 			pr_warn_ratelimited("knod_ipsec: GPU signal timeout nr=%d\n",
 					    work->nr_packets);

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_dispatcher_try_rx(struct knod_ipsec_dispatcher *disp,

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2166,4 +2148,3 @@ static bool knod_ipsec_dispatcher_try_rx(struct knod_ipsec_dispatcher *disp,
 				param->sub[si].bd_addr  =
 					cpu_to_le64(bd_gaddr);
 				param->sub[si].pkt_len  =
 					cpu_to_le32(bd->len);

[Severity: Critical]
If bd->len is smaller than the combined headers and ICV overhead, could this calculation underflow later in the shader, producing a massive block count that leads to out-of-bounds VRAM reads and a GPU crash?

Quoted 2 lines

param->sub[si].result_seq = 0;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_dispatcher_try_rx(struct knod_ipsec_dispatcher *disp, · line 2161

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2178,4 +2159,3 @@ static bool knod_ipsec_dispatcher_try_rx(struct knod_ipsec_dispatcher *disp,
 
 	if (static_branch_unlikely(&ipsec_stats_enabled_key)) {
 		struct knod_ipsec_stats *cs = this_cpu_ptr(priv->stats);

[Severity: Medium]
Can using this_cpu_ptr() here in preemptible kthread contexts trigger smp_processor_id() warnings and corrupt statistics upon migration?

Quoted 2 lines

cs->rx_peek_total += total_n;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static bool knod_ipsec_finalise_inflight(struct knod_ipsec_dispatcher *disp, · line 2310

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2328,4 +2308,3 @@ static bool knod_ipsec_finalise_inflight(struct knod_ipsec_dispatcher *disp,
 	if (work->n_sdma_pending > 0) {
 		if (stats_on) {
 			struct knod_ipsec_stats *s = this_cpu_ptr(priv->stats);

[Severity: Medium]
Is it safe to use this_cpu_ptr() here when preemption isn't disabled?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 			u64 build_ns = work->t_build_end - work->t_build_start;
 			u64 gpu_ns = t_gpu_end > work->t_submit

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static void knod_ipsec_dispatcher_drain(struct knod_ipsec_dispatcher *disp) · line 2562

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2581,4 +2560,3 @@ static void knod_ipsec_dispatcher_drain(struct knod_ipsec_dispatcher *disp)
 		if (w->state == KNOD_WORK_INFLIGHT) {
 			knod_ipsec_dispatch_wait(disp, w);
 			if (!knod_ipsec_finalise_inflight(disp, w)) {

[Severity: High]
If knod_ipsec_dispatch_wait() times out, does this unconditionally proceed to finalise the work item with incomplete GPU results?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 				/*
 				 * SDMA started - spin-wait, we can't

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_kat_h_one(const u8 *key, int key_len) · line 2761

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2782,4 +2760,3 @@ static int knod_ipsec_kat_h_one(const u8 *key, int key_len)
 {
 	u8 h_table[KNOD_GCM_H_TABLE_SIZE];

[Severity: Medium]
Given that KNOD_GCM_H_TABLE_SIZE is 4096 bytes, could allocating this array on the kernel stack cause a stack overflow?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	struct aes_enckey enckey;
 	u8 zero[AES_BLOCK_SIZE] = {};

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_run_shader_kat_n(struct knod_ipsec_priv *priv, int nr)

Quoted 5 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -2868,4 +2845,3 @@ static int knod_ipsec_run_shader_kat_n(struct knod_ipsec_priv *priv, int nr)
 	 * ALIGNed to 8 pages in the alloc path to dodge the KNOD VRAM
 	 * 7-page alloc bug; the logical payload is still SA_BO_SIZE.
 	 */
 	memset(e, 0, KNOD_IPSEC_SA_BO_SIZE);

[Severity: High]
Does clearing the live GPU SA table here without holding priv->slot_lock permanently overwrite any SA modifications made by concurrent administrative operations?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	for (i = 0; i < nr; i++) {
 		e[i].spi    = cpu_to_le32(SPI_KAT_BASE | (u32)i);

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_run_shader_kat_n(struct knod_ipsec_priv *priv, int nr) · line 2990

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -3012,4 +2988,3 @@ static int knod_ipsec_run_shader_kat_n(struct knod_ipsec_priv *priv, int nr)
 	knod_ipsec_dispatch_and_wait(&priv->disp[0], dbg_work);
 	if (priv->disp[0].kthread)
 		kthread_unpark(priv->disp[0].kthread);

[Severity: Medium]
If the device was previously stopped via knod_ipsec_nod_stop(), does this unconditionally unpark the thread and violate the stopped state of the device?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 	ret = 0;
 	pr_info("knod_ipsec: shader-dispatch[%d] submitted, signal %lld -> (done)\n",

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static int knod_ipsec_run_crypto_kat(struct knod_ipsec_priv *priv) · line 3315

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -3338,4 +3313,3 @@ static int knod_ipsec_run_crypto_kat(struct knod_ipsec_priv *priv)
 	}
 	memcpy(sa_backup, priv->sa_table->kaddr, KNOD_IPSEC_SA_BO_SIZE);
 	memset(priv->sa_table->kaddr, 0, KNOD_IPSEC_SA_BO_SIZE);

[Severity: High]
Can modifying the live SA table here without locks cause permanent corruption if a user adds or deletes an SA during this selftest?

Quoted 2 lines

sa_entry = (struct knod_ipsec_sa_entry *)priv->sa_table->kaddr;

[ ... ]

drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c static ssize_t knod_ipsec_stats_reset_write(struct file *file, · line 3910

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
@@ -3934,4 +3908,3 @@ static ssize_t knod_ipsec_stats_reset_write(struct file *file,
 		u64 sa_rekey = s->sa_rekey;
 
 		memset(s, 0, sizeof(*s));

[Severity: Medium]
Does using memset() on remote per-CPU statistics without synchronization cause data races and corruption if those CPUs are concurrently updating their local stats?

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/knod/knod_ipsec.c +0 -0
 		s->sa_add = sa_add;
 		s->sa_del = sa_del;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
#30 lore raw
In reply to Taehee Yoo
Quoted 36 lines · Taehee Yoo

knod (in-kernel network offload device) drives a GPU directly from the Linux kernel - with no userspace GPU runtime such as CUDA or ROCm, and no userspace component in the data path - to accelerate packet processing.

The kernel itself allocates the GPU's queues, JIT-compiles the per-packet program to GPU machine code, and dispatches it; the NIC DMAs received packets straight into GPU memory and the GPU returns a verdict. The GPU is programmed like any other in-kernel offload, not through a userspace framework, so existing XDP programs and IPsec SAs can be offloaded to it transparently.

The design and motivation were presented at Linux Plumbers Conference 2025:

https://lpc.events/event/19/contributions/2267/

Motivation
==========

Line-rate packet processing that does non-trivial per-packet work - an XDP program doing L4 load balancing, or IPsec crypto - is bound by the host CPU: each core handles one packet at a time, so scaling means spending more cores. A GPU is the opposite shape - thousands of lanes running the same small program over many packets at once (SIMT) - which happens to match the per-packet-program model of XDP.

knod moves that per-packet compute off the host CPU and onto a GPU. The NIC DMAs received packets directly into GPU memory, the GPU runs the program across a batch of packets in parallel, and only the result comes back: a verdict for every packet, plus the packet itself for the ones destined to the host. The CPU no longer pays the per-packet program cost, and throughput scales with GPU occupancy rather than core count.

Quoted 7 lines

Crucially this happens entirely inside the kernel. GPU packet processing today generally launches work from a userspace GPU runtime (CUDA and friends) and keeps that runtime in the data path; knod instead builds the GPU queues, compiles the program, and dispatches it from the kernel, so it plugs into existing offload paths (XDP, xfrm) with nothing to install or keep running in userspace.

TBH I found this motivation weak, especially since IIUC you're asking for almost ~50K lines of code to be merged to the kernel. My thinking is that (a) it's true that Native XDP takes up CPU cores, but HW-offload XDP already exists and takes up no CPU. (b) with AI, GPUs are very expensive and the work they're doing is critical, so you're unlikely to buy a GPU and use it for knod-offloaded XDP; you'd probably buy a cheaper smart-NIC? And in the cases where you do have a GPU, it's likely your system's money-maker, and you probably want to offload work from the GPU, not to your GPU.

But as far as I can tell there should be much more interesting applications for what you're doing rather than offloading XDP. Like wouldn't you with this feature be able to implement ML collectively like all-to-all/all-reduce/all-gather as purely XDP programs offloaded to the GPU? If you have that implemented and can positively compare performance to RoCE or RDMA that would be a much more interesting use case IMO.

Quoted 56 lines

Model
=====

knod binds two endpoints through a third object:

- a NIC-side netdev, registered by the NIC driver;
- an accelerator (the GPU), registered by a provider built into the

    GPU driver;

- an offload device, created on attach, that connects them and owns the per-queue data-path state.

The accelerator ops are feature-agnostic, so different per-packet programs plug in behind one interface. This series ships two features to show the framework is not tied to a single use case (the accel-type uAPI also reserves "dpu" for future non-GPU backends):

- BPF: an XDP program attached in offload mode is JIT-compiled from
eBPF to an AMD GCN shader and executed on the GPU.
- IPsec: RX ESP full-packet decrypt on the GPU (proof of concept, see below).

Data path
=========

For a NIC RX queue bound to an accelerator, a received packet flows as follows:

1. Attach reconfigures the NIC's page_pool so its pages come from GPU memory, exported as a dma-buf and plugged in through the devmem memory provider. The NIC therefore DMAs the packet straight into GPU memory - there is no host-side copy on RX.

2. Instead of building an skb and entering the stack, the NIC's NAPI pushes a small descriptor (memory ref + offset + length) onto a per-queue lock-free SPSC ring shared with the GPU.

3. A persistent GPU worker drains descriptors and runs the active feature's shader over a batch of packets in parallel - one workgroup per packet - then writes a per-packet verdict back onto the ring: PASS, DROP or TX.

4. Back on the NIC NAPI, each verdict is applied:
- DROP: the GPU page is recycled to the pool;
- TX: the packet is sent back out the NIC directly from GPU
memory (XDP_TX), with no host round-trip;
- PASS: the packet is copied out of GPU memory into a host delivery page by the GPU's DMA engine (async SDMA), wrapped as an skb, and handed to the normal stack.

So only PASS packets ever touch host memory, and even then the copy is done by the GPU, not the CPU.

The control plane is a generic netlink family ("knod") for attach/detach, accelerator/NIC inventory, and per-accelerator feature selection, with notifications on bind/unbind.

Quoted 9 lines

Code layout
===========

knod separates into a subsystem-neutral core and a GPU-specific provider; nothing in the core knows about GPUs. Roughly:

  - core + control plane + NIC drivers   net/, kernel/bpf/       (~4.4k)
  - GPU provider (queues, JIT, shaders)  drivers/gpu/drm/amd/    (~39k)

Since the changes are so large, I would break this work into many series, the first which only supports a very minimal set of work (which I would guess would be XDP_DROP and XDP_TX), and then follow up with another series which adds XDP_PASS support, and then IPSec, etc.

Signature · 2 lines
Thanks,
Mina
author
#31 lore raw
Quoted 1 line · Mina Almasry

Hi Mina,
Thanks a lot for your review!

Quoted 45 lines · Taehee Yoo
Quoted 36 lines

knod (in-kernel network offload device) drives a GPU directly from the Linux kernel - with no userspace GPU runtime such as CUDA or ROCm, and no userspace component in the data path - to accelerate packet processing.

The kernel itself allocates the GPU's queues, JIT-compiles the per-packet program to GPU machine code, and dispatches it; the NIC DMAs received packets straight into GPU memory and the GPU returns a verdict. The GPU is programmed like any other in-kernel offload, not through a userspace framework, so existing XDP programs and IPsec SAs can be offloaded to it transparently.

The design and motivation were presented at Linux Plumbers Conference 2025:

https://lpc.events/event/19/contributions/2267/

Motivation
==========

Line-rate packet processing that does non-trivial per-packet work - an XDP program doing L4 load balancing, or IPsec crypto - is bound by the host CPU: each core handles one packet at a time, so scaling means spending more cores. A GPU is the opposite shape - thousands of lanes running the same small program over many packets at once (SIMT) - which happens to match the per-packet-program model of XDP.

knod moves that per-packet compute off the host CPU and onto a GPU. The NIC DMAs received packets directly into GPU memory, the GPU runs the program across a batch of packets in parallel, and only the result comes back: a verdict for every packet, plus the packet itself for the ones destined to the host. The CPU no longer pays the per-packet program cost, and throughput scales with GPU occupancy rather than core count.

Quoted 7 lines

Crucially this happens entirely inside the kernel. GPU packet processing today generally launches work from a userspace GPU runtime (CUDA and friends) and keeps that runtime in the data path; knod instead builds the GPU queues, compiles the program, and dispatches it from the kernel, so it plugs into existing offload paths (XDP, xfrm) with nothing to install or keep running in userspace.

Quoted 10 lines

TBH I found this motivation weak, especially since IIUC you're asking for almost ~50K lines of code to be merged to the kernel. My thinking is that (a) it's true that Native XDP takes up CPU cores, but HW-offload XDP already exists and takes up no CPU. (b) with AI, GPUs are very expensive and the work they're doing is critical, so you're unlikely to buy a GPU and use it for knod-offloaded XDP; you'd probably buy a cheaper smart-NIC? And in the cases where you do have a GPU, it's likely your system's money-maker, and you probably want to offload work from the GPU, not to your GPU.

Thanks, these are fair concerns.

For a system where the GPU is already fully utilized, a SmartNIC may indeed be the better target for XDP offload. We are not arguing that GPU offload is always cheaper or more efficient.

The goal of KNOD is to provide an upstream path for using programmable accelerators already available in the system for kernel networking. Hardware XDP offload exists, but the available functionality is tied to the NIC, its firmware, and the features maintained by the vendor. KNOD instead aims to provide an execution layer that can be extended in the kernel community. XDP is the first workload used to validate this layer, rather than the intended limit of KNOD.

Quoted 8 lines

But as far as I can tell there should be much more interesting applications for what you're doing rather than offloading XDP. Like wouldn't you with this feature be able to implement ML collectively like all-to-all/all-reduce/all-gather as purely XDP programs offloaded to the GPU? If you have that implemented and can positively compare performance to RoCE or RDMA that would be a much more interesting use case IMO.

I agree that this is a promising direction, and collective communication could be a good fit for this model. At least for all-reduce, however, BPF/XDP does not currently support the required floating-point operations. If that support is discussed and added by the BPF community, this would be an interesting use case to explore with KNOD and evaluate against RoCE or RDMA.

Quoted 62 lines
Quoted 62 lines

Model
=====

knod binds two endpoints through a third object:

- a NIC-side netdev, registered by the NIC driver;
- an accelerator (the GPU), registered by a provider built into the

    GPU driver;

- an offload device, created on attach, that connects them and owns the per-queue data-path state.

The accelerator ops are feature-agnostic, so different per-packet programs plug in behind one interface. This series ships two features to show the framework is not tied to a single use case (the accel-type uAPI also reserves "dpu" for future non-GPU backends):

- BPF: an XDP program attached in offload mode is JIT-compiled from
eBPF to an AMD GCN shader and executed on the GPU.
- IPsec: RX ESP full-packet decrypt on the GPU (proof of concept, see below).

Data path
=========

For a NIC RX queue bound to an accelerator, a received packet flows as follows:

1. Attach reconfigures the NIC's page_pool so its pages come from GPU memory, exported as a dma-buf and plugged in through the devmem memory provider. The NIC therefore DMAs the packet straight into GPU memory - there is no host-side copy on RX.

2. Instead of building an skb and entering the stack, the NIC's NAPI pushes a small descriptor (memory ref + offset + length) onto a per-queue lock-free SPSC ring shared with the GPU.

3. A persistent GPU worker drains descriptors and runs the active feature's shader over a batch of packets in parallel - one workgroup per packet - then writes a per-packet verdict back onto the ring: PASS, DROP or TX.

4. Back on the NIC NAPI, each verdict is applied:
- DROP: the GPU page is recycled to the pool;
- TX: the packet is sent back out the NIC directly from GPU
memory (XDP_TX), with no host round-trip;
- PASS: the packet is copied out of GPU memory into a host delivery page by the GPU's DMA engine (async SDMA), wrapped as an skb, and handed to the normal stack.

So only PASS packets ever touch host memory, and even then the copy is done by the GPU, not the CPU.

The control plane is a generic netlink family ("knod") for attach/detach, accelerator/NIC inventory, and per-accelerator feature selection, with notifications on bind/unbind.

Code layout
===========

knod separates into a subsystem-neutral core and a GPU-specific provider; nothing in the core knows about GPUs. Roughly:

Quoted 9 lines
Quoted 3 lines
  - core + control plane + NIC drivers   net/, kernel/bpf/       (~4.4k)
  - GPU provider (queues, JIT, shaders)  drivers/gpu/drm/amd/    (~39k)

Since the changes are so large, I would break this work into many series, the first which only supports a very minimal set of work (which I would guess would be XDP_DROP and XDP_TX), and then follow up with another series which adds XDP_PASS support, and then IPSec, etc.

That makes sense. We will split the work and start with the minimum functionality needed to demonstrate the architecture, likely XDP_DROP and XDP_TX. XDP_PASS, IPsec, and the remaining functionality can follow in separate series.

Thanks a lot!
Taehee Yoo

Quoted 3 lines

--
Thanks,
Mina