Skip to contentSkip to conversation

command-line registry and gsp-rm logging

PATCH New Timur Tabi 3 participants 9 messages Feb 12, 2024 – Feb 14, 2024

Subsystems (3)
lore
Share & tools
lore

Timur Tabi

author
#0 lore raw

cover command-line registry and gsp-rm logging

Two patches that were previosly posted, but now updated for drm-next.

Timur Tabi (2):
[v3] nouveau: add command-line GSP-RM registry support [v3] drm/nouveau: expose GSP-RM logging buffers via debugfs

 .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h |  18 +
 .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c    | 494 +++++++++++++++++-
 2 files changed, 484 insertions(+), 28 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v3 1/2 nouveau: add command-line GSP-RM registry support

Add the NVreg_RegistryDwords command line parameter, which allows specifying additional registry keys to be sent to GSP-RM. This allows additional configuration, debugging, and experimentation with GSP-RM, which uses these keys to alter its behavior.

Note that these keys are passed as-is to GSP-RM, and Nouveau does not parse them. This is in contrast to the Nvidia driver, which may parse some of the keys to configure some functionality in concert with GSP-RM. Therefore, any keys which also require action by the driver may not function correctly when passed by Nouveau. Caveat emptor.

The name and format of NVreg_RegistryDwords is the same as used by the Nvidia driver, to maintain compatibility.

---
v3: rebased to drm-next

 .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h |   6 +
 .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c    | 279 ++++++++++++++++--
 2 files changed, 261 insertions(+), 24 deletions(-)
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +6 -0
index 6f5d376d8fcc..3fbc57b16a05 100644
@@ -211,6 +211,12 @@ struct nvkm_gsp {
 		struct mutex mutex;;
 		struct idr idr;
 	} client_id;
+
+	/* A linked list of registry items. The registry RPC will be built from it. */
+	struct list_head registry_list;
+
+	/* The size of the registry RPC */
+	size_t registry_rpc_size;
 };
 
 static inline bool
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +255 -24

312 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/2] [v3] nouveau: add command-line GSP-RM registry support

Quoted 53 lines · Timur Tabi

Add the NVreg_RegistryDwords command line parameter, which allows specifying additional registry keys to be sent to GSP-RM. This allows additional configuration, debugging, and experimentation with GSP-RM, which uses these keys to alter its behavior.

Note that these keys are passed as-is to GSP-RM, and Nouveau does not parse them. This is in contrast to the Nvidia driver, which may parse some of the keys to configure some functionality in concert with GSP-RM. Therefore, any keys which also require action by the driver may not function correctly when passed by Nouveau. Caveat emptor.

The name and format of NVreg_RegistryDwords is the same as used by the Nvidia driver, to maintain compatibility.

---
v3: rebased to drm-next

  .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h |   6 +
  .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c    | 279 ++++++++++++++++--
  2 files changed, 261 insertions(+), 24 deletions(-)
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +6 -0
index 6f5d376d8fcc..3fbc57b16a05 100644
@@ -211,6 +211,12 @@ struct nvkm_gsp {
  		struct mutex mutex;;
  		struct idr idr;
  	} client_id;
+
+	/* A linked list of registry items. The registry RPC will be built from it. */
+	struct list_head registry_list;
+
+	/* The size of the registry RPC */
+	size_t registry_rpc_size;
  };
 
  static inline bool
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -0
index 1c46e3f919be..86b62c7e1229 100644
@@ -54,6 +54,8 @@
  #include <nvrm/535.113.01/nvidia/kernel/inc/vgpu/rpc_global_enums.h>
 
  #include <linux/acpi.h>
+#include <linux/ctype.h>
+#include <linux/parser.h>
 
  #define GSP_MSG_MIN_SIZE GSP_PAGE_SIZE
  #define GSP_MSG_MAX_SIZE GSP_PAGE_MIN_SIZE * 16

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1088

Quoted 5 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -0
@@ -1082,51 +1084,280 @@ r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend)
  	return nvkm_gsp_rpc_wr(gsp, rpc, true);
  }
 
+struct registry_list_entry {
+	struct list_head list;

Nit: 'head' or 'entry' might be more suitable.

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1090

Quoted 2 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -0
+	size_t name_len;
+	u32 type;

I prefer to represent type as enum or use a define for '1 = integer' instead. This also gets you rid of the need to explicitly explain it's meaning in the documentation of add_registry().

Quoted 6 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +6 -0
+	u32 data;
+	u32 length;
+	char name[];
+};
+
+/**

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1106

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +10 -0
+ * add_registry -- adds a registry entry
+ * @gsp: gsp pointer
+ * @name: name of the registry key
+ * @type: type of data (1 = integer)
+ * @data: value
+ * @length: size of data, in bytes
+ *
+ * Adds a registry key/value pair to the registry database.
+ *
+ * Currently, only 32-bit integers (type == 1, length == 4) are supported.

What if we pass something else? This function doesn't seem to ensure nothing else is accepted. Although I recognize that it's only used by add_registry_num() enforcing this limitation by it's function signature.

Quoted 7 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +7 -0
+ *
+ * This function collects the registry information in a linked list.  After
+ * all registry keys have been added, build_registry() is used to create the
+ * RPC data structure.
+ *
+ * registry_rpc_size is a running total of the size of all registry keys.
+ * It's used to avoid an O(n) calculation of the size when the RPC is built.

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1123

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +10 -0
+ *
+ * Returns 0 on success, or negative error code on error.
+ */
+static int add_registry(struct nvkm_gsp *gsp, const char *name, u32 type, u32 data, u32 length)
+{
+	struct registry_list_entry *reg;
+	size_t nlen = strlen(name) + 1;
+
+	/* Set an arbitrary limit to avoid problems with broken command lines */
+	if (strlen(name) > 64)

Could re-use nlen for this check.

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1124

Quoted 1 line
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +1 -0
+		return -EFBIG;

This error code doesn't seem to apply here, prefer EINVAL.

Quoted 214 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +186 -10
+
+	reg = kmalloc(sizeof(struct registry_list_entry) + nlen, GFP_KERNEL);
+	if (!reg)
+		return -ENOMEM;
+
+	memcpy(reg->name, name, nlen);
+	reg->name_len = nlen;
+	reg->type = type;
+	reg->data = data;
+	reg->length = length;
+
+	nvkm_debug(&gsp->subdev, "adding GSP-RM registry '%s=%u'\n", name, data);
+	list_add_tail(&reg->list, &gsp->registry_list);
+	gsp->registry_rpc_size += sizeof(PACKED_REGISTRY_ENTRY) + nlen;
+
+	return 0;
+}
+
+static int add_registry_num(struct nvkm_gsp *gsp, const char *name, u32 value)
+{
+	return add_registry(gsp, name, 1, value, sizeof(u32));
+}
+
+/**
+ * build_registry -- create the registry RPC data
+ * @gsp: gsp pointer
+ * @registry: pointer to the RPC payload to fill
+ *
+ * After all registry key/value pairs have been added, call this function to
+ * build the RPC.
+ *
+ * The registry RPC looks like this:
+ *
+ * +-----------------+
+ * |NvU32 size;      |
+ * |NvU32 numEntries;|
+ * +-----------------+
+ * +---------------------+
+ * |PACKED_REGISTRY_ENTRY|
+ * +---------------------+
+ * |PACKED_REGISTRY_ENTRY|
+ * +---------------------+
+ * ... (one copy for each entry)
+ *
+ * +----------------------------------+
+ * |Null-terminated string for entry 0|
+ * +----------------------------------+
+ * |Null-terminated string for entry 1|
+ * +----------------------------------+
+ * ... (one copy for each entry)
+ *
+ * All memory allocated by add_registry() is released.
+ */
+static void build_registry(struct nvkm_gsp *gsp, PACKED_REGISTRY_TABLE *registry)
+{
+	struct registry_list_entry *reg, *n;
+	size_t str_offset;
+	unsigned int i = 0;
+
+	registry->numEntries = list_count_nodes(&gsp->registry_list);
+	str_offset = struct_size(registry, entries, registry->numEntries);
+
+	list_for_each_entry_safe(reg, n, &gsp->registry_list, list) {
+		registry->entries[i].type = reg->type;
+		registry->entries[i].data = reg->data;
+		registry->entries[i].length = reg->length;
+		registry->entries[i].nameOffset = str_offset;
+		memcpy((void *)registry + str_offset, reg->name, reg->name_len);
+		str_offset += reg->name_len;
+		i++;
+
+		list_del(&reg->list);
+		kfree(reg);
+	}
+
+	/* Double-check that we calculated the sizes correctly */
+	WARN_ON(gsp->registry_rpc_size != str_offset);
+
+	registry->size = gsp->registry_rpc_size;
+}
+
+/**
+ * clean_registry -- clean up registry memory in case of error
+ * @gsp: gsp pointer
+ *
+ * Call this function to clean up all memory allocated by add_registry()
+ * in case of error and build_registry() is not called.
+ */
+static void clean_registry(struct nvkm_gsp *gsp)
+{
+	struct registry_list_entry *reg, *n;
+
+	list_for_each_entry_safe(reg, n, &gsp->registry_list, list) {
+		list_del(&reg->list);
+		kfree(reg);
+	}
+
+	gsp->registry_rpc_size = sizeof(PACKED_REGISTRY_TABLE);
+}
+
+MODULE_PARM_DESC(NVreg_RegistryDwords,
+		 "A semicolon-separated list of key=integer pairs of GSP-RM registry keys");
+static char *NVreg_RegistryDwords;
+module_param(NVreg_RegistryDwords, charp, 0400);
+
  /* dword only */
  struct nv_gsp_registry_entries {
  	const char *name;
  	u32 value;
  };
 
+/**
+ * r535_registry_entries - required registry entries for GSP-RM
+ *
+ * This array lists registry entries that are required for GSP-RM to
+ * function correctly.
+ *
+ * RMSecBusResetEnable - enables PCI secondary bus reset
+ * RMForcePcieConfigSave - forces GSP-RM to preserve PCI configuration
+ *   registers on any PCI reset.
+ */
  static const struct nv_gsp_registry_entries r535_registry_entries[] = {
  	{ "RMSecBusResetEnable", 1 },
  	{ "RMForcePcieConfigSave", 1 },
  };
  #define NV_GSP_REG_NUM_ENTRIES ARRAY_SIZE(r535_registry_entries)
 
+/**
+ * strip - strips all characters in 'reject' from 's'
+ * @s: string to strip
+ * @reject: string of characters to remove
+ *
+ * 's' is modified.
+ *
+ * Returns the length of the new string.
+ */
+static size_t strip(char *s, const char *reject)
+{
+	char *p = s, *p2 = s;
+	size_t length = 0;
+	char c;
+
+	do {
+		while ((c = *p2) && strchr(reject, c))
+			p2++;
+
+		*p++ = c = *p2++;
+		length++;
+	} while (c);
+
+	return length;
+}
+
+/**
+ * r535_gsp_rpc_set_registry - build registry RPC and call GSP-RM
+ * @gsp: gsp pointer
+ *
+ * The GSP-RM registry is a set of key/value pairs that configure some aspects
+ * of GSP-RM. The keys are strings, and the values are 32-bit integers.
+ *
+ * The registry is built from a combination of a static hard-coded list (see
+ * above) and entries passed on the driver's command line.
+ */
  static int
  r535_gsp_rpc_set_registry(struct nvkm_gsp *gsp)
  {
  	PACKED_REGISTRY_TABLE *rpc;
-	char *strings;
-	int str_offset;
-	int i;
-	size_t rpc_size = struct_size(rpc, entries, NV_GSP_REG_NUM_ENTRIES);
+	unsigned int i;
+	int ret;
 
-	/* add strings + null terminator */
-	for (i = 0; i < NV_GSP_REG_NUM_ENTRIES; i++)
-		rpc_size += strlen(r535_registry_entries[i].name) + 1;
+	INIT_LIST_HEAD(&gsp->registry_list);
+	gsp->registry_rpc_size = sizeof(PACKED_REGISTRY_TABLE);
 
-	rpc = nvkm_gsp_rpc_get(gsp, NV_VGPU_MSG_FUNCTION_SET_REGISTRY, rpc_size);
-	if (IS_ERR(rpc))
-		return PTR_ERR(rpc);
+	/* Add the required registry entries first */
+	for (i = 0; i < NV_GSP_REG_NUM_ENTRIES; i++) {
+		ret = add_registry_num(gsp, r535_registry_entries[i].name,
+				 r535_registry_entries[i].value);
+		if (ret) {
+			clean_registry(gsp);
+			return ret;
+		}
+	}
+
+	/*
+	 * The NVreg_RegistryDwords parameter is a string of key=value
+	 * pairs separated by semicolons. We need to extract and trim each
+	 * substring, and then parse the substring to extract the key and
+	 * value.
+	 */
+	if (NVreg_RegistryDwords) {
+		char *p = kstrdup(NVreg_RegistryDwords, GFP_KERNEL);
+		char *start, *next = p, *equal;
+		size_t length;
+
+		/* Remove any whitespace from the parameter string */
+		length = strip(p, " \t\n");
+
+		while ((start = strsep(&next, ";"))) {
+			long value;
+
+			equal = strchr(start, '=');
+			if (!equal || (equal == start) || !isdigit(equal[1])) {
+				nvkm_error(&gsp->subdev,
+					"ignoring invalid registry string '%s'\n", start);

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1337

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +8 -1
+				continue;
+			}
 
-	rpc->numEntries = NV_GSP_REG_NUM_ENTRIES;
+			ret = kstrtol(equal + 1, 0, &value);
+			if (ret) {
+				nvkm_error(&gsp->subdev,
+					"ignoring invalid registry value in '%s'\n", start);
+				continue;
+			}

At a first glance, the string parsing looks correct to me. Did you test with invalid strings as well?

Quoted 5 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +0 -4
-	str_offset = offsetof(typeof(*rpc), entries[NV_GSP_REG_NUM_ENTRIES]);
-	strings = (char *)&rpc->entries[NV_GSP_REG_NUM_ENTRIES];
-	for (i = 0; i < NV_GSP_REG_NUM_ENTRIES; i++) {
-		int name_len = strlen(r535_registry_entries[i].name) + 1;

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1340

Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -8
-
-		rpc->entries[i].nameOffset = str_offset;
-		rpc->entries[i].type = 1;
-		rpc->entries[i].data = r535_registry_entries[i].value;
-		rpc->entries[i].length = 4;
-		memcpy(strings, r535_registry_entries[i].name, name_len);
-		strings += name_len;
-		str_offset += name_len;
+			/* Truncate the key=value string to just key */
+			*equal = 0;

What's the purpose for that?

Quoted 24 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +19 -1
+
+			ret = add_registry_num(gsp, start, value);
+			if (ret) {
+				nvkm_error(&gsp->subdev,
+					"ignoring invalid registry key/value '%s=%lu'\n",
+					start, value);
+				continue;
+			}
+		}
+
+		kfree(p);
  	}
-	rpc->size = str_offset;
+
+	rpc = nvkm_gsp_rpc_get(gsp, NV_VGPU_MSG_FUNCTION_SET_REGISTRY, gsp->registry_rpc_size);
+	if (IS_ERR(rpc)) {
+		clean_registry(gsp);
+		return PTR_ERR(rpc);
+	}
+
+	build_registry(gsp, rpc);
 
  	return nvkm_gsp_rpc_wr(gsp, rpc, false);
  }
author
#3 lore raw

Re: [PATCH 1/2] [v3] nouveau: add command-line GSP-RM registry support

Quoted 4 lines · Danilo Krummrich

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1088

Quoted 2 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -0
+struct registry_list_entry {
+	struct list_head list;

Nit: 'head' or 'entry' might be more suitable.

Will fix in v4.

Quoted 7 lines

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1090

Quoted 3 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -0
+	size_t name_len;
+	u32 type;

I prefer to represent type as enum or use a define for '1 = integer' instead. This also gets you rid of the need to explicitly explain it's meaning in the documentation of add_registry().

Will fix in v4.

Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +12 -0
+	u32 data;
+	u32 length;
+	char name[];
+};
+
+/**
+ * add_registry -- adds a registry entry
+ * @gsp: gsp pointer
+ * @name: name of the registry key
+ * @type: type of data (1 = integer)
+ * @data: value
+ * @length: size of data, in bytes
Quoted 8 lines
Quoted 4 lines

+ *
+ * Adds a registry key/value pair to the registry database.
+ *
+ * Currently, only 32-bit integers (type == 1, length == 4) are supported.

What if we pass something else? This function doesn't seem to ensure nothing else is accepted. Although I recognize that it's only used by add_registry_num() enforcing this limitation by it's function signature.

GSP-RM technically supports two other types: binary data and strings. For example, apparently it's possible to override the VBIOS itself with a registry key. However, I'm not familiar with any actual non-numeric registry keys that the user would set.

I can add support for binary data and strings.

Quoted 12 lines
Quoted 12 lines

+ *
+ * This function collects the registry information in a linked list. After
+ * all registry keys have been added, build_registry() is used to create the
+ * RPC data structure.
+ *
+ * registry_rpc_size is a running total of the size of all registry keys.
+ * It's used to avoid an O(n) calculation of the size when the RPC is built.
+ *
+ * Returns 0 on success, or negative error code on error.
+ */
+static int add_registry(struct nvkm_gsp *gsp, const char *name, u32 type, u32 data, u32 length)

Quoted 8 lines

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1123

Quoted 6 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +6 -0
+{
+	struct registry_list_entry *reg;
+	size_t nlen = strlen(name) + 1;
+
+	/* Set an arbitrary limit to avoid problems with broken command lines */
+	if (strlen(name) > 64)

Could re-use nlen for this check.

Will fix in v4.

Quoted 4 lines

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1124

Quoted 2 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +1 -0
+		return -EFBIG;

This error code doesn't seem to apply here, prefer EINVAL.

You don't like creative usage of error codes?

Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +9 -0
+		while ((start = strsep(&next, ";"))) {
+			long value;
+
+			equal = strchr(start, '=');
+			if (!equal || (equal == start) || !isdigit(equal[1])) {
+				nvkm_error(&gsp->subdev,
+					"ignoring invalid registry string '%s'\n", start);
+				continue;
+			}
Quoted 10 lines

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1337

Quoted 7 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +6 -1
-	rpc->numEntries = NV_GSP_REG_NUM_ENTRIES;
+			ret = kstrtol(equal + 1, 0, &value);
+			if (ret) {
+				nvkm_error(&gsp->subdev,
+					"ignoring invalid registry value in '%s'\n", start);
+				continue;
+			}

At a first glance, the string parsing looks correct to me. Did you test with invalid strings as well?

Yes. It would be nice if there were a regex parser in the kernel, but I think I did a good job testing and rejecting strings.

Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +0 -8
-	str_offset = offsetof(typeof(*rpc), entries[NV_GSP_REG_NUM_ENTRIES]);
-	strings = (char *)&rpc->entries[NV_GSP_REG_NUM_ENTRIES];
-	for (i = 0; i < NV_GSP_REG_NUM_ENTRIES; i++) {
-		int name_len = strlen(r535_registry_entries[i].name) + 1;
-
-		rpc->entries[i].nameOffset = str_offset;
-		rpc->entries[i].type = 1;
-		rpc->entries[i].data = r535_registry_entries[i].value;
Quoted 8 lines

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c r535_gsp_rpc_unloading_guest_driver(struct nvkm_gsp *gsp, bool suspend) · line 1340

Quoted 6 lines
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +2 -4
-		rpc->entries[i].length = 4;
-		memcpy(strings, r535_registry_entries[i].name, name_len);
-		strings += name_len;
-		str_offset += name_len;
+			/* Truncate the key=value string to just key */
+			*equal = 0;

What's the purpose for that?

Take for example, this parameter passed to Nouveau:

	NVreg_RegistryDwords="RM1457588=1;TEST=53"

The strsep() call points 'next' to "RM1457588=1", replacing the ; with \0.

'equal' then points to the '='. kstrtol() converts the "1" to 1. So all that's left is extracting the "RM1457588" into its own string. That's what "*equal = 0" does.

#4 lore raw

Re: [PATCH 1/2] [v3] nouveau: add command-line GSP-RM registry support

In reply to Timur Tabi
kernel test robot noticed the following build warnings:

Hi Timur,

kernel test robot noticed the following build warnings:

[auto build test WARNING on drm-intel/for-linux-next-fixes]
[also build test WARNING on drm-tip/drm-tip linus/master v6.8-rc4 next-20240213]
[cannot apply to drm-misc/drm-misc-next drm/drm-next drm-exynos/exynos-drm-next drm-intel/for-linux-next] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Timur-Tabi/nouveau-add-command-line-GSP-RM-registry-support/20240213-051852
base:   git://anongit.freedesktop.org/drm-intel for-linux-next-fixes
patch link:    https://lore.kernel.org/r/20240212211548.1094496-2-ttabi%40nvidia.com
patch subject: [PATCH 1/2] [v3] nouveau: add command-line GSP-RM registry support
config: parisc-defconfig (https://download.01.org/0day-ci/archive/20240214/[email protected]/config)
compiler: hppa-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240214/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c: In function 'r535_gsp_rpc_set_registry':

>> drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1317:24: warning: variable 'length' set but not used [-Wunused-but-set-variable]
    1317 |                 size_t length;
         |                        ^~~~~~
--
>> drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1246: warning: cannot understand function prototype: 'const struct nv_gsp_registry_entries r535_registry_entries[] = '

drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1673: warning: Function parameter or struct member 'priv' not described in 'r535_gsp_msg_run_cpu_sequencer' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1673: warning: Function parameter or struct member 'fn' not described in 'r535_gsp_msg_run_cpu_sequencer' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1673: warning: Function parameter or struct member 'repv' not described in 'r535_gsp_msg_run_cpu_sequencer' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:1673: warning: Function parameter or struct member 'repc' not described in 'r535_gsp_msg_run_cpu_sequencer' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:2046: warning: Function parameter or struct member 'gsp' not described in 'r535_gsp_libos_init' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:2186: warning: Function parameter or struct member 'gsp' not described in 'nvkm_gsp_radix3_sg' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:2186: warning: Function parameter or struct member 'sgt' not described in 'nvkm_gsp_radix3_sg' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:2186: warning: Function parameter or struct member 'size' not described in 'nvkm_gsp_radix3_sg' drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c:2186: warning: Function parameter or struct member 'rx3' not described in 'nvkm_gsp_radix3_sg'

vim +/length +1317 drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c

1235
1236 /**
1237 * r535_registry_entries - required registry entries for GSP-RM
1238 *
1239 * This array lists registry entries that are required for GSP-RM to
1240 * function correctly.
1241 *
1242 * RMSecBusResetEnable - enables PCI secondary bus reset
1243 * RMForcePcieConfigSave - forces GSP-RM to preserve PCI configuration
1244 * registers on any PCI reset.
1245 */

> 1246	static const struct nv_gsp_registry_entries r535_registry_entries[] = {
  1247		{ "RMSecBusResetEnable", 1 },
  1248		{ "RMForcePcieConfigSave", 1 },
  1249	};
  1250	#define NV_GSP_REG_NUM_ENTRIES ARRAY_SIZE(r535_registry_entries)
  1251	
  1252	/**
  1253	 * strip - strips all characters in 'reject' from 's'
  1254	 * @s: string to strip
  1255	 * @reject: string of characters to remove
  1256	 *
  1257	 * 's' is modified.
  1258	 *
  1259	 * Returns the length of the new string.
  1260	 */
  1261	static size_t strip(char *s, const char *reject)
  1262	{
  1263		char *p = s, *p2 = s;
  1264		size_t length = 0;
  1265		char c;
  1266	
  1267		do {
  1268			while ((c = *p2) && strchr(reject, c))
  1269				p2++;
  1270	
  1271			*p++ = c = *p2++;
  1272			length++;
  1273		} while (c);
  1274	
  1275		return length;
  1276	}
  1277	
  1278	/**
  1279	 * r535_gsp_rpc_set_registry - build registry RPC and call GSP-RM
  1280	 * @gsp: gsp pointer
  1281	 *
  1282	 * The GSP-RM registry is a set of key/value pairs that configure some aspects
  1283	 * of GSP-RM. The keys are strings, and the values are 32-bit integers.
  1284	 *
  1285	 * The registry is built from a combination of a static hard-coded list (see
  1286	 * above) and entries passed on the driver's command line.
  1287	 */
  1288	static int
  1289	r535_gsp_rpc_set_registry(struct nvkm_gsp *gsp)
  1290	{
  1291		PACKED_REGISTRY_TABLE *rpc;
  1292		unsigned int i;
  1293		int ret;
  1294	
  1295		INIT_LIST_HEAD(&gsp->registry_list);
  1296		gsp->registry_rpc_size = sizeof(PACKED_REGISTRY_TABLE);
  1297	
  1298		/* Add the required registry entries first */
  1299		for (i = 0; i < NV_GSP_REG_NUM_ENTRIES; i++) {
  1300			ret = add_registry_num(gsp, r535_registry_entries[i].name,
  1301					 r535_registry_entries[i].value);
  1302			if (ret) {
  1303				clean_registry(gsp);
  1304				return ret;
  1305			}
  1306		}
  1307	
  1308		/*
  1309		 * The NVreg_RegistryDwords parameter is a string of key=value
  1310		 * pairs separated by semicolons. We need to extract and trim each
  1311		 * substring, and then parse the substring to extract the key and
  1312		 * value.
  1313		 */
  1314		if (NVreg_RegistryDwords) {
  1315			char *p = kstrdup(NVreg_RegistryDwords, GFP_KERNEL);
  1316			char *start, *next = p, *equal;
> 1317			size_t length;

1318
1319 /* Remove any whitespace from the parameter string */
1320 length = strip(p, " \t\n");
1321
1322 while ((start = strsep(&next, ";"))) {
1323 long value;
1324
1325 equal = strchr(start, '=');
1326 if (!equal || (equal == start) || !isdigit(equal[1])) {
1327 nvkm_error(&gsp->subdev,
1328 "ignoring invalid registry string '%s'\n", start);
1329 continue;
1330 }
1331
1332 ret = kstrtol(equal + 1, 0, &value);
1333 if (ret) {
1334 nvkm_error(&gsp->subdev,
1335 "ignoring invalid registry value in '%s'\n", start);
1336 continue;
1337 }
1338
1339 /* Truncate the key=value string to just key */
1340 *equal = 0;
1341
1342 ret = add_registry_num(gsp, start, value);
1343 if (ret) {
1344 nvkm_error(&gsp->subdev,
1345 "ignoring invalid registry key/value '%s=%lu'\n",
1346 start, value);
1347 continue;
1348 }
1349 }
1350
1351 kfree(p);
1352 }
1353
1354 rpc = nvkm_gsp_rpc_get(gsp, NV_VGPU_MSG_FUNCTION_SET_REGISTRY, gsp->registry_rpc_size);
1355 if (IS_ERR(rpc)) {
1356 clean_registry(gsp);
1357 return PTR_ERR(rpc);
1358 }
1359
1360 build_registry(gsp, rpc);
1361
1362 return nvkm_gsp_rpc_wr(gsp, rpc, false);
1363 }
1364

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#5 lore raw

PATCH v3 2/2 drm/nouveau: expose GSP-RM logging buffers via debugfs

In reply to Timur Tabi

The LOGINIT, LOGINTR, LOGRM, and LOGPMU buffers are circular buffers that have printf-like logs from GSP-RM and PMU encoded in them.

LOGINIT, LOGINTR, and LOGRM are allocated by Nouveau and their DMA addresses are passed to GSP-RM during initialization. The buffers are required for GSP-RM to initialize properly.

LOGPMU is also allocated by Nouveau, but its contents are updated when Nouveau receives an NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT RPC from GSP-RM. Nouveau then copies the RPC to the buffer.

The messages are encoded as an array of variable-length structures that contain the parameters to an NV_PRINTF call. The format string and parameter count are stored in a special ELF image that contains only logging strings. This image is not currently shipped with the Nvidia driver.

There are two methods to extract the logs.

OpenRM tries to load the logging ELF, and if present, parses the log buffers in real time and outputs the strings to the kernel console.

Alternatively, and this is the method used by this patch, the buffers can be exposed to user space, and a user-space tool (along with the logging ELF image) can parse the buffer and dump the logs.

This method has the advantage that it allows the buffers to be parsed even when the logging ELF file is not available to the user. However, it has the disadvantage the debubfs entries need to remain until the driver is unloaded.

The buffers are exposed via debugfs. The debugfs entries must be created before GSP-RM is started, to ensure that they are available during GSP-RM initialization.

If GSP-RM fails to initialize, then Nouveau immediately shuts down the GSP interface. This would normally also deallocate the logging buffers, thereby preventing the user from capturing the debug logs. To avoid this, the keep-gsp-logging command line parameter can be specified. This parmater is marked as *unsafe* (thereby taining the kernel) because the DMA buffer and debugfs entries are never deallocated, even if the driver unloads. This gives the user the time to capture the logs, but it also means that resources can only be recovered by a reboot.

An end-user can capture the logs using the following commands:

cp /sys/kernel/debug/dri/<path>/loginit loginit cp /sys/kernel/debug/dri/<path>/logrm logrm cp /sys/kernel/debug/dri/<path>/logintr logintr cp /sys/kernel/debug/dri/<path>/logpmu logpmu

where <path> is the PCI ID of the GPU (e.g. 0000:65:00.0). If keep-gsp-logging is specified, then the <path> is the same but with -debug appended (e.g. 0000:65:00.0-debug).

Since LOGPMU is not needed for normal GSP-RM operation, it is only created if debugfs is available. Otherwise, the NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT RPCs are ignored.

---
v3: reworked r535_gsp_libos_debugfs_init, rebased for drm-next

 .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h |  12 +
 .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c    | 215 +++++++++++++++++-
 2 files changed, 223 insertions(+), 4 deletions(-)
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +12 -0
index 3fbc57b16a05..2ee44bdf8be7 100644
@@ -5,6 +5,8 @@
 #include <core/falcon.h>
 #include <core/firmware.h>
 
+#include <linux/debugfs.h>
+
 #define GSP_PAGE_SHIFT 12
 #define GSP_PAGE_SIZE  BIT(GSP_PAGE_SHIFT)
 
@@ -217,6 +219,16 @@ struct nvkm_gsp {
 
 	/* The size of the registry RPC */
 	size_t registry_rpc_size;
+
+	/*
+	 * Logging buffers in debugfs.  The wrapper objects need to remain
+	 * in memory until the dentry is deleted.
+	 */
+	struct debugfs_blob_wrapper blob_init;
+	struct debugfs_blob_wrapper blob_intr;
+	struct debugfs_blob_wrapper blob_rm;
+	struct debugfs_blob_wrapper blob_pmu;
+	struct dentry *debugfs_logging_dir;
 };
 
 static inline bool
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +211 -4

245 lines not on this page · raw message

Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH 2/2] [v3] drm/nouveau: expose GSP-RM logging buffers via debugfs

Quoted 85 lines · Timur Tabi

The LOGINIT, LOGINTR, LOGRM, and LOGPMU buffers are circular buffers that have printf-like logs from GSP-RM and PMU encoded in them.

LOGINIT, LOGINTR, and LOGRM are allocated by Nouveau and their DMA addresses are passed to GSP-RM during initialization. The buffers are required for GSP-RM to initialize properly.

LOGPMU is also allocated by Nouveau, but its contents are updated when Nouveau receives an NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT RPC from GSP-RM. Nouveau then copies the RPC to the buffer.

The messages are encoded as an array of variable-length structures that contain the parameters to an NV_PRINTF call. The format string and parameter count are stored in a special ELF image that contains only logging strings. This image is not currently shipped with the Nvidia driver.

There are two methods to extract the logs.

OpenRM tries to load the logging ELF, and if present, parses the log buffers in real time and outputs the strings to the kernel console.

Alternatively, and this is the method used by this patch, the buffers can be exposed to user space, and a user-space tool (along with the logging ELF image) can parse the buffer and dump the logs.

This method has the advantage that it allows the buffers to be parsed even when the logging ELF file is not available to the user. However, it has the disadvantage the debubfs entries need to remain until the driver is unloaded.

The buffers are exposed via debugfs. The debugfs entries must be created before GSP-RM is started, to ensure that they are available during GSP-RM initialization.

If GSP-RM fails to initialize, then Nouveau immediately shuts down the GSP interface. This would normally also deallocate the logging buffers, thereby preventing the user from capturing the debug logs. To avoid this, the keep-gsp-logging command line parameter can be specified. This parmater is marked as *unsafe* (thereby taining the kernel) because the DMA buffer and debugfs entries are never deallocated, even if the driver unloads. This gives the user the time to capture the logs, but it also means that resources can only be recovered by a reboot.

An end-user can capture the logs using the following commands:

cp /sys/kernel/debug/dri/<path>/loginit loginit cp /sys/kernel/debug/dri/<path>/logrm logrm cp /sys/kernel/debug/dri/<path>/logintr logintr cp /sys/kernel/debug/dri/<path>/logpmu logpmu

where <path> is the PCI ID of the GPU (e.g. 0000:65:00.0). If keep-gsp-logging is specified, then the <path> is the same but with -debug appended (e.g. 0000:65:00.0-debug).

Since LOGPMU is not needed for normal GSP-RM operation, it is only created if debugfs is available. Otherwise, the NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT RPCs are ignored.

---
v3: reworked r535_gsp_libos_debugfs_init, rebased for drm-next

  .../gpu/drm/nouveau/include/nvkm/subdev/gsp.h |  12 +
  .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c    | 215 +++++++++++++++++-
  2 files changed, 223 insertions(+), 4 deletions(-)
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +2 -0
index 3fbc57b16a05..2ee44bdf8be7 100644
@@ -5,6 +5,8 @@
  #include <core/falcon.h>
  #include <core/firmware.h>
 
+#include <linux/debugfs.h>
+
  #define GSP_PAGE_SHIFT 12
  #define GSP_PAGE_SIZE  BIT(GSP_PAGE_SHIFT)
 
@@ -217,6 +219,16 @@ struct nvkm_gsp {
 
  	/* The size of the registry RPC */
  	size_t registry_rpc_size;

drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h struct nvkm_gsp { · line 231

Quoted 10 lines
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +10 -0
+
+	/*
+	 * Logging buffers in debugfs.  The wrapper objects need to remain
+	 * in memory until the dentry is deleted.
+	 */
+	struct debugfs_blob_wrapper blob_init;
+	struct debugfs_blob_wrapper blob_intr;
+	struct debugfs_blob_wrapper blob_rm;
+	struct debugfs_blob_wrapper blob_pmu;
+	struct dentry *debugfs_logging_dir;

I think we should not create those from within the nvkm layer, but rather pass them down through nvkm_device_pci_new().

Lifecycle wise I think we should ensure that removing the Nouveau kernel module also cleans up those buffers. Even though keep-gsp-logging is considered unsafe, we shouldn't leak memory.

For instance, can we allocate corresponding buffers in the driver layer, copy things over and keep those buffers until nouveau_drm_exit()? This would also avoid exposing those DMA buffers via debugfs.

Quoted 257 lines
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +0 -0
  };
 
  static inline bool
drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +211 -4
index 86b62c7e1229..56209bf81360 100644
@@ -26,6 +26,7 @@
  #include <subdev/vfn.h>
  #include <engine/fifo/chan.h>
  #include <engine/sec2.h>
+#include <drm/drm_device.h>
 
  #include <nvfw/fw.h>
 
@@ -1979,6 +1980,196 @@ r535_gsp_rmargs_init(struct nvkm_gsp *gsp, bool resume)
  	return 0;
  }
 
+#define NV_GSP_MSG_EVENT_UCODE_LIBOS_CLASS_PMU		0xf3d722
+
+/**
+ * r535_gsp_msg_libos_print - capture log message from the PMU
+ * @priv: gsp pointer
+ * @fn: function number (ignored)
+ * @repv: pointer to libos print RPC
+ * @repc: message size
+ *
+ * See _kgspRpcUcodeLibosPrint
+ */
+static int r535_gsp_msg_libos_print(void *priv, u32 fn, void *repv, u32 repc)
+{
+	struct nvkm_gsp *gsp = priv;
+	struct nvkm_subdev *subdev = &gsp->subdev;
+	struct {
+		u32 ucodeEngDesc;
+		u32 libosPrintBufSize;
+		u8 libosPrintBuf[];
+	} *rpc = repv;
+	unsigned int class = rpc->ucodeEngDesc >> 8;
+
+	nvkm_debug(subdev, "received libos print from class 0x%x for %u bytes\n",
+		   class, rpc->libosPrintBufSize);
+
+	if (class != NV_GSP_MSG_EVENT_UCODE_LIBOS_CLASS_PMU) {
+		nvkm_warn(subdev,
+			  "received libos print from unknown class 0x%x\n",
+			  class);
+		return -ENOMSG;
+	}
+	if (rpc->libosPrintBufSize > GSP_PAGE_SIZE) {
+		nvkm_error(subdev, "libos print is too large (%u bytes)\n",
+			   rpc->libosPrintBufSize);
+		return -E2BIG;
+
+	}
+	memcpy(gsp->blob_pmu.data, rpc->libosPrintBuf, rpc->libosPrintBufSize);
+
+	return 0;
+}
+
+/*
+ * If GSP-RM load fails, then the GSP nvkm object will be deleted, the
+ * logging debugfs entries will be deleted, and it will not be possible to
+ * debug the load failure.  The keep_gsp_logging parameter tells Nouveau
+ * not to free these resources, even if the driver is unloading.  In this
+ * case, the only recovery is a reboot.
+ */
+static bool keep_gsp_logging;
+module_param_unsafe(keep_gsp_logging, bool, 0444);
+MODULE_PARM_DESC(keep_gsp_logging,
+		 "Do not remove the GSP-RM logging debugfs entries upon exit");
+
+/**
+ * r535_gsp_libos_debugfs_init - create logging debugfs entries
+ * @gsp: gsp pointer
+ *
+ * Create the debugfs entries.  This exposes the log buffers to
+ * userspace so that an external tool can parse it.
+ *
+ * The 'logpmu' contains exception dumps from the PMU. It is written via an
+ * RPC sent from GSP-RM and must be only 4KB.  We create it here because it's
+ * only useful if there is a debugfs entry to expose it.  If we get the PMU
+ * logging RPC and there is no debugfs entry, the RPC is just ignored.
+ *
+ * The blob_init, blob_rm, and blob_pmu objects can't be transient
+ * because debugfs_create_blob doesn't copy them.
+ *
+ * NOTE: OpenRM loads the logging elf image and prints the log messages
+ * in real-time. We may add that capability in the future, but that
+ * requires loading an ELF images that are not distributed with the driver,
+ * and adding the parsing code to Nouveau.
+ *
+ * Ideally, this should be part of nouveau_debugfs_init(), but that function
+ * is called too late.  We really want to create these debugfs entries before
+ * r535_gsp_booter_load() is called, so that if GSP-RM fails to initialize,
+ * there could still be a log to capture.
+ *
+ * If the unsafe command line pararameter 'keep-gsp-logging' is specified,
+ * then the logging buffer and debugfs entries will be retained when the
+ * driver shuts down.  This is necessary to debug initialization failures,
+ * because otherwise the buffers will disappear before the logs can be
+ * captured.
+ */
+static void r535_gsp_libos_debugfs_init(struct nvkm_gsp *gsp)
+{
+	struct dentry *dir_init, *dir_intr, *dir_rm, *dir_pmu;
+	struct dentry *root, *dir;
+	struct device *dev = gsp->subdev.device->dev;
+
+	/*
+	 * Under normal circumstances, we add our debugfs entries to the dentry
+	 * created by the DRM layer when the driver registered.  However, this
+	 * dentry and everything in it is deleted if GSP fails to initialize.
+	 *
+	 * If keep-gsp-logging is specified, then a different top-entry dentry
+	 * is created and that is used.  This dentry is never deleted, even if
+	 * the driver exits.
+	 */
+	if (keep_gsp_logging) {
+		char temp[64];
+
+		/* Find the 'dri' root debugfs entry. Every GPU has a dentry under it */
+		root = debugfs_lookup("dri", NULL);
+		if (IS_ERR(root)) {
+			/* No debugfs, or no root dentry for DRM */
+			return;
+		}
+
+		scnprintf(temp, sizeof(temp), "%s-debug", dev_name(dev));
+		dir = debugfs_create_dir(temp, root);
+		dput(root);
+		if (IS_ERR(dir)) {
+			nvkm_error(&gsp->subdev,
+				"failed to create %s debugfs entry\n", temp);
+			return;
+		}
+
+		gsp->debugfs_logging_dir = dir;
+	} else {
+		/* Each GPU has a subdir based on its device name, so find it */
+		struct drm_device *drm_dev = dev_get_drvdata(dev);
+
+		if (!drm_dev || !drm_dev->debugfs_root) {
+			nvkm_error(&gsp->subdev, "could not find debugfs path\n");
+			return;
+		}
+
+		dir = drm_dev->debugfs_root;
+	}
+
+	gsp->blob_init.data = gsp->loginit.data;
+	gsp->blob_init.size = gsp->loginit.size;
+	gsp->blob_intr.data = gsp->logintr.data;
+	gsp->blob_intr.size = gsp->logintr.size;
+	gsp->blob_rm.data = gsp->logrm.data;
+	gsp->blob_rm.size = gsp->logrm.size;
+
+	/*
+	 * Since the PMU buffer is copied from an RPC, it doesn't need to be
+	 * a DMA buffer.
+	 */
+	gsp->blob_pmu.size = GSP_PAGE_SIZE;
+	gsp->blob_pmu.data = kzalloc(gsp->blob_pmu.size, GFP_KERNEL);
+	if (!gsp->blob_pmu.data)
+		goto error;
+
+	dir_init = debugfs_create_blob("loginit", 0444, dir, &gsp->blob_init);
+	if (IS_ERR(dir_init)) {
+		nvkm_error(&gsp->subdev, "failed to create loginit debugfs entry\n");
+		goto error;
+	}
+
+	dir_intr = debugfs_create_blob("logintr", 0444, dir, &gsp->blob_intr);
+	if (IS_ERR(dir_intr)) {
+		nvkm_error(&gsp->subdev, "failed to create logintr debugfs entry\n");
+		goto error;
+	}
+
+	dir_rm = debugfs_create_blob("logrm", 0444, dir, &gsp->blob_rm);
+	if (IS_ERR(dir_rm)) {
+		nvkm_error(&gsp->subdev, "failed to create logrm debugfs entry\n");
+		goto error;
+	}
+
+	dir_pmu = debugfs_create_blob("logpmu", 0444, dir, &gsp->blob_pmu);
+	if (IS_ERR(dir_pmu)) {
+		nvkm_error(&gsp->subdev, "failed to create logpmu debugfs entry\n");
+		goto error;
+	}
+
+	i_size_write(d_inode(dir_init), gsp->blob_init.size);
+	i_size_write(d_inode(dir_intr), gsp->blob_intr.size);
+	i_size_write(d_inode(dir_rm), gsp->blob_rm.size);
+	i_size_write(d_inode(dir_pmu), gsp->blob_pmu.size);
+
+	r535_gsp_msg_ntfy_add(gsp, 0x0000100C, r535_gsp_msg_libos_print, gsp);
+
+	nvkm_debug(&gsp->subdev, "created debugfs GSP-RM logging entries\n");
+	return;
+
+error:
+	debugfs_remove(gsp->debugfs_logging_dir);
+	gsp->debugfs_logging_dir = NULL;
+
+	kfree(gsp->blob_pmu.data);
+	gsp->blob_pmu.data = NULL;
+}
+
  static inline u64
  r535_gsp_libos_id8(const char *name)
  {
@@ -2029,7 +2220,11 @@ static void create_pte_array(u64 *ptes, dma_addr_t addr, size_t size)
   * written to directly by GSP-RM and can be any multiple of GSP_PAGE_SIZE.
   *
   * The physical address map for the log buffer is stored in the buffer
- * itself, starting with offset 1. Offset 0 contains the "put" pointer.
+ * itself, starting with offset 1. Offset 0 contains the "put" pointer (pp).
+ * Initially, pp is equal to 0.  If the buffer has valid logging data in it,
+ * then pp points to index into the buffer where the next logging entry will
+ * be written.  Therefore, the logging data is valid if:
+ *   1 <= pp < sizeof(buffer)/sizeof(u64)
   *
   * The GSP only understands 4K pages (GSP_PAGE_SIZE), so even if the kernel is
   * configured for a larger page size (e.g. 64K pages), we need to give
@@ -2100,6 +2295,9 @@ r535_gsp_libos_init(struct nvkm_gsp *gsp)
  	args[3].size = gsp->rmargs.size;
  	args[3].kind = LIBOS_MEMORY_REGION_CONTIGUOUS;
  	args[3].loc  = LIBOS_MEMORY_REGION_LOC_SYSMEM;
+
+	r535_gsp_libos_debugfs_init(gsp);
+
  	return 0;
  }
 
@@ -2404,9 +2602,18 @@ r535_gsp_dtor(struct nvkm_gsp *gsp)
  	r535_gsp_dtor_fws(gsp);
 
  	nvkm_gsp_mem_dtor(gsp, &gsp->shm.mem);
-	nvkm_gsp_mem_dtor(gsp, &gsp->loginit);
-	nvkm_gsp_mem_dtor(gsp, &gsp->logintr);
-	nvkm_gsp_mem_dtor(gsp, &gsp->logrm);
+
+	if (keep_gsp_logging && gsp->debugfs_logging_dir)
+		nvkm_warn(&gsp->subdev,
+			"GSP-RM logging buffers retained, reboot required to recover\n");
+	else {
+		kfree(gsp->blob_pmu.data);
+		gsp->blob_pmu.data = NULL;
+
+		nvkm_gsp_mem_dtor(gsp, &gsp->loginit);
+		nvkm_gsp_mem_dtor(gsp, &gsp->logintr);
+		nvkm_gsp_mem_dtor(gsp, &gsp->logrm);
+	}
  }
 
  int
author
#7 lore raw

Re: [PATCH 2/2] [v3] drm/nouveau: expose GSP-RM logging buffers via debugfs

On Tue, 2024-02-13 at 17:57 +0100, Danilo Krummrich wrote:

+ struct debugfs_blob_wrapper blob_init;

+ struct debugfs_blob_wrapper blob_intr;

+ struct debugfs_blob_wrapper blob_rm;

+ struct debugfs_blob_wrapper blob_pmu;

+ struct dentry *debugfs_logging_dir;

I think we should not create those from within the nvkm layer, but rather pass

them down through nvkm_device_pci_new().

Should they be created in nvkm_device_pci_new() also, even though we have no idea whether GSP is involved at that point?

Lifecycle wise I think we should ensure that removing the Nouveau kernel module

also cleans up those buffers. Even though keep-gsp-logging is considered unsafe,

we shouldn't leak memory.

I agree, but then there needs to be some way to keep these debugfs entries until the driver unloads. I don't know how to do that without creating some ugly global variables.

For instance, can we allocate corresponding buffers in the driver layer, copy

things over and keep those buffers until nouveau_drm_exit()? This would also

avoid exposing those DMA buffers via debugfs.

The whole point behind this patch is to expose the buffers via debugfs. How else should they be exposed?

#8 lore raw

Re: [PATCH 2/2] [v3] drm/nouveau: expose GSP-RM logging buffers via debugfs

Quoted 11 lines · Timur Tabi
Quoted 8 lines · Danilo Krummrich

drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h struct nvkm_gsp { · line 231

Quoted 5 lines
drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +5 -0
+	struct debugfs_blob_wrapper blob_init;
+	struct debugfs_blob_wrapper blob_intr;
+	struct debugfs_blob_wrapper blob_rm;
+	struct debugfs_blob_wrapper blob_pmu;
+	struct dentry *debugfs_logging_dir;

I think we should not create those from within the nvkm layer, but rather pass them down through nvkm_device_pci_new().

Should they be created in nvkm_device_pci_new() also, even though we have no idea whether GSP is involved at that point?

We can pass some structure to nvkm_device_pci_new() where GSP sets the pointers to the buffers and maybe a callback to clean them up. Once nvkm_device_pci_new() returns we'd see if something has been set or not. If so, we can go ahead and create the debugfs nodes.

Quoted 7 lines
Quoted 5 lines

Lifecycle wise I think we should ensure that removing the Nouveau kernel module also cleans up those buffers. Even though keep-gsp-logging is considered unsafe, we shouldn't leak memory.

I agree, but then there needs to be some way to keep these debugfs entries until the driver unloads.  I don't know how to do that without creating some ugly global variables.

I fear that might be the only option. However, I still prefer a global list over a memory leak.

Quoted 8 lines
Quoted 5 lines

For instance, can we allocate corresponding buffers in the driver layer, copy things over and keep those buffers until nouveau_drm_exit()? This would also avoid exposing those DMA buffers via debugfs.

The whole point behind this patch is to expose the buffers via debugfs.  How else should they be exposed?

I think I only thought about the fail case where we could just copy over the data from the DMA buffer to another buffer and expose that one instead. However, that doesn't work if GSP loads successfully.

Hence, as mentioned above, we might just want to have some structure that we can add to the global list that contains the pointers to the DMA buffers and maybe a callback function to clean them up defined by the GSP code that we can simply call from nouveau_drm_exit().