Skip to contentSkip to conversation

daxctl: add a new reconfigure-device command

PATCH Superseded Vishal Verma 2 participants 14 messages May 8, 2019

Subsystems (2)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Vishal Verma

author maintainer
#0 lore raw

v2 cover daxctl: add a new reconfigure-device command

Changes in v2:
- Add examples to the documentation page (Dave Hansen)
- Clarify documentation regarding the conversion from system-ram to devdax
- Remove any references to a persistent config from the documentation - those can be added when the feature is added.
- device.c: validate option compatibility
- daxctl-list: display numa_node for device listings
- daxctl-list: display mode for device listings
- make the options more consistent by adding a '-O' short option for --attempt-offline

Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option).

Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work.

Example invocations:

1. Reconfigure dax0.0 to system-ram mode, don’t online the memory

    # daxctl reconfigure-device --mode=system-ram --no-online dax0.0
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "numa_node":2,
        "mode":"system-ram"
      }
    ]

2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory

    # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
    {
      "chardev":"dax0.0",
      "size":"15.63 GiB (16.78 GB)",
      "numa_node":2,
      "mode":"devdax"
    }

3. Reconfigure all dax devices on region0 to system-ram mode

    # daxctl reconfigure-device --mode=system-ram --region=0 all
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "numa_node":2,
        "mode":"system-ram"
      },
      {
        "chardev":"dax0.1",
        "size":16777216000,
        "numa_node":3,
        "mode":"system-ram"
      }
    ]

These patches can also be found in the 'kmem-pending' branch on github:
https://github.com/pmem/ndctl/tree/kmem-pending

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Pavel Tatashin <[email protected]>

Vishal Verma (10):
libdaxctl: add interfaces in support of device modes
libdaxctl: cache 'subsystem' in daxctl_ctx
libdaxctl: add interfaces to enable/disable devices
libdaxctl: add interfaces to get/set the online state for a node daxctl/list: add numa_node for device listings
libdaxctl: add an interface to get the mode for a dax device
daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for daxctl-reconfigure-device

 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 118 ++++
 contrib/ndctl                                 |  34 +-
 daxctl/Makefile.am                            |   2 +
 daxctl/builtin.h                              |   1 +
 daxctl/daxctl.c                               |   1 +
 daxctl/device.c                               | 237 ++++++++
 daxctl/lib/Makefile.am                        |   3 +-
 daxctl/lib/libdaxctl-private.h                |  21 +
 daxctl/lib/libdaxctl.c                        | 554 +++++++++++++++++-
 daxctl/lib/libdaxctl.sym                      |  14 +
 daxctl/libdaxctl.h                            |  16 +
 util/json.c                                   |  22 +
 13 files changed, 1015 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
 create mode 100644 daxctl/device.c
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#1 lore raw

PATCH v2 1/10 libdaxctl: add interfaces in support of device modes

In preparation for libdaxctl and daxctl to grow operational modes for DAX devices, add the following supporting APIs:

daxctl_dev_get_ctx
daxctl_dev_is_enabled

---
 daxctl/lib/libdaxctl.c   | 30 ++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym |  6 ++++++
 daxctl/libdaxctl.h       |  2 ++
 3 files changed, 38 insertions(+)
daxctl/lib/libdaxctl.c +30 -0
index c2e3a52..70f896b 100644
@@ -559,6 +559,36 @@ static void dax_regions_init(struct daxctl_ctx *ctx)
 	}
 }
 
+static int is_enabled(const char *drvpath)
+{
+	struct stat st;
+
+	if (lstat(drvpath, &st) < 0 || !S_ISLNK(st.st_mode))
+		return 0;
+	else
+		return 1;
+}
+
+DAXCTL_EXPORT int daxctl_dev_is_enabled(struct daxctl_dev *dev)
+{
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf;
+	int len = dev->buf_len;
+
+	if (snprintf(path, len, "%s/driver", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n",
+				daxctl_dev_get_devname(dev));
+		return 0;
+	}
+
+	return is_enabled(path);
+}
+
+DAXCTL_EXPORT struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev)
+{
+	return dev->region->ctx;
+}
+
 DAXCTL_EXPORT struct daxctl_dev *daxctl_dev_get_first(struct daxctl_region *region)
 {
 	dax_devices_init(region);
daxctl/lib/libdaxctl.sym +6 -0
index 84d3a69..c4af9a7 100644
@@ -50,3 +50,9 @@ LIBDAXCTL_5 {
 global:
 	daxctl_region_get_path;
 } LIBDAXCTL_4;
+
+LIBDAXCTL_6 {
+global:
+	daxctl_dev_get_ctx;
+	daxctl_dev_is_enabled;
+} LIBDAXCTL_5;
daxctl/libdaxctl.h +2 -0
index 1d13ea2..e20ccb4 100644
@@ -67,6 +67,8 @@ const char *daxctl_dev_get_devname(struct daxctl_dev *dev);
 int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
+struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
+int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#2 lore raw

PATCH v2 2/10 libdaxctl: cache 'subsystem' in daxctl_ctx

In reply to Vishal Verma

The 'DAX subsystem' in effect is determined at region or device init time, and dictates the sysfs base paths for all device/region operations. In preparation for adding bind/unbind functionality, cache the subsystem as determined at init time in the library context.

---
 daxctl/lib/libdaxctl.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
daxctl/lib/libdaxctl.c +10 -2
index 70f896b..f8f5b8c 100644
@@ -46,6 +46,7 @@ struct daxctl_ctx {
 	void *userdata;
 	int regions_init;
 	struct list_head regions;
+	enum dax_subsystem subsys;
 };
 
 /**
@@ -96,6 +97,7 @@ DAXCTL_EXPORT int daxctl_new(struct daxctl_ctx **ctx)
 	dbg(c, "log_priority=%d\n", c->ctx.log_priority);
 	*ctx = c;
 	list_head_init(&c->regions);
+	c->subsys = DAX_UNKNOWN;
 
 	return 0;
 }
@@ -454,14 +456,18 @@ static void dax_devices_init(struct daxctl_region *region)
 	for (i = 0; i < ARRAY_SIZE(dax_subsystems); i++) {
 		char *region_path;
 
-		if (i == DAX_BUS)
+		if (i == DAX_BUS) {
 			region_path = region->region_path;
-		else if (i == DAX_CLASS) {
+			if (ctx->subsys == DAX_UNKNOWN)
+				ctx->subsys = DAX_BUS;
+		} else if (i == DAX_CLASS) {
 			if (asprintf(&region_path, "%s/dax",
 						region->region_path) < 0) {
 				dbg(ctx, "region path alloc fail\n");
 				continue;
 			}
+			if (ctx->subsys == DAX_UNKNOWN)
+				ctx->subsys = DAX_CLASS;
 		} else
 			continue;
 		sysfs_device_parse(ctx, region_path, daxdev_fmt, region,
@@ -539,6 +545,8 @@ static void __dax_regions_init(struct daxctl_ctx *ctx, enum dax_subsystem subsys
 		free(dev_path);
 		if (!region)
 			err(ctx, "add_dax_region() for %s failed\n", de->d_name);
+		if (ctx->subsys == DAX_UNKNOWN)
+			ctx->subsys = subsys;
 	}
 	closedir(dir);
 }
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#3 lore raw

PATCH v2 3/10 libdaxctl: add interfaces to enable/disable devices

In reply to Vishal Verma

Add new libdaxctl interfaces to disable a device_dax based devices, and to enable it into a given mode. The modes available are 'device_dax', and 'system-ram', where device_dax is the normal device DAX mode used via a character device, and 'system-ram' uses the kernel's 'kmem' facility to hotplug the device into the system's memory space, and can be used as normal system memory.

This adds the following new interfaces:

daxctl_dev_disable;
daxctl_dev_enable_devdax;
daxctl_dev_enable_ram;

Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/lib/Makefile.am         |   3 +-
 daxctl/lib/libdaxctl-private.h |  15 ++
 daxctl/lib/libdaxctl.c         | 241 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   3 +
 daxctl/libdaxctl.h             |   9 ++
 5 files changed, 270 insertions(+), 1 deletion(-)
daxctl/lib/Makefile.am +2 -1
index d3d4852..9f0e444 100644
@@ -16,7 +16,8 @@ libdaxctl_la_SOURCES =\
 	libdaxctl.c
 
 libdaxctl_la_LIBADD =\
-	$(UUID_LIBS)
+	$(UUID_LIBS) \
+	$(KMOD_LIBS)
 
 daxctl_modprobe_data_DATA = daxctl.conf
 
daxctl/lib/libdaxctl-private.h +15 -0
index 4a462e7..e64d0a7 100644
@@ -13,6 +13,8 @@
 #ifndef _LIBDAXCTL_PRIVATE_H_
 #define _LIBDAXCTL_PRIVATE_H_
 
+#include <libkmod.h>
+
 #define DAXCTL_EXPORT __attribute__ ((visibility("default")))
 
 enum dax_subsystem {
@@ -26,6 +28,11 @@ static const char *dax_subsystems[] = {
 	[DAX_BUS] = "/sys/bus/dax/devices",
 };
 
+static const char *dax_modules[] = {
+	[DAXCTL_DEV_MODE_DEVDAX] = "device_dax",
+	[DAXCTL_DEV_MODE_RAM] = "kmem",
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -53,6 +60,14 @@ struct daxctl_dev {
 	char *dev_path;
 	struct list_node list;
 	unsigned long long size;
+	struct kmod_module *module;
+	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
 };
+
+static inline int check_kmod(struct kmod_ctx *kmod_ctx)
+{
+	return kmod_ctx ? 0 : -ENXIO;
+}
+
 #endif /* _LIBDAXCTL_PRIVATE_H_ */
daxctl/lib/libdaxctl.c +241 -0

309 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +3 -0
index c4af9a7..19904a2 100644
@@ -55,4 +55,7 @@ LIBDAXCTL_6 {
 global:
 	daxctl_dev_get_ctx;
 	daxctl_dev_is_enabled;
+	daxctl_dev_disable;
+	daxctl_dev_enable_devdax;
+	daxctl_dev_enable_ram;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +9 -0
index e20ccb4..b80488e 100644
@@ -69,6 +69,9 @@ int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
+int daxctl_dev_disable(struct daxctl_dev *dev);
+int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
+int daxctl_dev_enable_ram(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
@@ -81,6 +84,12 @@ int daxctl_dev_is_enabled(struct daxctl_dev *dev);
              region != NULL; \
              region = daxctl_region_get_next(region))
 
+enum daxctl_dev_mode {
+	DAXCTL_DEV_MODE_UNKNOWN,
+	DAXCTL_DEV_MODE_RAM,
+	DAXCTL_DEV_MODE_DEVDAX,
+};
+
 #ifdef __cplusplus
 } /* extern "C" */
 #endif
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#4 lore raw

PATCH v2 4/10 libdaxctl: add interfaces to get/set the online state for a node

In reply to Vishal Verma

In preparation for converting DAX devices for use as system-ram via the kernel's 'kmem' facility, add libndctl helpers to manipulate the sysfs interfaces to get the target_node of a DAX device, and to online, offline, and query the state of hotplugged memory sections associated with a given node.

This adds the following new interfaces:

daxctl_dev_get_target_node
daxctl_dev_online_node
daxctl_dev_offline_node
daxctl_dev_node_is_online

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/lib/libdaxctl-private.h |   6 +
 daxctl/lib/libdaxctl.c         | 228 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   4 +
 daxctl/libdaxctl.h             |   4 +
 4 files changed, 242 insertions(+)
daxctl/lib/libdaxctl-private.h +6 -0
index e64d0a7..ef443aa 100644
@@ -33,6 +33,11 @@ static const char *dax_modules[] = {
 	[DAXCTL_DEV_MODE_RAM] = "kmem",
 };
 
+enum node_state {
+	NODE_OFFLINE,
+	NODE_ONLINE,
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -63,6 +68,7 @@ struct daxctl_dev {
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
+	int target_node;
 };
 
 static inline int check_kmod(struct kmod_ctx *kmod_ctx)
daxctl/lib/libdaxctl.c +228 -0

243 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +4 -0
index 19904a2..cc47ed6 100644
@@ -54,8 +54,12 @@ global:
 LIBDAXCTL_6 {
 global:
 	daxctl_dev_get_ctx;
+	daxctl_dev_get_target_node;
 	daxctl_dev_is_enabled;
 	daxctl_dev_disable;
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
+	daxctl_dev_online_node;
+	daxctl_dev_offline_node;
+	daxctl_dev_node_is_online;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +4 -0
index b80488e..db0d4ea 100644
@@ -68,10 +68,14 @@ int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
+int daxctl_dev_get_target_node(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 int daxctl_dev_disable(struct daxctl_dev *dev);
 int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
 int daxctl_dev_enable_ram(struct daxctl_dev *dev);
+int daxctl_dev_online_node(struct daxctl_dev *dev);
+int daxctl_dev_offline_node(struct daxctl_dev *dev);
+int daxctl_dev_node_is_online(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#5 lore raw

PATCH v2 5/10 daxctl/list: add numa_node for device listings

In reply to Vishal Verma

The kernel provides a 'target_node' attribute for dax devices. When converting a dax device to the system-ram mode, the memory is hotplugged into this numa node. It would be helpful to print this in device listings so that it is easy for applications to detect the numa node to which the new memory belongs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 util/json.c | 8 ++++++++
 1 file changed, 8 insertions(+)
util/json.c +8 -0
index babdc8c..b7ce719 100644
@@ -271,6 +271,7 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 {
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
+	int node;
 
 	jdev = json_object_new_object();
 	if (!devname || !jdev)
@@ -284,6 +285,13 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 	if (jobj)
 		json_object_object_add(jdev, "size", jobj);
 
+	node = daxctl_dev_get_target_node(dev);
+	if (node >= 0) {
+		jobj = json_object_new_int(node);
+		if (jobj)
+			json_object_object_add(jdev, "numa_node", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#6 lore raw

PATCH v2 6/10 libdaxctl: add an interface to get the mode for a dax device

In reply to Vishal Verma

In preparation for a reconfigure-device command, add an interface to retrieve the 'mode' of a dax device. This will allow the reconfigure-device command (and via daxctl_dev_to_json()), also daxctl-list) to print the mode on device listings via a list command or immediately after a mode change.

Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/lib/libdaxctl.c   | 43 ++++++++++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym |  1 +
 daxctl/libdaxctl.h       |  1 +
 util/json.c              | 14 +++++++++++++
 4 files changed, 59 insertions(+)
daxctl/lib/libdaxctl.c +43 -0
index aab2364..a4919e0 100644
@@ -12,6 +12,8 @@
  */
 #include <stdio.h>
 #include <errno.h>
+#include <limits.h>
+#include <libgen.h>
 #include <stdlib.h>
 #include <dirent.h>
 #include <unistd.h>
@@ -840,6 +842,47 @@ DAXCTL_EXPORT int daxctl_dev_disable(struct daxctl_dev *dev)
 	return 0;
 }
 
+DAXCTL_EXPORT enum daxctl_dev_mode daxctl_dev_get_mode(struct daxctl_dev *dev)
+{
+	const char *devname = daxctl_dev_get_devname(dev);
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	int rc = DAXCTL_DEV_MODE_UNKNOWN;
+	char path[200];
+	const int len = sizeof(path);
+	char *mod_path, *mod_base;
+
+	if (!daxctl_dev_is_enabled(dev))
+		return rc;
+
+	if (ctx->subsys != DAX_BUS) {
+		err(ctx, "%s: invalid operation for dax subsystem\n", devname);
+		err(ctx, "%s:  see daxctl-migrate-device-model(1)\n", devname);
+		return -ENXIO;
+	}
+
+	if (snprintf(path, len, "%s/driver/module", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n", devname);
+		return -ENXIO;
+	}
+
+	mod_path = realpath(path, NULL);
+	if (!mod_path) {
+		rc = -errno;
+		err(ctx, "%s:  unable to determine module: %s\n", devname,
+			strerror(errno));
+		return rc;
+	}
+
+	mod_base = basename(mod_path);
+	if (strcmp(mod_base, dax_modules[DAXCTL_DEV_MODE_RAM]) == 0)
+		rc = DAXCTL_DEV_MODE_RAM;
+	else if (strcmp(mod_base, dax_modules[DAXCTL_DEV_MODE_DEVDAX]) == 0)
+		rc = DAXCTL_DEV_MODE_DEVDAX;
+
+	free(mod_path);
+	return rc;
+}
+
 DAXCTL_EXPORT struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev)
 {
 	return dev->region->ctx;
daxctl/lib/libdaxctl.sym +1 -0
index cc47ed6..d53976d 100644
@@ -62,4 +62,5 @@ global:
 	daxctl_dev_online_node;
 	daxctl_dev_offline_node;
 	daxctl_dev_node_is_online;
+	daxctl_dev_get_mode;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +1 -0
index db0d4ea..4f9088f 100644
@@ -76,6 +76,7 @@ int daxctl_dev_enable_ram(struct daxctl_dev *dev);
 int daxctl_dev_online_node(struct daxctl_dev *dev);
 int daxctl_dev_offline_node(struct daxctl_dev *dev);
 int daxctl_dev_node_is_online(struct daxctl_dev *dev);
+enum daxctl_dev_mode daxctl_dev_get_mode(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
util/json.c +14 -0
index b7ce719..4f13222 100644
@@ -271,6 +271,7 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 {
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
+	enum daxctl_dev_mode mode;
 	int node;
 
 	jdev = json_object_new_object();
@@ -292,6 +293,19 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 			json_object_object_add(jdev, "numa_node", jobj);
 	}
 
+	mode = daxctl_dev_get_mode(dev);
+	if (mode > 0) {
+		jobj = NULL;
+		if (mode == DAXCTL_DEV_MODE_RAM)
+			jobj = json_object_new_string("system-ram");
+		else if (mode == DAXCTL_DEV_MODE_DEVDAX)
+			jobj = json_object_new_string("devdax");
+		else
+			jobj = json_object_new_string("unknown");
+		if (jobj)
+			json_object_object_add(jdev, "mode", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#7 lore raw

PATCH v2 7/10 daxctl: add a new reconfigure-device command

In reply to Vishal Verma

Add a new command 'daxctl-reconfigure-device'. This is used to switch the mode of a dax device between regular 'device_dax' and 'system-memory'. The command also uses the memory hotplug sysfs interfaces to online the newly available memory when converting to 'system-ram', and to attempt to offline the memory when converting back to a DAX device.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/Makefile.am |   2 +
 daxctl/builtin.h   |   1 +
 daxctl/daxctl.c    |   1 +
 daxctl/device.c    | 237 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 241 insertions(+)
 create mode 100644 daxctl/device.c
daxctl/Makefile.am +2 -0
index 94f73f9..66dcc7f 100644
@@ -15,10 +15,12 @@ daxctl_SOURCES =\
 		daxctl.c \
 		list.c \
 		migrate.c \
+		device.c \
 		../util/json.c
 
 daxctl_LDADD =\
 	lib/libdaxctl.la \
 	../libutil.a \
 	$(UUID_LIBS) \
+	$(KMOD_LIBS) \
 	$(JSON_LIBS)
daxctl/builtin.h +1 -0
index 00ef5e9..756ba2a 100644
@@ -6,4 +6,5 @@
 struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +1 -0
index 2e41747..e1ba7b8 100644
@@ -71,6 +71,7 @@ static struct cmd_struct commands[] = {
 	{ "list", .d_fn = cmd_list },
 	{ "help", .d_fn = cmd_help },
 	{ "migrate-device-model", .d_fn = cmd_migrate },
+	{ "reconfigure-device", .d_fn = cmd_reconfig_device },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c new file +237 -0

237 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#8 lore raw

PATCH v2 8/10 Documentation/daxctl: add a man page for daxctl-reconfigure-device

In reply to Vishal Verma

Add a man page describing the new daxctl-reconfigure-device command.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 118 ++++++++++++++++++
 2 files changed, 120 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
Documentation/daxctl/Makefile.am +2 -1
index 6aba035..715fbad 100644
@@ -28,7 +28,8 @@ endif
 man1_MANS = \
 	daxctl.1 \
 	daxctl-list.1 \
-	daxctl-migrate-device-model.1
+	daxctl-migrate-device-model.1 \
+	daxctl-reconfigure-device.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-reconfigure-device.txt new file +118 -0

118 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#9 lore raw

PATCH v2 9/10 contrib/ndctl: fix region-id completions for daxctl

In reply to Vishal Verma

The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Add a new helper __daxctl_get_regions() to complete daxctl region IDs properly.

While at it, fix a useless use of 'echo' in __daxctl_get_devs() and quoting in __daxctl_comp_options()

Fixes d6790a32f32c ("daxctl: Add bash-completion")
---
 contrib/ndctl | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/ndctl +10 -4
index e17fb0b..d1f8bd6 100755
@@ -528,8 +528,14 @@ _ndctl()
 
 __daxctl_get_devs()
 {
-	local opts="--devices $*"
-	echo "$(daxctl list $opts | grep -E "^\s*\"chardev\":" | cut -d\" -f4)"
+	local opts=("--devices" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"chardev\":" | cut -d'"' -f4
+}
+
+__daxctl_get_regions()
+{
+	local opts=("--regions" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"id\":" | grep -Eo "[0-9]+"
 }
 
 __daxctlcomp()
@@ -558,10 +564,10 @@ __daxctl_comp_options()
 		local cur_arg=${cur##*=}
 		case $cur_subopt in
 		--region)
-			opts=$(__ndctl_get_regions -i)
+			opts="$(__daxctl_get_regions -i)"
 			;;
 		--dev)
-			opts=$(__daxctl_get_devs -i)
+			opts="$(__daxctl_get_devs -i)"
 			;;
 		*)
 			return
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#10 lore raw

PATCH v2 10/10 contrib/ndctl: add bash-completion for daxctl-reconfigure-device

In reply to Vishal Verma

Add bash completion helpers for the new daxctl-reconfigure-device command.

---
 contrib/ndctl | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
contrib/ndctl +17 -3
index d1f8bd6..32c4731 100755
@@ -544,7 +544,7 @@ __daxctlcomp()
 
 	COMPREPLY=( $( compgen -W "$1" -- "$2" ) )
 	for cword in "${COMPREPLY[@]}"; do
-		if [[ "$cword" == @(--region|--dev) ]]; then
+		if [[ "$cword" == @(--region|--dev|--mode) ]]; then
 			COMPREPLY[$i]="${cword}="
 		else
 			COMPREPLY[$i]="${cword} "
@@ -569,6 +569,9 @@ __daxctl_comp_options()
 		--dev)
 			opts="$(__daxctl_get_devs -i)"
 			;;
+		--mode)
+			opts="system-ram devdax"
+			;;
 		*)
 			return
 			;;
@@ -579,8 +582,19 @@ __daxctl_comp_options()
 
 __daxctl_comp_non_option_args()
 {
-	# there aren't any commands that accept non option arguments yet
-	return
+	local subcmd=$1
+	local cur=$2
+	local opts
+
+	case $subcmd in
+	reconfigure-device)
+		opts="$(__daxctl_get_devs -i) all"
+		;;
+	*)
+		return
+		;;
+	esac
+	__daxctlcomp "$opts" "$cur"
 }
 
 __daxctl_main()
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#11 lore raw
In reply to Vishal Verma

Hi Vishal,

I am trying to use your changes, and getting the following error:
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed

Here is full output:

# ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
[   26.891054] dax0.0 initialised, 524288 pages in 9ms
[   26.894239] random: ndctl: uninitialized urandom read (4 bytes read)
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
  "uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1",
  "daxregion":{
    "id":0,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152
  },
  "align":2097152
}
# [   73.670556] random: crng init done

Thank you,
Pasha

Quoted 98 lines · Vishal Verma

Changes in v2:
- Add examples to the documentation page (Dave Hansen)
- Clarify documentation regarding the conversion from system-ram to devdax
- Remove any references to a persistent config from the documentation - those can be added when the feature is added.
- device.c: validate option compatibility
- daxctl-list: display numa_node for device listings
- daxctl-list: display mode for device listings
- make the options more consistent by adding a '-O' short option for --attempt-offline

Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option).

Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work.

Example invocations:

1. Reconfigure dax0.0 to system-ram mode, don’t online the memory

    # daxctl reconfigure-device --mode=system-ram --no-online dax0.0
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "numa_node":2,
        "mode":"system-ram"
      }
    ]

2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory

    # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
    {
      "chardev":"dax0.0",
      "size":"15.63 GiB (16.78 GB)",
      "numa_node":2,
      "mode":"devdax"
    }

3. Reconfigure all dax devices on region0 to system-ram mode

    # daxctl reconfigure-device --mode=system-ram --region=0 all
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "numa_node":2,
        "mode":"system-ram"
      },
      {
        "chardev":"dax0.1",
        "size":16777216000,
        "numa_node":3,
        "mode":"system-ram"
      }
    ]

These patches can also be found in the 'kmem-pending' branch on github:
https://github.com/pmem/ndctl/tree/kmem-pending

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Pavel Tatashin <[email protected]>

Vishal Verma (10):
libdaxctl: add interfaces in support of device modes
libdaxctl: cache 'subsystem' in daxctl_ctx
libdaxctl: add interfaces to enable/disable devices
libdaxctl: add interfaces to get/set the online state for a node daxctl/list: add numa_node for device listings
libdaxctl: add an interface to get the mode for a dax device
daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for daxctl-reconfigure-device

 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 118 ++++
 contrib/ndctl                                 |  34 +-
 daxctl/Makefile.am                            |   2 +
 daxctl/builtin.h                              |   1 +
 daxctl/daxctl.c                               |   1 +
 daxctl/device.c                               | 237 ++++++++
 daxctl/lib/Makefile.am                        |   3 +-
 daxctl/lib/libdaxctl-private.h                |  21 +
 daxctl/lib/libdaxctl.c                        | 554 +++++++++++++++++-
 daxctl/lib/libdaxctl.sym                      |  14 +
 daxctl/libdaxctl.h                            |  16 +
 util/json.c                                   |  22 +
 13 files changed, 1015 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
 create mode 100644 daxctl/device.c
Quoted 3 lines

--
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#12 lore raw
Quoted 15 lines · Pavel Tatashin

Hi Vishal,

I am trying to use your changes, and getting the following error:
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed

Here is full output:

# ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
[   26.891054] dax0.0 initialised, 524288 pages in 9ms
[   26.894239] random: ndctl: uninitialized urandom read (4 bytes read)
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
Quoted 8 lines
  "uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1",
  "daxregion":{
    "id":0,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152
  },
  "align":2097152
}

Thanks for the report!

I suspect you may be using the "legacy" device-model. You can confirm by seeing if that device even shows up on the "dax" bus. Compare /sys/bus/dax/devices/ vs /sys/class/dax/

If you see stuff in /sys/class/dax and nothing in /sys/bus/dax/devices then you need to configure the system to switch to the device-dax bus-model by running either of the following commands:

# daxctl migrate-device-model

...or:

# cat <<EOF > /etc/modprobe.d/daxctl.conf
blacklist dax_pmem_compat
alias nd:t7* dax_pmem
EOF

...and then reload the subsystem:

ndctl disable-region all
ndctl enable-region all

That said, if this was the cause, we should improve the failure mode in this case - I'll look into that, and test by forcing dax-class too.

Thanks,
-Vishal
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#13 lore raw
Quoted 25 lines · Vishal L
Quoted 15 lines · Pavel Tatashin

Hi Vishal,

I am trying to use your changes, and getting the following error:
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed

Here is full output:

# ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
[   26.891054] dax0.0 initialised, 524288 pages in 9ms
[   26.894239] random: ndctl: uninitialized urandom read (4 bytes read)
libdaxctl: __sysfs_device_parse: dax0.0: add_dev() failed
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
Quoted 8 lines
  "uuid":"6684b3b0-4ab1-45ba-9ce6-48aa046b5fc1",
  "daxregion":{
    "id":0,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152
  },
  "align":2097152
}
Quoted 5 lines

Thanks for the report!

I suspect you may be using the "legacy" device-model. You can confirm by seeing if that device even shows up on the "dax" bus. Compare /sys/bus/dax/devices/ vs /sys/class/dax/

Hm, I can't reproduce it forcing the dax-class model. Is this with the qemu config I sent earlier? Also, could you post the output of ndctl list -BiRND

Thanks,
-Vishal
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm