Skip to contentSkip to conversation

daxctl: add a new reconfigure-device command

PATCH Superseded Vishal Verma 2 participants 24 messages Jul 24, 2019 – Jul 26, 2019

Subsystems (3)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Vishal Verma

author maintainer
#0 lore raw

v7 cover daxctl: add a new reconfigure-device command

Changes in v7:
- Fix a couple of checkaptch type errors in the new lines added in v6 (Dan).
- Get rid of daxctl_dev_get_mode. daxctl_dev_get_memory is sufficient to both check the mode and allocate the memory related structures on its first call. (Dan)
- Due to the above, daxctl_dev_mode is now private to libdaxctl, and not part of the API exported through libdaxctl.h
- Add a large enough buffer at init time to construct dynamic paths, and avoid asprintf() type allocations for memory blocks at runtime (Dan).

Changes in v6:
- For memory block online/offline operations, the kernel responds with an EINVAL for both 'real' errors, and if the memory was already in the requested state. Since there is a TOCTOU hole between checking the state and storing it, just perform a second check if the store results in an error. If the check shows the state to be the same as the one we're attempting, it means that another agent (usually udev) won the race, but we don't care so long as the state change happened, so don't report an error. (Fan Du)

Changes in v5:
- device.c: correctly set loglevel for daxctl_ctx for --verbose
- drop the subsys caching, its complexity started to exceed its benefit. dax-class device models will simply error out during reconfigure. (Dan)
- Add a note to the man page for the above.
- Clarify the onlining policy (online_movable) in the man page
- rename "numa_node" to "target_node" in device listings (Dan)
- When printing a device 'mode', assume devdax if !system-ram, avoiding a "mode: unknown" situation which can be confusing. (Dan)
- Add a "state: disabled" attribute to the device listing if a driver is not bound. This is more apt than the previous "mode: unknown" listing.
- add an api to get 'dev->resource' parsing /proc/iomem as a fallback for when the kernel doesn't provide the attribute (Dan)
- convert node_* apis to 'memory_* apis that act on a new daxctl_memory object (Dan)
- online only memory sections belonging to the device in question by cross referencing block indices with the dax device resource (Dan)
- Refuse to reconfigure a device that is already in the target mode. Until now, reconfiguring a system-ram device back to system-ram would result in a 'online memory may not be hot-removed' kernel warning.
- If the device was already in the system-ram mode, skip disabling/enabling, but still try to online the memory unless the --no-online option is in effect.
- In daxctl_unbind, also 'remove_id' to prevent devices automatically binding to the kmem driver on a disable + re-enable, which can be surprising (Dan).
- Rewrite the top half of daxctl/device.c to borrow elements from ndctl/namespace.c so that it can support growing additional commands that operate on devices (online-memory and offline-memory)
- Refactor the bottom half of daxctl/device.c so we only do the disabling/offlining steps if the device was enabled.
- Add new commands to online and offline memory sections (Dan) associated with a given dax device (Dan)
- Add a new test - daxctl-device.sh - to test daxctl reconfigure-device, online-memory, and offline-memory commands.
- Add an example in documentation demonstrating how to use numactl to bind a process to a node surfaced from a dax device (Andy Rudoff)

Changes in v4:
- Don't fail add_dax_dev for kmod failures. Instead fail only when the kmod list is actually used, i.e. during daxctl-reconfigure-device

Changes in v3:
- In daxctl_dev_get_mode(), remove the subsystem warning, detect dax-class and simply make it return devdax

Changes in v2:
- Add examples to the documentation page (Dave Hansen)
- Clarify documentation regarding the conversion from system-ram to devdax
- Remove any references to a persistent config from the documentation - those can be added when the feature is added.
- device.c: validate option compatibility
- daxctl-list: display numa_node for device listings
- daxctl-list: display mode for device listings
- make the options more consistent by adding a '-O' short option for --attempt-offline

Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option).

Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work.

Example invocations:

1. Reconfigure dax0.0 to system-ram mode, don’t online the memory

    # daxctl reconfigure-device --mode=system-ram --no-online dax0.0
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "target_node":2,
        "mode":"system-ram"
      }
    ]

2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory

    # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
    {
      "chardev":"dax0.0",
      "size":"15.63 GiB (16.78 GB)",
      "target_node":2,
      "mode":"devdax"
    }

3. Reconfigure all dax devices on region0 to system-ram mode

    # daxctl reconfigure-device --mode=system-ram --region=0 all
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "target_node":2,
        "mode":"system-ram"
      },
      {
        "chardev":"dax0.1",
        "size":16777216000,
        "target_node":3,
        "mode":"system-ram"
      }
    ]

These patches can also be found in the 'kmem-pending' branch on github:
https://github.com/pmem/ndctl/tree/kmem-pending

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Pavel Tatashin <[email protected]>

Vishal Verma (13):
libdaxctl: add interfaces to get ctx and check device state
libdaxctl: add interfaces to enable/disable devices
libdaxctl: add an interface to retrieve the device resource
libdaxctl: add a 'daxctl_memory' object for memory based operations daxctl/list: add target_node for device listings daxctl/list: display the mode for a dax device
daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device
daxctl: add commands to online and offline memory
Documentation: Add man pages for daxctl-{on,off}line-memory contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for the new daxctl commands
test: Add a unit test for daxctl-reconfigure-device and friends

 Documentation/daxctl/Makefile.am              |   5 +-
 .../daxctl/daxctl-offline-memory.txt          |  72 ++
 Documentation/daxctl/daxctl-online-memory.txt |  80 ++
 .../daxctl/daxctl-reconfigure-device.txt      | 139 ++++
 Makefile.am                                   |   3 +-
 contrib/ndctl                                 |  38 +-
 daxctl/Makefile.am                            |   2 +
 daxctl/builtin.h                              |   3 +
 daxctl/daxctl.c                               |   3 +
 daxctl/device.c                               | 476 ++++++++++++
 daxctl/lib/Makefile.am                        |   5 +-
 daxctl/lib/libdaxctl-private.h                |  38 +
 daxctl/lib/libdaxctl.c                        | 685 ++++++++++++++++++
 daxctl/lib/libdaxctl.sym                      |  18 +
 daxctl/libdaxctl.h                            |  16 +
 test/Makefile.am                              |   3 +-
 test/common                                   |  19 +-
 test/daxctl-devices.sh                        |  81 +++
 util/iomem.c                                  |  37 +
 util/iomem.h                                  |  12 +
 util/json.c                                   |  22 +
 21 files changed, 1743 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/daxctl/daxctl-offline-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-online-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
 create mode 100644 daxctl/device.c
 create mode 100755 test/daxctl-devices.sh
 create mode 100644 util/iomem.c
 create mode 100644 util/iomem.h
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#1 lore raw

PATCH v7 1/13 libdaxctl: add interfaces to get ctx and check device state

In preparation for libdaxctl and daxctl to grow operational modes for DAX devices, add the following supporting APIs:

daxctl_dev_get_ctx
daxctl_dev_is_enabled

---
 daxctl/lib/libdaxctl.c   | 30 ++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym |  6 ++++++
 daxctl/libdaxctl.h       |  2 ++
 3 files changed, 38 insertions(+)
daxctl/lib/libdaxctl.c +30 -0
index c2e3a52..70f896b 100644
@@ -559,6 +559,36 @@ static void dax_regions_init(struct daxctl_ctx *ctx)
 	}
 }
 
+static int is_enabled(const char *drvpath)
+{
+	struct stat st;
+
+	if (lstat(drvpath, &st) < 0 || !S_ISLNK(st.st_mode))
+		return 0;
+	else
+		return 1;
+}
+
+DAXCTL_EXPORT int daxctl_dev_is_enabled(struct daxctl_dev *dev)
+{
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf;
+	int len = dev->buf_len;
+
+	if (snprintf(path, len, "%s/driver", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n",
+				daxctl_dev_get_devname(dev));
+		return 0;
+	}
+
+	return is_enabled(path);
+}
+
+DAXCTL_EXPORT struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev)
+{
+	return dev->region->ctx;
+}
+
 DAXCTL_EXPORT struct daxctl_dev *daxctl_dev_get_first(struct daxctl_region *region)
 {
 	dax_devices_init(region);
daxctl/lib/libdaxctl.sym +6 -0
index 84d3a69..c4af9a7 100644
@@ -50,3 +50,9 @@ LIBDAXCTL_5 {
 global:
 	daxctl_region_get_path;
 } LIBDAXCTL_4;
+
+LIBDAXCTL_6 {
+global:
+	daxctl_dev_get_ctx;
+	daxctl_dev_is_enabled;
+} LIBDAXCTL_5;
daxctl/libdaxctl.h +2 -0
index 1d13ea2..e20ccb4 100644
@@ -67,6 +67,8 @@ const char *daxctl_dev_get_devname(struct daxctl_dev *dev);
 int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
+struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
+int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#2 lore raw

PATCH v7 2/13 libdaxctl: add interfaces to enable/disable devices

In reply to Vishal Verma

Add new libdaxctl interfaces to disable a device_dax based device, and to enable it into the given mode. The modes available are 'devdax', and 'system-ram', where devdax is the normal device DAX mode used via a character device, and 'system-ram' uses the kernel's 'kmem' facility to hotplug the device making it usable as normal memory.

This adds the following new interfaces:

daxctl_dev_disable;
daxctl_dev_enable_devdax;
daxctl_dev_enable_ram;

---
 daxctl/lib/Makefile.am         |   3 +-
 daxctl/lib/libdaxctl-private.h |  21 +++
 daxctl/lib/libdaxctl.c         | 234 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   3 +
 daxctl/libdaxctl.h             |   3 +
 5 files changed, 263 insertions(+), 1 deletion(-)
daxctl/lib/Makefile.am +2 -1
index d3d4852..9f0e444 100644
@@ -16,7 +16,8 @@ libdaxctl_la_SOURCES =\
 	libdaxctl.c
 
 libdaxctl_la_LIBADD =\
-	$(UUID_LIBS)
+	$(UUID_LIBS) \
+	$(KMOD_LIBS)
 
 daxctl_modprobe_data_DATA = daxctl.conf
 
daxctl/lib/libdaxctl-private.h +21 -0
index 4a462e7..120137f 100644
@@ -13,6 +13,8 @@
 #ifndef _LIBDAXCTL_PRIVATE_H_
 #define _LIBDAXCTL_PRIVATE_H_
 
+#include <libkmod.h>
+
 #define DAXCTL_EXPORT __attribute__ ((visibility("default")))
 
 enum dax_subsystem {
@@ -26,6 +28,17 @@ static const char *dax_subsystems[] = {
 	[DAX_BUS] = "/sys/bus/dax/devices",
 };
 
+enum daxctl_dev_mode {
+	DAXCTL_DEV_MODE_DEVDAX = 0,
+	DAXCTL_DEV_MODE_RAM,
+	DAXCTL_DEV_MODE_END,
+};
+
+static const char *dax_modules[] = {
+	[DAXCTL_DEV_MODE_DEVDAX] = "device_dax",
+	[DAXCTL_DEV_MODE_RAM] = "kmem",
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -53,6 +66,14 @@ struct daxctl_dev {
 	char *dev_path;
 	struct list_node list;
 	unsigned long long size;
+	struct kmod_module *module;
+	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
 };
+
+static inline int check_kmod(struct kmod_ctx *kmod_ctx)
+{
+	return kmod_ctx ? 0 : -ENXIO;
+}
+
 #endif /* _LIBDAXCTL_PRIVATE_H_ */
daxctl/lib/libdaxctl.c +234 -0

302 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +3 -0
index c4af9a7..19904a2 100644
@@ -55,4 +55,7 @@ LIBDAXCTL_6 {
 global:
 	daxctl_dev_get_ctx;
 	daxctl_dev_is_enabled;
+	daxctl_dev_disable;
+	daxctl_dev_enable_devdax;
+	daxctl_dev_enable_ram;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +3 -0
index e20ccb4..407f459 100644
@@ -69,6 +69,9 @@ int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
+int daxctl_dev_disable(struct daxctl_dev *dev);
+int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
+int daxctl_dev_enable_ram(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#3 lore raw

PATCH v7 3/13 libdaxctl: add an interface to retrieve the device resource

In reply to Vishal Verma

Add an interface to retrieve the 'resource' attribute for a dax device.

Attempt to retrieve it as usual via sysfs, but since older kernels may be missing this attribute, as a fallback, attempt to retrieve it from /proc/iomem

Cc Dan Williams <[email protected]>
[fscanf format string problem and diagnosis]
---
 Makefile.am                    |  3 ++-
 daxctl/lib/Makefile.am         |  2 ++
 daxctl/lib/libdaxctl-private.h |  1 +
 daxctl/lib/libdaxctl.c         | 12 +++++++++++
 daxctl/lib/libdaxctl.sym       |  1 +
 daxctl/libdaxctl.h             |  1 +
 util/iomem.c                   | 37 ++++++++++++++++++++++++++++++++++
 util/iomem.h                   | 12 +++++++++++
 8 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 util/iomem.c
 create mode 100644 util/iomem.h
Makefile.am +2 -1
index df8797e..8d10a10 100644
@@ -74,6 +74,7 @@ libutil_a_SOURCES = \
 	util/wrapper.c \
 	util/filter.c \
 	util/bitmap.c \
-	util/abspath.c
+	util/abspath.c \
+	util/iomem.c
 
 nobase_include_HEADERS = daxctl/libdaxctl.h
daxctl/lib/Makefile.am +2 -0
index 9f0e444..7704b1b 100644
@@ -9,6 +9,8 @@ lib_LTLIBRARIES = libdaxctl.la
 libdaxctl_la_SOURCES =\
 	../libdaxctl.h \
 	libdaxctl-private.h \
+	../../util/iomem.c \
+	../../util/iomem.h \
 	../../util/sysfs.c \
 	../../util/sysfs.h \
 	../../util/log.c \
daxctl/lib/libdaxctl-private.h +1 -0
index 120137f..fee67d1 100644
@@ -65,6 +65,7 @@ struct daxctl_dev {
 	size_t buf_len;
 	char *dev_path;
 	struct list_node list;
+	unsigned long long resource;
 	unsigned long long size;
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
daxctl/lib/libdaxctl.c +12 -0
index 85c4fbe..9bbf256 100644
@@ -24,6 +24,7 @@
 
 #include <util/log.h>
 #include <util/sysfs.h>
+#include <util/iomem.h>
 #include <daxctl/libdaxctl.h>
 #include "libdaxctl-private.h"
 
@@ -369,6 +370,12 @@ static void *add_dax_dev(void *parent, int id, const char *daxdev_base)
 	dev->major = major(st.st_rdev);
 	dev->minor = minor(st.st_rdev);
 
+	sprintf(path, "%s/resource", daxdev_base);
+	if (sysfs_read_attr(ctx, path, buf) == 0)
+		dev->resource = strtoull(buf, NULL, 0);
+	else
+		dev->resource = iomem_get_dev_resource(ctx, daxdev_base);
+
 	sprintf(path, "%s/size", daxdev_base);
 	if (sysfs_read_attr(ctx, path, buf) < 0)
 		goto err_read;
@@ -878,6 +885,11 @@ DAXCTL_EXPORT int daxctl_dev_get_minor(struct daxctl_dev *dev)
 	return dev->minor;
 }
 
+DAXCTL_EXPORT unsigned long long daxctl_dev_get_resource(struct daxctl_dev *dev)
+{
+	return dev->resource;
+}
+
 DAXCTL_EXPORT unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev)
 {
 	return dev->size;
daxctl/lib/libdaxctl.sym +1 -0
index 19904a2..1692624 100644
@@ -58,4 +58,5 @@ global:
 	daxctl_dev_disable;
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
+	daxctl_dev_get_resource;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +1 -0
index 407f459..adf55f3 100644
@@ -66,6 +66,7 @@ int daxctl_dev_get_id(struct daxctl_dev *dev);
 const char *daxctl_dev_get_devname(struct daxctl_dev *dev);
 int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
+unsigned long long daxctl_dev_get_resource(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
util/iomem.c new file +37 -0
new file mode 100644
index 0000000..a3c23f5
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2019 Intel Corporation. All rights reserved. */
+
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <util/log.h>
+#include <util/iomem.h>
+#include <util/sysfs.h>
+
+unsigned long long __iomem_get_dev_resource(struct log_ctx *ctx,
+		const char *devpath)
+{
+	const char *devname = devpath_to_devname(devpath);
+	FILE *fp = fopen("/proc/iomem", "r");
+	unsigned long long res;
+	char name[256];
+
+	if (fp == NULL) {
+		log_err(ctx, "%s: open /proc/iomem: %s\n", devname,
+				strerror(errno));
+		return 0;
+	}
+
+	while (fscanf(fp, "%llx-%*x : %254[^\n]\n", &res, name) == 2) {
+		if (strcmp(name, devname) == 0) {
+			log_dbg(ctx, "%s: got resource via iomem: %#llx\n",
+					devname, res);
+			fclose(fp);
+			return res;
+		}
+	}
+
+	log_dbg(ctx, "%s: not found in iomem\n", devname);
+	fclose(fp);
+	return 0;
+}
util/iomem.h new file +12 -0
new file mode 100644
index 0000000..aaaf6a7
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2019 Intel Corporation. All rights reserved. */
+#ifndef _NDCTL_IOMEM_H_
+#define _NDCTL_IOMEM_H_
+
+struct log_ctx;
+unsigned long long __iomem_get_dev_resource(struct log_ctx *ctx,
+		const char *path);
+
+#define iomem_get_dev_resource(c, p) __iomem_get_dev_resource(&(c)->ctx, (p))
+
+#endif /* _NDCTL_IOMEM_H_ */
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#4 lore raw

PATCH v7 4/13 libdaxctl: add a 'daxctl_memory' object for memory based operations

In reply to Vishal Verma

Introduce a new 'daxctl_memory' object, which will be used for operations related to managing dax devices in 'system-memory' modes.

Add libdaxctl APIs to get the target_node of a DAX device, and to online, offline, and query the state of hotplugged memory sections associated with a given device.

This adds the following new interfaces:

daxctl_dev_get_target_node
daxctl_dev_get_memory;
daxctl_memory_get_dev;
daxctl_memory_get_node_path;
daxctl_memory_get_block_size;
daxctl_memory_set_online
daxctl_memory_set_offline
daxctl_memory_is_online

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
[for the memblock-already-online TOCTOU hole]
---
 daxctl/lib/libdaxctl-private.h |  16 ++
 daxctl/lib/libdaxctl.c         | 407 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   8 +
 daxctl/libdaxctl.h             |  10 +
 4 files changed, 441 insertions(+)
daxctl/lib/libdaxctl-private.h +16 -0
index fee67d1..46a5782 100644
@@ -39,6 +39,11 @@ static const char *dax_modules[] = {
 	[DAXCTL_DEV_MODE_RAM] = "kmem",
 };
 
+enum memory_state {
+	MEM_OFFLINE,
+	MEM_ONLINE,
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -70,8 +75,19 @@ struct daxctl_dev {
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
+	struct daxctl_memory *mem;
+	int target_node;
+};
+
+struct daxctl_memory {
+	struct daxctl_dev *dev;
+	void *mem_buf;
+	size_t buf_len;
+	char *node_path;
+	unsigned long block_size;
 };
 
+
 static inline int check_kmod(struct kmod_ctx *kmod_ctx)
 {
 	return kmod_ctx ? 0 : -ENXIO;
daxctl/lib/libdaxctl.c +407 -0

440 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +8 -0
index 1692624..53eb700 100644
@@ -59,4 +59,12 @@ global:
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
 	daxctl_dev_get_resource;
+	daxctl_dev_get_target_node;
+	daxctl_dev_get_memory;
+	daxctl_memory_get_dev;
+	daxctl_memory_get_node_path;
+	daxctl_memory_get_block_size;
+	daxctl_memory_set_online;
+	daxctl_memory_set_offline;
+	daxctl_memory_is_online;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +10 -0
index adf55f3..d844359 100644
@@ -73,6 +73,16 @@ int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 int daxctl_dev_disable(struct daxctl_dev *dev);
 int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
 int daxctl_dev_enable_ram(struct daxctl_dev *dev);
+int daxctl_dev_get_target_node(struct daxctl_dev *dev);
+
+struct daxctl_memory;
+struct daxctl_memory *daxctl_dev_get_memory(struct daxctl_dev *dev);
+struct daxctl_dev *daxctl_memory_get_dev(struct daxctl_memory *mem);
+const char *daxctl_memory_get_node_path(struct daxctl_memory *mem);
+unsigned long daxctl_memory_get_block_size(struct daxctl_memory *mem);
+int daxctl_memory_set_online(struct daxctl_memory *mem);
+int daxctl_memory_set_offline(struct daxctl_memory *mem);
+int daxctl_memory_is_online(struct daxctl_memory *mem);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#5 lore raw

PATCH v7 5/13 daxctl/list: add target_node for device listings

In reply to Vishal Verma

The kernel provides a 'target_node' attribute for dax devices. When converting a dax device to the system-ram mode, the memory is hotplugged into this numa node. It would be helpful to print this in device listings so that it is easy for applications to detect the node to which the new memory belongs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 util/json.c | 8 ++++++++
 1 file changed, 8 insertions(+)
util/json.c +8 -0
index babdc8c..f521337 100644
@@ -271,6 +271,7 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 {
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
+	int node;
 
 	jdev = json_object_new_object();
 	if (!devname || !jdev)
@@ -284,6 +285,13 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 	if (jobj)
 		json_object_object_add(jdev, "size", jobj);
 
+	node = daxctl_dev_get_target_node(dev);
+	if (node >= 0) {
+		jobj = json_object_new_int(node);
+		if (jobj)
+			json_object_object_add(jdev, "target_node", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#6 lore raw

PATCH v7 6/13 daxctl/list: display the mode for a dax device

In reply to Vishal Verma

In preparation for a reconfigure-device command, allow JSON listings to display the 'mode' of a dax device. This will allow the reconfigure-device command (and via daxctl_dev_to_json(), also daxctl-list) to print the mode in device listings via a 'daxctl-list' command or immediately after a mode change.

Add a 'state' attribute to the json listings for devices, since a device could end up in a state where it is not bound to any driver, and hence, 'disabled'. The state attribute is only displayed for disabled devices.

Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/lib/libdaxctl.c |  2 ++
 util/json.c            | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
daxctl/lib/libdaxctl.c +2 -0
index 4684a07..5f8d2d0 100644
@@ -12,6 +12,8 @@
  */
 #include <stdio.h>
 #include <errno.h>
+#include <limits.h>
+#include <libgen.h>
 #include <stdlib.h>
 #include <dirent.h>
 #include <unistd.h>
util/json.c +14 -0
index f521337..9f80b5b 100644
@@ -269,6 +269,7 @@ struct json_object *util_dimm_to_json(struct ndctl_dimm *dimm,
 struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 		unsigned long flags)
 {
+	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
 	int node;
@@ -292,6 +293,19 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 			json_object_object_add(jdev, "target_node", jobj);
 	}
 
+	if (mem)
+		jobj = json_object_new_string("system-ram");
+	else
+		jobj = json_object_new_string("devdax");
+	if (jobj)
+		json_object_object_add(jdev, "mode", jobj);
+
+	if (!daxctl_dev_is_enabled(dev)) {
+		jobj = json_object_new_string("disabled");
+		if (jobj)
+			json_object_object_add(jdev, "state", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#7 lore raw

PATCH v7 7/13 daxctl: add a new reconfigure-device command

In reply to Vishal Verma

Add a new command 'daxctl-reconfigure-device'. This is used to switch the mode of a dax device between regular 'device_dax' and 'system-memory'. The command also uses the memory hotplug sysfs interfaces to online the newly available memory when converting to 'system-ram', and to attempt to offline the memory when converting back to a DAX device.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/Makefile.am |   2 +
 daxctl/builtin.h   |   1 +
 daxctl/daxctl.c    |   1 +
 daxctl/device.c    | 347 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 351 insertions(+)
 create mode 100644 daxctl/device.c
daxctl/Makefile.am +2 -0
index 94f73f9..66dcc7f 100644
@@ -15,10 +15,12 @@ daxctl_SOURCES =\
 		daxctl.c \
 		list.c \
 		migrate.c \
+		device.c \
 		../util/json.c
 
 daxctl_LDADD =\
 	lib/libdaxctl.la \
 	../libutil.a \
 	$(UUID_LIBS) \
+	$(KMOD_LIBS) \
 	$(JSON_LIBS)
daxctl/builtin.h +1 -0
index 00ef5e9..756ba2a 100644
@@ -6,4 +6,5 @@
 struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +1 -0
index 2e41747..e1ba7b8 100644
@@ -71,6 +71,7 @@ static struct cmd_struct commands[] = {
 	{ "list", .d_fn = cmd_list },
 	{ "help", .d_fn = cmd_help },
 	{ "migrate-device-model", .d_fn = cmd_migrate },
+	{ "reconfigure-device", .d_fn = cmd_reconfig_device },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c new file +347 -0

347 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#8 lore raw

PATCH v7 8/13 Documentation/daxctl: add a man page for daxctl-reconfigure-device

In reply to Vishal Verma

Add a man page describing the new daxctl-reconfigure-device command.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 139 ++++++++++++++++++
 2 files changed, 141 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
Documentation/daxctl/Makefile.am +2 -1
index 6aba035..715fbad 100644
@@ -28,7 +28,8 @@ endif
 man1_MANS = \
 	daxctl.1 \
 	daxctl-list.1 \
-	daxctl-migrate-device-model.1
+	daxctl-migrate-device-model.1 \
+	daxctl-reconfigure-device.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-reconfigure-device.txt new file +139 -0

139 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#9 lore raw

Re: [ndctl PATCH v7 08/13] Documentation/daxctl: add a man page for daxctl-reconfigure-device

Quoted 88 lines · Vishal Verma

Add a man page describing the new daxctl-reconfigure-device command.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 139 ++++++++++++++++++
 2 files changed, 141 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
Documentation/daxctl/Makefile.am +2 -1
index 6aba035..715fbad 100644
@@ -28,7 +28,8 @@ endif
 man1_MANS = \
        daxctl.1 \
        daxctl-list.1 \
-       daxctl-migrate-device-model.1
+       daxctl-migrate-device-model.1 \
+       daxctl-reconfigure-device.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-reconfigure-device.txt new file +55 -0
new file mode 100644
index 0000000..fb2b36b
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: GPL-2.0
+
+daxctl-reconfigure-device(1)
+============================
+
+NAME
+----
+daxctl-reconfigure-device - Reconfigure a dax device into a different mode
+
+SYNOPSIS
+--------
+[verse]
+'daxctl reconfigure-device' <dax0.0> [<dax1.0>...<daxY.Z>] [<options>]
+
+EXAMPLES
+--------
+
+* Reconfigure dax0.0 to system-ram mode, don't online the memory
+----
+# daxctl reconfigure-device --mode=system-ram --no-online dax0.0
+[
+  {
+    "chardev":"dax0.0",
+    "size":16777216000,
+    "target_node":2,
+    "mode":"system-ram"
+  }
+]
+----
+
+* Reconfigure dax0.0 to devdax mode, attempt to offline the memory
+----
+# daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
+{
+  "chardev":"dax0.0",
+  "size":"15.63 GiB (16.78 GB)",
+  "target_node":2,
+  "mode":"devdax"
+}
+----
+
+* Reconfigure all dax devices on region0 to system-ram mode
+----
+# daxctl reconfigure-device --mode=system-ram --region=0 all
+[
+  {
+    "chardev":"dax0.0",
+    "size":16777216000,
+    "target_node":2,
+    "mode":"system-ram"
+  },
+  {
+    "chardev":"dax0.1",
+    "size":16777216000,
+    "target_node":3,

Documentation/daxctl/daxctl-reconfigure-device.txt line 65

Quoted 10 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +10 -0
+    "mode":"system-ram"
+  }
+]
+----
+
+* Run a process called 'some-service' using numactl to restrict its cpu
+nodes to '0' and '1', and  memory allocations to node 2 (determined using
+daxctl_dev_get_target_node() or 'daxctl list')
+----
+# daxctl reconfigure-device --mode=system-ram --no-online dax0.0

Any reason to use --no-online in this example? Presumably some-service may not start if node2 has no online memory.

Quoted 7 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +7 -0
+[
+  {
+    "chardev":"dax0.0",
+    "size":16777216000,
+    "target_node":2,
+    "mode":"system-ram"
+  }

Documentation/daxctl/daxctl-reconfigure-device.txt line 82

Quoted 10 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +10 -0
+]
+
+# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2
+----
+
+DESCRIPTION
+-----------
+
+Reconfigure the operational mode of a dax device. This can be used to convert
+a regular 'devdax' mode device to the 'system-ram' mode which allows for the dax

s/allows/arranges/

Documentation/daxctl/daxctl-reconfigure-device.txt line 90

Quoted 8 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +8 -0
+range to be hot-plugged into the system as regular memory.
+
+NOTE: This is a destructive operation. Any data on the dax device *will* be
+lost.
+
+NOTE: Device reconfiguration depends on the dax-bus device model. If dax-class is
+in use (via the dax_pmem_compat driver), the reconfiguration will fail. See
+linkdaxctl:daxctl-migrate-device-model[1] for more information.

Let's make sure that do_reconfig() bails with a common error message for the compat case and quote that message here. You can check that by comparing the device path 'subsystem' to /sys/class/dax. I.e.

# ls -l /dev/dax0.0
crw------- 1 root root 253, 4 Jul 25 12:22 /dev/dax0.0
# readlink -f /sys/dev/char/253\:4/subsystem
/sys/class/dax

...it will be /sys/bus/dax otherwise.

Quoted 14 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +14 -0
+
+OPTIONS
+-------
+-r::
+--region=::
+       Restrict the operation to devices belonging to the specified region(s).
+       A device-dax region is a contiguous range of memory that hosts one or
+       more /dev/daxX.Y devices, where X is the region id and Y is the device
+       instance id.
+
+-m::
+--mode=::
+       Specify the mode to which the dax device(s) should be reconfigured.
+       - "system-ram": hotplug the device into system memory.

Documentation/daxctl/daxctl-reconfigure-device.txt line 114

Quoted 10 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +10 -0
+
+       - "devdax": switch to the normal "device dax" mode. This requires the
+         kernel to support hot-unplugging 'kmem' based memory. If this is not
+         available, a reboot is the only way to switch back to 'devdax' mode.
+
+-N::
+--no-online::
+       By default, memory sections provided by system-ram devices will be
+       brought online automatically and immediately with the 'online_movable'
+       policy. Use this option to disable the automatic onlining behavior.

Probably need to mention that the system might online the memory even if this is specified, or for extra credit, coordinate with that auto-online facility to arrange for it to be skipped.

Documentation/daxctl/daxctl-reconfigure-device.txt line 122

Quoted 8 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +8 -0
+
+-O::
+--attempt-offline::
+       When converting from "system-ram" mode to "devdax", it is expected
+       that all the memory sections are first made offline. By default,
+       daxctl won't touch online memory. However with this option, attempt
+       to offline the memory on the NUMA node associated with the dax device
+       before converting it back to "devdax" mode.

As mentioned in patch 7, this sounds like --force to me.
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#10 lore raw

Re: [ndctl PATCH v7 08/13] Documentation/daxctl: add a man page for daxctl-reconfigure-device

Quoted 13 lines · Dan Williams
Quoted 12 lines · Vishal Verma

Add a man page describing the new daxctl-reconfigure-device command.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 139 ++++++++++++++++++
 2 files changed, 141 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt

[..]

Quoted 9 lines

Documentation/daxctl/daxctl-reconfigure-device.txt line 65

Quoted 5 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +5 -0
+* Run a process called 'some-service' using numactl to restrict its cpu
+nodes to '0' and '1', and  memory allocations to node 2 (determined using
+daxctl_dev_get_target_node() or 'daxctl list')
+----
+# daxctl reconfigure-device --mode=system-ram --no-online dax0.0

Any reason to use --no-online in this example? Presumably some-service may not start if node2 has no online memory.

Yep just a copy/paste typo, removing it.

Quoted 12 lines
Quoted 12 lines

+[

Documentation/daxctl/daxctl-reconfigure-device.txt new file +10 -0
+  {
+    "chardev":"dax0.0",
+    "size":16777216000,
+    "target_node":2,
+    "mode":"system-ram"
+  }
+]
+
+# numactl --cpunodebind=0-1 --membind=2 -- some-service --opt1 --opt2
+----
Quoted 8 lines

Documentation/daxctl/daxctl-reconfigure-device.txt line 82

Quoted 6 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +6 -0
+
+DESCRIPTION
+-----------
+
+Reconfigure the operational mode of a dax device. This can be used to convert
+a regular 'devdax' mode device to the 'system-ram' mode which allows for the dax

s/allows/arranges/

ok.

Quoted 14 lines

Documentation/daxctl/daxctl-reconfigure-device.txt line 90

Quoted 9 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +8 -0
+range to be hot-plugged into the system as regular memory.
+
+NOTE: This is a destructive operation. Any data on the dax device *will* be
+lost.
+
+NOTE: Device reconfiguration depends on the dax-bus device model. If dax-class is
+in use (via the dax_pmem_compat driver), the reconfiguration will fail. See
+linkdaxctl:daxctl-migrate-device-model[1] for more information.

Let's make sure that do_reconfig() bails with a common error message for the compat case and quote that message here. You can check that by comparing the device path 'subsystem' to /sys/class/dax. I.e.

Quoted 7 lines
# ls -l /dev/dax0.0
crw------- 1 root root 253, 4 Jul 25 12:22 /dev/dax0.0
# readlink -f /sys/dev/char/253\:4/subsystem
/sys/class/dax

...it will be /sys/bus/dax otherwise.

Hm,. currently the failure is just:
libdaxctl: daxctl_dev_enable: dax3.0: failed to enable

Should we check the subsystem as above programatically, and print a better error in daxctl-reconfigure-device?

Quoted 21 lines
Quoted 21 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +20 -0
+
+OPTIONS
+-------
+-r::
+--region=::
+       Restrict the operation to devices belonging to the specified region(s).
+       A device-dax region is a contiguous range of memory that hosts one or
+       more /dev/daxX.Y devices, where X is the region id and Y is the device
+       instance id.
+
+-m::
+--mode=::
+       Specify the mode to which the dax device(s) should be reconfigured.
+       - "system-ram": hotplug the device into system memory.
+
+       - "devdax": switch to the normal "device dax" mode. This requires the
+         kernel to support hot-unplugging 'kmem' based memory. If this is not
+         available, a reboot is the only way to switch back to 'devdax' mode.
+
+-N::
Quoted 8 lines

Documentation/daxctl/daxctl-reconfigure-device.txt line 114

Quoted 4 lines

+--no-online::

Documentation/daxctl/daxctl-reconfigure-device.txt new file +3 -0
+       By default, memory sections provided by system-ram devices will be
+       brought online automatically and immediately with the 'online_movable'
+       policy. Use this option to disable the automatic onlining behavior.

Probably need to mention that the system might online the memory even if this is specified, or for extra credit, coordinate with that auto-online facility to arrange for it to be skipped.

Good point, added a note regarding this.

Quoted 11 lines

Documentation/daxctl/daxctl-reconfigure-device.txt line 122

Quoted 9 lines
Documentation/daxctl/daxctl-reconfigure-device.txt new file +8 -0
+
+-O::
+--attempt-offline::
+       When converting from "system-ram" mode to "devdax", it is expected
+       that all the memory sections are first made offline. By default,
+       daxctl won't touch online memory. However with this option, attempt
+       to offline the memory on the NUMA node associated with the dax device
+       before converting it back to "devdax" mode.

As mentioned in patch 7, this sounds like --force to me.

Done.

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#11 lore raw

PATCH v7 9/13 daxctl: add commands to online and offline memory

In reply to Vishal Verma

Add two new commands:

daxctl-online-memory
daxctl-offline-memory

to manage the state of hot-plugged memory from the system-ram mode for dax devices. This provides a way for the user to online/offline the memory as a separate step from the reconfiguration. Without this, a user that reconfigures a device into the system-ram mode with the --no-online option, would have no way to later online the memory, and would have to resort to shell scripting to online them manually via sysfs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 daxctl/builtin.h |   2 +
 daxctl/daxctl.c  |   2 +
 daxctl/device.c  | 131 ++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 134 insertions(+), 1 deletion(-)
daxctl/builtin.h +2 -0
index 756ba2a..f5a0147 100644
@@ -7,4 +7,6 @@ struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_online_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_offline_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +2 -0
index e1ba7b8..1ab0732 100644
@@ -72,6 +72,8 @@ static struct cmd_struct commands[] = {
 	{ "help", .d_fn = cmd_help },
 	{ "migrate-device-model", .d_fn = cmd_migrate },
 	{ "reconfigure-device", .d_fn = cmd_reconfig_device },
+	{ "online-memory", .d_fn = cmd_online_memory },
+	{ "offline-memory", .d_fn = cmd_offline_memory },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c +130 -1

177 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#12 lore raw

Re: [ndctl PATCH v7 09/13] daxctl: add commands to online and offline memory

Quoted 96 lines · Vishal Verma

Add two new commands:

daxctl-online-memory
daxctl-offline-memory

to manage the state of hot-plugged memory from the system-ram mode for dax devices. This provides a way for the user to online/offline the memory as a separate step from the reconfiguration. Without this, a user that reconfigures a device into the system-ram mode with the --no-online option, would have no way to later online the memory, and would have to resort to shell scripting to online them manually via sysfs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 daxctl/builtin.h |   2 +
 daxctl/daxctl.c  |   2 +
 daxctl/device.c  | 131 ++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 134 insertions(+), 1 deletion(-)
daxctl/builtin.h +2 -0
index 756ba2a..f5a0147 100644
@@ -7,4 +7,6 @@ struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_online_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_offline_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +2 -0
index e1ba7b8..1ab0732 100644
@@ -72,6 +72,8 @@ static struct cmd_struct commands[] = {
        { "help", .d_fn = cmd_help },
        { "migrate-device-model", .d_fn = cmd_migrate },
        { "reconfigure-device", .d_fn = cmd_reconfig_device },
+       { "online-memory", .d_fn = cmd_online_memory },
+       { "offline-memory", .d_fn = cmd_offline_memory },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c +17 -0
index a71ebbe..64eff04 100644
@@ -36,6 +36,8 @@ static unsigned long flags;
 
 enum device_action {
        ACTION_RECONFIG,
+       ACTION_ONLINE,
+       ACTION_OFFLINE,
 };
 
 #define BASE_OPTIONS() \
@@ -56,6 +58,11 @@ static const struct option reconfig_options[] = {
        OPT_END(),
 };
 
+static const struct option memory_options[] = {
+       BASE_OPTIONS(),
+       OPT_END(),
+};
+
 static const char *parse_device_options(int argc, const char **argv,
                enum device_action action, const struct option *options,
                const char *usage, struct daxctl_ctx *ctx)
@@ -76,6 +83,12 @@ static const char *parse_device_options(int argc, const char **argv,
                case ACTION_RECONFIG:
                        action_string = "reconfigure";
                        break;
+               case ACTION_ONLINE:
+                       action_string = "online memory for";
+                       break;
+               case ACTION_OFFLINE:
+                       action_string = "offline memory for";
+                       break;
                default:
                        action_string = "<>";
                        break;
@@ -124,6 +137,10 @@ static const char *parse_device_options(int argc, const char **argv,
                        }
                }
                break;
+       case ACTION_ONLINE:
+       case ACTION_OFFLINE:
+               /* nothing special */
+               break;
        }
        if (rc) {
                usage_with_options(u, options);

daxctl/device.c static int do_reconfig(struct daxctl_dev *dev, enum dev_mode mode, · line 312

Quoted 10 lines
daxctl/device.c +7 -0
@@ -286,10 +303,75 @@ static int do_reconfig(struct daxctl_dev *dev, enum dev_mode mode,
        return rc;
 }
 
+static int do_xline(struct daxctl_dev *dev, enum device_action action)
+{
+       struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
+       const char *devname = daxctl_dev_get_devname(dev);
+       int rc, num_online;
+
+       if (!daxctl_dev_is_enabled(dev)) {

Can this fail if daxctl_dev_get_memory() succeeded?

Other than that potential code reduction looks good to me.
_______________________________________________
Linux-nvdimm mailing list [email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#13 lore raw

Re: [ndctl PATCH v7 09/13] daxctl: add commands to online and offline memory

Quoted 125 lines · Dan Williams

On Wed, Jul 24, 2019 at 2:58 PM Vishal Verma <[email protected]

Quoted 111 lines

wrote:
Add two new commands:

daxctl-online-memory
daxctl-offline-memory

to manage the state of hot-plugged memory from the system-ram mode for
dax devices. This provides a way for the user to online/offline the memory as a separate step from the reconfiguration. Without this, a user
that reconfigures a device into the system-ram mode with the --no- online
option, would have no way to later online the memory, and would have to
resort to shell scripting to online them manually via sysfs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 daxctl/builtin.h |   2 +
 daxctl/daxctl.c  |   2 +
 daxctl/device.c  | 131
++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 134 insertions(+), 1 deletion(-)
daxctl/builtin.h +0 -0
index 756ba2a..f5a0147 100644
@@ -7,4 +7,6 @@ struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx
 *ctx);
 int cmd_reconfig_device(int argc, const char **argv, struct

daxctl_ctx *ctx);
+int cmd_online_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_offline_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
#endif /* _DAXCTL_BUILTIN_H_ */

daxctl/daxctl.c +2 -0
index e1ba7b8..1ab0732 100644
@@ -72,6 +72,8 @@ static struct cmd_struct commands[] = {
        { "help", .d_fn = cmd_help },
        { "migrate-device-model", .d_fn = cmd_migrate },
        { "reconfigure-device", .d_fn = cmd_reconfig_device },
+       { "online-memory", .d_fn = cmd_online_memory },
+       { "offline-memory", .d_fn = cmd_offline_memory },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c +7 -0
index a71ebbe..64eff04 100644
@@ -36,6 +36,8 @@ static unsigned long flags;
 
 enum device_action {
        ACTION_RECONFIG,
+       ACTION_ONLINE,
+       ACTION_OFFLINE,
 };
 
 #define BASE_OPTIONS() \
@@ -56,6 +58,11 @@ static const struct option reconfig_options[] = {
        OPT_END(),
 };
 
+static const struct option memory_options[] = {
+       BASE_OPTIONS(),
+       OPT_END(),
+};
+
 static const char *parse_device_options(int argc, const char
 **argv,
                enum device_action action, const struct option

*options,
const char *usage, struct daxctl_ctx *ctx)

daxctl/device.c +6 -0
@@ -76,6 +83,12 @@ static const char *parse_device_options(int argc,
 const char **argv,
                case ACTION_RECONFIG:
                        action_string = "reconfigure";
                        break;
+               case ACTION_ONLINE:
+                       action_string = "online memory for";
+                       break;
+               case ACTION_OFFLINE:
+                       action_string = "offline memory for";
+                       break;
                default:
                        action_string = "<>";
                        break;
daxctl/device.c +4 -0
@@ -124,6 +137,10 @@ static const char *parse_device_options(int
 argc, const char **argv,
                        }
                }
                break;
+       case ACTION_ONLINE:
+       case ACTION_OFFLINE:
+               /* nothing special */
+               break;
        }
        if (rc) {
                usage_with_options(u, options);
daxctl/device.c +0 -0
@@ -286,10 +303,75 @@ static int do_reconfig(struct daxctl_dev *dev,
 enum dev_mode mode,
        return rc;

daxctl/device.c static int do_reconfig(struct daxctl_dev *dev, · line 314

Quoted 10 lines
daxctl/device.c +7 -0
 }
 
+static int do_xline(struct daxctl_dev *dev, enum device_action
 action)
+{
+       struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
+       const char *devname = daxctl_dev_get_devname(dev);
+       int rc, num_online;
+
+       if (!daxctl_dev_is_enabled(dev)) {

Can this fail if daxctl_dev_get_memory() succeeded?

Quoted 1 line

Other than that potential code reduction looks good to me.

Nope, not needed. Removed, and got even more code reduction/consolidation as a result of adding the section count interface.

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#14 lore raw

PATCH v7 10/13 Documentation: Add man pages for daxctl-{on, off}line-memory

In reply to Vishal Verma

Add man pages for the two new commands: daxctl-online-memory, and daxctl-offline-memory.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 Documentation/daxctl/Makefile.am              |  4 +-
 .../daxctl/daxctl-offline-memory.txt          | 72 +++++++++++++++++
 Documentation/daxctl/daxctl-online-memory.txt | 80 +++++++++++++++++++
 3 files changed, 155 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-offline-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-online-memory.txt
Documentation/daxctl/Makefile.am +3 -1
index 715fbad..37c3bde 100644
@@ -29,7 +29,9 @@ man1_MANS = \
 	daxctl.1 \
 	daxctl-list.1 \
 	daxctl-migrate-device-model.1 \
-	daxctl-reconfigure-device.1
+	daxctl-reconfigure-device.1 \
+	daxctl-online-memory.1 \
+	daxctl-offline-memory.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-offline-memory.txt new file +72 -0

72 lines not on this page · raw message

Documentation/daxctl/daxctl-online-memory.txt new file +80 -0

80 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Reviewed-by on v7 10/13
#15 lore raw

Re: [ndctl PATCH v7 10/13] Documentation: Add man pages for daxctl-{on, off}line-memory

Quoted 7 lines · Vishal Verma

Add man pages for the two new commands: daxctl-online-memory, and daxctl-offline-memory.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>

Looks good:

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#16 lore raw

PATCH v7 11/13 contrib/ndctl: fix region-id completions for daxctl

In reply to Vishal Verma

The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Add a new helper __daxctl_get_regions() to complete daxctl region IDs properly.

While at it, fix a useless use of 'echo' in __daxctl_get_devs() and quoting in __daxctl_comp_options()

Fixes d6790a32f32c ("daxctl: Add bash-completion")
---
 contrib/ndctl | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/ndctl +10 -4
index 396a344..cacee2d 100755
@@ -531,8 +531,14 @@ _ndctl()
 
 __daxctl_get_devs()
 {
-	local opts="--devices $*"
-	echo "$(daxctl list $opts | grep -E "^\s*\"chardev\":" | cut -d\" -f4)"
+	local opts=("--devices" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"chardev\":" | cut -d'"' -f4
+}
+
+__daxctl_get_regions()
+{
+	local opts=("--regions" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"id\":" | grep -Eo "[0-9]+"
 }
 
 __daxctlcomp()
@@ -561,10 +567,10 @@ __daxctl_comp_options()
 		local cur_arg=${cur##*=}
 		case $cur_subopt in
 		--region)
-			opts=$(__ndctl_get_regions -i)
+			opts="$(__daxctl_get_regions -i)"
 			;;
 		--dev)
-			opts=$(__daxctl_get_devs -i)
+			opts="$(__daxctl_get_devs -i)"
 			;;
 		*)
 			return
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Reviewed-by on v7 11/13
#17 lore raw

Re: [ndctl PATCH v7 11/13] contrib/ndctl: fix region-id completions for daxctl

Quoted 5 lines · Vishal Verma

The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Oh, that's an unfortunate difference, but too late to change now I think, good find.

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#18 lore raw

Re: [ndctl PATCH v7 11/13] contrib/ndctl: fix region-id completions for daxctl

Quoted 11 lines · Dan Williams

On Wed, Jul 24, 2019 at 2:58 PM Vishal Verma <[email protected]

Quoted 5 lines

wrote:
The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Oh, that's an unfortunate difference, but too late to change now I think, good find.

Yep - though I think I like --region=0 better than --region=region0, less redundancy. But agreed, probably too late to change.

Thanks for reviewing these!
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#19 lore raw

Re: [ndctl PATCH v7 11/13] contrib/ndctl: fix region-id completions for daxctl

Quoted 10 lines · Vishal L
Quoted 7 lines · Dan Williams

On Wed, Jul 24, 2019 at 2:58 PM Vishal Verma <[email protected]

Quoted 5 lines

wrote:
The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Quoted 7 lines
Quoted 4 lines

Oh, that's an unfortunate difference, but too late to change now I think, good find.

Yep - though I think I like --region=0 better than --region=region0, less redundancy. But agreed, probably too late to change.

We could teach it to allow both. Something for the backlog, but at least the kernel is using a common number allocator for pmem regions and hmem regions.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#20 lore raw

PATCH v7 12/13 contrib/ndctl: add bash-completion for the new daxctl commands

In reply to Vishal Verma

Add bash completion helpers for the new daxctl-reconfigure-device, daxctl-online-memory, and daxctl-offline-memory commands.

---
 contrib/ndctl | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
contrib/ndctl +21 -3
index cacee2d..680fe6a 100755
@@ -547,7 +547,7 @@ __daxctlcomp()
 
 	COMPREPLY=( $( compgen -W "$1" -- "$2" ) )
 	for cword in "${COMPREPLY[@]}"; do
-		if [[ "$cword" == @(--region|--dev) ]]; then
+		if [[ "$cword" == @(--region|--dev|--mode) ]]; then
 			COMPREPLY[$i]="${cword}="
 		else
 			COMPREPLY[$i]="${cword} "
@@ -572,6 +572,9 @@ __daxctl_comp_options()
 		--dev)
 			opts="$(__daxctl_get_devs -i)"
 			;;
+		--mode)
+			opts="system-ram devdax"
+			;;
 		*)
 			return
 			;;
@@ -582,8 +585,23 @@ __daxctl_comp_options()
 
 __daxctl_comp_non_option_args()
 {
-	# there aren't any commands that accept non option arguments yet
-	return
+	local subcmd=$1
+	local cur=$2
+	local opts
+
+	case $subcmd in
+	reconfigure-device)
+		;&
+	online-memory)
+		;&
+	offline-memory)
+		opts="$(__daxctl_get_devs -i) all"
+		;;
+	*)
+		return
+		;;
+	esac
+	__daxctlcomp "$opts" "$cur"
 }
 
 __daxctl_main()
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Reviewed-by on v7 12/13
#21 lore raw

Re: [ndctl PATCH v7 12/13] contrib/ndctl: add bash-completion for the new daxctl commands

Quoted 4 lines · Vishal Verma

Add bash completion helpers for the new daxctl-reconfigure-device, daxctl-online-memory, and daxctl-offline-memory commands.

Looks good,

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author maintainer
#22 lore raw

PATCH v7 13/13 test: Add a unit test for daxctl-reconfigure-device and friends

In reply to Vishal Verma

Add a new unit test to test dax device reconfiguration and memory operations. This teaches test/common about daxctl, and adds an ACPI.NFIT bus variable. Since we have to operate on the ACPI.NFIT bus, the test is marked as destructive.

---
 test/Makefile.am       |  3 +-
 test/common            | 19 ++++++++--
 test/daxctl-devices.sh | 81 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+), 4 deletions(-)
 create mode 100755 test/daxctl-devices.sh
test/Makefile.am +2 -1
index 874c4bb..84474d0 100644
@@ -49,7 +49,8 @@ TESTS +=\
 	dax.sh \
 	device-dax \
 	device-dax-fio.sh \
-	mmap.sh
+	mmap.sh \
+	daxctl-devices.sh
 
 if ENABLE_KEYUTILS
 TESTS += security.sh
test/common +16 -3
index 1b9d3da..1814a0c 100644
@@ -15,12 +15,25 @@ else
 	exit 1
 fi
 
-# NFIT_TEST_BUS[01]
+# DAXCTL
 #
-NFIT_TEST_BUS0=nfit_test.0
-NFIT_TEST_BUS1=nfit_test.1
+if [ -f "../daxctl/daxctl" ] && [ -x "../daxctl/daxctl" ]; then
+	export DAXCTL=../daxctl/daxctl
+elif [ -f "./daxctl/daxctl" ] && [ -x "./daxctl/daxctl" ]; then
+	export DAXCTL=./daxctl/daxctl
+else
+	echo "Couldn't find an daxctl binary"
+	exit 1
+fi
 
 
+# NFIT_TEST_BUS[01]
+#
+NFIT_TEST_BUS0="nfit_test.0"
+NFIT_TEST_BUS1="nfit_test.1"
+ACPI_BUS="ACPI.NFIT"
+E820_BUS="e820"
+
 # Functions
 
 # err
test/daxctl-devices.sh new file +81 -0

81 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Reviewed-by on v7 13/13
#23 lore raw

Re: [ndctl PATCH v7 13/13] test: Add a unit test for daxctl-reconfigure-device and friends

Quoted 6 lines · Vishal Verma

Add a new unit test to test dax device reconfiguration and memory operations. This teaches test/common about daxctl, and adds an ACPI.NFIT bus variable. Since we have to operate on the ACPI.NFIT bus, the test is marked as destructive.

Looks good.

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm