Skip to contentSkip to conversation

daxctl: add a new reconfigure-device command

PATCH In review Vishal Verma 1 participant 15 messages Aug 1, 2019 – Aug 6, 2019

Subsystems (3)
lore
Share & tools
lore

Vishal Verma

author maintainer
#0 lore raw

v9 cover daxctl: add a new reconfigure-device command

Changes in v9:
- Move the device model checking into the library. This way, daxctl-list can correctly determine 'state' which only applies to the dax-bus model.

Changes in v8:
- rename the --attempt-offline option to --force (Dan)
- clarify the messages when device is already in the requested state (Dan)
- s/unable/failed/ in device.c error messages (Dan)
- daxctl_memory_{on,off}line() instead of daxctl_memory_set_{on,off}line (Dan)
- Add an interface to get a count of the memory sections associated with a device (Dan)
- As a result, refactor the readdir loop into a common memory_op function that can set the state, get the online state, and get a count of all blocks.
- Update the onlining/offlining routines used in both the reconfigure-device and {on,off}line-memory commands to use the new daxctl_memory_num_sections() interface to validate the number of sections for which we changed the state.
- Add some small clarifications in the daxctl-reconfigure-device man page (Dan)
- In device.c add a verify_dax_bus_model() helper to check for the dax-bus subsystem (Dan).

Changes in v7:
- Fix a couple of checkaptch type errors in the new lines added in v6 (Dan).
- Get rid of daxctl_dev_get_mode. daxctl_dev_get_memory is sufficient to both check the mode and allocate the memory related structures on its first call. (Dan)
- Due to the above, daxctl_dev_mode is now private to libdaxctl, and not part of the API exported through libdaxctl.h
- Add a large enough buffer at init time to construct dynamic paths, and avoid asprintf() type allocations for memory blocks at runtime (Dan).

Changes in v6:
- For memory block online/offline operations, the kernel responds with an EINVAL for both 'real' errors, and if the memory was already in the requested state. Since there is a TOCTOU hole between checking the state and storing it, just perform a second check if the store results in an error. If the check shows the state to be the same as the one we're attempting, it means that another agent (usually udev) won the race, but we don't care so long as the state change happened, so don't report an error. (Fan Du)

Changes in v5:
- device.c: correctly set loglevel for daxctl_ctx for --verbose
- drop the subsys caching, its complexity started to exceed its benefit. dax-class device models will simply error out during reconfigure. (Dan)
- Add a note to the man page for the above.
- Clarify the onlining policy (online_movable) in the man page
- rename "numa_node" to "target_node" in device listings (Dan)
- When printing a device 'mode', assume devdax if !system-ram, avoiding a "mode: unknown" situation which can be confusing. (Dan)
- Add a "state: disabled" attribute to the device listing if a driver is not bound. This is more apt than the previous "mode: unknown" listing.
- add an api to get 'dev->resource' parsing /proc/iomem as a fallback for when the kernel doesn't provide the attribute (Dan)
- convert node_* apis to 'memory_* apis that act on a new daxctl_memory object (Dan)
- online only memory sections belonging to the device in question by cross referencing block indices with the dax device resource (Dan)
- Refuse to reconfigure a device that is already in the target mode. Until now, reconfiguring a system-ram device back to system-ram would result in a 'online memory may not be hot-removed' kernel warning.
- If the device was already in the system-ram mode, skip disabling/enabling, but still try to online the memory unless the --no-online option is in effect.
- In daxctl_unbind, also 'remove_id' to prevent devices automatically binding to the kmem driver on a disable + re-enable, which can be surprising (Dan).
- Rewrite the top half of daxctl/device.c to borrow elements from ndctl/namespace.c so that it can support growing additional commands that operate on devices (online-memory and offline-memory)
- Refactor the bottom half of daxctl/device.c so we only do the disabling/offlining steps if the device was enabled.
- Add new commands to online and offline memory sections (Dan) associated with a given dax device (Dan)
- Add a new test - daxctl-device.sh - to test daxctl reconfigure-device, online-memory, and offline-memory commands.
- Add an example in documentation demonstrating how to use numactl to bind a process to a node surfaced from a dax device (Andy Rudoff)

Changes in v4:
- Don't fail add_dax_dev for kmod failures. Instead fail only when the kmod list is actually used, i.e. during daxctl-reconfigure-device

Changes in v3:
- In daxctl_dev_get_mode(), remove the subsystem warning, detect dax-class and simply make it return devdax

Changes in v2:
- Add examples to the documentation page (Dave Hansen)
- Clarify documentation regarding the conversion from system-ram to devdax
- Remove any references to a persistent config from the documentation - those can be added when the feature is added.
- device.c: validate option compatibility
- daxctl-list: display numa_node for device listings
- daxctl-list: display mode for device listings
- make the options more consistent by adding a '-O' short option for --attempt-offline

Add a new daxctl-reconfigure-device command that lets us reconfigure DAX devices back and forth between 'system-ram' and 'device-dax' modes. It also includes facilities to online any newly hot-plugged memory (default), and attempt to offline memory before converting away from the system-ram mode (not default, requires a --attempt-offline option).

Currently missing from this series is a way to persistently store which devices have been 'marked' for use as system-ram. This depends on a config system overhaul in ndctl, and patches for those will follow separately and are independent of this work.

Example invocations:

1. Reconfigure dax0.0 to system-ram mode, don’t online the memory

    # daxctl reconfigure-device --mode=system-ram --no-online dax0.0
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "target_node":2,
        "mode":"system-ram"
      }
    ]

2. Reconfigure dax0.0 to devdax mode, attempt to offline the memory

    # daxctl reconfigure-device --human --mode=devdax --attempt-offline dax0.0
    {
      "chardev":"dax0.0",
      "size":"15.63 GiB (16.78 GB)",
      "target_node":2,
      "mode":"devdax"
    }

3. Reconfigure all dax devices on region0 to system-ram mode

    # daxctl reconfigure-device --mode=system-ram --region=0 all
    [
      {
        "chardev":"dax0.0",
        "size":16777216000,
        "target_node":2,
        "mode":"system-ram"
      },
      {
        "chardev":"dax0.1",
        "size":16777216000,
        "target_node":3,
        "mode":"system-ram"
      }
    ]

These patches can also be found in the 'kmem-pending' branch on github:
https://github.com/pmem/ndctl/tree/kmem-pending

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Pavel Tatashin <[email protected]>

Vishal Verma (13):
libdaxctl: add interfaces to get ctx and check device state
libdaxctl: add interfaces to enable/disable devices
libdaxctl: add an interface to retrieve the device resource
libdaxctl: add a 'daxctl_memory' object for memory based operations daxctl/list: add target_node for device listings daxctl/list: display the mode for a dax device
daxctl: add a new reconfigure-device command Documentation/daxctl: add a man page for daxctl-reconfigure-device
daxctl: add commands to online and offline memory
Documentation: Add man pages for daxctl-{on,off}line-memory contrib/ndctl: fix region-id completions for daxctl contrib/ndctl: add bash-completion for the new daxctl commands
test: Add a unit test for daxctl-reconfigure-device and friends

 Documentation/daxctl/Makefile.am              |   5 +-
 .../daxctl/daxctl-offline-memory.txt          |  72 ++
 Documentation/daxctl/daxctl-online-memory.txt |  80 ++
 .../daxctl/daxctl-reconfigure-device.txt      | 157 ++++
 Makefile.am                                   |   3 +-
 contrib/ndctl                                 |  38 +-
 daxctl/Makefile.am                            |   2 +
 daxctl/builtin.h                              |   3 +
 daxctl/daxctl.c                               |   3 +
 daxctl/device.c                               | 543 +++++++++++++
 daxctl/lib/Makefile.am                        |   5 +-
 daxctl/lib/libdaxctl-private.h                |  40 +
 daxctl/lib/libdaxctl.c                        | 712 ++++++++++++++++++
 daxctl/lib/libdaxctl.sym                      |  19 +
 daxctl/libdaxctl.h                            |  17 +
 test/Makefile.am                              |   3 +-
 test/common                                   |  19 +-
 test/daxctl-devices.sh                        |  81 ++
 util/iomem.c                                  |  37 +
 util/iomem.h                                  |  12 +
 util/json.c                                   |  22 +
 21 files changed, 1859 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/daxctl/daxctl-offline-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-online-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
 create mode 100644 daxctl/device.c
 create mode 100755 test/daxctl-devices.sh
 create mode 100644 util/iomem.c
 create mode 100644 util/iomem.h
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#1 lore raw

PATCH v9 1/13 libdaxctl: add interfaces to get ctx and check device state

In preparation for libdaxctl and daxctl to grow operational modes for DAX devices, add the following supporting APIs:

daxctl_dev_get_ctx
daxctl_dev_is_enabled

It also adds and uses a helper to verify the device model for the _is_enabled API, since enable/disable only make sense for the dax-bus model.

---
 daxctl/lib/libdaxctl.c   | 70 ++++++++++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym |  6 ++++
 daxctl/libdaxctl.h       |  2 ++
 3 files changed, 78 insertions(+)
daxctl/lib/libdaxctl.c +70 -0
index c2e3a52..916a49e 100644
@@ -306,6 +306,43 @@ DAXCTL_EXPORT struct daxctl_region *daxctl_new_region(struct daxctl_ctx *ctx,
 	return region;
 }
 
+static bool device_model_is_dax_bus(struct daxctl_dev *dev)
+{
+	const char *devname = daxctl_dev_get_devname(dev);
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf, *resolved;
+	size_t len = dev->buf_len;
+	struct stat sb;
+
+	if (snprintf(path, len, "/dev/%s", devname) < 0)
+		return false;
+
+	if (lstat(path, &sb) < 0) {
+		err(ctx, "%s: stat for %s failed: %s\n",
+			devname, path, strerror(errno));
+		return false;
+	}
+
+	if (snprintf(path, len, "/sys/dev/char/%d:%d/subsystem",
+			major(sb.st_rdev), minor(sb.st_rdev)) < 0)
+		return false;
+
+	resolved = realpath(path, NULL);
+	if (!resolved) {
+		err(ctx, "%s:  unable to determine subsys: %s\n",
+			devname, strerror(errno));
+		return false;
+	}
+
+	if (strcmp(resolved, "/sys/bus/dax") == 0) {
+		free(resolved);
+		return true;
+	}
+
+	free(resolved);
+	return false;
+}
+
 static void *add_dax_dev(void *parent, int id, const char *daxdev_base)
 {
 	const char *devname = devpath_to_devname(daxdev_base);
@@ -559,6 +596,39 @@ static void dax_regions_init(struct daxctl_ctx *ctx)
 	}
 }
 
+static int is_enabled(const char *drvpath)
+{
+	struct stat st;
+
+	if (lstat(drvpath, &st) < 0 || !S_ISLNK(st.st_mode))
+		return 0;
+	else
+		return 1;
+}
+
+DAXCTL_EXPORT int daxctl_dev_is_enabled(struct daxctl_dev *dev)
+{
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf;
+	int len = dev->buf_len;
+
+	if (!device_model_is_dax_bus(dev))
+		return 1;
+
+	if (snprintf(path, len, "%s/driver", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n",
+				daxctl_dev_get_devname(dev));
+		return 0;
+	}
+
+	return is_enabled(path);
+}
+
+DAXCTL_EXPORT struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev)
+{
+	return dev->region->ctx;
+}
+
 DAXCTL_EXPORT struct daxctl_dev *daxctl_dev_get_first(struct daxctl_region *region)
 {
 	dax_devices_init(region);
daxctl/lib/libdaxctl.sym +6 -0
index 84d3a69..c4af9a7 100644
@@ -50,3 +50,9 @@ LIBDAXCTL_5 {
 global:
 	daxctl_region_get_path;
 } LIBDAXCTL_4;
+
+LIBDAXCTL_6 {
+global:
+	daxctl_dev_get_ctx;
+	daxctl_dev_is_enabled;
+} LIBDAXCTL_5;
daxctl/libdaxctl.h +2 -0
index 1d13ea2..e20ccb4 100644
@@ -67,6 +67,8 @@ const char *daxctl_dev_get_devname(struct daxctl_dev *dev);
 int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
+struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
+int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#2 lore raw

PATCH v9 2/13 libdaxctl: add interfaces to enable/disable devices

In reply to Vishal Verma

Add new libdaxctl interfaces to disable a device_dax based device, and to enable it into the given mode. The modes available are 'devdax', and 'system-ram', where devdax is the normal device DAX mode used via a character device, and 'system-ram' uses the kernel's 'kmem' facility to hotplug the device making it usable as normal memory.

This adds the following new interfaces:

daxctl_dev_disable;
daxctl_dev_enable_devdax;
daxctl_dev_enable_ram;

---
 daxctl/lib/Makefile.am         |   3 +-
 daxctl/lib/libdaxctl-private.h |  21 +++
 daxctl/lib/libdaxctl.c         | 244 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   3 +
 daxctl/libdaxctl.h             |   3 +
 5 files changed, 273 insertions(+), 1 deletion(-)
daxctl/lib/Makefile.am +2 -1
index d3d4852..9f0e444 100644
@@ -16,7 +16,8 @@ libdaxctl_la_SOURCES =\
 	libdaxctl.c
 
 libdaxctl_la_LIBADD =\
-	$(UUID_LIBS)
+	$(UUID_LIBS) \
+	$(KMOD_LIBS)
 
 daxctl_modprobe_data_DATA = daxctl.conf
 
daxctl/lib/libdaxctl-private.h +21 -0
index 4a462e7..120137f 100644
@@ -13,6 +13,8 @@
 #ifndef _LIBDAXCTL_PRIVATE_H_
 #define _LIBDAXCTL_PRIVATE_H_
 
+#include <libkmod.h>
+
 #define DAXCTL_EXPORT __attribute__ ((visibility("default")))
 
 enum dax_subsystem {
@@ -26,6 +28,17 @@ static const char *dax_subsystems[] = {
 	[DAX_BUS] = "/sys/bus/dax/devices",
 };
 
+enum daxctl_dev_mode {
+	DAXCTL_DEV_MODE_DEVDAX = 0,
+	DAXCTL_DEV_MODE_RAM,
+	DAXCTL_DEV_MODE_END,
+};
+
+static const char *dax_modules[] = {
+	[DAXCTL_DEV_MODE_DEVDAX] = "device_dax",
+	[DAXCTL_DEV_MODE_RAM] = "kmem",
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -53,6 +66,14 @@ struct daxctl_dev {
 	char *dev_path;
 	struct list_node list;
 	unsigned long long size;
+	struct kmod_module *module;
+	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
 };
+
+static inline int check_kmod(struct kmod_ctx *kmod_ctx)
+{
+	return kmod_ctx ? 0 : -ENXIO;
+}
+
 #endif /* _LIBDAXCTL_PRIVATE_H_ */
daxctl/lib/libdaxctl.c +244 -0

312 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +3 -0
index c4af9a7..19904a2 100644
@@ -55,4 +55,7 @@ LIBDAXCTL_6 {
 global:
 	daxctl_dev_get_ctx;
 	daxctl_dev_is_enabled;
+	daxctl_dev_disable;
+	daxctl_dev_enable_devdax;
+	daxctl_dev_enable_ram;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +3 -0
index e20ccb4..407f459 100644
@@ -69,6 +69,9 @@ int daxctl_dev_get_minor(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
+int daxctl_dev_disable(struct daxctl_dev *dev);
+int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
+int daxctl_dev_enable_ram(struct daxctl_dev *dev);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#3 lore raw

PATCH v9 3/13 libdaxctl: add an interface to retrieve the device resource

In reply to Vishal Verma

Add an interface to retrieve the 'resource' attribute for a dax device.

Attempt to retrieve it as usual via sysfs, but since older kernels may be missing this attribute, as a fallback, attempt to retrieve it from /proc/iomem

Cc Dan Williams <[email protected]>
[fscanf format string problem and diagnosis]
---
 Makefile.am                    |  3 ++-
 daxctl/lib/Makefile.am         |  2 ++
 daxctl/lib/libdaxctl-private.h |  1 +
 daxctl/lib/libdaxctl.c         | 12 +++++++++++
 daxctl/lib/libdaxctl.sym       |  1 +
 daxctl/libdaxctl.h             |  1 +
 util/iomem.c                   | 37 ++++++++++++++++++++++++++++++++++
 util/iomem.h                   | 12 +++++++++++
 8 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 util/iomem.c
 create mode 100644 util/iomem.h
Makefile.am +2 -1
index df8797e..8d10a10 100644
@@ -74,6 +74,7 @@ libutil_a_SOURCES = \
 	util/wrapper.c \
 	util/filter.c \
 	util/bitmap.c \
-	util/abspath.c
+	util/abspath.c \
+	util/iomem.c
 
 nobase_include_HEADERS = daxctl/libdaxctl.h
daxctl/lib/Makefile.am +2 -0
index 9f0e444..7704b1b 100644
@@ -9,6 +9,8 @@ lib_LTLIBRARIES = libdaxctl.la
 libdaxctl_la_SOURCES =\
 	../libdaxctl.h \
 	libdaxctl-private.h \
+	../../util/iomem.c \
+	../../util/iomem.h \
 	../../util/sysfs.c \
 	../../util/sysfs.h \
 	../../util/log.c \
daxctl/lib/libdaxctl-private.h +1 -0
index 120137f..fee67d1 100644
@@ -65,6 +65,7 @@ struct daxctl_dev {
 	size_t buf_len;
 	char *dev_path;
 	struct list_node list;
+	unsigned long long resource;
 	unsigned long long size;
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
daxctl/lib/libdaxctl.c +12 -0
index caf661e..aa0d2f2 100644
@@ -24,6 +24,7 @@
 
 #include <util/log.h>
 #include <util/sysfs.h>
+#include <util/iomem.h>
 #include <daxctl/libdaxctl.h>
 #include "libdaxctl-private.h"
 
@@ -406,6 +407,12 @@ static void *add_dax_dev(void *parent, int id, const char *daxdev_base)
 	dev->major = major(st.st_rdev);
 	dev->minor = minor(st.st_rdev);
 
+	sprintf(path, "%s/resource", daxdev_base);
+	if (sysfs_read_attr(ctx, path, buf) == 0)
+		dev->resource = strtoull(buf, NULL, 0);
+	else
+		dev->resource = iomem_get_dev_resource(ctx, daxdev_base);
+
 	sprintf(path, "%s/size", daxdev_base);
 	if (sysfs_read_attr(ctx, path, buf) < 0)
 		goto err_read;
@@ -928,6 +935,11 @@ DAXCTL_EXPORT int daxctl_dev_get_minor(struct daxctl_dev *dev)
 	return dev->minor;
 }
 
+DAXCTL_EXPORT unsigned long long daxctl_dev_get_resource(struct daxctl_dev *dev)
+{
+	return dev->resource;
+}
+
 DAXCTL_EXPORT unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev)
 {
 	return dev->size;
daxctl/lib/libdaxctl.sym +1 -0
index 19904a2..1692624 100644
@@ -58,4 +58,5 @@ global:
 	daxctl_dev_disable;
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
+	daxctl_dev_get_resource;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +1 -0
index 407f459..adf55f3 100644
@@ -66,6 +66,7 @@ int daxctl_dev_get_id(struct daxctl_dev *dev);
 const char *daxctl_dev_get_devname(struct daxctl_dev *dev);
 int daxctl_dev_get_major(struct daxctl_dev *dev);
 int daxctl_dev_get_minor(struct daxctl_dev *dev);
+unsigned long long daxctl_dev_get_resource(struct daxctl_dev *dev);
 unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev);
 struct daxctl_ctx *daxctl_dev_get_ctx(struct daxctl_dev *dev);
 int daxctl_dev_is_enabled(struct daxctl_dev *dev);
util/iomem.c new file +37 -0
new file mode 100644
index 0000000..a3c23f5
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2019 Intel Corporation. All rights reserved. */
+
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <util/log.h>
+#include <util/iomem.h>
+#include <util/sysfs.h>
+
+unsigned long long __iomem_get_dev_resource(struct log_ctx *ctx,
+		const char *devpath)
+{
+	const char *devname = devpath_to_devname(devpath);
+	FILE *fp = fopen("/proc/iomem", "r");
+	unsigned long long res;
+	char name[256];
+
+	if (fp == NULL) {
+		log_err(ctx, "%s: open /proc/iomem: %s\n", devname,
+				strerror(errno));
+		return 0;
+	}
+
+	while (fscanf(fp, "%llx-%*x : %254[^\n]\n", &res, name) == 2) {
+		if (strcmp(name, devname) == 0) {
+			log_dbg(ctx, "%s: got resource via iomem: %#llx\n",
+					devname, res);
+			fclose(fp);
+			return res;
+		}
+	}
+
+	log_dbg(ctx, "%s: not found in iomem\n", devname);
+	fclose(fp);
+	return 0;
+}
util/iomem.h new file +12 -0
new file mode 100644
index 0000000..aaaf6a7
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2019 Intel Corporation. All rights reserved. */
+#ifndef _NDCTL_IOMEM_H_
+#define _NDCTL_IOMEM_H_
+
+struct log_ctx;
+unsigned long long __iomem_get_dev_resource(struct log_ctx *ctx,
+		const char *path);
+
+#define iomem_get_dev_resource(c, p) __iomem_get_dev_resource(&(c)->ctx, (p))
+
+#endif /* _NDCTL_IOMEM_H_ */
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#4 lore raw

PATCH v9 4/13 libdaxctl: add a 'daxctl_memory' object for memory based operations

In reply to Vishal Verma

Introduce a new 'daxctl_memory' object, which will be used for operations related to managing dax devices in 'system-memory' modes.

Add libdaxctl APIs to get the target_node of a DAX device, and to online, offline, and query the state of hotplugged memory sections associated with a given device.

This adds the following new interfaces:

daxctl_dev_get_target_node
daxctl_dev_get_memory
daxctl_memory_get_dev
daxctl_memory_get_node_path
daxctl_memory_get_block_size
daxctl_memory_online
daxctl_memory_offline
daxctl_memory_is_online
daxctl_memory_num_sections

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
[for the memblock-already-online TOCTOU hole]
---
 daxctl/lib/libdaxctl-private.h |  18 ++
 daxctl/lib/libdaxctl.c         | 384 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   9 +
 daxctl/libdaxctl.h             |  11 +
 4 files changed, 422 insertions(+)
daxctl/lib/libdaxctl-private.h +18 -0
index fee67d1..01091de 100644
@@ -39,6 +39,13 @@ static const char *dax_modules[] = {
 	[DAXCTL_DEV_MODE_RAM] = "kmem",
 };
 
+enum memory_op {
+	MEM_SET_OFFLINE,
+	MEM_SET_ONLINE,
+	MEM_IS_ONLINE,
+	MEM_COUNT,
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -70,8 +77,19 @@ struct daxctl_dev {
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
+	struct daxctl_memory *mem;
+	int target_node;
+};
+
+struct daxctl_memory {
+	struct daxctl_dev *dev;
+	void *mem_buf;
+	size_t buf_len;
+	char *node_path;
+	unsigned long block_size;
 };
 
+
 static inline int check_kmod(struct kmod_ctx *kmod_ctx)
 {
 	return kmod_ctx ? 0 : -ENXIO;
daxctl/lib/libdaxctl.c +384 -0

417 lines not on this page · raw message

daxctl/lib/libdaxctl.sym +9 -0
index 1692624..bc18604 100644
@@ -59,4 +59,13 @@ global:
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
 	daxctl_dev_get_resource;
+	daxctl_dev_get_target_node;
+	daxctl_dev_get_memory;
+	daxctl_memory_get_dev;
+	daxctl_memory_get_node_path;
+	daxctl_memory_get_block_size;
+	daxctl_memory_online;
+	daxctl_memory_offline;
+	daxctl_memory_is_online;
+	daxctl_memory_num_sections;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +11 -0
index adf55f3..fb6c3b1 100644
@@ -73,6 +73,17 @@ int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 int daxctl_dev_disable(struct daxctl_dev *dev);
 int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
 int daxctl_dev_enable_ram(struct daxctl_dev *dev);
+int daxctl_dev_get_target_node(struct daxctl_dev *dev);
+
+struct daxctl_memory;
+struct daxctl_memory *daxctl_dev_get_memory(struct daxctl_dev *dev);
+struct daxctl_dev *daxctl_memory_get_dev(struct daxctl_memory *mem);
+const char *daxctl_memory_get_node_path(struct daxctl_memory *mem);
+unsigned long daxctl_memory_get_block_size(struct daxctl_memory *mem);
+int daxctl_memory_online(struct daxctl_memory *mem);
+int daxctl_memory_offline(struct daxctl_memory *mem);
+int daxctl_memory_is_online(struct daxctl_memory *mem);
+int daxctl_memory_num_sections(struct daxctl_memory *mem);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer Tested-by Fan Du on v9 4/13
#5 lore raw

Re: [ndctl PATCH v9 04/13] libdaxctl: add a 'daxctl_memory' object for memory based operations

Quoted 26 lines · Vishal Verma

Introduce a new 'daxctl_memory' object, which will be used for operations related to managing dax devices in 'system-memory' modes.

Add libdaxctl APIs to get the target_node of a DAX device, and to online, offline, and query the state of hotplugged memory sections associated with a given device.

This adds the following new interfaces:

daxctl_dev_get_target_node
daxctl_dev_get_memory
daxctl_memory_get_dev
daxctl_memory_get_node_path
daxctl_memory_get_block_size
daxctl_memory_online
daxctl_memory_offline
daxctl_memory_is_online
daxctl_memory_num_sections

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
[for the memblock-already-online TOCTOU hole]
Quoted 3 lines
---
 daxctl/lib/libdaxctl-private.h |  18 ++
 daxctl/lib/libdaxctl.c         | 384

+++++++++++++++++++++++++++++++++

Quoted 4 lines
 daxctl/lib/libdaxctl.sym       |   9 +
 daxctl/libdaxctl.h             |  11 +
 4 files changed, 422 insertions(+)

[..]

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+static bool memblock_in_dev(struct daxctl_dev *dev, const char

*memblock)

Review

Quoted 3 lines
(quoted diff) +2 -0
+{
+	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);

Static analysis complains that this can potentially cause a NULL dereference. Fix it by passing the mem object to memblock_in_dev(), since it has already been validated by that time.

3<----

Quoted 1 line

From e8bf803e359b784259f645d1ff68e964b2c8618f Mon Sep 17 00:00:00 2001

From: Vishal Verma <[email protected]>
Date: Fri, 3 May 2019 13:27:35 -0600
Subject: [ndctl PATCH] libdaxctl: add a 'daxctl_memory' object for memory
based operations

Introduce a new 'daxctl_memory' object, which will be used for operations related to managing dax devices in 'system-memory' modes.

Add libdaxctl APIs to get the target_node of a DAX device, and to online, offline, and query the state of hotplugged memory sections associated with a given device.

This adds the following new interfaces:

daxctl_dev_get_target_node
daxctl_dev_get_memory
daxctl_memory_get_dev
daxctl_memory_get_node_path
daxctl_memory_get_block_size
daxctl_memory_online
daxctl_memory_offline
daxctl_memory_is_online
daxctl_memory_num_sections

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
[for the memblock-already-online TOCTOU hole]
---
 daxctl/lib/libdaxctl-private.h |  18 ++
 daxctl/lib/libdaxctl.c         | 384 +++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |   9 +
 daxctl/libdaxctl.h             |  11 +
 4 files changed, 422 insertions(+)
daxctl/lib/libdaxctl- renamed from daxctl/lib/libdaxctl-private.h +0 -0

private.h
index fee67d1..01091de 100644

daxctl/lib/libdaxctl-private.h +18 -0
@@ -39,6 +39,13 @@ static const char *dax_modules[] = {
 	[DAXCTL_DEV_MODE_RAM] = "kmem",
 };
 
+enum memory_op {
+	MEM_SET_OFFLINE,
+	MEM_SET_ONLINE,
+	MEM_IS_ONLINE,
+	MEM_COUNT,
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
@@ -70,8 +77,19 @@ struct daxctl_dev {
 	struct kmod_module *module;
 	struct kmod_list *kmod_list;
 	struct daxctl_region *region;
+	struct daxctl_memory *mem;
+	int target_node;
+};
+
+struct daxctl_memory {
+	struct daxctl_dev *dev;
+	void *mem_buf;
+	size_t buf_len;
+	char *node_path;
+	unsigned long block_size;
 };
 
+
 static inline int check_kmod(struct kmod_ctx *kmod_ctx)
 {
 	return kmod_ctx ? 0 : -ENXIO;
daxctl/lib/libdaxctl.c +0 -0
index aa0d2f2..bcc77b6 100644
@@ -200,6 +200,15 @@ DAXCTL_EXPORT void daxctl_region_get_uuid(struct

daxctl_region *region, uuid_t u
uuid_copy(uu, region->uuid);
}

+static void free_mem(struct daxctl_dev *dev)
+{
+ if (dev && dev->mem) {
+ free(dev->mem->node_path);
+ free(dev->mem);
+ dev->mem = NULL;
+ }
+}
+
static void free_dev(struct daxctl_dev *dev, struct list_head *head) {
if (head)
@@ -207,6 +216,7 @@ static void free_dev(struct daxctl_dev *dev, struct list_head *head)
kmod_module_unref_list(dev->kmod_list); free(dev->dev_buf);
free(dev->dev_path);
+ free_mem(dev);
free(dev);
}

@@ -380,6 +390,94 @@ static struct kmod_list *to_module_list(struct daxctl_ctx *ctx,
return list;
}

+static int dev_is_system_ram_capable(struct daxctl_dev *dev) +{
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ char *mod_path, *mod_base;
+ char path[200];
+ const int len = sizeof(path);
+
+ if (!device_model_is_dax_bus(dev))
+ return false;
+
+ if (!daxctl_dev_is_enabled(dev))
+ return false;
+
+ if (snprintf(path, len, "%s/driver/module", dev->dev_path) >=
len) {
+ err(ctx, "%s: buffer too small!\n", devname);
+ return false;
+ }
+
+ mod_path = realpath(path, NULL);
+ if (!mod_path)
+ return false;
+
+ mod_base = basename(mod_path);
+ if (strcmp(mod_base, dax_modules[DAXCTL_DEV_MODE_RAM]) == 0) {
+ free(mod_path);
+ return true;
+ }
+
+ free(mod_path);
+ return false;
+}
+
+/*
+ * This checks for the device to be in system-ram mode, so calling
+ * daxctl_dev_get_memory() on a devdax mode device will always return NULL.
+ */
+static struct daxctl_memory *daxctl_dev_alloc_mem(struct daxctl_dev *dev)
+{
+ const char *size_path =
"/sys/devices/system/memory/block_size_bytes";
+ const char *node_base = "/sys/devices/system/node/node";
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ struct daxctl_memory *mem;
+ char buf[SYSFS_ATTR_SIZE];
+ int node_num;
+
+ if (!dev_is_system_ram_capable(dev))
+ return NULL;
+
+ mem = calloc(1, sizeof(*mem));
+ if (!mem)
+ return NULL;
+
+ mem->dev = dev;
+
+ if (sysfs_read_attr(ctx, size_path, buf) == 0) {
+ mem->block_size = strtoul(buf, NULL, 16);
+ if (mem->block_size == 0 || mem->block_size ==
ULONG_MAX) {
+ err(ctx, "%s: Unable to determine memblock size:
%s\n",
+ devname, strerror(errno));
+ mem->block_size = 0;
+ }
+ }
+
+ node_num = daxctl_dev_get_target_node(dev);
+ if (node_num >= 0) {
+ if (asprintf(&mem->node_path, "%s%d", node_base,
+ node_num) < 0) {
+ err(ctx, "%s: Unable to set node_path\n",
devname);
+ goto err_mem;
+ }
+ }
+
+ mem->mem_buf = calloc(1, strlen(node_base) + 256);
+ if (!mem->mem_buf)
+ goto err_node;
+ mem->buf_len = strlen(node_base) + 256;
+
+ return mem;
+
+err_node:
+ free(mem->node_path);
+err_mem:
+ free(mem);
+ return NULL;
+}
+
static void *add_dax_dev(void *parent, int id, const char *daxdev_base) {
const char *devname = devpath_to_devname(daxdev_base);
@@ -435,6 +533,12 @@ static void *add_dax_dev(void *parent, int id, const char *daxdev_base)
if (rc == 0)
dev->kmod_list = to_module_list(ctx, buf);

+ sprintf(path, "%s/target_node", daxdev_base);
+ if (sysfs_read_attr(ctx, path, buf) == 0)
+ dev->target_node = strtol(buf, NULL, 0);
+ else
+ dev->target_node = -1;
+
daxctl_dev_foreach(region, dev_dup)
if (dev_dup->id == dev->id) {
free_dev(dev, NULL);
@@ -862,6 +966,9 @@ DAXCTL_EXPORT int daxctl_dev_disable(struct daxctl_dev *dev)
if (!daxctl_dev_is_enabled(dev))
return 0;

+ /* If there is a memory object, first free that */
+ free_mem(dev);
+
daxctl_unbind(ctx, dev->dev_path);

if (daxctl_dev_is_enabled(dev)) {
@@ -944,3 +1051,280 @@ DAXCTL_EXPORT unsigned long long daxctl_dev_get_size(struct daxctl_dev *dev)
{
return dev->size;
}
+
+DAXCTL_EXPORT int daxctl_dev_get_target_node(struct daxctl_dev *dev) +{
+ return dev->target_node;
+}
+
+DAXCTL_EXPORT struct daxctl_memory *daxctl_dev_get_memory(struct daxctl_dev *dev)
+{
+ if (dev->mem)
+ return dev->mem;
+ else
+ return daxctl_dev_alloc_mem(dev);
+}
+
+DAXCTL_EXPORT struct daxctl_dev *daxctl_memory_get_dev(struct daxctl_memory *mem)
+{
+ return mem->dev;
+}
+
+DAXCTL_EXPORT const char *daxctl_memory_get_node_path(struct daxctl_memory *mem)
+{
+ return mem->node_path;
+}
+
+DAXCTL_EXPORT unsigned long daxctl_memory_get_block_size(struct daxctl_memory *mem)
+{
+ return mem->block_size;
+}
+
+static int online_one_memblock(struct daxctl_dev *dev, char *path) +{
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ const char *mode = "online_movable";
+ char buf[SYSFS_ATTR_SIZE];
+ int rc;
+
+ rc = sysfs_read_attr(ctx, path, buf);
+ if (rc) {
+ err(ctx, "%s: Failed to read %s: %s\n",
+ devname, path, strerror(-rc));
+ return rc;
+ }
+
+ /*
+ * if already online, possibly due to kernel config or a udev
rule,
+ * there is nothing to do and we can skip over the memblock
+ */
+ if (strncmp(buf, "online", 6) == 0)
+ return 1;
+
+ rc = sysfs_write_attr_quiet(ctx, path, mode);
+ if (rc) {
+ /*
+ * While we performed an already-online check above,
there
+ * is still a TOCTOU hole where someone (such as a udev
rule)
+ * may have raced to online the memory. In such a case,
+ * the sysfs store will fail, however we can check for
this
+ * by simply reading the state again. If it changed to
the
+ * desired state, then we don't have to error out.
+ */
+ if (sysfs_read_attr(ctx, path, buf) == 0) {
+ if (strncmp(buf, "online", 6) == 0)
+ return 1;
+ }
+ err(ctx, "%s: Failed to online %s: %s\n",
+ devname, path, strerror(-rc));
+ }
+ return rc;
+}
+
+static int offline_one_memblock(struct daxctl_dev *dev, char *path) +{
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ const char *mode = "offline";
+ char buf[SYSFS_ATTR_SIZE];
+ int rc;
+
+ rc = sysfs_read_attr(ctx, path, buf);
+ if (rc) {
+ err(ctx, "%s: Failed to read %s: %s\n",
+ devname, path, strerror(-rc));
+ return rc;
+ }
+
+ /* if already offline, there is nothing to do */
+ if (strncmp(buf, "offline", 7) == 0)
+ return 1;
+
+ rc = sysfs_write_attr_quiet(ctx, path, mode);
+ if (rc) {
+ /* Close the TOCTOU hole like in online_one_memblock()
above */
+ if (sysfs_read_attr(ctx, path, buf) == 0) {
+ if (strncmp(buf, "offline", 7) == 0)
+ return 1;
+ }
+ err(ctx, "%s: Failed to offline %s: %s\n",
+ devname, path, strerror(-rc));
+ }
+ return rc;
+}
+
+static int memblock_is_online(struct daxctl_dev *dev, char *path) +{
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ char buf[SYSFS_ATTR_SIZE];
+ int rc;
+
+ rc = sysfs_read_attr(ctx, path, buf);
+ if (rc) {
+ err(ctx, "%s: Failed to read %s: %s\n",
+ devname, path, strerror(-rc));
+ return rc;
+ }
+
+ if (strncmp(buf, "online", 6) == 0)
+ return 1;
+
+ /* offline */
+ return 0;
+}
+
+static bool memblock_in_dev(struct daxctl_memory *mem, const char *memblock)
+{
+ const char *mem_base = "/sys/devices/system/memory/";
+ struct daxctl_dev *dev = daxctl_memory_get_dev(mem);
+ unsigned long long memblock_res, dev_start, dev_end;
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ unsigned long memblock_size;
+ int path_len = mem->buf_len;
+ char buf[SYSFS_ATTR_SIZE];
+ unsigned long phys_index;
+ char *path = mem->mem_buf;
+
+ if (snprintf(path, path_len, "%s/%s/phys_index",
+ mem_base, memblock) < 0)
+ return false;
+
+ if (sysfs_read_attr(ctx, path, buf) == 0) {
+ phys_index = strtoul(buf, NULL, 16);
+ if (phys_index == 0 || phys_index == ULONG_MAX) {
+ err(ctx, "%s: %s: Unable to determine
phys_index: %s\n",
+ devname, memblock, strerror(errno));
+ return false;
+ }
+ } else {
+ err(ctx, "%s: %s: Unable to determine phys_index: %s\n",
+ devname, memblock, strerror(errno));
+ return false;
+ }
+
+ dev_start = daxctl_dev_get_resource(dev);
+ if (!dev_start) {
+ err(ctx, "%s: Unable to determine resource\n", devname);
+ return false;
+ }
+ dev_end = dev_start + daxctl_dev_get_size(dev);
+
+ memblock_size = daxctl_memory_get_block_size(mem);
+ if (!memblock_size) {
+ err(ctx, "%s: Unable to determine memory block size\n",
+ devname);
+ return false;
+ }
+ memblock_res = phys_index * memblock_size;
+
+ if (memblock_res >= dev_start && memblock_res <= dev_end)
+ return true;
+
+ return false;
+}
+
+static int op_for_one_memblock(struct daxctl_memory *mem, char *path,
+ enum memory_op op)
+{
+ struct daxctl_dev *dev = daxctl_memory_get_dev(mem);
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ int rc;
+
+ switch (op) {
+ case MEM_SET_ONLINE:
+ return online_one_memblock(dev, path);
+ case MEM_SET_OFFLINE:
+ return offline_one_memblock(dev, path);
+ case MEM_IS_ONLINE:
+ rc = memblock_is_online(dev, path);
+ if (rc < 0)
+ return rc;
+ /*
+ * Retain the 'normal' semantics for if
(memblock_is_online()),
+ * but since count needs rc == 0, we'll just flip rc for
this op
+ */
+ return !rc;
+ case MEM_COUNT:
+ return 0;
+ }
+
+ err(ctx, "%s: BUG: unknown op: %d\n", devname, op);
+ return -EINVAL;
+}
+
+static int daxctl_memory_op(struct daxctl_memory *mem, enum memory_op op)
+{
+ struct daxctl_dev *dev = daxctl_memory_get_dev(mem);
+ const char *devname = daxctl_dev_get_devname(dev);
+ struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+ const char *node_path;
+ int rc, count = 0;
+ struct dirent *de;
+ DIR *node_dir;
+
+ node_path = daxctl_memory_get_node_path(mem);
+ if (!node_path) {
+ err(ctx, "%s: Failed to get node_path\n", devname);
+ return -ENXIO;
+ }
+
+ node_dir = opendir(node_path);
+ if (!node_dir)
+ return -errno;
+
+ errno = 0;
+ while ((de = readdir(node_dir)) != NULL) {
+ char *path = mem->mem_buf;
+ int len = mem->buf_len;
+
+ if (strncmp(de->d_name, "memory", 6) == 0) {
+ if (!memblock_in_dev(mem, de->d_name))
+ continue;
+ rc = snprintf(path, len, "%s/%s/state",
+ node_path, de->d_name);
+ if (rc < 0) {
+ rc = -ENOMEM;
+ goto out_dir;
+ }
+ rc = op_for_one_memblock(mem, path, op);
+ if (rc < 0)
+ goto out_dir;
+ if (rc == 0)
+ count++;
+ }
+ errno = 0;
+ }
+ if (errno) {
+ rc = -errno;
+ goto out_dir;
+ }
+ rc = count;
+
+out_dir:
+ closedir(node_dir);
+ return rc;
+}
+
+DAXCTL_EXPORT int daxctl_memory_online(struct daxctl_memory *mem) +{
+ return daxctl_memory_op(mem, MEM_SET_ONLINE);
+}
+
+DAXCTL_EXPORT int daxctl_memory_offline(struct daxctl_memory *mem) +{
+ return daxctl_memory_op(mem, MEM_SET_OFFLINE);
+}
+
+DAXCTL_EXPORT int daxctl_memory_is_online(struct daxctl_memory *mem) +{
+ return daxctl_memory_op(mem, MEM_IS_ONLINE);
+}
+
+DAXCTL_EXPORT int daxctl_memory_num_sections(struct daxctl_memory *mem) +{
+ return daxctl_memory_op(mem, MEM_COUNT);
+}

daxctl/lib/libdaxctl.sym +9 -0
index 1692624..bc18604 100644
@@ -59,4 +59,13 @@ global:
 	daxctl_dev_enable_devdax;
 	daxctl_dev_enable_ram;
 	daxctl_dev_get_resource;
+	daxctl_dev_get_target_node;
+	daxctl_dev_get_memory;
+	daxctl_memory_get_dev;
+	daxctl_memory_get_node_path;
+	daxctl_memory_get_block_size;
+	daxctl_memory_online;
+	daxctl_memory_offline;
+	daxctl_memory_is_online;
+	daxctl_memory_num_sections;
 } LIBDAXCTL_5;
daxctl/libdaxctl.h +11 -0
index adf55f3..fb6c3b1 100644
@@ -73,6 +73,17 @@ int daxctl_dev_is_enabled(struct daxctl_dev *dev);
 int daxctl_dev_disable(struct daxctl_dev *dev);
 int daxctl_dev_enable_devdax(struct daxctl_dev *dev);
 int daxctl_dev_enable_ram(struct daxctl_dev *dev);
+int daxctl_dev_get_target_node(struct daxctl_dev *dev);
+
+struct daxctl_memory;
+struct daxctl_memory *daxctl_dev_get_memory(struct daxctl_dev *dev);
+struct daxctl_dev *daxctl_memory_get_dev(struct daxctl_memory *mem);
+const char *daxctl_memory_get_node_path(struct daxctl_memory *mem);
+unsigned long daxctl_memory_get_block_size(struct daxctl_memory *mem);
+int daxctl_memory_online(struct daxctl_memory *mem);
+int daxctl_memory_offline(struct daxctl_memory *mem);
+int daxctl_memory_is_online(struct daxctl_memory *mem);
+int daxctl_memory_num_sections(struct daxctl_memory *mem);
 
 #define daxctl_dev_foreach(region, dev) \
         for (dev = daxctl_dev_get_first(region); \
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#6 lore raw

PATCH v9 5/13 daxctl/list: add target_node for device listings

In reply to Vishal Verma

The kernel provides a 'target_node' attribute for dax devices. When converting a dax device to the system-ram mode, the memory is hotplugged into this numa node. It would be helpful to print this in device listings so that it is easy for applications to detect the node to which the new memory belongs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 util/json.c | 8 ++++++++
 1 file changed, 8 insertions(+)
util/json.c +8 -0
index babdc8c..f521337 100644
@@ -271,6 +271,7 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 {
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
+	int node;
 
 	jdev = json_object_new_object();
 	if (!devname || !jdev)
@@ -284,6 +285,13 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 	if (jobj)
 		json_object_object_add(jdev, "size", jobj);
 
+	node = daxctl_dev_get_target_node(dev);
+	if (node >= 0) {
+		jobj = json_object_new_int(node);
+		if (jobj)
+			json_object_object_add(jdev, "target_node", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#7 lore raw

PATCH v9 6/13 daxctl/list: display the mode for a dax device

In reply to Vishal Verma

In preparation for a reconfigure-device command, allow JSON listings to display the 'mode' of a dax device. This will allow the reconfigure-device command (and via daxctl_dev_to_json(), also daxctl-list) to print the mode in device listings via a 'daxctl-list' command or immediately after a mode change.

Add a 'state' attribute to the json listings for devices, since a device could end up in a state where it is not bound to any driver, and hence, 'disabled'. The state attribute is only displayed for disabled devices.

Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/lib/libdaxctl.c |  2 ++
 util/json.c            | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
daxctl/lib/libdaxctl.c +2 -0
index 949c56f..edb8257 100644
@@ -12,6 +12,8 @@
  */
 #include <stdio.h>
 #include <errno.h>
+#include <limits.h>
+#include <libgen.h>
 #include <stdlib.h>
 #include <dirent.h>
 #include <unistd.h>
util/json.c +14 -0
index f521337..9f80b5b 100644
@@ -269,6 +269,7 @@ struct json_object *util_dimm_to_json(struct ndctl_dimm *dimm,
 struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 		unsigned long flags)
 {
+	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj;
 	int node;
@@ -292,6 +293,19 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 			json_object_object_add(jdev, "target_node", jobj);
 	}
 
+	if (mem)
+		jobj = json_object_new_string("system-ram");
+	else
+		jobj = json_object_new_string("devdax");
+	if (jobj)
+		json_object_object_add(jdev, "mode", jobj);
+
+	if (!daxctl_dev_is_enabled(dev)) {
+		jobj = json_object_new_string("disabled");
+		if (jobj)
+			json_object_object_add(jdev, "state", jobj);
+	}
+
 	return jdev;
 }
 
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#8 lore raw

PATCH v9 7/13 daxctl: add a new reconfigure-device command

In reply to Vishal Verma

Add a new command 'daxctl-reconfigure-device'. This is used to switch the mode of a dax device between regular 'device_dax' and 'system-memory'. The command also uses the memory hotplug sysfs interfaces to online the newly available memory when converting to 'system-ram', and to attempt to offline the memory when converting back to a DAX device.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 daxctl/Makefile.am |   2 +
 daxctl/builtin.h   |   1 +
 daxctl/daxctl.c    |   1 +
 daxctl/device.c    | 455 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 459 insertions(+)
 create mode 100644 daxctl/device.c
daxctl/Makefile.am +2 -0
index 94f73f9..66dcc7f 100644
@@ -15,10 +15,12 @@ daxctl_SOURCES =\
 		daxctl.c \
 		list.c \
 		migrate.c \
+		device.c \
 		../util/json.c
 
 daxctl_LDADD =\
 	lib/libdaxctl.la \
 	../libutil.a \
 	$(UUID_LIBS) \
+	$(KMOD_LIBS) \
 	$(JSON_LIBS)
daxctl/builtin.h +1 -0
index 00ef5e9..756ba2a 100644
@@ -6,4 +6,5 @@
 struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +1 -0
index 2e41747..e1ba7b8 100644
@@ -71,6 +71,7 @@ static struct cmd_struct commands[] = {
 	{ "list", .d_fn = cmd_list },
 	{ "help", .d_fn = cmd_help },
 	{ "migrate-device-model", .d_fn = cmd_migrate },
+	{ "reconfigure-device", .d_fn = cmd_reconfig_device },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c new file +455 -0

455 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#9 lore raw

PATCH v9 8/13 Documentation/daxctl: add a man page for daxctl-reconfigure-device

In reply to Vishal Verma

Add a man page describing the new daxctl-reconfigure-device command.

Cc Pavel Tatashin <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Dan Williams <[email protected]>
---
 Documentation/daxctl/Makefile.am              |   3 +-
 .../daxctl/daxctl-reconfigure-device.txt      | 157 ++++++++++++++++++
 2 files changed, 159 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-reconfigure-device.txt
Documentation/daxctl/Makefile.am +2 -1
index 6aba035..715fbad 100644
@@ -28,7 +28,8 @@ endif
 man1_MANS = \
 	daxctl.1 \
 	daxctl-list.1 \
-	daxctl-migrate-device-model.1
+	daxctl-migrate-device-model.1 \
+	daxctl-reconfigure-device.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-reconfigure-device.txt new file +157 -0

157 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#10 lore raw

PATCH v9 9/13 daxctl: add commands to online and offline memory

In reply to Vishal Verma

Add two new commands:

daxctl-online-memory
daxctl-offline-memory

to manage the state of hot-plugged memory from the system-ram mode for dax devices. This provides a way for the user to online/offline the memory as a separate step from the reconfiguration. Without this, a user that reconfigures a device into the system-ram mode with the --no-online option, would have no way to later online the memory, and would have to resort to shell scripting to online them manually via sysfs.

Cc Dan Williams <[email protected]>
Cc Dave Hansen <[email protected]>
---
 daxctl/builtin.h |  2 ++
 daxctl/daxctl.c  |  2 ++
 daxctl/device.c  | 88 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+)
daxctl/builtin.h +2 -0
index 756ba2a..f5a0147 100644
@@ -7,4 +7,6 @@ struct daxctl_ctx;
 int cmd_list(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_migrate(int argc, const char **argv, struct daxctl_ctx *ctx);
 int cmd_reconfig_device(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_online_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
+int cmd_offline_memory(int argc, const char **argv, struct daxctl_ctx *ctx);
 #endif /* _DAXCTL_BUILTIN_H_ */
daxctl/daxctl.c +2 -0
index e1ba7b8..1ab0732 100644
@@ -72,6 +72,8 @@ static struct cmd_struct commands[] = {
 	{ "help", .d_fn = cmd_help },
 	{ "migrate-device-model", .d_fn = cmd_migrate },
 	{ "reconfigure-device", .d_fn = cmd_reconfig_device },
+	{ "online-memory", .d_fn = cmd_online_memory },
+	{ "offline-memory", .d_fn = cmd_offline_memory },
 };
 
 int main(int argc, const char **argv)
daxctl/device.c +88 -0

127 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#11 lore raw

PATCH v9 10/13 Documentation: Add man pages for daxctl-{on, off}line-memory

In reply to Vishal Verma

Add man pages for the two new commands: daxctl-online-memory, and daxctl-offline-memory.

---
 Documentation/daxctl/Makefile.am              |  4 +-
 .../daxctl/daxctl-offline-memory.txt          | 72 +++++++++++++++++
 Documentation/daxctl/daxctl-online-memory.txt | 80 +++++++++++++++++++
 3 files changed, 155 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/daxctl/daxctl-offline-memory.txt
 create mode 100644 Documentation/daxctl/daxctl-online-memory.txt
Documentation/daxctl/Makefile.am +3 -1
index 715fbad..37c3bde 100644
@@ -29,7 +29,9 @@ man1_MANS = \
 	daxctl.1 \
 	daxctl-list.1 \
 	daxctl-migrate-device-model.1 \
-	daxctl-reconfigure-device.1
+	daxctl-reconfigure-device.1 \
+	daxctl-online-memory.1 \
+	daxctl-offline-memory.1
 
 CLEANFILES = $(man1_MANS)
 
Documentation/daxctl/daxctl-offline-memory.txt new file +72 -0

72 lines not on this page · raw message

Documentation/daxctl/daxctl-online-memory.txt new file +80 -0

80 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#12 lore raw

PATCH v9 11/13 contrib/ndctl: fix region-id completions for daxctl

In reply to Vishal Verma

The completion helpers for daxctl assumed the region arguments for specifying daxctl regions were the same as ndctl regions, i.e. "regionX". This is not true - daxctl region arguments are a simple numeric 'id'.

Add a new helper __daxctl_get_regions() to complete daxctl region IDs properly.

While at it, fix a useless use of 'echo' in __daxctl_get_devs() and quoting in __daxctl_comp_options()

Fixes d6790a32f32c ("daxctl: Add bash-completion")
---
 contrib/ndctl | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/ndctl +10 -4
index 396a344..cacee2d 100755
@@ -531,8 +531,14 @@ _ndctl()
 
 __daxctl_get_devs()
 {
-	local opts="--devices $*"
-	echo "$(daxctl list $opts | grep -E "^\s*\"chardev\":" | cut -d\" -f4)"
+	local opts=("--devices" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"chardev\":" | cut -d'"' -f4
+}
+
+__daxctl_get_regions()
+{
+	local opts=("--regions" "$*")
+	daxctl list "${opts[@]}" | grep -E "^\s*\"id\":" | grep -Eo "[0-9]+"
 }
 
 __daxctlcomp()
@@ -561,10 +567,10 @@ __daxctl_comp_options()
 		local cur_arg=${cur##*=}
 		case $cur_subopt in
 		--region)
-			opts=$(__ndctl_get_regions -i)
+			opts="$(__daxctl_get_regions -i)"
 			;;
 		--dev)
-			opts=$(__daxctl_get_devs -i)
+			opts="$(__daxctl_get_devs -i)"
 			;;
 		*)
 			return
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#13 lore raw

PATCH v9 12/13 contrib/ndctl: add bash-completion for the new daxctl commands

In reply to Vishal Verma

Add bash completion helpers for the new daxctl-reconfigure-device, daxctl-online-memory, and daxctl-offline-memory commands.

---
 contrib/ndctl | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
contrib/ndctl +21 -3
index cacee2d..680fe6a 100755
@@ -547,7 +547,7 @@ __daxctlcomp()
 
 	COMPREPLY=( $( compgen -W "$1" -- "$2" ) )
 	for cword in "${COMPREPLY[@]}"; do
-		if [[ "$cword" == @(--region|--dev) ]]; then
+		if [[ "$cword" == @(--region|--dev|--mode) ]]; then
 			COMPREPLY[$i]="${cword}="
 		else
 			COMPREPLY[$i]="${cword} "
@@ -572,6 +572,9 @@ __daxctl_comp_options()
 		--dev)
 			opts="$(__daxctl_get_devs -i)"
 			;;
+		--mode)
+			opts="system-ram devdax"
+			;;
 		*)
 			return
 			;;
@@ -582,8 +585,23 @@ __daxctl_comp_options()
 
 __daxctl_comp_non_option_args()
 {
-	# there aren't any commands that accept non option arguments yet
-	return
+	local subcmd=$1
+	local cur=$2
+	local opts
+
+	case $subcmd in
+	reconfigure-device)
+		;&
+	online-memory)
+		;&
+	offline-memory)
+		opts="$(__daxctl_get_devs -i) all"
+		;;
+	*)
+		return
+		;;
+	esac
+	__daxctlcomp "$opts" "$cur"
 }
 
 __daxctl_main()
Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
author maintainer
#14 lore raw

PATCH v9 13/13 test: Add a unit test for daxctl-reconfigure-device and friends

In reply to Vishal Verma

Add a new unit test to test dax device reconfiguration and memory operations. This teaches test/common about daxctl, and adds an ACPI.NFIT bus variable. Since we have to operate on the ACPI.NFIT bus, the test is marked as destructive.

---
 test/Makefile.am       |  3 +-
 test/common            | 19 ++++++++--
 test/daxctl-devices.sh | 81 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 99 insertions(+), 4 deletions(-)
 create mode 100755 test/daxctl-devices.sh
test/Makefile.am +2 -1
index 874c4bb..84474d0 100644
@@ -49,7 +49,8 @@ TESTS +=\
 	dax.sh \
 	device-dax \
 	device-dax-fio.sh \
-	mmap.sh
+	mmap.sh \
+	daxctl-devices.sh
 
 if ENABLE_KEYUTILS
 TESTS += security.sh
test/common +16 -3
index 1b9d3da..1814a0c 100644
@@ -15,12 +15,25 @@ else
 	exit 1
 fi
 
-# NFIT_TEST_BUS[01]
+# DAXCTL
 #
-NFIT_TEST_BUS0=nfit_test.0
-NFIT_TEST_BUS1=nfit_test.1
+if [ -f "../daxctl/daxctl" ] && [ -x "../daxctl/daxctl" ]; then
+	export DAXCTL=../daxctl/daxctl
+elif [ -f "./daxctl/daxctl" ] && [ -x "./daxctl/daxctl" ]; then
+	export DAXCTL=./daxctl/daxctl
+else
+	echo "Couldn't find an daxctl binary"
+	exit 1
+fi
 
 
+# NFIT_TEST_BUS[01]
+#
+NFIT_TEST_BUS0="nfit_test.0"
+NFIT_TEST_BUS1="nfit_test.1"
+ACPI_BUS="ACPI.NFIT"
+E820_BUS="e820"
+
 # Functions
 
 # err
test/daxctl-devices.sh new file +81 -0

81 lines not on this page · raw message

Signature · 6 lines
2.20.1

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm