ndctl: monitor: monitor the smart events of
v3 cover ndctl: monitor: monitor the smart events of
This is a patch set of ndctl monitor, a tiny daemon to monitor the smart events of nvdimm dimms. When a smart event fires, monitor will output the notification which including dimm health status to syslog or a special file according to users' configuration. The output notification follows json format and can be consumed by log collectors like Fluentd.
Currently, I implemeted the following four commands to control monitor daemon. $ndctl create-monitor
$ndctl list-monitor
$ndctl show-monitor
$ndclt destroy-monitor
I will appreciate if you could give some comments.
Change log since v2:
- Changing the interface of daemon to the ndctl command line
- Changing the name of daemon form "nvdimmd" to "monitor"
- Removing the config file, unit_file, nvdimmd dir
- Removing nvdimmd_test program
- Adding ndctl/monitor.c
Change log since v1:
- Adding a config file(/etc/nvdimmd/nvdimmd.conf)
- Using struct log_ctx instead of syslog()
- Using log_syslog() to save the notify messages to syslog
- Using log_file() to save the notify messages to special file
- Adding LOG_NOTICE level to log_priority
- Using automake instead of Makefile
- Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon
- Adding nvdimmd_test program
QI Fuli (5):
ndctl: monitor: add LOG_NOTICE level to log_priority
ndctl: monitor: add ndclt create-monitor command
ndctl: monitor: add ndclt list-monitor command
ndctl: monitor: add ndclt show-monitor command
ndctl: monitor: add ndclt destroy-monitor command
builtin.h | 4 + configure.ac | 3 + ndctl/Makefile.am | 3 +- ndctl/monitor.c | 463 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 4 + util/log.c | 2 + util/log.h | 3 + 7 files changed, 481 insertions(+), 1 deletion(-) create mode 100644 ndctl/monitor.c
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
RFC PATCH v3 1/5 ndctl: nvdimmd: add LOG_NOTICE level to
This patch adds LOG_NOTICE level to log_priority for the notifications of nvdimm dimm over threshold event. Currently, the LOG_INFO level is too low, and the notifications are not up to LOG_ERROR level.
--- util/log.c | 2 ++ util/log.h | 3 +++ 2 files changed, 5 insertions(+)
util/log.c +2 -0
@@ -50,6 +50,8 @@ static int log_priority(const char *priority)return LOG_INFO ;if (strncmp (priority,"debug" ,5 ) ==0 )return LOG_DEBUG ; +if (strncmp (priority,"notice" ,6 ) ==0 ) +return LOG_NOTICE ;return 0 ; }
util/log.h +3 -0
@@ -48,15 +48,18 @@ do { \# endif # define log_info(ctx, arg...) log_cond(ctx, LOG_INFO, ## arg) # define log_err(ctx, arg...) log_cond(ctx, LOG_ERR, ## arg) +# define log_notice(ctx, arg...) log_cond(ctx, LOG_NOTICE, ## arg)#else # define log_dbg(ctx, arg...) log_null(ctx, ## arg) # define log_info(ctx, arg...) log_null(ctx, ## arg) # define log_err(ctx, arg...) log_null(ctx, ## arg) +# define log_notice(ctx, arg...) log_null(ctx, ## arg)#endif #define dbg(x, arg...) log_dbg(&(x)->ctx, ## arg) #define info(x, arg...) log_info(&(x)->ctx, ## arg) #define err(x, arg...) log_err(&(x)->ctx, ## arg) +#define notice(x, arg...) log_notice(&(x)->ctx, ## arg)#ifndef HAVE_SECURE_GETENV # ifdef HAVE___SECURE_GETENV
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 1/5] ndctl: nvdimmd: add LOG_NOTICE level to
Quoted 4 lines · QI Fuli
This patch adds
LOG_NOTICElevel tolog_priorityfor the notifications of nvdimm dimm over threshold event. Currently, theLOG_INFOlevel is too low, and the notifications are not up toLOG_ERRORlevel.
Looks good, applied.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 1/5] ndctl: nvdimmd: add LOG_NOTICE level to
Quoted 7 lines · Dan Williams
Quoted 4 lines · QI Fuli
This patch adds
LOG_NOTICElevel tolog_priorityfor the notifications of nvdimm dimm over threshold event. Currently, theLOG_INFOlevel is too low, and the notifications are not up toLOG_ERRORlevel.Looks good, applied.
Although, this patch has little to do with nvdimmd, so I'm changing the subject to:
"ndctl, log: add support for LOG_NOTICE messages"
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
RFC PATCH v3 2/5 ndctl: monitor: add ndctl create-monitor command
In reply to QI FuliThis patch is used to add $ndctl create-monitor command, by which users can create a new monitor. Users can select the DIMMS to be monitored by using [--dimm] [--bus] [--region] [--namespace] options. The notifications can be outputed to a special file or syslog by using [--output] option, the special file will be placed under /var/log/ndctl. A name is also required for a monitor,so users can destroy the monitor by the name. When a monitor is created successfully, a file with same name will be created under /var/ndctl/monitor.
Example:
#ndctl create-monitor --monitor m_nmem1 --dimm nmem1 --output m_nmem.log
--- builtin.h | 1 + configure.ac | 3 + ndctl/Makefile.am | 3 +- ndctl/monitor.c | 342 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 1 + 5 files changed, 349 insertions(+), 1 deletion(-) create mode 100644 ndctl/monitor.c
builtin.h +1 -0
@@ -36,6 +36,7 @@ int cmd_write_labels(int argc, const char **argv, void *ctx);int cmd_init_labels (int argc,const char **argv,void *ctx);int cmd_check_labels (int argc,const char **argv,void *ctx);int cmd_inject_error (int argc,const char **argv,void *ctx); +int cmd_create_monitor (int argc,const char **argv,void *ctx);int cmd_list (int argc,const char **argv,void *ctx);#ifdef ENABLE_TEST int cmd_test (int argc,const char **argv,void *ctx);
configure.ac +3 -0
@@ -160,6 +160,9 @@ AC_CONFIG_FILES([ Documentation/daxctl/Makefile ]) +AC_CONFIG_COMMANDS([monitorlogdir], [$MKDIR_P /var/log/ndctl]) +AC_CONFIG_COMMANDS([monitorprocdir], [$MKDIR_P /var/ndctl/monitor]) + AC_OUTPUT AC_MSG_RESULT([ $PACKAGE $VERSION
ndctl/Makefile.am +2 -1
@@ -13,7 +13,8 @@ ndctl_SOURCES = ndctl.c \ test.c \ ../util/json.c \ util/json-smart.c \ - inject-error.c + inject-error.c \ + monitor.c if ENABLE_DESTRUCTIVE ndctl_SOURCES += ../test/blk_namespaces.c \
ndctl/monitor.c new file +342 -0
342 lines not on this page · raw message
ndctl/ndctl.c +1 -0
@@ -84,6 +84,7 @@ static struct cmd_struct commands[] = { { "init-labels", cmd_init_labels }, { "check-labels", cmd_check_labels }, { "inject-error", cmd_inject_error }, + { "create-monitor", cmd_create_monitor }, { "list", cmd_list }, { "help", cmd_help }, #ifdef ENABLE_TEST
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 10 lines · QI Fuli
This patch is used to add $ndctl create-monitor command, by which users can create a new monitor. Users can select the DIMMS to be monitored by using [--dimm] [--bus] [--region] [--namespace] options. The notifications can be outputed to a special file or syslog by using [--output] option, the special file will be placed under
/var/log/ndctl. A name is also required for a monitor,so users can destroy the monitor by the name. When a monitor is created successfully, a file with same name will be created under/var/ndctl/monitor.
Example:
#ndctl create-monitor --monitorm_nmem1--dimm nmem1 --output m_nmem.log
Hi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.
The event list I have collected to date is:
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdown
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected
...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme can be handled with the bash tab-completion implementation:
ndctl monitor --dimm-events=spares-remaining,media-temperature
--namespace-events=all --regions-events --bus=ACPI.NFIT
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than duplicated as is done in this patch. In other words rework cmd_list() into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.
Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerRe: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 45 lines · Dan Williams
Quoted 9 lines · QI Fuli
This patch is used to add $ndctl create-monitor command, by which users can
create a new monitor. Users can select the DIMMS to be monitored by using
[--dimm] [--bus] [--region] [--namespace] options. The notifications can
be outputed to a special file or syslog by using [--output] option, the
special file will be placed under/var/log/ndctl. A name is alsoQuoted 8 lines
required for
a monitor,so users can destroy the monitor by the name. When a monitor is
created successfully, a file with same name will be created under/var/ndctl/monitor.
Example:
#ndctl create-monitor --monitorm_nmem1--dimm nmem1 --output m_nmem.logHi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the
monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.The event list I have collected to date is:
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdown
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected
Quoted 7 lines
...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme
can be handled with the bash tab-completion implementation:ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
Yes I think we should be able to extend bash completion for a comma separated list of arguments.
Quoted 13 lines
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than
duplicated as is done in this patch. In other words reworkcmd_list()into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.
Quoted 2 lines
Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 12 lines
Quoted 12 lines
Hi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the
monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.The event list I have collected to date is:
Quoted 13 lines
Quoted 5 lines
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdownQuoted 8 lines
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected
By referring to dimm_listen() in smart-listen.c, I understand how to monitor events come DIMMs. I have checked the ndctl, but I could not find how to monitor events come namespaces, region and bus. Could you please mention more?
Quoted 5 lines
Quoted 5 lines
...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme
can be handled with the bash tab-completion implementation:
Quoted 19 lines
Quoted 2 lines
ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
Yes I think we should be able to extend bash completion for a comma separated list of arguments.
Quoted 14 lines
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than
duplicated as is done in this patch. In other words reworkcmd_list()into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
Yes, I agree with refactoring ndctl_list and add a structure shared between ndctl list and ndctl monitor.
But I prefer to use tokens rather than callback functions.
I also think the option should support multiple parameters likes:
ndclt monitor --dimm nmem1,nmem2 --region region1,region2
The parameters from the same option can be stored in a list_node.
#define filter_node(field) \
struct filter_##field##_node { \
const char *name; \
struct list_node list; \
};
filter_node(bus);filter_node(dimm);filter_node(region);filter_node(namespace);
struct ndctl_filter {
struct list_head filter_bus;
struct list_head filter_dimm;
struct list_head filter_region;
struct list_head filter_namespace;
} nf;
Make the tokens like follows:
#define list_util_field_filter(field) \list_for_each(&nf.filter_##field, filter_##field##_node, list) { \
if (util_##field##_filter(field, filter_##field##_node->name)) { \
flag = true; \
break; \
} \
}
#define list_util_bus_filter_by_field(field) \list_for_each(&nf.filter_##field, filter_##field##_node, list) { \
if (util_bus_filter_by_##field(bus, filter_##field##_node->name)) { \
flag = true; \
break; \
} \
}
#define list_util_dimm_filter_by_field(field) \list_for_each(&nf.filter_##field, filter_##field##_node, list) { \
if (util_dimm_filter_by_##field(dimm, filter_##field##_node->name)) { \
flag = true; \
break; \
} \
}
Here is how to use the tokens:
ndctl_bus_foreach(ctx, bus) {
struct filter_bus_node *filter_bus_node; list_util_field_filter(bus);
if (!flag) {
struct filter_region_node *filter_region_node; list_util_bus_filter_by_field(region);
}
if (!flag) {
struct filter_namespace_node *filter_namespace_node; list_util_bus_filter_by_field(namespace);
}
if (!flag)
continue;
ndctl_dimm_foreach(bus, dimm) {
flag = false;
struct filter_dimm_node *filter_dimm_node; list_util_field_filter(dimm);
if (!flag) {
struct filter_region_node *filter_region_node; list_util_dimm_filter_by_field(region); }
if (!flag) {
struct filter_namespace_node *filter_namespace_node; list_util_dimm_filter_by_field(namespace); }
if (!flag)
continue;
}
}
To make sure the util_ndctl_filter() goes right, I would like you give some comments?
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 145 lines · Fuli
Quoted 12 lines
Quoted 12 lines
Hi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the
monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.The event list I have collected to date is:
Quoted 13 lines
Quoted 5 lines
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdownQuoted 8 lines
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detectedBy referring to
dimm_listen()insmart-listen.c, I understand how to monitor events
come DIMMs. I have checked the ndctl, but I could not find how to monitor events
come namespaces, region and bus. Could you please mention more?Quoted 6 lines
Quoted 6 lines
...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme
can be handled with the bash tab-completion implementation:Quoted 20 lines
Quoted 2 lines
ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
Yes I think we should be able to extend bash completion for a comma separated list of arguments.
Quoted 14 lines
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than
duplicated as is done in this patch. In other words reworkcmd_list()into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
Yes, I agree with refactoring
ndctl_listand add a structure shared between ndctl list and ndctl monitor.
But I prefer to use tokens rather than callback functions.I also think the option should support multiple parameters likes:
ndclt monitor --dimm nmem1,nmem2 --region region1,region2The parameters from the same option can be stored in a
list_node.#define
filter_node(field)\struct filter_##field##_node { \ const char *name; \ struct list_node list; \ };
filter_node(bus);filter_node(dimm);filter_node(region);filter_node(namespace);struct ndctl_filter { struct list_head filter_bus; struct list_head filter_dimm; struct list_head filter_region; struct list_head filter_namespace; } nf;Make the tokens like follows:
#define
list_util_field_filter(field)\list_for_each(&nf.filter_##field, filter_##field##_node, list) { \ if (util_##field##_filter(field, filter_##field##_node->name)) { \ flag = true; \ break; \ } \ }#define
list_util_bus_filter_by_field(field)\list_for_each(&nf.filter_##field, filter_##field##_node, list) { \ if (util_bus_filter_by_##field(bus, filter_##field##_node->name)) { \ flag = true; \ break; \ } \ }#define
list_util_dimm_filter_by_field(field)\list_for_each(&nf.filter_##field, filter_##field##_node, list) { \ if (util_dimm_filter_by_##field(dimm, filter_##field##_node->name)) { \ flag = true; \ break; \ } \ }Here is how to use the tokens:
ndctl_bus_foreach(ctx, bus) { struct filter_bus_node *filter_bus_node; list_util_field_filter(bus); if (!flag) { struct filter_region_node *filter_region_node; list_util_bus_filter_by_field(region); } if (!flag) { struct filter_namespace_node *filter_namespace_node; list_util_bus_filter_by_field(namespace); } if (!flag) continue; ndctl_dimm_foreach(bus, dimm) { flag = false; struct filter_dimm_node *filter_dimm_node; list_util_field_filter(dimm); if (!flag) { struct filter_region_node *filter_region_node; list_util_dimm_filter_by_field(region); } if (!flag) { struct filter_namespace_node *filter_namespace_node; list_util_dimm_filter_by_field(namespace); } if (!flag) continue; } }
Quoted 2 lines
To make sure the
util_ndctl_filter()goes right, I would like you give some comments?
I think the multi-layer macros makes the code difficult to read, and I'm not seeing how it handles the case where we're building up a combined json record representing the entire bus, i.e. "ndctl list -BRND". I'd rather re-write "ndctl list" in terms of this filter structure:
struct ndctl_filter_ops {
int (*match_bus)(struct ndctl_bus *bus, void **ctx, unsigned
long flags);
int (*match_dimm)(struct ndctl_bus *dimm, void *ctx, unsigned
long flags);
int (*match_region)(struct ndctl_bus *region, void **ctx,
unsigned long flags);
int (*match_namespace)(struct ndctl_bus *namespace, void *ctx,
unsigned long flags);
};
...where the match routine is only populated if that device is selected and it passes filter specific context via the "ctx" variable. In the case of "ndctl list" that might be the "jbus" container json object being returned from ->match_bus() and then passed to ->match_dimm() to hold "jdimm" instances. Note that ->match_dimm() and ->match_namespace() only take context as input because those objects don't have any children.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
In reply to Dan WilliamsHi,
Quoted 40 lines · QI Fuli
Quoted 10 lines
This patch is used to add $ndctl create-monitor command, by which users can create a new monitor. Users can select the DIMMS to be monitored by using [--dimm] [--bus] [--region] [--namespace] options. The notifications can be outputed to a special file or syslog by using [--output] option, the special file will be placed under
/var/log/ndctl. A name is also required for a monitor,so users can destroy the monitor by the name. When a monitor is created successfully, a file with same name will be created under/var/ndctl/monitor.
Example:
#ndctl create-monitor --monitorm_nmem1--dimm nmem1 --output m_nmem.logHi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.
The event list I have collected to date is:
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdown
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme can be handled with the bash tab-completion implementation:
Quoted 7 lines
ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Hmmmm....
Currently, I'm confusing what features/options are required for nvdimm daemon. For example, what is use-case of "--bus=ACPI.NFIT"?
For normal administorator of a server, what he/she's interest is "need to replace nvdimm module or not", and "need to backup/restore on the nvdimm module or not".
For normal programs, they just use device name or directory/filename of the filesystem on the nvdimm.
To backup thier data, he/she need to solve relationship between nvdimm modules and device name (/dev/pmem* or /dev/dax*).
So, IMHO, I suppose "namespace(device name) specifying (or all namespace)" is enough the following events which requires replace the nvdimm module.
- spare-remaining
- helth-state
- media-error
And I'm not sure what is use-case of specifying region, bus, and dimm on these events.
In addition, could you tell me what administrator/program can do on the following events? What nvdimm daemon should do on each event?
- media-temperature
- controller-temperature
- address-range-scrub-complete
- unclean-shutdown
Thanks,
Quoted 7 lines
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than duplicated as is done in this patch. In other words rework
cmd_list()into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.
Quoted 6 lines
Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 76 lines
Hi,
Quoted 40 lines · QI Fuli
Quoted 10 lines
This patch is used to add $ndctl create-monitor command, by which users can create a new monitor. Users can select the DIMMS to be monitored by using [--dimm] [--bus] [--region] [--namespace] options. The notifications can be outputed to a special file or syslog by using [--output] option, the special file will be placed under
/var/log/ndctl. A name is also required for a monitor,so users can destroy the monitor by the name. When a monitor is created successfully, a file with same name will be created under/var/ndctl/monitor.
Example:
#ndctl create-monitor --monitorm_nmem1--dimm nmem1 --output m_nmem.logHi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.
The event list I have collected to date is:
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdown
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme can be handled with the bash tab-completion implementation:
Quoted 7 lines
ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Hmmmm....
Currently, I'm confusing what features/options are required for nvdimm daemon. For example, what is use-case of "--bus=ACPI.NFIT"?
For normal administorator of a server, what he/she's interest is "need to replace nvdimm module or not", and "need to backup/restore on the nvdimm module or not".
For normal programs, they just use device name or directory/filename of the filesystem on the nvdimm.
To backup thier data, he/she need to solve relationship between nvdimm modules and device name (
/dev/pmem*or/dev/dax*).So, IMHO, I suppose "namespace(device name) specifying (or all namespace)" is enough the following events which requires replace the nvdimm module.
- spare-remaining
- helth-state
- media-errorAnd I'm not sure what is use-case of specifying region, bus, and dimm on these events.
Quoted 8 lines
In addition, could you tell me what administrator/program can do on the following events? What nvdimm daemon should do on each event?
- media-temperature
- controller-temperature
- address-range-scrub-complete
- unclean-shutdown
I would like to ask one more thing.
What should nvdimm daemon do on detected events of bus/region/namespace/dimm ? IIRC, udev handles such hotplug events.
What is relationship/roles between nvdimm daemon and udev?
Bye,
Quoted 17 lines
Thanks,
Quoted 13 lines
Since "ndctl list" already has this filtering implemented I'd like to see it refactored and shared between the 2 implementations rather than duplicated as is done in this patch. In other words rework
cmd_list()into a generic nvdimm object walking routine with callback functions to 'list' or 'monitor' a given object that matches the filter.Let me know if the above makes sense. I'm thinking the 'ndctl list' refactoring might be something I need to handle.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 1 line · Yasunori Goto
Quoted 1 line
Hi,
[..]
Quoted 5 lines
I would like to ask one more thing.
What should nvdimm daemon do on detected events of bus/region/namespace/dimm ? IIRC, udev handles such hotplug events.
What is relationship/roles between nvdimm daemon and udev?
UDEV handles the hotplug events, but I imagine device topology changes are of interest to whatever agent is going to be consuming monitor events, or monitoring changes that an administrator makes to a machine. My inspiration for including these events in the monitor comes from the monitor functionality in mdadm. It has events like:
DeviceDisappeared
RebuildStarted
RebuildNN
RebuildFinished
Fail
FailSpare
SpareActive
NewArray
DegradedArray
MoveSpare
SparesMissing
To be clear, I'm not suggesting that all of these events need to be supported in the first implementation of the monitor, but I want the architecture and the user interface to be ready to support events from any device in the topology.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
Quoted 27 lines · Yasunori Goto
Quoted 1 line
Quoted 1 line
Hi,
[..]
Quoted 5 lines
I would like to ask one more thing.
What should nvdimm daemon do on detected events of bus/region/namespace/dimm ? IIRC, udev handles such hotplug events.
What is relationship/roles between nvdimm daemon and udev?UDEV handles the hotplug events, but I imagine device topology changes are of interest to whatever agent is going to be consuming monitor events, or monitoring changes that an administrator makes to a machine. My inspiration for including these events in the monitor comes from the monitor functionality in mdadm. It has events like:
DeviceDisappeared
RebuildStarted
RebuildNN
RebuildFinished
Fail
FailSpare
SpareActive
NewArray
DegradedArray
MoveSpare
SparesMissing
Quoted 5 lines
To be clear, I'm not suggesting that all of these events need to be supported in the first implementation of the monitor, but I want the architecture and the user interface to be ready to support events from any device in the topology.
Ok, I got it.
Thank you for your clarification.
Bye,
---
Yasunori Goto
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v3 2/5] ndctl: monitor: add ndctl create-monitor command
In reply to Yasunori GotoQuoted 50 lines · Yasunori Goto
Hi,
Quoted 40 lines · QI Fuli
Quoted 10 lines
This patch is used to add $ndctl create-monitor command, by which users can create a new monitor. Users can select the DIMMS to be monitored by using [--dimm] [--bus] [--region] [--namespace] options. The notifications can be outputed to a special file or syslog by using [--output] option, the special file will be placed under
/var/log/ndctl. A name is also required for a monitor,so users can destroy the monitor by the name. When a monitor is created successfully, a file with same name will be created under/var/ndctl/monitor.
Example:
#ndctl create-monitor --monitorm_nmem1--dimm nmem1 --output m_nmem.logHi Qi,
This is getting closer to where I want to see this go, but still some architecture details to incorporate. I mentioned on the cover letter that systemd can handle starting, stopping, and show the status of the monitor. The other detail to incorporate is that monitor events can come DIMMs, but also namespaces, regions, and the bus.
The event list I have collected to date is:
dimm-spares-remaining
dimm-media-temperature
dimm-controller-temperature
dimm-health-state
dimm-unclean-shutdown
dimm-detected
namespace-media-error
namespace-detected
region-media-error
region-detected
bus-media-error
bus-address-range-scrub-complete
bus-detected...and I think all of those should be separate options, probably something like the following, but I'd Vishal to comment if this scheme can be handled with the bash tab-completion implementation:
Quoted 7 lines
ndctl monitor --dimm-events=spares-remaining,media-temperature--namespace-events=all --regions-events --bus=ACPI.NFIT
...where an empty --<object>-events option is equivalent to --<object>-events=all. Also, similar to "ndctl list" specifying specific buses, namespaces, etc causes the monitor to filter the objects based on those properties.
Quoted 4 lines
Hmmmm....
Currently, I'm confusing what features/options are required for nvdimm daemon. For example, what is use-case of "--bus=ACPI.NFIT"?
Other platforms may support different bus types, there are also proposals like this one for custom NVDIMM buses [1]. The other use case is allowing the user to monitor for any media error on the bus, or the completion of ARS.
Quoted 3 lines
For normal administorator of a server, what he/she's interest is "need to replace nvdimm module or not", and "need to backup/restore on the nvdimm module or not".
I think that's only part of it. Data center operations want to know more than just when it is time to replace a module, they want to collect almost any data that the operating system can provide about the platform.
Quoted 9 lines
For normal programs, they just use device name or directory/filename of the filesystem on the nvdimm.
To backup thier data, he/she need to solve relationship between nvdimm modules and device name (
/dev/pmem*or/dev/dax*).So, IMHO, I suppose "namespace(device name) specifying (or all namespace)" is enough the following events which requires replace the nvdimm module.
Quoted 6 lines
- spare-remaining
- helth-state
- media-errorAnd I'm not sure what is use-case of specifying region, bus, and dimm on these events.
The reason for supporting those other event sources in my mind is having a unified interface for tracking topology, health/status, and error events.
Quoted 8 lines
In addition, could you tell me what administrator/program can do on the following events? What nvdimm daemon should do on each event?
- media-temperature
- controller-temperature
- address-range-scrub-complete
- unclean-shutdown
Media temperature and controller temperature alarms can signal to data center operations that the server is getting too hot, and might need remediation, perhaps a specific fan has failed and replacing that fan becomes a high priority when these alarms start firing.
Address-range-scrub complete might be a signal that the server may get a boost in throughput since the overhead of the background operation is now complete. ARS may continue to run long after the server has booted, so the end of that event may be important to server loading decisions.
Unclean shutdown notification allows events that occurred at the last shutdown to be recorded at the next boot. Otherwise an operator would need to write a separate tool to go retrieve this information. Having it all in one place reduces the number of tools / data-sources that operations infrastructure needs to consider.
[1]: https://lists.01.org/pipermail/linux-nvdimm/2018-January/013926.html
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
RFC PATCH v3 3/5 ndctl: monitor: add ndclt list-monitor command
In reply to QI FuliThis patch adds $ndctl list-monitor command, by which users can list all currently running monitors.
Example: $ndctl list-monitor --all
--- builtin.h | 1 + ndctl/monitor.c | 32 ++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 1 + 3 files changed, 34 insertions(+)
builtin.h +1 -0
@@ -37,6 +37,7 @@ int cmd_init_labels(int argc, const char **argv, void *ctx);int cmd_check_labels (int argc,const char **argv,void *ctx);int cmd_inject_error (int argc,const char **argv,void *ctx);int cmd_create_monitor (int argc,const char **argv,void *ctx); +int cmd_list_monitor (int argc,const char **argv,void *ctx);int cmd_list (int argc,const char **argv,void *ctx);#ifdef ENABLE_TEST int cmd_test (int argc,const char **argv,void *ctx);
ndctl/monitor.c +32 -0
@@ -340,3 +340,35 @@ int cmd_create_monitor(int argc, const char **argv, void *ctx) out:return 1 ; } + +int cmd_list_monitor (int argc,const char **argv,void *ctx) +{ +const struct option options[] = { +OPT_BOOLEAN ('a' ,"all" , ¶m.all,"list all monitors" ) + }; +const char *const u[] = { +"ndctl list-monitor [<options>]" , +NULL + }; + argc =parse_options (argc, argv, options, u,0 ); +for (int i =0 ; i < argc; i++) { +error ("unknown parameter \"%s\"\n" , argv[i]); + goto out; + } +DIR *dir; +struct dirent *ent; + dir =opendir ("/var/ndctl/monitor/" ); +if (!dir) +return -1 ; +while ((ent =readdir (dir)) !=NULL ) + { +if (strcmp (ent->d_name,"." ) ==0 + ||strcmp (ent->d_name,".." ) ==0 ) +continue ; +printf (" %s\n" , ent->d_name); + } +return 0 ; + +out: +return 1 ; +}
ndctl/ndctl.c +1 -0
@@ -85,6 +85,7 @@ static struct cmd_struct commands[] = { { "check-labels", cmd_check_labels }, { "inject-error", cmd_inject_error }, { "create-monitor", cmd_create_monitor }, + { "list-monitor", cmd_list_monitor }, { "list", cmd_list }, { "help", cmd_help }, #ifdef ENABLE_TEST
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
RFC PATCH v3 4/5 ndctl: monitor: add ndclt show-monitor command
In reply to QI FuliThis patch is used to add $ndctl show-monitor command, by which users can check the specific of a monitor.
Example: $ndctl show-monitor --monitor m_nmem1
--- builtin.h | 1 + ndctl/monitor.c | 35 +++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 1 + 3 files changed, 37 insertions(+)
builtin.h +1 -0
@@ -38,6 +38,7 @@ int cmd_check_labels(int argc, const char **argv, void *ctx);int cmd_inject_error (int argc,const char **argv,void *ctx);int cmd_create_monitor (int argc,const char **argv,void *ctx);int cmd_list_monitor (int argc,const char **argv,void *ctx); +int cmd_show_monitor (int argc,const char **argv,void *ctx);int cmd_list (int argc,const char **argv,void *ctx);#ifdef ENABLE_TEST int cmd_test (int argc,const char **argv,void *ctx);
ndctl/monitor.c +35 -0
@@ -372,3 +372,38 @@ int cmd_list_monitor(int argc, const char **argv, void *ctx) out:return 1 ; } + +int cmd_show_monitor (int argc,const char **argv,void *ctx) +{ +const struct option options[] = { +OPT_STRING ('m' ,"monitor" , ¶m.monitor,"monitor name" , +"monitor name" ) + }; +const char *const u[] = { +"ndctl show-monitor <monitor>" , +NULL + }; + argc =parse_options (argc, argv, options, u,0 ); +for (int i =0 ; i < argc; i++) { +error ("unknown parameter \"%s\"\n" , argv[i]); + goto out; + } +if (!param.monitor) { +error ("monitor name --monitor is required\n" ); + goto out; + } + +char *filename; + filename =get_full_path_filename (proc_path, param.monitor); +struct json_object *jmonitors =json_object_from_file (filename); +if (jmonitors ==NULL ) { +error ("monitor %s is not exist\n" , param.monitor); + goto out; + } + +printf ("%s\n" ,json_object_to_json_string_ext (jmonitors, +JSON_C_TO_STRING_PRETTY )); +return 0 ; +out: +return 1 ; +}
ndctl/ndctl.c +1 -0
@@ -86,6 +86,7 @@ static struct cmd_struct commands[] = { { "inject-error", cmd_inject_error }, { "create-monitor", cmd_create_monitor }, { "list-monitor", cmd_list_monitor }, + { "show-monitor", cmd_show_monitor }, { "list", cmd_list }, { "help", cmd_help }, #ifdef ENABLE_TEST
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
RFC PATCH v3 5/5 ndctl: monitor: add ndclt destroy-monitor command
In reply to QI FuliThis patch is used to add $ndctl destroy-monitor command, by which users can destroy a monitor. After the monitor is destroyed, the file with same name under /var/ndctl/monitor will also be deleted.
Example: #ndctl destroy-monitor --monitor m_nmem1
--- builtin.h | 1 + ndctl/monitor.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 1 + 3 files changed, 56 insertions(+)
builtin.h +1 -0
@@ -39,6 +39,7 @@ int cmd_inject_error(int argc, const char **argv, void *ctx);int cmd_create_monitor (int argc,const char **argv,void *ctx);int cmd_list_monitor (int argc,const char **argv,void *ctx);int cmd_show_monitor (int argc,const char **argv,void *ctx); +int cmd_destroy_monitor (int argc,const char **argv,void *ctx);int cmd_list (int argc,const char **argv,void *ctx);#ifdef ENABLE_TEST int cmd_test (int argc,const char **argv,void *ctx);
ndctl/monitor.c +54 -0
57 lines not on this page · raw message
ndctl/ndctl.c +1 -0
@@ -87,6 +87,7 @@ static struct cmd_struct commands[] = { { "create-monitor", cmd_create_monitor }, { "list-monitor", cmd_list_monitor }, { "show-monitor", cmd_show_monitor }, + { "destroy-monitor", cmd_destroy_monitor }, { "list", cmd_list }, { "help", cmd_help }, #ifdef ENABLE_TEST
Signature · 7 lines
2.9.5 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Verma, Vishal L
maintainerQuoted 36 lines · QI Fuli
This is a patch set of ndctl monitor, a tiny daemon to monitor the smart
events of nvdimm dimms. When a smart event fires, monitor will output the notification which including dimm health status to syslog or a special file according to users' configuration. The output notification
follows json format and can be consumed by log collectors like Fluentd.Currently, I implemeted the following four commands to control monitor daemon.
$ndctl create-monitor
$ndctl list-monitor
$ndctl show-monitor
$ndclt destroy-monitorI will appreciate if you could give some comments.
Change log since v2:
- Changing the interface of daemon to the ndctl command line
- Changing the name of daemon form "nvdimmd" to "monitor"
- Removing the config file,unit_file, nvdimmd dir
- Removingnvdimmd_testprogram
- Addingndctl/monitor.cChange log since v1:
- Adding a config file(/etc/nvdimmd/nvdimmd.conf)
- Using structlog_ctxinstead ofsyslog()
- Usinglog_syslog()to save the notify messages to syslog
- Usinglog_file()to save the notify messages to special file
- AddingLOG_NOTICElevel tolog_priority
- Using automake instead ofMakefile
- Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon
- Addingnvdimmd_testprogram
Quoted 6 lines
QI Fuli (5):
ndctl: monitor: addLOG_NOTICElevel tolog_priority
ndctl: monitor: add ndclt create-monitor command
ndctl: monitor: add ndclt list-monitor command
ndctl: monitor: add ndclt show-monitor command
ndctl: monitor: add ndclt destroy-monitor command
^
I haven't had a chance to look at the rest of the series, but spotted a quick typo - this is in the subject line of each commit as well:
s/ndclt/ndctl/
Quoted 12 lines
builtin.h | 4 + configure.ac | 3 + ndctl/Makefile.am | 3 +- ndctl/monitor.c | 463 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 4 + util/log.c | 2 + util/log.h | 3 + 7 files changed, 481 insertions(+), 1 deletion(-) create mode 100644 ndctl/monitor.c
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 43 lines · Vishal L
Quoted 36 lines · QI Fuli
This is a patch set of ndctl monitor, a tiny daemon to monitor the smart
events of nvdimm dimms. When a smart event fires, monitor will output the notification which including dimm health status to syslog or a special file according to users' configuration. The output notification
follows json format and can be consumed by log collectors like Fluentd.Currently, I implemeted the following four commands to control monitor daemon.
$ndctl create-monitor
$ndctl list-monitor
$ndctl show-monitor
$ndclt destroy-monitorI will appreciate if you could give some comments.
Change log since v2:
- Changing the interface of daemon to the ndctl command line
- Changing the name of daemon form "nvdimmd" to "monitor"
- Removing the config file,unit_file, nvdimmd dir
- Removingnvdimmd_testprogram
- Addingndctl/monitor.cChange log since v1:
- Adding a config file(/etc/nvdimmd/nvdimmd.conf)
- Using structlog_ctxinstead ofsyslog()
- Usinglog_syslog()to save the notify messages to syslog
- Usinglog_file()to save the notify messages to special file
- AddingLOG_NOTICElevel tolog_priority
- Using automake instead ofMakefile
- Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon
- Addingnvdimmd_testprogramQuoted 6 lines
QI Fuli (5):
ndctl: monitor: addLOG_NOTICElevel tolog_priority
ndctl: monitor: add ndclt create-monitor command
ndctl: monitor: add ndclt list-monitor command
ndctl: monitor: add ndclt show-monitor command
ndctl: monitor: add ndclt destroy-monitor command
Quoted 4 lines
^I haven't had a chance to look at the rest of the series, but spotted a quick typo - this is in the subject line of each commit as well:
s/ndclt/ndctl/
Thank you for your comment, I will be more careful next time.
Quoted 12 lines
Quoted 10 lines
builtin.h | 4 + configure.ac | 3 + ndctl/Makefile.am | 3 +- ndctl/monitor.c | 463 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ ndctl/ndctl.c | 4 + util/log.c | 2 + util/log.h | 3 + 7 files changed, 481 insertions(+), 1 deletion(-)Quoted 2 lines
create mode 100644
ndctl/monitor.c
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 6 lines · QI Fuli
This is a patch set of ndctl monitor, a tiny daemon to monitor the smart events of nvdimm dimms. When a smart event fires, monitor will output the notification which including dimm health status to syslog or a special file according to users' configuration. The output notification follows json format and can be consumed by log collectors like Fluentd.
Quoted 7 lines
Currently, I implemeted the following four commands to control monitor daemon. $ndctl create-monitor
$ndctl list-monitor
$ndctl show-monitor
$ndclt destroy-monitorI will appreciate if you could give some comments.
Nice to see this new update.
I have a high level comment on the proposed command structure, as far as I can see the only new command we need is:
ndctl monitor
...that launches a monitor. We can have options to control whether that monitor is a one-shot check of events sources, or a daemon that forks into the background, but the 'create', 'list', 'show', and 'destroy' are the responsibility of systemd or System V services. For example:
create: systemctl start ndctl-monitor
show: systemctl status ndctl-monitor
destroy: systemctl stop ndctl-monitor
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 27 lines · Dan Williams
Quoted 6 lines · QI Fuli
This is a patch set of ndctl monitor, a tiny daemon to monitor the smart events of nvdimm dimms. When a smart event fires, monitor will output the notification which including dimm health status to syslog or a special file according to users' configuration. The output notification follows json format and can be consumed by log collectors like Fluentd.
Quoted 7 lines
Currently, I implemeted the following four commands to control monitor daemon. $ndctl create-monitor
$ndctl list-monitor
$ndctl show-monitor
$ndclt destroy-monitorI will appreciate if you could give some comments.
Nice to see this new update.
I have a high level comment on the proposed command structure, as far as I can see the only new command we need is:
ndctl monitor
...that launches a monitor. We can have options to control whether that monitor is a one-shot check of events sources, or a daemon that forks into the background, but the 'create', 'list', 'show', and 'destroy' are the responsibility of systemd or System V services. For example:
Quoted 5 lines
create: systemctl start ndctl-monitor
show: systemctl status ndctl-monitor
destroy: systemctl stop ndctl-monitor
Thank you for your comments, I will use systemd in next version.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm