powerpc/papr_scm: Add support for reporting nvdimm health
Subsystems (2)
v3 cover powerpc/papr_scm: Add support for reporting nvdimm health
The PAPR standard[1][3] provides mechanisms to query the health and performance stats of an NVDIMM via various hcalls as described in Ref[2]. Until now these stats were never available nor exposed to the user-space tools like 'ndctl'. This is partly due to PAPR platform not having support for ACPI and NFIT. Hence 'ndctl' is unable to query and report the dimm health status and a user had no way to determine the current health status of a NDVIMM.
To overcome this limitation, this patch-set updates papr_scm kernel module to query and fetch nvdimm health stats using hcalls described in Ref[2]. This health and performance stats are then exposed to userspace via syfs and Dimm-Specific-Methods(DSM) issued by libndctl.
These changes coupled with proposed ndtcl changes located at Ref[4] should provide a way for the user to retrieve NVDIMM health status using ndtcl.
Below is a sample output using proposed kernel + ndctl for PAPR NVDIMM in a emulation environment:
# ndctl list -DH
[
{
"dev":"nmem0",
"health":{
"health_state":"fatal",
"shutdown_state":"dirty"
}
}
]Dimm health report output on a pseries guest lpar with vPMEM or HMS based nvdimms that are in perfectly healthy conditions:
# ndctl list -d nmem0 -H
[
{
"dev":"nmem0",
"health":{
"health_state":"ok",
"shutdown_state":"clean"
}
}
]PAPR Dimm-Specific-Methods(DSM)
================================
As the name suggests DSMs are used by vendor specific code in libndctl to execute certain operations or fetch certain information for NVDIMMS. DSMs can be sent to papr_scm module via libndctl (userspace) and libnvdimm (kernel) using the ND_CMD_CALL ioctl which can be handled in the dimm control function papr_scm_ndctl(). For PAPR this patchset proposes a single DSM to retrieve DIMM health, defined in the newly introduced uapi header named 'papr_scm_dsm.h'. Support for more DSMs will be added in future.
Structure of the patch-set
==========================
The patchset starts with implementing support for fetching nvdimm health information from PHYP and partially exposing it to user-space via nvdimm flags.
Second & Third patches deal with implementing support for servicing DSM commands papr_scm.
Finally the Fourth patch implements support for servicing DSM 'DSM_PAPR_SCM_HEALTH' that returns the nvdimm health information to libndctl.
Change-log
==========
v2..v3:
* Updated the papr_scm_dsm.h header to be more confimant general kernel guidelines for UAPI headers. [Aneesh]
* Changed the definition of macro PAPR_SCM_DIMM_UNARMED_MASK to not include case when the nvdimm is unarmed because its a vPMEM nvdimm. [Aneesh]
v1..v2:
* Restructured the patch-set based on review comments on V1 patch-set to simplify the patch review. Multiple small patches have been combined into single patches to reduce cross referencing that was needed in earlier patch-set. Hence most of the patches in this patch-set as now new. [Aneesh]
* Removed the initial work done for fetch nvdimm performance statistics. These changes will be re-proposed in a separate patch-set. [Aneesh]
* Simplified handling of versioning of 'struct nd_papr_scm_dimm_health_stat_v1' as only one version of the structure is currently in existence.
References:
[1]: "Power Architecture Platform Reference"
https://en.wikipedia.org/wiki/Power_Architecture_Platform_Reference
[2]: commit 58b278f568f0
("powerpc: Provide initial documentation for PAPR hcalls")
[3]: "Linux on Power Architecture Platform Reference"
https://members.openpowerfoundation.org/document/dl/469
[4]: https://patchwork.kernel.org/project/linux-nvdimm/list/?series=244625
Vaibhav Jain (4):
powerpc/papr_scm: Fetch nvdimm health information from PHYP ndctl/uapi: Introduce NVDIMM_FAMILY_PAPR_SCM as a new NVDIMM DSM
family
powerpc/papr_scm,uapi: Add support for handling PAPR DSM commands powerpc/papr_scm: Implement support for DSM_PAPR_SCM_HEALTH
arch/powerpc/include/asm/papr_scm.h | 48 ++++ arch/powerpc/include/uapi/asm/papr_scm_dsm.h | 201 ++++++++++++++ arch/powerpc/platforms/pseries/papr_scm.c | 269 ++++++++++++++++++- include/uapi/linux/ndctl.h | 1 + 4 files changed, 510 insertions(+), 9 deletions(-) create mode 100644 arch/powerpc/include/asm/papr_scm.h create mode 100644 arch/powerpc/include/uapi/asm/papr_scm_dsm.h
Signature · 4 lines
2.24.1 _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
PATCH v3 1/4 powerpc/papr_scm: Fetch nvdimm health information from PHYP
Implement support for fetching nvdimm health information via H_SCM_HEALTH hcall as documented in Ref[1]. The hcall returns a pair of 64-bit big-endian integers which are then stored in 'struct papr_scm_priv' and subsequently partially exposed to user-space via newly introduced dimm specific attribute 'papr_flags'. Also a new asm header named 'papr-scm.h' is added that describes the interface between PHYP and guest kernel.
Following flags are reported via 'papr_flags' sysfs attribute contents of which are space separated string flags indicating various nvdimm states:
* "not_armed" : Indicating that nvdimm contents wont survive a power
cycle.
* "save_fail" : Indicating that nvdimm contents couldn't be flushed
during last shutdown event.
* "restore_fail": Indicating that nvdimm contents couldn't be restored during dimm initialization.
* "encrypted" : Dimm contents are encrypted.
* "smart_notify": There is health event for the nvdimm.
* "scrubbed" : Indicating that contents of the nvdimm have been
scrubbed.
* "locked" : Indicating that nvdimm contents cant be modified
until next power cycle.
[1]: commit 58b278f568f0 ("powerpc: Provide initial documentation for PAPR hcalls")
---
Change-log:
v2..v3 : Removed PAPR_SCM_DIMM_HEALTH_NON_CRITICAL as a condition for
NVDIMM unarmed [Aneesh]
v1..v2 : New patch in the series.
--- arch/powerpc/include/asm/papr_scm.h | 48 ++++++++++ arch/powerpc/platforms/pseries/papr_scm.c | 105 +++++++++++++++++++++- 2 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 arch/powerpc/include/asm/papr_scm.h
arch/powerpc/include/asm/papr_scm.h new file +48 -0
@@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Structures and defines needed to manage nvdimms for spapr guests. +*/ +#ifndef _ASM_POWERPC_PAPR_SCM_H_ +#define _ASM_POWERPC_PAPR_SCM_H_ + +#include <linux/types.h> +#include <asm/bitsperlong.h> + +/* DIMM health bitmap bitmap indicators */ +/* SCM device is unable to persist memory contents */ +#define PAPR_SCM_DIMM_UNARMED PPC_BIT(0) +/* SCM device failed to persist memory contents */ +#define PAPR_SCM_DIMM_SHUTDOWN_DIRTY PPC_BIT(1) +/* SCM device contents are persisted from previous IPL */ +#define PAPR_SCM_DIMM_SHUTDOWN_CLEAN PPC_BIT(2) +/* SCM device contents are not persisted from previous IPL */ +#define PAPR_SCM_DIMM_EMPTY PPC_BIT(3) +/* SCM device memory life remaining is critically low */ +#define PAPR_SCM_DIMM_HEALTH_CRITICAL PPC_BIT(4) +/* SCM device will be garded off next IPL due to failure */ +#define PAPR_SCM_DIMM_HEALTH_FATAL PPC_BIT(5) +/* SCM contents cannot persist due to current platform health status */ +#define PAPR_SCM_DIMM_HEALTH_UNHEALTHY PPC_BIT(6) +/* SCM device is unable to persist memory contents in certain conditions */ +#define PAPR_SCM_DIMM_HEALTH_NON_CRITICAL PPC_BIT(7) +/* SCM device is encrypted */ +#define PAPR_SCM_DIMM_ENCRYPTED PPC_BIT(8) +/* SCM device has been scrubbed and locked */ +#define PAPR_SCM_DIMM_SCRUBBED_AND_LOCKED PPC_BIT(9) + +/* Bits status indicators for health bitmap indicating unarmed dimm */ +#define PAPR_SCM_DIMM_UNARMED_MASK (PAPR_SCM_DIMM_UNARMED | \ + PAPR_SCM_DIMM_HEALTH_UNHEALTHY) + +/* Bits status indicators for health bitmap indicating unflushed dimm */ +#define PAPR_SCM_DIMM_BAD_SHUTDOWN_MASK (PAPR_SCM_DIMM_SHUTDOWN_DIRTY) + +/* Bits status indicators for health bitmap indicating unrestored dimm */ +#define PAPR_SCM_DIMM_BAD_RESTORE_MASK (PAPR_SCM_DIMM_EMPTY) + +/* Bit status indicators for smart event notification */ +#define PAPR_SCM_DIMM_SMART_EVENT_MASK (PAPR_SCM_DIMM_HEALTH_CRITICAL | \ + PAPR_SCM_DIMM_HEALTH_FATAL | \ + PAPR_SCM_DIMM_HEALTH_UNHEALTHY) + +#endif
arch/powerpc/platforms/pseries/papr_scm.c +103 -2
141 lines not on this page · raw message
Signature · 4 lines
2.24.1 _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
PATCH v3 2/4 ndctl/uapi: Introduce NVDIMM_FAMILY_PAPR_SCM as a new NVDIMM DSM family
In reply to Vaibhav JainAdd PAPR-scm family of DSM command-set to the white list of NVDIMM command sets.
---
Changelog:
v2..v3 : Updated the patch prefix to 'ndctl/uapi' [Aneesh]
v1..v2 : None
--- include/uapi/linux/ndctl.h | 1 + 1 file changed, 1 insertion(+)
include/uapi/linux/ndctl.h +1 -0
@@ -244,6 +244,7 @@ struct nd_cmd_pkg {#define NVDIMM_FAMILY_HPE2 2#define NVDIMM_FAMILY_MSFT 3#define NVDIMM_FAMILY_HYPERV 4 +#define NVDIMM_FAMILY_PAPR_SCM 5#define ND_IOCTL_CALL _IOWR(ND_IOCTL, ND_CMD_CALL,\ struct nd_cmd_pkg)
Signature · 4 lines
2.24.1 _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
PATCH v3 3/4 powerpc/papr_scm,uapi: Add support for handling PAPR DSM commands
In reply to Vaibhav JainImplement support for handling PAPR DSM commands in papr_scm module. We advertise support for ND_CMD_CALL for the dimm command mask and implement necessary scaffolding in the module to handle ND_CMD_CALL ioctl and DSM commands that we receive.
The layout of the DSM commands as we expect from libnvdimm/libndctl is described in newly introduced uapi header 'papr_scm_dsm.h' which defines a new 'struct nd_papr_scm_cmd_pkg' header. This header is used to communicate the DSM command via 'nd_pkg_papr_scm->nd_command' and size of payload that need to be sent/received for servicing the DSM.
The PAPR DSM commands are assigned indexes started from 0x10000 to prevent them from overlapping ND_CMD_* values and also makes handling dimm commands in papr_scm_ndctl() easier via a simplified switch-case block. For this a new function cmd_to_func() is implemented that reads the args to papr_scm_ndctl() , performs sanity tests on them and converts them to PAPR DSM commands which can then be handled via the switch-case block.
---
Changelog:
v2..v3: Updated the nd_papr_scm_cmd_pkg to use __xx types as its
exported to the userspace [Aneesh]
v1..v2: New patch in the series.
--- arch/powerpc/include/uapi/asm/papr_scm_dsm.h | 161 +++++++++++++++++++ arch/powerpc/platforms/pseries/papr_scm.c | 87 +++++++++- 2 files changed, 241 insertions(+), 7 deletions(-) create mode 100644 arch/powerpc/include/uapi/asm/papr_scm_dsm.h
arch/powerpc/include/uapi/asm/papr_scm_dsm.h new file +161 -0
@@ -0,0 +1,161 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +/* +* PAPR SCM Device specific methods and struct for libndctl and ndctl +* +* (C) Copyright IBM 2020 +* +* Author: Vaibhav Jain <vaibhav at linux.ibm.com> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2, or (at your option) +* any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#ifndef _UAPI_ASM_POWERPC_PAPR_SCM_DSM_H_ +#define _UAPI_ASM_POWERPC_PAPR_SCM_DSM_H_ + +#include <linux/types.h> + +#ifdef __KERNEL__ +#include <linux/ndctl.h> +#else +#include <ndctl.h> +#endif + +/* +* DSM Envelope: +* +* The ioctl ND_CMD_CALL transfers data between user-space and kernel via +* 'envelopes' which consists of a header and user-defined payload sections. +* The header is described by 'struct nd_papr_scm_cmd_pkg' which expects a +* payload following it and offset of which relative to the struct is provided +* by 'nd_papr_scm_cmd_pkg.payload_offset'. * +* +* +-------------+---------------------+---------------------------+ +* | 64-Bytes | 8-Bytes | Max 184-Bytes | +* +-------------+---------------------+---------------------------+ +* | nd_papr_scm_cmd_pkg | | +* |-------------+ | | +* | nd_cmd_pkg | | | +* +-------------+---------------------+---------------------------+ +* | nd_family | | | +* | nd_size_out | cmd_status | | +* | nd_size_in | payload_version | PAYLOAD | +* | nd_command | payload_offset -----> | +* | nd_fw_size | | | +* +-------------+---------------------+---------------------------+ +* +* DSM Header: +* +* The header is defined as 'struct nd_papr_scm_cmd_pkg' which embeds a +* 'struct nd_cmd_pkg' instance. The DSM command is assigned to member +* 'nd_cmd_pkg.nd_command'. Apart from size information of the envelop which is +* contained in 'struct nd_cmd_pkg', the header also has members following +* members: +* +* 'cmd_status' : (Out) Errors if any encountered while servicing DSM. +* 'payload_version' : (In/Out) Version number associated with the payload. +* 'payload_offset' : (In)Relative offset of payload from start of envelope. +* +* DSM Payload: +* +* The layout of the DSM Payload is defined by various structs shared between +* papr_scm and libndctl so that contents of payload can be interpreted. During +* servicing of a DSM the papr_scm module will read input args from the payload +* field by casting its contents to an appropriate struct pointer based on the +* DSM command. Similarly the output of servicing the DSM command will be copied +* to the payload field using the same struct. +* +* 'libnvdimm' enforces a hard limit of 256 bytes on the envelope size, which +* leaves around 184 bytes for the envelope payload (ignoring any padding that +* the compiler may silently introduce). +* +* Payload Version: +* +* A 'payload_version' field is present in DSM header that indicates a specific +* version of the structure present in DSM Payload for a given DSM command. This +* provides backward compatibility in case the DSM Payload structure evolves +* and different structures are supported by 'papr_scm' and 'libndctl'. +* +* When sending a DSM Payload to 'papr_scm', 'libndctl' should send the version +* of the payload struct it supports via 'payload_version' field. The 'papr_scm' +* module when servicing the DSM envelop checks the 'payload_version' and then +* uses 'payload struct version' == MIN('payload_version field', +* 'max payload-struct-version supported by papr_scm') to service the DSM. After +* servicing the DSM, 'papr_scm' put the negotiated version of payload struct in +* returned 'payload_version' field. +* +* Libndctl on receiving the envelop back from papr_scm again checks the +* 'payload_version' field and based on it use the appropriate version dsm +* struct to parse the results. +* +* Backward Compatibility: +* +* Above scheme of exchanging different versioned DSM struct between libndctl +* and papr_scm should provide backward compatibility until following two +* assumptions/conditions when defining new DSM structs hold: +* +* Let T(X) = { set of attributes in DSM struct 'T' versioned X } +* +* 1. T(X) is a proper subset of T(Y) if X > Y. +* i.e Each new version of DSM struct should retain existing struct +* attributes from previous version +* +* 2. If an entity (libndctl or papr_scm) supports a DSM struct T(X) then +* it should also support T(1), T(2)...T(X - 1). +* i.e When adding support for new version of a DSM struct, libndctl +* and papr_scm should retain support of the existing DSM struct +* version they support. +*/ + +/* Papr-scm-header + payload expected with ND_CMD_CALL ioctl from libnvdimm */ +struct nd_papr_scm_cmd_pkg { +struct nd_cmd_pkg hdr;/* Package header containing sub-cmd */ +__s32 cmd_status;/* Out: Sub-cmd status returned back */ +__u16 payload_offset;/* In: offset from start of struct */ +__u16 payload_version;/* In/Out: version of the payload */ +__u8 payload[];/* In/Out: Sub-cmd data buffer */ +}; + +/* +* Sub commands for ND_CMD_CALL. To prevent overlap from ND_CMD_*, values for +* these enums start at 0x10000. These values are then returned from +* cmd_to_func() making it easy to implement the switch-case block in +* papr_scm_ndctl(). These commands are sent to the kernel via +* 'nd_papr_scm_cmd_pkg.hdr.nd_command' +*/ +enum dsm_papr_scm { +DSM_PAPR_SCM_MIN =0x10000 , +DSM_PAPR_SCM_MAX , +}; + +/* Helpers to evaluate the size of PAPR_SCM envelope */ +/* Calculate the papr_scm-header size */ +#define ND_PAPR_SCM_ENVELOPE_CONTENT_HDR_SIZE \ + (sizeof(struct nd_papr_scm_cmd_pkg) - sizeof(struct nd_cmd_pkg)) + +/* Given a type calculate envelope-content size (papr_scm-header + payload) */ +#define ND_PAPR_SCM_ENVELOPE_CONTENT_SIZE (_type_) \ + (sizeof(_type_) + ND_PAPR_SCM_ENVELOPE_CONTENT_HDR_SIZE) + +/* Convert a libnvdimm nd_cmd_pkg to papr_scm specific pkg */ +static struct nd_papr_scm_cmd_pkg *nd_to_papr_cmd_pkg (struct nd_cmd_pkg *cmd) +{ +return (struct nd_papr_scm_cmd_pkg *) cmd; +} + +/* Return the payload pointer for a given pcmd */ +static void *papr_scm_pcmd_to_payload (struct nd_papr_scm_cmd_pkg *pcmd) +{ +if (pcmd->hdr.nd_size_in ==0 && pcmd->hdr.nd_size_out ==0 ) +return NULL ; +else +return (void *)((__u8 *) pcmd + pcmd->payload_offset); +} +#endif /* _UAPI_ASM_POWERPC_PAPR_SCM_DSM_H_ */
arch/powerpc/platforms/pseries/papr_scm.c +80 -7
125 lines not on this page · raw message
Signature · 4 lines
2.24.1 _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
PATCH v3 4/4 powerpc/papr_scm: Implement support for DSM_PAPR_SCM_HEALTH
In reply to Vaibhav JainThis patch implements support for papr_scm command 'DSM_PAPR_SCM_HEALTH' that returns a newly introduced 'struct nd_papr_scm_dimm_health_stat' instance containing dimm health information back to user space in response to ND_CMD_CALL. This functionality is implemented in newly introduced papr_scm_get_health() that queries the scm-dimm health information and then copies these bitmaps to the package payload whose layout is defined by 'struct papr_scm_ndctl_health'.
The patch also introduces a new member a new member 'struct papr_scm_priv.health' thats an instance of 'struct nd_papr_scm_dimm_health_stat' to cache the health information of a scm-dimm. As a result functions drc_pmem_query_health() and papr_flags_show() are updated to populate and use this new struct instead of two be64 integers that we earlier used.
---
Changelog:
v2..v3: Updated struct nd_papr_scm_dimm_health_stat_v1 to use '__xx'
types as its exported to the userspace [Aneesh] Changed the constants DSM_PAPR_SCM_DIMM_XX indicating dimm health from enum to #defines [Aneesh]
v1..v2: New patch in the series
--- arch/powerpc/include/uapi/asm/papr_scm_dsm.h | 40 +++++++ arch/powerpc/platforms/pseries/papr_scm.c | 111 ++++++++++++++++--- 2 files changed, 134 insertions(+), 17 deletions(-)
arch/powerpc/include/uapi/asm/papr_scm_dsm.h +40 -0
@@ -132,6 +132,7 @@ struct nd_papr_scm_cmd_pkg {*/ enum dsm_papr_scm {DSM_PAPR_SCM_MIN =0x10000 , +DSM_PAPR_SCM_HEALTH ,DSM_PAPR_SCM_MAX , };
@@ -158,4 +159,43 @@ static void *papr_scm_pcmd_to_payload(struct nd_papr_scm_cmd_pkg *pcmd) else return (void *)((__u8 *) pcmd + pcmd->payload_offset); } + +/* Various scm-dimm health indicators */ +#define DSM_PAPR_SCM_DIMM_HEALTHY 0 +#define DSM_PAPR_SCM_DIMM_UNHEALTHY 1 +#define DSM_PAPR_SCM_DIMM_CRITICAL 2 +#define DSM_PAPR_SCM_DIMM_FATAL 3 + +/* +* Struct exchanged between kernel & ndctl in for PAPR_DSM_PAPR_SMART_HEALTH +* Various bitflags indicate the health status of the dimm. +* +* dimm_unarmed : Dimm not armed. So contents wont persist. +* dimm_bad_shutdown : Previous shutdown did not persist contents. +* dimm_bad_restore : Contents from previous shutdown werent restored. +* dimm_scrubbed : Contents of the dimm have been scrubbed. +* dimm_locked : Contents of the dimm cant be modified until CEC reboot +* dimm_encrypted : Contents of dimm are encrypted. +* dimm_health : Dimm health indicator. +*/ +struct nd_papr_scm_dimm_health_stat_v1 { +__u8 dimm_unarmed; +__u8 dimm_bad_shutdown; +__u8 dimm_bad_restore; +__u8 dimm_scrubbed; +__u8 dimm_locked; +__u8 dimm_encrypted; +__u16 dimm_health; +}; + +/* +* Typedef the current struct for dimm_health so that any application +* or kernel recompiled after introducing a new version automatically +* supports the new version. +*/ +#define nd_papr_scm_dimm_health_stat nd_papr_scm_dimm_health_stat_v1 + +/* Current version number for the dimm health struct */ +#define ND_PAPR_SCM_DIMM_HEALTH_VERSION 1 +#endif /* _UAPI_ASM_POWERPC_PAPR_SCM_DSM_H_ */
arch/powerpc/platforms/pseries/papr_scm.c +94 -17
166 lines not on this page · raw message
Signature · 4 lines
2.24.1 _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]