Skip to contentSkip to conversation

powerpc/papr_scm: Workaround for failure of drc bind after kexec

PATCH In review Vaibhav Jain 7 participants 16 messages Jul 24, 2019 – Jul 30, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v5 cover powerpc/papr_scm: Workaround for failure of drc bind after kexec

Presently an error is returned in response to hcall H_SCM_BIND_MEM when a new kernel boots on lpar via kexec. This prevents papr_scm from registering drc memory regions with nvdimm. The error reported is of the form below:

"papr_scm ibm,persistent-memory:ibm,pmemory@44100002: bind err: -68"

On investigation it was revealed that phyp returns this error as previous kernel did not completely release bindings for drc scm-memory blocks and hence phyp rejected request for re-binding these block to lpar with error H_OVERLAP. Also support for a new H_SCM_UNBIND_ALL is recently added which is better suited for releasing all the bound scm-memory block from an lpar.

So leveraging new hcall H_SCM_UNBIND_ALL, we can workaround H_OVERLAP issue during kexec by forcing an unbind of all drm scm-memory blocks and issuing H_SCM_BIND_MEM to re-bind the drc scm-memory blocks to lpar. This sequence will also be needed when a new kernel boot on lpar after previous kernel panicked and it never got an opportunity to call H_SCM_UNBIND_MEM/ALL.

Hence this patch-set implements following changes to papr_scm module:

* Update hvcall.h to include opcodes for new hcall H_SCM_UNBIND_ALL.

* Update it to use H_SCM_UNBIND_ALL instead of H_SCM_UNBIND_MEM

* In case hcall H_SCM_BIND_MEM fails with error H_OVERLAP, force H_SCM_UNBIND_ALL and retry the bind operation again.

With the patch-set applied re-bind of drc scm-memory to lpar succeeds after a kexec to new kernel as illustrated below:

# Old kernel

$ sudo ndctl list -R
[
  {
    "dev":"region0",
    <snip>
    ....
  }
]
# kexec to new kernel
$ sudo kexec --initrd=... vmlinux
...
...
I'm in purgatory
...
papr_scm ibm,persistent-memory:ibm,pmemory@44100002: Un-binding and retrying
...
# New kernel
$ sudo ndctl list -R
[
  {
    "dev":"region0",
    <snip>
    ....
  }
]

---
Change-log:
v5:
* Added a new doc-patch describing the HCALL interface between a guest kernel and PAPR compliant hyper-visor like PowerVM/KVM.

v4:
* Updated the patch description of first patch in the series as suggested by Mpe.

v3:
* Fixed a build warning reported by kbuild test robot.
* Updated the hcall opcode from latest papr-scm specification.
* Fixed a minor code comment & patch description as pointed out by Oliver.

v2:
* Addressed review comments from Oliver on v1 patchset.

Vaibhav Jain (4):
powerpc: Document some HCalls for Storage Class Memory powerpc/pseries: Update SCM hcall op-codes in hvcall.h powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

 Documentation/powerpc/hcalls.txt          | 140 ++++++++++++++++++++++
 arch/powerpc/include/asm/hvcall.h         |  11 +-
 arch/powerpc/platforms/pseries/papr_scm.c |  44 +++++--
 3 files changed, 184 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/powerpc/hcalls.txt
Signature · 1 line
2.21.0
#1 lore raw

PATCH v5 1/4 powerpc: Document some HCalls for Storage Class Memory

This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).

Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.

---
Change-log:

v5
* First patch in this patchset.

---
 Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++
 1 file changed, 140 insertions(+)
 create mode 100644 Documentation/powerpc/hcalls.txt
Documentation/powerpc/hcalls.txt new file +140 -0
new file mode 100644
index 000000000000..cc9dd872cecd
@@ -0,0 +1,140 @@
+Hyper-visor Call Op-codes (HCALLS)
+====================================
+
+Overview
+=========
+
+Virtualization on PPC64 arch is based on the PAPR specification[1] which
+describes run-time environment for a guest operating system and how it should
+interact with the hyper-visor for privileged operations. Currently there are two
+PAPR compliant hypervisors (PHYP):
+
+IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as
+	     supported guests (termed as Logical Partitions or LPARS).
+
+Qemu/KVM:    Supports PPC64 linux guests running on a PPC64 linux host.
+
+On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0).
+Hence to perform a privileged operations the guest issues a Hyper-visor
+Call (HCALL) with necessary input operands. PHYP after performing the privilege
+operation returns a status code and output operands back to the guest.
+
+HCALL ABI
+=========
+The ABI specification for a HCall between guest os kernel and PHYP is
+described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments
+for the Hcall are provided in registers R4-R12. On return from 'HVCS'
+instruction the status code of HCall is available in R3 an the output parameters
+are returned in registers R4-R12.
+
+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in
+header 'hvcall.h' to issue HCalls from the linux kernel running as guest.
+
+
+DRC & DRC Indexes
+=================
+
+		 PAPR		     Guest
+  DR1          Hypervisor             OS
+  +--+        +----------+         +---------+
+  |  |<------>|          |         |  User   |
+  +--+  DRC1  |          |   DRC   |  Space  |
+	      |          |  Index  +---------+
+  DR2         |          |         |         |
+  +--+        |          |<------->|  Kernel |
+  |  |<----- >|          |  HCall  |         |
+  +--+  DRC2  +----------+         +---------+
+
+PHYP terms shared hardware resources like PCI devices, NVDimms etc available for
+use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP
+creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR
+access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index.
+The DRC-index value is provided to the LPAR via device-tree where its present
+as an attribute in the device tree node associated with the DR.
+
+HCALL Op-codes
+==============
+
+Below is a partial of of HCALLs that are supported by PHYP. For the
+corresponding opcode values please look into the header
+'arch/powerpc/include/asm/hvcall.h' :
+
+* H_SCM_READ_METADATA:
+  Input: drcIndex, offset, buffer-address, numBytesToRead
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, read N-bytes from the the meta data area
+  associated with it, at a specified offset and copy it to provided buffer.
+  The metadata area stores configuration information such as label information,
+  bad-blocks etc. The metadata area is located out-of-band of NVDimm storage
+  area hence a separate access semantics is provided.
+
+* H_SCM_WRITE_METADATA:
+  Input: drcIndex, offset, data, numBytesToWrite
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the
+  given offset to the the meta data area associated with the NVDimm.
+
+
+* H_SCM_BIND_MEM:
+  Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress
+  Out: guestMappedAddress, numScmBlockBound
+  Description:
+  Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to
+  continuous logical addresses in guest physical address space. The HCALL
+  arguments can be used to map partial range of SCM blocks instead of entire
+  NVDimm range to the LPAR.
+
+* H_SCM_UNBIND_MEM:
+  Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind
+  Out: numScmBlocksUnbound
+  Description:
+  Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest
+  physical address space. The HCALL can fail if the Guest has an active PTE
+  entry to the SCM block being unbinded.
+
+* H_SCM_QUERY_BLOCK_MEM_BINDING:
+  Input: drcIndex, scmBlockIndex
+  Out: Guest-Physical-Address
+  Description:
+  Given a DRC-Index and an SCM Block index return the guest physical address to
+  which the SCM block is mapped to.
+
+* H_SCM_QUERY_LOGICAL_MEM_BINDING:
+  Input: Guest-Physical-Address
+  Out: drcIndex, scmBlockIndex
+  Description:
+  Given a guest physical address return which DRC Index and SCM block is mapped
+  to that address.
+
+* H_SCM_UNBIND_ALL:
+  Input: scmTargetScope, drcIndex
+  Out: None
+  Description:
+  Depending on the Target scope unmap all scm blocks belonging to all NVDimms
+  or all scm blocks belonging to a single NVDimm identified by its drcIndex
+  from the LPAR memory.
+
+* H_SCM_HEALTH:
+  Input: drcIndex
+  Output: health-bitmap, health-bit-valid-bitmap
+  Description:
+  Given a DRC Index return the info on predictive failure and over all health of
+  the NVDimm. The asserted bits in the health-bitmap indicate a single predictive
+  failure and health-bit-valid-bitmap indicate which bits in health-bitmap are
+  valid.
+
+
+* H_SCM_PERFORMANCE_STATS:
+  Input: drcIndex, resultBuffer Addr
+  Out: None
+  Description:
+  Given a DRC Index collect the performance statistics for NVDimm and copy them
+  to the resultBuffer.
+
+
+References
+==========
+[1]: "Linux on Power Architecture Platform Reference"
+     https://members.openpowerfoundation.org/document/dl/469
Signature · 1 line
2.21.0
#2 lore raw

Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory

Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :

Quoted 8 lines

This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).

Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.

Hi Vaibhav,

That's a good idea to introduce such a documentation.

Quoted 35 lines

---
Change-log:

v5
* First patch in this patchset.

---
  Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++
  1 file changed, 140 insertions(+)

create mode 100644 Documentation/powerpc/hcalls.txt

Documentation/powerpc/hcalls.txt new file +18 -0
new file mode 100644
index 000000000000..cc9dd872cecd
@@ -0,0 +1,140 @@
+Hyper-visor Call Op-codes (HCALLS)
+====================================
+
+Overview
+=========
+
+Virtualization on PPC64 arch is based on the PAPR specification[1] which
+describes run-time environment for a guest operating system and how it should
+interact with the hyper-visor for privileged operations. Currently there are two
+PAPR compliant hypervisors (PHYP):
+
+IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as
+	     supported guests (termed as Logical Partitions or LPARS).
+
+Qemu/KVM:    Supports PPC64 linux guests running on a PPC64 linux host.
+
+On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0).
+Hence to perform a privileged operations the guest issues a Hyper-visor

Documentation/powerpc/hcalls.txt line 28

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+Call (HCALL) with necessary input operands. PHYP after performing the privilege
+operation returns a status code and output operands back to the guest.
+
+HCALL ABI
+=========
+The ABI specification for a HCall between guest os kernel and PHYP is
+described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments
+for the Hcall are provided in registers R4-R12. On return from 'HVCS'
+instruction the status code of HCall is available in R3 an the output parameters
+are returned in registers R4-R12.

Would it be good to mention that values passed through the memory must be stored in Big Endian format ?

Quoted 14 lines
Documentation/powerpc/hcalls.txt new file +14 -0
+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in
+header 'hvcall.h' to issue HCalls from the linux kernel running as guest.
+
+
+DRC & DRC Indexes
+=================
+
+		 PAPR		     Guest
+  DR1          Hypervisor             OS
+  +--+        +----------+         +---------+
+  |  |<------>|          |         |  User   |
+  +--+  DRC1  |          |   DRC   |  Space  |
+	      |          |  Index  +---------+
+  DR2         |          |         |         |

Documentation/powerpc/hcalls.txt line 53

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+  +--+        |          |<------->|  Kernel |
+  |  |<----- >|          |  HCall  |         |
+  +--+  DRC2  +----------+         +---------+
+
+PHYP terms shared hardware resources like PCI devices, NVDimms etc available for
+use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP
+creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR
+access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index.
+The DRC-index value is provided to the LPAR via device-tree where its present
+as an attribute in the device tree node associated with the DR.

Should you use the term 'Hypervisor' instead of 'PHYP' which is not usually designing only the proprietary one ?

Thanks,
Laurent.

Quoted 88 lines
Documentation/powerpc/hcalls.txt new file +87 -0
+
+HCALL Op-codes
+==============
+
+Below is a partial of of HCALLs that are supported by PHYP. For the
+corresponding opcode values please look into the header
+'arch/powerpc/include/asm/hvcall.h' :
+
+* H_SCM_READ_METADATA:
+  Input: drcIndex, offset, buffer-address, numBytesToRead
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, read N-bytes from the the meta data area
+  associated with it, at a specified offset and copy it to provided buffer.
+  The metadata area stores configuration information such as label information,
+  bad-blocks etc. The metadata area is located out-of-band of NVDimm storage
+  area hence a separate access semantics is provided.
+
+* H_SCM_WRITE_METADATA:
+  Input: drcIndex, offset, data, numBytesToWrite
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the
+  given offset to the the meta data area associated with the NVDimm.
+
+
+* H_SCM_BIND_MEM:
+  Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress
+  Out: guestMappedAddress, numScmBlockBound
+  Description:
+  Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to
+  continuous logical addresses in guest physical address space. The HCALL
+  arguments can be used to map partial range of SCM blocks instead of entire
+  NVDimm range to the LPAR.
+
+* H_SCM_UNBIND_MEM:
+  Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind
+  Out: numScmBlocksUnbound
+  Description:
+  Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest
+  physical address space. The HCALL can fail if the Guest has an active PTE
+  entry to the SCM block being unbinded.
+
+* H_SCM_QUERY_BLOCK_MEM_BINDING:
+  Input: drcIndex, scmBlockIndex
+  Out: Guest-Physical-Address
+  Description:
+  Given a DRC-Index and an SCM Block index return the guest physical address to
+  which the SCM block is mapped to.
+
+* H_SCM_QUERY_LOGICAL_MEM_BINDING:
+  Input: Guest-Physical-Address
+  Out: drcIndex, scmBlockIndex
+  Description:
+  Given a guest physical address return which DRC Index and SCM block is mapped
+  to that address.
+
+* H_SCM_UNBIND_ALL:
+  Input: scmTargetScope, drcIndex
+  Out: None
+  Description:
+  Depending on the Target scope unmap all scm blocks belonging to all NVDimms
+  or all scm blocks belonging to a single NVDimm identified by its drcIndex
+  from the LPAR memory.
+
+* H_SCM_HEALTH:
+  Input: drcIndex
+  Output: health-bitmap, health-bit-valid-bitmap
+  Description:
+  Given a DRC Index return the info on predictive failure and over all health of
+  the NVDimm. The asserted bits in the health-bitmap indicate a single predictive
+  failure and health-bit-valid-bitmap indicate which bits in health-bitmap are
+  valid.
+
+
+* H_SCM_PERFORMANCE_STATS:
+  Input: drcIndex, resultBuffer Addr
+  Out: None
+  Description:
+  Given a DRC Index collect the performance statistics for NVDimm and copy them
+  to the resultBuffer.
+
+
+References
+==========
+[1]: "Linux on Power Architecture Platform Reference"
+     https://members.openpowerfoundation.org/document/dl/469
#3 lore raw

Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory

Quoted 91 lines · On Wed, 24 Jul 2019 11:08:58 +0200 Laurent Dufour <[email protected]>…

Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :

Quoted 8 lines

This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).

Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux kernel and how information flows between kernel and PowerVM/KVM.

Hi Vaibhav,

That's a good idea to introduce such a documentation.

Quoted 35 lines

---
Change-log:

v5
* First patch in this patchset.

---
  Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++
  1 file changed, 140 insertions(+)

create mode 100644 Documentation/powerpc/hcalls.txt

Documentation/powerpc/hcalls.txt new file +18 -0
new file mode 100644
index 000000000000..cc9dd872cecd
@@ -0,0 +1,140 @@
+Hyper-visor Call Op-codes (HCALLS)
+====================================
+
+Overview
+=========
+
+Virtualization on PPC64 arch is based on the PAPR specification[1] which
+describes run-time environment for a guest operating system and how it should
+interact with the hyper-visor for privileged operations. Currently there are two
+PAPR compliant hypervisors (PHYP):
+
+IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as
+	     supported guests (termed as Logical Partitions or LPARS).
+
+Qemu/KVM:    Supports PPC64 linux guests running on a PPC64 linux host.
+
+On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0).
+Hence to perform a privileged operations the guest issues a Hyper-visor

Documentation/powerpc/hcalls.txt line 28

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+Call (HCALL) with necessary input operands. PHYP after performing the privilege
+operation returns a status code and output operands back to the guest.
+
+HCALL ABI
+=========
+The ABI specification for a HCall between guest os kernel and PHYP is
+described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments
+for the Hcall are provided in registers R4-R12. On return from 'HVCS'
+instruction the status code of HCall is available in R3 an the output parameters
+are returned in registers R4-R12.  

Would it be good to mention that values passed through the memory must be stored in Big Endian format ?

Quoted 14 lines
Documentation/powerpc/hcalls.txt new file +14 -0
+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in
+header 'hvcall.h' to issue HCalls from the linux kernel running as guest.
+
+
+DRC & DRC Indexes
+=================
+
+		 PAPR		     Guest
+  DR1          Hypervisor             OS
+  +--+        +----------+         +---------+
+  |  |<------>|          |         |  User   |
+  +--+  DRC1  |          |   DRC   |  Space  |
+	      |          |  Index  +---------+
+  DR2         |          |         |         |

Documentation/powerpc/hcalls.txt line 53

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+  +--+        |          |<------->|  Kernel |
+  |  |<----- >|          |  HCall  |         |
+  +--+  DRC2  +----------+         +---------+
+
+PHYP terms shared hardware resources like PCI devices, NVDimms etc available for
+use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP
+creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR
+access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index.
+The DRC-index value is provided to the LPAR via device-tree where its present
+as an attribute in the device tree node associated with the DR.  

Should you use the term 'Hypervisor' instead of 'PHYP' which is not usually designing only the proprietary one ?

Quoted 8 lines

Thanks,
Laurent.

Documentation/powerpc/hcalls.txt line 58

Quoted 5 lines
Documentation/powerpc/hcalls.txt new file +5 -0
+
+HCALL Op-codes
+==============
+
+Below is a partial of of HCALLs that are supported by PHYP. For the
                        ^^ list?

Thanks

Michal

Quoted 84 lines
Quoted 83 lines
Documentation/powerpc/hcalls.txt new file +82 -0
+corresponding opcode values please look into the header
+'arch/powerpc/include/asm/hvcall.h' :
+
+* H_SCM_READ_METADATA:
+  Input: drcIndex, offset, buffer-address, numBytesToRead
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, read N-bytes from the the meta data area
+  associated with it, at a specified offset and copy it to provided buffer.
+  The metadata area stores configuration information such as label information,
+  bad-blocks etc. The metadata area is located out-of-band of NVDimm storage
+  area hence a separate access semantics is provided.
+
+* H_SCM_WRITE_METADATA:
+  Input: drcIndex, offset, data, numBytesToWrite
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the
+  given offset to the the meta data area associated with the NVDimm.
+
+
+* H_SCM_BIND_MEM:
+  Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress
+  Out: guestMappedAddress, numScmBlockBound
+  Description:
+  Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to
+  continuous logical addresses in guest physical address space. The HCALL
+  arguments can be used to map partial range of SCM blocks instead of entire
+  NVDimm range to the LPAR.
+
+* H_SCM_UNBIND_MEM:
+  Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind
+  Out: numScmBlocksUnbound
+  Description:
+  Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest
+  physical address space. The HCALL can fail if the Guest has an active PTE
+  entry to the SCM block being unbinded.
+
+* H_SCM_QUERY_BLOCK_MEM_BINDING:
+  Input: drcIndex, scmBlockIndex
+  Out: Guest-Physical-Address
+  Description:
+  Given a DRC-Index and an SCM Block index return the guest physical address to
+  which the SCM block is mapped to.
+
+* H_SCM_QUERY_LOGICAL_MEM_BINDING:
+  Input: Guest-Physical-Address
+  Out: drcIndex, scmBlockIndex
+  Description:
+  Given a guest physical address return which DRC Index and SCM block is mapped
+  to that address.
+
+* H_SCM_UNBIND_ALL:
+  Input: scmTargetScope, drcIndex
+  Out: None
+  Description:
+  Depending on the Target scope unmap all scm blocks belonging to all NVDimms
+  or all scm blocks belonging to a single NVDimm identified by its drcIndex
+  from the LPAR memory.
+
+* H_SCM_HEALTH:
+  Input: drcIndex
+  Output: health-bitmap, health-bit-valid-bitmap
+  Description:
+  Given a DRC Index return the info on predictive failure and over all health of
+  the NVDimm. The asserted bits in the health-bitmap indicate a single predictive
+  failure and health-bit-valid-bitmap indicate which bits in health-bitmap are
+  valid.
+
+
+* H_SCM_PERFORMANCE_STATS:
+  Input: drcIndex, resultBuffer Addr
+  Out: None
+  Description:
+  Given a DRC Index collect the performance statistics for NVDimm and copy them
+  to the resultBuffer.
+
+
+References
+==========
+[1]: "Linux on Power Architecture Platform Reference"
+     https://members.openpowerfoundation.org/document/dl/469
#4 lore raw

Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory

Thanks everyone for reviewing this patch-set. The V4 got merged upstream which didn't have this DOC-patch. So, I will re-spin a separate and independent doc-patch incorporating your review comments.

Cheers,

Signature · 2 lines
Vaibhav Jain <[email protected]>
Linux Technology Center, IBM India Pvt. Ltd.
#5 lore raw

Re: [DOC][PATCH v5 1/4] powerpc: Document some HCalls for Storage Class Memory

In reply to Vaibhav Jain

Hi Vaibhav,

Thanks for writing this documentation.

Quoted 7 lines · Vaibhav Jain

This doc patch provides an initial description of the HCall op-codes that are used by Linux kernel running as a guest operating system (LPAR) on top of PowerVM or any other sPAPR compliant hyper-visor (e.g qemu).

Apart from documenting the HCalls the doc-patch also provides a rudimentary overview of how Hcalls are implemented inside the Linux

I prefer "hcall" rather than "HCall", "Hcall" or "HCALL".

Quoted 6 lines

kernel and how information flows between kernel and PowerVM/KVM.

---
Change-log:

Quoted 6 lines

v5
* First patch in this patchset.

---
 Documentation/powerpc/hcalls.txt | 140 +++++++++++++++++++++++++++++++
 1 file changed, 140 insertions(+)
 create mode 100644 Documentation/powerpc/hcalls.txt

During this merge window all the existing documentation in there has been converted to rst format. I suspect Jon will be annoyed at us if we immediately start adding new plain text docs.

So we need to make this an rst file and do at least a minimal job of making it valid ReST.

A while back I worked out how to actually build the rst docs, I'll try and find my notes on that.

Documentation/powerpc/hcalls.txt

Quoted 5 lines
Documentation/powerpc/hcalls.txt new file +0 -0
new file mode 100644
index 000000000000..cc9dd872cecd

Can we call it papr_hcalls please.

Documentation/powerpc/hcalls.txt line 2

Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +2 -0
@@ -0,0 +1,140 @@
+Hyper-visor Call Op-codes (HCALLS)
+====================================

I'm not sure if "hyper-visor" is correct, but regardless that's not what we commonly use, so please just spell it "hypervisor". And similarly for "opcode".

Documentation/powerpc/hcalls.txt line 7

Quoted 4 lines
Documentation/powerpc/hcalls.txt new file +4 -0
+Overview
+=========
+
+Virtualization on PPC64 arch is based on the PAPR specification[1] which

We should probably say on "64-bit Power Book3S platforms".

Also the link you provide is to the "LoPAPR" specification. Which is not quite the same as "PAPR", which used to be released via power.org, or "PAPR+" which is the IBM internal version.

Documentation/powerpc/hcalls.txt line 10

Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3 -0
+describes run-time environment for a guest operating system and how it should
+interact with the hyper-visor for privileged operations. Currently there are two
+PAPR compliant hypervisors (PHYP):

"PHYP" is only used as a another name for PowerVM (or part of PowerVM), not as a generic term for a PAPR hypervisor.

I also don't think it's accurate to say Qemu/KVM is a "PAPR compliant" hypervisor, it just implements (some of) the relevant parts of LoPAPR to support Linux guests.

Documentation/powerpc/hcalls.txt line 15

Quoted 4 lines
Documentation/powerpc/hcalls.txt new file +4 -0
+IBM PowerVM: IBM's proprietary hyper-visor that supports AIX, IBM-i and Linux as
+	     supported guests (termed as Logical Partitions or LPARS).
+
+Qemu/KVM:    Supports PPC64 linux guests running on a PPC64 linux host.

And actually other hosts via Qemu TCG but that's a bit of a tangent.

Documentation/powerpc/hcalls.txt line 17

Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1 -0
+On PPC64 arch a virtualized guest kernel runs in a non-privileged mode (HV=0).

"virtualized guest" is redundant, just "guest" is sufficient IMHO.

I know what you're trying to say there, but it actually contradicts the language that's used in the ISA.

If you look at Chapter 1 of Book III it defines these terms:

* hypervisor privileged
A term used to describe an instruction or facility that is available only when the thread is in hypervisor state.
* privileged state and supervisor mode
Used interchangeably to refer to a state in which privileged facilities are available.
* problem state and user mode
Used interchangeably to refer to a state in which privileged facilities are not available.

So you might want to write it more like:

When running under a PAPR hypervisor the guest kernel runs in supervisor mode (HV=0), and must issue hypercalls to the hypervisor whenever it needs to perform an action that is hypervisor privileged or for other services managed by the hypervisor.

Documentation/powerpc/hcalls.txt line 24

Quoted 7 lines
Documentation/powerpc/hcalls.txt new file +7 -0
+Hence to perform a privileged operations the guest issues a Hyper-visor
+Call (HCALL) with necessary input operands. PHYP after performing the privilege
+operation returns a status code and output operands back to the guest.
+
+HCALL ABI
+=========
+The ABI specification for a HCall between guest os kernel and PHYP is

^
"OS", but probably just "guest kernel" is fine.

Documentation/powerpc/hcalls.txt line 25

Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1 -0
+described in [1]. The Opcode for Hcall is set in R3 and subsequent in-arguments

Where in [1] ?

It's more common to spell the GPRs with lower case 'r', eg. 'r3'.

Documentation/powerpc/hcalls.txt line 26

Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1 -0
+for the Hcall are provided in registers R4-R12. On return from      'HVCS'

^the 'HVSC'

Documentation/powerpc/hcalls.txt line 27

Quoted 1 line
Documentation/powerpc/hcalls.txt new file +1 -0
+instruction the status code of HCall is available in R3 an the output parameters
                   ^             ^                         ^
                   return value  the                       and
Quoted 11 lines
Documentation/powerpc/hcalls.txt new file +11 -0
+are returned in registers R4-R12.
+
+Powerpc arch code provides convenient wrappers named plpar_hcall_xxx defined in
+header 'hvcall.h' to issue HCalls from the linux kernel running as guest.
+
+
+DRC & DRC Indexes
+=================
+
+		 PAPR		     Guest
+  DR1          Hypervisor             OS

Documentation/powerpc/hcalls.txt line 48

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+  +--+        +----------+         +---------+
+  |  |<------>|          |         |  User   |
+  +--+  DRC1  |          |   DRC   |  Space  |
+              |          |  Index  +---------+
+  DR2         |          |         |         |
+  +--+        |          |<------->|  Kernel |
+  |  |<----- >|          |  HCall  |         |
+  +--+  DRC2  +----------+         +---------+
+
+PHYP terms shared hardware resources like PCI devices, NVDimms etc available for

DIMM is an acronym too, so it should be either nvdimm or NVDIMM IMHO.

Quoted 12 lines
Documentation/powerpc/hcalls.txt new file +12 -0
+use by LPARs as Dynamic Resource (DR). When a DR is allocated to an LPAR, PHYP
+creates a data-structure called Dynamic Resource Connector (DRC) to manage LPAR
+access. An LPAR refers to a DRC via an opaque 32-bit number called DRC-Index.
+The DRC-index value is provided to the LPAR via device-tree where its present
+as an attribute in the device tree node associated with the DR.
+
+HCALL Op-codes
+==============
+
+Below is a partial of of HCALLs that are supported by PHYP. For the
+corresponding opcode values please look into the header
+'arch/powerpc/include/asm/hvcall.h' :

Documentation/powerpc/hcalls.txt line 70

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+
+* H_SCM_READ_METADATA:
+  Input: drcIndex, offset, buffer-address, numBytesToRead
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, read N-bytes from the the meta data area
+  associated with it, at a specified offset and copy it to provided buffer.
+  The metadata area stores configuration information such as label information,
+  bad-blocks etc. The metadata area is located out-of-band of NVDimm storage
+  area hence a separate access semantics is provided.

Can you also document the possible return values? (If they are defined).

Quoted 6 lines
Documentation/powerpc/hcalls.txt new file +6 -0
+* H_SCM_WRITE_METADATA:
+  Input: drcIndex, offset, data, numBytesToWrite
+  Out: None
+  Description:
+  Given a DRC Index of an NVDimm, write N-bytes from provided buffer at the
+  given offset to the the meta data area associated with the NVDimm.

Documentation/powerpc/hcalls.txt line 87

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+
+
+* H_SCM_BIND_MEM:
+  Input: drcIndex, startingScmBlockIndex, numScmBlocksToBind, targetAddress
+  Out: guestMappedAddress, numScmBlockBound
+  Description:
+  Given a DRC-Index of an NVDimm, maps the SCM (Storage Class Memory) blocks to
+  continuous logical addresses in guest physical address space. The HCALL
+  arguments can be used to map partial range of SCM blocks instead of entire
+  NVDimm range to the LPAR.

What address space do targetAddress and guestMappedAddress exist in?

What are these blocks and how do we find out about them? ie. how do I know what a valid block index is.

Documentation/powerpc/hcalls.txt line 90

Quoted 2 lines
Documentation/powerpc/hcalls.txt new file +2 -0
+* H_SCM_UNBIND_MEM:
+  Input: drcIndex, startingScmLogicalMemoryAddress, numScmBlocksToUnbind

Similar question for startingScmLogicalMemoryAddress.

Documentation/powerpc/hcalls.txt line 93

Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3 -0
+  Out: numScmBlocksUnbound
+  Description:
+  Given a DRC-Index of an NVDimm, unmap one or more the SCM blocks from guest
                                                      ^
                                                      of

Documentation/powerpc/hcalls.txt line 95

Quoted 2 lines
Documentation/powerpc/hcalls.txt new file +2 -0
+  physical address space. The HCALL can fail if the Guest has an active PTE
+  entry to the SCM block being unbinded.

unbound

Quoted 9 lines
Documentation/powerpc/hcalls.txt new file +9 -0
+* H_SCM_QUERY_BLOCK_MEM_BINDING:
+  Input: drcIndex, scmBlockIndex
+  Out: Guest-Physical-Address
+  Description:
+  Given a DRC-Index and an SCM Block index return the guest physical address to
+  which the SCM block is mapped to.
+
+* H_SCM_QUERY_LOGICAL_MEM_BINDING:
+  Input: Guest-Physical-Address

Documentation/powerpc/hcalls.txt line 115

Quoted 10 lines
Documentation/powerpc/hcalls.txt new file +10 -0
+  Out: drcIndex, scmBlockIndex
+  Description:
+  Given a guest physical address return which DRC Index and SCM block is mapped
+  to that address.
+
+* H_SCM_UNBIND_ALL:
+  Input: scmTargetScope, drcIndex
+  Out: None
+  Description:
+  Depending on the Target scope unmap all scm blocks belonging to all NVDimms

^
SCM for consistency.

Documentation/powerpc/hcalls.txt line 123

Quoted 8 lines
Documentation/powerpc/hcalls.txt new file +8 -0
+  or all scm blocks belonging to a single NVDimm identified by its drcIndex
+  from the LPAR memory.
+
+* H_SCM_HEALTH:
+  Input: drcIndex
+  Output: health-bitmap, health-bit-valid-bitmap
+  Description:
+  Given a DRC Index return the info on predictive failure and over all health of

overall

Documentation/powerpc/hcalls.txt line 126

Quoted 3 lines
Documentation/powerpc/hcalls.txt new file +3 -0
+  the NVDimm. The asserted bits in the health-bitmap indicate a single predictive
+  failure and health-bit-valid-bitmap indicate which bits in health-bitmap are
+  valid.

Presumably using bit endian bit ordering?

Documentation/powerpc/hcalls.txt line 140

Quoted 12 lines
Documentation/powerpc/hcalls.txt new file +12 -0
+* H_SCM_PERFORMANCE_STATS:
+  Input: drcIndex, resultBuffer Addr
+  Out: None
+  Description:
+  Given a DRC Index collect the performance statistics for NVDimm and copy them
+  to the resultBuffer.
+
+
+References
+==========
+[1]: "Linux on Power Architecture Platform Reference"
+     https://members.openpowerfoundation.org/document/dl/469

cheers

#6 lore raw

PATCH v5 2/4 powerpc/pseries: Update SCM hcall op-codes in hvcall.h

In reply to Vaibhav Jain

Update the hvcalls.h to include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.

The removed hcall op-codes H_SCM_MEM_QUERY, H_SCM_BLOCK_CLEAR were transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.

---
Change-log:

v5:
* None. Re-spinning the patchset.

v4:
* Updated the patch description mentioned current status of removed hcall opcodes. [Mpe]

v3:
* Added updated opcode for H_SCM_HEALTH [Oliver]

v2:
* None new patch in this series.

---
 arch/powerpc/include/asm/hvcall.h | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/hvcall.h +8 -3
index 463c63a9fcf1..11112023e327 100644
@@ -302,9 +302,14 @@
 #define H_SCM_UNBIND_MEM        0x3F0
 #define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4
 #define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8
-#define H_SCM_MEM_QUERY	        0x3FC
-#define H_SCM_BLOCK_CLEAR       0x400
-#define MAX_HCALL_OPCODE	H_SCM_BLOCK_CLEAR
+#define H_SCM_UNBIND_ALL        0x3FC
+#define H_SCM_HEALTH            0x400
+#define H_SCM_PERFORMANCE_STATS 0x418
+#define MAX_HCALL_OPCODE	H_SCM_PERFORMANCE_STATS
+
+/* Scope args for H_SCM_UNBIND_ALL */
+#define H_UNBIND_SCOPE_ALL (0x1)
+#define H_UNBIND_SCOPE_DRC (0x2)
 
 /* H_VIOCTL functions */
 #define H_GET_VIOA_DUMP_SIZE	0x01
Signature · 1 line
2.21.0
#7 lore raw

Re: [PATCH v5 2/4] powerpc/pseries: Update SCM hcall op-codes in hvcall.h

Quoted 10 lines · Vaibhav Jain

Update the hvcalls.h to include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.

The removed hcall op-codes H_SCM_MEM_QUERY, H_SCM_BLOCK_CLEAR were transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.

They really should not have been merged while only interim proposals. But since they have changed, better to update them than not, obviously.

Quoted 41 lines

---
Change-log:

v5:
* None. Re-spinning the patchset.

v4:
* Updated the patch description mentioned current status of removed hcall opcodes. [Mpe]

v3:
* Added updated opcode for H_SCM_HEALTH [Oliver]

v2:
* None new patch in this series.

---
 arch/powerpc/include/asm/hvcall.h | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/powerpc/include/asm/hvcall.h +8 -3
index 463c63a9fcf1..11112023e327 100644
@@ -302,9 +302,14 @@
 #define H_SCM_UNBIND_MEM        0x3F0
 #define H_SCM_QUERY_BLOCK_MEM_BINDING 0x3F4
 #define H_SCM_QUERY_LOGICAL_MEM_BINDING 0x3F8
-#define H_SCM_MEM_QUERY	        0x3FC
-#define H_SCM_BLOCK_CLEAR       0x400
-#define MAX_HCALL_OPCODE	H_SCM_BLOCK_CLEAR
+#define H_SCM_UNBIND_ALL        0x3FC
+#define H_SCM_HEALTH            0x400
+#define H_SCM_PERFORMANCE_STATS 0x418
+#define MAX_HCALL_OPCODE	H_SCM_PERFORMANCE_STATS
+
+/* Scope args for H_SCM_UNBIND_ALL */
+#define H_UNBIND_SCOPE_ALL (0x1)
+#define H_UNBIND_SCOPE_DRC (0x2)
 
 /* H_VIOCTL functions */
 #define H_GET_VIOA_DUMP_SIZE	0x01
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#8 lore raw

Re: [PATCH v5 2/4] powerpc/pseries: Update SCM hcall op-codes in hvcall.h

Quoted 10 lines · David Gibson
Quoted 9 lines · Vaibhav Jain

Update the hvcalls.h to include op-codes for new hcalls introduce to manage SCM memory. Also update existing hcall definitions to reflect current papr specification for SCM.

The removed hcall op-codes H_SCM_MEM_QUERY, H_SCM_BLOCK_CLEAR were transient proposals and there support was never implemented by Power-VM nor they were used anywhere in Linux kernel. Hence we don't expect anyone to be impacted by this change.

Quoted 4 lines
Quoted 1 line

They really should not have been merged while only interim proposals. But since they have changed, better to update them than not, obviously.

Yes, absolutely agree. It wasn't clear when I merged them that they were *unpublished* and *unstable*.

Unfortunately we can't realistically wait for these APIs to be in a published PAPR spec because it seems to take about 5 years for a PAPR spec to escape the IBM firewall.

cheers

#9 lore raw

PATCH v5 3/4 powerpc/papr_scm: Update drc_pmem_unbind() to use H_SCM_UNBIND_ALL

In reply to Vaibhav Jain

The new hcall named H_SCM_UNBIND_ALL has been introduce that can unbind all or specific scm memory assigned to an lpar. This is more efficient than using H_SCM_UNBIND_MEM as currently we don't support partial unbind of scm memory.

Hence this patch proposes following changes to drc_pmem_unbind():

* Update drc_pmem_unbind() to replace hcall H_SCM_UNBIND_MEM to H_SCM_UNBIND_ALL.

* Update drc_pmem_unbind() to handles cases when PHYP asks the guest kernel to wait for specific amount of time before retrying the hcall via the 'LONG_BUSY' return value.

* Ensure appropriate error code is returned back from the function in case of an error.

---
Change-log:

v5:
* None. Re-spinning the patchset.

v4:
* None. Re-spinning the patchset.

v3:
* Fixed a build warning reported by kbuild-robot.
* Updated patch description to put emphasis on 'scm memory' instead of 'scm drc memory blocks' as for phyp there is a stark difference between how drc are managed for scm memory v/s regular memory. [Oliver]

v2:
* Added a dev_dbg when unbind operation succeeds [Oliver]
* Changed type of variable 'rc' to int64_t [Oliver]
* Removed the code that was logging a warning in case bind operation takes >1-seconds [Oliver]
* Spinned off changes to hvcall.h as a separate patch. [Oliver]

---
 arch/powerpc/platforms/pseries/papr_scm.c | 29 +++++++++++++++++------
 1 file changed, 22 insertions(+), 7 deletions(-)
arch/powerpc/platforms/pseries/papr_scm.c +22 -7
index c8ec670ee924..82568a7e0a7c 100644
@@ -11,6 +11,7 @@
 #include <linux/sched.h>
 #include <linux/libnvdimm.h>
 #include <linux/platform_device.h>
+#include <linux/delay.h>
 
 #include <asm/plpar_wrappers.h>
 
@@ -78,22 +79,36 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
 static int drc_pmem_unbind(struct papr_scm_priv *p)
 {
 	unsigned long ret[PLPAR_HCALL_BUFSIZE];
-	uint64_t rc, token;
+	uint64_t token = 0;
+	int64_t rc;
 
-	token = 0;
+	dev_dbg(&p->pdev->dev, "unbind drc %x\n", p->drc_index);
 
-	/* NB: unbind has the same retry requirements mentioned above */
+	/* NB: unbind has the same retry requirements as drc_pmem_bind() */
 	do {
-		rc = plpar_hcall(H_SCM_UNBIND_MEM, ret, p->drc_index,
-				p->bound_addr, p->blocks, token);
+
+		/* Unbind of all SCM resources associated with drcIndex */
+		rc = plpar_hcall(H_SCM_UNBIND_ALL, ret, H_UNBIND_SCOPE_DRC,
+				 p->drc_index, token);
 		token = ret[0];
-		cond_resched();
+
+		/* Check if we are stalled for some time */
+		if (H_IS_LONG_BUSY(rc)) {
+			msleep(get_longbusy_msecs(rc));
+			rc = H_BUSY;
+		} else if (rc == H_BUSY) {
+			cond_resched();
+		}
+
 	} while (rc == H_BUSY);
 
 	if (rc)
 		dev_err(&p->pdev->dev, "unbind error: %lld\n", rc);
+	else
+		dev_dbg(&p->pdev->dev, "unbind drc %x complete\n",
+			p->drc_index);
 
-	return !!rc;
+	return rc == H_SUCCESS ? 0 : -ENXIO;
 }
 
 static int papr_scm_meta_get(struct papr_scm_priv *p,
Signature · 1 line
2.21.0
#10 lore raw

PATCH v5 4/4 powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

In reply to Vaibhav Jain

In some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the H_SCM_BIND_MEM return a H_OVERLAP error.

To mitigate such cases the patch updates papr_scm_probe() to force a call to drc_pmem_unbind() in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also update drc_pmem_bind() to handle the H_OVERLAP error returned by phyp and indicate it as a EBUSY error back to the caller.

---
Change-log:

v5:
* None. Re-spinning the patchset.

v4:
* None. Re-spinning the patchset.

v3:
* Minor update to a code comment. [Oliver]

v2:
* Moved the retry code from drc_pmem_bind() to papr_scm_probe() [Oliver]
* Changed the type of variable 'rc' in drc_pmem_bind() to int64_t. [Oliver]

---
 arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/papr_scm.c +14 -1
index 82568a7e0a7c..2c07908359b2 100644
@@ -44,8 +44,9 @@ struct papr_scm_priv {
 static int drc_pmem_bind(struct papr_scm_priv *p)
 {
 	unsigned long ret[PLPAR_HCALL_BUFSIZE];
-	uint64_t rc, token;
 	uint64_t saved = 0;
+	uint64_t token;
+	int64_t rc;
 
 	/*
 	 * When the hypervisor cannot map all the requested memory in a single
@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
 	} while (rc == H_BUSY);
 
 	if (rc) {
+		/* H_OVERLAP needs a separate error path */
+		if (rc == H_OVERLAP)
+			return -EBUSY;
+
 		dev_err(&p->pdev->dev, "bind err: %lld\n", rc);
 		return -ENXIO;
 	}
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
 
 	/* request the hypervisor to bind this region to somewhere in memory */
 	rc = drc_pmem_bind(p);
+
+	/* If phyp says drc memory still bound then force unbound and retry */
+	if (rc == -EBUSY) {
+		dev_warn(&pdev->dev, "Retrying bind after unbinding\n");
+		drc_pmem_unbind(p);
+		rc = drc_pmem_bind(p);
+	}
+
 	if (rc)
 		goto err;
 
Signature · 1 line
2.21.0
#11 lore raw

Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

Le 23/07/2019 à 18:13, Vaibhav Jain a écrit :

Quoted 63 lines

In some cases initial bind of scm memory for an lpar can fail if previously it wasn't released using a scm-unbind hcall. This situation can arise due to panic of the previous kernel or forced lpar fadump. In such cases the H_SCM_BIND_MEM return a H_OVERLAP error.

To mitigate such cases the patch updates papr_scm_probe() to force a call to drc_pmem_unbind() in case the initial bind of scm memory fails with EBUSY error. In case scm-bind operation again fails after the forced scm-unbind then we follow the existing error path. We also update drc_pmem_bind() to handle the H_OVERLAP error returned by phyp and indicate it as a EBUSY error back to the caller.

---
Change-log:

v5:
* None. Re-spinning the patchset.

v4:
* None. Re-spinning the patchset.

v3:
* Minor update to a code comment. [Oliver]

v2:
* Moved the retry code from drc_pmem_bind() to papr_scm_probe() [Oliver]
* Changed the type of variable 'rc' in drc_pmem_bind() to int64_t. [Oliver]

---
  arch/powerpc/platforms/pseries/papr_scm.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/papr_scm.c +6 -1
index 82568a7e0a7c..2c07908359b2 100644
@@ -44,8 +44,9 @@ struct papr_scm_priv {
  static int drc_pmem_bind(struct papr_scm_priv *p)
  {
  	unsigned long ret[PLPAR_HCALL_BUFSIZE];
-	uint64_t rc, token;
  	uint64_t saved = 0;
+	uint64_t token;
+	int64_t rc;
 
  	/*
  	 * When the hypervisor cannot map all the requested memory in a single
@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
  	} while (rc == H_BUSY);
 
  	if (rc) {
+		/* H_OVERLAP needs a separate error path */
+		if (rc == H_OVERLAP)
+			return -EBUSY;
+
  		dev_err(&p->pdev->dev, "bind err: %lld\n", rc);
  		return -ENXIO;
  	}

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417

Quoted 9 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -0
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
 
  	/* request the hypervisor to bind this region to somewhere in memory */
  	rc = drc_pmem_bind(p);
+
+	/* If phyp says drc memory still bound then force unbound and retry */
+	if (rc == -EBUSY) {
+		dev_warn(&pdev->dev, "Retrying bind after unbinding\n");
+		drc_pmem_unbind(p);
+		rc = drc_pmem_bind(p);

In the unlikely case where H_SCM_BIND_MEM is returning H_OVERLAP once the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?

Quoted 5 lines
arch/powerpc/platforms/pseries/papr_scm.c +2 -0
+	}
+
  	if (rc)
  		goto err;
 
#12 lore raw

Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain

*snip*

(quoted diff) +0 -0
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
 
Quoted 13 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417

Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -0
      /* request the hypervisor to bind this region to somewhere in memory */
      rc = drc_pmem_bind(p);
+
+     /* If phyp says drc memory still bound then force unbound and retry */
+     if (rc == -EBUSY) {
+             dev_warn(&pdev->dev, "Retrying bind after unbinding\n");
+             drc_pmem_unbind(p);
+             rc = drc_pmem_bind(p);

In the unlikely case where H_SCM_BIND_MEM is returning H_OVERLAP once the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?

drc_pmem_bind() prints the h-call error code if we get one, so it's not silent

#13 lore raw

Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

Le 24/07/2019 à 11:24, Oliver O'Halloran a écrit :

Quoted 21 lines
Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain

*snip*

(quoted diff) +0 -0
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
 
Quoted 13 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417

Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -0
       /* request the hypervisor to bind this region to somewhere in memory */
       rc = drc_pmem_bind(p);
+
+     /* If phyp says drc memory still bound then force unbound and retry */
+     if (rc == -EBUSY) {
+             dev_warn(&pdev->dev, "Retrying bind after unbinding\n");
+             drc_pmem_unbind(p);
+             rc = drc_pmem_bind(p);

In the unlikely case where H_SCM_BIND_MEM is returning H_OVERLAP once the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?

Quoted 1 line

drc_pmem_bind() prints the h-call error code if we get one, so it's not silent

That's no more the case whith this patch, H_OVERLAP is handled before writing the error message, which would make sense for the first try.

For the record, the patch introduces:

@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
} while (rc == H_BUSY);

if (rc) {
+ /* H_OVERLAP needs a separate error path */
+ if (rc == H_OVERLAP)
+ return -EBUSY;
+
dev_err(&p->pdev->dev, "bind err: %lld\n", rc); return -ENXIO;
}

#14 lore raw

Re: [PATCH v5 4/4] powerpc/papr_scm: Force a scm-unbind if initial scm-bind fails

Quoted 31 lines · Laurent Dufour
Quoted 21 lines · Oliver O'Halloran
Quoted 5 lines · Laurent Dufour
Quoted 3 lines · Vaibhav Jain

*snip*

(quoted diff) +0 -0
@@ -404,6 +409,14 @@ static int papr_scm_probe(struct platform_device *pdev)
 
Quoted 13 lines

arch/powerpc/platforms/pseries/papr_scm.c static int papr_scm_probe(struct platform_device *pdev) · line 417

Quoted 8 lines
arch/powerpc/platforms/pseries/papr_scm.c +6 -0
       /* request the hypervisor to bind this region to somewhere in memory */
       rc = drc_pmem_bind(p);
+
+     /* If phyp says drc memory still bound then force unbound and retry */
+     if (rc == -EBUSY) {
+             dev_warn(&pdev->dev, "Retrying bind after unbinding\n");
+             drc_pmem_unbind(p);
+             rc = drc_pmem_bind(p);

In the unlikely case where H_SCM_BIND_MEM is returning H_OVERLAP once the unbinding has been done, the error would be silently processed. That sounds really unlikely, but should an error message be displayed in this particular case ?

Quoted 1 line

drc_pmem_bind() prints the h-call error code if we get one, so it's not silent

That's no more the case whith this patch, H_OVERLAP is handled before writing the error message, which would make sense for the first try.

For the record, the patch introduces:

arch/powerpc/platforms/pseries/papr_scm.c static int drc_pmem_bind(struct papr_scm_priv *p) · line 75

Quoted 10 lines
arch/powerpc/platforms/pseries/papr_scm.c +4 -0
@@ -65,6 +66,10 @@ static int drc_pmem_bind(struct papr_scm_priv *p)
        } while (rc == H_BUSY);
 
        if (rc) {
+               /* H_OVERLAP needs a separate error path */
+               if (rc == H_OVERLAP)
+                       return -EBUSY;
+
                dev_err(&p->pdev->dev, "bind err: %lld\n", rc);
                return -ENXIO;
        }

Ah, good point. Getting H_OVERLAP is still an error case so I think it's reasonable to still print the message in that case.

Reviewed-by on the v5 series
#15 lore raw
In reply to Vaibhav Jain
Quoted 18 lines · Vaibhav Jain

Presently an error is returned in response to hcall H_SCM_BIND_MEM when a new kernel boots on lpar via kexec. This prevents papr_scm from registering drc memory regions with nvdimm. The error reported is of the form below:

"papr_scm ibm,persistent-memory:ibm,pmemory@44100002: bind err: -68"

On investigation it was revealed that phyp returns this error as previous kernel did not completely release bindings for drc scm-memory blocks and hence phyp rejected request for re-binding these block to lpar with error H_OVERLAP. Also support for a new H_SCM_UNBIND_ALL is recently added which is better suited for releasing all the bound scm-memory block from an lpar.

So leveraging new hcall H_SCM_UNBIND_ALL, we can workaround H_OVERLAP issue during kexec by forcing an unbind of all drm scm-memory blocks and issuing H_SCM_BIND_MEM to re-bind the drc scm-memory blocks to lpar. This sequence will also be needed when a new kernel boot on lpar after previous kernel panicked and it never got an opportunity to call H_SCM_UNBIND_MEM/ALL.

Quoted 9 lines

Hence this patch-set implements following changes to papr_scm module:

* Update hvcall.h to include opcodes for new hcall H_SCM_UNBIND_ALL.

* Update it to use H_SCM_UNBIND_ALL instead of H_SCM_UNBIND_MEM

* In case hcall H_SCM_BIND_MEM fails with error H_OVERLAP, force H_SCM_UNBIND_ALL and retry the bind operation again.

You can add for the series.