Skip to content

powerpc/pseries/vas: NXGZIP support with DLPAR

Haren Myneni linuxppc-dev 6 versions – Applied

Status from this message: Applied to powerpc/next.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v6 Latest9 patches 1 replyApplied
  2. v59 patches 0 repliesSuperseded
  3. v49 patches 13 repliesSuperseded
Earlier versions · 3
VersionPostedPatchesRepliesStatus
v310 patches16 repliesSuperseded
v210 patches0 repliesSuperseded
v110 patches0 repliesSuperseded

Changes in v6 from the cover letter

- Build fix in "Add paste address mmap fault handler" patch as reported by kernel test robot <[email protected]>

Patches in v6 Applied

#Subject and filesSizeRepliesApplied
1/9powerpc/pseries/vas: Use common names in VAS capability structurearch/powerpc/platforms/pseries/vas.c arch/powerpc/platforms/pseries/vas.h+7 -80 replies
2/9powerpc/pseries/vas: Save PID in pseries_vas_window structarch/powerpc/platforms/pseries/vas.c arch/powerpc/platforms/pseries/vas.h+6 -40 replies
3/9powerpc/vas: Add paste address mmap fault handlerarch/powerpc/include/asm/vas.h arch/powerpc/platforms/book3s/vas-api.c+78 -00 replies
4/9powerpc/vas: Return paste instruction failure if no active windowarch/powerpc/include/asm/ppc-opcode.h arch/powerpc/platforms/book3s/vas-api.c+56 -00 replies
5/9powerpc/vas: Map paste address only if window is activearch/powerpc/platforms/book3s/vas-api.c+22 -10 replies
6/9powerpc/pseries/vas: Close windows with DLPAR core removalarch/powerpc/include/asm/vas.h arch/powerpc/platforms/pseries/vas.c arch/powerpc/platforms/pseries/vas.h+204 -80 replies
7/9powerpc/pseries/vas: Reopen windows with DLPAR core addarch/powerpc/platforms/pseries/vas.c+90 -10 replies
8/9powerpc/pseries/vas: sysfs interface to export capabilitiesarch/powerpc/platforms/pseries/Makefile arch/powerpc/platforms/pseries/vas-sysfs.c arch/powerpc/platforms/pseries/vas.c +1 more+239 -10 replies
9/9powerpc/pseries/vas: Add 'update_total_credits' entry for QoS capabilitiesarch/powerpc/platforms/pseries/vas-sysfs.c arch/powerpc/platforms/pseries/vas.c arch/powerpc/platforms/pseries/vas.h+50 -70 replies

Reviews trailers per patch

4 of 9 patches have a Reviewed-by or Acked-by.

PatchNicholas Piggin
1/9R-b
2/9R-b
3/9R-b
4/9
5/9
6/9
7/9
8/9R-b
9/9

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v6 thread

Cover letter [PATCH v6 0/9] powerpc/pseries/vas: NXGZIP support with DLPAR

PowerPC provides HW compression with NX coprocessor. This feature is available on both PowerNV and PowerVM and included in Linux. Since each powerpc chip has one NX coprocessor, the VAS introduces the concept of windows / credits to manage access to this hardware resource. On powerVM, these limited resources should be available across all LPARs. So the hypervisor assigns the specific credits to each LPAR based on processor entitlement so that one LPAR does not overload NX. The hypervisor can reject the window open request to a partition if exceeds its credit limit (1 credit per window).

So the total number of target credits in a partition can be changed if the core configuration is modified. The hypervisor expects the partition to modify its window usage depends on new target credits. For example, if the partition uses more credits than the new target credits, it should close the excessive windows so that the NX resource will be available to other partitions.

This patch series enables OS to support this dynamic credit management with DLPAR core removal/add.

Core removal operation:
- Get new VAS capabilities from the hypervisor when the DLPAR notifier is received. This capabilities provides the new target credits based on new processor entitlement. In the case of QoS credit changes, the notification will be issued by updating the target_creds via sysfs.
- If the partition is already used more than the new target credits, the kernel selects windows, unmap the current paste address and close them in the hypervisor, It uses FIFO to identify these windows - last windows that are opened are the first ones to be closed.
- When the user space issue requests on these windows, NX generates page fault on the unmap paste address. The kernel handles the fault by returning the paste instruction failure if the window is not active (means unmap paste). Then up to the library / user space to fall back to SW compression or manage with the current windows.

Core add operation:
- The kernel can see increased target credits from the new VAS capabilities.
- Scans the window list for the closed windows in the hypervisor due to lost credit before and selects windows based on same FIFO.
- Make these corresponding windows active and create remap with the same VMA on the new paste address in the fault handler.
- Then the user space should expect paste successful later.

Patch 1: Define common names for sysfs target/used/avail_creds so
that same sysfs entries can be used even on PowerNV later.
Patch 2: Save PID in the vas window struct during initial window
open and use it when reopen later.
Patch 3: Add new mmap fault handler which handles the page fault
from NX on paste address.
Patch 4: Return the paste instruction failure if the window is not
active.
Patch 5: If the window is closed in the hypervisor before the user
space issue the initial mmap(), return -EACCES failure.
Patch 6: Close windows in the hypervisor when the partition exceeds
its usage than the new target credits.
Patch 7: When credits are available, reopen windows that are closed
before with core removal.
Patch 8 & 9: The user space determines the credit usage with sysfs
nr_total/nr_used_credits interfaces. drmgr uses update_total_credits to notify OS for QoS credit changes.

Thanks to Nicholas Piggin and Aneesh Kumar for the valuable suggestions on the NXGZIP design to support DLPAR operations.

Changes in v2:
- Rebase on 5.16-rc5
- Use list safe functions to iterate windows list
- Changes to show the actual value in sysfs used_credits even though some windows are inactive with core removal. Reflects -ve value in sysfs avail_creds to let userspace know that it opened more windows than the current maximum LPAR credits.

Changes in v3:
- Rebase on 5.16
- Reconfigure VAS windows only for CPU hotplug events.

Changes in v4:
- Rebase on 5.17-rc4
- Changes based on comments from Nicholas Piggin
- Included VAS DLPAR notifer code in 'Close windows with DLPAR' patch instead of as a separate patch
- Patches reordering and other changes

Changes in v5:
- Rebase on 5.17-rc5
- Add update_total_credits sysfs entry to update QoS target credits and other commit descriptions as suggested by Nicholas Piggin

Changed in v6:
- Build fix in "Add paste address mmap fault handler" patch as reported by kernel test robot <[email protected]>

Haren Myneni (9):
powerpc/pseries/vas: Use common names in VAS capability structure powerpc/pseries/vas: Save PID in pseries_vas_window struct powerpc/vas: Add paste address mmap fault handler powerpc/vas: Return paste instruction failure if no active window powerpc/vas: Map paste address only if window is active powerpc/pseries/vas: Close windows with DLPAR core removal powerpc/pseries/vas: Reopen windows with DLPAR core add powerpc/pseries/vas: sysfs interface to export capabilities powerpc/pseries/vas: Add 'update_total_credits' entry for QoS
capabilities

 arch/powerpc/include/asm/ppc-opcode.h      |   2 +
 arch/powerpc/include/asm/vas.h             |  12 +
 arch/powerpc/platforms/book3s/vas-api.c    | 145 +++++++++-
 arch/powerpc/platforms/pseries/Makefile    |   2 +-
 arch/powerpc/platforms/pseries/vas-sysfs.c | 268 +++++++++++++++++
 arch/powerpc/platforms/pseries/vas.c       | 321 +++++++++++++++++++--
 arch/powerpc/platforms/pseries/vas.h       |  16 +-
 7 files changed, 744 insertions(+), 22 deletions(-)
 create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
Signature · 1 line
2.27.0