powerpc/pseries/vas: NXGZIP support with DLPAR
v4 cover powerpc/pseries/vas: NXGZIP support with DLPAR
PowerPC provides HW compression with NX coprocessor. This feature is available on both PowerNV and PowerVM and included in Linux. Since each powerpc chip has one NX coprocessor, the VAS introduces the concept of windows / credits to manage access to this hardware resource. On powerVM, these limited resources should be available across all LPARs. So the hypervisor assigns the specific credits to each LPAR based on processor entitlement so that one LPAR does not overload NX. The hypervisor can reject the window open request to a partition if exceeds its credit limit (1 credit per window).
So the total number of target credits in a partition can be changed if the core configuration is modified. The hypervisor expects the partition to modify its window usage depends on new target credits. For example, if the partition uses more credits than the new target credits, it should close the excessive windows so that the NX resource will be available to other partitions.
This patch series enables OS to support this dynamic credit management with DLPAR core removal/add.
Core removal operation:
- Get new VAS capabilities from the hypervisor when the DLPAR notifier is received. This capabilities provides the new target credits based on new processor entitlement. In the case of QoS credit changes, the notification will be issued by updating the target_creds via sysfs.
- If the partition is already used more than the new target credits, the kernel selects windows, unmap the current paste address and close them in the hypervisor, It uses FIFO to identify these windows - last windows that are opened are the first ones to be closed.
- When the user space issue requests on these windows, NX generates page fault on the unmap paste address. The kernel handles the fault by returning the paste instruction failure if the window is not active (means unmap paste). Then up to the library / user space to fall back to SW compression or manage with the current windows.
Core add operation:
- The kernel can see increased target credits from the new VAS capabilities.
- Scans the window list for the closed windows in the hypervisor due to lost credit before and selects windows based on same FIFO.
- Make these corresponding windows active and create remap with the same VMA on the new paste address in the fault handler.
- Then the user space should expect paste successful later.
Patch 1: Define common names for sysfs target/used/avail_creds so
that same sysfs entries can be used even on PowerNV later.
Patch 2: Save PID in the vas window struct during initial window
open and use it when reopen later.
Patch 3: Add new mmap fault handler which handles the page fault
from NX on paste address.
Patch 4: Return the paste instruction failure if the window is not
active.
Patch 5: If the window is closed in the hypervisor before the user
space issue the initial mmap(), return -EACCES failure.
Patch 6: Close windows in the hypervisor when the partition exceeds
its usage than the new target credits.
Patch 7: When credits are available, reopen windows that are closed
before with core removal.
Patch 8 & 9: The user space determines the credit usage with sysfs
target/avail/used_creds interfaces. drmgr uses target_creds
to notify OS for QoS credit changes.
Thanks to Nicholas Piggin and Aneesh Kumar for the valuable suggestions on the NXGZIP design to support DLPAR operations.
Changes in v2:
- Rebase on 5.16-rc5
- Use list safe functions to iterate windows list
- Changes to show the actual value in sysfs used_credits even though some windows are inactive with core removal. Reflects -ve value in sysfs avail_creds to let userspace know that it opened more windows than the current maximum LPAR credits.
Changes in v3:
- Rebase on 5.16
- Reconfigure VAS windows only for CPU hotplug events.
Changes in v4:
- Rebase on 5.17-rc4
- Changes based on comments from Nicholas Piggin
- Included VAS DLPAR notifer code in 'Close windows with DLPAR' patch instead of as a separate patch
- Patches reordering and other changes
Haren Myneni (9):
powerpc/pseries/vas: Use common names in VAS capability structure powerpc/pseries/vas: Save PID in pseries_vas_window struct powerpc/vas: Add paste address mmap fault handler powerpc/vas: Return paste instruction failure if no active window powerpc/vas: Map paste address only if window is active powerpc/pseries/vas: Close windows with DLPAR core removal powerpc/pseries/vas: Reopen windows with DLPAR core add powerpc/pseries/vas: sysfs interface to export capabilities powerpc/pseries/vas: Write 'nr_total_credits' for QoS credits change
arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/include/asm/vas.h | 12 + arch/powerpc/platforms/book3s/vas-api.c | 141 ++++++++- arch/powerpc/platforms/pseries/Makefile | 2 +- arch/powerpc/platforms/pseries/vas-sysfs.c | 257 +++++++++++++++++ arch/powerpc/platforms/pseries/vas.c | 321 +++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 16 +- 7 files changed, 729 insertions(+), 22 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
Signature · 1 line
2.27.0
PATCH v4 1/9 powerpc/pseries/vas: Use common names in VAS capability structure
nr_total/nr_used_credits provides credits usage to user space via sysfs and the same interface can be used on PowerNV in future. Changed with proper naming so that applicable on both pseries and PowerNV.
--- arch/powerpc/platforms/pseries/vas.c | 10 +++++----- arch/powerpc/platforms/pseries/vas.h | 5 ++--- 2 files changed, 7 insertions(+), 8 deletions(-)
arch/powerpc/platforms/pseries/vas.c +5 -5
@@ -310,8 +310,8 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, cop_feat_caps = &caps->caps; -if (atomic_inc_return (&cop_feat_caps->used_lpar_creds) > -atomic_read (&cop_feat_caps->target_lpar_creds)) { +if (atomic_inc_return (&cop_feat_caps->nr_used_credits) > +atomic_read (&cop_feat_caps->nr_total_credits)) {pr_err ("Credits are not available to allocate window\n" ); rc = -EINVAL ; goto out;
@@ -385,7 +385,7 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags,free_irq_setup (txwin);h_deallocate_vas_window (txwin->vas_win.winid); out: -atomic_dec (&cop_feat_caps->used_lpar_creds); +atomic_dec (&cop_feat_caps->nr_used_credits);kfree (txwin);return ERR_PTR (rc); }
@@ -445,7 +445,7 @@ static int vas_deallocate_window(struct vas_window *vwin) }list_del (&win->win_list); -atomic_dec (&caps->used_lpar_creds); +atomic_dec (&caps->nr_used_credits);mutex_unlock (&vas_pseries_mutex);put_vas_user_win_ref (&vwin->task_ref);
@@ -521,7 +521,7 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, } caps->max_lpar_creds =be16_to_cpu (hv_caps->max_lpar_creds); caps->max_win_creds =be16_to_cpu (hv_caps->max_win_creds); -atomic_set (&caps->target_lpar_creds, +atomic_set (&caps->nr_total_credits,be16_to_cpu (hv_caps->target_lpar_creds));if (feat ==VAS_GZIP_DEF_FEAT ) { caps->def_lpar_creds =be16_to_cpu (hv_caps->def_lpar_creds);
arch/powerpc/platforms/pseries/vas.h +2 -3
@@ -72,9 +72,8 @@ struct vas_cop_feat_caps { };/* Total LPAR available credits. Can be different from max LPAR */ /* credits due to DLPAR operation */ -atomic_t target_lpar_creds; -atomic_t used_lpar_creds;/* Used credits so far */ -u16 avail_lpar_creds;/* Remaining available credits */ +atomic_t nr_total_credits;/* Total credits assigned to LPAR */ +atomic_t nr_used_credits;/* Used credits so far */ };/*
Signature · 1 line
2.27.0
PATCH v4 2/9 powerpc/pseries/vas: Save PID in pseries_vas_window struct
In reply to Haren MyneniThe kernel sets the VAS window with PID when it is opened in the hypervisor. During DLPAR operation, windows can be closed and reopened in the hypervisor when the credit is available. So saves this PID in pseries_vas_window struct when the window is opened initially and reuse it later during DLPAR operation.
--- arch/powerpc/platforms/pseries/vas.c | 9 +++++---- arch/powerpc/platforms/pseries/vas.h | 1 + 2 files changed, 6 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/vas.c +5 -4
@@ -107,7 +107,6 @@ static int h_deallocate_vas_window(u64 winid)static int h_modify_vas_window (struct pseries_vas_window *win) {long rc; -u32 lpid =mfspr (SPRN_PID );/* * AMR value is not supported in Linux VAS implementation.
@@ -115,7 +114,7 @@ static int h_modify_vas_window(struct pseries_vas_window *win)*/ do { rc = plpar_hcall_norets(H_MODIFY_VAS_WINDOW, - win->vas_win.winid, lpid, 0, + win->vas_win.winid, win->pid, 0, VAS_MOD_WIN_FLAGS, 0); rc = hcall_return_busy_check(rc);
@@ -124,8 +123,8 @@ static int h_modify_vas_window(struct pseries_vas_window *win)if (rc ==H_SUCCESS )return 0 ; -pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u lpid %u\n" , - rc, win->vas_win.winid, lpid); +pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u pid %u\n" , + rc, win->vas_win.winid, win->pid);return -EIO ; }
@@ -338,6 +337,8 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, } } + txwin->pid =mfspr (SPRN_PID ); +/* * Allocate / Deallocate window hcalls and setup / free IRQs * have to be protected with mutex.
arch/powerpc/platforms/pseries/vas.h +1 -0
@@ -114,6 +114,7 @@ struct pseries_vas_window {u64 domain[6 ];/* Associativity domain Ids */ /* this window is allocated */ u64 util; +u32 pid;/* PID associated with this window */ /* List of windows opened which is used for LPM */ struct list_head win_list;
Signature · 1 line
2.27.0
Re: [PATCH v4 2/9] powerpc/pseries/vas: Save PID in pseries_vas_window struct
Excerpts from Haren Myneni's message of February 20, 2022 5:55 am:
Quoted 6 lines
The kernel sets the VAS window with PID when it is opened in the hypervisor. During DLPAR operation, windows can be closed and reopened in the hypervisor when the credit is available. So saves this PID in
pseries_vas_windowstruct when the window is opened initially and reuse it later during DLPAR operation.
Thanks for renaming it lpid->pid and adding the comment.
Quoted 65 lines
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/vas.c | 9 +++++---- arch/powerpc/platforms/pseries/vas.h | 1 + 2 files changed, 6 insertions(+), 4 deletions(-)arch/powerpc/platforms/pseries/vas.c +5
-4@@ -107,7 +107,6 @@ static int h_deallocate_vas_window(u64 winid)static int h_modify_vas_window (struct pseries_vas_window *win) {long rc; -u32 lpid =mfspr (SPRN_PID );/* * AMR value is not supported in Linux VAS implementation. @@ -115,7 +114,7 @@ static int h_modify_vas_window(struct pseries_vas_window *win)*/ do { rc = plpar_hcall_norets(H_MODIFY_VAS_WINDOW, - win->vas_win.winid, lpid, 0, + win->vas_win.winid, win->pid, 0, VAS_MOD_WIN_FLAGS, 0); rc = hcall_return_busy_check(rc);@@ -124,8 +123,8 @@ static int h_modify_vas_window(struct pseries_vas_window *win)if (rc ==H_SUCCESS )return 0 ; -pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u lpid %u\n" , - rc, win->vas_win.winid, lpid); +pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u pid %u\n" , + rc, win->vas_win.winid, win->pid);return -EIO ; }@@ -338,6 +337,8 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, } } + txwin->pid =mfspr (SPRN_PID ); +/* * Allocate / Deallocate window hcalls and setup / free IRQs * have to be protected with mutex. arch/powerpc/platforms/pseries/vas.h +1
-0@@ -114,6 +114,7 @@ struct pseries_vas_window {u64 domain[6 ];/* Associativity domain Ids */ /* this window is allocated */ u64 util; +u32 pid;/* PID associated with this window */ /* List of windows opened which is used for LPM */ struct list_head win_list;Signature · 1 line
2.27.0
PATCH v4 3/9 powerpc/vas: Add paste address mmap fault handler
In reply to Haren MyneniThe user space opens VAS windows and issues NX requests by pasting CRB on the corresponding paste address mmap. When the system lost credits due to core removal, the kernel has to close the window in the hypervisor and make the window inactive by unmapping this paste address. Also the OS has to handle NX request page faults if the user space issue NX requests.
This handler maps the new paste address with the same VMA when the window is active again (due to core add with DLPAR). Otherwise returns paste failure.
--- arch/powerpc/include/asm/vas.h | 10 ++++ arch/powerpc/platforms/book3s/vas-api.c | 68 +++++++++++++++++++++++++ 2 files changed, 78 insertions(+)
arch/powerpc/include/asm/vas.h +10 -0
@@ -29,6 +29,12 @@#define VAS_THRESH_FIFO_GT_QTR_FULL 2#define VAS_THRESH_FIFO_GT_EIGHTH_FULL 3 +/* +* VAS window Linux status bits +*/ +#define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ +/* vas mmap() */ +/* * Get/Set bit fields */
@@ -59,6 +65,9 @@ struct vas_user_win_ref {struct pid *pid;/* PID of owner */ struct pid *tgid;/* Thread group ID of owner */ struct mm_struct *mm;/* Linux process mm_struct */ +struct mutex mmap_mutex;/* protects paste address mmap() */ +/* with DLPAR close/open windows */ +struct vm_area_struct *vma;/* Save VMA and used in DLPAR ops */ };/*
@@ -67,6 +76,7 @@ struct vas_user_win_ref {struct vas_window {u32 winid;u32 wcreds_max;/* Window credits */ +u32 status;/* Window status used in OS */ enum vas_cop_type cop;struct vas_user_win_ref task_ref;char *dbgname;
arch/powerpc/platforms/book3s/vas-api.c +68 -0
86 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH v4 3/9] powerpc/vas: Add paste address mmap fault handler
Excerpts from Haren Myneni's message of February 20, 2022 5:55 am:
Quoted 8 lines
The user space opens VAS windows and issues NX requests by pasting CRB on the corresponding paste address mmap. When the system lost credits due to core removal, the kernel has to close the window in the hypervisor and make the window inactive by unmapping this paste address. Also the OS has to handle NX request page faults if the user space issue NX requests.
Quoted 4 lines
This handler maps the new paste address with the same VMA when the window is active again (due to core add with DLPAR). Otherwise returns paste failure.
Quoted 139 lines
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/include/asm/vas.h | 10 ++++ arch/powerpc/platforms/book3s/vas-api.c | 68 +++++++++++++++++++++++++ 2 files changed, 78 insertions(+)arch/powerpc/include/asm/vas.h +10
-0@@ -29,6 +29,12 @@#define VAS_THRESH_FIFO_GT_QTR_FULL 2#define VAS_THRESH_FIFO_GT_EIGHTH_FULL 3 +/* +* VAS window Linux status bits +*/ +#define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ +/* vas mmap() */ +/* * Get/Set bit fields */ @@ -59,6 +65,9 @@ struct vas_user_win_ref {struct pid *pid;/* PID of owner */ struct pid *tgid;/* Thread group ID of owner */ struct mm_struct *mm;/* Linux process mm_struct */ +struct mutex mmap_mutex;/* protects paste address mmap() */ +/* with DLPAR close/open windows */ +struct vm_area_struct *vma;/* Save VMA and used in DLPAR ops */ };/* @@ -67,6 +76,7 @@ struct vas_user_win_ref {struct vas_window {u32 winid;u32 wcreds_max;/* Window credits */ +u32 status;/* Window status used in OS */ enum vas_cop_type cop;struct vas_user_win_ref task_ref;char *dbgname;arch/powerpc/platforms/book3s/vas-api.c +68
-0@@ -316,6 +316,7 @@ static int coproc_ioc_tx_win_open(struct file *fp, unsigned long arg)return PTR_ERR (txwin); } +mutex_init (&txwin->task_ref.mmap_mutex); cp_inst->txwin = txwin;return 0 ;@@ -350,6 +351,70 @@ static int coproc_release(struct inode *inode, struct file *fp)return 0 ; } +/* +* This fault handler is invoked when the core generates page fault on +* the paste address. Happens if the kernel closes window in hypervisor +* (on pseries) due to lost credit or the paste address is not mapped. +*/ +static vm_fault_t vas_mmap_fault (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct file *fp = vma->vm_file; +struct coproc_instance *cp_inst = fp->private_data; +struct vas_window *txwin; +u64 paste_addr; +int ret; + +/* +* window is not opened. Shouldn't expect this error. +*/ +if (!cp_inst || !cp_inst->txwin) { +pr_err ("%s(): Unexpected fault on paste address with TX window closed\n" , + __func__); +return VM_FAULT_SIGBUS ; + } + + txwin = cp_inst->txwin; +/* +* When the LPAR lost credits due to core removal or during +* migration, invalidate the existing mapping for the current +* paste addresses and set windows in-active (zap_page_range in +* reconfig_close_windows()). +* New mapping will be done later after migration or new credits +* available. So continue to receive faults if the user space +* issue NX request. +*/ +if (txwin->task_ref.vma != vmf->vma) { +pr_err ("%s(): No previous mapping with paste address\n" , + __func__); +return VM_FAULT_SIGBUS ; + } + +mutex_lock (&txwin->task_ref.mmap_mutex); +/* +* The window may be inactive due to lost credit (Ex: core +* removal with DLPAR). If the window is active again when +* the credit is available, map the new paste address at the +* the window virtual address. +*/ +if (txwin->status ==VAS_WIN_ACTIVE ) { + paste_addr = cp_inst->coproc->vops->paste_addr (txwin); +if (paste_addr) { + ret =vmf_insert_pfn (vma, vma->vm_start, + (paste_addr >>PAGE_SHIFT )); +mutex_unlock (&txwin->task_ref.mmap_mutex); +return ret; + } + } +mutex_unlock (&txwin->task_ref.mmap_mutex); + +return VM_FAULT_SIGBUS ; + +} +static const struct vm_operations_struct vas_vm_ops = { + .fault = vas_mmap_fault, +}; +static int coproc_mmap (struct file *fp,struct vm_area_struct *vma) {struct coproc_instance *cp_inst = fp->private_data;@@ -398,6 +463,9 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)pr_devel ("%s(): paste addr %llx at %lx, rc %d\n" , __func__, paste_addr, vma->vm_start, rc); + txwin->task_ref.vma = vma; + vma->vm_ops = &vas_vm_ops; +return rc; }Signature · 1 line
2.27.0
PATCH v4 4/9 powerpc/vas: Return paste instruction failure if no active window
In reply to Haren MyneniThe VAS window may not be active if the system looses credits and the NX generates page fault when it receives request on unmap paste address.
The kernel handles the fault by remap new paste address if the window is active again, Otherwise return the paste instruction failure if the executed instruction that caused the fault was a paste.
--- arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/platforms/book3s/vas-api.c | 55 ++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/ppc-opcode.h +2 -0
@@ -262,6 +262,8 @@#define PPC_INST_MFSPR_PVR 0x7c1f42a6#define PPC_INST_MFSPR_PVR_MASK 0xfc1ffffe#define PPC_INST_MTMSRD 0x7c000164 +#define PPC_INST_PASTE 0x7c20070d +#define PPC_INST_PASTE_MASK 0xfc2007ff#define PPC_INST_POPCNTB 0x7c0000f4#define PPC_INST_POPCNTB_MASK 0xfc0007fe#define PPC_INST_RFEBB 0x4c000124
arch/powerpc/platforms/book3s/vas-api.c +54 -1
69 lines not on this page · raw message
Signature · 1 line
2.27.0
Nicholas Piggin
reviewerRe: [PATCH v4 4/9] powerpc/vas: Return paste instruction failure if no active window
Excerpts from Haren Myneni's message of February 20, 2022 5:58 am:
Quoted 9 lines
The VAS window may not be active if the system looses credits and the NX generates page fault when it receives request on unmap paste address.
The kernel handles the fault by remap new paste address if the window is active again, Otherwise return the paste instruction failure if the executed instruction that caused the fault was a paste.
Looks good, thanks for fixin the SIGBUS thing, was that my fault? I vaguely remember writing some of this patch :P
Thanks,
Nick
Quoted 101 lines
Signed-off-by Nicholas Piggin <[email protected]>Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/platforms/book3s/vas-api.c | 55 ++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)arch/powerpc/include/asm/ppc-opcode.h +2
-0@@ -262,6 +262,8 @@#define PPC_INST_MFSPR_PVR 0x7c1f42a6#define PPC_INST_MFSPR_PVR_MASK 0xfc1ffffe#define PPC_INST_MTMSRD 0x7c000164 +#define PPC_INST_PASTE 0x7c20070d +#define PPC_INST_PASTE_MASK 0xfc2007ff#define PPC_INST_POPCNTB 0x7c0000f4#define PPC_INST_POPCNTB_MASK 0xfc0007fe#define PPC_INST_RFEBB 0x4c000124arch/powerpc/platforms/book3s/vas-api.c +54
-1@@ -351,6 +351,41 @@ static int coproc_release(struct inode *inode, struct file *fp)return 0 ; } +/* +* If the executed instruction that caused the fault was a paste, then +* clear regs CR0[EQ], advance NIP, and return 0. Else return error code. +*/ +static int do_fail_paste (void ) +{ +struct pt_regs *regs = current->thread.regs; +u32 instword; + +if (WARN_ON_ONCE (!regs)) +return -EINVAL ; + +if (WARN_ON_ONCE (!user_mode (regs))) +return -EINVAL ; + +/* +* If we couldn't translate the instruction, the driver should +* return success without handling the fault, it will be retried +* or the instruction fetch will fault. +*/ +if (get_user (instword, (u32 __user *)(regs->nip))) +return -EAGAIN ; + +/* +* Not a paste instruction, driver may fail the fault. +*/ +if ((instword &PPC_INST_PASTE_MASK ) !=PPC_INST_PASTE ) +return -ENOENT ; + + regs->ccr &= ~0xe0000000 ;/* Clear CR0[0-2] to fail paste */ +regs_add_return_ip (regs,4 );/* Emulate the paste */ + +return 0 ; +} +/* * This fault handler is invoked when the core generates page fault on * the paste address. Happens if the kernel closes window in hypervisor @@ -408,9 +443,27 @@ static vm_fault_t vas_mmap_fault(struct vm_fault *vmf) }mutex_unlock (&txwin->task_ref.mmap_mutex); -return VM_FAULT_SIGBUS ; +/* +* Received this fault due to closing the actual window. +* It can happen during migration or lost credits. +* Since no mapping, return the paste instruction failure +* to the user space. +*/ + ret =do_fail_paste (); +/* +* The user space can retry several times until success (needed +* for migration) or should fallback to SW compression or +* manage with the existing open windows if available. +* Looking at sysfs interface, it can determine whether these +* failures are coming during migration or core removal: +* nr_used_credits > nr_total_credits when lost credits +*/ +if (!ret || (ret == -EAGAIN )) +return VM_FAULT_NOPAGE ; +return VM_FAULT_SIGBUS ; } +static const struct vm_operations_struct vas_vm_ops = { .fault = vas_mmap_fault, };Signature · 1 line
2.27.0
Re: [PATCH v4 4/9] powerpc/vas: Return paste instruction failure if no active window
Quoted 5 lines · Nicholas Piggin
Excerpts from Haren Myneni's message of February 20, 2022 5:58 am:
Quoted 4 lines
The VAS window may not be active if the system looses credits and the NX generates page fault when it receives request on unmap paste address.
Quoted 7 lines
Quoted 4 lines
The kernel handles the fault by remap new paste address if the window is active again, Otherwise return the paste instruction failure if the executed instruction that caused the fault was a paste.
Looks good, thanks for fixin the SIGBUS thing, was that my fault? I vaguely remember writing some of this patch :P
Thanks for your reviews on all patches.
No, it was my fault not handling the -EGAIN error.
Quoted 114 lines
Thanks,
NickQuoted 110 lines
Signed-off-by Nicholas Piggin <[email protected]>Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/platforms/book3s/vas-api.c | 55 ++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-)arch/powerpc/include/asm/ppc-opcode.h +2
-0@@ -262,6 +262,8 @@#define PPC_INST_MFSPR_PVR 0x7c1f42a6#define PPC_INST_MFSPR_PVR_MASK 0xfc1ffffe#define PPC_INST_MTMSRD 0x7c000164 +#define PPC_INST_PASTE 0x7c20070d +#define PPC_INST_PASTE_MASK 0xfc2007ff#define PPC_INST_POPCNTB 0x7c0000f4#define PPC_INST_POPCNTB_MASK 0xfc0007fe#define PPC_INST_RFEBB 0x4c000124arch/powerpc/platforms/book3s/vas-api.c +2
-0@@ -351,6 +351,41 @@ static int coproc_release(struct inode *inode, struct file *fp)return 0 ; } +/* +* If the executed instruction that caused the fault was a paste, then
+ * clear regs CR0[EQ], advance NIP, and return 0. Else return error code.
+ */arch/powerpc/platforms/book3s/vas-api.c +14
-0+static int do_fail_paste (void ) +{ +struct pt_regs *regs = current->thread.regs; +u32 instword; + +if (WARN_ON_ONCE (!regs)) +return -EINVAL ; + +if (WARN_ON_ONCE (!user_mode (regs))) +return -EINVAL ; + +/* +* If we couldn't translate the instruction, the driver should +* return success without handling the fault, it will be retried
arch/powerpc/platforms/book3s/vas-api.c +17
-0+* or the instruction fetch will fault. +*/ +if (get_user (instword, (u32 __user *)(regs->nip))) +return -EAGAIN ; + +/* +* Not a paste instruction, driver may fail the fault. +*/ +if ((instword &PPC_INST_PASTE_MASK ) !=PPC_INST_PASTE ) +return -ENOENT ; + + regs->ccr &= ~0xe0000000 ;/* Clear CR0[0-2] to fail paste */ +regs_add_return_ip (regs,4 );/* Emulate the paste */ + +return 0 ; +} +/* * This fault handler is invoked when the core generates page fault on
* the paste address. Happens if the kernel closes window in
hypervisorarch/powerpc/platforms/book3s/vas-api.c +19
-1@@ -408,9 +443,27 @@ static vm_fault_t vas_mmap_fault(struct vm_fault *vmf) }mutex_unlock (&txwin->task_ref.mmap_mutex); -return VM_FAULT_SIGBUS ; +/* +* Received this fault due to closing the actual window. +* It can happen during migration or lost credits. +* Since no mapping, return the paste instruction failure +* to the user space. +*/ + ret =do_fail_paste (); +/* +* The user space can retry several times until success (needed +* for migration) or should fallback to SW compression or +* manage with the existing open windows if available. +* Looking at sysfs interface, it can determine whether these +* failures are coming during migration or core removal: +* nr_used_credits > nr_total_credits when lost credits +*/ +if (!ret || (ret == -EAGAIN )) +return VM_FAULT_NOPAGE ; +return VM_FAULT_SIGBUS ; } +static const struct vm_operations_struct vas_vm_ops = { .fault = vas_mmap_fault,};
Signature · 1 line
2.27.0
PATCH v4 5/9 powerpc/vas: Map paste address only if window is active
In reply to Haren MyneniThe paste address mapping is done with mmap() after the window is opened with ioctl. If the window is closed by OS in the hypervisor due to DLPAR after this mmap(), the paste instruction returns failure until the OS reopens this window again. But before mmap(), DLPAR core removal can happen which causes the corresponding window in-active. So if the window is not active, return mmap() failure with -EACCES and expects the user space reissue mmap() when the window is active or open a new window when the credit is available.
--- arch/powerpc/platforms/book3s/vas-api.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
arch/powerpc/platforms/book3s/vas-api.c +19 -1
@@ -496,10 +496,26 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)return -EACCES ; } +/* +* The initial mmap is done after the window is opened +* with ioctl. But before mmap(), this window can be closed in +* the hypervisor due to lost credit (core removal on pseries). +* So if the window is not active, return mmap() failure with +* -EACCES and expects the user space reissue mmap() when it +* is active again or open new window when the credit is available. +*/ +mutex_lock (&txwin->task_ref.mmap_mutex); +if (txwin->status !=VAS_WIN_ACTIVE ) { +pr_err ("%s(): Window is not active\n" , __func__); + rc = -EACCES ; + goto out; + } + paste_addr = cp_inst->coproc->vops->paste_addr (txwin);if (!paste_addr) {pr_err ("%s(): Window paste address failed\n" , __func__); -return -EINVAL ; + rc = -EINVAL ; + goto out; } pfn = paste_addr >>PAGE_SHIFT ;
@@ -519,6 +535,8 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) txwin->task_ref.vma = vma; vma->vm_ops = &vas_vm_ops; +out: +mutex_unlock (&txwin->task_ref.mmap_mutex);return rc; }
Signature · 1 line
2.27.0
Nicholas Piggin
reviewerRe: [PATCH v4 5/9] powerpc/vas: Map paste address only if window is active
Excerpts from Haren Myneni's message of February 20, 2022 5:59 am:
Quoted 4 lines
The paste address mapping is done with
mmap()after the window is opened with ioctl. If the window is closed by OS in the hypervisor due to DLPAR after thismmap(), the paste instruction returns
I don't think the changelog was improved here.
The window is closed by the OS in response to a DLPAR operation by the hypervisor? The OS can't be in the hypervisor.
Quoted 45 lines
failure until the OS reopens this window again. But before
mmap(), DLPAR core removal can happen which causes the corresponding window in-active. So if the window is not active, returnmmap()failure with-EACCESand expects the user space reissuemmap()when the window is active or open a new window when the credit is available.Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/book3s/vas-api.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)arch/powerpc/platforms/book3s/vas-api.c +17
-1@@ -496,10 +496,26 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)return -EACCES ; } +/* +* The initial mmap is done after the window is opened +* with ioctl. But before mmap(), this window can be closed in +* the hypervisor due to lost credit (core removal on pseries). +* So if the window is not active, return mmap() failure with +* -EACCES and expects the user space reissue mmap() when it +* is active again or open new window when the credit is available. +*/ +mutex_lock (&txwin->task_ref.mmap_mutex); +if (txwin->status !=VAS_WIN_ACTIVE ) { +pr_err ("%s(): Window is not active\n" , __func__); + rc = -EACCES ; + goto out; + } + paste_addr = cp_inst->coproc->vops->paste_addr (txwin);if (!paste_addr) {pr_err ("%s(): Window paste address failed\n" , __func__); -return -EINVAL ; + rc = -EINVAL ; + goto out; } pfn = paste_addr >>PAGE_SHIFT ;
arch/powerpc/platforms/book3s/vas-api.c static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) · line 539
Quoted 5 lines
arch/powerpc/platforms/book3s/vas-api.c +2
-0@@ -519,6 +535,8 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) txwin->task_ref.vma = vma; vma->vm_ops = &vas_vm_ops; +out: +mutex_unlock (&txwin->task_ref.mmap_mutex);
Did we have an explanation or what mmap_mutex is protecting? Sorry if you explained it and I forgot -- would be good to have a small comment (what is it protecting against).
Thanks,
Nick
Re: [PATCH v4 5/9] powerpc/vas: Map paste address only if window is active
Quoted 9 lines · Nicholas Piggin
Excerpts from Haren Myneni's message of February 20, 2022 5:59 am:
Quoted 3 lines
The paste address mapping is done with
mmap()after the window is opened with ioctl. If the window is closed by OS in the hypervisor due to DLPAR after thismmap(), the paste instruction returnsI don't think the changelog was improved here.
The window is closed by the OS in response to a DLPAR operation by the hypervisor? The OS can't be in the hypervisor.
From the user space point of view, this window is inactive. But from the hypervisor point of view, this window is closed. So my point is the window is closed by the OS in the hypervisor for the DLPAR event.
Quoted 55 lines
Quoted 55 lines
failure until the OS reopens this window again. But before
mmap(), DLPAR core removal can happen which causes the corresponding window in-active. So if the window is not active, returnmmap()failure with-EACCESand expects the user space reissuemmap()when the window is active or open a new window when the credit is available.Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/book3s/vas-api.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)arch/powerpc/platforms/book3s/vas-api.c +7
-0@@ -496,10 +496,26 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)return -EACCES ; } +/* +* The initial mmap is done after the window is opened +* with ioctl. But before mmap(), this window can be closed in +* the hypervisor due to lost credit (core removal on pseries). +* So if the window is not active, return mmap() failure with +* -EACCES and expects the user space reissue mmap() when it +* is active again or open new window when the credit is available.
arch/powerpc/platforms/book3s/vas-api.c +8
-0+*/ +mutex_lock (&txwin->task_ref.mmap_mutex); +if (txwin->status !=VAS_WIN_ACTIVE ) { +pr_err ("%s(): Window is not active\n" , __func__); + rc = -EACCES ; + goto out; + } + paste_addr = cp_inst->coproc->vops->paste_addr (txwin);if (!paste_addr) { pr_err("%s(): Window paste address failed\n",
__func__);arch/powerpc/platforms/book3s/vas-api.c +2
-1-return -EINVAL ; + rc = -EINVAL ; + goto out; } pfn = paste_addr >>PAGE_SHIFT ;@@ -519,6 +535,8 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) txwin->task_ref.vma = vma; vma->vm_ops = &vas_vm_ops;
Quoted 8 lines
arch/powerpc/platforms/book3s/vas-api.c static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) · line 539
Quoted 2 lines
arch/powerpc/platforms/book3s/vas-api.c +2
-0+out: +mutex_unlock (&txwin->task_ref.mmap_mutex);Did we have an explanation or what
mmap_mutexis protecting? Sorry if
you explained it and I forgot -- would be good to have a small comment
(what is it protecting against).
The comment should be in the struct definition. I will add some comment in this patch.
Thanks
Haren
Quoted 3 lines
Thanks,
Nick
PATCH v4 6/9 powerpc/pseries/vas: Close windows with DLPAR core removal
In reply to Haren MyneniThe hypervisor assigns vas credits (windows) for each LPAR based on the number of cores configured in that system. The OS is expected to release credits when cores are removed, and may allocate more when cores are added. So there is a possibility of using excessive credits (windows) in the LPAR and the hypervisor expects the system to close the excessive windows so that NX load can be equally distributed across all LPARs in the system.
When the OS closes the excessive windows in the hypervisor, it sets the window status in-active and invalidates window virtual address mapping. The user space receives paste instruction failure if any NX requests are issued on the in-active window.
This patch also adds the notifier for core removal/add to close windows in the hypervisor if the system lost credits (core removal) and reopen windows in the hypervisor when the previously lost credits are available.
--- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 207 +++++++++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 3 + 3 files changed, 204 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/vas.h +2 -0
@@ -34,6 +34,8 @@*/ #define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ /* vas mmap() */ +/* Window is closed in the hypervisor due to lost credit */ +#define VAS_WIN_NO_CRED_CLOSE 0x00000001/* * Get/Set bit fields
arch/powerpc/platforms/pseries/vas.c +199 -8
244 lines not on this page · raw message
arch/powerpc/platforms/pseries/vas.h +3 -0
@@ -83,6 +83,9 @@ struct vas_cop_feat_caps {struct vas_caps {struct vas_cop_feat_caps caps;struct list_head list;/* List of open windows */ +int nr_close_wins;/* closed windows in the hypervisor for DLPAR */ +int nr_open_windows;/* Number of successful open windows */ +u8 feat;/* Feature type */ };/*
Signature · 1 line
2.27.0
Nicholas Piggin
reviewerRe: [PATCH v4 6/9] powerpc/pseries/vas: Close windows with DLPAR core removal
Excerpts from Haren Myneni's message of February 20, 2022 6:00 am:
Quoted 9 lines
The hypervisor assigns vas credits (windows) for each LPAR based on the number of cores configured in that system. The OS is expected to release credits when cores are removed, and may allocate more when cores are added. So there is a possibility of using excessive credits (windows) in the LPAR and the hypervisor expects the system to close the excessive windows so that NX load can be equally distributed across all LPARs in the system.
Quoted 4 lines
When the OS closes the excessive windows in the hypervisor, it sets the window status in-active and invalidates window virtual address mapping. The user space receives paste instruction failure if any NX requests are issued on the in-active window.
Thanks for adding this paragraph. Then presumably userspace can update their windows and be able to re-try with an available open window?
in-active can be one word, not hyphenated.
Quoted 18 lines
This patch also adds the notifier for core removal/add to close windows in the hypervisor if the system lost credits (core removal) and reopen windows in the hypervisor when the previously lost credits are available.
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 207 +++++++++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 3 + 3 files changed, 204 insertions(+), 8 deletions(-)arch/powerpc/include/asm/vas.h +0
-0
arch/powerpc/include/asm/vas.h line 38
Quoted 5 lines
arch/powerpc/include/asm/vas.h +2
-0@@ -34,6 +34,8 @@*/ #define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ /* vas mmap() */ +/* Window is closed in the hypervisor due to lost credit */ +#define VAS_WIN_NO_CRED_CLOSE 0x00000001
I thought we were getting a different status for software status vs status rturned by hypervisor?
arch/powerpc/platforms/pseries/vas.h struct vas_cop_feat_caps { · line 89
Quoted 12 lines
arch/powerpc/platforms/pseries/vas.h +3
-0@@ -83,6 +83,9 @@ struct vas_cop_feat_caps {struct vas_caps {struct vas_cop_feat_caps caps;struct list_head list;/* List of open windows */ +int nr_close_wins;/* closed windows in the hypervisor for DLPAR */ +int nr_open_windows;/* Number of successful open windows */ +u8 feat;/* Feature type */ };
Still not entirely sold on the idea that nr_open_windows is a feature or capability, but if the code works out easier this way, sometimes these little hacks are reasonable.
Thanks,
Nick
Re: [PATCH v4 6/9] powerpc/pseries/vas: Close windows with DLPAR core removal
Quoted 9 lines · Nicholas Piggin
Excerpts from Haren Myneni's message of February 20, 2022 6:00 am:
Quoted 8 lines
The hypervisor assigns vas credits (windows) for each LPAR based on the number of cores configured in that system. The OS is expected to release credits when cores are removed, and may allocate more when cores are added. So there is a possibility of using excessive credits (windows) in the LPAR and the hypervisor expects the system to close the excessive windows so that NX load can be equally distributed across all LPARs in the system.
Quoted 8 lines
Quoted 4 lines
When the OS closes the excessive windows in the hypervisor, it sets the window status in-active and invalidates window virtual address mapping. The user space receives paste instruction failure if any NX requests are issued on the in-active window.
Thanks for adding this paragraph. Then presumably userspace can update their windows and be able to re-try with an available open window?
yes, the user space should be able to manage with the available open windows or fall back to SW compression if can. Added this comment in the fault hanlder patch.
Quoted 17 lines
in-active can be one word, not hyphenated.
Quoted 13 lines
This patch also adds the notifier for core removal/add to close windows in the hypervisor if the system lost credits (core removal) and reopen windows in the hypervisor when the previously lost credits are available.
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 207 +++++++++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 3 + 3 files changed, 204 insertions(+), 8 deletions(-)
Quoted 15 lines
arch/powerpc/include/asm/vas.h line 38
Quoted 12 lines
arch/powerpc/include/asm/vas.h +0
-0@@ -34,6 +34,8 @@*/ #define VAS_WIN_ACTIVE 0x0/* Used in platform independent */
arch/powerpc/include/asm/vas.h +2
-0/* vas mmap() */ +/* Window is closed in the hypervisor due to lost credit */ +#define VAS_WIN_NO_CRED_CLOSE 0x00000001I thought we were getting a different status for software status vs status rturned by hypervisor?
vas_window->status is only for Linux status bits which are used for active, DLPAR close or migration. We do not need status returned from hypervisor right now. In case if needed in future, hv_status will be added in pseries_vas_window.
Quoted 12 lines
Quoted 12 lines
arch/powerpc/platforms/pseries/vas.h +1
-0@@ -83,6 +83,9 @@ struct vas_cop_feat_caps {struct vas_caps {struct vas_cop_feat_caps caps;struct list_head list;/* List of open windows */ +int nr_close_wins;/* closed windows in the hypervisor for DLPAR */
Quoted 8 lines
arch/powerpc/platforms/pseries/vas.h struct vas_cop_feat_caps { · line 89
Quoted 4 lines
arch/powerpc/platforms/pseries/vas.h +1
-0+int nr_open_windows;/* Number of successful open windows */
arch/powerpc/platforms/pseries/vas.h +1
-0+u8 feat;/* Feature type */ };Still not entirely sold on the idea that
nr_open_windowsis a feature or capability, but if the code works out easier this way, sometimes these little hacks are reasonable.
nr_close_wins / nr_open_windows - not a capability or feature, but these are used to track active windows and needed for DLPAR / migration
- Means total number of open windows and the actual number of windows closed in the hypervisor. hence I did not add these elements in vas_cop_feat_caps struct.
Thanks
Haren
Quoted 3 lines
Thanks,
Nick
PATCH v4 7/9 powerpc/pseries/vas: Reopen windows with DLPAR core add
In reply to Haren MyneniVAS windows can be closed in the hypervisor due to lost credits when the core is removed and the kernel gets fault for NX requests on these in-active windows. If these credits are available later for core add, reopen these windows and set them active. When the OS sees page faults on these active windows, it creates mapping on the new paste address. Then the user space can continue to use these windows and send HW compression requests to NX successfully.
--- arch/powerpc/platforms/pseries/vas.c | 91 +++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/vas.c +90 -1
@@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type,return 0 ; } +/* +* VAS windows can be closed due to lost credits when the core is +* removed. So reopen them if credits are available due to DLPAR +* core add and set the window active status. When NX sees the page +* fault on the unmapped paste address, the kernel handles the fault +* by setting the remapping to new paste address if the window is +* active. +*/ +static int reconfig_open_windows (struct vas_caps *vcaps,int creds) +{ +long domain[PLPAR_HCALL9_BUFSIZE ] = {VAS_DEFAULT_DOMAIN_ID }; +struct vas_cop_feat_caps *caps = &vcaps->caps; +struct pseries_vas_window *win =NULL , *tmp; +int rc, mv_ents =0 ; + +/* +* Nothing to do if there are no closed windows. +*/ +if (!vcaps->nr_close_wins) +return 0 ; + +/* +* For the core removal, the hypervisor reduces the credits +* assigned to the LPAR and the kernel closes VAS windows +* in the hypervisor depends on reduced credits. The kernel +* uses LIFO (the last windows that are opened will be closed +* first) and expects to open in the same order when credits +* are available. +* For example, 40 windows are closed when the LPAR lost 2 cores +* (dedicated). If 1 core is added, this LPAR can have 20 more +* credits. It means the kernel can reopen 20 windows. So move +* 20 entries in the VAS windows lost and reopen next 20 windows. +*/ +if (vcaps->nr_close_wins > creds) + mv_ents = vcaps->nr_close_wins - creds; + +list_for_each_entry_safe (win, tmp, &vcaps->list, win_list) { +if (!mv_ents) +break ; + + mv_ents--; + } + +list_for_each_entry_safe_from (win, tmp, &vcaps->list, win_list) { +/* +* Nothing to do on this window if it is not closed +* with VAS_WIN_NO_CRED_CLOSE +*/ +if (!(win->vas_win.status &VAS_WIN_NO_CRED_CLOSE )) +continue ; + + rc =allocate_setup_window (win, (u64 *)&domain[0 ], + caps->win_type); +if (rc) +return rc; + + rc =h_modify_vas_window (win); +if (rc) + goto out; + +mutex_lock (&win->vas_win.task_ref.mmap_mutex); +/* +* Set window status to active +*/ + win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE ; +mutex_unlock (&win->vas_win.task_ref.mmap_mutex); + win->win_type = caps->win_type; +if (!--vcaps->nr_close_wins) +break ; + } + +return 0 ; +out: +/* +* Window modify HCALL failed. So close the window to the +* hypervisor and return. +*/ +free_irq_setup (win); +h_deallocate_vas_window (win->vas_win.winid); +return rc; +} +/* * The hypervisor reduces the available credits if the LPAR lost core. It * means the excessive windows should not be active and the user space
@@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)* closed / reopened. Hold the vas_pseries_mutex so that the * the user space can not open new windows. */ -if (old_nr_creds > new_nr_creds) { +if (old_nr_creds < new_nr_creds) { +/* +* If the existing target credits is less than the new +* target, reopen windows if they are closed due to +* the previous DLPAR (core removal). +*/ + rc =reconfig_open_windows (vcaps, new_nr_creds - old_nr_creds); + }else {/* * # active windows is more than new LPAR available * credits. So close the excessive windows.
Signature · 1 line
2.27.0
Nicholas Piggin
reviewerRe: [PATCH v4 7/9] powerpc/pseries/vas: Reopen windows with DLPAR core add
Excerpts from Haren Myneni's message of February 20, 2022 6:01 am:
Quoted 9 lines
VAS windows can be closed in the hypervisor due to lost credits when the core is removed and the kernel gets fault for NX requests on these in-active windows. If these credits are available later for core add, reopen these windows and set them active. When the OS sees page faults on these active windows, it creates mapping on the new paste address. Then the user space can continue to use these windows and send HW compression requests to NX successfully.
Just for my own ignorance, what happens if userspace does not get another page fault on that window? Presumably when it gets a page fault it changes to an available window and doesn't just keep re-trying. So in what situation does it attempt to re-access a faulting window?
Thanks,
Nick
Quoted 120 lines
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/vas.c | 91 +++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/vas.c +90
-1@@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type,return 0 ; } +/* +* VAS windows can be closed due to lost credits when the core is +* removed. So reopen them if credits are available due to DLPAR +* core add and set the window active status. When NX sees the page +* fault on the unmapped paste address, the kernel handles the fault +* by setting the remapping to new paste address if the window is +* active. +*/ +static int reconfig_open_windows (struct vas_caps *vcaps,int creds) +{ +long domain[PLPAR_HCALL9_BUFSIZE ] = {VAS_DEFAULT_DOMAIN_ID }; +struct vas_cop_feat_caps *caps = &vcaps->caps; +struct pseries_vas_window *win =NULL , *tmp; +int rc, mv_ents =0 ; + +/* +* Nothing to do if there are no closed windows. +*/ +if (!vcaps->nr_close_wins) +return 0 ; + +/* +* For the core removal, the hypervisor reduces the credits +* assigned to the LPAR and the kernel closes VAS windows +* in the hypervisor depends on reduced credits. The kernel +* uses LIFO (the last windows that are opened will be closed +* first) and expects to open in the same order when credits +* are available. +* For example, 40 windows are closed when the LPAR lost 2 cores +* (dedicated). If 1 core is added, this LPAR can have 20 more +* credits. It means the kernel can reopen 20 windows. So move +* 20 entries in the VAS windows lost and reopen next 20 windows. +*/ +if (vcaps->nr_close_wins > creds) + mv_ents = vcaps->nr_close_wins - creds; + +list_for_each_entry_safe (win, tmp, &vcaps->list, win_list) { +if (!mv_ents) +break ; + + mv_ents--; + } + +list_for_each_entry_safe_from (win, tmp, &vcaps->list, win_list) { +/* +* Nothing to do on this window if it is not closed +* with VAS_WIN_NO_CRED_CLOSE +*/ +if (!(win->vas_win.status &VAS_WIN_NO_CRED_CLOSE )) +continue ; + + rc =allocate_setup_window (win, (u64 *)&domain[0 ], + caps->win_type); +if (rc) +return rc; + + rc =h_modify_vas_window (win); +if (rc) + goto out; + +mutex_lock (&win->vas_win.task_ref.mmap_mutex); +/* +* Set window status to active +*/ + win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE ; +mutex_unlock (&win->vas_win.task_ref.mmap_mutex); + win->win_type = caps->win_type; +if (!--vcaps->nr_close_wins) +break ; + } + +return 0 ; +out: +/* +* Window modify HCALL failed. So close the window to the +* hypervisor and return. +*/ +free_irq_setup (win); +h_deallocate_vas_window (win->vas_win.winid); +return rc; +} +/* * The hypervisor reduces the available credits if the LPAR lost core. It * means the excessive windows should not be active and the user space @@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)* closed / reopened. Hold the vas_pseries_mutex so that the * the user space can not open new windows. */ -if (old_nr_creds > new_nr_creds) { +if (old_nr_creds < new_nr_creds) { +/* +* If the existing target credits is less than the new +* target, reopen windows if they are closed due to +* the previous DLPAR (core removal). +*/ + rc =reconfig_open_windows (vcaps, new_nr_creds - old_nr_creds); + }else {/* * # active windows is more than new LPAR available * credits. So close the excessive windows. Signature · 1 line
2.27.0
Re: [PATCH v4 7/9] powerpc/pseries/vas: Reopen windows with DLPAR core add
Quoted 10 lines · Nicholas Piggin
Excerpts from Haren Myneni's message of February 20, 2022 6:01 am:
Quoted 8 lines
VAS windows can be closed in the hypervisor due to lost credits when the core is removed and the kernel gets fault for NX requests on these in-active windows. If these credits are available later for core add, reopen these windows and set them active. When the OS sees page faults on these active windows, it creates mapping on the new paste address. Then the user space can continue to use these windows and send HW compression requests to NX successfully.
Quoted 5 lines
Just for my own ignorance, what happens if userspace does not get another page fault on that window? Presumably when it gets a page fault it changes to an available window and doesn't just keep re-trying. So in what situation does it attempt to re-access a faulting window?
We should except faults only when the user space issue NX request on these inactive windows.
Example:
- window is closed in the hypervisor to the DLPAR core removal. Means inactive in the OS.
- window is opened later when the previous credit is available - means active window in the OS.
- If the user space issue copy/paste on this window and fault on paste address:
- gets paste failure if the window is inactive
- map to the new paste address if the window is active and returns
So in case if the user space does not retry, new mapping will not be done. From the OS point of view, this window is not used by the user space.
Thanks
Haren
Quoted 135 lines
Thanks,
NickQuoted 131 lines
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/vas.c | 91 +++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/vas.c +4
-0@@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, return 0; } +/* +* VAS windows can be closed due to lost credits when the core is +* removed. So reopen them if credits are available due to DLPAR +* core add and set the window active status. When NX sees the page
+ * fault on the unmapped paste address, the kernel handles the fault
+ * by setting the remapping to new paste address if the window is
+ * active.
+ */
+static intreconfig_open_windows(struct vas_caps *vcaps, int creds)arch/powerpc/platforms/pseries/vas.c +20
-0+{ +long domain[PLPAR_HCALL9_BUFSIZE ] = {VAS_DEFAULT_DOMAIN_ID }; +struct vas_cop_feat_caps *caps = &vcaps->caps; +struct pseries_vas_window *win =NULL , *tmp; +int rc, mv_ents =0 ; + +/* +* Nothing to do if there are no closed windows. +*/ +if (!vcaps->nr_close_wins) +return 0 ; + +/* +* For the core removal, the hypervisor reduces the credits +* assigned to the LPAR and the kernel closes VAS windows +* in the hypervisor depends on reduced credits. The kernel +* uses LIFO (the last windows that are opened will be closed +* first) and expects to open in the same order when credits +* are available. +* For example, 40 windows are closed when the LPAR lost 2 cores
arch/powerpc/platforms/pseries/vas.c +3
-0+* (dedicated). If 1 core is added, this LPAR can have 20 more +* credits. It means the kernel can reopen 20 windows. So move +* 20 entries in the VAS windows lost and reopen next 20 windows.
arch/powerpc/platforms/pseries/vas.c +12
-0+*/ +if (vcaps->nr_close_wins > creds) + mv_ents = vcaps->nr_close_wins - creds; + +list_for_each_entry_safe (win, tmp, &vcaps->list, win_list) { +if (!mv_ents) +break ; + + mv_ents--; + } + +list_for_each_entry_safe_from (win, tmp, &vcaps->list, win_list){
arch/powerpc/platforms/pseries/vas.c +38
-0+/* +* Nothing to do on this window if it is not closed +* with VAS_WIN_NO_CRED_CLOSE +*/ +if (!(win->vas_win.status &VAS_WIN_NO_CRED_CLOSE )) +continue ; + + rc =allocate_setup_window (win, (u64 *)&domain[0 ], + caps->win_type); +if (rc) +return rc; + + rc =h_modify_vas_window (win); +if (rc) + goto out; + +mutex_lock (&win->vas_win.task_ref.mmap_mutex); +/* +* Set window status to active +*/ + win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE ; +mutex_unlock (&win->vas_win.task_ref.mmap_mutex); + win->win_type = caps->win_type; +if (!--vcaps->nr_close_wins) +break ; + } + +return 0 ; +out: +/* +* Window modify HCALL failed. So close the window to the +* hypervisor and return. +*/ +free_irq_setup (win); +h_deallocate_vas_window (win->vas_win.winid); +return rc; +} +/* * The hypervisor reduces the available credits if the LPAR lost core. It
* means the excessive windows should not be active and the user
spacearch/powerpc/platforms/pseries/vas.c +8
-1@@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)* closed / reopened. Hold the vas_pseries_mutex so that the * the user space can not open new windows. */ -if (old_nr_creds > new_nr_creds) { +if (old_nr_creds < new_nr_creds) { +/* +* If the existing target credits is less than the new +* target, reopen windows if they are closed due to +* the previous DLPAR (core removal). +*/ + rc =reconfig_open_windows (vcaps, new_nr_creds - old_nr_creds); + }else {/* * # active windows is more than new LPAR available * credits. So close the excessive windows.
Signature · 1 line
2.27.0
PATCH v4 8/9 powerpc/pseries/vas: sysfs interface to export capabilities
In reply to Haren MyneniThe hypervisor provides the available VAS GZIP capabilities such as default or QoS window type and the target available credits in each type. This patch creates sysfs entries and exports the target, used and the available credits for each feature.
This interface can be used by the user space to determine the credits usage or to set the target credits in the case of QoS type (for DLPAR).
/sys/devices/vas/vas0/gzip/default_capabilities (default GZIP capabilities)
nr_total_credits /* Total credits available. Can be
/* changed with DLPAR operation */
nr_used_credits /* Used credits */
/sys/devices/vas/vas0/gzip/qos_capabilities (QoS GZIP capabilities)
nr_total_credits
nr_used_credits
--- arch/powerpc/platforms/pseries/Makefile | 2 +- arch/powerpc/platforms/pseries/vas-sysfs.c | 226 +++++++++++++++++++++ arch/powerpc/platforms/pseries/vas.c | 6 + arch/powerpc/platforms/pseries/vas.h | 6 + 4 files changed, 239 insertions(+), 1 deletion(-) create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
arch/powerpc/platforms/pseries/Makefile +1 -1
@@ -29,6 +29,6 @@ obj-$(CONFIG_PPC_SVM) += svm.o obj-$(CONFIG_FA_DUMP) += rtas-fadump.o obj-$(CONFIG_SUSPEND) += suspend.o -obj-$(CONFIG_PPC_VAS) += vas.o +obj-$(CONFIG_PPC_VAS) += vas.o vas-sysfs.o obj-$(CONFIG_ARCH_HAS_CC_PLATFORM) += cc_platform.o
arch/powerpc/platforms/pseries/vas-sysfs.c new file +226 -0
226 lines not on this page · raw message
arch/powerpc/platforms/pseries/vas.c +6 -0
@@ -560,6 +560,10 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, } } + rc =sysfs_add_vas_caps (caps); +if (rc) +return rc; + copypaste_feat = true;return 0 ;
@@ -844,6 +848,8 @@ static int __init pseries_vas_init(void) caps_all.descriptor =be64_to_cpu (hv_caps->descriptor); caps_all.feat_type =be64_to_cpu (hv_caps->feat_type); +sysfs_pseries_vas_init (&caps_all); + hv_cop_caps =kmalloc (sizeof (*hv_cop_caps),GFP_KERNEL );if (!hv_cop_caps) { rc = -ENOMEM ;
arch/powerpc/platforms/pseries/vas.h +6 -0
@@ -30,6 +30,9 @@#define VAS_COPY_PASTE_USER_MODE 0x00000001#define VAS_COP_OP_USER_MODE 0x00000010 +#define VAS_GZIP_QOS_CAPABILITIES 0x56516F73477A6970 +#define VAS_GZIP_DEFAULT_CAPABILITIES 0x56446566477A6970 +/* * Co-processor feature - GZIP QoS windows or GZIP default windows */
@@ -125,4 +128,7 @@ struct pseries_vas_window {char *name; int fault_virq; }; + +int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps); +int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps);#endif /* _VAS_H */
Signature · 1 line
2.27.0
Re: [PATCH v4 8/9] powerpc/pseries/vas: sysfs interface to export capabilities
Excerpts from Haren Myneni's message of February 20, 2022 6:01 am:
Quoted 9 lines
The hypervisor provides the available VAS GZIP capabilities such as default or QoS window type and the target available credits in each type. This patch creates sysfs entries and exports the target, used and the available credits for each feature.
This interface can be used by the user space to determine the credits usage or to set the target credits in the case of QoS type (for DLPAR).
Quoted 9 lines
/sys/devices/vas/vas0/gzip/default_capabilities(default GZIP capabilities)
nr_total_credits/* Total credits available. Can be
/* changed with DLPAR operation */
nr_used_credits/* Used credits */
/sys/devices/vas/vas0/gzip/qos_capabilities(QoS GZIP capabilities)
nr_total_credits
nr_used_credits
Looks good, thanks
Quoted 304 lines
Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/Makefile | 2 +- arch/powerpc/platforms/pseries/vas-sysfs.c | 226 +++++++++++++++++++++ arch/powerpc/platforms/pseries/vas.c | 6 + arch/powerpc/platforms/pseries/vas.h | 6 + 4 files changed, 239 insertions(+), 1 deletion(-) create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.carch/powerpc/platforms/pseries/Makefile +1
-1@@ -29,6 +29,6 @@ obj-$(CONFIG_PPC_SVM) += svm.o obj-$(CONFIG_FA_DUMP) += rtas-fadump.o obj-$(CONFIG_SUSPEND) += suspend.o -obj-$(CONFIG_PPC_VAS) += vas.o +obj-$(CONFIG_PPC_VAS) += vas.o vas-sysfs.o obj-$(CONFIG_ARCH_HAS_CC_PLATFORM) += cc_platform.oarch/powerpc/platforms/pseries/vas-sysfs.c new file +226
-0@@ -0,0 +1,226 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright 2022-23 IBM Corp. +*/ + +#define pr_fmt (fmt) "vas: " fmt + +#include <linux/module.h> +#include <linux/kernel.h> +#include <linux/miscdevice.h> +#include <linux/kobject.h> +#include <linux/slab.h> +#include <linux/mm.h> + +#include "vas.h" + +#ifdef CONFIG_SYSFS +static struct kobject *pseries_vas_kobj; +static struct kobject *gzip_caps_kobj; + +struct vas_caps_entry { +struct kobject kobj; +struct vas_cop_feat_caps *caps; +}; + +#define to_caps_entry (entry) container_of(entry, struct vas_caps_entry, kobj) + +#define sysfs_caps_entry_read (_name) \ +static ssize_t _name##_show(struct vas_cop_feat_caps *caps, char *buf) \ +{ \ + return sprintf(buf, "%d\n", atomic_read(&caps->_name)); \ +} + +struct vas_sysfs_entry { +struct attribute attr; +ssize_t (*show)(struct vas_cop_feat_caps *,char *); +ssize_t (*store)(struct vas_cop_feat_caps *,const char *,size_t ); +}; + +#define VAS_ATTR_RO (_name) \ + sysfs_caps_entry_read(_name); \ + static struct vas_sysfs_entry _name##_attribute = __ATTR(_name, \ + 0444, _name##_show, NULL); + +/* +* Create sysfs interface: +* /sys/devices/vas/vas0/gzip/default_capabilities +* This directory contains the following VAS GZIP capabilities +* for the defaule credit type. +* /sys/devices/vas/vas0/gzip/default_capabilities/nr_total_credits +* Total number of default credits assigned to the LPAR which +* can be changed with DLPAR operation. +* /sys/devices/vas/vas0/gzip/default_capabilities/nr_used_credits +* Number of credits used by the user space. One credit will +* be assigned for each window open. +* +* /sys/devices/vas/vas0/gzip/qos_capabilities +* This directory contains the following VAS GZIP capabilities +* for the Quality of Service (QoS) credit type. +* /sys/devices/vas/vas0/gzip/qos_capabilities/nr_total_credits +* Total number of QoS credits assigned to the LPAR. The user +* has to define this value using HMC interface. It can be +* changed dynamically by the user. +* /sys/devices/vas/vas0/gzip/qos_capabilities/nr_used_credits +* Number of credits used by the user space. +*/ + +VAS_ATTR_RO (nr_total_credits); +VAS_ATTR_RO (nr_used_credits); + +static struct attribute *vas_capab_attrs[] = { + &nr_total_credits_attribute.attr, + &nr_used_credits_attribute.attr, +NULL , +}; + +static ssize_t vas_type_show (struct kobject *kobj,struct attribute *attr, +char *buf) +{ +struct vas_caps_entry *centry; +struct vas_cop_feat_caps *caps; +struct vas_sysfs_entry *entry; + + centry =to_caps_entry (kobj); + caps = centry->caps; + entry = container_of(attr, struct vas_sysfs_entry, attr); + +if (!entry->show) +return -EIO ; + +return entry->show (caps, buf); +} + +static ssize_t vas_type_store (struct kobject *kobj,struct attribute *attr, +const char *buf,size_t count) +{ +struct vas_caps_entry *centry; +struct vas_cop_feat_caps *caps; +struct vas_sysfs_entry *entry; + + centry =to_caps_entry (kobj); + caps = centry->caps; + entry = container_of(attr, struct vas_sysfs_entry, attr); +if (!entry->store) +return -EIO ; + +return entry->store (caps, buf, count); +} + +static void vas_type_release (struct kobject *kobj) +{ +struct vas_caps_entry *centry =to_caps_entry (kobj); +kfree (centry); +} + +static const struct sysfs_ops vas_sysfs_ops = { + .show = vas_type_show, + .store = vas_type_store, +}; + +static struct kobj_type vas_attr_type = { + .release = vas_type_release, + .sysfs_ops = &vas_sysfs_ops, + .default_attrs = vas_capab_attrs, +}; + +static char *vas_caps_kobj_name (struct vas_cop_feat_caps *caps, +struct kobject **kobj) +{ +if (caps->descriptor ==VAS_GZIP_QOS_CAPABILITIES ) { + *kobj = gzip_caps_kobj; +return "qos_capabilities" ; + }else if (caps->descriptor ==VAS_GZIP_DEFAULT_CAPABILITIES ) { + *kobj = gzip_caps_kobj; +return "default_capabilities" ; + }else +return "Unknown" ; +} + +/* +* Add feature specific capability dir entry. +* Ex: VDefGzip or VQosGzip +*/ +int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps) +{ +struct vas_caps_entry *centry; +struct kobject *kobj =NULL ; +int ret =0 ; +char *name; + + centry =kzalloc (sizeof (*centry),GFP_KERNEL ); +if (!centry) +return -ENOMEM ; + +kobject_init (¢ry->kobj, &vas_attr_type); + centry->caps = caps; + name =vas_caps_kobj_name (caps, &kobj); + +if (kobj) { + ret =kobject_add (¢ry->kobj, kobj,"%s" , name); + +if (ret) { +pr_err ("VAS: sysfs kobject add / event failed %d\n" , + ret); +kobject_put (¢ry->kobj); + } + } + +return ret; +} + +static struct miscdevice vas_miscdev = { + .minor =MISC_DYNAMIC_MINOR , + .name ="vas" , +}; + +/* +* Add VAS and VasCaps (overall capabilities) dir entries. +*/ +int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps) +{ +int ret; + + ret =misc_register (&vas_miscdev); +if (ret <0 ) { +pr_err ("%s: register vas misc device failed\n" , __func__); +return ret; + } + +/* +* The hypervisor does not expose multiple VAS instances, but can +* see multiple VAS instances on PowerNV. So create 'vas0' directory +* on pseries. +*/ + pseries_vas_kobj =kobject_create_and_add ("vas0" , + &vas_miscdev.this_device->kobj); +if (!pseries_vas_kobj) { +pr_err ("Failed to create VAS sysfs entry\n" ); +return -ENOMEM ; + } + +if ((vas_caps->feat_type &VAS_GZIP_QOS_FEAT_BIT ) || + (vas_caps->feat_type &VAS_GZIP_DEF_FEAT_BIT )) { + gzip_caps_kobj =kobject_create_and_add ("gzip" , + pseries_vas_kobj); +if (!gzip_caps_kobj) { +pr_err ("Failed to create VAS GZIP capability entry\n" ); +kobject_put (pseries_vas_kobj); +return -ENOMEM ; + } + } + +return 0 ; +} + +#else +int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps) +{ +return 0 ; +} + +int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps) +{ +return 0 ; +} +#endif arch/powerpc/platforms/pseries/vas.c +6
-0@@ -560,6 +560,10 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, } } + rc =sysfs_add_vas_caps (caps); +if (rc) +return rc; + copypaste_feat = true;return 0 ;@@ -844,6 +848,8 @@ static int __init pseries_vas_init(void) caps_all.descriptor =be64_to_cpu (hv_caps->descriptor); caps_all.feat_type =be64_to_cpu (hv_caps->feat_type); +sysfs_pseries_vas_init (&caps_all); + hv_cop_caps =kmalloc (sizeof (*hv_cop_caps),GFP_KERNEL );if (!hv_cop_caps) { rc = -ENOMEM ;arch/powerpc/platforms/pseries/vas.h +6
-0@@ -30,6 +30,9 @@#define VAS_COPY_PASTE_USER_MODE 0x00000001#define VAS_COP_OP_USER_MODE 0x00000010 +#define VAS_GZIP_QOS_CAPABILITIES 0x56516F73477A6970 +#define VAS_GZIP_DEFAULT_CAPABILITIES 0x56446566477A6970 +/* * Co-processor feature - GZIP QoS windows or GZIP default windows */ @@ -125,4 +128,7 @@ struct pseries_vas_window {char *name; int fault_virq; }; + +int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps); +int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps);#endif /* _VAS_H */ Signature · 1 line
2.27.0
PATCH v4 9/9 powerpc/pseries/vas: Write 'nr_total_credits' for QoS credits change
In reply to Haren Mynenipseries supports two types of credits - Default (uses normal priority FIFO) and Qality of service (QoS uses high priority FIFO). The user decides the number of QoS credits and sets this value with HMC interface. With the core add/removal, this value can be changed in HMC which invokes drmgr to communicate to the kernel.
This patch adds an interface so that drmgr command can write the new target QoS credits in sysfs. But the kernel gets the new QoS capabilities from the hypervisor whenever nr_total_credits is updated to make sure sync with the values in the hypervisor.
--- arch/powerpc/platforms/pseries/vas-sysfs.c | 33 +++++++++++++++++++++- arch/powerpc/platforms/pseries/vas.c | 2 +- arch/powerpc/platforms/pseries/vas.h | 1 + 3 files changed, 34 insertions(+), 2 deletions(-)
arch/powerpc/platforms/pseries/vas-sysfs.c +32 -1
@@ -25,6 +25,33 @@ struct vas_caps_entry {#define to_caps_entry (entry) container_of(entry, struct vas_caps_entry, kobj) +/* +* This function is used to get the notification from the drmgr when +* QoS credits are changed. Though receiving the target total QoS +* credits here, get the official QoS capabilities from the hypervisor. +*/ +static ssize_t nr_total_credits_store (struct vas_cop_feat_caps *caps, +const char *buf,size_t count) +{ +int err; +u16 creds; + +/* +* Nothing to do for default credit type. +*/ +if (caps->win_type ==VAS_GZIP_DEF_FEAT_TYPE ) +return -EOPNOTSUPP ; + + err =kstrtou16 (buf,0 , &creds); +if (!err) + err =vas_reconfig_capabilties (caps->win_type); + +if (err) +return -EINVAL ; + +return count; +} +#define sysfs_caps_entry_read (_name) \ static ssize_t _name##_show(struct vas_cop_feat_caps *caps, char *buf) \ { \
@@ -41,6 +68,10 @@ struct vas_sysfs_entry {sysfs_caps_entry_read (_name); \ static struct vas_sysfs_entry _name##_attribute = __ATTR(_name, \ 0444, _name##_show, NULL); +#define VAS_ATTR (_name) \ + sysfs_caps_entry_read(_name); \ + static struct vas_sysfs_entry _name##_attribute = __ATTR(_name, \ + 0644, _name##_show, _name##_store)/* * Create sysfs interface:
@@ -65,7 +96,7 @@ struct vas_sysfs_entry {* Number of credits used by the user space. */ -VAS_ATTR_RO (nr_total_credits); +VAS_ATTR (nr_total_credits);VAS_ATTR_RO (nr_used_credits);static struct attribute *vas_capab_attrs[] = {
arch/powerpc/platforms/pseries/vas.c +1 -1
@@ -722,7 +722,7 @@ static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds)* changes. Reconfig window configurations based on the credits * availability from this new capabilities. */ -static int vas_reconfig_capabilties (u8 type) +int vas_reconfig_capabilties (u8 type) {struct hv_vas_cop_feat_caps *hv_caps;struct vas_cop_feat_caps *caps;
arch/powerpc/platforms/pseries/vas.h +1 -0
@@ -130,5 +130,6 @@ struct pseries_vas_window { };int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps); +int vas_reconfig_capabilties (u8 type);int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps);#endif /* _VAS_H */
Signature · 1 line
2.27.0
Nicholas Piggin
reviewerRe: [PATCH v4 9/9] powerpc/pseries/vas: Write 'nr_total_credits' for QoS credits change
Excerpts from Haren Myneni's message of February 20, 2022 6:03 am:
Quoted 37 lines
pseries supports two types of credits - Default (uses normal priority FIFO) and Qality of service (QoS uses high priority FIFO). The user decides the number of QoS credits and sets this value with HMC interface. With the core add/removal, this value can be changed in HMC which invokes drmgr to communicate to the kernel.
This patch adds an interface so that drmgr command can write the new target QoS credits in sysfs. But the kernel gets the new QoS capabilities from the hypervisor whenever
nr_total_creditsis updated to make sure sync with the values in the hypervisor.Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/vas-sysfs.c | 33 +++++++++++++++++++++- arch/powerpc/platforms/pseries/vas.c | 2 +- arch/powerpc/platforms/pseries/vas.h | 1 + 3 files changed, 34 insertions(+), 2 deletions(-)arch/powerpc/platforms/pseries/vas-sysfs.c +10
-0@@ -25,6 +25,33 @@ struct vas_caps_entry {#define to_caps_entry (entry) container_of(entry, struct vas_caps_entry, kobj) +/* +* This function is used to get the notification from the drmgr when +* QoS credits are changed. Though receiving the target total QoS +* credits here, get the official QoS capabilities from the hypervisor. +*/ +static ssize_t nr_total_credits_store (struct vas_cop_feat_caps *caps, +const char *buf,size_t count) +{ +int err; +u16 creds;
arch/powerpc/platforms/pseries/vas-sysfs.c struct vas_caps_entry { · line 47
Quoted 10 lines
arch/powerpc/platforms/pseries/vas-sysfs.c +10
-0+ +/* +* Nothing to do for default credit type. +*/ +if (caps->win_type ==VAS_GZIP_DEF_FEAT_TYPE ) +return -EOPNOTSUPP ; + + err =kstrtou16 (buf,0 , &creds); +if (!err) + err =vas_reconfig_capabilties (caps->win_type);
So what's happening here? The creds value is ignored? Can it just be a write-only file which is named appropriately to indicate it can be written-to to trigger an update?
Thanks,
Nick
Re: [PATCH v4 9/9] powerpc/pseries/vas: Write 'nr_total_credits' for QoS credits change
Quoted 53 lines · Nicholas Piggin
Excerpts from Haren Myneni's message of February 20, 2022 6:03 am:
Quoted 52 lines
pseries supports two types of credits - Default (uses normal priority
FIFO) and Qality of service (QoS uses high priority FIFO). The user decides the number of QoS credits and sets this value with HMC interface. With the core add/removal, this value can be changed in HMC
which invokes drmgr to communicate to the kernel.This patch adds an interface so that drmgr command can write the new
target QoS credits in sysfs. But the kernel gets the new QoS capabilities from the hypervisor whenevernr_total_creditsis updated
to make sure sync with the values in the hypervisor.Signed-off-by Haren Myneni <[email protected]>--- arch/powerpc/platforms/pseries/vas-sysfs.c | 33 +++++++++++++++++++++- arch/powerpc/platforms/pseries/vas.c | 2 +- arch/powerpc/platforms/pseries/vas.h | 1 + 3 files changed, 34 insertions(+), 2 deletions(-)arch/powerpc/platforms/pseries/vas-sysfs.c +2
-0@@ -25,6 +25,33 @@ struct vas_caps_entry {#define to_caps_entry (entry) container_of(entry, struct vas_caps_entry, kobj) +/* +* This function is used to get the notification from the drmgr when
+ * QoS credits are changed. Though receiving the target total QoS
+ * credits here, get the official QoS capabilities from the hypervisor.
+ */
+staticssize_tnr_total_credits_store(structvas_cop_feat_caps*caps,arch/powerpc/platforms/pseries/vas-sysfs.c +10
-0+ const char *buf, size_t count) +{ + int err; +u16 creds; + +/* +* Nothing to do for default credit type. +*/ +if (caps->win_type ==VAS_GZIP_DEF_FEAT_TYPE ) +return -EOPNOTSUPP ;
Quoted 8 lines
arch/powerpc/platforms/pseries/vas-sysfs.c struct vas_caps_entry { · line 47
Quoted 4 lines
arch/powerpc/platforms/pseries/vas-sysfs.c +4
-0+ + err =kstrtou16 (buf,0 , &creds); +if (!err) + err =vas_reconfig_capabilties (caps->win_type);So what's happening here? The creds value is ignored? Can it just be a write-only file which is named appropriately to indicate it can be written-to to trigger an update?
Yes, new credits value is ignored. When the user changes QoS credits with the HMC interface, it should reflect in QoS capability in the hypervisor. So ignoring the credit value here and get the capability value from the hypervisor.
This file should be read/write - the user space should be able to read the current configured value for both credit types - default and QoS
Can I say nr_total_credits_update?
Thanks
Haren
Quoted 3 lines
Thanks,
Nick