powerpc/pseries/vas: NXGZIP support with DLPAR
v6 cover powerpc/pseries/vas: NXGZIP support with DLPAR
PowerPC provides HW compression with NX coprocessor. This feature is available on both PowerNV and PowerVM and included in Linux. Since each powerpc chip has one NX coprocessor, the VAS introduces the concept of windows / credits to manage access to this hardware resource. On powerVM, these limited resources should be available across all LPARs. So the hypervisor assigns the specific credits to each LPAR based on processor entitlement so that one LPAR does not overload NX. The hypervisor can reject the window open request to a partition if exceeds its credit limit (1 credit per window).
So the total number of target credits in a partition can be changed if the core configuration is modified. The hypervisor expects the partition to modify its window usage depends on new target credits. For example, if the partition uses more credits than the new target credits, it should close the excessive windows so that the NX resource will be available to other partitions.
This patch series enables OS to support this dynamic credit management with DLPAR core removal/add.
Core removal operation:
- Get new VAS capabilities from the hypervisor when the DLPAR notifier is received. This capabilities provides the new target credits based on new processor entitlement. In the case of QoS credit changes, the notification will be issued by updating the target_creds via sysfs.
- If the partition is already used more than the new target credits, the kernel selects windows, unmap the current paste address and close them in the hypervisor, It uses FIFO to identify these windows - last windows that are opened are the first ones to be closed.
- When the user space issue requests on these windows, NX generates page fault on the unmap paste address. The kernel handles the fault by returning the paste instruction failure if the window is not active (means unmap paste). Then up to the library / user space to fall back to SW compression or manage with the current windows.
Core add operation:
- The kernel can see increased target credits from the new VAS capabilities.
- Scans the window list for the closed windows in the hypervisor due to lost credit before and selects windows based on same FIFO.
- Make these corresponding windows active and create remap with the same VMA on the new paste address in the fault handler.
- Then the user space should expect paste successful later.
Patch 1: Define common names for sysfs target/used/avail_creds so
that same sysfs entries can be used even on PowerNV later.
Patch 2: Save PID in the vas window struct during initial window
open and use it when reopen later.
Patch 3: Add new mmap fault handler which handles the page fault
from NX on paste address.
Patch 4: Return the paste instruction failure if the window is not
active.
Patch 5: If the window is closed in the hypervisor before the user
space issue the initial mmap(), return -EACCES failure.
Patch 6: Close windows in the hypervisor when the partition exceeds
its usage than the new target credits.
Patch 7: When credits are available, reopen windows that are closed
before with core removal.
Patch 8 & 9: The user space determines the credit usage with sysfs
nr_total/nr_used_credits interfaces. drmgr uses update_total_credits to notify OS for QoS credit changes.
Thanks to Nicholas Piggin and Aneesh Kumar for the valuable suggestions on the NXGZIP design to support DLPAR operations.
Changes in v2:
- Rebase on 5.16-rc5
- Use list safe functions to iterate windows list
- Changes to show the actual value in sysfs used_credits even though some windows are inactive with core removal. Reflects -ve value in sysfs avail_creds to let userspace know that it opened more windows than the current maximum LPAR credits.
Changes in v3:
- Rebase on 5.16
- Reconfigure VAS windows only for CPU hotplug events.
Changes in v4:
- Rebase on 5.17-rc4
- Changes based on comments from Nicholas Piggin
- Included VAS DLPAR notifer code in 'Close windows with DLPAR' patch instead of as a separate patch
- Patches reordering and other changes
Changes in v5:
- Rebase on 5.17-rc5
- Add update_total_credits sysfs entry to update QoS target credits and other commit descriptions as suggested by Nicholas Piggin
Changed in v6:
- Build fix in "Add paste address mmap fault handler" patch as reported by kernel test robot <[email protected]>
Haren Myneni (9):
powerpc/pseries/vas: Use common names in VAS capability structure powerpc/pseries/vas: Save PID in pseries_vas_window struct powerpc/vas: Add paste address mmap fault handler powerpc/vas: Return paste instruction failure if no active window powerpc/vas: Map paste address only if window is active powerpc/pseries/vas: Close windows with DLPAR core removal powerpc/pseries/vas: Reopen windows with DLPAR core add powerpc/pseries/vas: sysfs interface to export capabilities powerpc/pseries/vas: Add 'update_total_credits' entry for QoS
capabilities
arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/include/asm/vas.h | 12 + arch/powerpc/platforms/book3s/vas-api.c | 145 +++++++++- arch/powerpc/platforms/pseries/Makefile | 2 +- arch/powerpc/platforms/pseries/vas-sysfs.c | 268 +++++++++++++++++ arch/powerpc/platforms/pseries/vas.c | 321 +++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 16 +- 7 files changed, 744 insertions(+), 22 deletions(-) create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
Signature · 1 line
2.27.0
PATCH v6 1/9 powerpc/pseries/vas: Use common names in VAS capability structure
nr_total/nr_used_credits provides credits usage to user space via sysfs and the same interface can be used on PowerNV in future. Changed with proper naming so that applicable on both pseries and PowerNV.
--- arch/powerpc/platforms/pseries/vas.c | 10 +++++----- arch/powerpc/platforms/pseries/vas.h | 5 ++--- 2 files changed, 7 insertions(+), 8 deletions(-)
arch/powerpc/platforms/pseries/vas.c +5 -5
@@ -310,8 +310,8 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, cop_feat_caps = &caps->caps; -if (atomic_inc_return (&cop_feat_caps->used_lpar_creds) > -atomic_read (&cop_feat_caps->target_lpar_creds)) { +if (atomic_inc_return (&cop_feat_caps->nr_used_credits) > +atomic_read (&cop_feat_caps->nr_total_credits)) {pr_err ("Credits are not available to allocate window\n" ); rc = -EINVAL ; goto out;
@@ -385,7 +385,7 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags,free_irq_setup (txwin);h_deallocate_vas_window (txwin->vas_win.winid); out: -atomic_dec (&cop_feat_caps->used_lpar_creds); +atomic_dec (&cop_feat_caps->nr_used_credits);kfree (txwin);return ERR_PTR (rc); }
@@ -445,7 +445,7 @@ static int vas_deallocate_window(struct vas_window *vwin) }list_del (&win->win_list); -atomic_dec (&caps->used_lpar_creds); +atomic_dec (&caps->nr_used_credits);mutex_unlock (&vas_pseries_mutex);put_vas_user_win_ref (&vwin->task_ref);
@@ -521,7 +521,7 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, } caps->max_lpar_creds =be16_to_cpu (hv_caps->max_lpar_creds); caps->max_win_creds =be16_to_cpu (hv_caps->max_win_creds); -atomic_set (&caps->target_lpar_creds, +atomic_set (&caps->nr_total_credits,be16_to_cpu (hv_caps->target_lpar_creds));if (feat ==VAS_GZIP_DEF_FEAT ) { caps->def_lpar_creds =be16_to_cpu (hv_caps->def_lpar_creds);
arch/powerpc/platforms/pseries/vas.h +2 -3
@@ -72,9 +72,8 @@ struct vas_cop_feat_caps { };/* Total LPAR available credits. Can be different from max LPAR */ /* credits due to DLPAR operation */ -atomic_t target_lpar_creds; -atomic_t used_lpar_creds;/* Used credits so far */ -u16 avail_lpar_creds;/* Remaining available credits */ +atomic_t nr_total_credits;/* Total credits assigned to LPAR */ +atomic_t nr_used_credits;/* Used credits so far */ };/*
Signature · 1 line
2.27.0
PATCH v6 2/9 powerpc/pseries/vas: Save PID in pseries_vas_window struct
In reply to Haren MyneniThe kernel sets the VAS window with PID when it is opened in the hypervisor. During DLPAR operation, windows can be closed and reopened in the hypervisor when the credit is available. So saves this PID in pseries_vas_window struct when the window is opened initially and reuse it later during DLPAR operation.
--- arch/powerpc/platforms/pseries/vas.c | 9 +++++---- arch/powerpc/platforms/pseries/vas.h | 1 + 2 files changed, 6 insertions(+), 4 deletions(-)
arch/powerpc/platforms/pseries/vas.c +5 -4
@@ -107,7 +107,6 @@ static int h_deallocate_vas_window(u64 winid)static int h_modify_vas_window (struct pseries_vas_window *win) {long rc; -u32 lpid =mfspr (SPRN_PID );/* * AMR value is not supported in Linux VAS implementation.
@@ -115,7 +114,7 @@ static int h_modify_vas_window(struct pseries_vas_window *win)*/ do { rc = plpar_hcall_norets(H_MODIFY_VAS_WINDOW, - win->vas_win.winid, lpid, 0, + win->vas_win.winid, win->pid, 0, VAS_MOD_WIN_FLAGS, 0); rc = hcall_return_busy_check(rc);
@@ -124,8 +123,8 @@ static int h_modify_vas_window(struct pseries_vas_window *win)if (rc ==H_SUCCESS )return 0 ; -pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u lpid %u\n" , - rc, win->vas_win.winid, lpid); +pr_err ("H_MODIFY_VAS_WINDOW error: %ld, winid %u pid %u\n" , + rc, win->vas_win.winid, win->pid);return -EIO ; }
@@ -338,6 +337,8 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, } } + txwin->pid =mfspr (SPRN_PID ); +/* * Allocate / Deallocate window hcalls and setup / free IRQs * have to be protected with mutex.
arch/powerpc/platforms/pseries/vas.h +1 -0
@@ -114,6 +114,7 @@ struct pseries_vas_window {u64 domain[6 ];/* Associativity domain Ids */ /* this window is allocated */ u64 util; +u32 pid;/* PID associated with this window */ /* List of windows opened which is used for LPM */ struct list_head win_list;
Signature · 1 line
2.27.0
PATCH v6 3/9 powerpc/vas: Add paste address mmap fault handler
In reply to Haren MyneniThe user space opens VAS windows and issues NX requests by pasting CRB on the corresponding paste address mmap. When the system lost credits due to core removal, the kernel has to close the window in the hypervisor and make the window inactive by unmapping this paste address. Also the OS has to handle NX request page faults if the user space issue NX requests.
This handler maps the new paste address with the same VMA when the window is active again (due to core add with DLPAR). Otherwise returns paste failure.
--- arch/powerpc/include/asm/vas.h | 10 ++++ arch/powerpc/platforms/book3s/vas-api.c | 68 +++++++++++++++++++++++++ 2 files changed, 78 insertions(+)
arch/powerpc/include/asm/vas.h +10 -0
@@ -29,6 +29,12 @@#define VAS_THRESH_FIFO_GT_QTR_FULL 2#define VAS_THRESH_FIFO_GT_EIGHTH_FULL 3 +/* +* VAS window Linux status bits +*/ +#define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ +/* vas mmap() */ +/* * Get/Set bit fields */
@@ -59,6 +65,9 @@ struct vas_user_win_ref {struct pid *pid;/* PID of owner */ struct pid *tgid;/* Thread group ID of owner */ struct mm_struct *mm;/* Linux process mm_struct */ +struct mutex mmap_mutex;/* protects paste address mmap() */ +/* with DLPAR close/open windows */ +struct vm_area_struct *vma;/* Save VMA and used in DLPAR ops */ };/*
@@ -67,6 +76,7 @@ struct vas_user_win_ref {struct vas_window {u32 winid;u32 wcreds_max;/* Window credits */ +u32 status;/* Window status used in OS */ enum vas_cop_type cop;struct vas_user_win_ref task_ref;char *dbgname;
arch/powerpc/platforms/book3s/vas-api.c +68 -0
86 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v6 4/9 powerpc/vas: Return paste instruction failure if no active window
In reply to Haren MyneniThe VAS window may not be active if the system looses credits and the NX generates page fault when it receives request on unmap paste address.
The kernel handles the fault by remap new paste address if the window is active again, Otherwise return the paste instruction failure if the executed instruction that caused the fault was a paste.
--- arch/powerpc/include/asm/ppc-opcode.h | 2 + arch/powerpc/platforms/book3s/vas-api.c | 54 +++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
arch/powerpc/include/asm/ppc-opcode.h +2 -0
@@ -262,6 +262,8 @@#define PPC_INST_MFSPR_PVR 0x7c1f42a6#define PPC_INST_MFSPR_PVR_MASK 0xfc1ffffe#define PPC_INST_MTMSRD 0x7c000164 +#define PPC_INST_PASTE 0x7c20070d +#define PPC_INST_PASTE_MASK 0xfc2007ff#define PPC_INST_POPCNTB 0x7c0000f4#define PPC_INST_POPCNTB_MASK 0xfc0007fe#define PPC_INST_RFEBB 0x4c000124
arch/powerpc/platforms/book3s/vas-api.c +54 -0
72 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v6 5/9 powerpc/vas: Map paste address only if window is active
In reply to Haren MyneniThe paste address mapping is done with mmap() after the window is opened with ioctl. The partition has to close VAS windows in the hypervisor if it lost credits due to DLPAR core removal. But the kernel marks these windows inactive until the previously lost credits are available later. If the window is inactive due to DLPAR after this mmap(), the paste instruction returns failure until the the OS reopens this window again.
Before the user space issuing mmap(), there is a possibility of happening DLPAR core removal event which causes the corresponding window inactive. So if the window is not active, return mmap() failure with -EACCES and expects the user space reissue mmap() when the window is active or open a new window when the credit is available.
--- arch/powerpc/platforms/book3s/vas-api.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
arch/powerpc/platforms/book3s/vas-api.c +22 -1
@@ -497,10 +497,29 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma)return -EACCES ; } +/* +* The initial mmap is done after the window is opened +* with ioctl. But before mmap(), this window can be closed in +* the hypervisor due to lost credit (core removal on pseries). +* So if the window is not active, return mmap() failure with +* -EACCES and expects the user space reissue mmap() when it +* is active again or open new window when the credit is available. +* mmap_mutex protects the paste address mmap() with DLPAR +* close/open event and allows mmap() only when the window is +* active. +*/ +mutex_lock (&txwin->task_ref.mmap_mutex); +if (txwin->status !=VAS_WIN_ACTIVE ) { +pr_err ("%s(): Window is not active\n" , __func__); + rc = -EACCES ; + goto out; + } + paste_addr = cp_inst->coproc->vops->paste_addr (txwin);if (!paste_addr) {pr_err ("%s(): Window paste address failed\n" , __func__); -return -EINVAL ; + rc = -EINVAL ; + goto out; } pfn = paste_addr >>PAGE_SHIFT ;
@@ -520,6 +539,8 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) txwin->task_ref.vma = vma; vma->vm_ops = &vas_vm_ops; +out: +mutex_unlock (&txwin->task_ref.mmap_mutex);return rc; }
Signature · 1 line
2.27.0
PATCH v6 6/9 powerpc/pseries/vas: Close windows with DLPAR core removal
In reply to Haren MyneniThe hypervisor assigns vas credits (windows) for each LPAR based on the number of cores configured in that system. The OS is expected to release credits when cores are removed, and may allocate more when cores are added. So there is a possibility of using excessive credits (windows) in the LPAR and the hypervisor expects the system to close the excessive windows so that NX load can be equally distributed across all LPARs in the system.
When the OS closes the excessive windows in the hypervisor, it sets the window status inactive and invalidates window virtual address mapping. The user space receives paste instruction failure if any NX requests are issued on the inactive window. Then the user space can use with the available open windows or retry NX requests until this window active again.
This patch also adds the notifier for core removal/add to close windows in the hypervisor if the system lost credits (core removal) and reopen windows in the hypervisor when the previously lost credits are available.
--- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 207 +++++++++++++++++++++++++-- arch/powerpc/platforms/pseries/vas.h | 3 + 3 files changed, 204 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/vas.h +2 -0
@@ -34,6 +34,8 @@*/ #define VAS_WIN_ACTIVE 0x0/* Used in platform independent */ /* vas mmap() */ +/* Window is closed in the hypervisor due to lost credit */ +#define VAS_WIN_NO_CRED_CLOSE 0x00000001/* * Get/Set bit fields
arch/powerpc/platforms/pseries/vas.c +199 -8
244 lines not on this page · raw message
arch/powerpc/platforms/pseries/vas.h +3 -0
@@ -83,6 +83,9 @@ struct vas_cop_feat_caps {struct vas_caps {struct vas_cop_feat_caps caps;struct list_head list;/* List of open windows */ +int nr_close_wins;/* closed windows in the hypervisor for DLPAR */ +int nr_open_windows;/* Number of successful open windows */ +u8 feat;/* Feature type */ };/*
Signature · 1 line
2.27.0
PATCH v6 7/9 powerpc/pseries/vas: Reopen windows with DLPAR core add
In reply to Haren MyneniVAS windows can be closed in the hypervisor due to lost credits when the core is removed and the kernel gets fault for NX requests on these inactive windows. If the NX requests are issued on these inactive windows, OS gets page faults and the paste failure will be returned to the user space. If the lost credits are available later with core add, reopen these windows and set them active. Later when the OS sees page faults on these active windows, it creates mapping on the new paste address. Then the user space can continue to use these windows and send HW compression requests to NX successfully.
--- arch/powerpc/platforms/pseries/vas.c | 91 +++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/vas.c +90 -1
@@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type,return 0 ; } +/* +* VAS windows can be closed due to lost credits when the core is +* removed. So reopen them if credits are available due to DLPAR +* core add and set the window active status. When NX sees the page +* fault on the unmapped paste address, the kernel handles the fault +* by setting the remapping to new paste address if the window is +* active. +*/ +static int reconfig_open_windows (struct vas_caps *vcaps,int creds) +{ +long domain[PLPAR_HCALL9_BUFSIZE ] = {VAS_DEFAULT_DOMAIN_ID }; +struct vas_cop_feat_caps *caps = &vcaps->caps; +struct pseries_vas_window *win =NULL , *tmp; +int rc, mv_ents =0 ; + +/* +* Nothing to do if there are no closed windows. +*/ +if (!vcaps->nr_close_wins) +return 0 ; + +/* +* For the core removal, the hypervisor reduces the credits +* assigned to the LPAR and the kernel closes VAS windows +* in the hypervisor depends on reduced credits. The kernel +* uses LIFO (the last windows that are opened will be closed +* first) and expects to open in the same order when credits +* are available. +* For example, 40 windows are closed when the LPAR lost 2 cores +* (dedicated). If 1 core is added, this LPAR can have 20 more +* credits. It means the kernel can reopen 20 windows. So move +* 20 entries in the VAS windows lost and reopen next 20 windows. +*/ +if (vcaps->nr_close_wins > creds) + mv_ents = vcaps->nr_close_wins - creds; + +list_for_each_entry_safe (win, tmp, &vcaps->list, win_list) { +if (!mv_ents) +break ; + + mv_ents--; + } + +list_for_each_entry_safe_from (win, tmp, &vcaps->list, win_list) { +/* +* Nothing to do on this window if it is not closed +* with VAS_WIN_NO_CRED_CLOSE +*/ +if (!(win->vas_win.status &VAS_WIN_NO_CRED_CLOSE )) +continue ; + + rc =allocate_setup_window (win, (u64 *)&domain[0 ], + caps->win_type); +if (rc) +return rc; + + rc =h_modify_vas_window (win); +if (rc) + goto out; + +mutex_lock (&win->vas_win.task_ref.mmap_mutex); +/* +* Set window status to active +*/ + win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE ; +mutex_unlock (&win->vas_win.task_ref.mmap_mutex); + win->win_type = caps->win_type; +if (!--vcaps->nr_close_wins) +break ; + } + +return 0 ; +out: +/* +* Window modify HCALL failed. So close the window to the +* hypervisor and return. +*/ +free_irq_setup (win); +h_deallocate_vas_window (win->vas_win.winid); +return rc; +} +/* * The hypervisor reduces the available credits if the LPAR lost core. It * means the excessive windows should not be active and the user space
@@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)* closed / reopened. Hold the vas_pseries_mutex so that the * the user space can not open new windows. */ -if (old_nr_creds > new_nr_creds) { +if (old_nr_creds < new_nr_creds) { +/* +* If the existing target credits is less than the new +* target, reopen windows if they are closed due to +* the previous DLPAR (core removal). +*/ + rc =reconfig_open_windows (vcaps, new_nr_creds - old_nr_creds); + }else {/* * # active windows is more than new LPAR available * credits. So close the excessive windows.
Signature · 1 line
2.27.0
PATCH v6 8/9 powerpc/pseries/vas: sysfs interface to export capabilities
In reply to Haren MyneniThe hypervisor provides the available VAS GZIP capabilities such as default or QoS window type and the target available credits in each type. This patch creates sysfs entries and exports the target, used and the available credits for each feature.
This interface can be used by the user space to determine the credits usage or to set the target credits in the case of QoS type (for DLPAR).
/sys/devices/vas/vas0/gzip/default_capabilities (default GZIP capabilities)
nr_total_credits /* Total credits available. Can be
/* changed with DLPAR operation */
nr_used_credits /* Used credits */
/sys/devices/vas/vas0/gzip/qos_capabilities (QoS GZIP capabilities)
nr_total_credits
nr_used_credits
--- arch/powerpc/platforms/pseries/Makefile | 2 +- arch/powerpc/platforms/pseries/vas-sysfs.c | 226 +++++++++++++++++++++ arch/powerpc/platforms/pseries/vas.c | 6 + arch/powerpc/platforms/pseries/vas.h | 6 + 4 files changed, 239 insertions(+), 1 deletion(-) create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
arch/powerpc/platforms/pseries/Makefile +1 -1
@@ -29,6 +29,6 @@ obj-$(CONFIG_PPC_SVM) += svm.o obj-$(CONFIG_FA_DUMP) += rtas-fadump.o obj-$(CONFIG_SUSPEND) += suspend.o -obj-$(CONFIG_PPC_VAS) += vas.o +obj-$(CONFIG_PPC_VAS) += vas.o vas-sysfs.o obj-$(CONFIG_ARCH_HAS_CC_PLATFORM) += cc_platform.o
arch/powerpc/platforms/pseries/vas-sysfs.c new file +226 -0
226 lines not on this page · raw message
arch/powerpc/platforms/pseries/vas.c +6 -0
@@ -560,6 +560,10 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, } } + rc =sysfs_add_vas_caps (caps); +if (rc) +return rc; + copypaste_feat = true;return 0 ;
@@ -844,6 +848,8 @@ static int __init pseries_vas_init(void) caps_all.descriptor =be64_to_cpu (hv_caps->descriptor); caps_all.feat_type =be64_to_cpu (hv_caps->feat_type); +sysfs_pseries_vas_init (&caps_all); + hv_cop_caps =kmalloc (sizeof (*hv_cop_caps),GFP_KERNEL );if (!hv_cop_caps) { rc = -ENOMEM ;
arch/powerpc/platforms/pseries/vas.h +6 -0
@@ -30,6 +30,9 @@#define VAS_COPY_PASTE_USER_MODE 0x00000001#define VAS_COP_OP_USER_MODE 0x00000010 +#define VAS_GZIP_QOS_CAPABILITIES 0x56516F73477A6970 +#define VAS_GZIP_DEFAULT_CAPABILITIES 0x56446566477A6970 +/* * Co-processor feature - GZIP QoS windows or GZIP default windows */
@@ -125,4 +128,7 @@ struct pseries_vas_window {char *name; int fault_virq; }; + +int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps); +int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps);#endif /* _VAS_H */
Signature · 1 line
2.27.0
PATCH v6 9/9 powerpc/pseries/vas: Add 'update_total_credits' entry for QoS capabilities
In reply to Haren Mynenipseries supports two types of credits - Default (uses normal priority FIFO) and Qality of service (QoS uses high priority FIFO). The user decides the number of QoS credits and sets this value with HMC interface. The total credits for QoS capabilities can be changed dynamically with HMC interface which invokes drmgr to communicate to the kernel.
This patch creats 'update_total_credits' entry for QoS capabilities so that drmgr command can write the new target QoS credits in sysfs. Instead of using this value, the kernel gets the new QoS capabilities from the hypervisor whenever update_total_credits is updated to make sure sync with the QoS target credits in the hypervisor.
--- arch/powerpc/platforms/pseries/vas-sysfs.c | 54 +++++++++++++++++++--- arch/powerpc/platforms/pseries/vas.c | 2 +- arch/powerpc/platforms/pseries/vas.h | 1 + 3 files changed, 50 insertions(+), 7 deletions(-)
arch/powerpc/platforms/pseries/vas-sysfs.c +48 -6
@@ -25,6 +25,27 @@ struct vas_caps_entry {#define to_caps_entry (entry) container_of(entry, struct vas_caps_entry, kobj) +/* +* This function is used to get the notification from the drmgr when +* QoS credits are changed. Though receiving the target total QoS +* credits here, get the official QoS capabilities from the hypervisor. +*/ +static ssize_t update_total_credits_trigger (struct vas_cop_feat_caps *caps, +const char *buf,size_t count) +{ +int err; +u16 creds; + + err =kstrtou16 (buf,0 , &creds); +if (!err) + err =vas_reconfig_capabilties (caps->win_type); + +if (err) +return -EINVAL ; + +return count; +} +#define sysfs_caps_entry_read (_name) \ static ssize_t _name##_show(struct vas_cop_feat_caps *caps, char *buf) \ { \
@@ -63,17 +84,29 @@ struct vas_sysfs_entry {* changed dynamically by the user. * /sys/devices/vas/vas0/gzip/qos_capabilities/nr_used_credits * Number of credits used by the user space. +* /sys/devices/vas/vas0/gzip/qos_capabilities/update_total_credits +* Update total QoS credits dynamically */ VAS_ATTR_RO (nr_total_credits);VAS_ATTR_RO (nr_used_credits); -static struct attribute *vas_capab_attrs[] = { +static struct vas_sysfs_entry update_total_credits_attribute = +__ATTR (update_total_credits,0200 ,NULL , update_total_credits_trigger); + +static struct attribute *vas_def_capab_attrs[] = { &nr_total_credits_attribute.attr, &nr_used_credits_attribute.attr,NULL , }; +static struct attribute *vas_qos_capab_attrs[] = { + &nr_total_credits_attribute.attr, + &nr_used_credits_attribute.attr, + &update_total_credits_attribute.attr, +NULL , +}; +static ssize_t vas_type_show (struct kobject *kobj,struct attribute *attr,char *buf) {
@@ -118,19 +151,29 @@ static const struct sysfs_ops vas_sysfs_ops = { .store = vas_type_store, }; -static struct kobj_type vas_attr_type = { +static struct kobj_type vas_def_attr_type = { .release = vas_type_release, .sysfs_ops = &vas_sysfs_ops, - .default_attrs = vas_capab_attrs, + .default_attrs = vas_def_capab_attrs, }; -static char *vas_caps_kobj_name (struct vas_cop_feat_caps *caps, +static struct kobj_type vas_qos_attr_type = { + .release = vas_type_release, + .sysfs_ops = &vas_sysfs_ops, + .default_attrs = vas_qos_capab_attrs, +}; + +static char *vas_caps_kobj_name (struct vas_caps_entry *centry,struct kobject **kobj) { +struct vas_cop_feat_caps *caps = centry->caps; +if (caps->descriptor ==VAS_GZIP_QOS_CAPABILITIES ) { +kobject_init (¢ry->kobj, &vas_qos_attr_type); *kobj = gzip_caps_kobj;return "qos_capabilities" ; }else if (caps->descriptor ==VAS_GZIP_DEFAULT_CAPABILITIES ) { +kobject_init (¢ry->kobj, &vas_def_attr_type); *kobj = gzip_caps_kobj;return "default_capabilities" ; }else
@@ -152,9 +195,8 @@ int sysfs_add_vas_caps(struct vas_cop_feat_caps *caps)if (!centry)return -ENOMEM ; -kobject_init (¢ry->kobj, &vas_attr_type); centry->caps = caps; - name =vas_caps_kobj_name (caps, &kobj); + name =vas_caps_kobj_name (centry, &kobj);if (kobj) { ret =kobject_add (¢ry->kobj, kobj,"%s" , name);
arch/powerpc/platforms/pseries/vas.c +1 -1
@@ -722,7 +722,7 @@ static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds)* changes. Reconfig window configurations based on the credits * availability from this new capabilities. */ -static int vas_reconfig_capabilties (u8 type) +int vas_reconfig_capabilties (u8 type) {struct hv_vas_cop_feat_caps *hv_caps;struct vas_cop_feat_caps *caps;
arch/powerpc/platforms/pseries/vas.h +1 -0
@@ -130,5 +130,6 @@ struct pseries_vas_window { };int sysfs_add_vas_caps (struct vas_cop_feat_caps *caps); +int vas_reconfig_capabilties (u8 type);int __init sysfs_pseries_vas_init (struct vas_all_caps *vas_caps);#endif /* _VAS_H */
Signature · 1 line
2.27.0
Michael Ellerman
AppliedQuoted 11 lines · Haren Myneni
PowerPC provides HW compression with NX coprocessor. This feature is available on both PowerNV and PowerVM and included in Linux. Since each powerpc chip has one NX coprocessor, the VAS introduces the concept of windows / credits to manage access to this hardware resource. On powerVM, these limited resources should be available across all LPARs. So the hypervisor assigns the specific credits to each LPAR based on processor entitlement so that one LPAR does not overload NX. The hypervisor can reject the window open request to a partition if exceeds its credit limit (1 credit per window).
[...]
Applied to powerpc/next.
[1/9] powerpc/pseries/vas: Use common names in VAS capability structure
https://git.kernel.org/powerpc/c/40562fe4fa3d94c7462ec909ab89b075e26c59ac [2/9] powerpc/pseries/vas: Save PID in pseries_vas_window struct https://git.kernel.org/powerpc/c/976410cd2cb4c6ed53bd12c192fc46bbcc0fbce7 [3/9] powerpc/vas: Add paste address mmap fault handler https://git.kernel.org/powerpc/c/1fe3a33ba0a37e7aa0df0acbe31d5dda7610c16e [4/9] powerpc/vas: Return paste instruction failure if no active window https://git.kernel.org/powerpc/c/b5c63d90cc2de8ac6724fec84d1d72cfebcae41d [5/9] powerpc/vas: Map paste address only if window is active https://git.kernel.org/powerpc/c/6a8d4ca891aa5f9402973eab5d7d9cf3929678b7 [6/9] powerpc/pseries/vas: Close windows with DLPAR core removal https://git.kernel.org/powerpc/c/8ef7b9e1765a52c8023d9133a2438ac9f6da486a [7/9] powerpc/pseries/vas: Reopen windows with DLPAR core add https://git.kernel.org/powerpc/c/c656cfe571a9b8b882e31177f554bd79141fc015 [8/9] powerpc/pseries/vas: sysfs interface to export capabilities https://git.kernel.org/powerpc/c/b903737bc522e0ef3f45a2a60c364ff547572c9b [9/9] powerpc/pseries/vas: Add 'update_total_credits' entry for QoS capabilities https://git.kernel.org/powerpc/c/45f06eac30e5abebecc66e41e7c89d5b4413bac1
cheers