powerpc/pseries/vas: VAS/NXGZIP support with LPM
cover powerpc/pseries/vas: VAS/NXGZIP support with LPM
Virtual Accelerator Switchboard (VAS) is an engine stays on the chip. So all windows opened on a specific engine belongs to VAS the chip. The hypervisor expects the partition to close all active windows on the sources system and reopen them after migration on the destination machine.
This patch series adds VAS support with the partition migration. When the migration initiates, the VAS code gets suspend notifier which closes all active windows and sets VAS_WIN_MIGRATE_CLOSE as window status. Whereas receives resume notifier after migration to reopen all windows which has VAS_WIN_MIGRATE_CLOSE status.
These patches depend on VAS/DLPAR support patch series
(https://lists.ozlabs.org/pipermail/linuxppc-dev/2021-December/238331.html)
Also the suspend/resume notifier code is added in a separate patch and will be posted later with the actual patches.
Haren Myneni (3):
powerpc/pseries/vas: Modify reconfig open/close functions for migration powerpc/pseries/vas: Add VAS suspend/resume notifier powerpc/pseries/vas: Disable window open during migration
arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 199 ++++++++++++++++++++++++--- 2 files changed, 181 insertions(+), 20 deletions(-)
Signature · 1 line
2.27.0
RFC PATCH 1/3 powerpc/pseries/vas: Modify reconfig open/close functions for migration
VAS is a hardware engine stays on the chip. So when the partition migrates, all VAS windows on the source system have to be closed and reopen them on the destination after migration.
This patch make changes to the current reconfig_open/close_windows functions to support migration:
- Set VAS_WIN_MIGRATE_CLOSE to the window status when closes and reopen windows with the same status during resume.
- Continue to close all windows even if deallocate HCALL failed (should not happen) since no way to stop migration with the current LPM implementation.
- If the DLPAR CPU event happens while migration is in progress, set VAS_WIN_NO_CRED_CLOSE to the window status. Close window happens with the first event (migration or DLPAR) and Reopen window happens only with the last event (migration or DLPAR).
--- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 88 ++++++++++++++++++++++------ 2 files changed, 73 insertions(+), 17 deletions(-)
arch/powerpc/include/asm/vas.h +2 -0
@@ -42,6 +42,8 @@/* Linux status bits */ #define VAS_WIN_NO_CRED_CLOSE 0x00000004/* Window is closed due to */ /* lost credit */ +#define VAS_WIN_MIGRATE_CLOSE 0x00000008/* Window is closed due to */ +/* migration */ /* * Get/Set bit fields */
arch/powerpc/platforms/pseries/vas.c +71 -17
174 lines not on this page · raw message
Signature · 1 line
2.27.0
RFC PATCH 2/3 powerpc/pseries/vas: Add VAS suspend/resume notifier
In reply to Haren MyneniSince the windows belong to the VAS hardware resource, the hypervisor expects the partition to close them on source partition and reopen them after the partition migrated on the destination machine.
This suspend/resume notifier invokes suspend operation before and resume operation after migration. All active windows for both default and QoS types will be closed during suspend and reopen them during resume. During migration, the user space should expect paste instruction failure if issues copy/paste on these active windows.
--- arch/powerpc/platforms/pseries/vas.c | 97 +++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/vas.c +96 -1
@@ -16,6 +16,7 @@#include <asm/machdep.h> #include <asm/hvcall.h> #include <asm/plpar_wrappers.h> +#include <asm/pseries-suspend.h> #include <asm/vas.h> #include "vas.h"
@@ -873,6 +874,98 @@ static struct notifier_block pseries_vas_nb = { .notifier_call = pseries_vas_notifier, }; +/* +* For LPM, all windows have to be closed on the source partition +* before migration and reopen them on the destination partition +* after migration. So closing windows during suspend and +* reopen them during resume. +*/ +static int vas_migrate_windows (bool suspend) +{ +struct hv_vas_cop_feat_caps *hv_caps; +struct vas_cop_feat_caps *caps; +int lpar_creds, new_creds =0 ; +struct vas_caps *vcaps; +int i, rc =0 ; + + hv_caps =kmalloc (sizeof (*hv_caps),GFP_KERNEL ); +if (!hv_caps) +return -ENOMEM ; + +mutex_lock (&vas_pseries_mutex); + +for (i =0 ; i <VAS_MAX_FEAT_TYPE ; i++) { + vcaps = &vascaps[i]; + caps = &vcaps->caps; + lpar_creds =atomic_read (&caps->target_creds); + + rc =h_query_vas_capabilities (H_QUERY_VAS_CAPABILITIES , vcaps->feat, + (u64 )virt_to_phys (hv_caps)); +if (!rc) { + new_creds =be16_to_cpu (hv_caps->target_lpar_creds); +/* +* Should not happen. But incase print messages, close all +* windows in the list during suspend and reopen windows +* based on new lpar_creds on the destination system. +*/ +if (lpar_creds != new_creds) { +pr_err ("%s: lpar creds: %d HV lpar creds: %d\n" , + suspend ?"Suspend" :"Resume" , lpar_creds, + new_creds); +pr_err ("Used creds: %d, Active creds: %d\n" , +atomic_read (&caps->used_creds), + vcaps->num_wins - vcaps->close_wins); + } + }else { +pr_err ("%s: Get VAS capabilities failed with %d\n" , + suspend ?"Suspend" :"Resume" , rc); +/* +* We can not stop migration with the current lpm +* implementation. So continue closing all windows in the +* list (during suspend) and return without opending windows +* (during resume) if VAS capabilities HCALL failed. +*/ +if (!suspend) + goto out; + } + +if (suspend) + rc =reconfig_close_windows (vcaps, vcaps->num_wins, true); +else { +atomic_set (&caps->target_creds, new_creds); + rc =reconfig_open_windows (vcaps, new_creds, true); + } + +/* +* Ignore errors during suspend and return for resume. +*/ +if (rc && !suspend) + goto out; + } + +out: +mutex_unlock (&vas_pseries_mutex); +kfree (hv_caps); +return rc; +} + +static int vas_migration_handler (struct notifier_block *nb, +unsigned long action,void *data) +{ +if (action ==PSERIES_RESUMING ) +return vas_migrate_windows (false); +else +return vas_migrate_windows (true); + +} + +static struct pseries_suspend_handler vas_suspend_handler = { + .notifier_block = { + .notifier_call = vas_migration_handler, + }, +}; + +static int __init pseries_vas_init (void ) {struct hv_vas_cop_feat_caps *hv_cop_caps;
@@ -929,8 +1022,10 @@ static int __init pseries_vas_init(void) }/* Processors can be added/removed only on LPAR */ -if (copypaste_feat &&firmware_has_feature (FW_FEATURE_LPAR )) +if (copypaste_feat &&firmware_has_feature (FW_FEATURE_LPAR )) {of_reconfig_notifier_register (&pseries_vas_nb); +pseries_register_suspend_handler (&vas_suspend_handler); + }pr_info ("GZIP feature is available\n" );
Signature · 1 line
2.27.0
RFC PATCH 3/3 powerpc/pseries/vas: Disable window open during migration
In reply to Haren MyneniThe current partition migration implementation does not freeze the user space and the user space can continue open VAS windows. So when migration_in_progress flag is enabled, VAS open window API returns -EBUSY.
--- arch/powerpc/platforms/pseries/vas.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
arch/powerpc/platforms/pseries/vas.c +12 -2
@@ -30,6 +30,7 @@ static bool copypaste_feat;static struct vas_caps vascaps[VAS_MAX_FEAT_TYPE ];static DEFINE_MUTEX (vas_pseries_mutex); +static bool migration_in_progress;static long hcall_return_busy_check (long rc) {
@@ -356,8 +357,11 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags,* same fault IRQ is not freed by the OS before. */ mutex_lock (&vas_pseries_mutex); - rc =allocate_setup_window (txwin, (u64 *)&domain[0 ], - cop_feat_caps->win_type); +if (migration_in_progress) + rc = -EBUSY ; +else + rc =allocate_setup_window (txwin, (u64 *)&domain[0 ], + cop_feat_caps->win_type);mutex_unlock (&vas_pseries_mutex);if (rc) goto out;
@@ -796,6 +800,7 @@ int vas_reconfig_capabilties(u8 type)return -ENOMEM ;mutex_lock (&vas_pseries_mutex); + rc =h_query_vas_capabilities (H_QUERY_VAS_CAPABILITIES , vcaps->feat, (u64 )virt_to_phys (hv_caps));if (rc)
@@ -894,6 +899,11 @@ static int vas_migrate_windows(bool suspend)mutex_lock (&vas_pseries_mutex); +if (suspend) + migration_in_progress = true; +else + migration_in_progress = false; +for (i =0 ; i <VAS_MAX_FEAT_TYPE ; i++) { vcaps = &vascaps[i]; caps = &vcaps->caps;
Signature · 1 line
2.27.0