powerpc/pseries: IPI doorbell improvements
Subsystems (2)
Nicholas Piggin
reviewerv3 cover powerpc/pseries: IPI doorbell improvements
Since v2:
- Fixed ppc32 compile error
- Tested-by from Cedric
Nicholas Piggin (3):
powerpc: inline doorbell sending functions powerpc/pseries: Use doorbells even if XIVE is available powerpc/pseries: Add KVM guest doorbell restrictions
arch/powerpc/include/asm/dbell.h | 67 ++++++++++++++++++++++++++-- arch/powerpc/include/asm/firmware.h | 6 +++ arch/powerpc/include/asm/kvm_para.h | 26 ++--------- arch/powerpc/kernel/Makefile | 5 +-- arch/powerpc/kernel/dbell.c | 55 ----------------------- arch/powerpc/kernel/firmware.c | 19 ++++++++ arch/powerpc/platforms/pseries/smp.c | 62 +++++++++++++++++-------- 7 files changed, 138 insertions(+), 102 deletions(-)
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v3 1/3 powerpc: inline doorbell sending functions
These are only called in one place for a given platform, so inline them for performance.
--- arch/powerpc/include/asm/dbell.h | 67 ++++++++++++++++++++++++++++++-- arch/powerpc/kernel/dbell.c | 55 -------------------------- 2 files changed, 64 insertions(+), 58 deletions(-)
arch/powerpc/include/asm/dbell.h +64 -3
@@ -13,6 +13,11 @@#include <asm/ppc-opcode.h> #include <asm/feature-fixups.h> +#ifdef CONFIG_KVM +#include <asm/kvm_ppc.h> +#else +static inline void kvmppc_set_host_ipi (int cpu) {} +#endif #define PPC_DBELL_MSG_BRDCAST (0x04000000)#define PPC_DBELL_TYPE (x) (((x) & 0xf) << (63-36))
@@ -87,9 +92,6 @@ static inline void ppc_msgsync(void)#endif /* CONFIG_PPC_BOOK3S */ -extern void doorbell_global_ipi (int cpu); -extern void doorbell_core_ipi (int cpu); -extern int doorbell_try_core_ipi (int cpu);extern void doorbell_exception (struct pt_regs *regs);static inline void ppc_msgsnd (enum ppc_dbell type,u32 flags,u32 tag)
@@ -100,4 +102,63 @@ static inline void ppc_msgsnd(enum ppc_dbell type, u32 flags, u32 tag)_ppc_msgsnd (msg); } +#ifdef CONFIG_SMP + +/* +* Doorbells must only be used if CPU_FTR_DBELL is available. +* msgsnd is used in HV, and msgsndp is used in !HV. +* +* These should be used by platform code that is aware of restrictions. +* Other arch code should use ->cause_ipi. +* +* doorbell_global_ipi() sends a dbell to any target CPU. +* Must be used only by architectures that address msgsnd target +* by PIR/get_hard_smp_processor_id. +*/ +static inline void doorbell_global_ipi (int cpu) +{ +u32 tag =get_hard_smp_processor_id (cpu); + +kvmppc_set_host_ipi (cpu); +/* Order previous accesses vs. msgsnd, which is treated as a store */ +ppc_msgsnd_sync (); +ppc_msgsnd (PPC_DBELL_MSGTYPE ,0 , tag); +} + +/* +* doorbell_core_ipi() sends a dbell to a target CPU in the same core. +* Must be used only by architectures that address msgsnd target +* by TIR/cpu_thread_in_core. +*/ +static inline void doorbell_core_ipi (int cpu) +{ +u32 tag =cpu_thread_in_core (cpu); + +kvmppc_set_host_ipi (cpu); +/* Order previous accesses vs. msgsnd, which is treated as a store */ +ppc_msgsnd_sync (); +ppc_msgsnd (PPC_DBELL_MSGTYPE ,0 , tag); +} + +/* +* Attempt to cause a core doorbell if destination is on the same core. +* Returns 1 on success, 0 on failure. +*/ +static inline int doorbell_try_core_ipi (int cpu) +{ +int this_cpu =get_cpu (); +int ret =0 ; + +if (cpumask_test_cpu (cpu,cpu_sibling_mask (this_cpu))) { +doorbell_core_ipi (cpu); + ret =1 ; + } + +put_cpu (); + +return ret; +} + +#endif /* CONFIG_SMP */ +#endif /* _ASM_POWERPC_DBELL_H */
arch/powerpc/kernel/dbell.c +0 -55
61 lines not on this page · raw message
Signature · 1 line
2.23.0
kernel test robot
build botRe: [PATCH v3 1/3] powerpc: inline doorbell sending functions
I love your patch! Yet something to improve:
Hi Nicholas,
I love your patch! Yet something to improve:
[auto build test ERROR on powerpc/next]
[also build test ERROR on scottwood/next v5.8-rc6 next-20200724] [cannot apply to kvm-ppc/kvm-ppc-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/powerpc-pseries-IPI-doorbell-improvements/20200726-115432 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next config: powerpc-randconfig-r012-20200726 (attached as .config) compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from arch/powerpc/kernel/asm-offsets.c:38:
arch/powerpc/include/asm/dbell.h: In function 'doorbell_core_ipi':
>> arch/powerpc/include/asm/dbell.h:135:12: error: implicit declaration of function 'cpu_thread_in_core' [-Werror=implicit-function-declaration]
135 | u32 tag = cpu_thread_in_core(cpu);
| ^~~~~~~~~~~~~~~~~~
In file included from arch/powerpc/include/asm/kvm_host.h:33,
from include/linux/kvm_host.h:36,
from arch/powerpc/kernel/asm-offsets.c:51:
arch/powerpc/include/asm/cputhreads.h: At top level:
>> arch/powerpc/include/asm/cputhreads.h:82:19: error: static declaration of 'cpu_thread_in_core' follows non-static declaration
82 | static inline int cpu_thread_in_core(int cpu)
| ^~~~~~~~~~~~~~~~~~
In file included from arch/powerpc/kernel/asm-offsets.c:38:
arch/powerpc/include/asm/dbell.h:135:12: note: previous implicit declaration of 'cpu_thread_in_core' was here
135 | u32 tag = cpu_thread_in_core(cpu);
| ^~~~~~~~~~~~~~~~~~ cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:114: arch/powerpc/kernel/asm-offsets.s] Error 1 make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1175: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +/cpu_thread_in_core +135 arch/powerpc/include/asm/dbell.h
127
128 /*
129 * doorbell_core_ipi() sends a dbell to a target CPU in the same core.
130 * Must be used only by architectures that address msgsnd target
131 * by TIR/cpu_thread_in_core.
132 */
133 static inline void doorbell_core_ipi(int cpu)
134 {
> 135 u32 tag = cpu_thread_in_core(cpu);
136
137 kvmppc_set_host_ipi(cpu);
138 /* Order previous accesses vs. msgsnd, which is treated as a store */
139 ppc_msgsnd_sync();
140 ppc_msgsnd(PPC_DBELL_MSGTYPE, 0, tag);
141 }
142
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Michael Ellerman
reviewerRe: [PATCH v3 1/3] powerpc: inline doorbell sending functions
In reply to Nicholas PigginQuoted 29 lines · Nicholas Piggin
arch/powerpc/include/asm/dbell.h +21
-0@@ -100,4 +102,63 @@ static inline void ppc_msgsnd(enum ppc_dbell type, u32 flags, u32 tag)_ppc_msgsnd (msg); } +#ifdef CONFIG_SMP + +/* +* Doorbells must only be used if CPU_FTR_DBELL is available. +* msgsnd is used in HV, and msgsndp is used in !HV. +* +* These should be used by platform code that is aware of restrictions. +* Other arch code should use ->cause_ipi. +* +* doorbell_global_ipi() sends a dbell to any target CPU. +* Must be used only by architectures that address msgsnd target +* by PIR/get_hard_smp_processor_id. +*/ +static inline void doorbell_global_ipi (int cpu) +{ +u32 tag =get_hard_smp_processor_id (cpu); + +kvmppc_set_host_ipi (cpu); +/* Order previous accesses vs. msgsnd, which is treated as a store */ +ppc_msgsnd_sync (); +ppc_msgsnd (PPC_DBELL_MSGTYPE ,0 , tag);
arch/powerpc/include/asm/dbell.h static inline void ppc_msgsnd(enum ppc_dbell type, u32 flags, u32 tag) · line 135
Quoted 10 lines
arch/powerpc/include/asm/dbell.h +10
-0+} + +/* +* doorbell_core_ipi() sends a dbell to a target CPU in the same core. +* Must be used only by architectures that address msgsnd target +* by TIR/cpu_thread_in_core. +*/ +static inline void doorbell_core_ipi (int cpu) +{ +u32 tag =cpu_thread_in_core (cpu);
corenet64_smp_defconfig gives me:
In file included from /linux/arch/powerpc/kernel/asm-offsets.c:38:
/linux/arch/powerpc/include/asm/dbell.h: In function 'doorbell_core_ipi':
/linux/arch/powerpc/include/asm/dbell.h:135:12: error: implicit declaration of function 'cpu_thread_in_core' [-Werror=implicit-function-declaration]
135 | u32 tag = cpu_thread_in_core(cpu);
| ^~~~~~~~~~~~~~~~~~Fixed by including cputhreads.h, but then that results in errors due to your addition of kvmppc_set_host_ipi().
Removing that gets us back to the fault_dear error.
I think I see a way around it, will do some build tests.
cheers
Nicholas Piggin
reviewerPATCH v3 2/3 powerpc/pseries: Use doorbells even if XIVE is available
In reply to Nicholas PigginKVM supports msgsndp in guests by trapping and emulating the instruction, so it was decided to always use XIVE for IPIs if it is available. However on PowerVM systems, msgsndp can be used and gives better performance. On large systems, high XIVE interrupt rates can have sub-linear scaling, and using msgsndp can reduce the load on the interrupt controller.
So switch to using core local doorbells even if XIVE is available. This reduces performance for KVM guests with an SMT topology by about 50% for ping-pong context switching between SMT vCPUs. An option vector (or dt-cpu-ftrs) could be defined to disable msgsndp to get KVM performance back.
--- arch/powerpc/platforms/pseries/smp.c | 54 ++++++++++++++++++---------- 1 file changed, 36 insertions(+), 18 deletions(-)
arch/powerpc/platforms/pseries/smp.c +36 -18
@@ -188,13 +188,16 @@ static int pseries_smp_prepare_cpu(int cpu)return 0 ; } -static void smp_pseries_cause_ipi (int cpu) +/* Cause IPI as setup by the interrupt controller (xics or xive) */ +static void (*ic_cause_ipi)(int cpu)__ro_after_init ; + +/* Use msgsndp doorbells target is a sibling, else use interrupt controller */ +static void dbell_or_ic_cause_ipi (int cpu) { -/* POWER9 should not use this handler */ if (doorbell_try_core_ipi (cpu))return ; - icp_ops->cause_ipi (cpu); +ic_cause_ipi (cpu); }static int pseries_cause_nmi_ipi (int cpu)
@@ -218,26 +221,41 @@ static int pseries_cause_nmi_ipi(int cpu)return 0 ; } -static __init void pSeries_smp_probe_xics (void ) -{ -xics_smp_probe (); - -if (cpu_has_feature (CPU_FTR_DBELL ) && !is_secure_guest ()) - smp_ops->cause_ipi = smp_pseries_cause_ipi; -else - smp_ops->cause_ipi = icp_ops->cause_ipi; -} -static __init void pSeries_smp_probe (void ) {if (xive_enabled ()) -/* -* Don't use P9 doorbells when XIVE is enabled. IPIs -* using MMIOs should be faster -*/ xive_smp_probe ();else -pSeries_smp_probe_xics (); +xics_smp_probe (); + +/* No doorbell facility, must use the interrupt controller for IPIs */ +if (!cpu_has_feature (CPU_FTR_DBELL )) +return ; + +/* Doorbells can only be used for IPIs between SMT siblings */ +if (!cpu_has_feature (CPU_FTR_SMT )) +return ; + +/* +* KVM emulates doorbells by disabling FSCR[MSGP] so msgsndp faults +* to the hypervisor which then reads the instruction from guest +* memory. This can't be done if the guest is secure, so don't use +* doorbells in secure guests. +* +* Under PowerVM, FSCR[MSGP] is enabled so doorbells could be used +* by secure guests if we distinguished this from KVM. +*/ +if (is_secure_guest ()) +return ; + +/* +* The guest can use doobells for SMT sibling IPIs, which stay in +* the core rather than going to the interrupt controller. This +* tends to be slower under KVM where doorbells are emulated, but +* faster for PowerVM where they're enabled. +*/ + ic_cause_ipi = smp_ops->cause_ipi; + smp_ops->cause_ipi = dbell_or_ic_cause_ipi; }static struct smp_ops_t pseries_smp_ops = {
Signature · 1 line
2.23.0
Nicholas Piggin
reviewerPATCH v3 3/3 powerpc/pseries: Add KVM guest doorbell restrictions
In reply to Nicholas PigginKVM guests have certain restrictions and performance quirks when using doorbells. This patch moves the EPAPR KVM guest test so it can be shared with PSERIES, and uses that in doorbell setup code to apply the KVM guest quirks and improves IPI performance for two cases:
- PowerVM guests may now use doorbells even if they are secure.
- KVM guests no longer use doorbells if XIVE is available.
There is a valid complaint that "KVM guest" is not a very reasonable thing to test for, it's preferable for the hypervisor to advertise particular behaviours to the guest so they could change if the hypervisor implementation or configuration changes. However in this case we were already assuming a KVM guest worst case, so this patch is about containing those quirks. If KVM later advertises fast doorbells, we should test for that and override the quirks.
--- arch/powerpc/include/asm/firmware.h | 6 +++++ arch/powerpc/include/asm/kvm_para.h | 26 +++---------------- arch/powerpc/kernel/Makefile | 5 ++-- arch/powerpc/kernel/firmware.c | 19 ++++++++++++++ arch/powerpc/platforms/pseries/smp.c | 38 +++++++++++++++++----------- 5 files changed, 53 insertions(+), 41 deletions(-)
arch/powerpc/include/asm/firmware.h +6 -0
@@ -132,6 +132,12 @@ extern int ibm_nmi_interlock_token;extern unsigned int __start___fw_ftr_fixup, __stop___fw_ftr_fixup; +#if defined(CONFIG_PPC_PSERIES ) || defined(CONFIG_KVM_GUEST ) +bool is_kvm_guest (void ); +#else +static inline bool is_kvm_guest (void ) {return false; } +#endif +#ifdef CONFIG_PPC_PSERIES void pseries_probe_fw_features (void );#else
arch/powerpc/include/asm/kvm_para.h +3 -23
@@ -8,35 +8,15 @@#ifndef __POWERPC_KVM_PARA_H__#define __POWERPC_KVM_PARA_H__ -#include <uapi/asm/kvm_para.h> - -#ifdef CONFIG_KVM_GUEST - -#include <linux/of.h> - -static inline int kvm_para_available (void ) -{ -struct device_node *hyper_node; - - hyper_node =of_find_node_by_path ("/hypervisor" ); -if (!hyper_node) -return 0 ; +#include <asm/firmware.h> -if (!of_device_is_compatible (hyper_node,"linux,kvm" )) -return 0 ; - -return 1 ; -} - -#else +#include <uapi/asm/kvm_para.h> static inline int kvm_para_available (void ) { -return 0 ; +return IS_ENABLED (CONFIG_KVM_GUEST ) &&is_kvm_guest (); } -#endif -static inline unsigned int kvm_arch_para_features (void ) {unsigned long r;
arch/powerpc/kernel/Makefile +2 -3
@@ -45,11 +45,10 @@ obj-y := cputable.o syscalls.o \ signal.o sysfs.o cacheinfo.o time.o \ prom.o traps.o setup-common.o \ udbg.o misc.o io.o misc_$(BITS).o \ - of_platform.o prom_parse.o + of_platform.o prom_parse.o firmware.o obj-y += ptrace/ obj-$(CONFIG_PPC64) += setup_64.o \ - paca.o nvram_64.o firmware.o note.o \ - syscall_64.o + paca.o nvram_64.o note.o syscall_64.o obj-$(CONFIG_COMPAT) += sys_ppc32.o signal_32.o obj-$(CONFIG_VDSO32) += vdso32/ obj-$(CONFIG_PPC_WATCHDOG) += watchdog.o
arch/powerpc/kernel/firmware.c +19 -0
@@ -11,8 +11,27 @@#include <linux/export.h> #include <linux/cache.h> +#include <linux/of.h> #include <asm/firmware.h> +#ifdef CONFIG_PPC64 unsigned long powerpc_firmware_features__read_mostly ;EXPORT_SYMBOL_GPL (powerpc_firmware_features); +#endif + +#if defined(CONFIG_PPC_PSERIES ) || defined(CONFIG_KVM_GUEST ) +bool is_kvm_guest (void ) +{ +struct device_node *hyper_node; + + hyper_node =of_find_node_by_path ("/hypervisor" ); +if (!hyper_node) +return 0 ; + +if (!of_device_is_compatible (hyper_node,"linux,kvm" )) +return 0 ; + +return 1 ; +} +#endif
arch/powerpc/platforms/pseries/smp.c +23 -15
@@ -236,24 +236,32 @@ static __init void pSeries_smp_probe(void)if (!cpu_has_feature (CPU_FTR_SMT ))return ; -/* -* KVM emulates doorbells by disabling FSCR[MSGP] so msgsndp faults -* to the hypervisor which then reads the instruction from guest -* memory. This can't be done if the guest is secure, so don't use -* doorbells in secure guests. -* -* Under PowerVM, FSCR[MSGP] is enabled so doorbells could be used -* by secure guests if we distinguished this from KVM. -*/ -if (is_secure_guest ()) -return ; +if (is_kvm_guest ()) { +/* +* KVM emulates doorbells by disabling FSCR[MSGP] so msgsndp +* faults to the hypervisor which then reads the instruction +* from guest memory, which tends to be slower than using XIVE. +*/ +if (xive_enabled ()) +return ; + +/* +* XICS hcalls aren't as fast, so we can use msgsndp (which +* also helps exercise KVM emulation), however KVM can't +* emulate secure guests because it can't read the instruction +* out of their memory. +*/ +if (is_secure_guest ()) +return ; + }/* -* The guest can use doobells for SMT sibling IPIs, which stay in -* the core rather than going to the interrupt controller. This -* tends to be slower under KVM where doorbells are emulated, but -* faster for PowerVM where they're enabled. +* Under PowerVM, FSCR[MSGP] is enabled as guest vCPU siblings are +* gang scheduled on the same physical core, so doorbells are always +* faster than the interrupt controller, and they can be used by +* secure guests. */ + ic_cause_ipi = smp_ops->cause_ipi; smp_ops->cause_ipi = dbell_or_ic_cause_ipi; }
Signature · 1 line
2.23.0
Michael Ellerman
AppliedQuoted 10 lines · Nicholas Piggin
Since v2:
- Fixed ppc32 compile error
- Tested-by from CedricNicholas Piggin (3):
powerpc: inline doorbell sending functions powerpc/pseries: Use doorbells even if XIVE is available powerpc/pseries: Add KVM guest doorbell restrictions[...]
Applied to powerpc/next.
[1/3] powerpc: Inline doorbell sending functions
https://git.kernel.org/powerpc/c/1f0ce497433f8944045ee1baae218e31a0d295ee [2/3] powerpc/pseries: Use doorbells even if XIVE is available https://git.kernel.org/powerpc/c/5b06d1679f2fe874ef49ea11324cd893ec9e2da8 [3/3] powerpc/pseries: Add KVM guest doorbell restrictions https://git.kernel.org/powerpc/c/107c55005fbd5243ee31fb13b6f166cde9e3ade1
cheers