powerpc/pseries/vas: Use usleep_range() to support HCALL delay
PATCH v3 1/1 powerpc/pseries/vas: Use usleep_range() to support HCALL delay
VAS allocate, modify and deallocate HCALLs returns H_LONG_BUSY_ORDER_1_MSEC or H_LONG_BUSY_ORDER_10_MSEC for busy delay and expects OS to reissue HCALL after that delay. But using msleep() will often sleep at least 20 msecs even though the hypervisor suggests OS reissue these HCALLs after 1 or 10msecs. The open and close VAS window functions hold mutex and then issue these HCALLs. So these operations can take longer than the necessary when multiple threads issue open or close window APIs simultaneously.
So instead of msleep(), use usleep_range() to ensure sleep with the expected value before issuing HCALL again.
---
v1 -> v2:
- Use usleep_range instead of using RTAS sleep routine as suggested by Nathan
v2 -> v3:
- Sleep 10MSecs even for HCALL delay > 10MSecs and the other commit / comemnt changes as suggested by Nathan and Ellerman.
--- arch/powerpc/platforms/pseries/vas.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
arch/powerpc/platforms/pseries/vas.c +24 -1
@@ -38,7 +38,30 @@ static long hcall_return_busy_check(long rc) {/* Check if we are stalled for some time */ if (H_IS_LONG_BUSY (rc)) { -msleep (get_longbusy_msecs (rc)); +unsigned int ms; +/* +* Allocate, Modify and Deallocate HCALLs returns +* H_LONG_BUSY_ORDER_1_MSEC or H_LONG_BUSY_ORDER_10_MSEC +* for the long delay. So the sleep time should always +* be either 1 or 10msecs, but in case if the HCALL +* returns the long delay > 10 msecs, clamp the sleep +* time to 10msecs. +*/ + ms =clamp (get_longbusy_msecs (rc),1 ,10 ); + +/* +* msleep() will often sleep at least 20 msecs even +* though the hypervisor suggests that the OS reissue +* HCALLs after 1 or 10msecs. Also the delay hint from +* the HCALL is just a suggestion. So OK to pause for +* less time than the hinted delay. Use usleep_range() +* to ensure we don't sleep much longer than actually +* needed. +* +* See Documentation/timers/timers-howto.rst for +* explanation of the range used here. +*/ +usleep_range (ms *100 , ms *1000 ); rc =H_BUSY ; }else if (rc ==H_BUSY ) {cond_resched ();
Signature · 1 line
2.26.3
Quoted 10 lines · Haren Myneni
VAS allocate, modify and deallocate HCALLs returns
H_LONG_BUSY_ORDER_1_MSECorH_LONG_BUSY_ORDER_10_MSECfor busy delay and expects OS to reissue HCALL after that delay. But usingmsleep()will often sleep at least 20 msecs even though the hypervisor suggests OS reissue these HCALLs after 1 or 10msecs. The open and close VAS window functions hold mutex and then issue these HCALLs. So these operations can take longer than the necessary when multiple threads issue open or close window APIs simultaneously.
Quoted 3 lines
So instead of
msleep(), useusleep_range()to ensure sleep with the expected value before issuing HCALL again.
Can you summarize if there an user observable impact for the current code? We have other code paths using msleep(get_longbusy_msec()). Should we audit those usages?
Quoted 53 lines
Signed-off-by Haren Myneni <[email protected]>Suggested-by Nathan Lynch <[email protected]>---
v1 -> v2:
- Useusleep_rangeinstead of using RTAS sleep routine as suggested by Nathan
v2 -> v3:
- Sleep 10MSecs even for HCALL delay > 10MSecs and the other commit / comemnt changes as suggested by Nathan and Ellerman.--- arch/powerpc/platforms/pseries/vas.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/vas.c +24
-1@@ -38,7 +38,30 @@ static long hcall_return_busy_check(long rc) {/* Check if we are stalled for some time */ if (H_IS_LONG_BUSY (rc)) { -msleep (get_longbusy_msecs (rc)); +unsigned int ms; +/* +* Allocate, Modify and Deallocate HCALLs returns +* H_LONG_BUSY_ORDER_1_MSEC or H_LONG_BUSY_ORDER_10_MSEC +* for the long delay. So the sleep time should always +* be either 1 or 10msecs, but in case if the HCALL +* returns the long delay > 10 msecs, clamp the sleep +* time to 10msecs. +*/ + ms =clamp (get_longbusy_msecs (rc),1 ,10 ); + +/* +* msleep() will often sleep at least 20 msecs even +* though the hypervisor suggests that the OS reissue +* HCALLs after 1 or 10msecs. Also the delay hint from +* the HCALL is just a suggestion. So OK to pause for +* less time than the hinted delay. Use usleep_range() +* to ensure we don't sleep much longer than actually +* needed. +* +* See Documentation/timers/timers-howto.rst for +* explanation of the range used here. +*/ +usleep_range (ms *100 , ms *1000 ); rc =H_BUSY ; }else if (rc ==H_BUSY ) {cond_resched ();Signature · 1 line
2.26.3
Quoted 12 lines · Aneesh Kumar K.V (IBM)
Quoted 10 lines · Haren Myneni
VAS allocate, modify and deallocate HCALLs returns
H_LONG_BUSY_ORDER_1_MSECorH_LONG_BUSY_ORDER_10_MSECfor busy delay and expects OS to reissue HCALL after that delay. But usingmsleep()will often sleep at least 20 msecs even though the hypervisor suggests OS reissue these HCALLs after 1 or 10msecs. The open and close VAS window functions hold mutex and then issue these HCALLs. So these operations can take longer than the necessary when multiple threads issue open or close window APIs simultaneously.
Quoted 7 lines
Quoted 3 lines
So instead of
msleep(), useusleep_range()to ensure sleep with the expected value before issuing HCALL again.Can you summarize if there an user observable impact for the current code? We have other code paths using msleep(get_longbusy_msec()). Should we audit those usages?
As mentioned in the description, the open and close VAS window APIs can take longer with simultaneous calls, especially might affect the performance in the case of repeat open/close APIs for each compression request. On the large machine configuration which allows more simultaneous open windows (Ex: 240 cores provides 4800 VAS credits), the user can observe mutex contention around open/close HCAlls and hung-up traces in dmesg. I will repost the patch with this update in the commit message.
I think applicable to use the similar approach for other HCALLs (like in rtas_busy_delay()) but I have not seen any impact so far with other HCALLs. So we can add this change later.
Thanks
Haren
Quoted 55 lines
Quoted 55 lines
Signed-off-by Haren Myneni <[email protected]>Suggested-by Nathan Lynch <[email protected]>---
v1 -> v2:
- Useusleep_rangeinstead of using RTAS sleep routine as suggested by Nathan
v2 -> v3:
- Sleep 10MSecs even for HCALL delay > 10MSecs and the other commit / comemnt changes as suggested by Nathan and Ellerman.--- arch/powerpc/platforms/pseries/vas.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)arch/powerpc/platforms/pseries/vas.c +24
-1@@ -38,7 +38,30 @@ static long hcall_return_busy_check(long rc) {/* Check if we are stalled for some time */ if (H_IS_LONG_BUSY (rc)) { -msleep (get_longbusy_msecs (rc)); +unsigned int ms; +/* +* Allocate, Modify and Deallocate HCALLs returns +* H_LONG_BUSY_ORDER_1_MSEC or H_LONG_BUSY_ORDER_10_MSEC +* for the long delay. So the sleep time should always +* be either 1 or 10msecs, but in case if the HCALL +* returns the long delay > 10 msecs, clamp the sleep +* time to 10msecs. +*/ + ms =clamp (get_longbusy_msecs (rc),1 ,10 ); + +/* +* msleep() will often sleep at least 20 msecs even +* though the hypervisor suggests that the OS reissue +* HCALLs after 1 or 10msecs. Also the delay hint from +* the HCALL is just a suggestion. So OK to pause for +* less time than the hinted delay. Use usleep_range() +* to ensure we don't sleep much longer than actually +* needed. +* +* See Documentation/timers/timers-howto.rst for +* explanation of the range used here. +*/ +usleep_range (ms *100 , ms *1000 ); rc =H_BUSY ; }else if (rc ==H_BUSY ) {cond_resched ();Signature · 1 line
2.26.3