irq_pipeline: Prevent returning to user space with pending inband IRQs
PATCH 1/1 irq_pipeline: Prevent returning to user space with pending inband IRQs
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/entry-common.h +2 -1
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif
--
2.26.2
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0
include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif
Good spot, real issue here, just like the CPU_IDLE code deals with. The fix looks good, but maybe we could do even better.
The way local_irq_disable_full() works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loop may be at fault in this case, which does not traverses synchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.
Inverting the order of the actions in local_irq_disable_full() may be an option (inband_irq_disable would allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.
Signature · 1 line
Philippe.
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.
Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.
So, always this way?
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enable
Or even flip local_irq_enable_full as well?
Was there ever code that required ordering local_irq_disable_full the other way around?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enable
Yes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?
Not this one, hard irqs must be on before local_irq_enable() is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version of local_irq_enable() force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason, inband_irq_enable() expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.
Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?
After review, I don't think so. The current order for local_irq_disable_full() was rather determined by applying a reverse sequence compared to local_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.
Signature · 1 line
Philippe.
Quoted 78 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 1 line
Quoted 9 lines
Quoted 3 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.
Is there a commit somewhere that I can drop already?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?
Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean up stage_info field and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when calling irqentry_enter with oob stalled
x86: dovetail: Fix TS flag reservation
2. This one should be replaced by a fix in local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQs
However, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context:
stall_inband();
trap_handler();
hard_local_irq_enable()
hard_local_irq_disable()
if (state.stage_info == IRQENTRY_INBAND_UNSTALLED ||
state.stage_info == IRQENTRY_OOB) {
unstall_inband();
synchronize_pipeline();
}3. I believe local_irq_restore_full() is fragile, it mixes hard and virtual irq states carelessly. I'll likely issue a patch for this one after testing.
Signature · 1 line
Philippe.
Quoted 6 lines · Philippe Gerum
2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQs
Tests went ok. So [1] should be replaced by a patch inverting the order in local_irq_disable_full() as we discussed.
[1] irq_pipeline: Prevent returning to user space with pending inband IRQs
Signature · 1 line
Philippe.
Quoted 5 lines · Philippe Gerum
Quoted 4 lines · Philippe Gerum
2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
Quoted 7 lines
Quoted 2 lines
irq_pipeline: Prevent returning to user space with pending inband IRQsTests went ok. So [1] should be replaced by a patch inverting the order in
local_irq_disable_full()as we discussed.[1]
irq_pipeline: Prevent returning to user space with pending inband IRQs
Likewise for local_irq_save_full() which may cause the same problem, i.e.:
include/linux/irqflags.h +2 -2
@@ -242,14 +242,14 @@ do { \#define local_irq_disable_full () \ do { \ - local_irq_disable(); \ hard_local_irq_disable(); \ + local_irq_disable(); \ } while (0)#define local_irq_save_full (__flags) \ do { \ - local_irq_save(__flags); \ hard_local_irq_disable(); \ + local_irq_save(__flags); \ } while (0)#define local_irq_restore_full (__flags) \
Signature · 1 line
Philippe.
Quoted 32 lines · Philippe Gerum
Quoted 5 lines · Philippe Gerum
Quoted 4 lines · Philippe Gerum
2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:Quoted 7 lines
Quoted 2 lines
irq_pipeline: Prevent returning to user space with pending inband IRQsTests went ok. So [1] should be replaced by a patch inverting the order in
local_irq_disable_full()as we discussed.[1]
irq_pipeline: Prevent returning to user space with pending inband IRQsLikewise for
local_irq_save_full()which may cause the same problem, i.e.:include/linux/irqflags.h +1
-1@@ -242,14 +242,14 @@ do { \#define local_irq_disable_full () \ do { \ - local_irq_disable(); \ hard_local_irq_disable(); \ + local_irq_disable(); \
include/linux/irqflags.h do { \ · line 255
Quoted 10 lines
include/linux/irqflags.h +1
-1} while (0)#define local_irq_save_full (__flags) \ do { \ - local_irq_save(__flags); \ hard_local_irq_disable(); \ + local_irq_save(__flags); \ } while (0)#define local_irq_restore_full (__flags) \
As you tested already, I assume you will make the corresponding commit available via -rebase. Let me know otherwise if I should do that.
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 40 lines · Jan Kiszka
Quoted 39 lines · Philippe Gerum
Quoted 5 lines · Philippe Gerum
Quoted 4 lines · Philippe Gerum
2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:Quoted 7 lines
Quoted 2 lines
irq_pipeline: Prevent returning to user space with pending inband IRQsTests went ok. So [1] should be replaced by a patch inverting the order in
local_irq_disable_full()as we discussed.[1]
irq_pipeline: Prevent returning to user space with pending inband IRQsLikewise for
local_irq_save_full()which may cause the same problem, i.e.:include/linux/irqflags.h +2
-2@@ -242,14 +242,14 @@ do { \#define local_irq_disable_full () \ do { \ - local_irq_disable(); \ hard_local_irq_disable(); \ + local_irq_disable(); \ } while (0)#define local_irq_save_full (__flags) \ do { \ - local_irq_save(__flags); \ hard_local_irq_disable(); \ + local_irq_save(__flags); \ } while (0)
Quoted 8 lines
include/linux/irqflags.h do { \ · line 255
Quoted 3 lines
include/linux/irqflags.h +0
-0#define local_irq_restore_full (__flags) \As you tested already, I assume you will make the corresponding commit available via -rebase. Let me know otherwise if I should do that.
Jan
Yes, I'll do that.
Signature · 1 line
Philippe.
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context:
stall_inband();
trap_handler();
hard_local_irq_enable()
hard_local_irq_disable()
if (state.stage_info == IRQENTRY_INBAND_UNSTALLED ||
state.stage_info == IRQENTRY_OOB) {
unstall_inband();
synchronize_pipeline();Oh, unstall_inband does not synchronize? Because of hard_local_irq_disable or in general?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 124 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?
Ah, found it - unstall != inband_irq_enable. Why not using the latter directly?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();
Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?
...because it's instrumented to detect hard_irqs_disabled.
Can we safely sync at that point? Will that also ensure that an inband rescheduling event will not be delayed? We are not running irqentry_exit_cond_resched after that.
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 134 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?
Quoted 6 lines
...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point? Will that also ensure that an inband rescheduling event will not be delayed? We are not running
irqentry_exit_cond_reschedafter that.
We should have hit irqexit_may_preempt_schedule() and rescheduled appropriately before branching to unstall_inband(). Since we cannot log more IRQs in the meantime, we should not miss any rescheduling opportunity.
[Yes, __inband_irq_enable() reschedules, but does not perform any of the expected rcu tweaks, so we should not count on it in this particular case].
Signature · 1 line
Philippe.
Quoted 144 lines · Philippe Gerum
Quoted 134 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?Quoted 6 lines
...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point? Will that also ensure that an inband rescheduling event will not be delayed? We are not running
irqentry_exit_cond_reschedafter that.
Quoted 9 lines
We should have hit
irqexit_may_preempt_schedule()and rescheduled appropriately before branching tounstall_inband(). Since we cannot log more IRQs in the meantime, we should not miss any rescheduling opportunity.[Yes,
__inband_irq_enable()reschedules, but does not perform any of the expected rcu tweaks, so we should not count on it in this particular case].
I don't get this argument yet: irqexit_may_preempt_schedule() runs before we replayed the log, no? So it cannot act on any reschedules that only replay the log will mark pending.
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 130 lines · Jan Kiszka
Quoted 129 lines · Jan Kiszka
Quoted 127 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?
Quoted 7 lines
Quoted 3 lines
Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Signature · 1 line
Philippe.
Quoted 137 lines · Philippe Gerum
Quoted 134 lines · Jan Kiszka
Quoted 132 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?
Quoted 9 lines
Quoted 3 lines
...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via irqentry_exit_cond_resched(). I'll write that one.
Signature · 1 line
Philippe.
Quoted 151 lines · Philippe Gerum
Quoted 148 lines · Jan Kiszka
Quoted 146 lines · Philippe Gerum
Quoted 130 lines · Jan Kiszka
Quoted 129 lines · Jan Kiszka
Quoted 127 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Quoted 7 lines
Quoted 3 lines
Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Quoted 8 lines
Quoted 3 lines
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 156 lines · Jan Kiszka
Quoted 155 lines · Philippe Gerum
Quoted 151 lines · Jan Kiszka
Quoted 137 lines · Philippe Gerum
Quoted 134 lines · Jan Kiszka
Quoted 132 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 30 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?Quoted 9 lines
Quoted 3 lines
...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Quoted 9 lines
Quoted 4 lines
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Signature · 1 line
Philippe.
Quoted 163 lines · Philippe Gerum
Quoted 161 lines · Jan Kiszka
Quoted 159 lines · Philippe Gerum
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 49 lines · Philippe Gerum via Xenomai
Jan Kiszka <[email protected]> writes:
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the CPU_IDLE code deals with. The
fix looks good, but maybe we could do even better.The way local_irq_disable_full() works tends to introduce this issue in
a sneaky way, when the code path does not synchronize the interrupt log
(exit_to_user_mode_loop may be at fault in this case, which does not
traverses synchronize_pipeline()). Lack of synchronization would still
hit us if some trap handler turns hard irqs off -> on -> off the same
way, and we don't leave through the user exit path.Inverting the order of the actions in local_irq_disable_full() may be an
option (inband_irq_disable would allow that), making sure we cannot be
caught in the same issue when returning to kernel mode is another
way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.
Quoted 6 lines
Quoted 4 lines
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Please test this (heavily) with cobalt. This survived a 2-hours long kvm test here, running loops of the evl test suite + hackbench over 64 vcpus, so far so good. No reason for any difference in behavior between cobalt and evl in this area, this is merely a dovetail business.
commit 30808d958277a525baad33f3c68ccb7910dc3176 (HEAD -> rebase/v5.10.y-evl)
Author: Philippe Gerum <[email protected]>
genirq: irq_pipeline: synchronize log on irq exit to kernel
We must make sure to play any IRQ which might be pending in the in-band log before leaving an interrupt frame for a preempted kernel context.
This completes "irq_pipeline: Account for stage migration across faults", so that we synchronize the log once the in-band stage is unstalled. In addition, we also care to do this before preempt_schedule_irq() runs, so that we won't miss any rescheduling request which might have been triggered by some IRQ we just played.
kernel/entry/common.c +57 -12
@@ -477,8 +477,57 @@ bool irqexit_may_preempt_schedule(irqentry_state_t state,#endif +#ifdef CONFIG_IRQ_PIPELINE + +static bool irqentry_syncstage (irqentry_state_t state)/* hard irqs off */ +{ +/* +* If pipelining interrupts, enable in-band IRQs then +* synchronize the interrupt log on exit if: +* +* - irqentry_enter() stalled the stage in order to mirror the +* hardware state. +* +* - we where coming from oob, thus went through a stage migration +* that was caused by taking a CPU exception, e.g., a fault. +* +* We run before preempt_schedule_irq() may be called later on +* by preemptible kernels, so that any rescheduling request +* triggered by in-band IRQ handlers is considered. +*/ +if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || + state.stage_info ==IRQENTRY_OOB ) { +unstall_inband_nocheck (); +synchronize_pipeline_on_irq (); +stall_inband_nocheck (); +return true; + } + +return false; +} + +static void irqentry_unstall (void ) +{ +unstall_inband_nocheck (); +} + +#else + +static bool irqentry_syncstage (irqentry_state_t state) +{ +return false; +} + +static void irqentry_unstall (void ) +{ +} + +#endif + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { +bool synchronized = false; +if (running_oob ())return ;
@@ -488,7 +537,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);return ; - }else if (irqexit_may_preempt_schedule (state, regs)) { + } + + synchronized =irqentry_syncstage (state); + +if (irqexit_may_preempt_schedule (state, regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing
@@ -521,17 +574,9 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) } out: -#ifdef CONFIG_IRQ_PIPELINE -/* -* If pipelining interrupts, clear the in-band stall bit if -* irqentry_enter() raised it in order to mirror the hardware -* state. Also clear it when we where coming from oob, thus went -* through a migration that was caused by taking, e.g., a fault. -*/ -if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || - state.stage_info ==IRQENTRY_OOB ) -unstall_inband (); -#endif +if (synchronized) +irqentry_unstall (); +return ; }
Signature · 1 line
Philippe.
Quoted 283 lines · Philippe Gerum
Quoted 163 lines · Philippe Gerum
Quoted 161 lines · Jan Kiszka
Quoted 159 lines · Philippe Gerum
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 61 lines · Jan Kiszka
On 06.06.21 16:43, Philippe Gerum via Xenomai wrote:
Jan Kiszka <[email protected]> writes:
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the CPU_IDLE code deals with. The
fix looks good, but maybe we could do even better.The way local_irq_disable_full() works tends to introduce this issue in
a sneaky way, when the code path does not synchronize the interrupt log
(exit_to_user_mode_loop may be at fault in this case, which does not
traverses synchronize_pipeline()). Lack of synchronization would still
hit us if some trap handler turns hard irqs off -> on -> off the same
way, and we don't leave through the user exit path.Inverting the order of the actions in local_irq_disable_full() may be an
option (inband_irq_disable would allow that), making sure we cannot be
caught in the same issue when returning to kernel mode is another
way. This needs more thought I think.So, always this way?
local_irq_disable_full:
hard_local_irq_disable
local_irq_disablelocal_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip local_irq_enable_full as well?
Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering local_irq_disable_full the
other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.Quoted 6 lines
Quoted 4 lines
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Please test this (heavily) with cobalt. This survived a 2-hours long kvm test here, running loops of the evl test suite + hackbench over 64 vcpus, so far so good. No reason for any difference in behavior between cobalt and evl in this area, this is merely a dovetail business.
commit
30808d958277a525baad33f3c68ccb7910dc3176(HEAD -> rebase/v5.10.y-evl)
Author: Philippe Gerum <[email protected]>genirq:
irq_pipeline: synchronize log on irq exit to kernelWe must make sure to play any IRQ which might be pending in the in-band log before leaving an interrupt frame for a preempted kernel context.
This completes "
irq_pipeline: Account for stage migration across faults", so that we synchronize the log once the in-band stage is unstalled. In addition, we also care to do this beforepreempt_schedule_irq()runs, so that we won't miss any rescheduling request which might have been triggered by some IRQ we just played.Signed-off-by Philippe Gerum <[email protected]>Suggested-by Jan Kiszka <[email protected]>kernel/entry/common.c +54
-8@@ -477,8 +477,57 @@ bool irqexit_may_preempt_schedule(irqentry_state_t state,#endif +#ifdef CONFIG_IRQ_PIPELINE + +static bool irqentry_syncstage (irqentry_state_t state)/* hard irqs off */ +{ +/* +* If pipelining interrupts, enable in-band IRQs then +* synchronize the interrupt log on exit if: +* +* - irqentry_enter() stalled the stage in order to mirror the +* hardware state. +* +* - we where coming from oob, thus went through a stage migration +* that was caused by taking a CPU exception, e.g., a fault. +* +* We run before preempt_schedule_irq() may be called later on +* by preemptible kernels, so that any rescheduling request +* triggered by in-band IRQ handlers is considered. +*/ +if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || + state.stage_info ==IRQENTRY_OOB ) { +unstall_inband_nocheck (); +synchronize_pipeline_on_irq (); +stall_inband_nocheck (); +return true; + } + +return false; +} + +static void irqentry_unstall (void ) +{ +unstall_inband_nocheck (); +} + +#else + +static bool irqentry_syncstage (irqentry_state_t state) +{ +return false; +} + +static void irqentry_unstall (void ) +{ +} + +#endif + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { +bool synchronized = false; +if (running_oob ())return ;@@ -488,7 +537,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);return ; - }else if (irqexit_may_preempt_schedule (state, regs)) { + } + + synchronized =irqentry_syncstage (state); + +if (irqexit_may_preempt_schedule (state, regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing @@ -521,17 +574,9 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) } out: -#ifdef CONFIG_IRQ_PIPELINE -/* -* If pipelining interrupts, clear the in-band stall bit if -* irqentry_enter() raised it in order to mirror the hardware -* state. Also clear it when we where coming from oob, thus went -* through a migration that was caused by taking, e.g., a fault. -*/
kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 582
Quoted 10 lines
kernel/entry/common.c +3
-4-if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || - state.stage_info ==IRQENTRY_OOB ) -unstall_inband (); -#endif +if (synchronized) +irqentry_unstall (); +return ; }
I've put that into a staging/v5.10.y-dovetail branch, pointed xenomai-images to that and started a run in our lab. That should give us some confidence that things work under cobalt across all archs, and we can make that commit official.
However, heavy stressing would require triggering migration-on-fault in tighter loops while running something like stress-ng in the background. Maybe it would make sense to add such a pattern to the switchtest (would also require a way to silence the related kernel warnings during runtime, not only build-time)?
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 300 lines · Jan Kiszka
Quoted 283 lines · Philippe Gerum
Quoted 163 lines · Philippe Gerum
Quoted 161 lines · Jan Kiszka
Quoted 159 lines · Philippe Gerum
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Jan Kiszka <[email protected]> writes:
On 06.06.21 16:43, Philippe Gerum via Xenomai wrote:
Jan Kiszka <[email protected]> writes:
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the CPU_IDLE code deals with. The
fix looks good, but maybe we could do even better.The way local_irq_disable_full() works tends to introduce this issue in
a sneaky way, when the code path does not synchronize the interrupt log
(exit_to_user_mode_loop may be at fault in this case, which does not
traverses synchronize_pipeline()). Lack of synchronization would still
hit us if some trap handler turns hard irqs off -> on -> off the same
way, and we don't leave through the user exit path.Inverting the order of the actions in local_irq_disable_full() may be an
option (inband_irq_disable would allow that), making sure we cannot be
caught in the same issue when returning to kernel mode is another
way. This needs more thought I think.So, always this way?
local_irq_disable_full:
hard_local_irq_disable
local_irq_disablelocal_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Or even flip local_irq_enable_full as well?
Not this one, hard irqs must be on before local_irq_enable() is issued
otherwise we would trigger a debug assertion. The reason for such
assertion is that the I-pipe version of local_irq_enable() force enables
hard irqs on, and this is something we want to depart from because
whether the caller expects or not such side-effect is error-prone. For
this reason, inband_irq_enable() expects hard irqs on, which should be
the current hard irq state for the caller under normal circumstances.Was there ever code that required ordering local_irq_disable_full the
other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full() was rather determined by applying a reverse
sequence compared to local_irq_enable_full(). So this looks good. I need
to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.Quoted 6 lines
Quoted 4 lines
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Please test this (heavily) with cobalt. This survived a 2-hours long kvm test here, running loops of the evl test suite + hackbench over 64 vcpus, so far so good. No reason for any difference in behavior between cobalt and evl in this area, this is merely a dovetail business.
commit
30808d958277a525baad33f3c68ccb7910dc3176(HEAD -> rebase/v5.10.y-evl)
Author: Philippe Gerum <[email protected]>genirq:
irq_pipeline: synchronize log on irq exit to kernelWe must make sure to play any IRQ which might be pending in the in-band log before leaving an interrupt frame for a preempted kernel context.
This completes "
irq_pipeline: Account for stage migration across faults", so that we synchronize the log once the in-band stage is unstalled. In addition, we also care to do this beforepreempt_schedule_irq()runs, so that we won't miss any rescheduling request which might have been triggered by some IRQ we just played.Signed-off-by Philippe Gerum <[email protected]>Suggested-by Jan Kiszka <[email protected]>kernel/entry/common.c +54
-8@@ -477,8 +477,57 @@ bool irqexit_may_preempt_schedule(irqentry_state_t state,#endif +#ifdef CONFIG_IRQ_PIPELINE + +static bool irqentry_syncstage (irqentry_state_t state)/* hard irqs off */ +{ +/* +* If pipelining interrupts, enable in-band IRQs then +* synchronize the interrupt log on exit if: +* +* - irqentry_enter() stalled the stage in order to mirror the +* hardware state. +* +* - we where coming from oob, thus went through a stage migration +* that was caused by taking a CPU exception, e.g., a fault. +* +* We run before preempt_schedule_irq() may be called later on +* by preemptible kernels, so that any rescheduling request +* triggered by in-band IRQ handlers is considered. +*/ +if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || + state.stage_info ==IRQENTRY_OOB ) { +unstall_inband_nocheck (); +synchronize_pipeline_on_irq (); +stall_inband_nocheck (); +return true; + } + +return false; +} + +static void irqentry_unstall (void ) +{ +unstall_inband_nocheck (); +} + +#else + +static bool irqentry_syncstage (irqentry_state_t state) +{ +return false; +} + +static void irqentry_unstall (void ) +{ +} + +#endif + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { +bool synchronized = false; +if (running_oob ())return ;@@ -488,7 +537,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);return ; - }else if (irqexit_may_preempt_schedule (state, regs)) { + } + + synchronized =irqentry_syncstage (state); + +if (irqexit_may_preempt_schedule (state, regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing @@ -521,17 +574,9 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) } out: -#ifdef CONFIG_IRQ_PIPELINE -/* -* If pipelining interrupts, clear the in-band stall bit if -* irqentry_enter() raised it in order to mirror the hardware -* state. Also clear it when we where coming from oob, thus went -* through a migration that was caused by taking, e.g., a fault. -*/ kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 582
Quoted 10 lines
kernel/entry/common.c +3
-4-if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || - state.stage_info ==IRQENTRY_OOB ) -unstall_inband (); -#endif +if (synchronized) +irqentry_unstall (); +return ; }I've put that into a staging/v5.10.y-dovetail branch, pointed xenomai-images to that and started a run in our lab. That should give us some confidence that things work under cobalt across all archs, and we can make that commit official.
Quoted 6 lines
However, heavy stressing would require triggering migration-on-fault in tighter loops while running something like stress-ng in the background. Maybe it would make sense to add such a pattern to the switchtest (would also require a way to silence the related kernel warnings during runtime, not only build-time)?
switchtest aims at exercising the core scheduling logic including the transition between execution stages, so migration-on-fault would fit there indeed.
Signature · 1 line
Philippe.
Quoted 310 lines · Philippe Gerum
Quoted 300 lines · Jan Kiszka
Quoted 283 lines · Philippe Gerum
Quoted 163 lines · Philippe Gerum
Quoted 161 lines · Jan Kiszka
Quoted 159 lines · Philippe Gerum
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 91 lines · Jan Kiszka
On 07.06.21 09:17, Philippe Gerum wrote:
Jan Kiszka <[email protected]> writes:
On 06.06.21 16:43, Philippe Gerum via Xenomai wrote:
Jan Kiszka <[email protected]> writes:
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the CPU_IDLE code deals with. The
fix looks good, but maybe we could do even better.The way local_irq_disable_full() works tends to introduce this issue in
a sneaky way, when the code path does not synchronize the interrupt log
(exit_to_user_mode_loop may be at fault in this case, which does not
traverses synchronize_pipeline()). Lack of synchronization would still
hit us if some trap handler turns hard irqs off -> on -> off the same
way, and we don't leave through the user exit path.Inverting the order of the actions in local_irq_disable_full() may be an
option (inband_irq_disable would allow that), making sure we cannot be
caught in the same issue when returning to kernel mode is another
way. This needs more thought I think.So, always this way?
local_irq_disable_full:
hard_local_irq_disable
local_irq_disablelocal_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Or even flip local_irq_enable_full as well?
Not this one, hard irqs must be on before local_irq_enable() is issued
otherwise we would trigger a debug assertion. The reason for such
assertion is that the I-pipe version of local_irq_enable() force enables
hard irqs on, and this is something we want to depart from because
whether the caller expects or not such side-effect is error-prone. For
this reason, inband_irq_enable() expects hard irqs on, which should be
the current hard irq state for the caller under normal circumstances.Was there ever code that required ordering local_irq_disable_full the
other way around?After review, I don't think so. The current order for
local_irq_disable_full() was rather determined by applying a reverse
sequence compared to local_irq_enable_full(). So this looks good. I need
to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.Quoted 6 lines
Quoted 4 lines
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Please test this (heavily) with cobalt. This survived a 2-hours long kvm test here, running loops of the evl test suite + hackbench over 64 vcpus, so far so good. No reason for any difference in behavior between cobalt and evl in this area, this is merely a dovetail business.
commit
30808d958277a525baad33f3c68ccb7910dc3176(HEAD -> rebase/v5.10.y-evl)
Author: Philippe Gerum <[email protected]>genirq:
irq_pipeline: synchronize log on irq exit to kernelWe must make sure to play any IRQ which might be pending in the in-band log before leaving an interrupt frame for a preempted kernel context.
This completes "
irq_pipeline: Account for stage migration across faults", so that we synchronize the log once the in-band stage is unstalled. In addition, we also care to do this beforepreempt_schedule_irq()runs, so that we won't miss any rescheduling request which might have been triggered by some IRQ we just played.Signed-off-by Philippe Gerum <[email protected]>Suggested-by Jan Kiszka <[email protected]>kernel/entry/common.c +54
-8@@ -477,8 +477,57 @@ bool irqexit_may_preempt_schedule(irqentry_state_t state,#endif +#ifdef CONFIG_IRQ_PIPELINE + +static bool irqentry_syncstage (irqentry_state_t state)/* hard irqs off */ +{ +/* +* If pipelining interrupts, enable in-band IRQs then +* synchronize the interrupt log on exit if: +* +* - irqentry_enter() stalled the stage in order to mirror the +* hardware state. +* +* - we where coming from oob, thus went through a stage migration +* that was caused by taking a CPU exception, e.g., a fault. +* +* We run before preempt_schedule_irq() may be called later on +* by preemptible kernels, so that any rescheduling request +* triggered by in-band IRQ handlers is considered. +*/ +if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || + state.stage_info ==IRQENTRY_OOB ) { +unstall_inband_nocheck (); +synchronize_pipeline_on_irq (); +stall_inband_nocheck (); +return true; + } + +return false; +} + +static void irqentry_unstall (void ) +{ +unstall_inband_nocheck (); +} + +#else + +static bool irqentry_syncstage (irqentry_state_t state) +{ +return false; +} + +static void irqentry_unstall (void ) +{ +} + +#endif + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { +bool synchronized = false; +if (running_oob ())return ;@@ -488,7 +537,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);return ; - }else if (irqexit_may_preempt_schedule (state, regs)) { + } + + synchronized =irqentry_syncstage (state); + +if (irqexit_may_preempt_schedule (state, regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing @@ -521,17 +574,9 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) } out: -#ifdef CONFIG_IRQ_PIPELINE -/* -* If pipelining interrupts, clear the in-band stall bit if -* irqentry_enter() raised it in order to mirror the hardware -* state. Also clear it when we where coming from oob, thus went -* through a migration that was caused by taking, e.g., a fault. -*/ kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 582
Quoted 10 lines
kernel/entry/common.c +3
-4-if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || - state.stage_info ==IRQENTRY_OOB ) -unstall_inband (); -#endif +if (synchronized) +irqentry_unstall (); +return ; }I've put that into a staging/v5.10.y-dovetail branch, pointed xenomai-images to that and started a run in our lab. That should give us some confidence that things work under cobalt across all archs, and we can make that commit official.
Quoted 6 lines
However, heavy stressing would require triggering migration-on-fault in tighter loops while running something like stress-ng in the background. Maybe it would make sense to add such a pattern to the switchtest (would also require a way to silence the related kernel warnings during runtime, not only build-time)?
Quoted 4 lines
switchtest aims at exercising the core scheduling logic including the transition between execution stages, so migration-on-fault would fit there indeed.
Do we need this testcase first? My feeling from review and testing is that is safe to move forward with the patch already.
Jan
Signature · 2 lines
Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux
Quoted 311 lines · Jan Kiszka
Quoted 310 lines · Philippe Gerum
Quoted 306 lines · Jan Kiszka
Quoted 283 lines · Philippe Gerum
Quoted 163 lines · Philippe Gerum
Quoted 161 lines · Jan Kiszka
Quoted 159 lines · Philippe Gerum
Quoted 142 lines · Jan Kiszka
Quoted 141 lines · Philippe Gerum
Quoted 137 lines · Jan Kiszka
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Jan Kiszka <[email protected]> writes:
On 07.06.21 09:17, Philippe Gerum wrote:
Jan Kiszka <[email protected]> writes:
On 06.06.21 16:43, Philippe Gerum via Xenomai wrote:
Jan Kiszka <[email protected]> writes:
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also
I-pipe. Otherwise, we risk to take an interrupt for the inband stage but
do not inject it before returning to user space.Signed-off-by: Jan Kiszka <[email protected]>
---Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into
dovetail..include/linux/entry-common.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 0fb45b2d6094..00540110985e 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);
#ifndef local_irq_disable_exit_to_user
static inline void local_irq_disable_exit_to_user(void)
{
- local_irq_disable_full();
+ hard_cond_local_irq_disable();
+ local_irq_disable();
}
#endifGood spot, real issue here, just like the CPU_IDLE code deals with. The
fix looks good, but maybe we could do even better.The way local_irq_disable_full() works tends to introduce this issue in
a sneaky way, when the code path does not synchronize the interrupt log
(exit_to_user_mode_loop may be at fault in this case, which does not
traverses synchronize_pipeline()). Lack of synchronization would still
hit us if some trap handler turns hard irqs off -> on -> off the same
way, and we don't leave through the user exit path.Inverting the order of the actions in local_irq_disable_full() may be an
option (inband_irq_disable would allow that), making sure we cannot be
caught in the same issue when returning to kernel mode is another
way. This needs more thought I think.So, always this way?
local_irq_disable_full:
hard_local_irq_disable
local_irq_disablelocal_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Or even flip local_irq_enable_full as well?
Not this one, hard irqs must be on before local_irq_enable() is issued
otherwise we would trigger a debug assertion. The reason for such
assertion is that the I-pipe version of local_irq_enable() force enables
hard irqs on, and this is something we want to depart from because
whether the caller expects or not such side-effect is error-prone. For
this reason, inband_irq_enable() expects hard irqs on, which should be
the current hard irq state for the caller under normal circumstances.Was there ever code that required ordering local_irq_disable_full the
other way around?After review, I don't think so. The current order for
local_irq_disable_full() was rather determined by applying a reverse
sequence compared to local_irq_enable_full(). So this looks good. I need
to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean up stage_info field and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when calling irqentry_enter with oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in local_irq_disable_full(),
pending some tests ongoing here:irq_pipeline: Prevent returning to user space with pending inband IRQs
However, as I mentioned earlier, this may not be sufficient per se, we
may have to synchronize the in-band log when unstalling the stage on the
kernel exit path, i.e. the exit point fixed up by the "Account for stage
migration across faults" patch. This would address the following
scenario:irqentry_enter on kernel context:
stall_inband();
trap_handler();
hard_local_irq_enable()
hard_local_irq_disable()
if (state.stage_info == IRQENTRY_INBAND_UNSTALLED ||
state.stage_info == IRQENTRY_OOB) {
unstall_inband();
synchronize_pipeline();Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?...because it's instrumented to detect
hard_irqs_disabled.Can we safely sync at that point?
Quoted 9 lines
Quoted 5 lines
Yes we may do this, because we restore the original -unstalled- state of the in-band stage. So if we have pending IRQs, we may play them at this point. What we may not do obviously, is creating synchronization points where the kernel does not expect irqs.
Again: When will any reschedules that might be requested by those replayed IRQs be handled then?
Nowhere unless we provide a log synchronization routine dedicated to the in-band kernel exit path, which would fold in the rescheduling call via
irqentry_exit_cond_resched(). I'll write that one.Quoted 6 lines
Quoted 4 lines
How about simply moving the replay up, before the kernel checks for pending reschedules anyway? We can stall inband then again, to please lockdep & Co., as long as there is no hard-irq-on section before the intret.
Yes, this is the right way to do this. I'm on it.
Please test this (heavily) with cobalt. This survived a 2-hours long kvm test here, running loops of the evl test suite + hackbench over 64 vcpus, so far so good. No reason for any difference in behavior between cobalt and evl in this area, this is merely a dovetail business.
commit
30808d958277a525baad33f3c68ccb7910dc3176(HEAD -> rebase/v5.10.y-evl)
Author: Philippe Gerum <[email protected]>genirq:
irq_pipeline: synchronize log on irq exit to kernelWe must make sure to play any IRQ which might be pending in the in-band log before leaving an interrupt frame for a preempted kernel context.
This completes "
irq_pipeline: Account for stage migration across faults", so that we synchronize the log once the in-band stage is unstalled. In addition, we also care to do this beforepreempt_schedule_irq()runs, so that we won't miss any rescheduling request which might have been triggered by some IRQ we just played.Signed-off-by Philippe Gerum <[email protected]>Suggested-by Jan Kiszka <[email protected]>kernel/entry/common.c +54
-8@@ -477,8 +477,57 @@ bool irqexit_may_preempt_schedule(irqentry_state_t state,#endif +#ifdef CONFIG_IRQ_PIPELINE + +static bool irqentry_syncstage (irqentry_state_t state)/* hard irqs off */ +{ +/* +* If pipelining interrupts, enable in-band IRQs then +* synchronize the interrupt log on exit if: +* +* - irqentry_enter() stalled the stage in order to mirror the +* hardware state. +* +* - we where coming from oob, thus went through a stage migration +* that was caused by taking a CPU exception, e.g., a fault. +* +* We run before preempt_schedule_irq() may be called later on +* by preemptible kernels, so that any rescheduling request +* triggered by in-band IRQ handlers is considered. +*/ +if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || + state.stage_info ==IRQENTRY_OOB ) { +unstall_inband_nocheck (); +synchronize_pipeline_on_irq (); +stall_inband_nocheck (); +return true; + } + +return false; +} + +static void irqentry_unstall (void ) +{ +unstall_inband_nocheck (); +} + +#else + +static bool irqentry_syncstage (irqentry_state_t state) +{ +return false; +} + +static void irqentry_unstall (void ) +{ +} + +#endif + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) { +bool synchronized = false; +if (running_oob ())return ;@@ -488,7 +537,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);return ; - }else if (irqexit_may_preempt_schedule (state, regs)) { + } + + synchronized =irqentry_syncstage (state); + +if (irqexit_may_preempt_schedule (state, regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing @@ -521,17 +574,9 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) } out: -#ifdef CONFIG_IRQ_PIPELINE -/* -* If pipelining interrupts, clear the in-band stall bit if -* irqentry_enter() raised it in order to mirror the hardware -* state. Also clear it when we where coming from oob, thus went -* through a migration that was caused by taking, e.g., a fault. -*/ kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 582
Quoted 10 lines
kernel/entry/common.c +3
-4-if (state.stage_info ==IRQENTRY_INBAND_UNSTALLED || - state.stage_info ==IRQENTRY_OOB ) -unstall_inband (); -#endif +if (synchronized) +irqentry_unstall (); +return ; }I've put that into a staging/v5.10.y-dovetail branch, pointed xenomai-images to that and started a run in our lab. That should give us some confidence that things work under cobalt across all archs, and we can make that commit official.
However, heavy stressing would require triggering migration-on-fault in tighter loops while running something like stress-ng in the background. Maybe it would make sense to add such a pattern to the switchtest (would also require a way to silence the related kernel warnings during runtime, not only build-time)?
Quoted 8 lines
Quoted 4 lines
switchtest aims at exercising the core scheduling logic including the transition between execution stages, so migration-on-fault would fit there indeed.
Do we need this testcase first? My feeling from review and testing is that is safe to move forward with the patch already.
Looks ok here too. The patch is fairly straightforward, and this has been tested on the EVL side of the universe on a handful of armv7, armv8 machines without any glitch. I'll merge this patch shortly.
Signature · 1 line
Philippe.
Quoted 125 lines · Jan Kiszka
Quoted 124 lines · Jan Kiszka
Quoted 122 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();
Quoted 7 lines
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?Ah, found it - unstall !=
inband_irq_enable. Why not using the latter directly?
Well, in this case I should have, hence the bug.
Signature · 1 line
Philippe.
Quoted 124 lines · Jan Kiszka
Quoted 113 lines · Philippe Gerum
Quoted 83 lines · Jan Kiszka
Quoted 82 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka
Quoted 44 lines · Philippe Gerum via Xenomai
Quoted 22 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
The correct order is hard IRQs off first, then stalling inband, see also I-pipe. Otherwise, we risk to take an interrupt for the inband stage but do not inject it before returning to user space.
Signed-off-by Jan Kiszka <[email protected]>---
Fixes the RCU stalls Florian reported, at least for me.
I'm afraid this wasn't the last regression of translating I-pipe into dovetail..
include/linux/entry-common.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/entry-common.h +0
-0include/linux/entry-common.h static inline void local_irq_disable_exit_to_user(void); · line 209
Quoted 8 lines
include/linux/entry-common.h +2
-1@@ -203,7 +203,8 @@ static inline void local_irq_disable_exit_to_user(void);#ifndef local_irq_disable_exit_to_userstatic inline void local_irq_disable_exit_to_user (void ) { -local_irq_disable_full (); +hard_cond_local_irq_disable (); +local_irq_disable (); }#endif Good spot, real issue here, just like the
CPU_IDLEcode deals with. The fix looks good, but maybe we could do even better.The way
local_irq_disable_full()works tends to introduce this issue in a sneaky way, when the code path does not synchronize the interrupt log (exit_to_user_mode_loopmay be at fault in this case, which does not traversessynchronize_pipeline()). Lack of synchronization would still hit us if some trap handler turns hard irqs off -> on -> off the same way, and we don't leave through the user exit path.Quoted 5 lines
Inverting the order of the actions in
local_irq_disable_full()may be an option (inband_irq_disablewould allow that), making sure we cannot be caught in the same issue when returning to kernel mode is another way. This needs more thought I think.So, always this way?
Quoted 8 lines
local_irq_disable_full:
hard_local_irq_disable
local_irq_disable
local_irq_enable_full:
hard_local_irq_enable
local_irq_enableYes.
Quoted 1 line
Or even flip
local_irq_enable_fullas well?Not this one, hard irqs must be on before
local_irq_enable()is issued otherwise we would trigger a debug assertion. The reason for such assertion is that the I-pipe version oflocal_irq_enable()force enables hard irqs on, and this is something we want to depart from because whether the caller expects or not such side-effect is error-prone. For this reason,inband_irq_enable()expects hard irqs on, which should be the current hard irq state for the caller under normal circumstances.Quoted 4 lines
Was there ever code that required ordering
local_irq_disable_fullthe other way around?Quoted 8 lines
Quoted 5 lines
After review, I don't think so. The current order for
local_irq_disable_full()was rather determined by applying a reverse sequence compared tolocal_irq_enable_full(). So this looks good. I need to test this on the armv[7/8] side here first before merging.Is there a commit somewhere that I can drop already?
1. These are fine with me in their latest iteration:
irq_pipeline: Clean upstage_infofield and users
irq_pipeline: Account for stage migration across faults
irq_pipeline: Warn when callingirqentry_enterwith oob stalled
x86: dovetail: Fix TS flag reservation2. This one should be replaced by a fix in
local_irq_disable_full(), pending some tests ongoing here:
irq_pipeline: Prevent returning to user space with pending inband IRQsHowever, as I mentioned earlier, this may not be sufficient per se, we may have to synchronize the in-band log when unstalling the stage on the kernel exit path, i.e. the exit point fixed up by the "Account for stage migration across faults" patch. This would address the following scenario:
Quoted 9 lines
irqentry_enter on kernel context: stall_inband(); trap_handler(); hard_local_irq_enable() hard_local_irq_disable() if (state.stage_info == IRQENTRY_INBAND_UNSTALLED || state.stage_info == IRQENTRY_OOB) { unstall_inband(); synchronize_pipeline();
Quoted 3 lines
Oh,
unstall_inbanddoes not synchronize? Because ofhard_local_irq_disableor in general?
In general, this is only a bit flip. The caller has to do the right thing afterwards, e.g. inband_irq_enable() or inband_irq_restore() do so.
Signature · 1 line
Philippe.