Question about interrupt enabling/disabling in kernel exit path
Hello:
I have a question about enabling interrupts in the kernel exit path. Especially, in do_notify_resume_user(). I have work that I need to defer to this point, but it needs to run with interrupts enabled. I wanted to piggy back off TIF_NOTIFY_RESUME, because that is more or less generic code and gets handled in a C function where I can play the usual header games: static inline wrapper function when feature is configured; null macro when not.
I see a comment at the beginning of ia64_leave_kernel that says that "work.need_resched, etc. mustn't get changed by this CPU before it returns to user- or fsys-mode, hence we disable interrupts early on." [This may be a stale comment. I.e., we now check for resched needed in current task's thread_info flags.] But, then I see later that when CONFIG_PREEMPT is set, we do enable interrupts and then re-disable them.
Then I notice that after we return from do_notify_resume_user() we don't
recheck TIF_NEED_RESCHED, etc. in the thread_info flags. So, maybe
this is the issue? If I open an interrupt window in do_notify_resume_user(), 'NEED_RESCHED might get set and we wouldn't notice it on return to the assembly language code. [I might even sleep here. Is that a problem?]
I took a look at the x86_64 and i386 versions and they always goes back to check for resched after calling their version of do_notify_resume() [no '_user suffix on the routine]. If this were the only issue with enabling interrupts in do_notify_resume_user(), I could change it to return non-zero if recheck is needed [zero in the usual case] and then go back and recheck thread_info flags.
Regards,
Lee
Lee,
As you may have seen there is a perfmon callback from do_notify_resume(). I am also using TIF_NOTIFY_RESUME to get back into perfmon just before leaving the kernel.
It is not true that interrupts are always off when getting there. Last year we ran into a issue for this reason, i.e., I was assume interrupts were always masked when getting there. But apparentlyy, that is not always the case. I don't recall exactly what is the path to get to this situation but I know it exists.
Quoted 39 lines · Lee Schermerhorn
Hello:
I have a question about enabling interrupts in the kernel exit path. Especially, in
do_notify_resume_user(). I have work that I need to defer to this point, but it needs to run with interrupts enabled. I wanted to piggy back offTIF_NOTIFY_RESUME, because that is more or less generic code and gets handled in a C function where I can play the usual header games: static inline wrapper function when feature is configured; null macro when not.I see a comment at the beginning of
ia64_leave_kernelthat says that "work.need_resched, etc. mustn't get changed by this CPU before it returns to user- or fsys-mode, hence we disable interrupts early on." [This may be a stale comment. I.e., we now check for resched needed in current task'sthread_infoflags.] But, then I see later that whenCONFIG_PREEMPTis set, we do enable interrupts and then re-disable them.Then I notice that after we return from
do_notify_resume_user()we don't
recheckTIF_NEED_RESCHED, etc. in thethread_infoflags. So, maybe
this is the issue? If I open an interrupt window indo_notify_resume_user(), 'NEED_RESCHEDmight get set and we wouldn't notice it on return to the assembly language code. [I might even sleep here. Is that a problem?]I took a look at the
x86_64and i386 versions and they always goes back to check for resched after calling their version ofdo_notify_resume()[no '_usersuffix on the routine]. If this were the only issue with enabling interrupts indo_notify_resume_user(), I could change it to return non-zero if recheck is needed [zero in the usual case] and then go back and recheckthread_infoflags.Regards,
Lee-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
-Stephane
Quoted 2 lines
Then I notice that after we return from
do_notify_resume_user()we don't recheckTIF_NEED_RESCHED, etc. in thethread_infoflags.
Why do you say this? It's not true. After processing pending work, we go back to .work_processed_{syscall,kernel), which will again invoke .work_pending, if necessary.
AFAIR, it's OK to reenable interrupts in notify_resume_user() because we know we're at the top of the kernel stack (i.e., there is no risk of nesting the kernel stack too deeply due to pending interrupts), provided the interrupts are off again when resuming execution at .work_processed_kernel.
--david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536
Quoted 6 lines · David Mosberger-Tang
Quoted 2 lines
Then I notice that after we return from
do_notify_resume_user()we don't recheckTIF_NEED_RESCHED, etc. in thethread_infoflags.Why do you say this? It's not true. After processing pending work, we go back to .work_processed_{syscall,kernel), which will again invoke .work_pending, if necessary.
OK. I guess I'm misinterpreting the comments on the line labeled '.ret10:'.
Say's it's setting p6 to zero, so no recheck. I'm not fluent [writing nor reading] ia64 assembler, but it appeared to me that with p6 set to zero on return from notify_resume(), the checks for more work back at .work_processed_* would be disabled.
Are the comments at the ia64_leave_kernel entry point stale?
Quoted 6 lines
AFAIR, it's OK to reenable interrupts in
notify_resume_user()because we know we're at the top of the kernel stack (i.e., there is no risk of nesting the kernel stack too deeply due to pending interrupts), provided the interrupts are off again when resuming execution at .work_processed_kernel.
OK. Ken said he thought it was OK to disable. And Stephane mentioned that he's seen interrupts enabled when entering do_notify_resume_user() as well.
Thanks,
Lee
Lee Schermerhorn wrote on Tuesday, November 15, 2005 12:21 PM
Quoted 8 lines · David Mosberger-Tang
Quoted 6 lines
Quoted 2 lines
Then I notice that after we return from
do_notify_resume_user()we don't recheckTIF_NEED_RESCHED, etc. in thethread_infoflags.Why do you say this? It's not true. After processing pending work, we go back to .work_processed_{syscall,kernel), which will again invoke .work_pending, if necessary.
Quoted 7 lines
OK. I guess I'm misinterpreting the comments on the line labeled '.ret10:'.
Say's it's setting p6 to zero, so no recheck. I'm not fluent [writing nor reading] ia64 assembler, but it appeared to me that with p6 set to zero on return fromnotify_resume(), the checks for more work back at .work_processed_* would be disabled.
Looking at the code, I think Lee is correct that upon returning from notify_resume_user, p6 is turned off and subsequently turns off all the check in work_pending.
- Ken
Quoted 17 lines · Kenneth W
Lee Schermerhorn wrote on Tuesday, November 15, 2005 12:21 PM
Quoted 8 lines · David Mosberger-Tang
Quoted 6 lines
Quoted 2 lines
Then I notice that after we return from
do_notify_resume_user()we don't recheckTIF_NEED_RESCHED, etc. in thethread_infoflags.Why do you say this? It's not true. After processing pending work, we go back to .work_processed_{syscall,kernel), which will again invoke .work_pending, if necessary.
Quoted 7 lines
OK. I guess I'm misinterpreting the comments on the line labeled '.ret10:'.
Say's it's setting p6 to zero, so no recheck. I'm not fluent [writing nor reading] ia64 assembler, but it appeared to me that with p6 set to zero on return fromnotify_resume(), the checks for more work back at .work_processed_* would be disabled.
Quoted 3 lines
Looking at the code, I think Lee is correct that upon returning from
notify_resume_user, p6 is turned off and subsequently turns off all the check inwork_pending.
Yes, I goofed.
The ia64 code shouldn't be different from other platforms in this respect though. If x86 nowadays does the re-check unconditionally, we need to look into what happened. Perhaps x86 got updated and we missed that update for ia64?
--david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536