eventpoll: Set epoll timeout if it's in the future
PATCH v2 1/1 eventpoll: Set epoll timeout if it's in the future
Avoid an edge case where epoll_wait arms a timer and calls schedule() even if the timer will expire immediately.
For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by calling schedule() (via schedule_hrtimeout_range).
This can be measured using a simple bpftrace script:
tracepoint:sched:sched_switch
/ args->prev_pid == $1 /
{
print(kstack());
print(ustack());
}
Before this patch is applied:
Testing an epoll_wait app with busy poll usecs set to 1000, and epoll_wait timeout set to 1ms using the script above shows:
__traceiter_sched_switch+69
__schedule+1495
schedule+32
schedule_hrtimeout_range+159
do_epoll_wait+1424
__x64_sys_epoll_wait+97
do_syscall_64+95
entry_SYSCALL_64_after_hwframe+118
epoll_wait+82Which is unexpected; the busy poll usecs should have consumed the entire timeout and there should be no reason to arm a timer.
After this patch is applied: the same test scenario does not generate a call to schedule() in the above edge case. If the busy poll usecs are reduced (for example usecs: 100, epoll_wait timeout 1ms) the timer is armed as expected.
---
v2:
- No longer an RFC and rebased on vfs/vfs.fixes
- Added Jan's Reviewed-by
- Added Fixes tag
- No functional changes from the RFC
rfcv1: https://lore.kernel.org/linux-fsdevel/[email protected]/
fs/eventpoll.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
fs/eventpoll.c +9 -1
@@ -1996,6 +1996,14 @@ static int ep_try_send_events(struct eventpoll *ep,return res; } +static int ep_schedule_timeout (ktime_t *to) +{ +if (to) +return ktime_after (*to,ktime_get ()); +else +return 1 ; +} +/** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer.
@@ -2103,7 +2111,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail) +if (!eavail &&ep_schedule_timeout (to)) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );
base-commit: a681b7c17dd21d5aa0da391ceb27a2007ba970a4
Signature · 1 line
2.43.0
Quoted 10 lines · Joe Damato
Avoid an edge case where
epoll_waitarms a timer and callsschedule()even if the timer will expire immediately.For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call
schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by callingschedule()(viaschedule_hrtimeout_range).[...]
Applied to the vfs.fixes branch of the vfs/vfs.git tree. Patches in the vfs.fixes branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs.fixes
[1/1] eventpoll: Set epoll timeout if it's in the future
https://git.kernel.org/vfs/vfs/c/0a65bc27bd64
Christian Brauner
maintainerQuoted 77 lines · Joe Damato
Avoid an edge case where
epoll_waitarms a timer and callsschedule()even if the timer will expire immediately.For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call
schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by callingschedule()(viaschedule_hrtimeout_range).This can be measured using a simple bpftrace script:
tracepoint:sched:sched_switch
/args->prev_pid== $1 /
{
print(kstack());
print(ustack());
}Before this patch is applied:
Testing an
epoll_waitapp with busy poll usecs set to 1000, andepoll_waittimeout set to 1ms using the script above shows:__traceiter_sched_switch+69 __schedule+1495 schedule+32 schedule_hrtimeout_range+159 do_epoll_wait+1424 __x64_sys_epoll_wait+97 do_syscall_64+95 entry_SYSCALL_64_after_hwframe+118 epoll_wait+82Which is unexpected; the busy poll usecs should have consumed the entire timeout and there should be no reason to arm a timer.
After this patch is applied: the same test scenario does not generate a call to
schedule()in the above edge case. If the busy poll usecs are reduced (for example usecs: 100,epoll_waittimeout 1ms) the timer is armed as expected.Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Joe Damato <[email protected]>Reviewed-by Jan Kara <[email protected]>---
v2:
- No longer an RFC and rebased on vfs/vfs.fixes
- Added Jan's Reviewed-by
- Added Fixes tag
- No functional changes from the RFCrfcv1: https://lore.kernel.org/linux-fsdevel/[email protected]/
fs/eventpoll.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/eventpoll.c +8
-0@@ -1996,6 +1996,14 @@ static int ep_try_send_events(struct eventpoll *ep,return res; } +static int ep_schedule_timeout (ktime_t *to) +{ +if (to) +return ktime_after (*to,ktime_get ()); +else +return 1 ; +} +/** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer.
fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2116
Quoted 7 lines
fs/eventpoll.c +1
-1@@ -2103,7 +2111,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail) +if (!eavail &&ep_schedule_timeout (to)) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );
Isn't this buggy? If @to is non-NULL and ep_schedule_timeout() returns false you want to set timed_out to 1 to break the wait. Otherwise you hang, no?
Jan Kara
reviewerQuoted 81 lines · Christian Brauner
Quoted 80 lines · Joe Damato
Avoid an edge case where
epoll_waitarms a timer and callsschedule()even if the timer will expire immediately.For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call
schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by callingschedule()(viaschedule_hrtimeout_range).This can be measured using a simple bpftrace script:
tracepoint:sched:sched_switch
/args->prev_pid== $1 /
{
print(kstack());
print(ustack());
}Before this patch is applied:
Testing an
epoll_waitapp with busy poll usecs set to 1000, andepoll_waittimeout set to 1ms using the script above shows:__traceiter_sched_switch+69 __schedule+1495 schedule+32 schedule_hrtimeout_range+159 do_epoll_wait+1424 __x64_sys_epoll_wait+97 do_syscall_64+95 entry_SYSCALL_64_after_hwframe+118 epoll_wait+82Which is unexpected; the busy poll usecs should have consumed the entire timeout and there should be no reason to arm a timer.
After this patch is applied: the same test scenario does not generate a call to
schedule()in the above edge case. If the busy poll usecs are reduced (for example usecs: 100,epoll_waittimeout 1ms) the timer is armed as expected.Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Joe Damato <[email protected]>Reviewed-by Jan Kara <[email protected]>---
v2:
- No longer an RFC and rebased on vfs/vfs.fixes
- Added Jan's Reviewed-by
- Added Fixes tag
- No functional changes from the RFCrfcv1: https://lore.kernel.org/linux-fsdevel/[email protected]/
fs/eventpoll.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/eventpoll.c +8
-0@@ -1996,6 +1996,14 @@ static int ep_try_send_events(struct eventpoll *ep,return res; } +static int ep_schedule_timeout (ktime_t *to) +{ +if (to) +return ktime_after (*to,ktime_get ()); +else +return 1 ; +} +/** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -2103,7 +2111,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock);
Quoted 8 lines
fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2116
Quoted 4 lines
fs/eventpoll.c +1
-1-if (!eavail) +if (!eavail &&ep_schedule_timeout (to)) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );Isn't this buggy? If @to is non-NULL and
ep_schedule_timeout()returns false you want to settimed_outto 1 to break the wait. Otherwise you hang, no?
Yep, looks like that. Good spotting!
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 91 lines · Jan Kara
Quoted 81 lines · Christian Brauner
Quoted 80 lines · Joe Damato
Avoid an edge case where
epoll_waitarms a timer and callsschedule()even if the timer will expire immediately.For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call
schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by callingschedule()(viaschedule_hrtimeout_range).This can be measured using a simple bpftrace script:
tracepoint:sched:sched_switch
/args->prev_pid== $1 /
{
print(kstack());
print(ustack());
}Before this patch is applied:
Testing an
epoll_waitapp with busy poll usecs set to 1000, andepoll_waittimeout set to 1ms using the script above shows:__traceiter_sched_switch+69 __schedule+1495 schedule+32 schedule_hrtimeout_range+159 do_epoll_wait+1424 __x64_sys_epoll_wait+97 do_syscall_64+95 entry_SYSCALL_64_after_hwframe+118 epoll_wait+82Which is unexpected; the busy poll usecs should have consumed the entire timeout and there should be no reason to arm a timer.
After this patch is applied: the same test scenario does not generate a call to
schedule()in the above edge case. If the busy poll usecs are reduced (for example usecs: 100,epoll_waittimeout 1ms) the timer is armed as expected.Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Joe Damato <[email protected]>Reviewed-by Jan Kara <[email protected]>---
v2:
- No longer an RFC and rebased on vfs/vfs.fixes
- Added Jan's Reviewed-by
- Added Fixes tag
- No functional changes from the RFCrfcv1: https://lore.kernel.org/linux-fsdevel/[email protected]/
fs/eventpoll.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/eventpoll.c +8
-0@@ -1996,6 +1996,14 @@ static int ep_try_send_events(struct eventpoll *ep,return res; } +static int ep_schedule_timeout (ktime_t *to) +{ +if (to) +return ktime_after (*to,ktime_get ()); +else +return 1 ; +} +/** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -2103,7 +2111,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock);Quoted 8 lines
fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2116
Quoted 4 lines
fs/eventpoll.c +1
-1-if (!eavail) +if (!eavail &&ep_schedule_timeout (to)) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );Isn't this buggy? If @to is non-NULL and
ep_schedule_timeout()returns false you want to settimed_outto 1 to break the wait. Otherwise you hang, no?
Quoted 2 lines
Yep, looks like that. Good spotting!
I second that. Also, isn't ep_schedule_timeout buggy too? It compares a timeout value with the current time, that has to be reworked as well.
I see this patch already queued for stable/linux-6.14.y. What's the plan with it?
Thanks,
ta
Quoted 2 lines · Tudor Ambarus
isn't
ep_schedule_timeoutbuggy too? It compares a
timeout value with the current time, that has to be reworked as well.
Ah, I see the timeout is relative to ktime_get_ts64() in SYSCALL_DEFINE6(epoll_pwait2, ...), please ignore.
Quoted 91 lines · Jan Kara
Quoted 81 lines · Christian Brauner
Quoted 80 lines · Joe Damato
Avoid an edge case where
epoll_waitarms a timer and callsschedule()even if the timer will expire immediately.For example: if the user has specified an epoll busy poll usecs which is equal or larger than the epoll_wait/epoll_pwait2 timeout, it is unnecessary to call
schedule_hrtimeout_range; the busy poll usecs have consumed the entire timeout duration so it is unnecessary to induce scheduling latency by callingschedule()(viaschedule_hrtimeout_range).This can be measured using a simple bpftrace script:
tracepoint:sched:sched_switch
/args->prev_pid== $1 /
{
print(kstack());
print(ustack());
}Before this patch is applied:
Testing an
epoll_waitapp with busy poll usecs set to 1000, andepoll_waittimeout set to 1ms using the script above shows:__traceiter_sched_switch+69 __schedule+1495 schedule+32 schedule_hrtimeout_range+159 do_epoll_wait+1424 __x64_sys_epoll_wait+97 do_syscall_64+95 entry_SYSCALL_64_after_hwframe+118 epoll_wait+82Which is unexpected; the busy poll usecs should have consumed the entire timeout and there should be no reason to arm a timer.
After this patch is applied: the same test scenario does not generate a call to
schedule()in the above edge case. If the busy poll usecs are reduced (for example usecs: 100,epoll_waittimeout 1ms) the timer is armed as expected.Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Joe Damato <[email protected]>Reviewed-by Jan Kara <[email protected]>---
v2:
- No longer an RFC and rebased on vfs/vfs.fixes
- Added Jan's Reviewed-by
- Added Fixes tag
- No functional changes from the RFCrfcv1: https://lore.kernel.org/linux-fsdevel/[email protected]/
fs/eventpoll.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/eventpoll.c +8
-0@@ -1996,6 +1996,14 @@ static int ep_try_send_events(struct eventpoll *ep,return res; } +static int ep_schedule_timeout (ktime_t *to) +{ +if (to) +return ktime_after (*to,ktime_get ()); +else +return 1 ; +} +/** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -2103,7 +2111,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock);Quoted 8 lines
fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2116
Quoted 4 lines
fs/eventpoll.c +1
-1-if (!eavail) +if (!eavail &&ep_schedule_timeout (to)) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );Isn't this buggy? If @to is non-NULL and
ep_schedule_timeout()returns false you want to settimed_outto 1 to break the wait. Otherwise you hang, no?
Quoted 1 line
Yep, looks like that. Good spotting!
Thank you for spotting that and sorry for the trouble.
Christian / Jan what would be the correct way for me to deal with this? Would it be to post a v3 (re-submitting the patch in its entirety) or to post a new patch that fixes the original and lists the commit sha from vfs.fixes with a Fixes tag ?
I was going to propose the following, which if I understand correctly, should fix the issue Christian identified:
fs/eventpoll.c +3 -1
@@ -2111,7 +2111,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail &&ep_schedule_timeout (to)) +if (!ep_schedule_timeout (to)) + timed_out =1 ; +else if (!eavail) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );
Carlos Llamas
Tested-byQuoted 1 line · Joe Damato
Thank you for spotting that and sorry for the trouble.
This was also flagged by our Android's epoll_pwait2 tests here:
https://android.googlesource.com/platform/bionic/+/refs/heads/main/tests/sys_epoll_test.cpp
They would all timeout, so the hang reported by Christian fits.
Quoted 4 lines
Christian / Jan what would be the correct way for me to deal with this? Would it be to post a v3 (re-submitting the patch in its entirety) or to post a new patch that fixes the original and lists the commit sha from vfs.fixes with a Fixes tag ?
The original commit has landed in mainline already, so it needs to be new patch at this point. If if helps, here is the tag:
Quoted 5 lines
fs/eventpoll.c +0
-0
fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2119
Quoted 10 lines
fs/eventpoll.c +3
-1@@ -2111,7 +2111,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail &&ep_schedule_timeout (to)) +if (!ep_schedule_timeout (to)) + timed_out =1 ; +else if (!eavail) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );
I've ran your change through our internal CI and I confirm it fixes the hangs seen on our end. If you send the fix feel free to add:
Cheers,
Carlos Llamas
Quoted 36 lines · Carlos Llamas
Quoted 1 line · Joe Damato
Thank you for spotting that and sorry for the trouble.
This was also flagged by our Android's
epoll_pwait2tests here:
https://android.googlesource.com/platform/bionic/+/refs/heads/main/tests/sys_epoll_test.cpp
They would all timeout, so the hang reported by Christian fits.Quoted 4 lines
Christian / Jan what would be the correct way for me to deal with this? Would it be to post a v3 (re-submitting the patch in its entirety) or to post a new patch that fixes the original and lists the commit sha from vfs.fixes with a Fixes tag ?
The original commit has landed in mainline already, so it needs to be new patch at this point. If if helps, here is the tag:
Fixes 0a65bc27bd64 ("eventpoll: Set epoll timeout if it's in the future")Quoted 5 lines
fs/eventpoll.c +0
-0fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2119
Quoted 10 lines
fs/eventpoll.c +3
-1@@ -2111,7 +2111,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail &&ep_schedule_timeout (to)) +if (!ep_schedule_timeout (to)) + timed_out =1 ; +else if (!eavail) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );I've ran your change through our internal CI and I confirm it fixes the hangs seen on our end. If you send the fix feel free to add:
Quoted 1 line
Tested-by Carlos Llamas <[email protected]>
Thanks, will do.
I was waiting to hear back from Christian / Jan if they are OK with the proposed fix before submitting something, but glad to hear it fixes the issue for you. Sorry for the trouble.
Jan Kara
reviewerQuoted 37 lines · Joe Damato
Quoted 36 lines · Carlos Llamas
Quoted 1 line · Joe Damato
Thank you for spotting that and sorry for the trouble.
This was also flagged by our Android's
epoll_pwait2tests here:
https://android.googlesource.com/platform/bionic/+/refs/heads/main/tests/sys_epoll_test.cpp
They would all timeout, so the hang reported by Christian fits.Quoted 4 lines
Christian / Jan what would be the correct way for me to deal with this? Would it be to post a v3 (re-submitting the patch in its entirety) or to post a new patch that fixes the original and lists the commit sha from vfs.fixes with a Fixes tag ?
The original commit has landed in mainline already, so it needs to be new patch at this point. If if helps, here is the tag:
Fixes 0a65bc27bd64 ("eventpoll: Set epoll timeout if it's in the future")Quoted 5 lines
fs/eventpoll.c +0
-0fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2119
Quoted 10 lines
fs/eventpoll.c +3
-1@@ -2111,7 +2111,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail &&ep_schedule_timeout (to)) +if (!ep_schedule_timeout (to)) + timed_out =1 ; +else if (!eavail) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );I've ran your change through our internal CI and I confirm it fixes the hangs seen on our end. If you send the fix feel free to add:
Quoted 7 lines
Quoted 1 line
Tested-by Carlos Llamas <[email protected]>Thanks, will do.
I was waiting to hear back from Christian / Jan if they are OK with the proposed fix before submitting something, but glad to hear it fixes the issue for you. Sorry for the trouble.
Yep, a new patch submission with proper Fixes tag is needed at this point.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Christian Brauner
maintainerQuoted 40 lines · Jan Kara
Quoted 39 lines · Joe Damato
Quoted 37 lines · Carlos Llamas
Quoted 1 line · Joe Damato
Thank you for spotting that and sorry for the trouble.
This was also flagged by our Android's
epoll_pwait2tests here:
https://android.googlesource.com/platform/bionic/+/refs/heads/main/tests/sys_epoll_test.cpp
They would all timeout, so the hang reported by Christian fits.Quoted 4 lines
Christian / Jan what would be the correct way for me to deal with this? Would it be to post a v3 (re-submitting the patch in its entirety) or to post a new patch that fixes the original and lists the commit sha from vfs.fixes with a Fixes tag ?
The original commit has landed in mainline already, so it needs to be new patch at this point. If if helps, here is the tag:
Fixes 0a65bc27bd64 ("eventpoll: Set epoll timeout if it's in the future")Quoted 5 lines
fs/eventpoll.c +0
-0fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 2119
Quoted 10 lines
fs/eventpoll.c +3
-1@@ -2111,7 +2111,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,write_unlock_irq (&ep->lock); -if (!eavail &&ep_schedule_timeout (to)) +if (!ep_schedule_timeout (to)) + timed_out =1 ; +else if (!eavail) timed_out = !schedule_hrtimeout_range (to, slack,HRTIMER_MODE_ABS );__set_current_state (TASK_RUNNING );I've ran your change through our internal CI and I confirm it fixes the hangs seen on our end. If you send the fix feel free to add:
Tested-by Carlos Llamas <[email protected]>
Quoted 7 lines
Quoted 5 lines
Thanks, will do.
I was waiting to hear back from Christian / Jan if they are OK with the proposed fix before submitting something, but glad to hear it fixes the issue for you. Sorry for the trouble.
Yep, a new patch submission with proper Fixes tag is needed at this point.
Yes, please send a new fixes patch that I can pick up!
Thanks!
Christian