Skip to contentSkip to conversation

eventpoll: compute timer slack lazily in ep_poll()

PATCH Applied Usama Arif 3 participants 3 messages Jul 7 – Jul 24 VFS

lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH 1/1 eventpoll: compute timer slack lazily in ep_poll()

ep_poll() computes the timer slack via select_estimate_accuracy() up front, before checking whether events are already available. select_estimate_accuracy() reads the clock (ktime_get_ts64()), and the resulting slack is only consumed by the schedule_hrtimeout_range() call on the blocking path.

A busy poller such as an L7 proxy event loop calls epoll_wait() at a very high rate and often finds events already pending, returning via ep_try_send_events() without ever blocking. In that case the up-front slack estimation - including its clock read - is pure overhead. read_tsc() attributable to select_estimate_accuracy() sometimes shows up in perf profiles of such a workload via the epoll_wait() path.

Move the slack estimation to the point where the thread is actually about to sleep. The timeout passed to ep_poll() is already an absolute deadline (ep_timeout_to_timespec()), so deferring the estimate does not change the wakeup time; taken closer to the sleep it is, if anything, marginally more accurate. On the common non-blocking path the clock read is skipped entirely.

Measured on a host running a Meta production workload with the following bpftrace script:

  #!/usr/bin/bpftrace
  fentry:__x64_sys_epoll_wait,
  fentry:__x64_sys_epoll_pwait   { @in[tid] = 1; }
  fexit:__x64_sys_epoll_wait,
  fexit:__x64_sys_epoll_pwait    { delete(@in, tid); }
  fentry:select_estimate_accuracy /@in[tid]/ { @sea++; }
  fentry:schedule_hrtimeout_range /@in[tid]/ { @shr++; }
  interval:s:30 {
      printf("sea=%lld shr=%lld wasted=%lld (%d%%)\n",
             @sea, @shr, @sea - @shr, (@sea - @shr) * 100 / @sea);
      exit();
  }

Over a 30s window:

sea=3,587,704 shr=3,003,920 wasted=583,784 (16%)

So ~16% of ep_poll invocations of select_estimate_accuracy have no consumer.

---
 fs/eventpoll.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/eventpoll.c +4 -2
index 0e65c7431dfc..128d7fd3d0ea 100644
@@ -2248,7 +2248,6 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 	lockdep_assert_irqs_enabled();
 
 	if (timeout && (timeout->tv_sec | timeout->tv_nsec)) {
-		slack = select_estimate_accuracy(timeout);
 		to = &expires;
 		*to = timespec64_to_ktime(*timeout);
 	} else if (timeout) {
@@ -2327,10 +2326,13 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 
 		spin_unlock_irq(&ep->lock);
 
-		if (!eavail)
+		if (!eavail) {
+			if (to)
+				slack = select_estimate_accuracy(timeout);
 			timed_out = !ep_schedule_timeout(to) ||
 				!schedule_hrtimeout_range(to, slack,
 							  HRTIMER_MODE_ABS);
+		}
 		__set_current_state(TASK_RUNNING);
 
 		/*
Signature · 1 line
2.53.0-Meta
maintainer Applied
#1 lore raw
Quoted 1 line · Usama Arif

eventpoll: compute timer slack lazily in ep_poll()

Applied to the vfs-7.3.misc branch of the vfs/vfs.git tree. Patches in the vfs-7.3.misc branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs-7.3.misc

[1/1] eventpoll: compute timer slack lazily in ep_poll()

      https://git.kernel.org/vfs/vfs/c/d0c242dda99c
Reviewed-by on v1 1/1
#2 lore raw
In reply to Usama Arif
Quoted 36 lines · Usama Arif

ep_poll() computes the timer slack via select_estimate_accuracy() up front, before checking whether events are already available. select_estimate_accuracy() reads the clock (ktime_get_ts64()), and the resulting slack is only consumed by the schedule_hrtimeout_range() call on the blocking path.

A busy poller such as an L7 proxy event loop calls epoll_wait() at a very high rate and often finds events already pending, returning via ep_try_send_events() without ever blocking. In that case the up-front slack estimation - including its clock read - is pure overhead. read_tsc() attributable to select_estimate_accuracy() sometimes shows up in perf profiles of such a workload via the epoll_wait() path.

Move the slack estimation to the point where the thread is actually about to sleep. The timeout passed to ep_poll() is already an absolute deadline (ep_timeout_to_timespec()), so deferring the estimate does not change the wakeup time; taken closer to the sleep it is, if anything, marginally more accurate. On the common non-blocking path the clock read is skipped entirely.

Measured on a host running a Meta production workload with the following bpftrace script:

  #!/usr/bin/bpftrace
  fentry:__x64_sys_epoll_wait,
  fentry:__x64_sys_epoll_pwait   { @in[tid] = 1; }
  fexit:__x64_sys_epoll_wait,
  fexit:__x64_sys_epoll_pwait    { delete(@in, tid); }
  fentry:select_estimate_accuracy /@in[tid]/ { @sea++; }
  fentry:schedule_hrtimeout_range /@in[tid]/ { @shr++; }
  interval:s:30 {
      printf("sea=%lld shr=%lld wasted=%lld (%d%%)\n",
             @sea, @shr, @sea - @shr, (@sea - @shr) * 100 / @sea);
      exit();
  }
Quoted 8 lines

Over a 30s window:

sea=3,587,704 shr=3,003,920 wasted=583,784 (16%)

So ~16% of ep_poll invocations of select_estimate_accuracy have no consumer.