Skip to contentSkip to conversation

hrtimer: make epoll_wait() use the hrtimer range feature

PATCH In review Shawn Bohrer 5 participants 10 messages Aug 8, 2010 – Nov 27, 2010 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hrtimer: make epoll_wait() use the hrtimer range feature

This make epoll use hrtimers for the timeout value which prevents epoll_wait() from timing out up to a millisecond early.

This mirrors the behavior of select() and poll().

---
 fs/eventpoll.c       |   35 +++++++++++++++++++----------------
 fs/select.c          |    2 +-
 include/linux/poll.h |    2 ++
 3 files changed, 22 insertions(+), 17 deletions(-)
fs/eventpoll.c +19 -16
index 3817149..728f56c 100644
@@ -77,9 +77,6 @@
 /* Maximum number of nesting allowed inside epoll sets */
 #define EP_MAX_NESTS 4
 
-/* Maximum msec timeout value storeable in a long int */
-#define EP_MAX_MSTIMEO min(1000ULL * MAX_SCHEDULE_TIMEOUT / HZ, (LONG_MAX - 999ULL) / HZ)
-
 #define EP_MAX_EVENTS (INT_MAX / sizeof(struct epoll_event))
 
 #define EP_UNACTIVE_PTR ((void *) -1L)
@@ -1116,18 +1113,22 @@ static int ep_send_events(struct eventpoll *ep,
 static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 		   int maxevents, long timeout)
 {
-	int res, eavail;
+	int res, eavail, timed_out = 0;
 	unsigned long flags;
-	long jtimeout;
+	long slack;
 	wait_queue_t wait;
-
-	/*
-	 * Calculate the timeout by checking for the "infinite" value (-1)
-	 * and the overflow condition. The passed timeout is in milliseconds,
-	 * that why (t * HZ) / 1000.
-	 */
-	jtimeout = (timeout < 0 || timeout >= EP_MAX_MSTIMEO) ?
-		MAX_SCHEDULE_TIMEOUT : (timeout * HZ + 999) / 1000;
+	struct timespec end_time;
+	ktime_t expires, *to = NULL;
+
+	if (timeout > 0) {
+		ktime_get_ts(&end_time);
+		timespec_add_ns(&end_time, (u64)timeout * NSEC_PER_MSEC);
+		slack = estimate_accuracy(&end_time);
+		to = &expires;
+		*to = timespec_to_ktime(end_time);
+	} else if (timeout == 0) {
+		timed_out = 1;
+	}
 
 retry:
 	spin_lock_irqsave(&ep->lock, flags);
@@ -1149,7 +1150,7 @@ retry:
 			 * to TASK_INTERRUPTIBLE before doing the checks.
 			 */
 			set_current_state(TASK_INTERRUPTIBLE);
-			if (!list_empty(&ep->rdllist) || !jtimeout)
+			if (!list_empty(&ep->rdllist) || timed_out)
 				break;
 			if (signal_pending(current)) {
 				res = -EINTR;
@@ -1157,7 +1158,9 @@ retry:
 			}
 
 			spin_unlock_irqrestore(&ep->lock, flags);
-			jtimeout = schedule_timeout(jtimeout);
+			if (!schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS))
+				timed_out = 1;
+
 			spin_lock_irqsave(&ep->lock, flags);
 		}
 		__remove_wait_queue(&ep->wq, &wait);
@@ -1175,7 +1178,7 @@ retry:
 	 * more luck.
 	 */
 	if (!res && eavail &&
-	    !(res = ep_send_events(ep, events, maxevents)) && jtimeout)
+	    !(res = ep_send_events(ep, events, maxevents)) && !timed_out)
 		goto retry;
 
 	return res;
fs/select.c +1 -1
index 500a669..003cb77 100644
@@ -67,7 +67,7 @@ static long __estimate_accuracy(struct timespec *tv)
 	return slack;
 }
 
-static long estimate_accuracy(struct timespec *tv)
+long estimate_accuracy(struct timespec *tv)
 {
 	unsigned long ret;
 	struct timespec now;
include/linux/poll.h +2 -0
index 600cc1f..52be81f 100644
@@ -73,6 +73,8 @@ extern void poll_initwait(struct poll_wqueues *pwq);
 extern void poll_freewait(struct poll_wqueues *pwq);
 extern int poll_schedule_timeout(struct poll_wqueues *pwq, int state,
 				 ktime_t *expires, unsigned long slack);
+extern long estimate_accuracy(struct timespec *tv);
+
 
 static inline int poll_schedule(struct poll_wqueues *pwq, int state)
 {
Signature · 1 line
1.7.2.1
#1 lore raw

I'd say this is an epoll patch, not an hrtimer patch. So I renamed it to "epoll: make epoll_wait() use the hrtimer range feature".

Davide looks after epoll, so let's cc him.

fs/eventpoll.c

Quoted 9 lines · On Sun, 8 Aug 2010 17:45:32 -0500 Shawn Bohrer <[email protected]> wrote:

This make epoll use hrtimers for the timeout value which prevents epoll_wait() from timing out up to a millisecond early.

This mirrors the behavior of select() and poll().

...

fs/eventpoll.c +0 -0

Davide stuff ;)

fs/select.c static long __estimate_accuracy(struct timespec *tv) · line 70

Quoted 8 lines
fs/select.c +1 -1
@@ -67,7 +67,7 @@ static long __estimate_accuracy(struct timespec *tv)
 	return slack;
 }
 
-static long estimate_accuracy(struct timespec *tv)
+long estimate_accuracy(struct timespec *tv)

"estimate_accuracy" is a rotten name for a global symbol. I queued a preparatory patch which renames this to "select_estimate_accuracy".

#2 lore raw
In reply to Shawn Bohrer
Quoted 4 lines · Shawn Bohrer

This make epoll use hrtimers for the timeout value which prevents epoll_wait() from timing out up to a millisecond early.

This mirrors the behavior of select() and poll().

I saw this now, since I got notifications from Andrew's patch machine. Is this really needed?
Is that really worth the extra code in the fast path? Aren't we breaking existing behavior?
I have some doubts.

- Davide

#3 lore raw
Quoted 9 lines · Davide Libenzi
Quoted 4 lines · Shawn Bohrer

This make epoll use hrtimers for the timeout value which prevents epoll_wait() from timing out up to a millisecond early.

This mirrors the behavior of select() and poll().

I saw this now, since I got notifications from Andrew's patch machine. Is this really needed?

Yes, as it is the last user space interface which is jiffies based.

Quoted 1 line

Is that really worth the extra code in the fast path?

It's not that much overhead and we had no complaints when we converted everything else

Quoted 1 line

Aren't we breaking existing behavior?

We had no problems when we moved all the other interfaces over

Quoted 1 line

I have some doubts.

/me not - though I need to look at the patch itself

Thanks,

tglx

#4 lore raw
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Davide Libenzi
Quoted 3 lines · Shawn Bohrer

This make epoll use hrtimers for the timeout value which prevents epoll_wait() from timing out up to a millisecond early.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

This mirrors the behavior of select() and poll().

I saw this now, since I got notifications from Andrew's patch machine. Is this really needed?

Yes, as it is the last user space interface which is jiffies based.

What? You killed jiffies?! :)
Alright then, the patch itself, besides the different bahavior on timeout greater/equal to EP_MAX_MSTIMEO (which is not documented, hence does not exist), looks OK to me.

- Davide

#5 lore raw
In reply to Shawn Bohrer
Quoted 43 lines · Shawn Bohrer
fs/eventpoll.c +14 -10
@@ -1116,18 +1113,22 @@ static int ep_send_events(struct eventpoll *ep,
  static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
                   int maxevents, long timeout)
  {
-       int res, eavail;
+       int res, eavail, timed_out = 0;
        unsigned long flags;
-       long jtimeout;
+       long slack;
        wait_queue_t wait;
-
-       /*
-        * Calculate the timeout by checking for the "infinite" value (-1)
-        * and the overflow condition. The passed timeout is in milliseconds,
-        * that why (t * HZ) / 1000.
-        */
-       jtimeout = (timeout < 0 || timeout >= EP_MAX_MSTIMEO) ?
-               MAX_SCHEDULE_TIMEOUT : (timeout * HZ + 999) / 1000;
+       struct timespec end_time;
+       ktime_t expires, *to = NULL;
+
+       if (timeout > 0) {
+               ktime_get_ts(&end_time);
+               timespec_add_ns(&end_time, (u64)timeout * NSEC_PER_MSEC);
+               slack = estimate_accuracy(&end_time);
+               to = &expires;
+               *to = timespec_to_ktime(end_time);
+       } else if (timeout == 0) {
+               timed_out = 1;
+       }
 

 retry:
       spin_lock_irqsave(&ep->lock, flags);

fs/eventpoll.c +1 -1
@@ -1149,7 +1150,7 @@ retry:
                         * to TASK_INTERRUPTIBLE before doing the checks.
                         */
                        set_current_state(TASK_INTERRUPTIBLE);
-                       if (!list_empty(&ep->rdllist) || !jtimeout)
+                       if (!list_empty(&ep->rdllist) || timed_out)
                                break;
                        if (signal_pending(current)) {
                                res = -EINTR;

fs/eventpoll.c retry: · line 1166

Quoted 10 lines
fs/eventpoll.c +3 -1
@@ -1157,7 +1158,9 @@ retry:
                        }
 
                        spin_unlock_irqrestore(&ep->lock, flags);
-                       jtimeout = schedule_timeout(jtimeout);
+                       if (!schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS))
+                               timed_out = 1;
+
                        spin_lock_irqsave(&ep->lock, flags);
                }
                __remove_wait_queue(&ep->wq, &wait);

this code introduces a warning:
fs/eventpoll.c: In function ‘ep_poll’:
fs/eventpoll.c:1119: warning: ‘slack’ may be used uninitialized in this function

looks to me like you arent properly handling negative timeouts. certainly epoll_wait() passes the timeout value straight from userspace to ep_poll() without any error checking, so if userspace passes a negative timeout value, it looks like "slack" will be used uninitialized.
-mike

#6 lore raw
Quoted 59 lines · Mike Frysinger
Quoted 43 lines · Shawn Bohrer
fs/eventpoll.c +14 -10
@@ -1116,18 +1113,22 @@ static int ep_send_events(struct eventpoll *ep,
  static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
                   int maxevents, long timeout)
  {
-       int res, eavail;
+       int res, eavail, timed_out = 0;
        unsigned long flags;
-       long jtimeout;
+       long slack;
        wait_queue_t wait;
-
-       /*
-        * Calculate the timeout by checking for the "infinite" value (-1)
-        * and the overflow condition. The passed timeout is in milliseconds,
-        * that why (t * HZ) / 1000.
-        */
-       jtimeout = (timeout < 0 || timeout >= EP_MAX_MSTIMEO) ?
-               MAX_SCHEDULE_TIMEOUT : (timeout * HZ + 999) / 1000;
+       struct timespec end_time;
+       ktime_t expires, *to = NULL;
+
+       if (timeout > 0) {
+               ktime_get_ts(&end_time);
+               timespec_add_ns(&end_time, (u64)timeout * NSEC_PER_MSEC);
+               slack = estimate_accuracy(&end_time);
+               to = &expires;
+               *to = timespec_to_ktime(end_time);
+       } else if (timeout == 0) {
+               timed_out = 1;
+       }
 

 retry:
       spin_lock_irqsave(&ep->lock, flags);

fs/eventpoll.c +1 -1
@@ -1149,7 +1150,7 @@ retry:
                         * to TASK_INTERRUPTIBLE before doing the checks.
                         */
                        set_current_state(TASK_INTERRUPTIBLE);
-                       if (!list_empty(&ep->rdllist) || !jtimeout)
+                       if (!list_empty(&ep->rdllist) || timed_out)
                                break;
                        if (signal_pending(current)) {
                                res = -EINTR;

fs/eventpoll.c retry: · line 1166

Quoted 10 lines
fs/eventpoll.c +3 -1
@@ -1157,7 +1158,9 @@ retry:
                        }
 
                        spin_unlock_irqrestore(&ep->lock, flags);
-                       jtimeout = schedule_timeout(jtimeout);
+                       if (!schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS))
+                               timed_out = 1;
+
                        spin_lock_irqsave(&ep->lock, flags);
                }
                __remove_wait_queue(&ep->wq, &wait);

this code introduces a warning:
fs/eventpoll.c: In function ‘ep_poll’:
fs/eventpoll.c:1119: warning: ‘slack’ may be used uninitialized in this function

Quoted 5 lines

looks to me like you arent properly handling negative timeouts. certainly epoll_wait() passes the timeout value straight from userspace to ep_poll() without any error checking, so if userspace passes a negative timeout value, it looks like "slack" will be used uninitialized.

If a negative timeout is passed in then 'to' remains NULL. When 'to is NULL schedule_hrtimeout_range() has an infinite timeout and the 'slack' parameter is never used. So technically everything should be fine here.

Of course it would be safest and best to simply initialize slack to 0. I can send a patch this evening with the fix.

--
Shawn
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 66 lines · Shawn Bohrer
Quoted 59 lines · Mike Frysinger
Quoted 43 lines · Shawn Bohrer
fs/eventpoll.c +14 -10
@@ -1116,18 +1113,22 @@ static int ep_send_events(struct eventpoll *ep,
  static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
                   int maxevents, long timeout)
  {
-       int res, eavail;
+       int res, eavail, timed_out = 0;
        unsigned long flags;
-       long jtimeout;
+       long slack;
        wait_queue_t wait;
-
-       /*
-        * Calculate the timeout by checking for the "infinite" value (-1)
-        * and the overflow condition. The passed timeout is in milliseconds,
-        * that why (t * HZ) / 1000.
-        */
-       jtimeout = (timeout < 0 || timeout >= EP_MAX_MSTIMEO) ?
-               MAX_SCHEDULE_TIMEOUT : (timeout * HZ + 999) / 1000;
+       struct timespec end_time;
+       ktime_t expires, *to = NULL;
+
+       if (timeout > 0) {
+               ktime_get_ts(&end_time);
+               timespec_add_ns(&end_time, (u64)timeout * NSEC_PER_MSEC);
+               slack = estimate_accuracy(&end_time);
+               to = &expires;
+               *to = timespec_to_ktime(end_time);
+       } else if (timeout == 0) {
+               timed_out = 1;
+       }
 

 retry:
       spin_lock_irqsave(&ep->lock, flags);

fs/eventpoll.c +1 -1
@@ -1149,7 +1150,7 @@ retry:
                         * to TASK_INTERRUPTIBLE before doing the checks.
                         */
                        set_current_state(TASK_INTERRUPTIBLE);
-                       if (!list_empty(&ep->rdllist) || !jtimeout)
+                       if (!list_empty(&ep->rdllist) || timed_out)
                                break;
                        if (signal_pending(current)) {
                                res = -EINTR;

fs/eventpoll.c retry: · line 1166

Quoted 10 lines
fs/eventpoll.c +3 -1
@@ -1157,7 +1158,9 @@ retry:
                        }
 
                        spin_unlock_irqrestore(&ep->lock, flags);
-                       jtimeout = schedule_timeout(jtimeout);
+                       if (!schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS))
+                               timed_out = 1;
+
                        spin_lock_irqsave(&ep->lock, flags);
                }
                __remove_wait_queue(&ep->wq, &wait);

this code introduces a warning:
fs/eventpoll.c: In function ‘ep_poll’:
fs/eventpoll.c:1119: warning: ‘slack’ may be used uninitialized in this function

Quoted 5 lines

looks to me like you arent properly handling negative timeouts. certainly epoll_wait() passes the timeout value straight from userspace to ep_poll() without any error checking, so if userspace passes a negative timeout value, it looks like "slack" will be used uninitialized.

Quoted 4 lines

If a negative timeout is passed in then 'to' remains NULL.  When 'to is NULL schedule_hrtimeout_range() has an infinite timeout and the 'slack' parameter is never used.  So technically everything should be fine here.

ok, but that depends on an external function never changing behavior and makes changing the API pretty hard since all callers must be closely analyzed

Quoted 2 lines

Of course it would be safest and best to simply initialize slack to 0. I can send a patch this evening with the fix.

thanks !
-mike
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

PATCH 1/1 epoll: initialize slack for negative timeout values

When a negative timeout value is passed to epoll the 'slack' variable is currently uninitialized:

fs/eventpoll.c: In function ‘ep_poll’:
fs/eventpoll.c:1119: warning: ‘slack’ may be used uninitialized in this function

In this case a NULL pointer is passed to schedule_hrtimeout_range() specifying an infinite timeout. The current implementation of schedule_hrtimeout_range() does not use slack in this case, but we should still initialize slack to 0 in case future implementations use it.

---
 fs/eventpoll.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
fs/eventpoll.c +1 -1
index 8cf0724..c24a032 100644
@@ -1116,7 +1116,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 {
 	int res, eavail, timed_out = 0;
 	unsigned long flags;
-	long slack;
+	long slack = 0;
 	wait_queue_t wait;
 	struct timespec end_time;
 	ktime_t expires, *to = NULL;
Signature · 1 line
1.7.3.2
Acked-by on v1 1/1
#9 lore raw

Re: [PATCH] epoll: initialize slack for negative timeout values

Quoted 10 lines · Shawn Bohrer

When a negative timeout value is passed to epoll the 'slack' variable is currently uninitialized:

fs/eventpoll.c: In function ‘ep_poll’:
fs/eventpoll.c:1119: warning: ‘slack’ may be used uninitialized in this function

In this case a NULL pointer is passed to schedule_hrtimeout_range() specifying an infinite timeout. The current implementation of schedule_hrtimeout_range() does not use slack in this case, but we should still initialize slack to 0 in case future implementations use it.

Thanks.

Quoted 1 line
Quoted 9 lines
---
 fs/eventpoll.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
fs/eventpoll.c +0 -0
index 8cf0724..c24a032 100644

fs/eventpoll.c static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, · line 1122

Quoted 8 lines
fs/eventpoll.c +1 -1
@@ -1116,7 +1116,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 {
 	int res, eavail, timed_out = 0;
 	unsigned long flags;
-	long slack;
+	long slack = 0;
 	wait_queue_t wait;
 	struct timespec end_time;
 	ktime_t expires, *to = NULL;

- Davide