fs/epoll: mark epoll_wait tasks TASK_FREEZABLE

Similar to epoll_wait(), threads sleeping in do_select() and do_poll() sleep in TASK_INTERRUPTIBLE via poll_schedule_timeout(). During system suspend and resume, these threads are woken up by the freezer solely to transition to TASK_FROZEN, and woken again upon thaw only to re-evaluate file descriptors with no pending activity and sleep again.

A previous attempt to make select() and poll() freezable in commit 9745cdb36da8 ("select: use freezable blocking call") was reverted in commit 59612d187912 ("Revert "select: use freezable blocking call"") alongside the epoll revert due to issues caused by the legacy freezer architecture.

Following the freezer overhaul in commit f5d39b020809 ("freezer,sched:
Rewrite core freezer logic"), tasks in TASK_FREEZABLE are safely frozen in-place into TASK_FROZEN without running userspace code or experiencing premature wakeups before resume completes.

Pass TASK_FREEZABLE to poll_schedule_timeout() in do_select() and do_poll() (TASK_INTERRUPTIBLE | TASK_FREEZABLE) so that polling threads remain asleep undisturbed across suspend/resume cycles.

Signed-off-by Sadrul Chowdhury <[email protected]>

---
 fs/select.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

fs/select.c +2 -2

index 8943f20e8300..5cbbe86135bc 100644
@@ -601,7 +601,7 @@ static noinline_for_stack int do_select(int n, fd_set_bits *fds, struct timespec
 			to = &expire;
 		}
 
-		if (!poll_schedule_timeout(&table, TASK_INTERRUPTIBLE,
+		if (!poll_schedule_timeout(&table, TASK_INTERRUPTIBLE | TASK_FREEZABLE,
 					   to, slack))
 			timed_out = 1;
 	}

@@ -961,7 +961,7 @@ static int do_poll(struct poll_list *list, struct poll_wqueues *wait,
 			to = &expire;
 		}
 
-		if (!poll_schedule_timeout(wait, TASK_INTERRUPTIBLE, to, slack))
+		if (!poll_schedule_timeout(wait, TASK_INTERRUPTIBLE | TASK_FREEZABLE, to, slack))
 			timed_out = 1;
 	}
 	return count;

Signature · 1 line

2.55.0.1082.g2b9226bbc0-goog

During system suspend (freeze_processes) and resume (thaw_processes), threads blocked in epoll_wait() sleep in TASK_INTERRUPTIBLE. When try_to_freeze_tasks() executes, it must wake up every sleeping thread with a synthetic signal so the thread can schedule, run try_to_freeze(), enter TASK_FROZEN, and sleep again. Upon resume, the freezer wakes them all up simultaneously, only for each thread to re-enter ep_poll(), observe no ready I/O events, and call schedule() to go back to sleep.

On systems with numerous event-driven daemons (where hundreds of threads idle in event loops), this causes a large wakeup stampede and thousands of redundant context switches per suspend/resume cycle.

An earlier attempt to make epoll_wait() freezable in commit 1c441e921201 ("epoll: use freezable blocking call") was reverted in commit c511851de162 ("Revert "epoll: use freezable blocking call"") due to reports of userspace corruption (Bugzilla #61781). Under the legacy freezer design, freezable tasks could wake up prematurely while hardware and peripheral drivers were still suspended.

With the 2022 core freezer rewrite (commit f5d39b020809 ("freezer,sched:
Rewrite core freezer logic")), the legacy fake-signal mechanism was replaced with the TASK_FROZEN state. Tasks sleeping in TASK_FREEZABLE are safely frozen in-place without waking them up, and the scheduler guarantees they cannot be woken until thaw_processes() explicitly clears the frozen state.

Mark sleeping tasks in ep_poll() as TASK_FREEZABLE. Benchmarks on an ARM64 test device demonstrate that combining this with freezable select/poll reduces thaw latency by 63% (from 20.64 ms down to 7.55 ms mean at fixed CPU frequency) and reduces context switches during freeze/thaw by 87%.

Signed-off-by Sadrul Chowdhury <[email protected]>

---
 fs/eventpoll.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

fs/eventpoll.c +2 -1

index 6cbef8b4c2e0..35992f6457bf 100644
@@ -39,6 +39,7 @@
 #include <linux/rculist.h>
 #include <linux/capability.h>
 #include <net/busy_poll.h>
+#include <linux/freezer.h>
 
 #include <trace/hooks/fs.h>
 

@@ -1987,7 +1988,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 		 * the same lock on wakeup ep_poll_callback() side, so it
 		 * is safe to avoid an explicit barrier.
 		 */
-		__set_current_state(TASK_INTERRUPTIBLE);
+		__set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE);
 
 		/*
 		 * Do the final check under the lock. ep_start/done_scan()

Signature · 1 line

2.55.0.1082.g2b9226bbc0-goog

Quoted 32 lines · Sadrul Chowdhury

During system suspend (freeze_processes) and resume (thaw_processes), threads blocked in epoll_wait() sleep in TASK_INTERRUPTIBLE. When try_to_freeze_tasks() executes, it must wake up every sleeping thread with a synthetic signal so the thread can schedule, run try_to_freeze(), enter TASK_FROZEN, and sleep again. Upon resume, the freezer wakes them all up simultaneously, only for each thread to re-enter ep_poll(), observe no ready I/O events, and call schedule() to go back to sleep.

On systems with numerous event-driven daemons (where hundreds of threads idle in event loops), this causes a large wakeup stampede and thousands of redundant context switches per suspend/resume cycle.

An earlier attempt to make epoll_wait() freezable in commit 1c441e921201 ("epoll: use freezable blocking call") was reverted in commit c511851de162 ("Revert "epoll: use freezable blocking call"") due to reports of userspace corruption (Bugzilla #61781). Under the legacy freezer design, freezable tasks could wake up prematurely while hardware and peripheral drivers were still suspended.

With the 2022 core freezer rewrite (commit f5d39b020809 ("freezer,sched:
Rewrite core freezer logic")), the legacy fake-signal mechanism was replaced with the TASK_FROZEN state. Tasks sleeping in TASK_FREEZABLE are safely frozen in-place without waking them up, and the scheduler guarantees they cannot be woken until thaw_processes() explicitly clears the frozen state.

Mark sleeping tasks in ep_poll() as TASK_FREEZABLE. Benchmarks on an ARM64 test device demonstrate that combining this with freezable select/poll reduces thaw latency by 63% (from 20.64 ms down to 7.55 ms mean at fixed CPU frequency) and reduces context switches during freeze/thaw by 87%.

Quoted 1 line

Signed-off-by Sadrul Chowdhury <[email protected]>

With similar caveat as on patch 2/2 feel free to add:

Reviewed-by Jan Kara <[email protected]>

Honza

Quoted 26 lines

---
 fs/eventpoll.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

fs/eventpoll.c +2 -2

index 6cbef8b4c2e0..35992f6457bf 100644
@@ -39,6 +39,7 @@
 #include <linux/rculist.h>
 #include <linux/capability.h>
 #include <net/busy_poll.h>
+#include <linux/freezer.h>
 
 #include <trace/hooks/fs.h>
 

@@ -1987,7 +1988,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
               * the same lock on wakeup ep_poll_callback() side, so it
               * is safe to avoid an explicit barrier.
               */
-             __set_current_state(TASK_INTERRUPTIBLE);
+             __set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE);
 
              /*
               * Do the final check under the lock. ep_start/done_scan()
--

Quoted 2 lines

2.55.0.1082.g2b9226bbc0-goog

--
Jan Kara <[email protected]>
SUSE Labs, CR