Fix epoll issue in 5.4 kernels
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest2 patches 4 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | epoll: call final ep_events_available() check under the lockfs/ | +27 | 2 replies | |
| 2/2 | epoll: check for events when removing a timed out thread from the wait queuefs/ | +16 | 1 reply |
Reviews trailers per patch
All 2 patches have a Reviewed-by or Acked-by.
| Patch | Jason Baron | Guantao Liu | Eric Dumazet | Willem de Bruijn | Khazhismel Kumykov | Davidlohr Bueso | Thadeu Lima de Souza Cascardo |
|---|---|---|---|---|---|---|---|
| 1/2 | R-b | A-b | |||||
| 2/2 | T-b | R-b | R-b | R-b | R-b | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 7 reviewers.
Cover letter [PATCH 5.4 0/2] Fix epoll issue in 5.4 kernels
Hi Greg
After upgrading to 5.4.211 we were started seeing some nodes getting stuck in our Kubernetes cluster. All nodes are running this kernel version. After taking a closer look it seems that runc was command getting stuck. Looking at the stack it appears the thread is stuck in epoll wait for sometime.
[<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1 [<0>] ep_poll+0x48d/0x4e0 [<0>] do_epoll_wait+0xab/0xc0 [<0>] __x64_sys_epoll_pwait+0x4d/0xa0 [<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1 [<0>] futex_wait_queue_me+0xb6/0x110 [<0>] futex_wait+0xe2/0x260 [<0>] do_futex+0x372/0x4f0 [<0>] __x64_sys_futex+0x134/0x180 [<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1
I noticed there are other discussions going on as well
regarding this.
https://lore.kernel.org/all/[email protected]/
Reverting the below patch does fix the issue:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.4.y&id=cf2db24ec4b8e9d399005ececd6f6336916ab6fc
We don't see this issue in latest upstream kernel or even latest 5.10 stable tree. Looking at the patches that went in for 5.10 stable there's one that stands out that seems to be missing in 5.4. 289caf5d8f6c61c6d2b7fd752a7f483cd153f182 (epoll: check for events when removing a timed out thread from the wait queue)
Backporting this patch to 5.4 we don't see the hangups anymore. Looks like this patch fixes time out scenarios which might cause missed wake ups. The other patch in the patch series also fixes a race and is needed for the second patch to apply.
Roman Penyaev (1):
epoll: call final ep_events_available() check under the lock
Soheil Hassas Yeganeh (1):
epoll: check for events when removing a timed out thread from the wait
queue
fs/eventpoll.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-)
Signature · 1 line
2.37.1