Fix hang in io_uring_get_cqe() with iopoll
v2 cover Fix hang in io_uring_get_cqe() with iopoll
v2: use relaxed load
fix errata
Pavel Begunkov (2):
barriers: add load relaxed
Fix hang in io_uring_get_cqe() with iopoll
src/include/liburing/barrier.h | 4 ++++ src/queue.c | 16 +++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)
Signature · 1 line
2.24.0
PATCH v2 1/2 barriers: add load relaxed
Add io_uring_smp_load_relaxed() for internal use.
--- src/include/liburing/barrier.h | 4 ++++ 1 file changed, 4 insertions(+)
src/include/liburing/barrier.h +4 -0
@@ -47,6 +47,8 @@ do { \ ___p1; \ }) +#define io_uring_smp_load_relaxed (p) IO_URING_READ_ONCE(*(p)) +#else /* defined(__x86_64__) || defined(__i386__) */ /* * Add arch appropriate definitions. Use built-in atomic operations for
@@ -55,6 +57,8 @@ do { \#define io_uring_smp_store_release (p, v) \ __atomic_store_n(p, v, __ATOMIC_RELEASE)#define io_uring_smp_load_acquire (p) __atomic_load_n(p, __ATOMIC_ACQUIRE) +#define io_uring_smp_load_relaxed (p) __atomic_load_n(p, __ATOMIC_RELAXED) +#endif /* defined(__x86_64__) || defined(__i386__) */ #endif /* defined(LIBURING_BARRIER_H) */
Signature · 1 line
2.24.0
PATCH v2 2/2 Fix hang in io_uring_get_cqe() with iopoll
In reply to Pavel BegunkovBecause of need_resched() check, io_uring_enter() -> io_iopoll_check() can return 0 even if @min_complete wasn't satisfied. If that's the case, __io_uring_get_cqe() sets submit=0 and wait_nr=0, disabling setting IORING_ENTER_GETEVENTS as well. So, it goes crazy calling io_uring_enter() in a loop, not actually submitting nor polling.
Set @wait_nr based on actual number of CQEs ready. It doesn't manifest extra CQEs if any, thus implements __io_uring_cq_ready() with relaxed semantics.
--- src/queue.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
src/queue.c +15 -1
@@ -32,6 +32,19 @@ static inline bool sq_ring_needs_enter(struct io_uring *ring,return false; } +static inline unsigned int __io_uring_cq_ready (struct io_uring *ring) +{ +return io_uring_smp_load_relaxed (ring->cq.ktail) - *ring->cq.khead; +} + +static inline unsigned int io_adjust_wait_nr (struct io_uring *ring, +unsigned int to_wait) +{ +unsigned int ready =__io_uring_cq_ready (ring); + +return (to_wait <= ready) ?0 : (to_wait - ready); +} +int __io_uring_get_cqe (struct io_uring *ring,struct io_uring_cqe **cqe_ptr,unsigned submit,unsigned wait_nr,sigset_t *sigmask) {
@@ -60,7 +73,8 @@ int __io_uring_get_cqe(struct io_uring *ring, struct io_uring_cqe **cqe_ptr, err = -errno; }else if (ret == (int )submit) { submit =0 ; - wait_nr =0 ; +if (to_wait) + wait_nr =io_adjust_wait_nr (ring, to_wait); }else { submit -= ret; }
Signature · 1 line
2.24.0
Jens Axboe
maintainerQuoted 10 lines · Pavel Begunkov
v2: use relaxed load
fix errataPavel Begunkov (2):
barriers: add load relaxed
Fix hang inio_uring_get_cqe()with iopollsrc/include/liburing/barrier.h | 4 ++++ src/queue.c | 16 +++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)
After checking again, I think your liburing is quite a bit out-of-date. Can you check if the issue still exists in the current git tree?
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Pavel Begunkov
v2: use relaxed load
fix errataPavel Begunkov (2):
barriers: add load relaxed
Fix hang inio_uring_get_cqe()with iopoll
Quoted 5 lines
Quoted 3 lines
src/include/liburing/barrier.h | 4 ++++ src/queue.c | 16 +++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-)After checking again, I think your liburing is quite a bit out-of-date.
It is. Apparently I checked out something old, my apologies.
Quoted 1 line
Can you check if the issue still exists in the current git tree?
This one is plumbed, though from a glance there is a similar problem with non-iopoll mode and early wake by a timeout, rendering it to idle-loop instead of sleeping. I'll leave it for a bit later.
Signature · 1 line
Pavel Begunkov