Skip to contentSkip to conversation

Is it safe to submit and reap IOs in different threads?

DISCUSSION Bhatia, Sumeet 2 participants 2 messages May 6, 2020 – May 7, 2020 io_uring

lore
Share & tools
lore
#0 lore raw

Hello,

My application has a thread per disk (aka producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.

producer_thread executes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.

My reading of fs/io_uring.c suggests it is safe to submit IOs from producer_thread and reap IOs from consumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too.

I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?

Thanks,
Sumeet

#1 lore raw
Quoted 9 lines · Sumeet

Hello,

My application has a thread per disk (aka producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.

producer_thread executes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.

My reading of fs/io_uring.c suggests it is safe to submit IOs from producer_thread and reap IOs from consumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too.

I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?

Shortly, if you don't mix up submissions and completions, it's safe. There is only pitfall I found in liburing -- you shouldn't pass a timeout in wait functions (e.g. io_uring_wait_cqes()), which would do submission.

Also take a look:
https://github.com/axboe/liburing/issues/108#issuecomment-619616721

Signature · 1 line
Pavel Begunkov