Is it safe to submit and reap IOs in different threads?
Hello,
My application has a thread per disk (aka producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.
producer_thread executes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.
My reading of fs/io_uring.c suggests it is safe to submit IOs from producer_thread and reap IOs from consumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too.
I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?
Thanks,
Sumeet
Quoted 9 lines · Sumeet
Hello,
My application has a thread per disk (aka
producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.
producer_threadexecutes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.My reading of
fs/io_uring.csuggests it is safe to submit IOs fromproducer_threadand reap IOs fromconsumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too.I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?
Shortly, if you don't mix up submissions and completions, it's safe. There is only pitfall I found in liburing -- you shouldn't pass a timeout in wait functions (e.g. io_uring_wait_cqes()), which would do submission.
Also take a look:
https://github.com/axboe/liburing/issues/108#issuecomment-619616721
Signature · 1 line
Pavel Begunkov