WRITEV with IOSQE_ASYNC broken?
Hi,
I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.
Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.
Sorry if I've made a mistake somehow, and thanks for all the great work on this game-changing feature!
Nick
Jens Axboe
maintainerQuoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.
Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Quoted 2 lines
Sorry if I've made a mistake somehow, and thanks for all the great work on this game-changing feature!
Thanks! Let's get to the bottom of this.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Quoted 3 lines
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 22 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Quoted 3 lines
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Quoted 5 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
fs/io_uring.c +4 -3
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw, bool force_nonblock) {struct io_async_rw *iorw = &req->io->rw; +struct iovec *iov;ssize_t ret; - iorw->iter.iov = iorw->fast_iov; - ret =__io_import_iovec (rw, req, (struct iovec **) &iorw->iter.iov, - &iorw->iter, !force_nonblock); + iorw->iter.iov = iov = iorw->fast_iov; + ret =__io_import_iovec (rw, req, &iov, &iorw->iter, !force_nonblock);if (unlikely (ret <0 ))return ret; + iorw->iter.iov = iov;io_req_map_rw (req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);return 0 ; }
Signature · 1 line
Jens Axboe
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Quoted 8 lines
Quoted 5 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.
Quoted 25 lines
fs/io_uring.c +4
-3@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw, bool force_nonblock) {struct io_async_rw *iorw = &req->io->rw; +struct iovec *iov;ssize_t ret; - iorw->iter.iov = iorw->fast_iov; - ret =__io_import_iovec (rw, req, (struct iovec **) &iorw->iter.iov, - &iorw->iter, !force_nonblock); + iorw->iter.iov = iov = iorw->fast_iov; + ret =__io_import_iovec (rw, req, &iov, &iorw->iter, !force_nonblock);if (unlikely (ret <0 ))return ret; + iorw->iter.iov = iov;io_req_map_rw (req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);return 0 ; }
Signature · 1 line
Pavel Begunkov
Quoted 62 lines · Pavel Begunkov
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Quoted 8 lines
Quoted 5 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const in
iter->iovis guarding from such cases, yet another proof that const casts are evil.Quoted 22 lines
fs/io_uring.c +3
-3@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw, bool force_nonblock) { struct io_async_rw *iorw = &req->io->rw; +struct iovec *iov;ssize_t ret; - iorw->iter.iov = iorw->fast_iov; - ret =__io_import_iovec (rw, req, (struct iovec **) &iorw->iter.iov, - &iorw->iter, !force_nonblock); + iorw->iter.iov = iov = iorw->fast_iov; + ret =__io_import_iovec (rw, req, &iov, &iorw->iter, !force_nonblock);if (unlikely (ret <0 ))return ret;
Quoted 5 lines
Quoted 5 lines
fs/io_uring.c +1
-0+ iorw->iter.iov = iov;io_req_map_rw (req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);return 0;
}
Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.
Nick
Yes … I will :) I am already compiling the kernel as we speak with the patch applied. Will report back later today.
Quoted 58 lines · On 5. Sep 2020, at 10:24, [email protected] wrote:
Quoted 43 lines · Pavel Begunkov
Quoted 18 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 7 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the nettyio_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.
Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when theIOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.Quoted 6 lines
Quoted 4 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const initer->iovis guarding from such cases, yet another proof that const casts are evil.Quoted 13 lines
fs/io_uring.c +1
-3@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw, bool force_nonblock) {struct io_async_rw *iorw = &req->io->rw; +struct iovec *iov;ssize_t ret; - iorw->iter.iov = iorw->fast_iov; - ret =__io_import_iovec (rw, req, (struct iovec **) &iorw->iter.iov, - &iorw->iter, !force_nonblock);Quoted 8 lines
fs/io_uring.c static inline int io_rw_prep_async(struct io_kiocb *req, int rw, · line 2994
Quoted 8 lines
fs/io_uring.c +3
-0+ iorw->iter.iov = iov = iorw->fast_iov; + ret =__io_import_iovec (rw, req, &iov, &iorw->iter, !force_nonblock);if (unlikely (ret <0 ))return ret; + iorw->iter.iov = iov;io_req_map_rw (req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);return 0 ; }Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.
Nick
I can confirm this fixed the problem for us.
Thanks a lot of the quick turnaround (as always!).
Bye
Norman
Quoted 65 lines · Norman Maurer
Yes … I will :) I am already compiling the kernel as we speak with the patch applied. Will report back later today.
Quoted 58 lines · On 5. Sep 2020, at 10:24, [email protected] wrote:
Quoted 43 lines · Pavel Begunkov
Quoted 18 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 7 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the nettyio_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.
Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when theIOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.Quoted 6 lines
Quoted 4 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const initer->iovis guarding from such cases, yet another proof that const casts are evil.Quoted 13 lines
fs/io_uring.c +1
-3@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw, bool force_nonblock) {struct io_async_rw *iorw = &req->io->rw; +struct iovec *iov;ssize_t ret; - iorw->iter.iov = iorw->fast_iov; - ret =__io_import_iovec (rw, req, (struct iovec **) &iorw->iter.iov, - &iorw->iter, !force_nonblock);Quoted 8 lines
fs/io_uring.c static inline int io_rw_prep_async(struct io_kiocb *req, int rw, · line 2994
Quoted 8 lines
fs/io_uring.c +3
-0+ iorw->iter.iov = iov = iorw->fast_iov; + ret =__io_import_iovec (rw, req, &iov, &iorw->iter, !force_nonblock);if (unlikely (ret <0 ))return ret; + iorw->iter.iov = iov;io_req_map_rw (req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);return 0 ; }Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.
Nick
Jens Axboe
maintainerQuoted 3 lines · Norman Maurer
I can confirm this fixed the problem for us.
Thanks a lot of the quick turnaround (as always!).
No problem, thanks for reporting! I've queued up a regression test as well so this won't ever happen again.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 30 lines · Pavel Begunkov
Quoted 29 lines · Jens Axboe
Quoted 27 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Quoted 3 lines
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
Quoted 8 lines
Quoted 2 lines
Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?
Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)>
BTW, const initer->iovis guarding from such cases, yet another proof that const casts are evil.
Definitely, not a great idea to begin with...
Signature · 1 line
Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:
Hi,
I am helping out with the netty
io_uringintegration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the
IOSQE_ASYNCflag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Quoted 3 lines
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.
Quoted 5 lines
Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.
I only started testing on 5.9-rc3 so not sure about earlier versions, but I'll try and report back.
It's a socket with O_NONBLOCK, iovec array length is ~30 and sum of buffer sizes ~1MB.
If it's not easy to recreate then please don't waste time since it could be my mistake - I'll try to make a standalone reproducer in that case.