Skip to contentSkip to conversation

WRITEV with IOSQE_ASYNC broken?

DISCUSSION [email protected] 4 participants 11 messages Sep 5, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Sorry if I've made a mistake somehow, and thanks for all the great work on this game-changing feature!

Nick

maintainer
#1 lore raw
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 2 lines

Sorry if I've made a mistake somehow, and thanks for all the great work on this game-changing feature!

Thanks! Let's get to the bottom of this.

Signature · 1 line
Jens Axboe
maintainer
#2 lore raw
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Quoted 3 lines

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

Signature · 1 line
Jens Axboe
maintainer
#3 lore raw
Quoted 22 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Quoted 3 lines

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 5 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

fs/io_uring.c +4 -3
index 0d7be2e9d005..000ae2acfd58 100644
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw,
 				   bool force_nonblock)
 {
 	struct io_async_rw *iorw = &req->io->rw;
+	struct iovec *iov;
 	ssize_t ret;
 
-	iorw->iter.iov = iorw->fast_iov;
-	ret = __io_import_iovec(rw, req, (struct iovec **) &iorw->iter.iov,
-				&iorw->iter, !force_nonblock);
+	iorw->iter.iov = iov = iorw->fast_iov;
+	ret = __io_import_iovec(rw, req, &iov, &iorw->iter, !force_nonblock);
 	if (unlikely(ret < 0))
 		return ret;
 
+	iorw->iter.iov = iov;
 	io_req_map_rw(req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);
 	return 0;
 }
Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 8 lines
Quoted 5 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)

BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.

Quoted 25 lines
fs/io_uring.c +4 -3
index 0d7be2e9d005..000ae2acfd58 100644
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw,
 				   bool force_nonblock)
 {
 	struct io_async_rw *iorw = &req->io->rw;
+	struct iovec *iov;
 	ssize_t ret;
 
-	iorw->iter.iov = iorw->fast_iov;
-	ret = __io_import_iovec(rw, req, (struct iovec **) &iorw->iter.iov,
-				&iorw->iter, !force_nonblock);
+	iorw->iter.iov = iov = iorw->fast_iov;
+	ret = __io_import_iovec(rw, req, &iov, &iorw->iter, !force_nonblock);
 	if (unlikely(ret < 0))
 		return ret;
 
+	iorw->iter.iov = iov;
 	io_req_map_rw(req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);
 	return 0;
 }
Signature · 1 line
Pavel Begunkov
#5 lore raw
Quoted 62 lines · Pavel Begunkov
Quoted 23 lines · Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 8 lines
Quoted 5 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)

BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.

Quoted 22 lines
fs/io_uring.c +3 -3
index 0d7be2e9d005..000ae2acfd58 100644
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct 
 io_kiocb *req, int rw,
 				   bool force_nonblock)
 {
 	struct io_async_rw *iorw = &req->io->rw;
+	struct iovec *iov;
 	ssize_t ret;
 
-	iorw->iter.iov = iorw->fast_iov;
-	ret = __io_import_iovec(rw, req, (struct iovec **) &iorw->iter.iov,
-				&iorw->iter, !force_nonblock);
+	iorw->iter.iov = iov = iorw->fast_iov;
+	ret = __io_import_iovec(rw, req, &iov, &iorw->iter, 
 !force_nonblock);
 	if (unlikely(ret < 0))
 		return ret;
 
Quoted 5 lines
Quoted 5 lines
fs/io_uring.c +1 -0
+	iorw->iter.iov = iov;
 	io_req_map_rw(req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);

return 0;
}

Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.

Nick

#6 lore raw

Yes … I will :) I am already compiling the kernel as we speak with the patch applied. Will report back later today.

Quoted 58 lines · On 5. Sep 2020, at 10:24, [email protected] wrote:
Quoted 43 lines · Pavel Begunkov
Quoted 18 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 7 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,
I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.
Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 6 lines
Quoted 4 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.

Quoted 13 lines
fs/io_uring.c +1 -3
index 0d7be2e9d005..000ae2acfd58 100644
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw,
 				   bool force_nonblock)
 {
 	struct io_async_rw *iorw = &req->io->rw;
+	struct iovec *iov;
 	ssize_t ret;
-	iorw->iter.iov = iorw->fast_iov;
-	ret = __io_import_iovec(rw, req, (struct iovec **) &iorw->iter.iov,
-				&iorw->iter, !force_nonblock);
Quoted 8 lines

fs/io_uring.c static inline int io_rw_prep_async(struct io_kiocb *req, int rw, · line 2994

Quoted 8 lines
fs/io_uring.c +3 -0
+	iorw->iter.iov = iov = iorw->fast_iov;
+	ret = __io_import_iovec(rw, req, &iov, &iorw->iter, !force_nonblock);
 	if (unlikely(ret < 0))
 		return ret;
+	iorw->iter.iov = iov;
 	io_req_map_rw(req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);
 	return 0;
 }

Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.

Nick

#7 lore raw

I can confirm this fixed the problem for us.

Thanks a lot of the quick turnaround (as always!).

Bye
Norman

Quoted 65 lines · Norman Maurer

Yes … I will :) I am already compiling the kernel as we speak with the patch applied. Will report back later today.

Quoted 58 lines · On 5. Sep 2020, at 10:24, [email protected] wrote:
Quoted 43 lines · Pavel Begunkov
Quoted 18 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Quoted 7 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,
I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.
Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.
Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 6 lines
Quoted 4 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.
Essentially all the details that I would need to recreate what you're seeing.

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)
BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.

Quoted 13 lines
fs/io_uring.c +1 -3
index 0d7be2e9d005..000ae2acfd58 100644
@@ -2980,14 +2980,15 @@ static inline int io_rw_prep_async(struct io_kiocb *req, int rw,
 				   bool force_nonblock)
 {
 	struct io_async_rw *iorw = &req->io->rw;
+	struct iovec *iov;
 	ssize_t ret;
-	iorw->iter.iov = iorw->fast_iov;
-	ret = __io_import_iovec(rw, req, (struct iovec **) &iorw->iter.iov,
-				&iorw->iter, !force_nonblock);
Quoted 8 lines

fs/io_uring.c static inline int io_rw_prep_async(struct io_kiocb *req, int rw, · line 2994

Quoted 8 lines
fs/io_uring.c +3 -0
+	iorw->iter.iov = iov = iorw->fast_iov;
+	ret = __io_import_iovec(rw, req, &iov, &iorw->iter, !force_nonblock);
 	if (unlikely(ret < 0))
 		return ret;
+	iorw->iter.iov = iov;
 	io_req_map_rw(req, iorw->iter.iov, iorw->fast_iov, &iorw->iter);
 	return 0;
 }

Thanks for the speedy replies and finding/fixing this so fast! I'm new to kernel dev and haven't built my own yet but I think Norman is going to try out your patch soon.

Nick

maintainer
#8 lore raw
Quoted 3 lines · Norman Maurer

I can confirm this fixed the problem for us.

Thanks a lot of the quick turnaround (as always!).

No problem, thanks for reporting! I've queued up a regression test as well so this won't ever happen again.

Signature · 1 line
Jens Axboe
maintainer
#9 lore raw
In reply to Pavel Begunkov
Quoted 30 lines · Pavel Begunkov
Quoted 29 lines · Jens Axboe
Quoted 27 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Quoted 3 lines

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

Quoted 8 lines
Quoted 2 lines

Turns out there was a bug in the explicit handling, new in the current -rc series. Can you try and add the below?

Hah, absolutely the same patch was in a series I was going to send today, but with a note that it works by luck so not a bug. Apparently, it is :)>
BTW, const in iter->iov is guarding from such cases, yet another proof that const casts are evil.

Definitely, not a great idea to begin with...

Signature · 1 line
Jens Axboe
#10 lore raw
In reply to Jens Axboe
Quoted 22 lines · Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 9 lines · On 9/4/20 9:22 PM, [email protected] wrote:

Hi,

I am helping out with the netty io_uring integration, and came across some strange behaviour which seems like it might be a bug related to async offload of read/write iovecs.

Basically a WRITEV SQE seems to fail reliably with -BADADDRESS when the IOSQE_ASYNC flag is set but works fine otherwise (everything else the same). This is with 5.9.0-rc3.

Do you see it just on 5.9-rc3, or also 5.8? Just curious... But that is very odd in any case, ASYNC writev is even part of the regular tests. Any sort of deferral, be it explicit via ASYNC or implicit through needing to retry, saves all the needed details to retry without needing any of the original context.

Quoted 3 lines

Can you narrow down what exactly is being written - like file type, buffered/O_DIRECT, etc. What file system, what device is hosting it. The more details the better, will help me narrow down what is going on.

Quoted 5 lines

Forgot, also size of the IO (both total, but also number of iovecs in that particular request.

Essentially all the details that I would need to recreate what you're seeing.

I only started testing on 5.9-rc3 so not sure about earlier versions, but I'll try and report back.

It's a socket with O_NONBLOCK, iovec array length is ~30 and sum of buffer sizes ~1MB.

If it's not easy to recreate then please don't waste time since it could be my mistake - I'll try to make a standalone reproducer in that case.