ceph: fix error handling in ceph_sync_write
PATCH 1/1 ceph: fix error handling in ceph_sync_write
ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.
Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -1
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len;
Signature · 1 line
2.37.2
Hi Jeff,
How reproducible of this ? Two weeks ago I have run some tests didn't hit this.
-- Xiubo
Quoted 38 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-1@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len;
Ilya Dryomov
maintainerQuoted 40 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --
Quoted 2 lines
2.37.2
Hi Jeff,
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
Thanks,
Ilya
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --Quoted 2 lines
2.37.2
Hi Jeff,
Quoted 5 lines
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
OK, I'm only seeing 'ret' being set to the write length only when enabling encryption (i.e. with test_dummy_encryption mount option). So, maybe the right fix is something like:
fs/ceph/file.c +1 -0
@@ -1889,6 +1889,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,ceph_release_page_vector (pages, num_pages);break ; } + ret =0 ; } req = ceph_osdc_new_request(osdc, &ci->i_layout,
Cheers,
--
Luís
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --Quoted 2 lines
2.37.2
Hi Jeff,
Quoted 6 lines
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
That's what I had thought too but I wasn't sure:
[ceph: root@quad1 /]# ceph --version
ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)I'll see if I can confirm that this is coming from the OSD and not some other layer as well.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 54 lines · Jeff Layton
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --Quoted 2 lines
2.37.2
Hi Jeff,
Quoted 6 lines
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
Quoted 7 lines
That's what I had thought too but I wasn't sure:
[ceph: root@quad1 /]# ceph --version ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)I'll see if I can confirm that this is coming from the OSD and not some other layer as well.
My mistake. This bug turns out to be a different bug in the fscrypt stack. We can drop this patch (and I probably should have sent it as an RFC in the first place). Sorry for the noise!
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 63 lines · Luís Henriques
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --Quoted 2 lines
2.37.2
Hi Jeff,
Quoted 5 lines
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
OK, I'm only seeing 'ret' being set to the write length only when enabling encryption (i.e. with
test_dummy_encryptionmount option). So, maybe the right fix is something like:fs/ceph/file.c +0
-0
fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1895
Quoted 7 lines
fs/ceph/file.c +1
-0@@ -1889,6 +1889,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,ceph_release_page_vector (pages, num_pages);break ; } + ret =0 ; } req = ceph_osdc_new_request(osdc, &ci->i_layout,
No, actually. I think the original patch I sent is just wrong. There was another bug (another missing ceph_osdc_wait_request) in the fscrypt stack that was causing r_result to not appear to be zero.
I've fixed this in the ceph-fscrypt branch in my tree and it seems to be working. I'll plan do a re-send of the fscrypt patches that are not yet in the testing branch today.
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 54 lines · Jeff Layton
Quoted 53 lines · Jeff Layton
Quoted 44 lines · Ilya Dryomov
Quoted 39 lines · Jeff Layton
ceph_sync_writehas assumed that a zero result inreq->r_resultmeans success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/file.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/ceph/file.c +9
-2@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, req->r_end_latency, len, ret); out:ceph_osdc_put_request (req); -if (ret !=0 ) { +if (ret <0 ) {ceph_set_error_write (ci);break ; } +/* +* FIXME: it's unclear whether all OSD versions return the +* length written on a write. For now, assume that a 0 return +* means that everything got written. +*/ +if (ret && ret < len) + len = ret; +ceph_clear_error_write (ci); pos += len; written += len; --Quoted 2 lines
2.37.2
Hi Jeff,
Quoted 6 lines
AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.
What is the cluster version you are testing against?
That's what I had thought too but I wasn't sure:
Quoted 9 lines
Quoted 5 lines
[ceph: root@quad1 /]# ceph --version ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)I'll see if I can confirm that this is coming from the OSD and not some other layer as well.
My mistake. This bug turns out to be a different bug in the fscrypt stack. We can drop this patch (and I probably should have sent it as an RFC in the first place). Sorry for the noise!
Cool, thanks Jeff.
I saw you new update about this, they look good to me and will test them.
- Xiubo