Skip to contentSkip to conversation

ceph: fix error handling in ceph_sync_write

PATCH In review Jeff Layton 4 participants 8 messages Aug 24, 2022 – Aug 26, 2022 Ceph client

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: fix error handling in ceph_sync_write

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -1
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 					  req->r_end_latency, len, ret);
 out:
 		ceph_osdc_put_request(req);
-		if (ret != 0) {
+		if (ret < 0) {
 			ceph_set_error_write(ci);
 			break;
 		}
 
+		/*
+		 * FIXME: it's unclear whether all OSD versions return the
+		 * length written on a write. For now, assume that a 0 return
+		 * means that everything got written.
+		 */
+		if (ret && ret < len)
+			len = ret;
+
 		ceph_clear_error_write(ci);
 		pos += len;
 		written += len;
Signature · 1 line
2.37.2
#1 lore raw

Hi Jeff,

How reproducible of this ? Two weeks ago I have run some tests didn't hit this.

-- Xiubo

Quoted 38 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
  fs/ceph/file.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -1
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
  					  req->r_end_latency, len, ret);
  out:
  		ceph_osdc_put_request(req);
-		if (ret != 0) {
+		if (ret < 0) {
  			ceph_set_error_write(ci);
  			break;
  		}
 
+		/*
+		 * FIXME: it's unclear whether all OSD versions return the
+		 * length written on a write. For now, assume that a 0 return
+		 * means that everything got written.
+		 */
+		if (ret && ret < len)
+			len = ret;
+
  		ceph_clear_error_write(ci);
  		pos += len;
  		written += len;
maintainer
#2 lore raw
In reply to Jeff Layton
Quoted 40 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                          req->r_end_latency, len, ret);
 out:
                ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                        ceph_set_error_write(ci);
                        break;
                }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                ceph_clear_error_write(ci);
                pos += len;
                written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

Thanks,

Ilya

#3 lore raw
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                          req->r_end_latency, len, ret);
 out:
                ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                        ceph_set_error_write(ci);
                        break;
                }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                ceph_clear_error_write(ci);
                pos += len;
                written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

Quoted 5 lines

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

OK, I'm only seeing 'ret' being set to the write length only when enabling encryption (i.e. with test_dummy_encryption mount option). So, maybe the right fix is something like:

fs/ceph/file.c +1 -0
index 16dcade66923..5119d87d61fb 100644
@@ -1889,6 +1889,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 				ceph_release_page_vector(pages, num_pages);
 				break;
 			}
+			ret = 0;
 		}
 
 		req = ceph_osdc_new_request(osdc, &ci->i_layout,

Cheers,
--
Luís

#4 lore raw
Quoted 63 lines · Luís Henriques
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                          req->r_end_latency, len, ret);
 out:
                ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                        ceph_set_error_write(ci);
                        break;
                }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                ceph_clear_error_write(ci);
                pos += len;
                written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

Quoted 5 lines

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

OK, I'm only seeing 'ret' being set to the write length only when enabling encryption (i.e. with test_dummy_encryption mount option). So, maybe the right fix is something like:

fs/ceph/file.c +0 -0
index 16dcade66923..5119d87d61fb 100644

fs/ceph/file.c ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos, · line 1895

Quoted 7 lines
fs/ceph/file.c +1 -0
@@ -1889,6 +1889,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
 				ceph_release_page_vector(pages, num_pages);
 				break;
 			}
+			ret = 0;
 		}
 
 		req = ceph_osdc_new_request(osdc, &ci->i_layout,

No, actually. I think the original patch I sent is just wrong. There was another bug (another missing ceph_osdc_wait_request) in the fscrypt stack that was causing r_result to not appear to be zero.

I've fixed this in the ceph-fscrypt branch in my tree and it seems to be working. I'll plan do a re-send of the fscrypt patches that are not yet in the testing branch today.

Thanks!

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
In reply to Ilya Dryomov
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                          req->r_end_latency, len, ret);
 out:
                ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                        ceph_set_error_write(ci);
                        break;
                }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                ceph_clear_error_write(ci);
                pos += len;
                written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

Quoted 6 lines

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

That's what I had thought too but I wasn't sure:

    [ceph: root@quad1 /]# ceph --version
    ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)

I'll see if I can confirm that this is coming from the OSD and not some other layer as well.

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw
Quoted 54 lines · Jeff Layton
Quoted 46 lines · Ilya Dryomov
Quoted 40 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
 fs/ceph/file.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                          req->r_end_latency, len, ret);
 out:
                ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                        ceph_set_error_write(ci);
                        break;
                }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                ceph_clear_error_write(ci);
                pos += len;
                written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

Quoted 6 lines

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

Quoted 7 lines

That's what I had thought too but I wasn't sure:

    [ceph: root@quad1 /]# ceph --version
    ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)

I'll see if I can confirm that this is coming from the OSD and not some other layer as well.

My mistake. This bug turns out to be a different bug in the fscrypt stack. We can drop this patch (and I probably should have sent it as an RFC in the first place). Sorry for the noise!

Signature · 1 line
Jeff Layton <[email protected]>
#7 lore raw
Quoted 54 lines · Jeff Layton
Quoted 53 lines · Jeff Layton
Quoted 44 lines · Ilya Dryomov
Quoted 39 lines · Jeff Layton

ceph_sync_write has assumed that a zero result in req->r_result means success. Testing with a recent cluster however shows the OSD returning a non-zero length written here. I'm not sure whether and when this changed, but fix the code to accept either result.

Assume a negative result means error, and anything else is a success. If we're given a short length, then return a short write.

---
  fs/ceph/file.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
fs/ceph/file.c +9 -2
index 86265713a743..c0b2c8968be9 100644
@@ -1632,11 +1632,19 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,
                                           req->r_end_latency, len, ret);
  out:
                 ceph_osdc_put_request(req);
-               if (ret != 0) {
+               if (ret < 0) {
                         ceph_set_error_write(ci);
                         break;
                 }
 
+               /*
+                * FIXME: it's unclear whether all OSD versions return the
+                * length written on a write. For now, assume that a 0 return
+                * means that everything got written.
+                */
+               if (ret && ret < len)
+                       len = ret;
+
                 ceph_clear_error_write(ci);
                 pos += len;
                 written += len;
--
Quoted 2 lines

2.37.2

Hi Jeff,

Quoted 6 lines

AFAIK OSDs aren't allowed to return any kind of length on a write and there is no such thing as a short write. This definitely needs deeper investigation.

What is the cluster version you are testing against?

That's what I had thought too but I wasn't sure:

Quoted 9 lines
Quoted 5 lines
     [ceph: root@quad1 /]# ceph --version
     ceph version 17.0.0-14400-gf61b38dc (f61b38dc82e94f14e7a0a5f6a5888c0c78fafa6c) quincy (dev)

I'll see if I can confirm that this is coming from the OSD and not some other layer as well.

My mistake. This bug turns out to be a different bug in the fscrypt stack. We can drop this patch (and I probably should have sent it as an RFC in the first place). Sorry for the noise!

Cool, thanks Jeff.

I saw you new update about this, they look good to me and will test them.

- Xiubo