Skip to contentSkip to conversation

libceph: fix sparse-read failure bug

PATCH Superseded [email protected] 3 participants 10 messages Dec 8, 2023 1 message not in the archive Ceph common code (libceph)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover libceph: fix sparse-read failure bug

From: Xiubo Li <[email protected]>

Xiubo Li (2):
libceph: fail the sparse-read if there still has data in socket
libceph: just wait for more data to be available on the socket

 net/ceph/messenger_v1.c | 18 ++++++++++--------
 net/ceph/osd_client.c   |  4 +++-
 2 files changed, 13 insertions(+), 9 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/2 libceph: fail the sparse-read if there still has data in socket

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/osd_client.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -1
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
 		fallthrough;
 	case CEPH_SPARSE_READ_DATA:
 		if (sr->sr_index >= count) {
-			if (sr->sr_datalen && count)
+			if (sr->sr_datalen) {
 				pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
 						    sr->sr_datalen, sr->sr_index,
 						    count);
+				return -EREMOTEIO;
+			}
 
 			sr->sr_state = CEPH_SPARSE_READ_HDR;
 			goto next_op;
Signature · 1 line
2.43.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket

Quoted 18 lines · On Fri, 2023-12-08 at 12:33 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/osd_client.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +0 -0
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
 		fallthrough;
 	case CEPH_SPARSE_READ_DATA:
 		if (sr->sr_index >= count) {

net/ceph/osd_client.c static int osd_sparse_read(struct ceph_connection *con, · line 5923

Quoted 10 lines
net/ceph/osd_client.c +3 -1
-			if (sr->sr_datalen && count)
+			if (sr->sr_datalen) {
 				pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
 						    sr->sr_datalen, sr->sr_index,
 						    count);
+				return -EREMOTEIO;
+			}
 
 			sr->sr_state = CEPH_SPARSE_READ_HDR;
 			goto next_op;

Nice catch. You can add:

maintainer
#3 lore raw

Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket

In reply to [email protected]
Quoted 30 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/osd_client.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -2
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
                fallthrough;
        case CEPH_SPARSE_READ_DATA:
                if (sr->sr_index >= count) {
-                       if (sr->sr_datalen && count)
+                       if (sr->sr_datalen) {
                                pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
                                                    sr->sr_datalen, sr->sr_index,
                                                    count);
+                               return -EREMOTEIO;
+                       }
 
                        sr->sr_state = CEPH_SPARSE_READ_HDR;
                        goto next_op;
--
Quoted 2 lines

2.43.0

Hi Xiubo,

There is a patch in linux-next, also from you, which is conflicting with this one: cca19d307d35 ("libceph: check the data length when sparse read finishes"). Do you want it replaced?

Thanks,

Ilya

author
#4 lore raw

Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket

Quoted 34 lines · Ilya Dryomov
Quoted 29 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
  net/ceph/osd_client.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -2
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
                 fallthrough;
         case CEPH_SPARSE_READ_DATA:
                 if (sr->sr_index >= count) {
-                       if (sr->sr_datalen && count)
+                       if (sr->sr_datalen) {
                                 pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
                                                     sr->sr_datalen, sr->sr_index,
                                                     count);
+                               return -EREMOTEIO;
+                       }
 
                         sr->sr_state = CEPH_SPARSE_READ_HDR;
                         goto next_op;
--
Quoted 2 lines

2.43.0

Hi Xiubo,

Quoted 3 lines

There is a patch in linux-next, also from you, which is conflicting with this one: cca19d307d35 ("libceph: check the data length when sparse read finishes"). Do you want it replaced?

Yeah, let me fold them.

Thanks Jeff and Ilya.

Quoted 4 lines

Thanks,

Ilya

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#6 lore raw

Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket

Quoted 39 lines · Xiubo Li

On 12/8/23 19:31, Ilya Dryomov wrote:

On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/osd_client.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -2
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
                fallthrough;
        case CEPH_SPARSE_READ_DATA:
                if (sr->sr_index >= count) {
-                       if (sr->sr_datalen && count)
+                       if (sr->sr_datalen) {
                                pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
                                                    sr->sr_datalen, sr->sr_index,
                                                    count);
+                               return -EREMOTEIO;
+                       }
 
                        sr->sr_state = CEPH_SPARSE_READ_HDR;
                        goto next_op;
--

2.43.0

Hi Xiubo,

Quoted 7 lines

There is a patch in linux-next, also from you, which is conflicting with this one: cca19d307d35 ("libceph: check the data length when sparse read finishes"). Do you want it replaced?

Ilya,

I found the commit cca19d307d35 has already in the master branch. Could you fold and update it ?

I would like to see the entire fix first. You seem to be going back and forth between just issuing a warning or also returning an error and the precise if condition there, so I'm starting to think that the bug is not fully understood and neither patch might be necessary.

Thanks,

Ilya

author
#7 lore raw

Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket

Quoted 43 lines · Ilya Dryomov
Quoted 42 lines · Xiubo Li

On 12/8/23 19:31, Ilya Dryomov wrote:

On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Once this happens that means there have bugs.

URL: https://tracker.ceph.com/issues/63586

---
  net/ceph/osd_client.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -2
index 5753036d1957..848ef19055a0 100644
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con,
                 fallthrough;
         case CEPH_SPARSE_READ_DATA:
                 if (sr->sr_index >= count) {
-                       if (sr->sr_datalen && count)
+                       if (sr->sr_datalen) {
                                 pr_warn_ratelimited("sr_datalen %u sr_index %d count %u\n",
                                                     sr->sr_datalen, sr->sr_index,
                                                     count);
+                               return -EREMOTEIO;
+                       }
 
                         sr->sr_state = CEPH_SPARSE_READ_HDR;
                         goto next_op;
--

2.43.0

Hi Xiubo,

There is a patch in linux-next, also from you, which is conflicting with this one: cca19d307d35 ("libceph: check the data length when sparse read finishes"). Do you want it replaced?

Quoted 7 lines
Quoted 3 lines

Ilya,

I found the commit cca19d307d35 has already in the master branch. Could you fold and update it ?

I would like to see the entire fix first. You seem to be going back and forth between just issuing a warning or also returning an error and the precise if condition there, so I'm starting to think that the bug is not fully understood and neither patch might be necessary.

Sure, I just sent out the second version.

Thanks

- Xiubo

Quoted 4 lines

Thanks,

Ilya

#8 lore raw

PATCH 2/2 libceph: just wait for more data to be available on the socket

In reply to [email protected]

From: Xiubo Li <[email protected]>

The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/messenger_v1.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/ceph/messenger_v1.c +10 -8
index f9a50d7f0d20..aff81fef932f 100644
@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)
 	/* header */
 	size = sizeof(con->v1.in_hdr);
 	end = size;
-	ret = read_partial(con, end, size, &con->v1.in_hdr);
-	if (ret <= 0)
-		return ret;
+	if (con->v1.in_base_pos < end) {
+		ret = read_partial(con, end, size, &con->v1.in_hdr);
+		if (ret <= 0)
+			return ret;
 
-	crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
-	if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
-		pr_err("read_partial_message bad hdr crc %u != expected %u\n",
-		       crc, con->v1.in_hdr.crc);
-		return -EBADMSG;
+		crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
+		if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
+			pr_err("read_partial_message bad hdr crc %u != expected %u\n",
+			       crc, con->v1.in_hdr.crc);
+			return -EBADMSG;
+		}
 	}
 
 	front_len = le32_to_cpu(con->v1.in_hdr.front_len);
Signature · 1 line
2.43.0
maintainer
#9 lore raw

Re: [PATCH 2/2] libceph: just wait for more data to be available on the socket

Quoted 44 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.

URL: https://tracker.ceph.com/issues/63586

---
 net/ceph/messenger_v1.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
net/ceph/messenger_v1.c +10 -9
index f9a50d7f0d20..aff81fef932f 100644
@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)
        /* header */
        size = sizeof(con->v1.in_hdr);
        end = size;
-       ret = read_partial(con, end, size, &con->v1.in_hdr);
-       if (ret <= 0)
-               return ret;
+       if (con->v1.in_base_pos < end) {
+               ret = read_partial(con, end, size, &con->v1.in_hdr);
+               if (ret <= 0)
+                       return ret;
 
-       crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
-       if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
-               pr_err("read_partial_message bad hdr crc %u != expected %u\n",
-                      crc, con->v1.in_hdr.crc);
-               return -EBADMSG;
+               crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
+               if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
+                       pr_err("read_partial_message bad hdr crc %u != expected %u\n",
+                              crc, con->v1.in_hdr.crc);
+                       return -EBADMSG;
+               }
        }
 
        front_len = le32_to_cpu(con->v1.in_hdr.front_len);
--
Quoted 2 lines

2.43.0

Hi Xiubo,

This doesn't seem right to me. read_partial() is supposed to be called unconditionally. On a short read (i.e. when it's unable to fill the destination buffer -- in this case the header), it returns 0 and the stack is supposed to unroll all the way up to ceph_con_workfn().

If the destination buffer is already filled, read_partial() does nothing and returns 1. Recomputing the header crc in case read_partial_message() is called repeatedly shouldn't be an issue because con->v1.in_hdr shouldn't be modified in the interim. If it gets modified, it's a bug.

It might help if you provide a step-by-step breakdown of the scenario that you are trying to address in the commit message.

Thanks,

Ilya

author
#10 lore raw

Re: [PATCH 2/2] libceph: just wait for more data to be available on the socket

Quoted 53 lines · Ilya Dryomov
Quoted 43 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.

URL: https://tracker.ceph.com/issues/63586

---
  net/ceph/messenger_v1.c | 18 ++++++++++--------
  1 file changed, 10 insertions(+), 8 deletions(-)
net/ceph/messenger_v1.c +10 -9
index f9a50d7f0d20..aff81fef932f 100644
@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)
         /* header */
         size = sizeof(con->v1.in_hdr);
         end = size;
-       ret = read_partial(con, end, size, &con->v1.in_hdr);
-       if (ret <= 0)
-               return ret;
+       if (con->v1.in_base_pos < end) {
+               ret = read_partial(con, end, size, &con->v1.in_hdr);
+               if (ret <= 0)
+                       return ret;
 
-       crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
-       if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
-               pr_err("read_partial_message bad hdr crc %u != expected %u\n",
-                      crc, con->v1.in_hdr.crc);
-               return -EBADMSG;
+               crc = crc32c(0, &con->v1.in_hdr, offsetof(struct ceph_msg_header, crc));
+               if (cpu_to_le32(crc) != con->v1.in_hdr.crc) {
+                       pr_err("read_partial_message bad hdr crc %u != expected %u\n",
+                              crc, con->v1.in_hdr.crc);
+                       return -EBADMSG;
+               }
         }
 
         front_len = le32_to_cpu(con->v1.in_hdr.front_len);
--
Quoted 2 lines

2.43.0

Hi Xiubo,

This doesn't seem right to me. read_partial() is supposed to be called unconditionally. On a short read (i.e. when it's unable to fill the destination buffer -- in this case the header), it returns 0 and the stack is supposed to unroll all the way up to ceph_con_workfn().

Quoted 8 lines

If the destination buffer is already filled, read_partial() does nothing and returns 1. Recomputing the header crc in case read_partial_message() is called repeatedly shouldn't be an issue because con->v1.in_hdr shouldn't be modified in the interim. If it gets modified, it's a bug.

It might help if you provide a step-by-step breakdown of the scenario that you are trying to address in the commit message.

Yeah, you are correct.

So my first change was correct, I just adjust the patch before sending it out.

Let me change it back.

Thanks

- Xiubo

Quoted 4 lines

Thanks,

Ilya