libceph: fix sparse-read failure bug
cover libceph: fix sparse-read failure bug
From: Xiubo Li <[email protected]>
Xiubo Li (2):
libceph: fail the sparse-read if there still has data in socket
libceph: just wait for more data to be available on the socket
net/ceph/messenger_v1.c | 18 ++++++++++-------- net/ceph/osd_client.c | 4 +++- 2 files changed, 13 insertions(+), 9 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 libceph: fail the sparse-read if there still has data in socket
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +3 -1
@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) { -if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op;
Signature · 1 line
2.43.0
Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket
Quoted 18 lines · On Fri, 2023-12-08 at 12:33 +0800, [email protected] wrote:
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/ceph/osd_client.c +0
-0@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) {
net/ceph/osd_client.c static int osd_sparse_read(struct ceph_connection *con, · line 5923
Quoted 10 lines
net/ceph/osd_client.c +3
-1-if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op;
Nice catch. You can add:
Ilya Dryomov
maintainerRe: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket
In reply to [email protected]Quoted 30 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/ceph/osd_client.c +3
-2@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) { -if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op; --
Quoted 2 lines
2.43.0
Hi Xiubo,
There is a patch in linux-next, also from you, which is conflicting with this one: cca19d307d35 ("libceph: check the data length when sparse read finishes"). Do you want it replaced?
Thanks,
Ilya
Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket
Quoted 34 lines · Ilya Dryomov
Quoted 29 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/ceph/osd_client.c +3
-2@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) { -if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op; --Quoted 2 lines
2.43.0
Hi Xiubo,
Quoted 3 lines
There is a patch in linux-next, also from you, which is conflicting with this one:
cca19d307d35("libceph: check the data length when sparse read finishes"). Do you want it replaced?
Yeah, let me fold them.
Thanks Jeff and Ilya.
Quoted 4 lines
Thanks,
Ilya
A message that is not in the archive (sent off-list or not yet received): [email protected]
Ilya Dryomov
maintainerRe: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket
Quoted 39 lines · Xiubo Li
On 12/8/23 19:31, Ilya Dryomov wrote:
On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/ceph/osd_client.c +3
-2@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) { -if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op; --2.43.0
Hi Xiubo,
Quoted 7 lines
There is a patch in linux-next, also from you, which is conflicting with this one:
cca19d307d35("libceph: check the data length when sparse read finishes"). Do you want it replaced?Ilya,
I found the commit
cca19d307d35has already in the master branch. Could you fold and update it ?
I would like to see the entire fix first. You seem to be going back and forth between just issuing a warning or also returning an error and the precise if condition there, so I'm starting to think that the bug is not fully understood and neither patch might be necessary.
Thanks,
Ilya
Re: [PATCH 1/2] libceph: fail the sparse-read if there still has data in socket
Quoted 43 lines · Ilya Dryomov
Quoted 42 lines · Xiubo Li
On 12/8/23 19:31, Ilya Dryomov wrote:
On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Once this happens that means there have bugs.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/osd_client.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)net/ceph/osd_client.c +3
-2@@ -5912,10 +5912,12 @@ static int osd_sparse_read(struct ceph_connection *con, fallthrough;case CEPH_SPARSE_READ_DATA :if (sr->sr_index >= count) { -if (sr->sr_datalen && count) +if (sr->sr_datalen) {pr_warn_ratelimited ("sr_datalen %u sr_index %d count %u\n" , sr->sr_datalen, sr->sr_index, count); +return -EREMOTEIO ; + } sr->sr_state =CEPH_SPARSE_READ_HDR ; goto next_op; --2.43.0
Hi Xiubo,
There is a patch in linux-next, also from you, which is conflicting with this one:
cca19d307d35("libceph: check the data length when sparse read finishes"). Do you want it replaced?
Quoted 7 lines
Quoted 3 lines
Ilya,
I found the commit
cca19d307d35has already in the master branch. Could you fold and update it ?I would like to see the entire fix first. You seem to be going back and forth between just issuing a warning or also returning an error and the precise if condition there, so I'm starting to think that the bug is not fully understood and neither patch might be necessary.
Sure, I just sent out the second version.
Thanks
- Xiubo
Quoted 4 lines
Thanks,
Ilya
PATCH 2/2 libceph: just wait for more data to be available on the socket
In reply to [email protected]From: Xiubo Li <[email protected]>
The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.
URL: https://tracker.ceph.com/issues/63586
--- net/ceph/messenger_v1.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
net/ceph/messenger_v1.c +10 -8
@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)/* header */ size =sizeof (con->v1.in_hdr); end = size; - ret =read_partial (con, end, size, &con->v1.in_hdr); -if (ret <=0 ) -return ret; +if (con->v1.in_base_pos < end) { + ret =read_partial (con, end, size, &con->v1.in_hdr); +if (ret <=0 ) +return ret; - crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); -if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { -pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , - crc, con->v1.in_hdr.crc); -return -EBADMSG ; + crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); +if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { +pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , + crc, con->v1.in_hdr.crc); +return -EBADMSG ; + } } front_len =le32_to_cpu (con->v1.in_hdr.front_len);
Signature · 1 line
2.43.0
Ilya Dryomov
maintainerRe: [PATCH 2/2] libceph: just wait for more data to be available on the socket
Quoted 44 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/messenger_v1.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)net/ceph/messenger_v1.c +10
-9@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)/* header */ size =sizeof (con->v1.in_hdr); end = size; - ret =read_partial (con, end, size, &con->v1.in_hdr); -if (ret <=0 ) -return ret; +if (con->v1.in_base_pos < end) { + ret =read_partial (con, end, size, &con->v1.in_hdr); +if (ret <=0 ) +return ret; - crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); -if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { -pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , - crc, con->v1.in_hdr.crc); -return -EBADMSG ; + crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); +if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { +pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , + crc, con->v1.in_hdr.crc); +return -EBADMSG ; + } } front_len =le32_to_cpu (con->v1.in_hdr.front_len); --
Quoted 2 lines
2.43.0
Hi Xiubo,
This doesn't seem right to me. read_partial() is supposed to be called unconditionally. On a short read (i.e. when it's unable to fill the destination buffer -- in this case the header), it returns 0 and the stack is supposed to unroll all the way up to ceph_con_workfn().
If the destination buffer is already filled, read_partial() does nothing and returns 1. Recomputing the header crc in case read_partial_message() is called repeatedly shouldn't be an issue because con->v1.in_hdr shouldn't be modified in the interim. If it gets modified, it's a bug.
It might help if you provide a step-by-step breakdown of the scenario that you are trying to address in the commit message.
Thanks,
Ilya
Re: [PATCH 2/2] libceph: just wait for more data to be available on the socket
Quoted 53 lines · Ilya Dryomov
Quoted 43 lines · On Fri, Dec 8, 2023 at 5:34 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
The messages from ceph maybe split into multiple socket packages and we just need to wait for all the data to be availiable on the sokcet.
URL: https://tracker.ceph.com/issues/63586
Signed-off-by Xiubo Li <[email protected]>--- net/ceph/messenger_v1.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)net/ceph/messenger_v1.c +10
-9@@ -1160,15 +1160,17 @@ static int read_partial_message(struct ceph_connection *con)/* header */ size =sizeof (con->v1.in_hdr); end = size; - ret =read_partial (con, end, size, &con->v1.in_hdr); -if (ret <=0 ) -return ret; +if (con->v1.in_base_pos < end) { + ret =read_partial (con, end, size, &con->v1.in_hdr); +if (ret <=0 ) +return ret; - crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); -if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { -pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , - crc, con->v1.in_hdr.crc); -return -EBADMSG ; + crc =crc32c (0 , &con->v1.in_hdr, offsetof(struct ceph_msg_header , crc)); +if (cpu_to_le32 (crc) != con->v1.in_hdr.crc) { +pr_err ("read_partial_message bad hdr crc %u != expected %u\n" , + crc, con->v1.in_hdr.crc); +return -EBADMSG ; + } } front_len =le32_to_cpu (con->v1.in_hdr.front_len); --Quoted 2 lines
2.43.0
Hi Xiubo,
This doesn't seem right to me.
read_partial()is supposed to be called unconditionally. On a short read (i.e. when it's unable to fill the destination buffer -- in this case the header), it returns 0 and the stack is supposed to unroll all the way up toceph_con_workfn().
Quoted 8 lines
If the destination buffer is already filled,
read_partial()does nothing and returns 1. Recomputing the header crc in caseread_partial_message()is called repeatedly shouldn't be an issue becausecon->v1.in_hdrshouldn't be modified in the interim. If it gets modified, it's a bug.It might help if you provide a step-by-step breakdown of the scenario that you are trying to address in the commit message.
Yeah, you are correct.
So my first change was correct, I just adjust the patch before sending it out.
Let me change it back.
Thanks
- Xiubo
Quoted 4 lines
Thanks,
Ilya