libceph: Reset sparse-read on fault
Subsystems (2)
PATCH 1/1 libceph: Reset sparse-read on fault
When a fault occurs, the connection is abandoned, reestablished, and any pending operations are retried. The OSD client tracks the progress of a sparse-read reply using a separate state machine, largely independent of the messenger's state.
If a connection is lost mid-payload or the sparse-read state machine returns an error, the sparse-read state is not reset. The OSD client will then interpret the beginning of a new reply as the continuation of the old one. If this makes the sparse-read machinery enter a failure state, it may never recover, producing loops like:
libceph: [0] got 0 extents
libceph: data len 142248331 != extent len 0
libceph: osd0 (1)...:6801 socket error on read
libceph: data len 142248331 != extent len 0
libceph: osd0 (1)...:6801 socket error on read
Therefore, reset the sparse-read state in osd_fault(), ensuring retries start from a clean state.
--- net/ceph/osd_client.c | 3 +++ 1 file changed, 3 insertions(+)
net/ceph/osd_client.c +3 -0
@@ -4281,6 +4281,9 @@ static void osd_fault(struct ceph_connection *con) goto out_unlock; } + osd->o_sparse_op_idx =-1 ; +ceph_init_sparse_read (&osd->o_sparse_read); +if (!reopen_osd (osd))kick_osd_requests (osd);maybe_request_map (osdc);
Signature · 1 line
2.51.2
Ilya Dryomov
maintainerQuoted 42 lines · Sam Edwards
When a fault occurs, the connection is abandoned, reestablished, and any pending operations are retried. The OSD client tracks the progress of a sparse-read reply using a separate state machine, largely independent of the messenger's state.
If a connection is lost mid-payload or the sparse-read state machine returns an error, the sparse-read state is not reset. The OSD client will then interpret the beginning of a new reply as the continuation of the old one. If this makes the sparse-read machinery enter a failure state, it may never recover, producing loops like:
libceph: [0] got 0 extents
libceph: data len 142248331 != extent len 0
libceph: osd0 (1)...:6801 socket error on read
libceph: data len 142248331 != extent len 0
libceph: osd0 (1)...:6801 socket error on readTherefore, reset the sparse-read state in
osd_fault(), ensuring retries start from a clean state.Signed-off-by Sam Edwards <[email protected]>--- net/ceph/osd_client.c | 3 +++ 1 file changed, 3 insertions(+)net/ceph/osd_client.c +3
-1@@ -4281,6 +4281,9 @@ static void osd_fault(struct ceph_connection *con) goto out_unlock; } + osd->o_sparse_op_idx =-1 ; +ceph_init_sparse_read (&osd->o_sparse_read); +if (!reopen_osd (osd))kick_osd_requests (osd);maybe_request_map (osdc); --
Quoted 2 lines
2.51.2
Hi Sam,
Good catch! Applied (with the sad note that support for sparse reads is officially the most problematic patchset that ever made it into libceph).
Thanks,
Ilya