Skip to contentSkip to conversation

libceph: init sent and completed when starting

PATCH In review Alex Elder 2 participants 3 messages May 11, 2013 Ceph common code (libceph)

lore
Share & tools
lore

Alex Elder

author
#0 lore raw

PATCH 1/1 libceph: init sent and completed when starting

The rbd code has a need to be able to restart an osd request that has already been started and completed once before. This currently wouldn't work right because the osd client code assumes an osd request will be started exactly once Certain fields in a request are never cleared and this leads to trouble if you try to reuse it.

Specifically, the r_sent, r_got_reply, and r_completed fields are never cleared. The r_sent field records the osd incarnation at the time the request was sent to that osd. If that's non-zero, the message won't get re-mapped to a target osd properly, and won't be put on the unsafe requests list the first time it's sent as it should. The r_got_reply field is used in handle_reply() to ensure the reply to a request is processed only once. And the r_completed field is used for lingering requests to avoid calling the callback function every time the osd client re-sends the request on behalf of its initiator.

Each osd request passes through ceph_osdc_start_request() when responsibility for the request is handed over to the osd client for completion. We can safely zero these three fields there each time a request gets started.

One last related change--clear the r_linger flag when a request is no longer registered as a linger request.

This resolves:

    http://tracker.ceph.com/issues/5026
---
 net/ceph/osd_client.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +0 -0
index a3395fd..d5953b8 100644
@@ -1204,6 +1204,7 @@ void ceph_osdc_unregister_linger_request(struct

ceph_osd_client *osdc,
mutex_lock(&osdc->request_mutex);
if (req->r_linger) {
__unregister_linger_request(osdc, req);
+ req->r_linger = 0;
ceph_osdc_put_request(req);
}
mutex_unlock(&osdc->request_mutex);
@@ -2120,7 +2121,9 @@ int ceph_osdc_start_request(struct ceph_osd_client *osdc,
down_read(&osdc->map_sem); mutex_lock(&osdc->request_mutex); __register_request(osdc, req);
- WARN_ON(req->r_sent);
+ req->r_sent = 0;
+ req->r_got_reply = 0;
+ req->r_completed = 0;
rc = __map_request(osdc, req, 0);
if (rc < 0) {
if (nofail) {

Signature · 1 line
1.7.9.5
author
#1 lore raw
Quoted 5 lines · Alex Elder

The rbd code has a need to be able to restart an osd request that has already been started and completed once before. This currently wouldn't work right because the osd client code assumes an osd request will be started exactly once Certain fields in a request are never cleared and this leads to trouble if you try to reuse it.

This patch and the next two, and the two series of patches that follow them, are available in the "review/wip-flatten" branch of the ceph-glient git repository.

-Alex

Quoted 53 lines

Specifically, the r_sent, r_got_reply, and r_completed fields are never cleared. The r_sent field records the osd incarnation at the time the request was sent to that osd. If that's non-zero, the message won't get re-mapped to a target osd properly, and won't be put on the unsafe requests list the first time it's sent as it should. The r_got_reply field is used in handle_reply() to ensure the reply to a request is processed only once. And the r_completed field is used for lingering requests to avoid calling the callback function every time the osd client re-sends the request on behalf of its initiator.

Each osd request passes through ceph_osdc_start_request() when responsibility for the request is handed over to the osd client for completion. We can safely zero these three fields there each time a request gets started.

One last related change--clear the r_linger flag when a request is no longer registered as a linger request.

This resolves:

    http://tracker.ceph.com/issues/5026
---
 net/ceph/osd_client.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +1 -0
index a3395fd..d5953b8 100644
@@ -1204,6 +1204,7 @@ void ceph_osdc_unregister_linger_request(struct
 ceph_osd_client *osdc,
 	mutex_lock(&osdc->request_mutex);
 	if (req->r_linger) {
 		__unregister_linger_request(osdc, req);
+		req->r_linger = 0;
 		ceph_osdc_put_request(req);
 	}
net/ceph/osd_client.c +3 -1
 	mutex_unlock(&osdc->request_mutex);
@@ -2120,7 +2121,9 @@ int ceph_osdc_start_request(struct ceph_osd_client
 *osdc,
 	down_read(&osdc->map_sem);
 	mutex_lock(&osdc->request_mutex);
 	__register_request(osdc, req);
-	WARN_ON(req->r_sent);
+	req->r_sent = 0;
+	req->r_got_reply = 0;
+	req->r_completed = 0;
 	rc = __map_request(osdc, req, 0);
 	if (rc < 0) {

if (nofail) {

Reviewed-by on v1 1/1
#2 lore raw
In reply to Alex Elder
Quoted 59 lines · Alex Elder

The rbd code has a need to be able to restart an osd request that has already been started and completed once before. This currently wouldn't work right because the osd client code assumes an osd request will be started exactly once Certain fields in a request are never cleared and this leads to trouble if you try to reuse it.

Specifically, the r_sent, r_got_reply, and r_completed fields are never cleared. The r_sent field records the osd incarnation at the time the request was sent to that osd. If that's non-zero, the message won't get re-mapped to a target osd properly, and won't be put on the unsafe requests list the first time it's sent as it should. The r_got_reply field is used in handle_reply() to ensure the reply to a request is processed only once. And the r_completed field is used for lingering requests to avoid calling the callback function every time the osd client re-sends the request on behalf of its initiator.

Each osd request passes through ceph_osdc_start_request() when responsibility for the request is handed over to the osd client for completion. We can safely zero these three fields there each time a request gets started.

One last related change--clear the r_linger flag when a request is no longer registered as a linger request.

This resolves:

     http://tracker.ceph.com/issues/5026
---
  net/ceph/osd_client.c |    5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
net/ceph/osd_client.c +1 -0
index a3395fd..d5953b8 100644
@@ -1204,6 +1204,7 @@ void ceph_osdc_unregister_linger_request(struct
 ceph_osd_client *osdc,
  	mutex_lock(&osdc->request_mutex);
  	if (req->r_linger) {
  		__unregister_linger_request(osdc, req);
+		req->r_linger = 0;
  		ceph_osdc_put_request(req);
  	}
  	mutex_unlock(&osdc->request_mutex);
net/ceph/osd_client.c +3 -1
@@ -2120,7 +2121,9 @@ int ceph_osdc_start_request(struct ceph_osd_client
 *osdc,
  	down_read(&osdc->map_sem);
  	mutex_lock(&osdc->request_mutex);
  	__register_request(osdc, req);
-	WARN_ON(req->r_sent);
+	req->r_sent = 0;
+	req->r_got_reply = 0;
+	req->r_completed = 0;
  	rc = __map_request(osdc, req, 0);
  	if (rc < 0) {
  		if (nofail) {