Skip to contentSkip to conversation

libceph: must hold mutex for reset_changed_osds()

PATCH In review Alex Elder 2 participants 3 messages May 15, 2013 – May 16, 2013 Ceph common code (libceph)

lore
Share & tools
lore

Alex Elder

author
#0 lore raw

PATCH 1/1 libceph: must hold mutex for reset_changed_osds()

An osd client has a red-black tree describing its osds, and occasionally we would get crashes due to one of these trees tree becoming corrupt somehow.

The problem turned out to be that reset_changed_osds() was being called without protection of the osd client request mutex. That function would call __reset_osd() for any osd that had changed, and __reset_osd() would call __remove_osd() for any osd with no outstanding requests, and finally __remove_osd() would remove the corresponding entry from the red-black tree. Thus, the tree was getting modified without having any lock protection, and was vulnerable to problems due to concurrent updates.

This appears to be the only osd tree updating path that has this problem. It can be fairly easily fixed by moving the call up a few lines, to just before the request mutex gets dropped in kick_requests().

This resolves:

    http://tracker.ceph.com/issues/5043
---
 net/ceph/osd_client.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/osd_client.c +0 -0
index d5953b8..3a246a6 100644
@@ -1675,13 +1675,13 @@ static void kick_requests(struct ceph_osd_client

*osdc, int force_resend)
__register_request(osdc, req); __unregister_linger_request(osdc, req);
}
+ reset_changed_osds(osdc);
mutex_unlock(&osdc->request_mutex);

if (needmap) {
dout("%d requests for down osds, need new map\n", needmap); ceph_monc_request_next_osdmap(&osdc->client->monc);
}
- reset_changed_osds(osdc);
}

Signature · 1 line
1.7.9.5
Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · Alex Elder

An osd client has a red-black tree describing its osds, and occasionally we would get crashes due to one of these trees tree becoming corrupt somehow.

The problem turned out to be that reset_changed_osds() was being called without protection of the osd client request mutex. That function would call __reset_osd() for any osd that had changed, and __reset_osd() would call __remove_osd() for any osd with no outstanding requests, and finally __remove_osd() would remove the corresponding entry from the red-black tree. Thus, the tree was getting modified without having any lock protection, and was vulnerable to problems due to concurrent updates.

Quoted 8 lines

This appears to be the only osd tree updating path that has this problem. It can be fairly easily fixed by moving the call up a few lines, to just before the request mutex gets dropped in kick_requests().

This resolves:

    http://tracker.ceph.com/issues/5043
Quoted 33 lines
---
 net/ceph/osd_client.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/osd_client.c +1 -1
index d5953b8..3a246a6 100644
@@ -1675,13 +1675,13 @@ static void kick_requests(struct ceph_osd_client
 *osdc, int force_resend)
 		__register_request(osdc, req);
 		__unregister_linger_request(osdc, req);
 	}
+	reset_changed_osds(osdc);
 	mutex_unlock(&osdc->request_mutex);
 
 	if (needmap) {
 		dout("%d requests for down osds, need new map\n", needmap);
 		ceph_monc_request_next_osdmap(&osdc->client->monc);
 	}
-	reset_changed_osds(osdc);
 }
 
Signature · 7 lines
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
Quoted 19 lines · Sage Weil
Quoted 18 lines · Alex Elder

An osd client has a red-black tree describing its osds, and occasionally we would get crashes due to one of these trees tree becoming corrupt somehow.

The problem turned out to be that reset_changed_osds() was being called without protection of the osd client request mutex. That function would call __reset_osd() for any osd that had changed, and __reset_osd() would call __remove_osd() for any osd with no outstanding requests, and finally __remove_osd() would remove the corresponding entry from the red-black tree. Thus, the tree was getting modified without having any lock protection, and was vulnerable to problems due to concurrent updates.

This appears to be the only osd tree updating path that has this problem. It can be fairly easily fixed by moving the call up a few lines, to just before the request mutex gets dropped in kick_requests().

Quoted 5 lines
Quoted 3 lines

This resolves:

    http://tracker.ceph.com/issues/5043

Before you commit this, can you see if this applies cleanly to the stable trees? If so, let's put the stable tag in the changelog and see if the automagic stable backport stuff works!

sage

Quoted 41 lines
Quoted 35 lines

---
 net/ceph/osd_client.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/osd_client.c +1 -1
index d5953b8..3a246a6 100644
@@ -1675,13 +1675,13 @@ static void kick_requests(struct ceph_osd_client
 *osdc, int force_resend)
 		__register_request(osdc, req);
 		__unregister_linger_request(osdc, req);
 	}
+	reset_changed_osds(osdc);
 	mutex_unlock(&osdc->request_mutex);
 
 	if (needmap) {
 		dout("%d requests for down osds, need new map\n", needmap);
 		ceph_monc_request_next_osdmap(&osdc->client->monc);
 	}
-	reset_changed_osds(osdc);
 }
 
Signature · 7 lines
-- 
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html