Fix for corrupted ceph cluster
PATCH 1/1 Fix for corrupted ceph cluster
Hi,
I wrote a small patch that ignores object_trim requests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.
As far as I understood, deleting something that does not exist should not cause an assert. It is wired, but should not cause abort.
This is regarding bug http://tracker.ceph.com/issues/6101
Any help is highly appreciated.
kind regards
Daniel
--- src/osd/ReplicatedPG.cc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
src/osd/ReplicatedPG.cc +0 -0
@@ -1491,7 +1491,7 @@ ReplicatedPG::RepGather
*ReplicatedPG::trim_object(const hobject_t &coid)
int r = find_object_context(coid, &obc, false, NULL);
if (r == -ENOENT || coid.snap != obc->obs.oi.soid.snap) {
derr << __func__ << "could not find coid " << coid << dendl;- assert(0);
+ return NULL;
}
assert(r == 0);
assert(obc->registered);
@@ -7866,7 +7866,10 @@ boost::statechart::result ReplicatedPG::TrimmingObjects::react(const SnapTrim&)
dout(10) << "TrimmingObjects react trimming " << pos << dendl; RepGather *repop = pg->trim_object(pos);
- assert(repop);
+ if (!repop) {
+ derr << "TrimmingObjects failed " << pos << dendl;
+ return discard_event();
+ }
repop->queue_snap_trimmer = true; eversion_t old_last_update = pg->pg_log.get_head();
Signature · 1 line
1.8.5.3
Quoted 4 lines · Daniel Poelzleithner
I wrote a small patch that ignores
object_trimrequests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.
[...]
Quoted 1 line
This is regarding bug http://tracker.ceph.com/issues/6101
The patch now ran for 2 weeks and the 4th node is working again. I think this patch is safe to apply, but not fixing the underlying problem. Some state in ceph causes the delete event to be triggered every some seconds and causes a log entry to be generated.
Do you need more informations to find the cause ? This definitely is some wired internal state and is no race condition.
kind regards
Daniel
Quoted 9 lines · Daniel Poelzleithner
Quoted 4 lines · Daniel Poelzleithner
I wrote a small patch that ignores
object_trimrequests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.[...]
Quoted 1 line
This is regarding bug http://tracker.ceph.com/issues/6101
Quoted 7 lines
The patch now ran for 2 weeks and the 4th node is working again. I think this patch is safe to apply, but not fixing the underlying problem. Some state in ceph causes the delete event to be triggered every some seconds and causes a log entry to be generated.
Do you need more informations to find the cause ? This definitely is some wired internal state and is no race condition.
Can you try, instead of the discard_event, to do
post_event(SnapTrim());
return transit< WaitingOnReplicas >();and see if that lets it move past the bad entry?
Thanks-
sage