Skip to contentSkip to conversation

Fix for corrupted ceph cluster

PATCH In review Daniel Poelzleithner 2 participants 3 messages Feb 11, 2014 – Mar 5, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix for corrupted ceph cluster

Hi,

I wrote a small patch that ignores object_trim requests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.

As far as I understood, deleting something that does not exist should not cause an assert. It is wired, but should not cause abort.

This is regarding bug http://tracker.ceph.com/issues/6101

Any help is highly appreciated.

kind regards
Daniel

---
 src/osd/ReplicatedPG.cc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
src/osd/ReplicatedPG.cc +0 -0
index 90d3e1d..d7e0b62 100644
@@ -1491,7 +1491,7 @@ ReplicatedPG::RepGather

*ReplicatedPG::trim_object(const hobject_t &coid)
int r = find_object_context(coid, &obc, false, NULL);

   if (r == -ENOENT || coid.snap != obc->obs.oi.soid.snap) {
     derr << __func__ << "could not find coid " << coid << dendl;

- assert(0);
+ return NULL;
}
assert(r == 0);
assert(obc->registered);
@@ -7866,7 +7866,10 @@ boost::statechart::result ReplicatedPG::TrimmingObjects::react(const SnapTrim&)

dout(10) << "TrimmingObjects react trimming " << pos << dendl; RepGather *repop = pg->trim_object(pos);
- assert(repop);
+ if (!repop) {
+ derr << "TrimmingObjects failed " << pos << dendl;
+ return discard_event();
+ }

repop->queue_snap_trimmer = true; eversion_t old_last_update = pg->pg_log.get_head();

Signature · 1 line
1.8.5.3
#1 lore raw
Quoted 4 lines · Daniel Poelzleithner

I wrote a small patch that ignores object_trim requests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.

[...]

Quoted 1 line

This is regarding bug http://tracker.ceph.com/issues/6101

The patch now ran for 2 weeks and the 4th node is working again. I think this patch is safe to apply, but not fixing the underlying problem. Some state in ceph causes the delete event to be triggered every some seconds and causes a log entry to be generated.

Do you need more informations to find the cause ? This definitely is some wired internal state and is no race condition.

kind regards
Daniel

#2 lore raw
Quoted 9 lines · Daniel Poelzleithner
Quoted 4 lines · Daniel Poelzleithner

I wrote a small patch that ignores object_trim requests when he does not find the context of this request.
We have a node that fails to start permanently and there is no way to get all nodes back up.

[...]

Quoted 1 line

This is regarding bug http://tracker.ceph.com/issues/6101

Quoted 7 lines

The patch now ran for 2 weeks and the 4th node is working again. I think this patch is safe to apply, but not fixing the underlying problem. Some state in ceph causes the delete event to be triggered every some seconds and causes a log entry to be generated.

Do you need more informations to find the cause ? This definitely is some wired internal state and is no race condition.

Can you try, instead of the discard_event, to do

    post_event(SnapTrim());
    return transit< WaitingOnReplicas >();

and see if that lets it move past the bad entry?

Thanks-
sage