Skip to contentSkip to conversation

ceph: re-enable rados_aio_set_callback

PATCH In review Christian Brunner 3 participants 4 messages May 5, 2010 – May 6, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: re-enable rados_aio_set_callback

While implementing the AIO parts of the qemu-kvm driver I realized that for every call to rados_aio_write the callback is called twice. The first time, when the write is complete (committed to journal?) and the second time, when the write is safe (committed to the object store?).

Usually I would think that a complete call is sufficient in most use cases. However when I call rados_aio_release on the first callback, I get an error message (on the second callback):

common/Mutex.h: In function 'void Mutex::Lock(bool)':
common/Mutex.h:97: FAILED assert(r == 0)
1: (RadosClient::C_aio_Safe::finish(int)+0x7f) [0x3ea0262e0f]
2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x5cb) [0x3ea027fc2b]
3: (RadosClient::_dispatch(Message*)+0x7c) [0x3ea0250bac]
4: (RadosClient::ms_dispatch(Message*)+0x33) [0x3ea0250ee3]
5: (SimpleMessenger::dispatch_entry()+0x6a2) [0x3ea0268aa2]
6: (SimpleMessenger::DispatchThread::entry()+0x51) [0x3ea025db41]
7: (Thread::_entry_func(void*)+0xc) [0x3ea027645c]
8: /lib64/libpthread.so.0() [0x3293406a3a]
9: (clone()+0x6d) [0x3292cde67d]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.
common/Mutex.h: In function 'void Mutex::Lock(bool)':
common/Mutex.h:97: FAILED assert(r == 0)
1: (RadosClient::C_aio_Safe::finish(int)+0x7f) [0x3ea0262e0f]
2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x5cb) [0x3ea027fc2b]
3: (RadosClient::_dispatch(Message*)+0x7c) [0x3ea0250bac]
4: (RadosClient::ms_dispatch(Message*)+0x33) [0x3ea0250ee3]
5: (SimpleMessenger::dispatch_entry()+0x6a2) [0x3ea0268aa2]
6: (SimpleMessenger::DispatchThread::entry()+0x51) [0x3ea025db41]
7: (Thread::_entry_func(void*)+0xc) [0x3ea027645c]
8: /lib64/libpthread.so.0() [0x3293406a3a]
9: (clone()+0x6d) [0x3292cde67d]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion*' Aborted (core dumped)

To work around this, I’d suggest to re-enable rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.

Regards,
Christian

---
 src/librados.cc |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
src/librados.cc +5 -0
index 76d4a53..c3a34d7 100644
@@ -1811,6 +1811,11 @@ extern "C" int rados_aio_create_completion(rados_callback_t cb, void *cba, rados
   return 0;
 }
 
+extern "C" int rados_aio_set_callback(rados_completion_t c, rados_callback_t cb, void *cba)
+{
+  return ((RadosClient::AioCompletion *)c)->set_callback(cb, cba);
+}
+
 extern "C" int rados_aio_wait_for_complete(rados_completion_t c)
 {
   return ((RadosClient::AioCompletion *)c)->wait_for_complete();
Signature · 11 lines
1.6.6.1


-- 
Christian Brunner                              MUC.DE e.V.
                                               Joseph-Dollinger-Bogen 14
                                               D-80807 Muenchen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#1 lore raw

On Wed, May 5, 2010 at 12:50 PM, Christian Brunner <[email protected]> wrote:
...

Quoted 4 lines

To work around this, I’d suggest to re-enable rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.

Although this would solve that specific problem I'd rather leave this function out, as it is racy and could be misused. I think having a solution in which we either set different callbacks for the complete and for the safe replies, or just set a flag that specifies when to use this callback would be better.

Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 8 lines · Yehuda Sadeh Weinraub

On Wed, May 5, 2010 at 12:50 PM, Christian Brunner <[email protected]> wrote:
...

Quoted 4 lines

To work around this, I?d suggest to re-enable rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.

Quoted 5 lines

Although this would solve that specific problem I'd rather leave this function out, as it is racy and could be misused. I think having a solution in which we either set different callbacks for the complete and for the safe replies, or just set a flag that specifies when to use this callback would be better.

I think being able to specify a callback for each would be the most flexible. Also, to be clear, setting adjusting the completions callback is safe if it happens prior to the completion being used (before calling aio_read or whatever). After that I don't think we should make any promises.

The patch below changes the API a bit. Since I think kvm-rbd is the only aio callback user so far, I'm going to be lazy and skip any compat cruft to maintain compatibility. Does this look okay?

sage

src/include/librados.h +2 -2
index 5f20c5a..5418c87 100644
@@ -79,8 +79,8 @@ int rados_exec(rados_pool_t pool, const char *oid, const char *cls, const char *
 typedef void *rados_completion_t;
 typedef void (*rados_callback_t)(rados_completion_t cb, void *arg);
 
-int rados_aio_create_completion(rados_callback_t, void *arg, rados_completion_t *pc);
-int rados_aio_set_callback(rados_completion_t c, rados_callback_t, void *arg);
+int rados_aio_create_completion(void *cb_arg, rados_callback_t cb_ack, rados_callback_t cb_safe,
+				rados_completion_t *pc);
 int rados_aio_wait_for_complete(rados_completion_t c);
 int rados_aio_wait_for_safe(rados_completion_t c);
 int rados_aio_is_complete(rados_completion_t c);
src/include/librados.hpp +3 -2
index 0e70e30..b9ec3c2 100644
@@ -100,7 +100,8 @@ public:
   struct AioCompletion {
     void *pc;
     AioCompletion(void *_pc) : pc(_pc) {}
-    int set_callback(callback_t cb, void *cba);
+    int set_ack_callback(void *cba, callback_t cb);
+    int set_safe_callback(void *cba, callback_t cb);
     int wait_for_complete();
     int wait_for_safe();
     bool is_complete();
@@ -114,7 +115,7 @@ public:
   int aio_write(pool_t pool, const std::string& oid, off_t off, const bufferlist& bl, size_t len,
 		AioCompletion *c);
   AioCompletion *aio_create_completion();
-  AioCompletion *aio_create_completion(callback_t cb, void *cba);
+  AioCompletion *aio_create_completion(void *cb_arg, callback_t cb_ack, callback_t cb_safe);
 };
 
 }
src/librados.cc +31 -16

100 lines not on this page · raw message

#3 lore raw
Quoted 10 lines · Sage Weil

I think being able to specify a callback for each would be the most flexible. Also, to be clear, setting adjusting the completions callback is safe if it happens prior to the completion being used (before calling aio_read or whatever). After that I don't think we should make any promises.

The patch below changes the API a bit. Since I think kvm-rbd is the only aio callback user so far, I'm going to be lazy and skip any compat cruft to maintain compatibility. Does this look okay?

The patch is looking good and it makes using the callback even simpler. I will send a patch for qemu in a few minutes.

Christian