ceph: re-enable rados_aio_set_callback
PATCH 1/1 ceph: re-enable rados_aio_set_callback
While implementing the AIO parts of the qemu-kvm driver I realized that for every call to rados_aio_write the callback is called twice. The first time, when the write is complete (committed to journal?) and the second time, when the write is safe (committed to the object store?).
Usually I would think that a complete call is sufficient in most use cases. However when I call rados_aio_release on the first callback, I get an error message (on the second callback):
common/Mutex.h: In function 'void Mutex::Lock(bool)':common/Mutex.h:97: FAILED assert(r == 0)
1: (RadosClient::C_aio_Safe::finish(int)+0x7f) [0x3ea0262e0f]
2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x5cb) [0x3ea027fc2b]
3: (RadosClient::_dispatch(Message*)+0x7c) [0x3ea0250bac]
4: (RadosClient::ms_dispatch(Message*)+0x33) [0x3ea0250ee3]
5: (SimpleMessenger::dispatch_entry()+0x6a2) [0x3ea0268aa2]
6: (SimpleMessenger::DispatchThread::entry()+0x51) [0x3ea025db41]
7: (Thread::_entry_func(void*)+0xc) [0x3ea027645c]
8: /lib64/libpthread.so.0() [0x3293406a3a]
9: (clone()+0x6d) [0x3292cde67d]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.common/Mutex.h: In function 'void Mutex::Lock(bool)':common/Mutex.h:97: FAILED assert(r == 0)
1: (RadosClient::C_aio_Safe::finish(int)+0x7f) [0x3ea0262e0f]
2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x5cb) [0x3ea027fc2b]
3: (RadosClient::_dispatch(Message*)+0x7c) [0x3ea0250bac]
4: (RadosClient::ms_dispatch(Message*)+0x33) [0x3ea0250ee3]
5: (SimpleMessenger::dispatch_entry()+0x6a2) [0x3ea0268aa2]
6: (SimpleMessenger::DispatchThread::entry()+0x51) [0x3ea025db41]
7: (Thread::_entry_func(void*)+0xc) [0x3ea027645c]
8: /lib64/libpthread.so.0() [0x3293406a3a]
9: (clone()+0x6d) [0x3292cde67d]
NOTE: a copy of the executable, or objdump -rdS <executable> is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion*' Aborted (core dumped)
To work around this, I’d suggest to re-enable rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.
Regards,
Christian
--- src/librados.cc | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
src/librados.cc +5 -0
@@ -1811,6 +1811,11 @@ extern "C" int rados_aio_create_completion(rados_callback_t cb, void *cba, rados return 0; } +extern "C" int rados_aio_set_callback(rados_completion_t c, rados_callback_t cb, void *cba) +{ + return ((RadosClient::AioCompletion *)c)->set_callback(cb, cba); +} + extern "C" int rados_aio_wait_for_complete(rados_completion_t c) { return ((RadosClient::AioCompletion *)c)->wait_for_complete();
Signature · 11 lines
1.6.6.1
--
Christian Brunner MUC.DE e.V.
Joseph-Dollinger-Bogen 14
D-80807 Muenchen
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlOn Wed, May 5, 2010 at 12:50 PM, Christian Brunner <[email protected]> wrote:
...
Quoted 4 lines
To work around this, I’d suggest to re-enable
rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.
Although this would solve that specific problem I'd rather leave this function out, as it is racy and could be misused. I think having a solution in which we either set different callbacks for the complete and for the safe replies, or just set a flag that specifies when to use this callback would be better.
Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Yehuda Sadeh Weinraub
On Wed, May 5, 2010 at 12:50 PM, Christian Brunner <[email protected]> wrote:
...Quoted 4 lines
To work around this, I?d suggest to re-enable
rados_aio_set_callback. This way it would possible to clear (cb=NULL) or change the callback handler.
Quoted 5 lines
Although this would solve that specific problem I'd rather leave this function out, as it is racy and could be misused. I think having a solution in which we either set different callbacks for the complete and for the safe replies, or just set a flag that specifies when to use this callback would be better.
I think being able to specify a callback for each would be the most flexible. Also, to be clear, setting adjusting the completions callback is safe if it happens prior to the completion being used (before calling aio_read or whatever). After that I don't think we should make any promises.
The patch below changes the API a bit. Since I think kvm-rbd is the only aio callback user so far, I'm going to be lazy and skip any compat cruft to maintain compatibility. Does this look okay?
sage
src/include/librados.h +2 -2
@@ -79,8 +79,8 @@ int rados_exec(rados_pool_t pool, const char *oid, const char *cls, const char *typedef void *rados_completion_t ;typedef void (*rados_callback_t )(rados_completion_t cb,void *arg); -int rados_aio_create_completion (rados_callback_t ,void *arg,rados_completion_t *pc); -int rados_aio_set_callback (rados_completion_t c,rados_callback_t ,void *arg); +int rados_aio_create_completion (void *cb_arg,rados_callback_t cb_ack,rados_callback_t cb_safe, +rados_completion_t *pc);int rados_aio_wait_for_complete (rados_completion_t c);int rados_aio_wait_for_safe (rados_completion_t c);int rados_aio_is_complete (rados_completion_t c);
src/include/librados.hpp +3 -2
@@ -100,7 +100,8 @@ public: struct AioCompletion { void *pc; AioCompletion(void *_pc) : pc(_pc) {} - int set_callback(callback_t cb, void *cba); + int set_ack_callback(void *cba, callback_t cb); + int set_safe_callback(void *cba, callback_t cb); int wait_for_complete(); int wait_for_safe(); bool is_complete();
@@ -114,7 +115,7 @@ public: int aio_write(pool_t pool, const std::string& oid, off_t off, const bufferlist& bl, size_t len, AioCompletion *c); AioCompletion *aio_create_completion(); - AioCompletion *aio_create_completion(callback_t cb, void *cba); + AioCompletion *aio_create_completion(void *cb_arg, callback_t cb_ack, callback_t cb_safe); }; }
src/librados.cc +31 -16
100 lines not on this page · raw message
Quoted 10 lines · Sage Weil
I think being able to specify a callback for each would be the most flexible. Also, to be clear, setting adjusting the completions callback is safe if it happens prior to the completion being used (before calling
aio_reador whatever). After that I don't think we should make any promises.The patch below changes the API a bit. Since I think kvm-rbd is the only aio callback user so far, I'm going to be lazy and skip any compat cruft to maintain compatibility. Does this look okay?
The patch is looking good and it makes using the callback even simpler. I will send a patch for qemu in a few minutes.
Christian