Fix: deadlock between UST registry lock and consumer lock
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 2/2 Fix: deadlock between UST registry lock and consumer lock
Reorganize locking of ust registry and consumer socket communication.
commit ce34fcd0 "Fix: per-uid flush and ust registry locking" attempted to fix locking related to the UST registry, but doing so introduced a deadlock. The actual solution is to reverse the order in which the UST registry and the consumer lock nest: the UST registry will now to responsible for serializing the registry content, and the consumer lock will only protect communication with the consumer, as it should. This deals with a TODO in the code.
The reason why this was not done from the beginning is that there was originally an intent to make sure the ust registry lock is not held for a long time, thus not while communicating with the consumer daemon. However, when live has been implemented, it required communication with the consumer daemon while the ust registry is held anyway. Therefore, there is not much point anymore in trying to make sure this lock is not held across the communication with consumerd in push_metadata. This allows us to greatly simplify locking of the UST registry.
--- src/bin/lttng-sessiond/consumer.c | 9 +++- src/bin/lttng-sessiond/ust-app.c | 84 +++++++++++++++-------------------- src/bin/lttng-sessiond/ust-consumer.c | 7 +-- src/bin/lttng-sessiond/ust-registry.h | 16 ++++--- 4 files changed, 59 insertions(+), 57 deletions(-)
src/bin/lttng-sessiond/consumer.c +7 -2
@@ -1158,7 +1158,8 @@ end: }/* -* Send a close metdata command to consumer using the given channel key. +* Send a close metadata command to consumer using the given channel key. +* Called with registry lock held. * * Return 0 on success else a negative value. */
@@ -1224,7 +1225,8 @@ end: }/* -* Send metadata string to consumer. Socket lock MUST be acquired. +* Send metadata string to consumer. +* RCU read-side lock must be held to guarantee existence of socket. * * Return 0 on success else a negative value. */
@@ -1239,6 +1241,8 @@ int consumer_push_metadata(struct consumer_socket *socket,DBG2 ("Consumer push metadata to consumer socket %d" , *socket->fd_ptr); +pthread_mutex_lock (socket->lock); +memset (&msg,0 ,sizeof (msg)); msg.cmd_type =LTTNG_CONSUMER_PUSH_METADATA ; msg.u.push_metadata.key = metadata_key;
@@ -1266,6 +1270,7 @@ int consumer_push_metadata(struct consumer_socket *socket, } end: +pthread_mutex_unlock (socket->lock);health_code_update ();return ret; }
src/bin/lttng-sessiond/ust-app.c +37 -47
169 lines not on this page · raw message
src/bin/lttng-sessiond/ust-consumer.c +4 -3
@@ -448,12 +448,12 @@ int ust_consumer_metadata_request(struct consumer_socket *socket)assert (socket);rcu_read_lock (); -pthread_mutex_lock (socket->lock); -health_code_update ();/* Wait for a metadata request */ +pthread_mutex_lock (socket->lock); ret =consumer_socket_recv (socket, &request,sizeof (request)); +pthread_mutex_unlock (socket->lock);if (ret <0 ) { goto end; }
@@ -488,7 +488,9 @@ int ust_consumer_metadata_request(struct consumer_socket *socket) }assert (ust_reg); +pthread_mutex_lock (&ust_reg->lock); ret_push =ust_app_push_metadata (ust_reg, socket,1 ); +pthread_mutex_unlock (&ust_reg->lock);if (ret_push <0 ) {ERR ("Pushing metadata" ); ret =-1 ;
@@ -498,7 +500,6 @@ int ust_consumer_metadata_request(struct consumer_socket *socket) ret =0 ; end: -pthread_mutex_unlock (socket->lock);rcu_read_unlock ();return ret; }
src/bin/lttng-sessiond/ust-registry.h +11 -5
@@ -33,8 +33,12 @@ struct ust_app;struct ust_registry_session {/* -* With multiple writers and readers, use this lock to access the registry. -* Can nest within the ust app session lock. +* With multiple writers and readers, use this lock to access +* the registry. Can nest within the ust app session lock. +* Also acts as a registry serialization lock. Used by registry +* readers to serialize the registry information sent from the +* sessiond to the consumerd. +* The consumer socket lock nests within this lock. */ pthread_mutex_t lock;/* Next channel ID available for a newly registered channel. */
@@ -63,11 +67,13 @@ struct ust_registry_session {/* Length of bytes sent to the consumer. */ size_t metadata_len_sent;/* -* Hash table containing channels sent by the UST tracer. MUST be accessed -* with a RCU read side lock acquired. +* Hash table containing channels sent by the UST tracer. MUST +* be accessed with a RCU read side lock acquired. */ struct lttng_ht *channels; -/* Unique key to identify the metadata on the consumer side. */ +/* +* Unique key to identify the metadata on the consumer side. +*/ uint64_t metadata_key;/* * Indicates if the metadata is closed on the consumer side. This is to
Signature · 1 line
2.1.1
Re: [PATCH lttng-tools 1/2] Fix: uninitialized return value
In reply to a message not in the archiveBoth patches were merged all the way back to 2.4.
Thanks!
Jérémie
Quoted 37 lines · Mathieu Desnoyers
Signed-off-by Mathieu Desnoyers <[email protected]>--- src/bin/lttng-sessiond/ust-app.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)src/bin/lttng-sessiond/ust-app.c +2
-3@@ -4132,7 +4132,7 @@ staticint ust_app_flush_session (struct ltt_ust_session *usess) { -int ret; +int ret =0 ;DBG ("Flushing session buffers for all ust apps" );@@ -4173,7 +4173,6 @@ int ust_app_flush_session(struct ltt_ust_session *usess)/* Push metadata. */ (void )push_metadata (ust_session_reg, usess->consumer); } - ret =0 ;break ; }case LTTNG_BUFFER_PER_PID :@@ -4192,6 +4191,7 @@ int ust_app_flush_session(struct ltt_ust_session *usess)break ; }default : + ret =-1 ;assert (0 );break ; } --2.1.1
Signature · 8 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev