Skip to contentSkip to conversation

Fix: deadlock between UST registry lock and consumer lock

PATCH New Mathieu Desnoyers 2 participants 2 messages Jan 23, 2015 – Jan 26, 2015 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 2/2 Fix: deadlock between UST registry lock and consumer lock

Reorganize locking of ust registry and consumer socket communication.

commit ce34fcd0 "Fix: per-uid flush and ust registry locking" attempted to fix locking related to the UST registry, but doing so introduced a deadlock. The actual solution is to reverse the order in which the UST registry and the consumer lock nest: the UST registry will now to responsible for serializing the registry content, and the consumer lock will only protect communication with the consumer, as it should. This deals with a TODO in the code.

The reason why this was not done from the beginning is that there was originally an intent to make sure the ust registry lock is not held for a long time, thus not while communicating with the consumer daemon. However, when live has been implemented, it required communication with the consumer daemon while the ust registry is held anyway. Therefore, there is not much point anymore in trying to make sure this lock is not held across the communication with consumerd in push_metadata. This allows us to greatly simplify locking of the UST registry.

---
 src/bin/lttng-sessiond/consumer.c     |  9 +++-
 src/bin/lttng-sessiond/ust-app.c      | 84 +++++++++++++++--------------------
 src/bin/lttng-sessiond/ust-consumer.c |  7 +--
 src/bin/lttng-sessiond/ust-registry.h | 16 ++++---
 4 files changed, 59 insertions(+), 57 deletions(-)
src/bin/lttng-sessiond/consumer.c +7 -2
index 9b8a0a7..b7394d9 100644
@@ -1158,7 +1158,8 @@ end:
 }
 
 /*
- * Send a close metdata command to consumer using the given channel key.
+ * Send a close metadata command to consumer using the given channel key.
+ * Called with registry lock held.
  *
  * Return 0 on success else a negative value.
  */
@@ -1224,7 +1225,8 @@ end:
 }
 
 /*
- * Send metadata string to consumer. Socket lock MUST be acquired.
+ * Send metadata string to consumer.
+ * RCU read-side lock must be held to guarantee existence of socket.
  *
  * Return 0 on success else a negative value.
  */
@@ -1239,6 +1241,8 @@ int consumer_push_metadata(struct consumer_socket *socket,
 
 	DBG2("Consumer push metadata to consumer socket %d", *socket->fd_ptr);
 
+	pthread_mutex_lock(socket->lock);
+
 	memset(&msg, 0, sizeof(msg));
 	msg.cmd_type = LTTNG_CONSUMER_PUSH_METADATA;
 	msg.u.push_metadata.key = metadata_key;
@@ -1266,6 +1270,7 @@ int consumer_push_metadata(struct consumer_socket *socket,
 	}
 
 end:
+	pthread_mutex_unlock(socket->lock);
 	health_code_update();
 	return ret;
 }
src/bin/lttng-sessiond/ust-app.c +37 -47

169 lines not on this page · raw message

src/bin/lttng-sessiond/ust-consumer.c +4 -3
index f120144..611b54d 100644
@@ -448,12 +448,12 @@ int ust_consumer_metadata_request(struct consumer_socket *socket)
 	assert(socket);
 
 	rcu_read_lock();
-	pthread_mutex_lock(socket->lock);
-
 	health_code_update();
 
 	/* Wait for a metadata request */
+	pthread_mutex_lock(socket->lock);
 	ret = consumer_socket_recv(socket, &request, sizeof(request));
+	pthread_mutex_unlock(socket->lock);
 	if (ret < 0) {
 		goto end;
 	}
@@ -488,7 +488,9 @@ int ust_consumer_metadata_request(struct consumer_socket *socket)
 	}
 	assert(ust_reg);
 
+	pthread_mutex_lock(&ust_reg->lock);
 	ret_push = ust_app_push_metadata(ust_reg, socket, 1);
+	pthread_mutex_unlock(&ust_reg->lock);
 	if (ret_push < 0) {
 		ERR("Pushing metadata");
 		ret = -1;
@@ -498,7 +500,6 @@ int ust_consumer_metadata_request(struct consumer_socket *socket)
 	ret = 0;
 
 end:
-	pthread_mutex_unlock(socket->lock);
 	rcu_read_unlock();
 	return ret;
 }
src/bin/lttng-sessiond/ust-registry.h +11 -5
index f195b74..0cba8a3 100644
@@ -33,8 +33,12 @@ struct ust_app;
 
 struct ust_registry_session {
 	/*
-	 * With multiple writers and readers, use this lock to access the registry.
-	 * Can nest within the ust app session lock.
+	 * With multiple writers and readers, use this lock to access
+	 * the registry. Can nest within the ust app session lock.
+	 * Also acts as a registry serialization lock. Used by registry
+	 * readers to serialize the registry information sent from the
+	 * sessiond to the consumerd.
+	 * The consumer socket lock nests within this lock.
 	 */
 	pthread_mutex_t lock;
 	/* Next channel ID available for a newly registered channel. */
@@ -63,11 +67,13 @@ struct ust_registry_session {
 	/* Length of bytes sent to the consumer. */
 	size_t metadata_len_sent;
 	/*
-	 * Hash table containing channels sent by the UST tracer. MUST be accessed
-	 * with a RCU read side lock acquired.
+	 * Hash table containing channels sent by the UST tracer. MUST
+	 * be accessed with a RCU read side lock acquired.
 	 */
 	struct lttng_ht *channels;
-	/* Unique key to identify the metadata on the consumer side. */
+	/*
+	 * Unique key to identify the metadata on the consumer side.
+	 */
 	uint64_t metadata_key;
 	/*
 	 * Indicates if the metadata is closed on the consumer side. This is to
Signature · 1 line
2.1.1
#2 lore raw

Re: [PATCH lttng-tools 1/2] Fix: uninitialized return value

In reply to a message not in the archive

Both patches were merged all the way back to 2.4.

Thanks!
Jérémie

Quoted 37 lines · Mathieu Desnoyers
---
 src/bin/lttng-sessiond/ust-app.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
src/bin/lttng-sessiond/ust-app.c +2 -3
index 41a6d39..bdee8e6 100644
@@ -4132,7 +4132,7 @@ static
 int ust_app_flush_session(struct ltt_ust_session *usess)
 
 {
-       int ret;
+       int ret = 0;
 
        DBG("Flushing session buffers for all ust apps");
 
@@ -4173,7 +4173,6 @@ int ust_app_flush_session(struct ltt_ust_session *usess)
                        /* Push metadata. */
                        (void) push_metadata(ust_session_reg, usess->consumer);
                }
-               ret = 0;
                break;
        }
        case LTTNG_BUFFER_PER_PID:
@@ -4192,6 +4191,7 @@ int ust_app_flush_session(struct ltt_ust_session *usess)
                break;
        }
        default:
+               ret = -1;
                assert(0);
                break;
        }
--

2.1.1

Signature · 8 lines
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev