Skip to contentSkip to conversation

virtiofsd: Fix lo_destroy crash in g_hash_table_foreach_remove

PATCH In review Dr. David Alan Gilbert (git) 3 participants 8 messages Aug 7, 2019 – Aug 9, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtiofsd: Fix lo_destroy crash in g_hash_table_foreach_remove

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Avoid the g_hash_table_foreach_remove and use a dummy iterator to find one element of the table at a time.

Fixes virtiofsd: fix lo_destroy() resource leaks
---
 contrib/virtiofsd/passthrough_ll.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
contrib/virtiofsd/passthrough_ll.c +13 -14
index cc9c175047..321bbb20be 100644
@@ -2445,18 +2445,6 @@ static void lo_removemapping(fuse_req_t req, struct fuse_session *se,
 	fuse_reply_err(req, ret);
 }
 
-static int destroy_inode_cb(gpointer key, gpointer value, gpointer user_data)
-{
-        struct lo_inode *inode = value;
-        struct lo_data *lo = user_data;
-
-        /* inode->nlookup is normally protected by lo->mutex but see the
-         * comment in lo_destroy().
-         */
-        unref_inode(lo, inode, inode->nlookup);
-        return TRUE;
-}
-
 static void lo_destroy(void *userdata, struct fuse_session *se)
 {
 	struct lo_data *lo = (struct lo_data*) userdata;
@@ -2474,10 +2462,21 @@ static void lo_destroy(void *userdata, struct fuse_session *se)
         /* Normally lo->mutex must be taken when traversing lo->inodes but
          * lo_destroy() is a serialized request so no races are possible here.
          *
-         * In addition, we cannot acquire lo->mutex since destroy_inode_cb() takes it
+         * In addition, we cannot acquire lo->mutex since unref_inode() takes it
          * too and this would result in a recursive lock.
          */
-        g_hash_table_foreach_remove(lo->inodes, destroy_inode_cb, lo);
+        while (true) {
+                GHashTableIter iter;
+                gpointer key, value;
+
+                g_hash_table_iter_init(&iter, lo->inodes);
+                if (!g_hash_table_iter_next(&iter, &key, &value)) {
+                        break;
+                }
+
+                struct lo_inode *inode = value;
+                unref_inode(lo, inode, inode->nlookup);
+        }
 }
 
 static struct fuse_lowlevel_ops lo_oper = {
Signature · 1 line
2.21.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Avoid the g_hash_table_foreach_remove and use a dummy iterator to find one element of the table at a time.

Quoted 6 lines
Fixes virtiofsd: fix lo_destroy() resource leaks
---
 contrib/virtiofsd/passthrough_ll.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
#2 lore raw
In reply to Dr. David Alan Gilbert (git)
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Hi David,

Got two questions.

- Shouldn't we take a lo->mutex to make sure parallel hash table modifications are not happening.

- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

Thanks
Vivek

Quoted 61 lines

Avoid the g_hash_table_foreach_remove and use a dummy iterator to find one element of the table at a time.

Fixes virtiofsd: fix lo_destroy() resource leaks
---
 contrib/virtiofsd/passthrough_ll.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
contrib/virtiofsd/passthrough_ll.c +13 -14
index cc9c175047..321bbb20be 100644
@@ -2445,18 +2445,6 @@ static void lo_removemapping(fuse_req_t req, struct fuse_session *se,
 	fuse_reply_err(req, ret);
 }
 
-static int destroy_inode_cb(gpointer key, gpointer value, gpointer user_data)
-{
-        struct lo_inode *inode = value;
-        struct lo_data *lo = user_data;
-
-        /* inode->nlookup is normally protected by lo->mutex but see the
-         * comment in lo_destroy().
-         */
-        unref_inode(lo, inode, inode->nlookup);
-        return TRUE;
-}
-
 static void lo_destroy(void *userdata, struct fuse_session *se)
 {
 	struct lo_data *lo = (struct lo_data*) userdata;
@@ -2474,10 +2462,21 @@ static void lo_destroy(void *userdata, struct fuse_session *se)
         /* Normally lo->mutex must be taken when traversing lo->inodes but
          * lo_destroy() is a serialized request so no races are possible here.
          *
-         * In addition, we cannot acquire lo->mutex since destroy_inode_cb() takes it
+         * In addition, we cannot acquire lo->mutex since unref_inode() takes it
          * too and this would result in a recursive lock.
          */
-        g_hash_table_foreach_remove(lo->inodes, destroy_inode_cb, lo);
+        while (true) {
+                GHashTableIter iter;
+                gpointer key, value;
+
+                g_hash_table_iter_init(&iter, lo->inodes);
+                if (!g_hash_table_iter_next(&iter, &key, &value)) {
+                        break;
+                }
+
+                struct lo_inode *inode = value;
+                unref_inode(lo, inode, inode->nlookup);
+        }
 }
 
 static struct fuse_lowlevel_ops lo_oper = {
Signature · 1 line
2.21.0
#3 lore raw
Quoted 7 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Quoted 6 lines

Hi David,

Got two questions.

- Shouldn't we take a lo->mutex to make sure parallel hash table modifications are not happening.

See Stefan's big comment (which I just changed the function name in) which explains that we can't take lo->mutex

Quoted 2 lines

- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

No, because in some situations lo_destroy does not happen at the end; e.g. it happens during a umount and we still have the connection to be able to remount it.

Dave

Quoted 60 lines

Thanks
Vivek

Quoted 57 lines

Avoid the g_hash_table_foreach_remove and use a dummy iterator to find one element of the table at a time.

Fixes virtiofsd: fix lo_destroy() resource leaks
---
 contrib/virtiofsd/passthrough_ll.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
contrib/virtiofsd/passthrough_ll.c +13 -14
index cc9c175047..321bbb20be 100644
@@ -2445,18 +2445,6 @@ static void lo_removemapping(fuse_req_t req, struct fuse_session *se,
 	fuse_reply_err(req, ret);
 }
 
-static int destroy_inode_cb(gpointer key, gpointer value, gpointer user_data)
-{
-        struct lo_inode *inode = value;
-        struct lo_data *lo = user_data;
-
-        /* inode->nlookup is normally protected by lo->mutex but see the
-         * comment in lo_destroy().
-         */
-        unref_inode(lo, inode, inode->nlookup);
-        return TRUE;
-}
-
 static void lo_destroy(void *userdata, struct fuse_session *se)
 {
 	struct lo_data *lo = (struct lo_data*) userdata;
@@ -2474,10 +2462,21 @@ static void lo_destroy(void *userdata, struct fuse_session *se)
         /* Normally lo->mutex must be taken when traversing lo->inodes but
          * lo_destroy() is a serialized request so no races are possible here.
          *
-         * In addition, we cannot acquire lo->mutex since destroy_inode_cb() takes it
+         * In addition, we cannot acquire lo->mutex since unref_inode() takes it
          * too and this would result in a recursive lock.
          */
-        g_hash_table_foreach_remove(lo->inodes, destroy_inode_cb, lo);
+        while (true) {
+                GHashTableIter iter;
+                gpointer key, value;
+
+                g_hash_table_iter_init(&iter, lo->inodes);
+                if (!g_hash_table_iter_next(&iter, &key, &value)) {
+                        break;
+                }
+
+                struct lo_inode *inode = value;
+                unref_inode(lo, inode, inode->nlookup);
+        }
 }
 
Quoted 4 lines
Quoted 4 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
 static struct fuse_lowlevel_ops lo_oper = {
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw
Quoted 10 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Hi David,

Quoted 7 lines
Quoted 4 lines

Got two questions.

- Shouldn't we take a lo->mutex to make sure parallel hash table modifications are not happening.

See Stefan's big comment (which I just changed the function name in) which explains that we can't take lo->mutex

That comment says that unref_inode() takes the lock and that's why we can't take it.

But that's easy to fix. We just have to come up with another helper which does not take lock and asssumes lock has already been taken.

/* This assumes lo->lock is already held */
__unref_inode()
{
}

Quoted 7 lines
Quoted 3 lines

- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

No, because in some situations lo_destroy does not happen at the end; e.g. it happens during a umount and we still have the connection to be able to remount it.

Ok, so we don't sever the connection completely. But we don't expect to process further requests till a new INIT has been done, isn't it?

IOW, once lo_destroy() is received, we cleanup any pending state from the mount and if any requests are received from client, we error them out.

And start processing requests normally when a new INIT has been received.

Atleast that was my understanding of the design.

Thanks
Vivek

Quoted 74 lines

Dave

Quoted 60 lines

Thanks
Vivek

Quoted 57 lines

Avoid the g_hash_table_foreach_remove and use a dummy iterator to find one element of the table at a time.

Fixes virtiofsd: fix lo_destroy() resource leaks
---
 contrib/virtiofsd/passthrough_ll.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)
contrib/virtiofsd/passthrough_ll.c +13 -14
index cc9c175047..321bbb20be 100644
@@ -2445,18 +2445,6 @@ static void lo_removemapping(fuse_req_t req, struct fuse_session *se,
 	fuse_reply_err(req, ret);
 }
 
-static int destroy_inode_cb(gpointer key, gpointer value, gpointer user_data)
-{
-        struct lo_inode *inode = value;
-        struct lo_data *lo = user_data;
-
-        /* inode->nlookup is normally protected by lo->mutex but see the
-         * comment in lo_destroy().
-         */
-        unref_inode(lo, inode, inode->nlookup);
-        return TRUE;
-}
-
 static void lo_destroy(void *userdata, struct fuse_session *se)
 {
 	struct lo_data *lo = (struct lo_data*) userdata;
@@ -2474,10 +2462,21 @@ static void lo_destroy(void *userdata, struct fuse_session *se)
         /* Normally lo->mutex must be taken when traversing lo->inodes but
          * lo_destroy() is a serialized request so no races are possible here.
          *
-         * In addition, we cannot acquire lo->mutex since destroy_inode_cb() takes it
+         * In addition, we cannot acquire lo->mutex since unref_inode() takes it
          * too and this would result in a recursive lock.
          */
-        g_hash_table_foreach_remove(lo->inodes, destroy_inode_cb, lo);
+        while (true) {
+                GHashTableIter iter;
+                gpointer key, value;
+
+                g_hash_table_iter_init(&iter, lo->inodes);
+                if (!g_hash_table_iter_next(&iter, &key, &value)) {
+                        break;
+                }
+
+                struct lo_inode *inode = value;
+                unref_inode(lo, inode, inode->nlookup);
+        }
 }
 
Quoted 4 lines
Quoted 4 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
 static struct fuse_lowlevel_ops lo_oper = {
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

#5 lore raw
Quoted 22 lines · Vivek Goyal
Quoted 10 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Hi David,

Quoted 7 lines
Quoted 4 lines

Got two questions.

- Shouldn't we take a lo->mutex to make sure parallel hash table modifications are not happening.

See Stefan's big comment (which I just changed the function name in) which explains that we can't take lo->mutex

That comment says that unref_inode() takes the lock and that's why we can't take it.

Quoted 7 lines

But that's easy to fix. We just have to come up with another helper which does not take lock and asssumes lock has already been taken.

/* This assumes lo->lock is already held */
__unref_inode()
{
}

Nice idea. It would be cleaner to drop the comments and introduce a unref_inode_locked() function so we can follow the usual locking regime.

(Linux uses __foo() but double underscore identifiers are reserved in the C language standard, so IMO it's better to avoid them).

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

No, because in some situations lo_destroy does not happen at the end; e.g. it happens during a umount and we still have the connection to be able to remount it.

Ok, so we don't sever the connection completely. But we don't expect to process further requests till a new INIT has been done, isn't it?

Quoted 8 lines

IOW, once lo_destroy() is received, we cleanup any pending state from the mount and if any requests are received from client, we error them out.

And start processing requests normally when a new INIT has been received.

Atleast that was my understanding of the design.

There is a state machine that does:

1. Drain in-flight requests when DESTROY is received.
2. Process DESTROY in a serialized fashion.
3. Forbid any requests until the next INIT. <-- requests queued after DESTROY are rejected here
4. Process INIT in a serialized fashion.

Stefan

#6 lore raw
Quoted 31 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 22 lines · Vivek Goyal
Quoted 10 lines · Dr. David Alan Gilbert
Quoted 9 lines · * Vivek Goyal ([email protected]) wrote:
Quoted 5 lines · Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

This fixes a crash in lo_destroy since g_hash_table_foreach_remove doesn't like the hashtable changing as it iterates, and unref_inode will remove entries.

Hi David,

Quoted 7 lines
Quoted 4 lines

Got two questions.

- Shouldn't we take a lo->mutex to make sure parallel hash table modifications are not happening.

See Stefan's big comment (which I just changed the function name in) which explains that we can't take lo->mutex

That comment says that unref_inode() takes the lock and that's why we can't take it.

Quoted 7 lines

But that's easy to fix. We just have to come up with another helper which does not take lock and asssumes lock has already been taken.

/* This assumes lo->lock is already held */
__unref_inode()
{
}

Quoted 5 lines

Nice idea. It would be cleaner to drop the comments and introduce a unref_inode_locked() function so we can follow the usual locking regime.

(Linux uses __foo() but double underscore identifiers are reserved in the C language standard, so IMO it's better to avoid them).

OK< yeh that's easy enough to do.

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 4 lines

- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

No, because in some situations lo_destroy does not happen at the end; e.g. it happens during a umount and we still have the connection to be able to remount it.

Ok, so we don't sever the connection completely. But we don't expect to process further requests till a new INIT has been done, isn't it?

IOW, once lo_destroy() is received, we cleanup any pending state from the mount and if any requests are received from client, we error them out.

Quoted 8 lines
Quoted 4 lines

And start processing requests normally when a new INIT has been received.

Atleast that was my understanding of the design.

There is a state machine that does:

1. Drain in-flight requests when DESTROY is received.

Hmm does that drain happen in the reboot case where there wasn't actually a destroy message? (fuse_session_process_buf_int where it calls se->op.destroy explicitly)

Dave

Quoted 7 lines

2. Process DESTROY in a serialized fashion.
3. Forbid any requests until the next INIT. <-- requests queued after DESTROY are rejected here
4. Process INIT in a serialized fashion.

Stefan

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#7 lore raw
Quoted 24 lines · Dr. David Alan Gilbert
Quoted 23 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 21 lines · Vivek Goyal
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 3 lines · * Vivek Goyal ([email protected]) wrote:

On Wed, Aug 07, 2019 at 10:31:52AM +0100, Dr. David Alan Gilbert (git) wrote:
- Also before destroying lo, should we sever connection so that any requests which come after lo_destroy() are not entertained.

No, because in some situations lo_destroy does not happen at the end; e.g. it happens during a umount and we still have the connection to be able to remount it.

Ok, so we don't sever the connection completely. But we don't expect to process further requests till a new INIT has been done, isn't it?

IOW, once lo_destroy() is received, we cleanup any pending state from the mount and if any requests are received from client, we error them out.

And start processing requests normally when a new INIT has been received.

Atleast that was my understanding of the design.

Quoted 7 lines
Quoted 3 lines

There is a state machine that does:

1. Drain in-flight requests when DESTROY is received.

Hmm does that drain happen in the reboot case where there wasn't actually a destroy message? (fuse_session_process_buf_int where it calls se->op.destroy explicitly)

struct fuse_session::init_rwlock has these semantics:

INIT and DESTROY requests acquire a write lock. All other requests acquire a read lock.

A write lock must wait for in-progress readers to complete. This is the drain. New readers must wait for writers (even for waiting writers, to ensure fairness).

In the reboot case INIT message acts as the drain operation.

Stefan