Skip to contentSkip to conversation

gfs2: Add missing rcu locking for glock lookup

PATCH New Andreas Gruenbacher 4 participants 12 messages Feb 22, 2017 – Feb 23, 2017 GFS2 file system

lore
Share & tools
lore

Andreas Gruenbacher

author maintainer
#0 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

From: Andrew Price <[email protected]>

We must hold the rcu read lock across looking up glocks and trying to bump their refcount to prevent the glocks from being freed in between.

---
 fs/gfs2/glock.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/gfs2/glock.c +5 -0
index 94f50ca..1d60f5f 100644
@@ -658,9 +658,11 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
 	struct kmem_cache *cachep;
 	int ret, tries = 0;
 
+	rcu_read_lock();
 	gl = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
 	if (gl && !lockref_get_not_dead(&gl->gl_lockref))
 		gl = NULL;
+	rcu_read_unlock();
 
 	*glp = gl;
 	if (gl)
@@ -728,15 +730,18 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
 
 	if (ret == -EEXIST) {
 		ret = 0;
+		rcu_read_lock();
 		tmp = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
 		if (tmp == NULL || !lockref_get_not_dead(&tmp->gl_lockref)) {
 			if (++tries < 100) {
+				rcu_read_unlock();
 				cond_resched();
 				goto again;
 			}
 			tmp = NULL;
 			ret = -ENOMEM;
 		}
+		rcu_read_unlock();
 	} else {
 		WARN_ON_ONCE(ret);
 	}
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Hi,

Looks good, but can we fix that "retry for 100 times" loop too at the same time? There does appear to be an rhashtable API function that does what we need there,

Steve.

Quoted 45 lines · Andreas Gruenbacher

From: Andrew Price <[email protected]>

We must hold the rcu read lock across looking up glocks and trying to bump their refcount to prevent the glocks from being freed in between.

---
  fs/gfs2/glock.c | 5 +++++
  1 file changed, 5 insertions(+)
fs/gfs2/glock.c +5 -0
index 94f50ca..1d60f5f 100644
@@ -658,9 +658,11 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
  	struct kmem_cache *cachep;
  	int ret, tries = 0;
 
+	rcu_read_lock();
  	gl = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
  	if (gl && !lockref_get_not_dead(&gl->gl_lockref))
  		gl = NULL;
+	rcu_read_unlock();
 
  	*glp = gl;
  	if (gl)
@@ -728,15 +730,18 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
 
  	if (ret == -EEXIST) {
  		ret = 0;
+		rcu_read_lock();
  		tmp = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
  		if (tmp == NULL || !lockref_get_not_dead(&tmp->gl_lockref)) {
  			if (++tries < 100) {
+				rcu_read_unlock();
  				cond_resched();
  				goto again;
  			}
  			tmp = NULL;
  			ret = -ENOMEM;
  		}
+		rcu_read_unlock();
  	} else {
  		WARN_ON_ONCE(ret);
  	}
author maintainer
#2 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Quoted 5 lines · Steven Whitehouse

Hi,

Looks good, but can we fix that "retry for 100 times" loop too at the same time? There does appear to be an rhashtable API function that does what we need there,

The rhashtable cleanup which I've posted separately doesn't actually affect the weird retry loop. The right thing to do here, so far, would be to retry forever I believe.

Andreas

author maintainer
#3 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Quoted 9 lines · Andreas Gruenbacher
Quoted 5 lines · Steven Whitehouse

Hi,

Looks good, but can we fix that "retry for 100 times" loop too at the same time? There does appear to be an rhashtable API function that does what we need there,

The rhashtable cleanup which I've posted separately doesn't actually affect the weird retry loop.

(But it does fix that at the same time.)

Andreas

#4 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

In reply to Andreas Gruenbacher

Hi,

Quoted 11 lines · Andreas Gruenbacher
Quoted 5 lines · Steven Whitehouse

Hi,

Looks good, but can we fix that "retry for 100 times" loop too at the same time? There does appear to be an rhashtable API function that does what we need there,

The rhashtable cleanup which I've posted separately doesn't actually affect the weird retry loop. The right thing to do here, so far, would be to retry forever I believe.

Andreas

There should be no need to retry at all. Either the new entry will be added to the hash table, or it will find an existing entry in the table. Is there really some way this can fail for some reason? We ought not to need a loop here,

Steve.

author maintainer
#5 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Quoted 3 lines · Steven Whitehouse

There should be no need to retry at all. Either the new entry will be added to the hash table, or it will find an existing entry in the table. Is there really some way this can fail for some reason?

Yes, we may find an existing entry but by the time we try to get a lockref, the entry may be dead. In that case, the right thing to do is to retry the insert.

Andreas

#6 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Hi,

Quoted 9 lines · Andreas Gruenbacher
Quoted 3 lines · Steven Whitehouse

There should be no need to retry at all. Either the new entry will be added to the hash table, or it will find an existing entry in the table. Is there really some way this can fail for some reason?

Yes, we may find an existing entry but by the time we try to get a lockref, the entry may be dead. In that case, the right thing to do is to retry the insert.

Andreas

Hmm. If it is dead but not yet removed from the list, then we will spin on this. The original code prior to the rhashtable conversion would have just looked at the next entry on the list, if it found a dead entry matching the key.

In gfs2_glock_put() we do remove the glock from the hash table, but not until after removing it from the lru and dropping the gl_lockref.lock, so there is some potential for this to happen. Whether it will be a problem in reality I'm not sure, but it is not ideal - I wonder how other rhashtable users deal with this issue... or maybe we are the only ones using lockref with rhashtable.

This is certainly a big improvement on what we had before, but at the same time, we should make a note to fix this properly in due course,

Steve.

#7 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

In reply to Andreas Gruenbacher
Quoted 6 lines · Andreas Gruenbacher

From: Andrew Price <[email protected]>

We must hold the rcu read lock across looking up glocks and trying to bump their refcount to prevent the glocks from being freed in between.

Thanks,
Andy

Quoted 40 lines
---
 fs/gfs2/glock.c | 5 +++++
 1 file changed, 5 insertions(+)
fs/gfs2/glock.c +5 -0
index 94f50ca..1d60f5f 100644
@@ -658,9 +658,11 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
 	struct kmem_cache *cachep;
 	int ret, tries = 0;
 
+	rcu_read_lock();
 	gl = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
 	if (gl && !lockref_get_not_dead(&gl->gl_lockref))
 		gl = NULL;
+	rcu_read_unlock();
 
 	*glp = gl;
 	if (gl)
@@ -728,15 +730,18 @@ int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
 
 	if (ret == -EEXIST) {
 		ret = 0;
+		rcu_read_lock();
 		tmp = rhashtable_lookup_fast(&gl_hash_table, &name, ht_parms);
 		if (tmp == NULL || !lockref_get_not_dead(&tmp->gl_lockref)) {
 			if (++tries < 100) {
+				rcu_read_unlock();
 				cond_resched();
 				goto again;
 			}
 			tmp = NULL;
 			ret = -ENOMEM;
 		}
+		rcu_read_unlock();
 	} else {
 		WARN_ON_ONCE(ret);
 	}
#8 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

In reply to Andreas Gruenbacher

----- Original Message -----

| From: Andrew Price <[email protected]>
| 
| We must hold the rcu read lock across looking up glocks and trying to
| bump their refcount to prevent the glocks from being freed in between.
| 
| Signed-off-by: Andreas Gruenbacher <[email protected]>
| ---

Hi,

Thanks. This is now applied to the for-next branch of the linux-gfs2 tree:
https://git.kernel.org/cgit/linux/kernel/git/gfs2/linux-gfs2.git/commit/?h=for-next&id=329414f6696541d1680f5867ce747050b8054e66

Regards,

Bob Peterson
Red Hat File Systems

#9 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Quoted 8 lines · Bob Peterson

----- Original Message -----

| From: Andrew Price <[email protected]>
|
| We must hold the rcu read lock across looking up glocks and trying to
| bump their refcount to prevent the glocks from being freed in between.
|
| Signed-off-by: Andreas Gruenbacher <[email protected]>
| ---
Quoted 4 lines

Hi,

Thanks. This is now applied to the for-next branch of the linux-gfs2 tree:
https://git.kernel.org/cgit/linux/kernel/git/gfs2/linux-gfs2.git/commit/?h=for-next&id=329414f6696541d1680f5867ce747050b8054e66

I think Andreas intended for me to be the "author" of this commit, which is what git am should have done with the "From:" line. Not that I'm overly bothered about it :)

Andy

#10 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

----- Original Message -----

| On 22/02/17 19:19, Bob Peterson wrote:
| > ----- Original Message -----
| > | From: Andrew Price <[email protected]>
| > |
| > | We must hold the rcu read lock across looking up glocks and trying to
| > | bump their refcount to prevent the glocks from being freed in between.
| > |
| > | Signed-off-by: Andreas Gruenbacher <[email protected]>
| > | ---
| > Hi,
| >
| > Thanks. This is now applied to the for-next branch of the linux-gfs2 tree:
| > https://git.kernel.org/cgit/linux/kernel/git/gfs2/linux-gfs2.git/commit/?h=for-next&id=329414f6696541d1680f5867ce747050b8054e66
| 
| I think Andreas intended for me to be the "author" of this commit, which
| is what git am should have done with the "From:" line. Not that I'm
| overly bothered about it :)
| 
| Andy
| 

Sorry about that. I fixed it now.

Regards,

Bob Peterson
Red Hat File Systems

#11 lore raw

PATCH 1/1 gfs2: Add missing rcu locking for glock lookup

Quoted 21 lines · Bob Peterson

----- Original Message -----

| On 22/02/17 19:19, Bob Peterson wrote:
| > ----- Original Message -----
| > | From: Andrew Price <[email protected]>
| > |
| > | We must hold the rcu read lock across looking up glocks and trying to
| > | bump their refcount to prevent the glocks from being freed in between.
| > |
| > | Signed-off-by: Andreas Gruenbacher <[email protected]>
| > | ---
| > Hi,
| >
| > Thanks. This is now applied to the for-next branch of the linux-gfs2 tree:
| > https://git.kernel.org/cgit/linux/kernel/git/gfs2/linux-gfs2.git/commit/?h=for-next&id=329414f6696541d1680f5867ce747050b8054e66
|
| I think Andreas intended for me to be the "author" of this commit, which
| is what git am should have done with the "From:" line. Not that I'm
| overly bothered about it :)
|
| Andy
|
Quoted 1 line

Sorry about that. I fixed it now.

Great, thanks :)

Andy