Skip to contentSkip to conversation

gfs2: Fix missed wakeups in find_insert_glock

PATCH New Andreas Gruenbacher 2 participants 4 messages Mar 6, 2019 – Mar 11, 2019 GFS2 file system

lore
Share & tools
lore

Andreas Gruenbacher

author maintainer
#0 lore raw

PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock

Mark Syms has reported seeing tasks that are stuck waiting in find_insert_glock. It turns out that struct lm_lockname contains four padding bytes on 64-bit architectures that function glock_waitqueue doesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.

Use offsetofend instead of sizeof to get the struct size without the padding.

---
 fs/gfs2/glock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +2 -1
index f66773c71bcde..01611f363cd59 100644
@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,
 
 static wait_queue_head_t *glock_waitqueue(struct lm_lockname *name)
 {
-	u32 hash = jhash2((u32 *)name, sizeof(*name) / 4, 0);
+	unsigned int size = offsetofend(struct lm_lockname, ln_type);
+	u32 hash = jhash2((u32 *)name, size / 4, 0);
 
 	return glock_wait_table + hash_32(hash, GLOCK_WAIT_TABLE_BITS);
 }
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock

Quoted 10 lines · Andreas Gruenbacher

Mark Syms has reported seeing tasks that are stuck waiting in find_insert_glock. It turns out that struct lm_lockname contains four padding bytes on 64-bit architectures that function glock_waitqueue doesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.

Use offsetofend instead of sizeof to get the struct size without the padding.

Thank you for your patch, initial testing looks promising when replacing the schedule_timeout patch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.

Quoted 9 lines
---
 fs/gfs2/glock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +0 -0
index f66773c71bcde..01611f363cd59 100644

fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 110

Quoted 5 lines
fs/gfs2/glock.c +1 -1
@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,
 
 static wait_queue_head_t *glock_waitqueue(struct lm_lockname *name)
 {
-	u32 hash = jhash2((u32 *)name, sizeof(*name) / 4, 0);
+	unsigned int size = offsetofend(struct lm_lockname, ln_type);

This is the same as ht_parms.key_len, which I assume was the intention.
Should we just use ht_parms.key_len for size instead of calculating it again to avoid the 2 values becoming different in the future again?

fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 114

Quoted 5 lines
fs/gfs2/glock.c +1 -0
+	u32 hash = jhash2((u32 *)name, size / 4, 0);
 
 	return glock_wait_table + hash_32(hash, GLOCK_WAIT_TABLE_BITS);
 }

Best regards,
--Edwin

author maintainer
#2 lore raw

PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock

Hi Edwin,

Quoted 7 lines · Edwin T?r?k
Quoted 6 lines · Andreas Gruenbacher

Mark Syms has reported seeing tasks that are stuck waiting in find_insert_glock. It turns out that struct lm_lockname contains four padding bytes on 64-bit architectures that function glock_waitqueue doesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.

Quoted 7 lines
Quoted 4 lines

Use offsetofend instead of sizeof to get the struct size without the padding.

Thank you for your patch, initial testing looks promising when replacing the schedule_timeout patch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.

okay great, thanks. Hope we can still get this into the 5.1 kernel.

Quoted 10 lines
Quoted 10 lines
---
 fs/gfs2/glock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +0 -0
index f66773c71bcde..01611f363cd59 100644
@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,
 
Quoted 8 lines

fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 110

Quoted 4 lines
fs/gfs2/glock.c +1 -1
 static wait_queue_head_t *glock_waitqueue(struct lm_lockname *name)
 {
-     u32 hash = jhash2((u32 *)name, sizeof(*name) / 4, 0);
+     unsigned int size = offsetofend(struct lm_lockname, ln_type);

This is the same as ht_parms.key_len, which I assume was the intention.
Should we just use ht_parms.key_len for size instead of calculating it again to avoid the 2 values becoming different in the future again?

We can do that, yes.

Quoted 4 lines

fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 114

Quoted 4 lines
fs/gfs2/glock.c +1 -0
+     u32 hash = jhash2((u32 *)name, size / 4, 0);
 
      return glock_wait_table + hash_32(hash, GLOCK_WAIT_TABLE_BITS);
 }

Thanks,
Andreas

#3 lore raw

PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock

Quoted 12 lines · Andreas Gruenbacher

Hi Edwin,

Quoted 9 lines · Edwin T?r?k
Quoted 8 lines · Andreas Gruenbacher

Mark Syms has reported seeing tasks that are stuck waiting in find_insert_glock. It turns out that struct lm_lockname contains four padding bytes on 64-bit architectures that function glock_waitqueue doesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.

Use offsetofend instead of sizeof to get the struct size without the padding.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Thank you for your patch, initial testing looks promising when replacing the schedule_timeout patch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.

okay great, thanks. Hope we can still get this into the 5.1 kernel.

Testing of this patch itself looked good (no new issues), however GFS2 still deadlocks on 4.19 even with this patch, see:
https://www.redhat.com/archives/cluster-devel/2019-March/msg00000.html
[the iomap deadlocks seems to be more rare now though]

Thanks,
--Edwin