gfs2: Fix missed wakeups in find_insert_glock
Andreas Gruenbacher
maintainerPATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock
Mark Syms has reported seeing tasks that are stuck waiting in find_insert_glock. It turns out that struct lm_lockname contains four padding bytes on 64-bit architectures that function glock_waitqueue doesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.
Use offsetofend instead of sizeof to get the struct size without the padding.
--- fs/gfs2/glock.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +2 -1
@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,static wait_queue_head_t *glock_waitqueue (struct lm_lockname *name) { -u32 hash =jhash2 ((u32 *)name,sizeof (*name) /4 ,0 ); + unsigned int size = offsetofend(struct lm_lockname, ln_type); +u32 hash =jhash2 ((u32 *)name, size /4 ,0 );return glock_wait_table +hash_32 (hash,GLOCK_WAIT_TABLE_BITS ); }
Signature · 1 line
2.20.1
PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock
Quoted 10 lines · Andreas Gruenbacher
Mark Syms has reported seeing tasks that are stuck waiting in
find_insert_glock. It turns out that structlm_locknamecontains four padding bytes on 64-bit architectures that functionglock_waitqueuedoesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.Use offsetofend instead of sizeof to get the struct size without the padding.
Reported-by Mark Syms <[email protected]>Signed-off-by Andreas Gruenbacher <[email protected]>
Thank you for your patch, initial testing looks promising when replacing the schedule_timeout patch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.
Quoted 9 lines
--- fs/gfs2/glock.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)fs/gfs2/glock.c +0
-0
fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 110
Quoted 5 lines
fs/gfs2/glock.c +1
-1@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,static wait_queue_head_t *glock_waitqueue (struct lm_lockname *name) { -u32 hash =jhash2 ((u32 *)name,sizeof (*name) /4 ,0 ); + unsigned int size = offsetofend(struct lm_lockname, ln_type);
This is the same as ht_parms.key_len, which I assume was the intention.
Should we just use ht_parms.key_len for size instead of calculating it again to avoid the 2 values becoming different in the future again?
fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 114
Quoted 5 lines
fs/gfs2/glock.c +1
-0+u32 hash =jhash2 ((u32 *)name, size /4 ,0 );return glock_wait_table +hash_32 (hash,GLOCK_WAIT_TABLE_BITS ); }
Best regards,
--Edwin
Andreas Gruenbacher
maintainerPATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock
Hi Edwin,
Quoted 7 lines · Edwin T?r?k
Quoted 6 lines · Andreas Gruenbacher
Mark Syms has reported seeing tasks that are stuck waiting in
find_insert_glock. It turns out that structlm_locknamecontains four padding bytes on 64-bit architectures that functionglock_waitqueuedoesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.
Quoted 7 lines
Quoted 4 lines
Use offsetofend instead of sizeof to get the struct size without the padding.
Reported-by Mark Syms <[email protected]>Signed-off-by Andreas Gruenbacher <[email protected]>Thank you for your patch, initial testing looks promising when replacing the
schedule_timeoutpatch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.
okay great, thanks. Hope we can still get this into the 5.1 kernel.
Quoted 10 lines
Quoted 10 lines
--- fs/gfs2/glock.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)fs/gfs2/glock.c +0
-0@@ -107,7 +107,8 @@ static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode,
Quoted 8 lines
fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 110
Quoted 4 lines
fs/gfs2/glock.c +1
-1static wait_queue_head_t *glock_waitqueue (struct lm_lockname *name) { -u32 hash =jhash2 ((u32 *)name,sizeof (*name) /4 ,0 ); + unsigned int size = offsetofend(struct lm_lockname, ln_type);This is the same as ht_parms.key_len, which I assume was the intention.
Should we just use ht_parms.key_len for size instead of calculating it again to avoid the 2 values becoming different in the future again?
We can do that, yes.
Quoted 4 lines
fs/gfs2/glock.c static int glock_wake_function(wait_queue_entry_t *wait, unsigned int mode, · line 114
Quoted 4 lines
fs/gfs2/glock.c +1
-0+u32 hash =jhash2 ((u32 *)name, size /4 ,0 );return glock_wait_table +hash_32 (hash,GLOCK_WAIT_TABLE_BITS ); }
Thanks,
Andreas
PATCH 1/1 gfs2: Fix missed wakeups in find_insert_glock
Quoted 12 lines · Andreas Gruenbacher
Hi Edwin,
Quoted 9 lines · Edwin T?r?k
Quoted 8 lines · Andreas Gruenbacher
Mark Syms has reported seeing tasks that are stuck waiting in
find_insert_glock. It turns out that structlm_locknamecontains four padding bytes on 64-bit architectures that functionglock_waitqueuedoesn't skip when hashing the glock name. As a result, we can end up waking up the wrong waitqueue, and the waiting tasks will be stuck.Use offsetofend instead of sizeof to get the struct size without the padding.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Reported-by Mark Syms <[email protected]>Signed-off-by Andreas Gruenbacher <[email protected]>Thank you for your patch, initial testing looks promising when replacing the
schedule_timeoutpatch with this one (no deadlocks/stuck tasks found so far). Will let you know when our testing completes.okay great, thanks. Hope we can still get this into the 5.1 kernel.
Testing of this patch itself looked good (no new issues), however GFS2 still deadlocks on 4.19 even with this patch, see:
https://www.redhat.com/archives/cluster-devel/2019-March/msg00000.html
[the iomap deadlocks seems to be more rare now though]
Thanks,
--Edwin