Skip to contentSkip to conversation

gfs2: Close timing window with GLF_INVALIDATE_IN_PROGRESS

PATCH New Bob Peterson 2 participants 4 messages Nov 15, 2019 – Nov 18, 2019 1 message not in the archive GFS2 file system

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 1/1 gfs2: Close timing window with GLF_INVALIDATE_IN_PROGRESS

Hi,

This patch closes a timing window in which two processes compete and overlap in the execution of do_xmote for the same glock:

Process A Process B
------------------------------------ -----------------------------
1. Grabs gl_lockref and calls do_xmote
2. Grabs gl_lockref but is blocked
3. Sets GLF_INVALIDATE_IN_PROGRESS
4. Unlocks gl_lockref
5. Calls do_xmote
6. Call glops->go_sync
7. test_and_clear_bit GLF_DIRTY
8. Call gfs2_log_flush Call glops->go_sync
9. (slow IO, so it blocks a long time) test_and_clear_bit GLF_DIRTY
It's not dirty (step 7) returns

10.                                       Tests GLF_INVALIDATE_IN_PROGRESS
11.                                       Calls go_inval (rgrp_go_inval)
12.                                       gfs2_rgrp_relse does brelse
13.                                       truncate_inode_pages_range
14.                                       Calls lm_lock UN

In step 14 we've just told dlm to give the glock to another node when, in fact, process A has not finished the IO and synced all buffer_heads to disk and make sure their revokes are done.

This patch fixes the problem by changing the GLF_INVALIDATE_IN_PROGRESS to use test_and_set_bit, and if the bit is already set, process B just ignores it and trusts that process A will do the do_xmote in the proper order.

---
 fs/gfs2/glock.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +13 -1
index faa88bd594e2..4a4a390ffd00 100644
@@ -558,7 +558,19 @@ __acquires(&gl->gl_lockref.lock)
 	GLOCK_BUG_ON(gl, gl->gl_state == gl->gl_target);
 	if ((target == LM_ST_UNLOCKED || target == LM_ST_DEFERRED) &&
 	    glops->go_inval) {
-		set_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags);
+		/*
+		 * If another process is already doing the invalidate we should
+		 * not interfere. If we call go_sync and it finds the glock is
+		 * not dirty, we might call go_inval prematurely before the
+		 * other go_sync has finished with its revokes. If we then call
+		 * lm_lock prematurely, we've really screwed up: we cannot tell
+		 * dlm to give the glock away until we're synced and
+		 * invalidated. Best thing is to return and trust the other
+		 * process will finish do_xmote tasks in their proper order.
+		 */
+		if (test_and_set_bit(GLF_INVALIDATE_IN_PROGRESS,
+				     &gl->gl_flags))
+			return;
 		do_error(gl, 0); /* Fail queued try locks */
 	}
 	gl->gl_req = target;
#2 lore raw

PATCH 1/1 gfs2: Close timing window with GLF_INVALIDATE_IN_PROGRESS

Quoted 47 lines · Bob Peterson

Hi,

This patch closes a timing window in which two processes compete and overlap in the execution of do_xmote for the same glock:

Process A Process B
------------------------------------ -----------------------------
1. Grabs gl_lockref and calls do_xmote
2. Grabs gl_lockref but is blocked
3. Sets GLF_INVALIDATE_IN_PROGRESS
4. Unlocks gl_lockref
5. Calls do_xmote
6. Call glops->go_sync
7. test_and_clear_bit GLF_DIRTY
8. Call gfs2_log_flush Call glops->go_sync
9. (slow IO, so it blocks a long time) test_and_clear_bit GLF_DIRTY
It's not dirty (step 7) returns

10.                                       Tests GLF_INVALIDATE_IN_PROGRESS
11.                                       Calls go_inval (rgrp_go_inval)
12.                                       gfs2_rgrp_relse does brelse
13.                                       truncate_inode_pages_range
14.                                       Calls lm_lock UN

In step 14 we've just told dlm to give the glock to another node when, in fact, process A has not finished the IO and synced all buffer_heads to disk and make sure their revokes are done.

This patch fixes the problem by changing the GLF_INVALIDATE_IN_PROGRESS to use test_and_set_bit, and if the bit is already set, process B just ignores it and trusts that process A will do the do_xmote in the proper order.

---
 fs/gfs2/glock.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +0 -1
index faa88bd594e2..4a4a390ffd00 100644
@@ -558,7 +558,19 @@ __acquires(&gl->gl_lockref.lock)
        GLOCK_BUG_ON(gl, gl->gl_state == gl->gl_target);
        if ((target == LM_ST_UNLOCKED || target == LM_ST_DEFERRED) &&
            glops->go_inval) {
-               set_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags);

fs/gfs2/glock.c __acquires(&gl->gl_lockref.lock) · line 570

Quoted 10 lines
fs/gfs2/glock.c +10 -0
+               /*
+                * If another process is already doing the invalidate we should
+                * not interfere. If we call go_sync and it finds the glock is
+                * not dirty, we might call go_inval prematurely before the
+                * other go_sync has finished with its revokes. If we then call
+                * lm_lock prematurely, we've really screwed up: we cannot tell
+                * dlm to give the glock away until we're synced and
+                * invalidated. Best thing is to return and trust the other
+                * process will finish do_xmote tasks in their proper order.
+                */

That's a bit too much information here. Can we please change that as follows?

* If another process is already doing the invalidate, let that
* finish first. The glock state machine will get back to this
* holder again later.

fs/gfs2/glock.c __acquires(&gl->gl_lockref.lock) · line 576

Quoted 7 lines
fs/gfs2/glock.c +3 -0
+               if (test_and_set_bit(GLF_INVALIDATE_IN_PROGRESS,
+                                    &gl->gl_flags))
+                       return;
                do_error(gl, 0); /* Fail queued try locks */
        }
        gl->gl_req = target;

Thanks,
Andreas

#3 lore raw

PATCH 1/1 gfs2: Close timing window with GLF_INVALIDATE_IN_PROGRESS

----- Original Message -----

Quoted 80 lines · Bob Peterson
Quoted 60 lines

Hi,

This patch closes a timing window in which two processes compete and overlap in the execution of do_xmote for the same glock:

Process A Process B
------------------------------------ -----------------------------
1. Grabs gl_lockref and calls do_xmote
2. Grabs gl_lockref but is blocked
3. Sets GLF_INVALIDATE_IN_PROGRESS
4. Unlocks gl_lockref
5. Calls do_xmote
6. Call glops->go_sync
7. test_and_clear_bit GLF_DIRTY
8. Call gfs2_log_flush Call glops->go_sync
9. (slow IO, so it blocks a long time) test_and_clear_bit GLF_DIRTY
It's not dirty (step 7) returns

10.                                       Tests GLF_INVALIDATE_IN_PROGRESS
11.                                       Calls go_inval (rgrp_go_inval)
12.                                       gfs2_rgrp_relse does brelse
13.                                       truncate_inode_pages_range
14.                                       Calls lm_lock UN

In step 14 we've just told dlm to give the glock to another node when, in fact, process A has not finished the IO and synced all buffer_heads to disk and make sure their revokes are done.

This patch fixes the problem by changing the GLF_INVALIDATE_IN_PROGRESS to use test_and_set_bit, and if the bit is already set, process B just ignores it and trusts that process A will do the do_xmote in the proper order.

---
 fs/gfs2/glock.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
fs/gfs2/glock.c +2 -1
index faa88bd594e2..4a4a390ffd00 100644
@@ -558,7 +558,19 @@ __acquires(&gl->gl_lockref.lock)
        GLOCK_BUG_ON(gl, gl->gl_state == gl->gl_target);
        if ((target == LM_ST_UNLOCKED || target == LM_ST_DEFERRED) &&
            glops->go_inval) {
-               set_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags);
+               /*
+                * If another process is already doing the invalidate we

should
+ * not interfere. If we call go_sync and it finds the glock
is

fs/gfs2/glock.c +2 -0
+                * not dirty, we might call go_inval prematurely before the
+                * other go_sync has finished with its revokes. If we then

call
+ * lm_lock prematurely, we've really screwed up: we cannot
tell

fs/gfs2/glock.c +3 -0
+                * dlm to give the glock away until we're synced and
+                * invalidated. Best thing is to return and trust the other
+                * process will finish do_xmote tasks in their proper
Quoted 2 lines

order.
+ */

That's a bit too much information here. Can we please change that as follows?

* If another process is already doing the invalidate, let that
* finish first. The glock state machine will get back to this
* holder again later.

fs/gfs2/glock.c __acquires(&gl->gl_lockref.lock) · line 576

Quoted 7 lines
fs/gfs2/glock.c +3 -0
+               if (test_and_set_bit(GLF_INVALIDATE_IN_PROGRESS,
+                                    &gl->gl_flags))
+                       return;
                do_error(gl, 0); /* Fail queued try locks */
        }
        gl->gl_req = target;
Quoted 4 lines

Thanks,
Andreas

Sure. Make it so.

Bob

#4 lore raw

PATCH 1/1 gfs2: Close timing window with GLF_INVALIDATE_IN_PROGRESS

Ok, pushed.

Thanks,
Andreas