Skip to contentSkip to conversation

fix hangup when multiple processes are trying to write to the same file

PATCH New Josef Whiter 2 participants 5 messages Feb 23, 2007 GFS2 file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fix hangup when multiple processes are trying to write to the same file

This fixes a problem I encountered while running bonnie++. When you have one thread that opens a file and starts to write to it, and then another thread that tries to open and write to the same file, the second thread will loop forever trying to grab the inode lock for that inode. Basically we come in through generic_buffered_file_write, which calls gfs2_prepare_write, which then attempts to grab the glock. Because we don't own the lock, gfs2_prepare_write gets GLR_TRYFAILED, which returns AOP_TRUNCATED_PAGE to generic_buffered_file_write. At this point generic_buffered_file_write loops around again and immediately retries the prepare_write. This means that the second process never gets off of the processor in order to allow the process that holds the lock to finish its work and let go of the lock. This patch makes gfs2_glock_nq schedule() if it gets back a GLR_TRYFAILED, which resolves this problem. Please let me know if this is in the wrong place, ie if it should go in gfs2_prepare_write instead of gfs2_glock_nq, or if its completely wrong :).

fs/gfs2/glock.c renamed from fs/gfs2/glock.c.josef +2 -1
@@ -1185,7 +1185,8 @@
 		if (error == GLR_CANCELED) {
 			msleep(100);
 			goto restart;
-		}
+		} else if (error == GLR_TRYFAILED) 
+			schedule();
 	}
 
 	return error;
#1 lore raw

PATCH 1/1 fix hangup when multiple processes are trying to write to the same file

Quoted 6 lines · Josef Whiter

This fixes a problem I encountered while running bonnie++. When you have one thread that opens a file and starts to write to it, and then another thread that tries to open and write to the same file, the second thread will loop forever trying to grab the inode lock for that inode. Basically we come in through generic_buffered_file_write, which calls gfs2_prepare_write, which then attempts to grab the glock. Because we don't own the lock, gfs2_prepare_write gets

Quoted 9 lines

GLR_TRYFAILED, which returns AOP_TRUNCATED_PAGE to generic_buffered_file_write. At this point generic_buffered_file_write loops around again and immediately retries the prepare_write. This means that the second process never gets off of the processor in order to allow the process that holds the lock to finish its work and let go of the lock. This patch makes gfs2_glock_nq schedule() if it gets back a GLR_TRYFAILED, which resolves this problem. Please let me know if this is in the wrong place, ie if it should go in gfs2_prepare_write instead of gfs2_glock_nq, or if its completely wrong :).

Ok at the suggestion of Steven I have made it so that we just yield in the places where this would be a problem, so in gfs2_prepare_write and gfs2_readpage.

fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +4 -1
@@ -266,8 +266,10 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
+	if (error == GLR_TRYFAILED) {
 		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	unlock_page(page);
 	if (do_unlock)
 		gfs2_holder_uninit(&gh);
@@ -364,6 +366,7 @@ static int gfs2_prepare_write(struct fil
 		if (error == GLR_TRYFAILED) {
 			unlock_page(page);
 			error = AOP_TRUNCATED_PAGE;
+			yield();
 		}
 		goto out_uninit;
 	}
#2 lore raw

PATCH 1/1 fix hangup when multiple processes are trying to write to the same file

Hi,

Quoted 17 lines · Josef Whiter
Quoted 6 lines · Josef Whiter

This fixes a problem I encountered while running bonnie++. When you have one thread that opens a file and starts to write to it, and then another thread that tries to open and write to the same file, the second thread will loop forever trying to grab the inode lock for that inode. Basically we come in through generic_buffered_file_write, which calls gfs2_prepare_write, which then attempts to grab the glock. Because we don't own the lock, gfs2_prepare_write gets

Quoted 9 lines

GLR_TRYFAILED, which returns AOP_TRUNCATED_PAGE to generic_buffered_file_write. At this point generic_buffered_file_write loops around again and immediately retries the prepare_write. This means that the second process never gets off of the processor in order to allow the process that holds the lock to finish its work and let go of the lock. This patch makes gfs2_glock_nq schedule() if it gets back a GLR_TRYFAILED, which resolves this problem. Please let me know if this is in the wrong place, ie if it should go in gfs2_prepare_write instead of gfs2_glock_nq, or if its completely wrong :).

Quoted 6 lines

Ok at the suggestion of Steven I have made it so that we just yield in the places where this would be a problem, so in gfs2_prepare_write and gfs2_readpage.

Thats looks good, but one small comment:

fs/gfs2/ops_address.c skip_lock: · line 273

Quoted 12 lines
fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +3 -1
@@ -266,8 +266,10 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
+	if (error == GLR_TRYFAILED) {
 		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	unlock_page(page);

Can you move the above unlock_page to directly after the out_unlock: ? That way we won't schedule while holding the page lock. The other part of the patch does that in the right order already and looks good,

Steve.

Quoted 11 lines
fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +1 -0
 	if (do_unlock)
 		gfs2_holder_uninit(&gh);
@@ -364,6 +366,7 @@ static int gfs2_prepare_write(struct fil
 		if (error == GLR_TRYFAILED) {
 			unlock_page(page);
 			error = AOP_TRUNCATED_PAGE;
+			yield();
 		}
 		goto out_uninit;
 	}
#3 lore raw

PATCH 1/1 fix hangup when multiple processes are trying to write to the same file

Quoted 44 lines · Steven Whitehouse

Hi,

Quoted 17 lines · Josef Whiter
Quoted 6 lines · Josef Whiter

This fixes a problem I encountered while running bonnie++. When you have one thread that opens a file and starts to write to it, and then another thread that tries to open and write to the same file, the second thread will loop forever trying to grab the inode lock for that inode. Basically we come in through generic_buffered_file_write, which calls gfs2_prepare_write, which then attempts to grab the glock. Because we don't own the lock, gfs2_prepare_write gets

Quoted 9 lines

GLR_TRYFAILED, which returns AOP_TRUNCATED_PAGE to generic_buffered_file_write. At this point generic_buffered_file_write loops around again and immediately retries the prepare_write. This means that the second process never gets off of the processor in order to allow the process that holds the lock to finish its work and let go of the lock. This patch makes gfs2_glock_nq schedule() if it gets back a GLR_TRYFAILED, which resolves this problem. Please let me know if this is in the wrong place, ie if it should go in gfs2_prepare_write instead of gfs2_glock_nq, or if its completely wrong :).

Quoted 6 lines

Ok at the suggestion of Steven I have made it so that we just yield in the places where this would be a problem, so in gfs2_prepare_write and gfs2_readpage.

Thats looks good, but one small comment:

fs/gfs2/ops_address.c skip_lock: · line 273

Quoted 12 lines
fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +3 -1
@@ -266,8 +266,10 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
+	if (error == GLR_TRYFAILED) {
 		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	unlock_page(page);

Can you move the above unlock_page to directly after the out_unlock: ? That way we won't schedule while holding the page lock. The other part of the patch does that in the right order already and looks good,

Quoted 2 lines

Steve.

Doh sorry about that. Here's the correct version

fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +5 -2
@@ -266,9 +266,11 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
-		error = AOP_TRUNCATED_PAGE;
 	unlock_page(page);
+	if (error == GLR_TRYFAILED) {
+		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	if (do_unlock)
 		gfs2_holder_uninit(&gh);
 	goto out;
@@ -364,6 +366,7 @@ static int gfs2_prepare_write(struct fil
 		if (error == GLR_TRYFAILED) {
 			unlock_page(page);
 			error = AOP_TRUNCATED_PAGE;
+			yield();
 		}
 		goto out_uninit;
 	}
#4 lore raw

PATCH 1/1 fix hangup when multiple processes are trying to write to the same file

Hi,

Now in the -nmw git tree. Thanks,

Steve.

Quoted 75 lines · Josef Whiter
Quoted 44 lines · Steven Whitehouse

Hi,

Quoted 17 lines · Josef Whiter
Quoted 6 lines · Josef Whiter

This fixes a problem I encountered while running bonnie++. When you have one thread that opens a file and starts to write to it, and then another thread that tries to open and write to the same file, the second thread will loop forever trying to grab the inode lock for that inode. Basically we come in through generic_buffered_file_write, which calls gfs2_prepare_write, which then attempts to grab the glock. Because we don't own the lock, gfs2_prepare_write gets

Quoted 9 lines

GLR_TRYFAILED, which returns AOP_TRUNCATED_PAGE to generic_buffered_file_write. At this point generic_buffered_file_write loops around again and immediately retries the prepare_write. This means that the second process never gets off of the processor in order to allow the process that holds the lock to finish its work and let go of the lock. This patch makes gfs2_glock_nq schedule() if it gets back a GLR_TRYFAILED, which resolves this problem. Please let me know if this is in the wrong place, ie if it should go in gfs2_prepare_write instead of gfs2_glock_nq, or if its completely wrong :).

Quoted 6 lines

Ok at the suggestion of Steven I have made it so that we just yield in the places where this would be a problem, so in gfs2_prepare_write and gfs2_readpage.

Thats looks good, but one small comment:

fs/gfs2/ops_address.c skip_lock: · line 273

Quoted 12 lines
fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +3 -1
@@ -266,8 +266,10 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
+	if (error == GLR_TRYFAILED) {
 		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	unlock_page(page);

Can you move the above unlock_page to directly after the out_unlock: ? That way we won't schedule while holding the page lock. The other part of the patch does that in the right order already and looks good,

Quoted 2 lines

Steve.

Doh sorry about that. Here's the correct version

fs/gfs2/ops_address.c renamed from fs/gfs2/ops_address.c.josef +5 -2
@@ -266,9 +266,11 @@ skip_lock:
 out:
 	return error;
 out_unlock:
-	if (error == GLR_TRYFAILED)
-		error = AOP_TRUNCATED_PAGE;
 	unlock_page(page);
+	if (error == GLR_TRYFAILED) {
+		error = AOP_TRUNCATED_PAGE;
+		yield();
+	}
 	if (do_unlock)
 		gfs2_holder_uninit(&gh);
 	goto out;
@@ -364,6 +366,7 @@ static int gfs2_prepare_write(struct fil
 		if (error == GLR_TRYFAILED) {
 			unlock_page(page);
 			error = AOP_TRUNCATED_PAGE;
+			yield();
 		}
 		goto out_uninit;
 	}