Skip to contentSkip to conversation

Fix freeze of cluster-2.03.11

PATCH New Kadlecsik Jozsef 3 participants 7 messages Apr 20, 2009 – Apr 22, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

I reported on the linux-cluster mailing list (see thread
https://www.redhat.com/archives/linux-cluster/2009-March/msg00176.html)
that version cluster-2.03.11 freezes.

As Wendy Cheng pointed out, the reason for this is that the filesystem API of the kernel changed and "put_inode" was replaced by "drop_inode": while "put_inode" was called without holding any lock, "drop_inode" is called under inode_lock held but the called gfs_sync_page_i may block
(https://www.redhat.com/archives/linux-cluster/2009-April/msg00060.html).

The patch below (applied in gfs-kernel/src/gfs/) fixes the bug by releasing the lock before calling gfs_sync_page_i and locking it back after the function call:

ops_super.c +5 -1
@@ -9,6 +9,7 @@
 #include <linux/statfs.h>
 #include <linux/seq_file.h>
 #include <linux/mount.h>
+#include <linux/writeback.h>
 
 #include "gfs.h"
 #include "dio.h"
@@ -68,8 +69,11 @@
 	if (ip &&
 	    !inode->i_nlink &&
 	    S_ISREG(inode->i_mode) &&
-	    !sdp->sd_args.ar_localcaching)
+	    !sdp->sd_args.ar_localcaching) {
+	    	spin_unlock(&inode_lock);
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
+		spin_lock(&inode_lock);
+	}
 	generic_drop_inode(inode);
 }
 

Additionally, the line

EXPORT_SYMBOL(inode_lock);

must be added to fs/inode.c in order to the inode_lock symbol get exported.

Best regards,
Jozsef
--
E-mail : kadlec at mail.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics

         H-1525 Budapest 114, POB. 49, Hungary
#1 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

Quoted 40 lines · Kadlecsik Jozsef

Hi,

I reported on the linux-cluster mailing list (see thread
https://www.redhat.com/archives/linux-cluster/2009-March/msg00176.html)
that version cluster-2.03.11 freezes.

As Wendy Cheng pointed out, the reason for this is that the filesystem API of the kernel changed and "put_inode" was replaced by "drop_inode": while "put_inode" was called without holding any lock, "drop_inode" is called under inode_lock held but the called gfs_sync_page_i may block
(https://www.redhat.com/archives/linux-cluster/2009-April/msg00060.html).

The patch below (applied in gfs-kernel/src/gfs/) fixes the bug by releasing the lock before calling gfs_sync_page_i and locking it back after the function call:

ops_super.c +5 -1
@@ -9,6 +9,7 @@
 #include <linux/statfs.h>
 #include <linux/seq_file.h>
 #include <linux/mount.h>
+#include <linux/writeback.h>
 
 #include "gfs.h"
 #include "dio.h"
@@ -68,8 +69,11 @@
 	if (ip &&
 	    !inode->i_nlink &&
 	    S_ISREG(inode->i_mode) &&
-	    !sdp->sd_args.ar_localcaching)
+	    !sdp->sd_args.ar_localcaching) {
+	    	spin_unlock(&inode_lock);
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
+		spin_lock(&inode_lock);
+	}
 	generic_drop_inode(inode);
 }
 
Quoted 6 lines

Additionally, the line

EXPORT_SYMBOL(inode_lock);

must be added to fs/inode.c in order to the inode_lock symbol get exported.

thanks a lot for you effort but we will need to find another solution.

it's been a long time since gfs1 required extra exported symbols from the kernel and asking distributions to do that again is not really an option (not sure if there any other option either).

I'll let the gfs guys look at it.

Thanks again!
Fabio

#2 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

Quoted 15 lines · Fabio M. Di Nitto

Hi,

Quoted 12 lines · Kadlecsik Jozsef

Hi,

I reported on the linux-cluster mailing list (see thread
https://www.redhat.com/archives/linux-cluster/2009-March/msg00176.html)
that version cluster-2.03.11 freezes.

As Wendy Cheng pointed out, the reason for this is that the filesystem API of the kernel changed and "put_inode" was replaced by "drop_inode": while "put_inode" was called without holding any lock, "drop_inode" is called under inode_lock held but the called gfs_sync_page_i may block
(https://www.redhat.com/archives/linux-cluster/2009-April/msg00060.html).

Its not a replacement as such, they are different operations. ->put_inode() was called on each and every iput() (despite what the docs said, as Christoph noted) whereas ->drop_inode() is only called when the inode is being removed from the cache.

The purpose of drop inode is to allow the fs to override the decision about whether the inode needs to be dallocated (->delete_inode()) or not (->clear_inode()).

Christoph's patch is here:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=33dcdac2df54e66c447ae03f58c95c7251aa5649

Quoted 13 lines
Quoted 13 lines

The patch below (applied in gfs-kernel/src/gfs/) fixes the bug by releasing the lock before calling gfs_sync_page_i and locking it back after the function call:

ops_super.c +1 -0
@@ -9,6 +9,7 @@
 #include <linux/statfs.h>
 #include <linux/seq_file.h>
 #include <linux/mount.h>
+#include <linux/writeback.h>
 
Quoted 15 lines
Quoted 5 lines
ops_super.c +0 -0
 #include "gfs.h"
 #include "dio.h"
@@ -68,8 +69,11 @@
 	if (ip &&
 	    !inode->i_nlink &&

ops_super.c line 79

Quoted 10 lines
ops_super.c +4 -1
 	    S_ISREG(inode->i_mode) &&
-	    !sdp->sd_args.ar_localcaching)
+	    !sdp->sd_args.ar_localcaching) {
+	    	spin_unlock(&inode_lock);
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
+		spin_lock(&inode_lock);
+	}
 	generic_drop_inode(inode);
 }
 

You can't do this in this particular way because you must set the inode's state before releasing the inode_lock. That is done in generic_drop_inode() and the functions that it calls.

Not to mention which, this doesn't seem to make any sense. It appears to be trying to write the inode's data back to disk, but if we've got to this point in the inode's life cycle, there should be nothing to write.

Since its only called when i_nlink is 0, thats even more strange since
(a) it could be moved to ->delete_inode() and (b) why do we care about forcing out dirty data for an inode thats being unlinked anyway....

This patch raises more questions than it answers I think.

Quoted 11 lines
Quoted 8 lines

Additionally, the line

EXPORT_SYMBOL(inode_lock);

must be added to fs/inode.c in order to the inode_lock symbol get exported.

thanks a lot for you effort but we will need to find another solution.

Quoted 9 lines

it's been a long time since gfs1 required extra exported symbols from the kernel and asking distributions to do that again is not really an option (not sure if there any other option either).

I'll let the gfs guys look at it.

Thanks again!
Fabio

Yes, we can't have any of our own exports now (since gfs isn't in the upstream kernel), as per the ruling from the Fedora people,

Steve.

#3 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

Quoted 14 lines · Steven Whitehouse
Quoted 7 lines · Fabio M. Di Nitto
Quoted 5 lines · Kadlecsik Jozsef

I reported on the linux-cluster mailing list (see thread
https://www.redhat.com/archives/linux-cluster/2009-March/msg00176.html)
that version cluster-2.03.11 freezes.

Quoted 6 lines
Quoted 6 lines

As Wendy Cheng pointed out, the reason for this is that the filesystem API of the kernel changed and "put_inode" was replaced by "drop_inode": while "put_inode" was called without holding any lock, "drop_inode" is called under inode_lock held but the called gfs_sync_page_i may block
(https://www.redhat.com/archives/linux-cluster/2009-April/msg00060.html).

Quoted 4 lines

Its not a replacement as such, they are different operations. ->put_inode() was called on each and every iput() (despite what the docs said, as Christoph noted) whereas ->drop_inode() is only called when the inode is being removed from the cache.

I knew I worded it incorrectly ;-).

Quoted 44 lines

The purpose of drop inode is to allow the fs to override the decision about whether the inode needs to be dallocated (->delete_inode()) or not (->clear_inode()).

Christoph's patch is here:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=33dcdac2df54e66c447ae03f58c95c7251aa5649

Quoted 13 lines
Quoted 13 lines

The patch below (applied in gfs-kernel/src/gfs/) fixes the bug by releasing the lock before calling gfs_sync_page_i and locking it back after the function call:

ops_super.c +1 -0
@@ -9,6 +9,7 @@
 #include <linux/statfs.h>
 #include <linux/seq_file.h>
 #include <linux/mount.h>
+#include <linux/writeback.h>
 
Quoted 15 lines
Quoted 5 lines
ops_super.c +0 -0
 #include "gfs.h"
 #include "dio.h"
@@ -68,8 +69,11 @@
 	if (ip &&
 	    !inode->i_nlink &&

ops_super.c line 79

Quoted 10 lines
ops_super.c +4 -1
 	    S_ISREG(inode->i_mode) &&
-	    !sdp->sd_args.ar_localcaching)
+	    !sdp->sd_args.ar_localcaching) {
+	    	spin_unlock(&inode_lock);
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
+		spin_lock(&inode_lock);
+	}
 	generic_drop_inode(inode);
 }
 

You can't do this in this particular way because you must set the inode's state before releasing the inode_lock. That is done in generic_drop_inode() and the functions that it calls.

Not to mention which, this doesn't seem to make any sense. It appears to be trying to write the inode's data back to disk, but if we've got to this point in the inode's life cycle, there should be nothing to write.

Quoted 5 lines

Since its only called when i_nlink is 0, thats even more strange since
(a) it could be moved to ->delete_inode() and (b) why do we care about forcing out dirty data for an inode thats being unlinked anyway....

This patch raises more questions than it answers I think.

Do you imply it is superfluous to call gfs_sync_page_i at all, and in consequence there's no need for a gfs-specific ->drop_inode function?

Just for the record: without unlocking/locking, the node having a mailmain queue over GFS got frozen in a few seconds as mailman queue manager was started. With the patch above it ran smoothly for a longer (test) period.

Best regards,
Jozsef
--
E-mail : kadlec at mail.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics

         H-1525 Budapest 114, POB. 49, Hungary
#4 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

Quoted 69 lines · Kadlecsik Jozsef

Hi,

Quoted 14 lines · Steven Whitehouse
Quoted 7 lines · Fabio M. Di Nitto
Quoted 5 lines · Kadlecsik Jozsef

I reported on the linux-cluster mailing list (see thread
https://www.redhat.com/archives/linux-cluster/2009-March/msg00176.html)
that version cluster-2.03.11 freezes.

Quoted 6 lines
Quoted 6 lines

As Wendy Cheng pointed out, the reason for this is that the filesystem API of the kernel changed and "put_inode" was replaced by "drop_inode": while "put_inode" was called without holding any lock, "drop_inode" is called under inode_lock held but the called gfs_sync_page_i may block
(https://www.redhat.com/archives/linux-cluster/2009-April/msg00060.html).

Quoted 4 lines

Its not a replacement as such, they are different operations. ->put_inode() was called on each and every iput() (despite what the docs said, as Christoph noted) whereas ->drop_inode() is only called when the inode is being removed from the cache.

I knew I worded it incorrectly ;-).

Quoted 44 lines

The purpose of drop inode is to allow the fs to override the decision about whether the inode needs to be dallocated (->delete_inode()) or not (->clear_inode()).

Christoph's patch is here:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=33dcdac2df54e66c447ae03f58c95c7251aa5649

Quoted 13 lines
Quoted 13 lines

The patch below (applied in gfs-kernel/src/gfs/) fixes the bug by releasing the lock before calling gfs_sync_page_i and locking it back after the function call:

ops_super.c +1 -0
@@ -9,6 +9,7 @@
 #include <linux/statfs.h>
 #include <linux/seq_file.h>
 #include <linux/mount.h>
+#include <linux/writeback.h>
 
Quoted 15 lines
Quoted 5 lines
ops_super.c +0 -0
 #include "gfs.h"
 #include "dio.h"
@@ -68,8 +69,11 @@
 	if (ip &&
 	    !inode->i_nlink &&

ops_super.c line 79

Quoted 10 lines
ops_super.c +4 -1
 	    S_ISREG(inode->i_mode) &&
-	    !sdp->sd_args.ar_localcaching)
+	    !sdp->sd_args.ar_localcaching) {
+	    	spin_unlock(&inode_lock);
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
+		spin_lock(&inode_lock);
+	}
 	generic_drop_inode(inode);
 }
 

You can't do this in this particular way because you must set the inode's state before releasing the inode_lock. That is done in generic_drop_inode() and the functions that it calls.

Not to mention which, this doesn't seem to make any sense. It appears to be trying to write the inode's data back to disk, but if we've got to this point in the inode's life cycle, there should be nothing to write.

Quoted 9 lines
Quoted 5 lines

Since its only called when i_nlink is 0, thats even more strange since
(a) it could be moved to ->delete_inode() and (b) why do we care about forcing out dirty data for an inode thats being unlinked anyway....

This patch raises more questions than it answers I think.

Do you imply it is superfluous to call gfs_sync_page_i at all, and in consequence there's no need for a gfs-specific ->drop_inode function?

Probably there is no need for it. Its not at all obvious what the gfs_sync_page_i was supposed to do, so its tricky to see that its not needed at all. I think we can be fairly certain that it shouldn't be in ->drop_inode() though, and that if it is needed, it can go into ->delete_inode()

Quoted 6 lines

Just for the record: without unlocking/locking, the node having a mailmain queue over GFS got frozen in a few seconds as mailman queue manager was started. With the patch above it ran smoothly for a longer (test) period.

Best regards,
Jozsef

Yes, it doesn't surprise me that you'd see lockups without your patch. To be on the safe side, try moving the call to gfs_sync_page_i into ->delete_inode so that you can do it after the inode lock has already been dropped (and after the state has been set correctly too). It won't harm anything to have that around, but it might slow things down a bit,

Steve.

#5 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Quoted 5 lines · Steven Whitehouse

Yes, it doesn't surprise me that you'd see lockups without your patch. To be on the safe side, try moving the call to gfs_sync_page_i into ->delete_inode so that you can do it after the inode lock has already been dropped (and after the state has been set correctly too). It won't harm anything to have that around, but it might slow things down a bit,

What do you think about this patch?

ops_super.c +6 -6
@@ -49,7 +49,7 @@
 }
 
 /**
- * gfs_drop_inode - drop an inode
+ * gfs_delete_inode - delete an inode
  * @inode: The inode
  *
  * If i_nlink is zero, any dirty data for the inode is thrown away.
@@ -58,19 +58,19 @@
  */
 
 static void
-gfs_drop_inode(struct inode *inode)
+gfs_delete_inode(struct inode *inode)
 {
 	struct gfs_sbd *sdp = get_v2sdp(inode->i_sb);
 	struct gfs_inode *ip = get_v2ip(inode);
 
-	atomic_inc(&sdp->sd_ops_super);
-
 	if (ip &&
 	    !inode->i_nlink &&
 	    S_ISREG(inode->i_mode) &&
 	    !sdp->sd_args.ar_localcaching)
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);
-	generic_drop_inode(inode);
+
+	truncate_inode_pages(&inode->i_data, 0);
+	clear_inode(inode);
 }
 
 /**
@@ -443,7 +443,7 @@
 
 struct super_operations gfs_super_ops = {
 	.write_inode = gfs_write_inode,
-	.drop_inode = gfs_drop_inode,
+	.delete_inode = gfs_delete_inode,
 	.put_super = gfs_put_super,
 	.write_super = gfs_write_super,
 	.write_super_lockfs = gfs_write_super_lockfs,

I'll be able to test it (or it's successor) around the next week.

Best regards,
Jozsef
--
E-mail : kadlec at mail.kfki.hu, kadlec at blackhole.kfki.hu PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics

         H-1525 Budapest 114, POB. 49, Hungary
#6 lore raw

PATCH 1/1 Fix freeze of cluster-2.03.11

Hi,

Quoted 25 lines · Kadlecsik Jozsef
Quoted 5 lines · Steven Whitehouse

Yes, it doesn't surprise me that you'd see lockups without your patch. To be on the safe side, try moving the call to gfs_sync_page_i into ->delete_inode so that you can do it after the inode lock has already been dropped (and after the state has been set correctly too). It won't harm anything to have that around, but it might slow things down a bit,

What do you think about this patch?

ops_super.c +1 -1
@@ -49,7 +49,7 @@
 }
 
 /**
- * gfs_drop_inode - drop an inode
+ * gfs_delete_inode - delete an inode
  * @inode: The inode
  *
  * If i_nlink is zero, any dirty data for the inode is thrown away.
@@ -58,19 +58,19 @@
  */
 
 static void

ops_super.c line 67

Quoted 10 lines
ops_super.c +1 -3
-gfs_drop_inode(struct inode *inode)
+gfs_delete_inode(struct inode *inode)
 {
 	struct gfs_sbd *sdp = get_v2sdp(inode->i_sb);
 	struct gfs_inode *ip = get_v2ip(inode);
 
-	atomic_inc(&sdp->sd_ops_super);
-
 	if (ip &&
 	    !inode->i_nlink &&

That looks much better, but you don't need to test for !inode->i_nlink as this function only gets called if that is the case.

ops_super.c line 70

Quoted 3 lines
ops_super.c +0 -0
 	    S_ISREG(inode->i_mode) &&
 	    !sdp->sd_args.ar_localcaching)
 		gfs_sync_page_i(inode, DIO_START | DIO_WAIT);

I still wonder whether gfs2_sync_page_i is really needed here at all, but its probably safer to keep it for now I guess.

Quoted 17 lines
ops_super.c +4 -2
-	generic_drop_inode(inode);
+
+	truncate_inode_pages(&inode->i_data, 0);
+	clear_inode(inode);
 }
 
 /**
@@ -443,7 +443,7 @@
 
 struct super_operations gfs_super_ops = {
 	.write_inode = gfs_write_inode,
-	.drop_inode = gfs_drop_inode,
+	.delete_inode = gfs_delete_inode,
 	.put_super = gfs_put_super,
 	.write_super = gfs_write_super,
 	.write_super_lockfs = gfs_write_super_lockfs,
Quoted 4 lines

I'll be able to test it (or it's successor) around the next week.

Best regards,
Jozsef

Ok, if we here some positive test results, then we can put this in. Thanks,

Steve.