ocfs2: flush dentry lock drop when sync ocfs2 volume.
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
In commit ea455f8ab68338ba69f5d3362b342c115bea8e13, we move the dentry lock put process into ocfs2_wq. This is OK for most case, but as for umount, it lead to at least 2 bugs. See
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1133 and
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1135. And it happens easily if we have opened a lot of inodes.
For 1135, the reason is that during umount will call generic_shutdown_super and it will do:
1. shrink_dcache_for_umount
2. sync_filesystem.
3. invalidate_inodes.
In shrink_dcache_for_umount, we will drop the dentry, and queue ocfs2_wq for dentry lock put. While in invalidate_inodes we will call invalidate_list which will iterate all the inodes for the sb. The bad thing is that in this function it will call cond_resched_lock(&inode_lock). So if in any case, we are scheduled out and ocfs2_wq is scheduled and drop some inodes, the "next" in invalidate_list will get damaged(have next->next = next). And the invalidate_list will enter dead loop and cause very high cpu.
So the only chance that we can solve this problem is flush dentry put in step 2 of generic_shutdown_super, that is sync_filesystem. And this patch is just adding dentry put flush process in ocfs2_sync_fs.
Jan,
Will dentry put in sync_fs have potential dead lock with quota
lock? If yes, maybe we have to revert that commit which cause this umount problem and find other ways instead.
--- fs/ocfs2/dcache.c | 16 ++++++++++++++++ fs/ocfs2/dcache.h | 1 + fs/ocfs2/super.c | 7 +++++++ 3 files changed, 24 insertions(+), 0 deletions(-)
fs/ocfs2/dcache.c +16 -0
@@ -316,6 +316,22 @@ static DEFINE_SPINLOCK(dentry_list_lock);* this limit so that we don't starve other users of ocfs2_wq. */ #define DL_INODE_DROP_COUNT 64 +void ocfs2_flush_dl_inode_drop (struct ocfs2_super *osb) +{ +struct ocfs2_dentry_lock *dl; + +spin_lock (&dentry_list_lock); +while (osb->dentry_lock_list) { + dl = osb->dentry_lock_list; + osb->dentry_lock_list = dl->dl_next; +spin_unlock (&dentry_list_lock); +iput (dl->dl_inode); +kfree (dl); +spin_lock (&dentry_list_lock); + } +spin_unlock (&dentry_list_lock); +} +/* Drop inode references from dentry locks */ void ocfs2_drop_dl_inodes (struct work_struct *work) {
fs/ocfs2/dcache.h +1 -0
@@ -62,4 +62,5 @@ void ocfs2_dentry_move(struct dentry *dentry, struct dentry *target,extern spinlock_t dentry_attach_lock; +void ocfs2_flush_dl_inode_drop (struct ocfs2_super *osb);#endif /* OCFS2_DCACHE_H */
fs/ocfs2/super.c +7 -0
@@ -384,6 +384,13 @@ static int ocfs2_sync_fs(struct super_block *sb, int wait)if (ocfs2_is_hard_readonly (osb))return -EROFS ; +if (osb->dentry_lock_list) { +if (wait) +ocfs2_flush_dl_inode_drop (osb); +else +queue_work (ocfs2_wq, &osb->dentry_lock_work); + } +if (wait) { status =ocfs2_flush_truncate_log (osb);if (status <0 )
Signature · 1 line
1.6.2.rc2.16.gf474c
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Hi,
Quoted 13 lines · Tao Ma
In commit
ea455f8ab68338ba69f5d3362b342c115bea8e13, we move the dentry lock put process intoocfs2_wq. This is OK for most case, but as for umount, it lead to at least 2 bugs. See
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1133 and
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1135. And it happens easily if we have opened a lot of inodes.For 1135, the reason is that during umount will call
generic_shutdown_superand it will do:
1.shrink_dcache_for_umount
2.sync_filesystem.
3.invalidate_inodes.
Quoted 8 lines
In
shrink_dcache_for_umount, we will drop the dentry, and queueocfs2_wqfor dentry lock put. While ininvalidate_inodeswe will callinvalidate_listwhich will iterate all the inodes for the sb. The bad thing is that in this function it will callcond_resched_lock(&inode_lock). So if in any case, we are scheduled out andocfs2_wqis scheduled and drop some inodes, the "next" ininvalidate_listwill getdamaged(have next->next = next). And theinvalidate_listwill enter dead loop and cause very high cpu.
Aw, well spotted. Sorry for the bug.
Quoted 8 lines
So the only chance that we can solve this problem is flush dentry put in step 2 of
generic_shutdown_super, that issync_filesystem. And this patch is just adding dentry put flush process inocfs2_sync_fs.Jan,
Will dentry put insync_fshave potential dead lock with quota
lock? If yes, maybe we have to revert that commit which cause this umount problem and find other ways instead.
Well, it might be OK but IMHO it's a crude hack. I think the right fix
should be different: OCFS2 should provide it's own .kill_sb function. In that function we should make sure ocfs2_wq stops putting inode references and then flush the list from ocfs2_put_super.
Regarding quota this should be safe as the only lock we hold isumount_sem.
Below is a patch doing what I'd imagine (lightly tested only). Could you
verify whether it fixes the issue you can see?
Honza
Signature · 42 lines
Jan Kara <[email protected]> SUSE Labs, CR --- From f96286432cc05f80e3aabc0c4795c62ffdd37d9a Mon Sep 17 00:00:00 2001 From: Jan Kara <[email protected]> Date: Mon, 20 Jul 2009 12:12:36 +0200 Subject: [PATCH] ocfs2: Fix deadlock on umount In commit ea455f8ab68338ba69f5d3362b342c115bea8e13, we move the dentry lock put process into ocfs2_wq. This is OK for most cases, but as for umount, it lead to at least 2 bugs. See http://oss.oracle.com/bugzilla/show_bug.cgi?id=1133 and http://oss.oracle.com/bugzilla/show_bug.cgi?id=1135. And it happens easily if we have opened a lot of inodes. For 1135, the reason is that during umount will call generic_shutdown_super and it will do: 1. shrink_dcache_for_umount 2. sync_filesystem. 3. invalidate_inodes. In shrink_dcache_for_umount, we will drop the dentry, and queue ocfs2_wq for dentry lock put. While in invalidate_inodes we will call invalidate_list which will iterate all the inodes for the sb. The bad thing is that in this function it will call cond_resched_lock(&inode_lock). So if in any case, we are scheduled out and ocfs2_wq is scheduled and drop some inodes, the "next" in invalidate_list will get damaged(have next->next = next). And the invalidate_list will enter dead loop and cause very high cpu. Fix the problem by making sure that ocfs2_wq does no more putting of inode references when umounting starts and drop all the references from ocfs2_put_super(). Signed-off-by: Jan Kara <[email protected]> --- fs/ocfs2/dcache.c | 35 +++++++++++++++++++++++++++-------- fs/ocfs2/dcache.h | 3 +++ fs/ocfs2/ocfs2.h | 13 +++++++++++++ fs/ocfs2/super.c | 25 ++++++++++++++++++++++--- 4 files changed, 65 insertions(+), 11 deletions(-)
fs/ocfs2/dcache.c +27 -8
@@ -310,22 +310,19 @@ out_attach:return ret; } -static DEFINE_SPINLOCK (dentry_list_lock); +DEFINE_SPINLOCK (dentry_list_lock);/* We limit the number of dentry locks to drop in one go. We have * this limit so that we don't starve other users of ocfs2_wq. */ #define DL_INODE_DROP_COUNT 64/* Drop inode references from dentry locks */ -void ocfs2_drop_dl_inodes (struct work_struct *work) +static void __ocfs2_drop_dl_inodes (struct ocfs2_super *osb,int drop_count) { - struct ocfs2_super *osb = container_of(work, struct ocfs2_super, - dentry_lock_work);struct ocfs2_dentry_lock *dl; -int drop_count =DL_INODE_DROP_COUNT ;spin_lock (&dentry_list_lock); -while (osb->dentry_lock_list && drop_count--) { +while (osb->dentry_lock_list && (drop_count <0 || drop_count--)) { dl = osb->dentry_lock_list; osb->dentry_lock_list = dl->dl_next;spin_unlock (&dentry_list_lock);
@@ -333,11 +330,32 @@ void ocfs2_drop_dl_inodes(struct work_struct *work)kfree (dl);spin_lock (&dentry_list_lock); } -if (osb->dentry_lock_list) +spin_unlock (&dentry_list_lock); +} + +void ocfs2_drop_dl_inodes (struct work_struct *work) +{ + struct ocfs2_super *osb = container_of(work, struct ocfs2_super, + dentry_lock_work); + +__ocfs2_drop_dl_inodes (osb,DL_INODE_DROP_COUNT ); +/* +* Don't queue dropping if umount is in progress. We flush the +* list in ocfs2_dismount_volume +*/ +spin_lock (&dentry_list_lock); +if (osb->dentry_lock_list && + !ocfs2_test_osb_flag (osb,OCFS2_OSB_UMOUNT ))queue_work (ocfs2_wq, &osb->dentry_lock_work);spin_unlock (&dentry_list_lock); } +/* Flush the whole work queue */ +void ocfs2_drop_all_dl_inodes (struct ocfs2_super *osb) +{ +__ocfs2_drop_dl_inodes (osb,-1 ); +} +/* * ocfs2_dentry_iput() and friends. *
@@ -368,7 +386,8 @@ static void ocfs2_drop_dentry_lock(struct ocfs2_super *osb,/* We leave dropping of inode reference to ocfs2_wq as that can * possibly lead to inode deletion which gets tricky */ spin_lock (&dentry_list_lock); -if (!osb->dentry_lock_list) +if (!osb->dentry_lock_list && + !ocfs2_test_osb_flag (osb,OCFS2_OSB_UMOUNT ))queue_work (ocfs2_wq, &osb->dentry_lock_work); dl->dl_next = osb->dentry_lock_list; osb->dentry_lock_list = dl;
fs/ocfs2/dcache.h +3 -0
@@ -49,10 +49,13 @@ struct ocfs2_dentry_lock {int ocfs2_dentry_attach_lock (struct dentry *dentry,struct inode *inode,u64 parent_blkno); +extern spinlock_t dentry_list_lock; +void ocfs2_dentry_lock_put (struct ocfs2_super *osb,struct ocfs2_dentry_lock *dl);void ocfs2_drop_dl_inodes (struct work_struct *work); +void ocfs2_drop_all_dl_inodes (struct ocfs2_super *osb);struct dentry *ocfs2_find_local_alias (struct inode *inode,u64 parent_blkno,int skip_unhashed);
fs/ocfs2/ocfs2.h +13 -0
@@ -227,6 +227,7 @@ enum ocfs2_mount_options#define OCFS2_OSB_SOFT_RO 0x0001#define OCFS2_OSB_HARD_RO 0x0002#define OCFS2_OSB_ERROR_FS 0x0004 +#define OCFS2_OSB_UMOUNT 0x0008#define OCFS2_DEFAULT_ATIME_QUANTUM 60struct ocfs2_journal ;
@@ -490,6 +491,18 @@ static inline void ocfs2_set_osb_flag(struct ocfs2_super *osb,spin_unlock (&osb->osb_lock); } + +static inline unsigned long ocfs2_test_osb_flag (struct ocfs2_super *osb, +unsigned long flag) +{ +unsigned long ret; + +spin_lock (&osb->osb_lock); + ret = osb->osb_flags & flag; +spin_unlock (&osb->osb_lock); +return ret; +} +static inline void ocfs2_set_ro_flag (struct ocfs2_super *osb,int hard) {
fs/ocfs2/super.c +22 -3
@@ -1213,14 +1213,27 @@ static int ocfs2_get_sb(struct file_system_type *fs_type, mnt); } +static void ocfs2_kill_sb (struct super_block *sb) +{ +struct ocfs2_super *osb =OCFS2_SB (sb); + +/* Prevent further queueing of inode drop events */ +spin_lock (&dentry_list_lock); +ocfs2_set_osb_flag (osb,OCFS2_OSB_UMOUNT ); +spin_unlock (&dentry_list_lock); +/* Wait for work to finish and/or remove it */ +cancel_work_sync (&osb->dentry_lock_work); + +kill_block_super (sb); +} +static struct file_system_type ocfs2_fs_type = { .owner =THIS_MODULE , .name ="ocfs2" , .get_sb = ocfs2_get_sb,/* is this called when we mount * the fs? */ - .kill_sb = kill_block_super,/* set to the generic one -* right now, but do we -* need to change that? */ + .kill_sb = ocfs2_kill_sb, + .fs_flags =FS_REQUIRES_DEV |FS_RENAME_DOES_D_MOVE , .next =NULL };
@@ -1819,6 +1832,12 @@ static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)debugfs_remove (osb->osb_ctxt); +/* +* Flush inode dropping work queue so that deletes are +* performed while the filesystem is still working +*/ +ocfs2_drop_all_dl_inodes (osb); +/* Orphan scan should be stopped as early as possible */ ocfs2_orphan_scan_stop (osb);
Signature · 1 line
1.6.0.2
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Jan Kara Wrote:
Quoted 38 lines
Hi,
Quoted 13 lines · Tao Ma
In commit
ea455f8ab68338ba69f5d3362b342c115bea8e13, we move the dentry lock put process intoocfs2_wq. This is OK for most case, but as for umount, it lead to at least 2 bugs. See
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1133 and
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1135. And it happens easily if we have opened a lot of inodes.For 1135, the reason is that during umount will call
generic_shutdown_superand it will do:
1.shrink_dcache_for_umount
2.sync_filesystem.
3.invalidate_inodes.Quoted 9 lines
In
shrink_dcache_for_umount, we will drop the dentry, and queueocfs2_wqfor dentry lock put. While ininvalidate_inodeswe will callinvalidate_listwhich will iterate all the inodes for the sb. The bad thing is that in this function it will callcond_resched_lock(&inode_lock). So if in any case, we are scheduled out andocfs2_wqis scheduled and drop some inodes, the "next" ininvalidate_listwill getdamaged(have next->next = next). And theinvalidate_listwill enter dead loop and cause very high cpu.Aw, well spotted. Sorry for the bug.
Quoted 9 lines
So the only chance that we can solve this problem is flush dentry put in step 2 of
generic_shutdown_super, that issync_filesystem. And this patch is just adding dentry put flush process inocfs2_sync_fs.Jan,
Will dentry put insync_fshave potential dead lock with quota
lock? If yes, maybe we have to revert that commit which cause this umount problem and find other ways instead.
Quoted 9 lines
Well, it might be OK but IMHO it's a crude hack. I think the right fix
should be different: OCFS2 should provide it's own .kill_sb function. In that function we should make sureocfs2_wqstops putting inode references and then flush the list fromocfs2_put_super.
Regarding quota this should be safe as the only lock we hold isumount_sem.
Below is a patch doing what I'd imagine (lightly tested only). Could you
verify whether it fixes the issue you can see?
yeah, your patch looks more natural. I will test it in my box. Great thanks.
Regards,
Tao
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
In reply to Jan KaraQuoted 6 lines · Jan Kara
fs/ocfs2/dcache.c +6
-0+void ocfs2_drop_dl_inodes (struct work_struct *work) +{ + struct ocfs2_super *osb = container_of(work, struct ocfs2_super, + dentry_lock_work); + +__ocfs2_drop_dl_inodes (osb,DL_INODE_DROP_COUNT );
fs/ocfs2/dcache.c void ocfs2_drop_dl_inodes(struct work_struct *work) · line 351
Quoted 10 lines
fs/ocfs2/dcache.c +7
-0+/* +* Don't queue dropping if umount is in progress. We flush the +* list in ocfs2_dismount_volume +*/ +spin_lock (&dentry_list_lock); +if (osb->dentry_lock_list && + !ocfs2_test_osb_flag (osb,OCFS2_OSB_UMOUNT ))queue_work (ocfs2_wq, &osb->dentry_lock_work);spin_unlock (&dentry_list_lock); }
The approach is good, but boy, I wish we could use vol_state
instead of a flag. We are now tracking the mount/umount state in how many places?
Joel
Signature · 9 lines
Life's Little Instruction Book #456 "Send your loved one flowers. Think of a reason later." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Quoted 13 lines · Joel Becker
Quoted 12 lines · Jan Kara
fs/ocfs2/dcache.c +12
-0+void ocfs2_drop_dl_inodes (struct work_struct *work) +{ + struct ocfs2_super *osb = container_of(work, struct ocfs2_super, + dentry_lock_work); + +__ocfs2_drop_dl_inodes (osb,DL_INODE_DROP_COUNT ); +/* +* Don't queue dropping if umount is in progress. We flush the +* list in ocfs2_dismount_volume +*/ +spin_lock (&dentry_list_lock); + if (osb->dentry_lock_list &&
Quoted 8 lines
fs/ocfs2/dcache.c void ocfs2_drop_dl_inodes(struct work_struct *work) · line 351
Quoted 4 lines
fs/ocfs2/dcache.c +1
-0+ !ocfs2_test_osb_flag(osb, OCFS2_OSB_UMOUNT))queue_work (ocfs2_wq, &osb->dentry_lock_work);spin_unlock (&dentry_list_lock); }The approach is good, but boy, I wish we could use
vol_state
instead of a flag. We are now tracking the mount/umount state in how many places?
A quick #ocfs2 discussion ensued. Let's rename OCFS2_OSB_UMOUNT
to OCFS2_OSB_DROP_DENTRY_LOCK_IMMED. That flag doesn't mean anything other than the dentry lock behavior, so we'll name it specific to its purpose.
I think the rest of the patch looked good, so please resend with
the renamed flag and a clean patch header and I'll verify and apply.
Joel
Signature · 9 lines
Life's Little Instruction Book #20 "Be forgiving of yourself and others." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Quoted 23 lines · Joel Becker
Quoted 13 lines · Joel Becker
Quoted 12 lines · Jan Kara
fs/ocfs2/dcache.c +12
-0+void ocfs2_drop_dl_inodes (struct work_struct *work) +{ + struct ocfs2_super *osb = container_of(work, struct ocfs2_super, + dentry_lock_work); + +__ocfs2_drop_dl_inodes (osb,DL_INODE_DROP_COUNT ); +/* +* Don't queue dropping if umount is in progress. We flush the +* list in ocfs2_dismount_volume +*/ +spin_lock (&dentry_list_lock); + if (osb->dentry_lock_list &&Quoted 8 lines
fs/ocfs2/dcache.c void ocfs2_drop_dl_inodes(struct work_struct *work) · line 351
Quoted 4 lines
fs/ocfs2/dcache.c +1
-0+ !ocfs2_test_osb_flag(osb, OCFS2_OSB_UMOUNT))queue_work (ocfs2_wq, &osb->dentry_lock_work);spin_unlock (&dentry_list_lock); }The approach is good, but boy, I wish we could use
vol_state
instead of a flag. We are now tracking the mount/umount state in how many places?
Quoted 6 lines
A quick #ocfs2 discussion ensued. Let's rename
OCFS2_OSB_UMOUNT
toOCFS2_OSB_DROP_DENTRY_LOCK_IMMED. That flag doesn't mean anything other than the dentry lock behavior, so we'll name it specific to its purpose.
I think the rest of the patch looked good, so please resend with
the renamed flag and a clean patch header and I'll verify and apply.
OK, attached is updated version of the patch.
BTW, Joel, did you have any chance of looking at the quota fixes I've sent you last week?
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-ocfs2-Fix-deadlock-on-umount.patch Type: text/x-patch Size: 6482 bytes Desc: not available Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20090721/5433e57b/attachment.bin
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Quoted 7 lines · Jan Kara
Quoted 6 lines
A quick #ocfs2 discussion ensued. Let's rename
OCFS2_OSB_UMOUNT
toOCFS2_OSB_DROP_DENTRY_LOCK_IMMED. That flag doesn't mean anything other than the dentry lock behavior, so we'll name it specific to its purpose.
I think the rest of the patch looked good, so please resend with
the renamed flag and a clean patch header and I'll verify and apply.OK, attached is updated version of the patch.
Still says OSB_UMOUNT. Attached the wrong version?
Quoted 2 lines
BTW, Joel, did you have any chance of looking at the quota fixes I've sent you last week?
Not yet, I'm going to try to get to them today. Patch review
requires brain power :-)
Joel
Signature · 9 lines
"The nice thing about egotists is that they don't talk about other
people."
- Lucille S. Harper
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Quoted 10 lines · Joel Becker
Quoted 7 lines · Jan Kara
Quoted 6 lines
A quick #ocfs2 discussion ensued. Let's rename
OCFS2_OSB_UMOUNT
toOCFS2_OSB_DROP_DENTRY_LOCK_IMMED. That flag doesn't mean anything other than the dentry lock behavior, so we'll name it specific to its purpose.
I think the rest of the patch looked good, so please resend with
the renamed flag and a clean patch header and I'll verify and apply.OK, attached is updated version of the patch.
Still says
OSB_UMOUNT. Attached the wrong version?
Drat, you're right. Now it should be the right version...
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-ocfs2-Fix-deadlock-on-umount.patch Type: text/x-patch Size: 6545 bytes Desc: not available Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20090722/954eb5ea/attachment-0001.bin
PATCH 1/1 ocfs2: flush dentry lock drop when sync ocfs2 volume.
Quoted 8 lines · Jan Kara
Quoted 7 lines · Joel Becker
Quoted 6 lines · Jan Kara
Quoted 6 lines
A quick #ocfs2 discussion ensued. Let's rename
OCFS2_OSB_UMOUNT
toOCFS2_OSB_DROP_DENTRY_LOCK_IMMED. That flag doesn't mean anything other than the dentry lock behavior, so we'll name it specific to its purpose.
I think the rest of the patch looked good, so please resend with
the renamed flag and a clean patch header and I'll verify and apply.
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
OK, attached is updated version of the patch.
Still says
OSB_UMOUNT. Attached the wrong version?Drat, you're right. Now it should be the right version...
Looks it. Thanks. This patch is now in the fixes branch of ocfs2.git.
Joel
Signature · 9 lines
"I am working for the time when unqualified blacks, browns, and women join the unqualified men in running our overnment." - Sissy Farenthold Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127