Quota locking fixes for OCFS2
This patch series fixes several issues I've found in OCFS2 quota code when I looked at lockdep warning Joel reported in
http://www.mail-archive.com/ocfs2-devel at oss.oracle.com/msg05269.html
Currently ocfs2 with quotas survives fsstress and fsx runs without oopsing and the first lockdep warning I get is
http://www.mail-archive.com/ocfs2-devel at oss.oracle.com/msg03188.html which is somewhere deep in DLM code and I don't have a clue how to fix it.
The patch set is on top of:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6 for_next
Since it really depends on some of the cleanups I have there we have to somehow resolve how to merge it. Either I can do it after getting your acks or if you wish to wait with the changes until after the merge window, I can merge all the generic quota bits and ocfs2 patches can be pulled into ocfs2 tree after that.
Honza
PATCH 1/7 quota: Refactor dquot_transfer code so that OCFS2 can pass in its references
Currently, __dquot_transfer() acquires its own references of dquot structures that will be put into inode. But for OCFS2, this creates a lock inversion between dq_lock (waited on in dqget) and transaction start (started in ocfs2_setattr). Currently, deadlock is impossible because dq_lock is acquired only during dquot_acquire and dquot_release and we already hold a reference to dquot structures in ocfs2_setattr so neither of these functions can be called while we call dquot_transfer. But this is rather subtle and it is hard to teach lockdep about it. So provide __dquot_transfer function that can be passed dquot references directly. OCFS2 can then pass acquired dquot references directly to __dquot_transfer with proper locking.
--- fs/quota/dquot.c | 61 +++++++++++++++++++-------------------------- include/linux/quotaops.h | 1 + 2 files changed, 27 insertions(+), 35 deletions(-)
fs/quota/dquot.c +26 -35
@@ -1703,16 +1703,19 @@ EXPORT_SYMBOL(dquot_free_inode);/* * Transfer the number of inode and blocks from one diskquota to an other. +* On success, dquot references in transfer_to are consumed and references +* to original dquots that need to be released are placed there. On failure, +* references are kept untouched. * * This operation can block, but only after everything is updated * A transaction must be started when entering this function. +* */ -static int __dquot_transfer (struct inode *inode,qid_t *chid,unsigned long mask) +int __dquot_transfer (struct inode *inode,struct dquot **transfer_to) {qsize_t space, cur_space;qsize_t rsv_space =0 ; -struct dquot *transfer_from[MAXQUOTAS ]; -struct dquot *transfer_to[MAXQUOTAS ]; +struct dquot *transfer_from[MAXQUOTAS ] = {};int cnt, ret =0 ;char warntype_to[MAXQUOTAS ];char warntype_from_inodes[MAXQUOTAS ], warntype_from_space[MAXQUOTAS ];
@@ -1722,19 +1725,12 @@ static int __dquot_transfer(struct inode *inode, qid_t *chid, unsigned long maskif (IS_NOQUOTA (inode))return 0 ;/* Initialize the arrays */ -for (cnt =0 ; cnt <MAXQUOTAS ; cnt++) { - transfer_from[cnt] =NULL ; - transfer_to[cnt] =NULL ; +for (cnt =0 ; cnt <MAXQUOTAS ; cnt++) warntype_to[cnt] =QUOTA_NL_NOWARN ; - } -for (cnt =0 ; cnt <MAXQUOTAS ; cnt++) { -if (mask & (1 << cnt)) - transfer_to[cnt] =dqget (inode->i_sb, chid[cnt], cnt); - }down_write (&sb_dqopt (inode->i_sb)->dqptr_sem);if (IS_NOQUOTA (inode)) {/* File without quota accounting? */ up_write (&sb_dqopt (inode->i_sb)->dqptr_sem); - goto put_all; +return 0 ; }spin_lock (&dq_data_lock); cur_space =inode_get_bytes (inode);
@@ -1786,46 +1782,41 @@ static int __dquot_transfer(struct inode *inode, qid_t *chid, unsigned long maskmark_all_dquot_dirty (transfer_from);mark_all_dquot_dirty (transfer_to); -/* The reference we got is transferred to the inode */ +/* Pass back references to put */ for (cnt =0 ; cnt <MAXQUOTAS ; cnt++) - transfer_to[cnt] =NULL ; -warn_put_all: + transfer_to[cnt] = transfer_from[cnt]; +warn:flush_warnings (transfer_to, warntype_to);flush_warnings (transfer_from, warntype_from_inodes);flush_warnings (transfer_from, warntype_from_space); -put_all: -dqput_all (transfer_from); -dqput_all (transfer_to);return ret; over_quota:spin_unlock (&dq_data_lock);up_write (&sb_dqopt (inode->i_sb)->dqptr_sem); -/* Clear dquot pointers we don't want to dqput() */ -for (cnt =0 ; cnt <MAXQUOTAS ; cnt++) - transfer_from[cnt] =NULL ; - goto warn_put_all; + goto warn; } +EXPORT_SYMBOL (__dquot_transfer);/* Wrapper for transferring ownership of an inode for uid/gid only * Called from FSXXX_setattr() */ int dquot_transfer (struct inode *inode,struct iattr *iattr) { -qid_t chid[MAXQUOTAS ]; -unsigned long mask =0 ; +struct dquot *transfer_to[MAXQUOTAS ] = {}; +struct super_block *sb = inode->i_sb; +int ret; -if (iattr->ia_valid &ATTR_UID && iattr->ia_uid != inode->i_uid) { - mask |=1 <<USRQUOTA ; - chid[USRQUOTA ] = iattr->ia_uid; - } -if (iattr->ia_valid &ATTR_GID && iattr->ia_gid != inode->i_gid) { - mask |=1 <<GRPQUOTA ; - chid[GRPQUOTA ] = iattr->ia_gid; - } -if (sb_any_quota_active (inode->i_sb) && !IS_NOQUOTA (inode)) -return __dquot_transfer (inode, chid, mask); +if (!sb_any_quota_active (sb) ||IS_NOQUOTA (inode)) +return 0 ; -return 0 ; +if (iattr->ia_valid &ATTR_UID && iattr->ia_uid != inode->i_uid) + transfer_to[USRQUOTA ] =dqget (sb, iattr->ia_uid,USRQUOTA ); +if (iattr->ia_valid &ATTR_GID && iattr->ia_gid != inode->i_gid) + transfer_to[GRPQUOTA ] =dqget (sb, iattr->ia_uid,GRPQUOTA ); + + ret =__dquot_transfer (inode, transfer_to); +dqput_all (transfer_to); +return ret; }EXPORT_SYMBOL (dquot_transfer);
include/linux/quotaops.h +1 -0
@@ -76,6 +76,7 @@ int vfs_get_dqblk(struct super_block *sb, int type, qid_t id,int vfs_set_dqblk (struct super_block *sb,int type,qid_t id,struct fs_disk_quota *di); +int __dquot_transfer (struct inode *inode,struct dquot **transfer_to);int dquot_transfer (struct inode *inode,struct iattr *iattr);int vfs_dq_quota_on_remount (struct super_block *sb);
Signature · 1 line
1.6.4.2
PATCH 1/7 quota: Refactor dquot_transfer code so that OCFS2 can pass in its references
Quoted 11 lines · Jan Kara
Currently,
__dquot_transfer()acquires its own references of dquot structures that will be put into inode. But for OCFS2, this creates a lock inversion betweendq_lock(waited on in dqget) and transaction start (started inocfs2_setattr). Currently, deadlock is impossible becausedq_lockis acquired only duringdquot_acquireanddquot_releaseand we already hold a reference to dquot structures inocfs2_setattrso neither of these functions can be called while we calldquot_transfer. But this is rather subtle and it is hard to teach lockdep about it. So provide__dquot_transferfunction that can be passed dquot references directly. OCFS2 can then pass acquired dquot references directly to__dquot_transferwith proper locking.
Quoted 1 line
Signed-off-by Jan Kara <[email protected]>
I like this patch. Very clean solution.
Joel
Signature · 8 lines
"You don't make the poor richer by making the rich poorer." - Sir Winston Churchill Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 2/7 ocfs2: Do not map blocks from local quota file on each write
In reply to Jan KaraThere is no need to map offset of local dquot structure to on disk block in each quota write. It is enough to map it just once and store the physical block number in quota structure in memory. Moreover this simplifies locking as we do not have to take ip_alloc_sem from quota write path.
--- fs/ocfs2/quota.h | 3 +++ fs/ocfs2/quota_global.c | 14 ++++++++++++++ fs/ocfs2/quota_local.c | 28 +++++++++++++++++++--------- 3 files changed, 36 insertions(+), 9 deletions(-)
fs/ocfs2/quota.h +3 -0
@@ -23,6 +23,7 @@struct ocfs2_dquot {struct dquot dq_dquot;/* Generic VFS dquot */ loff_t dq_local_off;/* Offset in the local quota file */ +u64 dq_local_phys_blk;/* Physical block carrying quota structure */ struct ocfs2_quota_chunk *dq_chunk;/* Chunk dquot is in */ unsigned int dq_use_count;/* Number of nodes having reference to this entry in global quota file */ s64 dq_origspace;/* Last globally synced space usage */
@@ -104,6 +105,8 @@ int ocfs2_lock_global_qf(struct ocfs2_mem_dqinfo *oinfo, int ex);void ocfs2_unlock_global_qf (struct ocfs2_mem_dqinfo *oinfo,int ex);int ocfs2_read_quota_block (struct inode *inode,u64 v_block,struct buffer_head **bh); +int ocfs2_read_quota_phys_block (struct inode *inode,u64 p_block, +struct buffer_head **bh);extern const struct dquot_operations ocfs2_quota_operations;extern struct quota_format_type ocfs2_quota_format;
fs/ocfs2/quota_global.c +14 -0
@@ -25,6 +25,7 @@#include "dlmglue.h" #include "uptodate.h" #include "super.h" +#include "buffer_head_io.h" #include "quota.h" static struct workqueue_struct *ocfs2_quota_wq =NULL ;
@@ -137,6 +138,19 @@ int ocfs2_read_quota_block(struct inode *inode, u64 v_block,return rc; } +int ocfs2_read_quota_phys_block (struct inode *inode,u64 p_block, +struct buffer_head **bhp) +{ +int rc; + + *bhp =NULL ; + rc =ocfs2_read_blocks (INODE_CACHE (inode), p_block,1 , bhp,0 , + ocfs2_validate_quota_block); +if (rc) +mlog_errno (rc); +return rc; +} +static int ocfs2_get_quota_block (struct inode *inode,int block,struct buffer_head **bh) {
fs/ocfs2/quota_local.c +19 -9
@@ -870,18 +870,17 @@ static int ocfs2_local_write_dquot(struct dquot *dquot) {struct super_block *sb = dquot->dq_sb;struct ocfs2_dquot *od =OCFS2_DQUOT (dquot); -struct buffer_head *bh =NULL ; +struct buffer_head *bh; +struct inode *lqinode =sb_dqopt (sb)->files[dquot->dq_type];int status; - status =ocfs2_read_quota_block (sb_dqopt (sb)->files[dquot->dq_type], -ol_dqblk_file_block (sb, od->dq_local_off), - &bh); + status =ocfs2_read_quota_phys_block (lqinode, od->dq_local_phys_blk, + &bh);if (status) {mlog_errno (status); goto out; } - status =ocfs2_modify_bh (sb_dqopt (sb)->files[dquot->dq_type], bh, - olq_set_dquot, od); + status =ocfs2_modify_bh (lqinode, bh, olq_set_dquot, od);if (status <0 ) {mlog_errno (status); goto out;
@@ -1219,17 +1218,27 @@ static int ocfs2_create_local_dquot(struct dquot *dquot)struct ocfs2_dquot *od =OCFS2_DQUOT (dquot);int offset;int status; +u64 pcount; +down_write (&OCFS2_I (lqinode)->ip_alloc_sem); chunk =ocfs2_find_free_entry (sb, type, &offset);if (!chunk) { chunk =ocfs2_extend_local_quota_file (sb, type, &offset); -if (IS_ERR (chunk)) -return PTR_ERR (chunk); +if (IS_ERR (chunk)) { + status =PTR_ERR (chunk); + goto out; + } }else if (IS_ERR (chunk)) { -return PTR_ERR (chunk); + status =PTR_ERR (chunk); + goto out; } od->dq_local_off =ol_dqblk_off (sb, chunk->qc_num, offset); od->dq_chunk = chunk; + status =ocfs2_extent_map_get_blocks (lqinode, +ol_dqblk_block (sb, chunk->qc_num, offset), + &od->dq_local_phys_blk, + &pcount, +NULL );/* Initialize dquot structure on disk */ status =ocfs2_local_write_dquot (dquot);
@@ -1246,6 +1255,7 @@ static int ocfs2_create_local_dquot(struct dquot *dquot) goto out; } out: +up_write (&OCFS2_I (lqinode)->ip_alloc_sem);return status; }
Signature · 1 line
1.6.4.2
PATCH 2/7 ocfs2: Do not map blocks from local quota file on each write
Quoted 6 lines · Jan Kara
There is no need to map offset of local dquot structure to on disk block in each quota write. It is enough to map it just once and store the physical block number in quota structure in memory. Moreover this simplifies locking as we do not have to take
ip_alloc_semfrom quota write path.Signed-off-by Jan Kara <[email protected]>
Am I correct that quota files cannot be shrunk in a running
filesystem?
Joel
Signature · 9 lines
"Reader, suppose you were and idiot. And suppose you were a member of Congress. But I repeat myself." - Mark Twain Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 2/7 ocfs2: Do not map blocks from local quota file on each write
Quoted 9 lines · Joel Becker
Quoted 6 lines · Jan Kara
There is no need to map offset of local dquot structure to on disk block in each quota write. It is enough to map it just once and store the physical block number in quota structure in memory. Moreover this simplifies locking as we do not have to take
ip_alloc_semfrom quota write path.Signed-off-by Jan Kara <[email protected]>Am I correct that quota files cannot be shrunk in a running
filesystem?
Yes, quota files are only grown when the filesystem is mounted. Shrinking
is done only by fsck.ocfs2.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 3/7 ocfs2: Avoid unnecessary block mapping when refreshing quota info
In reply to Jan KaraThe position of global quota file info does not change. So do not have to do logical -> physical block translation every time we reread it from disk. Thus we can also avoid taking ip_alloc_sem.
--- fs/ocfs2/dlmglue.c | 3 ++- fs/ocfs2/quota.h | 3 ++- fs/ocfs2/quota_global.c | 15 +++++++++++++++ fs/ocfs2/quota_local.c | 10 +++++----- 4 files changed, 24 insertions(+), 7 deletions(-)
fs/ocfs2/dlmglue.c +2 -1
@@ -3897,7 +3897,8 @@ static int ocfs2_refresh_qinfo(struct ocfs2_mem_dqinfo *oinfo) oinfo->dqi_gi.dqi_free_entry =be32_to_cpu (lvb->lvb_free_entry); }else { - status =ocfs2_read_quota_block (oinfo->dqi_gqinode,0 , &bh); + status =ocfs2_read_quota_phys_block (oinfo->dqi_gqinode, + oinfo->dqi_giblk, &bh);if (status) {mlog_errno (status); goto bail;
fs/ocfs2/quota.h +2 -1
@@ -52,8 +52,9 @@ struct ocfs2_mem_dqinfo {struct ocfs2_lock_res dqi_gqlock;/* Lock protecting quota information structure */ struct buffer_head *dqi_gqi_bh;/* Buffer head with global quota file inode - set only if inode lock is obtained */ int dqi_gqi_count;/* Number of holders of dqi_gqi_bh */ +u64 dqi_giblk;/* Number of block with global information header */ struct buffer_head *dqi_lqi_bh;/* Buffer head with local quota file inode */ -struct buffer_head *dqi_ibh;/* Buffer with information header */ +struct buffer_head *dqi_libh;/* Buffer with local information header */ struct qtree_mem_dqinfo dqi_gi;/* Info about global file */ struct delayed_work dqi_sync_work;/* Work for syncing dquots */ struct ocfs2_quota_recovery *dqi_rec;/* Pointer to recovery
fs/ocfs2/quota_global.c +15 -0
@@ -327,6 +327,7 @@ int ocfs2_global_read_info(struct super_block *sb, int type)struct ocfs2_global_disk_dqinfo dinfo;struct mem_dqinfo *info =sb_dqinfo (sb, type);struct ocfs2_mem_dqinfo *oinfo = info->dqi_priv; +u64 pcount;int status;mlog_entry_void ();
@@ -353,9 +354,19 @@ int ocfs2_global_read_info(struct super_block *sb, int type)mlog_errno (status); goto out_err; } + + status =ocfs2_extent_map_get_blocks (gqinode,0 , &oinfo->dqi_giblk, + &pcount,NULL ); +if (status <0 ) + goto out_unlock; + + status =ocfs2_qinfo_lock (oinfo,0 ); +if (status <0 ) + goto out_unlock; status = sb->s_op->quota_read (sb, type, (char *)&dinfo,sizeof (struct ocfs2_global_disk_dqinfo ),OCFS2_GLOBAL_INFO_OFF ); +ocfs2_qinfo_unlock (oinfo,0 );ocfs2_unlock_global_qf (oinfo,0 );if (status !=sizeof (struct ocfs2_global_disk_dqinfo )) { mlog(ML_ERROR, "Cannot read global quota info (%d).\n",
@@ -382,6 +393,10 @@ int ocfs2_global_read_info(struct super_block *sb, int type) out_err:mlog_exit (status);return status; +out_unlock: +ocfs2_unlock_global_qf (oinfo,0 ); +mlog_errno (status); + goto out_err; }/* Write information to global quota file. Expects exlusive lock on quota
fs/ocfs2/quota_local.c +5 -5
@@ -679,7 +679,7 @@ static int ocfs2_local_read_info(struct super_block *sb, int type)INIT_LIST_HEAD (&oinfo->dqi_chunk); oinfo->dqi_rec =NULL ; oinfo->dqi_lqi_bh =NULL ; - oinfo->dqi_ibh =NULL ; + oinfo->dqi_libh =NULL ; status =ocfs2_global_read_info (sb, type);if (status <0 )
@@ -705,7 +705,7 @@ static int ocfs2_local_read_info(struct super_block *sb, int type) info->dqi_flags =le32_to_cpu (ldinfo->dqi_flags); oinfo->dqi_chunks =le32_to_cpu (ldinfo->dqi_chunks); oinfo->dqi_blocks =le32_to_cpu (ldinfo->dqi_blocks); - oinfo->dqi_ibh = bh; + oinfo->dqi_libh = bh;/* We crashed when using local quota file? */ if (!(info->dqi_flags &OLQF_CLEAN )) {
@@ -767,7 +767,7 @@ static int ocfs2_local_write_info(struct super_block *sb, int type) {struct mem_dqinfo *info =sb_dqinfo (sb, type);struct buffer_head *bh = ((struct ocfs2_mem_dqinfo *)info->dqi_priv) - ->dqi_ibh; + ->dqi_libh;int status; status = ocfs2_modify_bh(sb_dqopt(sb)->files[type], bh, olq_update_info,
@@ -828,7 +828,7 @@ static int ocfs2_local_free_info(struct super_block *sb, int type)/* Mark local file as clean */ info->dqi_flags |=OLQF_CLEAN ; status =ocfs2_modify_bh (sb_dqopt (sb)->files[type], - oinfo->dqi_ibh, + oinfo->dqi_libh, olq_update_info, info);if (status <0 ) {
@@ -838,7 +838,7 @@ static int ocfs2_local_free_info(struct super_block *sb, int type) out:ocfs2_inode_unlock (sb_dqopt (sb)->files[type],1 ); -brelse (oinfo->dqi_ibh); +brelse (oinfo->dqi_libh);brelse (oinfo->dqi_lqi_bh);kfree (oinfo);return 0 ;
Signature · 1 line
1.6.4.2
PATCH 4/7 ocfs2: Fix quota locking
In reply to Jan KaraOCFS2 had two issues with quota locking:
a) When reading dquot from global quota file, we started a transaction while holding dqio_mutex which is prone to deadlocks because other paths to it other way around
b) During ocfs2_sync_dquot we were not protected against concurrent writers on the same node. Because we first copy data to local buffer, a race could happen resulting in old data being written to global quota file and thus causing quota inconsistency after a crash.
c) ip_alloc_sem of quota files was acquired while a transaction is started in ocfs2_quota_write which can deadlock because we first get ip_alloc_sem and then start a transaction when extending quota files.
We fix the problem a) by pulling all necessary code to ocfs2_acquire_dquot and ocfs2_release_dquot. Thus we no longer depend on generic dquot_acquire to do the locking and can force proper lock ordering.
Problems b) and c) are fixed by locking i_mutex and ip_alloc_sem of global quota file in ocfs2_lock_global_qf and removing ip_alloc_sem from ocfs2_quota_read and ocfs2_quota_write.
--- fs/ocfs2/quota.h | 5 +- fs/ocfs2/quota_global.c | 307 ++++++++++++++++++++++++++--------------------- fs/ocfs2/quota_local.c | 88 ++++++-------- 3 files changed, 214 insertions(+), 186 deletions(-)
fs/ocfs2/quota.h +3 -2
@@ -104,10 +104,11 @@ static inline int ocfs2_global_release_dquot(struct dquot *dquot)int ocfs2_lock_global_qf (struct ocfs2_mem_dqinfo *oinfo,int ex);void ocfs2_unlock_global_qf (struct ocfs2_mem_dqinfo *oinfo,int ex); -int ocfs2_read_quota_block (struct inode *inode,u64 v_block, -struct buffer_head **bh); +int ocfs2_validate_quota_block (struct super_block *sb,struct buffer_head *bh);int ocfs2_read_quota_phys_block (struct inode *inode,u64 p_block,struct buffer_head **bh); +int ocfs2_create_local_dquot (struct dquot *dquot); +int ocfs2_local_release_dquot (handle_t *handle,struct dquot *dquot);extern const struct dquot_operations ocfs2_quota_operations;extern struct quota_format_type ocfs2_quota_format;
fs/ocfs2/quota_global.c +171 -136
432 lines not on this page · raw message
fs/ocfs2/quota_local.c +40 -48
149 lines not on this page · raw message
Signature · 1 line
1.6.4.2
PATCH 5/7 ocfs2: Fix estimate of credits needed for quota allocation
In reply to Jan KaraWe were missing reservation of a journal credit for modification of quota file inode when creating new dquot structure in the global quota file.
--- fs/ocfs2/quota_global.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
fs/ocfs2/quota_global.c +3 -2
@@ -470,9 +470,10 @@ static int ocfs2_global_qinit_alloc(struct super_block *sb, int type)static int ocfs2_calc_global_qinit_credits (struct super_block *sb,int type) { -/* We modify all the allocated blocks, tree root, and info block */ +/* We modify all the allocated blocks, tree root, info block and +* the inode */ return (ocfs2_global_qinit_alloc (sb, type) +2 ) * -OCFS2_QUOTA_BLOCK_UPDATE_CREDITS ; +OCFS2_QUOTA_BLOCK_UPDATE_CREDITS +1 ; }/* Sync local information about quota modifications with global quota file.
Signature · 1 line
1.6.4.2
PATCH 6/7 ocfs2: Fix NULL pointer deref when writing local dquot
In reply to Jan Karacommit_dqblk() can write quota info to global file. That is actually a bad thing to do because if we are just modifying local quota file, we are not prepared (do not hold proper locks, do not have transaction credits) to do a modification of the global quota file. So do not use commit_dqblk() and instead call our writing function directly.
--- fs/ocfs2/quota.h | 1 + fs/ocfs2/quota_global.c | 20 ++++++++++---------- fs/ocfs2/quota_local.c | 3 +-- 3 files changed, 12 insertions(+), 12 deletions(-)
fs/ocfs2/quota.h +1 -0
@@ -109,6 +109,7 @@ int ocfs2_read_quota_phys_block(struct inode *inode, u64 p_block, struct buffer_head **bh);int ocfs2_create_local_dquot (struct dquot *dquot);int ocfs2_local_release_dquot (handle_t *handle,struct dquot *dquot); +int ocfs2_local_write_dquot (struct dquot *dquot);extern const struct dquot_operations ocfs2_quota_operations;extern struct quota_format_type ocfs2_quota_format;
fs/ocfs2/quota_global.c +10 -10
@@ -614,14 +614,13 @@ static int ocfs2_sync_dquot_helper(struct dquot *dquot, unsigned long type) }mutex_lock (&sb_dqopt (sb)->dqio_mutex); status =ocfs2_sync_dquot (dquot); -mutex_unlock (&sb_dqopt (sb)->dqio_mutex);if (status <0 )mlog_errno (status);/* We have to write local structure as well... */ -dquot_mark_dquot_dirty (dquot); - status =dquot_commit (dquot); + status =ocfs2_local_write_dquot (dquot);if (status <0 )mlog_errno (status); +mutex_unlock (&sb_dqopt (sb)->dqio_mutex);ocfs2_commit_trans (osb, handle); out_ilock:ocfs2_unlock_global_qf (oinfo,1 );
@@ -660,7 +659,9 @@ static int ocfs2_write_dquot(struct dquot *dquot)mlog_errno (status); goto out; } - status =dquot_commit (dquot); +mutex_lock (&sb_dqopt (dquot->dq_sb)->dqio_mutex); + status =ocfs2_local_write_dquot (dquot); +mutex_unlock (&sb_dqopt (dquot->dq_sb)->dqio_mutex);ocfs2_commit_trans (osb, handle); out:mlog_exit (status);
@@ -833,7 +834,6 @@ static int ocfs2_mark_dquot_dirty(struct dquot *dquot)struct ocfs2_super *osb =OCFS2_SB (sb);mlog_entry ("id=%u, type=%d" , dquot->dq_id, type); -dquot_mark_dquot_dirty (dquot);/* In case user set some limits, sync dquot immediately to global * quota file so that information propagates quicker */
@@ -858,14 +858,14 @@ static int ocfs2_mark_dquot_dirty(struct dquot *dquot) }mutex_lock (&sb_dqopt (sb)->dqio_mutex); status =ocfs2_sync_dquot (dquot); -mutex_unlock (&sb_dqopt (sb)->dqio_mutex);if (status <0 ) {mlog_errno (status); - goto out_trans; + goto out_dlock; }/* Now write updated local dquot structure */ - status =dquot_commit (dquot); -out_trans: + status =ocfs2_local_write_dquot (dquot); +out_dlock: +mutex_unlock (&sb_dqopt (sb)->dqio_mutex);ocfs2_commit_trans (osb, handle); out_ilock:ocfs2_unlock_global_qf (oinfo,1 );
@@ -917,7 +917,7 @@ static void ocfs2_destroy_dquot(struct dquot *dquot) }const struct dquot_operations ocfs2_quota_operations = { - .write_dquot = ocfs2_write_dquot, +/* We never make dquot dirty so .write_dquot is never called */ .acquire_dquot = ocfs2_acquire_dquot, .release_dquot = ocfs2_release_dquot, .mark_dirty = ocfs2_mark_dquot_dirty,
fs/ocfs2/quota_local.c +1 -2
@@ -900,7 +900,7 @@ static void olq_set_dquot(struct buffer_head *bh, void *private) }/* Write dquot to local quota file */ -static int ocfs2_local_write_dquot (struct dquot *dquot) +int ocfs2_local_write_dquot (struct dquot *dquot) {struct super_block *sb = dquot->dq_sb;struct ocfs2_dquot *od =OCFS2_DQUOT (dquot);
@@ -1335,7 +1335,6 @@ static const struct quota_format_ops ocfs2_format_ops = { .read_file_info = ocfs2_local_read_info, .write_file_info = ocfs2_global_write_info, .free_file_info = ocfs2_local_free_info, - .commit_dqblk = ocfs2_local_write_dquot, };struct quota_format_type ocfs2_quota_format = {
Signature · 1 line
1.6.4.2
PATCH 7/7 ocfs2: Use __dquot_transfer to avoid lock inversion
In reply to Jan Karadquot_transfer() acquires own references to dquots via dqget(). Thus it waits for dq_lock which creates a lock inversion because dq_lock ranks above transaction start but transaction is already started in ocfs2_setattr(). Fix the problem by passing own references directly to __dquot_transfer.
--- fs/ocfs2/file.c | 17 +++++------------ 1 files changed, 5 insertions(+), 12 deletions(-)
fs/ocfs2/file.c +5 -12
@@ -946,9 +946,8 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr)struct ocfs2_super *osb =OCFS2_SB (sb);struct buffer_head *bh =NULL ;handle_t *handle =NULL ; -int qtype; -struct dquot *transfer_from[MAXQUOTAS ] = { };struct dquot *transfer_to[MAXQUOTAS ] = { }; +int qtype;mlog_entry ("(0x%p, '%.*s')\n" , dentry, dentry->d_name.len, dentry->d_name.name);
@@ -1032,9 +1031,7 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr) OCFS2_FEATURE_RO_COMPAT_USRQUOTA)) { transfer_to[USRQUOTA ] =dqget (sb, attr->ia_uid,USRQUOTA ); - transfer_from[USRQUOTA ] =dqget (sb, inode->i_uid, -USRQUOTA ); -if (!transfer_to[USRQUOTA ] || !transfer_from[USRQUOTA ]) { +if (!transfer_to[USRQUOTA ]) { status = -ESRCH ; goto bail_unlock; }
@@ -1044,9 +1041,7 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr) OCFS2_FEATURE_RO_COMPAT_GRPQUOTA)) { transfer_to[GRPQUOTA ] =dqget (sb, attr->ia_gid,GRPQUOTA ); - transfer_from[GRPQUOTA ] =dqget (sb, inode->i_gid, -GRPQUOTA ); -if (!transfer_to[GRPQUOTA ] || !transfer_from[GRPQUOTA ]) { +if (!transfer_to[GRPQUOTA ]) { status = -ESRCH ; goto bail_unlock; }
@@ -1058,7 +1053,7 @@ int ocfs2_setattr(struct dentry *dentry, struct iattr *attr)mlog_errno (status); goto bail_unlock; } - status =dquot_transfer (inode, attr); + status =__dquot_transfer (inode, transfer_to);if (status <0 ) goto bail_commit; }else {
@@ -1098,10 +1093,8 @@ bail:brelse (bh);/* Release quota pointers in case we acquired them */ -for (qtype =0 ; qtype <MAXQUOTAS ; qtype++) { +for (qtype =0 ; qtype <MAXQUOTAS ; qtype++)dqput (transfer_to[qtype]); -dqput (transfer_from[qtype]); - }if (!status && attr->ia_valid &ATTR_MODE ) { status =ocfs2_acl_chmod (inode);
Signature · 1 line
1.6.4.2
Quoted 1 line · Jan Kara
Currently ocfs2 with quotas survives fsstress and fsx runs without oopsing
When you say "currently," do you mean mainline ocfs2 or
ocfs2 with these patches? ;-) I'm guessing you mean with these patches.
Quoted 8 lines
The patch set is on top of:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6for_nextSince it really depends on some of the cleanups I have there we have to somehow resolve how to merge it. Either I can do it after getting your acks or if you wish to wait with the changes until after the merge window, I can merge all the generic quota bits and ocfs2 patches can be pulled into ocfs2 tree after that.
I'm fine with Acking this series for you to send with your
cleanups. They're all going in the merge window, right?
Joel
Signature · 8 lines
"There is no sincerer love than the love of food."
- George Bernard Shaw
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Quoted 5 lines · Joel Becker
Quoted 1 line · Jan Kara
Currently ocfs2 with quotas survives fsstress and fsx runs without oopsing
When you say "currently," do you mean mainline ocfs2 or
ocfs2 with these patches? ;-) I'm guessing you mean with these patches.
I meant ocfs2 with these patches :)
Quoted 11 lines
Quoted 8 lines
The patch set is on top of:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6for_nextSince it really depends on some of the cleanups I have there we have to somehow resolve how to merge it. Either I can do it after getting your acks or if you wish to wait with the changes until after the merge window, I can merge all the generic quota bits and ocfs2 patches can be pulled into ocfs2 tree after that.
I'm fine with Acking this series for you to send with your
cleanups. They're all going in the merge window, right?
Yes, I plan to submit them during the merge window...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 11 lines · Jan Kara
Quoted 9 lines · Joel Becker
Quoted 5 lines · Jan Kara
Since it really depends on some of the cleanups I have there we have to somehow resolve how to merge it. Either I can do it after getting your acks or if you wish to wait with the changes until after the merge window, I can merge all the generic quota bits and ocfs2 patches can be pulled into ocfs2 tree after that.
I'm fine with Acking this series for you to send with your
cleanups. They're all going in the merge window, right?Yes, I plan to submit them during the merge window...
Ok, you can add my Acked-by to these and push them to Linus at
merge time.
Joel
Signature · 7 lines
f/8 and be there. Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Quoted 9 lines · Joel Becker
Quoted 8 lines · Jan Kara
Quoted 7 lines · Joel Becker
Quoted 5 lines · Jan Kara
Since it really depends on some of the cleanups I have there we have to somehow resolve how to merge it. Either I can do it after getting your acks or if you wish to wait with the changes until after the merge window, I can merge all the generic quota bits and ocfs2 patches can be pulled into ocfs2 tree after that.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
I'm fine with Acking this series for you to send with your
cleanups. They're all going in the merge window, right?Yes, I plan to submit them during the merge window...
Ok, you can add my Acked-by to these and push them to Linus at
merge time.
OK, will do. In the mean time I have found one more possible deadlock
which should be fixed by the attached patch so I'll add it to the series if you won't object.
Honza
Signature · 9 lines
Jan Kara <[email protected]> SUSE Labs, CR -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-ocfs2-Fix-lock-inversion-in-quotas-during-umount.patch Type: text/x-patch Size: 2089 bytes Desc: not available Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20100517/44d19161/attachment.bin
Quoted 3 lines · Jan Kara
OK, will do. In the mean time I have found one more possible deadlock
which should be fixed by the attached patch so I'll add it to the series if you won't object.
One question: is it possible for additional work to be added
between the time we call the cancel and the time we take the dqonoff_mutex? If it is not possible, ack on the patch and put it in the series.
Joel
Signature · 9 lines
"The real reason GNU ls is 8-bit-clean is so that they can start using ISO-8859-1 option characters." - Christopher Davis (ckd at loiosh.kei.com) Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Quoted 9 lines · Joel Becker
Quoted 3 lines · Jan Kara
OK, will do. In the mean time I have found one more possible deadlock
which should be fixed by the attached patch so I'll add it to the series if you won't object.One question: is it possible for additional work to be added
between the time we call the cancel and the time we take thedqonoff_mutex? If it is not possible, ack on the patch and put it in the series.
It isn't possible. There are only two places that queue work to this
workqueue. The first one is called when quota is being enabled (i.e., during mount) and the second one is called from qsync_work_fn() itself. Now cancel_delayed_work_sync() has an explicit note in the comment before it that "It is possible to use this function if @dwork rearms itself via queue_work() or queue_delayed_work()." so I believe everything is fine.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR