Skip to contentSkip to conversation

quota: Use s_umount for quota on/off serialization

PATCH In review Jan Kara 3 participants 11 messages Nov 24, 2016 – Nov 30, 2016

Subsystems (3)
lore
Share & tools
lore

Jan Kara

author reviewer
#0 lore raw

cover quota: Use s_umount for quota on/off serialization

Hello,

this patch set changes quota code to use s_umount semaphore for serialization of quota on/off operations among each other and with other quotactl and quota writeback operations. So far we have used dedicated dqonoff_mutex but that triggered lockdep warnings during fs freezing and also unnecessarily serialized some quotactl operations.

Al, any objections to patch 1/7 exporting functionality to get superblock with s_umount in exclusive mode? Alternatively I could add a wrapper around get_super_thawed() in quota code to drop s_umount & get it in exclusive mode and recheck that superblock didn't get unmounted / frozen but what I did here looked cleaner to me.

OCFS2 guys, it would be good if you could test ocfs2 quotas with this patch set in some multi-node setup (I have tested just with a single node), especially whether quota file recovery for other nodes still works as expected. Thanks.

If nobody objects, I'll push these changes through my tree to Linus.

Honza

author reviewer
#1 lore raw

PATCH 1/7 fs: Provide function to get superblock with exclusive s_umount

Quota code will need a variant of get_super_thawed() that returns superblock with s_umount held in exclusive mode to serialize quota on and quota off operations. Provide this functionality.

---
 fs/super.c         | 80 ++++++++++++++++++++++++++++++++++++++++--------------
 include/linux/fs.h |  2 ++
 2 files changed, 62 insertions(+), 20 deletions(-)
fs/super.c +60 -20
index c183835566c1..f7f724230e2b 100644
@@ -558,6 +558,13 @@ void drop_super(struct super_block *sb)
 
 EXPORT_SYMBOL(drop_super);
 
+void drop_super_exclusive(struct super_block *sb)
+{
+	up_write(&sb->s_umount);
+	put_super(sb);
+}
+EXPORT_SYMBOL(drop_super_exclusive);
+
 /**
  *	iterate_supers - call function for all active superblocks
  *	@f: function to call
@@ -628,15 +635,7 @@ void iterate_supers_type(struct file_system_type *type,
 
 EXPORT_SYMBOL(iterate_supers_type);
 
-/**
- *	get_super - get the superblock of a device
- *	@bdev: device to get the superblock for
- *	
- *	Scans the superblock list and finds the superblock of the file system
- *	mounted on the device given. %NULL is returned if no match is found.
- */
-
-struct super_block *get_super(struct block_device *bdev)
+static struct super_block *__get_super(struct block_device *bdev, bool excl)
 {
 	struct super_block *sb;
 
@@ -651,11 +650,17 @@ struct super_block *get_super(struct block_device *bdev)
 		if (sb->s_bdev == bdev) {
 			sb->s_count++;
 			spin_unlock(&sb_lock);
-			down_read(&sb->s_umount);
+			if (!excl)
+				down_read(&sb->s_umount);
+			else
+				down_write(&sb->s_umount);
 			/* still alive? */
 			if (sb->s_root && (sb->s_flags & MS_BORN))
 				return sb;
-			up_read(&sb->s_umount);
+			if (!excl)
+				up_read(&sb->s_umount);
+			else
+				up_write(&sb->s_umount);
 			/* nope, got unmounted */
 			spin_lock(&sb_lock);
 			__put_super(sb);
@@ -666,32 +671,67 @@ struct super_block *get_super(struct block_device *bdev)
 	return NULL;
 }
 
-EXPORT_SYMBOL(get_super);
-
 /**
- *	get_super_thawed - get thawed superblock of a device
+ *	get_super - get the superblock of a device
  *	@bdev: device to get the superblock for
  *
  *	Scans the superblock list and finds the superblock of the file system
- *	mounted on the device. The superblock is returned once it is thawed
- *	(or immediately if it was not frozen). %NULL is returned if no match
- *	is found.
+ *	mounted on the device given. %NULL is returned if no match is found.
  */
-struct super_block *get_super_thawed(struct block_device *bdev)
+struct super_block *get_super(struct block_device *bdev)
+{
+	return __get_super(bdev, false);
+}
+EXPORT_SYMBOL(get_super);
+
+static struct super_block *__get_super_thawed(struct block_device *bdev,
+					      bool excl)
 {
 	while (1) {
-		struct super_block *s = get_super(bdev);
+		struct super_block *s = __get_super(bdev, excl);
 		if (!s || s->s_writers.frozen == SB_UNFROZEN)
 			return s;
-		up_read(&s->s_umount);
+		if (!excl)
+			up_read(&s->s_umount);
+		else
+			up_write(&s->s_umount);
 		wait_event(s->s_writers.wait_unfrozen,
 			   s->s_writers.frozen == SB_UNFROZEN);
 		put_super(s);
 	}
 }
+
+/**
+ *	get_super_thawed - get thawed superblock of a device
+ *	@bdev: device to get the superblock for
+ *
+ *	Scans the superblock list and finds the superblock of the file system
+ *	mounted on the device. The superblock is returned once it is thawed
+ *	(or immediately if it was not frozen). %NULL is returned if no match
+ *	is found.
+ */
+struct super_block *get_super_thawed(struct block_device *bdev)
+{
+	return __get_super_thawed(bdev, false);
+}
 EXPORT_SYMBOL(get_super_thawed);
 
 /**
+ *	get_super_exclusive_thawed - get thawed superblock of a device
+ *	@bdev: device to get the superblock for
+ *
+ *	Scans the superblock list and finds the superblock of the file system
+ *	mounted on the device. The superblock is returned once it is thawed
+ *	(or immediately if it was not frozen) and s_umount semaphore is held
+ *	in exclusive mode. %NULL is returned if no match is found.
+ */
+struct super_block *get_super_exclusive_thawed(struct block_device *bdev)
+{
+	return __get_super_thawed(bdev, true);
+}
+EXPORT_SYMBOL(get_super_exclusive_thawed);
+
+/**
  * get_active_super - get an active reference to the superblock of a device
  * @bdev: device to get the superblock for
  *
include/linux/fs.h +2 -0
index dc0478c07b2a..d04cfdefcd11 100644
@@ -2949,8 +2949,10 @@ extern void put_filesystem(struct file_system_type *fs);
 extern struct file_system_type *get_fs_type(const char *name);
 extern struct super_block *get_super(struct block_device *);
 extern struct super_block *get_super_thawed(struct block_device *);
+extern struct super_block *get_super_exclusive_thawed(struct block_device *bdev);
 extern struct super_block *get_active_super(struct block_device *bdev);
 extern void drop_super(struct super_block *sb);
+extern void drop_super_exclusive(struct super_block *sb);
 extern void iterate_supers(void (*)(struct super_block *, void *), void *);
 extern void iterate_supers_type(struct file_system_type *,
 			        void (*)(struct super_block *, void *), void *);
Signature · 1 line
2.6.6
author reviewer
#2 lore raw

PATCH 2/7 quota: Hold s_umount in exclusive mode when enabling / disabling quotas

In reply to Jan Kara

Currently we hold s_umount semaphore only in shared mode when enabling or disabling quotas and use dqonoff_mutex for serializing quota state changes on a filesystem and also quota state changes with other places depending on current quota state. Using dedicated mutex for this causes possible deadlocks during filesystem freezing (see following commit for details) so we transition to using s_umount semaphore for the necessary synchronization whose lock ordering is properly handled by the filesystem freezing code. As a start grab s_umount in exclusive mode when enabling / disabling quotas.

---
 fs/quota/dquot.c | 11 +++++++++++
 fs/quota/quota.c | 15 +++++++++++++--
 2 files changed, 24 insertions(+), 2 deletions(-)
fs/quota/dquot.c +11 -0
index 1bfac28b7e7d..047afb966420 100644
@@ -2107,6 +2107,10 @@ int dquot_disable(struct super_block *sb, int type, unsigned int flags)
 	struct quota_info *dqopt = sb_dqopt(sb);
 	struct inode *toputinode[MAXQUOTAS];
 
+	/* s_umount should be held in exclusive mode */
+	if (WARN_ON_ONCE(down_read_trylock(&sb->s_umount)))
+		up_read(&sb->s_umount);
+
 	/* Cannot turn off usage accounting without turning off limits, or
 	 * suspend quotas and simultaneously turn quotas off. */
 	if ((flags & DQUOT_USAGE_ENABLED && !(flags & DQUOT_LIMITS_ENABLED))
@@ -2371,6 +2375,10 @@ int dquot_resume(struct super_block *sb, int type)
 	int ret = 0, cnt;
 	unsigned int flags;
 
+	/* s_umount should be held in exclusive mode */
+	if (WARN_ON_ONCE(down_read_trylock(&sb->s_umount)))
+		up_read(&sb->s_umount);
+
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		if (type != -1 && cnt != type)
 			continue;
@@ -2430,6 +2438,9 @@ int dquot_enable(struct inode *inode, int type, int format_id,
 
 	/* Just unsuspend quotas? */
 	BUG_ON(flags & DQUOT_SUSPENDED);
+	/* s_umount should be held in exclusive mode */
+	if (WARN_ON_ONCE(down_read_trylock(&sb->s_umount)))
+		up_read(&sb->s_umount);
 
 	if (!flags)
 		return 0;
fs/quota/quota.c +13 -2
index 2d445425aad7..790c61abc663 100644
@@ -790,6 +790,12 @@ static int quotactl_cmd_write(int cmd)
 	return 1;
 }
 
+/* Return true if quotactl command is manipulating quota on/off state */
+static bool quotactl_cmd_onoff(int cmd)
+{
+	return (cmd == Q_QUOTAON) || (cmd == Q_QUOTAOFF);
+}
+
 #endif /* CONFIG_BLOCK */
 
 /*
@@ -809,7 +815,9 @@ static struct super_block *quotactl_block(const char __user *special, int cmd)
 	putname(tmp);
 	if (IS_ERR(bdev))
 		return ERR_CAST(bdev);
-	if (quotactl_cmd_write(cmd))
+	if (quotactl_cmd_onoff(cmd))
+		sb = get_super_exclusive_thawed(bdev);
+	else if (quotactl_cmd_write(cmd))
 		sb = get_super_thawed(bdev);
 	else
 		sb = get_super(bdev);
@@ -872,7 +880,10 @@ SYSCALL_DEFINE4(quotactl, unsigned int, cmd, const char __user *, special,
 
 	ret = do_quotactl(sb, type, cmds, id, addr, pathp);
 
-	drop_super(sb);
+	if (!quotactl_cmd_onoff(cmds))
+		drop_super(sb);
+	else
+		drop_super_exclusive(sb);
 out:
 	if (pathp && !IS_ERR(pathp))
 		path_put(pathp);
Signature · 1 line
2.6.6
#3 lore raw

Re: [PATCH 2/7] quota: Hold s_umount in exclusive mode when enabling / disabling quotas

url: https://github.com/0day-ci/linux/commits/Jan-Kara/fs-Provide-function-to-get-superblock-with-exclusive-s_umount/20161124-192840 config: x86_64-randconfig-x013-201647 (attached as .config)…

Hi Jan,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.9-rc6 next-20161124]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Jan-Kara/fs-Provide-function-to-get-superblock-with-exclusive-s_umount/20161124-192840
config: x86_64-randconfig-x013-201647 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

fs/quota/quota.c: In function 'SYSC_quotactl':

>> fs/quota/quota.c:883:7: error: implicit declaration of function 'quotactl_cmd_onoff' [-Werror=implicit-function-declaration]
     if (!quotactl_cmd_onoff(cmds))
          ^~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/quotactl_cmd_onoff +883 fs/quota/quota.c

877 ret = PTR_ERR(sb);
878 goto out;
879 }
880
881 ret = do_quotactl(sb, type, cmds, id, addr, pathp);
882

 > 883		if (!quotactl_cmd_onoff(cmds))

884 drop_super(sb);
885 else
886 drop_super_exclusive(sb);

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 26410 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20161124/238f6683/attachment-0001.bin

author reviewer
#4 lore raw

PATCH 3/7 quota: Use s_umount protection for quota operations

In reply to Jan Kara

Writeback quota is protected by s_umount semaphore held for reading because every writeback must be protected by that lock (grabbed either by the generic writeback code or by quotactl handler). Getting next available ID in quota file, querying quota state, setting quota information, getting quota format are all quotactl operations protected by s_umount semaphore held for reading grabbed in quotactl handler.

This also fixes lockdep splat about possible deadlock during filesystem freezing where sync_filesystem() is called with page-faults already blocked but sync_filesystem() calls into dquot_writeback_dquots() which grabs dqonoff_mutex which ranks above i_mutex (vfs_load_quota_inode() grabs i_mutex under dqonoff_mutex) which clearly ranks below page fault freeze protection (e.g. via mmap_sem dependencies). The reported problem is not a real deadlock possibility since during quota on we check whether filesystem freezing is not in progress but still it is good to have this fixed.

---
 fs/quota/dquot.c | 39 ++++++++++-----------------------------
 fs/quota/quota.c |  6 +-----
 2 files changed, 11 insertions(+), 34 deletions(-)
fs/quota/dquot.c +10 -29
index 047afb966420..2a9dc3fb491c 100644
@@ -617,7 +617,8 @@ int dquot_writeback_dquots(struct super_block *sb, int type)
 	int cnt;
 	int err, ret = 0;
 
-	mutex_lock(&dqopt->dqonoff_mutex);
+	WARN_ON_ONCE(!rwsem_is_locked(&sb->s_umount));
+
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		if (type != -1 && cnt != type)
 			continue;
@@ -653,7 +654,6 @@ int dquot_writeback_dquots(struct super_block *sb, int type)
 		    && info_dirty(&dqopt->info[cnt]))
 			sb->dq_op->write_info(sb, cnt);
 	dqstats_inc(DQST_SYNCS);
-	mutex_unlock(&dqopt->dqonoff_mutex);
 
 	return ret;
 }
@@ -683,7 +683,6 @@ int dquot_quota_sync(struct super_block *sb, int type)
 	 * Now when everything is written we can discard the pagecache so
 	 * that userspace sees the changes.
 	 */
-	mutex_lock(&dqopt->dqonoff_mutex);
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		if (type != -1 && cnt != type)
 			continue;
@@ -693,7 +692,6 @@ int dquot_quota_sync(struct super_block *sb, int type)
 		truncate_inode_pages(&dqopt->files[cnt]->i_data, 0);
 		inode_unlock(dqopt->files[cnt]);
 	}
-	mutex_unlock(&dqopt->dqonoff_mutex);
 
 	return 0;
 }
@@ -2050,21 +2048,13 @@ int dquot_get_next_id(struct super_block *sb, struct kqid *qid)
 	struct quota_info *dqopt = sb_dqopt(sb);
 	int err;
 
-	mutex_lock(&dqopt->dqonoff_mutex);
-	if (!sb_has_quota_active(sb, qid->type)) {
-		err = -ESRCH;
-		goto out;
-	}
-	if (!dqopt->ops[qid->type]->get_next_id) {
-		err = -ENOSYS;
-		goto out;
-	}
+	if (!sb_has_quota_active(sb, qid->type))
+		return -ESRCH;
+	if (!dqopt->ops[qid->type]->get_next_id)
+		return -ENOSYS;
 	mutex_lock(&dqopt->dqio_mutex);
 	err = dqopt->ops[qid->type]->get_next_id(sb, qid);
 	mutex_unlock(&dqopt->dqio_mutex);
-out:
-	mutex_unlock(&dqopt->dqonoff_mutex);
-
 	return err;
 }
 EXPORT_SYMBOL(dquot_get_next_id);
@@ -2762,7 +2752,6 @@ int dquot_get_state(struct super_block *sb, struct qc_state *state)
 	struct quota_info *dqopt = sb_dqopt(sb);
 	int type;
   
-	mutex_lock(&sb_dqopt(sb)->dqonoff_mutex);
 	memset(state, 0, sizeof(*state));
 	for (type = 0; type < MAXQUOTAS; type++) {
 		if (!sb_has_quota_active(sb, type))
@@ -2784,7 +2773,6 @@ int dquot_get_state(struct super_block *sb, struct qc_state *state)
 		tstate->nextents = 1;	/* We don't know... */
 		spin_unlock(&dq_data_lock);
 	}
-	mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
 	return 0;
 }
 EXPORT_SYMBOL(dquot_get_state);
@@ -2798,18 +2786,13 @@ int dquot_set_dqinfo(struct super_block *sb, int type, struct qc_info *ii)
 	if ((ii->i_fieldmask & QC_WARNS_MASK) ||
 	    (ii->i_fieldmask & QC_RT_SPC_TIMER))
 		return -EINVAL;
-	mutex_lock(&sb_dqopt(sb)->dqonoff_mutex);
-	if (!sb_has_quota_active(sb, type)) {
-		err = -ESRCH;
-		goto out;
-	}
+	if (!sb_has_quota_active(sb, type))
+		return -ESRCH;
 	mi = sb_dqopt(sb)->info + type;
 	if (ii->i_fieldmask & QC_FLAGS) {
 		if ((ii->i_flags & QCI_ROOT_SQUASH &&
-		     mi->dqi_format->qf_fmt_id != QFMT_VFS_OLD)) {
-			err = -EINVAL;
-			goto out;
-		}
+		     mi->dqi_format->qf_fmt_id != QFMT_VFS_OLD))
+			return -EINVAL;
 	}
 	spin_lock(&dq_data_lock);
 	if (ii->i_fieldmask & QC_SPC_TIMER)
@@ -2826,8 +2809,6 @@ int dquot_set_dqinfo(struct super_block *sb, int type, struct qc_info *ii)
 	mark_info_dirty(sb, type);
 	/* Force write to disk */
 	sb->dq_op->write_info(sb, type);
-out:
-	mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
 	return err;
 }
 EXPORT_SYMBOL(dquot_set_dqinfo);
fs/quota/quota.c +1 -5
index 790c61abc663..0c8041688dcf 100644
@@ -104,13 +104,9 @@ static int quota_getfmt(struct super_block *sb, int type, void __user *addr)
 {
 	__u32 fmt;
 
-	mutex_lock(&sb_dqopt(sb)->dqonoff_mutex);
-	if (!sb_has_quota_active(sb, type)) {
-		mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
+	if (!sb_has_quota_active(sb, type))
 		return -ESRCH;
-	}
 	fmt = sb_dqopt(sb)->info[type].dqi_format->qf_fmt_id;
-	mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
 	if (copy_to_user(addr, &fmt, sizeof(fmt)))
 		return -EFAULT;
 	return 0;
Signature · 1 line
2.6.6
author reviewer
#5 lore raw

PATCH 4/7 ocfs2: Protect periodic quota syncing with s_umount semaphore

In reply to Jan Kara

New quota locking rules will require s_umount semaphore for all quota scanning functions. Add is for periodic quota syncing.

---
 fs/ocfs2/quota_global.c | 10 +++++++++-
 fs/ocfs2/super.c        |  1 -
 2 files changed, 9 insertions(+), 2 deletions(-)
fs/ocfs2/quota_global.c +9 -1
index 87e577a49b0d..cec495a921e3 100644
@@ -634,7 +634,15 @@ static void qsync_work_fn(struct work_struct *work)
 						      dqi_sync_work.work);
 	struct super_block *sb = oinfo->dqi_gqinode->i_sb;
 
-	dquot_scan_active(sb, ocfs2_sync_dquot_helper, oinfo->dqi_type);
+	/*
+	 * We have to be careful here not to deadlock on s_umount as umount
+	 * disabling quotas may be in progress and it waits for this work to
+	 * complete. If trylock fails, we'll do the sync next time...
+	 */
+	if (down_read_trylock(&sb->s_umount)) {
+		dquot_scan_active(sb, ocfs2_sync_dquot_helper, oinfo->dqi_type);
+		up_read(&sb->s_umount);
+	}
 	schedule_delayed_work(&oinfo->dqi_sync_work,
 			      msecs_to_jiffies(oinfo->dqi_syncms));
 }
fs/ocfs2/super.c +0 -1
index f56fe39fab04..5b9c0dfdb541 100644
@@ -985,7 +985,6 @@ static void ocfs2_disable_quotas(struct ocfs2_super *osb)
 	for (type = 0; type < OCFS2_MAXQUOTAS; type++) {
 		if (!sb_has_quota_loaded(sb, type))
 			continue;
-		/* Cancel periodic syncing before we grab dqonoff_mutex */
 		oinfo = sb_dqinfo(sb, type)->dqi_priv;
 		cancel_delayed_work_sync(&oinfo->dqi_sync_work);
 		inode = igrab(sb->s_dquot.files[type]);
Signature · 1 line
2.6.6
author reviewer
#6 lore raw

PATCH 5/7 quota: Remove dqonoff_mutex from dquot_scan_active()

In reply to Jan Kara

All callers of dquot_scan_active() now hold s_umount so we can rely on that lock to protect us against quota state changes.

---
 fs/quota/dquot.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/quota/dquot.c +2 -2
index 2a9dc3fb491c..d91aecc939c9 100644
@@ -572,7 +572,8 @@ int dquot_scan_active(struct super_block *sb,
 	struct dquot *dquot, *old_dquot = NULL;
 	int ret = 0;
 
-	mutex_lock(&sb_dqopt(sb)->dqonoff_mutex);
+	WARN_ON_ONCE(!rwsem_is_locked(&sb->s_umount));
+
 	spin_lock(&dq_list_lock);
 	list_for_each_entry(dquot, &inuse_list, dq_inuse) {
 		if (!test_bit(DQ_ACTIVE_B, &dquot->dq_flags))
@@ -603,7 +604,6 @@ int dquot_scan_active(struct super_block *sb,
 	spin_unlock(&dq_list_lock);
 out:
 	dqput(old_dquot);
-	mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
 	return ret;
 }
 EXPORT_SYMBOL(dquot_scan_active);
Signature · 1 line
2.6.6
author reviewer
#7 lore raw

PATCH 6/7 ocfs2: Use s_umount for quota recovery protection

In reply to Jan Kara

Currently we use dqonoff_mutex to serialize quota recovery protection and turning of quotas on / off. Use s_umount semaphore instead.

---
 fs/ocfs2/quota_local.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
fs/ocfs2/quota_local.c +7 -4
index 8a54fd8a4fa5..32c5a40c1257 100644
@@ -454,7 +454,7 @@ struct ocfs2_quota_recovery *ocfs2_begin_quota_recovery(
 /* Sync changes in local quota file into global quota file and
  * reinitialize local quota file.
  * The function expects local quota file to be already locked and
- * dqonoff_mutex locked. */
+ * s_umount locked in shared mode. */
 static int ocfs2_recover_local_quota_file(struct inode *lqinode,
 					  int type,
 					  struct ocfs2_quota_recovery *rec)
@@ -597,7 +597,7 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,
 	printk(KERN_NOTICE "ocfs2: Finishing quota recovery on device (%s) for "
 	       "slot %u\n", osb->dev_str, slot_num);
 
-	mutex_lock(&sb_dqopt(sb)->dqonoff_mutex);
+	down_read(&sb->s_umount);
 	for (type = 0; type < OCFS2_MAXQUOTAS; type++) {
 		if (list_empty(&(rec->r_list[type])))
 			continue;
@@ -674,7 +674,7 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,
 			break;
 	}
 out:
-	mutex_unlock(&sb_dqopt(sb)->dqonoff_mutex);
+	up_read(&sb->s_umount);
 	kfree(rec);
 	return status;
 }
@@ -840,7 +840,10 @@ static int ocfs2_local_free_info(struct super_block *sb, int type)
 	}
 	ocfs2_release_local_quota_bitmaps(&oinfo->dqi_chunk);
 
-	/* dqonoff_mutex protects us against racing with recovery thread... */
+	/*
+	 * s_umount held in exclusive mode protects us against racing with
+	 * recovery thread...
+	 */
 	if (oinfo->dqi_rec) {
 		ocfs2_free_quota_recovery(oinfo->dqi_rec);
 		mark_clean = 0;
Signature · 1 line
2.6.6
author reviewer
#8 lore raw

PATCH 7/7 quota: Remove dqonoff_mutex

In reply to Jan Kara

The only places that were grabbing dqonoff_mutex are functions turning quotas on and off and these are properly serialized using s_umount semaphore. Remove dqonoff_mutex.

---
 fs/quota/dquot.c      | 78 +++++++++++++++------------------------------------
 fs/super.c            |  1 -
 include/linux/quota.h |  1 -
 3 files changed, 23 insertions(+), 57 deletions(-)
fs/quota/dquot.c +23 -55
index d91aecc939c9..0d9dccac39a8 100644
@@ -119,8 +119,7 @@
  * spinlock to internal buffers before writing.
  *
  * Lock ordering (including related VFS locks) is the following:
- *   dqonoff_mutex > i_mutex > journal_lock > dquot->dq_lock > dqio_mutex
- * dqonoff_mutex > i_mutex comes from dquot_quota_sync, dquot_enable, etc.
+ *   s_umount > i_mutex > journal_lock > dquot->dq_lock > dqio_mutex
  */
 
 static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dq_list_lock);
@@ -933,7 +932,7 @@ static int dqinit_needed(struct inode *inode, int type)
 	return 0;
 }
 
-/* This routine is guarded by dqonoff_mutex mutex */
+/* This routine is guarded by s_umount semaphore */
 static void add_dquot_ref(struct super_block *sb, int type)
 {
 	struct inode *inode, *old_inode = NULL;
@@ -2108,18 +2107,14 @@ int dquot_disable(struct super_block *sb, int type, unsigned int flags)
 	    DQUOT_USAGE_ENABLED)))
 		return -EINVAL;
 
-	/* We need to serialize quota_off() for device */
-	mutex_lock(&dqopt->dqonoff_mutex);
-
 	/*
 	 * Skip everything if there's nothing to do. We have to do this because
 	 * sometimes we are called when fill_super() failed and calling
 	 * sync_fs() in such cases does no good.
 	 */
-	if (!sb_any_quota_loaded(sb)) {
-		mutex_unlock(&dqopt->dqonoff_mutex);
+	if (!sb_any_quota_loaded(sb))
 		return 0;
-	}
+
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		toputinode[cnt] = NULL;
 		if (type != -1 && cnt != type)
@@ -2173,7 +2168,6 @@ int dquot_disable(struct super_block *sb, int type, unsigned int flags)
 		dqopt->info[cnt].dqi_bgrace = 0;
 		dqopt->ops[cnt] = NULL;
 	}
-	mutex_unlock(&dqopt->dqonoff_mutex);
 
 	/* Skip syncing and setting flags if quota files are hidden */
 	if (dqopt->flags & DQUOT_QUOTA_SYS_FILE)
@@ -2191,19 +2185,13 @@ int dquot_disable(struct super_block *sb, int type, unsigned int flags)
 	 * changes done by userspace on the next quotaon() */
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++)
 		if (toputinode[cnt]) {
-			mutex_lock(&dqopt->dqonoff_mutex);
-			/* If quota was reenabled in the meantime, we have
-			 * nothing to do */
-			if (!sb_has_quota_loaded(sb, cnt)) {
-				inode_lock(toputinode[cnt]);
-				toputinode[cnt]->i_flags &= ~(S_IMMUTABLE |
+			WARN_ON_ONCE(sb_has_quota_loaded(sb, cnt));
+			inode_lock(toputinode[cnt]);
+			toputinode[cnt]->i_flags &= ~(S_IMMUTABLE |
 				  S_NOATIME | S_NOQUOTA);
-				truncate_inode_pages(&toputinode[cnt]->i_data,
-						     0);
-				inode_unlock(toputinode[cnt]);
-				mark_inode_dirty_sync(toputinode[cnt]);
-			}
-			mutex_unlock(&dqopt->dqonoff_mutex);
+			truncate_inode_pages(&toputinode[cnt]->i_data, 0);
+			inode_unlock(toputinode[cnt]);
+			mark_inode_dirty_sync(toputinode[cnt]);
 		}
 	if (sb->s_bdev)
 		invalidate_bdev(sb->s_bdev);
@@ -2275,6 +2263,10 @@ static int vfs_load_quota_inode(struct inode *inode, int type, int format_id,
 		error = -EINVAL;
 		goto out_fmt;
 	}
+	if (sb_has_quota_loaded(sb, type)) {
+		error = -EBUSY;
+		goto out_fmt;
+	}
 
 	if (!(dqopt->flags & DQUOT_QUOTA_SYS_FILE)) {
 		/* As we bypass the pagecache we must now flush all the
@@ -2286,11 +2278,6 @@ static int vfs_load_quota_inode(struct inode *inode, int type, int format_id,
 		sync_filesystem(sb);
 		invalidate_bdev(sb->s_bdev);
 	}
-	mutex_lock(&dqopt->dqonoff_mutex);
-	if (sb_has_quota_loaded(sb, type)) {
-		error = -EBUSY;
-		goto out_lock;
-	}
 
 	if (!(dqopt->flags & DQUOT_QUOTA_SYS_FILE)) {
 		/* We don't want quota and atime on quota files (deadlocks
@@ -2311,7 +2298,7 @@ static int vfs_load_quota_inode(struct inode *inode, int type, int format_id,
 	error = -EIO;
 	dqopt->files[type] = igrab(inode);
 	if (!dqopt->files[type])
-		goto out_lock;
+		goto out_file_flags;
 	error = -EINVAL;
 	if (!fmt->qf_ops->check_quota_file(sb, type))
 		goto out_file_init;
@@ -2334,14 +2321,13 @@ static int vfs_load_quota_inode(struct inode *inode, int type, int format_id,
 	spin_unlock(&dq_state_lock);
 
 	add_dquot_ref(sb, type);
-	mutex_unlock(&dqopt->dqonoff_mutex);
 
 	return 0;
 
 out_file_init:
 	dqopt->files[type] = NULL;
 	iput(inode);
-out_lock:
+out_file_flags:
 	if (oldflags != -1) {
 		inode_lock(inode);
 		/* Set the flags back (in the case of accidental quotaon()
@@ -2350,7 +2336,6 @@ static int vfs_load_quota_inode(struct inode *inode, int type, int format_id,
 		inode->i_flags |= oldflags;
 		inode_unlock(inode);
 	}
-	mutex_unlock(&dqopt->dqonoff_mutex);
 out_fmt:
 	put_quota_format(fmt);
 
@@ -2372,12 +2357,9 @@ int dquot_resume(struct super_block *sb, int type)
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		if (type != -1 && cnt != type)
 			continue;
-
-		mutex_lock(&dqopt->dqonoff_mutex);
-		if (!sb_has_quota_suspended(sb, cnt)) {
-			mutex_unlock(&dqopt->dqonoff_mutex);
+		if (!sb_has_quota_suspended(sb, cnt))
 			continue;
-		}
+
 		inode = dqopt->files[cnt];
 		dqopt->files[cnt] = NULL;
 		spin_lock(&dq_state_lock);
@@ -2386,7 +2368,6 @@ int dquot_resume(struct super_block *sb, int type)
 							cnt);
 		dqopt->flags &= ~dquot_state_flag(DQUOT_STATE_FLAGS, cnt);
 		spin_unlock(&dq_state_lock);
-		mutex_unlock(&dqopt->dqonoff_mutex);
 
 		flags = dquot_generic_flag(flags, cnt);
 		ret = vfs_load_quota_inode(inode, cnt,
@@ -2422,7 +2403,6 @@ EXPORT_SYMBOL(dquot_quota_on);
 int dquot_enable(struct inode *inode, int type, int format_id,
 		 unsigned int flags)
 {
-	int ret = 0;
 	struct super_block *sb = inode->i_sb;
 	struct quota_info *dqopt = sb_dqopt(sb);
 
@@ -2436,28 +2416,16 @@ int dquot_enable(struct inode *inode, int type, int format_id,
 		return 0;
 	/* Just updating flags needed? */
 	if (sb_has_quota_loaded(sb, type)) {
-		mutex_lock(&dqopt->dqonoff_mutex);
-		/* Now do a reliable test... */
-		if (!sb_has_quota_loaded(sb, type)) {
-			mutex_unlock(&dqopt->dqonoff_mutex);
-			goto load_quota;
-		}
 		if (flags & DQUOT_USAGE_ENABLED &&
-		    sb_has_quota_usage_enabled(sb, type)) {
-			ret = -EBUSY;
-			goto out_lock;
-		}
+		    sb_has_quota_usage_enabled(sb, type))
+			return -EBUSY;
 		if (flags & DQUOT_LIMITS_ENABLED &&
-		    sb_has_quota_limits_enabled(sb, type)) {
-			ret = -EBUSY;
-			goto out_lock;
-		}
+		    sb_has_quota_limits_enabled(sb, type))
+			return -EBUSY;
 		spin_lock(&dq_state_lock);
 		sb_dqopt(sb)->flags |= dquot_state_flag(flags, type);
 		spin_unlock(&dq_state_lock);
-out_lock:
-		mutex_unlock(&dqopt->dqonoff_mutex);
-		return ret;
+		return 0;
 	}
 
 load_quota:
fs/super.c +0 -1
index f7f724230e2b..1709ed029a2c 100644
@@ -244,7 +244,6 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
 	mutex_init(&s->s_vfs_rename_mutex);
 	lockdep_set_class(&s->s_vfs_rename_mutex, &type->s_vfs_rename_key);
 	mutex_init(&s->s_dquot.dqio_mutex);
-	mutex_init(&s->s_dquot.dqonoff_mutex);
 	s->s_maxbytes = MAX_NON_LFS;
 	s->s_op = &default_op;
 	s->s_time_gran = 1000000000;
include/linux/quota.h +0 -1
index 55107a8ff887..b281d198ee5b 100644
@@ -520,7 +520,6 @@ static inline void quota_send_warning(struct kqid qid, dev_t dev,
 struct quota_info {
 	unsigned int flags;			/* Flags for diskquotas on this device */
 	struct mutex dqio_mutex;		/* lock device while I/O in progress */
-	struct mutex dqonoff_mutex;		/* Serialize quotaon & quotaoff */
 	struct inode *files[MAXQUOTAS];		/* inodes of quotafiles */
 	struct mem_dqinfo info[MAXQUOTAS];	/* Information for each quota type */
 	const struct quota_format_ops *ops[MAXQUOTAS];	/* Operations for each type */
Signature · 1 line
2.6.6
#9 lore raw

Re: [PATCH 7/7] quota: Remove dqonoff_mutex

url: https://github.com/0day-ci/linux/commits/Jan-Kara/fs-Provide-function-to-get-superblock-with-exclusive-s_umount/20161124-192840 config: x86_64-randconfig-s0-11241854 (attached as .config)…

Hi Jan,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.9-rc6 next-20161124]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Jan-Kara/fs-Provide-function-to-get-superblock-with-exclusive-s_umount/20161124-192840
config: x86_64-randconfig-s0-11241854 (attached as .config)
compiler: gcc-4.4 (Debian 4.4.7-8) 4.4.7 reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All warnings (new ones prefixed by >>):

fs/quota/dquot.c: In function 'dquot_enable':
fs/quota/dquot.c:2431: warning: label 'load_quota' defined but not used

>> fs/quota/dquot.c:2407: warning: unused variable 'dqopt'

fs/quota/dquot.o: warning: objtool: inode_reserved_space()+0x20: function has unreachable instruction

vim +/dqopt +2407 fs/quota/dquot.c

f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2401   * of individual quota flags
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2402   */
287a8095 fs/quota/dquot.c Christoph Hellwig 2010-05-19  2403  int dquot_enable(struct inode *inode, int type, int format_id,
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2404  		 unsigned int flags)
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2405  {
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2406  	struct super_block *sb = inode->i_sb;
f55abc0f fs/dquot.c       Jan Kara          2008-08-20 @2407  	struct quota_info *dqopt = sb_dqopt(sb);
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2408  
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2409  	/* Just unsuspend quotas? */
0f0dd62f fs/quota/dquot.c Christoph Hellwig 2010-05-19  2410  	BUG_ON(flags & DQUOT_SUSPENDED);
a0a83c50 fs/quota/dquot.c Jan Kara          2016-11-24  2411  	/* s_umount should be held in exclusive mode */
a0a83c50 fs/quota/dquot.c Jan Kara          2016-11-24  2412  	if (WARN_ON_ONCE(down_read_trylock(&sb->s_umount)))
a0a83c50 fs/quota/dquot.c Jan Kara          2016-11-24  2413  		up_read(&sb->s_umount);
0f0dd62f fs/quota/dquot.c Christoph Hellwig 2010-05-19  2414  
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2415  	if (!flags)
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2416  		return 0;
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2417  	/* Just updating flags needed? */
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2418  	if (sb_has_quota_loaded(sb, type)) {
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2419  		if (flags & DQUOT_USAGE_ENABLED &&
1d852305 fs/quota/dquot.c Jan Kara          2016-11-24  2420  		    sb_has_quota_usage_enabled(sb, type))
1d852305 fs/quota/dquot.c Jan Kara          2016-11-24  2421  			return -EBUSY;
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2422  		if (flags & DQUOT_LIMITS_ENABLED &&
1d852305 fs/quota/dquot.c Jan Kara          2016-11-24  2423  		    sb_has_quota_limits_enabled(sb, type))
1d852305 fs/quota/dquot.c Jan Kara          2016-11-24  2424  			return -EBUSY;
cc33412f fs/dquot.c       Jan Kara          2009-01-12  2425  		spin_lock(&dq_state_lock);
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2426  		sb_dqopt(sb)->flags |= dquot_state_flag(flags, type);
cc33412f fs/dquot.c       Jan Kara          2009-01-12  2427  		spin_unlock(&dq_state_lock);
1d852305 fs/quota/dquot.c Jan Kara          2016-11-24  2428  		return 0;
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2429  	}
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2430  
f55abc0f fs/dquot.c       Jan Kara          2008-08-20 @2431  load_quota:
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2432  	return vfs_load_quota_inode(inode, type, format_id, flags);
f55abc0f fs/dquot.c       Jan Kara          2008-08-20  2433  }
287a8095 fs/quota/dquot.c Christoph Hellwig 2010-05-19  2434  EXPORT_SYMBOL(dquot_enable);

:::::: The code at line 2407 was first introduced by commit
:::::: f55abc0fb9c3189de3da829adf3220322c0da43e quota: Allow to separately enable quota accounting and enforcing limits

:::::: TO: Jan Kara <[email protected]>
:::::: CC: Mark Fasheh <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/gzip
Size: 29004 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20161124/b7758d62/attachment-0001.bin

Tested-by on the v1 series
#10 lore raw
In reply to Jan Kara

Hello,

Quoted 14 lines · Jan Kara

Hello,

this patch set changes quota code to use s_umount semaphore for serialization of quota on/off operations among each other and with other quotactl and quota writeback operations. So far we have used dedicated dqonoff_mutex but that triggered lockdep warnings during fs freezing and also unnecessarily serialized some quotactl operations.

Al, any objections to patch 1/7 exporting functionality to get superblock with s_umount in exclusive mode? Alternatively I could add a wrapper around get_super_thawed() in quota code to drop s_umount & get it in exclusive mode and recheck that superblock didn't get unmounted / frozen but what I did here looked cleaner to me.

Quoted 3 lines

OCFS2 guys, it would be good if you could test ocfs2 quotas with this patch set in some multi-node setup (I have tested just with a single node), especially whether quota file recovery for other nodes still works as expected. Thanks.

With this patch set, the quota file recovery works well for ocfs2 on multiple nodes.

Thanks,
Eric

Quoted 9 lines

If nobody objects, I'll push these changes through my tree to Linus.

Honza
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html