Skip to contentSkip to conversation

f2fs: change to use rwsem for cp_mutex

PATCH In review Sahitya Tummala 3 participants 6 messages Nov 23, 2020 – Nov 25, 2020 F2FS file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 f2fs: change to use rwsem for cp_mutex

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

---
 fs/f2fs/checkpoint.c | 8 ++++----
 fs/f2fs/f2fs.h       | 2 +-
 fs/f2fs/gc.c         | 4 ++--
 fs/f2fs/recovery.c   | 4 ++--
 fs/f2fs/super.c      | 2 +-
 5 files changed, 10 insertions(+), 10 deletions(-)
fs/f2fs/checkpoint.c +4 -4
index 023462e..7fb8c80 100644
@@ -348,13 +348,13 @@ static int f2fs_write_meta_pages(struct address_space *mapping,
 		goto skip_write;
 
 	/* if locked failed, cp will flush dirty pages instead */
-	if (!mutex_trylock(&sbi->cp_mutex))
+	if (!down_write_trylock(&sbi->cp_global_sem))
 		goto skip_write;
 
 	trace_f2fs_writepages(mapping->host, wbc, META);
 	diff = nr_pages_to_write(sbi, META, wbc);
 	written = f2fs_sync_meta_pages(sbi, META, wbc->nr_to_write, FS_META_IO);
-	mutex_unlock(&sbi->cp_mutex);
+	up_write(&sbi->cp_global_sem);
 	wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff);
 	return 0;
 
@@ -1572,7 +1572,7 @@ int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
 		f2fs_warn(sbi, "Start checkpoint disabled!");
 	}
 	if (cpc->reason != CP_RESIZE)
-		mutex_lock(&sbi->cp_mutex);
+		down_write(&sbi->cp_global_sem);
 
 	if (!is_sbi_flag_set(sbi, SBI_IS_DIRTY) &&
 		((cpc->reason & CP_FASTBOOT) || (cpc->reason & CP_SYNC) ||
@@ -1647,7 +1647,7 @@ int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
 	trace_f2fs_write_checkpoint(sbi->sb, cpc->reason, "finish checkpoint");
 out:
 	if (cpc->reason != CP_RESIZE)
-		mutex_unlock(&sbi->cp_mutex);
+		up_write(&sbi->cp_global_sem);
 	return err;
 }
 
fs/f2fs/f2fs.h +1 -1
index e082677..842c2ca 100644
@@ -1369,7 +1369,7 @@ struct f2fs_sb_info {
 	int cur_cp_pack;			/* remain current cp pack */
 	spinlock_t cp_lock;			/* for flag in ckpt */
 	struct inode *meta_inode;		/* cache meta blocks */
-	struct mutex cp_mutex;			/* checkpoint procedure lock */
+	struct rw_semaphore cp_global_sem;	/* checkpoint procedure lock */
 	struct rw_semaphore cp_rwsem;		/* blocking FS operations */
 	struct rw_semaphore node_write;		/* locking node writes */
 	struct rw_semaphore node_change;	/* locking node change */
fs/f2fs/gc.c +2 -2
index 05641a1..3ef84e6 100644
@@ -1986,7 +1986,7 @@ int f2fs_resize_fs(struct f2fs_sb_info *sbi, __u64 block_count)
 
 	freeze_super(sbi->sb);
 	down_write(&sbi->gc_lock);
-	mutex_lock(&sbi->cp_mutex);
+	down_write(&sbi->cp_global_sem);
 
 	spin_lock(&sbi->stat_lock);
 	if (shrunk_blocks + valid_user_blocks(sbi) +
@@ -2031,7 +2031,7 @@ int f2fs_resize_fs(struct f2fs_sb_info *sbi, __u64 block_count)
 		spin_unlock(&sbi->stat_lock);
 	}
 out_err:
-	mutex_unlock(&sbi->cp_mutex);
+	up_write(&sbi->cp_global_sem);
 	up_write(&sbi->gc_lock);
 	thaw_super(sbi->sb);
 	clear_sbi_flag(sbi, SBI_IS_RESIZEFS);
fs/f2fs/recovery.c +2 -2
index 0947d36..da75d5d 100644
@@ -799,7 +799,7 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
 	INIT_LIST_HEAD(&dir_list);
 
 	/* prevent checkpoint */
-	mutex_lock(&sbi->cp_mutex);
+	down_write(&sbi->cp_global_sem);
 
 	/* step #1: find fsynced inode numbers */
 	err = find_fsync_dnodes(sbi, &inode_list, check_only);
@@ -850,7 +850,7 @@ int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
 	if (!err)
 		clear_sbi_flag(sbi, SBI_POR_DOING);
 
-	mutex_unlock(&sbi->cp_mutex);
+	up_write(&sbi->cp_global_sem);
 
 	/* let's drop all the directory inodes for clean checkpoint */
 	destroy_fsync_dnodes(&dir_list, err);
fs/f2fs/super.c +1 -1
index 87f7a6e..e33b0da 100644
@@ -3552,7 +3552,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
 	sbi->valid_super_block = valid_super_block;
 	init_rwsem(&sbi->gc_lock);
 	mutex_init(&sbi->writepages);
-	mutex_init(&sbi->cp_mutex);
+	init_rwsem(&sbi->cp_global_sem);
 	init_rwsem(&sbi->node_write);
 	init_rwsem(&sbi->node_change);
 
Signature · 2 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
#1 lore raw

From: Sahitya Tummala

Quoted 5 lines

Sent: 23 November 2020 05:29

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

I can't see any read lock requests.

So why the change?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#2 lore raw
Quoted 10 lines · David Laight

From: Sahitya Tummala

Quoted 5 lines

Sent: 23 November 2020 05:29

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

I can't see any read lock requests.

So why the change?

Hi David,

You can check the context of this patch in below link:

https://lore.kernel.org/linux-f2fs-devel/[email protected]/T/#t

BTW, the root cause here is that mutex lock won't serialize callers, so there could be potential starvation problem when this lock is always grabbed by high priority tasks.

Thanks,

#3 lore raw

From: Chao Yu

Quoted 16 lines

Sent: 24 November 2020 03:12

Quoted 10 lines · David Laight

From: Sahitya Tummala

Quoted 5 lines

Sent: 23 November 2020 05:29

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

I can't see any read lock requests.

So why the change?

Hi David,

Quoted 7 lines

You can check the context of this patch in below link:

https://lore.kernel.org/linux-f2fs-devel/[email protected]/T/#t

BTW, the root cause here is that mutex lock won't serialize callers, so there could be potential starvation problem when this lock is always grabbed by high priority tasks.

That doesn't seem right.

If I read the above correctly it was high priority tasks that were being 'starved' precisely because mutex lock serializes wakers.

If you have a lock that is contended so much that it is held 100% of the time you need a different locking strategy.

IIRC mutex locks are 'ticket' locks so that only one thread is woken each time the mutex is released, and they are woken in the order they went to sleep.

While this behaviour might not be the one you want, relying on rwsem (which might happen currently to work differently) doesn't seem the correct long term fix.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

#4 lore raw

Hi David,

Quoted 21 lines · David Laight

From: Chao Yu

Quoted 20 lines

Sent: 24 November 2020 03:12

Quoted 10 lines · David Laight

From: Sahitya Tummala

Quoted 5 lines

Sent: 23 November 2020 05:29

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

I can't see any read lock requests.

So why the change?

Hi David,

You can check the context of this patch in below link:

https://lore.kernel.org/linux-f2fs-devel/[email protected]/T/#t

Quoted 8 lines
Quoted 3 lines

BTW, the root cause here is that mutex lock won't serialize callers, so there could be potential starvation problem when this lock is always grabbed by high priority tasks.

That doesn't seem right.

If I read the above correctly it was high priority tasks that were being 'starved' precisely because mutex lock serializes wakers.

Actually it can happen for any random task irrespective of the priority. In my case, I was observing that the thread that went to sleep first is not able to acquire the lock first and other new threads that came in just around the mutex unlock time were acquiring the lock.

Quoted 7 lines

If you have a lock that is contended so much that it is held 100% of the time you need a different locking strategy.

IIRC mutex locks are 'ticket' locks so that only one thread is woken each time the mutex is released, and they are woken in the order they went to sleep.

AFAIK mutex locks doesn't *strictly* enforce FIFO order. The lock is released before waking the first waiting task. The waiting task has to run to claim the lock. So the lock is available for other tasks in this *short* window.

Thanks,

Quoted 10 lines

While this behaviour might not be the one you want, relying on rwsem (which might happen currently to work differently) doesn't seem the correct long term fix.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

Signature · 3 lines
--
Sent by a consultant of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Reviewed-by on v1 1/1
#5 lore raw
In reply to Sahitya Tummala
Quoted 5 lines · Sahitya Tummala

Use rwsem to ensure serialization of the callers and to avoid starvation of high priority tasks, when the system is under heavy IO workload.

Thanks,

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel