Skip to contentSkip to conversation

Fix queueing work if !bdi_cap_writeback_dirty()

PATCH In review OGAWA Hirofumi 3 participants 29 messages Sep 11, 2012 – Sep 17, 2012 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix queueing work if !bdi_cap_writeback_dirty()

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +5 -2
diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;
+
 	/*
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
@@ -1310,7 +1313,7 @@ void writeback_inodes_sb_nr(struct super
 		.reason			= reason,
 	};
 
-	if (sb->s_bdi == &noop_backing_dev_info)
+	if (!bdi_cap_writeback_dirty(sb->s_bdi))
 		return;
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
@@ -1396,7 +1399,7 @@ void sync_inodes_sb(struct super_block *
 	};
 
 	/* Nothing to do? */
-	if (sb->s_bdi == &noop_backing_dev_info)
+	if (!bdi_cap_writeback_dirty(sb->s_bdi))
 		return;
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 

_

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#1 lore raw
Quoted 7 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

Thanks for catching this! Does this bug have any side effects other than memory leaking? It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.

Thanks,
Fengguang

Quoted 41 lines

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c

fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +5 -2
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;
+
 	/*
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
@@ -1310,7 +1313,7 @@ void writeback_inodes_sb_nr(struct super
 		.reason			= reason,
 	};
 
-	if (sb->s_bdi == &noop_backing_dev_info)
+	if (!bdi_cap_writeback_dirty(sb->s_bdi))
 		return;
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 	bdi_queue_work(sb->s_bdi, &work);
@@ -1396,7 +1399,7 @@ void sync_inodes_sb(struct super_block *
 	};
 
 	/* Nothing to do? */
-	if (sb->s_bdi == &noop_backing_dev_info)
+	if (!bdi_cap_writeback_dirty(sb->s_bdi))
 		return;
 	WARN_ON(!rwsem_is_locked(&sb->s_umount));
 

_

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#2 lore raw
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

Quoted 6 lines

Thanks for catching this! Does this bug have any side effects other than memory leaking?

It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.

If used custom bdi with BDI_CAP_NO_WRITEBACK, wait_for_completion() (e.g. sync_inodes_sb()) will be blocked forever.

I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#3 lore raw
Quoted 14 lines · OGAWA Hirofumi
Quoted 12 lines · Fengguang Wu
Quoted 7 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

Thanks for catching this! Does this bug have any side effects other than memory leaking?

Quoted 6 lines
Quoted 3 lines

It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.

If used custom bdi with BDI_CAP_NO_WRITEBACK, wait_for_completion() (e.g. sync_inodes_sb()) will be blocked forever.

The sync(2) block cannot be fixed by this patch?

Quoted 2 lines

I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.

What's your test script? How do you create/use that custom bdi?

Thanks,
Fengguang

#4 lore raw
Quoted 4 lines · Fengguang Wu
Quoted 2 lines

If used custom bdi with BDI_CAP_NO_WRITEBACK, wait_for_completion() (e.g. sync_inodes_sb()) will be blocked forever.

The sync(2) block cannot be fixed by this patch?

This patch fixes block problem too.

Quoted 4 lines
Quoted 2 lines

I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.

What's your test script? How do you create/use that custom bdi?

Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#5 lore raw
Quoted 8 lines · OGAWA Hirofumi
Quoted 4 lines · Fengguang Wu
Quoted 2 lines

If used custom bdi with BDI_CAP_NO_WRITEBACK, wait_for_completion() (e.g. sync_inodes_sb()) will be blocked forever.

The sync(2) block cannot be fixed by this patch?

This patch fixes block problem too.

Ah OK, my sight was somehow limited by the screen size and only saw the first chunk...

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.

What's your test script? How do you create/use that custom bdi?

Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.

OK. Got it. So I'd suggest to change the first chunk to a VM_BUG_ON(), which should be enough to catch new bad callers. The other two chunks look fine.

Thanks,
Fengguang

#6 lore raw
Quoted 11 lines · Fengguang Wu
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.

What's your test script? How do you create/use that custom bdi?

Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.

OK. Got it. So I'd suggest to change the first chunk to a VM_BUG_ON(), which should be enough to catch new bad callers. The other two chunks look fine.

Hm, if it changed to VM_BUG_ON(), we can't use BDI_CAP_NO_WRITEBACK to disable the writeback task (but allow inode cache reclaim) after all.

Or you meant we don't allow to disable the writeback task?

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#7 lore raw
In reply to OGAWA Hirofumi
Quoted 18 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 5 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

Thanks,
Fengguang

#8 lore raw
Quoted 22 lines · Fengguang Wu
Quoted 21 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c

fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +0 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
Quoted 7 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 2 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

I was going to use this to disable writeback task on my developing FS...

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#9 lore raw
Quoted 30 lines · OGAWA Hirofumi
Quoted 28 lines · Fengguang Wu
Quoted 18 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 5 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

Quoted 7 lines
Quoted 1 line

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

ping.

You are saying we should change bdi_cap_writeback_dirty(bdi) to VM_BUG_ON() too in __mark_inode_dirty()? And probably, you are going to remove the usage of BDI_CAP_NO_WRITEBACK for sb->s_bdi?

Quoted 1 line

I was going to use this to disable writeback task on my developing FS...

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#10 lore raw
Quoted 37 lines · OGAWA Hirofumi
Quoted 35 lines · OGAWA Hirofumi
Quoted 22 lines · Fengguang Wu
Quoted 21 lines · OGAWA Hirofumi

If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().

This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.

---

 fs/fs-writeback.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff -puN fs/fs-writeback.c~noop_backing_dev_info-check-fix fs/fs-writeback.c

fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +0 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
Quoted 7 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 2 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

Quoted 8 lines
Quoted 2 lines

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

ping.

You are saying we should change bdi_cap_writeback_dirty(bdi) to VM_BUG_ON() too in __mark_inode_dirty()? And probably, you are going to remove the usage of BDI_CAP_NO_WRITEBACK for sb->s_bdi?

No, I'm talking about this chunk only. My feeling is, adding the bdi_cap_writeback_dirty() detection in __bdi_start_writeback() sounds too late. If ever the test is false, it may well indicate a bug in the callers. Obviously the callers all assume success because __bdi_start_writeback() does not even return a value.

Thanks,
Fengguang

#11 lore raw
In reply to OGAWA Hirofumi
Quoted 12 lines
Quoted 11 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

Quoted 7 lines

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

I was going to use this to disable writeback task on my developing FS...

That sounds like an interesting use case. Can you elaborate a bit more? Note that even if you disable __bdi_start_writeback() here, the kernel may also start writeback in the page reclaim path, the fsync() path, and perhaps more.

Thanks,
Fengguang

#12 lore raw
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

Quoted 7 lines
Quoted 1 line

I was going to use this to disable writeback task on my developing FS...

That sounds like an interesting use case. Can you elaborate a bit more?

Note that even if you disable __bdi_start_writeback() here, the kernel may also start writeback in the page reclaim path, the fsync() path, and perhaps more.

page reclaim and fsync path have FS handler. So, FS can control those.

The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.

The writeback task is always called with sync_mode != WB_SYNC_ALL except sync_inodes_sb(). But FS has sb->s_op->sync_fs() handler for sync_inodes_sb() path. So, writeback task just bothers FS to control to flush.

Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.

I thought there are 2 options - 1) pin inode with iget(), and iput() on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.

(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.

So, I want to do (2).

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#13 lore raw
Quoted 28 lines · OGAWA Hirofumi
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

Quoted 7 lines
Quoted 1 line

I was going to use this to disable writeback task on my developing FS...

That sounds like an interesting use case. Can you elaborate a bit more?

Note that even if you disable __bdi_start_writeback() here, the kernel may also start writeback in the page reclaim path, the fsync() path, and perhaps more.

Quoted 5 lines

page reclaim and fsync path have FS handler. So, FS can control those.

The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.

Yeah, that test is mainly to improve IO efficiency for non-data-integrity writes.

Quoted 9 lines

The writeback task is always called with sync_mode != WB_SYNC_ALL except sync_inodes_sb(). But FS has sb->s_op->sync_fs() handler for sync_inodes_sb() path. So, writeback task just bothers FS to control to flush.

Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.

Quoted 6 lines

I thought there are 2 options - 1) pin inode with iget(), and iput() on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.

(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.

In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the BDI_CAP_NO_WRITEBACK approach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, see wb_check_background_flush().

And BDI_CAP_NO_WRITEBACK is expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change the BDI_CAP_NO_WRITEBACK flag dynamically.

Thanks,
Fengguang

#14 lore raw
Quoted 16 lines · Fengguang Wu
Quoted 9 lines

The writeback task is always called with sync_mode != WB_SYNC_ALL except sync_inodes_sb(). But FS has sb->s_op->sync_fs() handler for sync_inodes_sb() path. So, writeback task just bothers FS to control to flush.

Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.

Quoted 6 lines

I thought there are 2 options - 1) pin inode with iget(), and iput() on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.

(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.

Quoted 7 lines

In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the BDI_CAP_NO_WRITEBACK approach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, see wb_check_background_flush().

wb_check_background_flush() is called from,

bdi_forker_thread()
bdi_writeback_thread()
wb_do_writeback()
wb_check_background_flush()

But, bdi_forker_thread() never start bdi_writeback_thread() if !bdi_cap_writeback_dirty(bdi).

Or I'm seeing something wrong here?

Quoted 4 lines

And BDI_CAP_NO_WRITEBACK is expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change the BDI_CAP_NO_WRITEBACK flag dynamically.

I'm going to use it as static or per-sb by initialized in fill_super(). And it uses always BDI_CAP_NO_WRITEBACK if sb is available. Because own FS task flush instead.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#15 lore raw
Quoted 33 lines · OGAWA Hirofumi
Quoted 16 lines · Fengguang Wu
Quoted 9 lines

The writeback task is always called with sync_mode != WB_SYNC_ALL except sync_inodes_sb(). But FS has sb->s_op->sync_fs() handler for sync_inodes_sb() path. So, writeback task just bothers FS to control to flush.

Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.

Quoted 6 lines

I thought there are 2 options - 1) pin inode with iget(), and iput() on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.

(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.

Quoted 7 lines

In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the BDI_CAP_NO_WRITEBACK approach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, see wb_check_background_flush().

wb_check_background_flush() is called from,

bdi_forker_thread()
bdi_writeback_thread()
wb_do_writeback()
wb_check_background_flush()

Quoted 4 lines

But, bdi_forker_thread() never start bdi_writeback_thread() if !bdi_cap_writeback_dirty(bdi).

Or I'm seeing something wrong here?

Nothing wrong.

Quoted 8 lines
Quoted 4 lines

And BDI_CAP_NO_WRITEBACK is expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change the BDI_CAP_NO_WRITEBACK flag dynamically.

I'm going to use it as static or per-sb by initialized in fill_super(). And it uses always BDI_CAP_NO_WRITEBACK if sb is available. Because own FS task flush instead.

Ah OK, sorry I didn't quite catch your use case.

But then if you set BDI_CAP_NO_WRITEBACK in the beginning, how come __bdi_start_writeback() will be called at all?

Thanks,
Fengguang

#16 lore raw
Quoted 5 lines · Fengguang Wu
Quoted 5 lines
Quoted 4 lines

And BDI_CAP_NO_WRITEBACK is expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change the BDI_CAP_NO_WRITEBACK flag dynamically.

Quoted 8 lines
Quoted 3 lines

I'm going to use it as static or per-sb by initialized in fill_super(). And it uses always BDI_CAP_NO_WRITEBACK if sb is available. Because own FS task flush instead.

Ah OK, sorry I didn't quite catch your use case.

But then if you set BDI_CAP_NO_WRITEBACK in the beginning, how come __bdi_start_writeback() will be called at all?

If we call mark_inode_dirty(inode), inode goes into bdi->wb.b_dirty. And sync(2) calls __bdi_start_writeback() for all of bdi if bdi->wb.b_* is not empty.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#17 lore raw
In reply to OGAWA Hirofumi
Quoted 27 lines · OGAWA Hirofumi
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines

fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124

Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2 -0
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev
 {
 	struct wb_writeback_work *work;
 
+	if (!bdi_cap_writeback_dirty(bdi))
+		return;

Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?

If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.

I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?

If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?

Quoted 7 lines
Quoted 1 line

I was going to use this to disable writeback task on my developing FS...

That sounds like an interesting use case. Can you elaborate a bit more?

Note that even if you disable __bdi_start_writeback() here, the kernel may also start writeback in the page reclaim path, the fsync() path, and perhaps more.

Quoted 5 lines

page reclaim and fsync path have FS handler. So, FS can control those.

The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.

Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#18 lore raw
Quoted 8 lines · Jan Kara
Quoted 5 lines

page reclaim and fsync path have FS handler. So, FS can control those.

The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.

Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...

If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.

This job is nothing benefit to just for workaround of flusher, and is complex and racy.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#19 lore raw
Quoted 7 lines · OGAWA Hirofumi
Quoted 5 lines · Jan Kara
Quoted 5 lines

page reclaim and fsync path have FS handler. So, FS can control those.

The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.

Quoted 7 lines
Quoted 3 lines

Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...

If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.

Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...

Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#20 lore raw
Quoted 10 lines · Jan Kara
Quoted 3 lines

If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.

Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...

Basically all data can be data-integrity write like data logging, so it would be more than common. And ->writepages() will also ignore WBC_SYNC_NONE.

Quoted 4 lines

Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.

I'm not sure what you meant though. What is the difference with ignoring WBC_SYNC_NONE?

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#21 lore raw
Quoted 16 lines · OGAWA Hirofumi
Quoted 10 lines · Jan Kara
Quoted 3 lines

If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.

Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...

Basically all data can be data-integrity write like data logging, so it would be more than common. And ->writepages() will also ignore WBC_SYNC_NONE.

Quoted 7 lines
Quoted 4 lines

Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.

I'm not sure what you meant though. What is the difference with ignoring WBC_SYNC_NONE?

When you completely ignore WB_SYNC_NONE writeback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they do WB_SYNC_NONE writeback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#22 lore raw
Quoted 9 lines · Jan Kara
Quoted 2 lines

I'm not sure what you meant though. What is the difference with ignoring WBC_SYNC_NONE?

When you completely ignore WB_SYNC_NONE writeback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they do WB_SYNC_NONE writeback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.

But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.

Since situation was changed, will we accept it?

OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.

So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#23 lore raw
Quoted 12 lines · OGAWA Hirofumi
Quoted 9 lines · Jan Kara
Quoted 2 lines

I'm not sure what you meant though. What is the difference with ignoring WBC_SYNC_NONE?

When you completely ignore WB_SYNC_NONE writeback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they do WB_SYNC_NONE writeback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.

Quoted 2 lines

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.

Quoted 11 lines

Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.

But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.

Since situation was changed, will we accept it?

Quoted 7 lines

OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.

So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.

OK, thanks for explanation. Now I understand what you are trying to do.
Would it be enough if you could track dirty inodes inside your filesystem and provide some callback for flusher so that you can queue these inodes in the IO queue?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#24 lore raw
Quoted 6 lines · Jan Kara
Quoted 2 lines

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.

Right. But it is what current FSes are doing more or less.

Quoted 15 lines
Quoted 15 lines

Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.

But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.

Since situation was changed, will we accept it?

OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.

Quoted 7 lines
Quoted 3 lines

So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.

OK, thanks for explanation. Now I understand what you are trying to do.
Would it be enough if you could track dirty inodes inside your filesystem and provide some callback for flusher so that you can queue these inodes in the IO queue?

Yes, I guess so. I'm not doing the experiment this plan yet, so I'm not sure though. If we provide the callback something like ->writeback_sb_inodes(), it would work.

And the better design is to remove duplication of dirty inode tracking on writeback task and own FS though. (However, this is quite optional)

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#25 lore raw
Quoted 10 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.

Right. But it is what current FSes are doing more or less.

It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#26 lore raw
Quoted 10 lines · Jan Kara
Quoted 9 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.

Quoted 4 lines
Quoted 1 line

Right. But it is what current FSes are doing more or less.

It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...

Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.

Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on writepages()?

And if alternative plan is acceptable, maybe I will not have interest to this anymore.

Thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
reviewer
#27 lore raw
Quoted 12 lines · OGAWA Hirofumi
Quoted 10 lines · Jan Kara
Quoted 9 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines

I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.

Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Right. But it is what current FSes are doing more or less.

It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...

Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.

Well, because there is much more of data pages then there is metadata. So
when you do strange things (like refuse to write / reclaim) with metadata, it usually ends up in the noise. But when you start doing similar things with data pages, people will notice.

Quoted 2 lines

Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on writepages()?

Yes.

Quoted 2 lines

And if alternative plan is acceptable, maybe I will not have interest to this anymore.

Yes, the alternative plan looks better to me. But all in all I don't want
to stop you from your experiments :) I mostly just wanted to point out that disabling flusher thread for a filesystem has a complex consequences which IMHO bring more bad than good.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#28 lore raw
Quoted 7 lines · Jan Kara
Quoted 3 lines

Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.

Well, because there is much more of data pages then there is metadata. So
when you do strange things (like refuse to write / reclaim) with metadata, it usually ends up in the noise. But when you start doing similar things with data pages, people will notice.

Could you explain more. So, you are thinking we have to fix current behavior for metadata intensive applications? And what will people notice?

Quoted 10 lines
Quoted 2 lines

Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on writepages()?

Yes.

Quoted 2 lines

And if alternative plan is acceptable, maybe I will not have interest to this anymore.

Yes, the alternative plan looks better to me. But all in all I don't want
to stop you from your experiments :) I mostly just wanted to point out that disabling flusher thread for a filesystem has a complex consequences which IMHO bring more bad than good.

OK, thanks.

Signature · 1 line
OGAWA Hirofumi <[email protected]>