Fix queueing work if !bdi_cap_writeback_dirty()
PATCH 1/1 Fix queueing work if !bdi_cap_writeback_dirty()
If bdi has BDI_CAP_NO_WRITEBACK, bdi_forker_thread() doesn't start writeback thread. This means there is no consumer of work item made by bdi_queue_work().
This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling bdi_queue_work(), otherwise queued work never be consumed.
---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +5 -2
@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ; +/* * This is WB_SYNC_NONE writeback, so if allocation fails just * wakeup the thread for old dirty data writeback
@@ -1310,7 +1313,7 @@ void writeback_inodes_sb_nr(struct super .reason = reason, }; -if (sb->s_bdi == &noop_backing_dev_info) +if (!bdi_cap_writeback_dirty (sb->s_bdi))return ;WARN_ON (!rwsem_is_locked (&sb->s_umount));bdi_queue_work (sb->s_bdi, &work);
@@ -1396,7 +1399,7 @@ void sync_inodes_sb(struct super_block * };/* Nothing to do? */ -if (sb->s_bdi == &noop_backing_dev_info) +if (!bdi_cap_writeback_dirty (sb->s_bdi))return ;WARN_ON (!rwsem_is_locked (&sb->s_umount));
_
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 7 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.
Thanks for catching this! Does this bug have any side effects other than memory leaking? It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.
Thanks,
Fengguang
Quoted 41 lines
Signed-off-by OGAWA Hirofumi <[email protected]>---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)diff -puN
fs/fs-writeback.c~noop_backing_dev_info-check-fixfs/fs-writeback.cfs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +5
-2@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ; +/* * This is WB_SYNC_NONE writeback, so if allocation fails just * wakeup the thread for old dirty data writeback @@ -1310,7 +1313,7 @@ void writeback_inodes_sb_nr(struct super .reason = reason, }; -if (sb->s_bdi == &noop_backing_dev_info) +if (!bdi_cap_writeback_dirty (sb->s_bdi))return ;WARN_ON (!rwsem_is_locked (&sb->s_umount));bdi_queue_work (sb->s_bdi, &work);@@ -1396,7 +1399,7 @@ void sync_inodes_sb(struct super_block * };/* Nothing to do? */ -if (sb->s_bdi == &noop_backing_dev_info) +if (!bdi_cap_writeback_dirty (sb->s_bdi))return ;WARN_ON (!rwsem_is_locked (&sb->s_umount));_
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 9 lines · Fengguang Wu
Quoted 7 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.
Quoted 6 lines
Thanks for catching this! Does this bug have any side effects other than memory leaking?
It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.
If used custom bdi with BDI_CAP_NO_WRITEBACK, wait_for_completion() (e.g. sync_inodes_sb()) will be blocked forever.
I tested by custom bdi with BDI_CAP_NO_WRITEBACK - sync(2) blocked forever by this reason.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 14 lines · OGAWA Hirofumi
Quoted 12 lines · Fengguang Wu
Quoted 7 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.Thanks for catching this! Does this bug have any side effects other than memory leaking?
Quoted 6 lines
Quoted 3 lines
It may be possible for some caller that actually expect it to do some work to make progress, otherwise will eventually block. If so, we'll need to fix the caller.
If used custom bdi with
BDI_CAP_NO_WRITEBACK,wait_for_completion()(e.g.sync_inodes_sb()) will be blocked forever.
The sync(2) block cannot be fixed by this patch?
Quoted 2 lines
I tested by custom bdi with
BDI_CAP_NO_WRITEBACK- sync(2) blocked forever by this reason.
What's your test script? How do you create/use that custom bdi?
Thanks,
Fengguang
Quoted 4 lines · Fengguang Wu
Quoted 2 lines
If used custom bdi with
BDI_CAP_NO_WRITEBACK,wait_for_completion()(e.g.sync_inodes_sb()) will be blocked forever.The sync(2) block cannot be fixed by this patch?
This patch fixes block problem too.
Quoted 4 lines
Quoted 2 lines
I tested by custom bdi with
BDI_CAP_NO_WRITEBACK- sync(2) blocked forever by this reason.What's your test script? How do you create/use that custom bdi?
Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 8 lines · OGAWA Hirofumi
Quoted 4 lines · Fengguang Wu
Quoted 2 lines
If used custom bdi with
BDI_CAP_NO_WRITEBACK,wait_for_completion()(e.g.sync_inodes_sb()) will be blocked forever.The sync(2) block cannot be fixed by this patch?
This patch fixes block problem too.
Ah OK, my sight was somehow limited by the screen size and only saw the first chunk...
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I tested by custom bdi with
BDI_CAP_NO_WRITEBACK- sync(2) blocked forever by this reason.What's your test script? How do you create/use that custom bdi?
Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.
OK. Got it. So I'd suggest to change the first chunk to a VM_BUG_ON(), which should be enough to catch new bad callers. The other two chunks look fine.
Thanks,
Fengguang
Quoted 11 lines · Fengguang Wu
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I tested by custom bdi with
BDI_CAP_NO_WRITEBACK- sync(2) blocked forever by this reason.What's your test script? How do you create/use that custom bdi?
Ah, I wrote my kernel module to test. I guess there is no users in current kernel for now, because it doesn't work.
OK. Got it. So I'd suggest to change the first chunk to a
VM_BUG_ON(), which should be enough to catch new bad callers. The other two chunks look fine.
Hm, if it changed to VM_BUG_ON(), we can't use BDI_CAP_NO_WRITEBACK to disable the writeback task (but allow inode cache reclaim) after all.
Or you meant we don't allow to disable the writeback task?
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 18 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.Signed-off-by OGAWA Hirofumi <[email protected]>---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)diff -puN
fs/fs-writeback.c~noop_backing_dev_info-check-fixfs/fs-writeback.c
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 5 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;
Will someone in the current kernel actually call __bdi_start_writeback() on a BDI_CAP_NO_WRITEBACK bdi?
If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
Thanks,
Fengguang
Quoted 22 lines · Fengguang Wu
Quoted 21 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.Signed-off-by OGAWA Hirofumi <[email protected]>---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)diff -puN
fs/fs-writeback.c~noop_backing_dev_info-check-fixfs/fs-writeback.cfs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +0
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work;
Quoted 7 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 2 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0+if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in __mark_inode_dirty(), nobody should be using it, right?
If we defined it as the bug, I can't see what BDI_CAP_NO_WRITEBACK wants to do actually. We are not going to allow to disable the writeback task?
I was going to use this to disable writeback task on my developing FS...
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 30 lines · OGAWA Hirofumi
Quoted 28 lines · Fengguang Wu
Quoted 18 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.Signed-off-by OGAWA Hirofumi <[email protected]>---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)diff -puN
fs/fs-writeback.c~noop_backing_dev_info-check-fixfs/fs-writeback.cfs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 5 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?
Quoted 7 lines
Quoted 1 line
If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?
ping.
You are saying we should change bdi_cap_writeback_dirty(bdi) to VM_BUG_ON() too in __mark_inode_dirty()? And probably, you are going to remove the usage of BDI_CAP_NO_WRITEBACK for sb->s_bdi?
Quoted 1 line
I was going to use this to disable writeback task on my developing FS...
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 37 lines · OGAWA Hirofumi
Quoted 35 lines · OGAWA Hirofumi
Quoted 22 lines · Fengguang Wu
Quoted 21 lines · OGAWA Hirofumi
If bdi has
BDI_CAP_NO_WRITEBACK,bdi_forker_thread()doesn't start writeback thread. This means there is no consumer of work item made bybdi_queue_work().This adds to checking of !bdi_cap_writeback_dirty(sb->s_bdi) before calling
bdi_queue_work(), otherwise queued work never be consumed.Signed-off-by OGAWA Hirofumi <[email protected]>---
fs/fs-writeback.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)diff -puN
fs/fs-writeback.c~noop_backing_dev_info-check-fixfs/fs-writeback.cfs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +0
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work;Quoted 7 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 2 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0+if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?
Quoted 8 lines
Quoted 2 lines
If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?ping.
You are saying we should change
bdi_cap_writeback_dirty(bdi)toVM_BUG_ON()too in__mark_inode_dirty()? And probably, you are going to remove the usage ofBDI_CAP_NO_WRITEBACKforsb->s_bdi?
No, I'm talking about this chunk only. My feeling is, adding the bdi_cap_writeback_dirty() detection in __bdi_start_writeback() sounds too late. If ever the test is false, it may well indicate a bug in the callers. Obviously the callers all assume success because __bdi_start_writeback() does not even return a value.
Thanks,
Fengguang
Quoted 12 lines
Quoted 11 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
Quoted 7 lines
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?I was going to use this to disable writeback task on my developing FS...
That sounds like an interesting use case. Can you elaborate a bit more? Note that even if you disable __bdi_start_writeback() here, the kernel may also start writeback in the page reclaim path, the fsync() path, and perhaps more.
Thanks,
Fengguang
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?
Quoted 7 lines
Quoted 1 line
I was going to use this to disable writeback task on my developing FS...
That sounds like an interesting use case. Can you elaborate a bit more?
Note that even if you disable
__bdi_start_writeback()here, the kernel may also start writeback in the page reclaim path, thefsync()path, and perhaps more.
page reclaim and fsync path have FS handler. So, FS can control those.
The modern FS have to control to flush carefully. Many FSes are already ignoring if wbc->sync_mode != WB_SYNC_ALL (e.g. ext3_write_inode, nilfs_writepages), and have own FS task to flush.
The writeback task is always called with sync_mode != WB_SYNC_ALL except sync_inodes_sb(). But FS has sb->s_op->sync_fs() handler for sync_inodes_sb() path. So, writeback task just bothers FS to control to flush.
Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.
I thought there are 2 options - 1) pin inode with iget(), and iput() on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.
(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.
So, I want to do (2).
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 28 lines · OGAWA Hirofumi
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?Quoted 7 lines
Quoted 1 line
I was going to use this to disable writeback task on my developing FS...
That sounds like an interesting use case. Can you elaborate a bit more?
Note that even if you disable
__bdi_start_writeback()here, the kernel may also start writeback in the page reclaim path, thefsync()path, and perhaps more.
Quoted 5 lines
page reclaim and fsync path have FS handler. So, FS can control those.
The modern FS have to control to flush carefully. Many FSes are already ignoring if
wbc->sync_mode!=WB_SYNC_ALL(e.g.ext3_write_inode,nilfs_writepages), and have own FS task to flush.
Yeah, that test is mainly to improve IO efficiency for non-data-integrity writes.
Quoted 9 lines
The writeback task is always called with
sync_mode!=WB_SYNC_ALLexceptsync_inodes_sb(). But FS hassb->s_op->sync_fs()handler forsync_inodes_sb()path. So, writeback task just bothers FS to control to flush.Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.
Quoted 6 lines
I thought there are 2 options - 1) pin inode with
iget(), andiput()on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.
In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the BDI_CAP_NO_WRITEBACK approach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, see wb_check_background_flush().
And BDI_CAP_NO_WRITEBACK is expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change the BDI_CAP_NO_WRITEBACK flag dynamically.
Thanks,
Fengguang
Quoted 16 lines · Fengguang Wu
Quoted 9 lines
The writeback task is always called with
sync_mode!=WB_SYNC_ALLexceptsync_inodes_sb(). But FS hassb->s_op->sync_fs()handler forsync_inodes_sb()path. So, writeback task just bothers FS to control to flush.Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.
Quoted 6 lines
I thought there are 2 options - 1) pin inode with
iget(), andiput()on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.
Quoted 7 lines
In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the
BDI_CAP_NO_WRITEBACKapproach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, seewb_check_background_flush().
wb_check_background_flush() is called from,
bdi_forker_thread()
bdi_writeback_thread()
wb_do_writeback()
wb_check_background_flush()
But, bdi_forker_thread() never start bdi_writeback_thread() if !bdi_cap_writeback_dirty(bdi).
Or I'm seeing something wrong here?
Quoted 4 lines
And
BDI_CAP_NO_WRITEBACKis expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change theBDI_CAP_NO_WRITEBACKflag dynamically.
I'm going to use it as static or per-sb by initialized in fill_super(). And it uses always BDI_CAP_NO_WRITEBACK if sb is available. Because own FS task flush instead.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Quoted 33 lines · OGAWA Hirofumi
Quoted 16 lines · Fengguang Wu
Quoted 9 lines
The writeback task is always called with
sync_mode!=WB_SYNC_ALLexceptsync_inodes_sb(). But FS hassb->s_op->sync_fs()handler forsync_inodes_sb()path. So, writeback task just bothers FS to control to flush.Also it wants to control the reclaimable of inode cache too, because FS have to control to flush, and wants to use inode in own FS task, and it knows when inode is cleaned and can be reclaimed.
Quoted 6 lines
I thought there are 2 options - 1) pin inode with
iget(), andiput()on own FS task, 2) disable writeback task and care about inode reclaim by dirty flags.(1) was complex (e.g. inode can be the orphan inode), and seems to be ineffective workaround to survive with writeback task.
Quoted 7 lines
In principle, the VFS should of course give enough flexibility for the FS. But it's all about the details that matter. As for the
BDI_CAP_NO_WRITEBACKapproach, I'm afraid you'll not get the expected "FS control" through it. Because the flusher thread may already have a long queue of works which will take long time to finish. It even have its internal background/periodic works that's not controllable this way, seewb_check_background_flush().
wb_check_background_flush()is called from,
bdi_forker_thread()
bdi_writeback_thread()
wb_do_writeback()
wb_check_background_flush()
Quoted 4 lines
But,
bdi_forker_thread()never startbdi_writeback_thread()if !bdi_cap_writeback_dirty(bdi).Or I'm seeing something wrong here?
Nothing wrong.
Quoted 8 lines
Quoted 4 lines
And
BDI_CAP_NO_WRITEBACKis expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change theBDI_CAP_NO_WRITEBACKflag dynamically.I'm going to use it as static or per-sb by initialized in
fill_super(). And it uses alwaysBDI_CAP_NO_WRITEBACKif sb is available. Because own FS task flush instead.
Ah OK, sorry I didn't quite catch your use case.
But then if you set BDI_CAP_NO_WRITEBACK in the beginning, how come __bdi_start_writeback() will be called at all?
Thanks,
Fengguang
Quoted 5 lines · Fengguang Wu
Quoted 5 lines
Quoted 4 lines
And
BDI_CAP_NO_WRITEBACKis expected to be a static/constant flag that always evaluate to true/false for a given bdi. There will be correctness problems if you change theBDI_CAP_NO_WRITEBACKflag dynamically.
Quoted 8 lines
Quoted 3 lines
I'm going to use it as static or per-sb by initialized in
fill_super(). And it uses alwaysBDI_CAP_NO_WRITEBACKif sb is available. Because own FS task flush instead.Ah OK, sorry I didn't quite catch your use case.
But then if you set
BDI_CAP_NO_WRITEBACKin the beginning, how come__bdi_start_writeback()will be called at all?
If we call mark_inode_dirty(inode), inode goes into bdi->wb.b_dirty. And sync(2) calls __bdi_start_writeback() for all of bdi if bdi->wb.b_* is not empty.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 27 lines · OGAWA Hirofumi
Quoted 18 lines · Fengguang Wu
Quoted 17 lines
Quoted 11 lines
fs/fs-writeback.c __bdi_start_writeback(struct backing_dev · line 124
Quoted 6 lines
fs/fs-writeback.c renamed from fs/fs-writeback.c~noop_backing_dev_info-check-fix +2
-0@@ -120,6 +120,9 @@ __bdi_start_writeback(struct backing_dev {struct wb_writeback_work *work; +if (!bdi_cap_writeback_dirty (bdi)) +return ;Will someone in the current kernel actually call
__bdi_start_writeback()on aBDI_CAP_NO_WRITEBACKbdi?If the answer is no, VM_BUG_ON(!bdi_cap_writeback_dirty(bdi)) looks better.
I guess nobody call it in current kernel though. Hmm.., but we also have check in
__mark_inode_dirty(), nobody should be using it, right?If we defined it as the bug, I can't see what
BDI_CAP_NO_WRITEBACKwants to do actually. We are not going to allow to disable the writeback task?Quoted 7 lines
Quoted 1 line
I was going to use this to disable writeback task on my developing FS...
That sounds like an interesting use case. Can you elaborate a bit more?
Note that even if you disable
__bdi_start_writeback()here, the kernel may also start writeback in the page reclaim path, thefsync()path, and perhaps more.
Quoted 5 lines
page reclaim and fsync path have FS handler. So, FS can control those.
The modern FS have to control to flush carefully. Many FSes are already ignoring if
wbc->sync_mode!=WB_SYNC_ALL(e.g.ext3_write_inode,nilfs_writepages), and have own FS task to flush.
Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 8 lines · Jan Kara
Quoted 5 lines
page reclaim and fsync path have FS handler. So, FS can control those.
The modern FS have to control to flush carefully. Many FSes are already ignoring if
wbc->sync_mode!=WB_SYNC_ALL(e.g.ext3_write_inode,nilfs_writepages), and have own FS task to flush.Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...
If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.
This job is nothing benefit to just for workaround of flusher, and is complex and racy.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 7 lines · OGAWA Hirofumi
Quoted 5 lines · Jan Kara
Quoted 5 lines
page reclaim and fsync path have FS handler. So, FS can control those.
The modern FS have to control to flush carefully. Many FSes are already ignoring if
wbc->sync_mode!=WB_SYNC_ALL(e.g.ext3_write_inode,nilfs_writepages), and have own FS task to flush.
Quoted 7 lines
Quoted 3 lines
Out of curiosity, what exactly do you need to control in your filesystem
that makes flusher thread unusable for you? You still have a lot of flexibility with ->write_inode() and ->writepages() callbacks...If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.
Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...
Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 10 lines · Jan Kara
Quoted 3 lines
If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.
Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...
Basically all data can be data-integrity write like data logging, so it would be more than common. And ->writepages() will also ignore WBC_SYNC_NONE.
Quoted 4 lines
Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.
I'm not sure what you meant though. What is the difference with ignoring WBC_SYNC_NONE?
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 16 lines · OGAWA Hirofumi
Quoted 10 lines · Jan Kara
Quoted 3 lines
If flusher is working, it clears dirty flags of inode. But if those handers can't flush at the time, we have to do redirty or something to prevent the reclaim.
Well, if this is your only problem then I'd see better options than just
disabling flusher thread. If the inability to write inode is rare, then redirtying seems like a reasonable option (despite I agree it's a bit ugly). If the inability to write is common, then you'll probably have to do the dirty inode tracking yourself in some list and expose inodes to VM when they are ready to be written. Or you handle writing of inodes yourself but leave writing of pages on flusher thread...Basically all data can be data-integrity write like data logging, so it would be more than common. And ->writepages() will also ignore
WBC_SYNC_NONE.
Quoted 7 lines
Quoted 4 lines
Because when you disable flusher thread completely you have to put all the smarts to avoid livelocks, keep fairness among processes, write old data, keep number of dirty pages under control into your filesystem which leads to a lot of duplication.
I'm not sure what you meant though. What is the difference with ignoring
WBC_SYNC_NONE?
When you completely ignore WB_SYNC_NONE writeback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they do WB_SYNC_NONE writeback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 9 lines · Jan Kara
Quoted 2 lines
I'm not sure what you meant though. What is the difference with ignoring
WBC_SYNC_NONE?When you completely ignore
WB_SYNC_NONEwriteback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they doWB_SYNC_NONEwriteback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.
I think you know how to solve it though. You can add the periodic flush in own task. And you can check bdi->dirty_exceeded in any handlers.
Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.
But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.
Since situation was changed, will we accept it?
OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.
So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 12 lines · OGAWA Hirofumi
Quoted 9 lines · Jan Kara
Quoted 2 lines
I'm not sure what you meant though. What is the difference with ignoring
WBC_SYNC_NONE?When you completely ignore
WB_SYNC_NONEwriteback, you'll soon drive the
machine close to dirty limits and processes dirtying pages will get throttled. Because flusher threads won't be able to write pages - they doWB_SYNC_NONEwriteback when we have too many dirty pages - processes will be throttled until somebody calls sync(1) or someone writes the data for some other reason... So I suspect things won't really work as you expect.
Quoted 2 lines
I think you know how to solve it though. You can add the periodic flush in own task. And you can check
bdi->dirty_exceededin any handlers.
Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.
Quoted 11 lines
Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.
But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.
Since situation was changed, will we accept it?
Quoted 7 lines
OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.
So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.
OK, thanks for explanation. Now I understand what you are trying to do.
Would it be enough if you could track dirty inodes inside your filesystem and provide some callback for flusher so that you can queue these inodes in the IO queue?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 6 lines · Jan Kara
Quoted 2 lines
I think you know how to solve it though. You can add the periodic flush in own task. And you can check
bdi->dirty_exceededin any handlers.Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.
Right. But it is what current FSes are doing more or less.
Quoted 15 lines
Quoted 15 lines
Well, ok. The alternative plan but more bigger change is to add the handler to writeback task path. This would be better way, and core should be able to request to flush with usual way (I guess this is what you are concerning). And I believe some FS can implement the simpler and more efficient writeback path.
But this would look like what reiserfs4 was submitted in past (before bdi was introduced), and unfortunately never accepted though.
Since situation was changed, will we accept it?
OK, why my FS requires it? Because basic strategy try to keep the consistency of user view, not only internal metadata consistency. I.e. it works like to flush the snapshot of user view.
Quoted 7 lines
Quoted 3 lines
So, flushing metadata/data by arbitrary order like current writeback task does is unacceptable (of course, except request by user). And writeback task will never know the correct order of FS.
OK, thanks for explanation. Now I understand what you are trying to do.
Would it be enough if you could track dirty inodes inside your filesystem and provide some callback for flusher so that you can queue these inodes in the IO queue?
Yes, I guess so. I'm not doing the experiment this plan yet, so I'm not sure though. If we provide the callback something like ->writeback_sb_inodes(), it would work.
And the better design is to remove duplication of dirty inode tracking on writeback task and own FS though. (However, this is quite optional)
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 10 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines
I think you know how to solve it though. You can add the periodic flush in own task. And you can check
bdi->dirty_exceededin any handlers.Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.Right. But it is what current FSes are doing more or less.
It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 10 lines · Jan Kara
Quoted 9 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines
I think you know how to solve it though. You can add the periodic flush in own task. And you can check
bdi->dirty_exceededin any handlers.Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.
Quoted 4 lines
Quoted 1 line
Right. But it is what current FSes are doing more or less.
It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...
Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.
Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on writepages()?
And if alternative plan is acceptable, maybe I will not have interest to this anymore.
Thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>
Jan Kara
reviewerQuoted 12 lines · OGAWA Hirofumi
Quoted 10 lines · Jan Kara
Quoted 9 lines · OGAWA Hirofumi
Quoted 6 lines · Jan Kara
Quoted 2 lines
I think you know how to solve it though. You can add the periodic flush in own task. And you can check
bdi->dirty_exceededin any handlers.Sure, you can have your private thread. That is possible but you will
have to duplicate flusher logic and you will still get odd behavior e.g. when your filesystem is on one partition and another filesystem is on a different partition of the same disk.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Right. But it is what current FSes are doing more or less.
It's not. Page writeback is respected by all filesystems in most cases
AFAIK. Inode writeback is a different issue but that's not so interesting from mm point of view...Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.
Well, because there is much more of data pages then there is metadata. So
when you do strange things (like refuse to write / reclaim) with metadata, it usually ends up in the noise. But when you start doing similar things with data pages, people will notice.
Quoted 2 lines
Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on
writepages()?
Yes.
Quoted 2 lines
And if alternative plan is acceptable, maybe I will not have interest to this anymore.
Yes, the alternative plan looks better to me. But all in all I don't want
to stop you from your experiments :) I mostly just wanted to point out that disabling flusher thread for a filesystem has a complex consequences which IMHO bring more bad than good.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 7 lines · Jan Kara
Quoted 3 lines
Duplicate flusher - many FSes has own task to flush. Odd behavior in the case of partition - agree, but I'm not sure why metadata is ok, and it is not odd behavior.
Well, because there is much more of data pages then there is metadata. So
when you do strange things (like refuse to write / reclaim) with metadata, it usually ends up in the noise. But when you start doing similar things with data pages, people will notice.
Could you explain more. So, you are thinking we have to fix current behavior for metadata intensive applications? And what will people notice?
Quoted 10 lines
Quoted 2 lines
Sorry, I'm not sure your point in latest comment. You are just saying FS must flush pages on
writepages()?Yes.
Quoted 2 lines
And if alternative plan is acceptable, maybe I will not have interest to this anymore.
Yes, the alternative plan looks better to me. But all in all I don't want
to stop you from your experiments :) I mostly just wanted to point out that disabling flusher thread for a filesystem has a complex consequences which IMHO bring more bad than good.
OK, thanks.
Signature · 1 line
OGAWA Hirofumi <[email protected]>