Skip to contentSkip to conversation

Future writeback topics

LSF/MM TOPIC Boaz Harrosh 5 participants 10 messages Jan 22, 2012 – Jan 23, 2012

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

This can solve the IO fairness and latency bound (interactivness) of small IOs.
There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

Thanks
Boaz

maintainer
#1 lore raw

Since a lot of these are mm related; added linux-mm to cc list

Quoted 35 lines · Boaz Harrosh

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

This can solve the IO fairness and latency bound (interactivness) of small IOs.
There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

Quoted 8 lines

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

Quoted 9 lines

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

Since a bdi is coupled to a gendisk and a queue, why isn't optimal_io_size what you want?

Quoted 14 lines

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

Quoted 7 lines

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

This is essentially what we've been discussing in "Fixing Writeback" for the last two years, isn't it (the fact that we have multiple sources of writeback and they don't co-ordinate properly). I thought our solution was to prefer linear over seeky ... adding a mutex makes that more absolute than a preference, but are you sure it helps (especially as it adds a lock to the writeout path).

James

#2 lore raw
Quoted 2 lines · James Bottomley

Since a lot of these are mm related; added linux-mm to cc list

Hi James.

Thanks for reading, and sorry I missed linux-mm

Quoted 42 lines · Boaz Harrosh
Quoted 41 lines

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

This can solve the IO fairness and latency bound (interactivness) of small IOs.
There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

Quoted 8 lines
Quoted 2 lines

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

Some measurements have already been done and complained about. There were even attempts at IO-less page-reclaim by Dave Chinner if I recall correctly. Mainly the complains I'm addressing here are:
1. Very bad IO patterns of page-reclaim and it's avoidance.
2. The issue raised in that other thread about pages changing during IO penalty.
3. Oblivious-ness of the VFS writeback to fairness and the starvation of small IOs in filesystems that are not block based.

But I agree much more testing is needed specially for 3. I can't promise I'll be up to it for LSF.

Even more blasphemous of me is that I'm not the one that could code such changes, I'm not familiar and capable with the VFS code to do such a task. I only know that as a filesystem these are areas that are missed.

Quoted 5 lines
Quoted 5 lines

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

Quoted 8 lines
Quoted 4 lines

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

Since a bdi is coupled to a gendisk and a queue, why isn't optimal_io_size what you want?

Exactly for block-based devices these are intended here. The "register block BDI" will fill these in from there. It must be at the BDI level for these FSs that are not block based but have similar alignment needs. And/or also filesystems that are multidevice like BTRFS and ZFS(Fuse) which have conglomerated alignment needs.

Quoted 22 lines
Quoted 14 lines

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

Quoted 7 lines

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

Quoted 4 lines

This is essentially what we've been discussing in "Fixing Writeback" for the last two years, isn't it (the fact that we have multiple sources of writeback and they don't co-ordinate properly). I thought our solution was to prefer linear over seeky ...

Yes. Lots of work has been done, and as part of that a tremendous clean up has also been submitted and the code is kind of ready for the next round.

Some of these things we've been talking about for years as you said but are not yet done. For example my problem of seeky IO when the application just gave us perfectly linear writeout. This is why I said:
Are we ready for the second round?

Quoted 3 lines

adding a mutex makes that more
absolute than a preference, but are you sure it helps (especially as it adds a lock to the writeout path).

No, I'm not sure at all. I just gave an example at some example filesystem (exofs that I work on) where the penalty for non aligned IO is so bad (Raid 5) that a Mutex at every IO gave better performance then the above problem. I have not submitted this lock at the end because it is only for the large-file IO case, so in the General workloads I could not prove if it's better or not.

Quoted 4 lines

James

Thanks
Boaz

maintainer
#3 lore raw

[corrected linux-mm address I mistyped initially]

Quoted 67 lines · Boaz Harrosh
Quoted 2 lines · James Bottomley

Since a lot of these are mm related; added linux-mm to cc list

Hi James.

Thanks for reading, and sorry I missed linux-mm

Quoted 42 lines · Boaz Harrosh
Quoted 41 lines

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

This can solve the IO fairness and latency bound (interactivness) of small IOs.
There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

Quoted 8 lines
Quoted 2 lines

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

Some measurements have already been done and complained about. There were even attempts at IO-less page-reclaim by Dave Chinner if I recall correctly. Mainly the complains I'm addressing here are:
1. Very bad IO patterns of page-reclaim and it's avoidance.
2. The issue raised in that other thread about pages changing during IO penalty.
3. Oblivious-ness of the VFS writeback to fairness and the starvation of small IOs in filesystems that are not block based.

Quoted 2 lines

But I agree much more testing is needed specially for 3. I can't promise I'll be up to it for LSF.

As long as someone does them, I don't really care who.

Quoted 3 lines

Even more blasphemous of me is that I'm not the one that could code such changes, I'm not familiar and capable with the VFS code to do such a task. I only know that as a filesystem these are areas that are missed.

Well, OK, we'll treat this as a Call for a Topic rather than a topic (depending on whether someone is willing to do the work and talk about it) ... or we can just fold it into the general writeback discussion ... I'm sure there'll be one of those.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

Quoted 8 lines
Quoted 3 lines

Since a bdi is coupled to a gendisk and a queue, why isn't optimal_io_size what you want?

Exactly for block-based devices these are intended here. The "register block BDI" will fill these in from there. It must be at the BDI level for these FSs that are not block based but have similar alignment needs. And/or also filesystems that are multidevice like BTRFS and ZFS(Fuse) which have conglomerated alignment needs.

But this topic then becomes adding alignment for non block backed filesystems? I take it you're thinking NFS rather than MTD or MMC?

For multiple devices, you do a simple cascade ... a bit like dm does today ... but unless all the devices are aligned to optimal I/O it never really works (and it's not necessarily worth solving ... the idea that if you want performance from an array of devices, you match characteristics isn't a hugely hard one to get the industry to swallow).

Quoted 27 lines
Quoted 22 lines
Quoted 14 lines

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

Quoted 7 lines

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

Quoted 4 lines

This is essentially what we've been discussing in "Fixing Writeback" for the last two years, isn't it (the fact that we have multiple sources of writeback and they don't co-ordinate properly). I thought our solution was to prefer linear over seeky ...

Quoted 7 lines

Yes. Lots of work has been done, and as part of that a tremendous clean up has also been submitted and the code is kind of ready for the next round.

Some of these things we've been talking about for years as you said but are not yet done. For example my problem of seeky IO when the application just gave us perfectly linear writeout. This is why I said:
Are we ready for the second round?

OK, will defer to mm guys.

Quoted 10 lines
Quoted 3 lines

adding a mutex makes that more
absolute than a preference, but are you sure it helps (especially as it adds a lock to the writeout path).

No, I'm not sure at all. I just gave an example at some example filesystem (exofs that I work on) where the penalty for non aligned IO is so bad (Raid 5) that a Mutex at every IO gave better performance then the above problem. I have not submitted this lock at the end because it is only for the large-file IO case, so in the General workloads I could not prove if it's better or not.

Global mutexes add a latency to the fast path ... this latency rises with the NUMA ness or number of cores on the system ... that's why it hit my "are you really sure" detector.

James

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 4 lines · James Bottomley

But this topic then becomes adding alignment for non block backed filesystems? I take it you're thinking NFS rather than MTD or MMC?

Sorry to differ. But no this is for most making the IO aligned in the first place. Block-dev or not. Today VFS has no notion of alignment and IO is submitted as is with out any alignment considerations.

Quoted 6 lines

For multiple devices, you do a simple cascade ... a bit like dm does today ... but unless all the devices are aligned to optimal I/O it never really works (and it's not necessarily worth solving ... the idea that if you want performance from an array of devices, you match characteristics isn't a hugely hard one to get the industry to swallow).

No I'm talking about raid configurations like object raid in exofs/NFS or raid0/5 in BTRFS and ZFS and such, where there are other larger alignment structures to consider. Also for large-blocks filesystems/devices who would like IO aligned on bigger than a page sizes.

Thanks
Boaz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#5 lore raw
In reply to Boaz Harrosh

Since a lot of these are mm related; added linux-mm to cc list

Quoted 35 lines · Boaz Harrosh

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

This can solve the IO fairness and latency bound (interactivness) of small IOs.
There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

Quoted 8 lines

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

Quoted 9 lines

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

Since a bdi is coupled to a gendisk and a queue, why isn't optimal_io_size what you want?

Quoted 14 lines

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

Quoted 7 lines

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

This is essentially what we've been discussing in "Fixing Writeback" for the last two years, isn't it (the fact that we have multiple sources of writeback and they don't co-ordinate properly). I thought our solution was to prefer linear over seeky ... adding a mutex makes that more absolute than a preference, but are you sure it helps (especially as it adds a lock to the writeout path).

James

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 14 lines · James Bottomley

Since a lot of these are mm related; added linux-mm to cc list

Quoted 11 lines · Boaz Harrosh

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

Quoted 7 lines
Quoted 2 lines

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

The situations that required writeback for reclaim to make progress have shrunk a lot with this merge window because of respecting page reserves in the dirty limits, and per-zone dirty limits.

What's left to evaluate are certain NUMA configurations where the dirty pages are concentrated on a few nodes. Currently, we kick the flushers from direct reclaim, completely undirected, just "clean some pages, please". That works for systems up to a certain size, depending on the size of the node in relationship to the system as a whole (likelihood of pages cleaned being from the target node) and how fast the backing storage is (impact of cleaning 'wrong' pages).

So while the original problem is still standing, the urgency of it might have been reduced quite a bit or the problem itself might have been pushed into a corner where workarounds (spread dirty data more evenly e.g.) might be more economical than trying to make writeback node-aware and deal with all the implications (still have to guarantee dirty cache expiration times for integrity; can fail spectacularly when there is little or no relationship between disk placement and memory placement, imagine round-robin allocation of disk-contiguous dirty cache over a few nodes).

I agree with James: find scenarios where workarounds are not feasible but that are important enough that the complexity would be justified. Otherwise, talking about how to fix them is moot.

#7 lore raw
Quoted 45 lines · Johannes Weiner
Quoted 14 lines · James Bottomley

Since a lot of these are mm related; added linux-mm to cc list

Quoted 11 lines · Boaz Harrosh

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

Quoted 7 lines
Quoted 2 lines

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

All of the above are complex. The only reason for adding complexity in our writeback path should be because we can demonstrate that it's actually needed. In order to demonstrate this, you'd need performance measurements ... is there a plan to get these before the summit?

The situations that required writeback for reclaim to make progress have shrunk a lot with this merge window because of respecting page reserves in the dirty limits, and per-zone dirty limits.

What's left to evaluate are certain NUMA configurations where the dirty pages are concentrated on a few nodes. Currently, we kick the flushers from direct reclaim, completely undirected, just "clean some pages, please". That works for systems up to a certain size, depending on the size of the node in relationship to the system as a whole (likelihood of pages cleaned being from the target node) and how fast the backing storage is (impact of cleaning 'wrong' pages).

So while the original problem is still standing, the urgency of it might have been reduced quite a bit or the problem itself might have been pushed into a corner where workarounds (spread dirty data more evenly e.g.) might be more economical than trying to make writeback node-aware and deal with all the implications (still have to guarantee dirty cache expiration times for integrity; can fail spectacularly when there is little or no relationship between disk placement and memory placement, imagine round-robin allocation of disk-contiguous dirty cache over a few nodes).

Quoted 3 lines

I agree with James: find scenarios where workarounds are not feasible but that are important enough that the complexity would be justified. Otherwise, talking about how to fix them is moot.

Fine so IO-less page-reclaim is moot. What do I know I've never seen a NUMA machine. But that was just a by product of half a section of a list of 8 sections. Are all these moot? I must be smoking something good ;-)

Thanks
Boaz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

reviewer
#8 lore raw
In reply to Boaz Harrosh
Quoted 8 lines · Boaz Harrosh

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

Quoted 6 lines

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.
* BDI writeout should be smarter then a round robin cycle of SBs per BDI / inodes. It should be time based, writing the oldest data first. (Take the lowest indexed page of an inode as the dirty time of the inode. maybe also keep an oldest modified inode per-SB of a BDI)

As I wrote in the other thread, we are a bit smarter by using
i_dirtied_when timestamp. But not much. But it's hard to do without introducing rather big memory cost (e.g. something like per-page timestamps which you suggest). So if you have some solution without big overhead then I'm happy to listen to that.

Quoted 2 lines

This can solve the IO fairness and latency bound (interactivness) of small IOs.

As I also said in the other thread writeback IMO isn't the right place to
solve problems of small vs big IO. Writeback should more or less guarantee that data get to disk before certain time to assure reasonable behavior after crash. We also try to be fair among files but that's basically our way how to get data to disk early enough. I don't know about any other fairness that would make sense to be handled in writeback code.

Quoted 9 lines

There might be other solutions to this problem, any Ideas?

* Introduce an "aging time" factor of an inode which will postpone the writeout of an inode to the next writeback timer if the inode has "just changed".

This can solve the problem of an application doing heavy modification of some area of a file and the writeback timer sampling that change too soon and forcing pages to change during IO, as well as having split IO where waiting for the next cycle could have the complete modification in a singe submit.

But it also brings some problems - like avoiding to postpone writeback
forever. The devil is in the details here I believe and I thought about similar ideas some time ago and I didn't come up with anything reasonably simple and working better than current simple scheme.

Quoted 5 lines

[Targeted writeback (IO-less page-reclaim)]
Sometimes we would need to write a certain page or group of pages. It could be nice to prioritize/start the writeback on these pages, through the regular writeback mechanism instead of doing direct IO like today.

Quoted 8 lines

This is actually related to above where we can have a "write_now" time constant that makes the priority of that inode to be written first. Then we also need the page-info that we want to write as part of that inode's IO. Usually today we start at the lowest indexed page of the inode, right? In targeted writeback we should make sure the writeout is the longest contiguous (aligned) dirty region containing the targeted page.

With this in place we can also move to an IO-less page-reclaim. that is done entirely by the BDI thread writeback. (Need I say more)

Again, expensive to track IMHO. Also as Johannes wrote, IO-less
page-reclaim may be less urgent in recent kernels.

Quoted 9 lines

[Aligned IO]

Each BDI should have a way to specify it's Alignment preferences and optimum IO sizes and the VFS writeout can take that into consideration when submitting IO.

This can both reduce lots of work done at individual filesystems, as well as benefit lots of other filesystems that did not take care of this. It can also make the life of some of the FSs that do care, a lot easier. Producing IO patterns that are much better then what can be achieved today with the FS trying to second guess the VFS.

This is probably doable and may be reasonable. Just currently writeback
code has no idea where particular page lands on disk (mapping logical offset->physical block is in filesystem hands). But this might be reasonably doable. Just someone has to write a code to expose enough information from filesystems to writeback.

Quoted 14 lines

[IO less sync]

This topic is actually related to the above Aligned IO.

In today's code, in a regular write pattern, when an application is writing a long enough file, we have two sources of threads for the .write_pages vector. One is the BDI write_back thread, the other is the sync operation. This produces nightmarish IO patterns when the write_cache_pages() is re-entrant and each instance is fighting the other in garbing random pages, this is bad because of two reasons:
1. makes each instance grab a none contiguous set of pages which causes the IO to split and be none-aligned.
2. Causes Seeky IO where otherwise the application just wrote linear IO of a large file and then sync.

Quoted 7 lines

The IO pattern is so bad that in some cases it is better to serialize the call to write_cache_pages() to avoid it. Even with the cost of a Mutex at every call

Would it be hard to have "sync" set some info, raise a flag, fire up the writeback and wait for it to finish? writeback in it's turn should switch to a sync mode on that inode. (The sync operation need not change the writeback priority in my opinion like today)

We already have I_SYNC inode flag for this used in
writeback_single_inode(). Just fsync path currently seems to avoid writeback_single_inode() so that exclusion doesn't quite work it. I'm not sure if I_SYNC was originally intended to provide exlusion against fsync. But in either case I belive this particular problem can be somehow resolved.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
In reply to Boaz Harrosh
Quoted 5 lines · Boaz Harrosh

Hi

Now that we have the "IO-less dirty throttling" in and kicking (ass I might say) Are there plans for second stage? I can see few areas that need some love.

Quoted 7 lines

[IO Fairness, time sorted writeback, properly delayed writeback]

As we started to talk about in another thread: "[LSF/MM TOPIC] a few storage topics" I would like to propose the following topics:

* Do we have enough information for the time of dirty of pages, such as the IO-elevators information, readily available to be used at the VFS layer.

Assuming it is available, what's the plan. How would VFS layer make use of it?

Thanks
Vivek