new data logging and quota patches available
PATCH 1/1 new data logging and quota patches available
Hello all,
ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be updated with a new set of data logging and quota patches against 2.4.21-pre4
The data logging code is updated with another set of io stalling fixes, they should improve performance of data=ordered and data=writeback by being smarter about forcing commits under heavy write load and kicking kreiserfsd.
Treat these with care, they've gotten a ton of testing under the suse kernel, but the port to vanilla was just done today.
The quota patches include a fix for incorrect sd_block counts on symlinks.
-chris
Hi Chris,
Well, this is excellent :)
Does that include the jounal overflow fix ?
Thanks,
Philippe
On 21 Feb 2003 18:32:11 -0500
Chris Mason <[email protected]> wrote:
| Hello all, | | ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be | updated with a new set of data logging and quota patches against | 2.4.21-pre4 | | The data logging code is updated with another set of io stalling fixes, | they should improve performance of data=ordered and data=writeback by | being smarter about forcing commits under heavy write load and kicking | kreiserfsd. | | Treat these with care, they've gotten a ton of testing under the suse | kernel, but the port to vanilla was just done today. | | The quota patches include a fix for incorrect sd_block counts on | symlinks. | | -chris | |
Am Samstag, 22. Februar 2003 00:32 schrieb Chris Mason:
Quoted 6 lines
Hello all,
ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be updated with a new set of data logging and quota patches against 2.4.21-pre4
Quoted 7 lines
The data logging code is updated with another set of io stalling fixes, they should improve performance of data=ordered and data=writeback by being smarter about forcing commits under heavy write load and kicking kreiserfsd.
Treat these with care, they've gotten a ton of testing under the suse kernel, but the port to vanilla was just done today.
Which files are needed for 2.4.21-pre4aa3?
Have a nice weekend.
-Dieter
Quoted 1 line · Dieter Nützel
Which files are needed for 2.4.21-pre4aa3?
ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21-aa
I just uploaded a fresh set of data logging and quota patches for -aa
-chris
And here's one more patch for those of you that want to experiment a little. reiserfs_dirty_inode gets called to log the inode every time the vfs layer does an atime/mtime/ctime update, which is one of hte reasons mounting with -o noatime and -o nodiratime makes things faster. We had to do this because otherwise kswapd can end up trying to write inodes, which can lead to deadlock as he tries to wait on the log.
One of the patches in my data logging directory is kinoded-8, which changes things so a new kinoded does all inode writeback instead of kswapd.
That means that if you apply up to 05-data-logging-36 and then apply 09-kinoded-8 (you won't need any of the other quota patches), you can also apply this patch. It changes reiserfs to leave inodes dirty, which saves us lots of time on atime/mtime updates.
I'l upload this after it gets a little additional testing, but wanted to include it here in case anyone else was interested in benchmarking.
fs/reiserfs/inode.c +8 -1
@@ -1492,12 +1498,19 @@** does something when called for a synchronous update. */ void reiserfs_write_inode (struct inode * inode,int do_sync) { - +struct reiserfs_transaction_handle th;if (inode->i_sb->s_flags &MS_RDONLY ) {reiserfs_warning ("clm-6005: writing inode %lu on readonly FS\n" , inode->i_ino) ;return ; } + +lock_kernel () ; +journal_begin (&th, inode->i_sb,1 ) ; +reiserfs_update_sd (&th, inode); +journal_end (&th, inode->i_sb,1 ) ; +unlock_kernel () ; +/* memory pressure can sometimes initiate write_inode calls with sync == 1, ** these cases are just when the system needs ram, not when the ** inode needs to reach disk for safety, and they can safely be
fs/reiserfs/super.c +11 -8
@@ -421,15 +422,18 @@ inode->i_ino) ;return ; } -lock_kernel () ; - -/* this is really only used for atime updates, so they don't have -** to be included in O_SYNC or fsync +/* this is used for atime updates, so they don't have +** to be included in O_SYNC or fsync, but if we are nesting in to +** a transaction, journal_begin/end is free, so we might as well +** log the inode now (plus quota depends on it) */ -journal_begin (&th, inode->i_sb,1 ) ; -reiserfs_update_sd (&th, inode); -journal_end (&th, inode->i_sb,1 ) ; -unlock_kernel () ; +if (reiserfs_transaction_running (inode->i_sb)) { +lock_kernel () ; +journal_begin (&th, inode->i_sb,1 ) ; +reiserfs_update_sd (&th, inode); +journal_end (&th, inode->i_sb,1 ) ; +unlock_kernel () ; + } }static void reiserfs_clear_inode (struct inode *inode)
Quoted 15 lines · Chris Mason
And here's one more patch for those of you that want to experiment a
little.reiserfs_dirty_inodegets called to log the inode every time
the vfs layer does an atime/mtime/ctime update, which is one of hte
reasons mounting with -o noatime and -o nodiratime makes things faster. We had to do this because otherwise kswapd can end up trying to write inodes, which can lead to deadlock as he tries to wait on the log.One of the patches in my data logging directory is kinoded-8, which changes things so a new kinoded does all inode writeback instead of kswapd.
That means that if you apply up to 05-data-logging-36 and then apply 09-kinoded-8 (you won't need any of the other quota patches), you can also apply this patch. It changes reiserfs to leave inodes dirty, which saves us lots of time on atime/mtime updates.
Quoted 4 lines
I'l upload this after it gets a little additional testing, but wanted to include it here in case anyone else was interested in benchmarking.
[11.dirty-inodes-for-kinoded.diff]
Hi Chris!
At least I'm not able to copy through my partition via "cp -a ..." with the last supposed setup + preempt without quota ...
When umounting the destination partition it says "device is busy".
I don't know what is gone for now on my repository partition, but I know from earlier times this should not happen.
Bye,
Manuel
Quoted 22 lines · Manuel Krause
Quoted 15 lines · Chris Mason
And here's one more patch for those of you that want to experiment a
little.reiserfs_dirty_inodegets called to log the inode every time
the vfs layer does an atime/mtime/ctime update, which is one of hte
reasons mounting with -o noatime and -o nodiratime makes things faster. We had to do this because otherwise kswapd can end up trying to write inodes, which can lead to deadlock as he tries to wait on the log.One of the patches in my data logging directory is kinoded-8, which changes things so a new kinoded does all inode writeback instead of kswapd.
That means that if you apply up to 05-data-logging-36 and then apply 09-kinoded-8 (you won't need any of the other quota patches), you can also apply this patch. It changes reiserfs to leave inodes dirty, which saves us lots of time on atime/mtime updates.
Quoted 4 lines
I'l upload this after it gets a little additional testing, but wanted to include it here in case anyone else was interested in benchmarking.
[11.dirty-inodes-for-kinoded.diff]
Quoted 7 lines
Hi Chris!
At least I'm not able to copy through my partition via "cp -a ..." with the last supposed setup + preempt without quota ...
When umounting the destination partition it says "device is busy".
I don't know what is gone for now on my repository partition, but I know from earlier times this should not happen.
Sorry, I was incomplete with my report, at least with the following: After thinking over the remount strategy for some minutes with a pizza the partition was umountable again. I don't have exact values but it was under about 15 minutes max. with 3.5GB copied previously.
My setup in fact:
m02-akpm-b_journal_head-1.diff
m03-relocation-6.diff
m04-reiserfs-sync_fs-1.diff
m05-data-logging-36.diff
m06-logging-export.diff
m06-write_times.diff
m09-kinoded-8.diff
m11.inode-dirty-for-kinoded-ML.diff
(prefix m for you)
+ latest preempt patch (what was for 2.4.21-pre1).
Thanks,
Manuel
Quoted 32 lines · Manuel Krause
Quoted 31 lines · Manuel Krause
Quoted 17 lines · Chris Mason
And here's one more patch for those of you that want to experiment a little.
reiserfs_dirty_inodegets called to log the inode every time the vfs layer does an atime/mtime/ctime update, which is one of hte reasons mounting with -o noatime and -o nodiratime makes things faster. We had to do this because otherwise kswapd can end up trying to write
inodes, which can lead to deadlock as he tries to wait on the log.One of the patches in my data logging directory is kinoded-8, which changes things so a new kinoded does all inode writeback instead of kswapd.
That means that if you apply up to 05-data-logging-36 and then apply 09-kinoded-8 (you won't need any of the other quota patches), you can also apply this patch. It changes reiserfs to leave inodes dirty, which saves us lots of time on atime/mtime updates.
Quoted 4 lines
I'l upload this after it gets a little additional testing, but wanted to include it here in case anyone else was interested in benchmarking.
[11.dirty-inodes-for-kinoded.diff]
Hi Chris!
At least I'm not able to copy through my partition via "cp -a ..." with the last supposed setup + preempt without quota ...
When umounting the destination partition it says "device is busy".
Quoted 7 lines
Quoted 2 lines
I don't know what is gone for now on my repository partition, but I know from earlier times this should not happen.
Sorry, I was incomplete with my report, at least with the following:
After thinking over the remount strategy for some minutes with a pizza the partition was umountable again. I don't have exact values but it was under about 15 minutes max. with 3.5GB copied previously.
Hmmm, I think I need to changed around kinoded slightly. Thanks for giving it a try...
-chris
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
-chris
On Sunday 23 February 2003 18:18, Chris Mason wrote:
Hi Chris,
Quoted 2 lines
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
do you consider telling us the changes? ;)
ciao, Marc
On Sunday 23 February 2003 18:37, Marc-Christian Petersen wrote:
Hi again,
Quoted 3 lines
Quoted 2 lines
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
do you consider telling us the changes? ;)
_hopefully_ an SMP deadlock while massive disk I/O. :)
ciao, Marc
Quoted 8 lines · Marc-Christian Petersen
On Sunday 23 February 2003 18:37, Marc-Christian Petersen wrote:
Hi again,
Quoted 3 lines
Quoted 2 lines
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
do you consider telling us the changes? ;)
_hopefully_ an SMP deadlock while massive disk I/O. :)
No, the new kinoded-9.diff just changes how kinoded gets woken up, to do a better job of limiting the number of unused inodes (but still allocated) on the box.
So, which set of patches were you using when you saw the deadlock? If you can reproduce, could you please capture and decode the output of sysrq-t?
-chris
On Sunday 23 February 2003 20:33, Chris Mason wrote:
Hi Chris,
Quoted 7 lines
Quoted 1 line
_hopefully_ an SMP deadlock while massive disk I/O. :)
No, the new kinoded-9.diff just changes how kinoded gets woken up, to do a better job of limiting the number of unused inodes (but still allocated) on the box.
So, which set of patches were you using when you saw the deadlock? If you can reproduce, could you please capture and decode the output of sysrq-t?
Unfortunately my SMP box died some time ago (end Dec. 2002) so I cannot reproduce this yet. It works great with UP machines but I had bad deadlocks with SMP and kinoded-8 which was also in the SuSE 2.4.19 kernel tree.
Let me find some informations I wrote down that time. If I found them I write them down here.
ciao, Marc
Quoted 4 lines · Chris Mason
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
-chris
Hi and thank you!
I've replaced the old patch. And the problem is gone -- the partitions are umountable again after copying.
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
Bye,
Manuel
Quoted 11 lines · Manuel Krause
Quoted 4 lines · Chris Mason
I've attached a new kinoded-9.diff.gz, it will apply to both -aa and vanilla. Please give it a try.
-chris
Hi and thank you!
I've replaced the old patch. And the problem is gone -- the partitions are umountable again after copying.
Good to hear
Quoted 6 lines
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
That's odd, tests here show performance increasing but I'm testing 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
Could I talk you into trying to narrow this down? Perhaps starting with a benchmark of the same dataset on both kernels? The new data logging code kicks kreiserfsd in sooner to service async commits. So if your benchmark lasts < 30 seconds, it is possible the older kernels never did writes at all during the run.
-chris
Quoted 3 lines · Chris Mason
Quoted 2 lines · Manuel Krause
On 02/23/2003 06:18 PM, Chris Mason wrote:
[...]
Quoted 11 lines
Quoted 2 lines
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
That's odd, tests here show performance increasing but I'm testing 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
Could I talk you into trying to narrow this down? Perhaps starting with a benchmark of the same dataset on both kernels? The new data logging code kicks kreiserfsd in sooner to service async commits. So if your benchmark lasts < 30 seconds, it is possible the older kernels never did writes at all during the run.
No, the copy usually takes about 9 min. to complete and I umount the partitions to take (more) care everything went over correctly.
Yes, I can retest it with the old data-logging, too. But before just one clarification for "old data logging": Do you mean just the kinoded patches removed or even data-logging-35 instead of -36 ?
Thanks,
Manuel
Quoted 11 lines · Manuel Krause
Quoted 3 lines · Chris Mason
Quoted 2 lines · Manuel Krause
On 02/23/2003 06:18 PM, Chris Mason wrote:
[...]
Quoted 5 lines
Quoted 1 line
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
That's odd, tests here show performance increasing but I'm testing 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
Quoted 7 lines
Quoted 5 lines
Could I talk you into trying to narrow this down? Perhaps starting with a benchmark of the same dataset on both kernels? The new data logging code kicks kreiserfsd in sooner to service async commits. So if your benchmark lasts < 30 seconds, it is possible the older kernels never did writes at all during the run.
No, the copy usually takes about 9 min. to complete and I umount the partitions to take (more) care everything went over correctly.
(And this time is included when I compare the timing values.)
Quoted 2 lines
Yes, I can retest it with the old data-logging, too. But before just one clarification for "old data logging": Do you mean just the kinoded patches removed or even data-logging-35 instead of -36 ?
So you seem to agree...? ;-) O.k.
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
Best regards,
Manuel
Quoted 20 lines · Manuel Krause
Quoted 11 lines · Manuel Krause
Quoted 3 lines · Chris Mason
Quoted 2 lines · Manuel Krause
On 02/23/2003 06:18 PM, Chris Mason wrote:
[...]
Quoted 5 lines
Quoted 1 line
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
That's odd, tests here show performance increasing but I'm testing 2.4.21-pre4 + old data logging vs 2.4.21-pre4 + new data logging
Quoted 7 lines
Quoted 5 lines
Could I talk you into trying to narrow this down? Perhaps starting with a benchmark of the same dataset on both kernels? The new data logging code kicks kreiserfsd in sooner to service async commits. So if your benchmark lasts < 30 seconds, it is possible the older kernels never did writes at all during the run.
No, the copy usually takes about 9 min. to complete and I umount the partitions to take (more) care everything went over correctly.
Quoted 7 lines
(And this time is included when I compare the timing values.)
Quoted 1 line
Yes, I can retest it with the old data-logging, too. But before just one clarification for "old data logging": Do you mean just the kinoded patches removed or even data-logging-35 instead of -36 ?
So you seem to agree...? ;-) O.k.
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
But... though you were right with 2.4.21 data-logging performance relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???
A usual problem in my mind: Due to performance issues I would like to downgrade -- due to recent bugfixes and your additional enhancements, I would go with a performance loss of ~16.5%...
Should I really need to retest all kernels without data-logging patches on here to get the truth?
Mmmh. No, I think you should know the reasons and explain them when I'm just clueless upon my results...
Best regards to you and all others,
Manuel
Quoted 9 lines
Quoted 4 lines
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
But... though you were right with 2.4.21 data-logging performance
relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???
Did you exclude your changing dataset as a potential cause? If not I'm very interested in the details.
-chris
Quoted 7 lines · Chris Mason
Quoted 5 lines
Quoted 1 line
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
But... though you were right with 2.4.21 data-logging performance relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???
Did you exclude your changing dataset as a potential cause? If not I'm very interested in the details.
Please, tell me how to respond to this question!
I didn't want to _exclude_ it as a potential cause but felt in need to write it for the records. The test partition varies from day to day and to recreate a fillup rate 71%~73% relating to my previous tests was hard enough. I'm not able to hold a "reiserfs-special-testing-partition" on here... Mmh.
Unfortunately I didn't do the file-size-distribution one that time so you need to give me further hints on how to test my usual backup copies more valuably for you -- but only with these copies.
But whatever I did with that disk content at ~72% usage and staying error-free -- YOU can't blame it to the content distribution. Not 19% degression in copy rate since 2.4.19. Explain it, please!
... But now we have 2.4.21-pre5 with direct-io and iget5-patches already...
Thank you very much,
Manuel
P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs recent engagements...)
Quoted 5 lines · Manuel Krause
Quoted 4 lines · Chris Mason
Quoted 4 lines
Quoted 3 lines
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
But... though you were right with 2.4.21 data-logging performance
relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???Did you exclude your changing dataset as a potential cause? If not I'm very interested in the details.
Please, tell me how to respond to this question!
Sorry, I wasn't clear. It sounds like you've got two very different copy speeds on two possibly different data sets. The problem will be significantly easier to debug and discuss if you boot a 2.4.19 + data logging kernel, do the test, and then do the test again on a 2.4.21-pre data logging kernel.
I know this is a pain, I want to reduce the number of variables in place.
Quoted 2 lines
P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs recent engagements...)
Oleg is making patches for the -ac series. He also merged a special set for Philippe, they pay namesys for support ;-)
I'm remerging my work over the last week and should have a new set of patches out soon.
-chris
Quoted 16 lines · Chris Mason
Quoted 4 lines · Manuel Krause
Quoted 2 lines · Chris Mason
Quoted 2 lines
Quoted 1 line
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
But... though you were right with 2.4.21 data-logging performance relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???
Did you exclude your changing dataset as a potential cause? If not I'm very interested in the details.
Please, tell me how to respond to this question!
Quoted 9 lines
Sorry, I wasn't clear. It sounds like you've got two very different copy speeds on two possibly different data sets. The problem will be significantly easier to debug and discuss if you boot a 2.4.19 + data logging kernel, do the test, and then do the test again on a 2.4.21-pre data logging kernel.
I know this is a pain, I want to reduce the number of variables in place.
Yes. That all is why I asked. I'll try to find and retest the old things tomorrow night. The new setup will go fine with 2.4.21-pre5 ? Or is this one more step of irritation? Mmm. Or should I wait for the patches you announce below?
Thank you, good "night",
Manuel
Quoted 5 lines
Quoted 2 lines
P.S.: Do the data-logging things go to namesys now? (Me, watching Olegs recent engagements...)Oleg is making patches for the -ac series. He also merged a special set for Philippe, they pay namesys for support ;-)
My additional thanks go to Philippe ! :-)))
Quoted 4 lines
I'm remerging my work over the last week and should have a new set of patches out soon.
-chris
Quoted 25 lines · Manuel Krause
Quoted 23 lines · Chris Mason
Quoted 5 lines · Manuel Krause
Quoted 3 lines · Chris Mason
Quoted 3 lines
Quoted 1 line
Comparing data-logging-35 vs data-logging-36+kinoded-9+dirty_inode I have a performance increase of 2.5% for the new data-logging. Like you say above.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
But... though you were right with 2.4.21 data-logging performance relations: --> What was causing that performance loss between 2.4.19 +data-logging and now???
Did you exclude your changing dataset as a potential cause? If not I'm very interested in the details.
Please, tell me how to respond to this question!
Sorry, I wasn't clear. It sounds like you've got two very different copy speeds on two possibly different data sets. The problem will be significantly easier to debug and discuss if you boot a 2.4.19 + data logging kernel, do the test, and then do the test again on a 2.4.21-pre data logging kernel.
Quoted 6 lines
Quoted 3 lines
I know this is a pain, I want to reduce the number of variables in place.
Yes. That all is why I asked. I'll try to find and retest the old things tomorrow night. The new setup will go fine with 2.4.21-pre5 ? Or is this one more step of irritation? Mmm. Or should I wait for the patches you announce below?
Hi, and sorry for the delay.
So I copied over the same dataset several times. ;-)
The times measured include the umounts of both partitions. The test dataset was freshly copied to the freshly created
test FS under the same kernel version before timing.
This time I took the times of two subsequent runs. The test script does only simple "cp -a ..."
/dev/hda11 5076348 3762908 1313420 75% /mnt/alpha /dev/hdd11 5076348 3762908 1313440 75% /mnt/beta
2.4.21-pre5 data=ordered [A]
first run second run diff average rate real 9m05.375s 9m23.272s 3.2% 9m14.324s 6.63MB/s user 0m01.440s 0m01.730s sys 1m11.640s 1m08.870s
2.4.19 data=ordered [B]
first run second run diff average rate real 7m46.287s 7m34.031s 2.7% 7m40.159s 7.99MB/s user 0m01.520s 0m01.580s sys 1m21.920s 1m21.300s
-> I didn't know that subsequent runs on the same conditions
differ by ~3% ( -> So don't take my statement of the 2.5% difference in my previous mail for that serious as I only timed one run that time)
-> The throughput rate under 2.4.19 is about 20% higher
than under 2.4.21-pre5 with this dataset
I hope it's "visible" this time... ;-))
Best regards,
Manuel
P.S.: Disk data distribution file is available if you want.
Below the list of used patches just to be complete...
[A] Patches for 2.4.21-pre5:
01.m02-akpm-b_journal_head-1.diff 02.m03-relocation-6.diff 03.m04-reiserfs-sync_fs-1.diff 04.m05-data-logging-36.namesys.diff 05.m06-logging-export.diff 06.m06-write_times.diff 07.m09.kinoded-9.diff 08.m11.inode-dirty-for-kinoded.ML.diff 09.iget5_locked_for_datalogging.namesys.noquota.diff preempt-kernel-rml-2.4.21-pre5-pre1-0.patch
[B] Patches for 2.4.19:
01.change_help_wording.diff
02.change_menu_wording.diff
03.confusing_journal_replay_warning_removal.diff
05.hash_on_empty_fs-1.diff
07.m01-relocation-4.diff
08.m02-commit_super-9-relocation.diff
09.m03-data-logging-25.diff
10.m03-logging-export.diff
11.m04-write_times.diff
12.m05-search_reada-4.diff
preempt-kernel-rml-2.4.19-2.patch
Quoted 11 lines · Manuel Krause
Hi, and sorry for the delay.
So I copied over the same dataset several times. ;-)
The times measured include the umounts of both partitions. The test dataset was freshly copied to the freshly created
test FS under the same kernel version before timing.
This time I took the times of two subsequent runs. The test script does only simple "cp -a ..."
Thank you very much for the details. Just to make sure I understand how you are generating the numbers, could I please see the test script?
It sounds like you:
copy dataset to fresh fs
time copy dataset from freshfs sub1 to freshfs sub2?
is that right?
-chris
Quoted 18 lines · Chris Mason
Quoted 10 lines · Manuel Krause
Hi, and sorry for the delay.
So I copied over the same dataset several times. ;-)
The times measured include the umounts of both partitions. The test dataset was freshly copied to the freshly created
test FS under the same kernel version before timing.
This time I took the times of two subsequent runs. The test script does only simple "cp -a ..."Thank you very much for the details. Just to make sure I understand how you are generating the numbers, could I please see the test script?
Quoted 7 lines
It sounds like you:
copy dataset to fresh fs
time copy dataset from freshfs sub1 to freshfs sub2?is that right?
Hi!
What do you mean with "sub"? Sub-directories?
My procedure is the following:
->
mkreiserfs partition2 (second drive)
copy dataset from used partition1 to partition2
(identical, yes)
mkreiserfs partition1 (first drive)
time (copy dataset from partition2 to partition1
+ umount partitions 1 and 2)
<-
In fact that are two dumb scripts from years ago - one for the first direction and one for the second. The time command line is the following:
time sh -c "cp -ax /mnt/alpha/. /mnt/beta/ ; umount /mnt/alpha ;
umount /mnt/beta"
And again I may have forgotten to mention that I always mount my partitions with "notail":
mount -t reiserfs -o noatime,notail,data=ordered /dev/hd... /mnt/...
Does this information make it more understandable?
Bye,
Manuel
Quoted 21 lines · Manuel Krause
My procedure is the following:
->
mkreiserfs partition2 (second drive)copy dataset from used partition1 to partition2
(identical, yes)mkreiserfs partition1 (first drive)
time (copy dataset from partition2 to partition1
+ umount partitions 1 and 2)<-
In fact that are two dumb scripts from years ago - one for the first direction and one for the second. The time command line is the following:
time sh -c "cp -ax/mnt/alpha/./mnt/beta/; umount/mnt/alpha;
umount/mnt/beta"
Quoted 5 lines
And again I may have forgotten to mention that I always mount my partitions with "notail":
mount -t reiserfs -o noatime,notail,data=ordered/dev/hd.../mnt/...Does this information make it more understandable?
Very much, thanks. Thanks to some work Jeff recently did, my current thought is the slow down is related to the block allocator, and the search_by_reada patch. Jeff and I are working up numbers to see if this is right, more details on monday.
-chris
Quoted 1 line · Chris Mason
On Sun, 2003-03-09 at 12:07, Manuel Krause wrote:
[...]
Quoted 9 lines
Quoted 2 lines
Does this information make it more understandable?
Very much, thanks. Thanks to some work Jeff recently did, my current thought is the slow down is related to the block allocator, and the
search_by_readapatch. Jeff and I are working up numbers to see if this is right, more details on monday.
I hope you don't blame the search_by_reada patch itself for a slowdown. There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you integrate that patch in your other patches since then - and excluded it?
Bye,
Manuel
Quoted 9 lines · Manuel Krause
Quoted 5 lines
Very much, thanks. Thanks to some work Jeff recently did, my current thought is the slow down is related to the block allocator, and the
search_by_readapatch. Jeff and I are working up numbers to see if this is right, more details on monday.I hope you don't blame the
search_by_readapatch itself for a slowdown. There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you integrate that patch in your other patches since then - and excluded it?
Just the opposite, I think the search_reada patch helps, but not with the default allocator in 2.4.21. The metadata seems to be spread too far apart there. I promise, more details on monday after we get some additional numbers to back things up ;-)
-chris
Quoted 3 lines · Manuel Krause
I hope you don't blame the
search_by_readapatch itself for a slowdown. There is a 05-search_reada-4.diff in my said 2.4.19 setup. Or didn't you integrate that patch in your other patches since then - and excluded it?
Ok, our results so far are really interesting, they basically show the default allocator in 2.4.20 & 2.4.21 fragmenting badly as the number of concurrent writers increases.
But, you've only got one writer, so I need to do a better job of modeling your data set. Could you please send along the file size distribution?
-chris
[ slow down between 2.419 data logging and 2.4.21-pre data logging ]
Ok, could you please apply the search_reada-4 patch I sent and do your benchmark again on 2.4.21-preX with these mount options:
mount -o alloc=old_way:concentrating_formatted_nodes,notail
It's about 10% faster here, but my dataset is slightly different.
-chris
Hello!
Quoted 4 lines · Chris Mason
[ slow down between 2.419 data logging and 2.4.21-pre data logging ] Ok, could you please apply the search_reada-4 patch I sent and do your
It's about 10% faster here, but my dataset is slightly different.
Hm. Last time we had conversation with you on this patch (a year ago? may be even more), you said this patch makes no difference to you at all. And we thought this is because your system is all-scsi, and scsi appears to be more intelligent about caching/readahead.
Have you switched to IDE already? ;)
Bye,
Oleg
Quoted 6 lines · Oleg Drokin
Hello!
Quoted 2 lines · Chris Mason
[ slow down between 2.419 data logging and 2.4.21-pre data logging ] Ok, could you please apply the search_reada-4 patch I sent and do your
Quoted 7 lines
Quoted 1 line
It's about 10% faster here, but my dataset is slightly different.
Hm. Last time we had conversation with you on this patch (a year ago? may be even more), you said this patch makes no difference to you at all. And we thought this is because your system is all-scsi, and scsi appears to be more intelligent about caching/readahead.
Have you switched to IDE already? ;)
No, that was on a scsi drive ;-) Which patch, the search_reada? Or mounting with the border?
search_reada doesn't make much difference without the border, and even then it really only helps on long sequential reads.
-chris
Hello!
Quoted 9 lines · Chris Mason
Quoted 8 lines
Quoted 3 lines
[ slow down between 2.419 data logging and 2.4.21-pre data logging ] Ok, could you please apply the search_reada-4 patch I sent and do your It's about 10% faster here, but my dataset is slightly different.
Hm. Last time we had conversation with you on this patch (a year ago? may be even more), you said this patch makes no difference to you at all. And we thought this is because your system is all-scsi, and scsi appears to be more intelligent about caching/readahead.
Have you switched to IDE already? ;)No, that was on a scsi drive ;-) Which patch, the
search_reada? Or
Hm. Something have changed then. I can dig the email, if you want.
Quoted 1 line
mounting with the border?
search_reada patch.
Quoted 1 line
search_readadoesn't make much difference without the border, and even
Sure.
Quoted 1 line
then it really only helps on long sequential reads.
That's true, but it made no difference for you on SCSI for sequentional reads too.
Bye,
Oleg
Quoted 4 lines · Oleg Drokin
Quoted 2 lines
then it really only helps on long sequential reads.
That's true, but it made no difference for you on SCSI for sequentional reads too.
That's true, we were comparing your IDE drive results with my scsi for reading a single big file. It probably didn't help for the single big file because of the page cache readahead triggering parallel tree searches.
But Manuel's test is different, it's doing a long sequential read of multiple files. The page cache code only does readahead on one file at a time, reading fileA won't trigger a parallel read of the indirect items for fileB. search_reada will work better in that case, since it will grab indirect items for fileB.
Also, it will do readahead on the directory items, probably making directory scanning faster.
-chris
Quoted 9 lines · Chris Mason
[ slow down between 2.419 data logging and 2.4.21-pre data logging ]
Ok, could you please apply the search_reada-4 patch I sent and do your benchmark again on 2.4.21-preX with these mount options:
mount -o alloc=old_way:concentrating_formatted_nodes,notail
It's about 10% faster here, but my dataset is slightly different.
I did it and also tested without the alloc mount option from above.
Here is the overview including the "old" things to refer to. The dataset didn't change in the meantime.
(old)
2.4.21-pre5 data=ordered [A]
first run second run diff average rate real 9m05.375s 9m23.272s 3.2% 9m14.324s 6.63MB/s user 0m01.440s 0m01.730s sys 1m11.640s 1m08.870s
2.4.19 data=ordered [B]
first run second run diff average rate real 7m46.287s 7m34.031s 2.7% 7m40.159s 7.99MB/s user 0m01.520s 0m01.580s sys 1m21.920s 1m21.300s
(new)
2.4.21-pre5 [A] + search_reada-4 +
mount -o noatime,alloc=old_way:concentrating_formatted_nodes,notail [C]
first run second run diff average rate real 7m59.631s 7m57.167s 0.52% 7m58.399s 7.68MB/s user 0m01.580s 0m01.250s sys 1m11.200s 1m09.520s
2.4.21-pre5 [A] + search_reada-4 without special mount options
(is: mount -o noatime,notail,data=ordered) [D]
first run second run diff average rate real 7m57.365s 7m55.638s 0.36% 7m56.502s 7.71MB/s user 0m01.510s 0m01.570s sys 1m07.730s 1m08.020s
* What means a performance increase of 2.4.21-pre5 with [D] vs. without search_reada-4 [A] by ~16% !!!
* The difference of the special mount option [C] against my default [D] is about 0.3% ;-))
* With search_reada-4 2.4.21-pre5 [D] only looses ~3.6% throughput rate against 2.4.19 [B]
...in this test.
Do you think the group-alloc-4 patch would increase throughput with this test?
Thank you for your great work,
Manuel
Quoted 8 lines · Manuel Krause
* What means a performance increase of 2.4.21-pre5 with [D] vs. without
search_reada-4 [A] by ~16% !!! * The difference of the special mount option [C] against my default [D] is about 0.3% ;-)) * With search_reada-4 2.4.21-pre5 [D] only looses ~3.6% throughput rate against 2.4.19 [B]...in this test.
Outstanding, things are starting to make sense.
Quoted 4 lines
Do you think the group-alloc-4 patch would increase throughput with this test?
No, it will make this test slower (this is the default 2.4.21 allocator's best case)
Quoted 1 line
Thank you for your great work,
Thank you for all the testing and analysis, it really helps us.
-chris
Quoted 58 lines · Manuel Krause
Quoted 9 lines · Chris Mason
[ slow down between 2.419 data logging and 2.4.21-pre data logging ]
Ok, could you please apply the search_reada-4 patch I sent and do your benchmark again on 2.4.21-preX with these mount options:
mount -o alloc=old_way:concentrating_formatted_nodes,notail
It's about 10% faster here, but my dataset is slightly different.
I did it and also tested without the alloc mount option from above.
Here is the overview including the "old" things to refer to. The dataset didn't change in the meantime.
(old)
2.4.21-pre5 data=ordered [A]first run second run diff average rate real 9m05.375s 9m23.272s 3.2% 9m14.324s 6.63MB/s user 0m01.440s 0m01.730s sys 1m11.640s 1m08.870s2.4.19 data=ordered [B]
first run second run diff average rate real 7m46.287s 7m34.031s 2.7% 7m40.159s 7.99MB/s user 0m01.520s 0m01.580s sys 1m21.920s 1m21.300s(new)
2.4.21-pre5 [A] + search_reada-4 +
mount -o noatime,alloc=old_way:concentrating_formatted_nodes,notail [C]first run second run diff average rate real 7m59.631s 7m57.167s 0.52% 7m58.399s 7.68MB/s user 0m01.580s 0m01.250s sys 1m11.200s 1m09.520s2.4.21-pre5 [A] + search_reada-4 without special mount options
(is: mount -o noatime,notail,data=ordered) [D]first run second run diff average rate real 7m57.365s 7m55.638s 0.36% 7m56.502s 7.71MB/s user 0m01.510s 0m01.570s sys 1m07.730s 1m08.020s* What means a performance increase of 2.4.21-pre5 with [D] vs. without search_reada-4 [A] by ~16% !!!
* The difference of the special mount option [C] against my default [D] is about 0.3% ;-))
* With search_reada-4 2.4.21-pre5 [D] only looses ~3.6% throughput rate against 2.4.19 [B]
...in this test.
Quoted 10 lines
Do you think the group-alloc-4 patch would increase throughput with this test?
Thank you for your great work,
Manuel
Can you state what test is being performed, what the hardware is, etc. In general, we need benchmarks that can be collected onto webpages and reviewed a year or two later when we can barely remember why we did or did not do some design decision.
Signature · 1 line
Hans
Quoted 6 lines · Hans Reiser
Quoted 4 lines · Manuel Krause
Quoted 2 lines · Chris Mason
[ slow down between 2.419 data logging and 2.4.21-pre data logging ]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Ok, could you please apply the search_reada-4 patch I sent and do your benchmark again on 2.4.21-preX with these mount options:
mount -o alloc=old_way:concentrating_formatted_nodes,notail
It's about 10% faster here, but my dataset is slightly different.
[ timing data ]
Quoted 10 lines
Quoted 10 lines
* What means a performance increase of 2.4.21-pre5 with [D] vs. without search_reada-4 [A] by ~16% !!!
* The difference of the special mount option [C] against my default [D] is about 0.3% ;-))
* With search_reada-4 2.4.21-pre5 [D] only looses ~3.6% throughput rate against 2.4.19 [B]
...in this test.
Quoted 9 lines
Quoted 7 lines
Do you think the group-alloc-4 patch would increase throughput with this test?
Thank you for your great work,
Manuel
Can you state what test is being performed, what the hardware is, etc. In general, we need benchmarks that can be collected onto webpages and reviewed a year or two later when we can barely remember why we did or did not do some design decision.
Hi Hans,
I think the way I "test" (is: simple timed backup-copy / defrag cycle) was effectively clarified in this thread.
Last autumn I wanted to add the desired system info to the ReiserFS regarding parts of my homepage but skipped it at some point simply lacking time...
You'll receive a compiled file of that info you might want, with the results I snipped above and the file size distribution data in the followup mail in private so you could give it to the persons of your team who collect it for the archives and/or prepare it for publishing. If you need differnt or more information, please, let me know.
Best regards,
Manuel
On Monday 24 February 2003 02:42, Manuel Krause wrote:
Hi Manuel,
Quoted 6 lines
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
I can confirm that this is also true for me. I am using the -aa data-logging 36.
ciao, Marc
On Monday 24 February 2003 18:59, Marc-Christian Petersen wrote:
Hi again,
Quoted 8 lines
Quoted 6 lines
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
I can confirm that this is also true for me. I am using the -aa data-logging 36.
grmpf, I forgot to add one thing here ;(
I just tried it with:
- time seq -f "%06.0f" 1 100000 | xargs touch
v35-aa took about 4 seconds to complete this.
v36-aa took about 6 seconds to compelte this.
Mounted with:
nosuid,nodev,noexec,noatime,nodiratime,notail,block-allocator=noborder
It is my SQUID cache directory.
ciao, Marc
Quoted 9 lines · Marc-Christian Petersen
I just tried it with:
- time seq -f "%06.0f" 1 100000 | xargs touch
v35-aa took about 4 seconds to complete this.
v36-aa took about 6 seconds to compelte this.Mounted with:
nosuid,nodev,noexec,noatime,nodiratime,notail,block-allocator=noborder
For benchmarks that short, bdflush/kupdate can come in and screw the results. Also, when I create 100,000 files on my test box with these mount parameters, it results in 3 large async transactions.
In data-logging-36, kreiserfsd will get those async commits to disk sooner than in data-logging-35, which is the right thing to do because it helps make sure later writes won't stall while we have to flush that async transaction.
But for a very short benchmark that doesn't do more work later on, it can seem slower. If you want a more accurate picture, include the time it takes to sync after creating all the files.
-chris
Quoted 4 lines · Marc-Christian Petersen
On Monday 24 February 2003 02:42, Manuel Krause wrote:
Hi Manuel,
Quoted 8 lines
Quoted 6 lines
I see a performance decrease with my copies in data=ordered mode by about 7.5% compared to the 2.4.20-pre10-data-logging (it's about 19% decrease for 2.4.21-pre4 since 2.4.19-data-logging). What is causing this? Partly I can blame it on the disk content distribution that changed. If I removed the kinoded related patches would the throughput rate increase?
I can confirm that this is also true for me. I am using the -aa data-logging 36.
Sorry, you confirm that with kinoded related patches removed (and the dirty_inode patch) things get faster?
Also, are you using kinoded-8 or kinoded-9?
-chris
On Monday 24 February 2003 19:08, Chris Mason wrote:
Hi Chris,
Quoted 5 lines
Quoted 2 lines
I can confirm that this is also true for me. I am using the -aa data-logging 36.
Sorry, you confirm that with kinoded related patches removed (and the
dirty_inodepatch) things get faster?
Also, are you using kinoded-8 or kinoded-9?
Gna, sorry. There is no difference between kinoded-8 or kinoded-9 or w/o them.
It drops. See my other mail I just wrote to the list.
ciao, Marc
On Sat, 2003-02-22 at 19:50, Manuel Krause wrote:
[ patch to leave inodes dirty instead of constantly logging them ]
Quoted 9 lines
Hi Chris!
At least I'm not able to copy through my partition via "cp -a ..." with the last supposed setup + preempt without quota ...
When umounting the destination partition it says "device is busy".
I don't know what is gone for now on my repository partition, but I know from earlier times this should not happen.
A side not about this optimization. It only affects inode updates for mtime/ctime/atime, critical things like updating i_size or the block count or other stat data operations are still logged every time, and always logged in the same transaction as the critical metadata update.
In other words, this patch restores reiserfs to its 2.2.x inode update scheme.
-chris
Hello!
Quoted 11 lines · Chris Mason
ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be updated with a new set of data logging and quota patches against 2.4.21-pre4
The data logging code is updated with another set of io stalling fixes, they should improve performance of data=ordered and data=writeback by being smarter about forcing commits under heavy write load and kicking kreiserfsd.
Treat these with care, they've gotten a ton of testing under the suse kernel, but the port to vanilla was just done today.
The quota patches include a fix for incorrectsd_blockcounts on symlinks.
Replacement 05-data-logging-36.diff.gz file that applies to 2.4.21-pre4-ac5 is available at ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz It compiles, boots, survives my (simple) testing. (writing this email from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have correct blocks count too
The reason for rejects is mostly DIRECTIO fix that also went into current bk snapshot, so probably it will apply to Marcelo's bk tree, too.
Chris: Is it intended that directio only works on data=writeback mounted filesystems?
Also following README file diff should be considered:
README renamed from README.orig +1 -1
@@ -28,7 +28,7 @@ These add reiserfs quota support -07-quota-v2-2.4.21.diff.gz +07-quota-v2-2.4.21.diff.gz # you don't need this on -ac, too 08-reiserfs-quota-26.diff.gz 09-kinoded-8.diff.gz
Quoted 3 lines · Oleg Drokin
Replacement 05-data-logging-36.diff.gz file that applies to 2.4.21-pre4-ac5 is available at ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz
Thanks Oleg.
Quoted 9 lines
It compiles, boots, survives my (simple) testing. (writing this email from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have correct blocks count too
The reason for rejects is mostly DIRECTIO fix that also went into current bk snapshot, so probably it will apply to Marcelo's bk tree, too.Chris: Is it intended that directio only works on data=writeback mounted filesystems?
Yes, the way ordered writes work is the buffers are put onto a per transaction list that gets flushed before the commit. Since buffers can only be on one list, this means they don't get onto the list of buffers for that particular inode, and that makes it difficult to make sure all pending io to the file is finished before allowing the directio to proceed (just like the tail alias bug).
The only way to do it is by forcing a commit before the directio, which would be horribly slow so I've disabled the data=ordered o_direct support. The real solution is to allocate a data structure and point to it from the private journal pointer in the bh. That requires some other changes and other performance stuff is more important right now.
Quoted 2 lines
Also following README file diff should be considered:
Thanks, I'm uploading a few more optimizations shortly, I'll include this change.
-chris
Hi Chris and Oleg,
Oleg Drokin wrote (ao):
Quoted 5 lines · Chris Mason
Quoted 4 lines
ftp.suse.com/pub/people/mason/patches/data-logging/2.4.21 will soon be updated with a new set of data logging and quota patches against 2.4.21-pre4
[cut]
Quoted 6 lines
Replacement 05-data-logging-36.diff.gz file that applies to 2.4.21-pre4-ac5 is available at ftp://namesys.com/pub/reiserfs-for-2.4/testing/05-data-logging-36-ac5.diff.gz It compiles, boots, survives my (simple) testing. (writing this email from patched 2.4.21-pre4-ac5, too). Quota works. symlinks are now have correct blocks count too
Light testing on 2.4.21-pre4-ac6 with the following patches is 100% succesful:
(patch-2.4.21-pre4)
(patch-2.4.21-pre4-ac6)
03-relocation-6.diff
04-reiserfs-sync_fs-1.diff
05-data-logging-36-ac5.diff
06-logging-export.diff
08-reiserfs-quota-26.diff
09-kinoded-8.diff
10-reiserfs-quota-link-fix.diff
They apply with zero rejects.
Thanks a lot!!
If you have specific things you want to have tested, please let me know. Quota is a It Works For Me(tm), but only very light testing.