tons of logging patches

Hello everyone,

ftp.suse.com/pub/people/mason/data-logging

has beta code that gives you the following:

external logging devices
data logging mount option (makes fsync and O_SYNC faster) ordered data mount option (no more garbage in your files after a crash)

optimizations to keep kupdate from triggering commits every 5 seconds optimizations for tiny transactions to reduce CPU usage and IO. optimizations for better locking and fewer calls to wakeup()

In short, it adds a bunch of features and makes everything faster. The changes are very large, but they also build on older patches that have been circulating around for months. It has been through a weekend of load testing here, but I still won't be surprised if people find corruption causing bugs.

It needs broad testing and benchmarking, in all the data modes, with a standard and external journal.

-chris

Whoops, I should have noted the external logging patch is from namesys, my changes were just merging things around slightly.

-chris

You seem to mean:

ftp://ftp.suse.com/pub/people/mason/patches/data-logging/

Does this mean the current released ReiserFS versions are causing the persistent write activity at 5 s interval that I observe?

Or is it ext3fs also?

Matthias Andree

Correct, thanks.

No, there is a dirty flag on the super struct that should get cleaned after all commits are done. The commit_super-8 patch might fix a bug where we are not properly cleaning the flag.

-chris

Hi!

Just a short dumb question inbetween: Is it allowed (...wished and healthy?!) to use both options together? Should look like "mount -o data=journal,data=ordered ...", yes? (mount-options referring to your latest ftp...README)

[...]

Thank you,

Manuel

I really got rid of my "usual setup": the speedup-compound-patch(with iicache-15?), akpm and aa patches for this testing now (mostly due to all these many rejects I got that Dieter had promised already... and due to the fact I have no time to understand->adjust them all manually at the moment). I can't believe it, really, but I even left the other settings alone... ;-)

Now it is "only":
Kernel.2.4.19-pre9
ReiserFS.pending.01..03,05 (rest doesn't cooperate cleanly) ReiserFS.mason.01..03 (maybe slightly modified for this)
rml.preempt-kernel ( -"- )

/ mounted -o noatime,notail,data=logging

Related to my previous state I once assumed to be lightning-fast I get -3secs (of former 17+1secs) load time of NS7 with all my plugins and -13+10secs (of former 27+10secs) load time of OOo.1.0 on a freshly booted machine. That is really nice!

Now I'll see VMwares results/reliability...
;-)

Very-very impressing, so far,

Manuel

;-)

Great to hear, thanks for trying things out.

data=logging will be the slowest mode for everything except mail servers and write heavy databases (or other apps that hammer on O_SYNC/fsync). This is because all the data gets written twice, once to log and once to the main disk. It helps synchronous writes by writing to the log in quick sequential bursts, and then writing back to the main disk in larger chunks.

data=ordered will usually be fastest for streaming appends to a file. The rest of the time it will be the same or slightly slower than the default (data=writeback).

To answer your other question, you can only select one data mode at a time.

-chris

:-))

Makes some things and testings easier, I see, and some things faster??? ;-) Mmh...

So, VMware is stable with it, too, on my well known "heavy-private-test" of it (running Norton SpeedDisk at least twice within a most recent VMware Win98). It doesn't show greatly different timings than to my setup before though having a different disk i/o pattern (due to the missing aa patches)... and me having a reduced RAM from 512to256MB at the moment. And I should be honest to say I can't give exact timings as the important disk contents changed during last weeks. But the disk-access-times/related-to-the-content are definitively _not_ higher than before!

Yes, I assumed/learnt that from your very previous explanations. But I don't see the point "slowest", so far, on here.

This is a home/single session ?non?-production system on here. 1 to 2 IDE disks, one processor... notebook, you'd remember this setup from my
mails.

What would be my recommended mount option later then (mentionned in the manpages or on the webpage?!) Mmmh: I see I test that at the moment...

Why is no combo possible? Meant as: Could/would that bring advantages in future?!!

Thanks,

Manuel

same speed on 1/2 the ram isn't bad ;-)

try dbench, or any streaming write. It should be roughtly 1/2 as fast as the default mode. If it isn't data=journal probably isn't really activated (it sends a printk on mount about using journaled data mode).

data=ordered will work best for most people. The performance hit is very small, and you won't have any more garbage in files after a crash.

Since there are no userspace transactions, data=journal is only useful in a very limited set of cases. The fact that everything is logged allows me to cheat, and skip some of the steps to keep block allocations and deallocations safe in the writeback data mode.

intermezzo is the only app that really needs data=journal with the rest of the FS mounted with some other write mode (for speed). The patch has a way for them to enable data logging mode on a single file.

I think I'll forward the patch announcement to the intermezzo list...

-chris

[...]
Don't know where to reply best...

Hi, again!

I want to make some more comments on my latest words.

As I said I first used the data=journal mode and got nice timings. O.k. I really think after that "revision" my previous kernel setup wasn't that well configured as I thought and felt. Long time degression?!

I really had the reiserfs messages in my logs that it explicitely used
this mode. The only problem I obviously had, so far, was to distinguish the mount options at darkest night: data=logging is no mount-option but the description "data-logging", the mount option for it is data=journal -- Passing rootflags=data=journal in lilo.conf and data=logging in fstab results in an uncontrollable kernel ;-) Huh!
Sorry, for my thoughtless testing. But my posted timings are quite relieble on here.

Concerning VMware the "same speed on 1/2 RAM" results are even more
impressing as VMware seems to buffer it's memory contents to /tmp/... fs again since I reduced the RAM. With 512MB it didn't seem to need this method usually.

The data=ordered mode saves 1..2secs from of my previously posted load
times for NS7 and OOo-1.0 and seems to be stable itself in "everydays usage" and for my VMware sessions, too. I didn't test the "crash->no-garbage-in-files" case and the more recent 03-beta-data-logging-6.diff, yet.

I was extraordinary glad to see the explicit wording of the mounted partition in the logs we missed for so long time!

Thanks for your help,

Manuel

[...]

BTW, what is this "only" diff good for (is it worth to recompile, I mean):

# diff '03-beta-data-logging-6.diff' '03-beta-data-logging-5.diff'
2777c2777
< +  if (SB_JOURNAL(p_s_sb)->j_num_lists > 512) {
---

Thank you,

Manuel

It is a performance tweak, my next patch removes that bit entirely. Thanks to some hints from Andrew Morton I've been able to do many more optimizations for high load fsync heavy applications. The new patch will be out later tonight.

-chris

Ok, ftp.suse.com/pub/people/mason/patches/data-logging has my latest updates, which have more optimizations for many threads all doing synchronous transactions.

-chris

Mhh. I was just starting to try the ftp, but no-go-for-now. O.k now I've got an old Santana CD in... "Black magic ..." Thanks to Chris,

bye, very nice to listen to,

Manuel

Mmh. I recompiled with the new one when it appeared on the server and kept the kernel append line with the obvious problem not to be able to easily reboot to my ext2 maintenance partition.

I had some disappearing files/dirs from /lib/modules/* and from somewhere within /var/* the second time now. First occurrence is about 6 days ago. That takes action within a good uptime (I mean: not at startup or upon special programs' interaction).

Jun 12 20:52:48 firehead kernel: is_leaf: item location seems wrong (second one): *3.6* [5273 5274 0x1 IND], item_len 4, item_location 2120, free_space(entry_count) 0
Jun 12 20:52:48 firehead kernel: vs-5150: search_by_key: invalid format found in block 8422. Fsck?
Jun 12 20:52:48 firehead kernel: vs-13070: reiserfs_read_inode2: i/o failure occurred trying to find stat data of [5199 261934 0x0 SD]
Jun 12 20:52:48 firehead kernel: is_leaf: item location seems wrong (second one): *3.6* [5273 5274 0x1 IND], item_len 4, item_location 2120, free_space(entry_count) 0
Jun 12 20:52:48 firehead kernel: vs-5150: search_by_key: invalid format found in block 8422. Fsck?
[...] <- many many more
Jun 12 20:54:46 firehead kernel: vs-13070: reiserfs_read_inode2: i/o failure occurred trying to find stat data of [5199 261934 0x0 SD]
Jun 12 20:54:46 firehead modprobe: modprobe: Can't open dependencies file /lib/modules/2.4.19-pre9/modules.dep (Permission denied)

I then needed to do a reiserfsck --rebuild-tree, as suggested by most recent reiserfsck (3.x.1c-pre4), the 3.x.1b-one said, it was fixable by --fix-fixable, but I then decided to use the latest. And then I had to adjust the lost&found manually. -- Huh, that took a long time, but nothing's missing, if I experience my system correctly now.

My system/partitions still look like the recently posted setup:
Kernel.2.4.19-pre9
ReiserFS.pending.01..03,05 (rest doesn't cooperate cleanly) ReiserFS.mason.01..03 (most recent) rml.preempt-kernel (maybe slightly modified for this setup)

/ mounted -o noatime,notail,data=ordered
and from applied lilo.conf: append = "... rootflags=data=ordered"

Thanks, but the term "speedup" seems to be relative for me for now,

Manuel

Hmmm, I might have found it, but this isn't a new bug to the 07 patch. Have you been able to reproduce, or have you fallen back to the older patches?

-chris

You mean 03-beta-data-logging-7.diff.gz when saying "07 patch"? Then I didn't go back according to my disk content.

When I boot up I read on screen and in boot.msg:
...
EXT2-fs: Unrecognized mount option data
reiserfs: using ordered data mode
reiserfs: found format "3.6" with standard journal
reiserfs: checking transaction log (ide0(3,7)) for (ide0(3,7)) (...)
Using r5 hash to sort names
...

Addon is: I use the ReiserFS pending patches that apply as I mentionned previously(?).

No, it looks o.k. here.

Manuel

I've uploaded 03-beta-data-logging-8.diff.gz. It will fix the oops Ed hit, and hopefully your corruption problem as well.

-chris

ftp.suse.com/pub/people/mason/patches/data-logging has a new beta-data-logging-10.diff.gz

This one:

compiles as a module (thanks Jeff Mahoney)

has a highmem bug fixed with data=journal (help from Andrea)

Fixes a problem with relocated/different sized journal devices. This was a merge error on my part.

One tester reports a hang that I can't reproduce here, I'm still trying.

-chris

Many thanks to you, Chris!

I applied it on Sunday as I was on a short vacation to Hamburg.de, for one week before, and your recent patch works very well under noatime,notail,data=ordered on / still booted with the appropriate rootflag. (That is my old setup.)
I remarked one difference, I didn't realize this before on ReiserFS. Having a backup partition previously mounted with fstab settings:
"/dev/hda11 /mnt/beta reiserfs
noauto,user,noatime,notail,data=ordered 0 0",
I was not able to execute any executable. So it should be normally as I understand "mount" and its manpage. I had to add a "exec," to the mount options. Now it works and [this] is right?! ;-)

Is the rootflag needed any more? Does your patchset work with all pending ReiserFS patches together, now? (Yes, I remember your and your teams words on 2.4.20-pre...) I didn't try this complete combo again, just replaced the old beta-data-logging patch in my last (posted) setup.

Last week I got my old RAM renewed (to 512MB again) and some of my old problems disappeared for now (boot-up overheating symptoms, Oleg told me about, that are quite assumable (warranty time exceeded)). The load timings I posted stay the same, so, my aa-akpm-rml patch setup was really somekind of poor.

Mmh. Now I don't have the old crash pattern any more ;-) Maybe I push my notebooks power button "accidentally", soon, to test...

Best wishes and greetings,

Manuel

Hi everyone,

I've updated data logging patch, to fix a problem where reiserfs_writepage didn't properly unlock pages in data logging mode.

The new code will be on ftp.suse.com/pub/people/mason/patches/data-logging once the mirror updates.

If you've already got 03-beta-data-logging-10, you only need the incremental fix below.

-chris

@@ -2034,6 +2034,7 @@
     }
     if (reiserfs_active_handle(&th)) {
         journal_end(&th, sb, nr) ;
+        UnlockPage(page);
     }
 }
 

Hello again,

Yet another data logging patch is available (once the mirror updates) at:

ftp.suse.com/pub/people/mason/patches/data-logging

We're getting closer to stability here, so I'm very interested in workloads where the patch performs poorly.

New in 03-beta-data-logging-13.diff:

Fix BUG() due to missing BKL before calling journal_begin

Make reiserfs unpack ioctl force a log flush in data journaling mode. This is required to make things safe if you crash after running lilo. It is also slow, but there is no other fix possible at the moment.

When the running transaction has many writers, the count of log blocks reserved is much higher than the number actually used. If a new writer finds the transaction full, he now tries to let an existing writer finish instead of forcing a commit. This increases the average transaction size when you have many procs writing to the log.

Smarter code to collect writers before forcing a commit during fsync.

In benchmarks with akpm's synctest program (it tries to simulate postfix load), these last two changes brought the number of transactions during a 50 thread run down from 6000 to 3300. Not as good as I wanted, but it is 20% faster than the last patch, so I'll take it.

Overall, for most mail server workloads, using the new patches + data=journal should be around 3-4 times as fast as the stock 2.4.19-rc1 kernel. I'm trying to get verification on this from actual mail server benchmarks, instead of just simulations ;-)

-chris

And a buffer list corruption bug of some kind. I've pulled beta-data-logging-13.diff until I track the bug down. I'm heading off to the mountains for the 4th of July holiday, so it be fixed until monday.

-chris

So far the 13 prerelease works fine under rootflags boot and mount/remount option "data=ordered" on here.

Can you, please, answer my old questions as of:

On 06/25/2002 02:55 AM, Manuel Krause mainly wrote (but too much more):

Thanks,

I don't get a performance decrease with "13", and: Is it really dangerous to use "13" in this mode of operation?

Manuel

I think I've got the bug, it was newly introduced in -13. New stuff will be uploaded today.

Yes, you still need the rootflag. I haven't tried to merge it with all the namesys pending patches, except for the journal relocation stuff and the commit_super speedups.

It is dangerous to use -13 at all, although the bug is pretty difficult to trigger. Please run reiserfsck --check before upgrading.

-chris

On 07/08/2002 03:51 PM, Chris Mason wrote:

[...]

Thanks a lot,for your comments, Chris!

O.k., so far "no problems" with -13 "data=ordered". I write your advice to real paper and pin it over my desk to remember this advice after the new patch appeared... :-))

***

So far I assume I had more errors from my cheap hardware than from anything else (e.g. ReiserFS + any patch-combo) over the last 3..5..7? months. :-(

Last weekend after beeing heavily bugged by my notebook (no obvious errors any more) I decided to carefully disassemble it completely (warranty exceeded by 2 months), clean it and reassemble it:

Oleg was really right at the end! It was an overheating problem of a special kind: As my CPU lost its contact to its "little fixiating plastic piece part to the socket" (don't know the exact term ;-) ) the CPU was able to move some µm..mm in the socket. The CPU/fan - fan fixed with 4 screws over the CPU- obviously lost their contact when the fan went on working (torque) or the CPU/chipset decided to stop when the temperature went too high inbetween. Whooo, I have new 512MB Infineon notebook RAM through this case... Should I be quiet or report this to the manufacturer and/or their local outlet in .de, now?

The notebook now works like I bought it: fine, stable and the thermal (fan start/stop) patterns are quite well though I didn't replace the thermal pads between CPU&fan and the heatsink-to-graphix-and-chipset as always and everywhere recommended. (Oh, try to get these special parts once!!!) We had up to 35°C today here in Germany. That's a good restart!

***

Best regards,
I'm really sorry for the traffic and attention due to this circumstance! I was really encouraged by Olegs comments finally, thanks to you, Oleg!

Thanks to all,

Manuel

Ah, still on the todo list for the current patch is safe handling of remount. Currently data=logging and data=ordered are ignored on remount, which is probably why your results were so fast in data logging mode.

You'll want to pass rootflags=data=ordered or rootflags=data=journal at the lilo prompt.

-chris

O.k.

Yes, as-is on SuSE typical bootup. But, o.k., I know to handle, I've changed and reset my lilo-setup accordingly to your mail...

So, I just retested, with your recommended append to lilo.conf "rootflags=data=journal" -- but timings stay the same. And mount says still the same, too:
/dev/hda7 on / type reiserfs (rw,noatime,notail,data=logging)

So should you be glad or not!?
The NS7 and OOo.1.0 load timings are generally measured in secs +-1 and stay to be faster, like I posted before.

No, no mistake on this way, on my side, really, you could finally (if you don't find a solution within your new coding) blame it to my bad previous patches' setup... ;-))) (yes, finally -- at the real end, of course!!! ;-) )

Thank you, works very well,

Manuel

P.S.: Please, keep me informed, as Namesys seems to need a pushup for 2.4. kernels... ;-) Sorry, Hans! ;-)

BTW, for general usage, the most important speedup in these patches is the commit_super patch. The namesys guys already have that in the queue of stuff to send for 2.4.20-pre.

The tweaks included in the data logging code improve a few of the corner cases that were hurting us.

-chris

You will see substantial performance increases in 2.4.20

Oleg is doing a complete rewrite of our write function, and is seeing large file performance increases of 25%.

We are currently doing a complete analysis of our layout algorithms, and I expect some results from that.

Finally, there is V4 which is almost ready to work now.....

Hans

This is very impressive. Particularly considering I did some performance tests on reiserfs vs ext3 vs xfs last night & Reiserfs beat the pants off the other two for large file creation. This was on a 2.4.18 kernel & the xfs used was the new version 1.1.

In order to eliminate other variables, I was using the same partition for all tests (just remade it over & over), same kernel, same system load, etc.

I'm going to develop a script which will go through & do some formal testing on the different filesystems. Last night was just a play to see how it all went. The proper testing will be done in single user mode.

I also intend to patch for jfs if it will play nicely with the others.

I'm happy to post my results to the list when I finish if you like. Cheers,
-Rob

-- Robert Brockway B.Sc. email: [email protected] ICQ: 104781119
Linux counter project ID #16440 (http://counter.li.org)
"The earth is but one country and mankind its citizens" -Baha'u'llah

Hello!

Note that I see only 25+% performance increase CPU-wise. Since the task is io-bound real time does not change much (unless one runs tests on the ramdisk or similar (by speed) kind of storage).

Bye,
Oleg