Skip to contentSkip to conversation

Reiser4 status: benchmarked vs. V3 (and ext3)

DISCUSSION Hans Reiser 25 participants 88 messages Jul 23, 2003 – Sep 10, 2003

lore
Share & tools
lore
#0 lore raw

Please look at http://www.namesys.com/benchmarks/v4marks.html

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

We are able to perform all filesystem operations fully atomically, while getting dramatic performance improvements. (Other attempts at introducing transactions into filesystems are said to have failed for performance reasons.)

Balancing at flush time works well, not using blobs works well, allocating at flush time works well. CPU time is good enough to get by, and it will improve over the next few months as we tweak a lot of little details, but the IO performance is what matters, and this performance is quite good enough to use. In all the places where V3 sacrifices performance to save disk space, V4 saves more disk space and gains rather than loses performance.

The plugin infrastructure works well, expect lots of plugins over the next year or two.

Look for a repacker to come out in a few weeks that will make these numbers especially good for filesystems that have 80% of their files unmoving for long periods of time (which is to say most systems), and might otherwise suffer from fragmentation.

These benchmarks mean to me that our performance is now good enough to ship V4 to users (which means we need persons willing to try to crash it so that the stability can become good enough to ship to users). Sometime during the next week or two we will probably send a patch in, and ask for inclusion. We need to run another round of stress tests after our latest tweaks, and kill off two bugs that got added just recently, and then we will ask for testers.

I will be going to Budapest to discuss filesystem semantics with Peter Foldiak for a week, so V4 may get sent in for inclusion by members of my team while I am absent. If so, please include it in 2.5/2.6.

Signature · 1 line
Hans
#1 lore raw
Quoted 4 lines · Hans Reiser

Please look at http://www.namesys.com/benchmarks/v4marks.html

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

<snip>
I am interested in testing this out, but the latest patch on the namesys sight appears to be against 2.5.60 which was never usable on my hardware. If there is a later patch, or if somebody has adapted it to work against 2.6.0-test1(or anything comparably recent), please let me know.

-Tupshin

#2 lore raw

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Quoted 21 lines · Tupshin Harper
Quoted 6 lines · Hans Reiser

Please look at http://www.namesys.com/benchmarks/v4marks.html

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

<snip>
I am interested in testing this out, but the latest patch on the namesys sight appears to be against 2.5.60 which was never usable on my hardware. If there is a later patch, or if somebody has adapted it to work against 2.6.0-test1(or anything comparably recent), please let me know.

(quoted diff) +0 -2
-Tupshin
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 2 lines · Shawn

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a lack of success.

It applied cleanly, but it doesn't add a fs/reiser4 directory and asociated contents. Is there an additional patch, or is this one broken?

-Tupshin

#4 lore raw

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Quoted 13 lines · Tupshin Harper
Quoted 2 lines · Shawn

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a lack of success.

It applied cleanly, but it doesn't add a fs/reiser4 directory and asociated contents. Is there an additional patch, or is this one broken?

-Tupshin

#5 lore raw

Hmmm... Maybe he can get one of those swirly color guys on the namesys website to administer the spanking.
(http://www.namesys.com/v4/r4pics/finegrainingJ.jpg) I want to see video of that...

BTW Hans, I have a new swirly guy idea for the "Extensibility through plugins" graphic... ;P

Quoted 18 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Quoted 12 lines · Tupshin Harper
Quoted 2 lines · Shawn

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a lack of success.

It applied cleanly, but it doesn't add a fs/reiser4 directory and asociated contents. Is there an additional patch, or is this one broken?

-Tupshin

#6 lore raw
In reply to Shawn
Quoted 4 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

Quoted 15 lines · Tupshin Harper
Quoted 13 lines
Quoted 2 lines · Shawn

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a lack of success.

It applied cleanly, but it doesn't add a fs/reiser4 directory and asociated contents. Is there an additional patch, or is this one broken?

-Tupshin

#7 lore raw
Quoted 11 lines · Nikita Danilov
Quoted 4 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

I'm sorry, but I don't have a bitkeeper license. Please let me know if a fixed patch is available.

-Tupshin

#8 lore raw
Quoted 8 lines · Tupshin Harper
Quoted 6 lines · Nikita Danilov
Quoted 4 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Quoted 7 lines
Quoted 5 lines

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

I'm sorry, but I don't have a bitkeeper license. Please let me know if a fixed patch is available.

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#9 lore raw
Quoted 23 lines · Larry McVoy
Quoted 20 lines · Tupshin Harper
Quoted 16 lines · Nikita Danilov
Quoted 6 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

Quoted 8 lines
Quoted 3 lines

I'm sorry, but I don't have a bitkeeper license. Please let me know if a fixed patch is available.

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

Thanks...I'm just interested in a working version of the reiser 4 patches to apply against 2.6.0-test1 or any other very recent tree. The version at ftp://ftp.namesys.com/pub/tmp is incomplete.

-Tupshin

#10 lore raw
Quoted 4 lines · Larry McVoy
Quoted 1 line
Quoted 1 line

http://www.namesys.com/code.html (requires bitkeeper)?

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

I pulled the r4 tree into linux-2.5 (2.6) top of trunk, merged it (it all automerged, ya gotta love that, see below), and sent out a patch. If someone else needs that patch let me know.

takepatch: 2782 new revisions, 8 conflicts in 357 files 440459 bytes uncompressed to 2455963, 5.58X expansion Running resolve to apply new work ...
resolve: found 25 renames in pass 1
resolve: resolved 25 renames in pass 2
Content merge of Makefile OK
Content merge of arch/um/drivers/ubd_kern.c OK Content merge of fs/jbd/transaction.c OK Content merge of fs/proc/proc_misc.c OK Content merge of include/linux/sched.h OK Content merge of kernel/ksyms.c OK
Content merge of kernel/sched.c OK
resolve: resolved 7 conflicts in pass 3

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#11 lore raw

Re: Reiser4 and linux 2.6.0

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Perhaps it would be as simple as:

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

Quoted 26 lines · Larry McVoy
Quoted 4 lines · Larry McVoy
Quoted 1 line
Quoted 1 line

http://www.namesys.com/code.html (requires bitkeeper)?

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

I pulled the r4 tree into linux-2.5 (2.6) top of trunk, merged it (it all automerged, ya gotta love that, see below), and sent out a patch. If someone else needs that patch let me know.

takepatch: 2782 new revisions, 8 conflicts in 357 files 440459 bytes uncompressed to 2455963, 5.58X expansion Running resolve to apply new work ...
resolve: found 25 renames in pass 1
resolve: resolved 25 renames in pass 2
Content merge of Makefile OK
Content merge of arch/um/drivers/ubd_kern.c OK Content merge of fs/jbd/transaction.c OK Content merge of fs/proc/proc_misc.c OK Content merge of include/linux/sched.h OK Content merge of kernel/ksyms.c OK
Content merge of kernel/sched.c OK
resolve: resolved 7 conflicts in pass 3

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#12 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 10 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Quoted 9 lines

Perhaps it would be as simple as:

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

Larry was kind enough to send me the patch of his results of this merge, but unfortunately it suffered from the same incompleteness problems that the recent diffs on the namesys ftp site do. They apply cleanly, but they completely omit the contents of the fs/reiser4 directory which has the bulk of the code. It seem that the problem resides with the reiser folks. Maybe they have some more complicated structure involving two trees. Not sure, but a correct patch looks like it has to come from the reiser4 team.

-Tupshin

#13 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 21 lines · Tupshin Harper
Quoted 10 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Quoted 9 lines

Perhaps it would be as simple as:

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

Quoted 8 lines

Larry was kind enough to send me the patch of his results of this merge, but unfortunately it suffered from the same incompleteness problems that the recent diffs on the namesys ftp site do. They apply cleanly, but they completely omit the contents of the fs/reiser4 directory which has the bulk of the code. It seem that the problem resides with the reiser folks. Maybe they have some more complicated structure involving two trees. Not sure, but a correct patch looks like it has to come from the reiser4 team.

This is because Larry only made a diff against bk://bk.namesys.com/bk/reiser4-linux-2.5 which is a repository containing changes needed by the reiser4 in the core kernel code.

Bulk of reiser4 code (content of fs/reiser4) is in bk://bk.namesys.com/bk/reiser4 repository.

Quoted 3 lines

-Tupshin

Nikita.

#14 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 28 lines · Nikita Danilov
Quoted 14 lines · Tupshin Harper
Quoted 12 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Perhaps it would be as simple as:

Quoted 8 lines
Quoted 7 lines

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

Larry was kind enough to send me the patch of his results of this merge, but unfortunately it suffered from the same incompleteness problems that the recent diffs on the namesys ftp site do. They apply cleanly, but they completely omit the contents of the fs/reiser4 directory which has the bulk of the code. It seem that the problem resides with the reiser folks. Maybe they have some more complicated structure involving two trees. Not sure, but a correct patch looks like it has to come from the reiser4 team.

This is because Larry only made a diff against bk://bk.namesys.com/bk/reiser4-linux-2.5 which is a repository containing changes needed by the reiser4 in the core kernel code.

Quoted 12 lines

Bulk of reiser4 code (content of fs/reiser4) is in bk://bk.namesys.com/bk/reiser4 repository.

Quoted 3 lines

-Tupshin

Nikita.

make more effort to make this well documented for users. If they are asking these questions, it means that you failed to document.

Signature · 1 line
Hans
#15 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 13 lines · Hans Reiser
Quoted 11 lines · Nikita Danilov
Quoted 1 line · Tupshin Harper

Larry was kind enough to send me the patch of his results of this

merge, > but unfortunately it suffered from the same incompleteness problems that > the recent diffs on the namesys ftp site do. They apply cleanly, but > they completely omit the contents of the fs/reiser4 directory which has > the bulk of the code. It seem that the problem resides with the reiser > folks. Maybe they have some more complicated structure involving two > trees. Not sure, but a correct patch looks like it has to come from the > reiser4 team.

This is because Larry only made a diff against bk://bk.namesys.com/bk/reiser4-linux-2.5 which is a repository containing changes needed by the reiser4 in the core kernel code.

Bulk of reiser4 code (content of fs/reiser4) is in bk://bk.namesys.com/bk/reiser4 repository.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

-Tupshin

Nikita.

make more effort to make this well documented for users. If they are asking these questions, it means that you failed to document.

It would still be wonderful to have a way of getting such patches without going through bk. I requested that a working (complete) patch be made against a recent kernel version(2.6.0-test2 or later at this point) a few weeks ago, and while a got positive response, I still haven't seen anything. I would think you would want to make this very easy for people who are already going through the effort of testing 2.6 kernels.

-Tupshin

#16 lore raw

Re: Reiser4 and linux 2.6.0

Am Sonntag, 10. August 2003 04:02 schrieb Tupshin Harper:

Quoted 8 lines

It would still be wonderful to have a way of getting such patches without going through bk. I requested that a working (complete) patch be made against a recent kernel version(2.6.0-test2 or later at this point) a few weeks ago, and while a got positive response, I still haven't seen anything. I would think you would want to make this very easy for people who are already going through the effort of testing 2.6 kernels.

-Tupshin

Hello List,

i would love to see a patch against a 2.6.0-test kernel too. I don't want to obtain a bitkeeper licence.
A anoncvs-gateway as a alternative would be also ok ;)

As a happy reiserfs user, it is hard to read about the various changes in v4, and can't test them for yourself.

Henning

#17 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 10 lines · Henning Westerholt

Am Sonntag, 10. August 2003 04:02 schrieb Tupshin Harper:

Quoted 8 lines

It would still be wonderful to have a way of getting such patches without going through bk. I requested that a working (complete) patch be made against a recent kernel version(2.6.0-test2 or later at this point) a few weeks ago, and while a got positive response, I still haven't seen anything. I would think you would want to make this very easy for people who are already going through the effort of testing 2.6 kernels.

-Tupshin

Quoted 9 lines

Hello List,

i would love to see a patch against a 2.6.0-test kernel too. I don't want to obtain a bitkeeper licence.
A anoncvs-gateway as a alternative would be also ok ;)

As a happy reiserfs user, it is hard to read about the various changes in v4, and can't test them for yourself.

Snapshot will be done to-day (2003.08.11).

Quoted 3 lines

Henning

Nikita.

#18 lore raw

Re: Reiser4 and linux 2.6.0

In reply to Tupshin Harper

If you still have that small patch, I'll take it. Beats downloading old 2.5 code just to get a copy of a few files.

The I can pull the other repositories which should be much smaller.

I can't wait to get my test system toasted with Reiser4.

Marcelo

Quoted 30 lines · Tupshin Harper
Quoted 10 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Quoted 8 lines

Perhaps it would be as simple as:

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

Larry was kind enough to send me the patch of his results of this merge, but unfortunately it suffered from the same incompleteness problems that the recent diffs on the namesys ftp site do. They apply cleanly, but they completely omit the contents of the fs/reiser4 directory which has the bulk of the code. It seem that the problem resides with the reiser folks. Maybe they have some more complicated structure involving two trees. Not sure, but a correct patch looks like it has to come from the reiser4 team.

-Tupshin

#19 lore raw

Re: Reiser4 and linux 2.6.0

In reply to Robert August Vincent II
Quoted 10 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Quoted 8 lines

Perhaps it would be as simple as:

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

It is. Take a look at the http://www.namesys.com/code.html.

But:

1. reiser4 is highly experimental,
2. fsck is not yet ready.

so, be careful, don't trust it valuable data and do backup (just for case).

Quoted 12 lines · Larry McVoy
Quoted 10 lines · Larry McVoy
Quoted 4 lines
Quoted 1 line
Quoted 1 line

http://www.namesys.com/code.html (requires bitkeeper)?

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

I pulled the r4 tree into linux-2.5 (2.6) top of trunk, merged it (it all automerged, ya gotta love that, see below), and sent out a patch. If someone else needs that patch let me know.

Quoted 16 lines
Quoted 5 lines

takepatch: 2782 new revisions, 8 conflicts in 357 files 440459 bytes uncompressed to 2455963, 5.58X expansion Running resolve to apply new work ...
resolve: found 25 renames in pass 1
resolve: resolved 25 renames in pass 2

Quoted 11 lines

Content merge of Makefile OK
Content merge of arch/um/drivers/ubd_kern.c OK
Content merge of fs/jbd/transaction.c OK
Content merge of fs/proc/proc_misc.c OK
Content merge of include/linux/sched.h OK
Content merge of kernel/ksyms.c OK
Content merge of kernel/sched.c OK
resolve: resolved 7 conflicts in pass 3

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Nikita.

#20 lore raw

Re: Reiser4 and linux 2.6.0

Quoted 13 lines · Nikita Danilov
Quoted 12 lines · Robert August Vincent II

Could you explain, in simple terms, to somebody who is unfamiliar with bitkeeper, how to pull the latest 2.6.0 and reiser4 code and do whatever you did to merge them?

I'm not working on any SCM, and therefore downloaded bitkeeper using the "free" license.

But I'm unclear (even after perusing the FAQ and docs) on how to do what you just described.

Perhaps it would be as simple as:

Quoted 8 lines
Quoted 6 lines

bk clone http://linux.bkbits.net/linux-2.5
bk clone bk://bk.namesys.com/bk/reiser4-linux-2.5 linux-2.5 cd linux-2.5/fs
bk clone bk://bk.namesys.com/bk/reiser4

But I doubt it...

It is.

Actually, the above sequence fails on the second command.

Quoted 1 line

Take a look at the http://www.namesys.com/code.html.

I have. It tells me how to make a copy of the reiser bk repository. But does the reiser repository mirror the one at linux.bkbits.net? Does it have the latest 2.6 code? Or should I merge the reiser repository with the one at linux.bkbits.net?

In other words, how do I go about merging the latest code tree from linux.bkbits.net with the latest code tree from bk.namesys.com ?

Quoted 4 lines

But:

1. reiser4 is highly experimental,
2. fsck is not yet ready.

Understood.

Quoted 2 lines

so, be careful, don't trust it valuable data and do backup (just for case).

I've been following this list for over a year, and have been using Reiserfs since before Ext3 went stable.

I think you're worried that amateurs will try out the code, experience problems, and develop an unfavorable view of Reiser4 before it's production-ready.

Please understand that I'm fully aware of the consequences and I wouldn't blame you guys if anything goes wrong.

But I'm excited about Reiser4 and I'd like to play around with the latest code, on a non-production system.

#21 lore raw
In reply to Larry McVoy

Wow, it seems rather involved just to get a udiff against bk-cur. Pull kernel sources with reiser4 mods, then pull fs/reiser4/*, then generate a diff.

Well, at least I understand it now...

Quoted 19 lines · Larry McVoy
Quoted 8 lines · Tupshin Harper
Quoted 6 lines · Nikita Danilov
Quoted 4 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Hans? Time to long-distance spank someone?

Quoted 7 lines
Quoted 5 lines

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

I'm sorry, but I don't have a bitkeeper license. Please let me know if a fixed patch is available.

If someone can tell me what it is that you need and I'll do it and send you a patch. I'm cloning that tree now.

#22 lore raw
In reply to Tupshin Harper

Hello,

You don't need one for testing reiserfs v4.

Just download the bitkeeper software [1] Last time i did it i had to enter my email then i would receive the instructions to be able to download it ( login and password )

Thanks,

Philippe

[1] http://www.bitmover.com/cgi-bin/download.cgi

On Thu, 24 Jul 2003 08:10:19 -0700
Tupshin Harper <[email protected]> wrote:

  | I'm sorry, but I don't have a bitkeeper license. Please let me know if a 
  | fixed patch is available.
  | 
  | -Tupshin
#23 lore raw
Quoted 8 lines · Philippe Gramoullé

Hello,

You don't need one for testing reiserfs v4.

Just download the bitkeeper software [1] Last time i did it i had to enter my email then i would receive the instructions to be able to download it ( login and password )

Quoted 8 lines

Thanks,

Philippe

[1] http://www.bitmover.com/cgi-bin/download.cgi

Uhh...no. I have recently decided to contribute to the arch SCM project, so I am legally unable to use BitKeeper's free license which you have to agree to in order to download and use it.

-Tupshin

#24 lore raw

Hello,

Ok now i understand better your previous statement.

I personnaly don't contribute to any SCM project so far so i guess i can continue to use it without licence.

Thanks,

Philippe

On Thu, 24 Jul 2003 08:47:28 -0700
Tupshin Harper <[email protected]> wrote:

  | Uhh...no. I have recently decided to contribute to the arch SCM project, 
  | so I am legally unable to use BitKeeper's free license which you have to 
  | agree to in order to download and use it.
#25 lore raw

Hello,

I'd like to apologise if i made false statements (twice)

The corrected sentence should read :

"I actually need to have a free licence (i.e accepts its terms & conditions) to use BK but that i don't have to pay for it as long as i don't use BK for developping another SCM project"

Sorry for the confusion/noise.

Bye

Philippe

On Thu, 24 Jul 2003 17:57:45 +0200
Philippe Gramoullé <[email protected]> wrote:

| I personnaly don't contribute to any SCM project so far so i guess i can continue to use it | without licence.

#26 lore raw
In reply to Tupshin Harper
Quoted 14 lines · Tupshin Harper
Quoted 12 lines · Philippe Gramoullé

Hello,

You don't need one for testing reiserfs v4.

Just download the bitkeeper software [1] Last time i did it i had to enter my email then i would receive the instructions to be able to download it ( login and password )

Thanks,

Philippe

Quoted 7 lines
Quoted 4 lines

[1] http://www.bitmover.com/cgi-bin/download.cgi

Uhh...no. I have recently decided to contribute to the arch SCM project, so I am legally unable to use BitKeeper's free license which you have to agree to in order to download and use it.

Quoting from http://linux.bkbits.net/stats.html:

Wed Jul 23 17:30:55 PDT 2003 tupshin bastard.(none)
/usr/src/bks/linux-2.4 linux-2.4 2700 outgoing pull

This is another Tupshin?

Confused,
Carl-Daniel

#27 lore raw

I also have a machine ready to be fully installed with Reiser 4, but I need to be able to run it on 2.6.0-test1 and wouldn't like to need to install bk. If you could set up the cvs gateway, that would be ok.

#28 lore raw
In reply to Carl-Daniel Hailfinger
Quoted 33 lines · Carl-Daniel Hailfinger
Quoted 22 lines · Tupshin Harper
Quoted 8 lines · Philippe Gramoullé

Hello,

You don't need one for testing reiserfs v4.

Just download the bitkeeper software [1] Last time i did it i had to enter my email then i would receive the instructions to be able to download it ( login and password )

Quoted 10 lines

Thanks,

Philippe

[1] http://www.bitmover.com/cgi-bin/download.cgi

Quoted 5 lines

Uhh...no. I have recently decided to contribute to the arch SCM project, so I am legally unable to use BitKeeper's free license which you have to agree to in order to download and use it.

Quoting from http://linux.bkbits.net/stats.html:

Quoted 11 lines

Wed Jul 23 17:30:55 PDT 2003 tupshin bastard.(none)
/usr/src/bks/linux-2.4 linux-2.4 2700 outgoing pull

This is another Tupshin?

Confused,
Carl-Daniel

No confusion...just that the "recently" was in fact a decision I made yesterday. I had used bitkeeper a few different times since it was touted as the easiest way to get the latest kernel snapshots, and I had previously no intention of working on any SCM project. However, I joined the arch mailing list about a week ago, and last night I decided to start coding a few changes that will make it suit my needs a bit better. I have no idea if these changes will be incorporated into arch, and they don't have any overlap with bitkeeper functionality. However, I must abide by the bitkeeper license now that I have made that decision. I will be pulling linux kernels from the cvs exported version from here on out, and I will probably re-import it into arch in order to maintain a local kernel tree.

-Thanks
-Tupshin

#29 lore raw
In reply to Nikita Danilov
Quoted 4 lines · Nikita Danilov
Quoted 3 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Quoted 24 lines
Quoted 1 line

Hans? Time to long-distance spank someone?

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

Quoted 13 lines · Tupshin Harper
Quoted 11 lines
Quoted 2 lines · Shawn

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a lack of success.

It applied cleanly, but it doesn't add a fs/reiser4 directory and asociated contents. Is there an additional patch, or is this one broken?

-Tupshin

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we put a
tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

It is the usual American business english used in such cases.;-)

Signature · 1 line
Hans
#30 lore raw
Quoted 11 lines

[[email protected]]

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we put a
tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

It is the usual American business english used in such cases.;-)

Standard American puke inducing pretentious florid business language if you want to be completely accurate. 8)

Signature · 1 line
Tomas Szepe <[email protected]>
#31 lore raw
Quoted 7 lines · Tomas Szepe
Quoted 7 lines

[[email protected]]

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we put a
tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

Quoted 9 lines
Quoted 3 lines

It is the usual American business english used in such cases.;-)

Standard American puke inducing pretentious florid business language if you want to be completely accurate. 8)

Yeah, but Nikita really could have been more helpful....

Signature · 1 line
Hans
#32 lore raw
In reply to Tomas Szepe
Quoted 11 lines · Tomas Szepe
Quoted 11 lines

[[email protected]]

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we
put a tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

Quoted 4 lines
Quoted 1 line

It is the usual American business english used in such cases.;-)

Standard American puke inducing pretentious florid business language if you want to be completely accurate. 8)

Huh? I thought Hans was being facetious, or was practicing his standup comedy. I can see something like that coming out of a far western oriental type when dealing with an american that isn't really understanding his accent, but here in the states, after we've explained it once in plain english, the next exchange will more than likely go the other way, possibly even making it to the 'hey dumbf*ck' stage on about the 4th reply.

And I do object, albeit somewhat tongue-in-cheek, to applying that particular label/broad brush to all americans since I am one of those creatures.

Signature · 7 lines
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz  512M
99.27% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
#33 lore raw
Quoted 13 lines · Gene Heskett
Quoted 11 lines · Tomas Szepe
Quoted 11 lines

[[email protected]]

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we
put a tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

It is the usual American business english used in such cases.;-)

Standard American puke inducing pretentious florid business language if you want to be completely accurate. 8)

Huh? I thought Hans was being facetious, or was practicing his standup comedy. I can see something like that coming out of a far western oriental type when dealing with an american that isn't really understanding his accent, but here in the states, after we've explained it once in plain english, the next exchange will more than likely go the other way, possibly even making it to the 'hey dumbf*ck' stage on about the 4th reply.

Can you, please, make the last two sentences more readable for me, a "middle european type", in 'first' stage plain english?

Please, make your dialogues clear and clean for international usage! You're not "here in the states" when posting to an international list.

Manuel.

Quoted 3 lines

And I do object, albeit somewhat tongue-in-cheek, to applying that particular label/broad brush to all americans since I am one of those creatures.

#34 lore raw
In reply to Hans Reiser
Quoted 26 lines · Hans Reiser
Quoted 12 lines · Nikita Danilov
Quoted 3 lines · Shawn

Looks like the 2.5.74 is the last one of any respectable size. I'm thinking someone forgot a diff switch (N?) over at namesys...

Quoted 1 line

Hans? Time to long-distance spank someone?

Can you try following the instructions on the
http://www.namesys.com/code.html (requires bitkeeper)?

Nikita.

Quoted 2 lines
Quoted 2 lines · Tupshin Harper

Shawn wrote:

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

This is pretty f'ed, but it's on ftp://ftp.namesys.com/pub/tmp

Thanks, but I tried applying the 2.6.0-test1-reiser4-2.6.0-test1.diff from that location with a

lack of > > success.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

It applied cleanly, but it doesn't add a fs/reiser4 directory

and > > asociated contents. Is there an additional patch, or is this one broken?

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

-Tupshin

Quoted 7 lines

Nikita, how about phrasing this as:

`Dear and esteemed potential Reiser4 user, I apologize that we put a
tarball on our website and let it get so obsolete, thereby wasting your time. I am deleting it now, and will soon put a new one up. In the meantime, can you use bitkeeper if that is convenient for you? Here is the URL with the instructions for doing that.....'

It is the usual American business english used in such cases.;-)

No need to exaggerate... As I read these reiser4 related preliminary failure questions you can safely reuse
http://www.namesys.com/support.html

That page doesn't even need a correction therefore...

;-))

Bye,

Manuel

--
Pay 25€ for reading this mail! Just contact [email protected] on how to pay! No tax problems any more... You'll get internationally valid bills and receipts. :-))

#35 lore raw
In reply to Hans Reiser

Am Mit, 2003-07-23 um 23.02 schrieb Hans Reiser:

Quoted 3 lines

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

How do the wandering logs compare to the "wandering" logs of the log structured filesystem JFFS2? Does this mean I can achieve an implicit wear leveling for flash memory?

Quoted 4 lines

We are able to perform all filesystem operations fully atomically, while getting dramatic performance improvements. (Other attempts at introducing transactions into filesystems are said to have failed for performance reasons.)

How failsafe is it to switch off the power several times? When the filesystem really works atomically I should have either the old or the new version but no mixture. Does it still need to fsck or is the transaction replay done at mount time? In case one still needs fsck, what's the probability of needing user interaction? How long does it need to get a filesystem back into a consistent state after a powerloss (approx. per MB/GB)?

Background: I'm doing systems on compactflash cards and need a reliable filesystem for it. At the moment I'm using a compressed JFFS2 over the mtd emulation driver for block devices which works quite well but has a few catches...

Signature · 2 lines
Servus,
       Daniel
#36 lore raw
Quoted 9 lines · Daniel Egger

Am Mit, 2003-07-23 um 23.02 schrieb Hans Reiser:

Quoted 3 lines

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

How do the wandering logs compare to the "wandering" logs of the log structured filesystem JFFS2? Does this mean I can achieve an implicit wear leveling for flash memory?

I don't know enough about jffs2, but you can read about reiser4's "wandering logs" and transaction manager design at the
http://www.namesys.com/txn-doc.html.

Briefly speaking, in usual WAL (write-ahead logging) transaction system, whenever block is modified, journal record, describing changes to this block is forced to the on-disk journal before modified block is allowed to be written. In the worst case this means that data are written twice.

But if modified block is accessible through "pointer" of some kind stored in its "parent" block (one can think of ext2 inode addressing data blocks for example), we can

1. allocate new block location on the disk ("wandered location").

2. update parent block to point to the wandered location.

3. store modified block content to the wandered location.

4. add old block location to the journal. Old block is now journal record for the modified version.

Quoted 6 lines
Quoted 5 lines

We are able to perform all filesystem operations fully atomically, while getting dramatic performance improvements. (Other attempts at introducing transactions into filesystems are said to have failed for performance reasons.)

Quoted 7 lines

How failsafe is it to switch off the power several times? When the filesystem really works atomically I should have either the old or the new version but no mixture. Does it still need to fsck or is the transaction replay done at mount time? In case one still needs fsck, what's the probability of needing user interaction? How long does it need to get a filesystem back into a consistent state after a powerloss (approx. per MB/GB)?

I should warn everybody that reiser4 is _highly_ _experimental_ at this moment. Don't use it for production.

Quoted 9 lines

Background: I'm doing systems on compactflash cards and need a reliable filesystem for it. At the moment I'm using a compressed JFFS2 over the mtd emulation driver for block devices which works quite well but has a few catches...

Signature · 2 lines
Servus,
       Daniel

Calculemus!
Nikita.

#37 lore raw
Quoted 10 lines · Nikita Danilov
Quoted 8 lines · Daniel Egger

How failsafe is it to switch off the power several times? When the filesystem really works atomically I should have either the old or the new version but no mixture. Does it still need to fsck or is the transaction replay done at mount time? In case one still needs fsck, what's the probability of needing user interaction? How long does it need to get a filesystem back into a consistent state after a powerloss (approx. per MB/GB)?

Quoted 3 lines

I should warn everybody that reiser4 is _highly_ _experimental_ at this moment. Don't use it for production.

I'd like to ask this question differently: How failsafe is reiserfs4 *theoretically*. Assuming no bugs in implementation, what is the true import of its atomic nature? Strengths and potential weaknesses?

-Thanks
-Tupshin

#38 lore raw
Quoted 12 lines · Tupshin Harper
Quoted 10 lines · Nikita Danilov
Quoted 8 lines · Daniel Egger

How failsafe is it to switch off the power several times? When the filesystem really works atomically I should have either the old or the new version but no mixture. Does it still need to fsck or is the transaction replay done at mount time? In case one still needs fsck, what's the probability of needing user interaction? How long does it need to get a filesystem back into a consistent state after a powerloss (approx. per MB/GB)?

Quoted 6 lines
Quoted 3 lines

I should warn everybody that reiser4 is _highly_ _experimental_ at this moment. Don't use it for production.

I'd like to ask this question differently: How failsafe is reiserfs4 *theoretically*. Assuming no bugs in implementation, what is the true import of its atomic nature? Strengths and potential weaknesses?

Assuming no bugs in implementation it is very safe. :-)

This is lengthy topic. You may wish to read documents on the namesys.com. For example,

http://www.namesys.com/v4/reiser4_the_atomic_fs.html

Review

Quoted 4 lines
(quoted diff) +0 -2
-Thanks
-Tupshin

Nikita.

#39 lore raw
In reply to Nikita Danilov

Am Don, 2003-07-24 um 19.07 schrieb Nikita Danilov:

Quoted 3 lines

I don't know enough about jffs2, but you can read about reiser4's "wandering logs" and transaction manager design at the
http://www.namesys.com/txn-doc.html.

I've read it by now, thanks for the reference.

Quoted 4 lines

Briefly speaking, in usual WAL (write-ahead logging) transaction system, whenever block is modified, journal record, describing changes to this block is forced to the on-disk journal before modified block is allowed to be written. In the worst case this means that data are written twice.

Is there way to influence what is considered free space for the wandering blocks or is it a fixed algorithm? If the latter, what is the access pattern on the free space (like pseudorandom, cyclic linear, hashed)?

Quoted 2 lines

I should warn everybody that reiser4 is _highly_ _experimental_ at this moment. Don't use it for production.

That certainly doesn't stop me from trying... :)
Have you ran any tests to test the durabilty of your "transcrash" system for instance against sudden power dropouts?
Is the filesystem selfhealing or does one need fsck.reiserfs for it? If the latter: will it do the right thing (i.e. automatically bring the system into consistent shape not like ext3) when invoked with "-y"?

Signature · 2 lines
Servus,
       Daniel
#40 lore raw
Quoted 13 lines · Daniel Egger

Am Don, 2003-07-24 um 19.07 schrieb Nikita Danilov:

Quoted 3 lines

I don't know enough about jffs2, but you can read about reiser4's "wandering logs" and transaction manager design at the
http://www.namesys.com/txn-doc.html.

I've read it by now, thanks for the reference.

Quoted 4 lines

Briefly speaking, in usual WAL (write-ahead logging) transaction system, whenever block is modified, journal record, describing changes to this block is forced to the on-disk journal before modified block is allowed to be written. In the worst case this means that data are written twice.

Quoted 4 lines

Is there way to influence what is considered free space for the wandering blocks or is it a fixed algorithm? If the latter, what is the access pattern on the free space (like pseudorandom, cyclic linear, hashed)?

No special measures are taken to level block allocation. Wandered blocks are allocated to improve packing i.e., place blocks of the same file close to each other. Actually, it tries to place tree nodes in the parent-first order.

Quoted 10 lines
Quoted 3 lines

I should warn everybody that reiser4 is _highly_ _experimental_ at this moment. Don't use it for production.

That certainly doesn't stop me from trying... :)
Have you ran any tests to test the durabilty of your "transcrash" system for instance against sudden power dropouts?
Is the filesystem selfhealing or does one need fsck.reiserfs for it? If the latter: will it do the right thing (i.e. automatically bring the system into consistent shape not like ext3) when invoked with "-y"?

It should automatically replay journal on the mount. fsck.reiser4 is still needed owing to the bugs in the implementation, but, I am afraid, it cannot do much at the moment.

Quoted 4 lines
Signature · 3 lines
-- 
Servus,
       Daniel

Nikita.

#41 lore raw

Am Fre, 2003-07-25 um 15.02 schrieb Nikita Danilov:

Quoted 4 lines

No special measures are taken to level block allocation. Wandered blocks are allocated to improve packing i.e., place blocks of the same file close to each other. Actually, it tries to place tree nodes in the parent-first order.

So the new blocks are created as close as possible to the old blocks instead of say spreading them as far as possible. This is pretty bad for usage in the embedded world but I guess this is not the market you're aiming at. :(

Signature · 2 lines
Servus,
       Daniel
#42 lore raw
Quoted 8 lines · Daniel Egger

Am Fre, 2003-07-25 um 15.02 schrieb Nikita Danilov:

Quoted 4 lines

No special measures are taken to level block allocation. Wandered blocks are allocated to improve packing i.e., place blocks of the same file close to each other. Actually, it tries to place tree nodes in the parent-first order.

Quoted 4 lines

So the new blocks are created as close as possible to the old blocks instead of say spreading them as far as possible. This is pretty bad for usage in the embedded world but I guess this is not the market you're aiming at. :(

Reiser4 has plugin-based architecture. So, anybody is able to write new block allocator plugin.

Speaking about possible embedded usage... What kind of embedded devices do you mean. Reiser4 driver is big enough in size for some of them (for instance, for mine MPIO MP3 player :))

Signature · 1 line
We're flying high, we're watching the world passes by...
#43 lore raw

Am Fre, 2003-07-25 um 16.39 schrieb Yury Umanets:

Quoted 2 lines

Reiser4 has plugin-based architecture. So, anybody is able to write new block allocator plugin.

Cool.

Quoted 3 lines

Speaking about possible embedded usage... What kind of embedded devices do you mean. Reiser4 driver is big enough in size for some of them (for instance, for mine MPIO MP3 player :))

I'm talking about pretty standard ix86 hardware which has embedded like properties such as fanless and motorless use, hardware watchdog, flash memory but only few of the typical limitations like restricted memory (we are using 256 or 512 MB), slow CPU, few connectors.

So basically we do have pretty powerful hardware with huge storage and memory and now need a FS which is fast and reliable even on flash memory. JFFS2 is nice but way too slow once one has bigger sizes.

Signature · 2 lines
Servus,
       Daniel
#44 lore raw
Quoted 19 lines · Daniel Egger

Am Fre, 2003-07-25 um 16.39 schrieb Yury Umanets:

Quoted 2 lines

Reiser4 has plugin-based architecture. So, anybody is able to write new block allocator plugin.

Cool.

Quoted 3 lines

Speaking about possible embedded usage... What kind of embedded devices do you mean. Reiser4 driver is big enough in size for some of them (for instance, for mine MPIO MP3 player :))

I'm talking about pretty standard ix86 hardware which has embedded like properties such as fanless and motorless use, hardware watchdog, flash memory but only few of the typical limitations like restricted memory (we are using 256 or 512 MB), slow CPU, few connectors.

Quoted 3 lines

So basically we do have pretty powerful hardware with huge storage and memory and now need a FS which is fast and reliable even on flash memory. JFFS2 is nice but way too slow once one has bigger sizes.

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Signature · 1 line
We're flying high, we're watching the world passes by...
#45 lore raw

Am Sam, 2003-07-26 um 09.19 schrieb Yury Umanets:

Quoted 3 lines

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Linux is running just fine one the system, thanks. My question is whether reiserfs is suitable for flash devices. The chances to get some usable answers seem to be incredible low though...

Signature · 2 lines
Servus,
       Daniel
#46 lore raw
Quoted 10 lines · Daniel Egger

Am Sam, 2003-07-26 um 09.19 schrieb Yury Umanets:

Quoted 3 lines

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Linux is running just fine one the system, thanks. My question is whether reiserfs is suitable for flash devices. The chances to get some usable answers seem to be incredible low though...

Reiserfs cannot be used efficiently with flash, as it uses block size 4K (by default) and usual flash block size is in range 64K - 256K.

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000) and it is about three times as expensive as SDRAM.

So, it is better to use something more convenient. For instance jffs2.

But, if you are still want to use reiserfs for flash device, you should do at least the following:

(1) Make the journal substantial smaller of size.
(2) Don't turn tails off. This is useful to prolong flash live.

Regards.

Signature · 1 line
We're flying high, we're watching the world passes by...
#47 lore raw

Am Sam, 2003-07-26 um 16.54 schrieb Yury Umanets:

Now we're talking. :)

Quoted 2 lines

Reiserfs cannot be used efficiently with flash, as it uses block size 4K (by default) and usual flash block size is in range 64K - 256K.

Don't confuse block size with erase size. The former is the layout of the fs' data on the medium while the latter is the granulariy of the erase command which is important insofar that flash has to be erased (in most cases) before one can write new data on it.

However since you said that one can plug in a different block allocation scheme, I think it might be possible to work around that limitation by writing a block allocator which works around the limitations of the erase size.

Quoted 3 lines

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000)

I don't see what the compression has to do with the limited number of erase/write cycles.

Quoted 1 line

and it is about three times as expensive as SDRAM.

That's true but not important to us. The system right now fits nicely on a 128MB CF card when using ext2 or on 64MB when using JFFS2. The latter is far more stable and reliable but dogslow. Since the price difference between 128MB and 64CF is rather small and the cost of the overall system relatively high this is no argument for us.

Quoted 1 line

So, it is better to use something more convenient. For instance jffs2.

Convenient only insofar that it's more reliable. It's a pain in the neck to setup for non hardwired flash chips and to boot, it also takes forever to mount and to write on it.

Quoted 2 lines

(1) Make the journal substantial smaller of size.
(2) Don't turn tails off. This is useful to prolong flash live.

Thanks. But first I'll have a look at your plugin architecture to see how feasible a different implementation of block allocation especially for flash devices would be.

Signature · 2 lines
Servus,
       Daniel
#48 lore raw
Quoted 6 lines · Daniel Egger
Quoted 3 lines

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000)

I don't see what the compression has to do with the limited number of erase/write cycles.

It's a subtle point - let's say you have a 32K blob of data and a 4K block/
erase/whatever size on the flash. If you write it uncompressed, then 8 blocks
are going to get an erase cycle. If however you can compress it down to 12K (not at all unusual for text), then only 3 blocks get an erase cycle, and the other 5 blocks get to live longer...

#49 lore raw
In reply to Daniel Egger
Quoted 7 lines · Daniel Egger

Am Sam, 2003-07-26 um 16.54 schrieb Yury Umanets:

Now we're talking. :)

Quoted 7 lines
Quoted 2 lines

Reiserfs cannot be used efficiently with flash, as it uses block size 4K (by default) and usual flash block size is in range 64K - 256K.

Don't confuse block size with erase size. The former is the layout of the fs' data on the medium while the latter is the granulariy of the erase command which is important insofar that flash has to be erased (in most cases) before one can write new data on it.

So what? I mean, that if an IO request size does not equal to flash erase size, then corresponding block device driver can't just submit data to flash, but need maintain some cache, and cache size the same as erase size for particular flash device. And in the case when WRITE request is encountered, and write sector does not equal to start sector of cached data or cache is empty, block device driver should read data from flash first to fill cache up. This is redundant IO operation.

Quoted 5 lines

However since you said that one can plug in a different block allocation scheme, I think it might be possible to work around that limitation by writing a block allocator which works around the limitations of the erase size.

This is some misunderstanding :) First we've spoken about reiser4, then you asked how does reiserfs behave on flash devices and is it convenient for flash at all.

Just make sure, that we're speaking about the same thing:

Plugin-based architecture is used in reiser4, not in reiserfs (reiser3). Reiser4 is fully different, written from the scratch filesystem.

Quoted 9 lines
Quoted 4 lines

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000)

I don't see what the compression has to do with the limited number of erase/write cycles.

Compressed data which should be written is smaller then uncompressed one, thus, its writing affects smaller number of blocks. Each block will be erased rarely, that will prolong flash live.

Quoted 8 lines
Quoted 2 lines

and it is about three times as expensive as SDRAM.

That's true but not important to us. The system right now fits nicely on a 128MB CF card when using ext2 or on 64MB when using JFFS2. The latter is far more stable and reliable but dogslow. Since the price difference between 128MB and 64CF is rather small and the cost of the overall system relatively high this is no argument for us.

So, you prefer speed? What do you use for this x86 box with flash?

Quoted 5 lines
Quoted 2 lines

So, it is better to use something more convenient. For instance jffs2.

Convenient only insofar that it's more reliable.

I'd not say, that ext2 is too reliable though.

Quoted 11 lines

It's a pain in the neck
to setup for non hardwired flash chips and to boot, it also takes forever to mount and to write on it.

Quoted 2 lines

(1) Make the journal substantial smaller of size.
(2) Don't turn tails off. This is useful to prolong flash live.

Thanks. But first I'll have a look at your plugin architecture to see how feasible a different implementation of block allocation especially for flash devices would be.

You should take a look to reiser4, not to reiserfs. Don't forget :)

But I don't understand, why do you want to make changes in current block allocator plugin? In other words, what is wrong with current implementation, which is willing to allocate blocks closer one to another one?

I thought, if blocks lie side by side, as current block allocator does, this increases probability of flash block device cache hitting (take a look to drivers/mtd/mtdblock.c), what is definitely good. Isn't it?

Regards.

Signature · 1 line
We're flying high, we're watching the world passes by...
#50 lore raw

Am Son, 2003-07-27 um 12.30 schrieb Yury Umanets:

Quoted 7 lines

So what? I mean, that if an IO request size does not equal to flash erase size, then corresponding block device driver can't just submit data to flash, but need maintain some cache, and cache size the same as erase size for particular flash device. And in the case when WRITE request is encountered, and write sector does not equal to start sector of cached data or cache is empty, block device driver should read data from flash first to fill cache up. This is redundant IO operation.

Right, but it should be possible to ensure (by using a special encoding) that a part of the erased block can be detected as empty or already occupied by reading just a few bytes. Sure this is a tradeoff but one I'd be willing to make. :)

Quoted 8 lines

This is some misunderstanding :) First we've spoken about reiser4, then you asked how does reiserfs behave on flash devices and is it convenient for flash at all.

Just make sure, that we're speaking about the same thing:

Plugin-based architecture is used in reiser4, not in reiserfs (reiser3). Reiser4 is fully different, written from the scratch filesystem.

My bad, I thought you're using the term reiserfs also for reiser4. I was always talking about reiser4 when I said reiserfs.

Quoted 6 lines
Quoted 2 lines

I don't see what the compression has to do with the limited number of erase/write cycles.

Compressed data which should be written is smaller then uncompressed one, thus, its writing affects smaller number of blocks. Each block will be erased rarely, that will prolong flash live.

Only when the data is in motion. Considering that most of the data is quite fixed with only some bytes of configuration being written a few times and an update of a few packages every now and then I'm pretty sure the wear affect will hardly hit. It's more important, that the configuration bits are spread evenly over the full filesystem.

Quoted 1 line

So, you prefer speed?

Yes. Especially startup times are important to us but also execution times for cachecold executables.

Quoted 1 line

What do you use for this x86 box with flash?

This are VIA Eden boxes with 667 Mhz fanless x86 compatible CPUs. They come in a booksize chassis and deliver pretty impressive performance for their size.

Quoted 2 lines
Quoted 1 line

Convenient only insofar that it's more reliable.

I'd not say, that ext2 is too reliable though.

No it's not. Especially the fsck annoyance is a real killer because we can either not run it, thereby risking an inconsistent filesystem or run it unattended thereby risking a loss of files.

Quoted 1 line

You should take a look to reiser4, not to reiserfs. Don't forget :)

I'm aware, thanks. :)

Quoted 8 lines

But I don't understand, why do you want to make changes in current block allocator plugin? In other words, what is wrong with current implementation, which is willing to allocate blocks closer one to another one?

I thought, if blocks lie side by side, as current block allocator does, this increases probability of flash block device cache hitting (take a look to drivers/mtd/mtdblock.c), what is definitely good. Isn't it?

I've some doubts that placing blocks close to another wears out all of the flash equally. I imagine something like circular or hashed block allocator which ensures equal wear leveling taking the erasesize of the flash into account.

Signature · 2 lines
Servus,
       Daniel
#51 lore raw
Quoted 11 lines · Daniel Egger

Am Son, 2003-07-27 um 12.30 schrieb Yury Umanets:

Quoted 7 lines

So what? I mean, that if an IO request size does not equal to flash erase size, then corresponding block device driver can't just submit data to flash, but need maintain some cache, and cache size the same as erase size for particular flash device. And in the case when WRITE request is encountered, and write sector does not equal to start sector of cached data or cache is empty, block device driver should read data from flash first to fill cache up. This is redundant IO operation.

Quoted 4 lines

Right, but it should be possible to ensure (by using a special encoding) that a part of the erased block can be detected as empty or already occupied by reading just a few bytes. Sure this is a tradeoff but one I'd be willing to make. :)

This is probably tradeoff for flash producers first of all.

Quoted 5 lines
Quoted 4 lines

This is some misunderstanding :) First we've spoken about reiser4, then you asked how does reiserfs behave on flash devices and is it convenient for flash at all.

Quoted 7 lines
Quoted 4 lines

Just make sure, that we're speaking about the same thing:

Plugin-based architecture is used in reiser4, not in reiserfs (reiser3). Reiser4 is fully different, written from the scratch filesystem.

My bad, I thought you're using the term reiserfs also for reiser4. I was always talking about reiser4 when I said reiserfs.

Reiser4 will use compression. So, it will be more convenient or flash devices. But using XIP is problematic in this case.

Quoted 18 lines
Quoted 7 lines
Quoted 3 lines

I don't see what the compression has to do with the limited number of erase/write cycles.

Compressed data which should be written is smaller then uncompressed one, thus, its writing affects smaller number of blocks. Each block will be erased rarely, that will prolong flash live.

Only when the data is in motion. Considering that most of the data is quite fixed with only some bytes of configuration being written a few times and an update of a few packages every now and then I'm pretty sure the wear affect will hardly hit. It's more important, that the configuration bits are spread evenly over the full filesystem.

Quoted 1 line

So, you prefer speed?

Quoted 8 lines

Yes. Especially startup times are important to us but also execution times for cachecold executables.

Quoted 1 line

What do you use for this x86 box with flash?

This are VIA Eden boxes with 667 Mhz fanless x86 compatible CPUs. They come in a booksize chassis and deliver pretty impressive performance for their size.

My friend used something like this for video player :)

Quoted 23 lines
Quoted 3 lines
Quoted 2 lines

Convenient only insofar that it's more reliable.

I'd not say, that ext2 is too reliable though.

No it's not. Especially the fsck annoyance is a real killer because we can either not run it, thereby risking an inconsistent filesystem or run it unattended thereby risking a loss of files.

Quoted 1 line

You should take a look to reiser4, not to reiserfs. Don't forget :)

I'm aware, thanks. :)

Quoted 8 lines

But I don't understand, why do you want to make changes in current block allocator plugin? In other words, what is wrong with current implementation, which is willing to allocate blocks closer one to another one?

I thought, if blocks lie side by side, as current block allocator does, this increases probability of flash block device cache hitting (take a look to drivers/mtd/mtdblock.c), what is definitely good. Isn't it?

Quoted 4 lines

I've some doubts that placing blocks close to another wears out all of the flash equally. I imagine something like circular or hashed block allocator which ensures equal wear leveling taking the erasesize of the flash into account.

Probably you are right in general.

But erasesize is block device driver abstraction level related issue. General purpose filesystem should not be concerned about it.

Signature · 1 line
We're flying high, we're watching the world passes by...
#52 lore raw
In reply to Yury Umanets
Quoted 3 lines · Yury Umanets

I thought, if blocks lie side by side, as current block allocator does, this increases probability of flash block device cache hitting (take a look to drivers/mtd/mtdblock.c), what is definitely good. Isn't it?

Please do not use the word 'good' in the same sentence as mtdblock.

The mtdblock device is fundamentally not suitable for use in production in write mode -- by design.

'Normal' file systems such as ext3/reiser{fs,4} do not operate on flash directly. Flash is _different_ -- you can't atomically overwrite small-sized 'sectors', and you have to care about wear levelling.

There are two types of flash. On the original and more common NOR flash, the 'erase blocks' are typically of the order of 64KiB in size -- they start off filled with 0xFF, and you can clear bits individually until you're bored or they're all set to zero. At which point you can erase the _whole_ eraseblock back to all ones again.

On the newer and cheaper NAND flash, there are more restrictions. The eraseblocks are smaller (typ. 8KiB) and it's subdivided into 'pages' of typ. 512 bytes -- while with NOR flash you can clear bits individually in as many cycles as you like, with NAND you have a limited number of write cycles to any given 'page', after which point current leakage causes the contents of that page to become undefined.

Neither of these are suitable for use directly as a block device. You need some kind of 'translation layer' to make them emulate a hard drive in some way. Such a translation layer will ideally also handle wear levelling and bad block management for you.

The most naïve such 'translation layer' is that implemented in mtdblock.c. Upon a write request, it reads the eraseblock containing the 512-byte sector to be changed, modifies the in-memory copy, then erases the flash eraseblock and writes it back again. There is some caching present to prevent subsequent writes within the _same_ eraseblock from causing more than one erase cycle, but there's no wear levelling and no bad block management. There's a 1:1 mapping from 'logical' addresses within the fake block device and physical addresses on the flash.

If you lose power or crash between the 'erase' and 'writeback' portions of the above-described read/modify/erase/writeback cycle, you have lost not only the contents of the sector you were trying to modify, but also the 128KiB in which it was resident. This is _really_ not a good idea. The mtdblock device should be used only for read-only operation (e.g. with cramfs) in production, and for writing only during setup.

There also exist some more complicated 'translation layers', which are basically pseudo-filesystems used to emulate a block device using flash as backing store. They perform their own wear levelling and journalling to ensure reliability. The ones supported by Linux are FTL (used for NOR flash especially in PCMCIA devices) and NFTL (used on the NAND flash found in DiskOnChip devices). Although these aren't _fundamentally_ broken as mtdblock is, they are still somewhat suboptimal. There's a SmartMedia translation layer (SM is basically just NAND flash) too, which nobody's bothered to implement yet.

They still need to do some form of garbage-collection, copying around still-valid sectors from one place on the physical medium to another, to allow an eraseblock which contained some obsolete data to be completely obsoleted and hence erased to free up usable space. However, the block device layer is never told by the file system when a sector is no longer used -- so if you fill the file system with data then 'rm -rf *', the _block_ device will still think it's entirely full of data, and carefully copy that data around the physical medium for you in case you want it back.

Also, your file system needs its _own_ journalling to ensure data integrity at the higher level, since the block device 'translation layer' only ensures the same form of data integrity that a normal hard drive would achieve, and nothing more. So you (the file system) end up writing data (or at least metadata) changes out to the physical medium twice, once to the 'journal' on the faked block device, and once to the real location on the faked block device, while the underlying 'translation layer' is performing its own journalling underneath you to ensure integrity too. This is far from ideal for flash wear.

Hence the development of JFFS, JFFS2 and YAFFS -- file systems which operate directly on the flash chips rather than introducing the suboptimal 'fake' block devices. This isn't DOS any more -- we don't need to provide INT 13h Disk BIOS emulation and then expect everyone to use FAT atop it :)

For flash you can access directly as _flash_, a file system specifically designed for the purpose is the better approach. JFFS2 performance is being improved, and YAFFS (and soon YAFFS2) take different approaches to the problem.

Some devices, however, are made of flash but do their best to hide it from you.

CompactFlash may have flash internally but to all extents and purposes, as far as the computer can tell, it really is a (somewhat unreliable) IDE drive. It has a 'translation layer' built into its black box -- you can't tell whether it does its own wear levelling or not, but in the majority of cases we suspect not. Anecdotal evidence is that its internal firmware also tends to get the journalling part of the 'translation layer' wrong too, and can get its internal 'pseudo-filesystem' into an inconsistent state from which it cannot recover. Of course, since you can't access the flash directly from software, you cannot do anything but bin the unit when this happens. I assume the usb-storage devices are very similar.

The practice of using JFFS2 on CF (and other real block devices) isn't really something I encourage, but it seems to have happened because there isn't a 'real' block device based file system which is powerfail-save, optimised for space and which uses compression. If reiser4 can fill that gap, that would be pleasing to me.

Signature · 1 line
dwmw2
#53 lore raw
Quoted 5 lines · In article <[email protected]> you wrote:

The practice of using JFFS2 on CF (and other real block devices) isn't really something I encourage, but it seems to have happened because there isn't a 'real' block device based file system which is powerfail-save, optimised for space and which uses compression. If reiser4 can fill that gap, that would be pleasing to me.

Thanks for that great article, would you care to describe where the slowness of JFFS2 is coming from?

Do you have experiences with XFS and ext3 (datajournal) filesystems in terms of power fail security?

Greetings
Bernd

Signature · 2 lines
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/
#54 lore raw

You didn't Cc me; there are 7240 unread messages in my linux-kernel folder which are about to be read with the 'd' key... it's lucky I saw your reply at all. Please don't drop recipients.

Quoted 9 lines · Bernd Eckenfels
Quoted 5 lines · In article <[email protected]> you wrote:

The practice of using JFFS2 on CF (and other real block devices) isn't really something I encourage, but it seems to have happened because there isn't a 'real' block device based file system which is powerfail-save, optimised for space and which uses compression. If reiser4 can fill that gap, that would be pleasing to me.

Thanks for that great article, would you care to describe where the slowness of JFFS2 is coming from?

During mount, it comes from the fact that we need to find every single log entry (node) in the file system. It's a purely log-structured file system -- there is _no_ positional information; it's just a jumble of log entries with version numbers to show ordering, and we need to find them all and note which inode# they belong to.

We've already got about an order of magnitude improvement in mount time from 2.4 to 2.5 by deferring much of the work (in particular crc32 checking on all those nodes) from the actual mount to later, and by eliminating memcpy() into RAM of data on NOR flash where we can actually just use pointers directly into flash instead.

The scan is still slow on NAND flash, which we can't directly dereference and _have_ to copy from the flash to read it. As I said elsewhere, it's about 30 seconds for a 30%-full 144MiB DiskOnChip device. The fix for this, yet to be implemented, will be a 'tailer' at the end of each flash eraseblock, containing all the information which we would otherwise have to glean by scanning the whole of the eraseblock looking for log entries. Then we just need to read the tailer from the end of each block rather than reading the whole block.

In this way we trade off a small amount of space for a large improvement in mount time. This seems like an eminently sensible tradeoff especially given that it was the large size of these devices which caused the mount time to become problematic in the first place -- there's room to spare. It can also be a configurable option. For embedded devices which almost never cold-boot and usually suspend to RAM instead, a very slow mount isn't necessarily a show-stopper.

Performance during runtime isn't often cited as a problem, but can also be improved somewhat. Currently, our garbage collection is rather naïve; it picks a 'victim' eraseblock, generally one of the 'dirtiest' (i.e. containing the most log entries which are obsoleted by subsequent writes elsewhere). It then proceeds to obsolete all the nodes in that eraseblock which remain valid, by just writing out the same data elsewhere -- at which point it can erase the 'victim' and add it back to the free_list.

Garbage collection is most efficient when the 'victim' block in fact contains almost no still-valid nodes, and can just about be deleted straight away. It's at its least efficient when we pick a completely clean block in which _all_ the nodes need to be copied elsewhere -- we copy and entire eraseblock without making _any_ extra free space in that case (we do it very occasionally for wear levelling purposes).

This garbage collection is generally done in the background thread, which has some primitive heuristics to tell it how much space to keep available. But it's also done just-in-time, if you either kill the GC thread or if you're saturating the FS with writes and the GC thread hasn't had time to keep making space for you before you needed it.

The problem here is that GC writes which obsolete nodes in the victim eraseblock are mingled with the ongoing new writes driven from userspace. So we tend to end up writing out new eraseblocks full of data nodes which are half new and volatile data, and half old stuff which almost never changes like libraries and binaries. The new stuff like temporary datafiles tends to get changed, overwritten or deleted -- and the old stuff tends not to. So (referring to the paragraph on GC efficiency above) our behaviour is such that we tend to end up with none of the mostly-dirty blocks which are most efficient to garbage-collect from; instead we have blocks which are roughly half-clean and half-dirty.

The proposed fix for this is to split writes into two ongoing streams instead of the current one stream -- we write new data into one eraseblock while we write GC'd data into another. That way, the old stable data nodes tend to get grouped together into 100% clean blocks which we can just ignore (until we decide to wear level them), and 100% _dirty_ blocks which are nice and quick to GC.

I worry about the memory usage sometimes, and play some rather disgusting tricks to keep it down -- but in practice that's more of an issue for the eCos port of JFFS2 than Linux; Linux boxen generally have enough memory that it's not a problem.

Quoted 2 lines

Do you have experiences with XFS and ext3 (datajournal) filesystems in terms of power fail security?

No. I've been at the receiving end of an automated powerfail script which has been stress-testing JFFS2 and forcibly power cycling the device in question every few minutes -- which gave rise to all kinds of interesting observations about flash behaviour under those circumstances. But I'm not aware of similar tests being done on XFS or ext3 with repeated power failure and integrity tests.

Signature · 1 line
dwmw2
#55 lore raw

Hello David,

Quoted 3 lines · In article <[email protected]> you wrote:

I've been at the receiving end of an automated powerfail script which has been stress-testing JFFS2 and forcibly power cycling the device in question every few minutes

Is this needing some special hardware support, or is it kind of forcing apm/apci power downs? Can you publish that script? I would need that for some stress testing of applications and the kernel.

I also wonder, what the best method is to test those hard crashes, especially interesting is the case, where disks get power interruption at write, to see if the filesystem and block layer recovers from things like half written (format needing) blocks.

Greetings
Bernd

Signature · 2 lines
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/
#56 lore raw

This box is IPv4 only -- your email address with only an AAAA record is probably going to bounce again... :)

Quoted 3 lines · Bernd Eckenfels

Is this needing some special hardware support, or is it kind of forcing apm/apci power downs? Can you publish that script? I would need that for some stress testing of applications and the kernel.

I didn't do it myself -- I just got to fix the bugs which turned up ;)

I think it was done with X10 automated power switching stuff.

Quoted 4 lines

I also wonder, what the best method is to test those hard crashes, especially interesting is the case, where disks get power interruption at write, to see if the filesystem and block layer recovers from things like half written (format needing) blocks.

Journal at application layer to external network-attached storage. Check on-device fs integrity against your network journal at boot, continue stress testing from where you left off.

Signature · 1 line
dwmw2
#57 lore raw
In reply to Yury Umanets
Quoted 15 lines · Yury Umanets
Quoted 9 lines · Daniel Egger

Am Sam, 2003-07-26 um 09.19 schrieb Yury Umanets:

Quoted 5 lines

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Quoted 9 lines
Quoted 5 lines

Linux is running just fine one the system, thanks. My question is whether reiserfs is suitable for flash devices. The chances to get some usable answers seem to be incredible low though...

Reiserfs cannot be used efficiently with flash, as it uses block size 4K (by default) and usual flash block size is in range 64K - 256K.

This answer is incorrect. The device driver will hide this from us, slum squeezing will tend to write in large batches, and things will probably work. However, you should try it and see rather than theorize.

Quoted 5 lines

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000) and it is about three times as expensive as SDRAM.

We have compression plugins that will be ready soon. Go ask Edward in the chair behind you what he does for a living.;-)

Quoted 14 lines

So, it is better to use something more convenient. For instance jffs2.

But, if you are still want to use reiserfs for flash device, you should do at least the following:

(1) Make the journal substantial smaller of size.
(2) Don't turn tails off. This is useful to prolong flash live.

Regards.

He is asking about reiser4, not reiserfs V3.

Signature · 1 line
Hans
#58 lore raw
Quoted 23 lines · Hans Reiser
Quoted 21 lines · Yury Umanets
Quoted 17 lines · Daniel Egger

Am Sam, 2003-07-26 um 09.19 schrieb Yury Umanets:

Quoted 5 lines

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Linux is running just fine one the system, thanks. My question is whether reiserfs is suitable for flash devices. The chances to get some usable answers seem to be incredible low though...

Quoted 7 lines
Quoted 3 lines

Reiserfs cannot be used efficiently with flash, as it uses block size 4K (by default) and usual flash block size is in range 64K - 256K.

This answer is incorrect. The device driver will hide this from us, slum squeezing will tend to write in large batches, and things will probably work. However, you should try it and see rather than theorize.

See my explanation in last emails. Also I have not just theorized. I have been developing block device driver for MPIO players (Smart Card based one). And reiserfs does not use squeezing on flush (and I've spoken about reiserfs here).

Quoted 9 lines
Quoted 6 lines

Also reiserfs does not use compression, that would be very nice of it :), because flash has limited number of erase cycles per block (in range 100.000) and it is about three times as expensive as SDRAM.

We have compression plugins that will be ready soon. Go ask Edward in the chair behind you what he does for a living.;-)

Yes, we have compression plugins in reiser4, but we have spoken about reiserfs.

Quoted 16 lines
Quoted 4 lines

So, it is better to use something more convenient. For instance jffs2.

Quoted 11 lines

But, if you are still want to use reiserfs for flash device, you should do at least the following:

(1) Make the journal substantial smaller of size.
(2) Don't turn tails off. This is useful to prolong flash live.

Regards.

He is asking about reiser4, not reiserfs V3.

Signature · 1 line
We're flying high, we're watching the world passes by...
#59 lore raw
In reply to Daniel Egger
Quoted 10 lines · Daniel Egger

Am Sam, 2003-07-26 um 09.19 schrieb Yury Umanets:

Quoted 5 lines

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Quoted 6 lines

Linux is running just fine one the system, thanks. My question is whether reiserfs is suitable for flash devices. The chances to get some usable answers seem to be incredible low though...

it is suitable for any flash device that has wear leveling built into the hardware (e.g. all compact flash cards), or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

Signature · 1 line
Hans
#60 lore raw

Am Son, 2003-07-27 um 15.28 schrieb Hans Reiser:

Quoted 2 lines

it is suitable for any flash device that has wear leveling built into the hardware (e.g. all compact flash cards)

Are you sure CF cards have wear leveling? I'm pretty confident that they have defect sector management but no wear leveling. There's a huge difference between those two.

Quoted 2 lines

or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

This is normally done by the filesystem (e.g. JFFS2).

Signature · 2 lines
Servus,
       Daniel
#61 lore raw
Quoted 5 lines · Daniel Egger

Am Son, 2003-07-27 um 15.28 schrieb Hans Reiser:

Quoted 2 lines

it is suitable for any flash device that has wear leveling built into the hardware (e.g. all compact flash cards)

Quoted 8 lines

Are you sure CF cards have wear leveling? I'm pretty confident that they have defect sector management but no wear leveling. There's a huge difference between those two.

Quoted 2 lines

or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

This is normally done by the filesystem (e.g. JFFS2).

Normally device driver should be concerned about making wear out smaller. It is up to it IMHO.

Signature · 1 line
We're flying high, we're watching the world passes by...
#62 lore raw
Quoted 10 lines · Yury Umanets
Quoted 6 lines · Daniel Egger

Am Son, 2003-07-27 um 15.28 schrieb Hans Reiser:

Quoted 2 lines

or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

This is normally done by the filesystem (e.g. JFFS2).

Normally device driver should be concerned about making wear out smaller. It is up to it IMHO.

The driver should do the logical to physical mapping, but the portability vanishes if the filesystem to physical mapping is not the same for all machines and operating systems. For pluggable devices this is important.

The leveling seems to be done by JFFs2 in a portable way, and that's as it should be. If the leveling were in the driver I don't believe even FAT would work.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#63 lore raw
Quoted 14 lines · Bill Davidsen
Quoted 10 lines · Yury Umanets
Quoted 6 lines · Daniel Egger

Am Son, 2003-07-27 um 15.28 schrieb Hans Reiser:

Quoted 2 lines

or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

This is normally done by the filesystem (e.g. JFFS2).

Normally device driver should be concerned about making wear out smaller. It is up to it IMHO.

Quoted 7 lines

The driver should do the logical to physical mapping, but the portability vanishes if the filesystem to physical mapping is not the same for all machines and operating systems. For pluggable devices this is important.

The leveling seems to be done by JFFs2 in a portable way, and that's as it should be. If the leveling were in the driver I don't believe even FAT would work.

Hello Bill,

Yes, you are right. Device driver cannot take care about leveling.

It is able only to take care about simple caching (one erase block) in order to make wear out smaller and do not read/write whole block if one sector should be written.

Part of a filesystem called "block allocator" should take care about leveling.

#64 lore raw
Quoted 1 line · Yury Umanets

Yes, you are right. Device driver cannot take care about leveling.

The hardware device driver doesn't. The 'translation layer' does, in the case where you are using a traditional block-based file system.

If you consider the translation layer and the underlying raw hardware driver together to form the 'device driver' from the filesystem's perspective and in the context of the above sentence, then you're incorrect -- it can, and in general it _does_ take care of wear levelling.

Quoted 3 lines

It is able only to take care about simple caching (one erase block) in order to make wear out smaller and do not read/write whole block if one sector should be written.

Whatever meaning of 'device driver' you meant to use -- no.

The raw hardware driver provides only raw read/write/erase functionality; no caching is appropriate.

The optional translation layer which simulates a block device provides far more than simple caching -- it provides wear levelling, bad block management, etc. All using a standard layout on the flash hardware for portability.

(Except in the special case of the 'mtdblock' translation layer, which is not suitable for anything but read-only operation on devices without any bad blocks to be worked around.)

Quoted 2 lines

Part of a filesystem called "block allocator" should take care about leveling.

That's insufficient. In a traditional file system, blocks get overwritten without being freed and reallocated -- the allocator isn't always involved.

If you want to teach a file system about flash and wear levelling, you end up ditching the pretence that it's a block device entirely and working directly with the flash hardware driver.

Either that or use a translation layer which does it _all_ for the file system and then just use a standard file system on that simulated block device.

Between those two extremes, very little actually makes sense.

If you introduce the gratuitous extra 'block device' abstraction layer which doesn't really fit the reality of flash hardware very well at all, you end up wanting to violate the layering in so many ways that you realise you really shouldn't have been pretending to be a block device in the first place.

Signature · 1 line
dwmw2
#65 lore raw
Quoted 45 lines · David Woodhouse
Quoted 1 line · Yury Umanets

Yes, you are right. Device driver cannot take care about leveling.

The hardware device driver doesn't. The 'translation layer' does, in the case where you are using a traditional block-based file system.

If you consider the translation layer and the underlying raw hardware driver together to form the 'device driver' from the filesystem's perspective and in the context of the above sentence, then you're incorrect -- it can, and in general it _does_ take care of wear levelling.

Quoted 3 lines

It is able only to take care about simple caching (one erase block) in order to make wear out smaller and do not read/write whole block if one sector should be written.

Whatever meaning of 'device driver' you meant to use -- no.

The raw hardware driver provides only raw read/write/erase functionality; no caching is appropriate.

The optional translation layer which simulates a block device provides far more than simple caching -- it provides wear levelling, bad block management, etc. All using a standard layout on the flash hardware for portability.

(Except in the special case of the 'mtdblock' translation layer, which is not suitable for anything but read-only operation on devices without any bad blocks to be worked around.)

Quoted 2 lines

Part of a filesystem called "block allocator" should take care about leveling.

That's insufficient. In a traditional file system, blocks get overwritten without being freed and reallocated -- the allocator isn't always involved.

If you want to teach a file system about flash and wear levelling, you end up ditching the pretence that it's a block device entirely and working directly with the flash hardware driver.

Either that or use a translation layer which does it _all_ for the file system and then just use a standard file system on that simulated block device.

Quoted 7 lines

Between those two extremes, very little actually makes sense.

If you introduce the gratuitous extra 'block device' abstraction layer which doesn't really fit the reality of flash hardware very well at all, you end up wanting to violate the layering in so many ways that you realise you really shouldn't have been pretending to be a block device in the first place.

Agreed fully with you David. Thanks for explanation.

Only there are probably cannot be so fair borders between levels. Thus, some functions of translation layer may be passed to filesystem level (higher one). For instance some things about block allocating. And some other functions may be passed to device driver layer (lower one).

Regards.

#66 lore raw
In reply to David Woodhouse
Quoted 2 lines · David Woodhouse

The raw hardware driver provides only raw read/write/erase functionality; no caching is appropriate.

Okay, that's the model I have in mind as the driver, assuming you included seek in that list.

Quoted 6 lines

The optional translation layer which simulates a block device provides far more than simple caching -- it provides wear levelling, bad block management, etc. All using a standard layout on the flash hardware for portability.

Quoted 7 lines

(Except in the special case of the 'mtdblock' translation layer, which is not suitable for anything but read-only operation on devices without any bad blocks to be worked around.)

If you want to teach a file system about flash and wear levelling, you end up ditching the pretence that it's a block device entirely and working directly with the flash hardware driver.

I don't think that's right. A file system may very well be *optimized* for performance on a certain class of device, but that doesn't make it device dependent. For example some early SysV filesystems had the directory in the middle of the platters to minimize seek distance when the partition was only partially filled. I'd bet I could run JFFS2 on a normal drive, and I know I can run FAT, ext2, etc on a CF. Now if Linux only knew how to read SysV.4 drives I could save some critical old data from the 90's, but that's another issue...

Quoted 4 lines

Either that or use a translation layer which does it _all_ for the file system and then just use a standard file system on that simulated block device.

That sounds like a loopback mount, sort of. At least a feature which could be added fairly easily, like crypto mounts.

Quoted 8 lines

Between those two extremes, very little actually makes sense.

If you introduce the gratuitous extra 'block device' abstraction layer which doesn't really fit the reality of flash hardware very well at all, you end up wanting to violate the layering in so many ways that you realise you really shouldn't have been pretending to be a block device in the first place.

Agreed, if you're going to do that type of fakery it's probably better to take some overhead and not give up good design in the name of performance for something which is usually limited by other factors like device performance, or seldom used. Slow and robust is easier to maintain.

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
#67 lore raw
Quoted 7 lines · Bill Davidsen
Quoted 2 lines · David Woodhouse

The raw hardware driver provides only raw read/write/erase functionality; no caching is appropriate.

Okay, that's the model I have in mind as the driver, assuming you included seek in that list.

No seek -- offsets are passed to the read/write methods, like pread()/pwrite().

We have read(), which reads from the flash hardware, erase() which resets an eraseblock to all 0xFF, and write() which performs an bitwise AND operation between the contents of the flash and the buffer provided (you can only clear bits on flash, you can't set them except by erasing; qv.).

Quoted 4 lines
Quoted 3 lines

If you want to teach a file system about flash and wear levelling, you end up ditching the pretence that it's a block device entirely and working directly with the flash hardware driver.

Quoted 8 lines

I don't think that's right. A file system may very well be *optimized* for performance on a certain class of device, but that doesn't make it device dependent. For example some early SysV filesystems had the directory in the middle of the platters to minimize seek distance when the partition was only partially filled. I'd bet I could run JFFS2 on a normal drive, and I know I can run FAT, ext2, etc on a CF. Now if Linux only knew how to read SysV.4 drives I could save some critical old data from the 90's, but that's another issue...

CF != flash. For the purpose of this discussion, 'CF' and 'normal drive' are identical concepts. They are block devices; not MTD devices. JFFS2 does not work on them (without trickery) since JFFS2 is not written to use block devices.

Conversely, you cannot use FAT/ext/etc on real flash without some kind of 'translation layer' to make it pretend to be a block device.

CF just happens to have that translation layer built in to its hardware rather than doing it in software -- so as far as the computer is concerned, CF _is_ an IDE hard drive.

Quoted 6 lines
Quoted 3 lines

Either that or use a translation layer which does it _all_ for the file system and then just use a standard file system on that simulated block device.

That sounds like a loopback mount, sort of. At least a feature which could be added fairly easily, like crypto mounts.

No, it's nothing like a loopback mount. It's a pseudo-filesystem. Three sane implementations of such a thing are already in the kernel, and one hopelessly trivial readonly one.

Signature · 1 line
dwmw2
#68 lore raw
In reply to Bill Davidsen
Quoted 3 lines · Bill Davidsen

The driver should do the logical to physical mapping, but the portability vanishes if the filesystem to physical mapping is not the same for all machines and operating systems. For pluggable devices this is important.

The portability also vanishes if the file system layout is not the same for all machines and operating systems... what's your point?

Just like there are standard file systems, there are also standard 'translation layers' -- pseudofilesystems which are used to emulate a hard drive on flash storage -- and some of these are implemented for Linux.

Take a PCMCIA flash card (real flash, not CF) with FTL and FAT on it, and it'll work just fine under both Windows and Linux, because they both use the standard FTL and FAT formats.

FTL provides the logical<->physical mapping and the wear levelling, FAT is just normal FAT.

Quoted 2 lines

The leveling seems to be done by JFFs2 in a portable way, and that's as it should be.

You seem to be very confused here. JFFS2 works on flash directly; nothing's pretending to be a block device. It doesn't seem to be at all relevant to this discussion.

JFFS2 does its own wear levelling and flash management, because it works directly on the flash.

FAT can't do that -- it needs some other code (like the FTL code) to emulate a normal hard drive for it, providing wear levelling and logical<->physical translation for it.

See http://www.infradead.org/~dwmw2/mtd-upper-layers.jpeg

Wear levelling is not done in the driver -- the driver just drives the flash, and in fact is below the bottom of the diagram since it's largely irrelevant. It just gives you read/write/erase functions for the raw flash.

Wear levelling is done either in the file system which works directly on the flash (JFFS2, YAFFS), or in the 'translation layer' which uses the flash to pretend to be a block device (FTL, NFTL, INFTL, SMTL). (In the case of the extremely naïve 'mtdblock' translation layer, no translation and no wear levelling is done at all.)

Quoted 2 lines

If the leveling were in the driver I don't believe even FAT would work.

I think that by 'driver' you actually mean the 'translation layer' or the combination of translation layer and underlying hardware driver, in which case you would be incorrect to say that it wouldn't work. That _is_ how it works, portably.

Signature · 1 line
dwmw2
#69 lore raw
In reply to Daniel Egger
Quoted 1 line · In article <1059315015.10692.207.camel@sonja> you wrote:

This is normally done by the filesystem (e.g. JFFS2).

why is jffs2 so slow, if the cpu overhead can be totally neglected when writing to such slow media? I would asume a FS whic his optimized for not wearing out flash cards would reduce the IOs to the absolute minimum and therefore be fast be definition?

Greetings
Bernd

Signature · 2 lines
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/
#70 lore raw
Quoted 4 lines · Bernd Eckenfels

why is jffs2 so slow, if the cpu overhead can be totally neglected when writing to such slow media? I would asume a FS whic his optimized for not wearing out flash cards would reduce the IOs to the absolute minimum and therefore be fast be definition?

Flash cards are -slow-. Also jffs2 is mostly synchronous so it writes the long bit by bit. The flash wear is on erase not write. You could certainly teach jffs2 a bit more about batching writes. The other issue with jffs2 is startup because it is a log you have to read the entire log to know what state you are in

#71 lore raw
Quoted 5 lines · Alan Cox

Flash cards are -slow-. Also jffs2 is mostly synchronous so it writes the long bit by bit. The flash wear is on erase not write. You could certainly teach jffs2 a bit more about batching writes. The other issue with jffs2 is startup because it is a log you have to read the entire log to know what state you are in

Startup in 2.5 is a _lot_ better than in 2.4 -- we stopped checking the crc32 on every node during mount, and do it later instead. We also use a pointer directly into the flash if it's possible, rather than memcpying every node we look at during the mount.

The amount of state we need to rebuild during the mount isn't huge -- if you ignore nlink for the moment, all we really need to do is build up a list of

	{ physical address, length, inode # to which it belongs }

tuples for each log entry on the medium -- for larger media, we could add tailers to each eraseblock with a condensed version of that information, to prevent the need to scan the whole of each block during mount to work it out.

I suspect we're going to have to do that for the larger NAND devices, including DiskOnChip, in the fairly near future. It takes 30 seconds to mount a 144MiB DiskOnChip with JFFS2.

We already do some form of write batching on NAND flash too -- since we can't always write more than once to any given 512-byte 'page' on the flash, we have to have a write-back buffer and coalesce writes.

The other fairly simple thing we can do to improve runtime performance and device lifetime is start being more intelligent about garbage collection -- we should GC ancient and unchanged data to separate blocks on the flash, rather than mixing it in with new writes; then we will end up with more fully-clean eraseblocks (which can be ignored except for once in a blue moon when we decide to GC them for wear levelling purposes) and more mostly-dirty eraseblocks (on which we make rapid GC progress since not a lot needs to be copied before the block can be erased) -- and fewer of the 50%-dirty blocks we tend to see at the moment.

Signature · 1 line
dwmw2
#72 lore raw
In reply to Daniel Egger
Quoted 4 lines · Daniel Egger

Are you sure CF cards have wear leveling? I'm pretty confident that they have defect sector management but no wear leveling. There's a huge difference between those two.

I am told that they do by flx. After all, they are most used for the FAT filesystem.

Signature · 1 line
Hans
#73 lore raw
In reply to Hans Reiser
Quoted 1 line · Hans Reiser

it is suitable for any flash device that has wear leveling built into the hardware (e.g. all compact flash cards), or for which a wear leveling block device driver is used (I don't know if one exists for Linux).

I'm thinking about using ide flash disk for journal device - is it good idea? there are such disks with wear leveling available,
but can someone tell difference betwen dynamic and static wear leveling?

thanks

rafal

#74 lore raw
In reply to Yury Umanets
Quoted 7 lines · Yury Umanets
Quoted 3 lines

So basically we do have pretty powerful hardware with huge storage and memory and now need a FS which is fast and reliable even on flash memory. JFFS2 is nice but way too slow once one has bigger sizes.

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Most Linux filesystems can't be used properly with flash devices because of unability to handle write errors caused by flash wearing out. FS should mark the block as bad and relocate the data. Some devices report "read correctly, but had ECC" and when such happens data should also be relocated to not worn-out place and block marked as bad.

Signature · 1 line
Jussi Laako <[email protected]>
#75 lore raw
Quoted 15 lines · Jussi Laako
Quoted 10 lines · Yury Umanets
Quoted 5 lines

So basically we do have pretty powerful hardware with huge storage and memory and now need a FS which is fast and reliable even on flash memory. JFFS2 is nice but way too slow once one has bigger sizes.

I think this is more then enough for running reiser4. Reiser4 is a linux filesystem first of all, and linux is able to be ran on even worse hardware then you have.

Quoted 9 lines

Most Linux filesystems can't be used properly with flash devices because of unability to handle write errors caused by flash wearing out. FS should mark the block as bad and relocate the data. Some devices report "read correctly, but had ECC" and when such happens data should also be relocated to not worn-out place and block marked as bad.

I would be happy to accept a patch fixing that, or to fix it for a fee, or to fix it if we somehow get more funding from somewhere next year.;-)

Signature · 1 line
Hans
#76 lore raw
In reply to Jussi Laako
Quoted 3 lines · Jussi Laako

Most Linux filesystems can't be used properly with flash devices because of unability to handle write errors caused by flash wearing out. FS should mark the block as bad and relocate the data.

This is typically done by the pseudo-filesystem (FTL, NFTL, etc.) which is used to emulate a hard drive on flash storage; the 'real' file system itself doesn't need to do it for itself.

Signature · 1 line
dwmw2
#77 lore raw
In reply to Daniel Egger
Quoted 11 lines · Daniel Egger

Am Fre, 2003-07-25 um 15.02 schrieb Nikita Danilov:

Quoted 6 lines

No special measures are taken to level block allocation. Wandered blocks are allocated to improve packing i.e., place blocks of the same file close to each other. Actually, it tries to place tree nodes in the parent-first order.

Quoted 7 lines

So the new blocks are created as close as possible to the old blocks instead of say spreading them as far as possible. This is pretty bad for usage in the embedded world but I guess this is not the market you're aiming at. :(

I thought that close was fine, it was putting it in the same block that was the problem?

Again, I think this is best solved in the device layer.

Signature · 1 line
Hans
#78 lore raw

Am Son, 2003-07-27 um 14.59 schrieb Hans Reiser:

Quoted 2 lines

I thought that close was fine, it was putting it in the same block that was the problem?

This looks fine for normal harddrives put on flash you'd probably like to write the data evenly over the free space in some already formatted section still leaving the oportunity to format some other sectors to not run out of space.

Quoted 1 line

Again, I think this is best solved in the device layer.

A device layer that shuffles around sectors would have interesting semantics, like hardly being portable because one would have to use exactly the same device driver with the same parameters to use the filesystem and thus retrieve the data.

Signature · 2 lines
Servus,
       Daniel
#79 lore raw
Quoted 4 lines · In article <1059315409.10692.215.camel@sonja> you wrote:

A device layer that shuffles around sectors would have interesting semantics, like hardly being portable because one would have to use exactly the same device driver with the same parameters to use the filesystem and thus retrieve the data.

In fact it should not shuffle around, but support the filesystem in requesting new free blocks.

But I see that FS must support the flash by for example beeing prepared to move often used blocks (super blocks, bitmaps, ... ) around.

Greetings
Bernd

Signature · 2 lines
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/
#80 lore raw
Quoted 8 lines · Bernd Eckenfels
Quoted 4 lines · In article <1059315409.10692.215.camel@sonja> you wrote:

A device layer that shuffles around sectors would have interesting semantics, like hardly being portable because one would have to use exactly the same device driver with the same parameters to use the filesystem and thus retrieve the data.

In fact it should not shuffle around, but support the filesystem in requesting new free blocks.

In practice it _does_ shuffle around. It'll keep some kind of metadata somewhere logging which physical 512-byte 'sectors' on the medium contain data belonging to each logical 512-byte sector of the emulated block device. Each time a logical sector is overwritten, it'll just write it out elsewhere on the physical medium and adjust the metadata accordingly, and the original copy of that sector becomes obsolete.

When it (almost) runs out of 'elsewhere', it needs to garbage collect -- it'll pick an eraseblock which contains mostly obsolete data, copy the still-valid sectors into the remaining 'elsewhere' as if they'd been rewritten with the same data again, then erase the eraseblock which now _only_ contains obsolete sectors.

Quoted 2 lines

But I see that FS must support the flash by for example beeing prepared to move often used blocks (super blocks, bitmaps, ... ) around.

And by telling it which blocks no longer contain relevant data, so that the block 'translation layer' can discard them and stop copying them around the physical medium as described above...

Basically, if you're going to teach the filesystem about flash, you should teach it about flash properly and quit pretending to be a block device altogether. The artificial extra layer just begs you to violate the layering in _so_ many ways that you should just abolish it.

Signature · 1 line
dwmw2
#81 lore raw
In reply to Daniel Egger
Quoted 4 lines · Daniel Egger

Am Son, 2003-07-27 um 14.59 schrieb Hans Reiser:

Quoted 9 lines
Quoted 3 lines

I thought that close was fine, it was putting it in the same block that was the problem?

This looks fine for normal harddrives put on flash you'd probably like to write the data evenly over the free space in some already formatted section still leaving the oportunity to format some other sectors to not run out of space.

I was not able to parse the sentence above.;-)

Quoted 14 lines
Quoted 6 lines

Again, I think this is best solved in the device layer.

A device layer that shuffles around sectors would have interesting semantics, like hardly being portable because one would have to use exactly the same device driver with the same parameters to use the filesystem and thus retrieve the data.

No, you could be more clever than that.

Signature · 1 line
Hans
#82 lore raw

Am Mon, 2003-07-28 um 14.44 schrieb Hans Reiser:

Quoted 6 lines
Quoted 4 lines

This looks fine for normal harddrives put on flash you'd probably like to write the data evenly over the free space in some already formatted section still leaving the oportunity to format some other sectors to not run out of space.

I was not able to parse the sentence above.;-)

s/put/but/

As already mentioned the flash chips have to be erased before they can be written. The erasesize is much larger than the typical block size which means that although a block doesn't contain valid data it still contains something which means that it cannot be written until it was erased. That's why JFFS2 is using garbage collection to reclaim unused but (at the moment) unusable space.

Quoted 1 line

No, you could be more clever than that.

Sure. :)

Signature · 2 lines
Servus,
       Daniel
#83 lore raw
Quoted 28 lines · Daniel Egger

Am Mon, 2003-07-28 um 14.44 schrieb Hans Reiser:

Quoted 9 lines
Quoted 6 lines

This looks fine for normal harddrives put on flash you'd probably like to write the data evenly over the free space in some already formatted section still leaving the oportunity to format some other sectors to not run out of space.

Quoted 3 lines

I was not able to parse the sentence above.;-)

s/put/but/

As already mentioned the flash chips have to be erased before they can be written. The erasesize is much larger than the typical block size which means that although a block doesn't contain valid data it still contains something which means that it cannot be written until it was erased. That's why JFFS2 is using garbage collection to reclaim unused but (at the moment) unusable space.

Quoted 8 lines
Quoted 3 lines

No, you could be more clever than that.

Sure. :)

If you feel ambitious, try increasing the reiser4 node size to equal the erase size. This requires changes to VM though.

Signature · 1 line
Hans
#84 lore raw
In reply to Daniel Egger

So, in summary, reiser4 will do a good job of flushing in large batches using large bios, and that is most of what you can do to optimize for large erase size.

Things that could be added: improved compression for small files, garbage collection based freeing of unused blocks, increasing node size to equal erase size.

We can add such features for a fee, or you can code them yourself and send us a patch.

If I am wrong about compact flashes all having hardware wear leveling so that FAT can be used on them, then you can add wear leveling to the list of features desirable.

Signature · 1 line
Hans
#85 lore raw
In reply to Daniel Egger
Quoted 4 lines · Daniel Egger

Am Mit, 2003-07-23 um 23.02 schrieb Hans Reiser:

Quoted 8 lines
Quoted 3 lines

In brief, V4 is way faster than V3, and the wandering logs are indeed twice as fast as fixed location logs when performing writes in large batches.

How do the wandering logs compare to the "wandering" logs of the log structured filesystem JFFS2? Does this mean I can achieve an implicit wear leveling for flash memory?

Forgive me for answering your question with a question, but, wouldn't you want to do it at the block device layer? If no, then it would not be hard to code a block allocation plugin for it. Probably the main problem would be with the super block and bitmaps, which have fixed locations (and are written twice but we don't normally care because they are small and insignificant to performance.)

Quoted 10 lines
Quoted 9 lines

We are able to perform all filesystem operations fully atomically, while getting dramatic performance improvements. (Other attempts at introducing transactions into filesystems are said to have failed for performance reasons.)

Quoted 4 lines

How failsafe is it to switch off the power several times? When the filesystem really works atomically I should have either the old or the new version but no mixture.

It is safe.

Quoted 3 lines

Does it still need to fsck or is the
transaction replay done at mount time?

mount time.

Quoted 3 lines

In case one still needs fsck,
what's the probability of needing user interaction?

0, but an application can still write to two files, and if it does not use our atomic infrastructure (at this time none of them do;-) ), the two separate files will not be certain to be updated as one atom atomically.

Quoted 4 lines

How long does it
need to get a filesystem back into a consistent state after a powerloss (approx. per MB/GB)?

I don't have numbers, someone else will have to answer/measure it for you.

Quoted 8 lines

Background: I'm doing systems on compactflash cards and need a reliable filesystem for it. At the moment I'm using a compressed JFFS2 over the mtd emulation driver for block devices which works quite well but has a few catches...

Signature · 1 line
Hans
#86 lore raw
In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

Please look at http://www.namesys.com/benchmarks/v4marks.html

It says "but since most users use ext3 with only meta-data journaling" which isn't really correct. ext3's metadata-only journalling mode is writeback mode.

Most people in fact use ext3's ordered mode, which provides the same data consistency guarantees on recovery as data journalling.

Please compare against the ext3 in -mm. It has tweaks which aren't yet merged, but which will be submitted soon.

#87 lore raw
Quoted 11 lines · Andrew Morton
Quoted 3 lines · Hans Reiser

Please look at http://www.namesys.com/benchmarks/v4marks.html

It says "but since most users use ext3 with only meta-data journaling" which isn't really correct. ext3's metadata-only journalling mode is writeback mode.

Quoted 10 lines

Most people in fact use ext3's ordered mode, which provides the same data consistency guarantees on recovery as data journalling.

Please compare against the ext3 in -mm. It has tweaks which aren't yet merged, but which will be submitted soon.

We are going to run a bunch more benchmarks when I get back, probably doing things like turning on htrees and tail combining and stuff, in lots of different combinations. Ordered mode will be added, as well as making green have a uniform meaning for all the benchmarks;-). This benchmark was just what could be done before I got on a plane.

Signature · 1 line
Hans