Skip to contentSkip to conversation

the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion

DISCUSSION Hans Reiser 76 participants 353 messages Jul 21, 2006 – Aug 9, 2006 Page 1 of 2

lore
Share & tools
lore
#0 lore raw

Is http://wiki.kernelnewbies.org/WhyReiser4IsNotIn truly the "official" point of view as claimed by its author? An interesting method of expression for it. I heard about it from a user who suggested that I respond before it got slashdotted.

Let me ask that one compare and contrast the ext4 integration procedure outlined by Ted Tso, with the procedure experienced by other filesystems. The code isn't even written, benchmarked, or tested yet, and it is going into the kernel already so that its developers don't have to deal with maintaining patches separate from the tree. Wow. Kind of hard to argue that it is not politically differentiated, isn't it?

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations). They were kind enough to instruct me on what ideas I should take from XFS, and you know what, I listened. Reiser4's allocate on flush is the result of their kind instruction. I then took it a bit further, like a good student, and Reiser4 also has balance on flush, compress on flush, etc.

These guys wanted to port XFS to Linux, but there was a problem, which was that IRIX was better in some ways than Linux, and XFS depended on those advantages. Now I met them, and I have to tell you that it was pretty obvious that these guys knew what they were doing. Suggest that these guys needed supervision --- sorry, no way, we needed their supervision. What happened? They got hassled. Instead of learning from them, welcoming into our community two very senior developers who knew a lot more than any of us about the topics they chose to speak about, they got hassled, they get ignored, they felt rejected, and left the Linux community forever, never to return. XFS is still with us, but the loss of those two could only have been devastating for their
project. I think the whole kernel community suffered from their loss.

Linux has a problem, which is that with success it is attracting people with more skill than what it started with, and it is not doing a very good job of handling that. In fact, it downright stinks at it, behaving in the worst way it could choose for handling that. We have lost quite a number of FS developers who just don't want to deal with people who know less than they do but are obnoxious and disrespectful to submissions because they enjoy powertripping. We lost David Mazieres, for example, because he is very very bright, is one of DARPA's most promising security researchers, and he does not want to be bothered with Viro et al. so he develops for BSD instead. Linus, if you really want to prove that Linux welcomes talented people, go sweet talk Mazieres into giving Linux another try, you might succeed if you try. The odd thing is that Viro is not obnoxious at all in person. lkml suffers from email disease, and we need to make conscious efforts to reduce that.

Regarding distros accepting filesystems first, that is just completely backwards from what it ought to be. Linus, I respect you a lot, but I know this one is your idea, and some things I disagree with you on. Distros are marketed towards people who do not know how to tar and untar if an FS is dropped. A reasonable approach would be to say that any filesystem marked as experimental can be dropped at any time, so if you aren't able to tar and untar the partition it is on when a new kernel comes out, you should not use experimental filesystems. Then most distros will not make the experimental FS visible to users who don't press three buttons acknowledging that they were warned.... Linspire's view is pretty simple, they need to know that Reiser4 will be accepted BEFORE they make their distro depend on it. This is being responsible to their users. I could go ask Debian, etc., to include Reiser4, but it is the wrong way, so I am shy about it.

I am not saying that ext4 should not be accepted as an experimental FS, I don't even really believe that ext4 should only be accepted when it is higher performance than Reiser4, I am saying that the process should be the same for everyone. Reiser4 is the upgrade for ReiserFS V3, in which we fix all of V3's flaws without disturbing the mission critical servers using V3 by changing the V3 code underneath them. (Things like the bug affecting MythTV users on V3 at the moment just should not be happening. Experiments belong in V4, and I wish there was more respect for my views on this.). V4 contains bug fixes for several V3 bugs that are too deep to fix without deep rewrite, and since V3 does not have plugins, disk format changes should not get added to a stable branch. When submitted Reiser4 was more stable than V3 was when it was
accepted. (This is because we now have a much better test suite. I
would never submit code that I know has a bug unfixed. At the moment we can crash Reiser4 using our test suite, as some of the linux kernel inclusion related changes made recently were extensive, I hope we have that bug fixed by next week.)

We should develop a culture in which acceptance is more based on whose code measurably performs well than on who is friends with whom. We should not think that such a culture will develop without an effort being made to grow it.

Actually, if we just had a few more akpms to go around, things would be a lot better..... oh well. We need to recruit more people like him. You can't really have code reviewed by persons less experienced and proven than those being reviewed, it just doesn't work too well. Linux needs to look outward more, and welcome persons who have proven themselves in other arenas as though we were lucky to get their time. Because we are. Maybe when we don't have people with the expertise to review something, we should go outside the Linux community, like the way academic journals will solicit outside reviewers for particular articles as they need them. Our current attitude resembles that of BSD before it lost the market to Linux, I remember it well, there was a reason why I developed for Linux instead.

Avoiding the problems that some large corporations have with politics does not happen automatically as a result of it being free software, it requires as much effort as it does in the successful large corporations. Non-profits are in no way immune to being harmed by internal politics.

If it is true that Reiser4 is likely to go in for 2.6.19, this is good to hear, though an odd source to hear it from. If it is true, then I will skip lobbying distros to accept Reiser4 before the kernel does, because really it makes little sense for them to do so.

Hans

#1 lore raw

Losing Technologists [was: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Hi Hans,

Quoted 4 lines · On Friday July 21, [email protected] wrote:

Linux has a problem, which is that with success it is attracting people with more skill than what it started with, and it is not doing a very good job of handling that.

You seem to be saying that technological skills are the most important thing for Linux. I disagree. Linux is a success because of a great community as much (probably more) than because it has great technology.

And by "great community", I don't mean "completely harmonious" but rather "board and effective". Good community members follow the rule "Be liberal in what you accept, conservative in what you transmit". And the first if these is the more important.

Not everyone in the community is perfect at this, but it is clear to me that those who excel at this are what makes the community work.

Quoted 4 lines

We have lost quite
a number of FS developers who just don't want to deal with people who know less than they do but are obnoxious and disrespectful to submissions because they enjoy powertripping.

This just tells me that they wouldn't make good community members - they are not sufficiently liberal in what they accept.

It really isn't that hard to filter out the rubbish and look for the genuine technical content. For some people there isn't any of the later so you just ignore them. For others there is real technical content in what they say and you can respond to that while ignoring the rubbish.

Quoted 2 lines

lkml suffers from
email disease, and we need to make conscious efforts to reduce that.

Yes. But as this is a free and open community, we cannot exclude or censor people. The only emails that we can directly control are our
own. We should expect the general standard of the community to be
below the standard we present, and aim accordingly. I see this happening. It works. The community isn't perfect, but it could be a lot worse.

Quoted 3 lines

Actually, if we just had a few more akpms to go around, things would be a lot better..... oh well. We need to recruit more people like him.

Yes, and no.
Certainly a few more akpms would improve the technology, and would allow our current akpm to catch up with his workload.
But I don't think a few more of anything would dramatically change the community. There will always be people who attract flames, and others who are too willing to provide them.
New members to the community really need to sit back for a while and figure out who is who. Who is worth listening to. Who is best ignored (except for entertainment value). Who to ask questions of, etc.
It really is worth the effort.

There is plenty of historical evidence that the best technology doesn't always win out - there are plenty of other factors to success. So don't be too concerned if we lose a few brilliant technologists - it isn't the end of the world (domination).

NeilBrown

#2 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

Is http://wiki.kernelnewbies.org/WhyReiser4IsNotIn truly the "official" point of view as claimed by its author? An interesting method of expression for it. I heard about it from a user who suggested that I respond before it got slashdotted.

After two users asked the "Why is Reiser4 still not included?" question within a very short amount of time on this list (mailing lists aren't write-only, and this issue has been discussed often before...), Diego wrote an FAQ for this issue.

Emails by people like Linus, Andrew, Christoph or Al are what comes nearest to an official statement.

If there are factual errors or things you consider offensive in the FAQ, please try to discuss them with Diego privately first. But changing contents of this FAQ wouldn't change anything regarding the Reiser4 inclusion - the FAQ is only a high level description of the Reiser4 situation for end users.

Quoted 6 lines

Let me ask that one compare and contrast the ext4 integration procedure outlined by Ted Tso, with the procedure experienced by other filesystems. The code isn't even written, benchmarked, or tested yet, and it is going into the kernel already so that its developers don't have to deal with maintaining patches separate from the tree. Wow. Kind of hard to argue that it is not politically differentiated, isn't it?

The main difference seems to be that ext4 is being developed by people that have already shown and are trusted to develop a filesystem that follows the Linux way in all respects.

Note that I didn't say "right way" but "Linux way".

No matter whether it's coding style or the discussion which functionality belongs to the VFS level, the Linux kernel has it's (often unwritten) rules - but the same is true with other rules for any other operating system.

Quoted 5 lines

...
Actually, if we just had a few more akpms to go around, things would be a lot better..... oh well. We need to recruit more people like him. You can't really have code reviewed by persons less experienced and proven than those being reviewed, it just doesn't work too well. Linux

Quoted 8 lines

needs to look outward more, and welcome persons who have proven themselves in other arenas as though we were lucky to get their time. Because we are. Maybe when we don't have people with the expertise to review something, we should go outside the Linux community, like the way academic journals will solicit outside reviewers for particular articles as they need them. Our current attitude resembles that of BSD before it lost the market to Linux, I remember it well, there was a reason why I developed for Linux instead.

A very important part of a review is whether it follows the "Linux way" that might be quite different from what someone who comes from outwards has to learn before he can start doing a review.

Finding people for the cool stuff like developing a new filesystem is relatively easy compared to finding people why are both capable and willing to do the boring work of reviewing other people's code.

So we'd need people who are already acknowleged experts in the area, who are willing to learn the Linux way, and who will then do the not-fun work of reviewing other people's code.

How should this work?
Someone has to offer well-paid jobs for such people?

Quoted 5 lines

Avoiding the problems that some large corporations have with politics does not happen automatically as a result of it being free software, it requires as much effort as it does in the successful large corporations. Non-profits are in no way immune to being harmed by internal politics.

In my experience, the Linux kernel is a big open source project with a working structure.

The Linux kernel sometimes looses developers.

That seems to unavoidable, there are there are always problems like:
- Some developers leave the projects if their code was rejected because it didn't match the standards and policies of the project.
- Some developers leave the project if other people's code that didn't match the standards and policies of the project was accepted.

And it's not as if open source projects had in any respect better prerequisites than large corporations - open source lacks the "it's our job" glue forcing people to work together in large corporations.

Quoted 2 lines

...
Hans

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#4 lore raw
In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

Let me ask that one compare and contrast the ext4 integration procedure outlined by Ted Tso

"integration procedure" is hardly an accurate description, rather it is a development procedure that was developed after discussion and consensus building across LKML and the ext2/3/4 development team. It was not the original plan put forth by the ext2 developers, but after listening to the concerns and suggestions, we did not question the motives of the people making suggestions; we listened.

Quoted 1 line

The code isn't even written, benchmarked, or tested yet,

Actually, the first bits that we plan to merge have already been written and in use by hundreds of clusterfs customers, posted to LKML for comments (and we don't attack our reviewers, we thank them for their comments), and in fact they were written about at last year's OLS complete with benchmarks and graphs.
(http://ext2.sourceforge.net/2005-ols/2005-ols-ext3.html)

Quoted 4 lines

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations).

I believe you are referring to Jim Mostek and Steve Lord, and yes, they were very talented developers and engineers. I very much enjoyed talking to them at various filesystem and Linux conferences and workshops.

Quoted 5 lines

supervision. What happened? They got hassled. Instead of learning from them, welcoming into our community two very senior developers who knew a lot more than any of us about the topics they chose to speak about, they got hassled, they get ignored, they felt rejected, and left the Linux community forever, never to return.

That's hardly what happened. SGI went through layoffs, and they were hit. See: http://slashdot.org/articles/01/05/26/0743254.shtml

Quoted 8 lines

A reasonable approach would be to say that any
filesystem marked as experimental can be dropped at any time, so if you aren't able to tar and untar the partition it is on when a new kernel comes out, you should not use experimental filesystems. Then most distros will not make the experimental FS visible to users who don't press three buttons acknowledging that they were warned.... Linspire's view is pretty simple, they need to know that Reiser4 will be accepted BEFORE they make their distro depend on it.

You do realize these two statements are completely contradictory, don't you? If an experimental filesystem can be dropped at any time, then Linspire can't depend on it from the point of view of supporting their users. If there is a huge user base, then someone is going to have to maintain it, even if the developer community has moved on to supporting the next new exciting filesystem thing. Hence, it is critical that the resulting filesystem be fully maintainable before it is integrated. To put it in your words, it wouldn't be responsible to the user base to do otherwise.

- Ted

#5 lore raw
Quoted 11 lines · Theodore Tso
Quoted 4 lines · Hans Reiser

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations).

I believe you are referring to Jim Mostek and Steve Lord, and yes, they were very talented developers and engineers. I very much enjoyed talking to them at various filesystem and Linux conferences and workshops.

Quoted 8 lines
Quoted 5 lines

supervision. What happened? They got hassled. Instead of learning from them, welcoming into our community two very senior developers who knew a lot more than any of us about the topics they chose to speak about, they got hassled, they get ignored, they felt rejected, and left the Linux community forever, never to return.

That's hardly what happened. SGI went through layoffs, and they were hit. See: http://slashdot.org/articles/01/05/26/0743254.shtml

Jim & Steve were never laid off from SGI. SGI mgmt -was- smarter than that ;) Neither account above is 100% correct, but I won't speak further on behalf of these gentlemen...

-Eric

#3 lore raw
In reply to Adrian Bunk
Quoted 7 lines · Adrian Bunk

After two users asked the "Why is Reiser4 still not included?" question within a very short amount of time on this list (mailing lists aren't write-only, and this issue has been discussed often before...), Diego wrote an FAQ for this issue.

Emails by people like Linus, Andrew, Christoph or Al are what comes nearest to an official statement.

[I'm the guy who wrote the doc]

I didn't write "It could possibly be ready as soon as 2.6.19" literally, that was someone that reworked my (ugly) english. Being fair what make me wrote that pages was the huge amount of people FUDing about linux developers in online forums (including this list)

Quoted 2 lines
Quoted 2 lines

You can't really have code reviewed by persons less experienced and proven than those being reviewed, it just doesn't work too well. Linux

Hans, stop that "we're smarter than you" attitude. It's not surprising that reiser 4 creates so many flames and that it's getting so hard to make progress, with such strong arguments. As far as I can tell, most of the kernel hackers trust Al Viro and hch on their opinions, specially in fs-related issues - and for good reasons. They know linux and they know how a linux fs must behave. As long as a fs plays well with the rest of the system and the code doesn't suck, I doubt they care too much if it's very advanced or arcane, and the huge variety of filesystem available in linux confirms that.

Quoted 3 lines
Quoted 3 lines

as they need them. Our current attitude resembles that of BSD before it lost the market to Linux, I remember it well, there was a reason why I developed for Linux instead.

Hans, I don't agree. If anything, the problem is that right now there's not a "development" stage: people just takes more care about what goes in, it wouldn't happen the same under a development stage. That certainly could make the job of big projects like reiser 4 much harder.

#6 lore raw
In reply to Theodore Tso
Quoted 3 lines · Theodore Tso

Actually, the first bits

yes, the first bits.... other people send in completed filesystems....

Quoted 2 lines

that we plan to merge

I don't actually think that your merge approach is the wrong one, I think that it being exclusive to you is what is wrong.

Quoted 12 lines
Quoted 9 lines

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations).

I believe you are referring to Jim Mostek

Ah, Jim Mostek and Jim Gray. (Steve Lord was not a senior guy back then, and he is still with SGI last I heard.... I actually don't know Steve very well, hmm, maybe some future conference....) Thanks.

Quoted 4 lines

That's hardly what happened. SGI went through layoffs, and they were hit. See: http://slashdot.org/articles/01/05/26/0743254.shtml

As the other poster mentioned, they went off to startups, and did not become part of our community. How much of that was because their contributions were more hassled than welcomed, I cannot say with certainty, I can only say that they were discouraged by the difficulty of getting their stuff in, and this was not as it should have been. They were more knowledgeable than we were on the topics they spoke on, and this was not recognized and acknowledged.

Outsiders are not respected by the kernel community. This means we miss a lot.

Quoted 13 lines
Quoted 12 lines

A reasonable approach would be to say that any
filesystem marked as experimental can be dropped at any time, so if you aren't able to tar and untar the partition it is on when a new kernel comes out, you should not use experimental filesystems. Then most distros will not make the experimental FS visible to users who don't press three buttons acknowledging that they were warned.... Linspire's view is pretty simple, they need to know that Reiser4 will be accepted BEFORE they make their distro depend on it.

Quoted 3 lines

You do realize these two statements are completely contradictory, don't you?

No, because distros would wait until it is not experimental before giving it to their users by default, in my proposed release model. lkml is populated with people FAR more suited to experimenting with experimental filesystems than typical distro customer lists are. It is commercial and political reasons that motivate distros being the first with patches not tried yet by lkml, not the interests of the users.

Now, for other patches these commercial and political reasons may need to be catered to as the price of getting the Redhats of the world to fund kernel development, but that logic does not apply to Reiser4's particulars.

Hans

#7 lore raw
Quoted 5 lines · Hans Reiser
Quoted 2 lines · Theodore Tso

Actually, the first bits

yes, the first bits.... other people send in completed filesystems....

Completed filesystems have a much higher barrier to entry, because they require a fresh review.

ext4 will go upstream MUCH faster, because it follows the standard process of Linux evolution, building on top of existing code with progressive changes:

	cp -a ext3 ext4
	update ext4
	update ext4
	update ext4
	...

This process builds upon existing reviews and knowledge of existing code. This process also guarantees a higher degree of stability during development, because the interim changes must always form a complete, working, usable filesystem.

Quoted 9 lines

As the other poster mentioned, they went off to startups, and did not
become part of our community. How much of that was because their
contributions were more hassled than welcomed, I cannot say with
certainty, I can only say that they were discouraged by the difficulty
of getting their stuff in, and this was not as it should have been. They were more knowledgeable than we were on the topics they spoke on, and this was not recognized and acknowledged.

Outsiders are not respected by the kernel community. This means we miss a lot.

Anyone who fails to respect the kernel development process, the process of building consensus, is turn not respected, flamed, and/or ignored.

If you don't respect us, why should we respect you?

Quoted 2 lines

No, because distros would wait until it is not experimental before giving it to their users by default, in my proposed release model. lkml

Distros follow their own release model, and don't have a care about what Hans Reiser thinks they should do.

<vendor hat on>
Red Hat has a pipeline in place for offering new technologies to users:
Fedora Core -> RHEL, and sometimes RHEL technology previews. SuSE
presumably does something similar with OpenSUSE.
</vendor hat>

There is PLENTY of opportunity to be experimental.

Quoted 9 lines

is populated with people FAR more suited to experimenting with experimental filesystems than typical distro customer lists are. It is commercial and political reasons that motivate distros being the first with patches not tried yet by lkml, not the interests of the users.

Now, for other patches these commercial and political reasons may need to be catered to as the price of getting the Redhats of the world to fund kernel development, but that logic does not apply to Reiser4's particulars.

I always feel sad to hear technologists wail about politics.

In my experience, the cause of such is almost always the fault of the submittor, ignoring consensus. But once the submittor has decided that "politics" are cause of their troubles, the submittor focuses on that rather than addressing the technology objections that were raised.

With you in particular, you demonstrated NO interest in maintaining reiser3, once reiser4 began to make a splash. Linux kernel code exists for DECADES, and as such, long term maintenance is a CRITICAL aspect of development.

Regardless of whatever new whiz-bang technology exists in reiser4, there is a very real worry that you will abandon reiser4 once its in the tree for a few years, just like what happened with reiser3.

Jeff

#262 lore raw
In reply to Theodore Tso

Hi,

Quoted 8 lines · Theodore Tso
Quoted 1 line

The code isn't even written, benchmarked, or tested yet,

Actually, the first bits that we plan to merge have already been written and in use by hundreds of clusterfs customers, posted to LKML for comments (and we don't attack our reviewers, we thank them for their comments), and in fact they were written about at last year's OLS complete with benchmarks and graphs.
(http://ext2.sourceforge.net/2005-ols/2005-ols-ext3.html)

However I would estimate that Reiser4 is used by more people than yet aother ext2 patchups.

Yours,

Signature · 3 lines
René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany)
            http://exactcode.de | http://t2-project.org | http://rebe.name
            +49 (0)30 / 255 897 45
#8 lore raw
In reply to Jeff Garzik

Jeff, I think that a large part of what is going on is that any patch that can be read in 15 minutes gets reviewed immediately, and any patch that is worked on for 5 years and then takes a week to read gets neglected. This is true even if line for line the 1 week to read patch
is more valuable. What is more is that people know this is
irrational, but aren't able to cure it in themselves. Even I have a problem of paying too much attention to endless 5 minute emails when I know I should instead, say, read the compression plugin from beginning to end.

There is nothing about small patches that makes them better code. There is no reason we should favor them, if the developers are willing to work on something for 5 years to escape a local optimum, that is often the RIGHT thing to do.

It is importand that we embrace our diversity, and be happy for the strength it gives us. Some of us are good at small patches that evolve, and some are good at escaping local optimums. We all have value, both trees and grass have their place in the world.

Quoted 6 lines

With you in particular, you demonstrated NO interest in maintaining reiser3, once reiser4 began to make a splash. Linux kernel code exists for DECADES, and as such, long term maintenance is a CRITICAL aspect of development.

You are rejecting the development model which is based on stable branches getting only bugfixes. V3 is a stable branch. It just had a feature added to it which added a bug that MythTV users are hitting. Some of them are responding to it by walking away from Reiser3, and no doubt muttering about what an unstable pile of shit our code is. On monday one of my guys is stopping work on V4 to send in a bug fix for a feature that should have gone into V4 first, and then maybe gotten backported after it was proven in V4.

So, given that Jeff and Chris can often be gotten to fix bugs, do I ask them to do it whenever there is a bug to fix and they will fix it? Oh yes! The despiriting thing though is that there is usually another reason to let them fix it, which is that almost all v3 bugs are in features they have added to what ought to have been a stable branch, and since it is their code, they should be the ones to fix it. We might, maybe, get one bug report a year in code written by Namesys before I announced code freeze on V3.

I just got an email from the programmer who wrote the MythTV bug saying that he is just too busy to bother fixing the bug in his code..... so my response is that a Namesys programmer is going to fix it on Monday.

All this talk about how you guys worry that code is going to be abandoned, you know, try policing the kids in their 20's who do it, not those who have been working since 1984 on developing the thing you somehow are worried they will abandon. I am not 20 something anymore, I am getting fat no matter how much I exercise, and I stick with things, and I only wish some things didn't stick so much with my middle....

Quoted 4 lines

Regardless of whatever new whiz-bang technology exists in reiser4, there is a very real worry that you will abandon reiser4 once its in the tree for a few years, just like what happened with reiser3.

And look at how Linus abandoned 2.4! Users of 2.4 needed so many features that were put into 2.6 instead, and they were just abandoned and neglected and.... Do you think he will abandon 2.6.18 also?

The stable branch of code getting only bugfixes and the development branch getting all the new features model of development is something most release management professionals agree is the right way to do things. I worked with release management teams some, and I have to say that the dominant paradigm in the software industry is, in this case, the best one yet.

Of course, I want to make it a little better, you know how I am, and as I was just discussing on the reiserfs-list, with plugins we can now move to a model in which if you mount reiser4 using the -o reiser4.1-beta mount option, it changes what the default plugin is, and that is how we do releases, we put our beta code in different plugins, and let the user choose whether to upgrade to a new release by just choosing what plugins to use as his default. Now that we paid the 5 year development price tag to get everything as plugins, we can now upgrade in littler pieces than any other FS. Hmm, I need a buzz phrase, its not extreme programming, maybe "moderate programming". Does that sound exciting to others.;-) Seriously though, I am curious to see whether plugin based release management works out as pleasantly for users as I am hoping it will.

Hans

#9 lore raw
Quoted 3 lines · Hans Reiser

I just got an email from the programmer who wrote the MythTV bug saying that he is just too busy to bother fixing the bug in his code..... so my response is that a Namesys programmer is going to fix it on Monday.

The way you wrote this, makes it sound like a userspace issue, and _not_ a problem with reiserfs.

Quoted 3 lines

And look at how Linus abandoned 2.4! Users of 2.4 needed so many features that were put into 2.6 instead, and they were just abandoned and neglected and.... Do you think he will abandon 2.6.18 also?

Not entirely true, he did not abandon the 2.4 kernel branch, he passed on maintainership to Marcelo. Similar to how he passed the torch on the 2.2 kernel branch to Alan Cox. Also on a side note, many new features ( and a ton of bug fixes !! ) were added to the 2.4 series _after_ Linus started working on the 2.5 branch.

#242 lore raw
In reply to Hans Reiser
Quoted 1 line · Hans Reiser

There is nothing about small patches that makes them better code. There

Erm, a small patch is something which should _obviously_ fix one issue. A small patch, containing at max some 100 lines, can easily be read and understood.

A complete filesystem (I'm co-maintaining one for an ancient on-disk format, too) isn't really easy to understand or to verify from looking at it for 5min.

Quoted 3 lines

is no reason we should favor them, if the developers are willing to work on something for 5 years to escape a local optimum, that is often the RIGHT thing to do.

I give a shit of nothing to some 5 year work if I cannot verify that it won't hurt me at some point.

Quoted 4 lines

It is importand that we embrace our diversity, and be happy for the strength it gives us. Some of us are good at small patches that evolve, and some are good at escaping local optimums. We all have value, both trees and grass have their place in the world.

Just put reiser4 in some GIT tree and publish it. Maybe you can place it on git.kernel.org .

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                     ...und wenn Du denkst, es geht nicht mehr,
 the second  :                            kommt irgendwo ein Lichtlein her.
#243 lore raw
Quoted 10 lines · Jan-Benedict Glaw
Quoted 1 line · Hans Reiser

There is nothing about small patches that makes them better code. There

Erm, a small patch is something which should _obviously_ fix one issue. A small patch, containing at max some 100 lines, can easily be read and understood.

A complete filesystem (I'm co-maintaining one for an ancient on-disk format, too) isn't really easy to understand or to verify from looking at it for 5min.

Nonetheless, "There is nothing about small patches that makes them better code".

Hans is quite right. Long patches just take longer to read. This can make them harder to penetrate review, as he describes, with analogy.

Quoted 7 lines
Quoted 4 lines

is no reason we should favor them, if the developers are willing to work on something for 5 years to escape a local optimum, that is often the RIGHT thing to do.

I give a shit of nothing to some 5 year work if I cannot verify that it won't hurt me at some point.

Do you really review all patches to ensure this? It is well understood that only once r4 reaches mainline will it get the wider testing it must have to shake down.

Lucky Namesys is not deterred by ingratitude or there would be no "5 year work" for us to contemplate at all.

Quoted 8 lines
Quoted 5 lines

It is importand that we embrace our diversity, and be happy for the strength it gives us. Some of us are good at small patches that evolve, and some are good at escaping local optimums. We all have value, both trees and grass have their place in the world.

Just put reiser4 in some GIT tree and publish it. Maybe you can place it on git.kernel.org .

Why should Hans give up the aspiration to have r4 in mainline due to a small number of regressive personalities (a.k.a. politics)?

To much of the Linux world R3 has been an extremely valuable contribution; r4 promises to be even more so.

--T

Quoted 10 lines

MfG, JBG

Signature · 6 lines
       Jan-Benedict Glaw       [email protected]                 
+49-172-7608481
 Signature of:                     ...und wenn Du denkst, es geht  
nicht mehr,
 the second  :                            kommt irgendwo ein  
Lichtlein her.
#244 lore raw
In reply to Hans Reiser

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Hans Reiser

I just got an email from the programmer who wrote the MythTV bug saying that he is just too busy to bother fixing the bug in his code..... so my response is that a Namesys programmer is going to fix it on Monday.

Hans -

I'll accept blame when it's my bug, but the MythTV one isn't. I've been working with the bitmap code and did the analysis to track down what was happening, but that doesn't make it a bug in my code.

That particular bug isn't in the bitmap scanning code, it's a side effect of the write batching higher up. It's looking for a window of 32 blocks, and there's just no window that large available. It ends up scanning all the bitmaps looking for the window, and then backs off to single block allocations. The scanning code works fine, and it does skip where there aren't enough free blocks available in a particular bitmap.

It's a pathological case when the file system is seriously fragmented. A quick fix would be to set a flag indicating that future writes shouldn't bother trying to find a window that large, but that's a hack. A better allocation algorithm would keep track of free space extents in memory, subject to getting dropped by memory pressure. Since that information would be separate from the bitmaps themselves, we could get rid of that nasty "is this block free, but in the journal?" check that we need to do as well. It's invasive, and a quicker fix would just be to track the largest window, and rescan when it gets used or a block in that bitmap gets freed.

That said, I actually did start work on a fix for this one, but I really just don't have the time right now.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFEw+BBLPWxlyuTD7IRAtG8AKCOWW/AH3NAen6gd6BToJGVfzdnNACfYkVS j2/6yAAeWKAhs4ng9fdGW0Y=
=gB+v
-----END PGP SIGNATURE-----

#245 lore raw
Quoted 4 lines · Jeff Mahoney

That particular bug isn't in the bitmap scanning code, it's a side effect of the write batching higher up.

Did you write the code that looks for a window of 32 blocks? If not, and if this code has been around for a long time, I
apologize. I thought you did write it and added it in recent months.

Quoted 3 lines

It's a pathological case when the file system is seriously fragmented.

Most bugs are pathological cases.;-)

Quoted 3 lines

A
quick fix would be to set a flag indicating that future writes shouldn't bother trying to find a window that large,

There are lots of quick fixes. 1) The quickest is to not scan for the window at all. 2) The second quickest is to limit the number of bitmaps that will be scanned to some number like 3. 3) The not at all quickest is to track free extents like XFS does, which is not a hack, but it belongs in a development branch. I am not sure it is worth the complexity, but my mind is not closed.

On monday we will do 1) or 2), probably 1). After the repacker is
done, we should review all our block allocation algorithms. I have an idea for how to do things more optimally for streaming media that will avoid fragmentation over time, and when combined with the repacker may make 3 not worthwhile.

I am grateful that you and Chris do bug fixes, but when you guys are too busy, (and that can and will happen to any of us), the baton needs to get passed. V3 needs to be a zero defect product, and once we know it is a bug I don't want bugs in V3 to remain unfixed for more than a day
plus the time it takes to fix it. If you do add code, I want any bugs
that show up in the aftermath of mainstream merging to get jumped on.

Hans

#247 lore raw
In reply to Jeff Mahoney
Quoted 11 lines · Jeff Mahoney

Anyone up for it? :) There are changes I'd like to see in reiser3, particularly ones that address the severe problems observed in David Chinner's high bandwidth file system talk this year at OLS. Specifically, it ended up making very little progress and spending the majority of the time in the journal when the workload is streaming data at the disk at a very high rate on a very large file system. Yes, that is certainly XFS's sweet spot, but barely making progress at all is a bit more severe than "poor performance." Perhaps mkreiserfs should be a bit saner about choosing

Quoted 8 lines

journal sizes, since a 32 MB journal is not a good fit for all cases. Also,
I'd like to see the usage of the BKL gone as it severely limits performance
when more than one thread is writing to the file system, or even another reiserfs file system. It's not entirely low hanging fruit since the nested cases need to be audited, but it shouldn't be too hard to eliminate the inter-filesystem lock contention by replacing the BKL with a per-sb mutex.

Getting rid of the BKL is a huge task that was done in V4 for a reason. You are talking about 6+ man-months, and years of shake-out to fully debug. Actually, it is a tribute to Zam's skill that V4's locking got debugged so fast: I gave him the task knowing it was going to be the hardest code to debug, and he did it very well.

These things you discuss, except for the journal size, are not things to fix in a stable branch.

My apologies that I thought this was a new bug. Let us be glad that a user gave us enough detail we saw it.

Quoted 5 lines

I have some more things, but I have nowhere near the time to do them, and other file systems will perform fine.

#246 lore raw
In reply to Hans Reiser

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Hans Reiser
Quoted 3 lines · Jeff Mahoney

That particular bug isn't in the bitmap scanning code, it's a side effect of the write batching higher up.

Did you write the code that looks for a window of 32 blocks? If not, and if this code has been around for a long time, I
apologize. I thought you did write it and added it in recent months.

Nope. The scan_bitmap_block() code that looks for windows was added in a changeset from a bk merge against 2.5.33 in September 2002. The change to want a minimum size window was added in June 2004 to 2.6.8-rc2. That patch is actually credited to both Mason and I, but I don't recall who wrote that bit. It may well have been my code, after all, but it's certainly not a new bug. *shrug* I guess MythTV might just be generating an i/o pattern that hadn't been seen before.

Quoted 11 lines
Quoted 3 lines

A
quick fix would be to set a flag indicating that future writes shouldn't bother trying to find a window that large,

There are lots of quick fixes. 1) The quickest is to not scan for the window at all. 2) The second quickest is to limit the number of bitmaps that will be scanned to some number like 3. 3) The not at all quickest is to track free extents like XFS does, which is not a hack, but it belongs in a development branch. I am not sure it is worth the complexity, but my mind is not closed.

Quoted 5 lines

On monday we will do 1) or 2), probably 1). After the repacker is
done, we should review all our block allocation algorithms. I have an idea for how to do things more optimally for streaming media that will avoid fragmentation over time, and when combined with the repacker may make 3 not worthwhile.

If you want to go the 1) route, it's trivial. See patch below. It will restore the one-block-at-a-time behavior.

Quoted 6 lines

I am grateful that you and Chris do bug fixes, but when you guys are too busy, (and that can and will happen to any of us), the baton needs to get passed. V3 needs to be a zero defect product, and once we know it is a bug I don't want bugs in V3 to remain unfixed for more than a day
plus the time it takes to fix it. If you do add code, I want any bugs
that show up in the aftermath of mainstream merging to get jumped on.

Anyone up for it? :) There are changes I'd like to see in reiser3, particularly ones that address the severe problems observed in David Chinner's high bandwidth file system talk this year at OLS. Specifically, it ended up making very little progress and spending the majority of the time in the journal when the workload is streaming data at the disk at a very high rate on a very large file system. Yes, that is certainly XFS's sweet spot, but barely making progress at all is a bit more severe than "poor performance." Perhaps mkreiserfs should be a bit saner about choosing journal sizes, since a 32 MB journal is not a good fit for all cases. Also, I'd like to see the usage of the BKL gone as it severely limits performance when more than one thread is writing to the file system, or even another reiserfs file system. It's not entirely low hanging fruit since the nested cases need to be audited, but it shouldn't be too hard to eliminate the inter-filesystem lock contention by replacing the BKL with a per-sb mutex. I have some more things, but I have nowhere near the time to do them, and other file systems will perform fine.

- -Jeff

Patch:

- --- linux-2.6.17.orig/fs/reiserfs/bitmap.c 2006-01-02 22:21:10.000000000 -0500
+++ linux-2.6.17.orig.devel/fs/reiserfs/bitmap.c 2006-07-23 19:10:57.000000000 -0400
@@ -1020,7 +1020,6 @@
b_blocknr_t finish = SB_BLOCK_COUNT(s) - 1; int passno = 0;
int nr_allocated = 0;
- - int bigalloc = 0;

determine_prealloc_size(hint);
if (!hint->formatted_node) {
@@ -1047,28 +1046,9 @@
hint->preallocate = hint->prealloc_size = 0;
}
/* for unformatted nodes, force large allocations */
- - bigalloc = amount_needed;
}

do {
- - /* in bigalloc mode, nr_allocated should stay zero until
- - * the entire allocation is filled
- - */
- - if (unlikely(bigalloc && nr_allocated)) {
- - reiserfs_warning(s, "bigalloc is %d, nr_allocated %d\n",
- - bigalloc, nr_allocated);
- - /* reset things to a sane value */
- - bigalloc = amount_needed - nr_allocated;
- - }
- - /*
- - * try pass 0 and pass 1 looking for a nice big
- - * contiguous allocation. Then reset and look
- - * for anything you can find.
- - */
- - if (passno == 2 && bigalloc) {
- - passno = 0;
- - bigalloc = 0;
- - }
switch (passno++) {
case 0: /* Search from hint->search_start to end of disk */
start = hint->search_start;
@@ -1106,8 +1086,7 @@
new_blocknrs + nr_allocated, start, finish,
- - bigalloc ?
- - bigalloc : 1,
+ 1,
amount_needed - nr_allocated, hint->

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFExATJLPWxlyuTD7IRAmeKAJsFI/awPPAXpB2DI+kO19EZtr3tRwCfWduO Re+5kXNtj6St/LuUy9lbNm4=
=anQd
-----END PGP SIGNATURE-----

#318 lore raw
In reply to Theodore Tso
Quoted 6 lines · Theodore Tso
Quoted 4 lines · Hans Reiser

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations).

Not sure who Jim Grey was, he never worked on XFS, ah well.

Quoted 6 lines

I believe you are referring to Jim Mostek and Steve Lord, and yes, they were very talented developers and engineers. I very much enjoyed talking to them at various filesystem and Linux conferences and workshops.

Quoted 9 lines
Quoted 5 lines

supervision. What happened? They got hassled. Instead of learning from them, welcoming into our community two very senior developers who knew a lot more than any of us about the topics they chose to speak about, they got hassled, they get ignored, they felt rejected, and left the Linux community forever, never to return.

That's hardly what happened. SGI went through layoffs, and they were hit. See: http://slashdot.org/articles/01/05/26/0743254.shtml

Ted, you of all people should know not believe all you read on slashdot ;-) 'Linuxcare helping out with the funding' ha!

Both Jim Mostek and I left under our own steam at different times, Jim in 2000 and myself in 2003. SGI still has great technology to work on and, but the you can only take so many years of bad financial results and watching people get layed off.

I still work on Linux, and follow development as much as I can. I keep trying to get back to OLS, but circumstances keep conspiring against me, maybe next year.

Steve Lord

#10 lore raw
In reply to Matt Heler
Quoted 8 lines · Matt Heler

On Sunday 23 July 2006 12:20 am, Hans Reiser wrote:

The way you wrote this, makes it sound like a userspace issue, and _not_ a problem with reiserfs.

It was a problem with reiserfs. Code was added to search for the perfect spot to fit a file. If there is no perfect spot, it searches every bitmap for that spot before giving up. However, Jeff kindly gave us a little patch to fix this and made the whole issue moot. It also seems I was in error, and we actually have had this problem since 2002. Now some past remarks from users about fragmentation make more sense. What can I say, since I have no MP3s I never get anywhere near full on my personal hard drive.

Quoted 8 lines
Quoted 7 lines

And look at how Linus abandoned 2.4! Users of 2.4 needed so many features that were put into 2.6 instead, and they were just abandoned and neglected and.... Do you think he will abandon 2.6.18 also?

Quoted 7 lines

Not entirely true, he did not abandon the 2.4 kernel branch, he passed on maintainership to Marcelo. Similar to how he passed the torch on the 2.2 kernel branch to Alan Cox. Also on a side note, many new features ( and a ton of bug fixes !! ) were added to the 2.4 series _after_ Linus started working on the 2.5 branch.

You missed the sarcasm in my voice, my apologies, it is the trouble I have with email.

Just to balance everything with some nuance, let me add that when a development branch is first opened, there is usually a bit of gray as to whether particular small features should go into the development branch or the stable branch. As the stable branch gets more stable the incentive to not destabilize it increases, and as a development branch becomes usable, the delay to users due to putting features only there reduces.

I want reiserfs to be the filesystem that professional system administrators view as the one with both the fastest technological pace, and the most conservative release management.

I apologize to users that the technology required a 5 year gap between
releases. It just did, an outsider may not realize how deep the
changes we made were. Things like per node locking based on a whole new approach to tree locking that goes bottom up instead of the usual top
down are big tasks. Dancing trees are a big change, getting rid of
blobs is a big change, wandering logs..... We did a lot of things like that, and got very fortunate with them. If we had tried to add such changes to V3, the code would have been unstable the whole 5 years, and would not have come out right.

Experienced writers know that often, if you want to fix a passage, even a passage that is quite good in some parts, sometimes it is better to write the whole passage again without looking at the text of the first draft of the old passage, because sometimes your muse just needs the freedom, and without the freedom the awkwardness of the old passage is incurable. Probably there is some very sophisticated neurological reason why that is. Code can be the same. Sometimes. I knew that reiser4 HAD to be written from scratch without reference to the old code if it was to come out right.

If I cannot be a great artist, at least I can try to have the temperament of one, yes? :-)

I sincerely hope that using mount options to select default plugins, and making development code go into new plugins means that releases after this can be roughly quarterly, and that we can start doing a whole bunch of quick little plugins. Technically, I think it is going to be downhill skiing from here, and some very visible bits of functionality will get added much more easily than this difficult infrastructure we just coded.

Hans

#263 lore raw
In reply to Rene Rebe
Quoted 12 lines · Rene Rebe

Hi,

Quoted 8 lines · Theodore Tso
Quoted 1 line

The code isn't even written, benchmarked, or tested yet,

Actually, the first bits that we plan to merge have already been written and in use by hundreds of clusterfs customers, posted to LKML for comments (and we don't attack our reviewers, we thank them for their comments), and in fact they were written about at last year's OLS complete with benchmarks and graphs.
(http://ext2.sourceforge.net/2005-ols/2005-ols-ext3.html)

Quoted 2 lines

However I would estimate that Reiser4 is used by more people than yet aother ext2 patchups.

Any data backing up that estimation? Just to give an example, that "patchup" is used by (tens of) thousands of computers in governmental laboratories the US "national security" depends upon.

Quoted 2 lines

Yours,

Nikita.

#319 lore raw
In reply to Steve Lord
Quoted 9 lines · Steve Lord
Quoted 6 lines · Theodore Tso
Quoted 4 lines · Hans Reiser

Consider what happened with XFS as the article writer mentions. I met the original XFS team, led by two very senior developers (Jim Grey, and another fellow whose name I am blanking on, forgive me, I learned much from him in just a few conversations).

Not sure who Jim Grey was, he never worked on XFS, ah well.

I believe the (mis-)reference is to a famous data-base person, co-author of "Transaction Processing". He is with Microsoft now
(http://research.microsoft.com/~Gray/JimGrayHomePageSummary.htm).

Quoted 2 lines

Steve Lord

Nikita.

#12 lore raw
In reply to Matthias Andree
Quoted 2 lines · Matthias Andree

The father declared his child unsupported,

I never did that.

Quoted 5 lines

and that's the end
of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Quoted 8 lines

(I am well
aware this is exploiting worst-case behavior in a malicious sense but I simply cannot risk such nonsense on a 270 GB RAID5 if users have shared work directories.)

#259 lore raw
In reply to Jeff Garzik
Quoted 2 lines · Jeff Garzik

Anyone who fails to respect the kernel development process, the process of building consensus, is turn not respected, flamed, and/or ignored.

That reminds me of the old "layer 8 and 9" extensions to the OSI/ISO layering model. Layer 8: financial, Layer 9: policital.

SCNR.

Quoted 8 lines

With you in particular, you demonstrated NO interest in maintaining reiser3, once reiser4 began to make a splash. Linux kernel code exists for DECADES, and as such, long term maintenance is a CRITICAL aspect of development.

Regardless of whatever new whiz-bang technology exists in reiser4, there is a very real worry that you will abandon reiser4 once its in the tree for a few years, just like what happened with reiser3.

The most worrying point was that reiser3 maintenance was given up at the point where it was just about to transition from usable to mature.

Signature · 1 line
Matthias Andree
#11 lore raw
In reply to Hans Reiser
Quoted 3 lines · Hans Reiser

I want reiserfs to be the filesystem that professional system administrators view as the one with both the fastest technological pace, and the most conservative release management.

Well, I, with the administrator hat on, phased out all reiserfs file systems and replaced them by ext3. This got me rid of silent corruptions, immature reiserfsprogs and hash collision chain limits.

Quoted 9 lines

I apologize to users that the technology required a 5 year gap between
releases. It just did, an outsider may not realize how deep the
changes we made were. Things like per node locking based on a whole new approach to tree locking that goes bottom up instead of the usual top
down are big tasks. Dancing trees are a big change, getting rid of
blobs is a big change, wandering logs..... We did a lot of things like that, and got very fortunate with them. If we had tried to add such changes to V3, the code would have been unstable the whole 5 years, and would not have come out right.

And that is something that an administrator does not care the least about. It must simply work, and the tools must simply work. Once I hit issues like "xfs_check believes / were mounted R/W (not ignoring rootfs) and refuses the R/O check", "reiserfsck can't fix a R/O file system" (I believed this one got fixed before 3.6.19) or particularly silent corruptions that show up later in a routine fsck --check after a kernel update, the filesystem and its tools appear in a bad light. I've never had such troubles with ext2fs or ext3fs or FreeBSD's or Solaris's ufs.

I'm not sure what patches Chris added to SUSE's reiserfs, nor do I care any more. The father declared his child unsupported, and that's the end of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value". (I am well aware this is exploiting worst-case behavior in a malicious sense but I simply cannot risk such nonsense on a 270 GB RAID5 if users have shared work directories.)

Signature · 1 line
Matthias Andree
#260 lore raw
In reply to Matthias Andree
Quoted 5 lines · Matthias Andree

The most worrying point was that reiser3 maintenance was given up at the point where it was just about to transition from usable to mature.

Name a bug (not a feature) that has not been fixed by us, and which is not also so deep that it reasonably required waiting for a major release to fix it (for example, bug fixes that require disk format changes belong in v4 not v3 even if a case can be made that they are bug fixes).

I mean, god, sometimes I think users are like little children waiting for the pie that is in the oven and who want to take it out now before it finishes cooking so they can eat it, and they are very angry about it, and I should just understand that and not try to reason with it (but
also not give them the pie before it finishes cooking either). Someone
please tell me I don't understand the users and it all makes more sense than that, please....

No code before its time. No features in stable branches. Wait for it. Stop complaining about how you are abandoned, we are working hard. It's going to be the best pie ever. Wait for it.

Hans

#13 lore raw
In reply to Hans Reiser
Quoted 6 lines · Hans Reiser
Quoted 4 lines

and that's the end
of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

You see, I don't care a iota about "plugins" or other implementation details.

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

Signature · 1 line
Matthias Andree
#320 lore raw
In reply to Nikita Danilov
Quoted 5 lines · Nikita Danilov

I believe the (mis-)reference is to a famous data-base person, co-author of "Transaction Processing". He is with Microsoft now
(http://research.microsoft.com/~Gray/JimGrayHomePageSummary.htm).

That's what I thought when I saw the name Jim Gray, but as far as I knew he never worked on XFS and never left the Linux community dejected because Linux kernel coding standards requirements before changes were allowed to be merged, when Hans did his name dropping thing.

(I mean geez, if you want really high standards before new code is accepted, take a look at Open Solaris; they have *such* a heavyweight process, with two mandatory signoffs by core Solaris engineers who both have to do a line-by-line review, and with a promise of on-disk and ABI compatibility *forever* ---- that we do more commits in a week than they do in a year....)

- Ted

#261 lore raw
In reply to Hans Reiser
Quoted 7 lines · Hans Reiser

I mean, god, sometimes I think users are like little children waiting for the pie that is in the oven and who want to take it out now before it finishes cooking so they can eat it, and they are very angry about it, and I should just understand that and not try to reason with it (but
also not give them the pie before it finishes cooking either). Someone
please tell me I don't understand the users and it all makes more sense than that, please....

namesys.com doesn't list reiserfs 3.6 hash collision limits in an easy to find place (which would be the same place that boasts about 100,000 files per directory if it wants to be honest).

Quoted 3 lines

No code before its time. No features in stable branches. Wait for it. Stop complaining about how you are abandoned, we are working hard. It's going to be the best pie ever. Wait for it.

I'm not going to eat it while it's still steaming and fogging my glasses.

You're now making the same noise about how good reiser4 is that was made when reiserfs 3.5 was a patch for Linux 2.2 and that wedged local access when NFS exported, and 3.6 didn't fix the hash collision issue (but required a format change, too, right).

Signature · 1 line
Matthias Andree
#14 lore raw
In reply to Matthias Andree
Quoted 9 lines · Matthias Andree
Quoted 7 lines · Hans Reiser
Quoted 4 lines

and that's the end
of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Quoted 6 lines

You see, I don't care a iota about "plugins" or other implementation details.

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

And what do you do if you, say, run of of inodes on ext3? Do you think the users will care about that? Or what if the number of files in your mail queue or proxy cache* become large enough for your fs operations to slow to a crawl?

* Yes I know most programs work around this by using many subdirs, but that's really a bandaid solution.

Signature · 2 lines
Regards,
Christian Iversen
#321 lore raw
In reply to Theodore Tso
Quoted 6 lines · Theodore Tso

(I mean geez, if you want really high standards before new code is accepted, take a look at Open Solaris; they have *such* a heavyweight process, with two mandatory signoffs by core Solaris engineers who both have to do a line-by-line review, and with a promise of on-disk and ABI compatibility *forever* ---- that we do more commits in a week than they do in a year....)

That sounds almost like gcc, only worse.

I think there is something of a problem currently, tough. It is getting too hard to get code in if you're not a maintainer for an existing subsystem (reiser4, suspend2...), and too easy when you're a maintainer (ext4, uswsusp...). Ext patches don't get reviewed much outside of the developpers, and they go in pretty much without discussion in any case, except when Linus blows a fuse. Reiser4 would have be in without discussion if it had been a set of patches through time to reiser3, and would have been called 4 only when Linus yelled. I suspect some balancing would be useful.

OG.

#15 lore raw
In reply to Christian Iversen
Quoted 7 lines · Christian Iversen
Quoted 3 lines · Matthias Andree

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

And what do you do if you, say, run of of inodes on ext3? Do you think the users will care about that?

From what I've seen from our customers, that never happens. Yes, there are sometimes people with a million inodes in use, and we've seen four million once, but that's never been a problem, even not with a huge mail server with thousands of users having mailboxes in maildir format.

We usually limit our own filesystems to 12 million inodes and it's never been a problem to store files from our customers.

Quoted 3 lines

Or what if the number of files in your mail queue
or proxy cache* become large enough for your fs operations to slow to a crawl?

Not a problem anymore with htree dirextory indexing. If it's not yet enabled (dumpe2fs the filesystem and look for the "dir_index" feature), enable it with:

tune2fs -O dir_index /dev/whatever

After the next mount the filesystem will use it for new directories. To optimize existing directories, run e2fsck -D.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#322 lore raw
In reply to Olivier Galibert
Quoted 3 lines · Olivier Galibert

Ext patches don't get reviewed much
outside of the developpers, and they go in pretty much without discussion in any case, except when Linus blows a fuse.

Um, you're kidding, right? We certainly don't make the assumption that we can violate CodingStyle willy nilly and stuff in yacc grammers into ext3 and assume that no one will push back.

In fact we did a lot of work to make sure the patches were clean and mostly ready to be accepted to mainline even before we made the first proposal to push extents to LKML.

Quoted 4 lines

I think there is something of a problem currently, tough. It is getting too hard to get code in if you're not a maintainer for an existing subsystem (reiser4, suspend2...), and too easy when you're a maintainer (ext4, uswsusp...).

It's not fair to assume that the only reason why non-maintainers have a harder time getting changes is because their changes are getting more intensive review. (Although it is the case that we probably do need to get better at reviewing changes that go in via git trees.) A much more important effect is that non-maintainers aren't familiar with coding and patch submission guidelines. For example, in suspend2, Nigel first tried with patches that were too monolithic, and then his next series was too broken down such that it was too hard to review (and "git bisect" wouldn't work). And of course, there are people who assume that the rules shouldn't apply to their filesystem...

- Ted

#323 lore raw
Quoted 8 lines · Theodore Tso
Quoted 3 lines · Olivier Galibert

Ext patches don't get reviewed much
outside of the developpers, and they go in pretty much without discussion in any case, except when Linus blows a fuse.

Um, you're kidding, right? We certainly don't make the assumption that we can violate CodingStyle willy nilly and stuff in yacc grammers into ext3 and assume that no one will push back.

I'm not kidding. I do recognise that the ext* maintainers do a very good and clean job.

Quoted 3 lines

In fact we did a lot of work to make sure the patches were clean and mostly ready to be accepted to mainline even before we made the first proposal to push extents to LKML.

I'm no talking about extends only. Ext3 now is very, very different than the ext2+journal it was at the start, with backwards-incompatible format changes added all the time[1]. These changes went in no-questions-asked.

Quoted 8 lines
Quoted 4 lines

I think there is something of a problem currently, tough. It is getting too hard to get code in if you're not a maintainer for an existing subsystem (reiser4, suspend2...), and too easy when you're a maintainer (ext4, uswsusp...).

It's not fair to assume that the only reason why non-maintainers have a harder time getting changes is because their changes are getting more intensive review.

"only", no, definitively not. The impact is non-negligible though.

Quoted 2 lines

(Although it is the case that we probably do
need to get better at reviewing changes that go in via git trees.)

Ohh yes, a lot better. Just look at ALSA, most of SNDRV_HWDEP* should never have gotten in in the first place, especially some recent ones like SNDRV_HWDEP_IFACE_SB_RC, and even some (SNDRV_HWDEP_IFACE_USX2Y*) are security holes the size of Cleveland. I need to continue documenting the Alsa kernel interface, but I need a new bucket, the first one is overflowing.

Quoted 5 lines

A much more important effect is that non-maintainers aren't familiar with coding and patch submission guidelines. For example, in suspend2, Nigel first tried with patches that were too monolithic, and then his next series was too broken down such that it was too hard to review (and "git bisect" wouldn't work).

All his submissions since 2004 or so? It's a little easy to limit oneself to the last two ones.

Quoted 2 lines

And of course, there are people who assume that the rules shouldn't apply to their filesystem...

It may be a little hard to remove XFS at that point though... And, while it's not a filesystem, I'd love to be pointed to the technical discussion deciding whether uswsusp is a good idea.

OG.

[1] All optional, I know.

#324 lore raw
Quoted 4 lines · Olivier Galibert

I'm no talking about extends only. Ext3 now is very, very different than the ext2+journal it was at the start, with backwards-incompatible format changes added all the time[1]. These changes went in no-questions-asked.

The patches indeed were reviewed and changes made in response to the reviews. The philosophical/design question about whether or not optional features which, if enabled, would prevent older kernels to mount the filesystem was not asked, no. But that doesn't mean that the code was not reviewed; it was.

I would also note that we didn't intimate that we knew better than the reviewers, or question their motives, or otherwise insult the reviewers such that they might decide they have better things to do than to review our patches, and that might have had something to do with how the code got in relatively painlessly....

- Ted

#16 lore raw
In reply to Matthias Andree
Quoted 7 lines · Matthias Andree
Quoted 5 lines · Hans Reiser
Quoted 4 lines

and that's the end
of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Quoted 8 lines
Quoted 1 line

Requires a disk format change, in a filesystem without plugins, to fix it.

You see, I don't care a iota about "plugins" or other implementation details.

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly, which was reason enough for me to not use EXT3 and use ReiserFS instead.

Do you consider the EXT3 developers to have "abandoned" it because they haven't fixed this issue? I don't, I just think of it as using the right tool for the job.

I've been bitten by running out of inodes on several occasions, and by switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

I haven't been able to use EXT3 on a backup server for the last ~5 years due to inode limitations. Instead, ReiserFS has been filling that spot like a champ.

The bottom line is that every file system imposes some sort of limits that bite someone. In your case it sounds like EXT3 limits weren't an issue for you, in my case they were. Thats life.

Signature · 1 line
Mike Benoit <[email protected]>
#17 lore raw
Quoted 3 lines · Mike Benoit

I've been bitten by running out of inodes on several occasions, and by switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

ext3fs's inode density is configurable, reiserfs's hash overflow chain length is not, and it doesn't show in df -i either.

If you need lots of inodes, mkfs for lots. That's old Unix lore.

#325 lore raw
In reply to Theodore Tso
Quoted 5 lines · Theodore Tso

I would also note that we didn't intimate that we knew better than the reviewers, or question their motives, or otherwise insult the reviewers such that they might decide they have better things to do than to review our patches, and that might have had something to do with how the code got in relatively painlessly....

Now that has a very high impact :-) Hans is not very good at the diplomacy game.

The fact that the ext maintainers are very, very good helps quite a lot too. But I think it doesn't change the fact that if r4 has been a set of patches through time to r3, good or not, there wouldn't be a discussion.

It's maybe the lack of an official development branch, but it looks like the kernel development has become very risk-averse, and the bar is set much higher to accept anything that looks relatively new. Any reason is good to have it dropped, cosmetic or not.

Just to give you an idea, if the criteria applied to suspend2 or reiser4 had been applied to everything else, we wouldn't have at least XFS[1], ALSA[2], sysfs[3] and DRM[4]. Whether it is good or bad is an interesting question itself. But before, code just had to be reasonably sane, and it was expected to be fixed through time. Some even has been (sysfs got better). Now it has to attain an ever moving level of perfection before it has a chanc to be accepted.

Unless you're a maintainer, that is, for which you can get pretty much anything in that doesn't immediatly break the compile through git trees. Thankfully, most of the maintainers are sane.

Don't you think this is a problem?

OG.

[1] CodingStyle is for the weak

[2] You call that a kernel interface? HWDEP? ioctl to read/write samples instead of read/write?

[3] Lifetime rules and locking are for other people to care about

[4] Oh my, people are still trying to fix that one

#19 lore raw
In reply to Mike Benoit
Quoted 9 lines · Mike Benoit
Quoted 8 lines · Matthias Andree
Quoted 6 lines · Hans Reiser
Quoted 4 lines

and that's the end
of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Quoted 8 lines
Quoted 5 lines

You see, I don't care a iota about "plugins" or other implementation details.

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly,

Right. Plan ahead.

Quoted 2 lines

which was reason enough for me to not use EXT3 and use ReiserFS instead.

I don't see this following in any way.

Quoted 3 lines

Do you consider the EXT3 developers to have "abandoned" it because they haven't fixed this issue? I don't, I just think of it as using the right tool for the job.

Dangerous parallel, that one...

Quoted 1 line

I've been bitten by running out of inodes on several occasions,

Me too. It was rather painful each time, but fixable (and in hindsight, dumb user (setup) error).

Quoted 3 lines

and by
switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

How can a filesystem (which by basic requirements and design is almost transparent to applications) make such a difference?!

Quoted 2 lines

I haven't been able to use EXT3 on a backup server for the last ~5 years due to inode limitations.

See comment above. Read mke2fs(8) with care.

Quoted 2 lines

Instead, ReiserFS has been filling that spot
like a champ.

Nice for you.

Quoted 2 lines

The bottom line is that every file system imposes some sort of limits that bite someone.

Mostly that infinite disks are hard to come by ;-)

Quoted 2 lines

In your case it sounds like EXT3 limits weren't an
issue for you, in my case they were.

I'd suspect the limits you ran into weren't exactly in ext3.

Quoted 1 line

Thats life.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#18 lore raw
In reply to Matthias Andree
Quoted 8 lines · Matthias Andree
Quoted 3 lines · Mike Benoit

I've been bitten by running out of inodes on several occasions, and by switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

ext3fs's inode density is configurable, reiserfs's hash overflow chain length is not, and it doesn't show in df -i either.

Equally important - you can usually *see* "out of inodes" coming on a 'df -i' long before a reiser3 filesystem hits the wall on a hash issue.

#326 lore raw
In reply to Olivier Galibert
Quoted 4 lines · Olivier Galibert

The fact that the ext maintainers are very, very good helps quite a lot too. But I think it doesn't change the fact that if r4 has been a set of patches through time to r3, good or not, there wouldn't be a discussion.

That's a huuuuuge leap of logic.

Metadata plugins found in reiser4 are far better done at the VFS level, than burying "reiser4 can look like ext2, if it wishes" functionality inside the filesystem.

I guarantee such a patch to reiser3 would get rejected.

Quoted 4 lines

It's maybe the lack of an official development branch, but it looks like the kernel development has become very risk-averse, and the bar is set much higher to accept anything that looks relatively new. Any reason is good to have it dropped, cosmetic or not.

New stuff goes in all the time.

The bar is set too high in some cases (read: SCSI subsystem submissions), but reiser4 submission cannot be generalized as you have done here. There are very real issues present, that need to be dealt with.

Quoted 7 lines

Just to give you an idea, if the criteria applied to suspend2 or reiser4 had been applied to everything else, we wouldn't have at least XFS[1], ALSA[2], sysfs[3] and DRM[4]. Whether it is good or bad is an interesting question itself. But before, code just had to be reasonably sane, and it was expected to be fixed through time. Some even has been (sysfs got better). Now it has to attain an ever moving level of perfection before it has a chanc to be accepted.

reiser4 tries to be another VFS. That's a bit more than needing additional minor fixes over time.

Jeff

#178 lore raw
In reply to Matthias Andree

You see, I don't care a iota about "plugins" or other implementation

Quoted 7 lines

details.

The bottom line is reiserfs 3.6 imposes practial limits that ext3fs doesn't impose and that's reason enough for an administrator not to install reiserfs 3.6. Sorry.

I dont care a iota about ext3fs, reiser3 does its job for me. Ext3 got quite a number of its own limits, but your iotas and ext3 limit are not even close to ssubject of this conversation.

thank you.

Signature · 1 line
liberation loophole will make it clear
#20 lore raw
In reply to Horst H. von Brand
Quoted 4 lines · Horst H. von Brand
Quoted 2 lines

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly,

Right. Plan ahead.

That is great in theory. But back to reality, when your working for a company that is growing by leaps and bounds that isn't always possible. Why would I want to intentionally limit myself to a set number of inodes when I can get a performance increase and not have to worry about it by simply using ReiserFS?

It all boils down to using the right tool for the job, ReiserFS was the right tool for this job.

Quoted 11 lines
Quoted 1 line

I've been bitten by running out of inodes on several occasions,

Me too. It was rather painful each time, but fixable (and in hindsight, dumb user (setup) error).

Quoted 3 lines

and by
switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

How can a filesystem (which by basic requirements and design is almost transparent to applications) make such a difference?!

Very easily, the backup software the company had spent ~$75,000 on before I started working there used tiny little files as its "database". Once the inodes ran out the entire system pretty much came to a screeching halt. We basically had two options, use ReiserFS, or find another piece of software that didn't use tiny little files as its database. If I recall correctly the database went from about 2million files to over 40 million in the span of just a few months. No one could have predicted that. I believe this database was on an 18GB SCSI drive, so even using 1K blocks wouldn't have worked. According to the mke2fs man page:

"-i bytes-per-inode
This value generally shouldn't be smaller than the blocksize of the filesystem, since then too many inodes will be made."

The only other option at that time was to purchase Veritas backup and its not cheap. We ended up switching to ReiserFS, it increased our backup/restore time immensely and also bought us about 1year before the system finally outgrew itself for good. By that time the company could afford to drop $250,000 on high end backup software so we could grow past 10TB.

Signature · 1 line
Mike Benoit <[email protected]>
#21 lore raw
Quoted 7 lines · Mike Benoit

On Mon, 2006-07-24 at 14:06 -0400, Horst H. von Brand wrote:
The only other option at that time was to purchase Veritas backup and its not cheap. We ended up switching to ReiserFS, it increased our backup/restore time immensely and also bought us about 1year before the system finally outgrew itself for good. By that time the company could afford to drop $250,000 on high end backup software so we could grow past 10TB.

Erm, how does your data look like and why does it require a $250,000 backup solution? Sounds like I'm in the wrong business...

But that's for sure not lkml related...

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                       Don't believe in miracles: Rely on them!
 the second  :
#22 lore raw
In reply to Mike Benoit
Quoted 11 lines · Mike Benoit
Quoted 4 lines · Horst H. von Brand
Quoted 2 lines

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly,

Right. Plan ahead.

That is great in theory. But back to reality, when your working for a company that is growing by leaps and bounds that isn't always possible. Why would I want to intentionally limit myself to a set number of inodes when I can get a performance increase and not have to worry about it by simply using ReiserFS?

Place your filesystems on LVN, when they grow the number of inodes grows to match. Or did you run out of inodes and not of diskspace? Only ever happened to me on (static) /dev, or (wrong configured) newsspools...

Quoted 2 lines

It all boils down to using the right tool for the job, ReiserFS was the right tool for this job.

/One/ tool that did the job. Not "the" right one, perhaps not even the best one.

Quoted 9 lines
Quoted 8 lines
Quoted 1 line

I've been bitten by running out of inodes on several occasions,

Me too. It was rather painful each time, but fixable (and in hindsight, dumb user (setup) error).

Quoted 3 lines

and by
switching to ReiserFS it saved one company I worked for over $250,000 because they didn't need to buy a totally new piece of software.

Quoted 5 lines
Quoted 2 lines

How can a filesystem (which by basic requirements and design is almost transparent to applications) make such a difference?!

Very easily, the backup software the company had spent ~$75,000 on before I started working there used tiny little files as its "database".

If you know that, you configure your filesystem like a newsspool or some such.

Quoted 2 lines

Once the inodes ran out the entire system pretty much came to a screeching halt.

Get a clue by for, apply to the vendor (for the design, or at the very least for not warning unsuspecting users)?

Quoted 3 lines

We basically had two options, use ReiserFS, or find
another piece of software that didn't use tiny little files as its database.

Or reconfigure the filesystem with more inodes (you were willing to rebuild the filesystem in any case, so...)

Quoted 9 lines

If I recall correctly the database went from about 2million
files to over 40 million in the span of just a few months. No one could have predicted that. I believe this database was on an 18GB SCSI drive, so even using 1K blocks wouldn't have worked. According to the mke2fs man page:

"-i bytes-per-inode
This value generally shouldn't be smaller than the blocksize of the filesystem, since then too many inodes will be made."

18GiB = 18 million KiB, you do have a point there. But 40 million files on that, with some space to spare, just doesn't add up.

Quoted 1 line

The only other option at that time was to purchase Veritas backup

... or a larger disk...

Quoted 6 lines

and
its not cheap. We ended up switching to ReiserFS, it increased our backup/restore time immensely and also bought us about 1year before the system finally outgrew itself for good. By that time the company could afford to drop $250,000 on high end backup software so we could grow past 10TB.

#351 lore raw
In reply to Hans Reiser
Quoted 1 line · Hans

Jim Grey

Sounds like the name of Jim Gray, the famous database and transaction processing expert of IBM, Tandem, DEC and Microsoft.

Neither SGI nor XFS has had the honor of working with Jim Gray.

The original XFS developers and designers include Geoff Peck, Jeff Glover, Adam Sweeney and Doug Doucette, from what I can see from my notes dating back to 1993/1994 on this.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#23 lore raw
In reply to Horst H. von Brand
Quoted 5 lines · Horst H. von Brand
Quoted 2 lines

It all boils down to using the right tool for the job, ReiserFS was the right tool for this job.

/One/ tool that did the job. Not "the" right one, perhaps not even the best one.

Please, enlighten me as to what "the best" tool for the job would be in your opinion. Keep in mind this was around 2002-2003, which if I recall was before dir_index for EXT3 was around and stable too.

Quoted 10 lines
Quoted 9 lines

If I recall correctly the database went from about 2million
files to over 40 million in the span of just a few months. No one could have predicted that. I believe this database was on an 18GB SCSI drive, so even using 1K blocks wouldn't have worked. According to the mke2fs man page:

"-i bytes-per-inode
This value generally shouldn't be smaller than the blocksize of the filesystem, since then too many inodes will be made."

Quoted 6 lines

18GiB = 18 million KiB, you do have a point there. But 40 million files on that, with some space to spare, just doesn't add up.

Quoted 1 line

The only other option at that time was to purchase Veritas backup

... or a larger disk...

Why would we waste money purchasing larger disks just to _temporarily_ work around an EXT3 limitation? Even if we doubled the size of the disks (36gb) we would have run in to the same problem within months. I wouldn't consider this to be smart "planning ahead" as you put it.

Its not just an inode issue either, its a performance issue. Benchmarks we ran with just 100,000 files making up 1GB of data, ReiserFS blew EXT3 out of the water where EXT3 was over 50% slower. If I recall the "database" was about 5GB at 25million files. When you have a limited window of time to do the backups in, performance is critical.

http://fsbench.netnation.com/new_hardware/2.6.0-test9/scsi/bonnie.html
(4th table down)

Signature · 1 line
Mike Benoit <[email protected]>
#352 lore raw
In reply to Paul Jackson

No, it was not him, sigh, I'll have to ask Jim Mostek who it was if I can find a valid email for Jim. There was another guy who was Jim Mostek's peer. I might have the name wrong.

#264 lore raw
In reply to Nikita Danilov

I'm no reiserfs user (nor v3 nor v4), but I respect views of people that needs the fs to handle zillons of tiny files, for that reiser3 did a good job, and my humble opinion is that the only chance for reiser4 to stabilize is to grow the userbase and including it in mainline or in some distro would obviously help (like it happened to reiser3 in the past with SUSE inclusion first and mainline inclusion later, nothing has changed dramatically since then, reiser3 was very buggy at the time too, it was probably my backport of lfs that made us notice it couldn't even handle lfs on-disk so they had to change the fs format later on with some backwards compatibility mess with older kernels, hopefully this time the fixage of v4 for production usage won't require a change of the on-disk format).

Quoted 1 line · Nikita Danilov

Any data backing up that estimation? Just to give an example, that

Probably KLive is the only tool that can provide an estimate of the part of the reiser4 userbase compared to the other fs. reiser4 had so far 1926 days of total uptime and 4409 users using it at least once for root or anyway mounted nearly at boot. Avg uptime is 16 hours, max uptime is 28 days.

I can also provide the exact storage scsi/SATA/ATA chipsets used in combination with it, but it doesn't seem necessary.

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where f = 'reiser4' group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+------------------------------+-----------------+------------------
reiser4 | 4409 | 1926 days 27479:22:27.459534 | 16:42:59.666015 | 28 days 16:16:15
(1 row)

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where f = 'ext4' group by f order by sum(uptime) desc;
f | count | sum | avg | max
---+-------+-----+-----+-----
(0 rows)

klive=>

Full data of all fs (all kernels not just mainline ones) follows. In total uptime and avg uptime terms it beats many other common fs like isofs, autofs and ramfs. It'd be cool if I'd be able to receive a packet during sigterm so I could then differentiate sessions that rebooted cleanly from the ones that didn't. Receiving oopses (optionally encrypted) is also a long term planned feature. At the moment I can't (and I've no much time to work on klive), but this is still good to quantify the current userbase compared to the other more common fs.

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+--------------------------------+-------------------------+-------------------------

 sysfs                            | 40998 | 35138 days 259797:16:29.93576  | 26:54:23.100394         | 173 days 16:16:15
 tmpfs                            | 40507 | 35025 days 257298:57:48.690973 | 27:06:14.154311         | 173 days 16:16:15
 usbfs                            | 38803 | 26681 days 241169:00:36.268676 | 22:43:03.543444         | 173 days 16:16:15
 reiserfs                         | 19934 | 19257 days 133294:48:27.256508 | 29:52:18.181361         | 155 days 16:16:15
 ext3                             | 22077 | 16346 days 132190:42:38.862752 | 23:45:27.062502         | 173 days 16:16:15
 ext2                             |  9396 | 5629 days 63099:50:27.500173   | 21:05:37.103821         | 139 days 14:54:11
 nfs                              |  2824 | 6373 days 23368:45:22.667638   | 2 days 14:26:11.502361  | 86 days 16:16:15
 binfmt_misc                      |  4396 | 5421 days 33069:20:47.871638   | 1 day 13:07:06.944466   | 173 days 16:16:15
 nfsd                             |  2231 | 4727 days 19824:00:00.354481   | 2 days 11:44:11.187967  | 89 days 16:16:15
 xfs                              |  4326 | 4067 days 28000:38:11.762818   | 29:02:08.685105         | 139 days 14:54:11
 vfat                             |  8684 | 2470 days 46585:39:49.859957   | 12:11:27.193673         | 68 days 16:16:15
 ntfs                             |  8482 | 2382 days 47344:00:52.902741   | 12:19:17.846369         | 68 days 16:16:15
 rpc_pipefs                       |  1188 | 3078 days 11297:55:12.600343   | 2 days 23:41:30.667172  | 173 days 16:16:15
 reiser4                          |  4409 | 1926 days 27479:22:27.459534   | 16:42:59.666015         | 28 days 16:16:15
^^^^^^^^^^^^^
 autofs                           |  2030 | 2374 days 14512:14:16.617646   | 1 day 11:12:57.170748   | 54 days 16:16:15
 ramfs                            |  2847 | 817 days 14809:30:42.122875    | 12:05:20.562741         | 32 days 16:16:15
 selinuxfs                        |   600 | 1006 days 3466:41:58.91087     | 1 day 22:01:04.198185   | 173 days 16:16:15
 smbfs                            |   222 | 1053 days 2155:11:48.374153    | 4 days 27:32:45.353037  | 122 days 16:20:01
 jfs                              |   370 | 971 days 2435:07:17.805929     | 2 days 21:33:54.696773  | 160 days 16:16:15
 iso9660                          |   651 | 792 days 4698:41:27.153665     | 1 day 12:24:56.90807    | 84 days 16:06:15
 subfs                            |   611 | 736 days 3482:38:06.342656     | 1 day 10:36:35.558662   | 122 days 16:20:01
 devfs                            |    51 | 713 days 543:13:20.51142       | 13 days 34:10:50.99042  | 155 days 16:16:15
 supermount                       |   260 | 437 days 1281:43:13.500606     | 1 day 21:16:05.359618   | 68 days 16:16:15
 cifs                             |   506 | 222 days 2997:17:59.130448     | 16:27:11.381681         | 35 days 16:16:15
 openpromfs                       |    53 | 296 days 631:17:42.10027       | 5 days 25:56:56.266043  | 43 days 16:16:15
 fuse                             |   390 | 163 days 2443:31:37.846769     | 16:17:46.404735         | 17 days 16:16:15
 squashfs                         |    62 | 198 days 622:55:15.099784      | 3 days 14:41:32.179029  | 34 days 16:16:15
 configfs                         |    45 | 160 days 504:05:40.811143      | 3 days 24:32:07.573581  | 23 days 16:16:15
 capifs                           |    67 | 135 days 569:04:11.286495      | 2 days 08:51:06.437112  | 39 days 16:16:15
 minix                            |    35 | 128 days 428:57:59             | 3 days 28:01:39.40      | 16 days 16:16:15
 cramfs                           |     6 | 140 days 69:58:03.269142       | 23 days 19:39:40.544857 | 58 days 16:16:15
 udf                              |    15 | 102 days 161:35:54.111551      | 6 days 29:58:23.607437  | 51 days 16:16:15
 debugfs                          |   314 | 39 days 1144:50:09.253475      | 06:37:36.717368         | 11 days 16:14:23.049312
 ufs                              |   146 | 24 days 884:40:35.839055       | 10:00:16.683829         | 10 days 16:16:11.062917
 usbdevfs                         |   139 | 1076:04:34.054012              | 07:44:29.597511         | 19:39:58
 shm                              |   139 | 1076:04:34.054012              | 07:44:29.597511         | 19:39:58
 hfsplus                          |    80 | 14 days 480:38:46.186027       | 10:12:29.077325         | 2 days 23:36:48
 unionfs                          |    23 | 24 days 129:38:06              | 1 day 06:40:47.217391   | 22 days 16:16:15
 ocfs2_dlmfs                      |     1 | 23 days 16:16:15               | 23 days 16:16:15        | 23 days 16:16:15
 oprofilefs                       |     1 | 23 days 16:16:08.088967        | 23 days 16:16:08.088967 | 23 days 16:16:08.088967
 nfs4                             |     5 | 20 days 71:36:00               | 4 days 14:19:12         | 12 days 16:16:15
 nnpfs                            |     9 | 18 days 78:16:50               | 2 days 08:41:52.222222  | 8 days 16:16:15
 afs                              |     1 | 19 days 16:16:15               | 19 days 16:16:15        | 19 days 16:16:15
 securityfs                       |     9 | 4 days 22:13:22                | 13:08:09.111111         | 4 days 16:16:15
 ufsd                             |    15 | 75:38:57                       | 05:02:35.80             | 23:01:05
 vmware-hgfs                      |     1 | 1 day 04:56:21                 | 1 day 04:56:21          | 1 day 04:56:21
 shfs                             |     2 | 14:46:00                       | 07:23:00                | 11:27:35

(47 rows)

Now let's eliminate from the screen all sessions that rebooted (or crashed) in less than one day:

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where uptime > '1 day' group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+-------------------------------+-------------------------+-------------------------

 sysfs                            |  5385 | 35138 days 80491:02:11.763381 | 6 days 27:33:04.202742  | 173 days 16:16:15
 tmpfs                            |  5344 | 35025 days 79890:55:31.804764 | 6 days 28:14:51.192329  | 173 days 16:16:15
 usbfs                            |  4681 | 26681 days 69552:08:46.865054 | 5 days 31:39:17.301189  | 173 days 16:16:15
 reiserfs                         |  2959 | 19257 days 44003:36:16.956207 | 6 days 27:03:42.161864  | 155 days 16:16:15
 ext3                             |  2520 | 16346 days 37501:47:54.627011 | 6 days 26:33:28.283582  | 173 days 16:16:15
 nfs                              |   645 | 6373 days 10050:19:24.173861  | 9 days 36:43:00.409572  | 86 days 16:16:15
 ext2                             |  1012 | 5629 days 15173:13:30.06032   | 5 days 28:29:14.555396  | 139 days 14:54:11
 binfmt_misc                      |   806 | 5421 days 12168:47:27.060755  | 6 days 32:31:01.59685   | 173 days 16:16:15
 nfsd                             |   634 | 4727 days 9800:42:01.958542   | 7 days 26:23:54.892679  | 89 days 16:16:15
 xfs                              |   607 | 4067 days 9311:21:11.733095   | 6 days 32:08:38.075343  | 139 days 14:54:11
 rpc_pipefs                       |   445 | 3078 days 6789:18:13.704195   | 6 days 37:15:40.884729  | 173 days 16:16:15
 vfat                             |   669 | 2470 days 9424:11:09.850314   | 3 days 30:41:48.624589  | 68 days 16:16:15
 ntfs                             |   636 | 2382 days 9081:10:26.573867   | 3 days 32:09:55.324802  | 68 days 16:16:15
 autofs                           |   405 | 2374 days 6335:32:22.311385   | 5 days 36:19:29.240275  | 54 days 16:16:15
 reiser4                          |   509 | 1926 days 7617:09:58.618041   | 3 days 33:46:41.961921  | 28 days 16:16:15
^^^^^^^^^^
 smbfs                            |    96 | 1053 days 1476:55:37.466481   | 10 days 38:38:04.765276 | 122 days 16:20:01
 selinuxfs                        |    97 | 1006 days 1584:22:47.021437   | 10 days 25:14:27.701252 | 173 days 16:16:15
 jfs                              |    71 | 971 days 1070:04:09.186991    | 13 days 31:17:48.298408 | 160 days 16:16:15
 ramfs                            |   241 | 817 days 3394:18:10.874752    | 3 days 23:26:42.8667    | 32 days 16:16:15
 iso9660                          |    85 | 792 days 1321:54:28.689609    | 9 days 23:10:31.396348  | 84 days 16:06:15
 subfs                            |    89 | 736 days 1346:31:28.394797    | 8 days 21:36:05.038144  | 122 days 16:20:01
 devfs                            |    24 | 713 days 399:55:43.321687     | 29 days 33:39:49.30507  | 155 days 16:16:15
 supermount                       |    34 | 437 days 466:58:54.119926     | 12 days 34:12:19.238821 | 68 days 16:16:15
 openpromfs                       |    29 | 296 days 515:10:26.10027      | 10 days 22:43:48.486216 | 43 days 16:16:15
 cifs                             |    39 | 222 days 622:53:46.261055     | 5 days 32:35:13.493873  | 35 days 16:16:15
 squashfs                         |    23 | 198 days 410:38:26            | 8 days 32:27:45.478261  | 34 days 16:16:15
 fuse                             |    36 | 163 days 534:27:25.006209     | 4 days 27:30:45.694617  | 17 days 16:16:15
 configfs                         |    23 | 160 days 344:26:41.378742     | 6 days 37:55:56.581684  | 23 days 16:16:15
 minix                            |    24 | 128 days 393:11:05            | 5 days 24:22:57.708333  | 16 days 16:16:15
 cramfs                           |     4 | 140 days 66:29:40.119926      | 35 days 16:37:25.029981 | 58 days 16:16:15
 capifs                           |    11 | 135 days 170:13:33            | 12 days 22:01:13.909091 | 39 days 16:16:15
 udf                              |     4 | 102 days 64:26:53             | 25 days 28:06:43.25     | 51 days 16:16:15
 debugfs                          |    11 | 39 days 182:23:03.211977      | 3 days 29:40:16.655634  | 11 days 16:14:23.049312
 ufs                              |    11 | 24 days 151:36:11.289322      | 2 days 18:08:44.662666  | 10 days 16:16:11.062917
 unionfs                          |     2 | 24 days 39:53:03              | 12 days 19:56:31.50     | 22 days 16:16:15
 ocfs2_dlmfs                      |     1 | 23 days 16:16:15              | 23 days 16:16:15        | 23 days 16:16:15
 oprofilefs                       |     1 | 23 days 16:16:08.088967       | 23 days 16:16:08.088967 | 23 days 16:16:08.088967
 nfs4                             |     4 | 20 days 62:33:56              | 5 days 15:38:29         | 12 days 16:16:15
 nnpfs                            |     4 | 18 days 73:50:18              | 4 days 30:27:34.50      | 8 days 16:16:15
 afs                              |     1 | 19 days 16:16:15              | 19 days 16:16:15        | 19 days 16:16:15
 hfsplus                          |    11 | 14 days 109:35:09             | 1 day 16:30:28.090909   | 2 days 23:36:48
 securityfs                       |     1 | 4 days 16:16:15               | 4 days 16:16:15         | 4 days 16:16:15
 vmware-hgfs                      |     1 | 1 day 04:56:21                | 1 day 04:56:21          | 1 day 04:56:21

(43 rows)

Now let's eliminate everything that rebooted or crashed in less than 10 days:

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where uptime > '10 day' group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+-------------------------------+-------------------------+-------------------------

 sysfs                            |   948 | 22969 days 15224:50:33.717548 | 24 days 21:33:13.073542 | 173 days 16:16:15
 tmpfs                            |   949 | 22960 days 15240:45:20.729486 | 24 days 20:42:47.250505 | 173 days 16:16:15
 usbfs                            |   689 | 16015 days 11048:34:25.54762  | 23 days 21:53:15.450722 | 173 days 16:16:15
 reiserfs                         |   516 | 12330 days 8297:16:33.102261  | 23 days 37:34:06.110663 | 155 days 16:16:15
 ext3                             |   453 | 10925 days 7258:04:40.240741  | 24 days 18:49:48.698103 | 173 days 16:16:15
 nfs                              |   193 | 4902 days 3138:54:10.459336   | 25 days 25:50:19.950567 | 86 days 16:16:15
 binfmt_misc                      |   154 | 3581 days 2484:52:11.438999   | 23 days 22:12:48.385968 | 173 days 16:16:15
 ext2                             |   139 | 3359 days 2218:31:49.225427   | 24 days 19:55:54.742629 | 139 days 14:54:11
 nfsd                             |   135 | 3226 days 2186:35:50.40609    | 23 days 37:42:29.262267 | 89 days 16:16:15
 xfs                              |   116 | 2666 days 1848:31:20.130065   | 22 days 39:31:18.276983 | 139 days 14:54:11
 rpc_pipefs                       |    84 | 2034 days 1356:31:00.312955   | 24 days 21:17:30.718011 | 173 days 16:16:15
 autofs                           |    69 | 1366 days 1095:10:03.052435   | 19 days 35:00:08.73989  | 54 days 16:16:15
 ntfs                             |    56 | 1058 days 905:01:16.41019     | 18 days 37:35:22.793039 | 68 days 16:16:15
 vfat                             |    52 | 1060 days 846:41:12.168038    | 20 days 25:30:47.541693 | 68 days 16:16:15
 reiser4                          |    53 | 820 days 859:44:03.65926      | 15 days 27:32:31.767156 | 28 days 16:16:15
^^^^^^^^^^^^
 jfs                              |    21 | 837 days 336:58:57            | 39 days 36:37:05.571429 | 160 days 16:16:15
 smbfs                            |    30 | 815 days 475:06:57.099919     | 27 days 19:50:13.903331 | 122 days 16:20:01
 selinuxfs                        |    28 | 767 days 454:42:21.021437     | 27 days 25:40:05.03648  | 173 days 16:16:15
 devfs                            |    14 | 686 days 233:05:03.154639     | 49 days 16:38:55.939617 | 155 days 16:16:15
 iso9660                          |    24 | 596 days 386:04:44.125293     | 24 days 36:05:11.838554 | 84 days 16:06:15
 subfs                            |    20 | 542 days 306:50:12.359485     | 27 days 17:44:30.617974 | 122 days 16:20:01
 supermount                       |    13 | 383 days 210:20:37.087554     | 29 days 27:15:25.929812 | 68 days 16:16:15
 ramfs                            |    17 | 327 days 261:34:11.310952     | 19 days 21:02:00.66535  | 32 days 16:16:15
 openpromfs                       |    13 | 238 days 211:24:12.08448      | 18 days 23:38:47.083422 | 43 days 16:16:15
 squashfs                         |     8 | 142 days 130:10:00            | 17 days 34:16:15        | 34 days 16:16:15
 cramfs                           |     3 | 137 days 48:48:44.087554      | 45 days 32:16:14.695851 | 58 days 16:16:15
 cifs                             |     6 | 115 days 97:37:30             | 19 days 20:16:15        | 35 days 16:16:15
 capifs                           |     4 | 116 days 65:05:00             | 29 days 16:16:15        | 39 days 16:16:15
 configfs                         |     5 | 95 days 80:52:24.077093       | 19 days 16:10:28.815419 | 23 days 16:16:15
 udf                              |     3 | 96 days 48:10:38              | 32 days 16:03:32.666667 | 51 days 16:16:15
 fuse                             |     6 | 85 days 93:19:29              | 14 days 19:33:14.833333 | 17 days 16:16:15
 minix                            |     3 | 42 days 48:48:45              | 14 days 16:16:15        | 16 days 16:16:15
 ocfs2_dlmfs                      |     1 | 23 days 16:16:15              | 23 days 16:16:15        | 23 days 16:16:15
 oprofilefs                       |     1 | 23 days 16:16:08.088967       | 23 days 16:16:08.088967 | 23 days 16:16:08.088967
 unionfs                          |     1 | 22 days 16:16:15              | 22 days 16:16:15        | 22 days 16:16:15
 afs                              |     1 | 19 days 16:16:15              | 19 days 16:16:15        | 19 days 16:16:15
 nfs4                             |     1 | 12 days 16:16:15              | 12 days 16:16:15        | 12 days 16:16:15
 debugfs                          |     1 | 11 days 16:14:23.049312       | 11 days 16:14:23.049312 | 11 days 16:14:23.049312
 ufs                              |     1 | 10 days 16:16:11.062917       | 10 days 16:16:11.062917 | 10 days 16:16:11.062917

(39 rows)

klive=>

Now let's eliminate everything that rebooted or crashed in less than 20 days:

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where uptime > '20 day' group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+------------------------------+-------------------------+-------------------------

 sysfs                            |   425 | 15940 days 6844:38:13.264145 | 37 days 28:14:46.337092 | 173 days 16:16:15
 tmpfs                            |   425 | 15908 days 6844:19:50.221708 | 37 days 26:26:18.329933 | 173 days 16:16:15
 usbfs                            |   266 | 10341 days 4258:02:54.770525 | 38 days 37:01:48.927709 | 173 days 16:16:15
 reiserfs                         |   236 | 8553 days 3809:00:57.232965  | 36 days 21:56:11.428953 | 155 days 16:16:15
 ext3                             |   197 | 7465 days 3163:26:12.784865  | 37 days 37:29:58.846624 | 173 days 16:16:15
 nfs                              |   127 | 4029 days 2066:09:04.104158  | 31 days 33:39:17.04019  | 86 days 16:16:15
 binfmt_misc                      |    63 | 2318 days 1020:40:19.272961  | 36 days 35:14:55.544015 | 173 days 16:16:15
 nfsd                             |    63 | 2243 days 1023:44:21.136625  | 35 days 30:43:33.668835 | 89 days 16:16:15
 ext2                             |    53 | 2220 days 827:52:02.580825   | 41 days 36:54:11.36945  | 139 days 14:54:11
 xfs                              |    42 | 1665 days 662:12:10.297862   | 39 days 31:11:43.10233  | 139 days 14:54:11
 rpc_pipefs                       |    30 | 1286 days 486:09:06.068985   | 42 days 37:00:18.202299 | 173 days 16:16:15
 autofs                           |    23 | 756 days 374:13:45           | 32 days 37:08:25.434783 | 54 days 16:16:15
 jfs                              |    10 | 677 days 158:20:12           | 67 days 32:38:01.20     | 160 days 16:16:15
 vfat                             |    20 | 636 days 326:41:41.13894     | 31 days 35:32:05.056947 | 68 days 16:16:15
 smbfs                            |    15 | 620 days 231:13:12.099919    | 41 days 23:24:52.806661 | 122 days 16:20:01
 devfs                            |     9 | 609 days 151:46:53.100264    | 67 days 32:51:52.566696 | 155 days 16:16:15
 ntfs                             |    20 | 584 days 325:04:00.023183    | 29 days 21:03:12.001159 | 68 days 16:16:15
 selinuxfs                        |     9 | 485 days 145:33:36.021437    | 53 days 37:30:24.002382 | 173 days 16:16:15
 iso9660                          |    13 | 463 days 207:52:24           | 35 days 30:45:34.153846 | 84 days 16:06:15
 subfs                            |    10 | 403 days 144:54:16.30619     | 40 days 21:41:25.630619 | 122 days 16:20:01
 supermount                       |     9 | 325 days 145:38:07.087554    | 36 days 18:50:54.120839 | 68 days 16:16:15
 ramfs                            |    10 | 249 days 147:43:34.143084    | 24 days 36:22:21.414308 | 32 days 16:16:15
 reiser4                          |    10 | 240 days 161:33:44.187484    | 24 days 16:09:22.418748 | 28 days 16:16:15
^^^^^^^^^^^^^
 cramfs                           |     3 | 137 days 48:48:44.087554     | 45 days 32:16:14.695851 | 58 days 16:16:15
 openpromfs                       |     5 | 130 days 81:21:15            | 26 days 16:16:15        | 43 days 16:16:15
 capifs                           |     3 | 102 days 48:48:45            | 34 days 16:16:15        | 39 days 16:16:15
 udf                              |     3 | 96 days 48:10:38             | 32 days 16:03:32.666667 | 51 days 16:16:15
 cifs                             |     3 | 81 days 48:48:45             | 27 days 16:16:15        | 35 days 16:16:15
 squashfs                         |     3 | 79 days 48:48:45             | 26 days 24:16:15        | 34 days 16:16:15
 configfs                         |     3 | 64 days 48:28:45             | 21 days 24:09:35        | 23 days 16:16:15
 ocfs2_dlmfs                      |     1 | 23 days 16:16:15             | 23 days 16:16:15        | 23 days 16:16:15
 oprofilefs                       |     1 | 23 days 16:16:08.088967      | 23 days 16:16:08.088967 | 23 days 16:16:08.088967
 unionfs                          |     1 | 22 days 16:16:15             | 22 days 16:16:15        | 22 days 16:16:15

(33 rows)

klive=>

Currently the number of reiser4 users is 35, this lists only the kernels running in real time as I write this that are running reiser4:

klive=> select f, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where last + '1 day' > 'now' group by f order by sum(uptime) desc;
f | count | sum | avg | max
----------------------------------+-------+-----------------------------+-------------------------+-------------------------

 sysfs                            |   484 | 3928 days 4530:39:11.52799  | 8 days 12:08:15.76762   | 173 days 16:16:15
 tmpfs                            |   480 | 3927 days 4501:01:21.406313 | 8 days 13:43:37.669596  | 173 days 16:16:15
 usbfs                            |   437 | 2903 days 3824:39:39.983599 | 6 days 24:11:04.485088  | 173 days 16:16:15
 ext3                             |   279 | 2053 days 2355:05:46.852916 | 7 days 17:02:36.08191   | 173 days 16:16:15
 reiserfs                         |   230 | 1949 days 2245:30:53.639863 | 8 days 21:08:13.276695  | 155 days 16:16:15
 xfs                              |    58 | 624 days 587:32:09.569396   | 10 days 28:20:12.578783 | 139 days 14:54:11
 ext2                             |    91 | 608 days 933:47:49.912624   | 6 days 26:36:47.361677  | 139 days 14:54:11
 binfmt_misc                      |    58 | 600 days 619:39:09.233036   | 10 days 18:57:34.297121 | 173 days 16:16:15
 nfsd                             |    32 | 546 days 420:58:29.048813   | 17 days 14:39:19.657775 | 89 days 16:16:15
 jfs                              |    10 | 546 days 143:40:39          | 54 days 28:46:03.90     | 160 days 16:16:15
 nfs                              |    28 | 518 days 380:48:49.210675   | 18 days 25:36:01.757524 | 64 days 16:16:15
 rpc_pipefs                       |    25 | 473 days 347:10:34.048813   | 18 days 35:58:01.361953 | 173 days 16:16:15
 autofs                           |    17 | 211 days 218:50:13.090969   | 12 days 22:45:18.417116 | 54 days 16:16:15
 selinuxfs                        |    10 | 204 days 105:21:47          | 20 days 20:08:10.70     | 173 days 16:16:15
 devfs                            |     3 | 193 days 48:35:34.022351    | 64 days 24:11:51.340784 | 155 days 16:16:15
 ntfs                             |    74 | 93 days 459:41:24.880971    | 1 day 12:22:27.092986   | 30 days 16:16:15
 vfat                             |    64 | 83 days 437:49:49.197282    | 1 day 13:57:57.956208   | 37 days 16:16:15
 iso9660                          |    16 | 90 days 126:40:31.377898    | 5 days 22:55:01.961119  | 75 days 13:45:31
 smbfs                            |     7 | 72 days 122:03:20           | 10 days 24:17:37.142857 | 27 days 16:16:15
 reiser4                          |    35 | 56 days 314:41:54.194701    | 1 day 23:23:28.976991   | 24 days 16:16:15
^^^^^^^^^^^^^^^
 udf                              |     1 | 51 days 16:16:15            | 51 days 16:16:15        | 51 days 16:16:15
 ramfs                            |    11 | 45 days 82:10:59.027323     | 4 days 09:39:10.820666  | 30 days 16:16:15
 configfs                         |     2 | 24 days 25:15:41            | 12 days 12:37:50.50     | 22 days 16:06:15
 squashfs                         |     1 | 21 days 16:16:15            | 21 days 16:16:15        | 21 days 16:16:15
 subfs                            |     2 | 20 days 18:08:18.063223     | 10 days 09:04:09.031611 | 20 days 16:15:43.063223
 cifs                             |     6 | 15 days 66:54:31            | 2 days 23:09:05.166667  | 11 days 16:16:15
 openpromfs                       |     3 | 14 days 61:21:33.08448      | 4 days 36:27:11.02816   | 11 days 16:09:12.08448
 fuse                             |     4 | 10 days 34:01:09            | 2 days 20:30:17.25      | 8 days 16:16:15
 debugfs                          |     6 | 9 days 46:25:01             | 1 day 19:44:10.166667   | 8 days 16:16:15
 minix                            |     1 | 8 days 16:16:15             | 8 days 16:16:15         | 8 days 16:16:15
 hfsplus                          |     1 | 1 day 12:20:26              | 1 day 12:20:26          | 1 day 12:20:26
 ufs                              |     7 | 10:55:17                    | 01:33:36.714286         | 04:18:01
 supermount                       |     2 | 03:14:39                    | 01:37:19.50             | 01:52:35
 securityfs                       |     1 | 00:38:07                    | 00:38:07                | 00:38:07

(34 rows)

klive=>

And these are the respective kernel versions:

klive=> select kernel_release, count(*), sum(uptime), avg(uptime), max(uptime) from fs natural inner join klive where last + '1 day' > 'now' and f = 'reiser4' group by kernel_release order by sum(uptime) desc;
kernel_release | count | sum | avg | max
-------------------------+-------+------------------------+------------------------+------------------------

 2.6.17-ckpp1            |     1 | 24 days 16:16:15       | 24 days 16:16:15       | 24 days 16:16:15
 2.6.17-beyond2          |     3 | 9 days 61:26:13.021287 | 3 days 20:28:44.340429 | 6 days 16:13:52.021287
 2.6.15-gentoo-r1-blip-1 |     1 | 10 days 16:16:15       | 10 days 16:16:15       | 10 days 16:16:15
 2.6.16-gentoo-r12       |     1 | 6 days 16:06:15        | 6 days 16:06:15        | 6 days 16:06:15
 2.6.17-beyond1.1        |     1 | 3 days 17:41:00        | 3 days 17:41:00        | 3 days 17:41:00
 2.6.16-reiser4-r7       |     1 | 2 days 08:58:55.041969 | 2 days 08:58:55.041969 | 2 days 08:58:55.041969
 2.6.18-rc1-mm2-non14    |     1 | 1 day 21:35:33         | 1 day 21:35:33         | 1 day 21:35:33
 2.6.17-no4              |     1 | 1 day 04:56:21         | 1 day 04:56:21         | 1 day 04:56:21
 2.6.17-beyond2.1        |     2 | 24:53:40               | 12:26:50               | 23:01:05
 2.6.16-beyond4.1        |     1 | 22:51:05               | 22:51:05               | 22:51:05
 2.6.17-PRX5.1           |     2 | 18:16:52               | 09:08:26               | 18:16:52
 2.6.18-rc1-mm2          |     1 | 18:16:52               | 18:16:52               | 18:16:52
 2.6.17-gentoo-r3        |     3 | 11:02:11               | 03:40:43.666667        | 07:05:39
 2.6.17-rc1-no2          |     2 | 10:24:03.045926        | 05:12:01.522963        | 07:05:39
 2.6.17-beyond1          |     1 | 09:02:04               | 09:02:04               | 09:02:04
 2.6.17-beyond-git2      |     1 | 09:02:04               | 09:02:04               | 09:02:04
 2.6.16-beyond3          |     1 | 07:05:39               | 07:05:39               | 07:05:39
 2.6.17-no1              |     3 | 06:33:04               | 02:11:01.333333        | 03:18:25
 2.6.17-beyond2-r2       |     1 | 05:22:31               | 05:22:31               | 05:22:31
 2.6.17-gentoo-r4        |     1 | 04:18:01               | 04:18:01               | 04:18:01
 2.6.16-beyond4          |     3 | 03:19:24.068344        | 01:06:28.022781        | 03:09:24.068344
 2.6.17-reiser4-r4       |     2 | 00:57:37.017176        | 00:28:48.508588        | 00:57:37.017176
 2.6.16-reiser4-r11      |     1 | 00:00:00               | 00:00:00               | 00:00:00

(23 rows)

klive=>

This is the whole list of kernel_releases that ever run reiser4:
kernel_release | count | sum | avg | max
-------------------------------------+-------+-------------------------------+--------------------------+--------------------------

 2.6.15-gentoo-r1                    | 15726 | 12498 days 94186:27:26.881778 | 25:03:46.27794           | 139 days 14:54:11
 2.6.15.6                            |   262 | 6192 days 4014:29:48.915128   | 23 days 30:31:43.011126  | 114 days 16:16:15
 2.6.16-gentoo-r7                    |  5893 | 3852 days 34120:33:11.729484  | 21:28:40.005384          | 35 days 16:11:20.052466
 2.6.16-gentoo-r9                    |  7349 | 3505 days 39841:34:06.868008  | 16:52:04.132109          | 45 days 16:16:15
 2.6.14-gentoo-r5                    |  3363 | 3774 days 25871:49:28.023204  | 1 day 10:37:34.22778     | 85 days 16:16:15
 2.6.16-rc5                          |  1309 | 3909 days 10877:31:17.420649  | 2 days 31:58:47.179084   | 42 days 16:16:15
 2.6.15-gentoo-r7                    |  4183 | 2741 days 25823:28:50.761387  | 21:53:59.811322          | 48 days 16:10:16.072702
 2.6.16-gentoo-r8                    |  3300 | 2737 days 21674:06:24.528491  | 26:28:24.116524          | 55 days 16:16:15
 2.6.16-gentoo-r3                    |  4780 | 2369 days 26106:57:10.720939  | 17:21:22.558728          | 91 days 16:16:15
 2.6.16-gentoo-r2                    |  2303 | 2600 days 17412:40:17.134521  | 1 day 10:39:21.449038    | 89 days 16:16:15
 2.6.15.1                            |  1463 | 2818 days 10447:34:09.563652  | 1 day 29:22:10.177419    | 136 days 16:16:15
 2.6.16-gentoo-r1                    |  3608 | 2233 days 21934:44:07.471825  | 20:55:59.270364          | 64 days 16:03:18.045239
 2.6.16                              |  1165 | 2508 days 9354:44:26.049247   | 2 days 11:41:48.382875   | 53 days 16:06:15
 2.6.16-gentoo-r6                    |  2075 | 2300 days 13980:21:11.200159  | 1 day 09:20:23.745157    | 58 days 16:16:15
 2.6.15-gentoo-r5                    |  3321 | 1909 days 17474:22:48.504908  | 19:03:27.458147          | 104 days 16:15:50.041517
 2.6.17-gentoo                       |  3291 | 1753 days 19508:59:15.100422  | 18:42:42.97633           | 30 days 16:16:15
 2.6.17-rc1                          |   785 | 2053 days 8234:01:41.533576   | 2 days 25:15:21.912782   | 69 days 16:16:15
 2.6.15                              |  2206 | 1673 days 14112:53:10.800042  | 24:35:55.571532          | 45 days 16:16:15
 2.6.15-gentoo                       |  3284 | 1260 days 20322:09:18.149661  | 15:23:47.453761          | 26 days 16:16:15
 2.6.16-beyond4                      |  2123 | 1268 days 15252:17:49.21025   | 21:31:07.484319          | 21 days 16:16:15
 2.6.16-archck1                      |  1330 | 1388 days 10490:30:49.827973  | 1 day 08:56:03.195359    | 36 days 16:16:15
 2.6.15.4                            |   302 | 1683 days 2958:42:00.940999   | 5 days 23:32:43.314374   | 80 days 16:16:15
 2.6.12-12mdk                        |   360 | 1630 days 3696:44:27.785247   | 4 days 22:56:07.410515   | 84 days 16:06:15
 2.6.15-ck3-r1                       |   769 | 1364 days 6171:30:41.606372   | 1 day 26:35:41.796627    | 43 days 16:16:15
 2.6.15-nitro3                       |  1709 | 1142 days 10979:07:40.254166  | 22:27:42.293888          | 31 days 16:16:15
 2.6.16-beyond3                      |  1806 | 1030 days 13334:56:02.827243  | 21:04:17.011532          | 22 days 16:14:05.037859
 2.6.15-ck7                          |   637 | 1357 days 5372:37:06.322867   | 2 days 11:33:41.07743    | 31 days 16:16:15
 2.6.9-22.0.1.ELsmp                  |    13 | 1571 days 211:31:15           | 120 days 36:34:42.692308 | 173 days 16:16:15
 2.6.15-archck                       |  2579 | 865 days 15948:06:33.240877   | 14:14:00.478186          | 28 days 16:16:15
 2.6.15-1-k7                         |   164 | 1376 days 2179:27:47.995174   | 8 days 22:39:18.95119    | 35 days 16:16:15
 2.6.16-ck11                         |  1402 | 1016 days 10372:57:34.350942  | 24:47:27.542333          | 39 days 16:14:38.079596
 2.6.16-gentoo                       |  1927 | 942 days 11516:24:54.920025   | 17:42:30.853617          | 47 days 16:16:15
 2.6.14.5                            |   169 | 1318 days 1681:33:11          | 7 days 29:07:17.816568   | 86 days 16:16:15
 2.6.16-ck10                         |   814 | 1130 days 5987:58:35.725072   | 1 day 16:40:23.483692    | 40 days 16:16:15
 2.6.14-hardened-r5                  |   636 | 1194 days 4394:03:05.986415   | 1 day 27:57:55.76413     | 52 days 16:16:15
 2.6.12-gentoo-r10                   |   300 | 1213 days 3756:26:28.092538   | 4 days 13:33:41.293642   | 89 days 16:16:15
 2.6.17                              |  1038 | 918 days 10628:20:44.642394   | 31:27:52.875378          | 32 days 16:04:49.04156
 2.6.16-ck3                          |  1043 | 1053 days 6467:18:14.426646   | 1 day 06:25:50.809613    | 54 days 16:16:15
 2.6.16.1                            |   314 | 1161 days 3632:43:26.24459    | 3 days 28:18:28.937085   | 67 days 16:16:15
 2.6.15-archck2                      |  1738 | 727 days 12587:25:00.214091   | 17:16:53.751562          | 14 days 16:16:15
 2.6.13-gentoo-r5                    |   880 | 833 days 9688:10:28.922616    | 33:43:38.896503          | 34 days 16:16:15
 2.6.15-ck2pwr                       |    92 | 1127 days 1357:25:27.655134   | 12 days 20:45:16.604947  | 52 days 16:16:15

[..]

too long to list them all.

Quoted 2 lines

"patchup" is used by (tens of) thousands of computers in governmental laboratories the US "national security" depends upon.

I don't see any ext4 in the klive logs, it'd be nice if I would know how to track your ext4 patchset too to make a more accurate comparison. We can't compare the above numbers to the klive ones, the above is the interpolation of the reiser4 and klive userbase only. While if I could detect the ext4 "patchup" it would be feasible to attempt a comparison.

Disclaimer: the data should not be trusted, use it at your risk. Furthermore it may not represent a reliable sample, because the fs are only submitted at the first connection with the server after reboot, so if reiser4 or fuse are mounted later on (normally after an average of 5 minutes) they may not get logged in KLive. OTOH this also means that all computers showing reiser4 in the logs had it mounted since about the boot time (so if reiser4 would corrupt memory badly by just mounting nobody could reasonably hope to reach 20 days of uptime).

Hope this helps.

#265 lore raw

Hi,

On Tuesday 25 July 2006 14:35, Andrea Arcangeli wrote:
...

Quoted 5 lines

average of 5 minutes) they may not get logged in KLive. OTOH this also means that all computers showing reiser4 in the logs had it mounted since about the boot time (so if reiser4 would corrupt memory badly by just mounting nobody could reasonably hope to reach 20 days of uptime).

Well I tested Reiser4 on systems building our Linux flavour (based on the T2 framework) thus with excessive software compilation, a lot of tarball extraction, compilation of fat packages such as linux, mozilla, openoffice et.al. and huge per-package ccache directories -> zilions of small files and did not encounter any abnormal behaviour aside the yet-to-be-fixed commit storms. Performance was superiour over ext3, XFS*, JFS or Reiser3 in this workload (that is less time "wasted" for FS housekeeping and the build finished noticable earlier). Patchset was tested around 2.6.1{4,5]. But I do not have the numbers around anymore and was basically waiting for Reiser4 to make it into the kernel to schedule further testing.

You can see I'm not a blind Reiser4 follower but would rather like to see a fair chance for it.

*) XFS oopsed so often in this test that I'll never touch it again ...

Yours,

Signature · 3 lines
René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany)
            http://exactcode.de | http://t2-project.org | http://rebe.name
            +49 (0)30 / 255 897 45
#24 lore raw
In reply to Horst H. von Brand
Quoted 6 lines · Horst H. von Brand
Quoted 2 lines

Once the inodes ran out the entire system pretty much came to a screeching halt.

Get a clue by for, apply to the vendor (for the design, or at the very least for not warning unsuspecting users)?

Quoted 6 lines
Quoted 3 lines

We basically had two options, use ReiserFS, or find
another piece of software that didn't use tiny little files as its database.

Or reconfigure the filesystem with more inodes (you were willing to rebuild the filesystem in any case, so...)

The fact that _many_ UNIX filesystem formats have inode number limit (configurable only at mkfs time) doesn't make it a good design.

By the same virtue you may declare that it is okay to smoke
and drink lots of vodka. Why not? Sizable portion if population does that...

I, on the contrary, want software to impose as few limits on me as possible.
--
vda

#266 lore raw
In reply to Andrea Arcangeli

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

Hans

Quoted 7 lines · Andrea Arcangeli
reiserfs                         | 19934 | 19257 days 133294:48:27.256508 | 29:52:18.181361         | 155 days 16:16:15
ext3                             | 22077 | 16346 days 132190:42:38.862752 | 23:45:27.062502         | 173 days 16:16:15

reiser4 | 4409 | 1926 days 27479:22:27.459534 | 16:42:59.666015 | 28 days 16:16:15
^^^^^^^^^^^^^

#248 lore raw
In reply to Hans Reiser
Quoted 15 lines · Hans Reiser

Jeff, I think that a large part of what is going on is that any patch that can be read in 15 minutes gets reviewed immediately, and any patch that is worked on for 5 years and then takes a week to read gets neglected. This is true even if line for line the 1 week to read patch
is more valuable. What is more is that people know this is
irrational, but aren't able to cure it in themselves. Even I have a problem of paying too much attention to endless 5 minute emails when I know I should instead, say, read the compression plugin from beginning to end.

There is nothing about small patches that makes them better code. There is no reason we should favor them, if the developers are willing to work on something for 5 years to escape a local optimum, that is often the RIGHT thing to do.

Quoted 5 lines

It is importand that we embrace our diversity, and be happy for the strength it gives us. Some of us are good at small patches that evolve, and some are good at escaping local optimums. We all have value, both trees and grass have their place in the world.

Which is summed up quite well by:
http://en.wikipedia.org/wiki/Color_of_the_bikeshed

It seem to be a well know tendency for people to want to be involved in some way, thus keeping to much of the development cycle internal tends to generate friction.

-Russell Cattelan
[email protected]

#267 lore raw
In reply to Hans Reiser
Quoted 7 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

Hans

Yes, this was a pleasant surprise indeed :)

This useless mail from a mere user is just to testify that I've used reiser4 on my laptop since June 2004, and that, barring occasional hiccups, I never had serious problems with it. In fact I haven't lost a single bit of data despite crashes and cold shutdowns due to power outage.

I don't know if this means something or if I've just been lucky, but it seems to me that when reiser4 crashes (and sometimes it does, given its young age), it behaves very very well. That's the thing that impressed me the most (and it is really something, given for example the debacle of a mature filesystem like XFS in 2.6.17). Oh, and performance, of course ;)

I really hope that the technical difficulties that are preventing reiser4 from entering mainline will be sorted out soon.

Thanks,

Francesco (crawling back in the shadow)

Signature · 4 lines
Dr. Francesco Biscani
Dipartimento di Astronomia
Università di Padova
[email protected]
#25 lore raw
In reply to Denis Vlasenko
Quoted 2 lines · Denis Vlasenko

I, on the contrary, want software to impose as few limits on me as possible.

As long as it's choosing some limit, I'll pick the one with fewer surprises.

Signature · 1 line
Matthias Andree
#268 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

That makes the data /higly/ suspect: Most of the Linux users I know wouldn't know where to start to compile a kernel, forget about patching around. And of the minority who can, they mostly run machines in production, and won't dream of using non-distribution kernels, let alone custom patched ones.

BTW, where do you get the "twice as fast" number from?

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#350 lore raw
In reply to Theodore Tso
Quoted 2 lines · Theodore Tso

both have to do a line-by-line review, and with a promise of on-disk and ABI compatibility *forever* ---- that we do more commits in a week

"forever"? Man, that's hard-core. Is that *really* the guideline, or is it some "might as well be forever in this industry" rule like "5 years"?

#269 lore raw
In reply to Horst H. von Brand
Quoted 11 lines · Horst H. von Brand
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

That makes the data /higly/ suspect: Most of the Linux users I know wouldn't know where to start to compile a kernel, forget about patching around. And of the minority who can, they mostly run machines in production, and won't dream of using non-distribution kernels, let alone custom patched ones.

Well if you take a look at the KLive webpage you'll see that there's only 485 computers providing data. I'm not trying to knock Andrea's project, but the sample size is quite small and only really representative of people who would be inclined to read lkml, patch their kernels, etc.

Jim.

#26 lore raw
In reply to Matthias Andree
Quoted 7 lines · Matthias Andree
Quoted 2 lines · Denis Vlasenko

I, on the contrary, want software to impose as few limits on me as possible.

As long as it's choosing some limit, I'll pick the one with fewer surprises.

Running out of inodes would be pretty surprising for me.

But then, I guess it's a good thing I don't admin for a living anymore.

#271 lore raw
In reply to Horst H. von Brand

I suppose from the ones, lime me, who has production servers, but also some test system to play with.

Luigi

Quoted 13 lines · Horst H. von Brand
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

That makes the data /higly/ suspect: Most of the Linux users I know wouldn't know where to start to compile a kernel, forget about patching around. And of the minority who can, they mostly run machines in production, and won't dream of using non-distribution kernels, let alone custom patched ones.

BTW, where do you get the "twice as fast" number from?

#270 lore raw
In reply to Jim Crilly
Quoted 4 lines · Jim Crilly

Well if you take a look at the KLive webpage you'll see that there's only 485 computers providing data. I'm not trying to knock Andrea's project, but the sample size is quite small and only really representative of people who would be inclined to read lkml, patch their kernels, etc.

There are quite a few using the official distro kernels but the point remains that the KLive users are probably the ones most interested to try all new stuff and to live on the edge. They're not afraid to put the new technologies to work for them. That's probably why they're using reiser4 so much. OTOH they're probably the ones putting the fs under the most stress, a regular desktop user not capable of running klive.sh --install from the shell, would probably leave his CPU and harddisk idle most of the time too.

Being this only a sample I can't come up with absolute figures, but obviously there are more users than what is being recorded by KLive.

However the sample is not so small, in less then a year KLive logged 76740 days of uptime and 61355 reboots (or klive session restarts), and like you noticed an average of about 400-500 systems are always connected.

#249 lore raw
In reply to Russell Cattelan
Quoted 4 lines · Russell Cattelan
Quoted 3 lines · Hans Reiser

Jeff, I think that a large part of what is going on is that any patch that can be read in 15 minutes gets reviewed immediately, and any patch that is worked on for 5 years and then takes a week to read gets

[...]

Quoted 11 lines
Quoted 5 lines

It is importand that we embrace our diversity, and be happy for the strength it gives us. Some of us are good at small patches that evolve, and some are good at escaping local optimums. We all have value, both trees and grass have their place in the world.

Which is summed up quite well by:
http://en.wikipedia.org/wiki/Color_of_the_bikeshed

It seem to be a well know tendency for people to want to be involved in some way, thus keeping to much of the development cycle internal tends to generate friction.

No, I think Hans is right.

Although I should mention, Hans, that there is a really good reason to prefer the 15 minute patches. The patches that take a week are much harder to read during that week than any number of 15 minute incremental patches, because the incremental patches are already broken down into nice, small, readable, ordered chunks. And since development follows some sort of logical, orderly pattern, it can be much easier to read it that way than to try to consider the whole.

Think of it this way -- why are debuggers useful? One of the nicest thing about a debugger, especially for newbies, is the ability to step through a program a line at a time. It's the same principle -- you can understand the program state at one point in time, and the impact of one line of code, much more easily than the overall model of the program state (and all of its edge cases), or the impact of several hundred (thousand? tens of thousands?) lines of code.

So while I don't blame the Namesys team for putting off inclusion till it's done, I also can't really blame the kernel guys for not wanting to read it, especially if it's revolutionary. Revolutionary ideas are hard to grasp, and it's not their fault.

I mean, if revolutionary ideas were easy, why didn't you write Reiser4 for a system like, say, Tunes? (http://tunes.org/) Say what you will, but there are ways of doing fast filesystems which don't require that said filesystems be written in kernel C. Consider this:

http://www.cs.utah.edu/flux/oskit/

If I understand that right, it's a mechanism for writing kernel code in languages like "Java, Lisp, Scheme, or ML"...

If we could all grasp every single good (best) idea from every corner of software engineering, and write completely new software (including the OS) using those ideas, we could potentially replace all existing software in something like 3-5 years with software which has none of the problems ours does now. We'd never have inflexibility, insecurity, instability, user interface issues... Never have to worry about getting software out the door (it'd be so fast to develop), but always have it designed the right way the first time, yet be able to rearrange it completely with only 5-10 line patches.

So it's not always the computer hardware that's the limitation. Often it's our hardware as well. Human beings usually aren't equipped to be able to grok the whole universe all at once. If we were... see above.

#32 lore raw
In reply to Horst H. von Brand
Quoted 2 lines · Horst H. von Brand

18GiB = 18 million KiB, you do have a point there. But 40 million files on that, with some space to spare, just doesn't add up.

Right, ok...

Here's a quick check of my box. I've explicitly stated which root-level directories to search, to avoid nfs mounts, chrooted OSes, and virtual filesystems like /proc and /sys.

elite ~ # find /bin/ /boot/ /dev/ /emul/ /etc/ /home /lib32 /lib64 /opt /root /sbin /tmp /usr /var -type f -size 1 | wc -l
246127

According to the "find" manpage:

-size n[bckw]
File uses n units of space. The units are 512-byte blocks by default or if b' follows n, bytes if c' follows n, kilobytes
if k' follows n, or 2-byte words if w' follows n. The size
does not count indirect blocks, but it does count blocks in sparse files that are not actually allocated.

And I certainly didn't plan it that way. And this is my desktop box, and I'm just one user. Most of the space is taken up by movies.

And yet, I have almost 250k files at the moment whose size is less than 512 bytes. And this is a normal usage pattern. It's not hard to imagine something prone to creating lots of tiny files, combined with thousands of users, easily hitting some 40 mil files -- and since none of them are movies, it could fit in 18 gigs.

I mean, just for fun:

elite ~ # find /bin/ /boot/ /dev/ /emul/ /etc/ /home /lib32 /lib64 /opt /root /sbin /tmp /usr /var | wc -l
866160

It may not be a good idea, but it's possible. And one of the larger reasons it's not a good idea is that most filesystems can't handle it. Kind of like how BitTorrent is a very bad idea on dialup, but a very good idea on broadband.

#33 lore raw
Quoted 4 lines · David Masover
Quoted 2 lines · Horst H. von Brand

18GiB = 18 million KiB, you do have a point there. But 40 million files on that, with some space to spare, just doesn't add up.

if you have 18 million KiB and each file is a single block (512 Bytes = 0.5 Kib) then assuming zero overhead you could fit 18 Million KiB / 0.5 KiB = 36 Million files on the drive.

thus being scheptical about 40 million files on a 18G drive.

this is only possible if you are abel to have multiple files per 512 byte block.

David Lang

Quoted 41 lines

Right, ok...

Here's a quick check of my box. I've explicitly stated which root-level directories to search, to avoid nfs mounts, chrooted OSes, and virtual filesystems like /proc and /sys.

elite ~ # find /bin/ /boot/ /dev/ /emul/ /etc/ /home /lib32 /lib64 /opt /root /sbin /tmp /usr /var -type f -size 1 | wc -l
246127

According to the "find" manpage:

-size n[bckw]
File uses n units of space. The units are 512-byte blocks by default or if b' follows n, bytes if c' follows n, kilobytes
if k' follows n, or 2-byte words if w' follows n. The size
does not count indirect blocks, but it does count blocks in sparse files that are not actually allocated.

And I certainly didn't plan it that way. And this is my desktop box, and I'm just one user. Most of the space is taken up by movies.

And yet, I have almost 250k files at the moment whose size is less than 512 bytes. And this is a normal usage pattern. It's not hard to imagine something prone to creating lots of tiny files, combined with thousands of users, easily hitting some 40 mil files -- and since none of them are movies, it could fit in 18 gigs.

I mean, just for fun:

elite ~ # find /bin/ /boot/ /dev/ /emul/ /etc/ /home /lib32 /lib64 /opt /root /sbin /tmp /usr /var | wc -l
866160

It may not be a good idea, but it's possible. And one of the larger reasons it's not a good idea is that most filesystems can't handle it. Kind of like how BitTorrent is a very bad idea on dialup, but a very good idea on broadband.

#255 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

to use as his default. Now that we paid the 5 year development price tag to get everything as plugins, we can now upgrade in littler pieces than any other FS. Hmm, I need a buzz phrase, its not extreme programming, maybe "moderate programming". Does that sound exciting to

Hah! No, it doesn't sound exciting.

Plugins don't work well either, not as a marketing concept. People have had so many bad experiences with plugins, and they're only ever visible when you have a bad experience. Think about it -- missing plugin (so you have to download it),

On the other hand, it works for WordPress. My day job is work on a plugin for WordPress. Not including a link because I feel dirty for having to work with PHP...

Fluid programming? If you build a solution from the bottom up with gravel or large rocks, you leave gaps that are hard to fill without ripping off the top layer and redoing it. But if you can do fluid programming, your program just flows around any obstacle, and into every crack / between every space (metaphor for new customer requirements)...

#34 lore raw
In reply to David Lang
Quoted 8 lines · David Lang
Quoted 5 lines · David Masover
Quoted 3 lines · Horst H. von Brand

18GiB = 18 million KiB, you do have a point there. But 40 million files on
that, with some space to spare, just doesn't add up.

Quoted 8 lines

if you have 18 million KiB and each file is a single block (512 Bytes = 0.5 Kib) then assuming zero overhead you could fit 18 Million KiB / 0.5 KiB = 36 Million files on the drive.

thus being scheptical about 40 million files on a 18G drive.

this is only possible if you are abel to have multiple files per 512 byte block.

I believe Reiser4 does this. Does ext3? I know I heard somewhere in this thread that you can't set the blocksize lower than 1k...

#256 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to David Masover
Quoted 8 lines · David Masover
Quoted 4 lines · Hans Reiser

to use as his default. Now that we paid the 5 year development price tag to get everything as plugins, we can now upgrade in littler pieces than any other FS. Hmm, I need a buzz phrase, its not extreme programming, maybe "moderate programming". Does that sound exciting to

Quoted 8 lines

Hah! No, it doesn't sound exciting.

Plugins don't work well either, not as a marketing concept. People have had so many bad experiences with plugins, and they're only ever visible when you have a bad experience. Think about it -- missing plugin (so you have to download it),

marketing?
</joke mode on>
if you do not like the word "plugin" why don't you suggest some alternative? like "modules"?
</joke mode off>

Seriously, please leave out this kind of marketing. The plugin concept in reiser4 is probably the most interessant feature I filesystem some users could need.

Quoted 6 lines

Fluid programming? If you build a solution from the bottom up with gravel or large rocks, you leave gaps that are hard to fill without ripping off the top layer and redoing it. But if you can do fluid programming, your program just flows around any obstacle, and into every crack / between every space (metaphor for new customer requirements)...

probably I do not know english enought well to appraciate this metaphor and to understand what it means.

#258 lore raw
In reply to Hans Reiser
Quoted 9 lines · Hans Reiser

Jeff, I think that a large part of what is going on is that any patch that can be read in 15 minutes gets reviewed immediately, and any patch that is worked on for 5 years and then takes a week to read gets neglected. This is true even if line for line the 1 week to read patch
is more valuable. What is more is that people know this is
irrational, but aren't able to cure it in themselves. Even I have a problem of paying too much attention to endless 5 minute emails when I know I should instead, say, read the compression plugin from beginning to end.

Well, the problem is that someone actually went through the effort of doing the week-long reviews of your code, you flamed him every time he suggested that there was need for improvement, instead of saying thanks for all the hard work and implementing the needed fixes.

Not exactly the right way to make people fond of reviewing your code. Getting flamed by someone having a hard time taking criticism after spending 5 minutes to review a patch is bearable, after all, what's 5 minutes, right? Flaming someone that has put down a week or two of hard work on review is just disrespectful.

[snip]

Regards: David Weinehall

Signature · 3 lines
 /) David Weinehall <[email protected]> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/
#27 lore raw
In reply to David Masover
Quoted 10 lines · David Masover
Quoted 7 lines · Matthias Andree
Quoted 2 lines · Denis Vlasenko

I, on the contrary, want software to impose as few limits on me as possible.

As long as it's choosing some limit, I'll pick the one with fewer surprises.

Running out of inodes would be pretty surprising for me.

No offense: Then it was a surprise for you because you closed your eyes and didn't look at df -i or didn't have monitors in place.

There is no way to ask how many files with particular hash values you can still stuff into a reiserfs 3.X. There, you're running into a brick wall that only your forehead will "see" when you touch it.

Of course, different sites have different needs and if you need gazillions of inodes or file names, you may see trouble.

But the assertion that some backup was the cause for inode exhaustion on ext? is not very plausible since hard links do not take up inodes, symlinks are not backups and everything else requires disk blocks. So, since reformatting ext2/ext3 to one inode per block is possible (regardless of disk capacity), I see no way how a reformatted file system might run out of inodes before it runs out of blocks.

Signature · 1 line
Matthias Andree
#28 lore raw
Quoted 6 lines · Matthias Andree

But the assertion that some backup was the cause for inode exhaustion on ext? is not very plausible since hard links do not take up inodes, symlinks are not backups and everything else requires disk blocks. So, since reformatting ext2/ext3 to one inode per block is possible (regardless of disk capacity), I see no way how a reformatted file system might run out of inodes before it runs out of blocks.

OK; ext2/ext3 require 1k blocks, but still you need heaps of files < 1k to run out of inodes without running of space.

Signature · 1 line
Matthias Andree
#272 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

I doubt Andreas values have much value.

According to the numbers on the klive website, Gentoo has 47 times as many users as SuSE (sic).

Considering that klive is offered by someone working for SuSE, the Gentoo project could make a good news article ("Numbers by a SuSE developer confirm Gentoo has 47 times the market share of Suse.") out of it. ;-)

Quoted 1 line

Hans

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#29 lore raw
In reply to Matthias Andree

Hello,

I know thats not relevant for the discussion, but I wanted to share my experiences anyway (to emphasis how important df-i monitoring on smaller filesystems is):

Quoted 6 lines · Matthias Andree

But the assertion that some backup was the cause for inode exhaustion on ext? is not very plausible since hard links do not take up inodes, symlinks are not backups and everything else requires disk blocks. So, since reformatting ext2/ext3 to one inode per block is possible (regardless of disk capacity), I see no way how a reformatted file system might run out of inodes before it runs out of blocks.

Well I had actually the problem on a tmpfs where I had too many zero byte files...

Gruss
Bernd

Quoted 1 line
#179 lore raw
In reply to Hans Reiser

Hi!

Quoted 5 lines
Quoted 4 lines

of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Well, too bad, if reiser3 is so broken it needs on-disk-format-change, then I guess doing that change is the right thing to do...
Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#273 lore raw
In reply to Adrian Bunk
Quoted 8 lines · Adrian Bunk
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

I doubt Andreas values have much value.

I think nobody ever pretended them to have much value, they only need to have some minor value to be useful. And as far as I can tell no better source of data about the kernel testing userbase exists as of today.

Quoted 7 lines

According to the numbers on the klive website, Gentoo has 47 times as many users as SuSE (sic).

Considering that klive is offered by someone working for SuSE, the Gentoo project could make a good news article ("Numbers by a SuSE developer confirm Gentoo has 47 times the market share of Suse.") out of it. ;-)

Oh well, you misunderstood KLive completely. Please read again the top of the page: "this project aims to provide kernel live feedback about the usage of every different Linux Kernel version". Where did I ever mention anything about distributions?

There's absolutely no way to tell which distribution is running. I only can tell which _kernel_ is running. It could be 100% of the mainline users are running mainline on top of SUSE distro, or on top of Gentoo, or on top of Ubuntuu, there's absolutely no way to know about the distro. Nobody could ever make a claim like the above by using the KLive data.

To make an example I run mainline myself on top of opensuse 10.1, and I get rightfully accounted as a "mainline" and not as a "SUSE".

All you can tell is that there are many more people running KLive in combination with Gentoo kernels than with SUSE kernels. But you can't tell anything about what kind of distribution is running. One could guess the Gentoo kernels run on top of a Gentoo userland, but for the mainline ones you really can't tell, they could be slackware but they could be any other distro too.

Also note that the cpushare.com domain is not related to any distro, so the fact I also do consulting for SUSE is irrelevant to KLive or any other dealings at the cpushare.com domain. As long there is people running KLive and browsing the website like now, it means somebody thinks it's useful and so I keep delivering the service.

As of now, the Gentoo and mainline kernel users are the ones providing most of the feedback.

Perhaps I should have filed a patent on KLive too just to make you happy, right?

#274 lore raw
Quoted 8 lines · On Wed, Jul 26, 2006 at 03:29:57PM +0200, [email protected] wrote:
Quoted 7 lines · Adrian Bunk
Quoted 4 lines · Hans Reiser

Wow, I would never have guessed our market share was that high as 1/5th of ext3. I mean, you can't even get a distro which allows you to install onto reiser4 without hours of work so far as I know. I guess there are people who really do care about twice as fast.

Quoted 7 lines
Quoted 1 line

I doubt Andreas values have much value.

I think nobody ever pretended them to have much value, they only need to have some minor value to be useful. And as far as I can tell no better source of data about the kernel testing userbase exists as of today.
...

Hans said "Wow, I would never have guessed our market share was that high as 1/5th of ext3."

And estimate of the marked share based on klive data is simply wrong. That was my point.

My distribution example was just one example of what happens when you wrongly try to estimate a marked share based on klive data (and it seems you missed the smiley after the last paragraph of my email).

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#250 lore raw
In reply to David Masover
Quoted 11 lines · David Masover

Although I should mention, Hans, that there is a really good reason to prefer the 15 minute patches. The patches that take a week are much harder to read during that week than any number of 15 minute incremental patches, because the incremental patches are already broken down into nice, small, readable, ordered chunks. And since development follows some sort of logical, orderly pattern, it can be much easier to read it that way than to try to consider the whole.

No, I disagree, if the code is well commented, it is easier to read the whole thing at the end when it has its greatest coherence and refinement. A problem with Reiser4 is that its core algorithms are simply complex. We pushed the envelope in multiple areas all at once. Benchmarks don't always suggest simple algorithms are the ones that will be highest performance. Tree algorithms are notorious in the database industry for being simple on web pages but complex as code.

Some people program in small increments, some program things that require big increments of change, both kinds of people are needed.

#275 lore raw
In reply to Adrian Bunk
Quoted 2 lines · Adrian Bunk

My distribution example was just one example of what happens when you wrongly try to estimate a marked share based on klive data (and it seems

You're again wrong generalizing. With KLive I can attempt to estimate market share of _kernel_ code (what Hans did), but you can't attempt to estimate market share of userland (what you did). It won't be a smile at the end making your statement right.

The estimate market share of kernel code may not be accurate but it has some minor significance.

#257 lore raw
In reply to David Masover
Quoted 9 lines · David Masover
Quoted 5 lines · Hans Reiser

to use as his default. Now that we paid the 5 year development price tag to get everything as plugins, we can now upgrade in littler pieces than any other FS. Hmm, I need a buzz phrase, its not extreme programming, maybe "moderate programming".

This phrase was a bit tongue-in-cheek.

Quoted 6 lines
Quoted 3 lines

Does that sound exciting to

#276 lore raw
In reply to [email protected]
Quoted 8 lines · On Wed, Jul 26, 2006 at 04:28:54PM +0200, [email protected] wrote:
Quoted 2 lines · Adrian Bunk

My distribution example was just one example of what happens when you wrongly try to estimate a marked share based on klive data (and it seems

You're again wrong generalizing. With KLive I can attempt to estimate market share of _kernel_ code (what Hans did), but you can't attempt to estimate market share of userland (what you did). It won't be a smile at the end making your statement right.

No, you can't estimate the market share of kernel code based on klive data.

Someone said in this thread 'is used by (tens of) thousands of computers in governmental laboratories the US "national security" depends upon.'

klive will never get any feedback from such users.

But the computer freak using Gentoo and always trying the latest things like reiser4 is relatively likely to also use klive.

Quoted 2 lines

The estimate market share of kernel code may not be accurate but it has some minor significance.

You are able to say that based on klive data, reiser4 has at least 35 users in the world.

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#277 lore raw
Quoted 2 lines · Adrian Bunk

Someone said in this thread 'is used by (tens of) thousands of computers in governmental laboratories the US "national security" depends upon.'

There can be people using reiser4 behind the firewall too, what's the point? IIRC US .gov even sponsored part of reiser4 development, how do you know they're not testing it too?

You don't believe KLive has any relation to reality, but you have no way to proof your claim. JFYI: all statistics only take a sample of the larger space, the whole point of having a statistic is because you can't measure the total. The smaller the sample compared to the total, the less the stats are accurate, but they still have some statistical significance. And one can always hope that KLive will grow larger.

Last but not the least defining gentoo users as freaks isn't very nice from your part. For all new startups they're the ideal userbase to have, and they do a great deal of good work by testing all new stuff and they help speeding up innovation. Infact I think having a sample of what those brave users run is more important than the rest, the rest usually follows the ones living on the edge eventually.

#278 lore raw
Quoted 7 lines · On Wed, Jul 26, 2006 at 06:06:04PM +0200, [email protected] wrote:
Quoted 2 lines · Adrian Bunk

Someone said in this thread 'is used by (tens of) thousands of computers in governmental laboratories the US "national security" depends upon.'

There can be people using reiser4 behind the firewall too, what's the point? IIRC US .gov even sponsored part of reiser4 development, how do you know they're not testing it too?

I don't know.

The klive data might be inaccurate in any direction.

In this case I'd suspect an inaccuraty in one specific direction.

Quoted 6 lines

You don't believe KLive has any relation to reality, but you have no way to proof your claim. JFYI: all statistics only take a sample of the larger space, the whole point of having a statistic is because you can't measure the total. The smaller the sample compared to the total, the less the stats are accurate, but they still have some statistical significance. And one can always hope that KLive will grow larger.

Size isn't everything.
The problem is getting an accurate sample of data.

Let me try to make the problem clearer by making an example:

Consider you want to know the ratio between housewifes and women with a job for women aged between 30 and 40.

Consider you get your data by asking women in shopping malls between 10 and 11 o'clock in the morning on workdays.

Do you understand why the result might be quite different from the actual ratio?

Do you understand why asking a million women in shopping malls between 10 and 11 o'clock in the morning on workdays wouldn't make the data better?

And do you know the Linux Counter at [1]?

I remember several years ago Debian had some default setting in some package to send reports there.

smail was at that times the default Debian MTA, and therefore also the most popular MTA at the Linux counter...

Quoted 3 lines

Last but not the least defining gentoo users as freaks isn't very nice from your part. For all new startups they're the ideal userbase to have, and they do a great deal of good work by testing all new stuff

Please read carefully what I said.

I did NOT say:
"All Gentoo users are freaks."

It was more (implicitely) in the direction:
"Many freaks use Gentoo."

The latter is IMHO not that far from reality.

And "freak" wasn't meant negative.

Quoted 3 lines

and they help speeding up innovation. Infact I think having a sample of what those brave users run is more important than the rest, the rest usually follows the ones living on the edge eventually.

The majority of user will use the filesystem their distribution offers as default, no matter what people living on the edge today will use...

cu
Adrian

[1] http://counter.li.org/

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#279 lore raw
In reply to [email protected]
Quoted 4 lines · On Wed, Jul 26, 2006 at 06:06:04PM +0200, [email protected] wrote:

JFYI: all statistics only take a sample of the larger space, the whole point of having a statistic is because you can't measure the total. The smaller the sample compared to the total, the less the stats are accurate

Definitely not true in general. If I wanted to know the gender ratio at the latest OLS I'd take the results from a sample of a dozen chosen randomly over the results from a sample of hundreds all taken from the men's room.

For exactly the same quality of sampling, yes, the larger the better, but the point of diminishing returns comes pretty quickly. So given limited resources it's probably more important to work on the quality of the sample rather than on its size....

--b.

#280 lore raw
Quoted 10 lines · J. Bruce Fields
Quoted 4 lines · On Wed, Jul 26, 2006 at 06:06:04PM +0200, [email protected] wrote:

JFYI: all statistics only take a sample of the larger space, the whole point of having a statistic is because you can't measure the total. The smaller the sample compared to the total, the less the stats are accurate

Definitely not true in general. If I wanted to know the gender ratio at the latest OLS I'd take the results from a sample of a dozen chosen randomly over the results from a sample of hundreds all taken from the men's room.

Well, your example is perhaps the worst one since you wouldn't be decreasing the quality of your stats very much by only doing the sample in the men's room ;). I guess you meant the woman's room.

Quoted 4 lines

For exactly the same quality of sampling, yes, the larger the better, but the point of diminishing returns comes pretty quickly. So given limited resources it's probably more important to work on the quality of the sample rather than on its size....

No matter how you see it, the larger the better (in the worst case it won't make a difference). Certainly if I could work on the quality, that would be more important than adding 1 more user. But I can't work on the quality.

#251 lore raw
In reply to Hans Reiser
Quoted 21 lines · Hans Reiser
Quoted 11 lines · David Masover

Although I should mention, Hans, that there is a really good reason to prefer the 15 minute patches. The patches that take a week are much harder to read during that week than any number of 15 minute incremental patches, because the incremental patches are already broken down into nice, small, readable, ordered chunks. And since development follows some sort of logical, orderly pattern, it can be much easier to read it that way than to try to consider the whole.

No, I disagree, if the code is well commented, it is easier to read the whole thing at the end when it has its greatest coherence and refinement. A problem with Reiser4 is that its core algorithms are simply complex. We pushed the envelope in multiple areas all at once. Benchmarks don't always suggest simple algorithms are the ones that will be highest performance. Tree algorithms are notorious in the database industry for being simple on web pages but complex as code.

Quoted 3 lines

Some people program in small increments, some program things that require big increments of change, both kinds of people are needed.

FWIW I think both points are valid.

Personally I find it difficult to completely stop what I'm doing and spend several days studying a large patch/complete new subsystem.

But on the other hand it's important that code that goes in the kernel gets a proper review, I think we all come to rely on the gatekeepers job of making sure any given part of the kernel does not destabilize to the point were it interferes with our individual areas of development.

So exactly what level of granularity is appropriate for changes? Well that should probably be left of to the gatekeeper for each particular area. In the case of filesystems generic vfs changes obviously need to be small and easy to digest, and more importantly easy to bisect regressions. The core of the file system can be left to the person who can best manage the code, but hopefully that person applies a reasonable of granularity to the changes, thus allowing non familiar developers to at least keep up and possibly make helpful suggestions.

If we look at the current "beliefs" surrounding XFS you can see the affects of a code base that did not have an incremental development with regards to linux anyways. Ok so ya XFS was a close sourced IRIX FS for the first 8 years of it's life, and even once the Linux project started there was another year or so encumbrance investigation and cleaning before legal would sign off on its release.
So to this day the major hang up with XFS seems to be "it's to complex because it has x thousands lines of code". Hell by that argument Linux has way more lines of code than IRIX could ever hope to have and is therefore Linux is more complex than IRIX and to hard to understand. :-) Fortunately many developers (especially ones that have worked on other OS's) do not use "wc -l" as a tool to measure code quality/readability/complexity.
XFS unfortunately will probably always suffer from skepticism since it did not grow up in in Linux.

Guess it's sort of like adopting a 8 year old child vs a new born, hard to tell what happened in first 8 years.

-Russell Cattelan
[email protected]

#30 lore raw
In reply to Bernd Eckenfels
Quoted 11 lines · Bernd Eckenfels
Quoted 6 lines · Matthias Andree

But the assertion that some backup was the cause for inode exhaustion on ext? is not very plausible since hard links do not take up inodes, symlinks are not backups and everything else requires disk blocks. So, since reformatting ext2/ext3 to one inode per block is possible (regardless of disk capacity), I see no way how a reformatted file system might run out of inodes before it runs out of blocks.

Well I had actually the problem on a tmpfs where I had too many zero byte files...

Yes, I once ran out of inodes because logrotate kept rotating and compressing already compressed and empty logfiles. I can't remember how many seconds it took me to add 'df -i' to our monitoring system.

This, however, was not a feature of the software. I assume.

So, any company that considers the remote possibility of seeking a $250,000 solution, where the alternative is to buy a 36GB hard drive, please give me a call.

Cheers,
Buddy

#281 lore raw
In reply to [email protected]
Quoted 21 lines · On 7/27/06, [email protected] <[email protected]> wrote:
Quoted 10 lines · J. Bruce Fields
Quoted 4 lines · On Wed, Jul 26, 2006 at 06:06:04PM +0200, [email protected] wrote:

JFYI: all statistics only take a sample of the larger space, the whole point of having a statistic is because you can't measure the total. The smaller the sample compared to the total, the less the stats are accurate

Definitely not true in general. If I wanted to know the gender ratio at the latest OLS I'd take the results from a sample of a dozen chosen randomly over the results from a sample of hundreds all taken from the men's room.

Well, your example is perhaps the worst one since you wouldn't be decreasing the quality of your stats very much by only doing the sample in the men's room ;). I guess you meant the woman's room.

Quoted 4 lines

For exactly the same quality of sampling, yes, the larger the better, but the point of diminishing returns comes pretty quickly. So given limited resources it's probably more important to work on the quality of the sample rather than on its size....

Quoted 4 lines

No matter how you see it, the larger the better (in the worst case it won't make a difference). Certainly if I could work on the quality, that would be more important than adding 1 more user. But I can't work on the quality.

Maybe you could try pushing the use of klive by some distros arguing it's "a way of getting usage statistics in order to improve kernel quality and hardware support". I mean, not just an extra package but a service launch at the first boot so anyone can use it.

klive is a nice project, it just needs more _different_ users and maybe, a support of some distros.

Maybe, you could try add a page in the klive wiki for putting packages and RPMs of klive for several distros ? What do you think ? It's a first step to get it merge into distros.

In my case, I got some .tgz that feel lonely...

Maybe be in 2 years, we will know EXACTLY of much people use EXT4/Reiser5/CoincoinFS/CrashFS...

Signature · 2 lines
Jerome Pinot
http://ngc891.blogdns.net/
#282 lore raw

Removed some people from CC since we're quickly changing topic and I doubt they're interested.

Quoted 7 lines · Jerome Pinot

Maybe you could try pushing the use of klive by some distros arguing it's "a way of getting usage statistics in order to improve kernel quality and hardware support". I mean, not just an extra package but a service launch at the first boot so anyone can use it.

klive is a nice project, it just needs more _different_ users and maybe, a support of some distros.

That would be nice yes. KLive is already supported by Gentoo that created a proper emerge package, that's why there are so many gentoo kernels being tracked. Others are of course welcome. I'm open to suggestions on extending it as well.

At the moment the todo list is like this:

1) create an optional http transport to pass through most firewalls
2) send a packet when system reboots so I can track which sessions didn't cleanly reboot (it's not reliable if you disconnect from the internet first and you shutdown later, but it's better than nothing)
3) send the oops from kernel to klive server, so no oopses will be lost and I can use the pciids details to track down bad hardware as well
4) possibly send info on usb buses too and not only about the pci buses

Quoted 3 lines

Maybe, you could try add a page in the klive wiki for putting packages and RPMs of klive for several distros ? What do you think ? It's a first step to get it merge into distros.

If more distros will pickup KLive that will help like demonstrated by the emerge package from gentoo.

It's already very simple to install klive, I wrote an universal installer that runs on any flavour of any weird distro out there:

	wget klive.cpushare.com/klive.sh
	sh klive.sh --install

If you don't want it anymore:

       sh klive.sh --uninstall

that's it.

The rpm/deb packages would be just to make life easier for distro to pick it up but I hoped they had the resources to do the packaging work themself...

Perhaps once I'll create the CPUShare rpm/deb packages I'll be easy to share the work to create the KLive ones too. As far as spare time work goes, CPUShare currently has a much higher prio than KLive, and I'm not yet at the point of creating rpm/deb packages, now that CPUShare transaction works and you can already compute remotely, I'm currently fighting with the paypal sendbox so I can start accepting cash into the system ASAP. But the hardest part of the ebusiness side of CPUShare is the handling of the invoicing and receipts generation which is a non computer related problem. I almost solved it though.

Quoted 1 line

In my case, I got some .tgz that feel lonely...

The .tgz is only for the ones interested hacking or studying it (both client and server). You should be using the .sh script above.

Also note, the export of the whole database isn't available on the site, but I've no problem to publish it after filtering out all the ip addresses (which are collected only for purely paranoid security purposes, and tor doesn't route udp traffic).

Quoted 2 lines

Maybe be in 2 years, we will know EXACTLY of much people use EXT4/Reiser5/CoincoinFS/CrashFS...

BTW, CPUShare records a lot more than the fs already. 2 years or more, but I'm not in a hurry. The server will also soon require some caching trick to avoid recomputing the queries every time you click on it. Current KLive is the best I could do in a very little time I spent on it, it works well, but it still has an huge room for improvement. Also note, so far KLive logged 76989 cumulative days of uptime. When I see a number like that my mind thinks at the energy that can be recycled if all that time recorded on KLive would be routed inside CPUShare. That's why for me it's more urgent to give a chance to those brave 500 users to cash in with CPUShare than to push KLive beyond its current status ;). In due time KLive will get more attention too.

#283 lore raw
In reply to [email protected]
Quoted 11 lines · On Wed, Jul 26, 2006 at 07:20:29PM +0200, [email protected] wrote:

On Wed, Jul 26, 2006 at 01:02:36PM -0400, J. Bruce Fields wrote:
...

Quoted 4 lines

For exactly the same quality of sampling, yes, the larger the better, but the point of diminishing returns comes pretty quickly. So given limited resources it's probably more important to work on the quality of the sample rather than on its size....

No matter how you see it, the larger the better (in the worst case it won't make a difference). Certainly if I could work on the quality, that would be more important than adding 1 more user. But I can't work on the quality.

But depending on the nature of the error, the worst case might be the common case (as I've already explained in another email).

If you can't ensure the quality of your data, please don't use this data to wrongly draw any conclusions from them [1].

cu
Adrian

[1] the conclusion itself might or might not be true
e.g. there _could_ be an 1:5 ratio between reiser4 and ext3 users but your data is not in any way able to support or reject this statement

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#284 lore raw
Quoted 5 lines · Adrian Bunk

But depending on the nature of the error, the worst case might be the common case (as I've already explained in another email).

If you can't ensure the quality of your data, please don't use this data to wrongly draw any conclusions from them [1].

Please read the footer of the KLive pages:

"The use of the information and of the software in this website is at
your own risk. KLive probably doesn't represent a reliable sample of the real usage of the Linux Kernel."

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

#285 lore raw
Quoted 3 lines · On Wed, Jul 26, 2006 at 11:17:41PM +0200, [email protected] wrote:

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

This is where we disagree. It may be wrong, but it's certainly not nearly *as* wrong....

--b.

#286 lore raw
Quoted 2 lines · J. Bruce Fields

This is where we disagree. It may be wrong, but it's certainly not nearly *as* wrong....

;)

My point of view in saying you can't dismiss the current KLive stats completely, is simply that I didn't expect reiser4 numbers to be so high even in the small sample of the brave KLive project that is full of people willing to test new stuff (also note that I compared it to isofs, I didn't attempt a comparison with ext3 myself). That makes it a positive in my view.

If you think KLive numbers aren't a positive point for reiser4, then it can only mean you expected them to be even higher than they were...

#252 lore raw
In reply to Russell Cattelan
Quoted 2 lines · Russell Cattelan

Guess it's sort of like adopting a 8 year old child vs a new born, hard to tell what happened in first 8 years.

And, by that token, the newborn is sometimes preferable. It hasn't had time to develop severe emotional problems, it's physically harmless, and you get to help it form its ideas and beliefs.

On the other hand, the 8 year old is potty trained, you won't have to change a single diaper, it's intelligent and makes you question things, it understands what you want it to do...

So, it depends what kind of developer you are, what kind of gatekeeper, and what kind of project it is. I still believe in releasing early and often, but I can see many reasons not to. Some are financial -- Namesys is trying to operate a business, so any features they open up too early could be that much harder to sell as a commercial product. The repacker, for instance.

I'm not arguing for closed source, I'm just saying that once you open, there's no going back. Many times it's a good thing, but sometimes you want to wait and see.

#31 lore raw
In reply to Matthias Andree
Quoted 5 lines · Matthias Andree
Quoted 3 lines · David Masover
Quoted 2 lines · Matthias Andree

On Tue, 25 Jul 2006, Denis Vlasenko wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

I, on the contrary, want software to impose as few limits on me as possible.

As long as it's choosing some limit, I'll pick the one with fewer surprises.

Running out of inodes would be pretty surprising for me.

No offense: Then it was a surprise for you because you closed your eyes and didn't look at df -i or didn't have monitors in place.

Or because my (hypothetical) business exploded before I had the chance.

After all, you could make the same argument about bandwidth, until you get Slashdotted. Surprise!

Quoted 3 lines

There is no way to ask how many files with particular hash values you can still stuff into a reiserfs 3.X. There, you're running into a brick wall that only your forehead will "see" when you touch it.

That's true, so you may be correct about "less" surprises. So, it depends which is more valuable -- fewer surprises, or fewer limits?

That's not a hypothetical statement, and I don't really know. I can see both sides of this one. But I do hope that once Reiser4 is stable enough for you, it will be predictable enough.

Quoted 3 lines

But the assertion that some backup was the cause for inode exhaustion on ext? is not very plausible since hard links do not take up inodes, symlinks are not backups and everything else requires disk blocks. So,

Ok, where's the assertion that symlinks are not backups? Or not used in backup software? What about directories full of hardlinks -- the dirs themselves must use something, right?

Anyway, it wasn't my project that hit this limit.

#316 lore raw
In reply to Adrian Bunk
Quoted 6 lines · Adrian Bunk

[1] the conclusion itself might or might not be true
e.g. there _could_ be an 1:5 ratio between reiser4 and ext3 users but your data is not in any way able to support or reject this statement

It does however suggest that my surprise at how people at the last Linux Conference I went to all seemed to know that there exists a Reiser4 may be due to it being more widely used than I would have guessed. Maybe there is some coolness factor to having the faster FS that you can't get from any Distro that is enough to overcome the hassle of compiling reiser4progs and a kernel before inserting the DVD. I would not have guessed we had 1/5th of ext3's usage even among lkml readers..... I guess the market contains more people who like technology than I was guessing. Maybe there is a positive word of mouth effect going on too.

It would be nice if SuSE and others at least made it an unsupported option at install time. I shall have to find the time to go asking them all.....

Hans

#287 lore raw
In reply to [email protected]
Quoted 4 lines · On Wed, Jul 26, 2006 at 11:17:41PM +0200, [email protected] wrote:
Quoted 3 lines · Adrian Bunk

But depending on the nature of the error, the worst case might be the common case (as I've already explained in another email).

Quoted 8 lines
Quoted 2 lines

If you can't ensure the quality of your data, please don't use this data to wrongly draw any conclusions from them [1].

Please read the footer of the KLive pages:

"The use of the information and of the software in this website is at
your own risk. KLive probably doesn't represent a reliable sample of the real usage of the Linux Kernel."

It was you who wrongly said:
"With KLive I can attempt to estimate market share of _kernel_ code"

Hadn't you read your own disclaimer?

Quoted 3 lines

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

Possibly wrong conclusions about the general market share based on data not having the quality for being the basis of such statements are worse than having no data.

Every time someone will repeat the "1:5 ratio for reiser4:ext3 users", this will be an additional proof it's really worse than no data.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#288 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

Quoted 8 lines · Adrian Bunk
Quoted 4 lines

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

Possibly wrong conclusions about the general market share based on data not having the quality for being the basis of such statements are worse than having no data.

Am I missing something, or it is not marketing what we are talking about? please I do not understand your point.

For what I understand... I was not supposing reiser4 users to be so mutch, but this is very usefull because this means that there are more possibilities to test the code well and to fix bugs...

This also means that there are people who feels the need of this filesystem for their work because they are not satisfied with the other filesystems, or because anyway with reiser4 they see that they get better results about the work they need to be done.
And this is probably the most important point.

#289 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

Quoted 13 lines · Luigi Genoni
Quoted 8 lines · Adrian Bunk
Quoted 4 lines

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

Possibly wrong conclusions about the general market share based on data not having the quality for being the basis of such statements are worse than having no data.

Quoted 7 lines

Am I missing something, or it is not marketing what we are talking about? please I do not understand your point.

For what I understand... I was not supposing reiser4 users to be so mutch, but this is very usefull because this means that there are more possibilities to test the code well and to fix bugs...
...

I'm sorry for saying it that directly, and this is not meant against you personally:

Your statement is a good example why this data is worse than having no data.

As I already said in this thread:

<-- snip -->

You are able to say that based on klive data, reiser4 has at least 35 users in the world.

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

<-- snip -->

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data. But claiming it based on the klive data is worse since people like you are getting the wrong impression it was based on data that would have the quality for supporting such a statement.

The data simply has not the quality for such a statement. Please read my two examples in [1] if you want to get an impression why such problems can occur.

cu
Adrian

[1] http://lkml.org/lkml/2006/7/26/203

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#290 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

Still I am missing something.

I am not interested in discussing about 1:5 ora 5:1 or so on.
I am not even interested if reiser4 users are 35 or 35000.
But if some people felt they need reiser4 to get their work done, that means something. The numbers and the datum per se are meaningless.

Anyway you have a datum.
Some people need reiser4, period.

Why they need reiser4?
all of them are just playing with him to have a new game for their spare time? I don't think so.
Some of them are using reiser4 because it is the best solution for their work? this is a concrete possibility.

again... why that?
Because the other FSs are not suitable for certain work? Because the other FSs are suitable, but reiser4 offers better results? Because even reiser4 is not complitelly suitable for some kind of work, but the other FSs are a worser solution?

I don't think most reiser4 users decided to give it a try just to enjoy their time making tests just because they are curious.

that does not mean reiser4 should be merged into linux kernel because of this. On the other side you can agree reiser4 has its place, because there are users who need it. What we would need to understand, sine ira nec studio, is where this place is.

Quoted 81 lines · Adrian Bunk
Quoted 14 lines · Luigi Genoni
Quoted 8 lines · Adrian Bunk
Quoted 4 lines

Said that pretending that KLive data has absolutely no significance at all and that you can't draw any conclusion at all from it, to me seems as wrong as pretending it to perfect.

Possibly wrong conclusions about the general market share based on data
not having the quality for being the basis of such statements are
worse than having no data.

Quoted 7 lines

Am I missing something, or it is not marketing what we are talking about?
please I do not understand your point.

For what I understand... I was not supposing reiser4 users to be so mutch, but this is very usefull because this means that there are more possibilities to test the code well and to fix bugs... ...

I'm sorry for saying it that directly, and this is not meant against you personally:

Your statement is a good example why this data is worse than having no data.

As I already said in this thread:

<-- snip -->

You are able to say that based on klive data, reiser4 has at least 35 users in the world.

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

<-- snip -->

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data. But claiming it based on the klive data is worse since people like you are getting the wrong impression it was based on data that would have the quality for supporting such a statement.

The data simply has not the quality for such a statement. Please read my two examples in [1] if you want to get an impression why such problems can occur.

cu Adrian

[1] http://lkml.org/lkml/2006/7/26/203

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#291 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

Quoted 7 lines · Luigi Genoni

Still I am missing something.

I am not interested in discussing about 1:5 ora 5:1 or so on.
I am not even interested if reiser4 users are 35 or 35000.
But if some people felt they need reiser4 to get their work done, that means something. The numbers and the datum per se are meaningless.
...

You answered to an email where I talked about the wrong assumption the klive data could support the 1:5 market share claim.

This is completely independent from discussions about why users might need reiser4, or which technical or social problems prevent its merging (unless of course someone wants to justify its merging wrongly with the 1:5 claim), and therefore your answer does not apply to this part of the thread.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#292 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

I answered a mail about how klive data should not be took in account, and could even be dangerous...

Quoted 34 lines · Adrian Bunk
Quoted 8 lines · Luigi Genoni

Still I am missing something.

I am not interested in discussing about 1:5 ora 5:1 or so on. I am not even interested if reiser4 users are 35 or 35000. But if some people felt they need reiser4 to get their work done, that means something. The numbers and the datum per se are meaningless. ...

You answered to an email where I talked about the wrong assumption the klive data could support the 1:5 market share claim.

This is completely independent from discussions about why users might need reiser4, or which technical or social problems prevent its merging (unless of course someone wants to justify its merging wrongly with the 1:5 claim), and therefore your answer does not apply to this part of the thread.

cu Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

#306 lore raw
In reply to Adrian Bunk
Quoted 10 lines · Adrian Bunk
Quoted 9 lines · On Wed, Jul 26, 2006 at 11:17:41PM +0200, [email protected] wrote:
Quoted 5 lines · Adrian Bunk

But depending on the nature of the error, the worst case might be the common case (as I've already explained in another email).

If you can't ensure the quality of your data, please don't use this data to wrongly draw any conclusions from them [1].

Please read the footer of the KLive pages:

Quoted 8 lines
Quoted 3 lines

"The use of the information and of the software in this website is at
your own risk. KLive probably doesn't represent a reliable sample of the real usage of the Linux Kernel."

It was you who wrongly said:
"With KLive I can attempt to estimate market share of _kernel_ code"

Hadn't you read your own disclaimer?

There is no contradiction in the two statements. To attempt to estimate something I don't need a reliable sample of the whole population. Estimation is still a statistical thing. Also I said attempt to estimate, it doesn't mean I will make it.

If you don't consider those results a positive for reiser4, it can only mean you expected reiser4 to have a much higher share among the KLive users. This is obvious.

Quoted 2 lines

Every time someone will repeat the "1:5 ratio for reiser4:ext3 users", this will be an additional proof it's really worse than no data.

If they say "1:5 ratio for reiser4:ext3 KLive users" everything will be correct and nobody can object because it's a fact.

I said myself that I'm no reiserfs user, and I don't plan to become one any time soon (especially on my production systems), I'm only reporting plain numbers as KLive measured the stuff. I'm surprised as much as you are, but then I've to report facts, and not my own opinions.

As far as I'm concerned the thing I like less of reiser4 is the plugin thing, I'd be less concerned if that was a microkernel (fuse-like) userland plugin system. Anyway with time perhaps things can change and become userland based, and the stuff can be moved into vfs if that code really belongs there as some kernel developer says. That doesn't mean reiser4 can't be merged first and the stuff moved into vfs later. xfs when was merged also pratically rewrote a vfs internally that was meant to work with irix, if Christoph didn't complain about xfs being merged, I don't see what's the problem of reiser4 being merged even if it rewrites some part of vfs. xfs is also still having various special features like the pinhole one that only belongs to the vfs instead but nobody complains.

And if reiser4 is really so bad as they say, once people starts losing data they will spread the word of not using it. As long as it's marked experimental I don't see a big issue, the wireless driver for broadcom chip will eat your filesystem too if the reverse engineered dma operations writes into a buffer header instead of an skb.

#293 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Luigi Genoni
Quoted 2 lines · Luigi Genoni

I answered a mail about how klive data should not be took in account, and could even be dangerous...

... and you talking about people who might need reiser4 was therefore a wrong answer.

I never said reiser4 shouldn't be merged, but this wasn't the topic of this subthread.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#307 lore raw
In reply to [email protected]
Quoted 10 lines · On Thu, Jul 27, 2006 at 01:52:29PM +0200, [email protected] wrote:

On Thu, Jul 27, 2006 at 08:56:03AM +0200, Adrian Bunk wrote:
...

Quoted 4 lines

It was you who wrongly said:
"With KLive I can attempt to estimate market share of _kernel_ code"

Hadn't you read your own disclaimer?

There is no contradiction in the two statements. To attempt to estimate something I don't need a reliable sample of the whole population. Estimation is still a statistical thing. Also I said

Sure, you don't need any data to estimate anything.

But if your estimate is based on bad data it sounds better than if it was based on no data although the value of the estimate is the same.

Quoted 1 line

attempt to estimate, it doesn't mean I will make it.

Giving low quality raw data and then saying "it wasn't me who did the estimate" is silly.

Quoted 3 lines

If you don't consider those results a positive for reiser4, it can only mean you expected reiser4 to have a much higher share among the KLive users. This is obvious.

That's completely wrong.
I consider those results neither positive nor negative for reiser4.

They could only be considered positive if someone expected less than 35 reiser4 users worldwide.

Quoted 5 lines
Quoted 2 lines

Every time someone will repeat the "1:5 ratio for reiser4:ext3 users", this will be an additional proof it's really worse than no data.

If they say "1:5 ratio for reiser4:ext3 KLive users" everything will be correct and nobody can object because it's a fact.

And 90% of all people will misread it.

And many people will spread it wrong.

That's exactly where such data is worse than no data.

You could claim you know your the suboptimal quality of the data and that you had a disclaimer.

But did you read Hans' reaction on your email?

If you are intelligent (which I assume), you should have learned by this how to not present your data.

Quoted 6 lines

I said myself that I'm no reiserfs user, and I don't plan to become one any time soon (especially on my production systems), I'm only reporting plain numbers as KLive measured the stuff. I'm surprised as much as you are, but then I've to report facts, and not my own opinions.
...

I have no plans to use reiser4 myself (I'm a happy ext2 user). But I'm not against mergng reiser4, either.

I'll leave this technical discussion to the people who know more about this.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#302 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Adrian Bunk
Quoted 7 lines · Adrian Bunk

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data. But claiming it based on the klive data is worse since people like you are getting the wrong impression it was based on data that would have the quality for supporting such a statement.

Quoted 5 lines

The data simply has not the quality for such a statement. Please read my two examples in [1] if you want to get an impression why such problems can occur.

[1] http://lkml.org/lkml/2006/7/26/203

Also, one may wish to invest in the purchase of 'How to Lie with Statistics'[1] by Darrel Huff (ISBN 0-393-31072-8). An easy yet powerful little read.

[1] this is not meant to imply that anyone here is lieing, attempting to lie, thinking of lieing, wishing they were lieing or having anything to do with lieing, purposeful or otherwise.

Signature · 3 lines
    "To the extent that we overreact, we proffer the terrorists the
    greatest tribute."
    	- High Court Judge Michael Kirby
#308 lore raw
In reply to Adrian Bunk
Quoted 2 lines · Adrian Bunk

They could only be considered positive if someone expected less than 35 reiser4 users worldwide.

The only thing we know for sure is that 35 out of 500 KLive user are running reiser4, worldwide we have no clue.

Quoted 2 lines

If you are intelligent (which I assume), you should have learned by this how to not present your data.

I think readers are intelligent enough too to interpret the KLive data properly for themself, without you having to prevent their eyes to see the raw KLive data.

When you tell me how I should not present my data, you're asking me to censor part or all of the very output of the KLive project. I'd rather wipe out KLive completely, than to censor it. The way I presented it was absolutely not biased, if you can make more transparent and unbiased sql queries than the ones I did, please post them and I'll be glad to run them.

Quoted 1 line

[..] (I'm a happy ext2 user).

Oh my, I hope you're only choosing the fs for your own workstation.

Even though I don't pretend to fully understand someone who claims to be happy with ext2, I've no idea why you hate so much the stuff running at cpushare.com domain. But if it helps KLive is actually one of the non commercial projects I'm hosting there, and the only reason I keep it there, is to be sure not to find it filled by ads.

Now let's try to get some work done instead of only sending emails ;)

#317 lore raw
In reply to Hans Reiser
Quoted 9 lines · Hans Reiser
Quoted 4 lines · Adrian Bunk

[1] the conclusion itself might or might not be true
e.g. there _could_ be an 1:5 ratio between reiser4 and ext3 users but your data is not in any way able to support or reject this statement

It does however suggest that my surprise at how people at the last Linux Conference I went to all seemed to know that there exists a Reiser4

I'd be rather surprised to find someone at a Linux conference who hasn't at least heard of the recurrent flamewars on the topic here...

Quoted 4 lines

may be due to it being more widely used than I would have
guessed. Maybe there is some coolness factor to having the faster FS that you can't get from any Distro that is enough to overcome the hassle of compiling reiser4progs and a kernel before inserting the DVD.

Not seen any data backing up the "faster", let alone so much faster that the hassle (and the risk) would make it worth trying... No, Gentoo folks don't count, around here they are fond of claiming that their self-compiled systems are at least twice as fast as a binary distribution with the exact same software.

Quoted 3 lines

I
would not have guessed we had 1/5th of ext3's usage even among lkml readers.....

I'd guess something of the order of 1/100, for testing purposes and idle curiosity.

Quoted 3 lines

I guess the market contains more people who like
technology than I was guessing. Maybe there is a positive word of mouth effect going on too.

LKML (and Linux conferences) are exactly the places where you /only/ find this kind of people...

Quoted 3 lines

It would be nice if SuSE and others at least made it an unsupported option at install time. I shall have to find the time to go asking them all.....

At least Fedora is trying hard to just follow upstream packages (in this case, Linus' kernels) with the absolute minimum of local patches. It makes good sense, as it reduces the up-front work, and (more important) minimizes the pain when the later official version is somehow incompatible with the previews.

If you want ReiserFS 4 to get more exposure, do the legwork to get it into the official kernel. Distributions should be reluctant to pick it up as long as its status as an official Linux filesystem is in question.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#294 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Luigi Genoni

Luigi Genoni <[email protected]> wrote:

[...]

Quoted 2 lines

Anyway you have a datum.
Some people need reiser4, period.

Nope. Some people run kernels that include reiser4. That is all you can infer, and that I knew beforehand. They are at least 35, and that I'd have guessed in any case.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#295 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

Quoted 10 lines · Horst H. von Brand

Luigi Genoni <[email protected]> wrote:

[...]

Quoted 2 lines

Anyway you have a datum.
Some people need reiser4, period.

Nope. Some people run kernels that include reiser4. That is all you can infer, and that I knew beforehand. They are at least 35, and that I'd have guessed in any case.

35.5 as I'm testing it here on my workstation and it seems to be faster when you test some things involving many copies of large multi-level sourcetree directories each 3 to 6GiB big in size. 2.6.18-rc2-mm1 with Reiser4 looks ok so far and I had no sync() OOPS like the last time with one -mm revision.

speed tells us nothing about reliability of course, but compared to ext3 with dir_index,sparse_super Reiser4 seems to handle "du -sh" and "rm -r" much faster and without eating all of the CPU cycles as it finishes quicker although Reiser4 was meant to be CPU-heavy compared to ext*/reiserfs3.

#296 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Horst H. von Brand
Quoted 2 lines · Horst H. von Brand

Nope. Some people run kernels that include reiser4. That is all you can infer, and that I knew beforehand. They are at least 35, and that I'd have

Well, if they were sure they could never get any benefit by reiser4 they wouldn't be testing it in the first place...

But certainly the fact they're testing it, doesn't mean they're actually going to use it forever. You can monitor the 35 users live with this link and to see if they increase or decrease over time ;)

http://klive.cpushare.com/?order_by=kernel_group&where_machine=all&branch=all&scheduler=all&smp=all&live=live&ip=all

They're ordered by cumulative uptime and not by the number of users. For example there are only 10 jfs users but those 10 jfs users have a much larger average uptime (59 days) so they score much higher in the cumulative uptime too. Why the average reiser4 uptime is much lower is unknown, it could be because they shutdown the system by night, or because they upgrade kernel so frequently, or because the system crashes.

#309 lore raw
In reply to [email protected]
Quoted 14 lines · On Thu, Jul 27, 2006 at 03:10:32PM +0200, [email protected] wrote:
Quoted 2 lines · Adrian Bunk

They could only be considered positive if someone expected less than 35 reiser4 users worldwide.

The only thing we know for sure is that 35 out of 500 KLive user are running reiser4, worldwide we have no clue.

Quoted 2 lines

If you are intelligent (which I assume), you should have learned by this how to not present your data.

I think readers are intelligent enough too to interpret the KLive data properly for themself, without you having to prevent their eyes to see the raw KLive data.

Quoted 6 lines

When you tell me how I should not present my data, you're asking me to censor part or all of the very output of the KLive project. I'd rather wipe out KLive completely, than to censor it. The way I presented it was absolutely not biased, if you can make more transparent and unbiased sql queries than the ones I did, please post them and I'll be glad to run them.

What about the following statement:

"Gentoo is 47 times as popular as SuSE among KLive users (a service
offered by a SuSE employee gathering data from many users worldwide)."

Is any part of this information wrong?

Is it therefore OK to put this information to /. ?

Surely not, but you might get the point.

Quoted 6 lines
Quoted 1 line

[..] (I'm a happy ext2 user).

Oh my, I hope you're only choosing the fs for your own workstation.

Even though I don't pretend to fully understand someone who claims to be happy with ext2,

It's stable, and if the machine crashes the fsck is really great (Linus' tree contains at least a dozen patches that had their temporary home in my lost+found). (I don't care that the fsck takes 20 minutes on a 250 GB partition.)

Other people have other preferences and do therefore prefer other filesystems.

Quoted 4 lines

I've no idea why you hate so much the stuff
running at cpushare.com domain. But if it helps KLive is actually one of the non commercial projects I'm hosting there, and the only reason I keep it there, is to be sure not to find it filled by ads.

This isn't against cpushare.com, it's against publishing things people will easily misinterpret.

The Linux Counter [1] is another non commercial project gathering similar data (including live data about running kernels and uptimes) from far more users, and I've already given an example in this thread how someone could have misinterpreted the data offered there. And if someone would argue with data from there in a similar easy to misinterpret way on this list, I'd react similarly.

And another example would be that you could say "one third of all Linux Counter machines are still running kernels < 2.6". This would be based on current information submitted automatically from nearly 5000 machines. But people would read this is "one third of all machines are still running kernels < 2.6". But the data doesn't have the quality for this generalization. And this is exactly the problem.

Quoted 1 line

Now let's try to get some work done instead of only sending emails ;)

Agreed. ;-)

cu
Adrian

[1] http://counter.li.org/

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#310 lore raw
Quoted 10 lines · Adrian Bunk

What about the following statement:

"Gentoo is 47 times as popular as SuSE among KLive users (a service
offered by a SuSE employee gathering data from many users worldwide)."

Is any part of this information wrong?

Is it therefore OK to put this information to /. ?

Surely not, but you might get the point.

I already told you once that such a claim is totally bogus because KLive _doesn't_ record anything about the _userland_, it only records kernel stuff.

It's not Dlive (distributon Live), it's KLive and K stands for _kernel_ not distributions.

Not sure how many times I'll have to remind you about this.

Also note currently the current cumulative uptime ratio is 1:18 not 1:47 and even the ratio of the number of users is 1:42. The only claim based on actual facts you can try to publish on the press is this:

"The Gentoo kernels are 18 times more popular than the SUSE kernels among KLive users (a service offered in his spare time by a SUSE contractor that tries to gather data from many users worldwide)."

You can also add:

"Another data point is that among the KLive users the Fedora kernels are 24 times less popular then Gentoo ones."

Free to advertise the above if you find it interesting.

Quoted 2 lines

This isn't against cpushare.com, it's against publishing things people will easily misinterpret.

I think people deserve to think for themself without you acting as censorship filter for them.

#297 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Horst H. von Brand

Since reiser4 is not something enabled by default into a default kernel distribution, I assume they enabled it knowing what they where doing because they wanted to use it.

Quoted 25 lines · Horst H. von Brand

Luigi Genoni <[email protected]> wrote:

[...]

Quoted 3 lines

Anyway you have a datum.
Some people need reiser4, period.

Nope. Some people run kernels that include reiser4. That is all you can infer, and that I knew beforehand. They are at least 35, and that I'd have guessed in any case. --
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#311 lore raw
In reply to [email protected]
Quoted 19 lines · On Thu, Jul 27, 2006 at 04:45:29PM +0200, [email protected] wrote:
Quoted 10 lines · Adrian Bunk

What about the following statement:

"Gentoo is 47 times as popular as SuSE among KLive users (a service
offered by a SuSE employee gathering data from many users worldwide)."

Is any part of this information wrong?

Is it therefore OK to put this information to /. ?

Surely not, but you might get the point.

I already told you once that such a claim is totally bogus because KLive _doesn't_ record anything about the _userland_, it only records kernel stuff.

It's not Dlive (distributon Live), it's KLive and K stands for _kernel_ not distributions.

Quoted 6 lines

Not sure how many times I'll have to remind you about this.

Also note currently the current cumulative uptime ratio is 1:18 not 1:47 and even the ratio of the number of users is 1:42. The only claim based on actual facts you can try to publish on the press is this:

The 1:47 was what I calculated yesterday, but it seems to be 1:42 now.

But I'd measure popularity in current usage, not in cumulative uptime.

Quoted 4 lines

"The Gentoo kernels are 18 times more popular than the SUSE kernels among KLive users (a service offered in his spare time by a SUSE contractor that tries to gather data from many users worldwide)."
...

Slightly modified version is now in my .signature (I omitted the "spare time" part since the information is still true and sounds better without it - it would be cencorship if you'd try to force me to not omit it).

cu
Adrian

Signature · 3 lines
  The Gentoo kernels are 42 times more popular than the SUSE kernels 
  among KLive users (a service by SuSE contractor Andrea Arcangeli that 
  gathers data about kernels from many users worldwide).
#327 lore raw
In reply to Olivier Galibert

Hi!

Quoted 8 lines
Quoted 5 lines

A much more important effect is that non-maintainers aren't familiar with coding and patch submission guidelines. For example, in suspend2, Nigel first tried with patches that were too monolithic, and then his next series was too broken down such that it was too hard to review (and "git bisect" wouldn't work).

All his submissions since 2004 or so? It's a little easy to limit oneself to the last two ones.

Nigel did not do any submissions in 2004 or so. Check your fact, that stuff was marked 'RFC' and yes I did comment on it.

He did 1 (one) submission that looked like SubmittingPatches at the first sight, and that was very recent.

Stop spreading lies.

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#180 lore raw
In reply to Pavel Machek
Quoted 10 lines · Pavel Machek

Hi!

Quoted 5 lines
Quoted 4 lines

of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Well, too bad, if reiser3 is so broken it needs on-disk-format-change, then I guess doing that change is the right thing to do...

Sorry for my stupid question, but could you tell me why starting to make incompatible changes to reiserfs3 now (when reiserfs3 "technology" is rather old) and making reiserfs3 unstable (again), possibly for several months or even years is better than fixing big issues with reiser4 (if there are any really big left) merging it and trying to stabilize it?

For end user both ways will result in mkfs so...

Thanks,

Grzegorz Kulewski

#328 lore raw
In reply to Pavel Machek
Quoted 8 lines · Pavel Machek

Hi!

Quoted 6 lines
Quoted 5 lines

A much more important effect is that non-maintainers aren't familiar with coding and patch submission guidelines. For example, in suspend2, Nigel first tried with patches that were too monolithic, and then his next series was too broken down such that it was too hard to review (and "git bisect" wouldn't work).

Quoted 5 lines
Quoted 2 lines

All his submissions since 2004 or so? It's a little easy to limit oneself to the last two ones.

Nigel did not do any submissions in 2004 or so. Check your fact, that stuff was marked 'RFC' and yes I did comment on it.

2004-09-16 submission:

http://lkml.org/lkml/2004/9/16/76
http://lkml.org/lkml/2004/9/16/77
http://lkml.org/lkml/2004/9/16/78
http://lkml.org/lkml/2004/9/16/81
http://lkml.org/lkml/2004/9/16/82
http://lkml.org/lkml/2004/9/16/83
http://lkml.org/lkml/2004/9/16/86
http://lkml.org/lkml/2004/9/16/87
http://lkml.org/lkml/2004/9/16/89
http://lkml.org/lkml/2004/9/16/90

2004-11-24 submission:

http://lkml.org/lkml/2004/11/24/93
http://lkml.org/lkml/2004/11/24/94
http://lkml.org/lkml/2004/11/24/95
http://lkml.org/lkml/2004/11/24/96
http://lkml.org/lkml/2004/11/24/97
http://lkml.org/lkml/2004/11/24/98
http://lkml.org/lkml/2004/11/24/100
http://lkml.org/lkml/2004/11/24/101
[58 mails or so total]

Quoted 4 lines

He did 1 (one) submission that looked like SubmittingPatches at the first sight, and that was very recent.

Stop spreading lies.

I am awaiting your apologies.

OG.

#312 lore raw
In reply to Adrian Bunk
Quoted 4 lines · Adrian Bunk

Slightly modified version is now in my .signature (I omitted the "spare time" part since the information is still true and sounds better without it - it would be cencorship if you'd try to force me to not omit it).

Well, a signature like that, perfectly match the IQ of who writes it, please add a link to the klive homepage too if you don't mind.

This IMHO would be even more hilarious:

"I'm happy with ext2" by Adrain Bunk, the guy who pretends to provide useful feedback in the reiser4 discussion.

#349 lore raw
In reply to Pavel Machek

Ar Mer, 2006-07-26 am 13:08 +0000, ysgrifennodd Pavel Machek:

Quoted 4 lines

He did 1 (one) submission that looked like SubmittingPatches at the first sight, and that was very recent.

Stop spreading lies.

Pavel, I think you might want to check your facts and then apologize.

#181 lore raw
In reply to Grzegorz Kulewski
Quoted 7 lines · Grzegorz Kulewski

Sorry for my stupid question, but could you tell me why starting to make incompatible changes to reiserfs3 now (when reiserfs3 "technology" is rather old) and making reiserfs3 unstable (again), possibly for several months or even years is better than fixing big issues with reiser4 (if there are any really big left) merging it and trying to stabilize it?

For end user both ways will result in mkfs so...

ext2fs and ext3fs, without "plugins", added dir_index as a compatible upgrade, with an e2fsck option (that implies optional) to build indices for directories without them.

ext3fs is a compatible upgrade from ext2fs, it's as simple as unmount, tune2fs -j, mount.

reiserfs 3.6 could deal with 3.5 file systems, and "mount -o conv" with a 3.6 driver would convert a 3.5 file system to 3.6 level (ISTR it had to do with large file support and perhaps NFS exportability, but don't quote me on that).

I wonder what makes the hash overflow issue so complicated (other than differing business plans, that is) that upgrading in place isn't possible. Changes introduce instability, but namesys were proud of their regression testing - so how sustainable is their internal test suite?

Instead, we're told reiser4 would fix this (quite likely) and we should wait until it's ready (OK, we shouldn't be using experimental stuff for production but rather for /tmp, but the file system will take many months to mature after integration) and it will be "mkfs" time - so reiser4 better be mature before we go that way if there's no way back short of "amrecover", "restore" or "tar -x".

Smashing out most of the Cc:s in order not to bore people.

Signature · 1 line
Matthias Andree
#183 lore raw
In reply to Pavel Machek
Quoted 10 lines · Pavel Machek

Hi!

Quoted 5 lines
Quoted 4 lines

of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

Well, too bad, if reiser3 is so broken it needs on-disk-format-change, then I guess doing that change is the right thing to do...

Actually, there is reiser4 brokenness lurking in Hans' statement, too:

A filesystem WITH plugins must still handle the standard Linux compatibility stuff that other filesystems handle.

Plugins --do not-- mean that you can just change the filesystem format willy-nilly, with zero impact.

Jeff

#298 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Luigi Genoni
Quoted 5 lines · Luigi Genoni

Since reiser4 is not something enabled by default into a default kernel distribution, I assume they enabled it knowing what they where doing because they wanted to use it.

Or because they did 'allmodconfig' or 'allyesconfig'. Whenever I build a kernel I enabled everything possible as a module in case I ever need it. For instance, a few weeks ago I had the reiserfs module loaded because I was testing something, if I had klive running it would have said that I use reiserfs when in fact I don't.

Jim.

#300 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Luigi Genoni

Luigi Genoni <[email protected]> wrote:

[...]

Quoted 3 lines

Since reiser4 is not something enabled by default into a default kernel distribution, I assume they enabled it knowing what they where doing because they wanted to use it.

You know, I compile ReiserFS, JFS and XFS (and also CRAMFS and ROMFS) into my self-compiled kernels too. Haven't used any of them in ages (if ever), but one more go at compile-testing won't hurt...

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#184 lore raw
In reply to Jeff Garzik
Quoted 11 lines · Jeff Garzik
Quoted 6 lines · Pavel Machek

Hi!

Quoted 4 lines
Quoted 3 lines

of the story for me. There's nothing wrong about focusing on newer code, but the old code needs to be cared for, too, to fix remaining issues such as the "can only have N files with the same hash value".

Requires a disk format change, in a filesystem without plugins, to fix it.

A filesystem WITH plugins must still handle the standard Linux compatibility stuff that other filesystems handle.

Plugins --do not-- mean that you can just change the filesystem format willy-nilly, with zero impact.

They --do-- mean that you can change much of the filesystem behavior without requiring massive on-disk changes or massive interface changes.

After all, this is how many FUSE plugins work -- standard FS interface, usually uses another standard FS as storage, but does crazy things like compression, encryption, and other transformations in between.

#299 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Jim Crilly
Quoted 5 lines · Jim Crilly

Or because they did 'allmodconfig' or 'allyesconfig'. Whenever I build a kernel I enabled everything possible as a module in case I ever need it. For instance, a few weeks ago I had the reiserfs module loaded because I was testing something, if I had klive running it would have said that I use reiserfs when in fact I don't.

reiserfs would showup in the module list in such case, but _not_ in the fs list. KLive records both the modules loaded _and_ the mounted fs.

This is the code that records the FS:

	if CONFIG_FS:
		fs = {}
		for _fs in [ re.search(r'^[^\s]+\s[^\s]+\s([^\s]+)', x).group(1)
			     for x in file('/proc/mounts').readlines() ]:
			if _fs not in ['rootfs', 'proc', 'devpts']:
				fs[_fs] = None
		fs = ' '.join(fs.keys())
		KERNEL += str_append(fs)

/proc/mounts only lists the _mounted_ fs, not the fs loaded into the kernel statically or with insmod.

#301 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Horst H. von Brand
Quoted 2 lines · Horst H. von Brand

You know, I compile ReiserFS, JFS and XFS (and also CRAMFS and ROMFS) into my self-compiled kernels too. Haven't used any of them in ages (if ever),

If you never mounted them shortly after boot time, they would have never showed up in KLive fs list if that's what you mean.

#253 lore raw
In reply to David Masover

Hello David,

Quoted 2 lines · you

I'm not arguing for closed source, I'm just saying that once you open, there's no going back. Many times it's a good thing, but sometimes you

A sidenote.

Reiser4 is open and still we don't see people writing plugins as crazy. I belive there is one group that tried to be the first outside of namesys to write a plugin but still no success.

Regards,
Maciej

#254 lore raw
Quoted 10 lines · Maciej SoÅ‚tysiak

Hello David,

Quoted 2 lines · you

I'm not arguing for closed source, I'm just saying that once you open, there's no going back. Many times it's a good thing, but sometimes you

A sidenote.

Reiser4 is open and still we don't see people writing plugins as crazy. I belive there is one group that tried to be the first outside of namesys to write a plugin but still no success.

Kernel inclusion would help a lot. Decent documentation would be better, though. Someone should look at what FUSE is doing right. Plugins fill a lot of the same niches, but with significantly better performance.

#329 lore raw

suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Olivier Galibert

Hi!

Quoted 15 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

A much more important effect is that non-maintainers aren't familiar with coding and patch submission guidelines. For example, in suspend2, Nigel first tried with patches that were too monolithic, and then his next series was too broken down such that it was too hard to review (and "git bisect" wouldn't work).

All his submissions since 2004 or so? It's a little easy to limit oneself to the last two ones.

Nigel did not do any submissions in 2004 or so. Check your fact, that stuff was marked 'RFC' and yes I did comment on it.

2004-09-16 submission:

http://lkml.org/lkml/2004/9/16/76

...

Quoted 1 line

http://lkml.org/lkml/2004/9/16/90

Yes, Nigel submitted part of suspend2 here (I do not think he submitted compression code on that day, for example), but then

http://lkml.org/lkml/2004/9/16/347

came:

#I should mention, I'm planning on producing a BK tree with these patches #(and ones to fix issues raised), so you won't have to include a long #list in -mm eventually! It will be on suspend2.bkbits.net (not there #yet).

That sounds like "don't merge it yet, I'll prepare better version" to me.

Quoted 3 lines

2004-11-24 submission:

http://lkml.org/lkml/2004/11/24/93

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Okay, this one is important. It starts with

#Hi everyone.
#
#I know that I still have work to do on suspend2, but thought it was high #time I got around to properly submitting the code for review, so here goes.

That looks like request for review, not request for merge. Plus it was
to: lkml, while it should be to: maintainer (or maybe to: akpm). Well, it also says "Suspend 2 merge" in subject..

Quoted 6 lines
Quoted 4 lines

He did 1 (one) submission that looked like SubmittingPatches at the first sight, and that was very recent.

Stop spreading lies.

I am awaiting your apologies.

Okay, 11/24 looked closer to submission than I recalled. So I guess I have to say "sorry".

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#330 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 2 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

OG.

#185 lore raw
In reply to Jeff Garzik
Quoted 2 lines · Jeff Garzik

Actually, there is reiser4 brokenness lurking in Hans' statement, too:

Where! Someone tell me!;-)

Quoted 3 lines

A filesystem WITH plugins must still handle the standard Linux compatibility stuff that other filesystems handle.

Hmm, you mean we should first implement regular unix file plugins before implementing enhanced functionality ones? Are you aware that reiser4 plugins are per file, and thus if a user selects a plugin that is not the default, and which has user visible semantic differences, it means they said they want non-standard behavior?

Quoted 3 lines

Plugins --do not-- mean that you can just change the filesystem format willy-nilly, with zero impact.

Yes they do.....

Quoted 7 lines

Jeff

#182 lore raw
In reply to Matthias Andree
Quoted 7 lines · Matthias Andree

I wonder what makes the hash overflow issue so complicated (other than differing business plans, that is) that upgrading in place isn't possible. Changes introduce instability, but namesys were proud of their regression testing - so how sustainable is their internal test suite?

Never met a test suite the equal of a few million users.....

People have this image of Namesys as some large corporation that has large resources. We just barely are going to be able to ship reiser4, at the cost of a LOT of financial pain. We can't afford to go in two directions at once. We can add bugfixes to V3, but adding features, I have to tell you that we ain't got the staff for both that and shipping V4. Our whole corporation has a budget about what most corporations
spend on two programmers. We have 5 developers, including me, and
making little bits of money is a constant distraction from the main work.

Hans

#315 lore raw
In reply to [email protected]
Quoted 6 lines · [email protected] wrote:

As far as I'm concerned the thing I like less of reiser4 is the plugin thing, I'd be less concerned if that was a microkernel (fuse-like) userland plugin system.

Performance. If your plugin is performance valuing, it needs to be in
kernel. Also, FUSE does not have per-file plugins (correct me if I err
here). It would be nice to see it pay attention to reiser4 pluginids, and thus become per-file.

Thanks for your other words, which I will not comment on because I agree with them.;-)

#303 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Adrian Bunk
Quoted 11 lines · Adrian Bunk

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

<-- snip -->

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data.

Yes, but I have been surprised that linux conference attendees all know about reiser4, so I think it is consistent with the notion that reiser4 usage may well be much much higher than one would expect of a filesystem that is not in the main tree, and which requires a lot of hassle to install. It is a datapoint. I think the level of hobbyist enthusiasm seen suggests that distros may gain market advantage by adding reiser4 support.....

Hans

#304 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Hans Reiser
Quoted 13 lines · Hans Reiser
Quoted 11 lines · Adrian Bunk

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

<-- snip -->

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data.

Quoted 5 lines

Yes, but I have been surprised that linux conference attendees all know about reiser4, so I think it is consistent with the notion that reiser4 usage may well be much much higher than one would expect of a filesystem that is not in the main tree,
...

There were already several long flamewars^Wdiscussions about reiser4 on linux-kernel.

There has already been some amount of press articles covering reiser4.

But how much usage is the result of this?

There seem to be the the following possibilities how users get reiser4 into their kernel today:
1. -mm kernel
2. non -mm kernel
a) patch downloaded from namesys.com
b) through their distribution
- in a kernel image
- as patch
c) patch forwarded through other channels

2b) seems to be the only number that is easily measurable. Do you have some download statistics for namesys.com?

Multiplying the downloads for the 2.6.16 patches with 10 seems to be the best estimate for the order of magnitude of current reiser4 users that seems to be available.

Quoted 1 line

Hans

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#186 lore raw

metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 5 lines · Hans Reiser
Quoted 2 lines · Jeff Garzik

Plugins --do not-- mean that you can just change the filesystem format willy-nilly, with zero impact.

Yes they do.....

Take off your marketing cap for a second :)

Plugins do not eliminate the going-back-to-older-kernels issue, as discussed at length in the recent ext4 thread. Plugins complicate, rather than simplify, the issue of determining the compatibility between the kernel's metadata handling code and on-disk metadata.

Quite simply, there is always a compatibility impact when the filesystem format changes. Enterprise customers will inevitably run several -generations- of your software, and they are very aware of migration costs when dealing with hardware+software life cycles in the 5-10 year range. Power users and kernel developers bounce between kernel versions all the time.

This is the same problem I described in the ext4 thread, then summarized as "what ext3 am I talking to, today?" You can easily rewrite that as "what reiser4 set of plugins am I talking to, today?"

Plugins == a ton of new software to be versioned and tracked. Rather than just track "reiser4", you must now track the collection of plugin versions as well. OBVIOUS increased complexity, and OBVIOUS additional admin learning curve for the serious IT shop.

Now prepare for the bombshell revelation...

THAT SAID, I do think fs {algorithm|metadata} plugins are a very interesting idea.

The plugin infrastructure is the logical result of trying to support multiple incompatible {inode, dir, file data, ...} object types in the same Linux filesystem. Looking at in-tree Linux filesystems, one can see common design patterns in filesystems that hand-code support for multiple metadata format versions -- Al Viro's fs/sysv hacks being a simple and elegant example.

It is then simple to follow that train of logic: why not make it easy to replace the directory algorithm [and associated metadata]? or the file data space management algorithms? or even the inode handling? why not allow customers to replace a stock algorithm with an exotic, site-specific one?

In essence, reiser4 is not really a filesystem, but a second VFS, with a few stock metadata modules.

Furthermore, it completely changes the notion of what a Linux filesystem is. Currently, each Linux filesystem is a tightly constrained set of metadata support. reiser4 changes "tightly constrained" to "infinity".
While that freedom is certainly liberating, it also has obvious
support costs due to new admin paradigms and customer configuration possibilities.

I don't want to be the distro support person trying to fix a crash in "reiser4", where the customer has secretly replaced the standard inode data structure with a plugin written by an intern, and secretly replaced the directory algorithm with a closed source plugin from PickYourVendor.
Trying picking through that mess with a filesystem debugger.

The reiser4 plugin system is far better implemented as the VFS level, as an optional library, like fs/libfs.c. Such a library would contain factored-out infrastructure common to all filesystems that support multiple versions of the same object type, thereby shrinking the "real" filesystem code. Then add a tiny bit of code that allows addition and removal of object types within each filesystem.

At that point, you have reduced a Linux filesystem to binding between a common name ("reiser4", "ext3") and a set of cooperating modules.

A very interesting train of thought. But not one that belongs in the kernel in its current form. If you hide a second VFS inside fs/reiser4, I GUARANTEE that you will get the locking and multi-threading wrong. I guarantee your code will be under-reviewed, and NAK'd. You lose all the testing that would be gained by others travelling your code paths, if the code was implemented generically in fs/libplugin.c rather than fs/reiser4/*

To move forward towards the goal of merging reiser4 into the kernel, you must recognize two distinctly SEPARATE ISSUES, and deal with them separately:

1) merging the latest whiz-bang collection of filesystem algorithms, under the filesystem name "reiser4"
2) adding the plugin concept to the Linux VFS

I don't know if #2 will pass consensus, but I certainly think we should at least experiment in that direction. Your guys definitely have a lot of good ideas, and I'm glad Linux is seeing a fresh injection of new ideas.

I just don't think its a good idea to merge a filesystem called "foo" that can be completely redefined in the field. Or least, not without thinking a lot more on the impact. Its a VFS _and_ a filesystem, and should be evaluated as such.

Jeff

#190 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

In other words, if a filesystem wants to do something fancy, it needs to do so WITH THE VFS LAYER, not as some plugin architecture of its own.

Where does VFS store the plugin ids that specify per file variations? /etc/fstab? Also, is (current) VFS the interface for specifying where the hash directory plugin goes (specifies what order directory entries are within the directory)? What about the node layout plugin? The disk format plugin? Etc.? Our approach is different, but it has reasons.

We eliminated the layer of indirection that Hellwig objected to, in which VFS called the plugin which called its method.

(Let us try to avoid arguments over whether if you extend VFS it is still called VFS or is called reiser4's plugin layer, agreed?)

Regarding copyright, these plugins are compiled in. I have resisted dynamically loaded plugins for now, for reasons I will not go into here.

You can either portray reiser4 as duplicating VFS, or you can portray it as taking it to the next level, in which files (objects with classes and methods) vary rather than solely filesystems. I would prefer the latter.;-)

If you agree with taking it to the next level, then it is only to be expected that there are things that aren't well suited as they are, like parsing /etc/fstab when you have a trillion files. It is not very feasible to do it for all of the filesystems all at once given finite resources, it needs a prototype.

Quoted 6 lines

We
already have exactly the plugin interface we need, and it literally _is_ the VFS interfaces - you can plug in your own filesystems with "register_filesystem()", which in turn indirectly allows you to plug in your per-file and per-directory operations for things like lookup etc.

Quoted 9 lines

If that isn't enough, then the filesystem shouldn't make its own internal plug-in architecture that bypasses the VFS layer and exposes functionality that isn't necessarily sane. For example, reiser4 used to have (perhaps still does) these cool files that can be both directories and links, and I don't mind that at all, but I _do_ mind the fact that when Al Viro (long long ago) pointed out serious locking issues with them, those seemed to be totally brushed away.

We disabled them, and we won't enable them until them until the bug is fixed. It is fixable, but not within this year's programmer resources to fix it. I thank him for pointing out the bug, and it is not trivial to fix it.

Quoted 3 lines

I don't think I've ever had the cojones to argue with Al..

Linux needs all kinds of people, not just the kind that can audit locking and copy Plan 9 well (which was very valuable to do), but now that Linux is large in the market it also needs those who can take it
where Plan 9 has not already been. Why should we remain technology
trailers instead of moving into the role of leaders?

We have finite resources. We can give you a working filesystem with roughly twice the IO performance of the next fastest you have that does not disturb other filesystems,. (4x once the compression plugin is fully debugged). It also fixes various V3 bugs without disturbing that code with deep fixes. We cannot take every advantage reiser4 has and port it to every other filesystem in the form of genericized code as a prerequisite for going in, we just don't have the finances. Without plugins our per file compression plugins and encryption plugins cannot work. We can however let other filesystems use our code, and cooperate as they extend it and genericize it for their needs. Imposing code on other development teams is not how one best leads in open source, one sets an example and sees if others copy it. That is what I propose to do with our plugins. If no one copies, then we have harmed no one. Reasonable?

Hans

#192 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover

Let me put it from my perspective and stop pretending to be unbiased, so others can see where I am coming from. No one was interested in our plugins. We put the design on a website, spoke at conferences, no one but users were interested. No one would have conceived of having plugins if not for us. Our plugins affect no one else. Our self-contained code should not be delayed because other people delayed getting interested in our ideas and now they don't want us to have an advantage from leading. If they want to some distant day implement generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code. It is not fair or just to do otherwise. It also prevents users from getting advances they could be getting today, for no reason. Our code will not be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

#236 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hua Zhong
Quoted 6 lines · Hua Zhong

I remember someone said something along the lines of "Linux is evolution, not revolution". To me it seems unreasonable to put all the "revolutionary" VFS burden upon reiserfs team. It's not practical.

Thanks for saying that Hua. We have a guy named Nate Diller, who probably could fix VFS up pretty nicely if Namesys did not have him earning the consulting income the rest of the team lives on (he is doing
io scheduler work at the moment). That said, he would need a lot of
time, and stopping reiser4 inclusion to await his work merely ensures his work will never happen.

#331 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Olivier Galibert
Quoted 10 lines · Olivier Galibert
Quoted 2 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

Because Nigel has too much of code to start with, and refuses to fix his design because it would invalidate all the stabilization work.

Plus Nigel did not do very good job with submitting those patches.
Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#187 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Jeff Garzik

Jeff Garzik <[email protected]> wrote:

[...]

Quoted 5 lines

It is then simple to follow that train of logic: why not make it easy to replace the directory algorithm [and associated metadata]? or the file data space management algorithms? or even the inode handling? why not allow customers to replace a stock algorithm with an exotic, site-specific one?

IMVHO, such experiments should/must be done in userspace. And AFAICS, they can today.

Go wild, but leave the kernel alone.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#305 lore raw

Re: the ' 'official' point of view' expressed by kernelnewbies.org regarding reiser4 inclusion

In reply to Hans Reiser
Quoted 11 lines · Hans Reiser
Quoted 8 lines · Adrian Bunk

But you can not tell based on klive data whether the ratio of reiser4:ext3 users in the world is more like 1:5, 1:500 or 1:50000.

<-- snip -->

I can't prove that the 1:5 ratio is wrong, but the point is that claiming a 1:5 ratio was true based on the klive data is not better than claiming it based on no data.

Quoted 7 lines

Yes, but I have been surprised that linux conference attendees all know about reiser4, so I think it is consistent with the notion that reiser4 usage may well be much much higher than one would expect of a filesystem that is not in the main tree, and which requires a lot of hassle to install. It is a datapoint. I think the level of hobbyist enthusiasm seen suggests that distros may gain market advantage by adding reiser4 support.....

Great! Then /sit down and do the work to get it into the vanilla kernel/. You know then it is not a wasted effort... and you /do/ know by now (or should) that playing the "politics"/"whining" card goes nowhere.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#314 lore raw
In reply to Hans Reiser
Quoted 1 line

remember that, and try to all get along.....

FWIW the last thing I can be interested about is to get along with somebody that gratuitously attacks me every time he can.

#188 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Horst H. von Brand
Quoted 4 lines · Horst H. von Brand

Jeff Garzik <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 5 lines

It is then simple to follow that train of logic: why not make it easy to replace the directory algorithm [and associated metadata]? or the file data space management algorithms? or even the inode handling? why not allow customers to replace a stock algorithm with an exotic, site-specific one?

IMVHO, such experiments should/must be done in userspace. And AFAICS, they can today.

inode handling? Really?

But what's wrong with people doing such experiments outside the kernel?
AFAICS, "exotic, site-specific one" is not something that would be
considered for inclusion.

#189 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 4 lines · David Masover

But what's wrong with people doing such experiments outside the kernel? AFAICS, "exotic, site-specific one" is not something that would be considered for inclusion.

Here's a few ground rules at least from my viewpoint:

- as long you call them "plugins" and treat them as such, I (and I suspect a lot of other people) are totally uninterested, and in fact, a lot of people will suspect that the primary aim is to either subvert the kernel copyright rules, or at best to create a mess of incompatible semantics with no sane overlying rules for locking etc.

- the kernel does not, and _will_ not, support "hooks" that aren't actually used (and make sense) by normal kernel uses, for largely the same reasons. Interfaces need to be architected, make sense, and have real and valid GPL'd uses.

In other words, if a filesystem wants to do something fancy, it needs to do so WITH THE VFS LAYER, not as some plugin architecture of its own. We already have exactly the plugin interface we need, and it literally _is_ the VFS interfaces - you can plug in your own filesystems with "register_filesystem()", which in turn indirectly allows you to plug in your per-file and per-directory operations for things like lookup etc.

If that isn't enough, then the filesystem shouldn't make its own internal plug-in architecture that bypasses the VFS layer and exposes functionality that isn't necessarily sane. For example, reiser4 used to have (perhaps still does) these cool files that can be both directories and links, and I don't mind that at all, but I _do_ mind the fact that when Al Viro (long long ago) pointed out serious locking issues with them, those seemed to be totally brushed away.

So as far as I'm concerned, the problem with reiser4 is that it hasn't tried to work with the VFS people. Now, I realize that the main VFS people aren't always easy to work with (Al and Christoph, take a bow), but that doesn't really change the basic facts. Al in particular is _always_ right. I don't think I've ever had the cojones to argue with Al..

Linus

#191 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 8 lines · Hans Reiser
Quoted 3 lines · Linus Torvalds

In other words, if a filesystem wants to do something fancy, it needs to do so WITH THE VFS LAYER, not as some plugin architecture of its own.

(Let us try to avoid arguments over whether if you extend VFS it is still called VFS or is called reiser4's plugin layer, agreed?)

Ok, assuming you actually extend the VFS. The point is that if we want plugins, we don't have to implement them in ext3, but we have to put the plugin interface somewhere standard that is obviously not part of one filesystem (the VFS is the place) so that ext3 can implement a plugin system without having to read or touch a line of reiser4 code, and without compiling reiser4 into the kernel.

It may ultimately not be any different, technically. This seems more like an organizational and political thing. But that doesn't make it less important or valid.

Quoted 2 lines

Regarding copyright, these plugins are compiled in. I have resisted dynamically loaded plugins for now, for reasons I will not go into here.

Good point, there's no GPL issue here. Plugins will either not be distributed (used internally) or distributed as GPL.

Quoted 5 lines

If you agree with taking it to the next level, then it is only to be expected that there are things that aren't well suited as they are, like parsing /etc/fstab when you have a trillion files. It is not very feasible to do it for all of the filesystems all at once given finite resources, it needs a prototype.

Doesn't have to be in fstab, I hope, but think of it this way: ext3 uses JBD for its journaling. As I understand it, any other filesystem can also use JBD, and ext3 is mostly ext2 + JDB.

So, make the plugin interface generic enough that it compliments the VFS, doesn't duplicate it, and doesn't exist as part of Reiser4 (and requires Reiser4 to be present). This may be just a bunch of renaming or a lot of work, I don't know, but I suspect it would make a lot of people a lot happier.

Quoted 7 lines

We have finite resources. We can give you a working filesystem with roughly twice the IO performance of the next fastest you have that does not disturb other filesystems,. (4x once the compression plugin is fully debugged). It also fixes various V3 bugs without disturbing that code with deep fixes. We cannot take every advantage reiser4 has and port it to every other filesystem in the form of genericized code as a prerequisite for going in, we just don't have the finances.

This is a very compelling argument to me, but that's preaching to the choir, I've been running Reiser4 since before it was released, and before it looked like it was going to be stable anytime soon.

It may be bold of me to speak for the LKML, but I think the general consensus is:

The speed of a nonworking program is irrelevant -- no one cares how fast it is if it breaks things, either now or in the future. Currently, the concern is that it breaks things in the future, like adding plugin support to other filesystems.

And no one else cares what your finances are. Not out of compassion, but out of practicality. For instance, it would be a huge financial benefit to me if the kernel displayed, in big bold letters while booting, that DAVID MASOVER WROTE THIS! (I'm sure Linus knows what I'm talking about.) It would also be untrue in my case, and pointless for everyone else in the kernel, so I have to find another way to make money.

This is because one way Linux stays ahead of the competition (technologically) is by having quality be a much greater motivation than money.

Quoted 7 lines

Without
plugins our per file compression plugins and encryption plugins cannot work. We can however let other filesystems use our code, and cooperate as they extend it and genericize it for their needs. Imposing code on other development teams is not how one best leads in open source, one sets an example and sees if others copy it. That is what I propose to do with our plugins. If no one copies, then we have harmed no one. Reasonable?

Someone still has to maintain the FS. Anyway, like I said, this is a very compelling argument for me, but code speaks louder than words. Maybe, if you insist it's not in the VFS, maybe use some insanely simple FS like RomFS to demonstrate another FS using plugins?

Do that, and put it in the VFS. Maybe implement something like cramfs as a romfs plugin (another demo). Maybe even per-file -- implement zisofs as isofs + compression plugin. I think that would effectively kill any argument that plugins are bad because they are only in Reiser4.

Beyond that is all marketing, I guess. The word "plugin" is not helping here, too many people remember Plugins like Macromedia Flash...

#235 lore raw

RE: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 3 lines

Doesn't have to be in fstab, I hope, but think of it this
way: ext3 uses JBD for its journaling. As I understand it, any other filesystem can also use JBD, and ext3 is mostly ext2 + JDB.

The fact that no other major journaling filesystems use JBD except EXT3 might make this idea less appealing..

You can also say "other filesystems CAN use plugin". But who actually want to use it now? Even if there are people that want to use it, they should help doing the work.

I remember someone said something along the lines of "Linux is evolution, not revolution". To me it seems unreasonable to put all the "revolutionary" VFS burden upon reiserfs team. It's not practical.

Hua

#193 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser

Dnia Fri, 28 Jul 2006 15:34:36 +0200, Hans Reiser <[email protected]> napisa³:

Quoted 22 lines

Let me put it from my perspective and stop pretending to be unbiased, so others can see where I am coming from. No one was interested in our plugins. We put the design on a website, spoke at conferences, no one but users were interested. No one would have conceived of having plugins if not for us. Our plugins affect no one else. Our self-contained code should not be delayed because other people delayed getting interested in our ideas and now they don't want us to have an advantage from leading. If they want to some distant day implement generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code. It is not fair or just to do otherwise. It also prevents users from getting advances they could be getting today, for no reason. Our code will not be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Quoted 4 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

It just hited me that 90% of mails (those I've read and remember) in which You guys are talking why r4 should or should not be merged did not contain a patch or not even a line of code as a reference, most of complains feels so abstract and ahead of time, there is nothing wrong with planing ahead but does it still makes sense when nobody else actually said that he would want to use it in future? Can't it be pusshed up to vfs later if it proves itself and there is demand for it? My question is what does it break now? It's been in mm for some time now, what troubles does it couse? Did anyone complained that it breaks mm and should be dropped?
I'm just a end user and have no idea of linux internals, but all the fuzz about r4 seems so political and it's not just me who things so.

#194 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 4 lines · Łukasz Mierzwa

It just hited me that 90% of mails (those I've read and remember) in which You guys are talking why r4 should or should not be merged did not contain a patch or not even a line of code as a reference, most of complains feels so abstract and ahead of time, there is nothing wrong with planing ahead

Quoted 8 lines

but does it still makes sense when nobody else actually said that he would want to use it in future? Can't it be pusshed up to vfs later if it proves itself and there is demand for it?
My question is what does it break now? It's been in mm for some time now, what troubles does it couse? Did anyone complained that it breaks mm and should be dropped?
I'm just a end user and have no idea of linux internals, but all the fuzz about r4 seems so political and it's not just me who things so.

Agreed. As an outsider looking in trying to understand this all, it sounds like Hans can't win either way. People complain if the plugin code is contained entirely within Reiser4 and people complain if he tries to add functionality to the VFS that only Reiser4 will likely use for the foreseeable future. Or maybe ever use, we don't know yet?

Ignoring pseudo files which Hans has pulled for now anyways, what difference is their between Reiser4 plugins and what EXT3 does with mount options like dir_index? 3rd party plugins can't be dynamically loaded, and some of them need mount options to enable them even after they are compiled in. So people can't "mistakenly" load some plugin that corrupts their data, or changes the on-disk format making it impossible to downgrade kernel versions.

Is it the fact that Reiser4 plugins can be enabled/disable on a per file basis?

If you don't use the word "plugin", how is adding dir_index code to EXT3 any different then adding encryption/compression code to Reiser4?

Signature · 1 line
Mike Benoit <[email protected]>
#196 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

plugins if not for us. Our plugins affect no one else. Our self-contained code should not be delayed because other people delayed

And at the moment, I can still use Reiser4. If I ever make a distro, I will include Reiser4 support, probably as the default FS. That will help with getting into the kernel.

So, why is it that it's urgent to get into the kernel? It will have to be bootstrapped one way or another -- either get it into the kernel so distros are more likely to include it, or get it into distros so the kernel is more likely to include it.

But this is exactly the kind of thing that has happened before. With XFS, with Nvidia even -- clean it up, do it the way the kernel people want you to, because they're the ones who will have to maintain it for 20 years, and make sure it doesn't stop working or break anything else.

Quoted 6 lines

advantage from leading. If they want to some distant day implement generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code. It is not fair or just to do otherwise. It also prevents users from getting advances they could be getting today, for no reason.

It prevents users from doing nothing.

Quoted 3 lines

Our code will not
be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

Quoted 5 lines

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Let me put this in perspective the best way I know how, with an inane analogy:

Suppose there's a band. A good band, full of impossible superstars, led by a benevolent dictator -- for the sake of argument, let's call him Elvis. (the King -- dictator...) The band's doing really well, and Elvis & crew are getting paid fairly well just to share their music.

(Ok, maybe Elvis didn't write anything, but bear with me...)

Now, along comes a young Jimi Hendrix. He wants to be in the band, and Elvis says "Sure, just come up with a song we like and we'll play it, and you can even play it with us!" Sounds like a pretty good deal, so Jimi goes and tells all his friends, a couple of girls...

Now, Jimi finishes his song, Elvis listens to it, and if you know anything about the music Elvis did and the music Hendrix did, you can imagine what happens next. Elvis says "This song just isn't us. But if you change it here, and here, and maybe here, we'll play it."

Jimi is devastated. He'd been counting on playing it with them that night, and if he doesn't, he won't have any groupies, all his friends will laugh at him, and his life will kind of suck.

But, does anyone really think Elvis has any business singing Voodoo Child? Or Purple Haze? Is it really fair to ask Elvis to completely change his act and embarrass himself to help Jimi out?

The answer is, Jimi shouldn't have staked so much on something that was never a guarantee. And what's more, the real-life Jimi Hendrix never played with Elvis, but had a very successful band of his own. And if Elvis was still alive, seeing Jimi play might make him change his mind, maybe -- but at least with his own band, Jimi's success isn't pinned on playing with Elvis.

This analogy is flawed in many ways, aside from just being plain chronologically impossible, but while I'm sure Linus feels bad for you, I don't think it's his obligation to compromise his kernel to help you out with your financial situation. So it would help a lot if you wouldn't keep bringing it up in what should be a technical discussion.

So, if you can't make it work with VFS, then I guess you can't, and you're stuck either creating another interface which is not tied to any one filesystem and isn't tied to the VFS either, or coming up with a better (more specific) idea of how to make the Reiser4 plugin system acceptable to kernel maintainers without having to eat Ramen for a few years.

Understand that I'm putting on my devil's advocate hat right now. I'd love to see Reiser4 merged tomorrow, or a week from now, exactly as it's written today, but I just don't see it happening. I'd also love to get more technical, but I just don't know the Reiser4 internals well enough to understand the feasibility (or not) of any of my vague ideas.

#228 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 5 lines · Hans Reiser

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Using guilt as an argument in a technical discussion is a flashing red sign that says "I have no technical rebuttal"

Quoted 3 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

What delay? This is open source. There is nothing stopping you from worldwide distribution of the coolest Linux filesystem in the world.

Jeff

#197 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover
Quoted 4 lines · David Masover

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

None of the filesystems other than reiser4 have any interest in using plugins, and this whole argument over how it should be in VFS is nonsensical because nobody but us has any interest in using the functionality. The burden is on the generic code authors to prove that they will ever ever do anything at all besides complain. Frankly, I don't think they will. I think they will never produce one line of code.

Please cite one ext3 developer who is signed up to implement ext3 using plugins if they are supported by VFS.

Quoted 6 lines
Quoted 3 lines

. It also prevents users from getting advances they could be getting today, for no reason.

It prevents users from doing nothing.

Most users not only cannot patch a kernel, they don't know what a patch is. It most certainly does.

Hans

#229 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Jeff Garzik
Quoted 3 lines · Jeff Garzik

Using guilt as an argument in a technical discussion is a flashing red sign that says "I have no technical rebuttal"

Wow, that is really nervy. Let's recap this all:

* reiser4 has a 2x performance advantage over the next fastest FS (ext3), and when compression ships in a month that will double again as well as save space. See http://www.namesys.com/benchmarks.html, and then ask the [email protected] whether those benchmarks are fair representations of their experiences. This is in a field where a 25% advantage is a hard won big deal.

* we described our plugin architecture in 2001. No other FS developers were interested, only the users were, and it was presented quite a lot.

* So we implemented plugins for ourselves, because no other FS developers would possibly have supported us touching their code. (I do not say that they erred in this.)

* No one has actually made a serious case for it being genericizable when you get to the details, it is all just handwaving. I'd be surprised if >10% of it was FS independent, and unsurprised if making that 10% FS independent made the code ossified and hard to maintain. I do not in anyway claim that those who choose to implement Reiser4 plugins are not deeply affected by Reiser4 design choices. Most of the value of writing Reiser4 plugins comes from being able to reuse Reiser4 code as you choose to in the process, and if Reiser4 is not to your taste as a whole, then nobody should impose our plugins upon you. VFS is a bad enough straight jacket for FS developers, we don't need even more mandated design decisions for the FS developers to come who will be brighter than us. Actually, I would like to see Nate Diller implement a competing VFS layer, I think he would do a very good job of that.

* Here we are today, and Reiser4 plugins work. Now some say that because we did it for Reiser4 and not for every other FS, that we should be excluded from the kernel. So we are supposed to re-implement it as generic code, which will involve years of time, and then finally something will be coded and nobody but us will use it, and then they will tell us that because nobody but us wants to use it it cannot go in. If you disagree, find one ext3 developer who wants to rewrite ext3 to use plugins and change its disk format to do it.

And you have the nerve to say that this ever was a technical
discussion? Our code measurably works the best. If folks want to
imitate it, go ahead, but don't blame us for making our code work without first making those other folks's code work.

The technical rebuttal you ask for is
http://www.namesys.com/benchmarks.html.

The only time this argument gets technical is when akpm is involved. He was right about what should technically be done about batch write, which, by the way, was greeted upon completion with an if only reiser4 uses it then it should not go in response.

We are being penalized for thinking too differently, and this whole ping-pong between "no we don't want to do it your way" and "you did it your way for only you, redo it for us even though we won't ever use it" and "oh, you redid it for us but none of us want to use it, so no it is an imposition and cannot go in" is the Kafka-esque manifestation of that.

If only reiser4 wants to use something, then just let us do it in our
little corner without bothering anybody else. (Though any advice from
akpm that he has time for giving us is always welcome.) David, we
aren't asking to be in the band, we are asking to be in the jukebox. I
think enough users want to go 2x as fast that the users would benefit from our being in the jukebox.

Hans

#195 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Łukasz Mierzwa

Mike and Lukasz, please post your email to not just reiserfs-list, where only the reiserfs team will read it, but also to lkml if you could, please? Thanks for your support, user opinions count for a lot on lkml.

#198 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 3 lines

Most users not only cannot patch a kernel, they don't know what a patch is. It most certainly does.

obviously you can provide complete kernels, including precompiled ones. Most distros have a yum or apt or similar tool to suck down packages, it's trivial for users to add a site to that, so you could provide packages if you want and make it easy for them.

Signature · 1 line
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
#231 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser

Dnia Fri, 28 Jul 2006 15:34:36 +0200, Hans Reiser <[email protected]>

Quoted 22 lines · napisał:

Let me put it from my perspective and stop pretending to be unbiased, so others can see where I am coming from. No one was interested in our plugins. We put the design on a website, spoke at conferences, no one but users were interested. No one would have conceived of having plugins if not for us. Our plugins affect no one else. Our self-contained code should not be delayed because other people delayed getting interested in our ideas and now they don't want us to have an advantage from leading. If they want to some distant day implement generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code. It is not fair or just to do otherwise. It also prevents users from getting advances they could be getting today, for no reason. Our code will not be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Quoted 4 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

It just hited me that 90% of mails (those I've read and remember) in which You guys are talking why r4 should or should not be merged did not contain a patch or not even a line of code as a reference, most of complains feels so abstract and ahead of time, there is nothing wrong with planing ahead but does it still makes sense when nobody else actually said that he would want to use it in future? Can't it be pusshed up to vfs later if it proves itself and there is demand for it?
My question is what does it break now? It's been in mm for some time now, what troubles does it couse? Did anyone complained that it breaks mm and should be dropped?
I'm just a end user and have no idea of linux internals, but all the fuzz about r4 seems so political and it's not just me who things so.

#232 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser

Dnia Fri, 28 Jul 2006 15:34:36 +0200, Hans Reiser <[email protected]>

Quoted 22 lines · napisał:

Let me put it from my perspective and stop pretending to be unbiased, so others can see where I am coming from. No one was interested in our plugins. We put the design on a website, spoke at conferences, no one but users were interested. No one would have conceived of having plugins if not for us. Our plugins affect no one else. Our self-contained code should not be delayed because other people delayed getting interested in our ideas and now they don't want us to have an advantage from leading. If they want to some distant day implement generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code. It is not fair or just to do otherwise. It also prevents users from getting advances they could be getting today, for no reason. Our code will not be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Quoted 4 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

It just hited me that 90% of mails (those I've read and remember) in which You guys are talking why r4 should or should not be merged did not contain a patch or not even a line of code as a reference, most of complains feels so abstract and ahead of time, there is nothing wrong with planing ahead but does it still makes sense when nobody else actually said that he would want to use it in future? Can't it be pusshed up to vfs later if it proves itself and there is demand for it?
My question is what does it break now? It's been in mm for some time now, what troubles does it couse? Did anyone complained that it breaks mm and should be dropped?
I'm just a end user and have no idea of linux internals, but all the fuzz about r4 seems so political and it's not just me who thinks so.

#218 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 14 lines · Hans Reiser
Quoted 4 lines · David Masover

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

None of the filesystems other than reiser4 have any interest in using plugins, and this whole argument over how it should be in VFS is nonsensical because nobody but us has any interest in using the functionality. The burden is on the generic code authors to prove that they will ever ever do anything at all besides complain. Frankly, I don't think they will. I think they will never produce one line of code.

Quoted 2 lines

Please cite one ext3 developer who is signed up to implement ext3 using plugins if they are supported by VFS.

In fact, they all do:

struct inode_operations ext2_file_inode_operations; struct inode_operations ext2_dir_inode_operations; struct inode_operations ext2_special_inode_operations; struct inode_operations ext2_symlink_inode_operations; struct inode_operations ext2_fast_symlink_inode_operations;

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Quoted 3 lines

Hans

Nikita.

#199 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Arjan van de Ven
Quoted 8 lines · Arjan van de Ven
Quoted 2 lines

Most users not only cannot patch a kernel, they don't know what a patch is. It most certainly does.

obviously you can provide complete kernels, including precompiled ones. Most distros have a yum or apt or similar tool to suck down packages, it's trivial for users to add a site to that, so you could provide packages if you want and make it easy for them.

What's more, many distros patch their kernels extensively. They listen to their users, too. So if there are a lot of users wanting this to be in the kernel, let them complain -- loudly -- to their distro to patch for Reiser4.

It could be made even easier than that -- if Reiser4 is really so self-contained, it could be a whole separate set of modules, distributed on its own. Most gamers have to be content with doing something similar with the nvidia drivers -- for different reasons (licensing) but with the same results. I know Gentoo handles this automatically (emerge nvidia-kernel).

Hmm, maybe it makes it a pain to have it as a root filesystem, so that really needs distro support. And yet, we have a whole system designed specifically for being able to load modules and tweak settings before the root FS is available. It's called initrd, or more recently, initramfs. I use an old-style initrd on this box, because my root FS is on an nvidia RAID, so I have to run a program called "dmraid" before I mount my root FS -- it's really trivial for me to have Reiser4 as a module, and I do, despite it being a root FS.

I suspect that, all technical, political, and "mine is bigger" arguments aside, being available as a root FS of a distro, especially a default FS, would go a long way towards inclusion in the kernel. So all you have to do is find a reasonably popular and friendly distro, with people who are (for the moment) easier to deal with than kernel people.

Most people, if they even know what a filesystem or a kernel is, still won't bother compiling their own kernel, you're right. But that means they are more likely to be using a distro-patched kernel than a stock, vanilla one.

Is this enough to be "in the jukebox", Hans?

Of course, it's odd that I mention Gentoo, the Gentoo people (as a rule) hate ReiserFS, but there are far more distros than there are popular kernel forks. I'm sure someone will be interested.

That's assuming that making further changes (putting stuff in the VFS) is out of the question (for now).

#226 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 6 lines · Hans Reiser
Quoted 3 lines · David Masover

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

Quoted 6 lines

None of the filesystems other than reiser4 have any interest in using plugins, and this whole argument over how it should be in VFS is nonsensical because nobody but us has any interest in using the functionality. The burden is on the generic code authors to prove that they will ever ever do anything at all besides complain. Frankly, I don't think they will. I think they will never produce one line of code.

I think it's fair to say that 5-10 years from now, with different ext3 maintainers, when the Reiser4 concept has proven itself, people will want plugins for ext3, and the ext3 developers will like the idea.

ext* is one of those things that just refuses to die. I use ext3 for my /boot fs, so that I don't have to patch Grub for Reiser4, and so that at least I can mess with the bootloader from any rescue CD if something goes wrong. It's for kind of the same reason that Gentoo builds a 32-bit Grub, even though I'm booting a 64-bit OS -- just in case.

I also use ext2 for my initrd.

There are other monstrosities that will likely never die, also. ISO9660, UDF, and VFAT probably all have worse storage characteristics than Reiser4, in that as I understand it, they won't pack multiple files into a block. So Reiser4 might even make a good boot cd FS, storing things more efficiently -- but even if I'm right, those three filesystems will last forever, because they are currently well supported on every major OS, and I think one of ISO/UDF is required for DVDs.

So for whatever reason someone's using another filesystem, even if all they need is the on-disk format (my reason for ext3 /boot and vfat on USB thumbdrives), I think it's reasonable to expect that they may one day want plugin functionality. People who like Reiser filesystems will do just fine running Reiser4 with a (udf|iso|vfat) storage driver, but people who don't will just want the higher level stuff.

You're probably right and this is years of work for something that may not be worth anything, but I think this is what is going through people's heads as they look at this plugin system.

So see my comments about distro inclusion.

#219 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Nikita Danilov
Quoted 3 lines · Nikita Danilov

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Aha! So here's another question: Is it fair to ask Reiser4 to make its plugins generic, or should we be asking ext2/3 first?

#220 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 8 lines · David Masover
Quoted 3 lines · Nikita Danilov

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Aha! So here's another question: Is it fair to ask Reiser4 to make its plugins generic, or should we be asking ext2/3 first?

ext2/3 plugins are generic: in Linux every file system can implement per-object behavior by specifying {file,inode,dentry,address_space}_operations. This mechanism is provided by VFS (and, in fact, is the only way that VFS interacts with file system) and is completely generic.

Quoted 1 line

Nikita.

#332 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Pavel Machek
Quoted 7 lines · Pavel Machek
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

Quoted 6 lines
Quoted 3 lines

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

Because Nigel has too much of code to start with, and refuses to fix his design because it would invalidate all the stabilization work.

Why should he invalidate his stabilization work, and what's in need of fixing? The suspend in the kernel is great, but suspend2 includes both suspend and working resume code as well.

Quoted 2 lines

Plus Nigel did not do very good job with submitting those patches.

They apply, they work. What's not very good about that? Is this being blocked because of a spelling error, or did he mess up the indenting on "signed off by" or what? I realize you may have something other than the download version, but it's been years now.

I would like to see the working suspend (suspend2) in the kernel, and users wanting to debug the resume stuff currently in the kernel could get it under EXPERIMENTAL or some such.

Signature · 5 lines
Bill Davidsen <[email protected]>
  Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.
#200 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion )

In reply to David Masover
Quoted 1 line · David Masover

I know Gentoo handles this automatically (emerge nvidia-kernel).

I hate to say this again, but its not automatically. It requires more knowledge of ther user, and is more than asking users to patch the kernel. (I am in the support industry and I know of solaris admins, who have used that os for more than ten years ask me what the command patch does and what its impact is to production systems :-) )

Quoted 4 lines

I use an old-style initrd on this box, because my root FS is on an nvidia RAID, so I have to run a program called "dmraid" before I mount my root FS -- it's really trivial for me to have Reiser4 as a module, and I do, despite it being a root FS.

As a fact I have it as a root FS (and that too reiser4, and as a module). Surprisingly, this is a wiki article too in http://gentoo-wiki.com, by someone who's more patient enought to write one :-)

Quoted 5 lines

I suspect that, all technical, political, and "mine is bigger" arguments aside, being available as a root FS of a distro, especially a default FS, would go a long way towards inclusion in the kernel. So all you have to do is find a reasonably popular and friendly distro, with people who are (for the moment) easier to deal with than kernel people.

Its actually a matter of a hastle for the end user. That's where I would agree with Hans' comments quite earlier.

Quoted 4 lines

Most people, if they even know what a filesystem or a kernel is, still won't bother compiling their own kernel, you're right. But that means they are more likely to be using a distro-patched kernel than a stock, vanilla one.

Well, that's different, and that's the main problem in the linux empowerment that we see around ourselves. It finally revolves around the user, and as harsh as it may seem, it ultimately is the user who decides which fs is better (Give or take, they don't know the difference between a kernel or user-space. or for that matter far more basic things.)

Quoted 3 lines

Of course, it's odd that I mention Gentoo, the Gentoo people (as a rule) hate ReiserFS, but there are far more distros than there are popular kernel forks. I'm sure someone will be interested.

I do, and that's partly due to the speed of /usr/portage on reiser4, and the easiness of blowing everything and starting from scratch : -)

Cheers,
Sarath

Signature · 1 line
The gentlemen looked one another over with microscopic carelessness.
#201 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 4 lines · Sarath Menon
Quoted 1 line · David Masover

I know Gentoo handles this automatically (emerge nvidia-kernel).

I hate to say this again, but its not automatically. It requires more

My point is, there's a fairly large group of users who would be willing to do that, as they're willing to do that to get their video drivers working. Also, assuming a distro did choose to support it, the only reason nvidia-kernel isn't just distributed with a pre-built kernel (on pre-built OSes, anyway) is licensing. This isn't a problem for Reiser4, which is GPL'd.

Quoted 7 lines
Quoted 5 lines

I suspect that, all technical, political, and "mine is bigger" arguments aside, being available as a root FS of a distro, especially a default FS, would go a long way towards inclusion in the kernel. So all you have to do is find a reasonably popular and friendly distro, with people who are (for the moment) easier to deal with than kernel people.

Its actually a matter of a hastle for the end user. That's where I would agree with Hans' comments quite earlier.

Putting it in the kernel doesn't make it any more or less of a hassle for the end-user than getting distro support. I remember downloading a different set of Debian floppies which supported XFS, before XFS was mainstream.

In that sense, it's somewhat done already -- there is a Gentoo livecd that is kept patched for Reiser4. The problem with Gentoo, of course, is that if you're going to use Gentoo, you're going to be compiling your own kernel. So when it comes down to getting vanilla-sources or gentoo-sources, it wouldn't take much -- just a reiser4-sources, or a separate reiser4-module package.

Quoted 6 lines
Quoted 4 lines

Most people, if they even know what a filesystem or a kernel is, still won't bother compiling their own kernel, you're right. But that means they are more likely to be using a distro-patched kernel than a stock, vanilla one.

Well, that's different, and that's the main problem in the linux empowerment that we see around ourselves. It finally revolves around the user, and as harsh as it may seem, it ultimately is the user who decides which fs is better (Give or take, they don't know the difference between a kernel or user-space. or for that matter far more basic things.)

If I remember right, SuSe had ReiserFS as the default at one point. If even one moderately popular Linux had Reiser4 as the default FS, it would get a LOT more exposure than it would simply being included (as EXPERIMENTAL, at that) in the vanilla kernel.

Quoted 5 lines
Quoted 3 lines

Of course, it's odd that I mention Gentoo, the Gentoo people (as a rule) hate ReiserFS, but there are far more distros than there are popular kernel forks. I'm sure someone will be interested.

I do, and that's partly due to the speed of /usr/portage on reiser4, and the easiness of blowing everything and starting from scratch : -)

Yes, I love /var/lib/portage/world also.

Is /usr/portage still faster on Reiser4? I know it was when I switched, but that was years ago...

#333 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Bill Davidsen
Quoted 21 lines · Bill Davidsen
Quoted 7 lines · Pavel Machek
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

Quoted 6 lines
Quoted 3 lines

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

Because Nigel has too much of code to start with, and refuses to fix his design because it would invalidate all the stabilization work.

Why should he invalidate his stabilization work, and what's in need of fixing? The suspend in the kernel is great, but suspend2 includes both suspend and working resume code as well.

Quoted 2 lines

Plus Nigel did not do very good job with submitting those patches.

Quoted 8 lines

They apply, they work. What's not very good about that? Is this being blocked because of a spelling error, or did he mess up the indenting on "signed off by" or what? I realize you may have something other than the download version, but it's been years now.

I would like to see the working suspend (suspend2) in the kernel, and users wanting to debug the resume stuff currently in the kernel could get it under EXPERIMENTAL or some such.

You probably don't realize how offensive this is.

Actually some people have been working really hard to make the in-kernel code work and you could just respect that.

Now, please note the swsusp resume code works for me 100% of the time and I know of many people who use it without any problems. Also I know of some people for whom suspend2 doesn't work. Please take this into consideration.

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

Greetings,
Rafael

#334 lore raw

RE: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 3 lines

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

This statement worries me for several reasons.

First, I've seen repeatedly blame for "drivers". People might buy it if there was not a working suspend2. I never saw Nigal blame drivers; instead he makes sure to provide working code. In the end, users want a working suspend, and that's what counts.

Second, if the current swsusp maintainers have genuine interest for the users (not just "it works on my machine"), I would think they'd have taken a serious look at why suspend2 has a higher success rate. But from the above comment, you are not even sure why that is, and could only speculate "this probably means (drivers)". I could hardly be convinced that the current maintainers have been trying to work with Nigal to improve Linux suspend.

At last, "maintainer" is not just a title or a feeling of superiority, it's a responsibility of providing a great system to the users. I'll just quote Linus when he was flaming suspend-to-ram a couple of months ago. Replace it with suspend-to-disk at your own will:

"The fact that worries me is that suspend-to-ram DOES NOT WORK FOR PEOPLE. I have never _ever_ met a laptop or machine of mine that "just worked". I've always had to fix something, and people always end up having to do something ridiculous like unlink all modules etc.

If that isn't what worries you, you're on the wrong page."

http://article.gmane.org/gmane.linux.power-management.general/2105

Hua

#335 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 9 lines · Hua Zhong
Quoted 3 lines

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

This statement worries me for several reasons.

First, I've seen repeatedly blame for "drivers". People might buy it if there was not a working suspend2. I never saw Nigal blame drivers;

I don't _blame_ drivers. I only wanted to say this: "If Nigel knows that some drivers need to be fixed and he has working fixes for these drivers, he should have submitted these fixes for merging instead of just keeping them in suspend2". Period.

If I know of a fix for a driver, I always do my best to make sure the fix will get considered for merging at least. The problem is I'm not a driver expert and I can't provide the fixes myself.

Quoted 8 lines

instead he makes sure to provide working code. In the end, users want a working suspend, and that's what counts.

Second, if the current swsusp maintainers have genuine interest for the users (not just "it works on my machine"), I would think they'd have taken a serious look at why suspend2 has a higher success rate. But from the above comment, you are not even sure why that is, and could only speculate "this probably means (drivers)".

No, I'm not sure. If I were, I would fix the problem. Also I'm not quite sure about the higher success rate of suspend2.

Quoted 2 lines

I could hardly be convinced that the current maintainers have been trying to work with Nigal to improve Linux suspend.

This is an offensive comment.

Quoted 4 lines

At last, "maintainer" is not just a title or a feeling of superiority, it's a responsibility of providing a great system to the users. I'll just quote Linus when he was flaming suspend-to-ram a couple of months ago. Replace it with suspend-to-disk at your own will:

This also is offensive, because the current swsusp _does_ work for many people. Actually I've recently tried a couple of various machines on which it "just works", so I will not replace suspend-to-ram with suspend-to-disk below, sorry.

Moreover, if it doesn't and the problem is reported, we do our best to trace it and fix it, but sometimes we just can't do this without access to the hardware in question.

Quoted 6 lines

"The fact that worries me is that suspend-to-ram DOES NOT WORK FOR PEOPLE. I have never _ever_ met a laptop or machine of mine that "just worked". I've always had to fix something, and people always end up having to do something ridiculous like unlink all modules etc.

If that isn't what worries you, you're on the wrong page."

And that's very true. For example, the suspend-to-ram doesn't fully work on my own box and I'm not sure it ever will. However, that's not a fault of anyone who works on this, just broken BIOS and the lack of documentation. This is worrisome and many people work on improving these things, but the matter is notoriously difficult.

Greetings,
Rafael

#224 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Nikita Danilov
Quoted 31 lines · Nikita Danilov
Quoted 14 lines · Hans Reiser
Quoted 4 lines · David Masover

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

None of the filesystems other than reiser4 have any interest in using plugins, and this whole argument over how it should be in VFS is nonsensical because nobody but us has any interest in using the functionality. The burden is on the generic code authors to prove that they will ever ever do anything at all besides complain. Frankly, I don't think they will. I think they will never produce one line of code.

Quoted 2 lines

Please cite one ext3 developer who is signed up to implement ext3 using plugins if they are supported by VFS.

In fact, they all do:

struct inode_operations ext2_file_inode_operations; struct inode_operations ext2_dir_inode_operations; struct inode_operations ext2_special_inode_operations; struct inode_operations ext2_symlink_inode_operations; struct inode_operations ext2_fast_symlink_inode_operations;

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Quoted 1 line
Quoted 9 lines
Quoted 2 lines

Hans

Nikita.

So the job is already done. Good. Reiser4 can be included then.:)

Hans "The Easily Agreeable" Reiser

#221 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover
Quoted 4 lines · David Masover

Nikita Danilov wrote:

Quoted 11 lines
Quoted 5 lines

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Aha! So here's another question: Is it fair to ask Reiser4 to make its plugins generic, or should we be asking ext2/3 first?

Shhhhh....., ext* already made their plugins generic, job is done....:)

#343 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Hua Zhong
Quoted 16 lines · Hua Zhong
Quoted 1 line

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

This statement worries me for several reasons.

First, I've seen repeatedly blame for "drivers". People might buy it if there was not a working suspend2. I never saw Nigal blame drivers; instead he makes sure to provide working code. In the end, users want a working suspend, and that's what counts.

Second, if the current swsusp maintainers have genuine interest for the users (not just "it works on my machine"), I would think they'd have taken a serious look at why suspend2 has a higher success rate. But from the above comment, you are not even sure why that is, and could only speculate "this probably means (drivers)". I could hardly be convinced that the current maintainers have been trying to work with Nigal to improve Linux suspend.

At last, "maintainer" is not just a title or a feeling of superiority, it's a responsibility of providing a great system to the users. I'll just quote Linus when he was flaming suspend-to-ram a couple of months ago. Replace it with suspend-to-disk at your own will:

Quoted 5 lines

"The fact that worries me is that suspend-to-ram DOES NOT WORK FOR PEOPLE. I have never _ever_ met a laptop or machine of mine that "just worked". I've always had to fix something, and people always end up having to do something ridiculous like unlink all modules etc.

If that isn't what worries you, you're on the wrong page."

http://article.gmane.org/gmane.linux.power-management.general/2105

That's the way I feel, I'm glad someone who can't be ignored feels the same way. There is always a reason why the working version can't be put in mainline, there's always a reason why it doesn't work on my machines but I'm assured that it works on {someone}'s machine so I must have a bad {bios|setup|partition table|attitude}.

Signature · 5 lines
Bill Davidsen <[email protected]>
  Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.
#344 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Hi!

Quoted 5 lines

That's the way I feel, I'm glad someone who can't be ignored feels the same way. There is always a reason why the working version can't be put in mainline, there's always a reason why it doesn't work on my machines but I'm assured that it works on {someone}'s machine so I must have a bad {bios|setup|partition table|attitude}.

Do you have bugzilla # or are you just flaming? Please test 2.6.18-rc2.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#348 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Bill Davidsen

Hi!

Quoted 6 lines
Quoted 1 line

Plus Nigel did not do very good job with submitting those patches.

They apply, they work. What's not very good about that? Is this being blocked because of a spelling error, or did he mess up the indenting on "signed off by" or what? I realize you may have something other than the download version, but it's been years now.

Ask Nigel or search the archives. I will not repeat myself again.

Quoted 1 line

I would like to see the working suspend (suspend2) in the kernel,

Great. Help with debugging. Alternatively, help Nigel with fixing design problems and separating patches for merge -- that should keep you busy and stop you from trolling on the lists.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#345 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Hua Zhong

Hi!

Quoted 9 lines
Quoted 3 lines

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

This statement worries me for several reasons.

First, I've seen repeatedly blame for "drivers". People might buy it if there was not a working suspend2. I never saw Nigal blame drivers; instead he makes sure to provide working code. In the end, users want a working suspend, and that's what counts.

You seem to be claiming that swsusp is broken, but I do not remember any bugreport from you. So... either provide bugzilla #, or stop spreading FUD.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#214 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover

Hello David,

Quoted 4 lines · you

What's more, many distros patch their kernels extensively. They listen to their users, too. So if there are a lot of users wanting this to be in the kernel, let them complain -- loudly -- to their distro to patch for Reiser4.

Hmm, what about linspire / freespire ? Linsire is a proud reiser4 debugging sponsor as the website (http://www.namesys.com) says.

Wouldn't they want to include reiser4 in their distro first?

I removed the busy people from CC to save their time reading this.

Signature · 2 lines
Best regards,
Maciej
#215 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 8 lines · Maciej SoÅ‚tysiak

Hmm, what about linspire / freespire ? Linsire is a proud reiser4 debugging sponsor as the website (http://www.namesys.com) says.

Wouldn't they want to include reiser4 in their distro first?

Not if the mainstream kernel is not going to add it.....

Hans

#222 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover

Dnia Sat, 29 Jul 2006 20:31:59 +0200, David Masover <[email protected]> napisa³:

Quoted 9 lines
Quoted 3 lines · Nikita Danilov

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Aha! So here's another question: Is it fair to ask Reiser4 to make its plugins generic, or should we be asking ext2/3 first?

Doesn't iptables have plugins? Maybe we should make them generic so other packet filters can use them ;)

#216 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser

Dnia Sun, 30 Jul 2006 03:08:55 +0200, Hans Reiser <[email protected]> napisa³:

Quoted 11 lines
Quoted 8 lines · Maciej So³tysiak

Hmm, what about linspire / freespire ? Linsire is a proud reiser4 debugging sponsor as the website (http://www.namesys.com) says.

Wouldn't they want to include reiser4 in their distro first?

Not if the mainstream kernel is not going to add it.....

If I got it all rigth then the VFS issue is there from the moment when Hans made a request to merge reiser4 into mainstream, nobody nows what will happen tomorrow, it will go in without moving things to vfs, it can go in with moving some code to vfs or it can stay forever outside. If it's not sure then no distro will use it as a main fs (or at all) becouse if tommarow Hans will say "Ok, will move things to VFS" then for the next year or so reiser4 will be simply dead, it will take a lot of time to recode it and this change will provide a lot of bugs that will need to be debuged, it won't a be a single patch anymore as You will need to patch that adds reiser4 and a patch that alters VFS. What distro would toy with VFS? None until it's all done, fully and trully tested and merged into mainline, it won't happen quickly and it is not said that there won't be any more complains when this is done. In my opinion there is no best option for reiser4 only less evil ways, it's just too different from other fs, too independent.
Again I'm no expert, just enduser.

#202 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover
Quoted 3 lines · David Masover

Is /usr/portage still faster on Reiser4? I know it was when I switched, but that was years ago...

It is for sure. v4 is even better with fantastillions of small files than v3.

uziel

#346 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Rafael J. Wysocki
Quoted 36 lines · Rafael J. Wysocki
Quoted 33 lines · Bill Davidsen
Quoted 13 lines · Pavel Machek
Quoted 10 lines · Olivier Galibert
Quoted 4 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

Quoted 9 lines
Quoted 5 lines

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

Because Nigel has too much of code to start with, and refuses to fix his design because it would invalidate all the stabilization work.

Why should he invalidate his stabilization work, and what's in need of fixing? The suspend in the kernel is great, but suspend2 includes both suspend and working resume code as well.

Quoted 3 lines

Plus Nigel did not do very good job with submitting those patches.

They apply, they work. What's not very good about that? Is this being blocked because of a spelling error, or did he mess up the indenting on "signed off by" or what? I realize you may have something other than the download version, but it's been years now.

Quoted 10 lines
Quoted 3 lines

I would like to see the working suspend (suspend2) in the kernel, and users wanting to debug the resume stuff currently in the kernel could get it under EXPERIMENTAL or some such.

You probably don't realize how offensive this is.

Actually some people have been working really hard to make the in-kernel code work and you could just respect that.

By respect I take it you mean "don't call attention to the fact that it doesn't work for many people?"

Quoted 6 lines

Now, please note the swsusp resume code works for me 100% of the time and I know of many people who use it without any problems. Also I know of some people for whom suspend2 doesn't work. Please take this into consideration.

That's exactly the answer I've been getting for saveral years, "it works for me and my friends," and "try x.y.z release."

Quoted 4 lines

Moreover, if the swsusp's resume doesn't work for you and suspend2's resume does, this probably means that suspend2 contains some driver fixes that haven't been submitted for merging.

That's been addressed by other people, but the suspend2 patch has been submitted multiple times, if there is some driver fix it certainly has been submitted, just not accepted.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#347 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 47 lines · Bill Davidsen
Quoted 45 lines · Rafael J. Wysocki
Quoted 42 lines · Bill Davidsen
Quoted 13 lines · Pavel Machek
Quoted 10 lines · Olivier Galibert
Quoted 4 lines · Pavel Machek

So we have 1 submission for review in 11/2004 and 1 submission for -mm merge in 2006, right?

Wrong. I gave a list of dates at the beginning of the month, do you think I threw dice to get them?

Quoted 9 lines
Quoted 5 lines

And could you explain, as suspend maintainer for the linux kernel, how come code submitted for the first time two years ago and with a much better track record than the in-kernel one is still not in?

Because Nigel has too much of code to start with, and refuses to fix his design because it would invalidate all the stabilization work.

Why should he invalidate his stabilization work, and what's in need of fixing? The suspend in the kernel is great, but suspend2 includes both suspend and working resume code as well.

Quoted 3 lines

Plus Nigel did not do very good job with submitting those patches.

They apply, they work. What's not very good about that? Is this being blocked because of a spelling error, or did he mess up the indenting on "signed off by" or what? I realize you may have something other than the download version, but it's been years now.

I would like to see the working suspend (suspend2) in the kernel, and users wanting to debug the resume stuff currently in the kernel could get it under EXPERIMENTAL or some such.

Quoted 7 lines
Quoted 5 lines

You probably don't realize how offensive this is.

Actually some people have been working really hard to make the in-kernel code work and you could just respect that.

By respect I take it you mean "don't call attention to the fact that it doesn't work for many people?"

Sorry, you won't get anywhere by insulting people. If you want to call attention to a problem, please file a bug report.

Greetings,
Rafael

#223 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Łukasz Mierzwa
Quoted 8 lines · £ukasz Mierzwa

Dnia Sat, 29 Jul 2006 20:31:59 +0200, David Masover <[email protected]> napisa³:

Quoted 6 lines
Quoted 3 lines · Nikita Danilov

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Quoted 5 lines
Quoted 3 lines

Aha! So here's another question: Is it fair to ask Reiser4 to make its plugins generic, or should we be asking ext2/3 first?

Doesn't iptables have plugins? Maybe we should make them generic so other packet filters can use them ;)

Hey, yeah! I mean, not everyone wants to run the ipchains emulation on top of iptables! Some people really want to run ipchains with iptables plugins!

</sarcasm>

It is REALLY time for this discussion to get technical again, and to go way, way over my head. And it's time for me to go build my MythTV box, and see if I can shake out some Reiser4 bugs.

#233 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

Let me put it from my perspective and stop pretending to be unbiased, so others can see where I am coming from.

OK, but /that/ was pretty clear from day one...

Quoted 2 lines

No one was interested in our
plugins.

Should tell you something...

Quoted 2 lines

We put the design on a website, spoke at conferences, no one
but users were interested.

Again, should tell you something... "Look, a cool new gadget nobody ever imagined before" sure attracts lots of people. Intriguing. Play around a bit. Go for next "last novel gadget". Rinse, repeat.

Quoted 2 lines

No one would have conceived of having
plugins if not for us.

Perhaps because nobody else sees any sense in them?

Quoted 1 line

Our plugins affect no one else.

Wrong. Others will want plugins if they are any use, at the very least. And then there is the possibility of /massive/ maintainance problems ("So, you /are/ using Reiser 4, the broken file has attached plugin A version 5.3, and is reached through a "directory" with plugin D 3.4rc5, and ..."). /I/ wouldn't want to be at the receiving end of bug reports which after a dozen rounds boil down to this.

Quoted 2 lines

Our
self-contained code should not be delayed

Not your call to make.

Quoted 3 lines

because other people delayed
getting interested in our ideas and now they don't want us to have an advantage from leading.

Your problem, not Linux'.

Quoted 4 lines

If they want to some distant day implement
generic plugins, for which they have written not one line of code to date, fine, we'll use it when it exists, but right now those who haven't coded should get out of the way of people with working code.

But they are! Just branch the kernel, and be done with it.

Quoted 2 lines

It is not
fair or just to do otherwise.

/You/ are asking the kernel developers for a /huge/ favor. Even totally go out of their ways, and acting contrary to their set ways and beliefs. Saying "no" to that can't be called "unfair"...

Quoted 2 lines

It also prevents users from getting
advances they could be getting today, for no reason.

OK, so you have /never/ seen any reasons given here for not placing Reiser 4 into the kernel? Strange...

Quoted 3 lines

Our code will not
be harder to change once it is in the kernel, it will be easier, because there will be more staff funded to work on it.

Right. Just like Reiser 3 right now.

Quoted 6 lines

As for this "we are all too grand to be bothered with money to feed our families" business, building a system in which those who contribute can
find a way to be rewarded is what managers do. Free software
programmers may be willing to live on less than others, but they cannot live on nothing, and code that does not ever ship means living on nothing.

Then go do something else...

Quoted 3 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

How is that important in any way for the Linux kernel? This is not (and has not been for quite some time now) an experimental operating system. And it has /never/ been a dumpling ground for the next grand idea, it has always been about sound engineering.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#203 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Christian Trefzer

Hmm, looks like I have a partition to re-format now.

Wil

Quoted 12 lines · Christian Trefzer
Quoted 3 lines · David Masover

Is /usr/portage still faster on Reiser4? I know it was when I switched, but that was years ago...

It is for sure. v4 is even better with fantastillions of small files than v3.

uziel

#234 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Horst H. von Brand
Quoted 9 lines
Quoted 4 lines

If reiser4 is delayed enough, for reasons that have nothing to do with its needs, and without it having encumbered anyone else, it won't be ahead of the other filesystems when it ships.

How is that important in any way for the Linux kernel? This is not (and has not been for quite some time now) an experimental operating system. And it has /never/ been a dumpling ground for the next grand idea, it has always been about sound engineering.

One of the groundbreaking things about Linux' (modular) development model was, I thought, that it can succeed at both... not necessarily at the same time, but for different users?

--T

#204 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Wil Reichert

Hi Wil,

Quoted 1 line · Wil Reichert

Hmm, looks like I have a partition to re-format now.

In order to avoid having to pull the whole tree via rsync again, you might want to grab my script from the list and adapt it to your needs.

Kind regards,
Chris

#205 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 3 lines · Christian Trefzer

In order to avoid having to pull the whole tree via rsync again, you might want to grab my script from the list and adapt it to your needs.

Of course, you can tar it up manually instead. Silly me, but after approx. 9h of studying, little wonder ; )

Kind regards,
Chris

#336 lore raw

RE: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Rafael J. Wysocki
Quoted 10 lines

I don't _blame_ drivers. I only wanted to say this: "If Nigel knows that some drivers need to be fixed and he has working fixes for these drivers, he should have submitted these fixes for merging instead of just keeping them in suspend2". Period.

If I know of a fix for a driver, I always do my best to make sure the fix will get considered for merging at least. The problem is I'm not a driver expert and I can't provide the fixes myself.

Suspend2 patch is open source. You can always take a look. Moreover, if someone claims suspend2 isn't ready for merge, or the patches Nigal has submitted aren't up to standards, one would guess he'd have done so already. So I don't understand what the above means. Have you asked Nigal whether he had any driver fixes? If you guys are really working together, isn't it very easy to do?

Quoted 6 lines

And that's very true. For example, the suspend-to-ram doesn't fully work on my own box and I'm not sure it ever will. However, that's not a fault of anyone who works on this, just broken BIOS and the lack of documentation. This is worrisome and many people work on improving these things, but the matter is notoriously difficult.

I'm not exactly an expert, but I don't think suspend-to-ram is more difficult than suspend-to-disk (probably quite the contrary), and there are a lot in common.

I am sorry that you find some of the comments offensive, but there isn't really anything personal. It's merely a reflection of the frustration from linux users wrt suspend OVER THE YEARS. I know you guys are hard working, but in the end it's a top-quality suspend that counts, and people's patience has been wearing out, especially when there has been an out-of-tree implementation available for a long time already. Everyone wants to see the current maintainers take a more humble and proactive attitude when working with Nigal instead of just dismissing his "bad design" and bashing his patch quality. It's so easy to block someone's initiative than making it better.

Hua

#206 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Christian Trefzer
Quoted 6 lines · Christian Trefzer
Quoted 2 lines · Christian Trefzer

In order to avoid having to pull the whole tree via rsync again, you might want to grab my script from the list and adapt it to your needs.

Of course, you can tar it up manually instead. Silly me, but after approx. 9h of studying, little wonder ; )

In fact, the official install guide tells you to download a snapshot tarball first, then start syncing.

#337 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Hua Zhong
Quoted 6 lines · Hua Zhong
Quoted 6 lines

I don't _blame_ drivers. I only wanted to say this: "If Nigel knows that some drivers need to be fixed and he has working fixes for these drivers, he should have submitted these fixes for merging instead of just keeping them in suspend2". Period.

Quoted 6 lines
Quoted 4 lines

If I know of a fix for a driver, I always do my best to make sure the fix will get considered for merging at least. The problem is I'm not a driver expert and I can't provide the fixes myself.

Suspend2 patch is open source. You can always take a look.

swsusp is open source. You can always take a look. And you can always submit a patch.

Quoted 1 line

Moreover, if someone claims suspend2 isn't ready for merge, or the

Moreover, if someone claims swsusp is broken, they should attach bugzilla id.

Quoted 3 lines

I'm not exactly an expert, but I don't think suspend-to-ram is more difficult than suspend-to-disk (probably quite the contrary), and there are a lot in common.

As you said, you do not know what you are talking about.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#35 lore raw
In reply to Horst H. von Brand
Quoted 4 lines
Quoted 2 lines

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly,

Right. Plan ahead.

Ok: Assume that i've read the mke2fs manpage and added more inodes to my filesystem.

So: What happens if i need to grow my filesystem by 200% after 1-2 years? Can i add more inodes to Ext3 on-the-fly ?

A filesystem with a fixed number of inodes (= not readjustable while mounted) is ehr.. somewhat unuseable for a lot of people with big and *flexible* storage needs (Talking about NetApp/EMC owners)

Why are a lot of Solaris-people using (buying) VxFS? Maybe because UFS also has such silly limitations? (..and performs awkward with trillions of files..?..)

Ext3 may be a fine and stable Filesystem and works well for a lot of people. But there are also a lot of people who need 'something' better
like: VxFS, WAFL and Reiser4.

Btw: Do you know Adics 'StorNext Filesystem' ? IMHO Ext3 will never be able to do such things... But with Reiser4.. .. if someone wrote a plugin .. ;-)

Regards,
Adrian

#225 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 1 line · Hans Reiser

Nikita Danilov wrote:

[...]

Quoted 9 lines
Quoted 8 lines

As you see, ext2 code already has multiple file "plugins", with persistent "plugin id" (stored in i_mode field of on-disk struct ext2_inode).

Nikita.

So the job is already done. Good. Reiser4 can be included then.:)

Indeed, namesys already implemented requirements pointed to by Christoph Hellwig, as far as meta-data plugins are concerned, so this issue is not an obstacle.

Quoted 3 lines

Hans "The Easily Agreeable" Reiser

PS: please, don't trim CC lists.

Nikita.

#227 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Hans Reiser
Quoted 7 lines · Hans Reiser
Quoted 3 lines · David Masover

If indeed it can be changed easily at all. I think the burden is on you to prove that you can change it to be more generic, rather than saying "Well, we could do it later, if people want us to..."

None of the filesystems other than reiser4 have any interest in using plugins,

Then they are probably nonsense...

Quoted 5 lines

and this whole argument over how it should be in VFS is
nonsensical because nobody but us has any interest in using the functionality. The burden is on the generic code authors to prove that they will ever ever do anything at all besides complain. Frankly, I don't think they will. I think they will never produce one line of code.

It is /your/ burden to show that they are worthwhile. And if they are, thay aren't worthwhile only in your pet filesystem...

Quoted 2 lines

Please cite one ext3 developer who is signed up to implement ext3 using plugins if they are supported by VFS.

Have you asked?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

. It also prevents users from getting advances they could be getting today, for no reason.

It prevents users from doing nothing.

Most users not only cannot patch a kernel, they don't know what a patch is. It most certainly does.

Simple answer: Fork the kernel with all your grandiose plans, and see which one wins. Open source, remember? It is not as if your only chance is to get it into the official kernel.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#338 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Pavel Machek
Quoted 8 lines · Pavel Machek
Quoted 5 lines · Hua Zhong
Quoted 5 lines

I don't _blame_ drivers. I only wanted to say this: "If Nigel knows that some drivers need to be fixed and he has working fixes for these drivers, he should have submitted these fixes for merging instead of just keeping them in suspend2". Period.

If I know of a fix for a driver, I always do my best to make sure the fix will get considered for merging at least. The problem is I'm not a driver expert and I can't provide the fixes myself.

Quoted 8 lines
Quoted 3 lines

Suspend2 patch is open source. You can always take a look.

swsusp is open source. You can always take a look. And you can always submit a patch.

But you can't get the patch accepted, that's issue causing all this discussion.

Quoted 9 lines
Quoted 3 lines

Moreover, if someone claims suspend2 isn't ready for merge, or the

Moreover, if someone claims swsusp is broken, they should attach bugzilla id.

Quoted 10 lines
Quoted 5 lines

I'm not exactly an expert, but I don't think suspend-to-ram is more difficult than suspend-to-disk (probably quite the contrary), and there are a lot in common.

As you said, you do not know what you are talking about.

That's why people are frustrated. You blow off anyone who tells you the code doesn't work. Do you really think Linus doesn't know what he's talking about when he reported that it didn't work for him? Hua Zhong being "Not an expert" is not the same as not knowing what he's talking about.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#36 lore raw
In reply to Adrian Ulrich

Adrian Ulrich schrieb am 2006-07-31:

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

And EXT3 imposes practical limits that ReiserFS doesn't as well. The big one being a fixed number of inodes that can't be adjusted on the fly,

Right. Plan ahead.

Ok: Assume that i've read the mke2fs manpage and added more inodes to my filesystem.

So: What happens if i need to grow my filesystem by 200% after 1-2 years? Can i add more inodes to Ext3 on-the-fly ?

Since you "grow", you'll be using resize2fs (or growfs or mkfs -G for UFS). resize2fs and the other tools do exactly that: add inodes - and you could easily have told this either from reading the resize2fs code or just trying it on a temp file:

-- create file system
dd if=/dev/zero of=/tmp/foo bs=1k count=50000
/sbin/mke2fs -F -j /tmp/foo

-- check no. of inodes
/sbin/tune2fs -l /tmp/foo | grep -i inode | head -2
# Inode count: 12544
# Free inodes: 12533

-- resize
/sbin/e2fsck -f /tmp/foo
dd if=/dev/zero bs=1k count=50000 >>/tmp/foo
/sbin/resize2fs /tmp/foo

-- check no. of inodes
/sbin/tune2fs -l /tmp/foo | grep -i inode
# Inode count: 23296
# Free inodes: 23285

Trying the same after mke2fs -b 1024 -i 1024 shows that the inode density will continue to be respected.

FreeBSD 6.1's growfs(8) increases the number of inodes. This is documented to work since 4.4.

Solaris 8's mkfs -G also increases the number of inodes and apparently also works for mounted file systems.

This looks rather like an education issue rather than a technical limit.

Quoted 3 lines

A filesystem with a fixed number of inodes (= not readjustable while mounted) is ehr.. somewhat unuseable for a lot of people with big and *flexible* storage needs (Talking about NetApp/EMC owners)

Which is untrue at least for Solaris, which allows resizing a life file system. FreeBSD and Linux require an unmount.

Quoted 3 lines

Why are a lot of Solaris-people using (buying) VxFS? Maybe because UFS also has such silly limitations? (..and performs awkward with trillions of files..?..)

Well, such "silly limitations"... looks like they are mostly hot air spewn by marketroids that need to justify people spending money on their new filesystem.

The only problem remains if you grossly overestimate the average file size and with it underestimate the number of inodes needed. But even then, I'd be interested to know if that's a real problem for systems such as ZFS.

Signature · 1 line
Matthias Andree
#207 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Christian Trefzer

=)

That was sorta the plan.

Any idea how the fragmentation resulting from re-syncing the tree affects performance over time?

Wil

Quoted 13 lines · Christian Trefzer
Quoted 3 lines · Christian Trefzer

In order to avoid having to pull the whole tree via rsync again, you might want to grab my script from the list and adapt it to your needs.

Of course, you can tar it up manually instead. Silly me, but after approx. 9h of studying, little wonder ; )

Kind regards,
Chris

#208 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

Quoted 6 lines · Wil Reichert

=)

That was sorta the plan.

Any idea how the fragmentation resulting from re-syncing the tree affects performance over time?

Try to post replies at the bottom, or below the context.

Yes, it does affect it a lot. I have no idea how much, and I've never benchmarked it, but purely subjectively, my portage has gotten slower over time.

#37 lore raw
In reply to Matthias Andree

Hello Matthias,

Quoted 1 line

This looks rather like an education issue rather than a technical limit.

We aren't talking about the same issue: I was asking to do it on-the-fly. Umounting the filesystem, running e2fsck and resize2fs is something different ;-)

Quoted 2 lines

Which is untrue at least for Solaris, which allows resizing a life file system. FreeBSD and Linux require an unmount.

Correct: You can add more inodes to a Solaris UFS on-the-fly if you are lucky enough to have some free space available.

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a big deal with VxFS because such a problem is fixable within seconds. What would have happened if he had used UFS? mkfs -G wouldn't work because he had no additional Diskspace left... *ouch*..

Quoted 3 lines

Well, such "silly limitations"... looks like they are mostly hot air spewn by marketroids that need to justify people spending money on their new filesystem.

Have you ever seen VxFS or WAFL in action?

Quoted 2 lines

But even then, I'd be interested to know if that's a real problem for systems such as ZFS.

ZFS uses 'dnodes'. The dnodes are allocated on demand from your available space so running out of [di]nodes is impossible.

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

Regards,
Adrian

#209 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover

Dnia Mon, 31 Jul 2006 17:57:35 +0200, David Masover <[email protected]> napisa³:

Quoted 9 lines
Quoted 4 lines · Wil Reichert

=)
That was sorta the plan.
Any idea how the fragmentation resulting from re-syncing the tree
affects performance over time?

Try to post replies at the bottom, or below the context.

Yes, it does affect it a lot. I have no idea how much, and I've never benchmarked it, but purely subjectively, my portage has gotten slower over time.

I gues that extens are much harder to reuse then normal inodes so when You have something as big as portage tree filled with nano files wich are being modified all the time then You just can't keep performance all the time. You can always tar, rm -fr /usr/portage, untar and You will probably speed things up a lot.

#38 lore raw
In reply to Adrian Ulrich
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
Signature of:            Ich hatte in letzter Zeit ein bisschen viel Realitycheck.
the second  :                 Langsam möchte ich mal wieder weiterträumen können.
#339 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Bill Davidsen

Hi!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Suspend2 patch is open source. You can always take a look.

swsusp is open source. You can always take a look. And you can always submit a patch.

But you can't get the patch accepted, that's issue causing all this discussion.

If you send me a reasonably-sized patch, that actually fixes something, I'm likely to apply it. If you send me 14K line monster...

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

I'm not exactly an expert, but I don't think suspend-to-ram is more difficult than suspend-to-disk (probably quite the contrary), and there are a lot in common.

As you said, you do not know what you are talking about.

That's why people are frustrated. You blow off anyone who tells you the code doesn't work. Do you really think Linus doesn't know what he's talking about when he reported that it didn't work for him? Hua Zhong being "Not an expert" is not the same as not knowing what he's talking about.

He claims s-t-ram is easier than s-t-disk. That means that he did not do his homework, and did not check the archives on the subject.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#39 lore raw
In reply to Jan-Benedict Glaw
Quoted 6 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

So what?

Yes, you need to do preparation. But it is really nice if the filesystem can do that work for you.

Let me put it this way -- You're back in college, and it's time to write a thesis. You have a choice of software packages:

Package A: You have to specify how many pages, and how many words, you're likely to use before you start typing. Guess too high, and you'll print out a bunch of blank pages at the end. Guess too low, and you'll run out of space and have to start over, copy and paste your document back in, and hope it gets all the formatting right, which it probably won't.

Package B: Your document grows as you type. When it's time to print, only the pages you've actually written something on -- but all of the pages you've actually written something on -- are printed.

All other things being equal, which would you choose? Which one seems more modern?

Look, I understand the argument against ReiserFS v3 -- it has another limitation that you don't even know about. That other limitation is scary -- that's like being able to type as many words as you want, but once you type enough pages (no way of knowing how many), pages start randomly disappearing from the middle of your document.

But the argument that no one cares about inode limits? Really, stop kidding yourselves. It's 2006. The limits are starting to look ridiculous. Just because they're workable doesn't mean we should have to live with them.

#42 lore raw
In reply to Jan-Benedict Glaw
Quoted 8 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

MfG, JBG

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Cheers,

Rudy

#116 lore raw
In reply to Jan-Benedict Glaw
Quoted 9 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

MfG, JBG

I'd agree with that statement. The wrong filesystem was chosen at the beginning of the project.

As someone who is currently planning to migrate ~100GB of stored mail to the Maildirs format, it was pretty clear early on that EXT3 would not cut it (from past and current experiences), and not just for the sake of calculating inodes.

--Dan

#176 lore raw
In reply to Matthias Andree
Quoted 9 lines · Matthias Andree

Adrian Ulrich schrieb am 2006-07-31:

Quoted 3 lines

Why are a lot of Solaris-people using (buying) VxFS? Maybe because UFS also has such silly limitations? (..and performs awkward with trillions of files..?..)

Well, such "silly limitations"... looks like they are mostly hot air spewn by marketroids that need to justify people spending money on their new filesystem.

I think the limitations are silly, and I'm not paid to say this. Besides, we're talking about a filesystem that will be free (and libre), so I don't see the point of marketroids, certainly not in this context.

But let's not stoop to name-calling.

#120 lore raw
In reply to Adrian Ulrich

(resending complete message to the list).

Adrian Ulrich schrieb am 2006-07-31:

Quoted 7 lines

Hello Matthias,

Quoted 1 line

This looks rather like an education issue rather than a technical limit.

We aren't talking about the same issue: I was asking to do it on-the-fly. Umounting the filesystem, running e2fsck and resize2fs is something different ;-)

There was stuff by Andreas Dilger, to support "online" resizing of mounted ext2 file systems. I never cared to look for this (does it support ext3, does it work with current kernels, merge status) since offline resizing was always sufficient for me.

Quoted 3 lines

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes;

Well - easy to fix, newfs again with proper inode density (perhaps 1 per 2 kB) and redo the migration. Of course you're free to pay for a new file system if your fellow admin can't be bothered to remember newfs's -i option.

Quoted 5 lines
Quoted 3 lines

Well, such "silly limitations"... looks like they are mostly hot air spewn by marketroids that need to justify people spending money on their new filesystem.

Have you ever seen VxFS or WAFL in action?

No I haven't. As long as they are commercial, it's not likely that I will.

Quoted 2 lines

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

I think reallocating inodes for UFS and/or ext2/ext3 is possible, even online, but someone needs to write, debug and field-test the code to do that - possibly based on Andreas Dilger's earlier ext2 online resizing work.

Signature · 1 line
Matthias Andree
#117 lore raw
In reply to Dan Oglesby
Quoted 11 lines · Dan Oglesby
Quoted 5 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

As someone who is currently planning to migrate ~100GB of stored mail to the Maildirs format, it was pretty clear early on that EXT3 would not cut it (from past and current experiences), and not just for the sake of calculating inodes.

Uh? Where did you face a problem there?

With maildir, you shouldn't face any problems IMO. Even users with zillions of mails should work properly with the dir_index stuff:

tune2fs -O dir_index /dev/hdXX

or alternatively (to start that for already existing directories):

e2fsck -fD /dev/hdXX

Of course, you'll always face a problem with lots of files in one directory at getdents() time (eg. opendir()/readdir()/closedir()), but this is a common limit for all filesystems.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
Signature of:           Alles wird gut! ...und heute wirds schon ein bißchen besser.
the second  :
#43 lore raw
In reply to Rudy Zijlstra
Quoted 7 lines · Rudy Zijlstra
Quoted 6 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

Quoted 5 lines
Quoted 1 line

So preparation work wasn't done.

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Nope, I'm a quite happy ext3 user for quite divergent workloads. Starting with storing loads of small files in a hugh number of directories (mkfs -T news) over hugh numbers of files in few directories (mkfs -O dir_index) to only a small number of really hugh files (mkfs -T largefile4).

All that I still need to catch up with is 4TB sized files. But I guess this is really an uncommon workload, but ISTR that somebody even works on this.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                       Don't believe in miracles: Rely on them!
 the second  :
#44 lore raw
In reply to Rudy Zijlstra
Quoted 7 lines · Rudy Zijlstra
Quoted 6 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

Quoted 5 lines
Quoted 1 line

So preparation work wasn't done.

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Oh, and before people start laughing at me, here are some personal or friend's experiences with different filesystems:

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

* A friend's XFS data partition (portable USB/FireWire HDD) once crashed due to being hot-unplugged off the USB. The in-kernel XFS driver refused to mount that thing again, and the tools also refused to fix any errors. (Don't ask, no details at my hands...)

* JFS just always worked for me. Though I've never ever had a broken HDD where it (or it's tools) could have shown how well-done they were, so from a crash-recovery point of view, it's untested.

* Being a regular ext3 user, I had lots of broken HDDs containing ext3 filesystems. For every single case, it has been easy fixing the filesystem after cloning. Just _once_, fsck wasn't able to fix something, so I did it manually with some disk editor. This worked well because the on-disk data structures are actually as simple as they are.

ext3 always worked well for me, so why should I abandon it?

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                               If it doesn't work, force it.
 the second  :                      If it breaks, it needed replacing anyway.
#118 lore raw
In reply to Jan-Benedict Glaw

Jan-Benedict Glaw schrieb am 2006-07-31:

Quoted 10 lines

Uh? Where did you face a problem there?

With maildir, you shouldn't face any problems IMO. Even users with zillions of mails should work properly with the dir_index stuff:

tune2fs -O dir_index /dev/hdXX

or alternatively (to start that for already existing directories):

e2fsck -fD /dev/hdXX

hat is not "alternatively", but "tune2fs first", then "e2fsck -fD" (which can't happen on a RW-mounted FS and you should only try this on your rootfs if you can reboot with magic sysrq or from a rescue system).

Signature · 1 line
Matthias Andree
#40 lore raw
In reply to David Masover
Quoted 2 lines · David Masover

Yes, you need to do preparation. But it is really nice if the filesystem can do that work for you.

The filesystem defaults are more than useable for most situations. You simply dont migrate a filesystem of that size without doing some calculation.

Gruss
Bernd

#119 lore raw
In reply to Jan-Benedict Glaw
Quoted 6 lines · Jan-Benedict Glaw
Quoted 5 lines · Dan Oglesby
Quoted 4 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

So preparation work wasn't done.

As someone who is currently planning to migrate ~100GB of stored mail to the Maildirs format, it was pretty clear early on that EXT3 would not cut it (from past and current experiences), and not just for the sake of calculating inodes.

Uh? Where did you face a problem there?

Past experiences dealing with systems that generate several thousand to tens of thousands of files a day, adding up to well in the millions over the course of normal production (this is not for a mail server, BTW). Once we got close to a million files, filesystem transactions started bogging the system, driving the load over 50. Simply switching to ReiserFS v3 allowed us to go well past the number of files EXT3 could handle reasonably and the max load stayed right around the number of CPUs the system contained (typically 8).

There is a LOT going on with these systems, not just filesystem transactions. EXT3 does not do well in our environment at all.

Quoted 10 lines

With maildir, you shouldn't face any problems IMO. Even users with zillions of mails should work properly with the dir_index stuff:

tune2fs -O dir_index /dev/hdXX

or alternatively (to start that for already existing directories):

e2fsck -fD /dev/hdXX

I've been tuning EXT3 to see what performance I can get for the mail server, and it's just not there compared to ReiserFS with minimal tuning.

Quoted 6 lines

Of course, you'll always face a problem with lots of files in one directory at getdents() time (eg. opendir()/readdir()/closedir()), but this is a common limit for all filesystems.

MfG, JBG

Of course, but the issue is EXT3 does this a whole lot worse than ReiserFS v3 from my experiences.

At any rate, that's about all I have to say about this issue. I'll be patiently waiting to see ReiserFS v4 included in the main kernel, so I have less hoops to jump through to implement the latest and greatest from Namesys.

--Dan

#45 lore raw
In reply to Jan-Benedict Glaw
Quoted 39 lines · Jan-Benedict Glaw
Quoted 7 lines · Rudy Zijlstra
Quoted 6 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

Quoted 5 lines
Quoted 1 line

So preparation work wasn't done.

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Oh, and before people start laughing at me, here are some personal or friend's experiences with different filesystems:

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

* A friend's XFS data partition (portable USB/FireWire HDD) once crashed due to being hot-unplugged off the USB. The in-kernel XFS driver refused to mount that thing again, and the tools also refused to fix any errors. (Don't ask, no details at my hands...)

* JFS just always worked for me. Though I've never ever had a broken HDD where it (or it's tools) could have shown how well-done they were, so from a crash-recovery point of view, it's untested.

* Being a regular ext3 user, I had lots of broken HDDs containing ext3 filesystems. For every single case, it has been easy fixing the filesystem after cloning. Just _once_, fsck wasn't able to fix something, so I did it manually with some disk editor. This worked well because the on-disk data structures are actually as simple as they are.

Quoted 3 lines

ext3 always worked well for me, so why should I abandon it?

MfG, JBG

I've lost EXT2 and EXT3 filesystems from machines with no bad hardware (power loss during writes).

I've recovered all but a handful of files from a RAID-5 array running ReiserFS v3 that had two drives fail in rapid succession with bad sectors.

Sometimes you're lucky, sometimes you're not.

--Dan

#121 lore raw
In reply to Matthias Andree
Quoted 3 lines

Well - easy to fix, newfs again with proper inode density (perhaps 1 per 2 kB) and redo the migration.

Ehr: Such a migration (on a very busy system) takes *some* time (weeks). Re-Doing (migrate users back / recreate the FS / start again) the whole thing isn't really an option..

Quoted 3 lines

Of course you're free to pay for a new
file system if your fellow admin can't be bothered to remember newfs's -i option.

Let's face it: Shit happens and nobody is perfect. A filesystem should be flexible (modern..) and support Admin/User-needs.

We wouldn't need ECC / Raid / UPS's in a perfect world.

Quoted 4 lines
Quoted 1 line

Have you ever seen VxFS or WAFL in action?

No I haven't. As long as they are commercial, it's not likely that I will.

Why?

NetApp WAFL-Blurb:
http://www.netapp.com/library/tr/3002.pdf

Maybe we should crop the Cc: list .. this is getting OT.

-- Adrian

#46 lore raw
In reply to Jan-Benedict Glaw

Jan-Benedict Glaw schrieb am 2006-07-31:

Quoted 14 lines · Rudy Zijlstra
Quoted 7 lines · Jan-Benedict Glaw
Quoted 6 lines
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

Quoted 5 lines
Quoted 1 line

So preparation work wasn't done.

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Quoted 8 lines

Oh, and before people start laughing at me, here are some personal or friend's experiences with different filesystems:

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what idiot ordered a machine without ECC for a server, but anyways) and it fucked up every 64th bit - only in a certain region. Guess what happened to the fs when it went into e2fsck after a reboot. Boom. Same with a dead DPTA that lost every 16th block or so, the rescue in the first case was swapping the RAM and "amrecover" and in the second swapping the drive and "dsmc restore". OTOH, kernel panics on bad blocks are a no-no of course.

Quoted 4 lines

* A friend's XFS data partition (portable USB/FireWire HDD) once crashed due to being hot-unplugged off the USB. The in-kernel XFS driver refused to mount that thing again, and the tools also refused to fix any errors. (Don't ask, no details at my hands...)

Don't use write caches then. (Though I've seen NUL-filled blocks in new files or appended to files after in 2001 or 2002.)

Quoted 3 lines

* JFS just always worked for me. Though I've never ever had a broken HDD where it (or it's tools) could have shown how well-done they were, so from a crash-recovery point of view, it's untested.

SUSE removed JFS support from their installation tool for "technical reasons" they didn't specify in the release notes. Whatever.

Quoted 1 line

ext3 always worked well for me, so why should I abandon it?

Plus, it and its tools are maintained.

Signature · 1 line
Matthias Andree
#41 lore raw
In reply to David Masover
Quoted 19 lines · David Masover
Quoted 7 lines · Jan-Benedict Glaw
Quoted 3 lines · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and started to migrate Mailboxes (Maildir-style format = many small files (1-3kb)) to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

Let me put it this way -- You're back in college, and it's time to write a thesis. You have a choice of software packages:

Package A: You have to specify how many pages, and how many words, you're likely to use before you start typing. Guess too high, and you'll print out a bunch of blank pages at the end. Guess too low, and you'll run out of space and have to start over, copy and paste your document back in, and hope it gets all the formatting right, which it probably won't.

Quoted 6 lines

Package B: Your document grows as you type. When it's time to print, only the pages you've actually written something on -- but all of the pages you've actually written something on -- are printed.

All other things being equal, which would you choose? Which one seems more modern?

:) Well, given that TeX needs two (or even three!) runs to get all page references right, why should I choose MS Word, where you won't see that problem at all?

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                               If it doesn't work, force it.
 the second  :                      If it breaks, it needed replacing anyway.
#47 lore raw
In reply to Matthias Andree
Quoted 9 lines · Matthias Andree

Jan-Benedict Glaw schrieb am 2006-07-31:

Quoted 5 lines

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what

They do! Very much, actually. These happen In Real Life, so I have to pay attention to them. Once you're in setups with > 10000 machines, everything counts. At some certain point, you can even use HDD's temperature sensors in old machines to diagnose dead fans.

Everything that eases recovery for whatever reason is something you have to pay attention to. The simplicity of ext{2,3} is something I really fail to find proper words for. As well as the really good fsck. Once seen a SIGSEGV'ing fsck, you really don't want to go there.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                       Eine Freie Meinung in einem Freien Kopf
 the second  :                     für einen Freien Staat voll Freier Bürger.
#340 lore raw

RE: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Pavel Machek
Quoted 9 lines
Quoted 1 line

Suspend2 patch is open source. You can always take a look.

swsusp is open source. You can always take a look. And you can always submit a patch.

Quoted 1 line

Moreover, if someone claims suspend2 isn't ready for merge, or the

Moreover, if someone claims swsusp is broken, they should attach bugzilla id.

Pavel,

You can't blame me for not doing these things, because I am not a maintainer. However, you are, and you defend yourself so hard for that position, so if _you_ don't do these things, people complain.

Quoted 4 lines

As you said, you do not know what you are talking about.

He claims s-t-ram is easier than s-t-disk. That means that he did not do his homework, and did not check the archives on the subject.

Oh yeah? Let's check the archives:

"I seriously claim that STR _should_ be a lot simpler than suspend-to-disk, because it avoids all the memory management problems. The reason that we support suspend-to-disk but not STR is totally perverse - it's simply that it has been easier to debug, because unlike STR, we can do a "real boot" into a working system, and thus we don't have the debugging problems that the "easy" suspend/resume case has."

http://thread.gmane.org/gmane.linux.power-management.general/1884/focus=2105

Maybe it's why he didn't like the STR design you had?

Maybe I am still wrong, maybe Linus is wrong too, but you can't attack me not doing my homework.

Hua

#48 lore raw
In reply to Jan-Benedict Glaw
Quoted 8 lines · Jan-Benedict Glaw
Quoted 7 lines · Matthias Andree

Jan-Benedict Glaw schrieb am 2006-07-31:

Quoted 5 lines

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

Quoted 7 lines
Quoted 2 lines

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what

They do! Very much, actually. These happen In Real Life, so I have to pay attention to them. Once you're in setups with > 10000 machines, everything counts. At some certain point, you can even use HDD's temperature sensors in old machines to diagnose dead fans.

I think what he meant was that it is unfair to blame reiser3 for data loss in a massive failure situation as a case example by itself. What would be more appropriate (and fair) is saying something along the lines of "in a machine with DMA failure, reiser3 hosed a drive, while ext3 only lost x data (or nothing). In my situation, every bit of massive failure robustness counts... " This of course assumes you actually had the *exact* same problem with hardware under ext3, pretty much in every detail. Of course, so many subtleties interact in massive ways with filesystems and hardware problems, so we have to keep in mind how much difference that can make (all the difference in the world), and that, really, without a statistically significant sample size and some real analysis, hardware failure comparisons are impossible to do fairly.

Of course, if ext3 were proven to be more robust against failures, I bet the reiser team would be very interested in all the forensic data you can offer, since, from what I've seen, they are always trying to make reiser as good as possible---in speed, flexability, *and* robustness. If they didn't care about the last, they'd benifit greatly from not journaling! :-P

--Clay

Quoted 12 lines

Everything that eases recovery for whatever reason is something you have to pay attention to. The simplicity of ext{2,3} is something I really fail to find proper words for. As well as the really good fsck. Once seen a SIGSEGV'ing fsck, you really don't want to go there.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
 Signature of:                       Eine Freie Meinung in einem Freien Kopf
 the second  :                     für einen Freien Staat voll Freier Bürger.
#127 lore raw

Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Adrian Ulrich

Adrian Ulrich <[email protected]> wrote:

[...]

Quoted 5 lines

ZFS uses 'dnodes'. The dnodes are allocated on demand from your available space so running out of [di]nodes is impossible.

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

This would be worthwhile, if only to be able to futz around in Solaris-made filesystems.

Are you volunteering? You'd probably need a friend in Solaris-land who passes you information on how things are done, and copies of filessytems to take apart, and so on.

First question is if there are any restrictions (patent or otherwise) on doing this, just copying is out of the question due to (unfortunate) licence on Sun's part.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#49 lore raw
In reply to Clay Barnes

Ar Llu, 2006-07-31 am 12:17 -0700, ysgrifennodd Clay Barnes:

Quoted 4 lines

Of course, if ext3 were proven to be more robust against failures, I bet the reiser team would be very interested in all the forensic data you can offer, since, from what I've seen, they are always trying to make reiser as good as possible---in speed, flexability, *and* robustness.

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Alan

#62 lore raw
In reply to Clay Barnes
Quoted 3 lines · Clay Barnes
Quoted 2 lines · Jan-Benedict Glaw
Quoted 1 line · Matthias Andree

Jan-Benedict Glaw schrieb am 2006-07-31:

[Crippled DMA writes]

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what

They do! Very much, actually. These happen In Real Life, so I have to

I think what he meant was that it is unfair to blame reiser3 for data loss in a massive failure situation as a case example by itself. What

Crippling a few KB of metadata in the ext{2,3} case probably wouldn't fobar the filesystem...

Quoted 3 lines

failure robustness counts... " This of course assumes you actually had the *exact* same problem with hardware under ext3, pretty much in every detail. Of course, so many subtleties interact in massive ways with

The point is that it's quite hard to really fuck up ext{2,3} with only some KB being written while it seems (due to the fragile^Wsophisticated on-disk data structures) that it's just easy to kill a reiser3 filesystem.

MfG, JBG

Signature · 3 lines
       Jan-Benedict Glaw       [email protected]                +49-172-7608481
   Signature of:                            Zensur im Internet? Nein danke!
   the second  :
#50 lore raw
In reply to Alan Cox
Quoted 10 lines · Alan Cox

Ar Llu, 2006-07-31 am 12:17 -0700, ysgrifennodd Clay Barnes:

Quoted 4 lines

Of course, if ext3 were proven to be more robust against failures, I bet the reiser team would be very interested in all the forensic data you can offer, since, from what I've seen, they are always trying to make reiser as good as possible---in speed, flexability, *and* robustness.

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

Yes, that is true, and I think that's a big motivator for the reiser team to get reiser4 in a place where people can't say that. I suspect that they know that reiserfs's shortcomings in that respect are probably the biggest deterrent to using that fs, and they'll do everything they can to prevent such a problem in reiser4. That's pure conjecture based on the stuff I see on the list, so if I'm wrong, reiser people, please correct me.

--Clay

Quoted 4 lines

How reiser4 compares I've no idea.

Alan

#124 lore raw
In reply to Matthias Andree
Quoted 10 lines · Matthias Andree
Quoted 5 lines
Quoted 1 line

This looks rather like an education issue rather than a technical limit.

We aren't talking about the same issue: I was asking to do it on-the-fly. Umounting the filesystem, running e2fsck and resize2fs is something different ;-)

There was stuff by Andreas Dilger, to support "online" resizing of mounted ext2 file systems. I never cared to look for this (does it support ext3, does it work with current kernels, merge status) since offline resizing was always sufficient for me.

With the latest e2fsprogs and 2.6 kernels, the online resizing support has been merged in, and as long as the filesystem was created with space reserved for growing the filesystem (which is now the default, or if the filesystem has the off-line prepration step ext2prepare run on it), you can run resize2fs on a mounted filesystem and grow an ext2/3 filesystem on-line. And yes, you get more inodes as you add more disk blocks, using the original inode ratio that was established when the filesystem was created.

- Ted

#63 lore raw
In reply to Jan-Benedict Glaw
Quoted 5 lines · Jan-Benedict Glaw
Quoted 3 lines · Clay Barnes
Quoted 2 lines · Jan-Benedict Glaw
Quoted 1 line · Matthias Andree

Jan-Benedict Glaw schrieb am 2006-07-31:

[Crippled DMA writes]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what

They do! Very much, actually. These happen In Real Life, so I have to

I think what he meant was that it is unfair to blame reiser3 for data loss in a massive failure situation as a case example by itself. What

Crippling a few KB of metadata in the ext{2,3} case probably wouldn't fobar the filesystem...

Probably. By the time a few KB of metadata are corrupted, I'm reaching for my backup. I don't care what filesystem it is or how easy it is to edit the on-disk structures.

This isn't to say that having robust on-disk structures isn't a good thing. I have no idea how Reiser4 will hold up either way. But ultimately, what you want is the journaling (so power failure / crashes still leave you in an OK state), backups (so when blocks go bad, you don't care), and performance (so you can spend less money on hardware and more money on backup hardware).

#122 lore raw
In reply to Adrian Ulrich

Adrian Ulrich schrieb am 2006-07-31:

Quoted 3 lines

Ehr: Such a migration (on a very busy system) takes *some* time (weeks). Re-Doing (migrate users back / recreate the FS / start again) the whole thing isn't really an option..

All the more important to think about FS requirements *before* newfs-ing if a quick "one day for rsync/star/dump+restore" isn't available. If you're hitting, for instance, the hash collision problem in reiser3, you're as dead as with a FS without inodes.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Have you ever seen VxFS or WAFL in action?

No I haven't. As long as they are commercial, it's not likely that I will.

Why?

I'm trying to shift my focus away from computer administration and better file systems than old-style non-journalling, non-softupdates UFS are available today and more will follow.

Cc list weeded out.
Signature · 1 line
Matthias Andree
#128 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Horst H. von Brand
Quoted 13 lines · Horst H. von Brand

Adrian Ulrich <[email protected]> wrote:

[...]

Quoted 5 lines

ZFS uses 'dnodes'. The dnodes are allocated on demand from your available space so running out of [di]nodes is impossible.

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

This would be worthwhile, if only to be able to futz around in Solaris-made filesystems.

Quoted 7 lines

Are you volunteering? You'd probably need a friend in Solaris-land who passes you information on how things are done, and copies of filessytems to take apart, and so on.

First question is if there are any restrictions (patent or otherwise) on doing this, just copying is out of the question due to (unfortunate) licence on Sun's part.

This may eventually be good enough for "futzing around":
http://zfs-on-fuse.blogspot.com/

Quoted 5 lines
Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#123 lore raw
In reply to Matthias Andree
Quoted 5 lines

All the more important to think about FS requirements *before* newfs-ing if a quick "one day for rsync/star/dump+restore" isn't available. If you're hitting, for instance, the hash collision problem in reiser3, you're as dead as with a FS without inodes.

Quoting myself:

Quoted 2 lines
Quoted 2 lines

Let's face it: Shit happens and nobody is perfect. A filesystem should be flexible (modern..) and support Admin/User-needs.

Of COURSE you should think BEFORE creating the filesystem. But if you somehow failed to do it 'the right thing' your filesystem shouldn't let you down.

#64 lore raw

Re: the " 'official' point of view" expressed by kernelnewbies.orgregarding reiser4 inclusion

In reply to David Masover
Quoted 3 lines · David Masover

Probably. By the time a few KB of metadata are corrupted, I'm reaching for my backup. I don't care what filesystem it is or how easy it is to edit the on-disk structures.

This isn't to say that having robust on-disk structures isn't a good thing. I have no idea how Reiser4 will hold up either way. But ultimately, what you want is the journaling (so power failure / crashes still leave you in an OK state), backups (so when blocks go bad, you don't care), and performance (so you can spend less money on hardware and more money on backup hardware).

please read the discussion that took place at the filesystem summit a couple weeks ago (available on lwn.net)

one of the things that they pointed out there is that as disks get larger the ratio of bad spots per Gig of storage is remaining about the same. As is the rate of failures per Gig of storage.

As a result of this the idea of only running on perfect disks that never have any failures is becomeing significantly less realistic, instead you need to take measures to survive in the face of minor corruption (including robust filesystems, raid, etc)

David Lang

#129 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Horst H. von Brand
Quoted 6 lines
Quoted 3 lines

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

This would be worthwhile, if only to be able to futz around in Solaris-made filesystems.

s/I think linux should do the same/I think linux should include Reiser4/
;-)

Quoted 2 lines

First question is if there are any restrictions (patent or otherwise) on doing this,

Quoting from
http://it.sun.com/eventi/jc06/pdf/mi27_p5_poccia_virtualization.pdf

­> 47 ZFS patents added to CDDL patent commons

But i'd rather like to see a Linux version of WAFL :-)

ZFS didn't really impress me:
The Volume-Manager is nice but the Filesystem.. well: It beats UFS .. sometimes ;-)

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Regards,
Adrian

#51 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

#66 lore raw
In reply to Jan-Benedict Glaw
Quoted 6 lines · Jan-Benedict Glaw

The point is that it's quite hard to really fuck up ext{2,3} with only some KB being written while it seems (due to the fragile^Wsophisticated on-disk data structures) that it's just easy to kill a reiser3 filesystem.

Well, I was once very 'luckily' and after a system crash (*) e2fsck put all files into lost+found. Sure, I never experienced this again, but I also never experienced something like this with reiserfs. So please, stop this kind of FUD against reiser3.6.
While filesystem speed is nice, it also would be great if reiser4.x would be very robust against any kind of hardware failures.

(*) The problem was a specific mainboard + video-card + driver combination. As soon as X started up, the system entirely crashed. I don't believe many bytes were written, but I also can't prove it.

Signature · 5 lines
Bernd Schubert
PCI / Theoretische Chemie
Universität Heidelberg
INF 229
69120 Heidelberg
#65 lore raw

Re: the " 'official' point of view" expressed by kernelnewbies.orgregarding reiser4 inclusion

In reply to David Lang
Quoted 7 lines · David Lang
Quoted 3 lines · David Masover

Probably. By the time a few KB of metadata are corrupted, I'm reaching for my backup. I don't care what filesystem it is or how easy it is to edit the on-disk structures.

This isn't to say that having robust on-disk structures isn't a good thing. I have no idea how Reiser4 will hold up either way. But ultimately, what you want is the journaling (so power failure / crashes still leave you in an OK state), backups (so when blocks go bad, you don't care), and performance (so you can spend less money on hardware and more money on backup hardware).

please read the discussion that took place at the filesystem summit a couple weeks ago (available on lwn.net)

I think I will, but I don't have the time today, so...

Quoted 3 lines

one of the things that they pointed out there is that as disks get larger the ratio of bad spots per Gig of storage is remaining about the same. As is the rate of failures per Gig of storage.

As a result of this the idea of only running on perfect disks that never have any failures is becomeing significantly less realistic, instead you need to take measures to survive in the face of minor corruption (including robust filesystems, raid, etc)

RAID seems a much more viable solution to me. That and cheaper storage, so that you can actually afford to replace the disk when you find corruption, or have more redundancy so you don't have to.

Because "robust filesystems" is nice in theory, but in practice, you really never know what will get hit. RAID, at least, is predictable.

When it's not: Backups.

#52 lore raw
In reply to Gregory Maxwell
Quoted 17 lines · Gregory Maxwell
Quoted 6 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Quoted 6 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?

How's Wikimania going. :-)

What he says is correct. I have seen some serious issues with reiserfs in terms of stability and data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate. I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10 due to issues I have seen with data corruption and hard hangs on RAID 0 read/write sector errors. I have stopped using it for local drives and based everything on EXT3. Not to say it won't get there eventually, but file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

The Wikimedia appliances use Wolf Mountain, and I've tested it for about 4 months with few problems, but I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better than either EXT3 or ReiserFS. Despite this, for vertical wiki servers, its ok to go out with, folks can specifiy whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked" completely, though it does scale to 1 exabyte FS images.

Reiser does have issues still, and I hestitate to standardize on it until I stop seeing reports from the field about corruption and failover issues.

Jeff

#115 lore raw
In reply to Jan-Benedict Glaw

Dnia Mon, 31 Jul 2006 19:32:39 +0200, Jan-Benedict Glaw <[email protected]> napisa³:

Quoted 34 lines · Rudy Zijlstra
Quoted 6 lines · Jan-Benedict Glaw
Quoted 3 lines
Quoted 1 line · Adrian Ulrich

A colleague of mine happened to create a ~300gb filesystem and

started

Quoted 1 line
Quoted 1 line

to migrate Mailboxes (Maildir-style format = many small files

Quoted 8 lines

(1-3kb))

Quoted 3 lines
Quoted 1 line

to the new LUN. At about 70% the filesystem ran out of inodes; Not a

So preparation work wasn't done.

Of course you are right. Preparation work was not fully done. And using ext1 would also have been possible. I suspect you are still using ext1, cause with proper preparation it is perfectly usable.

Oh, and before people start laughing at me, here are some personal or friend's experiences with different filesystems:

* reiser3: A HDD containing a reiser3 filesystem was tried to be booted on a machine that fucked up DMA writes. Fortunately, it crashed really soon (right after going for read-write.) After rebooting the HDD on a sane PeeCee, it refused to boot. Starting off some rescue system showed an _empty_ root filesystem.

* A friend's XFS data partition (portable USB/FireWire HDD) once crashed due to being hot-unplugged off the USB. The in-kernel XFS driver refused to mount that thing again, and the tools also refused to fix any errors. (Don't ask, no details at my hands...)

* JFS just always worked for me. Though I've never ever had a broken HDD where it (or it's tools) could have shown how well-done they were, so from a crash-recovery point of view, it's untested.

Quoted 6 lines

* Being a regular ext3 user, I had lots of broken HDDs containing ext3 filesystems. For every single case, it has been easy fixing the filesystem after cloning. Just _once_, fsck wasn't able to fix something, so I did it manually with some disk editor. This worked well because the on-disk data structures are actually as simple as they are.

Is this some kind of "who lost more files on what fs" competition? What's the prize?
Topic subject sugests that it should cover something else. Please, let's get back on track.
First of all, it's about reiser4 so can't we forget about other filesystem's unless it's got something to do with reiser4 merge?

#60 lore raw
In reply to Gregory Maxwell

Ar Llu, 2006-07-31 am 17:00 -0400, ysgrifennodd Gregory Maxwell:

Quoted 9 lines · Alan Cox
Quoted 6 lines

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

Two sources, the cases I've looked at myself when IDE maintainer and also comments Hans made when we met at UKUUG a few years ago. Generally speaking on an IDE failure that lost chunks of disk the ext2/ext3 users got most of their data back. The reiserfs ones sometimes got it back and sometimes got catastrophic failure.

The ext3 fsck is extremely effective in the face of serious errors. Some of that is clever code that knows about things like rewriting inode blocks to force reallocation of failed metadata by the drive but the majority of it is simply because you *know* where inode X is on the disk rather than having to deal with data structures and walk them.

That's a tradeoff with the reiser performance with small files and until recently the reiser large directory performance. Which is right is another question altogether. Its also something I think XFS demonstrates can be done better so doesn't invalidate the theory behind reiserfs just the rev 3 implementation.

Quoted 4 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

man mount

Ext3 is configurable, and has been for years via the errors= option.

Alan

#53 lore raw
In reply to Jeff V. Merkey
Quoted 31 lines · Jeff V. Merkey
Quoted 18 lines · Gregory Maxwell
Quoted 6 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Quoted 6 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?
How's Wikimania going. :-)

Quoted 7 lines

What he says is correct. I have seen some serious issues with reiserfs in terms of stability and data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate. I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10 due to issues I have seen with data corruption and hard hangs on RAID 0 read/write sector errors. I have stopped using it for local drives and based everything on EXT3. Not to say it won't get there eventually, but file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

Correction,

That's "MediWiki" appliances. Two many transposed acronyms...

www.wolfmountaingroup.com

:-)

Jeff

Quoted 16 lines

The Wikimedia appliances use Wolf Mountain, and I've tested it for about 4 months with few problems, but I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better than either EXT3 or ReiserFS. Despite this, for vertical wiki servers, its ok to go out with, folks can specifiy whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked" completely, though it does scale to 1 exabyte FS images.
Reiser does have issues still, and I hestitate to standardize on it until I stop seeing reports from the field about corruption and failover issues.

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#61 lore raw
In reply to Alan Cox
Quoted 10 lines · Alan Cox

Ar Llu, 2006-07-31 am 17:00 -0400, ysgrifennodd Gregory Maxwell:

Quoted 4 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

man mount

Ext3 is configurable, and has been for years via the errors= option.

Sure, but I think the suggestion is that the reason we generally see more ReiserFS complaints than ext3 complaints might be because of the default level of errors logged.

#54 lore raw
In reply to Jeff V. Merkey
Quoted 4 lines

Correction,

Quoted 8 lines

That's "MediWiki" appliances. Two many transposed acronyms...

www.wolfmountaingroup.com

:-)

Jeff

And WMFS does not belong to the WolfMountainGroup any longer. It has been acquired by another company, so you won't see any info about it on the website. It's has been rolled into another company. I can bundle it with appliances but it is no longer the property of Wolf Mountain Group. WMG is a MediaWiki/Wikipedia Appliance company that does Machine translations of Wikipedia into several Native American Languages. The Translator is Linux and Windows based, but WMFS has a new home now.

Just to clarify.

Jeff

#130 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Adrian Ulrich
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Signature · 1 line
Matthias Andree
#210 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to Łukasz Mierzwa
Quoted 5 lines · £ukasz Mierzwa

I gues that extens are much harder to reuse then normal inodes so when You have something as big as portage tree filled with nano files wich are being modified all the time then You just can't keep performance all the time. You can always tar, rm -fr /usr/portage, untar and You will probably speed things up a lot.

I agree here. On my home system this gave me a visible improvement in the sync process. I blew up my portage tree and I rebuilt it, and it gave me around ~10% improvement - this was around 6 months ago. I am not all that greedy, so i didn't do this do this than once :-)

Sarath

#171 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Adrian Ulrich
Quoted 8 lines · Adrian Ulrich
Quoted 5 lines
Quoted 2 lines

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

This would be worthwhile, if only to be able to futz around in Solaris-made filesystems.

s/I think linux should do the same/I think linux should include Reiser4/
;-)

So ZFS isn't "state-of-the-art"?

[...]

Quoted 1 line

But i'd rather like to see a Linux version of WAFL :-)

WAFL is for high-turnover filesystems on RAID-5 (and assumes flash memory staging areas). Not your run-of-the-mill desktop...

Quoted 3 lines

ZFS didn't really impress me:
The Volume-Manager is nice but the Filesystem.. well: It beats UFS .. sometimes ;-)

OK, ext3 + LVM it is then.

Quoted 1 line

See also: http://spam.workaround.ch/dull/postmark.txt

Interesting.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#114 lore raw
In reply to Jan-Benedict Glaw

Jan-Benedict Glaw schrieb am 2006-07-31:

Quoted 8 lines
Quoted 2 lines

Massive hardware problems don't count. ext2/ext3 doesn't look much better in such cases. I had a machine with RAM gone bad (no ECC - I wonder what

They do! Very much, actually. These happen In Real Life, so I have to pay attention to them. Once you're in setups with > 10000 machines, everything counts. At some certain point, you can even use HDD's temperature sensors in old machines to diagnose dead fans.

Quoted 4 lines

Everything that eases recovery for whatever reason is something you have to pay attention to. The simplicity of ext{2,3} is something I really fail to find proper words for. As well as the really good fsck. Once seen a SIGSEGV'ing fsck, you really don't want to go there.

The point is: If you've written data with broken hardware (RAM, bus, controllers - loads of them, CPU), what is on your disks is untrustworthy anyways, and fsck isn't going to repair your gzip file where every 64th bit has become a 1 or when the battery-backed write cache threw 60 MB down the drain...

Of course, an fsck that crashes is unbearable, but that doesn't apply to "broken hardware" failures. You need backups with a few generations to avoid massively losing data.

Signature · 1 line
Matthias Andree
#125 lore raw
In reply to Theodore Tso

Theodore Tso schrieb am 2006-07-31:

Quoted 8 lines

With the latest e2fsprogs and 2.6 kernels, the online resizing support has been merged in, and as long as the filesystem was created with space reserved for growing the filesystem (which is now the default, or if the filesystem has the off-line prepration step ext2prepare run on it), you can run resize2fs on a mounted filesystem and grow an ext2/3 filesystem on-line. And yes, you get more inodes as you add more disk blocks, using the original inode ratio that was established when the filesystem was created.

That's cool.

The interesting part for some people would be, if I read past postings correctly, to change the inode ratio in an existing (perhaps even mounted) file system without losing data.

(I'm not sure how many blocks have to be moved and/or changed for that purpose, because I know too little about the on-disk ext2 layout, but since block relocating is already in place for shrink support in the offline resizer, some of the work appears to be done already.)

Signature · 1 line
Matthias Andree
#55 lore raw
In reply to Jeff V. Merkey
Quoted 43 lines · Jeff V. Merkey
Quoted 17 lines · Gregory Maxwell
Quoted 6 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system and the tree basis make it very hard to avoid such problems. XFS appears to have managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Quoted 6 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?

How's Wikimania going. :-)

What he says is correct. I have seen some serious issues with reiserfs in terms of stability and
data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate.
I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10
due to issues I have seen with data corruption and hard hangs on RAID 0 read/write sector errors. I have
stopped using it for local drives and based everything on EXT3. Not to say it won't get there eventually, but
file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

Quoted 9 lines

The Wikimedia appliances use Wolf Mountain, and I've tested it for about 4 months with few problems, but
I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better
than either EXT3 or ReiserFS. Despite this, for vertical wiki servers, its ok to go out with, folks can specifiy
whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked"
completely, though it does scale to 1 exabyte FS images.

i've seen you mention the Wolf Mountain FS in other emails, but google isn't telling me a lot about it. Do you have a whitepaper? are there any published benchmark results? what sort of workloads do you benchmark?

NATE

#341 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Hua Zhong

On Monday 31 July 2006 20:55, Hua Zhong wrote:
]--snip--[

Quoted 5 lines
Quoted 2 lines

He claims s-t-ram is easier than s-t-disk. That means that he did not do his homework, and did not check the archives on the subject.

Oh yeah? Let's check the archives:

Quoted 8 lines

"I seriously claim that STR _should_ be a lot simpler than suspend-to-disk, because it avoids all the memory management problems. The reason that we support suspend-to-disk but not STR is totally perverse - it's simply that it has been easier to debug, because unlike STR, we can do a "real boot" into a working system, and thus we don't have the debugging problems that the "easy" suspend/resume case has."

http://thread.gmane.org/gmane.linux.power-management.general/1884/focus=2105

The "_should_" is important here, IMHO.

I think the problems that we have with getting STR to work on many machines reflect the situation in which we are with respect to the hardware. From the programming point of view it's easy, because if you know how to handle the hardware, it doesn't take a lot to get it right. Still, you need to _know_, and we're talking of things that are hardly documented and require hours of trial-and-error to figure out. Usually, it can only be done by someone who
(a) has access to the hardware in question, (b) has a lot of time, (c) is _very_ determined to make the suspend and resume work on this particular device, because these things are notoriously difficult to debug, and (d),
ideally, knows how to write Linux device drivers. If you own an "exotic" notebook, there's practically no chance to find someone like that who owns one too.

Moreover, even if there are some hardware-related fixes in the wild, we can't just throw them at Andrew to merge, because some kernel developers may think they are not the right fixes. Each fix, before it gets merged, if ever, has to be reviewed by the appropriate driver maintainers and people who know how the related subsystems work. If the fix gets rejected, we can't help it.

For example, we've recently received a fix for a resume-related problem on some IDE chipsets (AMD and NVidia ones), but it has been vetoed by Alan Cox. The Alan's arguments are reasonable and everyone seems to agree that it should be done differently, but the net result, for now, is the problem remains - officially - unfixed (see
http://www.ussg.iu.edu/hypermail/linux/kernel/0607.3/1607.html).

However, Nigel maintains his patch in separation with the mainline kernel and he can include fixes like that into it just fine. He can include whatever he likes, as long as it works, but we just can't do that. You can say he has more freedom, because he doesn't have to take the other people's opinions into consideration, so he can make suspend2 work on a greater number of systems more easily. Yet, he faces the other people's opinions about the things that are in suspend2 whenever he submits it for merging. In other words, the same things that make suspend2 work on machines on which swsusp doesn't may also render it _very_ difficult to merge (eg. if Nigel had decided to include the patch vetoed by Alan into suspend2, the Alan's NAK would have applied to suspend2 as a whole).

Generally speaking suspend2 is a collection of many different solutions, some of them being more or less questionable, in many different areas which should not be considered all at once. There should be a separate patch for each of them, submitted and discussed on its own. Moreover, some of these solutions may be considered as not the right ones and some patches may get rejected, which may affect the next patches etc. Still, this is the way in which the kernel is developed and we have no other way to follow.

Greetings,
Rafael

#56 lore raw
In reply to Nate Diller
Quoted 58 lines · Nate Diller
Quoted 45 lines · Jeff V. Merkey
Quoted 4 lines · Gregory Maxwell
Quoted 2 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system

and the

Quoted 1 line
Quoted 1 line

tree basis make it very hard to avoid such problems. XFS appears

to have

Quoted 12 lines
Quoted 3 lines

managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Quoted 6 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?

How's Wikimania going. :-)

What he says is correct. I have seen some serious issues with reiserfs in terms of stability and
data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate.
I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10
due to issues I have seen with data corruption and hard hangs on RAID 0 read/write sector errors. I have
stopped using it for local drives and based everything on EXT3. Not to say it won't get there eventually, but
file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

Quoted 9 lines

The Wikimedia appliances use Wolf Mountain, and I've tested it for about 4 months with few problems, but
I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better
than either EXT3 or ReiserFS. Despite this, for vertical wiki servers, its ok to go out with, folks can specifiy
whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked"
completely, though it does scale to 1 exabyte FS images.

Quoted 7 lines

i've seen you mention the Wolf Mountain FS in other emails, but google isn't telling me a lot about it. Do you have a whitepaper? are there any published benchmark results? what sort of workloads do you benchmark?

NATE

Wikipedia is the app for now. I have not done any benchmarks on the FS side, just the capture side, and its been transferred to another entity. I have no idea what they are naming it to, but I expect you may hear about it soon. One of the incarnations of it is Solera's DSFS which can be reviewed here:

www.soleranetworks.com

I can sustain 850 MB/S throughput from user space with it -- about 5 x any other FS. On some hardware, I've broken the 1.25 GB/S (gigabyte/second) windows with it.

Jeff

#131 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Matthias Andree
Quoted 7 lines · Matthias Andree
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

why's that? postmark is one of the standard benchmarks...

Quoted 5 lines

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Data:
204.62 megabytes read (8.53 megabytes per second) 271.49 megabytes written (11.31 megabytes per second)

looks pretty I/O bound to me, 11.31 MB/s isn't exactly your latest DDR RAM bandwidth. as far as the synchronous I/O question, Reiser4 in this case acts more like a log-based FS. That allows it to "overlap" synchronous operations that are being submitted by multiple threads.

NATE

#172 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Horst H. von Brand
Quoted 11 lines · Horst H. von Brand
Quoted 8 lines · Adrian Ulrich
Quoted 5 lines
Quoted 2 lines

Great to see that Sun ships a state-of-the-art Filesystem with Solaris... I think linux should do the same...

This would be worthwhile, if only to be able to futz around in Solaris-made filesystems.

s/I think linux should do the same/I think linux should include Reiser4/
;-)

So ZFS isn't "state-of-the-art"?

maybe reiser4 and ZFS are. certainly, they optimize for different behavior though

Quoted 6 lines

[...]

Quoted 1 line

But i'd rather like to see a Linux version of WAFL :-)

WAFL is for high-turnover filesystems on RAID-5 (and assumes flash memory staging areas). Not your run-of-the-mill desktop...

yeah, good thing nobody tries to use linux for high-turnover servers with RAID. pish-posh.

Quoted 9 lines
Quoted 3 lines

ZFS didn't really impress me:
The Volume-Manager is nice but the Filesystem.. well: It beats UFS .. sometimes ;-)

OK, ext3 + LVM it is then.

Quoted 1 line

See also: http://spam.workaround.ch/dull/postmark.txt

Interesting.

yeah, i hadn't seen postmark numbers on linux before. maybe mongo isn't so biased after all

NATE

#132 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Nate Diller
Quoted 18 lines · Nate Diller
Quoted 7 lines · Matthias Andree
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

why's that? postmark is one of the standard benchmarks...

Quoted 5 lines

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Quoted 8 lines

Data:
204.62 megabytes read (8.53 megabytes per second) 271.49 megabytes written (11.31 megabytes per second)

looks pretty I/O bound to me, 11.31 MB/s isn't exactly your latest DDR RAM bandwidth. as far as the synchronous I/O question, Reiser4 in this case acts more like a log-based FS. That allows it to "overlap" synchronous operations that are being submitted by multiple threads.

what you are missing is that apps that need to do lots of syncing (databases, mail servers) need to wait for the data to hit non-volitile media before the write is complete. this limits such apps to ~1 write per revolution of the platters (yes it's possible for a limited time to have multiple writes to different things happen to be on the same track, but the counter is the extra seek time needed between tracks)

so any benchmark that shows more transactions then the media has revolutions is highly suspect (now if you have battery-backed cache, or the equivalent you can blow past these limits)

on consumer (7200 rpm) drives this limit is 120/sec, on high-end drives (15Krpm scsi's this is 250/sec, and on the 10k rpm drives in the middle it's about 166/sec.

David Lang

#57 lore raw
In reply to Jeff V. Merkey
Quoted 67 lines · Jeff V. Merkey
Quoted 58 lines · Nate Diller
Quoted 45 lines · Jeff V. Merkey
Quoted 4 lines · Gregory Maxwell
Quoted 2 lines · Alan Cox

Its well accepted that reiserfs3 has some robustness problems in the face of physical media errors. The structure of the file system

and the

Quoted 1 line
Quoted 1 line

tree basis make it very hard to avoid such problems. XFS appears

to have

Quoted 12 lines
Quoted 3 lines

managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research that I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss inducing failure conditions that Reiser3 detects an panics under.

Quoted 6 lines

Are you sure that you aren't commenting on cases where Reiser3 alerts the user to a critical data condition (via a panic) which leads to a trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?

How's Wikimania going. :-)

What he says is correct. I have seen some serious issues with reiserfs in terms of stability and
data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate.
I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10
due to issues I have seen with data corruption and hard hangs on RAID 0 read/write sector errors. I have
stopped using it for local drives and based everything on EXT3. Not to say it won't get there eventually, but
file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

Quoted 9 lines

The Wikimedia appliances use Wolf Mountain, and I've tested it for about 4 months with few problems, but
I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better
than either EXT3 or ReiserFS. Despite this, for vertical wiki servers, its ok to go out with, folks can specifiy
whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked"
completely, though it does scale to 1 exabyte FS images.

Quoted 7 lines

i've seen you mention the Wolf Mountain FS in other emails, but google isn't telling me a lot about it. Do you have a whitepaper? are there any published benchmark results? what sort of workloads do you benchmark?

NATE

Quoted 7 lines

Wikipedia is the app for now. I have not done any benchmarks on the FS side, just the capture side, and its been transferred to another entity. I have no idea what they are naming it to, but I expect you may hear about it soon. One of the incarnations
of it is Solera's DSFS which can be reviewed here:

www.soleranetworks.com

so this is a single stream, write only? ...

Quoted 3 lines

I can sustain 850 MB/S throughput from user space with it -- about 5 x any other FS. On some hardware, I've broken
the 1.25 GB/S (gigabyte/second) windows with it.

and you're saying it scales to much higher multi-spindle single-machine throughput. cool.

i'd love to see a whitepaper, or failing that, have an off-list discussion of your approach and the various kernel limitations you ran up against in testing. i don't suppose they invited you to the Kernel Summit to talk about it, heh.

NATE

#133 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Lang
Quoted 29 lines · David Lang
Quoted 18 lines · Nate Diller
Quoted 7 lines · Matthias Andree
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

why's that? postmark is one of the standard benchmarks...

Quoted 5 lines

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Quoted 8 lines

Data:
204.62 megabytes read (8.53 megabytes per second) 271.49 megabytes written (11.31 megabytes per second)

looks pretty I/O bound to me, 11.31 MB/s isn't exactly your latest DDR RAM bandwidth. as far as the synchronous I/O question, Reiser4 in this case acts more like a log-based FS. That allows it to "overlap" synchronous operations that are being submitted by multiple threads.

Quoted 6 lines

what you are missing is that apps that need to do lots of syncing (databases, mail servers) need to wait for the data to hit non-volitile media before the write is complete. this limits such apps to ~1 write per revolution of the platters (yes it's possible for a limited time to have multiple writes to different things happen to be on the same track, but the counter is the extra seek time needed between tracks)

this is true so long as there is only one thread submitting I/O and doing fsync(). for something like a mail server, it can run multi-threaded, and still get data integrity, if the changes are spread out across more than one file.

Quoted 3 lines

so any benchmark that shows more transactions then the media has revolutions is highly suspect (now if you have battery-backed cache, or the equivalent you can blow past these limits)

not all workloads are completely serial, transactions themselves may have no inter-dependencies at all. so it depends on the benchmark, and what workload you're measuring. in cases like this, threading can have a big advantage.

NATE

#134 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 37 lines · Nate Diller
Quoted 29 lines · David Lang
Quoted 18 lines · Nate Diller
Quoted 7 lines · Matthias Andree
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

why's that? postmark is one of the standard benchmarks...

Quoted 5 lines

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Quoted 8 lines

Data:
204.62 megabytes read (8.53 megabytes per second) 271.49 megabytes written (11.31 megabytes per second)

looks pretty I/O bound to me, 11.31 MB/s isn't exactly your latest DDR RAM bandwidth. as far as the synchronous I/O question, Reiser4 in this case acts more like a log-based FS. That allows it to "overlap" synchronous operations that are being submitted by multiple threads.

Quoted 6 lines

what you are missing is that apps that need to do lots of syncing (databases, mail servers) need to wait for the data to hit non-volitile media before the write is complete. this limits such apps to ~1 write per revolution of the platters (yes it's possible for a limited time to have multiple writes to different things happen to be on the same track, but the counter is the extra seek time needed between tracks)

Quoted 4 lines

this is true so long as there is only one thread submitting I/O and doing fsync(). for something like a mail server, it can run multi-threaded, and still get data integrity, if the changes are spread out across more than one file.

only if those multiple files all happen to live (along with their metadata) on the same track.

Quoted 8 lines
Quoted 3 lines

so any benchmark that shows more transactions then the media has revolutions is highly suspect (now if you have battery-backed cache, or the equivalent you can blow past these limits)

not all workloads are completely serial, transactions themselves may have no inter-dependencies at all. so it depends on the benchmark, and what workload you're measuring. in cases like this, threading can have a big advantage.

in the real-world (and benchmarks that simulate it fairly) the data spans multiple tracks so your best case is considerably less then the max I listed becouse you frequently have to seek around a lot to do your writes to multiple places on disk. more threads running should mean that you are attempting to write to more places on disk, which will cause more seeks, dropping you further below the max.

David Lang

#135 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 42 lines · David Lang
Quoted 40 lines · Nate Diller
Quoted 38 lines · David Lang
Quoted 18 lines · Nate Diller
Quoted 7 lines · Matthias Andree
Quoted 3 lines · Adrian Ulrich

See also: http://spam.workaround.ch/dull/postmark.txt

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Whatever Postmark does, this looks pretty besides the point.

why's that? postmark is one of the standard benchmarks...

Quoted 5 lines

Are these actual transactions with the "D"urability guarantee? 3000/s doesn't look too much like you're doing synchronous I/O (else figures around 70/s perhaps 100/s would be more adequate), and cache exercise is rather irrelevant for databases that manage real (=valuable) data...

Quoted 8 lines

Data:
204.62 megabytes read (8.53 megabytes per second) 271.49 megabytes written (11.31 megabytes per second)

looks pretty I/O bound to me, 11.31 MB/s isn't exactly your latest DDR RAM bandwidth. as far as the synchronous I/O question, Reiser4 in this case acts more like a log-based FS. That allows it to "overlap" synchronous operations that are being submitted by multiple threads.

what you are missing is that apps that need to do lots of syncing (databases,
mail servers) need to wait for the data to hit non-volitile media before the
write is complete. this limits such apps to ~1 write per revolution of the platters (yes it's possible for a limited time to have multiple writes to different things happen to be on the same track, but the counter is the extra
seek time needed between tracks)

Quoted 7 lines
Quoted 4 lines

this is true so long as there is only one thread submitting I/O and doing fsync(). for something like a mail server, it can run multi-threaded, and still get data integrity, if the changes are spread out across more than one file.

only if those multiple files all happen to live (along with their metadata) on the same track.

this is only a limitation for filesystems which do in-place data and metadata updates. this is why i mentioned the similarities to log file systems (see rosenblum and ousterhout, 1991). they observed an order-of-magnitude increase in performance for such workloads on their system.

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

so any benchmark that shows more transactions then the media has revolutions is
highly suspect (now if you have battery-backed cache, or the equivalent you can
blow past these limits)

not all workloads are completely serial, transactions themselves may have no inter-dependencies at all. so it depends on the benchmark, and what workload you're measuring. in cases like this, threading can have a big advantage.

Quoted 6 lines

in the real-world (and benchmarks that simulate it fairly) the data spans multiple tracks so your best case is considerably less then the max I listed becouse you frequently have to seek around a lot to do your writes to multiple places on disk. more threads running should mean that you are attempting to write to more places on disk, which will cause more seeks, dropping you further below the max.

postmark is very much real world. reiser4 just doesn't always do in-place writes.

NATE

#58 lore raw
In reply to Nate Diller
Quoted 98 lines · Nate Diller
Quoted 77 lines · Jeff V. Merkey
Quoted 7 lines · Nate Diller
Quoted 5 lines · Jeff V. Merkey
Quoted 3 lines · Gregory Maxwell
Quoted 1 line · Alan Cox

Its well accepted that reiserfs3 has some robustness problems

in the

Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

face of physical media errors. The structure of the file system

and the

Quoted 1 line
Quoted 1 line

tree basis make it very hard to avoid such problems. XFS appears

to have

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

managed to achieve both robustness and better data structures.

How reiser4 compares I've no idea.

Citation?

I ask because your clam differs from the only detailed research

that

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'm aware of on the subject[1]. In figure 2 of the iron filesystems paper that Ext3 is show to ignore a great number of data-loss

inducing

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

failure conditions that Reiser3 detects an panics under.

Are you sure that you aren't commenting on cases where Reiser3

alerts

Quoted 1 line
Quoted 1 line
Quoted 1 line

the user to a critical data condition (via a panic) which leads

to a

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

trouble report while ext3 ignores the problem which suppresses the trouble report from the user?

*1) http://www.cs.wisc.edu/adsl/Publications/iron-sosp05.pdf

Hi Gregory, Wikimedia Foundation and LKML?

How's Wikimania going. :-)

What he says is correct. I have seen some serious issues with

reiserfs

Quoted 6 lines
Quoted 6 lines

in terms of stability and
data corruption. Resier is however FASTER, but the statement is has robustness issues is accurate.
I was using reiserfs but we opted to make EXT3 the default for Solera appliances, even when using Suse 10
due to issues I have seen with data corruption and hard hangs on

RAID 0

Quoted 2 lines
Quoted 2 lines

read/write sector errors. I have
stopped using it for local drives and based everything on EXT3.

Not to

Quoted 5 lines
Quoted 5 lines

say it won't get there eventually, but
file systems have to endure a lot of time in the field and deployment befor they are ready for prime time.

The Wikimedia appliances use Wolf Mountain, and I've tested it for

about

Quoted 4 lines
Quoted 4 lines

4 months with few problems, but
I only use it for hosting the Cherokee Langauge Wikipedia. It's performance is several magnitudes better
than either EXT3 or ReiserFS. Despite this, for vertical wiki

servers,

Quoted 6 lines
Quoted 4 lines

its ok to go out with, folks can specifiy
whether they want appliances with EXT3, Reiser, or WMFS, but iit's a long way from being "cooked"
completely, though it does scale to 1 exabyte FS images.

Quoted 7 lines

i've seen you mention the Wolf Mountain FS in other emails, but google isn't telling me a lot about it. Do you have a whitepaper? are there any published benchmark results? what sort of workloads do you benchmark?

NATE

Quoted 7 lines

Wikipedia is the app for now. I have not done any benchmarks on the FS side, just the capture side, and its been transferred to another entity. I have no idea what they are naming it to, but I expect you may hear about it soon. One of the incarnations
of it is Solera's DSFS which can be reviewed here:

www.soleranetworks.com

so this is a single stream, write only? ...

Quoted 3 lines

I can sustain 850 MB/S throughput from user space with it -- about 5 x any other FS. On some hardware, I've broken
the 1.25 GB/S (gigabyte/second) windows with it.

and you're saying it scales to much higher multi-spindle single-machine throughput. cool.

Quoted 7 lines

i'd love to see a whitepaper, or failing that, have an off-list discussion of your approach and the various kernel limitations you ran up against in testing. i don't suppose they invited you to the Kernel Summit to talk about it, heh.

NATE

The patents have been filed for over a year, and will publish in several weeks at uspto.gov -- that's the only acclaim I care for -- one that results in value for the industry and more patent protection for Linux and profits for folks. No, I have not been invited to the summit, probably because of the lawsuit I filed against some folks who were threatening my family -- Peter Anvin booted me off Kernel.org after allowing folks to pinch my code and copy my bash history files all over the internet, and several folks have stiffed me. I could care less. I keep creating cool technology, make tons of money off of it, and I have cultivated an excellent relationship with the Wikimedia Foundation, and I am now the principal contributor on the Cherokee Wikipedia. Wales even deleted the article folks had used to smear me and made folks rewrite it. Wales is a very nice man and good dude.

I am content to contribute to Linux from a business viewpoint, and if the treatment I received from Anvin is par for kernel.org accounts,
I don't care for one -- IP addresses are rather cheap on the internet. I was and have remained loyal to Linux through it all.

I am appreciative of your interest. Check uspto.gov in next few weeks for published applications, it's all described there, distributed architecture and all.

All my Wikilove.

Jeff

#136 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Nate Diller
Quoted 5 lines · Nate Diller

this is only a limitation for filesystems which do in-place data and metadata updates. this is why i mentioned the similarities to log file systems (see rosenblum and ousterhout, 1991). they observed an order-of-magnitude increase in performance for such workloads on their system.

It's well known that transactions that would thrash on UFS or ext2fs may have quieter access patterns with shorter strokes can benefit from logging, data journaling, whatever else turns seeks into serial writes. And then, the other question with wandering logs (to avoid double writes) and such, you start wondering how much fragmentation you get as the price to pay for avoiding seeks and double writes at the same time. TANSTAAFL, or how long the system can sustain such access patterns, particularly if it gets under memory pressure and must move. Even with lazy allocation and other optimizations, I question the validity of 3000/s or faster transaction frequencies. Even the 500 on ext3 are suspect, particularly with 7200/min (s)ATA crap. This sounds pretty much like the drive doing its best to shuffle blocks around in its 8 MB cache and lazily writing back.

sdparm --clear=WCE /dev/sda # please.

Signature · 1 line
Matthias Andree
#342 lore raw

Re: suspend2 merge history [was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Hua Zhong
Quoted 7 lines · Hua Zhong
Quoted 7 lines
Quoted 1 line

Suspend2 patch is open source. You can always take a look.

swsusp is open source. You can always take a look. And you can always submit a patch.

Quoted 1 line

Moreover, if someone claims suspend2 isn't ready for merge, or the

Quoted 8 lines
Quoted 2 lines

Moreover, if someone claims swsusp is broken, they should attach bugzilla id.

Pavel,

You can't blame me for not doing these things, because I am not a maintainer. However, you are, and you defend yourself so hard for that position, so if _you_ don't do these things, people complain.

I have taken a look at suspend2 and decided I did not like it. (see archives). If you think parts of suspend2 are useful (== fix problem on hardware you have), separate them and submit them.

If you want to claim that swsusp is broken, you should attach bugzilla ids. Anything else is unhelpful.

I'm maintainer, but that does not mean that I have every possible notebook on earth.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#137 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Matthias Andree
Quoted 8 lines · Matthias Andree
Quoted 5 lines · Nate Diller

this is only a limitation for filesystems which do in-place data and metadata updates. this is why i mentioned the similarities to log file systems (see rosenblum and ousterhout, 1991). they observed an order-of-magnitude increase in performance for such workloads on their system.

Quoted 13 lines

It's well known that transactions that would thrash on UFS or ext2fs may have quieter access patterns with shorter strokes can benefit from logging, data journaling, whatever else turns seeks into serial writes. And then, the other question with wandering logs (to avoid double writes) and such, you start wondering how much fragmentation you get as the price to pay for avoiding seeks and double writes at the same time. TANSTAAFL, or how long the system can sustain such access patterns, particularly if it gets under memory pressure and must move. Even with lazy allocation and other optimizations, I question the validity of 3000/s or faster transaction frequencies. Even the 500 on ext3 are suspect, particularly with 7200/min (s)ATA crap. This sounds pretty much like the drive doing its best to shuffle blocks around in its 8 MB cache and lazily writing back.

it's not my benchmark, and you are right to be interested in more information. I would be curious about such things as write barrier support, average/min/max transaction latency, and number of individual threads, as well as hardware specs. i also suspect that the numbers would be altered a bit by testing with different I/O schedulers. unfortunately, namesys has considered mongo a replacement for postmark, so i cannot point to any more rigorous postmark tests ATM.

however, the results seem consistent with what i would expect for the various file systems, with a significant number of threads. after all, even ext3 has the benefit of a disk scheduler, especially if barriers are disabled

NATE

#138 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Matthias Andree
Quoted 8 lines · Matthias Andree
Quoted 5 lines · Nate Diller

this is only a limitation for filesystems which do in-place data and metadata updates. this is why i mentioned the similarities to log file systems (see rosenblum and ousterhout, 1991). they observed an order-of-magnitude increase in performance for such workloads on their system.

Quoted 6 lines

It's well known that transactions that would thrash on UFS or ext2fs may have quieter access patterns with shorter strokes can benefit from logging, data journaling, whatever else turns seeks into serial writes. And then, the other question with wandering logs (to avoid double writes) and such, you start wondering how much fragmentation you get as the price to pay for avoiding seeks and double writes at the same time.

So you use a repacker. Nice thing about a repacker is, everyone has downtime. Better to plan to be a little sluggish when you'll have 1/10th or 1/50th of the users than be MUCH slower all the time.

You're right, though, to ask the question:

Quoted 2 lines

TANSTAAFL, or how long the system can sustain such access patterns, particularly if it gets under memory pressure and must move.

Anyone care to run some very long benchmarks?

Quoted 6 lines

Even with
lazy allocation and other optimizations, I question the validity of 3000/s or faster transaction frequencies. Even the 500 on ext3 are suspect, particularly with 7200/min (s)ATA crap. This sounds pretty much like the drive doing its best to shuffle blocks around in its 8 MB cache and lazily writing back.

Oh, I'm curious -- do hard drives ever carry enough battery/capacitance to cover their caches? It doesn't seem like it would be that hard/expensive, and if it is done that way, then I think it's valid to leave them on. You could just say that other filesystems aren't taking as much advantage of newer drive features as Reiser :P

Anyway, remember that the primary tool of science is not logic. Logic is the primary tool of philosophy. The primary tool of science is observation.

Sorry, the only machines I could really run this on are about to be in remote only mode for a couple weeks. I'm hesitant to hit them too hard.

#126 lore raw
In reply to Matthias Andree
Quoted 9 lines · Matthias Andree
Quoted 4 lines

Have you ever seen VxFS or WAFL in action?

No I haven't. As long as they are commercial, it's not likely that I will.

WAFL was well done. It has several innovations that I admire,
including quota trees, non-support of fragments for performance reasons, and the basic WAFL notion applied to an NFS RAID special (though important) case.

#144 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Theodore Tso
Quoted 22 lines · Theodore Tso
Quoted 8 lines · David Lang

just becouse you have redundancy doesn't mean that your data is idle enough for you to run a repacker with your spare cycles. to run a repacker you need a time when the chunk of the filesystem that you are repacking is not being accessed or written to. it doesn't matter if that data lives on one disk or 9 disks all mirroring the same data, you can't just break off 1 of the copies and repack that becouse by the time you finish it won't match the live drives anymore.

Quoted 10 lines

database servers have a repacker (vaccum), and they are under tremendous preasure from their users to avoid having to use it becouse of the performance hit that it generates. (the theory in the past is exactly what was presented in this thread, make things run faster most of the time and accept the performance hit when you repack). the trend seems to be for a repacker thread that runs continuously, causing a small impact all the time (that can be calculated into the capacity planning) instead of a large impact once in a while.

Quoted 4 lines

Ah, but as soon as the repacker thread runs continuously, then you lose all or most of the claimed advantage of "wandering logs".

Wandering logs is a term specific to reiser4, and I think you are making a more general remark.

You are missing the implications of the oft-cited statistic that 80% of
files never or rarely move. You are also missing the implications of
the repacker being able to do larger IOs than occur for a random tiny IO workload which is impacting a filesystem that is performing allocations on the fly.

Quoted 12 lines

Specifically, the claim of the "wandering log" is that you don't have to write your data twice --- once to the log, and once to the final location on disk (whereas with ext3 you end up having to do double writes). But if the repacker is running continuously, you end up doing double writes anyway, as the repacker moves things from a location that is convenient for the log, to a location which is efficient for reading. Worse yet, if the repacker is moving disk blocks or objects which are no longer in cache, it may end up having to read objects in before writing them to a final location on disk. So instead of a write-write overhead, you end up with a write-read-write overhead.

Quoted 5 lines

But of course, people tend to disable the repacker when doing benchmarks because they're trying to play the "my filesystem/database has bigger performance numbers than yours" game....

When the repacker is done, we will just for you run one of our benchmarks the morning after the repacker is run (and reference this email);-).... that was what you wanted us to do to address your concern, yes?;-)

Quoted 5 lines

- Ted

#139 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Masover
Quoted 3 lines · David Masover

So you use a repacker. Nice thing about a repacker is, everyone has downtime. Better to plan to be a little sluggish when you'll have 1/10th or 1/50th of the users than be MUCH slower all the time.

Actually, that's a problem with log-structured filesystems in general. There are quite a few real-life workloads where you *don't* have downtime. The thing is, in a global economy, you move from the London/European stock exchanges, to the New York/US exchanges, to the Asian exchanges, with little to no downtime available. In addition, people have been getting more sophisticated with workload consolidation tricks so that you use your "downtime" for other applications (either to service other parts of the world, or to do daily summaries, 3-d frame rendering at animation companies, etc.) So the assumption that there will always be time to run the repacker is a dangerous one.

The problem is that many benchmarks (such as taring and untaring the kernel sources in reiser4 sort order) are overly simplistic, in that they don't really reflect how people use the filesystem in real life. (How many times can you guarantee that files will be written in the precise hash/tree order so that the filesystem gets the best possible time?) A more subtle version of this problem happens for filesystems where their performance degrades dramatically over-time without a repacker. If the benchmark doesn't take into account the need for repacker, or if the repacker is disabled or fails to run during the benchmark, the filesystem are in effect "cheating" on the benchmark because there is critical work which is necessary for the long-term health of the filesystem which is getting deferred until after the benchmark has finished measuring the performance of the system under test.

This sort of marketing benchmarks ("lies, d*mn lies, and benchmarks") may be useful for trying to scam mainline acceptance of the filesystem code, or to make pretty graphs that make log-structured filesystems look good on Usenix papers, but despite the fact that huge numbers of papers were written about the lfs filesystem two decades ago, it never was used in real-life by any of the commercial Unix systems. This wasn't an accident, and it wasn't due to a secret conspiracy of BSD fast filesystem hackers keeping people from using lfs. No, the BSD lfs died on its own merits....

- Ted

#152 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 8 lines · Theodore Tso
Quoted 1 line · David Masover

So you use a repacker. Nice thing about a repacker is, everyone has downtime. Better to plan to be a little sluggish when you'll have 1/10th or 1/50th of the users than be MUCH slower all the time.

Actually, that's a problem with log-structured filesystems in general. There are quite a few real-life workloads where you *don't* have downtime. The thing is, in a global economy, you move from the London/European stock exchanges, to the New York/US exchanges, to the Asian exchanges, with little to no downtime available.

Such systems must have redundancy, however. And if you have 2-3 servers hot in case one of them goes down, I can see switching between which is more active, and which is repacking.

This repacker is online, hence a filesystem being repacked would have to be less active, not necessarily down. So repack the backup server, then make the backup server the active one and repack the main server. If the main server goes down while the backup is repacking, kill the repack process.

I actually have a problem imagining a system where you don't have enough spare capacity (disk, CPU, spare servers) to run a repacker every now and then, but which also must have 100% uptime. What happens when a disk goes bad? Or when power to half the country goes out? Or... You get the idea.

Quoted 7 lines

In addition,
people have been getting more sophisticated with workload consolidation tricks so that you use your "downtime" for other applications (either to service other parts of the world, or to do daily summaries, 3-d frame rendering at animation companies, etc.) So the assumption that there will always be time to run the repacker is a dangerous one.

3D frame rendering in particular doesn't require much disk use, does it?
Daily summaries, I guess, depends on what kind of summaries they are. And anyway, those applications are making the same dangerous assumption.

And anyway, I suspect the repacker will work best once a week or so, but no one knows yet, as they haven't written it yet.

Quoted 3 lines

The problem is that many benchmarks (such as taring and untaring the kernel sources in reiser4 sort order) are overly simplistic, in that they don't really reflect how people use the filesystem in real life.

That's true. I'd also like to see lots more benchmarks.

Quoted 7 lines

If the benchmark doesn't take into account the need for repacker, or if the repacker is disabled or fails to run during the benchmark, the filesystem are in effect "cheating" on the benchmark because there is critical work which is necessary for the long-term health of the filesystem which is getting deferred until after the benchmark has finished measuring the performance of the system under test.

In this case, the only fair test would be to run the benchmark 24/7 for a week, and run the repacker on a weekend. Or however you're planning to do it. It wouldn't be fair to run a 10-minute or 1-hour benchmark and then immediately run the repacker.

But I'd also like to see more here, especially about fragmentation. If the repacker will cost money, the system should be reasonably good at avoiding fragmentation. I'm wondering if I should run a benchmark on my systems -- they're over a year old, and while they aren't under particularly heavy load, they should be getting somewhat fragmented by now.

#140 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Theodore Tso

Different users have different needs.

I agree, there are many users who can not afford any downtime.

I worked at the NYSE and they reboot all their computers once a week. We had a policy at NYSE. If you suspect a computer has hardware problems, take it off line. It is better to be short a few computers then have that computer bring everything down. And fix that computer off line. Until last year I worked at world wide webmail provider. And trust me they could not avoid downtime. But the fact is everyone has down time, because hardware breaks and software is broken.

However on the other hand if I am using a filesystem for recording TV programs or to play home computer games. Downtime is not the problem I really care about. I am rebooting lots anyway.

The problem I see is managing disk errors. NOT repackers, unless ofcourse I need to run it all the time just to keep the filesystem in a usable state.

The question is why not include lots of new filesystems. reiser4, ZFS
They both have their own markets. And perhaps a really good clustering filesystem for markets that require NO downtime.

-Tim

Quoted 6 lines · --- Theodore Tso
Quoted 1 line · David Masover

So you use a repacker. Nice thing about a

repacker is, everyone has

Quoted 1 line

downtime. Better to plan to be a little sluggish

when you'll have

Quoted 8 lines
Quoted 1 line

1/10th or 1/50th of the users than be MUCH slower

all the time.

Actually, that's a problem with log-structured filesystems in general.
There are quite a few real-life workloads where you *don't* have
downtime. The thing is, in a global economy, you

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#154 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Masover
Quoted 1 line · David Masover

Oh, I'm curious -- do hard drives ever carry enough battery/capacitance to cover their caches? It doesn't seem like it would be that hard/expensive, and if it is done that way, then I think it's valid to leave them on. You could just say that other filesystems aren't taking as much advantage of newer drive features as Reiser :P

there are no drives that have the ability to flush their cache after they loose power.

now, that being said, /. had a story within the last couple of days about hard drive manufacturers adding flash to their hard drives. they may be aiming to add some non-volitile cache capability to their drives, although I didn't think that flash writes were that fast (needed if you dump the cache to flash when you loose power), or that easy on power (given that you would first loose power), and flash has limited write cycles (needed if you always use the cache).

I've heard to many fancy-sounding drive technologies that never hit the market, I'll wait until thye are actually available before I start counting on them for anything (let alone design/run a filesystem that requires them :-)

external battery backed cache is readily available, either on high-end raid controllers or as seperate ram drives (and in raid array boxes), but nothing on individual drives.

David Lang

#141 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Timothy Webster
Quoted 1 line · Timothy Webster

Different users have different needs.

I'm having trouble thinking of users who need an FS that doesn't need a repacker.

The disk error problem, though, you're right -- most users will have to get bitten by this, hard, at least once, or they'll never get the importance of it. But it'd be nice if it's not too hard, and we can actually recover most of their files.

Still, I can see most people who are aware of this problem using RAID, backups, and not caring if their filesystem tolerates bad hardware.

Quoted 1 line

The problem I see is managing disk errors.

I see this kind of the same way. If your disk has errors, you should be getting a new disk. If you can't do that, you can run a mirrored RAID -- even on SATA, you should be able to hotswap it.

Even for a home/desktop user, disks are cheap, and getting cheaper all the time. All you have to do is run the mean time between failure numbers by them, and ask them if their backup is enough.

Quoted 3 lines

And perhaps a
really good clustering filesystem for markets that require NO downtime.

Thing is, a cluster is about the only FS I can imagine that could reasonably require (and MAYBE provide) absolutely no downtime. Everything else, the more you say it requires no downtime, the more I say it requires redundancy.

Am I missing any more obvious examples where you can't have enough redundancy, but you can't have downtime either?

#155 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Lang
Quoted 5 lines · David Lang
Quoted 1 line · David Masover

Oh, I'm curious -- do hard drives ever carry enough battery/capacitance to cover their caches? It doesn't seem like it would be that hard/expensive, and if it is done that way, then I think it's valid to leave them on. You could just say that other filesystems aren't taking as much advantage of newer drive features as Reiser :P

there are no drives that have the ability to flush their cache after they loose power.

Aha, so back to the usual argument: UPS! It takes a fraction of a second to flush that cache.

Quoted 1 line

now, that being said, /. had a story within the last couple of days about hard drive manufacturers adding flash to their hard drives. they may be aiming to add some non-volitile cache capability to their drives, although I didn't think that flash writes were that fast (needed if you dump the cache to flash when you loose power), or that easy on power (given that you would first loose power), and flash has limited write cycles (needed if you always use the cache).

But, the point of flash was not to replace the RAM cache, but to be another level. That is, you have your Flash which may be as fast as the disk, maybe faster, maybe less, and you have maybe a gig worth of it. Even the bloatiest of OSes aren't really all that big -- my OS X came installed, with all kinds of apps I'll never use, in less than 10 gigs.

And I think this story was awhile ago (a dupe? Not surprising), and the point of the Flash is that as long as your read/write cache doesn't run out, and you're still in that 1 gig of Flash, you're a bit safer than the RAM cache, and you can also leave the disk off, as in, spinned down.
Parked.

Very useful for a laptop -- I used to do this in Linux by using Reiser4, setting the disk to spin down, and letting lazy writes do their thing, but I didn't have enough RAM, and there's always the possibility of losing data. But leaving the disk off is nice, because in the event of sudden motion, it's safer that way. Besides, most hardware gets designed for That Other OS, which doesn't support any kind of Laptop Mode, so it's nice to be able to enforce this at a hardware level, in a safe way.

Quoted 1 line

I've heard to many fancy-sounding drive technologies that never hit the market, I'll wait until thye are actually available before I start counting on them for anything (let alone design/run a filesystem that requires them :-)

Or even remember their names.

Quoted 1 line

external battery backed cache is readily available, either on high-end raid controllers or as seperate ram drives (and in raid array boxes), but nothing on individual drives.

Ah. Curses.

UPS, then. If you have enough time, you could even do a Software Suspend first -- that way, when power comes back on, you boot back up, and if it's done quickly enough, connections won't even be dropped...

#153 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Theodore Tso
Quoted 6 lines · Theodore Tso

The problem is that many benchmarks (such as taring and untaring the kernel sources in reiser4 sort order) are overly simplistic, in that they don't really reflect how people use the filesystem in real life. (How many times can you guarantee that files will be written in the precise hash/tree order so that the filesystem gets the best possible time?) A more subtle version of this problem happens for filesystems

Quoted 8 lines

where their performance degrades dramatically over-time without a repacker. If the benchmark doesn't take into account the need for repacker, or if the repacker is disabled or fails to run during the benchmark, the filesystem are in effect "cheating" on the benchmark because there is critical work which is necessary for the long-term health of the filesystem which is getting deferred until after the benchmark has finished measuring the performance of the system under test.

If the file system that requires a repacker can do X operations in 1/2 the time all week long, even if the repacker takes several hours once a week to run, you're still ahead of the game. The load averages on the vast majority of servers have significant peaks and valleys throughout the day, and throughout the week, it wouldn't be hard to find a time where a online repacker is virtually unnoticeable to users. Delaying certain work until the server is less busy might be considered "cheating" on benchmarks, but in the real world most people would consider it good use of resources.

Just like RAID rebuilds, where you can set maximum IO speeds I could see a repacker working in a similar fashion.

Obviously there are some servers where this is unacceptable and in such cases, don't use Reiser4, but I would guess they are few and far between. No file system is perfect for 100% of the work loads thrown at it.

PostgreSQL and its vacuum process comes to mind as something similar to a repacker. PostgreSQL puts off doing some work to later, and it has proven itself over and over again, especially when it comes to scalability. PostgreSQL has recently (v8.0 I believe) moved to a system where it can automatically detect the need to vacuum specific tables, so tables that need it are vacuumed more often, and tables that don't are rarely touched. I don't see any reason why a repacker couldn't work in a similar fashion, once it detects a file to be fragmented over some value, it gets scheduled for repacking when there is idle disk IO available.

The bottom line is once you have a online repacker you instantly open up all sorts of doors. Its well known that disk drives have different sustained read/write performance depending on if the data is on the inside or outside tracks. Perhaps the repacker could also move files around on the disk to get further gains, for instance larger or most commonly used files could be moved to where the disk has the highest sustained read/write performance, and smaller less used files could be moved to the slowest sustained read/write performance areas.

Signature · 1 line
Mike Benoit <[email protected]>
#142 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Masover
Quoted 7 lines · David Masover
Quoted 3 lines

And perhaps a
really good clustering filesystem for markets that require NO downtime.

Thing is, a cluster is about the only FS I can imagine that could reasonably require (and MAYBE provide) absolutely no downtime. Everything else, the more you say it requires no downtime, the more I say it requires redundancy.

Am I missing any more obvious examples where you can't have enough redundancy, but you can't have downtime either?

just becouse you have redundancy doesn't mean that your data is idle enough for you to run a repacker with your spare cycles. to run a repacker you need a time when the chunk of the filesystem that you are repacking is not being accessed or written to. it doesn't matter if that data lives on one disk or 9 disks all mirroring the same data, you can't just break off 1 of the copies and repack that becouse by the time you finish it won't match the live drives anymore.

database servers have a repacker (vaccum), and they are under tremendous preasure from their users to avoid having to use it becouse of the performance hit that it generates. (the theory in the past is exactly what was presented in this thread, make things run faster most of the time and accept the performance hit when you repack). the trend seems to be for a repacker thread that runs continuously, causing a small impact all the time (that can be calculated into the capacity planning) instead of a large impact once in a while.

the other thing they are seeing as new people start useing them is that the newbys don't realize they need to do somthing as archaic as running a repacker periodicly, as a result they let things devolve down to where performance is really bad without understanding why.

David Lang

#156 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressedby kernelnewbies.org regarding reiser4 inclusion]

In reply to David Masover
Quoted 8 lines · David Masover
Quoted 5 lines · David Lang
Quoted 1 line · David Masover

Oh, I'm curious -- do hard drives ever carry enough battery/capacitance to cover their caches? It doesn't seem like it would be that hard/expensive, and if it is done that way, then I think it's valid to leave them on. You could just say that other filesystems aren't taking as much advantage of newer drive features as Reiser :P

there are no drives that have the ability to flush their cache after they loose power.

Aha, so back to the usual argument: UPS! It takes a fraction of a second to flush that cache.

which does absolutly no good if someone trips over the power cord, the fuse blows in the power supply, someone yanks the drive out of the hot-swap bay, etc.

Quoted 5 lines
Quoted 1 line

now, that being said, /. had a story within the last couple of days about hard drive manufacturers adding flash to their hard drives. they may be aiming to add some non-volitile cache capability to their drives, although I didn't think that flash writes were that fast (needed if you dump the cache to flash when you loose power), or that easy on power (given that you would first loose power), and flash has limited write cycles (needed if you always use the cache).

But, the point of flash was not to replace the RAM cache, but to be another level. That is, you have your Flash which may be as fast as the disk, maybe faster, maybe less, and you have maybe a gig worth of it. Even the bloatiest of OSes aren't really all that big -- my OS X came installed, with all kinds of apps I'll never use, in less than 10 gigs.

And I think this story was awhile ago (a dupe? Not surprising), and the point of the Flash is that as long as your read/write cache doesn't run out, and you're still in that 1 gig of Flash, you're a bit safer than the RAM cache, and you can also leave the disk off, as in, spinned down. Parked.

as I understand it flash reads are fast (ram speeds), but writes are pretty slow (comparable or worse to spinning media)

writing to a ram cache, but having a flash drive behind it doesn't gain you any protection. and I don't think you need it for reads

Quoted 5 lines
Quoted 1 line

external battery backed cache is readily available, either on high-end raid controllers or as seperate ram drives (and in raid array boxes), but nothing on individual drives.

Ah. Curses.

UPS, then. If you have enough time, you could even do a Software Suspend first -- that way, when power comes back on, you boot back up, and if it's done quickly enough, connections won't even be dropped...

remember, it can take 90W of power to run your CPU, 100+ to run your video card, plus everything else. even a few seconds of power for this is a very significant amount of energy storage.

however, I did get a pointer recently at a company makeing super-high capcity caps, up to 2600F (F, not uF!) in a 138mmx tall 57mm dia cyliner, however it only handles 2.7v (they have modules that handle higher voltages available)
http://www.maxwell.com/ultracapacitors/index.html

however I don't see these as being standard equipment in systems or on drives anytime soon

David Lang

#151 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Lang
Quoted 11 lines · David Lang
Quoted 7 lines · David Masover
Quoted 3 lines

And perhaps a
really good clustering filesystem for markets that require NO downtime.

Thing is, a cluster is about the only FS I can imagine that could reasonably require (and MAYBE provide) absolutely no downtime. Everything else, the more you say it requires no downtime, the more I say it requires redundancy.

Am I missing any more obvious examples where you can't have enough redundancy, but you can't have downtime either?

just becouse you have redundancy doesn't mean that your data is idle enough for you to run a repacker with your spare cycles.

Then you don't have redundancy, at least not for reliability. In that case, you have redundancy for speed.

Quoted 1 line

to run a repacker you need a time when the chunk of the filesystem that you are repacking is not being accessed or written to.

Reasonably, yes. But it will be an online repacker, so it will be somewhat tolerant of this.

Quoted 1 line

it doesn't matter if that data lives on one disk or 9 disks all mirroring the same data, you can't just break off 1 of the copies and repack that becouse by the time you finish it won't match the live drives anymore.

Aha. That really depends how you're doing the mirroring.

If you're doing it at the block level, then no, it won't work. But if you're doing it at the filesystem level (a cluster-based FS, or something that layers on top of an FS), or (most likely) the database/application level, then when you come back up, the new data is just pulled in from the logs as if it had been written to the FS.

The only example I can think of that I've actually used and seen working is MySQL tables, but that already covers a huge number of websites.

Quoted 1 line

database servers have a repacker (vaccum), and they are under tremendous preasure from their users to avoid having to use it becouse of the performance hit that it generates. (the theory in the past is exactly what was presented in this thread, make things run faster most of the time and accept the performance hit when you repack). the trend seems to be for a repacker thread that runs continuously, causing a small impact all the time (that can be calculated into the capacity planning) instead of a large impact once in a while.

Hmm, if that could be done right, it wouldn't be so bad -- if you get twice the performance but have to repack for 2 hrs at the end of the week, repacker is better, right? So if you could spread the 2 hours out over the week, in theory, you'd still be pretty close to twice the performance.

But that is fairly difficult to do, and may be more difficult to do well than to implement, say, a Reiser4 plugin that operates about on the level of rsync, but on every file modification.

Quoted 1 line

the other thing they are seeing as new people start useing them is that the newbys don't realize they need to do somthing as archaic as running a repacker periodicly, as a result they let things devolve down to where performance is really bad without understanding why.

Yikes. But then, that may be a failure of distro maintainers for not throwing it in cron for them.

I had a similar problem with MySQL. I turned on binary logging so I could do database replication, but I didn't realize I had to actually delete the logs. I now have a daily cron job that wipes out everything except the last day's logs. It could probably be modified pretty easily to run hourly, if I needed to.

Moral of the story? Maybe there's something to this "continuous repacker" idea, but don't ruin a good thing for the rest of us because of newbies.

#157 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressedby kernelnewbies.org regarding reiser4 inclusion]

In reply to David Lang
Quoted 5 lines · David Lang
Quoted 1 line · David Masover

Aha, so back to the usual argument: UPS! It takes a fraction of a second to flush that cache.

which does absolutly no good if someone trips over the power cord, the fuse blows in the power supply, someone yanks the drive out of the hot-swap bay, etc.

Power supply fuse... Yeah, it happens. Drives die, too. This seems fairly uncommon. And dear God, please tell me anyone smart enough to set up a UPS would also be smart enough to make tripping over the power cord rare or impossible.

My box has a cable that runs down behind a desk, between the desk and the wall. Power strip is on the floor, where a UPS will be when I get around to buying one. If someone kicks any cable, it would be where the UPS hits the wall -- but that's also behind the same desk.

Quoted 3 lines

as I understand it flash reads are fast (ram speeds), but writes are pretty slow (comparable or worse to spinning media)

writing to a ram cache, but having a flash drive behind it doesn't gain you any protection. and I don't think you need it for reads

Does gain you protection if you're not using the RAM cache, if you're that paranoid. I don't know if it's cheaper than RAM, but more read cache is always better. And losing power seems a lot less likely than crashing, especially on a Windows laptop, so it does make sense as a product. And a laptop, having a battery, will give you a good bit of warning before it dies. My Powerbook syncs and goes into Sleep mode when it runs low on power (~1%/5mins)

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

external battery backed cache is readily available, either on high-end raid controllers or as seperate ram drives (and in raid array boxes), but nothing on individual drives.

Ah. Curses.

UPS, then. If you have enough time, you could even do a Software Suspend first -- that way, when power comes back on, you boot back up, and if it's done quickly enough, connections won't even be dropped...

remember, it can take 90W of power to run your CPU, 100+ to run your video card, plus everything else. even a few seconds of power for this is a very significant amount of energy storage.

Suspend2 can take about 10-20 seconds. It should be possible to work out the maximum amount of time it can take.

Anyway, according to a quick Google search, my CPU is more like 70W. Video card isn't required on a server, but you may be right on mine. I haven't looked at UPSes lately, though. I need about 3 seconds for a sync, maybe 10 for a suspend, so to be safe I can say for sure I'd be down in about 30 seconds.

So, another Google search, and while you can get a cheap UPS for anywhere from $10 to $100, the sweet spot seems to be a little over $200.

$229, and it's 865W, supposedly for 3.7 minutes. Here's a review:

"This is a great product. It powers an AMD 64 3200+ with beefy (6800GT) graphics card, 21" CRT monitor, secondary 19" CRT, a linux server, a 15" CRT, Cisco 2800XL switch, Linksys WRTG54GS, cable modem, speakers, and many other things. The software says I will get 9 minutes runtime with all of that hooked up, realistically it's about 4 minutes."

This was the lowest time reported. Most of the other reviews say at least 15 minutes, sometimes 30 minutes, with fairly high-end computers listed (and monitors, sometimes two computers/monitors), but nowhere near as much stuff as this guy has.

I checked most of these for Linux support, and UPSes in general seem well supported. So yes, the box will shut off automatically. On a network, it shouldn't be too hard to get one box to shut off all the rest.

It's a lot of money, even at the low end, but when you're already spending a pile of money on a new computer, keep power in mind. And really, even 11 minutes would be fine, but 40 minutes of power is quite a lot compared to less than a minute of time taken to shut down normally -- not even suspend, but a normal shut down. I'd be tempted to try to ride it out for the first 20 minutes, see if power comes back up...

Quoted 4 lines

however, I did get a pointer recently at a company makeing super-high capcity caps, up to 2600F (F, not uF!) in a 138mmx tall 57mm dia cyliner, however it only handles 2.7v (they have modules that handle higher voltages available)
http://www.maxwell.com/ultracapacitors/index.html

however I don't see these as being standard equipment in systems or on drives anytime soon

This seems to be a whole different approach -- more along the lines of in the drive, which would be cool...

#177 lore raw
In reply to Matthias Andree
Quoted 10 lines
Quoted 4 lines

A filesystem with a fixed number of inodes (= not readjustable while mounted) is ehr.. somewhat unuseable for a lot of people with big and *flexible* storage needs (Talking about NetApp/EMC owners)

Which is untrue at least for Solaris, which allows resizing a life file system. FreeBSD and

Linux require an unmount.

Only for shrinking.

Jan Engelhardt

#59 lore raw
In reply to Jeffrey V. Merkey
Quoted 1 line · Jeffrey V. Merkey

... I was and have remained loyal to Linux through it all.

Except for that little fling with SCO, eh?

Off topic, but no more so than your self-aggrandising.

--T

#143 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to David Lang
Quoted 8 lines · David Lang

just becouse you have redundancy doesn't mean that your data is idle enough for you to run a repacker with your spare cycles. to run a repacker you need a time when the chunk of the filesystem that you are repacking is not being accessed or written to. it doesn't matter if that data lives on one disk or 9 disks all mirroring the same data, you can't just break off 1 of the copies and repack that becouse by the time you finish it won't match the live drives anymore.

Quoted 8 lines

database servers have a repacker (vaccum), and they are under tremendous preasure from their users to avoid having to use it becouse of the performance hit that it generates. (the theory in the past is exactly what was presented in this thread, make things run faster most of the time and accept the performance hit when you repack). the trend seems to be for a repacker thread that runs continuously, causing a small impact all the time (that can be calculated into the capacity planning) instead of a large impact once in a while.

Ah, but as soon as the repacker thread runs continuously, then you lose all or most of the claimed advantage of "wandering logs". Specifically, the claim of the "wandering log" is that you don't have to write your data twice --- once to the log, and once to the final location on disk (whereas with ext3 you end up having to do double writes). But if the repacker is running continuously, you end up doing double writes anyway, as the repacker moves things from a location that is convenient for the log, to a location which is efficient for reading. Worse yet, if the repacker is moving disk blocks or objects which are no longer in cache, it may end up having to read objects in before writing them to a final location on disk. So instead of a write-write overhead, you end up with a write-read-write overhead.

But of course, people tend to disable the repacker when doing benchmarks because they're trying to play the "my filesystem/database has bigger performance numbers than yours" game....

- Ted

#147 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 15 lines · Theodore Tso

Ah, but as soon as the repacker thread runs continuously, then you lose all or most of the claimed advantage of "wandering logs". Specifically, the claim of the "wandering log" is that you don't have to write your data twice --- once to the log, and once to the final location on disk (whereas with ext3 you end up having to do double writes). But if the repacker is running continuously, you end up doing double writes anyway, as the repacker moves things from a location that is convenient for the log, to a location which is efficient for reading. Worse yet, if the repacker is moving disk blocks or objects which are no longer in cache, it may end up having to read objects in before writing them to a final location on disk. So instead of a write-write overhead, you end up with a write-read-write overhead.

There's no reason to repack *all* of the data. Many workloads write and delete whole files, so file data should be contiguous. The repacker would only need to move metadata and small files.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#173 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Horst H. von Brand
Quoted 2 lines

So ZFS isn't "state-of-the-art"?

Of course it's state-of-the-art (on Solaris ;-) )

Quoted 2 lines

WAFL is for high-turnover filesystems on RAID-5 (and assumes flash memory staging areas).

s/RAID-5/RAID-4/

Quoted 1 line

Not your run-of-the-mill desktop...

The WAFL-Thing was just a joke ;-)

Regards,
Adrian

#165 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Matthias Andree
Quoted 1 line

suspect, particularly with 7200/min (s)ATA crap.

Quoted 1 line · myself
Quoted 1 line

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Yeah, the test ran on a single SATA-Harddisk (quick'n'dirty). I'm so sorry but i don't have access to a $$$ Raid-System at home.

Anyway: The test shows us that Reiser4 performed very good on my (common 0-8-15) hardware.

Quoted 1 line

sdparm --clear=WCE /dev/sda # please.

How about using /dev/emcpower* for the next benchmark?

I mighty be able to re-run it in a few weeks if people are interested and if i receive constructive suggestions (= Postmark parameters, mkfs options, etc..)

Regards,
Adrian

#212 lore raw

Re: metadata plugins (was Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion)

In reply to David Masover
Quoted 9 lines · David Masover
Quoted 2 lines · Wil Reichert

Any idea how the fragmentation resulting from re-syncing the tree affects performance over time?

Yes, it does affect it a lot. I have no idea how much, and I've never benchmarked it, but purely subjectively, my portage has gotten slower over time.

Delayed allocation still performs a lot better here than the v3 "immediate" allocation. In addition, tree balancing operations are performed on flush as well, so what you get on disk is basically an almost-optimal tree. Of course, this will change a bit over time, but with v4 it takes a lot longer for that to happen than with v3 afaict. There _has_ been some worthwile development in the meantime : )

Kind regards,
Chris

#166 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view" expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Adrian Ulrich

Adrian Ulrich schrieb am 2006-08-01:

Quoted 7 lines
Quoted 1 line

suspect, particularly with 7200/min (s)ATA crap.

Quoted 1 line · myself
Quoted 1 line

A quick'n'dirty ZFS-vs-UFS-vs-Reiser3-vs-Reiser4-vs-Ext3 'benchmark'

Yeah, the test ran on a single SATA-Harddisk (quick'n'dirty). I'm so sorry but i don't have access to a $$$ Raid-System at home.

I'm not asking for you to perform testing on a $$$$ RAID system with SCSI or SAS, but I consider the obtained data (I am focussing on transactions per unit of time) highly suspicious, and suspect write caches might have contributed their share - I haven't seen a drive that shipped with write cache disabled in the past years.

Quoted 3 lines
Quoted 1 line

sdparm --clear=WCE /dev/sda # please.

How about using /dev/emcpower* for the next benchmark?

No, it is valid to run the test on commodity hardware, but if you (or the benchmark rather) is claiming "transactions", I tend to think "ACID", and I highly doubt any 200 GB SATA drive manages 3000 synchronous writes per second without causing either serious fragmentation or background block moving.

This is a figure I'd expect for synchronous random access to RAM disks that have no seek and rotational latencies (and research for hybrid disks w/ flash or other nonvolatile fast random access media to cache actual rotating magnetic plattern access is going on elsewhere).

I didn't mean to say your particular drive were crap, but 200GB SATA drives are low end, like it or not -- still, I have one in my home computer because these Samsung SP2004C are so nicely quiet.

Quoted 3 lines

I mighty be able to re-run it in a few weeks if people are interested and if i receive constructive suggestions (= Postmark parameters, mkfs options, etc..)

I don't know Postmark, I did suggest to turn the write cache off. If your systems uses hdparm -W0 /dev/sda instead, go ahead. But you're right to collect and evaluate suggestions first if you don't want to run a new benchmark every day :)

Signature · 1 line
Matthias Andree
#148 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

In reply to Avi Kivity
Quoted 3 lines · Avi Kivity

There's no reason to repack *all* of the data. Many workloads write and delete whole files, so file data should be contiguous. The repacker would only need to move metadata and small files.

Move small files? What for?

Even if it is "only" moving metadata, it is not different from what ext3 or xfs are doing today (rewriting metadata from the intent log or block journal to the final location).

The UFS+softupdates from the BSD world looks pretty good at avoiding unnecessary writes (at the expense of a long-running but nice background fsck after a crash, which is however easy on the I/O as of recent FreeBSD versions). Which was their main point against logging/journaling BTW, but they are porting XFS as well to save those that need instant complete recovery.

Signature · 1 line
Matthias Andree
#149 lore raw

Re: Solaris ZFS on Linux [Was: Re: the " 'official' point of view"expressed by kernelnewbies.org regarding reiser4 inclusion]

Quoted 10 lines · Matthias Andree
Quoted 1 line · Avi Kivity

There's no reason to repack *all* of the data. Many workloads write

and

Quoted 2 lines

delete whole files, so file data should be contiguous. The repacker would only need to move metadata and small files.

Move small files? What for?

WAFL-style filesystems like contiguous space, so if small files are scattered in otherwise free space, the repacker should free them.

Quoted 4 lines

Even if it is "only" moving metadata, it is not different from what ext3 or xfs are doing today (rewriting metadata from the intent log or block journal to the final location).

There is no need to repack all metadata; only that which helps in creating free space.

For example: if you untar a source tree you'd get mixed metadata and small file data packed together, but there's no need to repack that data.

Signature · 1 line
error compiling committee.c: too many arguments to function