Skip to contentSkip to conversation

LogFS merge

DISCUSSION Jörn Engel 10 participants 23 messages May 2, 2008 – May 5, 2008 VFS

lore
Share & tools
lore
#0 lore raw

Stephen,

not being familiar with either maintaining my own git tree or the -next process, I'd still like to get logfs into mainline. It has gone through six rounds of reviews and the last has been mostly about crossing some i's here and dotting some t's there.

So should it simmer in -next and -mm for another month? Should it go straight into -linus?

Either way, please pull from master.kernel.org:/pub/scm/linux/kernel/git/joern/logfs.git/

Jörn

Signature · 4 lines
If System.PrivateProfileString("",
"HKEY_CURRENT_USER\Software\Microsoft\Office\9.0\Word\Security", "Level") <>
"" Then  CommandBars("Macro").Controls("Security...").Enabled = False
-- from the Melissa-source
#1 lore raw

Hi Jörn,

Quoted 7 lines · Jörn Engel

not being familiar with either maintaining my own git tree or the -next process, I'd still like to get logfs into mainline. It has gone through six rounds of reviews and the last has been mostly about crossing some i's here and dotting some t's there.

So should it simmer in -next and -mm for another month? Should it go straight into -linus?

You probably want an ACK from the VFS maintainers before aiming at mainline. But it surely makes sense to ask Andrew to pull it in -mm now.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 3 lines · Pekka Enberg

You probably want an ACK from the VFS maintainers before aiming at mainline. But it surely makes sense to ask Andrew to pull it in -mm now.

Definitively wants a re-review with all the bits from last time fixed.

How did the inode_lock abuse get fixed, btw? That one was rather lethal.

#4 lore raw
In reply to Jörn Engel
Quoted 10 lines · Jörn Engel

not being familiar with either maintaining my own git tree or the -next process, I'd still like to get logfs into mainline.  It has gone through six rounds of reviews and the last has been mostly about crossing some i's here and dotting some t's there.

So should it simmer in -next and -mm for another month?  Should it go straight into -linus?

Either way, please pull from master.kernel.org:/pub/scm/linux/kernel/git/joern/logfs.git/

There are still a few i's and t's left to dot and cross:

* the changeset comments needs a Signed-off-by: line
* The MAINTAINERS file should list your name and logfs mailing list
* you have a few instances of '#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 23)', that should go away for the merge
* The copyright notice says 2005-2007, it should probably be 2005-2008
* You may want to add a Documentation/filesystems/logfs.txt file explaining the supported mount options.
* CONFIG_LOGFS should be tristate, not bool. Unfortunately, you are still using three symbols that are not exported: swapper_space (through BUG_ON(!page_mapping(page)->a_ops->set_page_dirty)), add_to_page_cache_lru and inode_lock. Not sure what to do about this.
* You should really make sure the version you check in compiles, fs/logfs/logfs.h is missing an #endif. ;-)

Otherwise, I don't see any reasons why logfs shouldn't go in. The code is clean, feature-complete, and there is demand for it. The main question I can still see is the timing with the merge window. It's almost closed, so if logfs doesn't go in really soon, it should probably wait for the 2.6.27 window.

	Arnd <><

---

This patch fixes some of the problems mentined above.

MAINTAINERS +9 -0
index cae9001..4b45c5b 100644
@@ -2570,6 +2570,15 @@ L:	[email protected]
 W:	http://www.linux-ntfs.org/content/view/19/37/
 S:	Maintained
 
+LOGFS FILE SYSTEM
+P:	Joern Engel
+M:	[email protected]
+L:	[email protected]
+L:	[email protected]
+W:	http://www.logfs.org/
+T:	git://git.kernel.org/pub/scm/linux/kernel/git/joern/logfs.git
+S:	Maintained
+
 LSILOGIC MPT FUSION DRIVERS (FC/SAS/SPI)
 P:	Eric Moore
 M:	[email protected]
fs/logfs/compr.c +1 -1
index 8f01943..44bbfd2 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 #include <linux/vmalloc.h>
fs/logfs/dev_bdev.c +1 -1
index c6c5b5f..65106d7 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 #include <linux/buffer_head.h>
fs/logfs/dev_mtd.c +1 -5
index ea606cf..86e0a6b 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 #include <linux/completion.h>
@@ -354,11 +354,7 @@ static int mtd_get_sb(struct file_system_type *fs_type, int flags,
 	return get_sb_pseudo(fs_type, "mtd:", NULL, 0x6D746400, mnt);
 }
 
-#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 23)
 static void init_once(struct kmem_cache *cache, void *_mi)
-#else
-static void init_once(void *_mi, struct kmem_cache *cache, unsigned long flags)
-#endif
 {
 	struct mtd_inode *mi = _mi;
 
fs/logfs/dir.c +1 -1
index f0cf832..2a5ef8e 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 
fs/logfs/file.c +1 -1
index 17d2cde..105d470 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 #include <linux/sched.h>
fs/logfs/gc.c +1 -1
index 0767345..ee1c448 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  * GC design as it should be (and isn't, as of 15.3.07):
  * 1. Pick a good candidate for GC, constrained by the number of currently
fs/logfs/inode.c +1 -6
index 0d2b9be..d8f7b66 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 #include <linux/writeback.h>
@@ -539,12 +539,7 @@ struct inode *logfs_new_inode(struct inode *dir, int mode)
 	return inode;
 }
 
-#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 23)
 static void logfs_init_once(struct kmem_cache *cachep, void *_li)
-#else
-static void logfs_init_once(void *_li, struct kmem_cache *cachep,
-		unsigned long flags)
-#endif
 {
 	struct logfs_inode *li = _li;
 	int i;
fs/logfs/journal.c +1 -1
index a4cde56..cf07646 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  */
 #include "logfs.h"
 
fs/logfs/logfs.h +2 -1
index 471497f..48622b1 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  * Private header for logfs.
  */
@@ -552,3 +552,4 @@ static inline struct logfs_block *logfs_block(struct page *page)
 {
 	return (void *)page->private;
 }
+#endif /* FS_LOGFS_LOGFS_H */
fs/logfs/logfs_abi.h +1 -1
index 1cfd588..ae8548f 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  * Public header for logfs.
  */
fs/logfs/memtree.c +1 -1
index 68fe7ce..604eea4 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2007 Joern Engel <[email protected]>
+ * Copyright (c) 2007-2008 Joern Engel <[email protected]>
  *
  *
  * This could possibly get moved to lib/.
fs/logfs/readwrite.c +1 -1
index ed1226f..b35c3dc 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  *
  * Actually contains five sets of very similar functions:
fs/logfs/segment.c +1 -1
index 13d4301..77defd2 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  * Object store or ostore makes up the complete device with exception of
  * the superblock and journal areas.  Apart from its own metadata it stores
fs/logfs/super.c +1 -1
index 1f64494..d138a76 100644
@@ -3,7 +3,7 @@
  *
  * As should be obvious for Linux kernel code, license is GPLv2
  *
- * Copyright (c) 2005-2007 Joern Engel <[email protected]>
+ * Copyright (c) 2005-2008 Joern Engel <[email protected]>
  *
  * Generally contains mount/umount code and also serves as a dump area for
  * any functions that don't fit elsewhere and neither justify a file of their
#6 lore raw
In reply to Jörn Engel
Quoted 5 lines · Jörn Engel

not being familiar with either maintaining my own git tree or the -next process, I'd still like to get logfs into mainline. It has gone through six rounds of reviews and the last has been mostly about crossing some i's here and dotting some t's there.

The thing I'd like to see is:

- a more recent description of file system layout

I've read the original paper, and I assume things have changed when implementing stuff. They always do.

- some benchmarks and/or comments about regular usage (ie fragmentation etc). Yeah, it doesn't need to be all that extensive, but quite frankly, it sounds like this is meant to be at least a partial replacement for a GP filesystem (considering that seek/rotational delay are going away) and people are working on it with USB memory sticks etc, wouldn't it make sense to talk about disk usage (how much the GC wants free etc) and everyday performance?

Hmm?

Linus

#20 lore raw
In reply to Jörn Engel
Quoted 3 lines · Jörn Engel

So should it simmer in -next and -mm for another month? Should it go straight into -linus?

The main criteria for it going to Linus should be if you would really trust your data to it now. Would you put your $HOME on it? Merging file systems too early can quickly ruin their name and that taint is hard to ever get rid again then (e.g. happened to JFS)

-Andi

#7 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Jörn Engel

not being familiar with either maintaining my own git tree or the -next process, I'd still like to get logfs into mainline. It has gone through six rounds of reviews and the last has been mostly about crossing some i's here and dotting some t's there.

Quoted 6 lines

The thing I'd like to see is:

- a more recent description of file system layout

I've read the original paper, and I assume things have changed when implementing stuff. They always do.

The big picture has largely stayed the same, but many details haven't. Will do.

Quoted 7 lines

- some benchmarks and/or comments about regular usage (ie fragmentation etc). Yeah, it doesn't need to be all that extensive, but quite frankly, it sounds like this is meant to be at least a partial replacement for a GP filesystem (considering that seek/rotational delay are going away) and people are working on it with USB memory sticks etc, wouldn't it make sense to talk about disk usage (how much the GC wants free etc) and everyday performance?

Currently performance sucks badly on block device flashes (usb stick, etc.) when creating/removing/renaming files. The combination of logfs and the built-in logic can result in 1-2MB of data written to create a single empty file. Yuck!

"Real" block devices or real flash suffer a lot less and writing large amounts of data to existing files doesn't have this problem either.

Fragmentation is neither actively avoided nor actively enforced. If the workload writes files single-threaded, it will initially be fairly good. Over time GC will stir the soup and fragmentation grows. Several parallel writers give a pretty bad result for seek-bound devices, even initially.

GC wants 4095 + 28 bytes per segment (128KiB by default) to deal with not-quite-100% filled segments plus one free segment per level (12 by default, could become an mkfs option). Add the journal and superblock for about 2MiB minimum overhead. Some embedded people with 32MiB devices worry about that, although arguably they should still use jffs2 if minimal space overhead is a big issue.

I guess the above could go into Documentation/filesystems/logfs.txt. And some more.

Jörn

Signature · 4 lines
Fancy algorithms are slow when n is small, and n is usually small.
Fancy algorithms have big constants. Until you know that n is
frequently going to be big, don't get fancy.
-- Rob Pike
#3 lore raw
In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Pekka Enberg

You probably want an ACK from the VFS maintainers before aiming at mainline. But it surely makes sense to ask Andrew to pull it in -mm now.

Definitively wants a re-review with all the bits from last time fixed.

How did the inode_lock abuse get fixed, btw? That one was rather lethal.

That wart is still itching. I thought I'd need a core patch to remove it, but looking at it again, I might get away with a private spinlock.

Will get fixed.

Jörn

Signature · 6 lines
Happiness isn't having what you want, it's wanting what you have.
-- unknown
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw
In reply to Jörn Engel
Quoted 5 lines · Jörn Engel

Currently performance sucks badly on block device flashes (usb stick, etc.) when creating/removing/renaming files. The combination of logfs and the built-in logic can result in 1-2MB of data written to create a single empty file. Yuck!

Can you talk about why, and describe these kinds of things? Is it just because of deep directory trees and having to rebuild the tree from the root up, or is it something else going on?

Quoted 1 line

Fragmentation is neither actively avoided nor actively enforced.

I was more thinking about the fragmentation in terms of how much free space you need for reasonable performance behavior - these kinds of things tend to easily start behaving really badly when the disk fills up and you need to GC all the time just to make room for new erase blocks for the trivial inode mtime/atime updates etc.

Maybe logfs doesn't have that problem for some reason, but in many cases there are rules like "we will consider the filesystem full when it goes over 90% theoretical fill", and it's interesting to know.

Quoted 2 lines

I guess the above could go into Documentation/filesystems/logfs.txt. And some more.

I did try looking at gitweb to see if I could find some documentation file. I didn't find anything.

Linus

#9 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Jörn Engel

Currently performance sucks badly on block device flashes (usb stick, etc.) when creating/removing/renaming files. The combination of logfs and the built-in logic can result in 1-2MB of data written to create a single empty file. Yuck!

Can you talk about why, and describe these kinds of things? Is it just because of deep directory trees and having to rebuild the tree from the root up, or is it something else going on?

Logfs has the concept of "levels". Level 0 contains file data. Level 1 has indirect blocks, level 2 doubly indirect blocks, etc. Inodes are stored in an inode file, which is on level 6 for the inodes, 7 for indirect blocks, etc.

GC requires that data for each level is kept seperate from data for other levels. It is the only make deadlocks impossible, any alternative will just reduce deadlock likelyhood afaiks. Both regular files and the inode file can currently go up to 3x indirect, so you have up to 8 levels open for writing at any given time.

Writing data synchronously requires wandering the entire tree, i.e. writing a block on level 0, then one on level 1, 2 and 3 if indirect blocks are required, write the inode at level 6 and again writing blocks on levels 7, 8 and 9 if the inode number is high. When creating a file, both the dentry and the created inode are written synchronously.

So on a block device level, all this translates to several writes, none of them being adjacent. Each write is fairly small by itself. But the FTL inside your favorite type of consumer flash media will turn any small write into a write of the complete eraseblock. So somewhere on an internal bus, megabytes of data are happily shuffled around.

I have a solution for this, but it would require an incompatible change to the format. And right now I have fairly good confidence in the format wrt. ensuring correctness. So the plan is to merge logfs as-is (modulo bugfixes, review fallout, etc.) and handle the changes for this and other performance problems with compat flags. And someday rename the whole mess to log2fs and remove some support for old format variants.

Quoted 11 lines
Quoted 1 line

Fragmentation is neither actively avoided nor actively enforced.

I was more thinking about the fragmentation in terms of how much free space you need for reasonable performance behavior - these kinds of things tend to easily start behaving really badly when the disk fills up and you need to GC all the time just to make room for new erase blocks for the trivial inode mtime/atime updates etc.

Maybe logfs doesn't have that problem for some reason, but in many cases there are rules like "we will consider the filesystem full when it goes over 90% theoretical fill", and it's interesting to know.

For any flash filesystem there is what I call the "evil workload". Fill the filesystem 100% and randomly replace data. In the best case (jffs2) the filesystem has to GC one segment worth of free space to write one block, then GC another segment for the next block, etc. Non-compressed log-structured filesystems can cheat their way around that by avoiding GC and doing hole-plugging instead. Not sure if any actually do that.

Logfs tries hard to behave the same way on any medium - if only to increase test coverage and fix bugs for everyone when possible. And it has to write indirect blocks, which makes the evil workload worse yet. However, with just 1-2MiB free, it gets back to jffs2 behaviour or at least close, as it can GC a couple of segments and collect dirty indirect blocks, combining updates to those.

So in effect, the question is how fill the filesystem is and how random the write pattern. 50% full and completely random means on average logfs has to GC one segment to write one and performance is down to 50%. 90% full and completely random means GC 9 to write 1, so we're at 10%. 90% full, 85% of which remain untouched means 5% get written and 10% are full, so on average GC 1 to write 2.

I cannot give you a hard limit, but the math is simple enough.

Quoted 2 lines

I did try looking at gitweb to see if I could find some documentation file. I didn't find anything.

Noone else has asked for one before, I believe.

Jörn

Signature · 2 lines
Joern's library part 3:
http://inst.eecs.berkeley.edu/~cs152/fa05/handouts/clark-test.pdf
#21 lore raw
In reply to Andi Kleen
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Jörn Engel

So should it simmer in -next and -mm for another month? Should it go straight into -linus?

The main criteria for it going to Linus should be if you would really trust your data to it now. Would you put your $HOME on it? Merging file systems too early can quickly ruin their name and that taint is hard to ever get rid again then (e.g. happened to JFS)

Right now I don't, mainly because file creat performance is still too bad on the devices I can buy and attach to my notebook. But something like bittorrent would be an excellent testbed where few large files are created and performance should actually be good enough. Time to eat my dogfood.

Jörn

Signature · 2 lines
All art is but imitation of nature.
-- Lucius Annaeus Seneca
#10 lore raw
In reply to Jörn Engel
Quoted 8 lines · Jörn Engel

I have a solution for this, but it would require an incompatible change to the format. And right now I have fairly good confidence in the format wrt. ensuring correctness. So the plan is to merge logfs as-is (modulo bugfixes, review fallout, etc.) and handle the changes for this and other performance problems with compat flags. And someday rename the whole mess to log2fs and remove some support for old format variants.

Quite frankly, if that's the case, I'd *much* rather see that worked on first, so that there aren't any format changes that are already known to be pending before it even gets merged.

Would it be at all possible to try to do that, or is it just "too far out"?

Linus

#5 lore raw
In reply to Arnd Bergmann
Quoted 4 lines · Arnd Bergmann

There are still a few i's and t's left to dot and cross:

* the changeset comments needs a Signed-off-by: line

Doh! When sending patches that happens automatically. I should teach git the same trick.

Quoted 1 line

* The MAINTAINERS file should list your name and logfs mailing list

Definitely.

Quoted 2 lines

* you have a few instances of '#if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 23)', that should go away for the merge

Yes. I would like to keep the merge version roughly in sync with the external patch, at least for a while. Not sure how to deal with one needing the multi-version hackery and the other not wanting it.

Quoted 1 line

* The copyright notice says 2005-2007, it should probably be 2005-2008

On most files, yes.

Quoted 2 lines

* You may want to add a Documentation/filesystems/logfs.txt file explaining the supported mount options.

Sure. I don't have any logfs-specific ones yet, but even that fact should be made explicit.

Quoted 4 lines

* CONFIG_LOGFS should be tristate, not bool. Unfortunately, you are still using three symbols that are not exported: swapper_space (through BUG_ON(!page_mapping(page)->a_ops->set_page_dirty)), add_to_page_cache_lru and inode_lock. Not sure what to do about this.

inode_lock will get fixed. The BUG_ON could get removed. Not sure about add_to_page_cache_lru yet.

Quoted 2 lines

* You should really make sure the version you check in compiles, fs/logfs/logfs.h is missing an #endif. ;-)

Making a merge dash whilst moving is not always the wisest choice. ;)

Quoted 5 lines

Otherwise, I don't see any reasons why logfs shouldn't go in. The code is clean, feature-complete, and there is demand for it. The main question I can still see is the timing with the merge window. It's almost closed, so if logfs doesn't go in really soon, it should probably wait for the 2.6.27 window.

2.6.27 appears to make more sense, yes.

Quoted 1 line

This patch fixes some of the problems mentined above.

Excellent! Thanks a bunch!

Quoted 1 line

+L: [email protected]

I believe it is currently subscribers-only with the usual bounces everyone holds so dear. I should change that and add a spam filter to make it bearable.

Jörn

Signature · 2 lines
The only real mistake is the one from which we learn nothing.
-- John Powell
#11 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

Quite frankly, if that's the case, I'd *much* rather see that worked on first, so that there aren't any format changes that are already known to be pending before it even gets merged.

Would it be at all possible to try to do that, or is it just "too far out"?

Definitely possible. The last similar change happened in December and took until March until I ran out of stupid regressions from it. Most likely there are still some I just haven't found yet.

The question is when to draw the line and say "This is useful as-is for a sufficient number of users." I don't have a good answer to it. I certainly expect more changes in the future, including format changes. And if we wait for them all to happen, it won't get merged this decade.

Not sure.

Jörn

Signature · 2 lines
The only good bug is a dead bug.
-- Starship Troopers
#12 lore raw
Quoted 9 lines · Jörn Engel
Quoted 7 lines · Linus Torvalds

Quite frankly, if that's the case, I'd *much* rather see that worked on first, so that there aren't any format changes that are already known to be pending before it even gets merged.

Would it be at all possible to try to do that, or is it just "too far out"?

Quoted 8 lines

Definitely possible. The last similar change happened in December and took until March until I ran out of stupid regressions from it. Most likely there are still some I just haven't found yet.

The question is when to draw the line and say "This is useful as-is for a sufficient number of users." I don't have a good answer to it. I certainly expect more changes in the future, including format changes. And if we wait for them all to happen, it won't get merged this decade.

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

Willy

#13 lore raw
Quoted 14 lines · Willy Tarreau
Quoted 13 lines · Jörn Engel
Quoted 7 lines · Linus Torvalds

Quite frankly, if that's the case, I'd *much* rather see that worked on first, so that there aren't any format changes that are already known to be pending before it even gets merged.

Would it be at all possible to try to do that, or is it just "too far out"?

Definitely possible. The last similar change happened in December and took until March until I ran out of stupid regressions from it. Most likely there are still some I just haven't found yet.

Quoted 7 lines
Quoted 4 lines

The question is when to draw the line and say "This is useful as-is for a sufficient number of users." I don't have a good answer to it. I certainly expect more changes in the future, including format changes. And if we wait for them all to happen, it won't get merged this decade.

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

Andi already answered that one:
"Merging file systems too early can quickly ruin their name and that
taint is hard to ever get rid again then (e.g. happened to JFS)"

And a stable kernel shouldn't be something for getting "more testers", it should be for tested code ready to be used in production. What you call "more testers" would be people who try it in production (e.g. to overcome shortcomings of JFFS2) thinking it was stable.

And no, EXPERIMENTAL in the kernel is not usable for keeping people from trying known-whacky code.

Quoted 1 line

Willy

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#14 lore raw
Quoted 23 lines · Adrian Bunk
Quoted 14 lines · Willy Tarreau
Quoted 13 lines · Jörn Engel
Quoted 7 lines · Linus Torvalds

Quite frankly, if that's the case, I'd *much* rather see that worked on first, so that there aren't any format changes that are already known to be pending before it even gets merged.

Would it be at all possible to try to do that, or is it just "too far out"?

Definitely possible. The last similar change happened in December and took until March until I ran out of stupid regressions from it. Most likely there are still some I just haven't found yet.

Quoted 7 lines
Quoted 4 lines

The question is when to draw the line and say "This is useful as-is for a sufficient number of users." I don't have a good answer to it. I certainly expect more changes in the future, including format changes. And if we wait for them all to happen, it won't get merged this decade.

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

Quoted 7 lines

Andi already answered that one:
"Merging file systems too early can quickly ruin their name and that
taint is hard to ever get rid again then (e.g. happened to JFS)"

And a stable kernel shouldn't be something for getting "more testers", it should be for tested code ready to be used in production.

OK, -mm is generally more suited for this usage then.

Quoted 2 lines

What you call "more testers" would be people who try it in production (e.g. to overcome shortcomings of JFFS2) thinking it was stable.

No, it is just to gain more exposure by easing tester's job. People packaging distros for embedded systems do a lot of R&D, and having new features to experiment with is very important to them. And no, that does not mean they'll immediately use it in production. And even if some did, they would know why they did it and it's their problem.

Willy

#15 lore raw
Quoted 9 lines · Willy Tarreau

On Sat, May 03, 2008 at 12:11:11PM +0300, Adrian Bunk wrote:
...

Quoted 2 lines

What you call "more testers" would be people who try it in production (e.g. to overcome shortcomings of JFFS2) thinking it was stable.

No, it is just to gain more exposure by easing tester's job. People packaging distros for embedded systems do a lot of R&D, and having new features to experiment with is very important to them. And no, that does not mean they'll immediately use it in production.

If it's in the kernel it will end in distribution kernels.

And people will then use it.

You might be right, they might not immediately use it in production. They might use the current version one year later in the then one year old kernel they will then be using. Or the one year old version plus some random set of vendor patches.

Quoted 3 lines

And
even if some did, they would know why they did it and it's their problem.

You want to put experimental code into stable kernels and then blame people using it that they shouldn't have used it?

Quoted 1 line

Willy

cu
Adrian

BTW: This is not meant against the LogFS merge.

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#16 lore raw
In reply to Adrian Bunk

Hi Adrian,

Quoted 8 lines · Adrian Bunk
Quoted 2 lines

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

Andi already answered that one:

"Merging file systems too early can quickly ruin their name and that
taint is hard to ever get rid again then (e.g. happened to JFS)"

Quoted 7 lines

And a stable kernel shouldn't be something for getting "more testers", it should be for tested code ready to be used in production. What you call "more testers" would be people who try it in production (e.g. to overcome shortcomings of JFFS2) thinking it was stable.

And no, EXPERIMENTAL in the kernel is not usable for keeping people from trying known-whacky code.

I think ext4 already set the precedent that you _can_ do development within the 2.6 series, no?

#17 lore raw
Quoted 16 lines · Pekka Enberg

Hi Adrian,

Quoted 13 lines · Adrian Bunk
Quoted 2 lines

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

Andi already answered that one:

"Merging file systems too early can quickly ruin their name and that
taint is hard to ever get rid again then (e.g. happened to JFS)"

And a stable kernel shouldn't be something for getting "more testers", it should be for tested code ready to be used in production. What you call "more testers" would be people who try it in production (e.g. to overcome shortcomings of JFFS2) thinking it was stable.

Quoted 5 lines
Quoted 2 lines

And no, EXPERIMENTAL in the kernel is not usable for keeping people from trying known-whacky code.

I think ext4 already set the precedent that you _can_ do development within the 2.6 series, no?

I'd call the ext4 case a mistake we shouldn't repeat.

It's available in the kernel since 2006.

I've seen people using ext4 on their computers running with a corrupted filesystem since fsck was at that point not yet capable of fixing whatever was corrupted.

At least one distribution already has ext4 enabled in their kernels.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#18 lore raw
In reply to Willy Tarreau
Quoted 3 lines · Willy Tarreau

Why not merge it and mark it experimental then ? In fact, this is about what you're looking for : reduced merge hassle and more testers.

The real issue for me wrt a filesystem is the on-disk layout.

If we know that on-disk structures need change, we shouldn't merge it. It doesn't matter if that can be worked around with some backwards- compatibiltiy flag: we should simply not encourage that kind of behaviour. It would be much much better to just get a layout that is as final as possible and avoid the "there are two different formats, because the first format was known to be broken" issue.

Will extensions happen and add features anyway? Probably. But that's different from merging something knowing that the on-disk format will change.

Linus

#19 lore raw

Offtopic to: LogFS merge

Quoted 3 lines · Linus Torvalds

The real issue for me wrt a filesystem is the on-disk layout.

If we know that on-disk structures need change, we shouldn't merge it. It doesn't matter if that can be worked around with some backwards- compatibiltiy flag: we should simply not encourage that kind of behaviour.

I agree in particular, but not in principle (;-))

Changing the filesystem format was something that happened at
least twice on Multics, on production machines. I happened to be on during one of the changes and didn't even know it was happening until there was a broadcast message warning of poor performance.

I always thought that was cool, and got permission recently to
post a colleague's paper on it at http://www.multicians.org/stachour.html

It would be cool if data could change at run-time on Linux, just
like security-sensitive code.

--dave

Signature · 5 lines
David Collier-Brown            | Always do right. This will gratify
Sun Microsystems, Toronto      | some people and astonish the rest
[email protected]                 |                      -- Mark Twain
(905) 943-1983, cell: (647) 833-9377, (800) 555-9786 x56583
bridge: (877) 385-4099 code: 506 9191#
#22 lore raw
In reply to Jörn Engel
Quoted 1 line · On Fri, 2 May 2008 15:32:34 +0200 J__rn Engel <[email protected]> wrote:

master.kernel.org:/pub/scm/linux/kernel/git/joern/logfs.git/

I added this to the -mm pile.

Thank you for not putting your Makefile and Kconfig changes right at the end of the file like everyone else always does. It actually merges.