Porting Zfs features to ext2/3

Hello

I want to know that has any work been done to port the Zfs features to ext2/3

View this message in context: http://www.nabble.com/Porting-Zfs-features-to-ext2-3-tp18674437p18674437.html
Sent from the linux-ext4 mailing list archive at Nabble.com.

There are some new features in ext4, but the primary goal of ext4 development is to improve the filesystem by adding features such as extents, delayed allocation, etc., in an evolutionary way, so that an existing ext3 filesystem could be upgraded to ext4. So the primary goal is not to try to complete with the ZFS features set.

The btrfs filesystem effort is an attempt to create a filesystem that will leapfrog the ZFS feature set, but it will probably take longer to reach production ready status than ext4.

Regards,

- Ted

Hi Ted

Since you mention btrfs here and since I've read this earlier too, do you know if btrfs will be the default Linux file system in the future, like extX has been?

The nature of Linux is such that these sorts of decisions are not made by anyone other than each individual system administrator, and by the distributions who choose which filesystem they which to use as the "default". There is no such thing as an "official" default filesystem. For example, the Maemo distribution, which Nokia distributes for use on the N800/N810 devices, uses jffs2 as its default filesystem, since those devices use a flash storage device. DragonLinux, which is designed to be installed on top of DOS/Windows uses UMSDOS as its default distribution.

What happens in the future, who can say? At some point the ext2/3/4 filesystem, which is based fundamentally on a BSD Fast Filesystem design base, may get displaced by a filesystem which uses some very different design as a starting point, when the advantages of starting with that different design outweighs the advantages of backwards compatibility and broad base of support which is enjoyed by ext2/3/4.

To give one example from the past, filesystems like JFS were theoretically better than ext3 at the time, but unfortunately all of the expertise was concentrated in one company (IBM), and so distributions were slow to accept it. In the meantime, ext3 was able to add enough features (htree directories, better SMP scalability) to eventually meet and then surpass JFS's technical advantages.

XFS has a number of technical advantages over ext3, but the number of people who understand it are small, and people seem to like the tools built for ext3 --- and now ext4 has a number of features that were previously exclusive to XFS. XFS is still the best filesystem for very large, SGI-class machines, however. But for general purpose computing, most people are more comfortable with ext3.

Yet the fact that we are retaining backwards compatibility with ext3 does constraint our ability to add radical new features. So eventually some filesystem will probably overtake ext2/3/4. Will that btrfs? I don't think anyone can answer that question. I *have* been helping out the btrfs design team, though, giving them advice such as making sure that they try to gather contributors from a wide variety of distributions and other Linux companies. So I hope they do become successful. In the meantime, though, ext4 is a great extension to the ext2/3 filesystem family. But in the long run, it may very well be that btrfs will be more successful than some future attempt to create an ext5; and that's fine.

Regards,

- Ted

Thanks. That sums up the trade-off pretty clearly.

-Shehjar

Did you know that ZFS is available for Linux?

Cheers,
Eric

ZFS is available in a FUSE filesystem. As a userspace filesystem, it means a huge number of context switches to get data between the disk, to the kernel, to the FUSE userspace, back to the kernel, and to the process trying to access the ZFS file. That's not going to be high performance. For someone who wants to migrate from Solaris to Linux, it might be useful, but I'm not sure you would really want to use a ZFS/FUSE implementation in production.

- Ted

It's true that ZFS on FUSE performance isn't all it could be right now. However, ZFS on FUSE is currently not taking advantage of mechanisms FUSE provides to improve performance. For an example of what can be achieved, check out http://www.ntfs-3g.org/performance.html .

FWIW, I am satisfied with its performance for backups of my home directory and for my fileserver, which is limited by a 100Mbps connection to the rest of the network. I do not recommend it as a root filesystem yet.

Cheers,
Eric

Yes... and take a look at the metadata operations numbers. FUSE can do things to accellerate bulk read/write, but metadata-intensive operations will (I suspect) always be slow. I also question whether the FUSE implementation will have the safety that has always been the Raison d'être of ZFS. Have you or the ZFS/FUSE developers done tests where you are writing to the filesystem, and then someone pulls the plug on the fileserver while ZFS is writing? Does the filesystem recovery cleanly from such a scenario?

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Please let me know if I'm getting off topic for the ext4-devel list. My point is not to advocate ZFS over ext3/4 since ZFS still has its share of issues. No resizing raidz vdevs, for example, and performance in certain areas. My only point is to make it clear that ZFS on Linux is available (and not necessarily a bad choice) to people reading the ext4-devel mailing list looking for ZFS-like features like the original poster.

It doesn't seem too much worse than the other non-ext3 filesystems in the comparison. I'm sure everyone would prefer a non-FUSE implementation and the licensing issues aren't going to go away, but this post on Jeff Bonwick's blog gives some hope:
http://blogs.sun.com/bonwick/entry/casablanca . Even so, not everyone needs a whole lot of speed in the metadata operations area.

I haven't personally tried pulling the plug, but I've tried holding down the power button on my laptop until it powers off. Everything works fine and scrubs (the closest ZFS gets to fsck) don't report any checksum errors. The filesystem driver updates the on-disk filesystem atomically every five seconds (less time in special circumstances) so there's never any point at which the filesystem would need recovery. The next time the filesystem is mounted the system sees the state the filesystem was in up to five seconds before the power went out. The FUSEness of the filesystem driver doesn't seem to affect this.

Cheers,
Eric

Does that mean you always lose the last 5 seconds of data before the power outage?

We had an earlier thread where Chris had a good test for making a case for the write barrier code being enabled by default. It would be neat to try that on ZFS ;-) The expected behaviour should be that any fsync()'ed files should be there (regardless of the 5 seconds) and other non-fsync'ed files might or might not be there, but that all file system integrity is complete.

It would also be very interesting to try and do a drive hot pull.

Thanks!

Ric

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

> The expected behaviour should be that any fsync()'ed

fsync()s are one of the triggers that cause the filesystem driver to update the disk more than once every five seconds. This can also happen when the filesystem is very full and someone is writing to the disk.

To my knowledge, write barriers are not available to FUSE filesystems. Also, the maintainer of ZFS FUSE discovered that calling fsync() on an open device file just plain does not work on Linux, so currently the driver requests write cache flushes directly from the hardware.

I've got some vacation coming up...sounds like fun! Unless someone specifically asks on here, I'll post the results in the ZFS on FUSE mailing list.

Cheers,
Eric

Those are old numbers of the unoptimized ntfs-3g driver, which could be at least 3-30 times better.

- create: until recently, when ext3 defaulted htree, the unoptimized ntfs-3g was 2-4x faster. But nobody seems to really care because it's not a real-world benchmark (creation of 0 byte size files).

- lookup: by enabling FUSE entry cache the performance will be exactly the same (no user space involvement), or the bottleneck will be the disk seek time and how an fs optimizes for it.

FUSE can also cache attributes, positive/negative lookups, file data and hopefully the new performance improving features, infrastructure being worked on will be added in the future too.

Basically FUSE file systems can be considered as in-kernel network file systems where the network latency is a context switch. Yes, things need to be done a bit differently sometimes but achieving high-performance even for metadata operations is not impossible.

This is an implementation detail, irrelevant to FUSE.

Regards, Szaka

--
NTFS-3G: http://ntfs-3g.org

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Transferring 4 kB from a commodity disk (disk seek + rotational delay + transfer) takes usually between 100-4,000 usec. Two context switches take about 2 usec, i.e. only 2-0.05% of the full data transfer time.

In other words, there isn't really time for huge number of context switches because most of the time is spent waiting for the disk.

I did also an in memory test on a [email protected], with disk I/O completely eliminated. Results:

tmpfs:    975 MB/sec
ntfs-3g:  889 MB/sec  (note, this FUSE driver is not optimized yet)
ext3:     675 MB/sec

It seems the problem is that the only active ZFS-FUSE developer was hired by SUN and since then not much (visible) is happening.

Regards, Szaka

--
NTFS-3G: http://ntfs-3g.org

Again, I agree that you can optimize bulk data transfer. It'll be metadata operations where I'm really not convinced FUSE will be able to be acceptable for many workloads. If you are doing a sequential I/O in huge chunks, sure you can amortize the overhead of the userspace context switches.

The test you did above looks bad because ext3 does lots of small I/O to the loop device. The CPU overhead is not a big deal for real disks, but when you do a pure memory test, it definitely becomes an issue. Try doing an in-memory test with ext2, and you'll see much better results, much closer to tmpfs. The reason? Blktrace tells the tale. Ext2 looks like this:

254,4    1        1     0.000000000 23109  Q   W 180224 + 96 [pdflush]
254,4    1        2     0.000030032 23109  Q   W 180320 + 8 [pdflush]
254,4    1        3     0.000328538 23109  Q   W 180328 + 1024 [pdflush]
254,4    1        4     0.000628162 23109  Q   W 181352 + 1024 [pdflush]
254,4    1        5     0.000925550 23109  Q   W 182376 + 1024 [pdflush]
254,4    1        6     0.001317715 23109  Q   W 183400 + 1024 [pdflush]
254,4    1        7     0.001619783 23109  Q   W 184424 + 1024 [pdflush]
254,4    1        8     0.001913400 23109  Q   W 185448 + 1024 [pdflush]
254,4    1        9     0.002206738 23109  Q   W 186472 + 1024 [pdflush]

Ext3 looks like this:

254,4    0        1     0.000000000 23109  Q   W 131072 + 8 [pdflush]
254,4    0        2     0.000040578 23109  Q   W 131080 + 8 [pdflush]
254,4    0        3     0.000059575 23109  Q   W 131088 + 8 [pdflush]
254,4    0        4     0.000076617 23109  Q   W 131096 + 8 [pdflush]
254,4    0        5     0.000093728 23109  Q   W 131104 + 8 [pdflush]
254,4    0        6     0.000110211 23109  Q   W 131112 + 8 [pdflush]
254,4    0        7     0.000127253 23109  Q   W 131120 + 8 [pdflush]
254,4    0        8     0.000143735 23109  Q   W 131128 + 8 [pdflush]

So it's issueing lots of 4k writes, one page at a time, because it needs to track the completion of each block. This creates a significant CPU overhead, which dominates in an all-memory test. Although this is not an issue in real-life today, it will likely become an issue in real-life solid state disks (SSD's).

Fortunately, ext4's blktrace when copying a large file looks like this:

254,4    1        1     0.000000000 24574  Q   R 648 + 8 [cp]
254,4    1        2     0.000059855 24574  U   N [cp] 0
254,4    0        1     0.000427435     0  C   R 648 + 8 [0]
254,4    1        3     0.385530672 24313  Q   R 520 + 8 [pdflush]
254,4    1        4     0.385558400 24313  U   N [pdflush] 0
254,4    1        5     0.385969143     0  C   R 520 + 8 [0]
254,4    1        6     0.387101706 24313  Q   W 114688 + 1024 [pdflush]
254,4    1        7     0.387269327 24313  Q   W 115712 + 1024 [pdflush]
254,4    1        8     0.387434854 24313  Q   W 116736 + 1024 [pdflush]
254,4    1        9     0.387598425 24313  Q   W 117760 + 1024 [pdflush]
254,4    1       10     0.387831698 24313  Q   W 118784 + 1024 [pdflush]
254,4    1       11     0.387996037 24313  Q   W 119808 + 1024 [pdflush]
254,4    1       12     0.388162890 24313  Q   W 120832 + 1024 [pdflush]
254,4    1       13     0.388325204 24313  Q   W 121856 + 1024 [pdflush]

*Much* better. :-)

- Ted

...

This already is a major issue for us. We are starting to use SAS raid boxes that deliver >350MB/s write and >600MB/s read performance with lustre, which is ext3 with patches. It is somewhat between ext3 and ext4 in that it has some of its features but not all.

That is promising. Once the 64BIT feature becomes usable we plan to port lustre to use ext4 as base filesystem. The current 8TiB limit is a real pain.

MfG
Goswin

Am I write in guessing that this test involved copying a single large file, with no seeks? What happens if you try benchmarking unpacking a kernel source tar.bz2 file? My guess is that ntfs-3g won't look as good. :-)

- Ted

Yes, it was writing a large file on a newly created, loop mounted image file.

It seems the tar.bz2 number is not so bad relatively but the metadata performance difference is much more visible by eliminating the compression overhead. The results are in second.

ext3 ntfs-3g

  tar.bz2      7.7      12.4
  tar.gz       3.1       8.7
  tar          1.4       7.6

A few seconds could be improved but I think getting similar numbers will need major FUSE changes and non-trivial work to get rid of most contexts switches which indeed seem to be the bottleneck according to the profiling data.

Szaka

--
NTFS-3G: http://ntfs-3g.org

Sorry, I didn't use the currently best performing ntfs-3g version. Corrected results:

               ext3    ntfs-3g
               ----    -------
   tar.bz2      7.7      9.9
   tar.gz       3.1      6.0
   tar          1.4      4.9

Compilation of e2fsprogs:

                 ext3     ntfs-3g
                 ----     -------
   unpack        0.27       0.47
   configure     8.42       9.73
   make         21.35      23.33
   make -j      13.21      14.09

make clean 0.20 0.24

Please not that ntfs-3g and fuse is not yet optimized for metadata operations.

Szaka

--
NTFS-3G: http://ntfs-3g.org

In most situations the limiting factor is the users harddisk. Esspecially for metadata the seek time is the real limiting factor. In the case of ZFS there is another big factor though, the checksumming. A kernel implementation could use the async crypto engine and take advantage of hardware checksumming.

To truely compete with a kernel ZFS there would have to be an interface for passing data to the crypto engine from userspace. Combine that with splice or some other zero copy solution and the performance will be mostly the same.

On the other hand never forget how complex the filesystem code is in the kernel. One of the biggest advantages of fuse is the simplicity of the code. As such it is much easier to get the codebase stable.

MfG
Goswin