OMFS filesystem version 3
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · On Mon, 14 Apr 2008 Anton Altaparmakov wrote:
Quoted 8 lines · Andrew Morton
Quoted 3 lines · On Mon, 14 Apr 2008 09:25:46 +0200 Miklos Szeredi <[email protected]> wrote:
And I didn't advocate moving
ntfs to fuse, still that was done and the resulting filesystem at the moment happens to outperform the kernel one in every respect ;)Gad. Why?
Well, not only the kernel ntfs (in metadata operations) but also XFS, JFS, ext2 in metadata operations, ext3 in large bulk writes on commodity hardware.
It's also important to note that ntfs-3g isn't optimized yet at all. There are several major performance bootlenecks still in the kernel, FUSE and ntfs-3g itself.
Answering the why, the issue is almost never the context switch but the file system design and the quality of the implementation. While the latter still needs a lot of work on several areas (the size of the Microsoft NTFS driver is almost like all the in-kernel file system altogether) but the design can very nicely compensate the latter. E.g. ntfs (and ntfs-3g) already supports almost all the things what ext4 will include.
Quoted 1 line
No-one has "moved" ntfs to fuse.
Yes, historically the user space ntfs code was moved to the kernel. The kernel always lagged somehow behind when this work was stopped a few years ago.
Quoted 7 lines
However the kernel one as available in the kernel source tree doesn't have many write-features, it can only overwrite files, it cannot create/delete files, etc. So I guess if you define "performance" to mean "features" then sure ntfsmount/ntfs-3g have more features than the public kernel driver. If you define "performance" to mean "speed" then no ntfsmount/ntfs-3g can't compare to the kernel except in very limited and meaningless benchmarks...
First comes reliability for ntfs-3g. There are just too many million users. This is non-negotiable. Because of this, ntfs-3g is indeed not optimized yet.
However the bootlenecks are almost always storage related (seeks, bandwidth, buses) on commodity desktop systems. Embedded devices and servers come a bit later with the CPU usage optimizations.
Quoted 2 lines
btw. even comparing features, the fuse solutions lag behind in some respects, e.g. no-one can "kill -9" the kernel driver
I consider this as a benefit for FUSE file systems. An unloadable kernel module results reboot which is much more intrusive.
Quoted 1 line
leaving a corrupt file system on the volume
That would be a file system driver implementation bug. What I never denied for ntfs-3g. Surely it's solvable but this was never a priority based on user wishes.
Quoted 1 line
(and under no-one I include the OOM killer for example!)
The OOM killer can be configured and if the fs still uses too much memory then probably it's better to be killed/restarted with journaling support. The important here would be the kernel finally fixing the non-sync behavior when it clams to do so (see recent kernel threads).
On the other hand, when in-kernel drivers leak then you get a system panic.
Quoted 2 lines
and another example is that the fuse solutions require large amounts of ram whereas the kernel driver can happily function in 1MiB ram
A FUSE driver can also happily function in much less than 1 MB RAM.
Quoted 2 lines
when they find a large directory and the user has only 32MiB ram for example...
This is a FUSE fs implementation, not FUSE problem.
Szaka
--
NTFS-3G: http://ntfs-3g.org
Hi,
Quoted 28 lines · Szabolcs Szakacsits
Quoted 10 lines · On Mon, 14 Apr 2008 Anton Altaparmakov wrote:
Quoted 9 lines · Andrew Morton
On Mon, 14 Apr 2008 09:25:46 +0200 Miklos Szeredi <[email protected]
Quoted 1 line
Quoted 3 lines · wrote:
And I didn't advocate moving
ntfs to fuse, still that was done and the resulting filesystem at the moment happens to outperform the kernel one in every respect ;)Gad. Why?
Well, not only the kernel ntfs (in metadata operations) but also XFS, JFS, ext2 in metadata operations, ext3 in large bulk writes on commodity hardware.
It's also important to note that ntfs-3g isn't optimized yet at all. There are several major performance bootlenecks still in the kernel, FUSE and ntfs-3g itself.
Answering the why, the issue is almost never the context switch but the file system design and the quality of the implementation. While the latter still needs a lot of work on several areas (the size of the Microsoft NTFS driver is almost like all the in-kernel file system altogether) but the design can very nicely compensate the latter. E.g. ntfs (and ntfs-3g) already supports almost all the things what ext4 will include.
Quoted 5 lines
Quoted 1 line
No-one has "moved" ntfs to fuse.
Yes, historically the user space ntfs code was moved to the kernel. The kernel always lagged somehow behind when this work was stopped a few years ago.
Yes. It was much easier for me to write the ntfs code for user space in ntfsprogs/libntfs and then once I had the ntfs side of it working reliably port it to the kernel so I then only had to deal with kernel interface/threading/etc issues but at least the NTFS code would be there. Over time this changed for me and I started working primarily on the kernel and pushing things back into libntfs and as these things go I eventually was so busy that I no longer had the time to do any back porting to libntfs and since then I have only been working on the kernel driver. Unfortunately I haven't been allowed to release my work yet as it is used in a closed source production environment but hopefully one day I will be allowed to release it and update the kernel driver with my code which will move it up to being a fully function read-write driver. Or if I never end up being able to do that then things are not too bad either as my MacOSX NTFS driver (Leopard already has the read-only driver in it) will eventually be released and that is BSD/GPL dual licensed so anyone will be able to take the source and port it to Linux thus cutting me out of the loop...
Quoted 16 lines
Quoted 7 lines
However the kernel one as available in the kernel source tree doesn't have many write-features, it can only overwrite files, it cannot create/delete files, etc. So I guess if you define "performance" to mean "features" then sure ntfsmount/ntfs-3g have more features than the public kernel driver. If you define "performance" to mean "speed" then no ntfsmount/ntfs-3g can't compare to the kernel except in very limited and meaningless benchmarks...
First comes reliability for ntfs-3g. There are just too many million users. This is non-negotiable. Because of this, ntfs-3g is indeed not optimized yet.
However the bootlenecks are almost always storage related (seeks, bandwidth, buses) on commodity desktop systems. Embedded devices and servers come a bit later with the CPU usage optimizations.
Quoted 5 lines
Quoted 2 lines
btw. even comparing features, the fuse solutions lag behind in some respects, e.g. no-one can "kill -9" the kernel driver
I consider this as a benefit for FUSE file systems. An unloadable kernel module results reboot which is much more intrusive.
Kernel modules don't become "unloadable" unless there is a bug. The "kill -9" can happen inadvertently even without any bugs in the FUSE or the FUSE-fs. And that IMO is unacceptable.
Quoted 6 lines
Quoted 1 line
leaving a corrupt file system on the volume
That would be a file system driver implementation bug. What I never denied for ntfs-3g. Surely it's solvable but this was never a priority based on user wishes.
Quoted 6 lines
Quoted 1 line
(and under no-one I include the OOM killer for example!)
The OOM killer can be configured and if the fs still uses too much memory then probably it's better to be killed/restarted with journaling support. The important here would be the kernel finally fixing the non-sync behavior when it clams to do so (see recent kernel threads).
You don't get the point. Any process in the system can be using too much memory and trigger the OOM killer even when the FS is behaving just fine...
Quoted 2 lines
On the other hand, when in-kernel drivers leak then you get a system panic.
So don't leak then! (-;
Quoted 4 lines
Quoted 2 lines
and another example is that the fuse solutions require large amounts of ram whereas the kernel driver can happily function in 1MiB ram
A FUSE driver can also happily function in much less than 1 MB RAM.
Obviously it can! We use a FUSE based file system I wrote here at the University on about 850 machines and it doesn't use any RAM at all (well, ok a few kiB but that is hardly worth talking about!).
Quoted 4 lines
Quoted 2 lines
when they find a large directory and the user has only 32MiB ram for example...
This is a FUSE fs implementation, not FUSE problem.
I never said it was a FUSE problem! It is a ntfsmount/ntfs-3g problem. At least a few years ago someone was trying to use ntfsmount (or ntfs-3g I can't remember if you had already forked it then) on a 32MiB RAM embedded ARM box and he was running OOM when trying to list directories due to the ntfs/fuse implementation. In the kernel ntfs driver that does not happen.
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer, http://www.linux-ntfs.org/
Quoted 4 lines · Anton Altaparmakov
Quoted 2 lines
I consider this as a benefit for FUSE file systems. An unloadable kernel module results reboot which is much more intrusive.
Kernel modules don't become "unloadable" unless there is a bug.
That's exactly what I meant. The majority of the system crashes are due to kernel drivers.
Quoted 2 lines
The "kill -9" can happen inadvertently even without any bugs in the FUSE or the FUSE-fs.
Not really. And if so then distros solve it, as some of them already did (e.g. during system shutdown).
Quoted 7 lines
Quoted 4 lines
The OOM killer can be configured and if the fs still uses too much memory then probably it's better to be killed/restarted with journaling support. The important here would be the kernel finally fixing the non-sync behavior when it clams to do so (see recent kernel threads).
You don't get the point. Any process in the system can be using too much memory and trigger the OOM killer even when the FS is behaving just fine...
Actually you missed when I wrote "the OOM killer can be configured".
FUSE is a new thing which sometimes requires non-conventional thinking and minor adjustments here and there. These works are ongoing for some years now.
Quoted 5 lines
I never said it was a FUSE problem! It is a ntfsmount/ntfs-3g problem. At least a few years ago someone was trying to use ntfsmount (or ntfs-3g I can't remember if you had already forked it then) on a 32MiB RAM embedded ARM box and he was running OOM when trying to list directories due to the ntfs/fuse implementation. In the kernel ntfs driver that does not happen.
Listing a directory with over 100k files can be still an ENOMEM problem using 32 MB RAM but of course it's solvable. Nobody was interested so far.
Szaka
--
NTFS-3G: http://ntfs-3g.org