Skip to contentSkip to conversation

Filesystem for mobile hard drive

DISCUSSION Nicolas George 4 participants 10 messages Feb 12, 2006 – Feb 13, 2006

lore
Share & tools
lore
#0 lore raw

Hi.

I am about to buy a mobile hard drive (actually, a FireWire/USB box and a normal hard drive), and it raises the question of which filesystem to put on it. I am not wondering which of ext3, reiserfs, XFS or JFS is best, but more basically whether I should use a Linux/Unix-style filesystem or the horrible FAT.

The drawbacks of FAT are numerous and well-known: poor efficiency with big files, fragmentation, bad handling of file names, lack of robustness, and worst of all, the 4 Go limit. On the other hand, FAT gives the possibility to easyly read the drive on non-Unix systems (I know there are ext2 and reiserfs readers for windows, I do not know for XFS or JFS, but at the worst it should be possible to do something with colinux).

All these elements are rather feeble, but the Unix-style filesystems have a big drawback as mobile filesystems: they store UIDs. UIDs make sense inside a given system, but not across systems. On the most annoying case, I can have my disk automatically mounted on a system where I am not root, and all my files unreadable because they belong to another user.

Since big mobile mass-storage devices which require efficient filesystems will become more and more common, I think this problem should be addressed. Someone suggested to me to use some sort of network filesystem (NFS or SMB), and its UID mapping facility. That should work, but that is rather an ugly solution, and that is not something that can be done in five minutes while visiting a friend.

I believe that we lack an option at the VFS to completely override file ownership of a filesystem. But maybe there are other solutions.

Did someone already think in depths about this issue?

Regards,

Signature · 1 line
  Nicolas George
#1 lore raw
Quoted 8 lines · Nicolas George

Hi.

I am about to buy a mobile hard drive (actually, a FireWire/USB box and a normal hard drive), and it raises the question of which filesystem to put on it. I am not wondering which of ext3, reiserfs, XFS or JFS is best, but more basically whether I should use a Linux/Unix-style filesystem or the horrible FAT.

Quoted 7 lines

The drawbacks of FAT are numerous and well-known: poor efficiency with big files, fragmentation, bad handling of file names, lack of robustness, and worst of all, the 4 Go limit. On the other hand, FAT gives the possibility to easyly read the drive on non-Unix systems (I know there are ext2 and reiserfs readers for windows, I do not know for XFS or JFS, but at the worst it should be possible to do something with colinux).

If by FAT you mean FAT16, then yes, you have an 8 GB limit for the entire filesystem. Fat32 on the other hand, can handle much more and so should be suitable in this aspect. Fragmentation is also a property not of the filesystem, but of Microsoft's filesystem drivers. I'm fairly sure that the linux fat implementations do not use absurdly stupid allocation algorithms that lead to lots of fragmentation.

Quoted 5 lines

All these elements are rather feeble, but the Unix-style filesystems have a big drawback as mobile filesystems: they store UIDs. UIDs make sense inside a given system, but not across systems. On the most annoying case, I can have my disk automatically mounted on a system where I am not root, and all my files unreadable because they belong to another user.

This can be overcome with the UDF filesystem by using the uid and gid mount options, allowing the files to appear to be owned by the correct local user. It would be nice if the other filesystems were patched to allow such options as well.

Quoted 8 lines

Since big mobile mass-storage devices which require efficient filesystems will become more and more common, I think this problem should be addressed. Someone suggested to me to use some sort of network filesystem (NFS or SMB), and its UID mapping facility. That should work, but that is rather an ugly solution, and that is not something that can be done in five minutes while visiting a friend.

Network filesystems are not on disk filesystems, so they have nothing to do with this discussion; you can't format a disk as "nfs" or "smb".

Quoted 2 lines

I believe that we lack an option at the VFS to completely override file ownership of a filesystem. But maybe there are other solutions.

I agree. I recently submitted a patch fixing a bug in the udf filesystem so it correctly uses the uid and gid mount options to translate the ownership of files owned on disc by id -1 to the values you specify. It would be nice if you could also specify that should be done for all files, not just those owned by nobody, and if all filesystems allowed you to do this.

Quoted 6 lines

Did someone already think in depths about this issue?

Regards,

#2 lore raw

Le quartidi 24 pluviôse, an CCXIV, Phillip Susi a écrit :

Quoted 3 lines

If by FAT you mean FAT16, then yes, you have an 8 GB limit for the entire filesystem. Fat32 on the other hand, can handle much more and so should be suitable in this aspect.

According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To for the whole filesystem. But the limit I was talking of is the file size
limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a DVD-R does not fit, for example.

Quoted 4 lines

Fragmentation is also a property not
of the filesystem, but of Microsoft's filesystem drivers. I'm fairly sure that the linux fat implementations do not use absurdly stupid allocation algorithms that lead to lots of fragmentation.

I am not sure about that: you can not do really good algorthms on bad data structures, and the data structures of FAT do not provide any support to do smart allocation.

Quoted 3 lines

This can be overcome with the UDF filesystem by using the uid and gid mount options, allowing the files to appear to be owned by the correct local user.

That is interesting. Do you know how efficient UDF is compared to others filesystems on normal hard drives? It is optimized for CDs and DVDs, I would not be surprised if the performances were poor on different supports.

Quoted 2 lines

It would be nice if the other filesystems were patched to
allow such options as well.

I believe that such options should not be done on a per-filesystem basis. Something in the common code of the VFS would be more logical.

Quoted 2 lines

Network filesystems are not on disk filesystems, so they have nothing to do with this discussion; you can't format a disk as "nfs" or "smb".

The idea was to mount the disk with its haphazard UIDs, and then export it and mount it as a network filesystem over the loopback. By itself, it is absolutely useless, but networked filesystems have UIDs mapping facilities.

Regards,

Signature · 1 line
  Nicolas George
#3 lore raw
Quoted 5 lines · Nicolas George

According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To for the whole filesystem. But the limit I was talking of is the file size
limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a DVD-R does not fit, for example.

Ahh yes, the per file limit. BTW, why are you saying "To" and "Go" when you apparently mean "TB" and "GB"?

Quoted 4 lines

I am not sure about that: you can not do really good algorthms on bad data structures, and the data structures of FAT do not provide any support to do smart allocation.

The fat data structures do not encourage fragmentation any more or less than ext2/3. NTFS is slightly better, more comparable to reiserfs than ext2/3, but the difference is small. What causes massive fragmentation is how the driver chooses to allocate new blocks as you write to files.
Microsoft has always used about the worst possible algorithm for doing
this you can imagine, which is why fragmentation has always been a big problem on their OSes. Linux is smarter and allocates blocks such that fragmentation is kept to a minimum.

Quoted 4 lines

That is interesting. Do you know how efficient UDF is compared to others filesystems on normal hard drives? It is optimized for CDs and DVDs, I would not be surprised if the performances were poor on different supports.

I have not done any testing, but I know no reason why it would be worse than fat. It does not do transaction logging, and there currently is no fsck for it, so for safety reasons, it may not be such a good choice. If you want reliability, efficiency, and posix semantics, I'd go with reiserfs or ext3.

Quoted 3 lines

I believe that such options should not be done on a per-filesystem basis. Something in the common code of the VFS would be more logical.

I agree. I think the VFS layer should process the uid/gid options. By default it should replace nobody with the specified id, and fat and ntfs
should just report all files as owned by nobody. Then a new option
should be added to force the translation for all ids, not just nobody.

Quoted 4 lines

The idea was to mount the disk with its haphazard UIDs, and then export it and mount it as a network filesystem over the loopback. By itself, it is absolutely useless, but networked filesystems have UIDs mapping facilities.

Eww, that is quite a kludge.

#4 lore raw
Quoted 8 lines
Quoted 4 lines

According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To for the whole filesystem. But the limit I was talking of is the file size
limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a DVD-R does not fit, for example.

Ahh yes, the per file limit. BTW, why are you saying "To" and "Go" when you apparently mean "TB" and "GB"?

French wording...

Jan Engelhardt

#5 lore raw
In reply to Phillip Susi
Quoted 1 line · Phillip Susi

Nicolas George wrote:

[snip]

Quoted 7 lines
Quoted 2 lines

I believe that such options should not be done on a per-filesystem basis. Something in the common code of the VFS would be more logical.

I agree. I think the VFS layer should process the uid/gid options. By default it should replace nobody with the specified id, and fat and ntfs
should just report all files as owned by nobody. Then a new option
should be added to force the translation for all ids, not just nobody.

I might be wrong, but I always thought that NTFS has user/group and a bunch of other attributes, so it might not be a good idea to replace them hard under linux. Or am I wrong? I never used NTFS much, the few windoze machines around me use FAT32 for compatibility.

Kalin.

Signature · 3 lines
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|
#7 lore raw
In reply to Phillip Susi

Le quartidi 24 pluviôse, an CCXIV, Phillip Susi a écrit :

Quoted 2 lines

Ahh yes, the per file limit. BTW, why are you saying "To" and "Go" when you apparently mean "TB" and "GB"?

I use the french word octet instead of byte, because it is less error prone (when you read "mb", does-it really mean megabit, or does it mean that the author is lazy about capitalization?) and a little bit more precise. Tough I actually am French, I did not start using a French word in English by myself. I copy a practice of the IETF: the RFCs use octet more than byte.

Quoted 8 lines

The fat data structures do not encourage fragmentation any more or less than ext2/3. NTFS is slightly better, more comparable to reiserfs than ext2/3, but the difference is small. What causes massive fragmentation is how the driver chooses to allocate new blocks as you write to files.
Microsoft has always used about the worst possible algorithm for doing
this you can imagine, which is why fragmentation has always been a big problem on their OSes. Linux is smarter and allocates blocks such that fragmentation is kept to a minimum.

I believe you about that.

Quoted 2 lines

I have not done any testing, but I know no reason why it would be worse than fat.

That is a very good point. If windows can read UDF on hard drives and not only DVD, UDF could probably supersede FAT completely.

Thank you for pointing me that direction.

Quoted 2 lines

It does not do transaction logging, and there currently is no
fsck for it, so for safety reasons, it may not be such a good choice.

I have a Solaris 9 near at hand, and I see a /lib/fs/udfs/fsck, and in the source tarball of OpenSolaris, I find a directory usr/src/cmd/fs.d/udfs/fsck/. It does not compile out of the box, but it may be possible to port it with limited effort.

Quoted 4 lines

I agree. I think the VFS layer should process the uid/gid options. By default it should replace nobody with the specified id, and fat and ntfs should just report all files as owned by nobody. Then a new option should be added to force the translation for all ids, not just nobody.

I agree with that (except maybe for the NTFS part, which I do not know; let us just say "UID-less filesystems"). Maybe a full UID translation system similar th the one in NFS could be useful, or a generic hook for modules, but having basic UID overriding would be great.

Unfortunately, the VFS subsystem is something too complex for me at this time.

#8 lore raw
Quoted 6 lines · Nicolas George

I use the french word octet instead of byte, because it is less error prone (when you read "mb", does-it really mean megabit, or does it mean that the author is lazy about capitalization?) and a little bit more precise. Tough I actually am French, I did not start using a French word in English by myself. I copy a practice of the IETF: the RFCs use octet more than byte.

Ahh, I see. I've never seen anyone use it in conjunction with an si prefix. I also think that they use it in RFCs because at the time they started writing them, bytes were not always 8 bits on all machines. Today it is a pretty safe assumption that a byte is 8 bits, so most people use the two terms interchangeably ;)

As for 1000 vs 1024 powers, I personally use the general rule that B = bytes and is in powers of 1024, and b = bits, and is in powers of 1000. For some reason the telecom industry likes to talk in terms of 1000 bits per second, but the rest of the sane world uses bytes in even powers of two. Generally it's people trying to sell you something that use powers of 1000 so they can make it sound bigger. I really hate that.

Quoted 6 lines

That is a very good point. If windows can read UDF on hard drives and not only DVD, UDF could probably supersede FAT completely.

Thank you for pointing me that direction.

I had that same thought a few weeks ago so I gave it a try. I formatted a partition with UDF, put some files on it, then booted windows to see if it would take it. It didn't :(

Quoted 6 lines

I have a Solaris 9 near at hand, and I see a /lib/fs/udfs/fsck, and in the source tarball of OpenSolaris, I find a directory usr/src/cmd/fs.d/udfs/fsck/. It does not compile out of the box, but it may be possible to port it with limited effort.

Hrm... interesting, I wonder how complete it is and what it's license is? The fsck in the udftools package was never actually implemented unfortunately.

Quoted 11 lines
Quoted 2 lines

I agree. I think the VFS layer should process the uid/gid options. By default it should replace nobody with the specified id, and fat and ntfs should just report all files as owned by nobody. Then a new option should be added to force the translation for all ids, not just nobody.

I agree with that (except maybe for the NTFS part, which I do not know; let us just say "UID-less filesystems"). Maybe a full UID translation system similar th the one in NFS could be useful, or a generic hook for modules, but having basic UID overriding would be great.

Unfortunately, the VFS subsystem is something too complex for me at this time.

#6 lore raw
In reply to Kalin KOZHUHAROV
Quoted 6 lines · Kalin KOZHUHAROV

I might be wrong, but I always thought that NTFS has user/group and a bunch of other attributes, so it might not be a good idea to replace them hard under linux. Or am I wrong? I never used NTFS much, the few windoze machines around me use FAT32 for compatibility.

Kalin.

IIRC, NTFS has the capability to store a posix uid and gid, but it is never actually used. There is no good way to get NT to understand mappings to linux uid/gids, so the information it puts there is useless. As a result, the Linux NTFS driver just makes all files owned by a fixed id you can specify at mount time, or defaults to root.

#9 lore raw
In reply to Phillip Susi

Le quintidi 25 pluviôse, an CCXIV, Phillip Susi a écrit :

Quoted 5 lines

Ahh, I see. I've never seen anyone use it in conjunction with an si prefix. I also think that they use it in RFCs because at the time they started writing them, bytes were not always 8 bits on all machines. Today it is a pretty safe assumption that a byte is 8 bits, so most people use the two terms interchangeably ;)

They continue using more octet than bytes even in recent RFCs. I have read I do not remember where that the goal was to avoid byte/bit confusion.

I am sorry, I did not intend to start an off-topic subthread. I think I should stick with kB/MB/GB unless I already used the full word earlier.

Quoted 3 lines

I had that same thought a few weeks ago so I gave it a try. I formatted a partition with UDF, put some files on it, then booted windows to see if it would take it. It didn't :(

So bad... Perhaps it was asking too much...

Quoted 2 lines

Hrm... interesting, I wonder how complete it is and what it's license is?

The man page (<URL:
http://docs.sun.com/app/docs/doc/816-5166/6mbb1kq22?a=view > for Solaris 10, I believe OpenSolaris is based on it) tells briefly that the checked inconsistencies are (I quote):

- Blocks claimed by more than one file or the free list
- Blocks claimed by a file or the free list outside the range of the file system
- Incorrect link counts in file entries
- Incorrect directory sizes
- Bad file entry format
- Blocks not accounted for anywhere
- Directory checks, file pointing to unallocated file entry and absence of a parent directory entry
- Descriptor checks, more blocks for files than there are in the file system
- Bad free block list format
- Total free block count incorrect

I do not know UDF at all, so I can not tell if this is enough or not.

As for the licence, it is the one of OpenSolaris <URL:
http://www.opensolaris.org/os/licensing/opensolaris_license/ >, which is free enough for the FSF to make efforts to have GPL3 compatible with it.

Regards,

Signature · 1 line
  Nicolas George