Posix file attribute support on VFAT (take #2)

This is a take 2 of posix file attribute support on VFAT.

I made a couple of changes from previous revision, according feedbacks on LKML.
- clean up
- restrict file attribute operations to prevent attribute lose at runtime.
- added condition checking to special files like device files and symlink.

I did not yet implement short symlik optimization, sugessted at <[email protected]> from [email protected]

Main purpose of this patch is to build root file system with VFAT for smallembedded device. FAT is widely used for embedded device to exchange data, and also small embedded device has resource limitation. So it's very handy that VFAT has capability to built root fs, even that has restrictions.

Details are described within a patch. I think this feature still needs improvemnts, however it is very helpful for most embedded developpers.

This patch is against 2.6.12 kernel.

Hiroyuki Machida		[email protected]		
SSW Dept. HENC, Sony Corp.

This patch enables "posix_attr" option described as following;

		 vfat posix attr option "posix_attr" 

					Mon Aug 15 21:29:54 JST 2005

* FEATURES

Following attributes/modes are supported in posix attributea mapping in VFAT.

- FileType
This supports following special files and it's attributes;
symbolic link, block device node,
char device node, fifo, socket
Regular files/dirs also may have POSIX attributes.
- DeviceFile
Major and minor number would be held at ctime
and both values are limited to 255.
- Owner's User ID/Group ID
This can be used to distinguish root and others, because this has just one bit width.
Value of UID/GID for non-root user will be taken from uid/gid option on mounting. If nothing is specified, system uses (u16)-1 as last resort. That means change-uid may affect on gid.
- Permission for Group/Other (rwx)
Those modes will be kept in ctime_cs.
Also permission modes for "others" will be
same as "group", due to lack of fields.
That means set-group-mode may affect on other-mode. On the other hand, set-other-mode has no affect to group-mode.

- Permission for Owner (rwx)
These modes will be mapped to FAT attributes. Just same as mapping under VFAT.
- Others
no sticky, setgid nor setuid bits are not supported.

* ALGORITHM FOR MAPPING DECISION
- Regular file/dir
To distinguish regular files/dirs, look if this fat dir entry doesn't have ATTR_SYS, first. If it doesn't have ATTR_SYS, then check if TYPE field (MSB 3bits) in ctime_cs is equal to 7. If so, this regular file/dir is created and/or modified under VFAT with "posix_attr". And posix attribute mapping can be take place. Otherwise, conventional VFAT attribute mapping is used.

- Special file
To distinguish special files, look if this fat dir entry has ATTR_SYS, first. Also we need to check it not to have ATT_EXT.
If it has ATTR_SYS, then check 1st. LSB bit in ctime_cs, referred as "special file flag".
If set, this file is created under VFAT with "posix_attr". Look up TYPE field to decide special file type.

This special file detection method has some flaw to make potential confusion. E.g. some system file created under dos/win may be treated as special file. However in most case, user don't create system file under dos/win.
To reduce possiblity of this confusion, system makes sure special files except symlink have size ZERO. For symlink, system checks it's size not to exceed page size and PATH_MAX.

* FAT DIR ENTRY FIELDS
- ctime_cs
8bit byte
7 6 5 4 3 2 1 0

 	|===| | | | | |
 	TYPE  | | | | +- special file flag (valid if ATTR_SYS)
 	      | | | +--- User/Group ID(root or others)
 	      | | +----- !group X
 	      | +------- !group W
 	      +--------- !group R

special file flag
Indicate this entry has posix attribute mapping. This field is valid for fat dir entry, which have ATTR_SYS.

special file TYPE
val type on VFS(val) Description
------------------------------------------------
0 (place folder for backward compat)
1 DT_LNK (10) symbolic link
2 DT_BLK (6) block dev
3 DT_CHR (4) char dev
4 DT_FIFO (1) fifo
5 DT_SOCK (12) socket

7*) (reserved for DT_REG/DT_DIR)

*)Value 7 is reserved for regular file/dir (DT_REG/DT_DIR). Normally ctime_cs would have 0-199 value to stand for up to 2sec. The value for DT_REG/DT_DIR is selected to be over this range to distinguish if file was created under POSIX_ATTR or not.

- attr
FAT attribute (val) mapped attribute
------------------------------------------------
ATTR_RO 0x01 !owner W
ATTR_HIDDEN 0x02 !owner R
ATTR_SYS 0x04
ATTR_VOLUME 0x08
ATTR_DIR 0x10 DIR
ATTR_ARCH 0x20 !owner X

- ctime
16bit word
f e d c b a 9 8 7 6 5 4 3 2 1 0

 	|=============| |-------------|
 	  major		  minor

* vfat-posix_attr.patch:

 fs/fat/file.c            |   52 ++++++++
 fs/fat/inode.c           |   27 ++++
 fs/vfat/namei.c          |  272 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/msdos_fs.h |  280 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 629 insertions(+), 2 deletions(-)

@@ -162,6 +162,8 @@
 	struct msdos_sb_info *sbi = MSDOS_SB(dentry->d_sb);
 	struct inode *inode = dentry->d_inode;
 	int mask, error = 0;
+	int px_uid = sbi->options.fs_uid ? sbi->options.fs_uid : (u16)-1;
+	int px_gid = sbi->options.fs_gid ? sbi->options.fs_gid : (u16)-1;
 
 	lock_kernel();
 

@@ -173,12 +175,56 @@
 		}
 	}
 
+	if (sbi->options.posix_attr) {
+		if (attr->ia_valid & ATTR_UID) {
+			/* root can change uid to any value */
+			if (capable(CAP_CHOWN) && 
+			    (attr->ia_uid && attr->ia_uid != px_uid))
+				attr->ia_uid = px_uid;
+			/* change-uid affects gid */
+			attr->ia_valid |= ATTR_GID;
+			attr->ia_gid = attr->ia_uid ? px_gid : 0;
+		} else {
+			/* chown syscall sets both uid and gid */
+			if (attr->ia_valid & ATTR_GID) {
+				/* root can change gid to any value */
+				if (capable(CAP_CHOWN) && 
+				    (attr->ia_gid && attr->ia_gid != px_gid))
+					attr->ia_gid = px_gid;
+				/* change-gid affects uid */
+				attr->ia_valid |= ATTR_UID;
+				attr->ia_uid = attr->ia_gid ? px_uid : 0;
+			}
+		}
+		/* change-group-mode affects on others-mode */
+		if (attr->ia_valid & ATTR_MODE) { 
+			int others_mode = (attr->ia_mode & S_IRWXG) >> 3;
+			attr->ia_mode &=  ~S_IRWXO;
+			attr->ia_mode |=  others_mode;
+		}
+	}
+
 	error = inode_change_ok(inode, attr);
 	if (error) {
 		if (sbi->options.quiet)
 			error = 0;
 		goto out;
 	}
+
+	if (sbi->options.posix_attr) {
+		if (((attr->ia_valid & ATTR_UID) &&
+		       ((attr->ia_uid) && (attr->ia_uid != px_uid))) ||
+		    ((attr->ia_valid & ATTR_GID) &&
+		       ((attr->ia_gid) && (attr->ia_gid != px_gid))) ||
+		    ((attr->ia_valid & ATTR_MODE) &&
+		     (attr->ia_mode & ~VFAT_POSIX_ATTR_VALID_MODE)))
+			error = -EPERM;
+		if (!error)  
+			error = inode_setattr(inode, attr);
+		if (sbi->options.quiet)
+			error = 0;
+		goto out;
+	}
 	if (((attr->ia_valid & ATTR_UID) &&
 	     (attr->ia_uid != sbi->options.fs_uid)) ||
 	    ((attr->ia_valid & ATTR_GID) &&

@@ -306,3 +352,9 @@
 	.truncate	= fat_truncate,
 	.setattr	= fat_notify_change,
 };
+
+struct inode_operations fat_symlink_inode_operations = {
+       .readlink       = page_readlink,
+       .follow_link    = page_follow_link_light,
+       .setattr        = fat_notify_change,
+};

Index: alp-2.6.12-p8/fs/fat/inode.c
===================================================================
@@ -274,6 +274,10 @@
 		MSDOS_I(inode)->i_logstart = MSDOS_I(inode)->i_start;
 		inode->i_size = le32_to_cpu(de->size);
 		inode->i_op = &fat_file_inode_operations;
+		if (sbi->options.posix_attr
+			&& is_vfat_posix_symlink(inode, de)){
+				inode->i_op = &fat_symlink_inode_operations;
+		}
 		inode->i_fop = &fat_file_operations;
 		inode->i_mapping->a_ops = &fat_aops;
 		MSDOS_I(inode)->mmu_private = inode->i_size;

@@ -499,8 +503,11 @@
 	fat_date_unix2dos(inode->i_mtime.tv_sec, &raw_entry->time, &raw_entry->date);
 	if (sbi->options.isvfat) {
 		fat_date_unix2dos(inode->i_ctime.tv_sec,&raw_entry->ctime,&raw_entry->cdate);
-		raw_entry->ctime_cs = (inode->i_ctime.tv_sec & 1) * 100 +
-			inode->i_ctime.tv_nsec / 10000000;
+		if (set_vfat_posix_attr(raw_entry, inode) == -1) {
+			raw_entry->ctime_cs 
+				= (inode->i_ctime.tv_sec & 1) * 100 +
+					inode->i_ctime.tv_nsec / 10000000;
+		}
 	}
 	spin_unlock(&sbi->inode_hash_lock);
 	mark_buffer_dirty(bh);

@@ -742,6 +749,8 @@
 			seq_puts(m, ",uni_xlate");
 		if (!opts->numtail)
 			seq_puts(m, ",nonumtail");
+		if (opts->posix_attr)
+			seq_puts(m, ",posix_attr");
 	}
 
 	return 0;

@@ -755,6 +764,7 @@
 	Opt_charset, Opt_shortname_lower, Opt_shortname_win95,
 	Opt_shortname_winnt, Opt_shortname_mixed, Opt_utf8_no, Opt_utf8_yes,
 	Opt_uni_xl_no, Opt_uni_xl_yes, Opt_nonumtail_no, Opt_nonumtail_yes,
+	Opt_posix_attr_no, Opt_posix_attr_yes,
 	Opt_obsolate, Opt_err,
 };
 

@@ -823,6 +833,13 @@
 	{Opt_nonumtail_yes, "nonumtail=yes"},
 	{Opt_nonumtail_yes, "nonumtail=true"},
 	{Opt_nonumtail_yes, "nonumtail"},
+	{Opt_posix_attr_no, "posix_attr=0"},		/* 0 or no or false */
+	{Opt_posix_attr_no, "posix_attr=no"},
+	{Opt_posix_attr_no, "posix_attr=false"},
+	{Opt_posix_attr_yes, "posix_attr=1"},	/* empty or 1 or yes or true */
+	{Opt_posix_attr_yes, "posix_attr=yes"},
+	{Opt_posix_attr_yes, "posix_attr=true"},
+	{Opt_posix_attr_yes, "posix_attr"},
 	{Opt_err, NULL}
 };
 

@@ -980,6 +997,12 @@
 		case Opt_nonumtail_yes:		/* empty or 1 or yes or true */
 			opts->numtail = 0;	/* negated option */
 			break;
+		case Opt_posix_attr_no:		/* 0 or no or false */
+			opts->posix_attr = 0;
+			break;
+		case Opt_posix_attr_yes:	/* empty or 1 or yes or true */
+			opts->posix_attr = 1;
+			break;
 
 		/* obsolete mount options */
 		case Opt_obsolate:

Sorry, but this is far too scary. Please just use one of the sane filesystems linux supports.

I'm trying to explain background....

I would say that purpose of the feature is having ability to
build root fs for small embedded device, not support full posix attributes top of VFAT. I think the situation is like uclinux, which has no MMU support and many restriction, however it's still very helpful for small embedded device.

To reduce resource consumption, developers for embedded system
would like to select one file system type to be used, if possible. And in most case, FAT is required for data exchange.
E.g. memory/storage card or USB client.

So adding small feature to FAT could have ability to build root fs, it's very helpful. It's not required to support full attributes.

What do you think ?

Thanks,
Hiroyuki Machida

This doesn't seem so different from umsdos to me -- which was only
removed from the kernel because it was unmaintained. It might have its place.

However, I'd guess you'd need to demonstrate that someone is
actually going to use and maintain this feature before it would be considered for inclusion in the kernel.

--
Joseph Fannin
[email protected]

"Bull in pure form is rare; there is usually some contamination by data."
-- William Graves Perry Jr.

Hi!

Unfortunately, it makes sense. If you have compact flash card, you really want to have VFAT there, so that it is a) compatible with windows and b) so that you don't kill the hardware.

I guess being able to use CF card for root filesystem is usefull, too....

Pavel

if you have sharp zaurus hardware you don't need... you know my address

Why not using fuse in a initramfs?

Why not doing this with fuse? Write a filesystem that will use the base fat file system (or any other stupid file system) but using a database (BSD) or a file for non existent field. This new layer will implement:
o mapping between true name and short name
o links (soft and emulated hard)
o permissions (and special files)

This solution is more general and can apply to all the stupid filesystems. In the initramfs we need only to put:
o fuse module
o our fuse filesystem statically linked or again klibc.

Moreover this doesn't add more fat (fat because previous solution is really ugly) to the kernel and will implement an universal umsdos.

I believe a shell program example will give you a better idea of my minds:
-----
modprobe fuse
mount -t vfat /dev/hda1 /mnt
# fuse filesystem use the old /mnt
# It do a chdir in /mnt (base) before the mount was efective mount -t fuse_extenddumbfilesystem none /mnt -o base=/mnt

# chroot
chroot /mnt
-------

But some could object that fuse like a support of write shared memory mmaped segment, but it is not different than nfs root!

Slowness is not a valid argument because fat and other file system are not really suitable for unix computing...

I join as an attachment (I know, it is not really good), my 1c simple implementation of such filesystem. This program do only the fuse bind mount nothing else but if somebody want to go further...

/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2005 Miklos Szeredi <[email protected]>

This program can be distributed under the terms of the GNU GPL. See the file COPYING.
*/

//#include <config.h>

//#ifdef linux
/* For pread()/pwrite() */
#define _XOPEN_SOURCE 500
#define FUSE_USE_VERSION 22
#define _FILE_OFFSET_BITS 64
//#endif

#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <sys/statfs.h>
#ifdef HAVE_SETXATTR
#include <sys/xattr.h>
#endif

/*
* remove the absolute path part
*/
static char *local=".";
static inline const char *rel(const char *path)
{
if(strcmp(path,"/")==0)

    return local;
  else
    return (path+1);
}

static int xmp_getattr(const char *path, struct stat *stbuf)

{
    int res;

    res = lstat(rel(path), stbuf);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_readlink(const char *path, char *buf, size_t size)

{
    int res;

    res = readlink(rel(path), buf, size - 1);
    
    if(res == -1)
        return -errno;

    buf[res] = '\0';
    return 0;
}

static int xmp_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
off_t offset, struct fuse_file_info *fi)

{
    DIR *dp;
    struct dirent *de;

    (void) offset;
    (void) fi;

    dp = opendir(rel(path));
    if(dp == NULL)
        return -errno;

    while((de = readdir(dp)) != NULL) {
        struct stat st;
        memset(&st, 0, sizeof(st));
        st.st_ino = de->d_ino;
        st.st_mode = de->d_type << 12;
        if (filler(buf, de->d_name, &st, 0))
            break;
    }

    closedir(dp);
    return 0;
}

static int xmp_mknod(const char *path, mode_t mode, dev_t rdev)

{
    int res;

    res = mknod(rel(path), mode, rdev);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_mkdir(const char *path, mode_t mode)

{
    int res;

    res = mkdir(rel(path), mode);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_unlink(const char *path)

{
    int res;

    res = unlink(rel(path));
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_rmdir(const char *path)

{
    int res;

    res = rmdir(rel(path));
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_symlink(const char *from, const char *to)

{
    int res;

    res = symlink(rel(from), to);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_rename(const char *from, const char *to)

{
    int res;

    res = rename(rel(from), rel(to));
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_link(const char *from, const char *to)

{
    int res;

    res = link(rel(from), rel(to));
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_chmod(const char *path, mode_t mode)

{
    int res;

    res = chmod(rel(path), mode);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_chown(const char *path, uid_t uid, gid_t gid)

{
    int res;

    res = lchown(rel(path), uid, gid);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_truncate(const char *path, off_t size)

{
    int res;

    res = truncate(rel(path), size);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_utime(const char *path, struct utimbuf *buf)

{
    int res;

    res = utime(rel(path), buf);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_open(const char *path, struct fuse_file_info *fi)

{
    int res;
    
    res = open(rel(path), fi->flags);
    if(res == -1)
        return -errno;

    close(res);
    return 0;
}

static int xmp_read(const char *path, char *buf, size_t size, off_t offset,
struct fuse_file_info *fi)

{
    int fd;
    int res;

    (void) fi;
    fd = open(rel(path), O_RDONLY);
    if(fd == -1)
        return -errno;

    res = pread(fd, buf, size, offset);
    if(res == -1)
        res = -errno;

    close(fd);
    return res;
}

static int xmp_write(const char *path, const char *buf, size_t size,
off_t offset, struct fuse_file_info *fi)

{
    int fd;
    int res;

    (void) fi;
    fd = open(rel(path), O_WRONLY);
    if(fd == -1)
        return -errno;

    res = pwrite(fd, buf, size, offset);
    if(res == -1)
        res = -errno;

    close(fd);
    return res;
}

static int xmp_statfs(const char *path, struct statfs *stbuf)

{
    int res;

    res = statfs(rel(path), stbuf);
    if(res == -1)
        return -errno;

    return 0;
}

static int xmp_release(const char *path, struct fuse_file_info *fi)

{
    /* Just a stub.  This method is optional and can safely be left
       unimplemented */

    (void) path;
    (void) fi;
    return 0;
}

static int xmp_fsync(const char *path, int isdatasync,
struct fuse_file_info *fi)

{
    /* Just a stub.  This method is optional and can safely be left
       unimplemented */

    (void) path;
    (void) isdatasync;
    (void) fi;
    return 0;
}

#ifdef HAVE_SETXATTR
/* xattr operations are optional and can safely be left unimplemented */ static int xmp_setxattr(const char *path, const char *name, const char *value,
size_t size, int flags)

{
    int res = lsetxattr(rel(path), name, value, size, flags);
    if(res == -1)
        return -errno;
    return 0;
}

static int xmp_getxattr(const char *path, const char *name, char *value,
size_t size)

{
    int res = lgetxattr(rel(path), name, value, size);
    if(res == -1)
        return -errno;
    return res;
}

static int xmp_listxattr(const char *path, char *list, size_t size)

{
    int res = llistxattr(rel(path), list, size);
    if(res == -1)
        return -errno;
    return res;
}

static int xmp_removexattr(const char *path, const char *name)

{
    int res = lremovexattr(rel(path), name);
    if(res == -1)
        return -errno;
    return 0;
}

#endif /* HAVE_SETXATTR */

static struct fuse_operations xmp_oper = {
    .getattr	= xmp_getattr,
    .readlink	= xmp_readlink,
    .readdir	= xmp_readdir,
    .mknod	= xmp_mknod,
    .mkdir	= xmp_mkdir,
    .symlink	= xmp_symlink,
    .unlink	= xmp_unlink,
    .rmdir	= xmp_rmdir,
    .rename	= xmp_rename,
    .link	= xmp_link,
    .chmod	= xmp_chmod,
    .chown	= xmp_chown,
    .truncate	= xmp_truncate,
    .utime	= xmp_utime,
    .open	= xmp_open,
    .read	= xmp_read,
    .write	= xmp_write,
    .statfs	= xmp_statfs,
    .release	= xmp_release,
    .fsync	= xmp_fsync,

#ifdef HAVE_SETXATTR
.setxattr = xmp_setxattr,
.getxattr = xmp_getxattr,
.listxattr = xmp_listxattr,
.removexattr= xmp_removexattr,
#endif
};

int main(int argc, char *argv[])

{
    chdir(argv[argc-1]);
    return fuse_main(argc-1, argv, &xmp_oper);
}

VFAT is plenty good at killing hardware. It's a terrible filesystem for flash cards (if they don't do their own wear leveling properly). Most of the linux filesystems may not be any better but they are also no worse. Windows compatibility is completely irrelevant if the card is being used as your root filesystem since any extensions you make to vfat wouldn't be understood by windows anyhow, so at best it makes a mess of it.

I run ext3 on CF and so far, no problems. I run with noatime and try to avoid writing in general as much as possible. VFAT would be crap since, well, I run linux on the system.

Len Sorensen

Hi!

Well, they actually do test those cards with VFAT. Rumors are, they have some VFAT specific hacks in flash card firmware.

64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms