Skip to contentSkip to conversation

What's a realistic size for xattr?

DISCUSSION Charles Manning 8 participants 11 messages Mar 12, 2008 – Mar 15, 2008 VFS

lore
Share & tools
lore
#0 lore raw

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

Quoted 1 line

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Charles

#1 lore raw
Quoted 9 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

From my (limited) knowledge of xattr it seems that in theory you could store a multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

XATTR_SIZE_MAX.

Signature · 1 line
Nathan
#2 lore raw
In reply to Charles Manning
Quoted 11 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

Quoted 1 line

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Quoted 7 lines

Charles

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I just realized that I never really answered your question. I've yet to see a reasonable SELinux context over 128 characters and Smack labels should be shorter than that. I don't know what beagle places in xattrs but I think that would be a good place to take a look for more practical xattr sizes.

Dave

#3 lore raw
Quoted 23 lines · On Thursday 13 March 2008 12:32:33 Dave Quigley wrote:
Quoted 14 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

Quoted 2 lines

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Quoted 7 lines

Charles

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

I just realized that I never really answered your question. I've yet to see a reasonable SELinux context over 128 characters and Smack labels should be shorter than that. I don't know what beagle places in xattrs but I think that would be a good place to take a look for more practical xattr sizes.

Dave

Is that 128 characters per xattr value or for the whole set of xattribs attached to a file?

What I'm trying to figure out is a reasonable size for all the xattribs together. ie. If I store xattribs as a single blob containing all the name:value pairs, how big will that blob need to be?

I should have perhaps given some extra context here.

I'm trying to find the bang-for-bucks trade-off point for a simple implementation.

YAFFS is a flash file system and is typically used in embedded systems (phones, routers, printers, point-of-sale,... etc). I guess limited use of SELinux is a reasonable benchmark for what is used here.

Huge fs for corporate servers don't really fit in this picture so I guess vastly complex ACLs , while theoretically possible, don't really fall into the yaffs space.

#4 lore raw
Quoted 30 lines · Charles Manning
Quoted 29 lines · On Thursday 13 March 2008 12:32:33 Dave Quigley wrote:
Quoted 14 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

Quoted 2 lines

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Quoted 7 lines

Charles

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I just realized that I never really answered your question. I've yet to see a reasonable SELinux context over 128 characters and Smack labels should be shorter than that. I don't know what beagle places in xattrs but I think that would be a good place to take a look for more practical xattr sizes.

Quoted 5 lines
Quoted 1 line

Dave

Is that 128 characters per xattr value or for the whole set of xattribs attached to a file?

I'm talking about the value part so your name will be security.selinux and a value of usually no more than 128 characters. However with multiple LSMs its possible that you may have security.selinux security.smack security.newlsmnotcreatedyet.

Quoted 27 lines

What I'm trying to figure out is a reasonable size for all the xattribs together. ie. If I store xattribs as a single blob containing all the name:value pairs, how big will that blob need to be?

I should have perhaps given some extra context here.

I'm trying to find the bang-for-bucks trade-off point for a simple implementation.

YAFFS is a flash file system and is typically used in embedded systems (phones, routers, printers, point-of-sale,... etc). I guess limited use of SELinux is a reasonable benchmark for what is used here.

Huge fs for corporate servers don't really fit in this picture so I guess vastly complex ACLs , while theoretically possible, don't really fall into the yaffs space.

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to Dave Quigley
Quoted 5 lines · Dave Quigley
Quoted 5 lines
Quoted 1 line

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

Quoted 7 lines
Quoted 1 line

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

I just realized that I never really answered your question. I've yet to see a reasonable SELinux context over 128 characters and Smack labels should be shorter than that. I don't know what beagle places in xattrs but I think that would be a good place to take a look for more practical xattr sizes.

People copy files from other systems using rsync, so it's not just Linux uses of xattrs. Also, some programs can store their own data, such as search metadata, desktop metadata, checksums, descriptions (similar to MP3 id tags, but in xattrs) and all sorts of things.

On Macs, xattrs are used to store the "resource fork", which contains icons etc. so can be quite large. Of course that's not done on Linux. People do sometimes try to copy those files to Linux and back, and it generally fails because Linux refuses the large xattrs.

-- Jamie

#6 lore raw
Quoted 4 lines · Jamie Lokier

On Macs, xattrs are used to store the "resource fork", which contains icons etc. so can be quite large. Of course that's not done on Linux. People do sometimes try to copy those files to Linux and back, and it generally fails because Linux refuses the large xattrs.

The resource fork in HFS/HFS+ is not considered an xattr at all. In fact, it's effectively another file that just happens to be attached to the same inode-equivalent. There is a data fork (seen as a normal file for unix) and the resource fork, and it's just the posix style API that treats one as more important than the other. The two are considered peers in the on-disk format and in the classic mac API.

The only way I've seen the resource fork stored on a non-Apple fs is as a named stream style format or some kind of encoded single file format like AppleSingle/AppleDouble or BinHex.

If you really want to talk about size, there is a lot more than icons and metadata stored in the resource fork. In the m68k era, the compiled code was also stored in the resource fork with each segment showing up as a numbered resource of type 'CODE' that could be dynamically loaded and released. If you look at an application from those days, it most likely has a 0-length data fork.

Brad Boyer
[email protected]

#7 lore raw
In reply to Jamie Lokier
Quoted 19 lines · Jamie Lokier
Quoted 5 lines · Dave Quigley
Quoted 5 lines
Quoted 1 line

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

Quoted 7 lines
Quoted 1 line

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

I just realized that I never really answered your question. I've yet to see a reasonable SELinux context over 128 characters and Smack labels should be shorter than that. I don't know what beagle places in xattrs but I think that would be a good place to take a look for more practical xattr sizes.

People copy files from other systems using rsync, so it's not just Linux uses of xattrs. Also, some programs can store their own data, such as search metadata, desktop metadata, checksums, descriptions (similar to MP3 id tags, but in xattrs) and all sorts of things.

Quoted 6 lines

On Macs, xattrs are used to store the "resource fork", which contains icons etc. so can be quite large. Of course that's not done on Linux. People do sometimes try to copy those files to Linux and back, and it generally fails because Linux refuses the large xattrs.

-- Jamie

Linux xattrs are nothing more than named extensions to struct stat. It's unfortunate that Solaris reused the xattr name for a completely different concept that already had existing names (named streams AKA forks), but they aren't equivalent and are not intended to be equivalent.

Pretending that xattrs are named streams is just a path to madness and suffering, considering the Linux API is based around atomic attribute querying and modification instead of the standard Unix I/O primitives and the Powers That Be are vehemently against the implementation of named streams in the Linux VFS.

Signature · 1 line
Nicholas Miell <[email protected]>
#8 lore raw
In reply to Charles Manning
Quoted 11 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

Quoted 1 line

From my (limited) knowledge of xattr it seems that in theory you could store a

multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

Thanx.

Quoted 7 lines

Charles

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I don't know about specific file system limits on xattrs but from a user-space perspective the sys_setxattr->setxattr call chain enforces a limit of no more than 64k on an xattr value. In addition the list of xattr names is also limited to 64k with a name being no longer than 255 chars. From your wording above it seems that you want to store the xattrs as their own records. One approach (which is different from what I think you want) is to store the xattrs in the inode upto a point and then if necessary allocate space from them elsewhere. I'm not familiar with what you are trying to implement but I believe this is the way XFS handles xattrs.

#define XATTR_NAME_MAX   255    /* # chars in an extended attribute name */
#define XATTR_SIZE_MAX 65536    /* size of an extended attribute value (64k) */
#define XATTR_LIST_MAX 65536    /* size of extended attribute namelist (64k) */

Dave

#9 lore raw
In reply to Charles Manning
Quoted 8 lines · Charles Manning

I'm trying to figure out a reasonable approach to implementing xattr in YAFFS.

From my (limited) knowledge of xattr it seems that in theory you could store a multi-Mbyte database in xattr, but in practice a smaller size is far more reasonable. Clearly storing/managing a small blob is going to be a lot simpler.

What is the cut off of a reasonable xattr blob size? 1kbytes? 2kbytes?...

As a point of reference, ext2/3/4 limit the total xattr size to 1 data block (4kB commonly), and I believe XFS has a total limit of 64kB. ZFS and NTFS have resource forks, which can store arbitrary data, but they have a different API. On NTFS the word is that resource forks are mostly used by root kits.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#10 lore raw

* Andreas Dilger:

Quoted 1 line

On NTFS the word is that resource forks are mostly used by root kits.

They are also used to mark programs you've downloaded from the Internet, so that a warning can be displayed when you open them.