directory quota survey on xfs

Hi all,

As discussed with ext4 folks, I will try to make ext4 file system support directory quota (a.k.a., project id in xfs).

For keeping consistency
with xfs's implementation, I did some tests on xfs, and I summarized the result as following. This will help us understand what we can do and what we can't do. Please correct me if I miss doing some tests or mis- understand something.

I just do some tests about rename/hardlink because they are the key issue from our discussion.

+ unaccounted dir
x accounted dir

rename(mv)
==========

+ -> +: ok

+ -> x: ok

I use strace(1) to lookup which syscall is used, and I found that xfs will return EXDEV when mv(1) tries to use rename(2) to move a dir from a unaccounted dir to a accounted dir. Then mv uses creat(2)/read(2)/ write(2) syscalls to move this dir.

x -> +: wrong (design by feature?)

If we move a dir from a accounted dir to a unaccounted dir, the quota is still accounted. It seems that xfs don't remove project id from this dir and its subdirs/files on this case.

x -> x: ok

Xfs returns EXDEV error when mv(1) uses rename(2) to move a dir from a accounted dir to another accounted dir (These dirs have different project ids). Then mv(1) uses creat(1)/read(1)/write(1) syscalls to move this dir.

summary:

rename        +        x
+             ok       ok (EXDEV)
x             wrong    ok (EXDEV)

hardlink(ln)
========

+ -> +: ok

+ -> x: error

Xfs also returns EXDEV error to forbid this operation. So that means that we don't allow users to do a hardlink for a file from a unaccount dir to a accounted dir.

x -> +: ok

This operation can be executed and no any error is reported. After that the quota doesn't be changed. When both of two hardlinks are removed, the quota will be discharged.

As always, any comment or idea are welcome.

Quoted 1 line

As always, any comment or idea are welcome.

I'd suggest that you implement project quotas, not directory quotas. They are way more flexible than pure directory quotas, but with only a few lines of code and a special directory flag they can be used to

Quoted 1 line

implement directory subtree quotas....

Quoted 6 lines · Dave Chinner

Quoted 1 line

As always, any comment or idea are welcome.

I'd suggest that you implement project quotas, not directory quotas. They are way more flexible than pure directory quotas, but with only a few lines of code and a special directory flag they can be used to

Quoted 1 line

implement directory subtree quotas....

Would be also nice to allow a file to belong to more than one project.

Let say I want to have

/projects/ with 10GB quota
and
/projects/projectA/ with 1GB quota /projects/projectB/ with 2GB quota
and so on that's still limited by /projects/ 10GB quota limit.

Quoted 4 lines · Zheng Liu

Hi all,

As discussed with ext4 folks, I will try to make ext4 file system support directory quota (a.k.a., project id in xfs).

Firstly, project quotas are not directory quotas. Project groups are simply an aggregation of unrelated inodes with a specific identifier (i.e. the project ID). Files accessable only to individual users can shared project ID and hence be accounted to the same quota, and hence accounting is independent of uid/gid.

By themselves, project quotas cannot be used to implement direct subtree quotas - thay requires help from a special inode flag that is used on directory inodes: XFS_DIFLAG_PROJINHERIT

This flag indicates that the directory and all inodes created in the directory inherit the project ID of the directory. Hence the act of creating a file in a XFS_DIFLAG_PROJINHERIT marked directory associates the new file with s a specific project group. New directories also get marked with XFS_DIFLAG_PROJINHERIT so the behaviour is propagated down the directory tree.

Now, there is nothing to stop us from having files outside the inheritance subtree from also having the same project ID, and hence be accounted to the same project group. Indeed, you can have multiple sub-trees that all use the same project ID and hence are accounted together. e.g. a project has subdirs in various directories:

	/documentation/project_A
	/src/project_A
	/build/project_A
	/test/project_A
	.....
	/home/bill/project_A
	/home/barry/project_A
	/home/benito/project_A
	/home/beryl/project_A
	.....

All of these directories can be set up with the same project ID and XFS_DIFLAG_PROJINHERIT flag, and hence all be accounted to the same project quota, despite being separate, disconnected subtrees.

IOWs, project groups are an accounting mechanism associated with the inode's project ID, while XFS_DIFLAG_PROJINHERIT is a policy used to direct how project IDs are applied.

Quoted 15 lines

For keeping consistency
with xfs's implementation, I did some tests on xfs, and I summarized the result as following. This will help us understand what we can do and what we can't do. Please correct me if I miss doing some tests or mis- understand something.

I just do some tests about rename/hardlink because they are the key issue from our discussion.

+ unaccounted dir
x accounted dir

rename(mv)
==========

Quoted 8 lines

+ -> +: ok

+ -> x: ok

I use strace(1) to lookup which syscall is used, and I found that xfs will return EXDEV when mv(1) tries to use rename(2) to move a dir from a unaccounted dir to a accounted dir. Then mv uses creat(2)/read(2)/ write(2) syscalls to move this dir.

That's purely an implementation detail, designed to simplify the change of project ID for an inode. By forcing the new inode to be created from scratch under the destination's project ID, we don't have to care about whether rename needs to allocate or free source directory metadata, what anonymous metadata was accounted to the source project ID as the srouce file and directory was modified, etc. It's as simple as this:

        /*
         * If we are using project inheritance, we only allow renames
         * into our tree when the project IDs are the same; else the
         * tree quota mechanism would be circumvented.
         */
        if (unlikely((target_dp->i_d.di_flags & XFS_DIFLAG_PROJINHERIT) &&
                     (xfs_get_projid(target_dp) != xfs_get_projid(src_ip)))) {
                error = XFS_ERROR(EXDEV);
                goto error_return;
        }

It also documents the policy for accounting directory tree quotas:
that quota is accounted for when moving *into* an accounted directory tree, not when moving out of a directory tree.

Quoted 5 lines

x -> +: wrong (design by feature?)

If we move a dir from a accounted dir to a unaccounted dir, the quota is still accounted. It seems that xfs don't remove project id from this dir and its subdirs/files on this case.

That's the way the directory tree policy was designed: it's designed to allow project quotas to account for individual files as well as subtrees. Remember: projects are not confined to a single subtree and directory tree accounting is done when moving *into* a controlled tree, not the other way around.

Quoted 15 lines

x -> x: ok

Xfs returns EXDEV error when mv(1) uses rename(2) to move a dir from a accounted dir to another accounted dir (These dirs have different project ids). Then mv(1) uses creat(1)/read(1)/write(1) syscalls to move this dir.

summary:

rename        +        x
+             ok       ok (EXDEV)
x             wrong    ok (EXDEV)

hardlink(ln)
========

Quoted 7 lines

+ -> +: ok

+ -> x: error

Xfs also returns EXDEV error to forbid this operation. So that means that we don't allow users to do a hardlink for a file from a unaccount dir to a accounted dir.

Of course - who do you account new changes to? It's the same problem as linking across directory trees with different project IDs....

Quoted 6 lines

x -> +: ok

This operation can be executed and no any error is reported. After that the quota doesn't be changed. When both of two hardlinks are removed, the quota will be discharged.

Consistent with the rename case - checking is done based on the destination directory - you can link out to an uncontrolled destination as the inode is still accounted to the project ID, but you can't link into a controlled destination with a different project ID. The check is identical to the one I quoted for rename above.

Quoted 1 line

As always, any comment or idea are welcome.

I'd suggest that you implement project quotas, not directory quotas. They are way more flexible than pure directory quotas, but with only a few lines of code and a special directory flag they can be used to implement directory subtree quotas....

I'd also strongly suggest that you use the XFS userspace quota API for managing project quotas, so that we can use the same management tools and tests to verify that they behave the same. Please don't invent a new version of the quota API to implement this - everything you need ifor managing project/directory quotas is already there in xfs_quota.....

Quoted 6 lines · Dave Chinner

I'd also strongly suggest that you use the XFS userspace quota API for managing project quotas, so that we can use the same management tools and tests to verify that they behave the same. Please don't invent a new version of the quota API to implement this - everything you need ifor managing project/directory quotas is already there in xfs_quota.....

Frankly, I don't like this, really. Now we have quota-tool to manage the quota in ext4. So IMHO we'd better go on using this tool because it is natural for ext4 users. I still couldn't accept this fact that I need to install xfsprogs for using a feature of ext4. Further, it could make users puzzled because they use quota to control user/group quota in ext4, but it uses xfs_quota to control project quota. It could bring some troubles for the ext4 users who have written some scripts to manage their machines.

Quoted 8 lines · Zheng Liu

Quoted 6 lines · Dave Chinner

I'd also strongly suggest that you use the XFS userspace quota API for managing project quotas, so that we can use the same management tools and tests to verify that they behave the same. Please don't invent a new version of the quota API to implement this - everything you need ifor managing project/directory quotas is already there in xfs_quota.....

Quoted 3 lines

Frankly, I don't like this, really. Now we have quota-tool to manage the quota in ext4. So IMHO we'd better go on using this tool because it is natural for ext4 users.

Quoted 6 lines

I still couldn't accept this fact that I
need to install xfsprogs for using a feature of ext4. Further, it could make users puzzled because they use quota to control user/group quota in ext4, but it uses xfs_quota to control project quota. It could bring some troubles for the ext4 users who have written some scripts to manage their machines.

Please see Li Xi's recent email "Directory/Project quota supports" on the linux-ext4 list. He has already added some prototype support for project quotas to quota-tools.

I think it might make sense to keep the same API as XFS for the ext4 quotas (to keep compatibility for existing XFS deployments), but add support into quota-tools so that it is usable by all filesystems.

Quoted 9 lines · Andreas Dilger

Quoted 8 lines · Zheng Liu

Quoted 6 lines · Dave Chinner

I'd also strongly suggest that you use the XFS userspace quota API for managing project quotas, so that we can use the same management tools and tests to verify that they behave the same. Please don't invent a new version of the quota API to implement this - everything you need ifor managing project/directory quotas is already there in xfs_quota.....

Quoted 3 lines

Quoted 3 lines

Frankly, I don't like this, really. Now we have quota-tool to manage the quota in ext4. So IMHO we'd better go on using this tool because it is natural for ext4 users.

Zheng - you're confusing the userspace tool that users run with the quotactl API the tool uses to communicate with the kernel.

Quoted 8 lines

Quoted 7 lines

I still couldn't accept this fact that I
need to install xfsprogs for using a feature of ext4. Further, it could make users puzzled because they use quota to control user/group quota in ext4, but it uses xfs_quota to control project quota. It could bring some troubles for the ext4 users who have written some scripts to manage their machines.

Quoted 3 lines

Please see Li Xi's recent email "Directory/Project quota supports" on the linux-ext4 list. He has already added some prototype support for project quotas to quota-tools.

So, while it is a prototype, lets do it the right way. i.e. let's not reinvent the wheel.

Quoted 4 lines

I think it might make sense to keep the same API as XFS for the ext4 quotas (to keep compatibility for existing XFS deployments), but add support into quota-tools so that it is usable by all filesystems.

Well, yes. If you are writing a generic quota tool, then it needs to support all filesystems. We already have a fully featured quota API that can provide this support - it's the API that XFS has been using since it was ported to Linux. We have the opportunity to unify the quota APIs that ext4 and XFS, so we should take the opportunity while it is here. Don't create a new API for ext4 simply because of NIH syndrome.

Hi Dave,

Quoted 11 lines · Dave Chinner

Quoted 2 lines · Andreas Dilger

quotas (to keep compatibility for existing XFS deployments), but add support into quota-tools so that it is usable by all filesystems.

Well, yes. If you are writing a generic quota tool, then it needs to support all filesystems. We already have a fully featured quota API that can provide this support - it's the API that XFS has been using since it was ported to Linux. We have the opportunity to unify the quota APIs that ext4 and XFS, so we should take the opportunity while it is here. Don't create a new API for ext4 simply because of NIH syndrome.

These days I was thinking about your comment that uses quotactl API to communicate the userspace tool with the kernel. But I am still confusing about your comment that unifies the quota API between ext4 and XFS.

Now we have two flag sets in quotactl(2). One (Q_QUOTAON, Q_GETQUOTA, etc...) is used by extN file system (I am not sure whether other file systems use these flags or not), and another (Q_XQUOTAON, Q_XGETQSTAT, etc...) is used by XFS.

In xfs_quota it uses an ioctl(2) to get/set/check project id,

and calls
quotactl(2) with Q_XSETQLIM/Q_XGETQUOTA to set/get project quota.

On
kernel side, ->set_dqblk()/get_dqblk() is called when we try to set/get project quota in XFS.

In ext4 the same callback functions are used to
set/get user/group quota, although on userspace we use Q_SETQUOTA/ Q_GETQUOTA to set/get quota. I am not sure I fully understand your meaning that unifies the quota API between ext4 and XFS. Do you mean that we should use Q_XSETQLIM/Q_XGETQUOTA flags to set/get quota on ext4?

Quoted 20 lines · Zheng Liu

Hi Dave,

Quoted 11 lines · Dave Chinner

Quoted 2 lines · Andreas Dilger

quotas (to keep compatibility for existing XFS deployments), but add support into quota-tools so that it is usable by all filesystems.

Well, yes. If you are writing a generic quota tool, then it needs to support all filesystems. We already have a fully featured quota API that can provide this support - it's the API that XFS has been using since it was ported to Linux. We have the opportunity to unify the quota APIs that ext4 and XFS, so we should take the opportunity while it is here. Don't create a new API for ext4 simply because of NIH syndrome.

These days I was thinking about your comment that uses quotactl API to communicate the userspace tool with the kernel. But I am still confusing about your comment that unifies the quota API between ext4 and XFS.

Quoted 4 lines

Now we have two flag sets in quotactl(2). One (Q_QUOTAON, Q_GETQUOTA, etc...) is used by extN file system (I am not sure whether other file systems use these flags or not), and another (Q_XQUOTAON, Q_XGETQSTAT, etc...) is used by XFS.

I'm talking about making ext4 be able to use Q_XQUOTAON, Q_XGETQSTAT, etc.

Quoted 1 line

In xfs_quota it uses an ioctl(2) to get/set/check project id,

Right, because that's a filesystem specific operation that has no equivalent in any other filesystem at this point in time. Same for the project inheritence inode flag.

You're going to need to add such an interface to ext4 to do this, so add a generic ioctl and wire XFS up to it as well. This is kind of why I want a generic xattr namespace for inode flags/attribute at the VFS - so we don't have to keep inventing new ioctl/fcntl interfaces to make this sort of functionality common between different filesystems - we just define a new attribute string and values and let individual filesystems handle how they store them.