separate project quota from group quota (questions, design issues)
Dave thinks that this subject should be discussed first here.
My goal is to be able to use user, group and project quota at the same time. Example usage is a web hosting company which gives some space for customer (group quota), customer splits it into accounts (user quota) and then each account can have multiple vhosts with own quota (project quota).
Since on-disk format keeps group and project quota separate already then why not making it usable for a user as separate things, too?
I started doing that by splitting every group+project handling code into separate group and project one. Added superblock field for project quota. New feature flag (SEPARATEPQUOTA).
If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load sb_gquotino into mp->m_sb.sb_pquotino which I think is enough to handle old fs (since new kernel operates on separate structures for project quota).
New filesystem (without quota inodes allocated yet), when mounted, gets new quota inode allocated for project quota and SEPARATEQUOTA feature bit set.
Converting old fs to separate quota isn't currently solved. User could need two choices - use old group quota still as group or old group as new project quota. Not sure how to handle these yet.
Also, quota data is keept in extents and extents have d_flags representing type of quota (user/group/proj). Now what happens (in current code) with extents of group/project quota inode if mouting once with grpquota and once with projquota? Are the other extent entries ignored? (like mounted with group -> extents with proj type ignored or deleted or something else?).
There is also issue with kernel generic quota interface fs_quota_stat which doesn't handle third quota at all. It simply won't be able to operate on project quota (as I guess it isn't able to do that even now).
Initial, not working yet (due to locking issues) patch is here (on top of projid32bit patch actually, but both features are unrelated):
http://carme.pld-linux.org/~arekm/xfs-parallel-grp-prj-quota-2.patch
My current problem is lack of understanding of locking issues here:
http://pastebin.com/brBHPavR
How locking should look like when there is also project quota? Lines 1390-1397 and 1419-1428.
With current code as pasted on pastebin mount hangs trying to do lock on 1426 line which was unlocked just few lines before.
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines · On Samstag, 4. September 2010 Arkadiusz Miskiewicz wrote:
My goal is to be able to use user, group and project quota at the
same time. Example usage is a web hosting company which gives some space for customer (group quota), customer splits it into accounts (user quota) and then each account can have multiple vhosts with own quota (project quota).
Nice idea. I'd also like that, but also mixed:
userdir1 -> user quota
userdir2
prjdir1 -> project quota
prjdir2
groupdir1 -> group quota for a group of users
groupdir2 -> for a group of projects
From what you write, I'm sure you can't mix groups with users and projects, instead it needs to be defined at mount time. It would be very good if that can be mixed at the same time.
I'd like to say users1-5 can have 500G each, together 2T max. And projects can have 2T each, but project1+2 together max. 2T. users6-9 can have 100G each, together 100G max. Currently I defined everything as a project, and manually check quotas for groups of projects if they cross the "group max" border.
But I'd also be very interested in what you write above - it would be a very welcome expansion of quotas which seems good. Just a question for
setup: We use all dirs side-by-side like
domain1
domain2
|- prj1
|- prj2
domain3
Now customer1 can have domain1 and domain2, customer2 has domain3. prj1 and prj2 can have prjquota. Am I right that this is possible with your approach? Seems like, and I just want to ask to be sure I understand you correctly.
Just how would you account that? Say each customer gets 20G, each domain 20G, projects 10G each. When above customer1 has 12G in domain1, he has left 8G for domain2. Are projects also limited to 8G in sum, or is their 10G limit the one that's valid? If prj1 has 5G, is the 8G-5G=3G what's available for domain2, or are projects completely separate? I believe there are lots of variations and some people would like to have it accounted, some not. Can this ever be solved?
Signature · 13 lines
mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services http://proteger.at [gesprochen: Prot-e-schee] Tel: 0660 / 415 65 31 ****** Aktuelles Radiointerview! ****** http://www.it-podcast.at/aktuelle-sendung.html // Wir haben im Moment zwei Häuser zu verkaufen: // http://zmi.at/langegg/ // http://zmi.at/haus2009/
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Thanks for doing this work Arek, I think it will be useful for some users and defintively is cleaners than what we have now.
Quoted 3 lines
I started doing that by splitting every group+project handling code into separate group and project one. Added superblock field for project quota. New feature flag (SEPARATEPQUOTA).
Ok.
Quoted 3 lines
If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
sb_gquotinointomp->m_sb.sb_pquotinowhich I think is enough to handle old fs (since new kernel operates on separate structures for project quota).
Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
Quoted 2 lines
New filesystem (without quota inodes allocated yet), when mounted, gets new quota inode allocated for project quota and SEPARATEQUOTA feature bit set.
I think we need the SEPARATEQUOTA bit set explicitly by mkfs.xfs/xfs_admin. But once we start using project quotas with the bit set the above sounds fine.
Quoted 3 lines
Converting old fs to separate quota isn't currently solved. User could need two choices - use old group quota still as group or old group as new project quota. Not sure how to handle these yet.
We know what sb_gquotino is used for currently from sb_qflags. If it's used for group quota we can simply set the SEPARATEQUOTA bit and we're done. To convert a filesystem using project quotas to SEPARATEQUOTA sb_gquotino needs to be cleared and instead the new sb_pquotino field needs to point to the project quota inode.
Need to go now, I'll answer to your other questions once I had some more time to review them.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Christoph Hellwig
Thanks for doing this work Arek, I think it will be useful for some users and defintively is cleaners than what we have now.
Quoted 3 lines
I started doing that by splitting every group+project handling code into separate group and project one. Added superblock field for project quota. New feature flag (SEPARATEPQUOTA).
Quoted 8 lines
Ok.
Quoted 4 lines
If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
sb_gquotinointomp->m_sb.sb_pquotinowhich I think is enough to handle old fs (since new kernel operates on separate structures for project quota).Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
Yes. When trying to load existing project quota inode in xfs_qm_init_quotainos() I'm doing:
+ if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
+ ASSERT(!XFS_IS_GQUOTA_ON(mp)); + mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino; + mp->m_sb.sb_gquotino = 0;
+ }
While xfs_mod_sb() now does:
/*
* Filesystem has no separatepquota turned on,
* so we need to store project quota in group
* quota inode on disk.
*/
if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
mp->m_sb.sb_gquotino = mp->m_sb.sb_pquotino;
mp->m_sb.sb_pquotino = 0;
}
xfs_sb_to_disk(XFS_BUF_TO_SBP(bp), &mp->m_sb, fields);[...]
/*
* Restore original in-memory project quota inode state.
*/
if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) {
mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino;
mp->m_sb.sb_gquotino = 0;
} so in-memory sb_pquotaino is always used for project quota (regardless of SEPARATEPQUOTA feature) while on-disk is depending on SEPARATEPQUOTA feature bit. SB modify part is untested yet.
Quoted 9 lines
Quoted 3 lines
Converting old fs to separate quota isn't currently solved. User could need two choices - use old group quota still as group or old group as new project quota. Not sure how to handle these yet.
We know what
sb_gquotinois used for currently fromsb_qflags. If it's used for group quota we can simply set the SEPARATEQUOTA bit and we're done. To convert a filesystem using project quotas to SEPARATEQUOTAsb_gquotinoneeds to be cleared and instead the newsb_pquotinofield needs to point to the project quota inode.
Ok, I'll add new "upgrade <feature>" command to xfs_db (and call it from xfs_admin, too) where for now the only feature will be "separatepquota" one.
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 40 lines · Arkadiusz Miskiewicz
Quoted 7 lines · Christoph Hellwig
Thanks for doing this work Arek, I think it will be useful for some users and defintively is cleaners than what we have now.
Quoted 3 lines
I started doing that by splitting every group+project handling code into separate group and project one. Added superblock field for project quota. New feature flag (SEPARATEPQUOTA).
Quoted 8 lines
Ok.
Quoted 4 lines
If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
sb_gquotinointomp->m_sb.sb_pquotinowhich I think is enough to handle old fs (since new kernel operates on separate structures for project quota).Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
Yes. When trying to load existing project quota inode in
xfs_qm_init_quotainos()I'm doing:(quoted diff) +5
-0+ if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) { + ASSERT(!XFS_IS_GQUOTA_ON(mp)); + mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino; + mp->m_sb.sb_gquotino = 0; + }While
xfs_mod_sb()now does:/* * Filesystem has no separatepquota turned on, * so we need to store project quota in group * quota inode on disk. */ if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) { mp->m_sb.sb_gquotino = mp->m_sb.sb_pquotino; mp->m_sb.sb_pquotino = 0; } xfs_sb_to_disk(XFS_BUF_TO_SBP(bp), &mp->m_sb, fields);
Quoted 8 lines
[...]
/* * Restore original in-memory project quota inode state. */ if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) { mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino; mp->m_sb.sb_gquotino = 0; }
I don't think that is safe - we can have concurrent access to the in-core superblock (mp->m_sb) without locking the superblock, so something that races with xfs_mod_sb() looking up project quota could die a horrible death here.
The only time that you should need to do this juggling is when the quota inode changes. That is, when the XFS_SB_GQUOTINO field is set. Otherwise the field won't be modified and so we don't need to convert the values. That only occurs when quotas are being initialised (xfs_qm_qino_alloc()) during mount, so in that case there can't be any concurrent operations occurring. Hence swizzling the inode fields only when the XFS_SB_GQUOTINO filed is set should be safe.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Dave Chinner
Quoted 8 lines · Arkadiusz Miskiewicz
* Restore original in-memory project quota inode state. */ if (!xfs_sb_version_hasseparatepquota(&mp->m_sb)) { mp->m_sb.sb_pquotino = mp->m_sb.sb_gquotino; mp->m_sb.sb_gquotino = 0;
Quoted 6 lines
Quoted 1 line
}I don't think that is safe - we can have concurrent access to the in-core superblock (
mp->m_sb) without locking the superblock, so something that races withxfs_mod_sb()looking up project quota could die a horrible death here.
I was thinking about working on a copy, too, like this
+ xfs_sb_t sb_copy;
[...]
+ memcpy(&sb_copy, from, sizeof(sb_copy));
+ from_ptr = (xfs_caddr_t)&sb_copy;
+
+ if (!xfs_sb_version_hasseparatepquota(from) &&
+ (sb_copy.sb_qflags & XFS_PQUOTA_ACCT)) {
+ sb_copy.sb_gquotino = from->sb_pquotino;
+ sb_copy.sb_pquotino = 0;
+
+ }
Quoted 9 lines
The only time that you should need to do this juggling is when the quota inode changes. That is, when the
XFS_SB_GQUOTINOfield is set. Otherwise the field won't be modified and so we don't need to convert the values. That only occurs when quotas are being initialised (xfs_qm_qino_alloc()) during mount, so in that case there can't be any concurrent operations occurring. Hence swizzling the inode fields only when theXFS_SB_GQUOTINOfiled is set should be safe.
Unfortunately it turns out that I need to convert sb_qflags, too (see below) and sb_qflags seem to be changes in few different places, so xfs_mod_sb() looks a good place to do a conversion for this.
sb_qflags come into play. Old XFS_OQUOTA_CHKD and XFS_PQUOTA_CHKD are no longer used by kernel but existing filesystems can have these, so I'm going to translate old into new flags currently in xfs_sb_from_disk() and vice versa in xfs_mod_sb().
@@ -575,6 +588,26 @@ xfs_sb_from_disk(
to->sb_logsunit = be32_to_cpu(from->sb_logsunit);
to->sb_features2 = be32_to_cpu(from->sb_features2);
to->sb_bad_features2 = be32_to_cpu(from->sb_bad_features2);+ to->sb_pquotino = be64_to_cpu(from->sb_pquotino);
+
+ /*
+ * Convert old quota to separatepquota flags.
+ */
+ if (to->sb_qflags & XFS_OQUOTA_CHKD_NOTUSED) {
+ if (to->sb_qflags & XFS_GQUOTA_ACCT) {
+ to->sb_qflags &= ~XFS_OQUOTA_CHKD_NOTUSED;
+ to->sb_qflags |= XFS_GQUOTA_CHKD;
+ } else if (to->sb_qflags & XFS_PQUOTA_ACCT) {
+ to->sb_qflags &= ~XFS_PQUOTA_CHKD_NOTUSED;
+ to->sb_qflags |= XFS_PQUOTA_CHKD;
+ }
+ }
+
+ if (!xfs_sb_version_hasseparatepquota(to) &&
+ (to->sb_qflags & XFS_PQUOTA_ACCT)) {
+ to->sb_pquotino = to->sb_gquotino;
+ to->sb_gquotino = 0;
+ }
}
New flags are going to be like this:
#define XFS_UQUOTA_ENFD 0x0002 /* user quota limits enforced */ #define XFS_UQUOTA_CHKD 0x0004 /* quotacheck run on usr quotas */ #define XFS_PQUOTA_ACCT 0x0008 /* project quota accounting ON */ -#define XFS_OQUOTA_ENFD 0x0010 /* other (grp/prj) quota limits enforced */ -#define XFS_OQUOTA_CHKD 0x0020 /* quotacheck run on other (grp/prj) quotas */ +#define XFS_OQUOTA_ENFD_NOTUSED 0x0010 /* other (grp/prj) quota limits enforced (NO LONGER USED)*/ +#define XFS_OQUOTA_CHKD_NOTUSED 0x0020 /* quotacheck run on other (grp/prj) quotas (NO LONGER USED)*/ #define XFS_GQUOTA_ACCT 0x0040 /* group quota accounting ON */ +#define XFS_GQUOTA_ENFD 0x0080 /* group quota limits enforced */ +#define XFS_GQUOTA_CHKD 0x0100 /* group quota accoungint ON */ +#define XFS_PQUOTA_CHKD 0x0200 /* quotacheck run on project quotas */ +#define XFS_PQUOTA_ENFD 0x0400 /* project quota limits enforced */
Quoted 3 lines
Cheers,
Dave.
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 13 lines · Christoph Hellwig
Thanks for doing this work Arek, I think it will be useful for some users and defintively is cleaners than what we have now.
Quoted 3 lines
I started doing that by splitting every group+project handling code into separate group and project one. Added superblock field for project quota. New feature flag (SEPARATEPQUOTA).
Ok.
Quoted 3 lines
If old filesystem (for SEPARATEQUOTA) is mounted then I'll simply load
sb_gquotinointomp->m_sb.sb_pquotinowhich I think is enough to handle old fs (since new kernel operates on separate structures for project quota).
Quoted 8 lines
Do you mean an old filesystem without the SEPARATEQUOTA bit set here?
Quoted 2 lines
New filesystem (without quota inodes allocated yet), when mounted, gets new quota inode allocated for project quota and SEPARATEQUOTA feature bit set.
I think we need the SEPARATEQUOTA bit set explicitly by mkfs.xfs/xfs_admin. But once we start using project quotas with the bit set the above sounds fine.
Agreed - we can't set the bit by default until the kernel has suppoerted the feature for some time. A user who upgrades from a kernel that doesn't support separate quotas can't downgrade if the bit is automatically set if there are problems (and the problems requiring downgrade might be completely unrelated to XFS).
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs