trying to avoid a lengthy quotacheck by deleting all quota data

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:
-- disable
-- off
-- remove
- remount the drive with -o prjquota, hoping that there will not be a
quotacheck, because we've deleted all the old quota data
- run a script gradually restore all the quotas, one by one and in
good time, from our own external backups (we've got the quotas in a database basically).

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with -o prjquota but that quota accounting is *not* active)
- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

Answers on a postcard, received with the utmost gratitude.

Rgds,
Harry + the PythonAnywhere team.

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:
-- disable
-- off
-- remove
- remount the drive with -o prjquota, hoping that there will not be a quotacheck, because we've deleted all the old quota data
- run a script gradually restore all the quotas, one by one and in good time, from our own external backups (we've got the quotas in a database basically).

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with -o prjquota but that quota accounting is *not* active)
- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

Hi Harry,

Quoted 7 lines · Harry

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

The 'noquota' mount option will disable quotacheck at mount time. That may do what you need.

Quoted 12 lines

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:

   -- disable
   -- off
   -- remove

- remount the drive with -o prjquota, hoping that there will not be a quotacheck, because we've deleted all the old quota data
- run a script gradually restore all the quotas, one by one and in good time, from our own external backups (we've got the quotas in a database basically).

Quoted 6 lines

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with -o prjquota but that quota accounting is *not* active)
- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

Quotacheck is implemented by truncating the quota inode and then rebuilding the dquots from scratch as it traverses all the inodes in the filesystem. Unfortunately the filesystem needs to be idle during this process or the accounting could be incorrect, so there is no gradual option for restoring quotas.

Regards,
Ben

Hi Ben, thanks for replying.

We're using project quotas, and we'd be prepared to wipe the slate clean and start by saying there are no projects and no quotas at all, to start with. Is there still no way of starting from scratch and avoiding that quotacheck?

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:
-- disable
-- off
-- remove
- remount the drive with -o prjquota, hoping that there will not be a quotacheck, because we've deleted all the old quota data

- run a script gradually restore all the quotas, one by one and in good time, from our own external backups (we've got the quotas in a database basically).

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with

-o prjquota but that quota accounting is *not* active)

- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

Quoted 7 lines · Harry

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

What tests are you performing to suggest a quotacheck of a small filesystem will take hours? (yes, 2TB is a *small* filesystem).

(xfs_info, df -i, df -h, storage hardware, etc are all relevant here).

Quoted 8 lines

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:

   -- disable
   -- off
   -- remove

- remount the drive with -o prjquota, hoping that there will not be a quotacheck, because we've deleted all the old quota data

Mounting with a quota enabled *forces* a quota check if quotas aren't currently enabled. You cannot avoid it; it's the way quota consistency is created.

Quoted 3 lines

- run a script gradually restore all the quotas, one by one and in good time, from our own external backups (we've got the quotas in a database basically).

Can't be done - quotas need to be consistent with what is currently on disk, not what you have in a backup somewhere.

Quoted 3 lines

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with

mount with quotas on and turn them off via xfs_quota,i or mount without quota options at all. Then run the remove command in xfs_quota.

Quoted 1 line

-o prjquota but that quota accounting is *not* active)

Not possible.

Quoted 2 lines

- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

No.

Cheers,

Dave.

Thanks Dave,

* The main filesystem is currently online and seems ok, but quotas are not active.
* We want to estimate how long the quotacheck will take when we reboot/remount
* We're even a bit worried the disk might be in a broken state, such that the quotacheck won't actually complete successfully at all.

A brief description of our setup:
- we're on AWS
- using mdadm to make a raid array out of 8x 200GB SSD EBS drives (and lvm)
- we're using DRBD to make a live backup of all writes to another instance with a similar raid array

We're not doing our experiments on our live system. Instead, we're using the drives from the DRBD target system. We take DRBD offline, so it's no longer writing, then we take snapshots of the drives, then remount those elsewhere so we can experiment without disturbing the live system.

We've managed to mount the backup drives ok, with the 'noquota' option. Files look ok. But, so far, we haven't been able to get a quotacheck to complete. We've waited 12 hours+. Do you think it's possible DRBD is giving us copies of the live disks that are inconsistent somehow?

How can we reassure ourselves that this live disk *will* mount successfully if we reboot the machine, and can we estimate how long it will take?

/mount | grep log_storage/
/dev/drbd0 on /mnt/log_storage type xfs
(rw,prjquota,allocsize=64k,_netdev)

/df -i /mnt/log_storage//
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/drbd0 938210704 72929413 865281291 8% /mnt/log_storage

/df -h /mnt/log_storage//
Filesystem Size Used Avail Use% Mounted on
/dev/drbd0 1.6T 1.4T 207G 88% /mnt/log_storage

/xfs_info ///mnt/log_storage////
/<lots of errors re: cannot find mount point path xyz>/
meta-data=/dev/drbd0 isize=256 agcount=64,
agsize=6553600 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=418906112,
imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=12800, version=2
= sectsz=512 sunit=0 blks,
lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

The missing paths errors are, I think, from folders we've deleted but not yet removed from the projid/projects folders. I *think* they're a red herring here.

We've also tried running xfs_repair on the backup drives. It takes about 3 hours, and shows a lot of errors about incorrect directory flags on inodes. here's one from the bottom of the log of a recent attempt:

directory flags set on non-directory inode 268702898

rgds,
Confused in London.

Quoted 44 lines · Dave Chinner

Quoted 7 lines · Harry

Hi there,

We've got a moderately large disk (~2TB) into an inconsistent state, such that it's going to want a quotacheck the next time we mount it (it's currently mounted with quota accounting inactive). Our tests suggest this is going to take several hours, and cause an outage we can't afford.

What tests are you performing to suggest a quotacheck of a small filesystem will take hours? (yes, 2TB is a *small* filesystem).

(xfs_info, df -i, df -h, storage hardware, etc are all relevant here).

Quoted 8 lines

We're wondering whether there's a 'nuke the site from orbit' option that will let us avoid it. The plan would be to:
- switch off quotas and delete them completely, using the commands:

   -- disable
   -- off
   -- remove

- remount the drive with -o prjquota, hoping that there will not be a quotacheck, because we've deleted all the old quota data

Mounting with a quota enabled *forces* a quota check if quotas aren't currently enabled. You cannot avoid it; it's the way quota consistency is created.

Quoted 3 lines

- run a script gradually restore all the quotas, one by one and in good time, from our own external backups (we've got the quotas in a database basically).

Can't be done - quotas need to be consistent with what is currently on disk, not what you have in a backup somewhere.

Quoted 3 lines

So the questions are:
- is there a way to remove all quota information from a mounted drive? (the current mount status seems to be that it tried to mount it with

mount with quotas on and turn them off via xfs_quota,i or mount without quota options at all. Then run the remove command in xfs_quota.

Quoted 1 line

-o prjquota but that quota accounting is *not* active)

Not possible.

Quoted 2 lines

- will it work and let us remount the drive with -o prjquota without causing a quotacheck?

Quoted 5 lines

No.

Cheers,

Dave.

Rgds,
Harry + the PythonAnywhere team.

Signature · 12 lines

Harry Percival
Developer
[email protected]

PythonAnywhere - a fully browser-based Python development and hosting environment
<http://www.pythonanywhere.com/>

PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK

Update -- so far, we've not managed to gain any confidence that we'll ever be able to re-mount that disk. The general consensus seems to be to fish all the data off the disk using rsync, and then move off XFS to ext4.

Not a very helpful message for y'all to hear, I know. But if it's any help in prioritising your future work, i think the dealbreaker for us was the inescapable quotacheck on mount, which means that any time a fileserver goes down unexpectedly, we have an unavoidable, indeterminate-but-long period of downtime...

hp

* We're even a bit worried the disk might be in a broken state, such that the quotacheck won't actually complete successfully at all.

We take DRBD offline, so it's no longer writing, then we take snapshots of the drives, then remount those elsewhere so we can experiment without disturbing the live system.

Quoted 5 lines · Harry

Update -- so far, we've not managed to gain any confidence that we'll ever be able to re-mount that disk. The general consensus seems to be to fish all the data off the disk using rsync, and then move off XFS to ext4.

Quoted 7 lines

Not a very helpful message for y'all to hear, I know. But if it's any help in prioritising your future work, i think the dealbreaker for us was the inescapable quotacheck on mount, which means that any time a fileserver goes down unexpectedly, we have an unavoidable, indeterminate-but-long period of downtime...

hp

What you decide to use is up to you of course, and causes us no heartbreak. :) But I think you fundamentally misunderstand the situation; an unexpected fileserver failure should not result in a lengthy quotacheck on xfs, because xfs quota is journaled, and will simply be replayed along with the rest of the log.

I honestly don't know what has led you to the conclusion that remounting the filesystem will lead to any quotacheck at all, let alone a lengthy one.

Quoted 2 lines

* We're even a bit worried the disk might be in a broken state, such that the quotacheck won't actually complete successfully at all.

If your disk is broken, that's not a filesystem issue. It seems possible that whatever drbd manipulation you're doing is causing an issue, but because you haven't really explained it in detail, I don't know.

Quoted 3 lines

We take DRBD offline, so it's no longer writing, then we take snapshots of the drives, then remount those elsewhere so we can experiment without disturbing the live system.

Did you quiesce the filesystem first with i.e. xfs_freeze?

So far this thread has been long on prose and speculation, and short on actual analysis, log messages, etc. Feel free to use ext4 or whatever suits you, but given that nothing in this thread has implicated misbehavior by xfs, I don't think that switching filesystems will solve the perceived problem.

-Eric

Thanks for the reply Eric.

One of our problems is that we're limited in terms of what manipulations we can apply to the live system, and so instead we've been running our experiments against the backup system, and you're quite right that DRBD may be introducing some weirdness of its own, so those experiments may not be safe to draw conclusions from.

Here's what we know about the live system
-> it had an outage, equivalent to having its power cable yanked, or doing an 'echo b > /proc/sysrq-trigger'
-> when it came back, it decided to mount the drive without quotas. -> we saw a message in syslog saying " Failed to initialize disk quotas" -> last time we had to run a quotacheck (several months ago) it took about 2 hours.

We can repro the quotacheck issue on our test clusters, as follows:
-> kick off a job that writes to the disk
-> hard reboot with "echo b > /proc/sysrq-trigger"
-> on next boot, see "Failed to initialize disk quotas" message, xfs mounts without quotas
-> soft reboot with "reboot"
-> on next boot, see "Quotacheck needed: Please wait." message. -> Quotacheck completes some time later.

So our best-case scenario is that, next time we reboot, we'll have an outage of about 2 hours. And our paranoid worst-case scenario, induced by our experiments with our drbd backup drives, are that the disk will actually turn out not to be mountable at all.

is that "quotacheck always required after hard reboot" behaviour that we're observing something you expected? you seemed to be saying that the fact that quota are journaled should mean it's not needed?

HP

Quoted 39 lines · Eric Sandeen

Quoted 5 lines · Harry

Update -- so far, we've not managed to gain any confidence that we'll ever be able to re-mount that disk. The general consensus seems to be to fish all the data off the disk using rsync, and then move off XFS to ext4.

Quoted 7 lines

Not a very helpful message for y'all to hear, I know. But if it's any help in prioritising your future work, i think the dealbreaker for us was the inescapable quotacheck on mount, which means that any time a fileserver goes down unexpectedly, we have an unavoidable, indeterminate-but-long period of downtime...

hp

What you decide to use is up to you of course, and causes us no heartbreak. :) But I think you fundamentally misunderstand the situation; an unexpected fileserver failure should not result in a lengthy quotacheck on xfs, because xfs quota is journaled, and will simply be replayed along with the rest of the log.

I honestly don't know what has led you to the conclusion that remounting the filesystem will lead to any quotacheck at all, let alone a lengthy one.

Quoted 2 lines

* We're even a bit worried the disk might be in a broken state, such that the quotacheck won't actually complete successfully at all.

If your disk is broken, that's not a filesystem issue. It seems possible that whatever drbd manipulation you're doing is causing an issue, but because you haven't really explained it in detail, I don't know.

Quoted 3 lines

We take DRBD offline, so it's no longer writing, then we take snapshots of the drives, then remount those elsewhere so we can experiment without disturbing the live system.

Did you quiesce the filesystem first with i.e. xfs_freeze?

So far this thread has been long on prose and speculation, and short on actual analysis, log messages, etc. Feel free to use ext4 or whatever suits you, but given that nothing in this thread has implicated misbehavior by xfs, I don't think that switching filesystems will solve the perceived problem.

Quoted 1 line

-Eric

Rgds,
Harry + the PythonAnywhere team.

Signature · 12 lines

Harry Percival
Developer
[email protected]

PythonAnywhere - a fully browser-based Python development and hosting environment
<http://www.pythonanywhere.com/>

PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK

Thanks for the reply Eric.

One of our problems is that we're limited in terms of what manipulations we can apply to the live system, and so instead we've been running our experiments against the backup system, and you're quite right that DRBD may be introducing some weirdness of its own, so those experiments may not be safe to draw conclusions from.

Here's what we know about the live system
-> it had an outage, equivalent to having its power cable yanked, or doing an 'echo b > /proc/sysrq-trigger' -> when it came back, it decided to mount the drive without quotas.
-> we saw a message in syslog saying " Failed to initialize disk quotas"
-> last time we had to run a quotacheck (several months ago) it took about 2 hours.

We can repro the quotacheck issue on our test clusters, as follows:
-> kick off a job that writes to the disk
-> hard reboot with "echo b > /proc/sysrq-trigger"
-> on next boot, see "Failed to initialize disk quotas" message, xfs mounts without quotas -> soft reboot with "reboot"
-> on next boot, see "Quotacheck needed: Please wait." message.
-> Quotacheck completes some time later.

So our best-case scenario is that, next time we reboot, we'll have an outage of about 2 hours. And our paranoid worst-case scenario, induced by our experiments with our drbd backup drives, are that the disk will actually turn out not to be mountable at all.

is that "quotacheck always required after hard reboot" behaviour that we're observing something you expected? you seemed to be saying that the fact that quota are journaled should mean it's not needed?

Quoted 22 lines · Harry

Thanks for the reply Eric.

One of our problems is that we're limited in terms of what manipulations we can apply to the live system, and so instead we've been running our experiments against the backup system, and you're quite right that DRBD may be introducing some weirdness of its own, so those experiments may not be safe to draw conclusions from.

Here's what we know about the live system
-> it had an outage, equivalent to having its power cable yanked, or doing an 'echo b > /proc/sysrq-trigger' -> when it came back, it decided to mount the drive without quotas.
-> we saw a message in syslog saying " Failed to initialize disk quotas"
-> last time we had to run a quotacheck (several months ago) it took about 2 hours.

We can repro the quotacheck issue on our test clusters, as follows:
-> kick off a job that writes to the disk
-> hard reboot with "echo b > /proc/sysrq-trigger"
-> on next boot, see "Failed to initialize disk quotas" message, xfs mounts without quotas -> soft reboot with "reboot"
-> on next boot, see "Quotacheck needed: Please wait." message.
-> Quotacheck completes some time later.

Quoted 8 lines

So our best-case scenario is that, next time we reboot, we'll have an outage of about 2 hours. And our paranoid worst-case scenario, induced by our experiments with our drbd backup drives, are that the disk will actually turn out not to be mountable at all.

is that "quotacheck always required after hard reboot" behaviour that we're observing something you expected? you seemed to be saying that the fact that quota are journaled should mean it's not needed?

In general, that's correct. It's not clear why "Failed to initialize disk quotas" appeared; that seems closer to the root cause. But again, we don't have your full logs to look at, I don't know if anything else offers a clue. (For that matter, we don't even know what kernel version you're on...)

here, on a recent 4.0-rc1 kernel:

# mount -o quota /dev/sdc6 /mnt/test
# cp -aR /lib/modules/ /mnt/test
# echo b > /proc/sysrq-trigger

[152807.209688] sysrq: SysRq : Resetting

...
<reboots>

# mount -o quota /dev/sdc6 /mnt/test
# dmesg | tail -n 3
[   90.822601] XFS (sdc6): Mounting V4 Filesystem
[   90.921346] XFS (sdc6): Starting recovery (logdev: internal)
[   93.399133] XFS (sdc6): Ending recovery (logdev: internal)
#

-Eric

We're on 3.13.0-39 (Ubuntu Trusty).

If you're interested in looking into it further, I'd be happy to provide any extra info you'd like?

But just to make sure I'm not wasting any of your time -- I think the team have pretty much decided to make the switch no matter what. The quotacheck issue is one thing, but actually the switch to ext4 simplifies lots of other aspects of our quota system (one of the reasons we picked nfs was to be able to use project quotas, but it turns out we don't need them any more, so user quotas are simpler...)

Quoted 3 lines · Harry

We're on 3.13.0-39 (Ubuntu Trusty).

If you're interested in looking into it further, I'd be happy to provide any extra info you'd like?

Well, not really. It all works here, and you have an ... interesting setup, so if you've decided that somehow ext4 will save you from quotachecks in the future, I'm not going to dig a lot further here.

I did already ask for logs, which might tell us why the original quota init failed, but ...

Quoted 6 lines

But just to make sure I'm not wasting any of your time -- I think the team have pretty much decided to make the switch no matter what. The quotacheck issue is one thing, but actually the switch to ext4 simplifies lots of other aspects of our quota system (one of the reasons we picked nfs was to be able to use project quotas, but it turns out we don't need them any more, so user quotas are simpler...)

... it sounds like you've already picked your solution to this AFAICT not-well-understood problem.

*shrug* knock yourself out. :) You should use what works best meets your needs, of course.

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Here's the syslog, if you're curious.

http://pastebin.com/raw.php?i=kKvWJcze

Search for "Failed to initialize"

So your best guess is that it's the drbd layer that's causing the quotacheck? Out of curiosity, i may try mounting a non-drbd drive with xfs, and seeing if we can still repro the hard-reboot-causes-quotacheck thing... Unless you think it's just an old behaviour that's more to do with the version of the kernel we're using?

HP

Quoted 5 lines · Harry

Here's the syslog, if you're curious.

http://pastebin.com/raw.php?i=kKvWJcze

Search for "Failed to initialize"

Ok, there is no other message offering more info, sadly.

Quoted 6 lines

So your best guess is that it's the drbd layer that's causing the quotacheck? Out of curiosity, i may try mounting a non-drbd drive with xfs, and seeing if we can still repro the hard-reboot-causes-quotacheck thing... Unless you think it's just an old behaviour that's more to do with the version of the kernel we're using?

I really don't have a good guess at this point..... oh, wait, finally, a bell goes off:

commit 5ef828c4152726f56751c78ea844f08d2b2a4fa3

Author: Eric Sandeen <[email protected]>
Date:   Mon Aug 4 11:35:44 2014 +1000

xfs: avoid false quotacheck after unclean shutdown

The commit

83e782e xfs: Remove incore use of XFS_OQUOTA_ENFD and XFS_OQUOTA_CHKD

added a new function xfs_sb_quota_from_disk() which swaps on-disk XFS_OQUOTA_* flags for in-core XFS_GQUOTA_* and XFS_PQUOTA_* flags after the superblock is read.

However, if log recovery is required, the superblock is read again, and the modified in-core flags are re-read from disk, so we have XFS_OQUOTA_* flags in memory again. This causes the XFS_QM_NEED_QUOTACHECK() test to be true, because the XFS_OQUOTA_CHKD is still set, and not XFS_GQUOTA_CHKD or XFS_PQUOTA_CHKD.

Change xfs_sb_from_disk to call xfs_sb_quota_from disk and always convert the disk flags to in-memory flags.

Add a lower-level function which can be called with "false" to not convert the flags, so that the sb verifier can verify exactly what was on disk, per Brian Foster's suggestion.

Reported-by Cyril B. <[email protected]>

Signed-off-by Eric Sandeen <[email protected]>

83e782e went in at v3.11; the above commit hit v3.17, so it was broken for a while.

I still can't explain the "quota init failed" bit, but the above probably explains the unexpected quotacheck problem.

-Eric

Quoted 1 line

HP

Glad we managed to nail down a probable culprit! Here's hoping
Debian and Ubuntu pull in a new kernel :)

In other news, any advice on running this

xfstests:src/bstat

command as a way of estimating how long a quotacheck will take?

Would it still be a useful estimator? Do you think it would significantly affect the performance of a disk that's under fairly heavy use?

Quoted 8 lines · Harry Percival

Glad we managed to nail down a probable culprit! Here's hoping
Debian and Ubuntu pull in a new kernel :)

In other news, any advice on running this

xfstests:src/bstat

command as a way of estimating how long a quotacheck will take?

It will give you an idea - quotacheck uses bulkstat, too.

Quoted 3 lines

Would it still be a useful estimator? Do you think it would significantly affect the performance of a disk that's under fairly heavy use?

Of course. Bulkstat drives the disks as hard as they will go.

Cheers,

Dave.

Here's the syslog, if you're curious.

http://pastebin.com/raw.php?i=kKvWJcze

Search for "Failed to initialize"