quotacheck speed
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Arkadiusz Miśkiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
How long does a repair vs quotacheck of that same filesystem take? repair has to iterate the inodes 2-3 times, so if that is faster than quotacheck, then that is really important to know....
Quoted 2 lines
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
quotacheck makes the assumption that it is run on an otherwise idle filesystem that nobody is accessing. Well, what it requires is that nobody is modifying it. What we could do is bring the filesystem up in a frozen state so that read-only access could be made but modifications are blocked until the quotacheck is completed.
Also, quotacheck uses the bulkstat code to iterate all the inodes quickly. Improvements in bulkstat speed will translate directly into faster quotachecks. quotacheck could probably drive bulkstat in a parallel manner to do the quotacheck faster, but that assumes that the underlying storage is not already seek bound. What is the utilisation of the underlying storage and CPU while quotacheck is running?
Otherwise, bulkstat inode prefetching could be improved like xfs_repair was to look at inode chunk density and change IO patterns and to slice and dice large IO buffers into smaller inode buffers. We can actually do that efficiently now that we don't use the page cache for metadata caching. If repair is iterating inodes faster than bulkstat, then this optimisation will be the reason and having that data point is very important....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Dave Chinner
Quoted 5 lines · Arkadiusz Miśkiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
How long does a repair vs quotacheck of that same filesystem take? repair has to iterate the inodes 2-3 times, so if that is faster than quotacheck, then that is really important to know....
Don't have exact times but looking at nagios and dmesg it took about:
repair ~20 minutes, quotacheck ~10 minutes (it's 800GB of maildirs).
Quoted 10 lines
Quoted 4 lines
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
quotacheck makes the assumption that it is run on an otherwise idle filesystem that nobody is accessing. Well, what it requires is that nobody is modifying it. What we could do is bring the filesystem up in a frozen state so that read-only access could be made but modifications are blocked until the quotacheck is completed.
Read-only is better than no access at all. I was hoping that there is a way to make quotacheck being recalculated on the fly with taking all write accesses that happen in meantime into account.
Quoted 7 lines
Also, quotacheck uses the bulkstat code to iterate all the inodes quickly. Improvements in bulkstat speed will translate directly into faster quotachecks. quotacheck could probably drive bulkstat in a parallel manner to do the quotacheck faster, but that assumes that the underlying storage is not already seek bound. What is the utilisation of the underlying storage and CPU while quotacheck is running?
Will try to gather more information then.
Quoted 12 lines
Otherwise, bulkstat inode prefetching could be improved like
xfs_repairwas to look at inode chunk density and change IO patterns and to slice and dice large IO buffers into smaller inode buffers. We can actually do that efficiently now that we don't use the page cache for metadata caching. If repair is iterating inodes faster than bulkstat, then this optimisation will be the reason and having that data point is very important....Cheers,
Dave.
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Arkadiusz Miśkiewicz
Quoted 7 lines · Dave Chinner
Quoted 5 lines · Arkadiusz Miśkiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
Quoted 6 lines
Quoted 3 lines
How long does a repair vs quotacheck of that same filesystem take? repair has to iterate the inodes 2-3 times, so if that is faster than quotacheck, then that is really important to know....
Don't have exact times but looking at nagios and dmesg it took about:
repair ~20 minutes, quotacheck ~10 minutes (it's 800GB of maildirs).
Ok. Seems like repair is a little faster than quotacheck, then.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
quotacheck makes the assumption that it is run on an otherwise idle filesystem that nobody is accessing. Well, what it requires is that nobody is modifying it. What we could do is bring the filesystem up in a frozen state so that read-only access could be made but modifications are blocked until the quotacheck is completed.
Quoted 3 lines
Read-only is better than no access at all. I was hoping that there is a way to make quotacheck being recalculated on the fly with taking all write accesses that happen in meantime into account.
The problem is that we'd need to keep two sets of dquots in memory for each quota user while the quota check is being done - one to track modifications being made, and the other to track quotacheck progress. It gets complex quite rapidly then - where do we account changes to an inode that hasn't been quota-checked yet? Or vice versa? How do we even know if an inode has been quota checked?
THese are probably all things that can be solved, but I get lost in the complexity when just thinking about it....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Arkadiusz Mi??kiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair. It's fairly simple given that quotacheck simply walks all inodes and adds their space usage to the correct user/group/project, and given that repair already walks all inodes, and checks their block maps it does most of that work already. The only downside would be that the memory usage of repair increases a bit by keeping the dquots in memoryb, but even for your 130000 dquot setup that would add about 100 bytes * 130000 please a bit of in-memory metadata (less than 20MB total) of memory usage, so it probably is a good tradeoff.
In what cases do you regularly run quotacheck when you did not do a repair first?
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines
Quoted 3 lines
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
For tight downtime minimization requirements wishful thinking is not a strategy: whole filetree metadata scans are not cheap. If you require fast scan of metadata of the whole filetree, ensure filetrees don't have a lot of metadata (or fund the development of a parallel whole tree metadata scanner).
Also 10 minutes is not that long; file system checks/repairs can take days or weeks.
Quoted 5 lines
Quoted 3 lines
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
From 'man 8 quotacheck':
"It is strongly recommended to run quotacheck with quotas turned off for the filesys-
tem. Otherwise, possible damage or loss to data in the quota files can result. It is also unwise to run quotacheck on a live filesystem as actual usage may change during
the scan. To prevent this, quotacheck tries to remount the filesystem read-only
before starting the scan. After the scan is done it remounts the filesystem read- write. You can disable this with option -m. You can also make quotacheck ignore the failure to remount the filesystem read-only with option -M."
Accoridng to this the only consequence of parallel running of 'quotacheck' is a somewhat inaccurate accounting of quotas.
Quoted 2 lines
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair.
Probably this should be 'xfs_check' more than 'xfs_repair'...
[ ... ]
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Arkadiusz Mi??kiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
Quoted 6 lines
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair. It's fairly simple given that quotacheck simply walks all inodes and adds their space usage to the correct user/group/project, and given that repair already walks all inodes, and checks their block maps it does most of that work already.
That would be interesting and probably make
Quoted 9 lines
The only downside would be that the memory usage
of repair increases a bit by keeping the dquots in memoryb, but even for your 130000 dquot setup that would add about 100 bytes * 130000 please a bit of in-memory metadata (less than 20MB total) of memory usage, so it probably is a good tradeoff.In what cases do you regularly run quotacheck when you did not do a repair first?
I don't initiate quotacheck manually. AFAIK internal xfs quotacheck happens in two cases here:
1) repair->mount
2) filesystem has quotacheck done properly some time ago -> umount -> mount-
Quoted 1 line
oops/reset/something like that happens while mounting -> new mount
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 28 lines · Arkadiusz Miśkiewicz
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Arkadiusz Mi??kiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
Quoted 6 lines
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair. It's fairly simple given that quotacheck simply walks all inodes and adds their space usage to the correct user/group/project, and given that repair already walks all inodes, and checks their block maps it does most of that work already.
That would be interesting and probably make
Quoted 7 lines
The only downside would be that the memory usage
of repair increases a bit by keeping the dquots in memoryb, but even for your 130000 dquot setup that would add about 100 bytes * 130000 please a bit of in-memory metadata (less than 20MB total) of memory usage, so it probably is a good tradeoff.
Quoted 8 lines
Quoted 2 lines
In what cases do you regularly run quotacheck when you did not do a repair first?
I don't initiate quotacheck manually. AFAIK internal xfs quotacheck happens in two cases here:
1)repair->mount
2) filesystem has quotacheck done properly some time ago -> umount -> mount-Quoted 1 line
oops/reset/something like that happens while mounting -> new mount
So you'd like both quotacheck to be sped up and repair to do it as well? ;)
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 38 lines · Dave Chinner
Quoted 37 lines · Arkadiusz Miśkiewicz
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Arkadiusz Mi??kiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
Quoted 6 lines
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair. It's fairly simple given that quotacheck simply walks all inodes and adds their space usage to the correct user/group/project, and given that repair already walks all inodes, and checks their block maps it does most of that work already.
That would be interesting and probably make
Quoted 9 lines
The only downside would be that the memory usage
of repair increases a bit by keeping the dquots in memoryb, but even for your 130000 dquot setup that would add about 100 bytes * 130000 please a bit of in-memory metadata (less than 20MB total) of memory usage, so it probably is a good tradeoff.In what cases do you regularly run quotacheck when you did not do a repair first?
I don't initiate quotacheck manually. AFAIK internal xfs quotacheck happens in two cases here:
1)repair->mount
2) filesystem has quotacheck done properly some time ago -> umount -> mount-
Quoted 4 lines
Quoted 1 line
Quoted 1 line
oops/reset/something like that happens while mounting -> new mount
So you'd like both quotacheck to be sped up and repair to do it as well? ;)
Well, 1) is happening much more often than 2) :-)
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 39 lines · Arkadiusz Miśkiewicz
Quoted 38 lines · Dave Chinner
Quoted 37 lines · Arkadiusz Miśkiewicz
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Arkadiusz Mi??kiewicz
Hi,
When mounting 800GB filesystem (after repair for example) here quotacheck takes 10 minutes. Quite long time that adds to whole time of filesystem downtime (repair + quotacheck).
I wonder if quotacheck can be somehow improved or done differently like doing it in parallel with normal fs usage (so there will be no downtime) ?
Quoted 6 lines
I think the best idea to improve the performance in case you did a repair is to integrate the quotacheck code into repair. It's fairly simple given that quotacheck simply walks all inodes and adds their space usage to the correct user/group/project, and given that repair already walks all inodes, and checks their block maps it does most of that work already.
That would be interesting and probably make
Quoted 9 lines
The only downside would be that the memory usage
of repair increases a bit by keeping the dquots in memoryb, but even for your 130000 dquot setup that would add about 100 bytes * 130000 please a bit of in-memory metadata (less than 20MB total) of memory usage, so it probably is a good tradeoff.In what cases do you regularly run quotacheck when you did not do a repair first?
I don't initiate quotacheck manually. AFAIK internal xfs quotacheck happens in two cases here:
1)repair->mount
2) filesystem has quotacheck done properly some time ago -> umount -> mount-
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
oops/reset/something like that happens while mounting -> new mount
So you'd like both quotacheck to be sped up and repair to do it as well? ;)
Well, 1) is happening much more often than 2) :-)
Oh, and one more scenario. Running system, sysrq u, s, b -> new boot, mount -> quotacheck runs.
Does it need to run in such case?
Signature · 7 lines
Arkadiusz Miśkiewicz PLD/Linux Team arekm / maven.pl http://ftp.pld-linux.org/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 18 lines · Arkadiusz Miśkiewicz
Quoted 17 lines · Arkadiusz Miśkiewicz
Quoted 15 lines · Dave Chinner
Quoted 11 lines · Arkadiusz Miśkiewicz
Quoted 2 lines · Christoph Hellwig
In what cases do you regularly run quotacheck when you did not do a repair first?
I don't initiate quotacheck manually. AFAIK internal xfs quotacheck happens in two cases here:
1)repair->mount
2) filesystem has quotacheck done properly some time ago -> umount -> mount-Quoted 1 line
oops/reset/something like that happens while mounting -> new mount
So you'd like both quotacheck to be sped up and repair to do it as well? ;)
Quoted 6 lines
Quoted 1 line
Well, 1) is happening much more often than 2) :-)
Oh, and one more scenario. Running system, sysrq u, s, b -> new boot, mount -> quotacheck runs.
Does it need to run in such case?
That's no different to an unclean shutdown. I'm not sure why that would trigger a quotacheck. Anything inthe log that might indicate why it started a quotacheck?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs