Skip to contentSkip to conversation

Performance problem with multiple parallel rm -rf's

DISCUSSION Jens Rosenboom 4 participants 4 messages Dec 2, 2009 – Dec 3, 2009

lore
Share & tools
lore
#0 lore raw

On a large 13TB XFS volume that is being used for backups, I am seeing bad performance if multiple "rm -rf" processes are running in parallel. The backups are being done with rsnapshot and the first operation it does is removing the oldest snapshot. A single rsnapshot does this in reasonable time, but if four jobs are started at the same time, all their rm processes run for hours without making much progress.

This seems to be related to the planned optimizations in

http://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead

Are there any other tuning options I might try? I'm already using "noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k" as mount options and did enable lazy_counters for the fs.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

This I think is not dependent on XFS. Most File systems tend to give bad performance when multiple rm -rf is being run. I think the Linux(XFS or any) are object based and tend to lock on the filesystem, but donot have any exact knowledge
of the Level at which it is being held.

Thanks,
Sujit

Quoted 11 lines · Jens Rosenboom

On a large 13TB XFS volume that is being used for backups, I am seeing bad performance if multiple "rm -rf" processes are running in parallel. The backups are being done with rsnapshot and the first operation it does is removing the oldest snapshot. A single rsnapshot does this in reasonable time, but if four jobs are started at the same time, all their rm processes run for hours without making much progress.

This seems to be related to the planned optimizations in

http://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead

Quoted 9 lines

Are there any other tuning options I might try? I'm already using "noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k" as mount options and did enable lazy_counters for the fs.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
In reply to Jens Rosenboom
Quoted 4 lines · Jens Rosenboom

On a large 13TB XFS volume that is being used for backups, I am seeing bad performance if multiple "rm -rf" processes are running in parallel. The backups are being done with rsnapshot and the first operation it does is removing the oldest snapshot. A single rsnapshot does this in reasonable time, but if four jobs are started at the same time, all their rm processes run for hours without making much progress.

This seems to be related to the planned optimizations in

Quoted 9 lines

http://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead

Are there any other tuning options I might try? I'm already using "noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k" as mount options and did enable lazy_counters for the fs.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

For faster rm performance you need a bigger log (128m-256mb), that is shown to increase delete performance. But I am not sure there is a way to change the size of it once the array has been created. An alternative may be creating another volume for a log.

Also: nodiratime is not needed, as it is implied by noatime.

Justin.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to Jens Rosenboom
Quoted 10 lines · Jens Rosenboom

On a large 13TB XFS volume that is being used for backups, I am seeing bad performance if multiple "rm -rf" processes are running in parallel. The backups are being done with rsnapshot and the first operation it does is removing the oldest snapshot. A single rsnapshot does this in reasonable time, but if four jobs are started at the same time, all their rm processes run for hours without making much progress.

This seems to be related to the planned optimizations in

http://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead

Not directly, I think. More likely is the effect of cold caches on the inode read rate.

That is, if you are running cold-cache 'rm -rf' operations, there is a substantial amount of *read* IO executed to pull the inodes into memory before they are unlinked. (i.e. an unlink is roughly 1 read IO and two write IOs).

If you are doing multiple cold-cache 'rm -rf' in parallel, you will be causing more disk seeks for reading the inodes you are trying to unlink, and as such this will slow down the unlink rate as the unlink can only go as fast as the inodes can be read off disk.

Effectively there is not much you can do about this - you could try doing a traversal of the old snapshot first (e.g
ls -lR <snapshot> > /dev/null) to get the cache populated as fast as possible before doing the unlink traversal, but that requires you have plenty of memory available (i.e. to hold more inodes than multiple parallel snapshot traversals will read).

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs