Skip to contentSkip to conversation

Deleting files with extended attributes is dead slow

DISCUSSION Bernd Schubert 3 participants 8 messages Aug 12, 2011 – Aug 22, 2011 1 message not in the archive XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hi there,

with FhGFS we may use extended attributes to store file meta data and with ext3/ext4 that also works very well and the rate to create files and to write those EAs (create() + fsetxattr() is about 2.5 to 3 times faster than with a create() + write(). Size of those EA data is about 256 bytes depending on the number of storage stripes. However, with XFS using extended attributes is *extremely* slow. Here are some numbers with ext4 and xfs using a patched [1] bonnie++

Quoted 26 lines

schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestXFS/ -s0 -n 1:256:256:1 -r 0 -X Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fsdevel3 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

          1:256:256   330   6 +++++ +++    47   1   469   9 +++++ +++    31   1
Latency               962ms      36us     140ms     866ms      28us     311ms

1.96,1.96,fsdevel3,1,1314378941,,,,,,,,,,,,,,1,256,256,,,330,6,+++++,+++,47,1,469,9,+++++,+++,31,1,,,,,,,962ms,36us,140ms,866ms,28us,311ms

schubert@fsdevel3 bonnie++-1.96>./bonnie++ -d /mnt/fstestEXT4/ -s0 -n 100:256:256:10 -r 0 -X Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Create------ --------Random Create--------
fsdevel3 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

Quoted 3 lines
     100:256:256/10 21348  52 +++++ +++ 32878  61 25286  60 +++++ +++ 25873  53

Latency 746us 1926us 2553us 653us 118us 32250us
1.96,1.96,fsdevel3,1,1314379136,,,,,,,,,,,,,,100,256,256,,10,21348,52,+++++,+++,32878,61,25286,60,+++++,+++,25873,53,,,,,,,746us,1926us,2553us,653us,118us,32250us

NOTE: For ext4 I had to increase the number of files by factor 100 to get any sane number (it would only print '+++++' otherwise). Running the benchmark with the same numbers on xfs deleted in so slow delete numbers, that it probably still would be busy to delete files by tomorrow.

The xfs file system was formated with these parameters:

Quoted 1 line

mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/mapper/vg0fsdev3-XFStest

ext4 was formated to have an inode size of 512B and to have a joural size of 400MB.

Both file systems are mounted with "nobarrier" (real FhGFS production systems usually have raid controllers with battery backups).

Any ideas why xfs is so slow with extended attributes? Without extended attributes, so skipping the "-X" option in our patched bonnie also results in slower Create numbers (about 4500 on xfs vs. 7500 on ext4) compared to ext4, but that is 'only' factor 1.67 and not 50 or more as with EAs.

Thanks,
Bernd

[1] Updated bonnie to support extended attributes:
https://bitbucket.org/aakef/bonnie/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 9 lines · Bernd Schubert

Hi there,

with FhGFS we may use extended attributes to store file meta data and with ext3/ext4 that also works very well and the rate to create files and to write those EAs (create() + fsetxattr() is about 2.5 to 3 times faster than with a create() + write(). Size of those EA data is about 256 bytes depending on the number of storage stripes. However, with XFS using extended attributes is *extremely* slow. Here are some numbers with ext4 and xfs using a patched [1] bonnie++

Currently XFS uses synchronous transactions when retiring inodes with xattrs on them. So far I've not actually found a good reason for that, so I suspect we can remove it. I'll do another audit of it, and will send you a patch for it after doing basic QA.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

The patch below should help with the performance of deleting files that have external attributes. But I still wonder why you see attributes outside the inode if you are using 512 byte inodes and around 256k of attrs. Can you check tat you really have external blocks for attributes using xfs_bmap -a on the files in your workload. Can you also check if this goes away using 1k or 2k inode sizes?

---
From: Christoph Hellwig <[email protected]>
Subject: xfs: avoid synchronous transactions when deleting attr blocks

Currently xfs_attr_inactive causes a synchronous transactions if we are removing a file that has any extents allocated to the attribute fork. The code looks a like a relict from the days before the busy extent list, but with the busy extent list we avoid reusing data and attr extents that have been freed but not commited yet, so this code is just as superflous as the synchronous transactions for data blocks.

fs/xfs/xfs_attr.c +0 -12
Index: xfs/fs/xfs/xfs_attr.c
===================================================================
@@ -823,18 +823,6 @@ xfs_attr_inactive(xfs_inode_t *dp)
 	if (error)
 		goto out;
 
-	/*
-	 * Signal synchronous inactive transactions unless this is a
-	 * synchronous mount filesystem in which case we know that we're here
-	 * because we've been called out of xfs_inactive which means that the
-	 * last reference is gone and the unlink transaction has already hit
-	 * the disk so async inactive transactions are safe.
-	 */
-	if (!(mp->m_flags & XFS_MOUNT_WSYNC)) {
-		if (dp->i_d.di_anextents > 0)
-			xfs_trans_set_sync(trans);
-	}
-
 	error = xfs_itruncate_extents(&trans, dp, XFS_ATTR_FORK, 0);
 	if (error)
 		goto out;
fs/xfs/xfs_bmap.c +1 -9
Index: xfs/fs/xfs/xfs_bmap.c
===================================================================
@@ -3782,19 +3782,11 @@ xfs_bmap_compute_maxlevels(
  * Routine to be called at transaction's end by xfs_bmapi, xfs_bunmapi
  * caller.  Frees all the extents that need freeing, which must be done
  * last due to locking considerations.  We never free any extents in
- * the first transaction.  This is to allow the caller to make the first
- * transaction a synchronous one so that the pointers to the data being
- * broken in this transaction will be permanent before the data is actually
- * freed.  This is necessary to prevent blocks from being reallocated
- * and written to before the free and reallocation are actually permanent.
- * We do not just make the first transaction synchronous here, because
- * there are more efficient ways to gain the same protection in some cases
- * (see the file truncation code).
+ * the first transaction.
  *
  * Return 1 if the given transaction was committed and a new one
  * started, and 0 otherwise in the committed parameter.
  */
-/*ARGSUSED*/
 int						/* error */
 xfs_bmap_finish(
 	xfs_trans_t		**tp,		/* transaction pointer addr */

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 6 lines · Bernd Schubert
Quoted 3 lines
(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
       0: [0..7]: 92304..92311

(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell me).

It means that you are having an extent spanning 8 blocks for xattr storage, that map to physical blocks 92304 to 9231 in the filesystem.

It sounds to me like your workload has a lot more than 256 bytes of xattrs, or the underlying code is doing something rather stupid.

Quoted 4 lines

Looking at 'top' and 'iostat -x' outout, I noticed we are actually not limited by io to disk, but CPU bound. If you should be interested, I have attached 'perf record -g' and 'perf report -g' outout, of the bonnie file create (create + fsetfattr() ) phase.

It's mostly spending a lot of time on copying things into the CIL buffers, which is expected and intentional as that allows for additional parallelity. I you'd switch the workload to multiple intances doing the create in parallel you should be able to scale to better numbers.

Quoted 7 lines
Quoted 3 lines
    100:256:256/10 37026  91 +++++ +++ 43691  93 35960  92 +++++ +++ 40708  92

Latency 4328us 765us 2960us 527us 440us 1075us
1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us

xfs:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/sdd

Do 64k dir blocks actually help you with the workload? They also tend to do a lot of useless memcpys in their current form, although these didn't show up on your profile. Did you try using a larger inode size as suggested in my previous mail?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Bernd Schubert
Quoted 3 lines
(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
        0: [0..7]: 92304..92311
Quoted 8 lines
Quoted 2 lines

(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell me).

It means that you are having an extent spanning 8 blocks for xattr storage, that map to physical blocks 92304 to 9231 in the filesystem.

It sounds to me like your workload has a lot more than 256 bytes of xattrs, or the underlying code is doing something rather stupid.

Well, the workload I described here is a controlled bonnie test, so there cannot be more than 256 bytes (unless there is a bug in the code, will double check later on).

Quoted 10 lines
Quoted 5 lines

Looking at 'top' and 'iostat -x' outout, I noticed we are actually not limited by io to disk, but CPU bound. If you should be interested, I have attached 'perf record -g' and 'perf report -g' outout, of the bonnie file create (create + fsetfattr() ) phase.

It's mostly spending a lot of time on copying things into the CIL buffers, which is expected and intentional as that allows for additional parallelity. I you'd switch the workload to multiple intances doing the create in parallel you should be able to scale to better numbers.

I just tried to bonnies in parallel and that didn't improve anything. FhGFS code has several threads anyway. But it would be good, if the underlying file system wouldn't take all the CPU time...

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
     100:256:256/10 37026  91 +++++ +++ 43691  93 35960  92 +++++ +++ 40708  92

Latency 4328us 765us 2960us 527us 440us 1075us
1.96,1.96,fslab2,1,1313594619,,,,,,,,,,,,,,100,256,256,,10,37026,91,+++++,+++,43691,93,35960,92,+++++,+++,40708,92,,,,,,,4328us,765us,2960us,527us,440us,1075us

xfs:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/sdd

Do 64k dir blocks actually help you with the workload? They also tend

Also just tested, with or without doesn't improve anything.

Quoted 3 lines

to do a lot of useless memcpys in their current form, although these didn't show up on your profile. Did you try using a larger inode size as suggested in my previous mail?

I just tried and now that I understand the xfs_bmap output, it is interesting to see, that an xattr size up to 128 byte does not need an extent + blocks, but 256 byte do have one extent and 8 blocks even with an inode size of 2K. xfs_info tells me that isize=2048 was accepted. I didn't test any sizes in between 128 and 256 byte yet. Now while I can set the data/xattr size for the bonnie test to less than 256 byte, that is not so easy with our real target FhGFS ;)

Btw, the create() + fsetfattr() reate with 128B xattr data is between 13000 and 17000, so due to inlining much better than with 256 xattr.

Thanks,
Bernd

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 30 lines · Bernd Schubert
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Bernd Schubert
Quoted 3 lines
(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
       0: [0..7]: 92304..92311
Quoted 8 lines
Quoted 2 lines

(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell me).

It means that you are having an extent spanning 8 blocks for xattr storage, that map to physical blocks 92304 to 9231 in the filesystem.

It sounds to me like your workload has a lot more than 256 bytes of xattrs, or the underlying code is doing something rather stupid.

Well, the workload I described here is a controlled bonnie test, so there cannot be more than 256 bytes (unless there is a bug in the code, will double check later on).

Quoted 10 lines
Quoted 5 lines

Looking at 'top' and 'iostat -x' outout, I noticed we are actually not limited by io to disk, but CPU bound. If you should be interested, I have attached 'perf record -g' and 'perf report -g' outout, of the bonnie file create (create + fsetfattr() ) phase.

It's mostly spending a lot of time on copying things into the CIL buffers, which is expected and intentional as that allows for additional parallelity. I you'd switch the workload to multiple intances doing the create in parallel you should be able to scale to better numbers.

Quoted 4 lines

I just tried to bonnies in parallel and that didn't improve anything. FhGFS code has several threads anyway. But it would be good, if the underlying file system wouldn't take all the CPU time...

XFS directory algorithms are significantly more complex than ext4. They trade off CPU usage for significantly better layout and scalability at large sizes. i.e. CPU costs less than IO so we burn more CPU to reduce IO. You don't see the benefits of that until directories start to get large (e.g. > 100k entries) and you are doing cold cache lookups.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

xfs:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/sdd

What is the output of this command?

Quoted 3 lines
Quoted 1 line

Do 64k dir blocks actually help you with the workload? They also tend

Also just tested, with or without doesn't improve anything.

Right, 64k directory blocks make a difference on cold cache traversals and lookups by flattening the btrees. They also make a difference in create/unlink performance once you get over a few million files in the one directory (once again due to reduced IO).

Quoted 11 lines
Quoted 3 lines

to do a lot of useless memcpys in their current form, although these didn't show up on your profile. Did you try using a larger inode size as suggested in my previous mail?

I just tried and now that I understand the xfs_bmap output, it is interesting to see, that an xattr size up to 128 byte does not need an extent + blocks, but 256 byte do have one extent and 8 blocks even with an inode size of 2K. xfs_info tells me that isize=2048 was accepted. I didn't test any sizes in between 128 and 256 byte yet. Now while I can set the data/xattr size for the bonnie test to less than 256 byte, that is not so easy with our real target FhGFS ;)

That tells me your filesystem is either not using dynamic attribute fork offsets or that code is broken. The output of the above mkfs command will tell us what attribute fork behaviour is expected, so which of the two cases you are seeing.

Also, what kernel are you testing on?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw
Quoted 69 lines · Dave Chinner
Quoted 30 lines · Bernd Schubert
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Bernd Schubert
Quoted 3 lines
(squeeze-x86_64)fslab2:~# xfs_bmap -a /mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n
/mnt/xfs/Bonnie.29243/00000/00000027faJxifNb0n:
       0: [0..7]: 92304..92311
Quoted 8 lines
Quoted 2 lines

(Sorry, I have no idea what "0: [0..7]: 92304..9231" is supposed to tell me).

It means that you are having an extent spanning 8 blocks for xattr storage, that map to physical blocks 92304 to 9231 in the filesystem.

It sounds to me like your workload has a lot more than 256 bytes of xattrs, or the underlying code is doing something rather stupid.

Well, the workload I described here is a controlled bonnie test, so there cannot be more than 256 bytes (unless there is a bug in the code, will double check later on).

Quoted 10 lines
Quoted 5 lines

Looking at 'top' and 'iostat -x' outout, I noticed we are actually not limited by io to disk, but CPU bound. If you should be interested, I have attached 'perf record -g' and 'perf report -g' outout, of the bonnie file create (create + fsetfattr() ) phase.

It's mostly spending a lot of time on copying things into the CIL buffers, which is expected and intentional as that allows for additional parallelity. I you'd switch the workload to multiple intances doing the create in parallel you should be able to scale to better numbers.

Quoted 4 lines

I just tried to bonnies in parallel and that didn't improve anything. FhGFS code has several threads anyway. But it would be good, if the underlying file system wouldn't take all the CPU time...

XFS directory algorithms are significantly more complex than ext4. They trade off CPU usage for significantly better layout and scalability at large sizes. i.e. CPU costs less than IO so we burn more CPU to reduce IO. You don't see the benefits of that until directories start to get large (e.g. > 100k entries) and you are doing cold cache lookups.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

xfs:
mkfs.xfs -f -i size=512 -i maxpct=90 -l lazy-count=1 -n size=64k /dev/sdd

What is the output of this command?

Quoted 3 lines
Quoted 1 line

Do 64k dir blocks actually help you with the workload? They also tend

Also just tested, with or without doesn't improve anything.

Right, 64k directory blocks make a difference on cold cache traversals and lookups by flattening the btrees. They also make a difference in create/unlink performance once you get over a few million files in the one directory (once again due to reduced IO).

Quoted 11 lines
Quoted 3 lines

to do a lot of useless memcpys in their current form, although these didn't show up on your profile. Did you try using a larger inode size as suggested in my previous mail?

I just tried and now that I understand the xfs_bmap output, it is interesting to see, that an xattr size up to 128 byte does not need an extent + blocks, but 256 byte do have one extent and 8 blocks even with an inode size of 2K. xfs_info tells me that isize=2048 was accepted. I didn't test any sizes in between 128 and 256 byte yet. Now while I can set the data/xattr size for the bonnie test to less than 256 byte, that is not so easy with our real target FhGFS ;)

Quoted 2 lines

That tells me your filesystem is either not using dynamic attribute fork offsets or that code is broken.

It is neither of these.

Quoted 5 lines

The output of the above mkfs
command will tell us what attribute fork behaviour is expected, so which of the two cases you are seeing.

Also, what kernel are you testing on?

Ok, I've reproduced it on a 3.0-rc2 kernel with attr=2 and 2k inodes. An attribute of 254 bytes stays in line:

....
u = (empty)
a.sfattr.hdr.totsize = 270
a.sfattr.hdr.count = 1
a.sfattr.list[0].namelen = 9
a.sfattr.list[0].valuelen = 254
a.sfattr.list[0].root = 0
a.sfattr.list[0].secure = 0
a.sfattr.list[0].name = "user.test" a.sfattr.list[0].value = " ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten chars ten"

but at 255 bytes:

u = (empty)
a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,40,1,0]

it goes out of line. There's something strange happening there....

/me looks

#define XFS_ATTR_SF_ENTSIZE_MAX                 /* max space for name&value */ \
        ((1 << (NBBY*(int)sizeof(__uint8_t))) - 1)

There's the issue - shortform attribute structures (i.e. inline attributes) have only 8 bits for the value/name lengths, so the attribute is going out of line at a value length of 255 bytes.

So, the reason you are seeing this is that in line attributes must be 254 bytes or smaller to remain in line regardless of the inode size. If FhGFS is using multiple attributes smaller than 254 bytes, then they will all stay inline until all the inode attribute fork space is used. However, the first one that goes over this limit will push them all out of line.

Unfortunately, that limit is fixed into the on-disk format for attributes, so is kind of hard to change. :/

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

Btw, one interesting thing I noticed is that xfs_attr_leaf_create always zeroes and logs whe whole attr block. I'll need to audit this a bit more, but from the highlevel POV I don't see why we can't simply just logs the bits we actually added.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs