xfs: dynamic speculative allocation beyond EOF V4
cover xfs: dynamic speculative allocation beyond EOF V4
This is the latest version of the dynamic speculative allocation beyond EOF patch set. The description of the patchset can be found here:
http://oss.sgi.com/archives/xfs/2010-10/msg00040.html
Version 4:
- factored prealloc size into separate function to keep xfs_iomap_write_delay() easy to read.
- convert i_dirty_releases counter to a flag.
Version 3:
- allocsize mount option returned to fixed preallocation size only.
- reduces maximum dynamic prealloc size as the filesytem gets near full.
- split i_delayed_blks bug fixes into new patch (posted in 2.6.37-rc bug fix series)
Version 2:
- base speculative execution size on current inode size, not the number of previous speculative allocations.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/2 xfs: dynamic speculative EOF preallocation
From: Dave Chinner <[email protected]>
Currently the size of the speculative preallocation during delayed allocation is fixed by either the allocsize mount option of a default size. We are seeing a lot of cases where we need to recommend using the allocsize mount option to prevent fragmentation when buffered writes land in the same AG.
Rather than using a fixed preallocation size by default (up to 64k), make it dynamic by basing it on the current inode size. That way the EOF preallocation will increase as the file size increases. Hence for streaming writes we are much more likely to get large preallocations exactly when we need it to reduce fragementation.
For default settings, the size of the initial extents is determined by the number of parallel writers and the amount of memory in the machine. For 4GB RAM and 4 concurrent 32GB file writes:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..1048575]: 1048672..2097247 0 (1048672..2097247) 1048576 1: [1048576..2097151]: 5242976..6291551 0 (5242976..6291551) 1048576 2: [2097152..4194303]: 12583008..14680159 0 (12583008..14680159) 2097152 3: [4194304..8388607]: 25165920..29360223 0 (25165920..29360223) 4194304 4: [8388608..16777215]: 58720352..67108959 0 (58720352..67108959) 8388608 5: [16777216..33554423]: 117440584..134217791 0 (117440584..134217791) 16777208 6: [33554424..50331511]: 184549056..201326143 0 (184549056..201326143) 16777088 7: [50331512..67108599]: 251657408..268434495 0 (251657408..268434495) 16777088
and for 16 concurrent 16GB file writes:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..262143]: 2490472..2752615 0 (2490472..2752615) 262144 1: [262144..524287]: 6291560..6553703 0 (6291560..6553703) 262144 2: [524288..1048575]: 13631592..14155879 0 (13631592..14155879) 524288 3: [1048576..2097151]: 30408808..31457383 0 (30408808..31457383) 1048576 4: [2097152..4194303]: 52428904..54526055 0 (52428904..54526055) 2097152 5: [4194304..8388607]: 104857704..109052007 0 (104857704..109052007) 4194304 6: [8388608..16777215]: 209715304..218103911 0 (209715304..218103911) 8388608
7: [16777216..33554423]: 452984848..469762055 0 (452984848..469762055) 16777208
Because it is hard to take back specualtive preallocation, cases where there are large slow growing log files on a nearly full filesystem may cause premature ENOSPC. Hence as the filesystem nears full, the maximum dynamic prealloc size іs reduced according to this table (based on 4k block size):
freespace max prealloc size
Quoted 1 line
5% full extent (8GB)
4-5% 2GB (8GB >> 2)
3-4% 1GB (8GB >> 3)
2-3% 512MB (8GB >> 4) 1-2% 256MB (8GB >> 5) <1% 128MB (8GB >> 6)
This should reduce the amount of space held in speculative preallocation for such cases.
The allocsize mount option turns off the dynamic behaviour and fixes the prealloc size to whatever the mount option specifies. i.e. the behaviour is unchanged.
--- fs/xfs/xfs_fsops.c | 1 + fs/xfs/xfs_iomap.c | 85 +++++++++++++++++++++++++++++++++++++++++++++------ fs/xfs/xfs_mount.c | 21 +++++++++++++ fs/xfs/xfs_mount.h | 14 ++++++++ 4 files changed, 111 insertions(+), 10 deletions(-)
fs/xfs/xfs_fsops.c +1 -0
@@ -374,6 +374,7 @@ xfs_growfs_data_private( mp->m_maxicount = icount << mp->m_sb.sb_inopblog; }else mp->m_maxicount =0 ; +xfs_set_low_space_thresholds (mp);/* update secondary superblocks. */ for (agno =1 ; agno < nagcount; agno++) {
fs/xfs/xfs_iomap.c +75 -10
138 lines not on this page · raw message
fs/xfs/xfs_mount.c +21 -0
@@ -1101,6 +1101,24 @@ xfs_set_rw_sizes(xfs_mount_t *mp) }/* +* precalculate the low space thresholds for dynamic speculative preallocation. +*/ +void +xfs_set_low_space_thresholds ( +struct xfs_mount *mp) +{ +int i; + +for (i =0 ; i <XFS_LOWSP_MAX ; i++) { +__uint64_t space = mp->m_sb.sb_dblocks; + +do_div (space,100 ); + mp->m_low_space[i] = space * (i +1 ); + } +} + + +/* * Set whether we're using inode alignment. */ STATIC void
@@ -1322,6 +1340,9 @@ xfs_mountfs(*/ xfs_set_rw_sizes (mp); +/* set the low space thresholds for dynamic preallocation */ +xfs_set_low_space_thresholds (mp); +/* * Set the inode cluster size. * This may still be overridden by the file system
fs/xfs/xfs_mount.h +14 -0
@@ -75,6 +75,16 @@ enum { XFS_ICSB_MAX, }; +/* dynamic preallocation free space thresholds, 5% down to 1% */ +enum { +XFS_LOWSP_1_PCNT =0 , +XFS_LOWSP_2_PCNT , +XFS_LOWSP_3_PCNT , +XFS_LOWSP_4_PCNT , +XFS_LOWSP_5_PCNT , +XFS_LOWSP_MAX , +}; + typedef struct xfs_mount { struct super_block *m_super; xfs_tid_t m_tid; /* next unused tid for fs */
@@ -169,6 +179,8 @@ typedef struct xfs_mount {on the next remount,rw */ struct shrinker m_inode_shrink;/* inode reclaim shrinker */ struct percpu_counter m_icsb[XFS_ICSB_MAX ]; +int64_t m_low_space[XFS_LOWSP_MAX ]; +/* low free space thresholds */ } xfs_mount_t;/*
@@ -333,6 +345,8 @@ extern void xfs_icsb_sync_counters(struct xfs_mount *);extern int xfs_icsb_modify_inodes (struct xfs_mount *,int ,int64_t );extern int xfs_icsb_modify_free_blocks (struct xfs_mount *,int64_t ,int ); +extern void xfs_set_low_space_thresholds (struct xfs_mount *); +#endif /* __KERNEL__ */ extern void xfs_mod_sb (struct xfs_trans *,__int64_t );
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/2] xfs: dynamic speculative EOF preallocation
With this patch my 32-bit x86 VM hangs when running test 014, although I can interrupt it. With the full patchset applied I get softlockup warnings in this test for xfsaild, which seems related. Also test 012 fails, but that might be intentional (?).
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/2 xfs: don't truncate prealloc from frequently accessed inodes
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
A long standing problem for streaming writeѕ through the NFS server has been that the NFS server opens and closes file descriptors on an inode for every write. The result of this behaviour is that the ->release() function is called on every close and that results in XFS truncating speculative preallocation beyond the EOF. This has an adverse effect on file layout when multiple files are being written at the same time - they interleave their extents and can result in severe fragmentation.
To avoid this problem, keep a count of the number of ->release calls made on an inode. For most cases, an inode is only going to be opened once for writing and then closed again during it's lifetime in cache. Hence if there are multiple ->release calls, there is a good chance that the inode is being accessed by the NFS server. Hence count up every time ->release is called while there are delalloc blocks still outstanding on the inode.
If this count is non-zero when ->release is next called, then do no truncate away the speculative preallocation - leave it there so that subsequent writes do not need to reallocate the delalloc space. This will prevent interleaving of extents of different inodes written concurrently to the same AG.
If we get this wrong, it is not a big deal as we truncate speculative allocation beyond EOF anyway in xfs_inactive() when the inode is thrown out of the cache.
The new counter in the struct xfs_inode fits into a hole in the structure on 64 bit machines, so does not grow the size of the inode at all.
--- fs/xfs/xfs_inode.h | 13 +++++----- fs/xfs/xfs_vnodeops.c | 61 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 47 insertions(+), 27 deletions(-)
fs/xfs/xfs_inode.h +7 -6
@@ -376,12 +376,13 @@ static inline void xfs_ifunlock(xfs_inode_t *ip)/* * In-core inode flags. */ -#define XFS_IRECLAIM 0x0001/* we have started reclaiming this inode */ -#define XFS_ISTALE 0x0002/* inode has been staled */ -#define XFS_IRECLAIMABLE 0x0004/* inode can be reclaimed */ -#define XFS_INEW 0x0008/* inode has just been allocated */ -#define XFS_IFILESTREAM 0x0010/* inode is in a filestream directory */ -#define XFS_ITRUNCATED 0x0020/* truncated down so flush-on-close */ +#define XFS_IRECLAIM 0x0001/* started reclaiming this inode */ +#define XFS_ISTALE 0x0002/* inode has been staled */ +#define XFS_IRECLAIMABLE 0x0004/* inode can be reclaimed */ +#define XFS_INEW 0x0008/* inode has just been allocated */ +#define XFS_IFILESTREAM 0x0010/* inode is in a filestream directory */ +#define XFS_ITRUNCATED 0x0020/* truncated down so flush-on-close */ +#define XFS_IDIRTY_RELEASE 0x0040/* dirty release already seen */ /* * Flags for inode locking.
fs/xfs/xfs_vnodeops.c +40 -21
69 lines not on this page · raw message
Signature · 6 lines
1.7.2.3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/2] xfs: don't truncate prealloc from frequently accessed inodes
Quoted 3 lines · Dave Chinner
The new counter in the struct
xfs_inodefits into a hole in the structure on 64 bit machines, so does not grow the size of the inode at all.
The count actually is gone now.
But this patch totally changes xfstests 203 output for me. Given that the test doesn't care about the allocation pattern it's probably fine, but we'll a fix for the testcase.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs