xfs: optimise CIL insertion during transaction commit [RFC]

From: Dave Chinner <[email protected]>

Note: This is an RFC right now - it'll need to be broken up into several patches for final submission.

The CIL insertion during transaction commit currently does multiple passes across the transaction objects and requires multiple memory allocations per object that is to be inserted into the CIL. It is quite inefficient, and as such xfs_log_commit_cil() and it's children show up quite highly in profiles under metadata modification intensive workloads.

The current insertion tries to minimise ithe number of times the xc_cil_lock is grabbed and the hold times via a couple of methods:

1. an initial loop across the transaction items outside the lock to allocate log vectors, buffers and copy the data into them.
2. a second pass across the log vectors that then inserts them into the CIL, modifies the CIL state and frees the old vectors.

This is somewhat inefficient. While it minimises lock grabs, the hold time is still quite high because we are freeing objects with the spinlock held and so the hold times are much higher than they need to be.

Optimisations that can be made:

1. change IOP_SIZE() to return the size of the metadata to be logged with the number of vectors required. This means we can allocate the buffer to hold the metadata as part of the allocation for the log vector array. Once allocation instead of two.

2. Store the size of the allocated region for the log vector in the log vector itself. This allows us to determine if we can just reuse the existing log vector for the new transactions. Avoids all memory allocation for that item.

3. when setting up a new log vector, we don't need to hold the CIL lock while determining the difference in it's size when compared to the old one. Hence we can do all the accounting, swapping of the log vectors and freeing the old log vectors outside the xc_cil_lock.

4. We can do all the CIL insertation of items and busy extents and the accounting under a single grab of the xc_cil_lock. This minimises the number of times we need to grab it and hence contention points are kept to a minimum

5. the xc_cil_lock is used for serialising both the CIL and the push/commit ordering. Separate the two functions into different locks so push/commit ordering does not affect CIL insertion

6. the xc_cil_lock shares cachelines with other locks. Separate them onto different cachelines.

The result is that my standard fsmark benchmark (8-way, 50m files) on my standard test VM (8-way, 4GB RAM, 4xSSD in RAID0, 100TB fs) gives the following results with a xfs-oss tree. No CRCs:

                vanilla         patched         Difference
create  (time)  483s            435s            -10.0%  (faster)
        (rate)  109k+/6k        122k+/-7k       +11.9%  (faster)

walk            339s            335s            (noise)
     (sys cpu)  1134s           1135s           (noise)

unlink 692s 645s -6.8% (faster)

So it's significantly faster than the current code, and lock_stat reports lower contention on the xc_cil_lock, too. So, big win here.

With CRCs:

                vanilla         patched         Difference
create  (time)  510s            460s             -9.8%  (faster)
        (rate)  105k+/5.4k      117k+/-5k       +11.4%  (faster)

walk            494s            486s            (noise)
     (sys cpu)  1324s           1290s           (noise)

unlink 959s 889s -7.3% (faster)

Gains are of the same order, with walk and unlink still affected by VFS LRU lock contention. IOWs, with this changes, filesystems with CRCs enabled will still be faster than the old non-CRC kernels...

---
 fs/xfs/xfs_buf_item.c     |   52 ++++---
 fs/xfs/xfs_dquot_item.c   |   22 +--
 fs/xfs/xfs_extfree_item.c |   50 ++++---
 fs/xfs/xfs_icreate_item.c |    9 +-
 fs/xfs/xfs_inode_item.c   |   52 ++++---
 fs/xfs/xfs_log.h          |    1 +
 fs/xfs/xfs_log_cil.c      |  366 +++++++++++++++++++++++----------------------
 fs/xfs/xfs_log_priv.h     |    7 +-
 fs/xfs/xfs_trans.h        |    4 +-
 9 files changed, 311 insertions(+), 252 deletions(-)

index bfc4e0c..9358504 100644
@@ -39,6 +39,14 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip)
 
 STATIC void	xfs_buf_do_callbacks(struct xfs_buf *bp);
 
+static inline int
+xfs_buf_log_format_size(
+	struct xfs_buf_log_format *blfp)
+{
+	return offsetof(struct xfs_buf_log_format, blf_data_map) +
+			(blfp->blf_map_size * sizeof(blfp->blf_data_map[0]));
+}
+
 /*
  * This returns the number of log iovecs needed to log the
  * given buf log item.

@@ -49,25 +57,27 @@ STATIC void	xfs_buf_do_callbacks(struct xfs_buf *bp);
  *
  * If the XFS_BLI_STALE flag has been set, then log nothing.
  */
-STATIC uint
+STATIC void
 xfs_buf_item_size_segment(
 	struct xfs_buf_log_item	*bip,
-	struct xfs_buf_log_format *blfp)
+	struct xfs_buf_log_format *blfp,
+	int			*nvecs,
+	int			*nbytes)
 {
 	struct xfs_buf		*bp = bip->bli_buf;
-	uint			nvecs;
 	int			next_bit;
 	int			last_bit;
 
 	last_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, 0);
 	if (last_bit == -1)
-		return 0;
+		return;
 
 	/*
 	 * initial count for a dirty buffer is 2 vectors - the format structure
 	 * and the first dirty region.
 	 */
-	nvecs = 2;
+	*nvecs += 2;
+	*nbytes += xfs_buf_log_format_size(blfp) + XFS_BLF_CHUNK;
 
 	while (last_bit != -1) {
 		/*

@@ -87,18 +97,17 @@ xfs_buf_item_size_segment(
 			break;
 		} else if (next_bit != last_bit + 1) {
 			last_bit = next_bit;
-			nvecs++;
+			(*nvecs)++;
 		} else if (xfs_buf_offset(bp, next_bit * XFS_BLF_CHUNK) !=
 			   (xfs_buf_offset(bp, last_bit * XFS_BLF_CHUNK) +
 			    XFS_BLF_CHUNK)) {
 			last_bit = next_bit;
-			nvecs++;
+			(*nvecs)++;
 		} else {
 			last_bit++;
 		}
+		*nbytes += XFS_BLF_CHUNK;
 	}
-
-	return nvecs;
 }
 
 /*

@@ -118,12 +127,13 @@ xfs_buf_item_size_segment(
  * If the XFS_BLI_STALE flag has been set, then log nothing but the buf log
  * format structures.
  */
-STATIC uint
+STATIC void
 xfs_buf_item_size(
-	struct xfs_log_item	*lip)
+	struct xfs_log_item	*lip,
+	int			*nvecs,
+	int			*nbytes)
 {
 	struct xfs_buf_log_item	*bip = BUF_ITEM(lip);
-	uint			nvecs;
 	int			i;
 
 	ASSERT(atomic_read(&bip->bli_refcount) > 0);

@@ -135,7 +145,11 @@ xfs_buf_item_size(
 		 */
 		trace_xfs_buf_item_size_stale(bip);
 		ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL);
-		return bip->bli_format_count;
+		*nvecs += bip->bli_format_count;
+		for (i = 0; i < bip->bli_format_count; i++) {
+			*nbytes += xfs_buf_log_format_size(&bip->bli_formats[i]);
+		}
+		return;
 	}
 
 	ASSERT(bip->bli_flags & XFS_BLI_LOGGED);

@@ -147,7 +161,8 @@ xfs_buf_item_size(
 		 * commit, so no vectors are used at all.
 		 */
 		trace_xfs_buf_item_size_ordered(bip);
-		return XFS_LOG_VEC_ORDERED;
+		*nvecs = XFS_LOG_VEC_ORDERED;
+		return;
 	}
 
 	/*

@@ -159,13 +174,11 @@ xfs_buf_item_size(
 	 * count for the extra buf log format structure that will need to be
 	 * written.
 	 */
-	nvecs = 0;
 	for (i = 0; i < bip->bli_format_count; i++) {
-		nvecs += xfs_buf_item_size_segment(bip, &bip->bli_formats[i]);
+		xfs_buf_item_size_segment(bip, &bip->bli_formats[i],
+					  nvecs, nbytes);
 	}
-
 	trace_xfs_buf_item_size(bip);
-	return nvecs;
 }
 
 static struct xfs_log_iovec *

@@ -192,8 +205,7 @@ xfs_buf_item_format_segment(
 	 * the actual size of the dirty bitmap rather than the size of the in
 	 * memory structure.
 	 */
-	base_size = offsetof(struct xfs_buf_log_format, blf_data_map) +
-			(blfp->blf_map_size * sizeof(blfp->blf_data_map[0]));
+	base_size = xfs_buf_log_format_size(blfp);
 
 	nvecs = 0;
 	first_bit = xfs_next_bit(blfp->blf_data_map, blfp->blf_map_size, 0);

index f07a436..60c6e1f 100644
@@ -44,14 +44,15 @@ static inline struct xfs_dq_logitem *DQUOT_ITEM(struct xfs_log_item *lip)
 /*
  * returns the number of iovecs needed to log the given dquot item.
  */
-STATIC uint
+STATIC void
 xfs_qm_dquot_logitem_size(
-	struct xfs_log_item	*lip)
+	struct xfs_log_item	*lip,
+	int			*nvecs,
+	int			*nbytes)
 {
-	/*
-	 * we need only two iovecs, one for the format, one for the real thing
-	 */
-	return 2;
+	*nvecs += 2;
+	*nbytes += sizeof(struct xfs_dq_logformat) +
+		   sizeof(struct xfs_disk_dquot);
 }
 
 /*

@@ -286,11 +287,14 @@ static inline struct xfs_qoff_logitem *QOFF_ITEM(struct xfs_log_item *lip)
  * We only need 1 iovec for an quotaoff item.  It just logs the
  * quotaoff_log_format structure.
  */
-STATIC uint
+STATIC void
 xfs_qm_qoff_logitem_size(
-	struct xfs_log_item	*lip)
+	struct xfs_log_item	*lip,
+	int			*nvecs,
+	int			*nbytes)
 {
-	return 1;
+	*nvecs += 1;
+	*nbytes += sizeof(struct xfs_qoff_logitem);
 }
 
 /*

index 441a78a..5a5a593 100644
@@ -40,11 +40,14 @@ static inline struct xfs_icreate_item *ICR_ITEM(struct xfs_log_item *lip)
  *
  * We only need one iovec for the icreate log structure.
  */
-STATIC uint
+STATIC void
 xfs_icreate_item_size(
-	struct xfs_log_item	*lip)
+	struct xfs_log_item	*lip,
+	int			*nvecs,
+	int			*nbytes)
 {
-	return 1;
+	*nvecs += 1;
+	*nbytes += sizeof(struct xfs_icreate_log);
 }
 
 /*

index e63d9e1..1c45848 100644
@@ -27,6 +27,7 @@ struct xfs_log_vec {
 	struct xfs_log_item	*lv_item;	/* owner */
 	char			*lv_buf;	/* formatted buffer */
 	int			lv_buf_len;	/* size of formatted buffer */
+	int			lv_size;	/* size of allocated lv */
 };
 
 #define XFS_LOG_VEC_ORDERED	(-1)

index edd0964..56654931 100644
@@ -278,13 +278,16 @@ struct xfs_cil {
 	struct xlog		*xc_log;
 	struct list_head	xc_cil;
 	spinlock_t		xc_cil_lock;
-	struct xfs_cil_ctx	*xc_ctx;
+
+	struct xfs_cil_ctx	*xc_ctx ____cacheline_aligned_in_smp;
 	struct rw_semaphore	xc_ctx_lock;
+
+	spinlock_t		xc_push_lock ____cacheline_aligned_in_smp;
 	struct list_head	xc_committing;
 	wait_queue_head_t	xc_commit_wait;
+	xfs_lsn_t		xc_push_seq;
 	xfs_lsn_t		xc_current_sequence;
 	struct work_struct	xc_push_work;
-	xfs_lsn_t		xc_push_seq;
 };
 
 /*

index 8e67284..157e9c4 100644
@@ -140,7 +140,7 @@ typedef struct xfs_log_item {
 	{ XFS_LI_ABORTED,	"ABORTED" }
 
 struct xfs_item_ops {
-	uint (*iop_size)(xfs_log_item_t *);
+	void (*iop_size)(xfs_log_item_t *, int *, int *);
 	void (*iop_format)(xfs_log_item_t *, struct xfs_log_iovec *);
 	void (*iop_pin)(xfs_log_item_t *);
 	void (*iop_unpin)(xfs_log_item_t *, int remove);

@@ -150,8 +150,6 @@ struct xfs_item_ops {
 	void (*iop_committing)(xfs_log_item_t *, xfs_lsn_t);
 };
 
-#define IOP_SIZE(ip)		(*(ip)->li_ops->iop_size)(ip)
-#define IOP_FORMAT(ip,vp)	(*(ip)->li_ops->iop_format)(ip, vp)
 #define IOP_PIN(ip)		(*(ip)->li_ops->iop_pin)(ip)
 #define IOP_UNPIN(ip, remove)	(*(ip)->li_ops->iop_unpin)(ip, remove)
 #define IOP_PUSH(ip, list)	(*(ip)->li_ops->iop_push)(ip, list)

1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Dave,

The idea to separate the transaction commits from the CIL push looks good.

I am still concerned that the xc_ctx->space_used comparison to XLOG_CIL_SPACE_LIMIT(log) does not also contain any information on the "stolen" ticket log space. Maybe the future Liu/Chinner minimum log space series is the best place to address this and my other log space concerns.

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

[cc fsdevel because after all the XFS stuff I did a some testing on mmotm w.r.t per-node LRU lock contention avoidance, and also some scalability tests against ext4 and btrfs for comparison on some new hardware. That bit ain't pretty. ]

.....

FWIW, I have new hardware here that I'll be using for benchmarking like this, so here's a quick baseline comparison using the same 8p/4GB RAM VM (just migrated across) and same SSD based storage (physically moved) and 100TB filesystem. The disks are behind a faster RAID controller w/ 1GB of BBWC, so random read and write IOPS are higher and hence traversal times will due to lower IO latency.

Create times
wall time(s) rate (files/s)
vanilla patched diff vanilla patched diff

Old system	  483	  435	 -10.0%	   109k+-6k 122k+-7k +11.9%
New system	  378	  342	  -9.5%	   143k+-9k 158k+-8k +10.5%
diff		-21.7%	-21.4%		    +31.2%   +29.5%

Walk times
wall time(s)
vanilla patched diff

Old system	  339	  335	 (noise)
New system	  194	  197	 (noise)

diff -42.7% -41.2%

Unlink times
wall time(s)
vanilla patched diff

Old system	  692	  645	  -7.3%
New system	  457	  405	 -11.4%

diff -34.0% -37.2%

So, overall, the new system is 20-40% faster than the old one on a comparitive test. but I have a few more cores and a lot more memory to play with, so a 16-way test on the same machine with the VM expanded to 16p/16GB RAM, 4 fake numa nodes follows:

New system, patched kernel:

Threads create walk unlink

	time(s)	 rate		time(s)		time(s)

8 342 158k+-8k 197 405
16 222 266k+-32k 170 295
diff -35.1% +68.4% -13.7% -27.2%

Create rates are much more variable because the memory reclaim behaviour appears to be very harsh, pulling 4-6 million inodes out of memory every 10s or so and thrashing on the LRU locks, and then doing nothing until another large step occurs.

Walk rates improve, but not much because of lock contention. I added 8 cpu cores to the workload, and I'm burning at least 4 of those cores on the inode LRU lock.

- 30.61% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 65.33% _raw_spin_lock
+ 88.19% inode_add_lru
+ 7.31% dentry_lru_del
+ 1.07% shrink_dentry_list
+ 0.90% dput
+ 0.83% inode_sb_list_add
+ 0.59% evict
+ 27.79% do_raw_spin_lock
+ 4.03% do_raw_spin_trylock
+ 2.85% _raw_spin_trylock

The current mmotm (and hence probably 3.11) has the new per-node LRU code in it, so this variance and contention should go away very soon.

Unlinks go lots faster because they don't cause inode LRU lock contention, but we are still a long way from linear scalability from 8- to 16-way.

FWIW, the mmotm kernel (which has a fair bit of debug enabled, so not quite comparitive) doesn't have any LRU lock contention to speak of. For create:

- 7.81% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 70.98% _raw_spin_lock
+ 97.55% xfs_log_commit_cil
+ 0.93% __d_instantiate
+ 0.58% inode_sb_list_add
- 29.02% do_raw_spin_lock
- _raw_spin_lock
+ 41.14% xfs_log_commit_cil
+ 8.29% _xfs_buf_find
+ 8.00% xfs_iflush_cluster

And the walk:

- 26.37% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 49.10% _raw_spin_lock
- 50.65% evict
dispose_list
prune_icache_sb
super_cache_scan
+ shrink_slab
- 26.99% list_lru_add
+ 89.01% inode_add_lru
+ 10.95% dput
+ 7.03% __remove_inode_hash
- 40.65% do_raw_spin_lock
- _raw_spin_lock
- 41.96% evict
dispose_list
prune_icache_sb
super_cache_scan
+ shrink_slab
- 13.55% list_lru_add
84.33% inode_add_lru
iput
d_kill
shrink_dentry_list
prune_dcache_sb
super_cache_scan
shrink_slab
15.01% dput
0.66% xfs_buf_rele
+ 10.10% __remove_inode_hash
system_call_fastpath

There's quite a different pattern of contention - it has moved inward to evict which implies the inode_sb_list_lock is the next obvious point of contention. I have patches in the works for that. Also, the inode_hash_lock is causing some contention, even though we fake inode hashing. I have a patch to fix that for XFS as well.

I also note an interesting behaviour of the per-node inode LRUs - the contention is coming from the dentry shrinker on one node freeing inodes allocated on a different node during reclaim. There's scope for improvement there.

But here' the interesting part:

Kernel create walk unlink

	time(s)	 rate		time(s)		time(s)

3.10-cil 222 266k+-32k 170 295
mmotm 251 222k+-16k 128 356

Even with all the debug enabled, the overall walk time dropped by 25% to 128s. So performance in this workload has substantially improved because of the per-node LRUs and variability is also down as well, as predicted. Once I add all the tweaks I have in the 3.10-cil tree to mmotm, I expect significant improvements to create and unlink performance as well...

So, lets look at ext4 vs btrfs vs XFS at 16-way (this is on the 3.10-cil kernel I've been testing XFS on):

	    create		 walk		unlink
	 time(s)   rate		time(s)		time(s)

xfs 222 266k+-32k 170 295
ext4 978 54k+- 2k 325 2053
btrfs 1223 47k+- 8k 366 12000(*)

(*) Estimate based on a removal rate of 18.5 minutes for the first 4.8 million inodes.

Basically, neither btrfs or ext4 have any concurrency scaling to demonstrate, and unlinks on btrfs a just plain woeful.

ext4 create rate is limited by the extent cache LRU locking:

- 41.81% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 60.67% _raw_spin_lock
- 99.60% ext4_es_lru_add
+ 99.63% ext4_es_lookup_extent
- 39.15% do_raw_spin_lock
- _raw_spin_lock
+ 95.38% ext4_es_lru_add
0.51% insert_inode_locked
__ext4_new_inode
- 16.20% [kernel] [k] native_read_tsc
- native_read_tsc
- 60.91% delay_tsc
__delay
do_raw_spin_lock
+ _raw_spin_lock
- 39.09% __delay
do_raw_spin_lock
+ _raw_spin_lock

Ext4 unlink is serialised on orphan list processing:

- 12.67% [kernel] [k] __mutex_unlock_slowpath
- __mutex_unlock_slowpath
- 99.95% mutex_unlock
+ 54.37% ext4_orphan_del
+ 43.26% ext4_orphan_add
+ 5.33% [kernel] [k] __mutex_lock_slowpath

btrfs create has tree lock problems:

- 21.68% [kernel] [k] __write_lock_failed
- __write_lock_failed
- 99.93% do_raw_write_lock
- _raw_write_lock
- 79.04% btrfs_try_tree_write_lock
- btrfs_search_slot
- 97.48% btrfs_insert_empty_items 99.82% btrfs_new_inode
+ 2.52% btrfs_lookup_inode
- 20.37% btrfs_tree_lock
- 99.38% btrfs_search_slot 99.92% btrfs_insert_empty_items 0.52% btrfs_lock_root_node btrfs_search_slot btrfs_insert_empty_items
- 21.24% [kernel] [k] _raw_spin_unlock_irqrestore
- _raw_spin_unlock_irqrestore
- 61.22% prepare_to_wait
+ 61.52% btrfs_tree_lock
+ 32.31% btrfs_tree_read_lock
6.17% reserve_metadata_bytes
btrfs_block_rsv_add

btrfs walk phase hammers the inode_hash_lock:

- 18.45% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 47.38% _raw_spin_lock
+ 42.99% iget5_locked
+ 15.17% __remove_inode_hash
+ 13.77% btrfs_get_delayed_node
+ 11.27% inode_tree_add
+ 9.32% btrfs_destroy_inode
.....
- 46.77% do_raw_spin_lock
- _raw_spin_lock
+ 30.51% iget5_locked
+ 11.40% __remove_inode_hash
+ 11.38% btrfs_get_delayed_node
+ 9.45% inode_tree_add
+ 7.28% btrfs_destroy_inode
.....

I have a RCU inode hash lookup patch floating around somewhere if someone wants it...

And, well, the less said about btrfs unlinks the better:

+  37.14%  [kernel]  [k] _raw_spin_unlock_irqrestore
+  33.18%  [kernel]  [k] __write_lock_failed
+  17.96%  [kernel]  [k] __read_lock_failed
+   1.35%  [kernel]  [k] _raw_spin_unlock_irq
+   0.82%  [kernel]  [k] __do_softirq
+   0.53%  [kernel]  [k] btrfs_tree_lock
+   0.41%  [kernel]  [k] btrfs_tree_read_lock
+   0.41%  [kernel]  [k] do_raw_read_lock
+   0.39%  [kernel]  [k] do_raw_write_lock
+   0.38%  [kernel]  [k] btrfs_clear_lock_blocking_rw
+   0.37%  [kernel]  [k] free_extent_buffer
+   0.36%  [kernel]  [k] btrfs_tree_read_unlock
+   0.32%  [kernel]  [k] do_raw_write_unlock

Cheers,

Dave.

Dave Chinner
[email protected]

On Mon 08-07-13 22:44:53, Dave Chinner wrote:
<snipped some nice XFS results ;)>

Thanks for posting the numbers. There isn't anyone seriously testing ext4
SMP scalability AFAIK so it's not surprising it sucks.

At least this should improve with the patches in 3.11-rc1.

Jan Kara <[email protected]>
SUSE Labs, CR

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Il 08/07/2013 15:59, Jan Kara ha scritto:

Funny, if I well remember Google guys switched android from yaffs2 to ext4 due to its superiority on SMP :)

Marco

Well, there's SMP and SMP. Ext4 is perfectly OK for desktop kind of SMP -
that's what lots of people use. When we speak of heavy IO load with 16 CPUs on enterprise grade storage so that CPU (and not IO) bottlenecks are actually visible, that's not so easily available and so we don't have serious performance work in that direction...

Honza

Jan Kara <[email protected]>
SUSE Labs, CR

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

It's worse than that - nobody picked up on review that taking a global lock on every extent lookup might be a scalability issue? Scalability is not an afterthought anymore - new filesystem and kernel features need to be designed from the ground up with this in mind. We're living in a world where even phones have 4 CPU cores....

Barely. It tops out in parallelism at between 2-4 threads depending on the metadata operations being done.

I'm not testing with "enterprise grade" storage. The filesystem I'm testing on is hosted on less than $300 of SSDs. The "enterprise" RAID controller they sit behind is actually an IOPS bottleneck, not an improvement.

My 2.5 year old desktop has a pair of cheap, no name sandforce SSDs in RAID0 and they can do at least 2x the read and write IOPS of the new hardware I just tested. And yes, I run XFS on my desktop.

And then there's my 3 month old laptop, which has a recent SATA SSD in it. It also has 8 threads, but twice the memory and about 1.5x the IOPS and bandwidth of my desktop machine.

The bottlenecks showing up in ext4 and btrfs don't magically show up at 16 threads - they are present and reproducable at 2-4 threads. Indeed, I didn't bother testing at 32 threads - even though my new server can do that - because that will just hammer the same bottlenecks even harder. Fundamentally, I'm not testing anything you can't test on a $2000 desktop PC....

FWIW, the SSDs are making ext4 and btrfs look good in these workloads. XFS is creating >250k files/s doing about 1500 IOPS. ext4 is making 50k files/s at 23,000 IOPS. btrfs has peaks every 30s of over 30,000 IOPS. Which filesystem is going to scale better on desktops with spinning rust?

Cheers,

Dave.

Dave Chinner
[email protected]

The bigger reason why was because raw NAND flash doesn't really make sense any more; especially as the feature size of flash cells has shrunk and with the introduction of MLC and TLC, you really need to use hardware assist to make flash sufficiently reliable. Modern flash storage uses dynamic adjustment of voltage levels as the flash cells age, and error correcting codes to compensate for flash reliability challenges. This means accessing flash using eMMC, SATA, SAS, etc., and that rules out YAFFS2.

- Ted

Hi Dave,

On Mon, Jul 08, 2013 at 10:44:53PM +1000, Dave Chinner wrote:
[...]

I have a patch to fix this problem and the patch has been applied into 3.11-rc1. The patch is (d3922a77):
ext4: improve extent cache shrink mechanism to avoid to burn CPU time

I do really appreicate that if you could try your testing again against this patch. I just want to make sure that this problem has been fixed. At least in my own testing it looks fine.

Thanks,
- Zheng

I'll redo them when 3.11-rc1 comes around. I'll let you know how much better it is, and where the next ring of the onion lies.

Cheers,

Dave.

Dave Chinner
[email protected]

Quoting Dave Chinner (2013-07-08 08:44:53)

Sure, but you can't do that most workloads. Only on specialised workloads (e.g. hashed directory tree based object stores) is this really a viable option....

It will be interesting to see how these new structures play out ;)

Cheers,

Dave.

Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoting Dave Chinner (2013-07-08 21:26:14)

Yes and no. It makes a huge difference even when you have 8 procs all working on the same 8 subvolumes. It's not perfect but it's all I have ;)

The skiplists don't translate well to the tree roots, so I'll probably have to do something different there. But I'll get the onion peeled one way or another.

-chris

A quick follow on mmotm:

So i just ported all my prototype sync and inode_sb_list_lock changes across to mmotm, as well as the XFS CIL optimisations.

- 2.33% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 70.14% do_raw_spin_lock
- _raw_spin_lock
+ 16.91% _xfs_buf_find
+ 15.20% list_lru_add
+ 12.83% xfs_log_commit_cil
+ 11.18% d_alloc
+ 7.43% dput
+ 4.56% __d_instantiate
....

Most of the spinlock contention has gone away.

...

....

....

- 15.44% [kernel] [k] __ticket_spin_trylock
- __ticket_spin_trylock
- 46.59% _raw_spin_lock
+ 69.40% list_lru_add
17.65% list_lru_del
5.70% list_lru_count_node
2.44% shrink_dentry_list
prune_dcache_sb
super_cache_scan
shrink_slab
0.86% __page_check_address
- 33.06% do_raw_spin_lock
- _raw_spin_lock
+ 36.96% list_lru_add
+ 11.98% list_lru_del
+ 6.68% shrink_dentry_list
+ 6.43% d_alloc
+ 4.79% _xfs_buf_find
.....
+ 11.48% do_raw_spin_trylock
+ 8.87% _raw_spin_trylock

So now we see that CPU wasted on contention is down by 40%. Observation shows that most of the list_lru_add/list_lru_del contention occurs when reclaim is running - before memory filled up the lookup rate was on the high side of 600,000 inodes/s, but fell back to about 425,000/s once reclaim started working.

mmotm-cil 225 258k+-26k 122 296

So even with all the debug on, the mmotm kernel with most of the mods as I was running in 3.10-cil, plus the s_inodes ->list_lru conversion gets the same throughput for create and unlink and has much better walk times.

So, let's run these again on my current mmotm tree - it has the ext4 extent tree fixes in it and my rcu inode hash lookup patch...

	    create		 walk		unlink
	 time(s)   rate		time(s)		time(s)

xfs 225 258k+-26k 122 296
ext4 456 118k+- 4k 128 1632
btrfs 1122 51k+- 3k 281 3200(*)

(*) about 4.7 million inodes removed in 5 minutes.

ext4 is a lot healthier: create speed doubles from the extent cache lock contention fixes, and the walk time halves due to the rcu inode cache lookup. That said, it is still burning a huge amount of CPU on the inode_hash_lock adding and removing inodes. Unlink perf is a bit faster, but still slow. So, yeah, things will get better in the not-too distant future...

And for btrfs? Well, create is a tiny bit faster, the walk is 20% faster thanks to the rcu hash lookups, and unlinks are markedly faster (3x). Still not fast enough for me to hang around waiting for them to complete, though.

FWIW, while the results are a lot better for ext4, let me just point out how hard it is driving the storage to get that performance:

load | create | walk | unlink
IO type | write | read | read | write
| IOPS BW | IOPS BW | IOPS BW | IOPS BW
--------+------------+--------------+---------------+--------------
xfs | 900 200 | 18000 140 | 7500 50 | 400 50
ext4 |23000 390 | 55000 200 | 2000 10 | 13000 160
btrfs(*)|peaky 75 | 26000 100 | decay 10 | peaky peaky

ext4 is hammering the SSDs far harder than XFS, both in terms of IOPS and bandwidth. You do not want to run ext4 on your SSD if you have a metadata intensive workload as it will age the SSD much, much faster than XFS with that sort of write behaviour.

(*) the btrfs create IO pattern is 5s peaks of write IOPS every 30s. The baseline is about 500 IOPS, but the peaks reach upwards of 30,000 write IOPS. Unlink does this as well. There are also short bursts of 2-3000 read IOPS just before the write IOPS bursts in the create workload. For the unlink, it starts off with about 10,000 read IOPS, and goes quickly into exponential decay down to about 2000 read IOPS in 90s. Then it hits some trigger and the cycle starts again. The trigger appears to co-incide with the reclaim 1-2 million dentries being reclaimed.

Cheers,

Dave.

Dave Chinner
[email protected]