Skip to contentSkip to conversation

xfs: current patch stack for 2.6.38 window

PATCH In review Dave Chinner 3 participants 42 messages Nov 8, 2010 – Nov 10, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover xfs: current patch stack for 2.6.38 window

Folks,

FYI, here is my current XFS patch stack that I'll be trying to get ready in time for the 2.6.38 merge window. Note that the first two patches are candidates for 2.6.37-rc. They are a perag reference counting fix and the movement of a trace point.

My tree is currently based on the VFS locking changes I have out for review, so there's a couple fo patches that won't apply sanely to a mainline or OSS xfs dev tree. See below for a pointer to a git tree with all the patches in it.

First patch is a per-cpu superblock counter rewrite. This uses the generic per-cpu coutner infrastructure to do the heavy lifting. Needs to be split into two patches.

Following this is the dynamic speculative allocation patches. These have been rewritten to be base don the current inode size rather than a thumb-in-the-air how-many-preallocs-have-we-already-done algorithm. There are also some fixes in the second patch that fix assumptions about ip->i_delayed_blks being zero after a flush.

Next up we have the inode cache RCU freeing and lookup patches, including one that avoids putting the inode in the VFS hash (similar to Christoph's patch, but using the different VFS code).

Then there are buffer cache reclaim changes. First is a per-buftarg shrinker interface, followed by a lazily updated per-buftarg buffer LRU. building on this connecting up the prioritised buffer reclaim hooks that ensure more critical buffers are harder to reclaim.

AIL lock contention fixes are next, with bulk AIL insert and removal functions being implemented and connected up to the transaction commit and inode buffer IO completion routines. These significantly reduce AIL lock contention, and combined with a reduction in the granularity of xfsaild push wakeups, the AIL lock drops out of the "top 10" contended locks on ۸-way workloads.

There's a fix to avoid error injection from burning CPU on debug kernels - with a badly fragmented freespace tree, the btree block validation was taking ~60% of the CPU time, with most of that running error injection checks.

Finally, there's a patch to split up the log grant lock. This needs splitting into 4 or 5 smaller patches (as you can see it was originally from the commit log). It splits the grant lock into two list locks (reserve and write queues), and converts all the other variables that the grant lock protected into atomic variables. Grant head calculations are made atomic by converting them into 64 bit "LSNs" and the use of cmpxchg loops on atomic 64 bit variables. All log tail and sync LSNs updates are made atomic via conversion to atomic variables. With this, the grant lock goes away completely, and the transaction reserve fast path now only has two cmpxchg loops instead of a heavily contended spin lock.

The result of all this is raw cpu bound 8-way create performance of just over 100,000 inodes/s, and unlink performance of over 90,000 inodes/s. 8-way dbench performance is improved from ~1150MB/s to ~1650MB/s by this patchset.

For 8-way creation and unlink of small files (~50 million), the lockstat profiles look like:

contended total Lock
Lock acquistions acquisitions Description
----------------------------- ----------- ------------ -------------------

           inode_wb_list_lock:    496330785    836287347	VFS
                  dcache_lock:    116299583    681450027	VFS
        &(&vblk->lock)->rlock:     52829329    131054495	virtio block device
    &sb->s_type->i_lock_key#1:     41772196   2375571240	VFS (inode->i_lock)
  &(&cil->xc_cil_lock)->rlock:     29549897    410553961	XFS (CIL commit lock)
         &irq_desc_lock_class:     27520142     63908701	IRQ edge lock
 &(&pag->pag_buf_lock)->rlock:     11756249   1838039685	XFS (buffer cache lock)
    &(&dentry->d_lock)->rlock:      5735657   1225028487	VFS
 &(&parent->list_lock)->rlock:      4356293    249408696	VM (SLAB list lock)
           inode_sb_list_lock:      3616366    203712449	VFS
                        key#5:      2075310    139221312	XFS SB percpu counter
              inode_hash_lock:      1529969    102359626	VFS
             rcu_node_level_0:      1363470     13730113	RCU
        &(&zone->lock)->rlock:      1247467     16469316	VM (free list lock)
 &(&pag->pag_ici_lock)->rlock:       770880    337090972	XFS (inode cache lock)
                    &rq->lock:       589111    184220946	Scheduler
               inode_lru_lock:       527163    102791204	VFS
g->l_grant_write_lock)->rlock:       526471     51279626	XFS (grant write lock)
    &(&pag->pagb_lock)->rlock:       402878    208861744	XFS (busy extent list)
    &(&zone->lru_lock)->rlock:       167692     25383748	VM (page cache LRU)
              &on_slab_l3_key:       166183     58470153	VM (slab cache)
            semaphore->lock#2:       161321   3659173925	???
     &(&ailp->xa_lock)->rlock:       143859    164470123	XFS (AIL lock)
          &cil->xc_ctx_lock-W:        32850       173279	XFS (CIL push lock)
          &cil->xc_ctx_lock-R:        90868    357572724	XFS (CIL push lock)

I'm still to determine if I'll have the time to finish the removal of the page cache from the buffer cache yet - for pure inode create/unlink workloads the buftarg mapping tree lock is the second most heavily contended lock in the system. Hence this definitely needs solving in some way or another....

Anyway, comments are welcome - just keep in mind that there is still some polish required for these patches. ;)

If you want the git version, everything is here:

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git working

Dave Chinner (16):
xfs: fix per-ag reference counting in inode reclaim tree walking
xfs: move delayed write buffer trace
[RFC] xfs: use generic per-cpu counter infrastructure
xfs: dynamic speculative EOF preallocation
xfs: don't truncate prealloc from frequently accessed inodes patch xfs-inode-hash-fake
xfs: convert inode cache lookups to use RCU locking
xfs: convert pag_ici_lock to a spin lock
xfs: convert xfsbud shrinker to a per-buftarg shrinker.
xfs: add a lru to the XFS buffer cache
xfs: connect up buffer reclaim priority hooks
xfs: bulk AIL insertion during transaction commit
xfs: reduce the number of AIL push wakeups
xfs: remove all the inodes on a buffer from the AIL in bulk
xfs: only run xfs_error_test if error injection is active
xfs: make xlog_space_left() independent of the grant lock

 fs/xfs/linux-2.6/xfs_buf.c     |  239 ++++++++----
 fs/xfs/linux-2.6/xfs_buf.h     |   43 ++-
 fs/xfs/linux-2.6/xfs_iops.c    |   11 +-
 fs/xfs/linux-2.6/xfs_linux.h   |    9 -
 fs/xfs/linux-2.6/xfs_super.c   |   22 +-
 fs/xfs/linux-2.6/xfs_sync.c    |   28 +-
 fs/xfs/linux-2.6/xfs_trace.h   |   36 +-
 fs/xfs/quota/xfs_dquot.c       |    2 +-
 fs/xfs/quota/xfs_qm_syscalls.c |    3 +
 fs/xfs/xfs_ag.h                |    2 +-
 fs/xfs/xfs_alloc.c             |    4 +-
 fs/xfs/xfs_bmap.c              |    9 +-
 fs/xfs/xfs_btree.c             |   11 +-
 fs/xfs/xfs_buf_item.c          |   17 +-
 fs/xfs/xfs_da_btree.c          |    4 +-
 fs/xfs/xfs_dfrag.c             |   13 +
 fs/xfs/xfs_error.c             |    3 +
 fs/xfs/xfs_error.h             |    5 +-
 fs/xfs/xfs_extfree_item.c      |   85 +++--
 fs/xfs/xfs_extfree_item.h      |   12 +-
 fs/xfs/xfs_fsops.c             |    4 +-
 fs/xfs/xfs_ialloc.c            |    2 +-
 fs/xfs/xfs_iget.c              |   55 ++-
 fs/xfs/xfs_inode.c             |   24 +-
 fs/xfs/xfs_inode.h             |    1 +
 fs/xfs/xfs_inode_item.c        |  112 +++++-
 fs/xfs/xfs_iomap.c             |   53 ++-
 fs/xfs/xfs_log.c               |  678 +++++++++++++++++---------------
 fs/xfs/xfs_log_cil.c           |    9 +-
 fs/xfs/xfs_log_priv.h          |   40 ++-
 fs/xfs/xfs_log_recover.c       |   27 +-
 fs/xfs/xfs_mount.c             |  837 +++++++++++-----------------------------
 fs/xfs/xfs_mount.h             |   80 +---
 fs/xfs/xfs_trans.c             |   70 ++++-
 fs/xfs/xfs_trans.h             |    2 +-
 fs/xfs/xfs_trans_ail.c         |  189 ++++++++-
 fs/xfs/xfs_trans_extfree.c     |    4 +-
 fs/xfs/xfs_trans_priv.h        |   13 +-
 fs/xfs/xfs_vnodeops.c          |   61 ++-
 include/linux/percpu_counter.h |   16 +
 lib/percpu_counter.c           |   79 ++++
 41 files changed, 1593 insertions(+), 1321 deletions(-)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/16 xfs: fix per-ag reference counting in inode reclaim tree walking

From: Dave Chinner <[email protected]>

The walk fails to decrement the per-ag reference count when the non-blocking walk fails to obtain the per-ag reclaim lock, leading to an assert failure on debug kernels when unmounting a filesystem.

---
 fs/xfs/linux-2.6/xfs_sync.c |    1 +
 fs/xfs/xfs_mount.c          |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)
fs/xfs/linux-2.6/xfs_sync.c +1 -0
index 37d3325..afb0d7c 100644
@@ -853,6 +853,7 @@ restart:
 		if (trylock) {
 			if (!mutex_trylock(&pag->pag_ici_reclaim_lock)) {
 				skipped++;
+				xfs_perag_put(pag);
 				continue;
 			}
 			first_index = pag->pag_ici_reclaim_cursor;
fs/xfs/xfs_mount.c +1 -0
index b1498ab..19e9dfa 100644
@@ -275,6 +275,7 @@ xfs_free_perag(
 		pag = radix_tree_delete(&mp->m_perag_tree, agno);
 		spin_unlock(&mp->m_perag_lock);
 		ASSERT(pag);
+		ASSERT(atomic_read(&pag->pag_ref) == 0);
 		call_rcu(&pag->rcu_head, __xfs_free_perag);
 	}
 }
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 1/16
#2 lore raw

Re: [PATCH 01/16] xfs: fix per-ag reference counting in inode reclaim tree walking

fs/xfs/linux-2.6/xfs_sync.c restart: · line 859

Quoted 9 lines

+++ b/fs/xfs/linux-2.6/xfs_sync.c

fs/xfs/linux-2.6/xfs_sync.c +1 -0
@@ -853,6 +853,7 @@ restart:
 		if (trylock) {
 			if (!mutex_trylock(&pag->pag_ici_reclaim_lock)) {
 				skipped++;
+				xfs_perag_put(pag);
 				continue;
 			}
 			first_index = pag->pag_ici_reclaim_cursor;

One way to make this loop more maintainable is to split the guts of it into a xfs_reclaim_inodes_ag helper, and make the existing function a wrapper around it (and remove the incorrect _ag prefix), but that's .38 material.

For .37 the patch looks good,

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

PATCH 2/16 xfs: move delayed write buffer trace

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The delayed write buffer split trace currently issues a trace for every buffer it scans. These buffers are not necessarily queued for delayed write. Indeed, when buffers are pinned, there can be thousands of traces of buffers that aren't actually queued for delayed write and the ones that are are lost in the noise. Move the trace point to record only buffers that are split out for IO to be issued on.

---
 fs/xfs/linux-2.6/xfs_buf.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
fs/xfs/linux-2.6/xfs_buf.c +1 -1
index 63fd2c0..aa1d353 100644
@@ -1781,7 +1781,6 @@ xfs_buf_delwri_split(
 	INIT_LIST_HEAD(list);
 	spin_lock(dwlk);
 	list_for_each_entry_safe(bp, n, dwq, b_list) {
-		trace_xfs_buf_delwri_split(bp, _RET_IP_);
 		ASSERT(bp->b_flags & XBF_DELWRI);
 
 		if (!XFS_BUF_ISPINNED(bp) && !xfs_buf_cond_lock(bp)) {
@@ -1795,6 +1794,7 @@ xfs_buf_delwri_split(
 					 _XBF_RUN_QUEUES);
 			bp->b_flags |= XBF_WRITE;
 			list_move_tail(&bp->b_list, list);
+			trace_xfs_buf_delwri_split(bp, _RET_IP_);
 		} else
 			skipped++;
 	}
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 2/16
#4 lore raw

Re: [PATCH 02/16] xfs: move delayed write buffer trace

Quoted 9 lines · Dave Chinner

From: Dave Chinner <[email protected]>

The delayed write buffer split trace currently issues a trace for every buffer it scans. These buffers are not necessarily queued for delayed write. Indeed, when buffers are pinned, there can be thousands of traces of buffers that aren't actually queued for delayed write and the ones that are are lost in the noise. Move the trace point to record only buffers that are split out for IO to be issued on.

Looks good,

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

RFC PATCH 3/16 xfs: use generic per-cpu counter infrastructure

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

XFS has a per-cpu counter implementation for in-core superblock counters that pre-dated the generic implementation. It is complex and baroque as it is tailored directly to the needs of ENOSPC detection. Implement the complex accurate-compare-and-add calculation in the generic per-cpu counter code and convert the XFS counters to use the much simpler generic counter code.

Passes xfsqa on SMP system.

Still to do:

1. UP build and test.
2. split into separate patches

For discussion:
1. kill the no-per-cpu-counter mode?
2. does we need a custom batch size?
3. do we need to factor xfs_mod_sb_incore()?
4. should all the readers just sum the counters themselves and kill the wrapperѕ?

---
 fs/xfs/linux-2.6/xfs_linux.h   |    9 -
 fs/xfs/linux-2.6/xfs_super.c   |    4 +-
 fs/xfs/xfs_fsops.c             |    4 +-
 fs/xfs/xfs_mount.c             |  834 +++++++++++-----------------------------
 fs/xfs/xfs_mount.h             |   80 +---
 include/linux/percpu_counter.h |   16 +
 lib/percpu_counter.c           |   79 ++++
 7 files changed, 341 insertions(+), 685 deletions(-)
fs/xfs/linux-2.6/xfs_linux.h +0 -9
index 214ddd7..9fa4f2a 100644
@@ -88,15 +88,6 @@
 #include <xfs_super.h>
 #include <xfs_buf.h>
 
-/*
- * Feature macros (disable/enable)
- */
-#ifdef CONFIG_SMP
-#define HAVE_PERCPU_SB	/* per cpu superblock counters are a 2.6 feature */
-#else
-#undef  HAVE_PERCPU_SB	/* per cpu superblock counters are a 2.6 feature */
-#endif
-
 #define irix_sgid_inherit	xfs_params.sgid_inherit.val
 #define irix_symlink_mode	xfs_params.symlink_mode.val
 #define xfs_panic_mask		xfs_params.panic_mask.val
fs/xfs/linux-2.6/xfs_super.c +2 -2
index 53ab47f..fa789b7 100644
@@ -1230,9 +1230,9 @@ xfs_fs_statfs(
 	statp->f_fsid.val[0] = (u32)id;
 	statp->f_fsid.val[1] = (u32)(id >> 32);
 
-	xfs_icsb_sync_counters(mp, XFS_ICSB_LAZY_COUNT);
-
 	spin_lock(&mp->m_sb_lock);
+	xfs_icsb_sync_counters_locked(mp);
+
 	statp->f_bsize = sbp->sb_blocksize;
 	lsize = sbp->sb_logstart ? sbp->sb_logblocks : 0;
 	statp->f_blocks = sbp->sb_dblocks - lsize;
fs/xfs/xfs_fsops.c +2 -2
index a7c116e..44ecf1b 100644
@@ -478,7 +478,7 @@ xfs_fs_counts(
 	xfs_mount_t		*mp,
 	xfs_fsop_counts_t	*cnt)
 {
-	xfs_icsb_sync_counters(mp, XFS_ICSB_LAZY_COUNT);
+	xfs_icsb_sync_counters(mp);
 	spin_lock(&mp->m_sb_lock);
 	cnt->freedata = mp->m_sb.sb_fdblocks - XFS_ALLOC_SET_ASIDE(mp);
 	cnt->freertx = mp->m_sb.sb_frextents;
@@ -540,7 +540,7 @@ xfs_reserve_blocks(
 	 */
 retry:
 	spin_lock(&mp->m_sb_lock);
-	xfs_icsb_sync_counters_locked(mp, 0);
+	xfs_icsb_sync_counters_locked(mp);
 
 	/*
 	 * If our previous reservation was larger than the current value,
fs/xfs/xfs_mount.c +220 -614

891 lines not on this page · raw message

fs/xfs/xfs_mount.h +22 -58

116 lines not on this page · raw message

include/linux/percpu_counter.h +16 -0
index 46f6ba5..32014a4 100644
@@ -41,6 +41,8 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch);
 s64 __percpu_counter_sum(struct percpu_counter *fbc);
 int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs);
+int percpu_counter_add_unless_lt(struct percpu_counter *fbc, s64 amount,
+							s64 threshold);
 
 static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount)
 {
@@ -153,6 +155,20 @@ static inline int percpu_counter_initialized(struct percpu_counter *fbc)
 	return 1;
 }
 
+static inline int percpu_counter_test_and_add_delta(struct percpu_counter *fbc, s64 delta)
+{
+	s64 count;
+
+	preempt_disable();
+	count = fbc->count + delta;
+	if (count < 0)
+		return -1;
+	fbc->count = count;
+	preempt_enable();
+	return count ? 1 : 0;
+}
+
+
 #endif	/* CONFIG_SMP */
 
 static inline void percpu_counter_inc(struct percpu_counter *fbc)
lib/percpu_counter.c +79 -0

85 lines not on this page · raw message

Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw

Re: [PATCH 03/16] [RFC] xfs: use generic per-cpu counter infrastructure

Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

XFS has a per-cpu counter implementation for in-core superblock counters that pre-dated the generic implementation. It is complex and baroque as it is tailored directly to the needs of ENOSPC detection. Implement the complex accurate-compare-and-add calculation in the generic per-cpu counter code and convert the XFS counters to use the much simpler generic counter code.

Passes xfsqa on SMP system.

Some mostly cosmetic comments below. I haven't looked at the more hairy bits like the changes to the generic percpu code and the reservation handling yet.

Quoted 1 line

1. kill the no-per-cpu-counter mode?

already done.

Quoted 1 line

3. do we need to factor xfs_mod_sb_incore()?

Doesn't exist anymore.

fs/xfs/xfs_fsops.c xfs_fs_counts( · line 482

Quoted 3 lines
fs/xfs/xfs_fsops.c +1 -1
-	xfs_icsb_sync_counters(mp, XFS_ICSB_LAZY_COUNT);
+	xfs_icsb_sync_counters(mp);
 	spin_lock(&mp->m_sb_lock);

Can be moved inside the lock and use the unlocked version, too.

Quoted 30 lines
fs/xfs/xfs_mount.c +30 -0
+static inline int
+xfs_icsb_add(
+	struct xfs_mount	*mp,
+	int			counter,
+	int64_t			delta,
+	int64_t			threshold)
+{
+	int			ret;
+
+	ret = percpu_counter_add_unless_lt(&mp->m_icsb[counter], delta,
+								threshold);
+	if (ret < 0)
+		return -ENOSPC;
+	return 0;
+}
+
+static inline void
+xfs_icsb_set(
+	struct xfs_mount	*mp,
+	int			counter,
+	int64_t			value)
+{
+	percpu_counter_set(&mp->m_icsb[counter], value);
+}
+
+static inline int64_t
+xfs_icsb_sum(
+	struct xfs_mount	*mp,
+	int			counter)
+{

fs/xfs/xfs_mount.c xfs_free_perag( · line 324

Quoted 10 lines
fs/xfs/xfs_mount.c +10 -0
+	return percpu_counter_sum_positive(&mp->m_icsb[counter]);
+}
+
+static inline int64_t
+xfs_icsb_read(
+	struct xfs_mount	*mp,
+	int			counter)
+{
+	return percpu_counter_read_positive(&mp->m_icsb[counter]);
+}

I would just opencode all these helpers in their callers. There's generally just one caller of each, which iterates over the three counters anyway.

fs/xfs/xfs_mount.c unwind: · line 2007

Quoted 6 lines
fs/xfs/xfs_mount.c +6 -0
+int
+xfs_icsb_modify_counters(
+	xfs_mount_t	*mp,
+	xfs_sb_field_t	field,
+	int64_t		delta,
+	int		rsvd)

I can't see the point of keeping this multiplexer. The inode counts are handled entirely different from the block count, so they should have separate functions.

Quoted 12 lines
fs/xfs/xfs_mount.c +12 -0
+{
+	int64_t		lcounter;
+	int64_t		res_used;
+	int		ret = 0;
+
+
+	switch (field) {
+	case XFS_SBS_ICOUNT:
+		ret = xfs_icsb_add(mp, XFS_ICSB_ICOUNT, delta, 0);
+		if (ret < 0) {
+			ASSERT(0);
+			return XFS_ERROR(EINVAL);

fs/xfs/xfs_mount.c unwind: · line 2029

Quoted 10 lines
fs/xfs/xfs_mount.c +10 -0
+		}
+		return 0;
+
+	case XFS_SBS_IFREE:
+		ret = xfs_icsb_add(mp, XFS_ICSB_IFREE, delta, 0);
+		if (ret < 0) {
+			ASSERT(0);
+			return XFS_ERROR(EINVAL);
+		}
+		return 0;

If you're keeping a common helper for both inode counts this can be simplified by sharing the code and just passing on the field instead of having two cases.

fs/xfs/xfs_mount.h typedef struct xfs_mount { · line 174

Quoted 1 line
fs/xfs/xfs_mount.h +1 -0
+	struct percpu_counter	m_icsb[XFS_ICSB_MAX];

I wonder if there's all that much of a point in keeping the array. We basically only use the fact it's an array for the init/destroy code. Maybe it would be a tad cleaner to just have three separate percpu counters.

fs/xfs/xfs_mount.h extern int xfs_sb_validate_fsb_count(struct xfs_sb *, __uint64_t); · line 344

Quoted 8 lines
fs/xfs/xfs_mount.h +8 -0
+static inline void
+xfs_icsb_sync_counters(
+	struct xfs_mount	*mp)
+{
+	spin_lock(&mp->m_sb_lock);
+	xfs_icsb_sync_counters_locked(mp);
+	spin_unlock(&mp->m_sb_lock);
+}

There's only one callers of this left after my comment above is addressed. I'd just make xfs_icsb_sync_counters the locked version, throw in an assert_spin_locked and have the one remaining caller take the lock opencoded as well.

Quoted 12 lines
include/linux/percpu_counter.h +2 -0
@@ -41,6 +41,8 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch);
 s64 __percpu_counter_sum(struct percpu_counter *fbc);
 int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs);
+int percpu_counter_add_unless_lt(struct percpu_counter *fbc, s64 amount,
+							s64 threshold);
 
 static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount)
 {

include/linux/percpu_counter.h static inline int percpu_counter_initialized(struct percpu_counter *fbc) · line 158

Quoted 4 lines
include/linux/percpu_counter.h +1 -0
@@ -153,6 +155,20 @@ static inline int percpu_counter_initialized(struct percpu_counter *fbc)
 	return 1;
 }
 
+static inline int percpu_counter_test_and_add_delta(struct percpu_counter *fbc, s64 delta)

This doesn't match the function provided for CONFIG_SMP.

include/linux/percpu_counter.h static inline int percpu_counter_initialized(struct percpu_counter *fbc) · line 160

Quoted 2 lines
include/linux/percpu_counter.h +2 -0
+/**
+ *

spurious line.

include/linux/percpu_counter.h static inline int percpu_counter_initialized(struct percpu_counter *fbc) · line 162

Quoted 2 lines
include/linux/percpu_counter.h +2 -0
+int percpu_counter_add_unless_lt(struct percpu_counter *fbc, s64 amount, s64
+threshold)

too long line

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw

Re: [PATCH 03/16] [RFC] xfs: use generic per-cpu counter infrastructure

Quoted 16 lines · Christoph Hellwig
Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

XFS has a per-cpu counter implementation for in-core superblock counters that pre-dated the generic implementation. It is complex and baroque as it is tailored directly to the needs of ENOSPC detection. Implement the complex accurate-compare-and-add calculation in the generic per-cpu counter code and convert the XFS counters to use the much simpler generic counter code.

Passes xfsqa on SMP system.

Some mostly cosmetic comments below. I haven't looked at the more hairy bits like the changes to the generic percpu code and the reservation handling yet.

Quoted 7 lines
Quoted 1 line

1. kill the no-per-cpu-counter mode?

already done.

Quoted 1 line

3. do we need to factor xfs_mod_sb_incore()?

Doesn't exist anymore.

Ah, forgot to update the commit message ;)

Quoted 5 lines

fs/xfs/xfs_fsops.c xfs_fs_counts( · line 482

Quoted 3 lines
fs/xfs/xfs_fsops.c +1 -1
-	xfs_icsb_sync_counters(mp, XFS_ICSB_LAZY_COUNT);
+	xfs_icsb_sync_counters(mp);
 	spin_lock(&mp->m_sb_lock);

Can be moved inside the lock and use the unlocked version, too.

OK, I just went for the straight transformation approach.

Quoted 36 lines
Quoted 36 lines
fs/xfs/xfs_mount.c +36 -0
+static inline int
+xfs_icsb_add(
+	struct xfs_mount	*mp,
+	int			counter,
+	int64_t			delta,
+	int64_t			threshold)
+{
+	int			ret;
+
+	ret = percpu_counter_add_unless_lt(&mp->m_icsb[counter], delta,
+								threshold);
+	if (ret < 0)
+		return -ENOSPC;
+	return 0;
+}
+
+static inline void
+xfs_icsb_set(
+	struct xfs_mount	*mp,
+	int			counter,
+	int64_t			value)
+{
+	percpu_counter_set(&mp->m_icsb[counter], value);
+}
+
+static inline int64_t
+xfs_icsb_sum(
+	struct xfs_mount	*mp,
+	int			counter)
+{
+	return percpu_counter_sum_positive(&mp->m_icsb[counter]);
+}
+
+static inline int64_t
+xfs_icsb_read(
+	struct xfs_mount	*mp,
Quoted 8 lines

fs/xfs/xfs_mount.c xfs_free_perag( · line 324

Quoted 4 lines
fs/xfs/xfs_mount.c +4 -0
+	int			counter)
+{
+	return percpu_counter_read_positive(&mp->m_icsb[counter]);
+}

I would just opencode all these helpers in their callers. There's generally just one caller of each, which iterates over the three counters anyway.

That seems reasonable, but I had is a good reason for adding the wrappers. That is, I'm not sure that the fixed percpu counter batch size (32) scales well enough for large systems. In the bdi code, a custom batch size that is logarithmicaly scaled with the number of CPUs is used and I suspect we'll need to do this here, too. Hence I'd like to keep the wrappers to minimise the number of places we'd need to modify to handle customised batch sizes.

Quoted 10 lines

fs/xfs/xfs_mount.c unwind: · line 2007

Quoted 6 lines
fs/xfs/xfs_mount.c +6 -0
+int
+xfs_icsb_modify_counters(
+	xfs_mount_t	*mp,
+	xfs_sb_field_t	field,
+	int64_t		delta,
+	int		rsvd)

I can't see the point of keeping this multiplexer. The inode counts are handled entirely different from the block count, so they should have separate functions.

I just went for the simple approach - I wanted to get it working without having to modify lots of other code. Now that it is working, I can see why getting rid of the wrapper altogether would be good.

Quoted 28 lines
Quoted 13 lines
fs/xfs/xfs_mount.c +12 -0
+{
+	int64_t		lcounter;
+	int64_t		res_used;
+	int		ret = 0;
+
+
+	switch (field) {
+	case XFS_SBS_ICOUNT:
+		ret = xfs_icsb_add(mp, XFS_ICSB_ICOUNT, delta, 0);
+		if (ret < 0) {
+			ASSERT(0);
+			return XFS_ERROR(EINVAL);

fs/xfs/xfs_mount.c unwind: · line 2029

Quoted 10 lines
fs/xfs/xfs_mount.c +10 -0
+		}
+		return 0;
+
+	case XFS_SBS_IFREE:
+		ret = xfs_icsb_add(mp, XFS_ICSB_IFREE, delta, 0);
+		if (ret < 0) {
+			ASSERT(0);
+			return XFS_ERROR(EINVAL);
+		}
+		return 0;

If you're keeping a common helper for both inode counts this can be simplified by sharing the code and just passing on the field instead of having two cases.

Quoted 6 lines

fs/xfs/xfs_mount.h typedef struct xfs_mount { · line 174

Quoted 1 line
fs/xfs/xfs_mount.h +1 -0
+	struct percpu_counter	m_icsb[XFS_ICSB_MAX];

I wonder if there's all that much of a point in keeping the array. We basically only use the fact it's an array for the init/destroy code. Maybe it would be a tad cleaner to just have three separate percpu counters.

Not sure - I'd like to extend the per-cpu counters to more fields in the superblock (e.g. the rt extent counter), and having an array makes that pretty simple...

Quoted 14 lines
Quoted 14 lines

+++ b/include/linux/percpu_counter.h

include/linux/percpu_counter.h +2 -0
@@ -41,6 +41,8 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount);
 void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch);
 s64 __percpu_counter_sum(struct percpu_counter *fbc);
 int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs);
+int percpu_counter_add_unless_lt(struct percpu_counter *fbc, s64 amount,
+							s64 threshold);
 
 static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount)
 {
@@ -153,6 +155,20 @@ static inline int percpu_counter_initialized(struct percpu_counter *fbc)
 	return 1;
 }
 
Quoted 4 lines

include/linux/percpu_counter.h static inline int percpu_counter_initialized(struct percpu_counter *fbc) · line 158

Quoted 1 line
include/linux/percpu_counter.h +1 -0
+static inline int percpu_counter_test_and_add_delta(struct percpu_counter *fbc, s64 delta)

This doesn't match the function provided for CONFIG_SMP.

Doh - I hadn't retested UP since I renamed the function that did all the work.

And I just realised that with UP using the icsb functions, I can kill all the cases in the locked variant....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

PATCH 4/16 xfs: dynamic speculative EOF preallocation

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Currently the size of the speculative preallocation during delayed allocation is fixed by either the allocsize mount option of a default size. We are seeing a lot of cases where we need to recommend using the allocsize mount option to prevent fragmentation when buffered writes land in the same AG.

Rather than using a fixed preallocation size by default (up to 64k), make it dynamic by basing it on the current inode size. That way the EOF preallocation will increase as the file size increases. Hence for streaming writes we are much more likely to get large preallocations exactly when we need it to reduce fragementation.

For default settings, ひe size and the initial extents is determined by the number of parallel writers and the amount of memory in the machine. For 4GB RAM and 4 concurrent 32GB file writes:

EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL

   0: [0..1048575]:         1048672..2097247      0 (1048672..2097247)      1048576
   1: [1048576..2097151]:   5242976..6291551      0 (5242976..6291551)      1048576
   2: [2097152..4194303]:   12583008..14680159    0 (12583008..14680159)    2097152
   3: [4194304..8388607]:   25165920..29360223    0 (25165920..29360223)    4194304
   4: [8388608..16777215]:  58720352..67108959    0 (58720352..67108959)    8388608
   5: [16777216..33554423]: 117440584..134217791  0 (117440584..134217791) 16777208
   6: [33554424..50331511]: 184549056..201326143  0 (184549056..201326143) 16777088
   7: [50331512..67108599]: 251657408..268434495  0 (251657408..268434495) 16777088

and for 16 concurrent 16GB file writes:

 EXT: FILE-OFFSET           BLOCK-RANGE          AG AG-OFFSET                 TOTAL
   0: [0..262143]:          2490472..2752615      0 (2490472..2752615)       262144
   1: [262144..524287]:     6291560..6553703      0 (6291560..6553703)       262144
   2: [524288..1048575]:    13631592..14155879    0 (13631592..14155879)     524288
   3: [1048576..2097151]:   30408808..31457383    0 (30408808..31457383)    1048576
   4: [2097152..4194303]:   52428904..54526055    0 (52428904..54526055)    2097152
   5: [4194304..8388607]:   104857704..109052007  0 (104857704..109052007)  4194304
   6: [8388608..16777215]:  209715304..218103911  0 (209715304..218103911)  8388608

7: [16777216..33554423]: 452984848..469762055 0 (452984848..469762055) 16777208

The allocsize mount option still controls the minimum preallocation size, so the smallest extent size can stil be bound in situations where this behaviour is not sufficient.

---
 fs/xfs/xfs_iomap.c |   53 ++++++++++++++++++++++++++++++++++++++++++---------
 1 files changed, 43 insertions(+), 10 deletions(-)
fs/xfs/xfs_iomap.c +43 -10
index 2057614..0227ac1 100644
@@ -389,6 +389,9 @@ error_out:
  * If the caller is doing a write at the end of the file, then extend the
  * allocation out to the file system's write iosize.  We clean up any extra
  * space left over when the file is closed in xfs_inactive().
+ *
+ * If we find we already have delalloc preallocation beyond EOF, don't do more
+ * preallocation as it it not needed.
  */
 STATIC int
 xfs_iomap_eof_want_preallocate(
@@ -405,6 +408,7 @@ xfs_iomap_eof_want_preallocate(
 	xfs_filblks_t   count_fsb;
 	xfs_fsblock_t	firstblock;
 	int		n, error, imaps;
+	int		found_delalloc = 0;
 
 	*prealloc = 0;
 	if ((offset + count) <= ip->i_size)
@@ -427,11 +431,16 @@ xfs_iomap_eof_want_preallocate(
 			if ((imap[n].br_startblock != HOLESTARTBLOCK) &&
 			    (imap[n].br_startblock != DELAYSTARTBLOCK))
 				return 0;
+
 			start_fsb += imap[n].br_blockcount;
 			count_fsb -= imap[n].br_blockcount;
+
+			if (imap[n].br_startblock == DELAYSTARTBLOCK)
+				found_delalloc = 1;
 		}
 	}
-	*prealloc = 1;
+	if (!found_delalloc)
+		*prealloc = 1;
 	return 0;
 }
 
@@ -469,6 +478,7 @@ xfs_iomap_write_delay(
 	extsz = xfs_get_extsz_hint(ip);
 	offset_fsb = XFS_B_TO_FSBT(mp, offset);
 
+
 	error = xfs_iomap_eof_want_preallocate(mp, ip, offset, count,
 				ioflag, imap, XFS_WRITE_IMAPS, &prealloc);
 	if (error)
@@ -476,9 +486,23 @@ xfs_iomap_write_delay(
 
 retry:
 	if (prealloc) {
+		xfs_fileoff_t	alloc_blocks = 0;
+		/*
+		 * If we don't have a user specified preallocation size, dynamically
+		 * increase the preallocation size as the size of the file
+		 * grows. Cap the maximum size at a single extent.
+		 */
+		if (!(mp->m_flags & XFS_MOUNT_DFLT_IOSIZE)) {
+			alloc_blocks = XFS_B_TO_FSB(mp, ip->i_size);
+			alloc_blocks = XFS_FILEOFF_MIN(MAXEXTLEN,
+					rounddown_pow_of_two(alloc_blocks));
+		}
+		if (alloc_blocks < mp->m_writeio_blocks)
+			alloc_blocks = mp->m_writeio_blocks;
+
 		aligned_offset = XFS_WRITEIO_ALIGN(mp, (offset + count - 1));
 		ioalign = XFS_B_TO_FSBT(mp, aligned_offset);
-		last_fsb = ioalign + mp->m_writeio_blocks;
+		last_fsb = ioalign + alloc_blocks;
 	} else {
 		last_fsb = XFS_B_TO_FSB(mp, ((xfs_ufsize_t)(offset + count)));
 	}
@@ -496,22 +520,31 @@ retry:
 			  XFS_BMAPI_DELAY | XFS_BMAPI_WRITE |
 			  XFS_BMAPI_ENTIRE, &firstblock, 1, imap,
 			  &nimaps, NULL);
-	if (error && (error != ENOSPC))
+	switch (error) {
+	case 0:
+	case ENOSPC:
+	case EDQUOT:
+		break;
+	default:
 		return XFS_ERROR(error);
+	}
 
 	/*
-	 * If bmapi returned us nothing, and if we didn't get back EDQUOT,
-	 * then we must have run out of space - flush all other inodes with
-	 * delalloc blocks and retry without EOF preallocation.
+	 * If bmapi returned us nothing, we got either ENOSPC or EDQUOT.  For
+	 * ENOSPC, * flush all other inodes with delalloc blocks to free up
+	 * some of the excess reserved metadata space. For both cases, retry
+	 * without EOF preallocation.
 	 */
 	if (nimaps == 0) {
 		trace_xfs_delalloc_enospc(ip, offset, count);
 		if (flushed)
-			return XFS_ERROR(ENOSPC);
+			return XFS_ERROR(error ? error : ENOSPC);
 
-		xfs_iunlock(ip, XFS_ILOCK_EXCL);
-		xfs_flush_inodes(ip);
-		xfs_ilock(ip, XFS_ILOCK_EXCL);
+		if (error == ENOSPC) {
+			xfs_iunlock(ip, XFS_ILOCK_EXCL);
+			xfs_flush_inodes(ip);
+			xfs_ilock(ip, XFS_ILOCK_EXCL);
+		}
 
 		flushed = 1;
 		error = 0;
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw

Re: [PATCH 04/16] xfs: dynamic speculative EOF preallocation

Quoted 1 line

For default settings, ???e size and the initial extents is determined

weird character.

Quoted 3 lines

The allocsize mount option still controls the minimum preallocation size, so the smallest extent size can stil be bound in situations where this behaviour is not sufficient.

Do we also need a way to keep an upper boundary? Think lots of slowly growing log files on a filesystem not having tons of free space.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw

Re: [PATCH 04/16] xfs: dynamic speculative EOF preallocation

Quoted 10 lines · Christoph Hellwig
Quoted 1 line

For default settings, ???e size and the initial extents is determined

weird character.

Quoted 3 lines

The allocsize mount option still controls the minimum preallocation size, so the smallest extent size can stil be bound in situations where this behaviour is not sufficient.

Do we also need a way to keep an upper boundary? Think lots of slowly growing log files on a filesystem not having tons of free space.

Perhaps - it's one of the things I've been debating backwards and forwards and done nothing about yet.

It's hard to trim back preallocation before we hit ENOSPC via a static threshold (e.g. 1% free space could be terabytes of space), but once ENOSPC is hit we drop new preallocation completely. Perhaps a gradual decrease in the maximum prealloc size based on freespace remaining? e.g.

freespace max prealloc size

Quoted 1 line

5% full extent (8GB)

4-5% 2GB (8GB >> 2)
3-4% 1GB (8GB >> 3)
2-3% 512MB (8GB >> 4)
1-2% 256MB (8GB >> 5)
<1% 128MB (8GB >> 6)

I'm open to other ideas on what to do here.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw

PATCH 5/16 xfs: don't truncate prealloc from frequently accessed inodes

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

A long standing problem for streaming writeѕ through the NFS server has been that the NFS server opens and closes file descriptors on an inode for every write. The result of this behaviour is that the ->release() function is called on every close and that results in XFS truncating speculative preallocation beyond the EOF. This has an adverse effect on file layout when multiple files are being written at the same time - they interleave their extents and can result in severe fragmentation.

To avoid this problem, keep a count of the number of ->release calls made on an inode. For most cases, an inode is only going to be opened once for writing and then closed again during it's lifetime in cache. Hence if there are multiple ->release calls, there is a good chance that the inode is being accessed by the NFS server. Hence count up every time ->release is called while there are delalloc blocks still outstanding on the inode.

If this count is non-zero when ->release is next called, then do no truncate away the speculative preallocation - leave it there so that subsequent writes do not need to reallocate the delalloc space. This will prevent interleaving of extents of different inodes written concurrently to the same AG.

If we get this wrong, it is not a big deal as we truncate speculative allocation beyond EOF anyway in xfs_inactive() when the inode is thrown out of the cache.

The new counter in the struct xfs_inode fits into a hole in the structure on 64 bit machines, so does not grow the size of the inode at all.

---
 fs/xfs/xfs_bmap.c     |    9 +++++-
 fs/xfs/xfs_dfrag.c    |   13 ++++++++++
 fs/xfs/xfs_iget.c     |    1 +
 fs/xfs/xfs_inode.h    |    1 +
 fs/xfs/xfs_vnodeops.c |   61 ++++++++++++++++++++++++++++++++-----------------
 5 files changed, 62 insertions(+), 23 deletions(-)
fs/xfs/xfs_bmap.c +7 -2
index 8abd12e..7764a4f 100644
@@ -5471,8 +5471,13 @@ xfs_getbmap(
 			if (error)
 				goto out_unlock_iolock;
 		}
-
-		ASSERT(ip->i_delayed_blks == 0);
+		/*
+		 * even after flushing the inode, there can still be delalloc
+		 * blocks on the inode beyond EOF due to speculative
+		 * preallocation. These are not removed until the release
+		 * function is called or the inode is inactivated. Hence we
+		 * cannot assert here that ip->i_delayed_blks == 0.
+		 */
 	}
 
 	lock = xfs_ilock_map_shared(ip);
fs/xfs/xfs_dfrag.c +13 -0
index 3b9582c..e60490b 100644
@@ -377,6 +377,19 @@ xfs_swap_extents(
 	ip->i_d.di_format = tip->i_d.di_format;
 	tip->i_d.di_format = tmp;
 
+	/*
+	 * The extents in the source inode could still contain speculative
+	 * preallocation beyond EOF (e.g. the file is open but not modified
+	 * while defrag is in progress). In that case, we need to copy over the
+	 * number of delalloc blocks the data fork in the source inode is
+	 * tracking beyond EOF so that when the fork is truncated away when the
+	 * temporary inode is unlinked we don't underrun the i_delayed_blks
+	 * counter on that inode.
+	 */
+	ASSERT(tip->i_delayed_blks == 0);
+	tip->i_delayed_blks = ip->i_delayed_blks;
+	ip->i_delayed_blks = 0;
+
 	ilf_fields = XFS_ILOG_CORE;
 
 	switch(ip->i_d.di_format) {
fs/xfs/xfs_iget.c +1 -0
index 0cdd269..18991a9 100644
@@ -84,6 +84,7 @@ xfs_inode_alloc(
 	memset(&ip->i_d, 0, sizeof(xfs_icdinode_t));
 	ip->i_size = 0;
 	ip->i_new_size = 0;
+	ip->i_dirty_releases = 0;
 
 	/* prevent anyone from using this yet */
 	VFS_I(ip)->i_state = I_NEW;
fs/xfs/xfs_inode.h +1 -0
index fb2ca2e..ea2f34e 100644
@@ -260,6 +260,7 @@ typedef struct xfs_inode {
 	xfs_fsize_t		i_size;		/* in-memory size */
 	xfs_fsize_t		i_new_size;	/* size when write completes */
 	atomic_t		i_iocount;	/* outstanding I/O count */
+	int			i_dirty_releases; /* dirty ->release calls */
 
 	/* VFS inode */
 	struct inode		i_vnode;	/* embedded VFS inode */
fs/xfs/xfs_vnodeops.c +40 -21

69 lines not on this page · raw message

Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#12 lore raw

Re: [PATCH 05/16] xfs: don't truncate prealloc from frequently accessed inodes

I'd be much more happy about fixing this properly in nfsd. But I guess the fix is simple enough that we can put it into XFS for now. Any reason you use up a whole int in the inode instead of using a flag in i_flags?

fs/xfs/xfs_bmap.c xfs_getbmap( · line 5480

Quoted 9 lines
fs/xfs/xfs_bmap.c +7 -2
-
-		ASSERT(ip->i_delayed_blks == 0);
+		/*
+		 * even after flushing the inode, there can still be delalloc
+		 * blocks on the inode beyond EOF due to speculative
+		 * preallocation. These are not removed until the release
+		 * function is called or the inode is inactivated. Hence we
+		 * cannot assert here that ip->i_delayed_blks == 0.
+		 */

Shouldn't this be in a separate patch given that we can fail the flush due to iolock contention? I think this and the swapext fix are .37 material in fact.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw

Re: [PATCH 05/16] xfs: don't truncate prealloc from frequently accessed inodes

Quoted 1 line · Christoph Hellwig

I'd be much more happy about fixing this properly in nfsd.

So would I, but we've been saying that for years and it still ain't done....

Quoted 4 lines

But I guess
the fix is simple enough that we can put it into XFS for now. Any reason you use up a whole int in the inode instead of using a flag in i_flags?

I wasn't sure how many dirty releases we wanted before triggering the change of behaviour. A single dirty release seems to be fine in my testing so far, and if that continues then I think that , like you suggest, changing it to a flag in i_flags is the right thing to do.

Quoted 5 lines
Quoted 5 lines
fs/xfs/xfs_bmap.c +3 -2
-
-		ASSERT(ip->i_delayed_blks == 0);
+		/*
+		 * even after flushing the inode, there can still be delalloc
+		 * blocks on the inode beyond EOF due to speculative
Quoted 8 lines

fs/xfs/xfs_bmap.c xfs_getbmap( · line 5480

Quoted 4 lines
fs/xfs/xfs_bmap.c +4 -0
+		 * preallocation. These are not removed until the release
+		 * function is called or the inode is inactivated. Hence we
+		 * cannot assert here that ip->i_delayed_blks == 0.
+		 */

Shouldn't this be in a separate patch given that we can fail the flush due to iolock contention? I think this and the swapext fix are .37 material in fact.

Agreed. I should have noted in the series preamble that I thought these probably need splitting out into separate bug fixing patches rather than being lumped in here.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw

PATCH 6/16 patch xfs-inode-hash-fake

In reply to Dave Chinner
---
 fs/xfs/linux-2.6/xfs_iops.c |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)
fs/xfs/linux-2.6/xfs_iops.c +5 -1
index 496455a..8b46867 100644
@@ -753,6 +753,10 @@ xfs_diflags_to_iflags(
  * We are always called with an uninitialised linux inode here.
  * We need to initialise the necessary fields and take a reference
  * on it.
+ *
+ * We don't use the VFS inode hash for lookups anymore, so make the inode look
+ * hashed to the VFS by faking it. This avoids needing to touch inode hash
+ * locks in this path, but makes the VFS believe the inode is validly hashed.
  */
 void
 xfs_setup_inode(
@@ -764,7 +768,7 @@ xfs_setup_inode(
 	inode->i_state = I_NEW;
 
 	inode_sb_list_add(inode);
-	insert_inode_hash(inode);
+	hlist_nulls_add_fake(&inode->i_hash);
 
 	inode->i_mode	= ip->i_d.di_mode;
 	inode->i_nlink	= ip->i_d.di_nlink;
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#16 lore raw

PATCH 7/16 xfs: convert inode cache lookups to use RCU locking

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

With delayed logging greatly increasing the sustained parallelism of inode operations, the inode cache locking is showing significant read vs write contention when inode reclaim runs at the same time as lookups. There is also a lot more write lock acquistions than there are read locks (4:1 ratio) so the read locking is not really buying us much in the way of parallelism.

To avoid the read vs write contention, change the cache to use RCU locking on the read side. To avoid needing to RCU free every single inode, use the built in slab RCU freeing mechanism. This requires us to be able to detect lookups of freed inodes, so enѕure that ever freed inode has an inode number of zero and the XFS_IRECLAIM flag set. We already check the XFS_IRECLAIM flag in cache hit lookup path, but also add a check for a zero inode number as well.

We canthen convert all the read locking lockups to use RCU read side locking and hence remove all read side locking.

---
 fs/xfs/linux-2.6/xfs_iops.c    |    7 +++++-
 fs/xfs/linux-2.6/xfs_sync.c    |   13 +++++++++--
 fs/xfs/quota/xfs_qm_syscalls.c |    3 ++
 fs/xfs/xfs_iget.c              |   44 ++++++++++++++++++++++++++++++---------
 fs/xfs/xfs_inode.c             |   22 ++++++++++++-------
 5 files changed, 67 insertions(+), 22 deletions(-)
fs/xfs/linux-2.6/xfs_iops.c +6 -1
index 8b46867..909bd9c 100644
@@ -757,6 +757,8 @@ xfs_diflags_to_iflags(
  * We don't use the VFS inode hash for lookups anymore, so make the inode look
  * hashed to the VFS by faking it. This avoids needing to touch inode hash
  * locks in this path, but makes the VFS believe the inode is validly hashed.
+ * We initialise i_state and i_hash under the i_lock so that we follow the same
+ * setup rules that the rest of the VFS follows.
  */
 void
 xfs_setup_inode(
@@ -765,10 +767,13 @@ xfs_setup_inode(
 	struct inode		*inode = &ip->i_vnode;
 
 	inode->i_ino = ip->i_ino;
+
+	spin_lock(&inode->i_lock);
 	inode->i_state = I_NEW;
+	hlist_nulls_add_fake(&inode->i_hash);
+	spin_unlock(&inode->i_lock);
 
 	inode_sb_list_add(inode);
-	hlist_nulls_add_fake(&inode->i_hash);
 
 	inode->i_mode	= ip->i_d.di_mode;
 	inode->i_nlink	= ip->i_d.di_nlink;
fs/xfs/linux-2.6/xfs_sync.c +10 -3
index afb0d7c..9a53cc9 100644
@@ -53,6 +53,10 @@ xfs_inode_ag_walk_grab(
 {
 	struct inode		*inode = VFS_I(ip);
 
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return ENOENT;
+
 	/* nothing to sync during shutdown */
 	if (XFS_FORCED_SHUTDOWN(ip->i_mount))
 		return EFSCORRUPTED;
@@ -98,12 +102,12 @@ restart:
 		int		error = 0;
 		int		i;
 
-		read_lock(&pag->pag_ici_lock);
+		rcu_read_lock();
 		nr_found = radix_tree_gang_lookup(&pag->pag_ici_root,
 					(void **)batch, first_index,
 					XFS_LOOKUP_BATCH);
 		if (!nr_found) {
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 			break;
 		}
 
@@ -129,7 +133,7 @@ restart:
 		}
 
 		/* unlock now we've grabbed the inodes. */
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 
 		for (i = 0; i < nr_found; i++) {
 			if (!batch[i])
@@ -639,6 +643,9 @@ xfs_reclaim_inode_grab(
 	struct xfs_inode	*ip,
 	int			flags)
 {
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return 1;
 
 	/*
 	 * do some unlocked checks first to avoid unnecceary lock traffic.
fs/xfs/quota/xfs_qm_syscalls.c +3 -0
index bdebc18..8b207fc 100644
@@ -875,6 +875,9 @@ xfs_dqrele_inode(
 	struct xfs_perag	*pag,
 	int			flags)
 {
+	if (!ip->i_ino)
+		return ENOENT;
+
 	/* skip quota inodes */
 	if (ip == ip->i_mount->m_quotainfo->qi_uquotaip ||
 	    ip == ip->i_mount->m_quotainfo->qi_gquotaip) {
fs/xfs/xfs_iget.c +34 -10

109 lines not on this page · raw message

fs/xfs/xfs_inode.c +14 -8
index 108c7a0..25becb1 100644
@@ -2000,13 +2000,14 @@ xfs_ifree_cluster(
 		 */
 		for (i = 0; i < ninodes; i++) {
 retry:
-			read_lock(&pag->pag_ici_lock);
+			rcu_read_lock();
 			ip = radix_tree_lookup(&pag->pag_ici_root,
 					XFS_INO_TO_AGINO(mp, (inum + i)));
 
 			/* Inode not in memory or stale, nothing to do */
-			if (!ip || xfs_iflags_test(ip, XFS_ISTALE)) {
-				read_unlock(&pag->pag_ici_lock);
+			if (!ip || !ip->i_ino ||
+			    xfs_iflags_test(ip, XFS_ISTALE)) {
+				rcu_read_unlock();
 				continue;
 			}
 
@@ -2019,11 +2020,11 @@ retry:
 			 */
 			if (ip != free_ip &&
 			    !xfs_ilock_nowait(ip, XFS_ILOCK_EXCL)) {
-				read_unlock(&pag->pag_ici_lock);
+				rcu_read_unlock();
 				delay(1);
 				goto retry;
 			}
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 
 			xfs_iflock(ip);
 			xfs_iflags_set(ip, XFS_ISTALE);
@@ -2629,7 +2630,7 @@ xfs_iflush_cluster(
 
 	mask = ~(((XFS_INODE_CLUSTER_SIZE(mp) >> mp->m_sb.sb_inodelog)) - 1);
 	first_index = XFS_INO_TO_AGINO(mp, ip->i_ino) & mask;
-	read_lock(&pag->pag_ici_lock);
+	rcu_read_lock();
 	/* really need a gang lookup range call here */
 	nr_found = radix_tree_gang_lookup(&pag->pag_ici_root, (void**)ilist,
 					first_index, inodes_per_cluster);
@@ -2640,6 +2641,11 @@ xfs_iflush_cluster(
 		iq = ilist[i];
 		if (iq == ip)
 			continue;
+
+		/* check we've got a valid inode */
+		if (!iq->i_ino)
+			continue;
+
 		/* if the inode lies outside this cluster, we're done. */
 		if ((XFS_INO_TO_AGINO(mp, iq->i_ino) & mask) != first_index)
 			break;
@@ -2692,7 +2698,7 @@ xfs_iflush_cluster(
 	}
 
 out_free:
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	kmem_free(ilist);
 out_put:
 	xfs_perag_put(pag);
@@ -2704,7 +2710,7 @@ cluster_corrupt_out:
 	 * Corruption detected in the clustering loop.  Invalidate the
 	 * inode buffer and shut down the filesystem.
 	 */
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	/*
 	 * Clean up the buffer.  If it was B_DELWRI, just release it --
 	 * brelse can handle it with no problems.  If not, shut down the
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#17 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Quoted 42 lines · Dave Chinner

From: Dave Chinner <[email protected]>

With delayed logging greatly increasing the sustained parallelism of inode operations, the inode cache locking is showing significant read vs write contention when inode reclaim runs at the same time as lookups. There is also a lot more write lock acquistions than there are read locks (4:1 ratio) so the read locking is not really buying us much in the way of parallelism.

To avoid the read vs write contention, change the cache to use RCU locking on the read side. To avoid needing to RCU free every single inode, use the built in slab RCU freeing mechanism. This requires us to be able to detect lookups of freed inodes, so en??ure that ever freed inode has an inode number of zero and the XFS_IRECLAIM flag set. We already check the XFS_IRECLAIM flag in cache hit lookup path, but also add a check for a zero inode number as well.

We canthen convert all the read locking lockups to use RCU read side locking and hence remove all read side locking.

---
 fs/xfs/linux-2.6/xfs_iops.c    |    7 +++++-
 fs/xfs/linux-2.6/xfs_sync.c    |   13 +++++++++--
 fs/xfs/quota/xfs_qm_syscalls.c |    3 ++
 fs/xfs/xfs_iget.c              |   44 ++++++++++++++++++++++++++++++---------
 fs/xfs/xfs_inode.c             |   22 ++++++++++++-------
 5 files changed, 67 insertions(+), 22 deletions(-)
fs/xfs/linux-2.6/xfs_iops.c +2 -0
index 8b46867..909bd9c 100644
@@ -757,6 +757,8 @@ xfs_diflags_to_iflags(
  * We don't use the VFS inode hash for lookups anymore, so make the inode look
  * hashed to the VFS by faking it. This avoids needing to touch inode hash
  * locks in this path, but makes the VFS believe the inode is validly hashed.
+ * We initialise i_state and i_hash under the i_lock so that we follow the same
+ * setup rules that the rest of the VFS follows.
  */
 void
 xfs_setup_inode(

fs/xfs/linux-2.6/xfs_iops.c xfs_setup_inode( · line 774

Quoted 8 lines
fs/xfs/linux-2.6/xfs_iops.c +4 -0
@@ -765,10 +767,13 @@ xfs_setup_inode(
 	struct inode		*inode = &ip->i_vnode;
 
 	inode->i_ino = ip->i_ino;
+
+	spin_lock(&inode->i_lock);
 	inode->i_state = I_NEW;
+	hlist_nulls_add_fake(&inode->i_hash);
+	spin_unlock(&inode->i_lock);

This screams for another VFS helper, even if it's XFS-specific for now. Having to duplicate inode.c-private locking rules in XFS seems a bit nasty to me.

Quoted 11 lines
fs/xfs/linux-2.6/xfs_iops.c +0 -1
 	inode_sb_list_add(inode);
-	hlist_nulls_add_fake(&inode->i_hash);
 
 	inode->i_mode	= ip->i_d.di_mode;
 	inode->i_nlink	= ip->i_d.di_nlink;
fs/xfs/linux-2.6/xfs_sync.c +0 -0
index afb0d7c..9a53cc9 100644

fs/xfs/linux-2.6/xfs_sync.c xfs_inode_ag_walk_grab( · line 58

Quoted 6 lines
fs/xfs/linux-2.6/xfs_sync.c +3 -0
@@ -53,6 +53,10 @@ xfs_inode_ag_walk_grab(
 {
 	struct inode		*inode = VFS_I(ip);
 
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return ENOENT;

Assuming i_ino is never 0 is fine for XFS, unlike for the generic VFS code, so ACK.

Quoted 42 lines
fs/xfs/linux-2.6/xfs_sync.c +6 -3
 	/* nothing to sync during shutdown */
 	if (XFS_FORCED_SHUTDOWN(ip->i_mount))
 		return EFSCORRUPTED;
@@ -98,12 +102,12 @@ restart:
 		int		error = 0;
 		int		i;
 
-		read_lock(&pag->pag_ici_lock);
+		rcu_read_lock();
 		nr_found = radix_tree_gang_lookup(&pag->pag_ici_root,
 					(void **)batch, first_index,
 					XFS_LOOKUP_BATCH);
 		if (!nr_found) {
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 			break;
 		}
 
@@ -129,7 +133,7 @@ restart:
 		}
 
 		/* unlock now we've grabbed the inodes. */
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 
 		for (i = 0; i < nr_found; i++) {
 			if (!batch[i])
@@ -639,6 +643,9 @@ xfs_reclaim_inode_grab(
 	struct xfs_inode	*ip,
 	int			flags)
 {
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return 1;
 
 	/*
 	 * do some unlocked checks first to avoid unnecceary lock traffic.
fs/xfs/quota/xfs_qm_syscalls.c +0 -0
index bdebc18..8b207fc 100644

fs/xfs/quota/xfs_qm_syscalls.c xfs_dqrele_inode( · line 880

Quoted 6 lines
fs/xfs/quota/xfs_qm_syscalls.c +3 -0
@@ -875,6 +875,9 @@ xfs_dqrele_inode(
 	struct xfs_perag	*pag,
 	int			flags)
 {
+	if (!ip->i_ino)
+		return ENOENT;
+

Why do we need the check here again? Having it in xfs_inode_ag_walk_grab should be enough.

Quoted 208 lines
fs/xfs/quota/xfs_qm_syscalls.c +0 -0
 	/* skip quota inodes */
 	if (ip == ip->i_mount->m_quotainfo->qi_uquotaip ||
 	    ip == ip->i_mount->m_quotainfo->qi_gquotaip) {
fs/xfs/xfs_iget.c +34 -10
index 18991a9..edeb918 100644
@@ -69,6 +69,7 @@ xfs_inode_alloc(
 	ASSERT(atomic_read(&ip->i_pincount) == 0);
 	ASSERT(!spin_is_locked(&ip->i_flags_lock));
 	ASSERT(completion_done(&ip->i_flush));
+	ASSERT(ip->i_ino == 0);
 
 	mrlock_init(&ip->i_iolock, MRLOCK_BARRIER, "xfsio", ip->i_ino);
 
@@ -86,9 +87,6 @@ xfs_inode_alloc(
 	ip->i_new_size = 0;
 	ip->i_dirty_releases = 0;
 
-	/* prevent anyone from using this yet */
-	VFS_I(ip)->i_state = I_NEW;
-
 	return ip;
 }
 
@@ -135,6 +133,16 @@ xfs_inode_free(
 	ASSERT(!spin_is_locked(&ip->i_flags_lock));
 	ASSERT(completion_done(&ip->i_flush));
 
+	/*
+	 * because we use SLAB_DESTROY_BY_RCU freeing, ensure the inode
+	 * always appears to be reclaimed with an invalid inode number
+	 * when in the free state. The ip->i_flags_lock provides the barrier
+	 * against lookup races.
+	 */
+	spin_lock(&ip->i_flags_lock);
+	ip->i_flags = XFS_IRECLAIM;
+	ip->i_ino = 0;
+	spin_unlock(&ip->i_flags_lock);
 	kmem_zone_free(xfs_inode_zone, ip);
 }
 
@@ -146,12 +154,28 @@ xfs_iget_cache_hit(
 	struct xfs_perag	*pag,
 	struct xfs_inode	*ip,
 	int			flags,
-	int			lock_flags) __releases(pag->pag_ici_lock)
+	int			lock_flags) __releases(RCU)
 {
 	struct inode		*inode = VFS_I(ip);
 	struct xfs_mount	*mp = ip->i_mount;
 	int			error;
 
+	/*
+	 * check for re-use of an inode within an RCU grace period due to the
+	 * radix tree nodes not being updated yet. We monitor for this by
+	 * setting the inode number to zero before freeing the inode structure.
+	 * We don't need to recheck this after taking the i_flags_lock because
+	 * the check against XFS_IRECLAIM will catch a freed inode.
+	 */
+	if (ip->i_ino == 0) {
+		trace_xfs_iget_skip(ip);
+		XFS_STATS_INC(xs_ig_frecycle);
+		rcu_read_unlock();
+		/* Expire the grace period so we don't trip over it again. */
+		synchronize_rcu();
+		return EAGAIN;
+	}
+
 	spin_lock(&ip->i_flags_lock);
 
 	/*
@@ -195,7 +219,7 @@ xfs_iget_cache_hit(
 		ip->i_flags |= XFS_IRECLAIM;
 
 		spin_unlock(&ip->i_flags_lock);
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 
 		error = -inode_init_always(mp->m_super, inode);
 		if (error) {
@@ -203,7 +227,7 @@ xfs_iget_cache_hit(
 			 * Re-initializing the inode failed, and we are in deep
 			 * trouble.  Try to re-add it to the reclaim list.
 			 */
-			read_lock(&pag->pag_ici_lock);
+			rcu_read_lock();
 			spin_lock(&ip->i_flags_lock);
 
 			ip->i_flags &= ~XFS_INEW;
@@ -231,7 +255,7 @@ xfs_iget_cache_hit(
 
 		/* We've got a live one. */
 		spin_unlock(&ip->i_flags_lock);
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 		trace_xfs_iget_hit(ip);
 	}
 
@@ -245,7 +269,7 @@ xfs_iget_cache_hit(
 
 out_error:
 	spin_unlock(&ip->i_flags_lock);
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	return error;
 }
 
@@ -376,7 +400,7 @@ xfs_iget(
 
 again:
 	error = 0;
-	read_lock(&pag->pag_ici_lock);
+	rcu_read_lock();
 	ip = radix_tree_lookup(&pag->pag_ici_root, agino);
 
 	if (ip) {
@@ -384,7 +408,7 @@ again:
 		if (error)
 			goto out_error_or_again;
 	} else {
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 		XFS_STATS_INC(xs_ig_missed);
 
 		error = xfs_iget_cache_miss(mp, pag, tp, ino, &ip,
fs/xfs/xfs_inode.c +14 -8
index 108c7a0..25becb1 100644
@@ -2000,13 +2000,14 @@ xfs_ifree_cluster(
 		 */
 		for (i = 0; i < ninodes; i++) {
 retry:
-			read_lock(&pag->pag_ici_lock);
+			rcu_read_lock();
 			ip = radix_tree_lookup(&pag->pag_ici_root,
 					XFS_INO_TO_AGINO(mp, (inum + i)));
 
 			/* Inode not in memory or stale, nothing to do */
-			if (!ip || xfs_iflags_test(ip, XFS_ISTALE)) {
-				read_unlock(&pag->pag_ici_lock);
+			if (!ip || !ip->i_ino ||
+			    xfs_iflags_test(ip, XFS_ISTALE)) {
+				rcu_read_unlock();
 				continue;
 			}
 
@@ -2019,11 +2020,11 @@ retry:
 			 */
 			if (ip != free_ip &&
 			    !xfs_ilock_nowait(ip, XFS_ILOCK_EXCL)) {
-				read_unlock(&pag->pag_ici_lock);
+				rcu_read_unlock();
 				delay(1);
 				goto retry;
 			}
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 
 			xfs_iflock(ip);
 			xfs_iflags_set(ip, XFS_ISTALE);
@@ -2629,7 +2630,7 @@ xfs_iflush_cluster(
 
 	mask = ~(((XFS_INODE_CLUSTER_SIZE(mp) >> mp->m_sb.sb_inodelog)) - 1);
 	first_index = XFS_INO_TO_AGINO(mp, ip->i_ino) & mask;
-	read_lock(&pag->pag_ici_lock);
+	rcu_read_lock();
 	/* really need a gang lookup range call here */
 	nr_found = radix_tree_gang_lookup(&pag->pag_ici_root, (void**)ilist,
 					first_index, inodes_per_cluster);
@@ -2640,6 +2641,11 @@ xfs_iflush_cluster(
 		iq = ilist[i];
 		if (iq == ip)
 			continue;
+
+		/* check we've got a valid inode */
+		if (!iq->i_ino)
+			continue;
+
 		/* if the inode lies outside this cluster, we're done. */
 		if ((XFS_INO_TO_AGINO(mp, iq->i_ino) & mask) != first_index)
 			break;
@@ -2692,7 +2698,7 @@ xfs_iflush_cluster(
 	}
 
 out_free:
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	kmem_free(ilist);
 out_put:
 	xfs_perag_put(pag);
@@ -2704,7 +2710,7 @@ cluster_corrupt_out:
 	 * Corruption detected in the clustering loop.  Invalidate the
 	 * inode buffer and shut down the filesystem.
 	 */
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	/*
 	 * Clean up the buffer.  If it was B_DELWRI, just release it --
 	 * brelse can handle it with no problems.  If not, shut down the
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

---end quoted text---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#18 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

Quoted 48 lines · Christoph Hellwig

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Quoted 44 lines · Dave Chinner

From: Dave Chinner <[email protected]>

With delayed logging greatly increasing the sustained parallelism of inode operations, the inode cache locking is showing significant read vs write contention when inode reclaim runs at the same time as lookups. There is also a lot more write lock acquistions than there are read locks (4:1 ratio) so the read locking is not really buying us much in the way of parallelism.

To avoid the read vs write contention, change the cache to use RCU locking on the read side. To avoid needing to RCU free every single inode, use the built in slab RCU freeing mechanism. This requires us to be able to detect lookups of freed inodes, so en??ure that ever freed inode has an inode number of zero and the XFS_IRECLAIM flag set. We already check the XFS_IRECLAIM flag in cache hit lookup path, but also add a check for a zero inode number as well.

We canthen convert all the read locking lockups to use RCU read side locking and hence remove all read side locking.

---
 fs/xfs/linux-2.6/xfs_iops.c    |    7 +++++-
 fs/xfs/linux-2.6/xfs_sync.c    |   13 +++++++++--
 fs/xfs/quota/xfs_qm_syscalls.c |    3 ++
 fs/xfs/xfs_iget.c              |   44 ++++++++++++++++++++++++++++++---------
 fs/xfs/xfs_inode.c             |   22 ++++++++++++-------
 5 files changed, 67 insertions(+), 22 deletions(-)
fs/xfs/linux-2.6/xfs_iops.c +2 -0
index 8b46867..909bd9c 100644
@@ -757,6 +757,8 @@ xfs_diflags_to_iflags(
  * We don't use the VFS inode hash for lookups anymore, so make the inode look
  * hashed to the VFS by faking it. This avoids needing to touch inode hash
  * locks in this path, but makes the VFS believe the inode is validly hashed.
+ * We initialise i_state and i_hash under the i_lock so that we follow the same
+ * setup rules that the rest of the VFS follows.
  */
 void
 xfs_setup_inode(
@@ -765,10 +767,13 @@ xfs_setup_inode(
 	struct inode		*inode = &ip->i_vnode;
 
Quoted 10 lines

fs/xfs/linux-2.6/xfs_iops.c xfs_setup_inode( · line 774

Quoted 6 lines
fs/xfs/linux-2.6/xfs_iops.c +4 -0
 	inode->i_ino = ip->i_ino;
+
+	spin_lock(&inode->i_lock);
 	inode->i_state = I_NEW;
+	hlist_nulls_add_fake(&inode->i_hash);
+	spin_unlock(&inode->i_lock);

This screams for another VFS helper, even if it's XFS-specific for now. Having to duplicate inode.c-private locking rules in XFS seems a bit nasty to me.

Agreed. I was thinking that it would be a good idea to do this, but I hadn't decided on how to do it yet....

Quoted 6 lines
Quoted 6 lines
fs/xfs/quota/xfs_qm_syscalls.c +0 -0
index bdebc18..8b207fc 100644
@@ -875,6 +875,9 @@ xfs_dqrele_inode(
 	struct xfs_perag	*pag,
Quoted 8 lines

fs/xfs/quota/xfs_qm_syscalls.c xfs_dqrele_inode( · line 880

Quoted 5 lines
fs/xfs/quota/xfs_qm_syscalls.c +3 -0
 	int			flags)
 {
+	if (!ip->i_ino)
+		return ENOENT;
+

Why do we need the check here again? Having it in xfs_inode_ag_walk_grab should be enough.

Yes, you are right. I'll fix that.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#19 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

In reply to Christoph Hellwig
Quoted 2 lines · Christoph Hellwig

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Is there a git tree, tarball, or whatever? For example, I don't see how this patch handles the case of an inode being freed just as an RCU reader gains a reference to it, but then reallocated as some other inode (so that ->ino is nonzero) before the RCU reader gets a chance to actually look at the inode. But such a check might well be in the code that this patch didn't change...

Thanx, Paul

Quoted 333 lines · Dave Chinner
Quoted 42 lines

From: Dave Chinner <[email protected]>

With delayed logging greatly increasing the sustained parallelism of inode operations, the inode cache locking is showing significant read vs write contention when inode reclaim runs at the same time as lookups. There is also a lot more write lock acquistions than there are read locks (4:1 ratio) so the read locking is not really buying us much in the way of parallelism.

To avoid the read vs write contention, change the cache to use RCU locking on the read side. To avoid needing to RCU free every single inode, use the built in slab RCU freeing mechanism. This requires us to be able to detect lookups of freed inodes, so en??ure that ever freed inode has an inode number of zero and the XFS_IRECLAIM flag set. We already check the XFS_IRECLAIM flag in cache hit lookup path, but also add a check for a zero inode number as well.

We canthen convert all the read locking lockups to use RCU read side locking and hence remove all read side locking.

---
 fs/xfs/linux-2.6/xfs_iops.c    |    7 +++++-
 fs/xfs/linux-2.6/xfs_sync.c    |   13 +++++++++--
 fs/xfs/quota/xfs_qm_syscalls.c |    3 ++
 fs/xfs/xfs_iget.c              |   44 ++++++++++++++++++++++++++++++---------
 fs/xfs/xfs_inode.c             |   22 ++++++++++++-------
 5 files changed, 67 insertions(+), 22 deletions(-)
fs/xfs/linux-2.6/xfs_iops.c +2 -0
index 8b46867..909bd9c 100644
@@ -757,6 +757,8 @@ xfs_diflags_to_iflags(
  * We don't use the VFS inode hash for lookups anymore, so make the inode look
  * hashed to the VFS by faking it. This avoids needing to touch inode hash
  * locks in this path, but makes the VFS believe the inode is validly hashed.
+ * We initialise i_state and i_hash under the i_lock so that we follow the same
+ * setup rules that the rest of the VFS follows.
  */
 void
 xfs_setup_inode(

fs/xfs/linux-2.6/xfs_iops.c xfs_setup_inode( · line 774

Quoted 8 lines
fs/xfs/linux-2.6/xfs_iops.c +4 -0
@@ -765,10 +767,13 @@ xfs_setup_inode(
 	struct inode		*inode = &ip->i_vnode;
 
 	inode->i_ino = ip->i_ino;
+
+	spin_lock(&inode->i_lock);
 	inode->i_state = I_NEW;
+	hlist_nulls_add_fake(&inode->i_hash);
+	spin_unlock(&inode->i_lock);

This screams for another VFS helper, even if it's XFS-specific for now. Having to duplicate inode.c-private locking rules in XFS seems a bit nasty to me.

Quoted 11 lines
fs/xfs/linux-2.6/xfs_iops.c +0 -1
 	inode_sb_list_add(inode);
-	hlist_nulls_add_fake(&inode->i_hash);
 
 	inode->i_mode	= ip->i_d.di_mode;
 	inode->i_nlink	= ip->i_d.di_nlink;
fs/xfs/linux-2.6/xfs_sync.c +0 -0
index afb0d7c..9a53cc9 100644

fs/xfs/linux-2.6/xfs_sync.c xfs_inode_ag_walk_grab( · line 58

Quoted 6 lines
fs/xfs/linux-2.6/xfs_sync.c +3 -0
@@ -53,6 +53,10 @@ xfs_inode_ag_walk_grab(
 {
 	struct inode		*inode = VFS_I(ip);
 
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return ENOENT;

Assuming i_ino is never 0 is fine for XFS, unlike for the generic VFS code, so ACK.

Quoted 42 lines
fs/xfs/linux-2.6/xfs_sync.c +6 -3
 	/* nothing to sync during shutdown */
 	if (XFS_FORCED_SHUTDOWN(ip->i_mount))
 		return EFSCORRUPTED;
@@ -98,12 +102,12 @@ restart:
 		int		error = 0;
 		int		i;
 
-		read_lock(&pag->pag_ici_lock);
+		rcu_read_lock();
 		nr_found = radix_tree_gang_lookup(&pag->pag_ici_root,
 					(void **)batch, first_index,
 					XFS_LOOKUP_BATCH);
 		if (!nr_found) {
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 			break;
 		}
 
@@ -129,7 +133,7 @@ restart:
 		}
 
 		/* unlock now we've grabbed the inodes. */
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 
 		for (i = 0; i < nr_found; i++) {
 			if (!batch[i])
@@ -639,6 +643,9 @@ xfs_reclaim_inode_grab(
 	struct xfs_inode	*ip,
 	int			flags)
 {
+	/* check for stale RCU freed inode */
+	if (!ip->i_ino)
+		return 1;
 
 	/*
 	 * do some unlocked checks first to avoid unnecceary lock traffic.
fs/xfs/quota/xfs_qm_syscalls.c +0 -0
index bdebc18..8b207fc 100644

fs/xfs/quota/xfs_qm_syscalls.c xfs_dqrele_inode( · line 880

Quoted 6 lines
fs/xfs/quota/xfs_qm_syscalls.c +3 -0
@@ -875,6 +875,9 @@ xfs_dqrele_inode(
 	struct xfs_perag	*pag,
 	int			flags)
 {
+	if (!ip->i_ino)
+		return ENOENT;
+

Why do we need the check here again? Having it in xfs_inode_ag_walk_grab should be enough.

Quoted 204 lines
fs/xfs/quota/xfs_qm_syscalls.c +0 -0
 	/* skip quota inodes */
 	if (ip == ip->i_mount->m_quotainfo->qi_uquotaip ||
 	    ip == ip->i_mount->m_quotainfo->qi_gquotaip) {
fs/xfs/xfs_iget.c +34 -10
index 18991a9..edeb918 100644
@@ -69,6 +69,7 @@ xfs_inode_alloc(
 	ASSERT(atomic_read(&ip->i_pincount) == 0);
 	ASSERT(!spin_is_locked(&ip->i_flags_lock));
 	ASSERT(completion_done(&ip->i_flush));
+	ASSERT(ip->i_ino == 0);
 
 	mrlock_init(&ip->i_iolock, MRLOCK_BARRIER, "xfsio", ip->i_ino);
 
@@ -86,9 +87,6 @@ xfs_inode_alloc(
 	ip->i_new_size = 0;
 	ip->i_dirty_releases = 0;
 
-	/* prevent anyone from using this yet */
-	VFS_I(ip)->i_state = I_NEW;
-
 	return ip;
 }
 
@@ -135,6 +133,16 @@ xfs_inode_free(
 	ASSERT(!spin_is_locked(&ip->i_flags_lock));
 	ASSERT(completion_done(&ip->i_flush));
 
+	/*
+	 * because we use SLAB_DESTROY_BY_RCU freeing, ensure the inode
+	 * always appears to be reclaimed with an invalid inode number
+	 * when in the free state. The ip->i_flags_lock provides the barrier
+	 * against lookup races.
+	 */
+	spin_lock(&ip->i_flags_lock);
+	ip->i_flags = XFS_IRECLAIM;
+	ip->i_ino = 0;
+	spin_unlock(&ip->i_flags_lock);
 	kmem_zone_free(xfs_inode_zone, ip);
 }
 
@@ -146,12 +154,28 @@ xfs_iget_cache_hit(
 	struct xfs_perag	*pag,
 	struct xfs_inode	*ip,
 	int			flags,
-	int			lock_flags) __releases(pag->pag_ici_lock)
+	int			lock_flags) __releases(RCU)
 {
 	struct inode		*inode = VFS_I(ip);
 	struct xfs_mount	*mp = ip->i_mount;
 	int			error;
 
+	/*
+	 * check for re-use of an inode within an RCU grace period due to the
+	 * radix tree nodes not being updated yet. We monitor for this by
+	 * setting the inode number to zero before freeing the inode structure.
+	 * We don't need to recheck this after taking the i_flags_lock because
+	 * the check against XFS_IRECLAIM will catch a freed inode.
+	 */
+	if (ip->i_ino == 0) {
+		trace_xfs_iget_skip(ip);
+		XFS_STATS_INC(xs_ig_frecycle);
+		rcu_read_unlock();
+		/* Expire the grace period so we don't trip over it again. */
+		synchronize_rcu();
+		return EAGAIN;
+	}
+
 	spin_lock(&ip->i_flags_lock);
 
 	/*
@@ -195,7 +219,7 @@ xfs_iget_cache_hit(
 		ip->i_flags |= XFS_IRECLAIM;
 
 		spin_unlock(&ip->i_flags_lock);
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 
 		error = -inode_init_always(mp->m_super, inode);
 		if (error) {
@@ -203,7 +227,7 @@ xfs_iget_cache_hit(
 			 * Re-initializing the inode failed, and we are in deep
 			 * trouble.  Try to re-add it to the reclaim list.
 			 */
-			read_lock(&pag->pag_ici_lock);
+			rcu_read_lock();
 			spin_lock(&ip->i_flags_lock);
 
 			ip->i_flags &= ~XFS_INEW;
@@ -231,7 +255,7 @@ xfs_iget_cache_hit(
 
 		/* We've got a live one. */
 		spin_unlock(&ip->i_flags_lock);
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 		trace_xfs_iget_hit(ip);
 	}
 
@@ -245,7 +269,7 @@ xfs_iget_cache_hit(
 
 out_error:
 	spin_unlock(&ip->i_flags_lock);
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	return error;
 }
 
@@ -376,7 +400,7 @@ xfs_iget(
 
 again:
 	error = 0;
-	read_lock(&pag->pag_ici_lock);
+	rcu_read_lock();
 	ip = radix_tree_lookup(&pag->pag_ici_root, agino);
 
 	if (ip) {
@@ -384,7 +408,7 @@ again:
 		if (error)
 			goto out_error_or_again;
 	} else {
-		read_unlock(&pag->pag_ici_lock);
+		rcu_read_unlock();
 		XFS_STATS_INC(xs_ig_missed);
 
 		error = xfs_iget_cache_miss(mp, pag, tp, ino, &ip,
fs/xfs/xfs_inode.c +14 -8
index 108c7a0..25becb1 100644
@@ -2000,13 +2000,14 @@ xfs_ifree_cluster(
 		 */
 		for (i = 0; i < ninodes; i++) {
 retry:
-			read_lock(&pag->pag_ici_lock);
+			rcu_read_lock();
 			ip = radix_tree_lookup(&pag->pag_ici_root,
 					XFS_INO_TO_AGINO(mp, (inum + i)));
 
 			/* Inode not in memory or stale, nothing to do */
-			if (!ip || xfs_iflags_test(ip, XFS_ISTALE)) {
-				read_unlock(&pag->pag_ici_lock);
+			if (!ip || !ip->i_ino ||
+			    xfs_iflags_test(ip, XFS_ISTALE)) {
+				rcu_read_unlock();
 				continue;
 			}
 
@@ -2019,11 +2020,11 @@ retry:
 			 */
 			if (ip != free_ip &&
 			    !xfs_ilock_nowait(ip, XFS_ILOCK_EXCL)) {
-				read_unlock(&pag->pag_ici_lock);
+				rcu_read_unlock();
 				delay(1);
 				goto retry;
 			}
-			read_unlock(&pag->pag_ici_lock);
+			rcu_read_unlock();
 
 			xfs_iflock(ip);
 			xfs_iflags_set(ip, XFS_ISTALE);
@@ -2629,7 +2630,7 @@ xfs_iflush_cluster(
 
 	mask = ~(((XFS_INODE_CLUSTER_SIZE(mp) >> mp->m_sb.sb_inodelog)) - 1);
 	first_index = XFS_INO_TO_AGINO(mp, ip->i_ino) & mask;
-	read_lock(&pag->pag_ici_lock);
+	rcu_read_lock();
 	/* really need a gang lookup range call here */
 	nr_found = radix_tree_gang_lookup(&pag->pag_ici_root, (void**)ilist,
 					first_index, inodes_per_cluster);
@@ -2640,6 +2641,11 @@ xfs_iflush_cluster(
 		iq = ilist[i];
 		if (iq == ip)
 			continue;
+
+		/* check we've got a valid inode */
+		if (!iq->i_ino)
+			continue;
+
 		/* if the inode lies outside this cluster, we're done. */
 		if ((XFS_INO_TO_AGINO(mp, iq->i_ino) & mask) != first_index)
 			break;
@@ -2692,7 +2698,7 @@ xfs_iflush_cluster(
 	}
 
 out_free:
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	kmem_free(ilist);
 out_put:
 	xfs_perag_put(pag);
@@ -2704,7 +2710,7 @@ cluster_corrupt_out:
 	 * Corruption detected in the clustering loop.  Invalidate the
 	 * inode buffer and shut down the filesystem.
 	 */
-	read_unlock(&pag->pag_ici_lock);
+	rcu_read_unlock();
 	/*
 	 * Clean up the buffer.  If it was B_DELWRI, just release it --
 	 * brelse can handle it with no problems.  If not, shut down the
Signature · 1 line
1.7.2.3
Quoted 5 lines
Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

---end quoted text---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#20 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Christoph Hellwig

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Is there a git tree, tarball, or whatever?

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git working

contains the series that this patch is in.

Quoted 3 lines

For example, I don't see
how this patch handles the case of an inode being freed just as an RCU reader gains a reference to it,

XFS_IRECLAIM flag is set on inodes as they transition into the reclaim state long before they are freed. The XFS_IRECLAIM flag is left there once freed. Hence lookups in xfs_iget_cache_hit() will see this.

If the inode has been reallocated, the inode number will not yet be set, or the inode state will have changed to XFS_INEW, both of which xfs_iget_cache_hit() will also reject.

Quoted 3 lines

but then reallocated as some other inode
(so that ->ino is nonzero) before the RCU reader gets a chance to actually look at the inode.

XFS_INEW is not cleared until well after a new ->i_ino is set, so the lookup should find trip over XFS_INEW in that case. I think that I may need to move the inode number check under the i_flags_lock after validating the flags - more to check that we've got the correct inode than to validate we have a freed inode.

Quoted 2 lines

But such a check might well be in the code that this patch didn't change...

Yeah, most of the XFS code is already in a form compatible with such RCU use because inodes have always had a quiescent "reclaimable" state between active and reclaim (XFS_INEW -> active -> XFS_IRECLAIMABLE -> XFS_IRECLAIM) where the inode can be reused before being freed. The result is that lookups have always had to handle races with inodes that have just transitioned into the XFS_IRECLAIM state and hence cannot be immediately reused...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#21 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

Quoted 8 lines · Dave Chinner
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Christoph Hellwig

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Is there a git tree, tarball, or whatever?

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git working

Thank you -- I have downloaded this and will look it over.

Once the C++ guys get done grilling me on memory-model issues...

Quoted 18 lines

contains the series that this patch is in.

Quoted 3 lines

For example, I don't see
how this patch handles the case of an inode being freed just as an RCU reader gains a reference to it,

XFS_IRECLAIM flag is set on inodes as they transition into the reclaim state long before they are freed. The XFS_IRECLAIM flag is left there once freed. Hence lookups in xfs_iget_cache_hit() will see this.

If the inode has been reallocated, the inode number will not yet be set, or the inode state will have changed to XFS_INEW, both of which xfs_iget_cache_hit() will also reject.

Quoted 3 lines

but then reallocated as some other inode
(so that ->ino is nonzero) before the RCU reader gets a chance to actually look at the inode.

Quoted 5 lines

XFS_INEW is not cleared until well after a new ->i_ino is set, so the lookup should find trip over XFS_INEW in that case. I think that I may need to move the inode number check under the i_flags_lock after validating the flags - more to check that we've got the correct inode than to validate we have a freed inode.

OK, this sounds promising. Of course, the next question is "how quickly can the inode number be available for reuse?"

Quoted 10 lines
Quoted 2 lines

But such a check might well be in the code that this patch didn't change...

Yeah, most of the XFS code is already in a form compatible with such RCU use because inodes have always had a quiescent "reclaimable" state between active and reclaim (XFS_INEW -> active -> XFS_IRECLAIMABLE -> XFS_IRECLAIM) where the inode can be reused before being freed. The result is that lookups have always had to handle races with inodes that have just transitioned into the XFS_IRECLAIM state and hence cannot be immediately reused...

Cool!!!

Thanx, Paul

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#22 lore raw

Re: [PATCH 07/16] xfs: convert inode cache lookups to use RCU locking

Quoted 11 lines · Paul E. McKenney
Quoted 8 lines · Dave Chinner
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Christoph Hellwig

This patch generally looks good to me, but with so much RCU magic I'd prefer if Paul & Eric could look over it.

Is there a git tree, tarball, or whatever?

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfsdev.git working

Thank you -- I have downloaded this and will look it over.

fs/xfs/xfs_iget.c is the place to start - that's where the inode cache lookups occur...

Quoted 20 lines

Once the C++ guys get done grilling me on memory-model issues...

Quoted 18 lines

contains the series that this patch is in.

Quoted 3 lines

For example, I don't see
how this patch handles the case of an inode being freed just as an RCU reader gains a reference to it,

XFS_IRECLAIM flag is set on inodes as they transition into the reclaim state long before they are freed. The XFS_IRECLAIM flag is left there once freed. Hence lookups in xfs_iget_cache_hit() will see this.

If the inode has been reallocated, the inode number will not yet be set, or the inode state will have changed to XFS_INEW, both of which xfs_iget_cache_hit() will also reject.

Quoted 3 lines

but then reallocated as some other inode
(so that ->ino is nonzero) before the RCU reader gets a chance to actually look at the inode.

Quoted 8 lines
Quoted 5 lines

XFS_INEW is not cleared until well after a new ->i_ino is set, so the lookup should find trip over XFS_INEW in that case. I think that I may need to move the inode number check under the i_flags_lock after validating the flags - more to check that we've got the correct inode than to validate we have a freed inode.

OK, this sounds promising. Of course, the next question is "how quickly can the inode number be available for reuse?"

Immediately. Indeed, an inode number can be reused even before the inode is reclaimed. However, looking at the case of having already freed the inode when the new lookup comes in, I think checking everything under the i_flags_lock is safe.

That is, if we've freed inode #X (@ &A) and find &A during the RCU protected lookup for inode #X, the only way the inode number in the structure at &A would match #X is that if the new #X was reallocated at &A again. In that case, if the inode wasn't fully set up, we'd find either XFS_INEW|XFS_IRECLAIM still set on it and we'd back off and try the lookup again. However, if inode #X was reallocated at address &B then the inode at &A would not match #X regardless of whether &A had been reallocated or not.

Hence I think checking the inode number under the i_flags_lock after checking XFS_INEW|XFS_IRECLAIM are not set is sufficient to validate we have both an active inode and the correct inode.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#23 lore raw

PATCH 8/16 xfs: convert pag_ici_lock to a spin lock

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

now that we are using RCU protection for the inode cache lookups, the lock is only needed on the modification side. Hence it is not necessary for the lock to be a rwlock as there are no read side holders anymore. Convert it to a spin lock to reflect it's exclusive nature.

---
 fs/xfs/linux-2.6/xfs_sync.c |   14 +++++++-------
 fs/xfs/xfs_ag.h             |    2 +-
 fs/xfs/xfs_iget.c           |   10 +++++-----
 fs/xfs/xfs_mount.c          |    2 +-
 4 files changed, 14 insertions(+), 14 deletions(-)
fs/xfs/linux-2.6/xfs_sync.c +7 -7
index 9a53cc9..0b3d367 100644
@@ -596,12 +596,12 @@ xfs_inode_set_reclaim_tag(
 	struct xfs_perag *pag;
 
 	pag = xfs_perag_get(mp, XFS_INO_TO_AGNO(mp, ip->i_ino));
-	write_lock(&pag->pag_ici_lock);
+	spin_lock(&pag->pag_ici_lock);
 	spin_lock(&ip->i_flags_lock);
 	__xfs_inode_set_reclaim_tag(pag, ip);
 	__xfs_iflags_set(ip, XFS_IRECLAIMABLE);
 	spin_unlock(&ip->i_flags_lock);
-	write_unlock(&pag->pag_ici_lock);
+	spin_unlock(&pag->pag_ici_lock);
 	xfs_perag_put(pag);
 }
 
@@ -802,12 +802,12 @@ reclaim:
 	 * added to the tree assert that it's been there before to catch
 	 * problems with the inode life time early on.
 	 */
-	write_lock(&pag->pag_ici_lock);
+	spin_lock(&pag->pag_ici_lock);
 	if (!radix_tree_delete(&pag->pag_ici_root,
 				XFS_INO_TO_AGINO(ip->i_mount, ip->i_ino)))
 		ASSERT(0);
 	__xfs_inode_clear_reclaim(pag, ip);
-	write_unlock(&pag->pag_ici_lock);
+	spin_unlock(&pag->pag_ici_lock);
 
 	/*
 	 * Here we do an (almost) spurious inode lock in order to coordinate
@@ -871,14 +871,14 @@ restart:
 			struct xfs_inode *batch[XFS_LOOKUP_BATCH];
 			int	i;
 
-			write_lock(&pag->pag_ici_lock);
+			spin_lock(&pag->pag_ici_lock);
 			nr_found = radix_tree_gang_lookup_tag(
 					&pag->pag_ici_root,
 					(void **)batch, first_index,
 					XFS_LOOKUP_BATCH,
 					XFS_ICI_RECLAIM_TAG);
 			if (!nr_found) {
-				write_unlock(&pag->pag_ici_lock);
+				spin_unlock(&pag->pag_ici_lock);
 				break;
 			}
 
@@ -905,7 +905,7 @@ restart:
 			}
 
 			/* unlock now we've grabbed the inodes. */
-			write_unlock(&pag->pag_ici_lock);
+			spin_unlock(&pag->pag_ici_lock);
 
 			for (i = 0; i < nr_found; i++) {
 				if (!batch[i])
fs/xfs/xfs_ag.h +1 -1
index 63c7a1a..58632cc 100644
@@ -227,7 +227,7 @@ typedef struct xfs_perag {
 
 	atomic_t        pagf_fstrms;    /* # of filestreams active in this AG */
 
-	rwlock_t	pag_ici_lock;	/* incore inode lock */
+	spinlock_t	pag_ici_lock;	/* incore inode cache lock */
 	struct radix_tree_root pag_ici_root;	/* incore inode cache root */
 	int		pag_ici_reclaimable;	/* reclaimable inodes */
 	struct mutex	pag_ici_reclaim_lock;	/* serialisation point */
fs/xfs/xfs_iget.c +5 -5
index edeb918..e00d88c 100644
@@ -237,14 +237,14 @@ xfs_iget_cache_hit(
 			goto out_error;
 		}
 
-		write_lock(&pag->pag_ici_lock);
+		spin_lock(&pag->pag_ici_lock);
 		spin_lock(&ip->i_flags_lock);
 		ip->i_flags &= ~(XFS_IRECLAIMABLE | XFS_IRECLAIM);
 		ip->i_flags |= XFS_INEW;
 		__xfs_inode_clear_reclaim_tag(mp, pag, ip);
 		inode->i_state = I_NEW;
 		spin_unlock(&ip->i_flags_lock);
-		write_unlock(&pag->pag_ici_lock);
+		spin_unlock(&pag->pag_ici_lock);
 	} else {
 		/* If the VFS inode is being torn down, pause and try again. */
 		if (!igrab(inode)) {
@@ -322,7 +322,7 @@ xfs_iget_cache_miss(
 			BUG();
 	}
 
-	write_lock(&pag->pag_ici_lock);
+	spin_lock(&pag->pag_ici_lock);
 
 	/* insert the new inode */
 	error = radix_tree_insert(&pag->pag_ici_root, agino, ip);
@@ -337,14 +337,14 @@ xfs_iget_cache_miss(
 	ip->i_udquot = ip->i_gdquot = NULL;
 	xfs_iflags_set(ip, XFS_INEW);
 
-	write_unlock(&pag->pag_ici_lock);
+	spin_unlock(&pag->pag_ici_lock);
 	radix_tree_preload_end();
 
 	*ipp = ip;
 	return 0;
 
 out_preload_end:
-	write_unlock(&pag->pag_ici_lock);
+	spin_unlock(&pag->pag_ici_lock);
 	radix_tree_preload_end();
 	if (lock_flags)
 		xfs_iunlock(ip, lock_flags);
fs/xfs/xfs_mount.c +1 -1
index 0d9a030..7544258 100644
@@ -564,7 +564,7 @@ xfs_initialize_perag(
 			goto out_unwind;
 		pag->pag_agno = index;
 		pag->pag_mount = mp;
-		rwlock_init(&pag->pag_ici_lock);
+		spin_lock_init(&pag->pag_ici_lock);
 		mutex_init(&pag->pag_ici_reclaim_lock);
 		INIT_RADIX_TREE(&pag->pag_ici_root, GFP_ATOMIC);
 		spin_lock_init(&pag->pag_buf_lock);
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 8/16
#24 lore raw

Re: [PATCH 08/16] xfs: convert pag_ici_lock to a spin lock

Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

now that we are using RCU protection for the inode cache lookups, the lock is only needed on the modification side. Hence it is not necessary for the lock to be a rwlock as there are no read side holders anymore. Convert it to a spin lock to reflect it's exclusive nature.

Looks good, and this will make Thomas happy given that XFS is now rwlock_t-free.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#25 lore raw

PATCH 9/16 xfs: convert xfsbud shrinker to a per-buftarg shrinker.

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Before we introduce per-buftarg LRU lists, split the shrinker implementation into per-buftarg shrinker callbacks. At the moment we wake all the xfsbufds to run the delayed write queues to free the dirty buffers and make their pages available for reclaim. However, with an LRU, we want to be able to free clean, unused buffers as well, so we need to separate the xfsbufd from the shrinker callbacks.

---
 fs/xfs/linux-2.6/xfs_buf.c |   89 ++++++++++++--------------------------------
 fs/xfs/linux-2.6/xfs_buf.h |    4 +-
 2 files changed, 27 insertions(+), 66 deletions(-)
fs/xfs/linux-2.6/xfs_buf.c +24 -65
index aa1d353..f21803b 100644
@@ -44,12 +44,7 @@
 
 static kmem_zone_t *xfs_buf_zone;
 STATIC int xfsbufd(void *);
-STATIC int xfsbufd_wakeup(struct shrinker *, int, gfp_t);
 STATIC void xfs_buf_delwri_queue(xfs_buf_t *, int);
-static struct shrinker xfs_buf_shake = {
-	.shrink = xfsbufd_wakeup,
-	.seeks = DEFAULT_SEEKS,
-};
 
 static struct workqueue_struct *xfslogd_workqueue;
 struct workqueue_struct *xfsdatad_workqueue;
@@ -337,7 +332,6 @@ _xfs_buf_lookup_pages(
 					__func__, gfp_mask);
 
 			XFS_STATS_INC(xb_page_retries);
-			xfsbufd_wakeup(NULL, 0, gfp_mask);
 			congestion_wait(BLK_RW_ASYNC, HZ/50);
 			goto retry;
 		}
@@ -1464,28 +1458,23 @@ xfs_wait_buftarg(
 	}
 }
 
-/*
- *	buftarg list for delwrite queue processing
- */
-static LIST_HEAD(xfs_buftarg_list);
-static DEFINE_SPINLOCK(xfs_buftarg_lock);
-
-STATIC void
-xfs_register_buftarg(
-	xfs_buftarg_t           *btp)
-{
-	spin_lock(&xfs_buftarg_lock);
-	list_add(&btp->bt_list, &xfs_buftarg_list);
-	spin_unlock(&xfs_buftarg_lock);
-}
-
-STATIC void
-xfs_unregister_buftarg(
-	xfs_buftarg_t           *btp)
+int
+xfs_buftarg_shrink(
+	struct shrinker		*shrink,
+	int			nr_to_scan,
+	gfp_t			mask)
 {
-	spin_lock(&xfs_buftarg_lock);
-	list_del(&btp->bt_list);
-	spin_unlock(&xfs_buftarg_lock);
+	struct xfs_buftarg	*btp = container_of(shrink,
+					struct xfs_buftarg, bt_shrinker);
+	if (nr_to_scan) {
+		if (test_bit(XBT_FORCE_SLEEP, &btp->bt_flags))
+			return -1;
+		if (list_empty(&btp->bt_delwrite_queue))
+			return -1;
+		set_bit(XBT_FORCE_FLUSH, &btp->bt_flags);
+		wake_up_process(btp->bt_task);
+	}
+	return list_empty(&btp->bt_delwrite_queue) ? -1 : 1;
 }
 
 void
@@ -1493,17 +1482,14 @@ xfs_free_buftarg(
 	struct xfs_mount	*mp,
 	struct xfs_buftarg	*btp)
 {
+	unregister_shrinker(&btp->bt_shrinker);
+
 	xfs_flush_buftarg(btp, 1);
 	if (mp->m_flags & XFS_MOUNT_BARRIER)
 		xfs_blkdev_issue_flush(btp);
 	iput(btp->bt_mapping->host);
 
-	/* Unregister the buftarg first so that we don't get a
-	 * wakeup finding a non-existent task
-	 */
-	xfs_unregister_buftarg(btp);
 	kthread_stop(btp->bt_task);
-
 	kmem_free(btp);
 }
 
@@ -1600,20 +1586,13 @@ xfs_alloc_delwrite_queue(
 	xfs_buftarg_t		*btp,
 	const char		*fsname)
 {
-	int	error = 0;
-
-	INIT_LIST_HEAD(&btp->bt_list);
 	INIT_LIST_HEAD(&btp->bt_delwrite_queue);
 	spin_lock_init(&btp->bt_delwrite_lock);
 	btp->bt_flags = 0;
 	btp->bt_task = kthread_run(xfsbufd, btp, "xfsbufd/%s", fsname);
-	if (IS_ERR(btp->bt_task)) {
-		error = PTR_ERR(btp->bt_task);
-		goto out_error;
-	}
-	xfs_register_buftarg(btp);
-out_error:
-	return error;
+	if (IS_ERR(btp->bt_task))
+		return PTR_ERR(btp->bt_task);
+	return 0;
 }
 
 xfs_buftarg_t *
@@ -1636,6 +1615,9 @@ xfs_alloc_buftarg(
 		goto error;
 	if (xfs_alloc_delwrite_queue(btp, fsname))
 		goto error;
+	btp->bt_shrinker.shrink = xfs_buftarg_shrink;
+	btp->bt_shrinker.seeks = DEFAULT_SEEKS;
+	register_shrinker(&btp->bt_shrinker);
 	return btp;
 
 error:
@@ -1740,27 +1722,6 @@ xfs_buf_runall_queues(
 	flush_workqueue(queue);
 }
 
-STATIC int
-xfsbufd_wakeup(
-	struct shrinker		*shrink,
-	int			priority,
-	gfp_t			mask)
-{
-	xfs_buftarg_t		*btp;
-
-	spin_lock(&xfs_buftarg_lock);
-	list_for_each_entry(btp, &xfs_buftarg_list, bt_list) {
-		if (test_bit(XBT_FORCE_SLEEP, &btp->bt_flags))
-			continue;
-		if (list_empty(&btp->bt_delwrite_queue))
-			continue;
-		set_bit(XBT_FORCE_FLUSH, &btp->bt_flags);
-		wake_up_process(btp->bt_task);
-	}
-	spin_unlock(&xfs_buftarg_lock);
-	return 0;
-}
-
 /*
  * Move as many buffers as specified to the supplied list
  * idicating if we skipped any buffers to prevent deadlocks.
@@ -1955,7 +1916,6 @@ xfs_buf_init(void)
 	if (!xfsconvertd_workqueue)
 		goto out_destroy_xfsdatad_workqueue;
 
-	register_shrinker(&xfs_buf_shake);
 	return 0;
 
  out_destroy_xfsdatad_workqueue:
@@ -1971,7 +1931,6 @@ xfs_buf_init(void)
 void
 xfs_buf_terminate(void)
 {
-	unregister_shrinker(&xfs_buf_shake);
 	destroy_workqueue(xfsconvertd_workqueue);
 	destroy_workqueue(xfsdatad_workqueue);
 	destroy_workqueue(xfslogd_workqueue);
fs/xfs/linux-2.6/xfs_buf.h +3 -1
index 383a3f3..9344103 100644
@@ -128,10 +128,12 @@ typedef struct xfs_buftarg {
 
 	/* per device delwri queue */
 	struct task_struct	*bt_task;
-	struct list_head	bt_list;
 	struct list_head	bt_delwrite_queue;
 	spinlock_t		bt_delwrite_lock;
 	unsigned long		bt_flags;
+
+	/* LRU control structures */
+	struct shrinker		bt_shrinker;
 } xfs_buftarg_t;
 
 /*
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#26 lore raw

PATCH 10/16 xfs: add a lru to the XFS buffer cache

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Introduce a per-buftarg LRU for memory reclaim to operate on. This is the last piece we need to put in place so that we can fully control the buffer lifecycle. This allows XFS to be responsibile for maintaining the working set of buffers under memory pressure instead of relying on the VM reclaim not to take pages we need out from underneath us.

The implementation introduces a b_lru_ref counter into the buffer. This is currently set to 1 whenever the buffer is referenced and so is used to determine if the buffer should be added to the LRU or not when freed. Effectively it allows lazy LRU initialisation of the buffer so we do not need to touch the LRU list and locks in xfs_buf_find().

Instead, when the buffer is being released and we drop the last reference to it, we check the b_lru_ref count and if it is none zero we re-add the buffer reference and add the inode to the LRU. The b_lru_ref counter is decremented by the shrinker, and whenever the shrinker comes across a buffer with a zero b_lru_ref counter, if released the LRU reference on the buffer. In the absence of a lookup race, this will result in the buffer being freed.

This counting mechanism is used instead of a reference flag so that it is simple to re-introduce buffer-type specific reclaim reference counts to prioritise reclaim more effectively. We still have all those hooks in the XFS code, so this will provide the infrastructure to re-implement that functionality.

---
 fs/xfs/linux-2.6/xfs_buf.c |  168 ++++++++++++++++++++++++++++++++++++++------
 fs/xfs/linux-2.6/xfs_buf.h |    8 ++-
 2 files changed, 153 insertions(+), 23 deletions(-)
fs/xfs/linux-2.6/xfs_buf.c +145 -21
index f21803b..80d9f13 100644
@@ -163,8 +163,79 @@ test_page_region(
 }
 
 /*
- *	Internal xfs_buf_t object manipulation
+ * xfs_buf_lru_add - add a buffer to the LRU.
+ *
+ * The LRU takes a new reference to the buffer so that it will only be freed
+ * once the shrinker takes the buffer off the LRU.
+ */
+STATIC void
+xfs_buf_lru_add(
+	struct xfs_buf	*bp)
+{
+	struct xfs_buftarg *btp = bp->b_target;
+
+	spin_lock(&btp->bt_lru_lock);
+	if (list_empty(&bp->b_lru)) {
+		atomic_inc(&bp->b_hold);
+		list_add_tail(&bp->b_lru, &btp->bt_lru);
+		btp->bt_lru_nr++;
+	}
+	spin_unlock(&btp->bt_lru_lock);
+}
+
+/*
+ * xfs_buf_lru_del - remove a buffer from the LRU
+ *
+ * The unlocked check is safe here because it only occurs when there are not
+ * b_lru_ref counts left on the inode under the pag->pag_buf_lock. it is there
+ * to optimise the shrinker removing the buffer from the LRU and calling
+ * xfs_buf_free(). i.e. it removes an unneccessary round trip on the
+ * bt_lru_lock.
+ */
+STATIC void
+xfs_buf_lru_del(
+	struct xfs_buf	*bp)
+{
+	struct xfs_buftarg *btp = bp->b_target;
+
+	if (list_empty(&bp->b_lru))
+		return;
+
+	spin_lock(&btp->bt_lru_lock);
+	if (!list_empty(&bp->b_lru)) {
+		list_del_init(&bp->b_lru);
+		btp->bt_lru_nr--;
+	}
+	spin_unlock(&btp->bt_lru_lock);
+}
+
+/*
+ * When we mark a buffer stale, we remove the buffer from the LRU and clear the
+ * b_lru_ref count so that the buffer is freed immediately when the buffer
+ * reference count falls to zero. If the buffer is already on the LRU, we need
+ * to remove the reference that LRU holds on the buffer.
+ *
+ * This prevents build-up of stale buffers on the LRU.
  */
+void
+xfs_buf_stale(
+	struct xfs_buf	*bp)
+{
+	bp->b_flags |= XBF_STALE;
+	atomic_set(&(bp)->b_lru_ref, 0);
+	if (!list_empty(&bp->b_lru)) {
+		struct xfs_buftarg *btp = bp->b_target;
+
+		spin_lock(&btp->bt_lru_lock);
+		if (!list_empty(&bp->b_lru)) {
+			list_del_init(&bp->b_lru);
+			btp->bt_lru_nr--;
+			atomic_dec(&bp->b_hold);
+		}
+		spin_unlock(&btp->bt_lru_lock);
+	}
+	ASSERT(atomic_read(&bp->b_hold) >= 1);
+}
 
 STATIC void
 _xfs_buf_initialize(
@@ -181,7 +252,9 @@ _xfs_buf_initialize(
 
 	memset(bp, 0, sizeof(xfs_buf_t));
 	atomic_set(&bp->b_hold, 1);
+	atomic_set(&bp->b_lru_ref, 1);
 	init_completion(&bp->b_iowait);
+	INIT_LIST_HEAD(&bp->b_lru);
 	INIT_LIST_HEAD(&bp->b_list);
 	RB_CLEAR_NODE(&bp->b_rbnode);
 	sema_init(&bp->b_sema, 0); /* held, no waiters */
@@ -257,6 +330,8 @@ xfs_buf_free(
 {
 	trace_xfs_buf_free(bp, _RET_IP_);
 
+	ASSERT(list_empty(&bp->b_lru));
+
 	if (bp->b_flags & (_XBF_PAGE_CACHE|_XBF_PAGES)) {
 		uint		i;
 
@@ -471,6 +546,8 @@ _xfs_buf_find(
 		/* the buffer keeps the perag reference until it is freed */
 		new_bp->b_pag = pag;
 		spin_unlock(&pag->pag_buf_lock);
+
+		xfs_buf_lru_add(new_bp);
 	} else {
 		XFS_STATS_INC(xb_miss_locked);
 		spin_unlock(&pag->pag_buf_lock);
@@ -835,6 +912,7 @@ xfs_buf_rele(
 
 	if (!pag) {
 		ASSERT(!bp->b_relse);
+		ASSERT(list_empty(&bp->b_lru));
 		ASSERT(RB_EMPTY_NODE(&bp->b_rbnode));
 		if (atomic_dec_and_test(&bp->b_hold))
 			xfs_buf_free(bp);
@@ -842,13 +920,19 @@ xfs_buf_rele(
 	}
 
 	ASSERT(!RB_EMPTY_NODE(&bp->b_rbnode));
+
 	ASSERT(atomic_read(&bp->b_hold) > 0);
 	if (atomic_dec_and_lock(&bp->b_hold, &pag->pag_buf_lock)) {
 		if (bp->b_relse) {
 			atomic_inc(&bp->b_hold);
 			spin_unlock(&pag->pag_buf_lock);
 			bp->b_relse(bp);
+		} else if (!(bp->b_flags & XBF_STALE) &&
+			   atomic_read(&bp->b_lru_ref)) {
+			xfs_buf_lru_add(bp);
+			spin_unlock(&pag->pag_buf_lock);
 		} else {
+			xfs_buf_lru_del(bp);
 			ASSERT(!(bp->b_flags & (XBF_DELWRI|_XBF_DELWRI_Q)));
 			rb_erase(&bp->b_rbnode, &pag->pag_buf_tree);
 			spin_unlock(&pag->pag_buf_lock);
@@ -1435,27 +1519,35 @@ xfs_buf_iomove(
  */
 
 /*
- *	Wait for any bufs with callbacks that have been submitted but
- *	have not yet returned... walk the hash list for the target.
+ * Wait for any bufs with callbacks that have been submitted but have not yet
+ * returned. These buffers will have an elevated hold count, so wait on those
+ * while freeing all the buffers only held by the LRU.
  */
 void
 xfs_wait_buftarg(
 	struct xfs_buftarg	*btp)
 {
-	struct xfs_perag	*pag;
-	uint			i;
+	struct xfs_buf		*bp;
 
-	for (i = 0; i < btp->bt_mount->m_sb.sb_agcount; i++) {
-		pag = xfs_perag_get(btp->bt_mount, i);
-		spin_lock(&pag->pag_buf_lock);
-		while (rb_first(&pag->pag_buf_tree)) {
-			spin_unlock(&pag->pag_buf_lock);
+restart:
+	spin_lock(&btp->bt_lru_lock);
+	while (!list_empty(&btp->bt_lru)) {
+		bp = list_first_entry(&btp->bt_lru, struct xfs_buf, b_lru);
+		if (atomic_read(&bp->b_hold) > 1) {
+			spin_unlock(&btp->bt_lru_lock);
 			delay(100);
-			spin_lock(&pag->pag_buf_lock);
+			goto restart;
 		}
-		spin_unlock(&pag->pag_buf_lock);
-		xfs_perag_put(pag);
+		/*
+		 * clear the LRU reference count so the bufer doesn't get
+		 * ignored in xfs_buf_rele().
+		 */
+		atomic_set(&bp->b_lru_ref, 0);
+		spin_unlock(&btp->bt_lru_lock);
+		xfs_buf_rele(bp);
+		spin_lock(&btp->bt_lru_lock);
 	}
+	spin_unlock(&btp->bt_lru_lock);
 }
 
 int
@@ -1466,15 +1558,45 @@ xfs_buftarg_shrink(
 {
 	struct xfs_buftarg	*btp = container_of(shrink,
 					struct xfs_buftarg, bt_shrinker);
-	if (nr_to_scan) {
-		if (test_bit(XBT_FORCE_SLEEP, &btp->bt_flags))
-			return -1;
-		if (list_empty(&btp->bt_delwrite_queue))
-			return -1;
-		set_bit(XBT_FORCE_FLUSH, &btp->bt_flags);
-		wake_up_process(btp->bt_task);
+	struct xfs_buf		*bp;
+	LIST_HEAD(dispose);
+
+	if (!nr_to_scan)
+		return btp->bt_lru_nr;
+
+	spin_lock(&btp->bt_lru_lock);
+	while (!list_empty(&btp->bt_lru)) {
+		if (nr_to_scan-- <= 0)
+			break;
+
+		bp = list_first_entry(&btp->bt_lru, struct xfs_buf, b_lru);
+
+		/*
+		 * Decrement the b_lru_ref count unless the value is already
+		 * zero. If the value is already zero, we need to reclaim the
+		 * buffer, otherwise it gets another trip through the LRU.
+		 */
+		if (!atomic_add_unless(&bp->b_lru_ref, -1, 0)) {
+			list_move_tail(&bp->b_lru, &btp->bt_lru);
+			continue;
+		}
+
+		/*
+		 * remove the buffer from the LRU now to avoid needing another
+		 * lock round trip inside xfs_buf_rele().
+		 */
+		list_move(&bp->b_lru, &dispose);
+		btp->bt_lru_nr--;
 	}
-	return list_empty(&btp->bt_delwrite_queue) ? -1 : 1;
+	spin_unlock(&btp->bt_lru_lock);
+
+	while (!list_empty(&dispose)) {
+		bp = list_first_entry(&dispose, struct xfs_buf, b_lru);
+		list_del_init(&bp->b_lru);
+		xfs_buf_rele(bp);
+	}
+
+	return btp->bt_lru_nr;
 }
 
 void
@@ -1609,6 +1731,8 @@ xfs_alloc_buftarg(
 	btp->bt_mount = mp;
 	btp->bt_dev =  bdev->bd_dev;
 	btp->bt_bdev = bdev;
+	INIT_LIST_HEAD(&btp->bt_lru);
+	spin_lock_init(&btp->bt_lru_lock);
 	if (xfs_setsize_buftarg_early(btp, bdev))
 		goto error;
 	if (xfs_mapping_buftarg(btp, bdev))
fs/xfs/linux-2.6/xfs_buf.h +7 -1
index 9344103..4601eab 100644
@@ -134,6 +134,9 @@ typedef struct xfs_buftarg {
 
 	/* LRU control structures */
 	struct shrinker		bt_shrinker;
+	struct list_head	bt_lru;
+	spinlock_t		bt_lru_lock;
+	unsigned int		bt_lru_nr;
 } xfs_buftarg_t;
 
 /*
@@ -166,9 +169,11 @@ typedef struct xfs_buf {
 	xfs_off_t		b_file_offset;	/* offset in file */
 	size_t			b_buffer_length;/* size of buffer in bytes */
 	atomic_t		b_hold;		/* reference count */
+	atomic_t		b_lru_ref;	/* lru reclaim ref count */
 	xfs_buf_flags_t		b_flags;	/* status flags */
 	struct semaphore	b_sema;		/* semaphore for lockables */
 
+	struct list_head	b_lru;		/* lru list */
 	wait_queue_head_t	b_waiters;	/* unpin waiters */
 	struct list_head	b_list;
 	struct xfs_perag	*b_pag;		/* contains rbtree root */
@@ -266,7 +271,8 @@ extern void xfs_buf_terminate(void);
 #define XFS_BUF_ZEROFLAGS(bp)	((bp)->b_flags &= \
 		~(XBF_READ|XBF_WRITE|XBF_ASYNC|XBF_DELWRI|XBF_ORDERED))
 
-#define XFS_BUF_STALE(bp)	((bp)->b_flags |= XBF_STALE)
+void xfs_buf_stale(struct xfs_buf *bp);
+#define XFS_BUF_STALE(bp)	xfs_buf_stale(bp);
 #define XFS_BUF_UNSTALE(bp)	((bp)->b_flags &= ~XBF_STALE)
 #define XFS_BUF_ISSTALE(bp)	((bp)->b_flags & XBF_STALE)
 #define XFS_BUF_SUPER_STALE(bp)	do {				\
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#27 lore raw

Re: [PATCH 10/16] xfs: add a lru to the XFS buffer cache

fs/xfs/linux-2.6/xfs_buf.c _xfs_buf_find( · line 550

Quoted 6 lines
fs/xfs/linux-2.6/xfs_buf.c +2 -0
@@ -471,6 +546,8 @@ _xfs_buf_find(
 		/* the buffer keeps the perag reference until it is freed */
 		new_bp->b_pag = pag;
 		spin_unlock(&pag->pag_buf_lock);
+
+		xfs_buf_lru_add(new_bp);

Why do we add the buffer to the lru when we find it? Normally we would remove it here (unless we want a lazy lru scheme), and potentially increment b_lru_ref - although that seems to be done by the callers in the next patch.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#28 lore raw

Re: [PATCH 10/16] xfs: add a lru to the XFS buffer cache

Quoted 9 lines · Christoph Hellwig

fs/xfs/linux-2.6/xfs_buf.c _xfs_buf_find( · line 550

Quoted 6 lines
fs/xfs/linux-2.6/xfs_buf.c +2 -0
@@ -471,6 +546,8 @@ _xfs_buf_find(
 		/* the buffer keeps the perag reference until it is freed */
 		new_bp->b_pag = pag;
 		spin_unlock(&pag->pag_buf_lock);
+
+		xfs_buf_lru_add(new_bp);

Why do we add the buffer to the lru when we find it? Normally we would remove it here (unless we want a lazy lru scheme),

Oh, I forgot to remove that from the patch when rewriting it to use lazy updateѕ. Good catch! (*)

Quoted 3 lines

and potentially increment b_lru_ref - although that seems to be done by the callers in the next patch.

b_lru_ref is initialised to 1 when the buffer is first initialised, so it doesn't need to be done here. And yes, the next patch allows the users of the buffers to set the reclaim reference count themselves when the buffer is read for prioritisation. ideally I don't want to have to touch the reclaim state of the buffer during xfs_buf_find....

Cheers,

Dave.

(*) Catching this sort of bug is exactly why I posted the series early. Little details are easy to miss in a forest of changes this large....

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#29 lore raw

PATCH 11/16 xfs: connect up buffer reclaim priority hooks

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Now that the buffer reclaim infrastructure can handle different reclaim priorities for different types of buffers, reconnect the hooks in the XFS code that has been sitting dormant since it was ported to Linux. This should finally give use reclaim prioritisation that is on a par with the functionality that Irix provided XFS 15 years ago.

---
 fs/xfs/linux-2.6/xfs_buf.h |   31 +++++++++++++++++++++++++++++--
 fs/xfs/quota/xfs_dquot.c   |    2 +-
 fs/xfs/xfs_alloc.c         |    4 ++--
 fs/xfs/xfs_btree.c         |   11 +++++------
 fs/xfs/xfs_da_btree.c      |    4 ++--
 fs/xfs/xfs_ialloc.c        |    2 +-
 fs/xfs/xfs_inode.c         |    2 +-
 fs/xfs/xfs_trans.h         |    2 +-
 8 files changed, 42 insertions(+), 16 deletions(-)
fs/xfs/linux-2.6/xfs_buf.h +29 -2
index 4601eab..2dff03f 100644
@@ -336,9 +336,36 @@ void xfs_buf_stale(struct xfs_buf *bp);
 #define XFS_BUF_SIZE(bp)		((bp)->b_buffer_length)
 #define XFS_BUF_SET_SIZE(bp, cnt)	((bp)->b_buffer_length = (cnt))
 
-#define XFS_BUF_SET_VTYPE_REF(bp, type, ref)	do { } while (0)
+/*
+ * buffer types 
+ */
+#define	B_FS_DQUOT	1
+#define	B_FS_AGFL	2
+#define	B_FS_AGF	3
+#define	B_FS_ATTR_BTREE	4
+#define	B_FS_DIR_BTREE	5
+#define	B_FS_MAP	6
+#define	B_FS_INOMAP	7
+#define	B_FS_AGI	8
+#define	B_FS_INO	9
+
+static inline void
+xfs_buf_set_vtype_ref(
+	struct xfs_buf	*bp,
+	int		type,
+	int		lru_ref)
+{
+	atomic_set(&bp->b_lru_ref, lru_ref);
+}
+
+static inline void
+xfs_buf_set_ref(
+	struct xfs_buf	*bp,
+	int		lru_ref)
+{
+	atomic_set(&bp->b_lru_ref, lru_ref);
+}
 #define XFS_BUF_SET_VTYPE(bp, type)		do { } while (0)
-#define XFS_BUF_SET_REF(bp, ref)		do { } while (0)
 
 #define XFS_BUF_ISPINNED(bp)	atomic_read(&((bp)->b_pin_count))
 
fs/xfs/quota/xfs_dquot.c +1 -1
index faf8e1a..682cbf5 100644
@@ -607,7 +607,7 @@ xfs_qm_dqread(
 	dqp->q_res_rtbcount = be64_to_cpu(ddqp->d_rtbcount);
 
 	/* Mark the buf so that this will stay incore a little longer */
-	XFS_BUF_SET_VTYPE_REF(bp, B_FS_DQUOT, XFS_DQUOT_REF);
+	xfs_buf_set_vtype_ref(bp, B_FS_DQUOT, XFS_DQUOT_REF);
 
 	/*
 	 * We got the buffer with a xfs_trans_read_buf() (in dqtobp())
fs/xfs/xfs_alloc.c +2 -2
index 112abc4..702b097 100644
@@ -463,7 +463,7 @@ xfs_alloc_read_agfl(
 		return error;
 	ASSERT(bp);
 	ASSERT(!XFS_BUF_GETERROR(bp));
-	XFS_BUF_SET_VTYPE_REF(bp, B_FS_AGFL, XFS_AGFL_REF);
+	xfs_buf_set_vtype_ref(bp, B_FS_AGFL, XFS_AGFL_REF);
 	*bpp = bp;
 	return 0;
 }
@@ -2160,7 +2160,7 @@ xfs_read_agf(
 		xfs_trans_brelse(tp, *bpp);
 		return XFS_ERROR(EFSCORRUPTED);
 	}
-	XFS_BUF_SET_VTYPE_REF(*bpp, B_FS_AGF, XFS_AGF_REF);
+	xfs_buf_set_vtype_ref(*bpp, B_FS_AGF, XFS_AGF_REF);
 	return 0;
 }
 
fs/xfs/xfs_btree.c +5 -6
index 04f9cca..20cec22 100644
@@ -634,9 +634,8 @@ xfs_btree_read_bufl(
 		return error;
 	}
 	ASSERT(!bp || !XFS_BUF_GETERROR(bp));
-	if (bp != NULL) {
-		XFS_BUF_SET_VTYPE_REF(bp, B_FS_MAP, refval);
-	}
+	if (bp)
+		xfs_buf_set_vtype_ref(bp, B_FS_MAP, refval);
 	*bpp = bp;
 	return 0;
 }
@@ -944,13 +943,13 @@ xfs_btree_set_refs(
 	switch (cur->bc_btnum) {
 	case XFS_BTNUM_BNO:
 	case XFS_BTNUM_CNT:
-		XFS_BUF_SET_VTYPE_REF(*bpp, B_FS_MAP, XFS_ALLOC_BTREE_REF);
+		xfs_buf_set_vtype_ref(bp, B_FS_MAP, XFS_ALLOC_BTREE_REF);
 		break;
 	case XFS_BTNUM_INO:
-		XFS_BUF_SET_VTYPE_REF(*bpp, B_FS_INOMAP, XFS_INO_BTREE_REF);
+		xfs_buf_set_vtype_ref(bp, B_FS_INOMAP, XFS_INO_BTREE_REF);
 		break;
 	case XFS_BTNUM_BMAP:
-		XFS_BUF_SET_VTYPE_REF(*bpp, B_FS_MAP, XFS_BMAP_BTREE_REF);
+		xfs_buf_set_vtype_ref(bp, B_FS_MAP, XFS_BMAP_BTREE_REF);
 		break;
 	default:
 		ASSERT(0);
fs/xfs/xfs_da_btree.c +2 -2
index 1c00bed..eea90ff 100644
@@ -2056,10 +2056,10 @@ xfs_da_do_buf(
 			continue;
 		if (caller == 1) {
 			if (whichfork == XFS_ATTR_FORK) {
-				XFS_BUF_SET_VTYPE_REF(bp, B_FS_ATTR_BTREE,
+				xfs_buf_set_vtype_ref(bp, B_FS_ATTR_BTREE,
 						XFS_ATTR_BTREE_REF);
 			} else {
-				XFS_BUF_SET_VTYPE_REF(bp, B_FS_DIR_BTREE,
+				xfs_buf_set_vtype_ref(bp, B_FS_DIR_BTREE,
 						XFS_DIR_BTREE_REF);
 			}
 		}
fs/xfs/xfs_ialloc.c +1 -1
index 0626a32..7fe7f35 100644
@@ -1517,7 +1517,7 @@ xfs_read_agi(
 		return XFS_ERROR(EFSCORRUPTED);
 	}
 
-	XFS_BUF_SET_VTYPE_REF(*bpp, B_FS_AGI, XFS_AGI_REF);
+	xfs_buf_set_vtype_ref(*bpp, B_FS_AGI, XFS_AGI_REF);
 
 	xfs_check_agi_unlinked(agi);
 	return 0;
fs/xfs/xfs_inode.c +1 -1
index 25becb1..fc09a22 100644
@@ -887,7 +887,7 @@ xfs_iread(
 	 * around for a while.  This helps to keep recently accessed
 	 * meta-data in-core longer.
 	 */
-	XFS_BUF_SET_REF(bp, XFS_INO_REF);
+	xfs_buf_set_ref(bp, XFS_INO_REF);
 
 	/*
 	 * Use xfs_trans_brelse() to release the buffer containing the
fs/xfs/xfs_trans.h +1 -1
index 246286b..c2042b7 100644
@@ -294,8 +294,8 @@ struct xfs_log_item_desc {
 #define	XFS_ALLOC_BTREE_REF	2
 #define	XFS_BMAP_BTREE_REF	2
 #define	XFS_DIR_BTREE_REF	2
+#define	XFS_INO_REF		2
 #define	XFS_ATTR_BTREE_REF	1
-#define	XFS_INO_REF		1
 #define	XFS_DQUOT_REF		1
 
 #ifdef __KERNEL__
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#30 lore raw

Re: [PATCH 11/16] xfs: connect up buffer reclaim priority hooks

fs/xfs/linux-2.6/xfs_buf.h void xfs_buf_stale(struct xfs_buf *bp); · line 350

Quoted 12 lines
fs/xfs/linux-2.6/xfs_buf.h +12 -0
+/*
+ * buffer types 
+ */
+#define	B_FS_DQUOT	1
+#define	B_FS_AGFL	2
+#define	B_FS_AGF	3
+#define	B_FS_ATTR_BTREE	4
+#define	B_FS_DIR_BTREE	5
+#define	B_FS_MAP	6
+#define	B_FS_INOMAP	7
+#define	B_FS_AGI	8
+#define	B_FS_INO	9

Is there any good reason to keep/reintroduce the buffer types? In this series we're only using the refcounts, and I can't see any good use for the types either.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#31 lore raw

Re: [PATCH 11/16] xfs: connect up buffer reclaim priority hooks

Quoted 8 lines · Christoph Hellwig
Quoted 8 lines
fs/xfs/linux-2.6/xfs_buf.h +8 -0
+/*
+ * buffer types 
+ */
+#define	B_FS_DQUOT	1
+#define	B_FS_AGFL	2
+#define	B_FS_AGF	3
+#define	B_FS_ATTR_BTREE	4
+#define	B_FS_DIR_BTREE	5
Quoted 8 lines

fs/xfs/linux-2.6/xfs_buf.h void xfs_buf_stale(struct xfs_buf *bp); · line 350

Quoted 4 lines
fs/xfs/linux-2.6/xfs_buf.h +4 -0
+#define	B_FS_MAP	6
+#define	B_FS_INOMAP	7
+#define	B_FS_AGI	8
+#define	B_FS_INO	9

Is there any good reason to keep/reintroduce the buffer types? In this series we're only using the refcounts, and I can't see any good use for the types either.

I just wanted to use the existing hooks without modifying them to be plain "set reference" hooks. I'm not really fussed - this was just the simplest way to get the hookѕ working.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#32 lore raw

PATCH 12/16 xfs: bulk AIL insertion during transaction commit

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

When inserting items into the AIL from the transaction committed callbacks, we take the AIL lock for every single item that is to be inserted. For a CIL checkpoint commit, this can be tens of thousands of individual inserts, yet almost all of the items will be inserted at the same point in the AIL because they have the same index.

To reduce the overhead and contention on the AIL lock for such operations, introduce a "bulk insert" operation which allows a list of log items with the same LSN to be inserted in a single operation via a list splice. To do this, we need to pre-sort the log items being committed into a temporary list for insertion.

The complexity is that not every log item will end up with the same LSN, and not every item is actually inserted into the AIL. Items that don't match the commit LSN will be inserted and unpinned as per the current one-at-a-time method (relatively rare), while items that are not to be inserted will be unpinned and freed immediately. Items that are to be inserted at the given commit lsn are placed in a temporary array and inserted into the AIL in bulk each time the array fills up.

As a result of this, we trade off AIL hold time for a significant reduction in traffic. lock_stat output shows that the worst case hold time is unchanged, but contention from AIL inserts drops by an order of magnitude and the number of lock traversal decreases significantly.

---
 fs/xfs/xfs_extfree_item.c  |   85 +++++++++++++++++-------------
 fs/xfs/xfs_extfree_item.h  |   12 ++--
 fs/xfs/xfs_inode_item.c    |   20 +++++++
 fs/xfs/xfs_log_cil.c       |    9 +---
 fs/xfs/xfs_log_recover.c   |    4 +-
 fs/xfs/xfs_trans.c         |   70 ++++++++++++++++++++++++-
 fs/xfs/xfs_trans_ail.c     |  124 +++++++++++++++++++++++++++++++++++++-------
 fs/xfs/xfs_trans_extfree.c |    4 +-
 fs/xfs/xfs_trans_priv.h    |    9 +++-
 9 files changed, 259 insertions(+), 78 deletions(-)
fs/xfs/xfs_extfree_item.c +47 -38
index a55e687..5e16d7d 100644
@@ -74,7 +74,8 @@ xfs_efi_item_format(
 	struct xfs_efi_log_item	*efip = EFI_ITEM(lip);
 	uint			size;
 
-	ASSERT(efip->efi_next_extent == efip->efi_format.efi_nextents);
+	ASSERT(atomic_read(&efip->efi_next_extent) ==
+				efip->efi_format.efi_nextents);
 
 	efip->efi_format.efi_type = XFS_LI_EFI;
 
@@ -99,10 +100,10 @@ xfs_efi_item_pin(
 }
 
 /*
- * While EFIs cannot really be pinned, the unpin operation is the
- * last place at which the EFI is manipulated during a transaction.
- * Here we coordinate with xfs_efi_cancel() to determine who gets to
- * free the EFI.
+ * While EFIs cannot really be pinned, the unpin operation is the last place at
+ * which the EFI is manipulated during a transaction.  Here we coordinate with
+ * xfs_efi_release() (via XFS_EFI_COMMITTED) to determine who gets to free
+ * the EFI.
  */
 STATIC void
 xfs_efi_item_unpin(
@@ -112,18 +113,18 @@ xfs_efi_item_unpin(
 	struct xfs_efi_log_item	*efip = EFI_ITEM(lip);
 	struct xfs_ail		*ailp = lip->li_ailp;
 
-	spin_lock(&ailp->xa_lock);
-	if (efip->efi_flags & XFS_EFI_CANCELED) {
-		if (remove)
-			xfs_trans_del_item(lip);
-
-		/* xfs_trans_ail_delete() drops the AIL lock. */
-		xfs_trans_ail_delete(ailp, lip);
-		xfs_efi_item_free(efip);
-	} else {
-		efip->efi_flags |= XFS_EFI_COMMITTED;
-		spin_unlock(&ailp->xa_lock);
+	if (remove) {
+		/* transaction cancel - delete and free the item */
+		xfs_trans_del_item(lip);
+	} else if (test_and_clear_bit(XFS_EFI_COMMITTED, &efip->efi_flags)) {
+		/* efd has not be processed yet, it will free the efi */
+		return;
 	}
+
+	spin_lock(&ailp->xa_lock);
+	/* xfs_trans_ail_delete() drops the AIL lock. */
+	xfs_trans_ail_delete(ailp, lip);
+	xfs_efi_item_free(efip);
 }
 
 /*
@@ -152,16 +153,22 @@ xfs_efi_item_unlock(
 }
 
 /*
- * The EFI is logged only once and cannot be moved in the log, so
- * simply return the lsn at which it's been logged.  The canceled
- * flag is not paid any attention here.  Checking for that is delayed
- * until the EFI is unpinned.
+ * The EFI is logged only once and cannot be moved in the log, so simply return
+ * the lsn at which it's been logged.  The canceled flag is not paid any
+ * attention here.  Checking for that is delayed until the EFI is unpinned.
+ *
+ * For bulk transaction committed processing, the EFI may be processed but not
+ * yet unpinned prior to the EFD being processed. Set the XFS_EFI_COMMITTED
+ * flag so this case can be detected when processing the EFD.
  */
 STATIC xfs_lsn_t
 xfs_efi_item_committed(
 	struct xfs_log_item	*lip,
 	xfs_lsn_t		lsn)
 {
+	struct xfs_efi_log_item	*efip = EFI_ITEM(lip);
+
+	set_bit(XFS_EFI_COMMITTED, &efip->efi_flags);
 	return lsn;
 }
 
@@ -289,36 +296,38 @@ xfs_efi_copy_format(xfs_log_iovec_t *buf, xfs_efi_log_format_t *dst_efi_fmt)
 }
 
 /*
- * This is called by the efd item code below to release references to
- * the given efi item.  Each efd calls this with the number of
- * extents that it has logged, and when the sum of these reaches
- * the total number of extents logged by this efi item we can free
- * the efi item.
+ * This is called by the efd item code below to release references to the given
+ * efi item.  Each efd calls this with the number of extents that it has
+ * logged, and when the sum of these reaches the total number of extents logged
+ * by this efi item we can free the efi item.
+ *
+ * Freeing the efi item requires that we remove it from the AIL if it has
+ * already been placed there. However, the EFI may not yet have been placed in
+ * the AIL due to a bulk insert operation, so we have to be careful here. This
+ * case is detected if the XFS_EFI_COMMITTED flag is set. This code is
+ * tricky - both xfs_efi_item_unpin() and this code do test_and_clear_bit()
+ * operations on this flag - if it is not set here, then it means that the
+ * unpin has run and we don't need to free it. If it is set here, then we clear
+ * it to tell the unpin we have run and that the unpin needs to free the EFI.
  *
- * Freeing the efi item requires that we remove it from the AIL.
- * We'll use the AIL lock to protect our counters as well as
- * the removal from the AIL.
  */
 void
 xfs_efi_release(xfs_efi_log_item_t	*efip,
 		uint			nextents)
 {
 	struct xfs_ail		*ailp = efip->efi_item.li_ailp;
-	int			extents_left;
 
-	ASSERT(efip->efi_next_extent > 0);
-	ASSERT(efip->efi_flags & XFS_EFI_COMMITTED);
+	ASSERT(atomic_read(&efip->efi_next_extent) > 0);
 
-	spin_lock(&ailp->xa_lock);
-	ASSERT(efip->efi_next_extent >= nextents);
-	efip->efi_next_extent -= nextents;
-	extents_left = efip->efi_next_extent;
-	if (extents_left == 0) {
+	ASSERT(atomic_read(&efip->efi_next_extent) >= nextents);
+	if (!atomic_sub_and_test(nextents, &efip->efi_next_extent))
+		return;
+
+	if (!test_and_clear_bit(XFS_EFI_COMMITTED, &efip->efi_flags)) {
+		spin_lock(&ailp->xa_lock);
 		/* xfs_trans_ail_delete() drops the AIL lock. */
 		xfs_trans_ail_delete(ailp, (xfs_log_item_t *)efip);
 		xfs_efi_item_free(efip);
-	} else {
-		spin_unlock(&ailp->xa_lock);
 	}
 }
 
fs/xfs/xfs_extfree_item.h +6 -6
index 0d22c56..26a7550 100644
@@ -111,11 +111,11 @@ typedef struct xfs_efd_log_format_64 {
 #define	XFS_EFI_MAX_FAST_EXTENTS	16
 
 /*
- * Define EFI flags.
+ * Define EFI flag bits. Manipulated by set/clear/test_bit operators.
  */
-#define	XFS_EFI_RECOVERED	0x1
-#define	XFS_EFI_COMMITTED	0x2
-#define	XFS_EFI_CANCELED	0x4
+#define	XFS_EFI_RECOVERED	1
+#define	XFS_EFI_CANCELED	2
+#define	XFS_EFI_COMMITTED	3
 
 /*
  * This is the "extent free intention" log item.  It is used
@@ -125,8 +125,8 @@ typedef struct xfs_efd_log_format_64 {
  */
 typedef struct xfs_efi_log_item {
 	xfs_log_item_t		efi_item;
-	uint			efi_flags;	/* misc flags */
-	uint			efi_next_extent;
+	atomic_t		efi_next_extent;
+	unsigned long		efi_flags;	/* misc flags */
 	xfs_efi_log_format_t	efi_format;
 } xfs_efi_log_item_t;
 
fs/xfs/xfs_inode_item.c +20 -0
index c7ac020..3be7bdc 100644
@@ -663,12 +663,32 @@ xfs_inode_item_unlock(
  * all dirty data in an inode, the latest copy in the on disk log
  * is the only one that matters.  Therefore, simply return the
  * given lsn.
+ *
+ * If the inode has been marked stale because the cluster is being freed,
+ * we don't want to (re-)insert this inode into the AIL. There is a race
+ * condition where the cluster buffer may be unpinned before the inode is
+ * inserted into the AIL during transaction committed processing. If the buffer
+ * is unpinned before the inode item has been committed and inserted, then
+ * it is possible for the buffer to be written and process IO completions
+ * before the inode is inserted into the AIL. In that case, we'd be inserting a
+ * clean, stale inode into the AIL which will never get removed. It will,
+ * however, get reclaimed which triggers an assert in xfs_inode_free()
+ * complaining about freein an inode still in the AIL.
+ *
+ * To avoid this, return a lower LSN than the one passed in so that the
+ * transaction committed code will not move the inode forward in the AIL
+ * but will still unpin it properly.
  */
 STATIC xfs_lsn_t
 xfs_inode_item_committed(
 	struct xfs_log_item	*lip,
 	xfs_lsn_t		lsn)
 {
+	struct xfs_inode_log_item *iip = INODE_ITEM(lip);
+	struct xfs_inode	*ip = iip->ili_inode;
+
+	if (xfs_iflags_test(ip, XFS_ISTALE))
+		return lsn - 1;
 	return lsn;
 }
 
fs/xfs/xfs_log_cil.c +2 -7
index 23d6ceb..f36f1a2 100644
@@ -361,15 +361,10 @@ xlog_cil_committed(
 	int	abort)
 {
 	struct xfs_cil_ctx	*ctx = args;
-	struct xfs_log_vec	*lv;
-	int			abortflag = abort ? XFS_LI_ABORTED : 0;
 	struct xfs_busy_extent	*busyp, *n;
 
-	/* unpin all the log items */
-	for (lv = ctx->lv_chain; lv; lv = lv->lv_next ) {
-		xfs_trans_item_committed(lv->lv_item, ctx->start_lsn,
-							abortflag);
-	}
+	xfs_trans_committed_bulk(ctx->cil->xc_log->l_ailp, ctx->lv_chain,
+					ctx->start_lsn, abort);
 
 	list_for_each_entry_safe(busyp, n, &ctx->busy_extents, list)
 		xfs_alloc_busy_clear(ctx->cil->xc_log->l_mp, busyp);
fs/xfs/xfs_log_recover.c +2 -2
index 966d3f9..baad94a 100644
@@ -2717,8 +2717,8 @@ xlog_recover_do_efi_trans(
 		xfs_efi_item_free(efip);
 		return error;
 	}
-	efip->efi_next_extent = efi_formatp->efi_nextents;
-	efip->efi_flags |= XFS_EFI_COMMITTED;
+	atomic_set(&efip->efi_next_extent, efi_formatp->efi_nextents);
+	clear_bit(XFS_EFI_COMMITTED, &efip->efi_flags);
 
 	spin_lock(&log->l_ailp->xa_lock);
 	/*
fs/xfs/xfs_trans.c +69 -1

82 lines not on this page · raw message

fs/xfs/xfs_trans_ail.c +104 -20

203 lines not on this page · raw message

fs/xfs/xfs_trans_extfree.c +2 -2
index f783d5e..143ff840 100644
@@ -69,12 +69,12 @@ xfs_trans_log_efi_extent(xfs_trans_t		*tp,
 	tp->t_flags |= XFS_TRANS_DIRTY;
 	efip->efi_item.li_desc->lid_flags |= XFS_LID_DIRTY;
 
-	next_extent = efip->efi_next_extent;
+	next_extent = atomic_read(&efip->efi_next_extent);
 	ASSERT(next_extent < efip->efi_format.efi_nextents);
 	extp = &(efip->efi_format.efi_extents[next_extent]);
 	extp->ext_start = start_block;
 	extp->ext_len = ext_len;
-	efip->efi_next_extent++;
+	atomic_inc(&efip->efi_next_extent);
 }
 
 
fs/xfs/xfs_trans_priv.h +7 -2
index 62da86c..d25460f 100644
@@ -22,15 +22,17 @@ struct xfs_log_item;
 struct xfs_log_item_desc;
 struct xfs_mount;
 struct xfs_trans;
+struct xfs_ail;
+struct xfs_log_vec;
 
 void	xfs_trans_add_item(struct xfs_trans *, struct xfs_log_item *);
 void	xfs_trans_del_item(struct xfs_log_item *);
 void	xfs_trans_free_items(struct xfs_trans *tp, xfs_lsn_t commit_lsn,
 				int flags);
-void	xfs_trans_item_committed(struct xfs_log_item *lip,
-				xfs_lsn_t commit_lsn, int aborted);
 void	xfs_trans_unreserve_and_mod_sb(struct xfs_trans *tp);
 
+void	xfs_trans_committed_bulk(struct xfs_ail *ailp, struct xfs_log_vec *lv,
+				xfs_lsn_t commit_lsn, int aborted);
 /*
  * AIL traversal cursor.
  *
@@ -76,6 +78,9 @@ struct xfs_ail {
 void			xfs_trans_ail_update(struct xfs_ail *ailp,
 					struct xfs_log_item *lip, xfs_lsn_t lsn)
 					__releases(ailp->xa_lock);
+void			 xfs_trans_ail_update_bulk(struct xfs_ail *ailp,
+					struct xfs_log_item **lgia,
+					int nr_items, xfs_lsn_t lsn);
 void			xfs_trans_ail_delete(struct xfs_ail *ailp,
 					struct xfs_log_item *lip)
 					__releases(ailp->xa_lock);
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#33 lore raw

PATCH 13/16 xfs: reduce the number of AIL push wakeups

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The xfaild often tries to rest to wait for congestion to pass of for IO to complete, but is regularly woken in tail-pushing situations. In severe cases, the xfsaild is getting woken tens of thousands of times a second. Reduce the number needless wakeups by only waking the xfsaild if the new target is larger than the old one. Further make short sleeps uninterruptible as they occur when the xfsaild has decided it needs to back off to allow some IO to complete and being woken early is counter-productive.

---
 fs/xfs/linux-2.6/xfs_super.c |   18 +++++++++++++++---
 1 files changed, 15 insertions(+), 3 deletions(-)
fs/xfs/linux-2.6/xfs_super.c +15 -3
index fa789b7..668b010 100644
@@ -837,8 +837,11 @@ xfsaild_wakeup(
 	struct xfs_ail		*ailp,
 	xfs_lsn_t		threshold_lsn)
 {
-	ailp->xa_target = threshold_lsn;
-	wake_up_process(ailp->xa_task);
+	/* only ever move the target forwards */
+	if (XFS_LSN_CMP(threshold_lsn, ailp->xa_target) > 0) {
+		ailp->xa_target = threshold_lsn;
+		wake_up_process(ailp->xa_task);
+	}
 }
 
 STATIC int
@@ -850,8 +853,17 @@ xfsaild(
 	long		tout = 0; /* milliseconds */
 
 	while (!kthread_should_stop()) {
-		schedule_timeout_interruptible(tout ?
+		/*
+		 * for short sleeps indicating congestion, don't allow us to
+		 * get woken early. Otherwise all we do is bang on the AIL lock
+		 * without making progress.
+		 */
+		if (tout && tout <= 20) {
+			schedule_timeout_uninterruptible(msecs_to_jiffies(tout));
+		} else {
+			schedule_timeout_interruptible(tout ?
 				msecs_to_jiffies(tout) : MAX_SCHEDULE_TIMEOUT);
+		}
 
 		/* swsusp */
 		try_to_freeze();
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#34 lore raw

Re: [PATCH 13/16] xfs: reduce the number of AIL push wakeups

Quoted 7 lines

STATIC int

fs/xfs/linux-2.6/xfs_super.c +1 -1
@@ -850,8 +853,17 @@ xfsaild(
 	long		tout = 0; /* milliseconds */
 
 	while (!kthread_should_stop()) {
-		schedule_timeout_interruptible(tout ?
+		/*

fs/xfs/linux-2.6/xfs_super.c xfsaild( · line 866

Quoted 10 lines
fs/xfs/linux-2.6/xfs_super.c +9 -0
+		 * for short sleeps indicating congestion, don't allow us to
+		 * get woken early. Otherwise all we do is bang on the AIL lock
+		 * without making progress.
+		 */
+		if (tout && tout <= 20) {
+			schedule_timeout_uninterruptible(msecs_to_jiffies(tout));
+		} else {
+			schedule_timeout_interruptible(tout ?
 				msecs_to_jiffies(tout) : MAX_SCHEDULE_TIMEOUT);
+		}

How about just setting the state ourselves and calling schedule_timeout? That seems a lot more readable to me. Also we can switch to TASK_KILLABLE for the short sleeps, just to not introduce any delay in shutting down aild when kthread_stop is called. It would look something like this:

		if (tout && tout <= 20)
			__set_current_state(TASK_KILLABLE);
		else
			__set_current_state(TASK_UNINTERRUPTIBLE);
		schedule_timeout(tout ?
				 msecs_to_jiffies(tout) : MAX_SCHEDULE_TIMEOUT);

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#35 lore raw

Re: [PATCH 13/16] xfs: reduce the number of AIL push wakeups

Quoted 24 lines · Christoph Hellwig
Quoted 7 lines

STATIC int

fs/xfs/linux-2.6/xfs_super.c +1 -1
@@ -850,8 +853,17 @@ xfsaild(
 	long		tout = 0; /* milliseconds */
 
 	while (!kthread_should_stop()) {
-		schedule_timeout_interruptible(tout ?
+		/*

fs/xfs/linux-2.6/xfs_super.c xfsaild( · line 866

Quoted 10 lines
fs/xfs/linux-2.6/xfs_super.c +9 -0
+		 * for short sleeps indicating congestion, don't allow us to
+		 * get woken early. Otherwise all we do is bang on the AIL lock
+		 * without making progress.
+		 */
+		if (tout && tout <= 20) {
+			schedule_timeout_uninterruptible(msecs_to_jiffies(tout));
+		} else {
+			schedule_timeout_interruptible(tout ?
 				msecs_to_jiffies(tout) : MAX_SCHEDULE_TIMEOUT);
+		}

How about just setting the state ourselves and calling schedule_timeout? That seems a lot more readable to me. Also we can switch to TASK_KILLABLE for the short sleeps, just to not introduce any delay in shutting down aild when kthread_stop is called. It would look something like this:

Quoted 6 lines
		if (tout && tout <= 20)
			__set_current_state(TASK_KILLABLE);
		else
			__set_current_state(TASK_UNINTERRUPTIBLE);
		schedule_timeout(tout ?
				 msecs_to_jiffies(tout) : MAX_SCHEDULE_TIMEOUT);

Yes, seems reasonable. I'll convert it to do this.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#36 lore raw

PATCH 14/16 xfs: remove all the inodes on a buffer from the AIL in bulk

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

When inode buffer IO completes, usually all of the inodes are removed from the AIL. This involves processing them one at a time and taking the AIL lock once for every inode. When all CPUs are processing inode IO completions, this causes excessive amount sof contention on the AIL lock.

Instead, change the way we process inode IO completion in the buffer IO done callback. Allow the inode IO done callback to walk the list of IO done callbacks and pull all the inodes off the buffer in one go and then process them as a batch.

Once all the inodes for removal are collected, take the AIL lock once and do a bulk removal operation to minimise traffic on the AIL lock.

---
 fs/xfs/xfs_buf_item.c   |   17 +++++++--
 fs/xfs/xfs_inode_item.c |   92 ++++++++++++++++++++++++++++++++++++++---------
 fs/xfs/xfs_trans_ail.c  |   65 +++++++++++++++++++++++++++++++++
 fs/xfs/xfs_trans_priv.h |    4 ++
 4 files changed, 158 insertions(+), 20 deletions(-)
fs/xfs/xfs_buf_item.c +14 -3
index 2686d0d..46a7ef2 100644
@@ -918,6 +918,18 @@ xfs_buf_attach_iodone(
 	XFS_BUF_SET_IODONE_FUNC(bp, xfs_buf_iodone_callbacks);
 }
 
+/*
+ * We can have many callbacks on a buffer. Running the callbacks individually
+ * can cause a lot of contention on the AIL lock, so we allow for a single
+ * callback to be able to scan the remaining lip->li_bio_list for other items
+ * of the same type and callback to be processed in the first call.
+ *
+ * As a result, the loop walking the callback list below will also modify the
+ * list. it removes the first item from the list and then runs the callback.
+ * The loop then restarts from the new head of the list. This allows the
+ * callback to scan and modify the list attached to the buffer and we don't
+ * have to care about maintaining a next item pointer.
+ */
 STATIC void
 xfs_buf_do_callbacks(
 	xfs_buf_t	*bp,
@@ -925,8 +937,8 @@ xfs_buf_do_callbacks(
 {
 	xfs_log_item_t	*nlip;
 
-	while (lip != NULL) {
-		nlip = lip->li_bio_list;
+	while ((lip = XFS_BUF_FSPRIVATE(bp, xfs_log_item_t *)) != NULL) {
+		XFS_BUF_SET_FSPRIVATE(bp, lip->li_bio_list);
 		ASSERT(lip->li_cb != NULL);
 		/*
 		 * Clear the next pointer so we don't have any
@@ -936,7 +948,6 @@ xfs_buf_do_callbacks(
 		 */
 		lip->li_bio_list = NULL;
 		lip->li_cb(bp, lip);
-		lip = nlip;
 	}
 }
 
fs/xfs/xfs_inode_item.c +74 -16

117 lines not on this page · raw message

fs/xfs/xfs_trans_ail.c +65 -0

71 lines not on this page · raw message

fs/xfs/xfs_trans_priv.h +4 -0
index d25460f..8110e6f 100644
@@ -84,6 +84,10 @@ void			 xfs_trans_ail_update_bulk(struct xfs_ail *ailp,
 void			xfs_trans_ail_delete(struct xfs_ail *ailp,
 					struct xfs_log_item *lip)
 					__releases(ailp->xa_lock);
+void			xfs_trans_ail_delete_bulk(struct xfs_ail *ailp,
+					struct xfs_log_item **lgia,
+					int nr_items)
+					__releases(ailp->xa_lock);
 void			xfs_trans_ail_push(struct xfs_ail *, xfs_lsn_t);
 void			xfs_trans_unlocked_item(struct xfs_ail *,
 					xfs_log_item_t *);
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#37 lore raw

PATCH 15/16 xfs: only run xfs_error_test if error injection is active

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Recent tests writing lots of small files showed the flusher thread being CPU bound and taking a long time to do allocations on a debug kernel. perf showed this as the prime reason:

             samples  pcnt function                    DSO
             _______ _____ ___________________________ _________________

           224648.00 36.8% xfs_error_test              [kernel.kallsyms]
            86045.00 14.1% xfs_btree_check_sblock      [kernel.kallsyms]
            39778.00  6.5% prandom32                   [kernel.kallsyms]
            37436.00  6.1% xfs_btree_increment         [kernel.kallsyms]
            29278.00  4.8% xfs_btree_get_rec           [kernel.kallsyms]
            27717.00  4.5% random32                    [kernel.kallsyms]

Walking btree blocks during allocation checking them requires each block (a cache hit, so no I/O) call xfs_error_test(), which then does a random32() call as teh first operation. IOWs, ~50% of the CPU is being consumed just testing whether we need to inject an error, even though error injection is not active.

Kill this overhead when error injection is not active by adding a global counter of active error traps and only calling into xfs_error_test when fault injection is active.

---
 fs/xfs/xfs_error.c |    3 +++
 fs/xfs/xfs_error.h |    5 +++--
 2 files changed, 6 insertions(+), 2 deletions(-)
fs/xfs/xfs_error.c +3 -0
index ed99902..c78cc6a 100644
@@ -58,6 +58,7 @@ xfs_error_trap(int e)
 int	xfs_etest[XFS_NUM_INJECT_ERROR];
 int64_t	xfs_etest_fsid[XFS_NUM_INJECT_ERROR];
 char *	xfs_etest_fsname[XFS_NUM_INJECT_ERROR];
+int	xfs_error_test_active;
 
 int
 xfs_error_test(int error_tag, int *fsidp, char *expression,
@@ -108,6 +109,7 @@ xfs_errortag_add(int error_tag, xfs_mount_t *mp)
 			len = strlen(mp->m_fsname);
 			xfs_etest_fsname[i] = kmem_alloc(len + 1, KM_SLEEP);
 			strcpy(xfs_etest_fsname[i], mp->m_fsname);
+			xfs_error_test_active++;
 			return 0;
 		}
 	}
@@ -137,6 +139,7 @@ xfs_errortag_clearall(xfs_mount_t *mp, int loud)
 			xfs_etest_fsid[i] = 0LL;
 			kmem_free(xfs_etest_fsname[i]);
 			xfs_etest_fsname[i] = NULL;
+			xfs_error_test_active--;
 		}
 	}
 
fs/xfs/xfs_error.h +3 -2
index c2c1a07..f338847 100644
@@ -127,13 +127,14 @@ extern void xfs_corruption_error(const char *tag, int level,
 #define	XFS_RANDOM_BMAPIFORMAT				XFS_RANDOM_DEFAULT
 
 #ifdef DEBUG
+extern int xfs_error_test_active;
 extern int xfs_error_test(int, int *, char *, int, char *, unsigned long);
 
 #define	XFS_NUM_INJECT_ERROR				10
 #define XFS_TEST_ERROR(expr, mp, tag, rf)		\
-	((expr) || \
+	((expr) || (xfs_error_test_active && \
 	 xfs_error_test((tag), (mp)->m_fixedfsid, "expr", __LINE__, __FILE__, \
-			(rf)))
+			(rf))))
 
 extern int xfs_errortag_add(int error_tag, xfs_mount_t *mp);
 extern int xfs_errortag_clearall(xfs_mount_t *mp, int loud);
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 15/16
#38 lore raw

Re: [PATCH 15/16] xfs: only run xfs_error_test if error injection is active

Quoted 9 lines

Walking btree blocks during allocation checking them requires each block (a cache hit, so no I/O) call xfs_error_test(), which then does a random32() call as teh first operation. IOWs, ~50% of the CPU is being consumed just testing whether we need to inject an error, even though error injection is not active.

Kill this overhead when error injection is not active by adding a global counter of active error traps and only calling into xfs_error_test when fault injection is active.

Looks good. And a good reminder that we should optimize the code to not call xfs_btree_check_block on catch matches once putting the CRC checks into it later.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#39 lore raw

PATCH 16/16 xfs: make xlog_space_left() independent of the grant lock

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Convert the xlog_space_left() calculation to take the tail_lsn as a parameter. This allows the function to be called with fixed values rather than sampling the tail_lsn during the call and hence requiring it to be called under the log grant lock.

Header from folded patch 'xfs-log-ail-push-tail-unlocked':

xfs: make AIL tail pushing independent of the grant lock

Convert the xlog_grant_push_ail() calculation to take the tail_lsn and the last_sync_lsn as parameters. This allows the function to be called with fixed values rather than sampling variables protected by the grant lock. This allows us to move the grant lock outside the push function which immediately reduces unnecessary grant lock traffic, but also allows use to split the function away from the grant lock in future.

Header from folded patch 'xfs-log-ticket-queue-list-head':

xfs: Convert the log space ticket queue to use list_heads

The current code uses a roll-your-own double linked list, so convert it to a standard list_head structure and convert all the list traversals to use list_for_each_entry(). We can also get rid of the XLOG_TIC_IN_Q flag as we can use the list_empty() check to tell if the ticket is in a list or not.

---
 fs/xfs/linux-2.6/xfs_trace.h |   36 +--
 fs/xfs/xfs_log.c             |  678 ++++++++++++++++++++++--------------------
 fs/xfs/xfs_log_priv.h        |   40 ++-
 fs/xfs/xfs_log_recover.c     |   23 +-
 4 files changed, 409 insertions(+), 368 deletions(-)
fs/xfs/linux-2.6/xfs_trace.h +17 -19
index acef2e9..1a029bd 100644
@@ -766,12 +766,10 @@ DECLARE_EVENT_CLASS(xfs_loggrant_class,
 		__field(int, curr_res)
 		__field(int, unit_res)
 		__field(unsigned int, flags)
-		__field(void *, reserve_headq)
-		__field(void *, write_headq)
-		__field(int, grant_reserve_cycle)
-		__field(int, grant_reserve_bytes)
-		__field(int, grant_write_cycle)
-		__field(int, grant_write_bytes)
+		__field(void *, reserveq)
+		__field(void *, writeq)
+		__field(xfs_lsn_t, grant_reserve_lsn)
+		__field(xfs_lsn_t, grant_write_lsn)
 		__field(int, curr_cycle)
 		__field(int, curr_block)
 		__field(xfs_lsn_t, tail_lsn)
@@ -784,15 +782,15 @@ DECLARE_EVENT_CLASS(xfs_loggrant_class,
 		__entry->curr_res = tic->t_curr_res;
 		__entry->unit_res = tic->t_unit_res;
 		__entry->flags = tic->t_flags;
-		__entry->reserve_headq = log->l_reserve_headq;
-		__entry->write_headq = log->l_write_headq;
-		__entry->grant_reserve_cycle = log->l_grant_reserve_cycle;
-		__entry->grant_reserve_bytes = log->l_grant_reserve_bytes;
-		__entry->grant_write_cycle = log->l_grant_write_cycle;
-		__entry->grant_write_bytes = log->l_grant_write_bytes;
+		__entry->reserveq = log->l_reserveq.next;
+		__entry->writeq = log->l_writeq.next;
+		__entry->grant_reserve_lsn =
+				atomic64_read(&log->l_grant_reserve_lsn);
+		__entry->grant_write_lsn =
+				atomic64_read(&log->l_grant_write_lsn);
 		__entry->curr_cycle = log->l_curr_cycle;
 		__entry->curr_block = log->l_curr_block;
-		__entry->tail_lsn = log->l_tail_lsn;
+		__entry->tail_lsn = atomic64_read(&log->l_tail_lsn);
 	),
 	TP_printk("dev %d:%d type %s t_ocnt %u t_cnt %u t_curr_res %u "
 		  "t_unit_res %u t_flags %s reserve_headq 0x%p "
@@ -807,12 +805,12 @@ DECLARE_EVENT_CLASS(xfs_loggrant_class,
 		  __entry->curr_res,
 		  __entry->unit_res,
 		  __print_flags(__entry->flags, "|", XLOG_TIC_FLAGS),
-		  __entry->reserve_headq,
-		  __entry->write_headq,
-		  __entry->grant_reserve_cycle,
-		  __entry->grant_reserve_bytes,
-		  __entry->grant_write_cycle,
-		  __entry->grant_write_bytes,
+		  __entry->reserveq,
+		  __entry->writeq,
+		  CYCLE_LSN(__entry->grant_reserve_lsn),
+		  BLOCK_LSN(__entry->grant_reserve_lsn),
+		  CYCLE_LSN(__entry->grant_write_lsn),
+		  BLOCK_LSN(__entry->grant_write_lsn),
 		  __entry->curr_cycle,
 		  __entry->curr_block,
 		  CYCLE_LSN(__entry->tail_lsn),
fs/xfs/xfs_log.c +356 -322

1,094 lines not on this page · raw message

fs/xfs/xfs_log_priv.h +24 -16
index edcdfe0..4d6bf38 100644
@@ -133,12 +133,10 @@ static inline uint xlog_get_client_id(__be32 i)
  */
 #define XLOG_TIC_INITED		0x1	/* has been initialized */
 #define XLOG_TIC_PERM_RESERV	0x2	/* permanent reservation */
-#define XLOG_TIC_IN_Q		0x4
 
 #define XLOG_TIC_FLAGS \
 	{ XLOG_TIC_INITED,	"XLOG_TIC_INITED" }, \
-	{ XLOG_TIC_PERM_RESERV,	"XLOG_TIC_PERM_RESERV" }, \
-	{ XLOG_TIC_IN_Q,	"XLOG_TIC_IN_Q" }
+	{ XLOG_TIC_PERM_RESERV,	"XLOG_TIC_PERM_RESERV" }
 
 #endif	/* __KERNEL__ */
 
@@ -245,8 +243,7 @@ typedef struct xlog_res {
 
 typedef struct xlog_ticket {
 	sv_t		   t_wait;	 /* ticket wait queue            : 20 */
-	struct xlog_ticket *t_next;	 /*			         :4|8 */
-	struct xlog_ticket *t_prev;	 /*				 :4|8 */
+	struct list_head   t_queue;	 /* reserve/write queue */
 	xlog_tid_t	   t_tid;	 /* transaction identifier	 : 4  */
 	atomic_t	   t_ref;	 /* ticket reference count       : 4  */
 	int		   t_curr_res;	 /* current reservation in bytes : 4  */
@@ -509,23 +506,34 @@ typedef struct log {
 						 * log entries" */
 	xlog_in_core_t		*l_iclog;       /* head log queue	*/
 	spinlock_t		l_icloglock;    /* grab to change iclog state */
-	xfs_lsn_t		l_tail_lsn;     /* lsn of 1st LR with unflushed
-						 * buffers */
-	xfs_lsn_t		l_last_sync_lsn;/* lsn of last LR on disk */
 	int			l_curr_cycle;   /* Cycle number of log writes */
 	int			l_prev_cycle;   /* Cycle number before last
 						 * block increment */
 	int			l_curr_block;   /* current logical log block */
 	int			l_prev_block;   /* previous logical log block */
 
-	/* The following block of fields are changed while holding grant_lock */
-	spinlock_t		l_grant_lock ____cacheline_aligned_in_smp;
-	xlog_ticket_t		*l_reserve_headq;
-	xlog_ticket_t		*l_write_headq;
-	int			l_grant_reserve_cycle;
-	int			l_grant_reserve_bytes;
-	int			l_grant_write_cycle;
-	int			l_grant_write_bytes;
+	/*
+	 * The l_tail_lsn and l_last_sync_lsn variables are set up as atomic
+	 * variables so they can be safely set and read without locking. While
+	 * they are often read together, they are updated differently with the
+	 * l_tail_lsn being quite hot, so place them on spearate cachelines.
+	 */
+	/* lsn of 1st LR with unflushed buffers */
+	atomic64_t		l_tail_lsn ____cacheline_aligned_in_smp;
+	/* lsn of last LR on disk */
+	atomic64_t		l_last_sync_lsn ____cacheline_aligned_in_smp;
+
+	/*
+	 * ticket grant locks, queues and accounting have their own cachlines
+	 * as these are quite hot and can be operated on concurrently.
+	 */
+	spinlock_t		l_grant_reserve_lock ____cacheline_aligned_in_smp;
+	struct list_head	l_reserveq;
+	atomic64_t		l_grant_reserve_lsn;
+
+	spinlock_t		l_grant_write_lock ____cacheline_aligned_in_smp;
+	struct list_head	l_writeq;
+	atomic64_t		l_grant_write_lsn;
 
 	/* The following field are used for debugging; need to hold icloglock */
 #ifdef DEBUG
fs/xfs/xfs_log_recover.c +12 -11
index baad94a..f73a215 100644
@@ -925,12 +925,13 @@ xlog_find_tail(
 	log->l_curr_cycle = be32_to_cpu(rhead->h_cycle);
 	if (found == 2)
 		log->l_curr_cycle++;
-	log->l_tail_lsn = be64_to_cpu(rhead->h_tail_lsn);
-	log->l_last_sync_lsn = be64_to_cpu(rhead->h_lsn);
-	log->l_grant_reserve_cycle = log->l_curr_cycle;
-	log->l_grant_reserve_bytes = BBTOB(log->l_curr_block);
-	log->l_grant_write_cycle = log->l_curr_cycle;
-	log->l_grant_write_bytes = BBTOB(log->l_curr_block);
+	atomic64_set(&log->l_tail_lsn, be64_to_cpu(rhead->h_tail_lsn));
+	atomic64_set(&log->l_last_sync_lsn, be64_to_cpu(rhead->h_lsn));
+
+	atomic64_set(&log->l_grant_reserve_lsn,
+		xlog_assign_lsn(log->l_curr_cycle, BBTOB(log->l_curr_block)));
+	atomic64_set(&log->l_grant_write_lsn,
+		xlog_assign_lsn(log->l_curr_cycle, BBTOB(log->l_curr_block)));
 
 	/*
 	 * Look for unmount record.  If we find it, then we know there
@@ -960,7 +961,7 @@ xlog_find_tail(
 	}
 	after_umount_blk = (i + hblks + (int)
 		BTOBB(be32_to_cpu(rhead->h_len))) % log->l_logBBsize;
-	tail_lsn = log->l_tail_lsn;
+	tail_lsn = atomic64_read(&log->l_tail_lsn);
 	if (*head_blk == after_umount_blk &&
 	    be32_to_cpu(rhead->h_num_logops) == 1) {
 		umount_data_blk = (i + hblks) % log->l_logBBsize;
@@ -975,12 +976,12 @@ xlog_find_tail(
 			 * log records will point recovery to after the
 			 * current unmount record.
 			 */
-			log->l_tail_lsn =
+			atomic64_set(&log->l_tail_lsn,
 				xlog_assign_lsn(log->l_curr_cycle,
-						after_umount_blk);
-			log->l_last_sync_lsn =
+						after_umount_blk));
+			atomic64_set(&log->l_last_sync_lsn,
 				xlog_assign_lsn(log->l_curr_cycle,
-						after_umount_blk);
+						after_umount_blk));
 			*tail_blk = after_umount_blk;
 
 			/*
Signature · 6 lines
1.7.2.3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#40 lore raw
In reply to Dave Chinner
Quoted 3 lines · Dave Chinner

My tree is currently based on the VFS locking changes I have out for review, so there's a couple fo patches that won't apply sanely to a mainline or OSS xfs dev tree. See below for a pointer to a git tree with all the patches in it.

The only think that should depend on it are the inode hash changes. I suspect it might be a better idea if we feed those via Al together with the VFS scalability bits, and only feed the rest through the XFS tree to avoid having nasty dependencies.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#41 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Dave Chinner

My tree is currently based on the VFS locking changes I have out for review, so there's a couple fo patches that won't apply sanely to a mainline or OSS xfs dev tree. See below for a pointer to a git tree with all the patches in it.

The only think that should depend on it are the inode hash changes. I suspect it might be a better idea if we feed those via Al together with the VFS scalability bits, and only feed the rest through the XFS tree to avoid having nasty dependencies.

Yes, sounds reasonable.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs