xfs: lockless buffer lookups
v2 cover xfs: lockless buffer lookups
Hi folks,
Current work to merge the XFS inode life cycle with the VFS indoe life cycle is finding some interesting issues. If we have a path that hits buffer trylocks fairly hard (e.g. a non-blocking background inode freeing function), we end up hitting massive contention on the buffer cache hash locks:
- 92.71% 0.05% [kernel] [k] xfs_inodegc_worker
- 92.67% xfs_inodegc_worker
- 92.13% xfs_inode_unlink
- 91.52% xfs_inactive_ifree
- 85.63% xfs_read_agi
- 85.61% xfs_trans_read_buf_map
- 85.59% xfs_buf_read_map
- xfs_buf_get_map
- 85.55% xfs_buf_find
- 72.87% _raw_spin_lock
- do_raw_spin_lock
71.86% __pv_queued_spin_lock_slowpath
- 8.74% xfs_buf_rele
- 7.88% _raw_spin_lock
- 7.88% do_raw_spin_lock 7.63% __pv_queued_spin_lock_slowpath
- 1.70% xfs_buf_trylock
- 1.68% down_trylock
- 1.41% _raw_spin_lock_irqsave
- 1.39% do_raw_spin_lock __pv_queued_spin_lock_slowpath
- 0.76% _raw_spin_unlock
0.75% do_raw_spin_unlock
This is basically hammering the pag->pag_buf_lock from lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.
We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by creating a lookup fast path which leverages the rhashtable's ability to do rcu protected lookups.
This is a rework of the initial lockless buffer lookup patch I sent here:
https://lore.kernel.org/linux-xfs/[email protected]/
And the alternative cleanup sent by Christoph here:
https://lore.kernel.org/linux-xfs/[email protected]/
This version isn't quite a short as Christophs, but it does roughly the same thing in killing the two-phase _xfs_buf_find() call mechanism. It separates the fast and slow paths a little more cleanly and doesn't have context dependent buffer return state from the slow path that the caller needs to handle. It also picks up the rhashtable insert optimisation that Christoph added.
This series passes fstests under several different configs and does not cause any obvious regressions in scalability testing that has been performed. Hence I'm proposing this as potential 5.20 cycle material.
Thoughts, comments?
Version 2:
- based on 5.19-rc2
- high speed collision of original proposals.
Initial versions:
- https://lore.kernel.org/linux-xfs/[email protected]/
- https://lore.kernel.org/linux-xfs/[email protected]/
PATCH v2 1/6 xfs: rework xfs_buf_incore() API
From: Dave Chinner <[email protected]>
Make it consistent with the other buffer APIs to return a error and the buffer is placed in a parameter.
--- fs/xfs/libxfs/xfs_attr_remote.c | 15 ++++++++++----- fs/xfs/scrub/repair.c | 15 +++++++++------ fs/xfs/xfs_buf.c | 19 ++----------------- fs/xfs/xfs_buf.h | 20 ++++++++++++++++---- fs/xfs/xfs_qm.c | 9 ++++----- 5 files changed, 41 insertions(+), 37 deletions(-)
fs/xfs/libxfs/xfs_attr_remote.c +10 -5
@@ -543,6 +543,7 @@ xfs_attr_rmtval_stale( {struct xfs_mount *mp = ip->i_mount;struct xfs_buf *bp; +int error;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));
@@ -550,14 +551,18 @@ xfs_attr_rmtval_stale( XFS_IS_CORRUPT(mp, map->br_startblock == HOLESTARTBLOCK)) return -EFSCORRUPTED; - bp =xfs_buf_incore (mp->m_ddev_targp, + error =xfs_buf_incore (mp->m_ddev_targp,XFS_FSB_TO_DADDR (mp, map->br_startblock), -XFS_FSB_TO_BB (mp, map->br_blockcount), incore_flags); -if (bp) { -xfs_buf_stale (bp); -xfs_buf_relse (bp); +XFS_FSB_TO_BB (mp, map->br_blockcount), + incore_flags, &bp); +if (error) { +if (error == -ENOENT ) +return 0 ; +return error; } +xfs_buf_stale (bp); +xfs_buf_relse (bp);return 0 ; }
fs/xfs/scrub/repair.c +9 -6
@@ -457,16 +457,19 @@ xrep_invalidate_blocks(* assume it's owned by someone else. */ for_each_xbitmap_block (fsbno, bmr, n, bitmap) { +int error; +/* Skip AG headers and post-EOFS blocks */ if (!xfs_verify_fsbno (sc->mp, fsbno))continue ; - bp =xfs_buf_incore (sc->mp->m_ddev_targp, + error =xfs_buf_incore (sc->mp->m_ddev_targp,XFS_FSB_TO_DADDR (sc->mp, fsbno), -XFS_FSB_TO_BB (sc->mp,1 ),XBF_TRYLOCK ); -if (bp) { -xfs_trans_bjoin (sc->tp, bp); -xfs_trans_binval (sc->tp, bp); - } +XFS_FSB_TO_BB (sc->mp,1 ),XBF_TRYLOCK , &bp); +if (error) +continue ; + +xfs_trans_bjoin (sc->tp, bp); +xfs_trans_binval (sc->tp, bp); }return 0 ;
fs/xfs/xfs_buf.c +2 -17
@@ -616,23 +616,6 @@ xfs_buf_find(return 0 ; } -struct xfs_buf * -xfs_buf_incore ( -struct xfs_buftarg *target, -xfs_daddr_t blkno, -size_t numblks, -xfs_buf_flags_t flags) -{ -struct xfs_buf *bp; -int error; -DEFINE_SINGLE_BUF_MAP (map, blkno, numblks); - - error =xfs_buf_find (target, &map,1 , flags,NULL , &bp); -if (error) -return NULL ; -return bp; -} -/* * Assembles a buffer covering the specified range. The code is optimised for * cache hits, as metadata intensive workloads will see 3 orders of magnitude
@@ -656,6 +639,8 @@ xfs_buf_get_map( goto found;if (error != -ENOENT )return error; +if (flags &XBF_INCORE ) +return -ENOENT ; error =_xfs_buf_alloc (target, map, nmaps, flags, &new_bp);if (error)
fs/xfs/xfs_buf.h +16 -4
@@ -42,9 +42,11 @@ struct xfs_buf;#define _XBF_DELWRI_Q (1u << 22)/* buffer on a delwri queue */ /* flags used only as arguments to access routines */ +#define XBF_INCORE (1u << 29)/* lookup only, return if found in cache */ #define XBF_TRYLOCK (1u << 30)/* lock requested, but do not wait */ #define XBF_UNMAPPED (1u << 31)/* do not map the buffer */ +typedef unsigned int xfs_buf_flags_t ;#define XFS_BUF_FLAGS \
@@ -63,6 +65,7 @@ typedef unsigned int xfs_buf_flags_t; { _XBF_KMEM, "KMEM" }, \ { _XBF_DELWRI_Q, "DELWRI_Q" }, \/* The following interface flags should never be set */ \ + { XBF_INCORE, "INCORE" }, \ { XBF_TRYLOCK, "TRYLOCK" }, \ { XBF_UNMAPPED, "UNMAPPED" }
@@ -196,10 +199,6 @@ struct xfs_buf { };/* Finding and Reading Buffers */ -struct xfs_buf *xfs_buf_incore (struct xfs_buftarg *target, -xfs_daddr_t blkno,size_t numblks, -xfs_buf_flags_t flags); -int xfs_buf_get_map (struct xfs_buftarg *target,struct xfs_buf_map *map,int nmaps,xfs_buf_flags_t flags,struct xfs_buf **bpp); int xfs_buf_read_map(struct xfs_buftarg *target, struct xfs_buf_map *map,
@@ -209,6 +208,19 @@ void xfs_buf_readahead_map(struct xfs_buftarg *target, struct xfs_buf_map *map, int nmaps, const struct xfs_buf_ops *ops); +static inline int +xfs_buf_incore ( +struct xfs_buftarg *target, +xfs_daddr_t blkno, +size_t numblks, +xfs_buf_flags_t flags, +struct xfs_buf **bpp) +{ +DEFINE_SINGLE_BUF_MAP (map, blkno, numblks); + +return xfs_buf_get_map (target, &map,1 ,XBF_INCORE | flags, bpp); +} + static inline int xfs_buf_get( struct xfs_buftarg *target,
fs/xfs/xfs_qm.c +4 -5
@@ -1229,12 +1229,11 @@ xfs_qm_flush_one(*/ if (!xfs_dqflock_nowait (dqp)) {/* buf is pinned in-core by delwri list */ - bp =xfs_buf_incore (mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 ); -if (!bp) { - error = -EINVAL ; + error =xfs_buf_incore (mp->m_ddev_targp, dqp->q_blkno, + mp->m_quotainfo->qi_dqchunklen,0 , &bp); +if (error) goto out_unlock; - } +xfs_buf_unlock (bp);xfs_buf_delwri_pushbuf (bp, buffer_list);
Signature · 1 line
2.36.1
Re: [PATCH 1/6] xfs: rework xfs_buf_incore() API
Quoted 4 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Make it consistent with the other buffer APIs to return a error and the buffer is placed in a parameter.
Looks good, although I'd just drop the xfs_buf_incore wrapper now as well..
Re: [PATCH 1/6] xfs: rework xfs_buf_incore() API
In reply to Dave ChinnerQuoted 6 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Make it consistent with the other buffer APIs to return a error and the buffer is placed in a parameter.
Signed-off-by Dave Chinner <[email protected]>
LGTM
--D
Quoted 190 lines
--- fs/xfs/libxfs/xfs_attr_remote.c | 15 ++++++++++----- fs/xfs/scrub/repair.c | 15 +++++++++------ fs/xfs/xfs_buf.c | 19 ++----------------- fs/xfs/xfs_buf.h | 20 ++++++++++++++++---- fs/xfs/xfs_qm.c | 9 ++++----- 5 files changed, 41 insertions(+), 37 deletions(-)fs/xfs/libxfs/xfs_attr_remote.c +10
-5@@ -543,6 +543,7 @@ xfs_attr_rmtval_stale( {struct xfs_mount *mp = ip->i_mount;struct xfs_buf *bp; +int error;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));@@ -550,14 +551,18 @@ xfs_attr_rmtval_stale( XFS_IS_CORRUPT(mp, map->br_startblock == HOLESTARTBLOCK)) return -EFSCORRUPTED; - bp =xfs_buf_incore (mp->m_ddev_targp, + error =xfs_buf_incore (mp->m_ddev_targp,XFS_FSB_TO_DADDR (mp, map->br_startblock), -XFS_FSB_TO_BB (mp, map->br_blockcount), incore_flags); -if (bp) { -xfs_buf_stale (bp); -xfs_buf_relse (bp); +XFS_FSB_TO_BB (mp, map->br_blockcount), + incore_flags, &bp); +if (error) { +if (error == -ENOENT ) +return 0 ; +return error; } +xfs_buf_stale (bp); +xfs_buf_relse (bp);return 0 ; }fs/xfs/scrub/repair.c +9
-6@@ -457,16 +457,19 @@ xrep_invalidate_blocks(* assume it's owned by someone else. */ for_each_xbitmap_block (fsbno, bmr, n, bitmap) { +int error; +/* Skip AG headers and post-EOFS blocks */ if (!xfs_verify_fsbno (sc->mp, fsbno))continue ; - bp =xfs_buf_incore (sc->mp->m_ddev_targp, + error =xfs_buf_incore (sc->mp->m_ddev_targp,XFS_FSB_TO_DADDR (sc->mp, fsbno), -XFS_FSB_TO_BB (sc->mp,1 ),XBF_TRYLOCK ); -if (bp) { -xfs_trans_bjoin (sc->tp, bp); -xfs_trans_binval (sc->tp, bp); - } +XFS_FSB_TO_BB (sc->mp,1 ),XBF_TRYLOCK , &bp); +if (error) +continue ; + +xfs_trans_bjoin (sc->tp, bp); +xfs_trans_binval (sc->tp, bp); }return 0 ;fs/xfs/xfs_buf.c +2
-17@@ -616,23 +616,6 @@ xfs_buf_find(return 0 ; } -struct xfs_buf * -xfs_buf_incore ( -struct xfs_buftarg *target, -xfs_daddr_t blkno, -size_t numblks, -xfs_buf_flags_t flags) -{ -struct xfs_buf *bp; -int error; -DEFINE_SINGLE_BUF_MAP (map, blkno, numblks); - - error =xfs_buf_find (target, &map,1 , flags,NULL , &bp); -if (error) -return NULL ; -return bp; -} -/* * Assembles a buffer covering the specified range. The code is optimised for * cache hits, as metadata intensive workloads will see 3 orders of magnitude @@ -656,6 +639,8 @@ xfs_buf_get_map( goto found;if (error != -ENOENT )return error; +if (flags &XBF_INCORE ) +return -ENOENT ; error =_xfs_buf_alloc (target, map, nmaps, flags, &new_bp);if (error)fs/xfs/xfs_buf.h +16
-4@@ -42,9 +42,11 @@ struct xfs_buf;#define _XBF_DELWRI_Q (1u << 22)/* buffer on a delwri queue */ /* flags used only as arguments to access routines */ +#define XBF_INCORE (1u << 29)/* lookup only, return if found in cache */ #define XBF_TRYLOCK (1u << 30)/* lock requested, but do not wait */ #define XBF_UNMAPPED (1u << 31)/* do not map the buffer */ +typedef unsigned int xfs_buf_flags_t ;#define XFS_BUF_FLAGS \@@ -63,6 +65,7 @@ typedef unsigned int xfs_buf_flags_t; { _XBF_KMEM, "KMEM" }, \ { _XBF_DELWRI_Q, "DELWRI_Q" }, \/* The following interface flags should never be set */ \ + { XBF_INCORE, "INCORE" }, \ { XBF_TRYLOCK, "TRYLOCK" }, \ { XBF_UNMAPPED, "UNMAPPED" }@@ -196,10 +199,6 @@ struct xfs_buf { };/* Finding and Reading Buffers */ -struct xfs_buf *xfs_buf_incore (struct xfs_buftarg *target, -xfs_daddr_t blkno,size_t numblks, -xfs_buf_flags_t flags); -int xfs_buf_get_map (struct xfs_buftarg *target,struct xfs_buf_map *map,int nmaps,xfs_buf_flags_t flags,struct xfs_buf **bpp); int xfs_buf_read_map(struct xfs_buftarg *target, struct xfs_buf_map *map,@@ -209,6 +208,19 @@ void xfs_buf_readahead_map(struct xfs_buftarg *target, struct xfs_buf_map *map, int nmaps, const struct xfs_buf_ops *ops); +static inline int +xfs_buf_incore ( +struct xfs_buftarg *target, +xfs_daddr_t blkno, +size_t numblks, +xfs_buf_flags_t flags, +struct xfs_buf **bpp) +{ +DEFINE_SINGLE_BUF_MAP (map, blkno, numblks); + +return xfs_buf_get_map (target, &map,1 ,XBF_INCORE | flags, bpp); +} + static inline int xfs_buf_get( struct xfs_buftarg *target,fs/xfs/xfs_qm.c +4
-5@@ -1229,12 +1229,11 @@ xfs_qm_flush_one(*/ if (!xfs_dqflock_nowait (dqp)) {/* buf is pinned in-core by delwri list */ - bp =xfs_buf_incore (mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 ); -if (!bp) { - error = -EINVAL ; + error =xfs_buf_incore (mp->m_ddev_targp, dqp->q_blkno, + mp->m_quotainfo->qi_dqchunklen,0 , &bp); +if (error) goto out_unlock; - } +xfs_buf_unlock (bp);xfs_buf_delwri_pushbuf (bp, buffer_list);Signature · 1 line
2.36.1
PATCH v2 2/6 xfs: break up xfs_buf_find() into individual pieces
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
xfs_buf_find() is made up of three main parts: lookup, insert and locking. The interactions with xfs_buf_get_map() require it to be called twice - once for a pure lookup, and again on lookup failure so the insert path can be run. We want to simplify this down a lot, so split it into a fast path lookup, a slow path insert and a "lock the found buffer" helper. This will then let use integrate these operations more effectively into xfs_buf_get_map() in future patches.
--- fs/xfs/xfs_buf.c | 159 +++++++++++++++++++++++++++++++---------------- 1 file changed, 105 insertions(+), 54 deletions(-)
fs/xfs/xfs_buf.c +105 -54
@@ -503,77 +503,60 @@ xfs_buf_hash_destroy(rhashtable_destroy (&pag->pag_buf_hash); } -/* -* Look up a buffer in the buffer cache and return it referenced and locked -* in @found_bp. -* -* If @new_bp is supplied and we have a lookup miss, insert @new_bp into the -* cache. -* -* If XBF_TRYLOCK is set in @flags, only try to lock the buffer and return -* -EAGAIN if we fail to lock it. -* -* Return values are: -* -EFSCORRUPTED if have been supplied with an invalid address -* -EAGAIN on trylock failure -* -ENOENT if we fail to find a match and @new_bp was NULL -* 0, with @found_bp: -* - @new_bp if we inserted it into the cache -* - the buffer we found and locked. -*/ static int -xfs_buf_find ( +xfs_buf_find_verify (struct xfs_buftarg *btp, -struct xfs_buf_map *map, -int nmaps, -xfs_buf_flags_t flags, -struct xfs_buf *new_bp, -struct xfs_buf **found_bp) +struct xfs_buf_map *map) { -struct xfs_perag *pag; -struct xfs_buf *bp; -struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn };xfs_daddr_t eofs; -int i; - - *found_bp =NULL ; - -for (i =0 ; i < nmaps; i++) - cmap.bm_len += map[i].bm_len;/* Check for IOs smaller than the sector size / not sector aligned */ -ASSERT (!(BBTOB (cmap.bm_len) < btp->bt_meta_sectorsize)); -ASSERT (!(BBTOB (cmap.bm_bn) & (xfs_off_t )btp->bt_meta_sectormask)); +ASSERT (!(BBTOB (map->bm_len) < btp->bt_meta_sectorsize)); +ASSERT (!(BBTOB (map->bm_bn) & (xfs_off_t )btp->bt_meta_sectormask));/* * Corrupted block numbers can get through to here, unfortunately, so we * have to check that the buffer falls within the filesystem bounds. */ eofs =XFS_FSB_TO_BB (btp->bt_mount, btp->bt_mount->m_sb.sb_dblocks); -if (cmap.bm_bn <0 || cmap.bm_bn >= eofs) { +if (map->bm_bn <0 || map->bm_bn >= eofs) {xfs_alert (btp->bt_mount,"%s: daddr 0x%llx out of range, EOFS 0x%llx" , - __func__, cmap.bm_bn, eofs); + __func__, map->bm_bn, eofs);WARN_ON (1 );return -EFSCORRUPTED ; } +return 0 ; +} - pag =xfs_perag_get (btp->bt_mount, -xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); +static inline struct xfs_buf * +xfs_buf_find_fast ( +struct xfs_perag *pag, +struct xfs_buf_map *map) +{ +struct xfs_buf *bp; -spin_lock (&pag->pag_buf_lock); - bp =rhashtable_lookup_fast (&pag->pag_buf_hash, &cmap, - xfs_buf_hash_params); -if (bp) { -atomic_inc (&bp->b_hold); - goto found; - } + bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); +if (!bp) +return NULL ; +atomic_inc (&bp->b_hold); +return bp; +} +/* +* Insert the new_bp into the hash table. This consumes the perag reference +* taken for the lookup. +*/ +static int +xfs_buf_find_insert ( +struct xfs_buftarg *btp, +struct xfs_perag *pag, +struct xfs_buf *new_bp) +{/* No match found */ if (!new_bp) { -XFS_STATS_INC (btp->bt_mount, xb_miss_locked); -spin_unlock (&pag->pag_buf_lock);xfs_perag_put (pag); +XFS_STATS_INC (btp->bt_mount, xb_miss_locked);return -ENOENT ; }
@@ -581,14 +564,15 @@ xfs_buf_find( new_bp->b_pag = pag;rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, xfs_buf_hash_params); -spin_unlock (&pag->pag_buf_lock); - *found_bp = new_bp;return 0 ; +} -found: -spin_unlock (&pag->pag_buf_lock); -xfs_perag_put (pag); - +static int +xfs_buf_find_lock ( +struct xfs_buftarg *btp, +struct xfs_buf *bp, +xfs_buf_flags_t flags) +{if (!xfs_buf_trylock (bp)) {if (flags &XBF_TRYLOCK ) {xfs_buf_rele (bp);
@@ -609,6 +593,73 @@ xfs_buf_find( bp->b_flags &= _XBF_KMEM | _XBF_PAGES; bp->b_ops =NULL ; } +return 0 ; +} + +/* +* Look up a buffer in the buffer cache and return it referenced and locked +* in @found_bp. +* +* If @new_bp is supplied and we have a lookup miss, insert @new_bp into the +* cache. +* +* If XBF_TRYLOCK is set in @flags, only try to lock the buffer and return +* -EAGAIN if we fail to lock it. +* +* Return values are: +* -EFSCORRUPTED if have been supplied with an invalid address +* -EAGAIN on trylock failure +* -ENOENT if we fail to find a match and @new_bp was NULL +* 0, with @found_bp: +* - @new_bp if we inserted it into the cache +* - the buffer we found and locked. +*/ +static int +xfs_buf_find ( +struct xfs_buftarg *btp, +struct xfs_buf_map *map, +int nmaps, +xfs_buf_flags_t flags, +struct xfs_buf *new_bp, +struct xfs_buf **found_bp) +{ +struct xfs_perag *pag; +struct xfs_buf *bp; +struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn }; +int error; +int i; + + *found_bp =NULL ; + +for (i =0 ; i < nmaps; i++) + cmap.bm_len += map[i].bm_len; + + error =xfs_buf_find_verify (btp, &cmap); +if (error) +return error; + + pag =xfs_perag_get (btp->bt_mount, +xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); + +spin_lock (&pag->pag_buf_lock); + bp =xfs_buf_find_fast (pag, &cmap); +if (bp) + goto found; + + error =xfs_buf_find_insert (btp, pag, new_bp); +spin_unlock (&pag->pag_buf_lock); +if (error) +return error; + *found_bp = new_bp; +return 0 ; + +found: +spin_unlock (&pag->pag_buf_lock); +xfs_perag_put (pag); + + error =xfs_buf_find_lock (btp, bp, flags); +if (error) +return error;trace_xfs_buf_find (bp, flags, _RET_IP_);XFS_STATS_INC (btp->bt_mount, xb_get_locked);
Signature · 1 line
2.36.1
Re: [PATCH 2/6] xfs: break up xfs_buf_find() into individual pieces
Quoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfs_buf_find()is made up of three main parts: lookup, insert and locking. The interactions withxfs_buf_get_map()require it to be called twice - once for a pure lookup, and again on lookup failure so the insert path can be run. We want to simplify this down a lot, so split it into a fast path lookup, a slow path insert and a "lock the found buffer" helper. This will then let use integrate these
"let us"
Re: [PATCH 2/6] xfs: break up xfs_buf_find() into individual pieces
In reply to Dave Chinnerfs/xfs/xfs_buf.c xfs_buf_hash_destroy( · line 533
Quoted 2 lines
fs/xfs/xfs_buf.c +2
-0+static inline struct xfs_buf * +xfs_buf_find_fast(
I'd probably name this xfs_buf_lookup as the _fast suffix is a little weird.
Quoted 6 lines
fs/xfs/xfs_buf.c +6
-0+/* +* Insert the new_bp into the hash table. This consumes the perag reference +* taken for the lookup. +*/ +static int +xfs_buf_find_insert(
fs/xfs/xfs_buf.c xfs_buf_hash_destroy( · line 561
Quoted 10 lines
fs/xfs/xfs_buf.c +5
-0+struct xfs_buftarg *btp, + struct xfs_perag *pag, + struct xfs_buf *new_bp) +{ /* No match found */ if (!new_bp) {xfs_perag_put (pag ); +XFS_STATS_INC (btp->bt_mount, xb_miss_locked);return -ENOENT ; }
Having this condition here vs in the caller seems a bit odd, but I'll see what the rest of the series brings.
Re: [PATCH 2/6] xfs: break up xfs_buf_find() into individual pieces
In reply to Dave ChinnerQuoted 35 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfs_buf_find()is made up of three main parts: lookup, insert and locking. The interactions withxfs_buf_get_map()require it to be called twice - once for a pure lookup, and again on lookup failure so the insert path can be run. We want to simplify this down a lot, so split it into a fast path lookup, a slow path insert and a "lock the found buffer" helper. This will then let use integrate these operations more effectively intoxfs_buf_get_map()in future patches.Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf.c | 159 +++++++++++++++++++++++++++++++---------------- 1 file changed, 105 insertions(+), 54 deletions(-)fs/xfs/xfs_buf.c +0
-11@@ -503,77 +503,60 @@ xfs_buf_hash_destroy(rhashtable_destroy (&pag->pag_buf_hash); } -/* -* Look up a buffer in the buffer cache and return it referenced and locked -* in @found_bp. -* -* If @new_bp is supplied and we have a lookup miss, insert @new_bp into the -* cache. -* -* If XBF_TRYLOCK is set in @flags, only try to lock the buffer and return -* -EAGAIN if we fail to lock it. -* -* Return values are:
fs/xfs/xfs_buf.c xfs_buf_hash_destroy( · line 507
Quoted 10 lines
fs/xfs/xfs_buf.c +1
-8-* -EFSCORRUPTED if have been supplied with an invalid address -* -EAGAIN on trylock failure -* -ENOENT if we fail to find a match and @new_bp was NULL -* 0, with @found_bp: -* - @new_bp if we inserted it into the cache -* - the buffer we found and locked. -*/ static int -xfs_buf_find( +xfs_buf_find_verify(
Isn't this more of a xfs_buf_map verifier? Why not call it xfs_buf_map_verify()?
--D
Quoted 183 lines
fs/xfs/xfs_buf.c +104
-35struct xfs_buftarg *btp, - struct xfs_buf_map *map, - int nmaps, - xfs_buf_flags_t flags, - struct xfs_buf *new_bp, - struct xfs_buf **found_bp) + struct xfs_buf_map *map) { - struct xfs_perag *pag; -struct xfs_buf *bp; -struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn }; xfs_daddr_t eofs; -int i; - - *found_bp =NULL ; - -for (i =0 ; i < nmaps; i++) - cmap.bm_len += map[i].bm_len;/* Check for IOs smaller than the sector size / not sector aligned */ -ASSERT (!(BBTOB (cmap.bm_len) < btp->bt_meta_sectorsize)); -ASSERT (!(BBTOB (cmap.bm_bn) & (xfs_off_t )btp->bt_meta_sectormask)); +ASSERT (!(BBTOB (map->bm_len) < btp->bt_meta_sectorsize)); +ASSERT (!(BBTOB (map->bm_bn) & (xfs_off_t )btp->bt_meta_sectormask));/* * Corrupted block numbers can get through to here, unfortunately, so we * have to check that the buffer falls within the filesystem bounds. */ eofs =XFS_FSB_TO_BB (btp->bt_mount, btp->bt_mount->m_sb.sb_dblocks); -if (cmap.bm_bn <0 || cmap.bm_bn >= eofs) { +if (map->bm_bn <0 || map->bm_bn >= eofs) {xfs_alert (btp->bt_mount,"%s: daddr 0x%llx out of range, EOFS 0x%llx" , - __func__, cmap.bm_bn, eofs); + __func__, map->bm_bn, eofs);WARN_ON (1 );return -EFSCORRUPTED ; } +return 0 ; +} - pag =xfs_perag_get (btp->bt_mount, -xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); +static inline struct xfs_buf * +xfs_buf_find_fast ( +struct xfs_perag *pag, +struct xfs_buf_map *map) +{ +struct xfs_buf *bp; -spin_lock (&pag->pag_buf_lock); - bp =rhashtable_lookup_fast (&pag->pag_buf_hash, &cmap, - xfs_buf_hash_params); -if (bp) { -atomic_inc (&bp->b_hold); - goto found; - } + bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); +if (!bp) +return NULL ; +atomic_inc (&bp->b_hold); +return bp; +} +/* +* Insert the new_bp into the hash table. This consumes the perag reference +* taken for the lookup. +*/ +static int +xfs_buf_find_insert ( +struct xfs_buftarg *btp, +struct xfs_perag *pag, +struct xfs_buf *new_bp) +{/* No match found */ if (!new_bp) { -XFS_STATS_INC (btp->bt_mount, xb_miss_locked); -spin_unlock (&pag->pag_buf_lock);xfs_perag_put (pag); +XFS_STATS_INC (btp->bt_mount, xb_miss_locked);return -ENOENT ; }@@ -581,14 +564,15 @@ xfs_buf_find( new_bp->b_pag = pag;rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, xfs_buf_hash_params); -spin_unlock (&pag->pag_buf_lock); - *found_bp = new_bp;return 0 ; +} -found: -spin_unlock (&pag->pag_buf_lock); -xfs_perag_put (pag); - +static int +xfs_buf_find_lock ( +struct xfs_buftarg *btp, +struct xfs_buf *bp, +xfs_buf_flags_t flags) +{if (!xfs_buf_trylock (bp)) {if (flags &XBF_TRYLOCK ) {xfs_buf_rele (bp);@@ -609,6 +593,73 @@ xfs_buf_find( bp->b_flags &= _XBF_KMEM | _XBF_PAGES; bp->b_ops =NULL ; } +return 0 ; +} + +/* +* Look up a buffer in the buffer cache and return it referenced and locked +* in @found_bp. +* +* If @new_bp is supplied and we have a lookup miss, insert @new_bp into the +* cache. +* +* If XBF_TRYLOCK is set in @flags, only try to lock the buffer and return +* -EAGAIN if we fail to lock it. +* +* Return values are: +* -EFSCORRUPTED if have been supplied with an invalid address +* -EAGAIN on trylock failure +* -ENOENT if we fail to find a match and @new_bp was NULL +* 0, with @found_bp: +* - @new_bp if we inserted it into the cache +* - the buffer we found and locked. +*/ +static int +xfs_buf_find ( +struct xfs_buftarg *btp, +struct xfs_buf_map *map, +int nmaps, +xfs_buf_flags_t flags, +struct xfs_buf *new_bp, +struct xfs_buf **found_bp) +{ +struct xfs_perag *pag; +struct xfs_buf *bp; +struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn }; +int error; +int i; + + *found_bp =NULL ; + +for (i =0 ; i < nmaps; i++) + cmap.bm_len += map[i].bm_len; + + error =xfs_buf_find_verify (btp, &cmap); +if (error) +return error; + + pag =xfs_perag_get (btp->bt_mount, +xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); + +spin_lock (&pag->pag_buf_lock); + bp =xfs_buf_find_fast (pag, &cmap); +if (bp) + goto found; + + error =xfs_buf_find_insert (btp, pag, new_bp); +spin_unlock (&pag->pag_buf_lock); +if (error) +return error; + *found_bp = new_bp; +return 0 ; + +found: +spin_unlock (&pag->pag_buf_lock); +xfs_perag_put (pag); + + error =xfs_buf_find_lock (btp, bp, flags); +if (error) +return error;trace_xfs_buf_find (bp, flags, _RET_IP_);XFS_STATS_INC (btp->bt_mount, xb_get_locked);Signature · 1 line
2.36.1
PATCH v2 3/6 xfs: merge xfs_buf_find() and xfs_buf_get_map()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we factored xfs_buf_find(), we can start separating into distinct fast and slow paths from xfs_buf_get_map(). We start by moving the lookup map and perag setup to _get_map(), and then move all the specifics of the fast path lookup into xfs_buf_find_fast() and call it directly from _get_map(). We the move all the slow path code to xfs_buf_find_insert(), which is now also called directly from _get_map(). As such, xfs_buf_find() now goes away.
--- fs/xfs/xfs_buf.c | 233 ++++++++++++++++++++++------------------------- 1 file changed, 108 insertions(+), 125 deletions(-)
fs/xfs/xfs_buf.c +108 -125
@@ -529,58 +529,18 @@ xfs_buf_find_verify(return 0 ; } -static inline struct xfs_buf * -xfs_buf_find_fast ( -struct xfs_perag *pag, -struct xfs_buf_map *map) -{ -struct xfs_buf *bp; - - bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); -if (!bp) -return NULL ; -atomic_inc (&bp->b_hold); -return bp; -} - -/* -* Insert the new_bp into the hash table. This consumes the perag reference -* taken for the lookup. -*/ -static int -xfs_buf_find_insert ( -struct xfs_buftarg *btp, -struct xfs_perag *pag, -struct xfs_buf *new_bp) -{ -/* No match found */ -if (!new_bp) { -xfs_perag_put (pag); -XFS_STATS_INC (btp->bt_mount, xb_miss_locked); -return -ENOENT ; - } - -/* the buffer keeps the perag reference until it is freed */ - new_bp->b_pag = pag; -rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, - xfs_buf_hash_params); -return 0 ; -} -static int xfs_buf_find_lock ( -struct xfs_buftarg *btp,struct xfs_buf *bp,xfs_buf_flags_t flags) {if (!xfs_buf_trylock (bp)) {if (flags &XBF_TRYLOCK ) { -xfs_buf_rele (bp); -XFS_STATS_INC (btp->bt_mount, xb_busy_locked); +XFS_STATS_INC (bp->b_mount, xb_busy_locked);return -EAGAIN ; }xfs_buf_lock (bp); -XFS_STATS_INC (btp->bt_mount, xb_get_locked_waited); +XFS_STATS_INC (bp->b_mount, xb_get_locked_waited); }/*
@@ -596,75 +556,97 @@ xfs_buf_find_lock(return 0 ; } +static inline int +xfs_buf_find_fast ( +struct xfs_perag *pag, +struct xfs_buf_map *map, +xfs_buf_flags_t flags, +struct xfs_buf **bpp) +{ +struct xfs_buf *bp; +int error; + +spin_lock (&pag->pag_buf_lock); + bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); +if (!bp) { +spin_unlock (&pag->pag_buf_lock); +return -ENOENT ; + } +atomic_inc (&bp->b_hold); +spin_unlock (&pag->pag_buf_lock); + + error =xfs_buf_find_lock (bp, flags); +if (error) { +xfs_buf_rele (bp); +return error; + } + +trace_xfs_buf_find (bp, flags, _RET_IP_); + *bpp = bp; +return 0 ; +} +/* -* Look up a buffer in the buffer cache and return it referenced and locked -* in @found_bp. -* -* If @new_bp is supplied and we have a lookup miss, insert @new_bp into the -* cache. -* -* If XBF_TRYLOCK is set in @flags, only try to lock the buffer and return -* -EAGAIN if we fail to lock it. -* -* Return values are: -* -EFSCORRUPTED if have been supplied with an invalid address -* -EAGAIN on trylock failure -* -ENOENT if we fail to find a match and @new_bp was NULL -* 0, with @found_bp: -* - @new_bp if we inserted it into the cache -* - the buffer we found and locked. +* Insert the new_bp into the hash table. This consumes the perag reference +* taken for the lookup regardless of the result of the insert. */ static int -xfs_buf_find ( +xfs_buf_find_insert (struct xfs_buftarg *btp, +struct xfs_perag *pag, +struct xfs_buf_map *cmap,struct xfs_buf_map *map,int nmaps,xfs_buf_flags_t flags, -struct xfs_buf *new_bp, -struct xfs_buf **found_bp) +struct xfs_buf **bpp) { -struct xfs_perag *pag; +struct xfs_buf *new_bp;struct xfs_buf *bp; -struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn };int error; -int i; - - *found_bp =NULL ; -for (i =0 ; i < nmaps; i++) - cmap.bm_len += map[i].bm_len; - - error =xfs_buf_find_verify (btp, &cmap); + error =_xfs_buf_alloc (btp, map, nmaps, flags, &new_bp);if (error) -return error; + goto out_drop_pag; - pag =xfs_perag_get (btp->bt_mount, -xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); +/* +* For buffers that fit entirely within a single page, first attempt to +* allocate the memory from the heap to minimise memory usage. If we +* can't get heap memory for these small buffers, we fall back to using +* the page allocator. +*/ +if (BBTOB (new_bp->b_length) >=PAGE_SIZE || +xfs_buf_alloc_kmem (new_bp, flags) <0 ) { + error =xfs_buf_alloc_pages (new_bp, flags); +if (error) + goto out_free_buf; + }spin_lock (&pag->pag_buf_lock); - bp =xfs_buf_find_fast (pag, &cmap); -if (bp) - goto found; + bp =rhashtable_lookup (&pag->pag_buf_hash, cmap, xfs_buf_hash_params); +if (bp) { +atomic_inc (&bp->b_hold); +spin_unlock (&pag->pag_buf_lock); + error =xfs_buf_find_lock (bp, flags); +if (error) +xfs_buf_rele (bp); +else + *bpp = bp; + goto out_free_buf; + } - error =xfs_buf_find_insert (btp, pag, new_bp); +/* The buffer keeps the perag reference until it is freed. */ + new_bp->b_pag = pag; +rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, + xfs_buf_hash_params);spin_unlock (&pag->pag_buf_lock); -if (error) -return error; - *found_bp = new_bp; + *bpp = new_bp;return 0 ; -found: -spin_unlock (&pag->pag_buf_lock); +out_free_buf: +xfs_buf_free (new_bp); +out_drop_pag:xfs_perag_put (pag); - - error =xfs_buf_find_lock (btp, bp, flags); -if (error) -return error; - -trace_xfs_buf_find (bp, flags, _RET_IP_); -XFS_STATS_INC (btp->bt_mount, xb_get_locked); - *found_bp = bp; -return 0 ; +return error; }/*
@@ -674,54 +656,54 @@ xfs_buf_find(*/ int xfs_buf_get_map ( -struct xfs_buftarg *target, +struct xfs_buftarg *btp,struct xfs_buf_map *map,int nmaps,xfs_buf_flags_t flags,struct xfs_buf **bpp) { -struct xfs_buf *bp; -struct xfs_buf *new_bp; +struct xfs_perag *pag; +struct xfs_buf *bp =NULL ; +struct xfs_buf_map cmap = { .bm_bn = map[0 ].bm_bn };int error; +int i; - *bpp =NULL ; - error =xfs_buf_find (target, map, nmaps, flags,NULL , &bp); -if (!error) - goto found; -if (error != -ENOENT ) -return error; -if (flags &XBF_INCORE ) -return -ENOENT ; +for (i =0 ; i < nmaps; i++) + cmap.bm_len += map[i].bm_len; - error =_xfs_buf_alloc (target, map, nmaps, flags, &new_bp); + error =xfs_buf_find_verify (btp, &cmap);if (error)return error; -/* -* For buffers that fit entirely within a single page, first attempt to -* allocate the memory from the heap to minimise memory usage. If we -* can't get heap memory for these small buffers, we fall back to using -* the page allocator. -*/ -if (BBTOB (new_bp->b_length) >=PAGE_SIZE || -xfs_buf_alloc_kmem (new_bp, flags) <0 ) { - error =xfs_buf_alloc_pages (new_bp, flags); -if (error) - goto out_free_buf; - } + pag =xfs_perag_get (btp->bt_mount, +xfs_daddr_to_agno (btp->bt_mount, cmap.bm_bn)); - error =xfs_buf_find (target, map, nmaps, flags, new_bp, &bp); -if (error) - goto out_free_buf; + error =xfs_buf_find_fast (pag, &cmap, flags, &bp); +if (error && error != -ENOENT ) + goto out_put_perag; -if (bp != new_bp) -xfs_buf_free (new_bp); +/* cache hits always outnumber misses by at least 10:1 */ +if (unlikely (!bp)) { +XFS_STATS_INC (btp->bt_mount, xb_miss_locked); -found: +if (flags &XBF_INCORE ) + goto out_put_perag; + +/* xfs_buf_find_insert() consumes the perag reference. */ + error =xfs_buf_find_insert (btp, pag, &cmap, map, nmaps, + flags, &bp); +if (error) +return error; + }else { +XFS_STATS_INC (btp->bt_mount, xb_get_locked); +xfs_perag_put (pag); + } + +/* We do not hold a perag reference anymore. */ if (!bp->b_addr) { error =_xfs_buf_map_pages (bp, flags);if (unlikely (error)) { -xfs_warn_ratelimited (target->bt_mount, +xfs_warn_ratelimited (btp->bt_mount,"%s: failed to map %u pages" , __func__, bp->b_page_count);xfs_buf_relse (bp);
@@ -736,12 +718,13 @@ xfs_buf_get_map(if (!(flags &XBF_READ ))xfs_buf_ioerror (bp,0 ); -XFS_STATS_INC (target->bt_mount, xb_get); +XFS_STATS_INC (btp->bt_mount, xb_get);trace_xfs_buf_get (bp, flags, _RET_IP_); *bpp = bp;return 0 ; -out_free_buf: -xfs_buf_free (new_bp); + +out_put_perag: +xfs_perag_put (pag);return error; }
Signature · 1 line
2.36.1
Re: [PATCH 3/6] xfs: merge xfs_buf_find() and xfs_buf_get_map()
Quoted 9 lines
fs/xfs/xfs_buf.c +0
-8-static inline struct xfs_buf * -xfs_buf_find_fast ( -struct xfs_perag *pag, -struct xfs_buf_map *map) -{ -struct xfs_buf *bp; - - bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params);
fs/xfs/xfs_buf.c xfs_buf_find_verify(
Quoted 10 lines
fs/xfs/xfs_buf.c +0
-9-if (!bp) -return NULL ; -atomic_inc (&bp->b_hold); -return bp; -} -static int -xfs_buf_find_insert( - struct xfs_buftarg *btp, - struct xfs_perag *pag,
Adding the function just in the last patch and moving them around here and slighty changing seems a little counter productive. I think just merging the two might actually end up with a result that is easier to review.
Re: [PATCH 3/6] xfs: merge xfs_buf_find() and xfs_buf_get_map()
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines
fs/xfs/xfs_buf.c +0
-13-static inline struct xfs_buf * -xfs_buf_find_fast ( -struct xfs_perag *pag, -struct xfs_buf_map *map) -{ -struct xfs_buf *bp; - - bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); -if (!bp) -return NULL ; -atomic_inc (&bp->b_hold); -return bp; -}
Quoted 9 lines
fs/xfs/xfs_buf.c xfs_buf_find_verify( · line 553
Quoted 4 lines
fs/xfs/xfs_buf.c +0
-4-static int -xfs_buf_find_insert( - struct xfs_buftarg *btp, - struct xfs_perag *pag,Adding the function just in the last patch and moving them around here and slighty changing seems a little counter productive. I think just merging the two might actually end up with a result that is easier to review.
I read the second patch and it makes sense, but I'm also curious if hch's suggestion here would make this change easier to read?
--D
Re: [PATCH 3/6] xfs: merge xfs_buf_find() and xfs_buf_get_map()
Quoted 26 lines · Darrick J. Wong
Quoted 15 lines · Christoph Hellwig
Quoted 14 lines
fs/xfs/xfs_buf.c +0
-13-static inline struct xfs_buf * -xfs_buf_find_fast ( -struct xfs_perag *pag, -struct xfs_buf_map *map) -{ -struct xfs_buf *bp; - - bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); -if (!bp) -return NULL ; -atomic_inc (&bp->b_hold); -return bp; -}Quoted 9 lines
fs/xfs/xfs_buf.c xfs_buf_find_verify( · line 553
Quoted 4 lines
fs/xfs/xfs_buf.c +0
-4-static int -xfs_buf_find_insert( - struct xfs_buftarg *btp, - struct xfs_perag *pag,Adding the function just in the last patch and moving them around here and slighty changing seems a little counter productive. I think just merging the two might actually end up with a result that is easier to review.
Quoted 2 lines
I read the second patch and it makes sense, but I'm also curious if hch's suggestion here would make this change easier to read?
I moved the initial placement of these functions around and it took a big chunk out of the diff in this patch. That should make it easier to read without combining the two patches together...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 4/6 xfs: reduce the number of atomic when locking a buffer after lookup
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Avoid an extra atomic operation in the non-trylock case by only doing a trylock if the XBF_TRYLOCK flag is set. This follows the pattern in the IO path with NOWAIT semantics where the "trylock-fail-lock" path showed 5-10% reduced throughput compared to just using single lock call when not under NOWAIT conditions. So make that same change here, too.
See commit 942491c9e6d6 ("xfs: fix AIM7 regression") for details.
--- fs/xfs/xfs_buf.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/xfs/xfs_buf.c +3 -2
@@ -534,11 +534,12 @@ xfs_buf_find_lock( struct xfs_buf *bp, xfs_buf_flags_t flags) { -if (!xfs_buf_trylock (bp)) { -if (flags &XBF_TRYLOCK ) { +if (flags &XBF_TRYLOCK ) { +if (!xfs_buf_trylock (bp)) {XFS_STATS_INC (bp->b_mount, xb_busy_locked);return -EAGAIN ; } + }else {xfs_buf_lock (bp);XFS_STATS_INC (bp->b_mount, xb_get_locked_waited); }
Signature · 1 line
2.36.1
Re: [PATCH 4/6] xfs: reduce the number of atomic when locking a buffer after lookup
Quoted 9 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Avoid an extra atomic operation in the non-trylock case by only doing a trylock if the
XBF_TRYLOCKflag is set. This follows the pattern in the IO path with NOWAIT semantics where the "trylock-fail-lock" path showed 5-10% reduced throughput compared to just using single lock call when not under NOWAIT conditions. So make that same change here, too.
Quoted 5 lines
See commit
942491c9e6d6("xfs: fix AIM7 regression") for details.Signed-off-by Dave Chinner <[email protected]>[hch: split from a larger patch]Signed-off-by Christoph Hellwig <[email protected]>
LGTM
--D
Quoted 26 lines
--- fs/xfs/xfs_buf.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/xfs/xfs_buf.c +3
-2@@ -534,11 +534,12 @@ xfs_buf_find_lock( struct xfs_buf *bp, xfs_buf_flags_t flags) { -if (!xfs_buf_trylock (bp)) { -if (flags &XBF_TRYLOCK ) { +if (flags &XBF_TRYLOCK ) { +if (!xfs_buf_trylock (bp)) {XFS_STATS_INC (bp->b_mount, xb_busy_locked);return -EAGAIN ; } + }else {xfs_buf_lock (bp);XFS_STATS_INC (bp->b_mount, xb_get_locked_waited); }Signature · 1 line
2.36.1
PATCH v2 5/6 xfs: remove a superflous hash lookup when inserting new buffers
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Currently on the slow path insert we repeat the initial hash table lookup before we attempt the insert, resulting in a two traversals of the hash table to ensure the insert is valid. The rhashtable API provides a method for an atomic lookup and insert operation, so we can avoid one of the hash table traversals by using this method.
Adapted from a large patch containing this optimisation by Christoph Hellwig.
--- fs/xfs/xfs_buf.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
fs/xfs/xfs_buf.c +9 -4
@@ -623,8 +623,15 @@ xfs_buf_find_insert( }spin_lock (&pag->pag_buf_lock); - bp =rhashtable_lookup (&pag->pag_buf_hash, cmap, xfs_buf_hash_params); + bp =rhashtable_lookup_get_insert_fast (&pag->pag_buf_hash, + &new_bp->b_rhash_head, xfs_buf_hash_params); +if (IS_ERR (bp)) { + error =PTR_ERR (bp); +spin_unlock (&pag->pag_buf_lock); + goto out_free_buf; + }if (bp) { +/* found an existing buffer */ atomic_inc (&bp->b_hold);spin_unlock (&pag->pag_buf_lock); error =xfs_buf_find_lock (bp, flags);
@@ -635,10 +642,8 @@ xfs_buf_find_insert( goto out_free_buf; } -/* The buffer keeps the perag reference until it is freed. */ +/* The new buffer keeps the perag reference until it is freed. */ new_bp->b_pag = pag; -rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, - xfs_buf_hash_params);spin_unlock (&pag->pag_buf_lock); *bpp = new_bp;return 0 ;
Signature · 1 line
2.36.1
Re: [PATCH 5/6] xfs: remove a superflous hash lookup when inserting new buffers
Looks good:
Re: [PATCH 5/6] xfs: remove a superflous hash lookup when inserting new buffers
In reply to Dave ChinnerQuoted 8 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Currently on the slow path insert we repeat the initial hash table lookup before we attempt the insert, resulting in a two traversals of the hash table to ensure the insert is valid. The rhashtable API provides a method for an atomic lookup and insert operation, so we can avoid one of the hash table traversals by using this method.
Quoted 4 lines
Adapted from a large patch containing this optimisation by Christoph Hellwig.
Signed-off-by Dave Chinner <[email protected]>
That's pretty neat!
--D
Quoted 40 lines
--- fs/xfs/xfs_buf.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)fs/xfs/xfs_buf.c +9
-4@@ -623,8 +623,15 @@ xfs_buf_find_insert( }spin_lock (&pag->pag_buf_lock); - bp =rhashtable_lookup (&pag->pag_buf_hash, cmap, xfs_buf_hash_params); + bp =rhashtable_lookup_get_insert_fast (&pag->pag_buf_hash, + &new_bp->b_rhash_head, xfs_buf_hash_params); +if (IS_ERR (bp)) { + error =PTR_ERR (bp); +spin_unlock (&pag->pag_buf_lock); + goto out_free_buf; + }if (bp) { +/* found an existing buffer */ atomic_inc (&bp->b_hold);spin_unlock (&pag->pag_buf_lock); error =xfs_buf_find_lock (bp, flags);@@ -635,10 +642,8 @@ xfs_buf_find_insert( goto out_free_buf; } -/* The buffer keeps the perag reference until it is freed. */ +/* The new buffer keeps the perag reference until it is freed. */ new_bp->b_pag = pag; -rhashtable_insert_fast (&pag->pag_buf_hash, &new_bp->b_rhash_head, - xfs_buf_hash_params);spin_unlock (&pag->pag_buf_lock); *bpp = new_bp;return 0 ;Signature · 1 line
2.36.1
PATCH v2 6/6 xfs: lockless buffer lookup
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we have a standalone fast path for buffer lookup, we can easily convert it to use rcu lookups. When we continually hammer the buffer cache with trylock lookups, we end up with a huge amount of lock contention on the per-ag buffer hash locks:
- 92.71% 0.05% [kernel] [k] xfs_inodegc_worker
- 92.67% xfs_inodegc_worker
- 92.13% xfs_inode_unlink
- 91.52% xfs_inactive_ifree
- 85.63% xfs_read_agi
- 85.61% xfs_trans_read_buf_map
- 85.59% xfs_buf_read_map
- xfs_buf_get_map
- 85.55% xfs_buf_find
- 72.87% _raw_spin_lock
- do_raw_spin_lock
71.86% __pv_queued_spin_lock_slowpath
- 8.74% xfs_buf_rele
- 7.88% _raw_spin_lock
- 7.88% do_raw_spin_lock 7.63% __pv_queued_spin_lock_slowpath
- 1.70% xfs_buf_trylock
- 1.68% down_trylock
- 1.41% _raw_spin_lock_irqsave
- 1.39% do_raw_spin_lock __pv_queued_spin_lock_slowpath
- 0.76% _raw_spin_unlock
0.75% do_raw_spin_unlock
This is basically hammering the pag->pag_buf_lock from lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.
We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by converting the lookup fast path to leverages the rhashtable's ability to do rcu protected lookups.
We avoid races with the buffer release path by using atomic_inc_not_zero() on the buffer hold count. Any buffer that is in the LRU will have a non-zero count, thereby allowing the lockless fast path to be taken in most cache hit situations. If the buffer hold count is zero, then it is likely going through the release path so in that case we fall back to the existing lookup miss slow path.
The slow path will then do an atomic lookup and insert under the buffer hash lock and hence serialise correctly against buffer release freeing the buffer.
The use of rcu protected lookups means that buffer handles now need to be freed by RCU callbacks (same as inodes). We still free the buffer pages before the RCU callback - we won't be trying to access them at all on a buffer that has zero references - but we need the buffer handle itself to be present for the entire rcu protected read side to detect a zero hold count correctly.
--- fs/xfs/xfs_buf.c | 22 +++++++++++++++------- fs/xfs/xfs_buf.h | 1 + 2 files changed, 16 insertions(+), 7 deletions(-)
fs/xfs/xfs_buf.c +15 -7
@@ -294,6 +294,16 @@ xfs_buf_free_pages( bp->b_flags &= ~_XBF_PAGES; } +static void +xfs_buf_free_callback ( +struct callback_head *cb) +{ + struct xfs_buf *bp = container_of(cb, struct xfs_buf, b_rcu); + +xfs_buf_free_maps (bp); +kmem_cache_free (xfs_buf_cache, bp); +} +static void xfs_buf_free (struct xfs_buf *bp)
@@ -307,8 +317,7 @@ xfs_buf_free(else if (bp->b_flags & _XBF_KMEM)kmem_free (bp->b_addr); -xfs_buf_free_maps (bp); -kmem_cache_free (xfs_buf_cache, bp); +call_rcu (&bp->b_rcu, xfs_buf_free_callback); } static int
@@ -567,14 +576,13 @@ xfs_buf_find_fast(struct xfs_buf *bp;int error; -spin_lock (&pag->pag_buf_lock); +rcu_read_lock (); bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); -if (!bp) { -spin_unlock (&pag->pag_buf_lock); +if (!bp || !atomic_inc_not_zero (&bp->b_hold)) { +rcu_read_unlock ();return -ENOENT ; } -atomic_inc (&bp->b_hold); -spin_unlock (&pag->pag_buf_lock); +rcu_read_unlock (); error =xfs_buf_find_lock (bp, flags);if (error) {
fs/xfs/xfs_buf.h +1 -0
@@ -196,6 +196,7 @@ struct xfs_buf {int b_last_error;const struct xfs_buf_ops *b_ops; +struct rcu_head b_rcu; };/* Finding and Reading Buffers */
Signature · 1 line
2.36.1
Re: [PATCH 6/6] xfs: lockless buffer lookup
Looks good:
Re: [PATCH 6/6] xfs: lockless buffer lookup
In reply to Dave ChinnerQuoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have a standalone fast path for buffer lookup, we can easily convert it to use rcu lookups. When we continually hammer the buffer cache with trylock lookups, we end up with a huge amount of lock contention on the per-ag buffer hash locks:
- 92.71% 0.05% [kernel] [k]
xfs_inodegc_worker
- 92.67%xfs_inodegc_worker
- 92.13%xfs_inode_unlink
- 91.52%xfs_inactive_ifree
- 85.63%xfs_read_agi
- 85.61%xfs_trans_read_buf_map
- 85.59%xfs_buf_read_map
-xfs_buf_get_map
- 85.55%xfs_buf_find
- 72.87%_raw_spin_lock
-do_raw_spin_lock
71.86%__pv_queued_spin_lock_slowpath
- 8.74%xfs_buf_rele
- 7.88%_raw_spin_lock
- 7.88%do_raw_spin_lock7.63%__pv_queued_spin_lock_slowpath
- 1.70%xfs_buf_trylock
- 1.68%down_trylock
- 1.41%_raw_spin_lock_irqsave
- 1.39%do_raw_spin_lock__pv_queued_spin_lock_slowpath
- 0.76%_raw_spin_unlock
0.75%do_raw_spin_unlockThis is basically hammering the
pag->pag_buf_lockfrom lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by converting the lookup fast path to leverages the rhashtable's ability to do rcu protected lookups.
We avoid races with the buffer release path by using
atomic_inc_not_zero()on the buffer hold count. Any buffer that is in the LRU will have a non-zero count, thereby allowing the lockless fast path to be taken in most cache hit situations. If the buffer hold count is zero, then it is likely going through the release path so in that case we fall back to the existing lookup miss slow path.The slow path will then do an atomic lookup and insert under the buffer hash lock and hence serialise correctly against buffer release freeing the buffer.
Quoted 6 lines
The use of rcu protected lookups means that buffer handles now need to be freed by RCU callbacks (same as inodes). We still free the buffer pages before the RCU callback - we won't be trying to access them at all on a buffer that has zero references - but we need the buffer handle itself to be present for the entire rcu protected read side to detect a zero hold count correctly.
Hmm, so what still uses pag_buf_lock? Are we still using it to serialize xfs_buf_rele calls?
--D
Quoted 71 lines
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf.c | 22 +++++++++++++++------- fs/xfs/xfs_buf.h | 1 + 2 files changed, 16 insertions(+), 7 deletions(-)fs/xfs/xfs_buf.c +15
-7@@ -294,6 +294,16 @@ xfs_buf_free_pages( bp->b_flags &= ~_XBF_PAGES; } +static void +xfs_buf_free_callback ( +struct callback_head *cb) +{ + struct xfs_buf *bp = container_of(cb, struct xfs_buf, b_rcu); + +xfs_buf_free_maps (bp); +kmem_cache_free (xfs_buf_cache, bp); +} +static void xfs_buf_free (struct xfs_buf *bp)@@ -307,8 +317,7 @@ xfs_buf_free(else if (bp->b_flags & _XBF_KMEM)kmem_free (bp->b_addr); -xfs_buf_free_maps (bp); -kmem_cache_free (xfs_buf_cache, bp); +call_rcu (&bp->b_rcu, xfs_buf_free_callback); } static int@@ -567,14 +576,13 @@ xfs_buf_find_fast(struct xfs_buf *bp;int error; -spin_lock (&pag->pag_buf_lock); +rcu_read_lock (); bp =rhashtable_lookup (&pag->pag_buf_hash, map, xfs_buf_hash_params); -if (!bp) { -spin_unlock (&pag->pag_buf_lock); +if (!bp || !atomic_inc_not_zero (&bp->b_hold)) { +rcu_read_unlock ();return -ENOENT ; } -atomic_inc (&bp->b_hold); -spin_unlock (&pag->pag_buf_lock); +rcu_read_unlock (); error =xfs_buf_find_lock (bp, flags);if (error) {fs/xfs/xfs_buf.h +1
-0@@ -196,6 +196,7 @@ struct xfs_buf {int b_last_error;const struct xfs_buf_ops *b_ops; +struct rcu_head b_rcu; };/* Finding and Reading Buffers */ Signature · 1 line
2.36.1
Re: [PATCH 6/6] xfs: lockless buffer lookup
Quoted 61 lines · Darrick J. Wong
Quoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have a standalone fast path for buffer lookup, we can easily convert it to use rcu lookups. When we continually hammer the buffer cache with trylock lookups, we end up with a huge amount of lock contention on the per-ag buffer hash locks:
- 92.71% 0.05% [kernel] [k]
xfs_inodegc_worker
- 92.67%xfs_inodegc_worker
- 92.13%xfs_inode_unlink
- 91.52%xfs_inactive_ifree
- 85.63%xfs_read_agi
- 85.61%xfs_trans_read_buf_map
- 85.59%xfs_buf_read_map
-xfs_buf_get_map
- 85.55%xfs_buf_find
- 72.87%_raw_spin_lock
-do_raw_spin_lock
71.86%__pv_queued_spin_lock_slowpath
- 8.74%xfs_buf_rele
- 7.88%_raw_spin_lock
- 7.88%do_raw_spin_lock7.63%__pv_queued_spin_lock_slowpath
- 1.70%xfs_buf_trylock
- 1.68%down_trylock
- 1.41%_raw_spin_lock_irqsave
- 1.39%do_raw_spin_lock__pv_queued_spin_lock_slowpath
- 0.76%_raw_spin_unlock
0.75%do_raw_spin_unlockThis is basically hammering the
pag->pag_buf_lockfrom lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by converting the lookup fast path to leverages the rhashtable's ability to do rcu protected lookups.
We avoid races with the buffer release path by using
atomic_inc_not_zero()on the buffer hold count. Any buffer that is in the LRU will have a non-zero count, thereby allowing the lockless fast path to be taken in most cache hit situations. If the buffer hold count is zero, then it is likely going through the release path so in that case we fall back to the existing lookup miss slow path.The slow path will then do an atomic lookup and insert under the buffer hash lock and hence serialise correctly against buffer release freeing the buffer.
Quoted 6 lines
The use of rcu protected lookups means that buffer handles now need to be freed by RCU callbacks (same as inodes). We still free the buffer pages before the RCU callback - we won't be trying to access them at all on a buffer that has zero references - but we need the buffer handle itself to be present for the entire rcu protected read side to detect a zero hold count correctly.
Quoted 2 lines
Hmm, so what still uses
pag_buf_lock? Are we still using it to serializexfs_buf_relecalls?
slow path lookup/insert and xfs_buf_rele calls.
The only thing we are allowing lockless lookups on are buffers with at least one reference. With the LRU holding a reference, that means it the buffer is still actively cached or referenced by something else so won't disappear from under us. If the ref count is zero, then it means the buffer is being removed from the cache, so we need to go the slow way and take the pag_buf_lock to serialise the lookup against the release of the unreferenced buffer we found in the cache.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Darrick J. Wong
Reviewed-byRe: [PATCH 6/6] xfs: lockless buffer lookup
Quoted 67 lines · Dave Chinner
Quoted 61 lines · Darrick J. Wong
Quoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have a standalone fast path for buffer lookup, we can easily convert it to use rcu lookups. When we continually hammer the buffer cache with trylock lookups, we end up with a huge amount of lock contention on the per-ag buffer hash locks:
- 92.71% 0.05% [kernel] [k]
xfs_inodegc_worker
- 92.67%xfs_inodegc_worker
- 92.13%xfs_inode_unlink
- 91.52%xfs_inactive_ifree
- 85.63%xfs_read_agi
- 85.61%xfs_trans_read_buf_map
- 85.59%xfs_buf_read_map
-xfs_buf_get_map
- 85.55%xfs_buf_find
- 72.87%_raw_spin_lock
-do_raw_spin_lock
71.86%__pv_queued_spin_lock_slowpath
- 8.74%xfs_buf_rele
- 7.88%_raw_spin_lock
- 7.88%do_raw_spin_lock7.63%__pv_queued_spin_lock_slowpath
- 1.70%xfs_buf_trylock
- 1.68%down_trylock
- 1.41%_raw_spin_lock_irqsave
- 1.39%do_raw_spin_lock__pv_queued_spin_lock_slowpath
- 0.76%_raw_spin_unlock
0.75%do_raw_spin_unlockThis is basically hammering the
pag->pag_buf_lockfrom lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by converting the lookup fast path to leverages the rhashtable's ability to do rcu protected lookups.
We avoid races with the buffer release path by using
atomic_inc_not_zero()on the buffer hold count. Any buffer that is in the LRU will have a non-zero count, thereby allowing the lockless fast path to be taken in most cache hit situations. If the buffer hold count is zero, then it is likely going through the release path so in that case we fall back to the existing lookup miss slow path.The slow path will then do an atomic lookup and insert under the buffer hash lock and hence serialise correctly against buffer release freeing the buffer.
Quoted 6 lines
The use of rcu protected lookups means that buffer handles now need to be freed by RCU callbacks (same as inodes). We still free the buffer pages before the RCU callback - we won't be trying to access them at all on a buffer that has zero references - but we need the buffer handle itself to be present for the entire rcu protected read side to detect a zero hold count correctly.
Quoted 2 lines
Hmm, so what still uses
pag_buf_lock? Are we still using it to serializexfs_buf_relecalls?slow path lookup/insert and
xfs_buf_relecalls.
Quoted 8 lines
The only thing we are allowing lockless lookups on are buffers with at least one reference. With the LRU holding a reference, that means it the buffer is still actively cached or referenced by something else so won't disappear from under us. If the ref count is zero, then it means the buffer is being removed from the cache, so we need to go the slow way and take the
pag_buf_lockto serialise the lookup against the release of the unreferenced buffer we found in the cache.
Ah, right, got it.
--D
Quoted 6 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 6/6] xfs: lockless buffer lookup
In reply to Dave ChinnerQuoted 5 lines · Dave Chinner
Quoted 3 lines
Hmm, so what still uses
pag_buf_lock? Are we still using it to serializexfs_buf_relecalls?slow path lookup/insert and
xfs_buf_relecalls.
Note that we might be able to eventually remove the locking in rele if we switch the lookup slow path to use atomic_inc_unless_zero as well. But it would need a very careful audit.
Quoted 59 lines · Dave Chinner
Hi folks,
Current work to merge the XFS inode life cycle with the VFS indoe life cycle is finding some interesting issues. If we have a path that hits buffer trylocks fairly hard (e.g. a non-blocking background inode freeing function), we end up hitting massive contention on the buffer cache hash locks:
- 92.71% 0.05% [kernel] [k]
xfs_inodegc_worker
- 92.67%xfs_inodegc_worker
- 92.13%xfs_inode_unlink
- 91.52%xfs_inactive_ifree
- 85.63%xfs_read_agi
- 85.61%xfs_trans_read_buf_map
- 85.59%xfs_buf_read_map
-xfs_buf_get_map
- 85.55%xfs_buf_find
- 72.87%_raw_spin_lock
-do_raw_spin_lock
71.86%__pv_queued_spin_lock_slowpath
- 8.74%xfs_buf_rele
- 7.88%_raw_spin_lock
- 7.88%do_raw_spin_lock7.63%__pv_queued_spin_lock_slowpath
- 1.70%xfs_buf_trylock
- 1.68%down_trylock
- 1.41%_raw_spin_lock_irqsave
- 1.39%do_raw_spin_lock__pv_queued_spin_lock_slowpath
- 0.76%_raw_spin_unlock
0.75%do_raw_spin_unlockThis is basically hammering the
pag->pag_buf_lockfrom lots of CPUs doing trylocks at the same time. Most of the buffer trylock operations ultimately fail after we've done the lookup, so we're really hammering the buf hash lock whilst making no progress.We can also see significant spinlock traffic on the same lock just under normal operation when lots of tasks are accessing metadata from the same AG, so let's avoid all this by creating a lookup fast path which leverages the rhashtable's ability to do rcu protected lookups.
This is a rework of the initial lockless buffer lookup patch I sent here:
https://lore.kernel.org/linux-xfs/[email protected]/
And the alternative cleanup sent by Christoph here:
https://lore.kernel.org/linux-xfs/[email protected]/
This version isn't quite a short as Christophs, but it does roughly the same thing in killing the two-phase
_xfs_buf_find()call mechanism. It separates the fast and slow paths a little more cleanly and doesn't have context dependent buffer return state from the slow path that the caller needs to handle. It also picks up the rhashtable insert optimisation that Christoph added.
Quoted 6 lines
This series passes fstests under several different configs and does not cause any obvious regressions in scalability testing that has been performed. Hence I'm proposing this as potential 5.20 cycle material.
Thoughts, comments?
Any chance there'll be a v3 (or just responses to the replies sent so far) in time for 5.20?
--D
Quoted 9 lines
Version 2:
- based on 5.19-rc2
- high speed collision of original proposals.Initial versions:
- https://lore.kernel.org/linux-xfs/[email protected]/
- https://lore.kernel.org/linux-xfs/[email protected]/