xfs: Improve CIL scalability
Subsystems (2)
cover xfs: Improve CIL scalability
This series aims to improve the scalability of XFS transaction commits on large CPU count machines. My 32p machine hits contention limits in xlog_cil_commit() at about 700,000 transaction commits a section. It hits this at 16 thread workloads, and 32 thread workloads go no faster and just burn CPU on the CIL spinlocks.
This patchset gets rid of spinlocks and global serialisation points in the xlog_cil_commit() path. It does this by moving to a combination of per-cpu counters, unordered per-cpu lists and post-ordered per-cpu lists.
This results in transaction commit rates exceeding 1.6 million commits/s under unlink certain workloads, and while the log lock contention is largely gone there is still significant lock contention at the VFS at 600,000 transactions/s:
19.39% [kernel] [k] __pv_queued_spin_lock_slowpath 6.40% [kernel] [k] do_raw_spin_lock 4.07% [kernel] [k] __raw_callee_save___pv_queued_spin_unlock 3.08% [kernel] [k] memcpy_erms 1.93% [kernel] [k] xfs_buf_find 1.69% [kernel] [k] xlog_cil_commit 1.50% [kernel] [k] syscall_exit_to_user_mode 1.18% [kernel] [k] memset_erms
- 64.23% 0.22% [kernel] [k] path_openat
- 64.01% path_openat
- 48.69% xfs_vn_create
- 48.60% xfs_generic_create
- 40.96% xfs_create
- 20.39% xfs_dir_ialloc
- 7.05% xfs_setup_inode
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- 6.87%
inode_sb_list_add
- 6.54% _raw_spin_lock
- 6.53% do_raw_spin_lock
6.08% __pv_queued_spin_lock_slowpath
.....
- 11.27% xfs_trans_commit
- 11.23% __xfs_trans_commit
- 10.85% xlog_cil_commit
2.47% memcpy_erms
- 1.77% xfs_buf_item_committing
- 1.70% xfs_buf_item_release
- 0.79% xfs_buf_unlock 0.68% up
0.61% xfs_buf_rele
0.80% xfs_buf_item_format
0.73% xfs_inode_item_format 0.68% xfs_buf_item_size
- 0.55% kmem_alloc_large
- 0.55% kmem_alloc
0.52% __kmalloc
.....
- 7.08% d_instantiate
- 6.66% security_d_instantiate
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- 6.63%
selinux_d_instantiate
- 6.48% inode_doinit_with_dentry
- 6.11% _raw_spin_lock
- 6.09% do_raw_spin_lock
5.60% __pv_queued_spin_lock_slowpath
....
- 1.77% terminate_walk
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- 1.69% dput
- 1.55% _raw_spin_lock
- do_raw_spin_lock
1.19% __pv_queued_spin_lock_slowpath
But when we extend out to 1.5M commits/s we see that the contention starts to shift to the atomics in the lockless log reservation path:
14.81% [kernel] [k] __pv_queued_spin_lock_slowpath 7.88% [kernel] [k] xlog_grant_add_space 7.18% [kernel] [k] xfs_log_ticket_ungrant 4.82% [kernel] [k] do_raw_spin_lock 3.58% [kernel] [k] xlog_space_left 3.51% [kernel] [k] xlog_cil_commit
There's still substantial spin lock contention occurring at the VFS, too, but it's indicating that multiple atomic variable updates per transaction reservation/commit pair is starting to reach scalability limits here.
This is largely a re-implementation of a past RFC patchsets. While that were good enough proof of concept to perf test, they did not preserve transaction order correctly and failed shutdown tests all the time. The changes to the CIL accounting and behaviour, combined with the structural changes to xlog_write() in prior patchsets make the per-cpu restructuring possible and sane.
Instead of trying to account for continuation log opheaders on a "growth" basis, we pre-calculate how many iclogs we'll need to write out a maximally sized CIL checkpoint and just reserve that space one per commit until the CIL has a full reservation. If we ever run a commit when we are already at the hard limit (because post-throttling) we simply take an extra reservation from each commit that is run when over the limit. Hence we don't need to do space usage math in the fast path and so never need to sum the per-cpu counters in this path.
Similarly, per-cpu lists have the problem of ordering - we can't remove an item from a per-cpu list if we want to move it forward in the CIL. We solve this problem by using an atomic counter to give every commit a sequence number that is copied into the log items in that transaction. Hence relogging items just overwrites the sequence number in the log item, and does not move it in the per-cpu lists. Once we reaggregate the per-cpu lists back into a single list in the CIL push work, we can run it through list-sort() and reorder it back into a globally ordered list. This costs a bit of CPU time, but now that the CIL can run multiple works and pipelines properly, this is not a limiting factor for performance. It does increase fsync latency when the CIL is full, but workloads issuing large numbers of fsync()s or sync transactions end up with very small CILs and so the latency impact or sorting is not measurable for such workloads.
This seems to be solid so far. It passes fstests on top of all the patchsets that it is built on and I've posted in the past couple of days. The performance is good, but largely limited now by VFS lock contention issues that are outside the scope of this work.
Thoughts, comments?
-Dave.
Dave Chinner (12):
xfs: use the CIL space used counter for emptiness checks
xfs: lift init CIL reservation out of xc_cil_lock
xfs: rework per-iclog header CIL reservation
xfs: introduce per-cpu CIL tracking sructure
xfs: implement percpu cil space used calculation
xfs: track CIL ticket reservation in percpu structure
xfs: convert CIL busy extents to per-cpu
xfs: Add order IDs to log items in CIL
xfs: convert CIL to unordered per cpu lists
xfs: move CIL ordering to the logvec chain
xfs: __percpu_counter_compare() inode count debug too expensive
xfs: avoid cil push lock if possible
fs/xfs/libxfs/xfs_log_rlimit.c | 2 +- fs/xfs/libxfs/xfs_shared.h | 3 +- fs/xfs/xfs_log.c | 55 +++-- fs/xfs/xfs_log.h | 3 +- fs/xfs/xfs_log_cil.c | 388 +++++++++++++++++++++++++-------- fs/xfs/xfs_log_priv.h | 48 ++-- fs/xfs/xfs_trans.c | 15 +- fs/xfs/xfs_trans.h | 1 + fs/xfs/xfs_trans_priv.h | 4 +- include/linux/cpuhotplug.h | 1 + 10 files changed, 380 insertions(+), 140 deletions(-)
Signature · 1 line
2.28.0
PATCH 1/12 xfs: use the CIL space used counter for emptiness checks
From: Dave Chinner <[email protected]>
In the next patches we are going to make the CIL list itself per-cpu, and so we cannot use list_empty() to check is the list is empty. Replace the list_empty() checks with a flag in the CIL to indicate we have committed at least one transaction to the CIL and hence the CIL is not empty.
We need this flag to be an atomic so that we can clear it without holding any locks in the commit fast path, but we also need to be careful to avoid atomic operations in the fast path. Hence we use the fact that test_bit() is not an atomic op to first check if the flag is set and then run the atomic test_and_clear_bit() operation to clear it and steal the initial unit reservation for the CIL context checkpoint.
When we are switching to a new context in a push, we place the setting of the XLOG_CIL_EMPTY flag under the xc_push_lock. THis allows all the other places that need to check whether the CIL is empty to use test_bit() and still be serialised correctly with the CIL context swaps that set the bit.
--- fs/xfs/xfs_log_cil.c | 49 +++++++++++++++++++++++-------------------- fs/xfs/xfs_log_priv.h | 4 ++++ 2 files changed, 30 insertions(+), 23 deletions(-)
fs/xfs/xfs_log_cil.c +26 -23
@@ -70,6 +70,7 @@ xlog_cil_ctx_switch(struct xfs_cil *cil, struct xfs_cil_ctx *ctx) { + set_bit(XLOG_CIL_EMPTY, &cil->xc_flags); ctx->sequence = ++cil->xc_current_sequence; ctx->cil = cil; cil->xc_ctx = ctx;
@@ -436,13 +437,12 @@ xlog_cil_insert_items(list_splice_init (&tp->t_busy, &ctx->busy_extents);/* -* Now transfer enough transaction reservation to the context ticket -* for the checkpoint. The context ticket is special - the unit -* reservation has to grow as well as the current reservation as we -* steal from tickets so we can correctly determine the space used -* during the transaction commit. +* We need to take the CIL checkpoint unit reservation on the first +* commit into the CIL. Test the XLOG_CIL_EMPTY bit first so we don't +* unnecessarily do an atomic op in the fast path here. */ -if (ctx->ticket->t_curr_res ==0 ) { +if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags) && +test_and_clear_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) { ctx_res = ctx->ticket->t_unit_res; ctx->ticket->t_curr_res = ctx_res; tp->t_ticket->t_curr_res -= ctx_res;
@@ -770,7 +770,7 @@ xlog_cil_push_work(* move on to a new sequence number and so we have to be able to push * this sequence again later. */ -if (list_empty (&cil->xc_cil)) { +if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) { cil->xc_push_seq =0 ;spin_unlock (&cil->xc_push_lock); goto out_skip;
@@ -1014,9 +1014,10 @@ xlog_cil_push_background(/* * The cil won't be empty because we are called while holding the -* context lock so whatever we added to the CIL will still be there +* context lock so whatever we added to the CIL will still be there. */ ASSERT (!list_empty (&cil->xc_cil)); +ASSERT (!test_bit (XLOG_CIL_EMPTY , &cil->xc_flags));/* * Don't do a background push if we haven't used up all the
@@ -1103,7 +1104,8 @@ xlog_cil_push_now(* there's no work we need to do. */ spin_lock (&cil->xc_push_lock); -if (list_empty (&cil->xc_cil) || push_seq <= cil->xc_push_seq) { +if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags) || + push_seq <= cil->xc_push_seq) {spin_unlock (&cil->xc_push_lock);return ; }
@@ -1123,7 +1125,7 @@ xlog_cil_empty(bool empty = false;spin_lock (&cil->xc_push_lock); -if (list_empty (&cil->xc_cil)) +if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) empty = true;spin_unlock (&cil->xc_push_lock);return empty;
@@ -1284,7 +1286,7 @@ xlog_cil_force_seq(* we would have found the context on the committing list. */ if (sequence == cil->xc_current_sequence && - !list_empty (&cil->xc_cil)) { + !test_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) {spin_unlock (&cil->xc_push_lock); goto restart; }
@@ -1315,21 +1317,19 @@ xlog_cil_force_seq(*/ bool xfs_log_item_in_current_chkpt ( -struct xfs_log_item *lip) +struct xfs_log_item *lip) { -struct xfs_cil_ctx *ctx; +struct xfs_cil *cil = lip->li_mountp->m_log->l_cilp; -if (list_empty (&lip->li_cil)) +if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags))return false; - ctx = lip->li_mountp->m_log->l_cilp->xc_ctx; -/* * li_seq is written on the first commit of a log item to record the * first checkpoint it is written to. Hence if it is different to the * current sequence, we're in a new checkpoint. */ -if (XFS_LSN_CMP (lip->li_seq, ctx->sequence) !=0 ) +if (XFS_LSN_CMP (lip->li_seq, cil->xc_ctx->sequence) !=0 )return false;return true; }
@@ -1368,13 +1368,16 @@ voidxlog_cil_destroy ( struct xlog *log) { -if (log->l_cilp->xc_ctx) { -if (log->l_cilp->xc_ctx->ticket) -xfs_log_ticket_put (log->l_cilp->xc_ctx->ticket); -kmem_free (log->l_cilp->xc_ctx); +struct xfs_cil *cil = log->l_cilp; + +if (cil->xc_ctx) { +if (cil->xc_ctx->ticket) +xfs_log_ticket_put (cil->xc_ctx->ticket); +kmem_free (cil->xc_ctx); } -ASSERT (list_empty (&log->l_cilp->xc_cil)); -kmem_free (log->l_cilp); +ASSERT (list_empty (&cil->xc_cil)); +ASSERT (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags)); +kmem_free (cil); }
fs/xfs/xfs_log_priv.h +4 -0
@@ -248,6 +248,7 @@ struct xfs_cil_ctx {*/ struct xfs_cil {struct xlog *xc_log; +unsigned long xc_flags;struct list_head xc_cil;spinlock_t xc_cil_lock;
@@ -263,6 +264,9 @@ struct xfs_cil {wait_queue_head_t xc_push_wait;/* background push throttle */ }____cacheline_aligned_in_smp ; +/* xc_flags bit values */ +#define XLOG_CIL_EMPTY 1 +/* * The amount of log space we allow the CIL to aggregate is difficult to size. * Whatever we choose, we have to make sure we can get a reservation for the
Signature · 1 line
2.28.0
PATCH 2/12 xfs: lift init CIL reservation out of xc_cil_lock
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The xc_cil_lock is the most highly contended lock in XFS now. To start the process of getting rid of it, lift the initial reservation of the CIL log space out from under the xc_cil_lock.
--- fs/xfs/xfs_log_cil.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-)
fs/xfs/xfs_log_cil.c +12 -15
@@ -430,23 +430,19 @@ xlog_cil_insert_items(*/ xlog_cil_insert_format_items (log, tp, &len); -spin_lock (&cil->xc_cil_lock); - -/* attach the transaction to the CIL if it has any busy extents */ -if (!list_empty (&tp->t_busy)) -list_splice_init (&tp->t_busy, &ctx->busy_extents); -/* * We need to take the CIL checkpoint unit reservation on the first * commit into the CIL. Test the XLOG_CIL_EMPTY bit first so we don't -* unnecessarily do an atomic op in the fast path here. +* unnecessarily do an atomic op in the fast path here. We don't need to +* hold the xc_cil_lock here to clear the XLOG_CIL_EMPTY bit as we are +* under the xc_ctx_lock here and that needs to be held exclusively to +* reset the XLOG_CIL_EMPTY bit. */ if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags) && -test_and_clear_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) { +test_and_clear_bit (XLOG_CIL_EMPTY , &cil->xc_flags)) ctx_res = ctx->ticket->t_unit_res; - ctx->ticket->t_curr_res = ctx_res; - tp->t_ticket->t_curr_res -= ctx_res; - } + +spin_lock (&cil->xc_cil_lock);/* do we need space for more log record headers? */ iclog_space = log->l_iclog_size - log->l_iclog_hsize;
@@ -456,11 +452,9 @@ xlog_cil_insert_items(/* need to take into account split region headers, too */ split_res *= log->l_iclog_hsize +sizeof (struct xlog_op_header ); ctx->ticket->t_unit_res += split_res; - ctx->ticket->t_curr_res += split_res; - tp->t_ticket->t_curr_res -= split_res; -ASSERT (tp->t_ticket->t_curr_res >= len); } - tp->t_ticket->t_curr_res -= len; + tp->t_ticket->t_curr_res -= split_res + ctx_res + len; + ctx->ticket->t_curr_res += split_res + ctx_res; ctx->space_used += len;/*
@@ -498,6 +492,9 @@ xlog_cil_insert_items(list_move_tail (&lip->li_cil, &cil->xc_cil); } +/* attach the transaction to the CIL if it has any busy extents */ +if (!list_empty (&tp->t_busy)) +list_splice_init (&tp->t_busy, &ctx->busy_extents);spin_unlock (&cil->xc_cil_lock);if (tp->t_ticket->t_curr_res <0 )
Signature · 1 line
2.28.0
PATCH 3/12 xfs: rework per-iclog header CIL reservation
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
For every iclog that a CIL push will use up, we need to ensure we have space reserved for the iclog header in each iclog. It is extremely difficult to do this accurately with a per-cpu counter without expensive summing of the counter in every commit. However, we know what the maximum CIL size is going to be because of the hard space limit we have, and hence we know exactly how many iclogs we are going to need to write out the CIL.
We are constrained by the requirement that small transactions only have reservation space for a single iclog header built into them. At commit time we don't know how much of the current transaction reservation is made up of iclog header reservations as calculated by xfs_log_calc_unit_res() when the ticket was reserved. As larger reservations have multiple header spaces reserved, we can steal more than one iclog header reservation at a time, but we only steal the exact number needed for the given log vector size delta.
As a result, we don't know exactly when we are going to steal iclog header reservations, nor do we know exactly how many we are going to need for a given CIL.
To make things simple, start by calculating the worst case number of iclog headers a full CIL push will require. Record this into an atomic variable in the CIL. Then add a byte counter to the log ticket that records exactly how much iclog header space has been reserved in this ticket by xfs_log_calc_unit_res(). This tells us exactly how much space we can steal from the ticket at transaction commit time.
Now, at transaction commit time, we can check if the CIL has a full iclog header reservation and, if not, steal the entire reservation the current ticket holds for iclog headers. This minimises the number of times we need to do atomic operations in the fast path, but still guarantees we get all the reservations we need.
--- fs/xfs/libxfs/xfs_log_rlimit.c | 2 +- fs/xfs/libxfs/xfs_shared.h | 3 +- fs/xfs/xfs_log.c | 12 +++++--- fs/xfs/xfs_log_cil.c | 55 ++++++++++++++++++++++++++-------- fs/xfs/xfs_log_priv.h | 20 +++++++------ 5 files changed, 64 insertions(+), 28 deletions(-)
fs/xfs/libxfs/xfs_log_rlimit.c +1 -1
@@ -88,7 +88,7 @@ xfs_log_calc_minimum_size(xfs_log_get_max_trans_res (mp, &tres); - max_logres =xfs_log_calc_unit_res (mp, tres.tr_logres); + max_logres =xfs_log_calc_unit_res (mp, tres.tr_logres,NULL );if (tres.tr_logcount >1 ) max_logres *= tres.tr_logcount;
fs/xfs/libxfs/xfs_shared.h +2 -1
@@ -48,7 +48,8 @@ extern const struct xfs_buf_ops xfs_symlink_buf_ops;extern const struct xfs_buf_ops xfs_rtbuf_ops;/* log size calculation functions */ -int xfs_log_calc_unit_res (struct xfs_mount *mp,int unit_bytes); +int xfs_log_calc_unit_res (struct xfs_mount *mp,int unit_bytes, +int *niclogs);int xfs_log_calc_minimum_size (struct xfs_mount *);struct xfs_trans_res ;
fs/xfs/xfs_log.c +8 -4
@@ -3334,7 +3334,8 @@ xfs_log_ticket_get(static int xlog_calc_unit_res (struct xlog *log, -int unit_bytes) +int unit_bytes, +int *niclogs) {int iclog_space;uint num_headers;
@@ -3414,15 +3415,18 @@ xlog_calc_unit_res(/* roundoff padding for transaction data and one for commit record */ unit_bytes +=2 * log->l_iclog_roundoff; +if (niclogs) + *niclogs = num_headers;return unit_bytes; }int xfs_log_calc_unit_res (struct xfs_mount *mp, -int unit_bytes) +int unit_bytes, +int *niclogs) { -return xlog_calc_unit_res (mp->m_log, unit_bytes); +return xlog_calc_unit_res (mp->m_log, unit_bytes, niclogs); }/*
@@ -3440,7 +3444,7 @@ xlog_ticket_alloc( tic =kmem_cache_zalloc (xfs_log_ticket_zone,GFP_NOFS | __GFP_NOFAIL); - unit_res =xlog_calc_unit_res (log, unit_bytes); + unit_res =xlog_calc_unit_res (log, unit_bytes, &tic->t_iclog_hdrs);atomic_set (&tic->t_ref,1 ); tic->t_task = current;
fs/xfs/xfs_log_cil.c +42 -13
89 lines not on this page · raw message
fs/xfs/xfs_log_priv.h +11 -9
@@ -140,15 +140,16 @@ enum xlog_iclog_state {#define XLOG_TIC_LEN_MAX 15typedef struct xlog_ticket { -struct list_head t_queue;/* reserve/write queue */ -struct task_struct *t_task;/* task that owns this ticket */ -xlog_tid_t t_tid;/* transaction identifier : 4 */ -atomic_t t_ref;/* ticket reference count : 4 */ -int t_curr_res;/* current reservation in bytes : 4 */ -int t_unit_res;/* unit reservation in bytes : 4 */ -char t_ocnt;/* original count : 1 */ -char t_cnt;/* current count : 1 */ -char t_flags;/* properties of reservation : 1 */ +struct list_head t_queue;/* reserve/write queue */ +struct task_struct *t_task;/* task that owns this ticket */ +xlog_tid_t t_tid;/* transaction identifier */ +atomic_t t_ref;/* ticket reference count */ +int t_curr_res;/* current reservation */ +int t_unit_res;/* unit reservation */ +char t_ocnt;/* original count */ +char t_cnt;/* current count */ +char t_flags;/* properties of reservation */ +int t_iclog_hdrs;/* iclog hdrs in t_curr_res */ }xlog_ticket_t ;/*
@@ -249,6 +250,7 @@ struct xfs_cil_ctx {struct xfs_cil {struct xlog *xc_log;unsigned long xc_flags; +atomic_t xc_iclog_hdrs;struct list_head xc_cil;spinlock_t xc_cil_lock;
Signature · 1 line
2.28.0
PATCH 4/12 xfs: introduce per-cpu CIL tracking sructure
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The CIL push lock is highly contended on larger machines, becoming a hard bottleneck that about 700,000 transaction commits/s on >16p machines. To address this, start moving the CIL tracking infrastructure to utilise per-CPU structures.
We need to track the space used, the amount of log reservation space reserved to write the CIL, the log items in the CIL and the busy extents that need to be completed by the CIL commit. This requires a couple of per-cpu counters, an unordered per-cpu list and a globally ordered per-cpu list.
Create a per-cpu structure to hold these and all the management interfaces needed, as well as the hooks to handle hotplug CPUs.
--- fs/xfs/xfs_log_cil.c | 94 ++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_log_priv.h | 15 ++++++ include/linux/cpuhotplug.h | 1 + 3 files changed, 110 insertions(+)
fs/xfs/xfs_log_cil.c +94 -0
@@ -1360,6 +1360,93 @@ xfs_log_item_in_current_chkpt(return true; } +#ifdef CONFIG_HOTPLUG_CPU +static LIST_HEAD (xlog_cil_pcp_list); +static DEFINE_SPINLOCK (xlog_cil_pcp_lock); +static bool xlog_cil_pcp_init; + +static int +xlog_cil_pcp_dead ( +unsigned int cpu) +{ +struct xfs_cil *cil; + +spin_lock (&xlog_cil_pcp_lock); +list_for_each_entry (cil, &xlog_cil_pcp_list, xc_pcp_list) { +/* move stuff on dead CPU to context */ + } +spin_unlock (&xlog_cil_pcp_lock); +return 0 ; +} + +static int +xlog_cil_pcp_hpadd ( +struct xfs_cil *cil) +{ +if (!xlog_cil_pcp_init) { +int ret; + ret =cpuhp_setup_state_nocalls (CPUHP_XFS_CIL_DEAD , +"xfs/cil_pcp:dead" ,NULL , + xlog_cil_pcp_dead); +if (ret <0 ) { +xfs_warn (cil->xc_log->l_mp, +"Failed to initialise CIL hotplug, error %d. XFS is non-functional." , + ret); +ASSERT (0 ); +return -ENOMEM ; + } + xlog_cil_pcp_init = true; + } + +INIT_LIST_HEAD (&cil->xc_pcp_list); +spin_lock (&xlog_cil_pcp_lock); +list_add (&cil->xc_pcp_list, &xlog_cil_pcp_list); +spin_unlock (&xlog_cil_pcp_lock); +return 0 ; +} + +static void +xlog_cil_pcp_hpremove ( +struct xfs_cil *cil) +{ +spin_lock (&xlog_cil_pcp_lock); +list_del (&cil->xc_pcp_list); +spin_unlock (&xlog_cil_pcp_lock); +} + +#else /* !CONFIG_HOTPLUG_CPU */ +static inline void xlog_cil_pcp_hpadd (struct xfs_cil *cil) {} +static inline void xlog_cil_pcp_hpremove (struct xfs_cil *cil) {} +#endif + +static void __percpu * +xlog_cil_pcp_alloc ( +struct xfs_cil *cil) +{ +struct xlog_cil_pcp *cilpcp; + + cilpcp = alloc_percpu(struct xlog_cil_pcp); +if (!cilpcp) +return NULL ; + +if (xlog_cil_pcp_hpadd (cil) <0 ) { +free_percpu (cilpcp); +return NULL ; + } +return cilpcp; +} + +static void +xlog_cil_pcp_free ( +struct xfs_cil *cil, +struct xlog_cil_pcp *cilpcp) +{ +if (!cilpcp) +return ; +xlog_cil_pcp_hpremove (cil); +free_percpu (cilpcp); +} +/* * Perform initial CIL structure initialisation. */
@@ -1374,6 +1461,12 @@ xlog_cil_init(if (!cil)return -ENOMEM ; + cil->xc_pcp =xlog_cil_pcp_alloc (cil); +if (!cil->xc_pcp) { +kmem_free (cil); +return -ENOMEM ; + } +INIT_LIST_HEAD (&cil->xc_cil);INIT_LIST_HEAD (&cil->xc_committing);spin_lock_init (&cil->xc_cil_lock);
@@ -1404,6 +1497,7 @@ xlog_cil_destroy(ASSERT (list_empty (&cil->xc_cil));ASSERT (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags)); +xlog_cil_pcp_free (cil, cil->xc_pcp);kmem_free (cil); }
fs/xfs/xfs_log_priv.h +15 -0
@@ -231,6 +231,16 @@ struct xfs_cil_ctx {struct work_struct push_work; }; +/* +* Per-cpu CIL tracking items +*/ +struct xlog_cil_pcp { +uint32_t space_used; +uint32_t curr_res; +struct list_head busy_extents; +struct list_head log_items; +}; +/* * Committed Item List structure *
@@ -264,6 +274,11 @@ struct xfs_cil {wait_queue_head_t xc_commit_wait;uint64_t xc_current_sequence;wait_queue_head_t xc_push_wait;/* background push throttle */ + +struct xlog_cil_pcp __percpu *xc_pcp; +#ifdef CONFIG_HOTPLUG_CPU +struct list_head xc_pcp_list; +#endif }____cacheline_aligned_in_smp ;/* xc_flags bit values */
include/linux/cpuhotplug.h +1 -0
@@ -52,6 +52,7 @@ enum cpuhp_state {CPUHP_FS_BUFF_DEAD ,CPUHP_PRINTK_DEAD ,CPUHP_MM_MEMCQ_DEAD , +CPUHP_XFS_CIL_DEAD ,CPUHP_PERCPU_CNT_DEAD ,CPUHP_RADIX_DEAD , CPUHP_PAGE_ALLOC_DEAD,
Signature · 1 line
2.28.0
PATCH 5/12 xfs: implement percpu cil space used calculation
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we have the CIL percpu structures in place, implement the space used counter with a fast sum check similar to the percpu_counter infrastructure.
--- fs/xfs/xfs_log_cil.c | 42 ++++++++++++++++++++++++++++++++++++------ fs/xfs/xfs_log_priv.h | 2 +- 2 files changed, 37 insertions(+), 7 deletions(-)
fs/xfs/xfs_log_cil.c +36 -6
@@ -433,6 +433,8 @@ xlog_cil_insert_items(struct xfs_log_item *lip;int len =0 ;int iovhdr_res =0 , split_res =0 , ctx_res =0 ; +int space_used; +struct xlog_cil_pcp *cilpcp;ASSERT (tp);
@@ -469,8 +471,9 @@ xlog_cil_insert_items(* * This can steal more than we need, but that's OK. */ + space_used =atomic_read (&ctx->space_used);if (atomic_read (&cil->xc_iclog_hdrs) >0 || - ctx->space_used + len >=XLOG_CIL_BLOCKING_SPACE_LIMIT (log)) { + space_used + len >=XLOG_CIL_BLOCKING_SPACE_LIMIT (log)) {int split_res = log->l_iclog_hsize +sizeof (struct xlog_op_header );if (ctx_res)
@@ -480,16 +483,34 @@ xlog_cil_insert_items(atomic_sub (tp->t_ticket->t_iclog_hdrs, &cil->xc_iclog_hdrs); } +/* +* Update the CIL percpu pointer. This updates the global counter when +* over the percpu batch size or when the CIL is over the space limit. +* This means low lock overhead for normal updates, and when over the +* limit the space used is immediately accounted. This makes enforcing +* the hard limit much more accurate. The per cpu fold threshold is +* based on how close we are to the hard limit. +*/ + cilpcp =get_cpu_ptr (cil->xc_pcp); + cilpcp->space_used += len; +if (space_used >=XLOG_CIL_SPACE_LIMIT (log) || + cilpcp->space_used > + ((XLOG_CIL_BLOCKING_SPACE_LIMIT (log) - space_used) / +num_online_cpus ())) { +atomic_add (cilpcp->space_used, &ctx->space_used); + cilpcp->space_used =0 ; + } +put_cpu_ptr (cilpcp); +spin_lock (&cil->xc_cil_lock); - tp->t_ticket->t_curr_res -= ctx_res + len; ctx->ticket->t_unit_res += ctx_res; ctx->ticket->t_curr_res += ctx_res; - ctx->space_used += len;/* * If we've overrun the reservation, dump the tx details before we move * the log items. Shutdown is imminent... */ + tp->t_ticket->t_curr_res -= ctx_res + len;if (WARN_ON (tp->t_ticket->t_curr_res <0 )) {xfs_warn (log->l_mp,"Transaction log reservation overrun:" ); xfs_warn(log->l_mp,
@@ -768,12 +789,20 @@ xlog_cil_push_work(xfs_lsn_t push_seq;DECLARE_COMPLETION_ONSTACK (bdev_flush);bool commit_iclog_sync = false; +int cpu; +struct xlog_cil_pcp *cilpcp; new_ctx =xlog_cil_ctx_alloc (); new_ctx->ticket =xlog_cil_ticket_alloc (log);down_write (&cil->xc_ctx_lock); +/* Reset the CIL pcp counters */ +for_each_online_cpu (cpu) { + cilpcp =per_cpu_ptr (cil->xc_pcp, cpu); + cilpcp->space_used =0 ; + } +spin_lock (&cil->xc_push_lock); push_seq = cil->xc_push_seq;ASSERT (push_seq <= ctx->sequence);
@@ -1037,6 +1066,7 @@ xlog_cil_push_background( struct xlog *log) __releases(cil->xc_ctx_lock) {struct xfs_cil *cil = log->l_cilp; +int space_used =atomic_read (&cil->xc_ctx->space_used);/* * The cil won't be empty because we are called while holding the
@@ -1049,7 +1079,7 @@ xlog_cil_push_background(* Don't do a background push if we haven't used up all the * space available yet. */ -if (cil->xc_ctx->space_used <XLOG_CIL_SPACE_LIMIT (log)) { +if (space_used <XLOG_CIL_SPACE_LIMIT (log)) {up_read (&cil->xc_ctx_lock);return ; }
@@ -1078,10 +1108,10 @@ xlog_cil_push_background(* The ctx->xc_push_lock provides the serialisation necessary for safely * using the lockless waitqueue_active() check in this context. */ -if (cil->xc_ctx->space_used >=XLOG_CIL_BLOCKING_SPACE_LIMIT (log) || +if (space_used >=XLOG_CIL_BLOCKING_SPACE_LIMIT (log) ||waitqueue_active (&cil->xc_push_wait)) {trace_xfs_log_cil_wait (log, cil->xc_ctx->ticket); -ASSERT (cil->xc_ctx->space_used < log->l_logsize); +ASSERT (space_used < log->l_logsize);xlog_wait (&cil->xc_push_wait, &cil->xc_push_lock);return ; }
fs/xfs/xfs_log_priv.h +1 -1
@@ -222,7 +222,7 @@ struct xfs_cil_ctx {xfs_lsn_t commit_lsn;/* chkpt commit record lsn */ struct xlog_ticket *ticket;/* chkpt ticket */ int nvecs;/* number of regions */ -int space_used;/* aggregate size of regions */ +atomic_t space_used;/* aggregate size of regions */ struct list_head busy_extents;/* busy extents in chkpt */ struct xfs_log_vec *lv_chain;/* logvecs being pushed */ struct list_head iclog_entry;
Signature · 1 line
2.28.0
PATCH 6/12 xfs: track CIL ticket reservation in percpu structure
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
To get it out from under the cil spinlock.
--- fs/xfs/xfs_log_cil.c | 11 ++++++----- fs/xfs/xfs_log_priv.h | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-)
fs/xfs/xfs_log_cil.c +6 -5
@@ -492,6 +492,7 @@ xlog_cil_insert_items(* based on how close we are to the hard limit. */ cilpcp =get_cpu_ptr (cil->xc_pcp); + cilpcp->space_reserved += ctx_res; cilpcp->space_used += len;if (space_used >=XLOG_CIL_SPACE_LIMIT (log) || cilpcp->space_used >
@@ -502,10 +503,6 @@ xlog_cil_insert_items( }put_cpu_ptr (cilpcp); -spin_lock (&cil->xc_cil_lock); - ctx->ticket->t_unit_res += ctx_res; - ctx->ticket->t_curr_res += ctx_res; -/* * If we've overrun the reservation, dump the tx details before we move * the log items. Shutdown is imminent...
@@ -527,6 +524,7 @@ xlog_cil_insert_items(* We do this here so we only need to take the CIL lock once during * the transaction commit. */ +spin_lock (&cil->xc_cil_lock);list_for_each_entry (lip, &tp->t_items, li_trans) {/* Skip items which aren't dirty in this transaction. */
@@ -797,10 +795,13 @@ xlog_cil_push_work(down_write (&cil->xc_ctx_lock); -/* Reset the CIL pcp counters */ +/* Aggregate and reset the CIL pcp counters */ for_each_online_cpu (cpu) { cilpcp =per_cpu_ptr (cil->xc_pcp, cpu); + ctx->ticket->t_curr_res += cilpcp->space_reserved; cilpcp->space_used =0 ; + cilpcp->space_reserved =0 ; + }spin_lock (&cil->xc_push_lock);
fs/xfs/xfs_log_priv.h +1 -1
@@ -236,7 +236,7 @@ struct xfs_cil_ctx {*/ struct xlog_cil_pcp {uint32_t space_used; -uint32_t curr_res; +uint32_t space_reserved;struct list_head busy_extents;struct list_head log_items; };
Signature · 1 line
2.28.0
PATCH 7/12 xfs: convert CIL busy extents to per-cpu
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
To get them out from under the CIL lock.
This is an unordered list, so we can simply punt it to per-cpu lists during transaction commits and reaggregate it back into a single list during the CIL push work.
--- fs/xfs/xfs_log_cil.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
fs/xfs/xfs_log_cil.c +18 -8
@@ -501,6 +501,9 @@ xlog_cil_insert_items(atomic_add (cilpcp->space_used, &ctx->space_used); cilpcp->space_used =0 ; } +/* attach the transaction to the CIL if it has any busy extents */ +if (!list_empty (&tp->t_busy)) +list_splice_init (&tp->t_busy, &cilpcp->busy_extents);put_cpu_ptr (cilpcp);/*
@@ -540,9 +543,6 @@ xlog_cil_insert_items(list_move_tail (&lip->li_cil, &cil->xc_cil); } -/* attach the transaction to the CIL if it has any busy extents */ -if (!list_empty (&tp->t_busy)) -list_splice_init (&tp->t_busy, &ctx->busy_extents);spin_unlock (&cil->xc_cil_lock);if (tp->t_ticket->t_curr_res <0 )
@@ -801,7 +801,10 @@ xlog_cil_push_work( ctx->ticket->t_curr_res += cilpcp->space_reserved; cilpcp->space_used =0 ; cilpcp->space_reserved =0 ; - +if (!list_empty (&cilpcp->busy_extents)) { +list_splice_init (&cilpcp->busy_extents, + &ctx->busy_extents); + } }spin_lock (&cil->xc_push_lock);
@@ -1454,17 +1457,24 @@ static void __percpu *xlog_cil_pcp_alloc ( struct xfs_cil *cil) { +void __percpu *pcptr;struct xlog_cil_pcp *cilpcp; +int cpu; - cilpcp = alloc_percpu(struct xlog_cil_pcp); -if (!cilpcp) + pcptr = alloc_percpu(struct xlog_cil_pcp); +if (!pcptr)return NULL ; +for_each_possible_cpu (cpu) { + cilpcp =per_cpu_ptr (pcptr, cpu); +INIT_LIST_HEAD (&cilpcp->busy_extents); + } +if (xlog_cil_pcp_hpadd (cil) <0 ) { -free_percpu (cilpcp); +free_percpu (pcptr);return NULL ; } -return cilpcp; +return pcptr; } static void
Signature · 1 line
2.28.0
PATCH 8/12 xfs: Add order IDs to log items in CIL
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Before we split the ordered CIL up into per cpu lists, we need a mechanism to track the order of the items in the CIL. We need to do this because there are rules around the order in which related items must physically appear in the log even inside a single checkpoint transaction.
An example of this is intents - an intent must appear in the log before it's intent done record so taht log recovery can cancel the intent correctly. If we have these two records misordered in the CIL, then they will not be recovered correctly by journal replay.
We also will not be able to move items to the tail of the CIL list when they are relogged, hence the log items will need some mechanism to allow the correct log item order to be recreated before we write log items to the hournal.
Hence we need to have a mechanism for recording global order of transactions in the log items so that we can recover that order from un-ordered per-cpu lists.
Do this with a simple monotonic increasing commit counter in the CIL context. Each log item in the transaction gets stamped with the current commit order ID before it is added to the CIL. If the item is already in the CIL, leave it where it is instead of moving it to the tail of the list and instead sort the list before we start the push work.
XXX: list_sort() under the cil_ctx_lock held exclusive starts hurting that >16 threads. Front end commits are waiting on the push to switch contexts much longer. The item order id should likely be moved into the logvecs when they are detacted from the items, then the sort can be done on the logvec after the cil_ctx_lock has been released. logvecs will need to use a list_head for this rather than a single linked list like they do now....
--- fs/xfs/xfs_log_cil.c | 34 ++++++++++++++++++++++++++-------- fs/xfs/xfs_log_priv.h | 1 + fs/xfs/xfs_trans.h | 1 + 3 files changed, 28 insertions(+), 8 deletions(-)
fs/xfs/xfs_log_cil.c +26 -8
@@ -434,6 +434,7 @@ xlog_cil_insert_items(int len =0 ;int iovhdr_res =0 , split_res =0 , ctx_res =0 ;int space_used; +int order;struct xlog_cil_pcp *cilpcp;ASSERT (tp);
@@ -523,10 +524,12 @@ xlog_cil_insert_items( }/* -* Now (re-)position everything modified at the tail of the CIL. +* Now update the order of everything modified in the transaction +* and insert items into the CIL if they aren't already there. * We do this here so we only need to take the CIL lock once during * the transaction commit. */ + order =atomic_inc_return (&ctx->order_id);spin_lock (&cil->xc_cil_lock);list_for_each_entry (lip, &tp->t_items, li_trans) {
@@ -534,13 +537,10 @@ xlog_cil_insert_items(if (!test_bit (XFS_LI_DIRTY , &lip->li_flags))continue ; -/* -* Only move the item if it isn't already at the tail. This is -* to prevent a transient list_empty() state when reinserting -* an item that is already the only item in the CIL. -*/ -if (!list_is_last (&lip->li_cil, &cil->xc_cil)) -list_move_tail (&lip->li_cil, &cil->xc_cil); + lip->li_order_id = order; +if (!list_empty (&lip->li_cil)) +continue ; +list_add (&lip->li_cil, &cil->xc_cil); }spin_unlock (&cil->xc_cil_lock);
@@ -753,6 +753,22 @@ xlog_cil_build_trans_hdr( tic->t_curr_res -= lvhdr->lv_bytes; } +static int +xlog_cil_order_cmp ( +void *priv, +struct list_head *a, +struct list_head *b) +{ + struct xfs_log_item *l1 = container_of(a, struct xfs_log_item, li_cil); + struct xfs_log_item *l2 = container_of(b, struct xfs_log_item, li_cil); + +if (l1->li_order_id > l2->li_order_id) +return 1 ; +if (l1->li_order_id < l2->li_order_id) +return -1 ; +return 0 ; +} +/* * Push the Committed Item List to the log. *
@@ -889,6 +905,7 @@ xlog_cil_push_work(* needed on the transaction commit side which is currently locked out * by the flush lock. */ +list_sort (NULL , &cil->xc_cil, xlog_cil_order_cmp); lv =NULL ;while (!list_empty (&cil->xc_cil)) {struct xfs_log_item *item;
@@ -896,6 +913,7 @@ xlog_cil_push_work( item =list_first_entry (&cil->xc_cil, struct xfs_log_item, li_cil);list_del_init (&item->li_cil); + item->li_order_id =0 ;if (!ctx->lv_chain) ctx->lv_chain = item->li_lv;else
fs/xfs/xfs_log_priv.h +1 -0
@@ -229,6 +229,7 @@ struct xfs_cil_ctx {struct list_head committing;/* ctx committing list */ struct work_struct discard_endio_work;struct work_struct push_work; +atomic_t order_id; };/*
fs/xfs/xfs_trans.h +1 -0
@@ -44,6 +44,7 @@ struct xfs_log_item {struct xfs_log_vec *li_lv;/* active log vector */ struct xfs_log_vec *li_lv_shadow;/* standby vector */ uint64_t li_seq;/* CIL commit seq */ +uint32_t li_order_id;/* CIL commit order */ };/*
Signature · 1 line
2.28.0
PATCH 9/12 xfs: convert CIL to unordered per cpu lists
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
So that we can remove the cil_lock which is a global serialisation point. We've already got ordering sorted, so all we need to do is treat the CIL list like the busy extent list and reconstruct it before the push starts.
This is what we're trying to avoid:
- 75.35% 1.83% [kernel] [k] xfs_log_commit_cil
- 46.35% xfs_log_commit_cil
- 41.54% _raw_spin_lock
- 67.30% do_raw_spin_lock
66.96% __pv_queued_spin_lock_slowpath
Which happens on a 32p system when running a 32-way 'rm -rf' workload. After this patch:
- 20.90% 3.23% [kernel] [k] xfs_log_commit_cil
- 17.67% xfs_log_commit_cil
- 6.51% xfs_log_ticket_ungrant
1.40% xfs_log_space_wake
2.32% memcpy_erms
- 2.18% xfs_buf_item_committing
- 2.12% xfs_buf_item_release
- 1.03% xfs_buf_unlock
0.96% up
0.72% xfs_buf_rele
1.33% xfs_inode_item_format
1.19% down_read
0.91% up_read
0.76% xfs_buf_item_format
- 0.68% kmem_alloc_large
- 0.67% kmem_alloc
0.64% __kmalloc
0.50% xfs_buf_item_sizeIt kinda looks like the workload is running out of log space all the time. But all the spinlock contention is gone and the transaction commit rate has gone from 800k/s to 1.3M/s so the amount of real work being done has gone up a *lot*.
--- fs/xfs/xfs_log_cil.c | 61 ++++++++++++++++++++----------------------- fs/xfs/xfs_log_priv.h | 2 -- 2 files changed, 29 insertions(+), 34 deletions(-)
fs/xfs/xfs_log_cil.c +29 -32
@@ -448,10 +448,9 @@ xlog_cil_insert_items(/* * We need to take the CIL checkpoint unit reservation on the first * commit into the CIL. Test the XLOG_CIL_EMPTY bit first so we don't -* unnecessarily do an atomic op in the fast path here. We don't need to -* hold the xc_cil_lock here to clear the XLOG_CIL_EMPTY bit as we are -* under the xc_ctx_lock here and that needs to be held exclusively to -* reset the XLOG_CIL_EMPTY bit. +* unnecessarily do an atomic op in the fast path here. We can clear the +* XLOG_CIL_EMPTY bit as we are under the xc_ctx_lock here and that +* needs to be held exclusively to reset the XLOG_CIL_EMPTY bit. */ if (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags) &&test_and_clear_bit (XLOG_CIL_EMPTY , &cil->xc_flags))
@@ -505,24 +504,6 @@ xlog_cil_insert_items(/* attach the transaction to the CIL if it has any busy extents */ if (!list_empty (&tp->t_busy))list_splice_init (&tp->t_busy, &cilpcp->busy_extents); -put_cpu_ptr (cilpcp); - -/* -* If we've overrun the reservation, dump the tx details before we move -* the log items. Shutdown is imminent... -*/ - tp->t_ticket->t_curr_res -= ctx_res + len; -if (WARN_ON (tp->t_ticket->t_curr_res <0 )) { -xfs_warn (log->l_mp,"Transaction log reservation overrun:" ); -xfs_warn (log->l_mp, -" log items: %d bytes (iov hdrs: %d bytes)" , - len, iovhdr_res); -xfs_warn (log->l_mp," split region headers: %d bytes" , - split_res); -xfs_warn (log->l_mp," ctx ticket: %d bytes" , ctx_res); -xlog_print_trans (tp); - } -/* * Now update the order of everything modified in the transaction * and insert items into the CIL if they aren't already there.
@@ -530,7 +511,6 @@ xlog_cil_insert_items(* the transaction commit. */ order =atomic_inc_return (&ctx->order_id); -spin_lock (&cil->xc_cil_lock);list_for_each_entry (lip, &tp->t_items, li_trans) {/* Skip items which aren't dirty in this transaction. */
@@ -540,10 +520,26 @@ xlog_cil_insert_items( lip->li_order_id = order;if (!list_empty (&lip->li_cil))continue ; -list_add (&lip->li_cil, &cil->xc_cil); +list_add (&lip->li_cil, &cilpcp->log_items); + } +put_cpu_ptr (cilpcp); + +/* +* If we've overrun the reservation, dump the tx details before we move +* the log items. Shutdown is imminent... +*/ + tp->t_ticket->t_curr_res -= ctx_res + len; +if (WARN_ON (tp->t_ticket->t_curr_res <0 )) { +xfs_warn (log->l_mp,"Transaction log reservation overrun:" ); +xfs_warn (log->l_mp, +" log items: %d bytes (iov hdrs: %d bytes)" , + len, iovhdr_res); +xfs_warn (log->l_mp," split region headers: %d bytes" , + split_res); +xfs_warn (log->l_mp," ctx ticket: %d bytes" , ctx_res); +xlog_print_trans (tp); } -spin_unlock (&cil->xc_cil_lock);if (tp->t_ticket->t_curr_res <0 )xfs_force_shutdown (log->l_mp,SHUTDOWN_LOG_IO_ERROR );
@@ -805,6 +801,7 @@ xlog_cil_push_work(bool commit_iclog_sync = false;int cpu;struct xlog_cil_pcp *cilpcp; +LIST_HEAD (log_items); new_ctx =xlog_cil_ctx_alloc (); new_ctx->ticket =xlog_cil_ticket_alloc (log);
@@ -821,6 +818,9 @@ xlog_cil_push_work(list_splice_init (&cilpcp->busy_extents, &ctx->busy_extents); } +if (!list_empty (&cilpcp->log_items)) { +list_splice_init (&cilpcp->log_items, &log_items); + } }spin_lock (&cil->xc_push_lock);
@@ -905,12 +905,12 @@ xlog_cil_push_work(* needed on the transaction commit side which is currently locked out * by the flush lock. */ -list_sort (NULL , &cil->xc_cil, xlog_cil_order_cmp); +list_sort (NULL , &log_items, xlog_cil_order_cmp); lv =NULL ; -while (!list_empty (&cil->xc_cil)) { +while (!list_empty (&log_items)) {struct xfs_log_item *item; - item =list_first_entry (&cil->xc_cil, + item =list_first_entry (&log_items, struct xfs_log_item, li_cil);list_del_init (&item->li_cil); item->li_order_id =0 ;
@@ -1094,7 +1094,6 @@ xlog_cil_push_background(* The cil won't be empty because we are called while holding the * context lock so whatever we added to the CIL will still be there. */ -ASSERT (!list_empty (&cil->xc_cil));ASSERT (!test_bit (XLOG_CIL_EMPTY , &cil->xc_flags));/*
@@ -1486,6 +1485,7 @@ xlog_cil_pcp_alloc(for_each_possible_cpu (cpu) { cilpcp =per_cpu_ptr (pcptr, cpu);INIT_LIST_HEAD (&cilpcp->busy_extents); +INIT_LIST_HEAD (&cilpcp->log_items); }if (xlog_cil_pcp_hpadd (cil) <0 ) {
@@ -1526,9 +1526,7 @@ xlog_cil_init(return -ENOMEM ; } -INIT_LIST_HEAD (&cil->xc_cil);INIT_LIST_HEAD (&cil->xc_committing); -spin_lock_init (&cil->xc_cil_lock);spin_lock_init (&cil->xc_push_lock);init_waitqueue_head (&cil->xc_push_wait);init_rwsem (&cil->xc_ctx_lock);
@@ -1554,7 +1552,6 @@ xlog_cil_destroy(kmem_free (cil->xc_ctx); } -ASSERT (list_empty (&cil->xc_cil));ASSERT (test_bit (XLOG_CIL_EMPTY , &cil->xc_flags));xlog_cil_pcp_free (cil, cil->xc_pcp);kmem_free (cil);
fs/xfs/xfs_log_priv.h +0 -2
@@ -262,8 +262,6 @@ struct xfs_cil {struct xlog *xc_log;unsigned long xc_flags;atomic_t xc_iclog_hdrs; -struct list_head xc_cil; -spinlock_t xc_cil_lock;struct rw_semaphore xc_ctx_lock____cacheline_aligned_in_smp ;struct xfs_cil_ctx *xc_ctx;
Signature · 1 line
2.28.0
PATCH 10/12 xfs: move CIL ordering to the logvec chain
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Adding a list_sort() call to the CIL push work while the xc_ctx_lock is held exclusively has resulted in fairly long lock hold times and that stops all front end transaction commits from making progress.
We can move the sorting out of the xc_ctx_lock if we can transfer the ordering information to the log vectors as they are detached from the log items and then we can sort the log vectors. This requires log vectors to use a list_head rather than a single linked list and to hold an order ID field. With these changes, we can move the list_sort() call to just before we call xlog_write() when we aren't holding any locks at all.
--- fs/xfs/xfs_log.c | 43 ++++++++++++++++++++-------- fs/xfs/xfs_log.h | 3 +- fs/xfs/xfs_log_cil.c | 63 +++++++++++++++++++++++++---------------- fs/xfs/xfs_log_priv.h | 4 +-- fs/xfs/xfs_trans.c | 4 +-- fs/xfs/xfs_trans_priv.h | 4 +-- 6 files changed, 77 insertions(+), 44 deletions(-)
fs/xfs/xfs_log.c +31 -12
@@ -843,11 +843,14 @@ xlog_write_unmount_record( .lv_niovecs = 1, .lv_iovecp = ®, }; +LIST_HEAD (lv_chain); +INIT_LIST_HEAD (&vec.lv_chain); +list_add (&vec.lv_chain, &lv_chain);/* account for space used by record data */ ticket->t_curr_res -=sizeof (unmount_rec); -return xlog_write (log, &vec, ticket,NULL ,NULL ,XLOG_UNMOUNT_TRANS , - reg.i_len); +return xlog_write (log, &lv_chain, ticket,NULL ,NULL , +XLOG_UNMOUNT_TRANS , reg.i_len); }/*
@@ -1560,14 +1563,17 @@ xlog_commit_record( .lv_iovecp = ®, };int error; +LIST_HEAD (lv_chain); +INIT_LIST_HEAD (&vec.lv_chain); +list_add (&vec.lv_chain, &lv_chain);if (XLOG_FORCED_SHUTDOWN (log))return -EIO ;/* account for space used by record data */ ticket->t_curr_res -= reg.i_len; - error =xlog_write (log, &vec, ticket, lsn, iclog,XLOG_COMMIT_TRANS , - reg.i_len); + error =xlog_write (log, &lv_chain, ticket, lsn, iclog, +XLOG_COMMIT_TRANS , reg.i_len);if (error)xfs_force_shutdown (log->l_mp,SHUTDOWN_LOG_IO_ERROR );return error;
@@ -2117,6 +2123,7 @@ xlog_print_trans(*/ static struct xfs_log_vec * xlog_write_single( + struct list_head *lv_chain, struct xfs_log_vec *log_vector, struct xlog_ticket *ticket, struct xlog_in_core *iclog,
@@ -2135,7 +2142,8 @@ xlog_write_single( iclog->ic_state == XLOG_STATE_WANT_SYNC); ptr = iclog->ic_datap + *log_offset; -while (lv && (!lv->lv_niovecs || index < lv->lv_niovecs)) { +while ( + (!lv->lv_niovecs || index < lv->lv_niovecs)) {/* ordered log vectors have no regions to write */
@@ -2171,7 +2179,11 @@ xlog_write_single(continue ; next_lv:/* move to the next logvec */ - lv = lv->lv_next; + lv =list_next_entry (lv, lv_chain); +if (list_entry_is_head (lv, lv_chain, lv_chain)) { + lv =NULL ; +break ; + } index =0 ; }ASSERT (*len ==0 || lv);
@@ -2219,6 +2231,7 @@ xlog_write_get_more_iclog_space(static struct xfs_log_vec * xlog_write_partial( struct xlog *log, + struct list_head *lv_chain, struct xfs_log_vec *log_vector, struct xlog_ticket *ticket, struct xlog_in_core **iclogp,
@@ -2358,7 +2371,10 @@ xlog_write_partial(* the caller so it can go back to fast path copying. */ *iclogp = iclog; -return lv->lv_next; + lv =list_next_entry (lv, lv_chain); +if (list_entry_is_head (lv, lv_chain, lv_chain)) +return NULL ; +return lv; }/*
@@ -2404,7 +2420,7 @@ xlog_write_partial( int xlog_write( struct xlog *log, - struct xfs_log_vec *log_vector, + struct list_head *lv_chain, struct xlog_ticket *ticket, xfs_lsn_t *start_lsn, struct xlog_in_core **commit_iclog,
@@ -2412,7 +2428,7 @@ xlog_write( uint32_t len) {struct xlog_in_core *iclog =NULL ; -struct xfs_log_vec *lv = log_vector; +struct xfs_log_vec *lv;int record_cnt =0 ;int data_cnt =0 ;int error =0 ;
@@ -2444,15 +2460,18 @@ xlog_write(if (optype & (XLOG_COMMIT_TRANS |XLOG_UNMOUNT_TRANS )) iclog->ic_flags |= (XLOG_ICL_NEED_FLUSH |XLOG_ICL_NEED_FUA ); + lv = list_first_entry_or_null(lv_chain, struct xfs_log_vec, lv_chain);while (lv) { - lv =xlog_write_single (lv, ticket, iclog, &log_offset, + + lv =xlog_write_single (lv_chain, lv, ticket, iclog, &log_offset, &len, &record_cnt, &data_cnt);if (!lv)break ;ASSERT (!(optype & (XLOG_COMMIT_TRANS |XLOG_UNMOUNT_TRANS ))); - lv =xlog_write_partial (log, lv, ticket, &iclog, &log_offset, - &len, &record_cnt, &data_cnt); + lv =xlog_write_partial (log, lv_chain, lv, ticket, &iclog, + &log_offset, &len, &record_cnt, + &data_cnt);if (IS_ERR (lv)) { error =PTR_ERR (lv);break ;
fs/xfs/xfs_log.h +2 -1
@@ -9,7 +9,8 @@struct xfs_cil_ctx ;struct xfs_log_vec { -struct xfs_log_vec *lv_next;/* next lv in build list */ +struct list_head lv_chain;/* lv chain ptrs */ +int lv_order_id;/* chain ordering info */ int lv_niovecs;/* number of iovecs in lv */ struct xfs_log_iovec *lv_iovecp;/* iovec array */ struct xfs_log_item *lv_item;/* owner */
fs/xfs/xfs_log_cil.c +38 -25
152 lines not on this page · raw message
fs/xfs/xfs_log_priv.h +2 -2
@@ -224,7 +224,7 @@ struct xfs_cil_ctx {int nvecs;/* number of regions */ atomic_t space_used;/* aggregate size of regions */ struct list_head busy_extents;/* busy extents in chkpt */ -struct xfs_log_vec *lv_chain;/* logvecs being pushed */ +struct list_head lv_chain;/* logvecs being pushed */ struct list_head iclog_entry;struct list_head committing;/* ctx committing list */ struct work_struct discard_endio_work;
@@ -480,7 +480,7 @@ xlog_write_adv_cnt(void **ptr, int *len, int *off, size_t bytes)void xlog_print_tic_res (struct xfs_mount *mp,struct xlog_ticket *ticket);void xlog_print_trans (struct xfs_trans *); -int xlog_write (struct xlog *log,struct xfs_log_vec *log_vector, +int xlog_write (struct xlog *log,struct list_head *lv_chain,struct xlog_ticket *tic,xfs_lsn_t *start_lsn,struct xlog_in_core **commit_iclog,uint optype,uint32_t len); int xlog_commit_record(struct xlog *log, struct xlog_ticket *ticket,
fs/xfs/xfs_trans.c +2 -2
@@ -745,7 +745,7 @@ xfs_log_item_batch_insert(void xfs_trans_committed_bulk (struct xfs_ail *ailp, -struct xfs_log_vec *log_vector, +struct list_head *lv_chain,xfs_lsn_t commit_lsn,bool aborted) {
@@ -760,7 +760,7 @@ xfs_trans_committed_bulk(spin_unlock (&ailp->ail_lock);/* unpin all the log items */ -for (lv = log_vector; lv; lv = lv->lv_next ) { +list_for_each_entry (lv, lv_chain, lv_chain) {struct xfs_log_item *lip = lv->lv_item;xfs_lsn_t item_lsn;
fs/xfs/xfs_trans_priv.h +2 -2
@@ -18,8 +18,8 @@ void xfs_trans_add_item(struct xfs_trans *, struct xfs_log_item *);void xfs_trans_del_item (struct xfs_log_item *);void xfs_trans_unreserve_and_mod_sb (struct xfs_trans *tp); -void xfs_trans_committed_bulk (struct xfs_ail *ailp,struct xfs_log_vec *lv, -xfs_lsn_t commit_lsn,bool aborted); +void xfs_trans_committed_bulk (struct xfs_ail *ailp, +struct list_head *lv_chain,xfs_lsn_t commit_lsn,bool aborted);/* * AIL traversal cursor. *
Signature · 1 line
2.28.0
PATCH 11/12 xfs: __percpu_counter_compare() inode count debug too expensive
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
- 21.92% __xfs_trans_commit
- 21.62% xfs_log_commit_cil
- 11.69% xfs_trans_unreserve_and_mod_sb
- 11.58% __percpu_counter_compare
- 11.45% __percpu_counter_sum
- 10.29% _raw_spin_lock_irqsave
- 10.28% do_raw_spin_lock __pv_queued_spin_lock_slowpath
We debated just getting rid of it last time this came up and there was no real objection to removing it. Now it's the biggest scalability limitation for debug kernels even on smallish machines, so let's just get rid of it.
--- fs/xfs/xfs_trans.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-)
fs/xfs/xfs_trans.c +2 -9
@@ -614,19 +614,12 @@ xfs_trans_unreserve_and_mod_sb(ASSERT (!error); } -if (idelta) { +if (idelta)percpu_counter_add_batch (&mp->m_icount, idelta,XFS_ICOUNT_BATCH ); -if (idelta <0 ) -ASSERT (__percpu_counter_compare (&mp->m_icount,0 , -XFS_ICOUNT_BATCH ) >=0 ); - } -if (ifreedelta) { +if (ifreedelta)percpu_counter_add (&mp->m_ifree, ifreedelta); -if (ifreedelta <0 ) -ASSERT (percpu_counter_compare (&mp->m_ifree,0 ) >=0 ); - }if (rtxdelta ==0 && !(tp->t_flags &XFS_TRANS_SB_DIRTY ))return ;
Signature · 1 line
2.28.0
PATCH 12/12 xfs: avoid cil push lock if possible
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Because now it hurts when the CIL fills up.
- 37.20% __xfs_trans_commit
- 35.84% xfs_log_commit_cil
- 19.34% _raw_spin_lock
- do_raw_spin_lock
19.01% __pv_queued_spin_lock_slowpath
- 4.20% xfs_log_ticket_ungrant
0.90% xfs_log_space_wake
--- fs/xfs/xfs_log_cil.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
fs/xfs/xfs_log_cil.c +11 -3
@@ -1110,10 +1110,18 @@ xlog_cil_push_background(ASSERT (!test_bit (XLOG_CIL_EMPTY , &cil->xc_flags));/* -* Don't do a background push if we haven't used up all the -* space available yet. +* We are done if: +* - we haven't used up all the space available yet; or +* - we've already queued up a push; and +* - we're not over the hard limit; and +* - nothing has been over the hard limit. +* +* If so, we don't need to take the push lock as there's nothing to do. */ -if (space_used <XLOG_CIL_SPACE_LIMIT (log)) { +if (space_used <XLOG_CIL_SPACE_LIMIT (log) || + (cil->xc_push_seq == cil->xc_current_sequence && + space_used <XLOG_CIL_BLOCKING_SPACE_LIMIT (log) && + !waitqueue_active (&cil->xc_push_wait))) {up_read (&cil->xc_ctx_lock);return ; }
Signature · 1 line
2.28.0