NFSv4/flexfiles: scale layoutstats for striped layouts
Subsystems (2)
cover NFSv4/flexfiles: scale layoutstats for striped layouts
Hi,
Since striped flexfiles layouts were introduced, each stripe of a mirror carries its own read_stat and write_stat, but the lock protecting them stayed in the mirror. For a RAID0-style layout (one mirror, many stripes) that lock is effectively per-file: every DS RPC takes it twice, once at prepare and once at completion, so striping multiplies RPC parallelism without multiplying lock parallelism.
On a 64-core client reading one file from a 32-stripe layout with a 128KiB stripe unit (~36.7 GB/s, ~275k NFSv3 READ RPCs/s), perf reports 45% of all CPU in native_queued_spin_lock_slowpath, and 73% of the lock contention events come from the two layoutstats paths.
This series removes that bottleneck and, along the way, fixes what the server is told in LAYOUTSTATS for a striped mirror.
Patches 1-2 (Jonathan) move the layoutstats lock into the stripe, next to the statistics it protects, and remove the now-unused mirror lock.
Patches 3-8 keep the per-stripe hot path to the stripe's own cachelines:
3: cacheline align the per-stripe statistics so neighbouring stripes
do not false-share
4: make dss_info->start_time write-once (try_cmpxchg64), removing its
dependence on any lock
5: drop the redundant atomic from the busy timer
6: pair the in-flight count with the layoutstats updates, under the
same lock, with lockdep assertions
7: derive ffil_ops_requested rather than storing it
8: give each direction its own lock and cacheline, so an I/O dirties
one line (the stripe shrinks from 384 to 320 bytes)
Patches 9-10 fix LAYOUTSTATS reporting for striped mirrors:
9: drop NFS4_FF_MIRROR_STAT_AVAIL. The per-mirror bit cost two atomic
RMWs per RPC on the mirror's shared cacheline, and since the first stripe reported cleared it, a report carried only one ff_layoutupdate4 per mirror: always the lowest usable dss_id.
10: rotate reporting across a mirror's stripes. Callers cap a report
at four entries; resume each mirror's scan where the previous report stopped, so every busy stripe is reported within ceil(busy_stripes / 4) report intervals instead of only the first four ever being reported.
Patch 11 takes the report decision (pointer chases into the mirror and layout, ktime_to_ms(), the last_report_time cmpxchg) out of the critical section, leaving the lock to cover two writes on one cacheline.
The wire encoding is unchanged; the values reported remain cumulative since the layout was granted.
The series is based on Anna's linux-next (nfs-for-7.4-1).
Thanks,
Mike
Jonathan Flynn (2):
NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock NFSv4/flexfiles: drop the now-unused per-mirror lock
Mike Snitzer (9):
NFSv4/flexfiles: cacheline align the per-stripe layoutstats NFSv4/flexfiles: make dss_info->start_time write-once NFSv4/flexfiles: drop the redundant atomic from the busy timer NFSv4/flexfiles: pair the in-flight count with the layoutstats updates NFSv4/flexfiles: derive ffil_ops_requested rather than storing it NFSv4/flexfiles: give each direction its own lock and cacheline NFSv4/flexfiles: drop NFS4_FF_MIRROR_STAT_AVAIL NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes NFSv4/flexfiles: take the report decision out of the critical section
fs/nfs/flexfilelayout/flexfilelayout.c | 251 ++++++++++++++++--------- fs/nfs/flexfilelayout/flexfilelayout.h | 74 ++++++-- 2 files changed, 218 insertions(+), 107 deletions(-)
base-commit: ef30b3efb7e8bae94d790a6503a595e1051323e7
Signature · 1 line
2.52.0
PATCH 1/11 NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock
From: Jonathan Flynn <[email protected]>
Since striped layouts were introduced, a mirror owns an array of struct nfs4_ff_layout_ds_stripe and each stripe carries its own read_stat and write_stat. The lock protecting those statistics stayed behind in the mirror, so every DS RPC in the layout serialises on the same spinlock twice: once in ff_layout_{read,write}_prepare_common() and again on completion in ff_layout_{read,write}_count_stats().
For a RAID0-style layout (one mirror, many stripes) that lock is effectively per-file, and striping then multiplies RPC parallelism without multiplying lock parallelism.
On a 64-core client reading a single file from a 32-stripe layout with a 128KiB stripe unit (2MB O_DIRECT reads, ~36.7 GB/s, ~275k NFSv3 READ RPCs/s, i.e. ~550k acquisitions/s of one lock), perf reports 45% of all CPU in native_queued_spin_lock_slowpath, and lock:contention_begin attributes 73% of the contention events to the two layoutstats paths:
36.4% rpc_exit_task -> ff_layout_read_count_stats ->
ff_layout_read_record_layoutstats_done -> _raw_spin_lock
36.3% ff_layout_read_prepare_v3 ->
ff_layout_read_record_layoutstats_start -> _raw_spin_lockMove the lock into struct nfs4_ff_layout_ds_stripe, next to the statistics it protects, so that I/O to different stripes no longer contends. The reporting paths take the same per-stripe lock.
NFS4_FF_MIRROR_STAT_AVAIL is a mirror-wide bit operation and does not need the lock; set it outside the critical section so the mirror's cacheline is not written under it.
ffl->last_report_time is shared by every mirror and stripe of a layout and was already updated under different mirror locks, so it was never serialised. Make that explicit with a cmpxchg(): whichever racer wins sends the LAYOUTSTATS report.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 45 +++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 ++ 2 files changed, 28 insertions(+), 19 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +26 -19
@@ -293,8 +293,10 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count,return NULL ; } -for (u32 dss_id =0 ; dss_id < mirror->dss_count; dss_id++) +for (u32 dss_id =0 ; dss_id < mirror->dss_count; dss_id++) { +spin_lock_init (&mirror->dss[dss_id].lock);nfs_localio_file_init (&mirror->dss[dss_id].nfl); + }return mirror; }
@@ -736,6 +738,7 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, {s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;struct nfs4_flexfile_layout *ffl =FF_LAYOUT_FROM_HDR (mirror->layout); +ktime_t last_report;nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now);if (!mirror->dss[dss_id].start_time)
@@ -744,13 +747,17 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, report_interval = (s64 )mirror->report_interval *1000LL ; else if (layoutstats_timer != 0) report_interval = (s64 )layoutstats_timer *1000LL ; -if (ktime_to_ms (ktime_sub (now, ffl->last_report_time)) >= - report_interval) { - ffl->last_report_time = now; -return true; - } -return false; +/* +* last_report_time is shared by every mirror and stripe of this +* layout, so it was never serialised by the mirror lock either. +* Make that explicit: whoever wins the cmpxchg() reports. +*/ + last_report =READ_ONCE (ffl->last_report_time); +if (ktime_to_ms (ktime_sub (now, last_report)) < report_interval) +return false; + +return try_cmpxchg64 (&ffl->last_report_time, &last_report, now); } static void
@@ -794,13 +801,13 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode, {bool report; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock); report =nfs4_ff_layoutstat_start_io ( mirror, dss_id, &mirror->dss[dss_id].read_stat, now);nfs4_ff_layout_stat_io_update_requested ( &mirror->dss[dss_id].read_stat, requested); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());
@@ -813,12 +820,12 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, __u64 requested, __u64 completed) { -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock);nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].read_stat, requested, completed,ktime_get (), task->tk_start); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock); } static void
@@ -829,7 +836,7 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode, {bool report; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock); report = nfs4_ff_layoutstat_start_io( mirror, dss_id,
@@ -838,8 +845,8 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,nfs4_ff_layout_stat_io_update_requested ( &mirror->dss[dss_id].write_stat, requested); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());
@@ -856,11 +863,11 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock);nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].write_stat, requested, completed,ktime_get (), task->tk_start); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock); } static struct nfs4_ff_layout_ds *
@@ -3071,13 +3078,13 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr, p =xdr_reserve_space (xdr,4 + fh->size);xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ -spin_lock (&dss_info->mirror->lock); +spin_lock (&dss_info->lock);ff_layout_encode_io_latency (xdr, &dss_info->read_stat.io_stat);/* ff_io_latency4 write */ ff_layout_encode_io_latency (xdr, &dss_info->write_stat.io_stat); -spin_unlock (&dss_info->mirror->lock); +spin_unlock (&dss_info->lock);/* nfstime4 */ ff_layout_encode_nfstime(xdr, ktime_sub(ktime_get(),
@@ -3153,7 +3160,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&mirror->lock); +spin_lock (&dss_info->lock); devinfo->read_count = dss_info->read_stat.io_stat.ops_completed; devinfo->read_bytes =
@@ -3162,7 +3169,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, dss_info->write_stat.io_stat.ops_completed; devinfo->write_bytes = dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&mirror->lock); +spin_unlock (&dss_info->lock); devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops; priv->dss_info = dss_info;
fs/nfs/flexfilelayout/flexfilelayout.h +2 -0
@@ -86,6 +86,8 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; +/* Protects read_stat, write_stat and start_time below */ +spinlock_t lock;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat;ktime_t start_time;
Signature · 1 line
2.52.0
PATCH 2/11 NFSv4/flexfiles: drop the now-unused per-mirror lock
In reply to Mike SnitzerFrom: Jonathan Flynn <[email protected]>
With layoutstats accounting moved to the per-stripe lock, nothing takes mirror->lock any more. Remove it.
The only users of mirror->lock in this tree were the layoutstats paths the previous patch converted.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 1 - fs/nfs/flexfilelayout/flexfilelayout.h | 1 - 2 files changed, 2 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +0 -1
@@ -280,7 +280,6 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count,if (mirror ==NULL )return NULL ; -spin_lock_init (&mirror->lock);refcount_set (&mirror->ref,1 );INIT_LIST_HEAD (&mirror->mirrors);
fs/nfs/flexfilelayout/flexfilelayout.h +0 -1
@@ -99,7 +99,6 @@ struct nfs4_ff_layout_mirror {u32 dss_count;struct nfs4_ff_layout_ds_stripe *dss;refcount_t ref; -spinlock_t lock;unsigned long flags;u32 report_interval; };
Signature · 1 line
2.52.0
PATCH 3/11 NFSv4/flexfiles: cacheline align the per-stripe layoutstats
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
Moving the layoutstats lock into struct nfs4_ff_layout_ds_stripe stopped stripes serialising on one spinlock, but it left them sharing cachelines with each other.
The lock and the statistics it guards sat at the tail of a struct whose sizeof() was 296 and whose alignof() was only 8, so in mirror->dss[] stripe i began at i * 296. 296 mod 64 = 40, so every stripe started 40 bytes further into a line than the last and no stripe's lock was line aligned. Acquiring stripe i's lock therefore invalidated the line holding stripe i+1's lock and counters. A RAID0-style layout -- one mirror, many stripes -- is exactly the case the per-stripe lock was introduced for, so this false sharing ate into the win.
Align the first member of the guarded set. That does two things: the lock now starts on a line boundary, so the guarded set occupies lines of its own and shares none with the read-mostly members above; and it raises the struct's alignment to 64, which forces sizeof() up to a multiple of 64 and makes the array stride line aligned. The stride is what actually separates stripe i from stripe i+1.
Verified with pahole on flexfilelayout.o built with DWARF5. The layout is identical on x86_64, arm64 4K pages and arm64 64K pages -- cachelines are 64 bytes on both architectures, so page size does not enter into it -- and was taken with neither CONFIG_DEBUG_SPINLOCK nor CONFIG_PROVE_LOCKING, both of which fatten spinlock_t and change it:
/* --- cacheline 3 boundary (192 bytes) --- */ spinlock_t lock __aligned(64); /* 192 4 */ /* XXX 4 bytes hole */ struct nfs4_ff_layoutstat read_stat; /* 200 72 */ struct nfs4_ff_layoutstat write_stat; /* 272 72 */ ktime_t start_time; /* 344 8 */ /* size: 384, cachelines: 6 */ } __attribute__((__aligned__(64)));
A read RPC now dirties two lines of its own stripe and a write three, and neither disturbs a neighbouring stripe. Later patches give each direction a line of its own.
Cost is 296 -> 384 bytes per stripe, i.e. 2816 bytes for a 32 stripe layout, which is negligible beside the rest of the per-file layout state. 384 is also a multiple of 128, so the stride holds for implementations with a 128 byte coherency granule.
Note that the alignment attribute only orders the stripes relative to the start of mirror->dss[], not that array's own base. The base is aligned too, under either allocator kvmalloc() may pick for it:
- From the slab. kmalloc() documents a size based alignment guarantee for power-of-two sizes only, but the smallest possible request is now 384 bytes, and every kmalloc bucket at or above that is a power of two whose objects SLUB lays out at multiples of the object size from a page aligned slab, so the base is at least 512 byte aligned. The exceptions, kmalloc-96 and kmalloc-192, are both smaller than one stripe and cannot be selected.
- From vmalloc. A stripe array wide enough to need a high order allocation is served by the vmalloc() fallback instead, which returns page aligned memory.
Either is comfortably more than the 64 bytes this needs.
--- fs/nfs/flexfilelayout/flexfilelayout.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.h +12 -2
@@ -14,6 +14,7 @@#define FF_FLAGS_NO_IO_THRU_MDS 2#define FF_FLAGS_NO_READ_IO 4 +#include <linux/cache.h> #include <linux/refcount.h> #include "../pnfs.h"
@@ -86,8 +87,17 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; -/* Protects read_stat, write_stat and start_time below */ -spinlock_t lock; +/* +* Protects read_stat, write_stat and start_time below. +* +* Cacheline aligned so that the layoutstats accounting done on +* every I/O to this stripe does not share cachelines with the +* read-mostly members above, nor with the neighbouring stripes +* in mirror->dss[]: aligning the first member of the guarded +* set also rounds sizeof() up to a multiple of the cacheline, +* which is what makes the array stride line-aligned. +*/ +spinlock_t lock____cacheline_aligned_in_smp ;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat;ktime_t start_time;
Signature · 1 line
2.52.0
PATCH 4/11 NFSv4/flexfiles: make dss_info->start_time write-once
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
start_time was written under the stripe lock in nfs4_ff_layoutstat_start_io() and read outside it in ff_layout_encode_ff_layoutupdate(). Pre-existing -- mirror->lock had the same shape -- and benign, since it is a single naturally aligned 8 byte value, but it is an unannotated race either way.
The lock is the wrong tool for it. ffl_duration is measured from this stripe's first I/O, so the field is written once and read-only for the rest of the stripe's life. Say that instead: publish it with try_cmpxchg64() and let the first I/O win, and read it with READ_ONCE(). Only the first I/O to a stripe pays the read-modify-write; every one after it short circuits on the READ_ONCE().
ffl->last_report_time two lines away is already published this way, so the pattern is not new to this file.
This also stops start_time depending on there being exactly one lock per stripe, which a later patch changes.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 21 ++++++++++++++++++--- fs/nfs/flexfilelayout/flexfilelayout.h | 3 ++- 2 files changed, 20 insertions(+), 4 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +18 -3
@@ -729,6 +729,22 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now)return ktime_sub (now, start); } +/* +* ffl_duration is measured from this stripe's first I/O, so start_time is +* written once and read-only for the rest of the stripe's life. Publish it +* with a cmpxchg and let the first I/O win, rather than depending on a lock +* that every reader would then have to take. +*/ +static void +nfs4_ff_layoutstat_set_start_time (struct nfs4_ff_layout_ds_stripe *dss_info, +ktime_t now) +{ +ktime_t unset =0 ; + +if (!READ_ONCE (dss_info->start_time)) +try_cmpxchg64 (&dss_info->start_time, &unset, now); +} + static bool nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, u32 dss_id,
@@ -740,8 +756,7 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror,ktime_t last_report;nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); -if (!mirror->dss[dss_id].start_time) - mirror->dss[dss_id].start_time = now; +nfs4_ff_layoutstat_set_start_time (&mirror->dss[dss_id], now);if (mirror->report_interval !=0 ) report_interval = (s64 )mirror->report_interval *1000LL ;else if (layoutstats_timer !=0 )
@@ -3087,7 +3102,7 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr,/* nfstime4 */ ff_layout_encode_nfstime (xdr,ktime_sub (ktime_get (), - dss_info->start_time)); +READ_ONCE (dss_info->start_time)));/* bool */ p =xdr_reserve_space (xdr,4 ); *p =cpu_to_be32 (false);
fs/nfs/flexfilelayout/flexfilelayout.h +2 -1
@@ -88,7 +88,7 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *rw_cred;struct nfs_file_localio nfl;/* -* Protects read_stat, write_stat and start_time below. +* Protects read_stat and write_stat below. * * Cacheline aligned so that the layoutstats accounting done on * every I/O to this stripe does not share cachelines with the
@@ -100,6 +100,7 @@ struct nfs4_ff_layout_ds_stripe {spinlock_t lock____cacheline_aligned_in_smp ;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat; +/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */ ktime_t start_time; };
Signature · 1 line
2.52.0
PATCH 5/11 NFSv4/flexfiles: drop the redundant atomic from the busy timer
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
nfs4_ff_busy_timer::n_ops is an atomic_t, but every access to it is already serialised by the stripe lock:
nfs4_ff_start_busy_timer() has one caller, nfs4_ff_layoutstat_start_io(), reached only from ff_layout_{read,write}_record_layoutstats_start() with dss[dss_id].lock held.
nfs4_ff_end_busy_timer() has one caller, nfs4_ff_layout_stat_io_update_completed(), reached only from ff_layout_{read,write}_record_layoutstats_done() with the same lock held.
The two reporting paths, ff_layout_mirror_prepare_stats() and ff_layout_encode_ff_layoutupdate(), take the stripe lock too and do not touch n_ops at all. There is no lockless reader.
So the atomic_inc_return()/atomic_dec_return() are locked read-modify- writes on a line the CPU already owns exclusively -- pure serialisation overhead, paid twice per DS RPC on the hottest path in this file, at the 275k-450k RPC/s the per-stripe lock work was measured at.
Make n_ops a plain int and document the locking on the struct. Its size is unchanged, so the stripe layout is unaffected. The underflow check is preserved.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 6 ++---- fs/nfs/flexfilelayout/flexfilelayout.h | 6 +++++- 2 files changed, 7 insertions(+), 5 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +2 -4
@@ -711,9 +711,8 @@ static void nfs4_ff_start_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {/* first IO request? */ -if (atomic_inc_return (&timer->n_ops) ==1 ) { +if (++timer->n_ops ==1 ) timer->start_time = now; - } } static ktime_t
@@ -721,8 +720,7 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {ktime_t start; -if (atomic_dec_return (&timer->n_ops) <0 ) -WARN_ON_ONCE (1 ); +WARN_ON_ONCE (--timer->n_ops <0 ); start = timer->start_time; timer->start_time = now;
fs/nfs/flexfilelayout/flexfilelayout.h +5 -1
@@ -64,9 +64,13 @@ struct nfs4_ff_io_stat {ktime_t aggregate_completion_time; }; +/* +* Both members are serialised by the enclosing stripe's +* nfs4_ff_layout_ds_stripe::lock; see nfs4_ff_{start,end}_busy_timer(). +*/ struct nfs4_ff_busy_timer {ktime_t start_time; -atomic_t n_ops; +int n_ops; };struct nfs4_ff_layoutstat {
Signature · 1 line
2.52.0
PATCH 6/11 NFSv4/flexfiles: pair the in-flight count with the layoutstats updates
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
The busy timer's in-flight op count and the requested-side statistics are two halves of the same accounting, but they lived in two helpers that the callers invoked back to back. Nothing stopped a new requested-side path from updating one and missing the other.
Fold nfs4_ff_layout_stat_io_update_requested() into nfs4_ff_layoutstat_start_io() so both happen in one place, and rename nfs4_ff_layout_stat_io_update_completed() to nfs4_ff_layoutstat_end_io() for symmetry -- it already dropped the in-flight count alongside the completed-side statistics.
Rename nfs4_ff_busy_timer::n_ops to ops_in_flight while here. It is what the field holds, and it makes its relationship to the op counters legible at the point of use.
Both helpers now take the stripe rather than mirror + dss_id, which lets them assert the lock they run under. Since the previous patch made ops_in_flight a plain int, that lock is the only thing keeping it consistent, so assert it rather than document it; lockdep_assert_held() compiles out without CONFIG_PROVE_LOCKING. The callers get a dss_info local, which is what they were open coding.
No functional change.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 104 ++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 +- 2 files changed, 58 insertions(+), 48 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +57 -47
@@ -711,7 +711,7 @@ static void nfs4_ff_start_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {/* first IO request? */ -if (++timer->n_ops ==1 ) +if (++timer->ops_in_flight ==1 ) timer->start_time = now; }
@@ -720,7 +720,7 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {ktime_t start; -WARN_ON_ONCE (--timer->n_ops <0 ); +WARN_ON_ONCE (--timer->ops_in_flight <0 ); start = timer->start_time; timer->start_time = now;
@@ -743,18 +743,31 @@ nfs4_ff_layoutstat_set_start_time(struct nfs4_ff_layout_ds_stripe *dss_info,try_cmpxchg64 (&dss_info->start_time, &unset, now); } +/* +* Account for an I/O about to be sent to this stripe, and return true if a +* LAYOUTSTATS report is due. +* +* The in-flight count and the requested-side counters are bumped here +* together, so a new requested-side path cannot update one and miss the +* other. +*/ static bool -nfs4_ff_layoutstat_start_io (struct nfs4_ff_layout_mirror *mirror, -u32 dss_id, +nfs4_ff_layoutstat_start_io (struct nfs4_ff_layout_ds_stripe *dss_info,struct nfs4_ff_layoutstat *layoutstat, -ktime_t now) +__u64 requested,ktime_t now) { -s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ; +struct nfs4_ff_layout_mirror *mirror = dss_info->mirror;struct nfs4_flexfile_layout *ffl =FF_LAYOUT_FROM_HDR (mirror->layout); +s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;ktime_t last_report; +lockdep_assert_held (&dss_info->lock); +nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); -nfs4_ff_layoutstat_set_start_time (&mirror->dss[dss_id], now); + layoutstat->io_stat.ops_requested++; + layoutstat->io_stat.bytes_requested += requested; + +nfs4_ff_layoutstat_set_start_time (dss_info, now);if (mirror->report_interval !=0 ) report_interval = (s64 )mirror->report_interval *1000LL ;else if (layoutstats_timer !=0 )
@@ -772,27 +785,25 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror,return try_cmpxchg64 (&ffl->last_report_time, &last_report, now); } +/* +* Account for an I/O that has completed on this stripe. The in-flight count +* is dropped here, alongside the completed-side counters, for the same reason +* it is bumped in nfs4_ff_layoutstat_start_io(). +*/ static void -nfs4_ff_layout_stat_io_update_requested (struct nfs4_ff_layoutstat *layoutstat, -__u64 requested) -{ -struct nfs4_ff_io_stat *iostat = &layoutstat->io_stat; - - iostat->ops_requested++; - iostat->bytes_requested += requested; -} - -static void -nfs4_ff_layout_stat_io_update_completed (struct nfs4_ff_layoutstat *layoutstat, -__u64 requested, -__u64 completed, -ktime_t time_completed, -ktime_t time_started) +nfs4_ff_layoutstat_end_io (struct nfs4_ff_layout_ds_stripe *dss_info, +struct nfs4_ff_layoutstat *layoutstat, +__u64 requested, +__u64 completed, +ktime_t time_completed, +ktime_t time_started) {struct nfs4_ff_io_stat *iostat = &layoutstat->io_stat;ktime_t completion_time =ktime_sub (time_completed, time_started);ktime_t timer; +lockdep_assert_held (&dss_info->lock); + iostat->ops_completed++; iostat->bytes_completed += completed; iostat->bytes_not_delivered += requested - completed;
@@ -811,14 +822,13 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,u32 dss_id, __u64 requested, ktime_t now) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&mirror->dss[dss_id].lock); - report =nfs4_ff_layoutstat_start_io ( - mirror, dss_id, &mirror->dss[dss_id].read_stat, now); -nfs4_ff_layout_stat_io_update_requested ( - &mirror->dss[dss_id].read_stat, requested); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); + report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, + requested, now); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)
@@ -832,11 +842,13 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, __u64 requested, __u64 completed) { -spin_lock (&mirror->dss[dss_id].lock); -nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].read_stat, - requested, completed, -ktime_get (), task->tk_start); -spin_unlock (&mirror->dss[dss_id].lock); +struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; + +spin_lock (&dss_info->lock); +nfs4_ff_layoutstat_end_io (dss_info, &dss_info->read_stat, + requested, completed, +ktime_get (), task->tk_start); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }
@@ -846,18 +858,13 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,u32 dss_id, __u64 requested, ktime_t now) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&mirror->dss[dss_id].lock); - report =nfs4_ff_layoutstat_start_io ( - mirror, - dss_id, - &mirror->dss[dss_id].write_stat, - now); -nfs4_ff_layout_stat_io_update_requested ( - &mirror->dss[dss_id].write_stat, - requested); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); + report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, + requested, now); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)
@@ -872,13 +879,16 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,__u64 completed, enum nfs3_stable_how committed) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; +if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&mirror->dss[dss_id].lock); -nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].write_stat, - requested, completed,ktime_get (), task->tk_start); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); +nfs4_ff_layoutstat_end_io (dss_info, &dss_info->write_stat, + requested, completed, +ktime_get (), task->tk_start); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }
fs/nfs/flexfilelayout/flexfilelayout.h +1 -1
@@ -70,7 +70,7 @@ struct nfs4_ff_io_stat {*/ struct nfs4_ff_busy_timer {ktime_t start_time; -int n_ops; +int ops_in_flight; };struct nfs4_ff_layoutstat {
Signature · 1 line
2.52.0
PATCH 7/11 NFSv4/flexfiles: derive ffil_ops_requested rather than storing it
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
nfs4_ff_io_stat::ops_requested carries no information that the stripe does not already hold. An op is in flight from the moment it is counted as requested until it is counted as completed, and the previous patch put both halves of that accounting in one place each, so
ops_requested == ops_completed + ops_in_flight
holds at all times under the lock.
Drop the field and compute the value at encode time. It had exactly one producer and one consumer, the LAYOUTSTATS encoder, so nothing else needs to change and the wire format is untouched -- ffil_ops_requested is still a uint64, and nfs4_ff_ops_requested() returns __u64. ff_layout_encode_io_latency() now takes the whole nfs4_ff_layoutstat, since the in-flight count lives beside the statistics rather than in them.
This takes 8 bytes off each direction's statistics, which is what lets each direction fit in a cacheline of its own; see the next patch.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 12 +++++------- fs/nfs/flexfilelayout/flexfilelayout.h | 19 ++++++++++++++++++- 2 files changed, 23 insertions(+), 8 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +5 -7
@@ -764,7 +764,6 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_ds_stripe *dss_info,lockdep_assert_held (&dss_info->lock);nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); - layoutstat->io_stat.ops_requested++; layoutstat->io_stat.bytes_requested += requested;nfs4_ff_layoutstat_set_start_time (dss_info, now);
@@ -3066,12 +3065,13 @@ ff_layout_encode_nfstime(struct xdr_stream *xdr,static void ff_layout_encode_io_latency (struct xdr_stream *xdr, -struct nfs4_ff_io_stat *stat) +const struct nfs4_ff_layoutstat *layoutstat) { +const struct nfs4_ff_io_stat *stat = &layoutstat->io_stat;__be32 *p; p =xdr_reserve_space (xdr,5 *8 ); - p =xdr_encode_hyper (p, stat->ops_requested); + p =xdr_encode_hyper (p,nfs4_ff_ops_requested (layoutstat)); p =xdr_encode_hyper (p, stat->bytes_requested); p =xdr_encode_hyper (p, stat->ops_completed); p =xdr_encode_hyper (p, stat->bytes_completed);
@@ -3101,11 +3101,9 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr,xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ spin_lock (&dss_info->lock); -ff_layout_encode_io_latency (xdr, - &dss_info->read_stat.io_stat); +ff_layout_encode_io_latency (xdr, &dss_info->read_stat);/* ff_io_latency4 write */ -ff_layout_encode_io_latency (xdr, - &dss_info->write_stat.io_stat); +ff_layout_encode_io_latency (xdr, &dss_info->write_stat);spin_unlock (&dss_info->lock);/* nfstime4 */ ff_layout_encode_nfstime(xdr,
fs/nfs/flexfilelayout/flexfilelayout.h +18 -1
@@ -55,7 +55,6 @@ struct nfs4_ff_layout_ds_err { };struct nfs4_ff_io_stat { -__u64 ops_requested;__u64 bytes_requested;__u64 ops_completed;__u64 bytes_completed;
@@ -78,6 +77,24 @@ struct nfs4_ff_layoutstat {struct nfs4_ff_busy_timer busy_timer; }; +/* +* ffil_ops_requested is not stored. An op is in flight from the moment it is +* counted as requested until it is counted as completed, so +* +* ops_requested == ops_completed + ops_in_flight +* +* holds by construction; see nfs4_ff_layoutstat_start_io() and +* nfs4_ff_layoutstat_end_io(), which maintain both halves together. +* +* Caller must hold the lock guarding this nfs4_ff_layoutstat. +*/ +static inline __u64 +nfs4_ff_ops_requested (const struct nfs4_ff_layoutstat *layoutstat) +{ +return layoutstat->io_stat.ops_completed + + layoutstat->busy_timer.ops_in_flight; +} +struct nfs4_ff_layout_mirror ;struct nfs4_ff_layout_ds_stripe {
Signature · 1 line
2.52.0
PATCH 8/11 NFSv4/flexfiles: give each direction its own lock and cacheline
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
struct nfs4_ff_layout_ds_stripe shrinks from 384 bytes to 320, and an I/O to a stripe now dirties one cacheline instead of two. The 384 was what "NFSv4/flexfiles: cacheline align the per-stripe layoutstats" had to pay to get the array stride line aligned; this gets that back and takes another 64 bytes off, on top of removing the last shared lock between the two I/O directions.
Cacheline aligning the stripe's lock stopped stripes sharing lines with each other, but within a stripe an I/O still dirtied more than one line, and reads and writes still serialised on one spinlock.
The set one RPC touches is not the whole guarded region. A read takes the lock, bumps the in-flight count and updates every field of read_stat -- bytes_requested and the busy timer on the way out, ops_completed, bytes_completed, bytes_not_delivered and the two ktime_t sums on completion -- while write_stat stays untouched, and a write is the mirror image. The two directions share nothing but the lock.
So stop sharing the lock. Move it into struct nfs4_ff_layoutstat, alongside the in-flight count and the statistics it guards, and fold nfs4_ff_busy_timer in as a bare ktime_t. With ops_requested derived rather than stored, that comes to
spinlock_t lock 4 int ops_in_flight 4
struct nfs4_ff_io_stat io_stat 48
ktime_t busy_start_time 8
which is 64 bytes with no padding. Align the structure to a cacheline and each direction occupies one, exactly.
An I/O now dirties a single line, and a read and a write to the same stripe touch neither the same lock nor the same line. The two reporting paths, ff_layout_mirror_prepare_stats() and ff_layout_encode_ff_layoutupdate(), take each lock in turn; they need no atomicity across the pair, as the statistics are advisory and were never sampled atomically with respect to in-flight I/O anyway.
pahole on flexfilelayout.o built with DWARF5, identical on x86_64, arm64 4K pages and arm64 64K pages:
struct nfs4_ff_layoutstat {
spinlock_t lock; /* 0 4 */
int ops_in_flight; /* 4 4 */
struct nfs4_ff_io_stat io_stat; /* 8 48 */
ktime_t busy_start_time; /* 56 8 */
/* size: 64, cachelines: 1, members: 4 */
} __attribute__((__aligned__(64))); ktime_t start_time; /* 136 8 */
/* XXX 48 bytes hole */
/* --- cacheline 3 boundary (192 bytes) --- */
struct nfs4_ff_layoutstat read_stat; /* 192 64 */
/* --- cacheline 4 boundary (256 bytes) --- */
struct nfs4_ff_layoutstat write_stat; /* 256 64 */
/* size: 320, cachelines: 5 */
start_time stays out of both lines, since it neither fits nor needs to. "NFSv4/flexfiles: make dss_info->start_time write-once" removed its dependence on any lock, and nothing writes it after the stripe's first I/O, so its line never bounces; the same I/O already reads fh_versions, stateid and the creds from the lines around it. In the hole ahead of read_stat it is free.
The array stride stays a multiple of 64 and of 128, so stripes remain separated on implementations with a 128 byte coherency granule.
Note that CONFIG_DEBUG_SPINLOCK or CONFIG_PROVE_LOCKING fatten spinlock_t enough to push nfs4_ff_layoutstat to 128 bytes and the stripe to 448. Correctness is unaffected; such kernels simply do not get the packing.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 53 ++++++++++++++------------ fs/nfs/flexfilelayout/flexfilelayout.h | 46 +++++++++++----------- 2 files changed, 52 insertions(+), 47 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +29 -24
@@ -293,7 +293,8 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count, }for (u32 dss_id =0 ; dss_id < mirror->dss_count; dss_id++) { -spin_lock_init (&mirror->dss[dss_id].lock); +spin_lock_init (&mirror->dss[dss_id].read_stat.lock); +spin_lock_init (&mirror->dss[dss_id].write_stat.lock);nfs_localio_file_init (&mirror->dss[dss_id].nfl); }
@@ -708,22 +709,22 @@ static u32 calc_dss_id_from_commit(struct pnfs_layout_segment *lseg, }static void -nfs4_ff_start_busy_timer (struct nfs4_ff_busy_timer *timer,ktime_t now) +nfs4_ff_start_busy_timer (struct nfs4_ff_layoutstat *layoutstat,ktime_t now) {/* first IO request? */ -if (++timer->ops_in_flight ==1 ) - timer->start_time = now; +if (++layoutstat->ops_in_flight ==1 ) + layoutstat->busy_start_time = now; }static ktime_t -nfs4_ff_end_busy_timer (struct nfs4_ff_busy_timer *timer,ktime_t now) +nfs4_ff_end_busy_timer (struct nfs4_ff_layoutstat *layoutstat,ktime_t now) {ktime_t start; -WARN_ON_ONCE (--timer->ops_in_flight <0 ); +WARN_ON_ONCE (--layoutstat->ops_in_flight <0 ); - start = timer->start_time; - timer->start_time = now; + start = layoutstat->busy_start_time; + layoutstat->busy_start_time = now;return ktime_sub (now, start); }
@@ -761,9 +762,9 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_ds_stripe *dss_info,s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;ktime_t last_report; -lockdep_assert_held (&dss_info->lock); +lockdep_assert_held (&layoutstat->lock); -nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); +nfs4_ff_start_busy_timer (layoutstat, now); layoutstat->io_stat.bytes_requested += requested;nfs4_ff_layoutstat_set_start_time (dss_info, now);
@@ -801,13 +802,13 @@ nfs4_ff_layoutstat_end_io(struct nfs4_ff_layout_ds_stripe *dss_info,ktime_t completion_time =ktime_sub (time_completed, time_started);ktime_t timer; -lockdep_assert_held (&dss_info->lock); +lockdep_assert_held (&layoutstat->lock); iostat->ops_completed++; iostat->bytes_completed += completed; iostat->bytes_not_delivered += requested - completed; - timer =nfs4_ff_end_busy_timer (&layoutstat->busy_timer, time_completed); + timer =nfs4_ff_end_busy_timer (layoutstat, time_completed); iostat->total_busy_time =ktime_add (iostat->total_busy_time, timer); iostat->aggregate_completion_time =
@@ -824,10 +825,10 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock); report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, requested, now); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->read_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)
@@ -843,11 +844,11 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, {struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock);nfs4_ff_layoutstat_end_io (dss_info, &dss_info->read_stat, requested, completed,ktime_get (), task->tk_start); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->read_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }
@@ -860,10 +861,10 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->write_stat.lock); report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, requested, now); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)
@@ -883,11 +884,11 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->write_stat.lock);nfs4_ff_layoutstat_end_io (dss_info, &dss_info->write_stat, requested, completed,ktime_get (), task->tk_start); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }
@@ -3100,11 +3101,13 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr, p =xdr_reserve_space (xdr,4 + fh->size);xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock);ff_layout_encode_io_latency (xdr, &dss_info->read_stat); +spin_unlock (&dss_info->read_stat.lock);/* ff_io_latency4 write */ +spin_lock (&dss_info->write_stat.lock);ff_layout_encode_io_latency (xdr, &dss_info->write_stat); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);/* nfstime4 */ ff_layout_encode_nfstime(xdr, ktime_sub(ktime_get(),
@@ -3180,16 +3183,18 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock); devinfo->read_count = dss_info->read_stat.io_stat.ops_completed; devinfo->read_bytes = dss_info->read_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->read_stat.lock); +spin_lock (&dss_info->write_stat.lock); devinfo->write_count = dss_info->write_stat.io_stat.ops_completed; devinfo->write_bytes = dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock); devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops; priv->dss_info = dss_info;
fs/nfs/flexfilelayout/flexfilelayout.h +23 -23
70 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH 9/11 NFSv4/flexfiles: drop NFS4_FF_MIRROR_STAT_AVAIL
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
The "this needs reporting" hint was a bit in nfs4_ff_layout_mirror::flags, set on every DS RPC and consumed once per report interval. It was wrong in both directions.
It cost too much. set_bit() ran twice per RPC -- once from nfs4_ff_layout_stat_io_start_{read,write}() and again from the matching _end_ -- on a mirror-wide cacheline that every stripe of the mirror shares. That is the same false sharing the per-stripe lock was introduced to remove, and after the preceding patches gave each direction a line of its own it was the last write to the mirror's line left on the hot path. Two unconditional atomic read-modify-writes per RPC, at the 275k-450k RPC/s this series was measured at, to set a hint read once every 60 seconds.
And it said too little. The bit is per mirror, but the statistics it gates are per stripe, and ff_layout_mirror_prepare_stats() tests it inside the per-stripe loop. The first stripe with a usable deviceid cleared it, so every other stripe of that mirror hit the test_and_clear and skipped.
What the server sees today: for a mirror with N stripes, a LAYOUTSTATS report carries exactly one ff_layoutupdate4, for the lowest usable dss_id, no matter how many stripes are busy or how the I/O is spread across them. Stripes 1..N-1 are reported only when the layout is returned. For the 32-stripe RAID0 layout this series targets, the server is picking data servers on 1/32 of the picture, and always the same 1/32 -- a stripe that is idle while the other 31 are saturated reports zeros, and a stripe that is saturated while stripe 0 is idle never reports at all.
So drop the bit rather than fix it. The reporting path already reads ops_completed for both directions under their locks; remember what it reported and skip a stripe whose count has not moved since. That is strictly more accurate than the flag was -- it is per stripe by construction, and it cannot be left set by I/O that has already been reported -- and it costs the I/O paths nothing.
What the server sees after this patch: up to dev_limit ff_layoutupdate4 entries, one per stripe that has completed I/O since its own last report, in dss_id order. The numbers themselves are unchanged -- same cumulative ops_completed and bytes_completed, same deviceid, same encoding -- there are just more of them, and they describe the stripes that did the work. A stripe with no completions since its last report is now omitted rather than reported as an unchanged repeat; a server differencing successive reports sees the same thing either way. LAYOUTRETURN still reports every stripe it can fit, as before.
nfs4_ff_layout_ds_stripe::last_reported_ops is deliberately out in the read-mostly region rather than in either nfs4_ff_layoutstat: it is touched only by ff_layout_mirror_prepare_stats(), about once per report interval, so it must not consume one of the 64 bytes an I/O dirties. It lands in the existing hole ahead of read_stat, so sizeof is still 320. Both callers of ff_layout_mirror_prepare_stats() hold the inode's i_lock, which serialises it; no new lock is needed.
NFS4_FF_MIRROR_STAT_AVAIL was the only bit ever defined on nfs4_ff_layout_mirror::flags, so the field goes too.
Note that dev_limit is 4 in both callers, so N > 4 stripes still cannot all be reported at once; the next patch rotates which ones are.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 43 +++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 12 +++++-- 2 files changed, 34 insertions(+), 21 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +25 -18
@@ -829,7 +829,6 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode, report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, requested, now);spin_unlock (&dss_info->read_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());
@@ -849,7 +848,6 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, requested, completed, ktime_get(), task->tk_start);spin_unlock (&dss_info->read_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); } static void
@@ -865,7 +863,6 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode, report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, requested, now);spin_unlock (&dss_info->write_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());
@@ -889,7 +886,6 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task, requested, completed, ktime_get(), task->tk_start);spin_unlock (&dss_info->write_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); } static struct nfs4_ff_layout_ds *
@@ -3156,6 +3152,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,struct nfs4_ff_layout_mirror *mirror;struct nfs4_ff_layout_ds_stripe *dss_info;struct nfs4_ff_layout_ds *mirror_ds; +__u64 read_count, read_bytes, write_count, write_bytes, ops;int i =0 , dss_id;rcu_read_lock ();
@@ -3167,13 +3164,31 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, mirror_ds =rcu_dereference (dss_info->mirror_ds);if (IS_ERR_OR_NULL (mirror_ds))continue ; -if (!test_and_clear_bit (NFS4_FF_MIRROR_STAT_AVAIL , - &mirror->flags) && + +spin_lock (&dss_info->read_stat.lock); + read_count = dss_info->read_stat.io_stat.ops_completed; + read_bytes = dss_info->read_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->read_stat.lock); +spin_lock (&dss_info->write_stat.lock); + write_count = dss_info->write_stat.io_stat.ops_completed; + write_bytes = dss_info->write_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->write_stat.lock); + +/* +* Nothing has completed on this stripe since it last +* reported, so do not spend one of the few devinfo +* slots on repeating those numbers. A LAYOUTRETURN is +* the last chance to report and says everything. +*/ + ops = read_count + write_count; +if (ops == dss_info->last_reported_ops && type !=NFS4_FF_OP_LAYOUTRETURN )continue ; +/* mirror refcount put in cleanup_layoutstats */ if (!refcount_inc_not_zero (&mirror->ref))continue ; + dss_info->last_reported_ops = ops;/* The pin holds a reference; it is exchanged out only * under i_lock. Put in ff_layout_free_layoutstats(). */
@@ -3183,18 +3198,10 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&dss_info->read_stat.lock); - devinfo->read_count = - dss_info->read_stat.io_stat.ops_completed; - devinfo->read_bytes = - dss_info->read_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->read_stat.lock); -spin_lock (&dss_info->write_stat.lock); - devinfo->write_count = - dss_info->write_stat.io_stat.ops_completed; - devinfo->write_bytes = - dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->write_stat.lock); + devinfo->read_count = read_count; + devinfo->read_bytes = read_bytes; + devinfo->write_count = write_count; + devinfo->write_bytes = write_bytes; devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops; priv->dss_info = dss_info;
fs/nfs/flexfilelayout/flexfilelayout.h +9 -3
@@ -113,6 +113,15 @@ struct nfs4_ff_layout_ds_stripe {struct nfs_file_localio nfl;/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */ ktime_t start_time; +/* +* Completed ops, both directions, as of this stripe's last +* LAYOUTSTATS report. Touched only by +* ff_layout_mirror_prepare_stats(), so roughly once per report +* interval, and serialised by the inode's i_lock, which both of its +* callers hold. Deliberately out here rather than in either +* nfs4_ff_layoutstat: it must not cost the I/O paths a cacheline. +*/ +__u64 last_reported_ops;/* * A line each, and each carrying its own lock, so that a read and a * write to this stripe neither serialise against each other nor
@@ -131,12 +140,9 @@ struct nfs4_ff_layout_mirror {u32 dss_count;struct nfs4_ff_layout_ds_stripe *dss;refcount_t ref; -unsigned long flags;u32 report_interval; }; -#define NFS4_FF_MIRROR_STAT_AVAIL (0) -struct nfs4_ff_layout_segment {struct pnfs_layout_segment generic_hdr;u64 stripe_unit;
Signature · 1 line
2.52.0
PATCH 10/11 NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
Both callers of ff_layout_mirror_prepare_stats() cap it at four ff_layoutupdate4 entries -- PNFS_LAYOUTSTATS_MAXDEV for LAYOUTSTATS and ARRAY_SIZE(ff_args->devinfo), which is FF_LAYOUTSTATS_MAXDEV, for LAYOUTRETURN. The scan walks mirror->dss[] from index 0 every time and stops once the budget is spent.
What the server sees without this patch: for a mirror with more than four busy stripes, the same low dss_ids every report, forever. The previous patch widened that from one stripe to four -- it did not stop it being the same four. On the 32-stripe RAID0 layout this series targets, stripes 4..31 are reported only when the layout is returned, so a data server that is only ever backing a high stripe index stays invisible to the server's placement decisions no matter how much I/O it does.
Remember where each mirror's scan stopped and resume there. The cursor advances past each stripe that actually fills a slot, so a report picks up where its predecessor left off and idle stripes, which are skipped without consuming a slot, do not stall the rotation.
What the server sees with this patch: still up to four entries per report, but rotating, so every busy stripe of a mirror is reported within ceil(busy_stripes / 4) report intervals -- eight intervals, or about eight minutes at the default 60 second interval, for 32 busy stripes. A given stripe's numbers therefore arrive less often than before but, unlike before, they arrive. The values are unchanged:
cumulative ops_completed and bytes_completed since the layout was granted, so a server differencing successive reports for a stripe still gets a correct delta, just over a longer window. Nothing about the encoding, the deviceid or the ordering within a report changes.
The mirror loop gets the same dev_limit test as the stripe loop, so a full report stops scanning instead of walking the remaining mirrors to break immediately on each.
[jonathan.flynn: read dss_report_start once per mirror, before the stripe
loop. The loop advances it after each stripe it reports, so re-reading it moved the index twice: stripes were skipped, and on a mirror with stripes not yet written (and so skipped) the index wrapped back onto a stripe already reported. That put the same deviceid in one ff_iostats4 array twice with identical counters.]
--- fs/nfs/flexfilelayout/flexfilelayout.c | 22 +++++++++++++++++++--- fs/nfs/flexfilelayout/flexfilelayout.h | 7 +++++++ 2 files changed, 26 insertions(+), 3 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +19 -3
@@ -3153,14 +3153,28 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,struct nfs4_ff_layout_ds_stripe *dss_info;struct nfs4_ff_layout_ds *mirror_ds;__u64 read_count, read_bytes, write_count, write_bytes, ops; -int i =0 , dss_id; +u32 dss_id, scanned, base; +int i =0 ;rcu_read_lock ();list_for_each_entry (mirror, &ff_layout->mirrors, mirrors) { -for (dss_id =0 ; dss_id < mirror->dss_count; ++dss_id) { - dss_info = &mirror->dss[dss_id]; +if (i >= dev_limit) +break ; + base = mirror->dss_report_start; +for (scanned =0 ; scanned < mirror->dss_count; ++scanned) {if (i >= dev_limit)break ; +/* Resume where the last report of this mirror left +* off; one subtraction suffices to wrap, as both +* terms are below dss_count. Use the start as it +* was on entry: this loop advances it for the next +* report, and re-reading it would skip stripes and +* wrap back onto one already reported. +*/ + dss_id = base + scanned; +if (dss_id >= mirror->dss_count) + dss_id -= mirror->dss_count; + dss_info = &mirror->dss[dss_id]; mirror_ds =rcu_dereference (dss_info->mirror_ds);if (IS_ERR_OR_NULL (mirror_ds))continue ;
@@ -3189,6 +3203,8 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,if (!refcount_inc_not_zero (&mirror->ref))continue ; dss_info->last_reported_ops = ops; + mirror->dss_report_start = + dss_id +1 < mirror->dss_count ? dss_id +1 :0 ;/* The pin holds a reference; it is exchanged out only * under i_lock. Put in ff_layout_free_layoutstats(). */
fs/nfs/flexfilelayout/flexfilelayout.h +7 -0
@@ -141,6 +141,13 @@ struct nfs4_ff_layout_mirror {struct nfs4_ff_layout_ds_stripe *dss;refcount_t ref;u32 report_interval; +/* +* dss[] index to begin the next LAYOUTSTATS scan of this mirror at, +* so that a mirror with more stripes than a report has room for +* rotates through them rather than always reporting the lowest. +* Serialised by the inode's i_lock, like last_reported_ops. +*/ +u32 dss_report_start; };struct nfs4_ff_layout_segment {
Signature · 1 line
2.52.0
PATCH 11/11 NFSv4/flexfiles: take the report decision out of the critical section
In reply to Mike SnitzerFrom: Mike Snitzer <[email protected]>
Now that each direction's statistics are one cacheline behind their own lock, what is left inside that lock matters. On the requested side it was not just the counters:
- dss_info->mirror and mirror->layout, two pointer chases into other objects
- mirror->report_interval, on the mirror's cacheline
- the layoutstats_timer module parameter
- ktime_to_ms(), a division
- ffl->last_report_time, on the layout header's cacheline, plus a try_cmpxchg64() on it once per report interval
- dss_info->start_time, on the stripe's read-mostly cacheline
None of that is covered by the lock, and none of it touches the data the lock guards. mirror->report_interval is written once when the layout segment is decoded and read-only afterwards; ffl->last_report_time is shared by every mirror and stripe of the layout and has carried its own cmpxchg since the statistics moved to the stripe; start_time has been write-once, published by cmpxchg, since "NFSv4/flexfiles: make dss_info->start_time write-once". So none of them were being protected, they were only being waited for.
Split the report decision out into nfs4_ff_layoutstat_report_due() and call it, and nfs4_ff_layoutstat_set_start_time(), after the unlock. The critical section becomes what it should be: bump the in-flight count, add to bytes_requested. Two writes, one cacheline, no dependent loads into other objects and no division.
The reads themselves are unchanged and still happen once per RPC. They are on read-mostly lines, so they share rather than bounce and are not worth eliminating without a profile that says otherwise; the point here is only that an I/O no longer waits behind them.
Ordering is unchanged: the statistics are updated before the decision is taken, as before, and pnfs_report_layoutstat() was already called after the unlock.
nfs4_ff_layoutstat_end_io() loses its dss_info argument, unused since the lock moved into nfs4_ff_layoutstat.
--- fs/nfs/flexfilelayout/flexfilelayout.c | 60 +++++++++++++++----------- 1 file changed, 35 insertions(+), 25 deletions(-)
fs/nfs/flexfilelayout/flexfilelayout.c +35 -25
@@ -745,39 +745,48 @@ nfs4_ff_layoutstat_set_start_time(struct nfs4_ff_layout_ds_stripe *dss_info, }/* -* Account for an I/O about to be sent to this stripe, and return true if a -* LAYOUTSTATS report is due. +* Account for an I/O about to be sent to this stripe. * * The in-flight count and the requested-side counters are bumped here * together, so a new requested-side path cannot update one and miss the -* other. +* other. Everything this touches is in the cacheline the caller's lock +* guards; anything else belongs outside that lock. */ -static bool -nfs4_ff_layoutstat_start_io (struct nfs4_ff_layout_ds_stripe *dss_info, -struct nfs4_ff_layoutstat *layoutstat, +static void +nfs4_ff_layoutstat_start_io (struct nfs4_ff_layoutstat *layoutstat,__u64 requested,ktime_t now) +{ +lockdep_assert_held (&layoutstat->lock); + +nfs4_ff_start_busy_timer (layoutstat, now); + layoutstat->io_stat.bytes_requested += requested; +} + +/* +* Is a LAYOUTSTATS report due? Deliberately outside the statistics locks: +* none of this is covered by them, and all of it lives in other objects -- +* mirror->report_interval and ffl->last_report_time, plus a division -- so +* running it under a lock would only lengthen a critical section that is +* otherwise two writes to one cacheline. +* +* last_report_time is shared by every mirror and stripe of this layout, so it +* was never serialised by the mirror lock either. Make that explicit: +* whoever wins the cmpxchg() reports. +*/ +static bool +nfs4_ff_layoutstat_report_due (struct nfs4_ff_layout_ds_stripe *dss_info, +ktime_t now) {struct nfs4_ff_layout_mirror *mirror = dss_info->mirror;struct nfs4_flexfile_layout *ffl =FF_LAYOUT_FROM_HDR (mirror->layout);s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;ktime_t last_report; -lockdep_assert_held (&layoutstat->lock); - -nfs4_ff_start_busy_timer (layoutstat, now); - layoutstat->io_stat.bytes_requested += requested; - -nfs4_ff_layoutstat_set_start_time (dss_info, now);if (mirror->report_interval !=0 ) report_interval = (s64 )mirror->report_interval *1000LL ;else if (layoutstats_timer !=0 ) report_interval = (s64 )layoutstats_timer *1000LL ; -/* -* last_report_time is shared by every mirror and stripe of this -* layout, so it was never serialised by the mirror lock either. -* Make that explicit: whoever wins the cmpxchg() reports. -*/ last_report =READ_ONCE (ffl->last_report_time);if (ktime_to_ms (ktime_sub (now, last_report)) < report_interval)return false;
@@ -791,8 +800,7 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_ds_stripe *dss_info,* it is bumped in nfs4_ff_layoutstat_start_io(). */ static void -nfs4_ff_layoutstat_end_io(struct nfs4_ff_layout_ds_stripe *dss_info, - struct nfs4_ff_layoutstat *layoutstat, +nfs4_ff_layoutstat_end_io(struct nfs4_ff_layoutstat *layoutstat, __u64 requested, __u64 completed, ktime_t time_completed,
@@ -826,10 +834,11 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,bool report;spin_lock (&dss_info->read_stat.lock); - report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, - requested, now); +nfs4_ff_layoutstat_start_io (&dss_info->read_stat, requested, now);spin_unlock (&dss_info->read_stat.lock); +nfs4_ff_layoutstat_set_start_time (dss_info, now); + report =nfs4_ff_layoutstat_report_due (dss_info, now);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ()); }
@@ -844,7 +853,7 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task,struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];spin_lock (&dss_info->read_stat.lock); -nfs4_ff_layoutstat_end_io (dss_info, &dss_info->read_stat, +nfs4_ff_layoutstat_end_io (&dss_info->read_stat, requested, completed,ktime_get (), task->tk_start);spin_unlock (&dss_info->read_stat.lock);
@@ -860,10 +869,11 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,bool report;spin_lock (&dss_info->write_stat.lock); - report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, - requested, now); +nfs4_ff_layoutstat_start_io (&dss_info->write_stat, requested, now);spin_unlock (&dss_info->write_stat.lock); +nfs4_ff_layoutstat_set_start_time (dss_info, now); + report =nfs4_ff_layoutstat_report_due (dss_info, now);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ()); }
@@ -882,7 +892,7 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task, requested = completed =0 ;spin_lock (&dss_info->write_stat.lock); -nfs4_ff_layoutstat_end_io (dss_info, &dss_info->write_stat, +nfs4_ff_layoutstat_end_io (&dss_info->write_stat, requested, completed,ktime_get (), task->tk_start);spin_unlock (&dss_info->write_stat.lock);
Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 03/11] NFSv4/flexfiles: cacheline align the per-stripe layoutstats
In reply to Mike SnitzerHi Mike,
Quoted 58 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
Moving the layoutstats lock into struct
nfs4_ff_layout_ds_stripestopped stripes serialising on one spinlock, but it left them sharing cachelines with each other.The lock and the statistics it guards sat at the tail of a struct whose
sizeof()was 296 and whosealignof()was only 8, so in mirror->dss[] stripe i began at i * 296. 296 mod 64 = 40, so every stripe started 40 bytes further into a line than the last and no stripe's lock was line aligned. Acquiring stripe i's lock therefore invalidated the line holding stripe i+1's lock and counters. A RAID0-style layout -- one mirror, many stripes -- is exactly the case the per-stripe lock was introduced for, so this false sharing ate into the win.Align the first member of the guarded set. That does two things: the lock now starts on a line boundary, so the guarded set occupies lines of its own and shares none with the read-mostly members above; and it raises the struct's alignment to 64, which forces
sizeof()up to a multiple of 64 and makes the array stride line aligned. The stride is what actually separates stripe i from stripe i+1.Verified with pahole on flexfilelayout.o built with DWARF5. The layout is identical on
x86_64, arm64 4K pages and arm64 64K pages -- cachelines are 64 bytes on both architectures, so page size does not enter into it -- and was taken with neitherCONFIG_DEBUG_SPINLOCKnorCONFIG_PROVE_LOCKING, both of which fattenspinlock_tand change it:/* --- cacheline 3 boundary (192 bytes) --- */ spinlock_t lock __aligned(64); /* 192 4 */ /* XXX 4 bytes hole */ struct nfs4_ff_layoutstat read_stat; /* 200 72 */ struct nfs4_ff_layoutstat write_stat; /* 272 72 */ ktime_t start_time; /* 344 8 */ /* size: 384, cachelines: 6 */ } __attribute__((__aligned__(64)));A read RPC now dirties two lines of its own stripe and a write three, and neither disturbs a neighbouring stripe. Later patches give each direction a line of its own.
Cost is 296 -> 384 bytes per stripe, i.e. 2816 bytes for a 32 stripe layout, which is negligible beside the rest of the per-file layout state. 384 is also a multiple of 128, so the stride holds for implementations with a 128 byte coherency granule.
Note that the alignment attribute only orders the stripes relative to the start of mirror->dss[], not that array's own base. The base is aligned too, under either allocator
kvmalloc()may pick for it:- From the slab.
kmalloc()documents a size based alignment guarantee for power-of-two sizes only, but the smallest possible request is now 384 bytes, and every kmalloc bucket at or above that is a power of two whose objects SLUB lays out at multiples of the object size from a page aligned slab, so the base is at least 512 byte aligned. The exceptions, kmalloc-96 and kmalloc-192, are both smaller than one stripe and cannot be selected.
Quoted 5 lines
- From vmalloc. A stripe array wide enough to need a high order allocation is served by the
vmalloc()fallback instead, which returns page aligned memory.Either is comfortably more than the 64 bytes this needs.
This commit message is very wordy, and can probably be edited down.
Quoted 24 lines
Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.h +1
-1@@ -14,6 +14,7 @@#define FF_FLAGS_NO_IO_THRU_MDS 2#define FF_FLAGS_NO_READ_IO 4 +#include <linux/cache.h> #include <linux/refcount.h> #include "../pnfs.h" @@ -86,8 +87,17 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; -/* Protects read_stat, write_stat and start_time below */
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 98
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.h +9
-1-spinlock_t lock; +/* +* Protects read_stat, write_stat and start_time below. +* +* Cacheline aligned so that the layoutstats accounting done on +* every I/O to this stripe does not share cachelines with the +* read-mostly members above, nor with the neighbouring stripes +* in mirror->dss[]: aligning the first member of the guarded +* set also rounds sizeof() up to a multiple of the cacheline, +* which is what makes the array stride line-aligned.
Let's leave the original comment alone, since most of this information is duplicated in the commit message and can be easily found with git blame.
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 100
Quoted 2 lines
fs/nfs/flexfilelayout/flexfilelayout.h +2
-0+*/ +spinlock_t lock____cacheline_aligned_in_smp ;
Making this cacheline aligned seems like a good idea!
Anna
Quoted 5 lines
fs/nfs/flexfilelayout/flexfilelayout.h +0
-0struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat;ktime_t start_time;Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 01/11] NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock
In reply to Mike SnitzerHi Mike,
Quoted 83 lines · Mike Snitzer
From: Jonathan Flynn <[email protected]>
Since striped layouts were introduced, a mirror owns an array of struct
nfs4_ff_layout_ds_stripeand each stripe carries its ownread_statandwrite_stat. The lock protecting those statistics stayed behind in the mirror, so every DS RPC in the layout serialises on the same spinlock twice: once in ff_layout_{read,write}_prepare_common() and again on completion in ff_layout_{read,write}_count_stats().For a RAID0-style layout (one mirror, many stripes) that lock is effectively per-file, and striping then multiplies RPC parallelism without multiplying lock parallelism.
On a 64-core client reading a single file from a 32-stripe layout with a 128KiB stripe unit (2MB
O_DIRECTreads, ~36.7 GB/s, ~275k NFSv3 READ RPCs/s, i.e. ~550k acquisitions/s of one lock), perf reports 45% of all CPU innative_queued_spin_lock_slowpath, and lock:contention_begin attributes 73% of the contention events to the two layoutstats paths:36.4%
rpc_exit_task->ff_layout_read_count_stats->ff_layout_read_record_layoutstats_done -> _raw_spin_lock 36.3% ff_layout_read_prepare_v3 -> ff_layout_read_record_layoutstats_start -> _raw_spin_lockMove the lock into struct
nfs4_ff_layout_ds_stripe, next to the statistics it protects, so that I/O to different stripes no longer contends. The reporting paths take the same per-stripe lock.
NFS4_FF_MIRROR_STAT_AVAILis a mirror-wide bit operation and does not need the lock; set it outside the critical section so the mirror's cacheline is not written under it.
ffl->last_report_timeis shared by every mirror and stripe of a layout and was already updated under different mirror locks, so it was never serialised. Make that explicit with acmpxchg(): whichever racer wins sends the LAYOUTSTATS report.Signed-off-by Jonathan Flynn <[email protected]>Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 45 +++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 ++ 2 files changed, 28 insertions(+), 19 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +3
-1@@ -293,8 +293,10 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count, return NULL; } - for (u32 dss_id = 0; dss_id < mirror->dss_count; dss_id++) + for (u32 dss_id = 0; dss_id < mirror->dss_count; dss_id++) { + spin_lock_init(&mirror->dss[dss_id].lock); nfs_localio_file_init(&mirror->dss[dss_id].nfl); + } return mirror;}
fs/nfs/flexfilelayout/flexfilelayout.c +1
-0@@ -736,6 +738,7 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, {s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;struct nfs4_flexfile_layout *ffl =FF_LAYOUT_FROM_HDR (mirror->layout); +ktime_t last_report;nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now);fs/nfs/flexfilelayout/flexfilelayout.c +0
-6if (!mirror->dss[dss_id].start_time)@@ -744,13 +747,17 @@ nfs4_ff_layoutstat_start_io(structnfs4_ff_layout_mirror *mirror, report_interval = (s64 )mirror->report_interval *1000LL ; else if (layoutstats_timer != 0) report_interval = (s64 )layoutstats_timer *1000LL ; -if (ktime_to_ms (ktime_sub (now, ffl->last_report_time)) >= - report_interval) { - ffl->last_report_time = now; -return true; - } -return false;
fs/nfs/flexfilelayout/flexfilelayout.c nfs4_ff_layoutstat_start_io(struct · line 761
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.c +10
-0+/* +* last_report_time is shared by every mirror and stripe of this +* layout, so it was never serialised by the mirror lock either. +* Make that explicit: whoever wins the cmpxchg() reports. +*/ + last_report =READ_ONCE (ffl->last_report_time); +if (ktime_to_ms (ktime_sub (now, last_report)) < report_interval) +return false; + +return try_cmpxchg64 (&ffl->last_report_time, &last_report, now);
As far as I can tell, this is the first use of try_cmpxchg64 in any filesystem code. This has me worried it's the wrong function to be calling here. Is there any reason we can't use atomic64_try_cmpxchg(), which is what I see being used by other filesystem code?
Thanks,
Anna
Quoted 120 lines
fs/nfs/flexfilelayout/flexfilelayout.c +0
-0}static void
fs/nfs/flexfilelayout/flexfilelayout.c +10
-10@@ -794,13 +801,13 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode, {bool report; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock); report =nfs4_ff_layoutstat_start_io ( mirror, dss_id, &mirror->dss[dss_id].read_stat, now);nfs4_ff_layout_stat_io_update_requested ( &mirror->dss[dss_id].read_stat, requested); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());@@ -813,12 +820,12 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, __u64 requested, __u64 completed) { -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock);nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].read_stat, requested, completed,ktime_get (), task->tk_start); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock); } static void@@ -829,7 +836,7 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode, {bool report; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock); report = nfs4_ff_layoutstat_start_io( mirror, dss_id,@@ -838,8 +845,8 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,nfs4_ff_layout_stat_io_update_requested ( &mirror->dss[dss_id].write_stat, requested); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());@@ -856,11 +863,11 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&mirror->lock); +spin_lock (&mirror->dss[dss_id].lock);nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].write_stat, requested, completed,ktime_get (), task->tk_start); +spin_unlock (&mirror->dss[dss_id].lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); -spin_unlock (&mirror->lock); } static struct nfs4_ff_layout_ds *@@ -3071,13 +3078,13 @@ ff_layout_encode_ff_layoutupdate(structxdr_stream *xdr, p =xdr_reserve_space (xdr,4 + fh->size);xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ -spin_lock (&dss_info->mirror->lock); +spin_lock (&dss_info->lock);ff_layout_encode_io_latency (xdr, &dss_info->read_stat.io_stat);/* ff_io_latency4 write */ ff_layout_encode_io_latency (xdr, &dss_info->write_stat.io_stat); -spin_unlock (&dss_info->mirror->lock); +spin_unlock (&dss_info->lock);/* nfstime4 */ ff_layout_encode_nfstime(xdr,fs/nfs/flexfilelayout/flexfilelayout.c +1
-1ktime_sub(ktime_get(),@@ -3153,7 +3160,7 @@ ff_layout_mirror_prepare_stats(structpnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&mirror->lock); +spin_lock (&dss_info->lock); devinfo->read_count = dss_info->read_stat.io_stat.ops_completed;fs/nfs/flexfilelayout/flexfilelayout.c +1
-1devinfo->read_bytes =@@ -3162,7 +3169,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, dss_info->write_stat.io_stat.ops_completed; devinfo->write_bytes = dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&mirror->lock); +spin_unlock (&dss_info->lock); devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops;fs/nfs/flexfilelayout/flexfilelayout.c +0
-0priv->dss_info = dss_info;fs/nfs/flexfilelayout/flexfilelayout.h +2
-0@@ -86,6 +86,8 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; +/* Protects read_stat, write_stat and start_time below */ +spinlock_t lock;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat;ktime_t start_time;Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 04/11] NFSv4/flexfiles: make dss_info->start_time write-once
In reply to Mike SnitzerHi MIke,
Quoted 7 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
start_timewas written under the stripe lock innfs4_ff_layoutstat_start_io()and read outside it inff_layout_encode_ff_layoutupdate(). Pre-existing --mirror->lockhad the same shape -- and benign, since it is a single naturally aligned 8 byte value, but it is an unannotated race either way.
We can probably drop this last sentence.
Quoted 4 lines
The lock is the wrong tool for it.
ffl_durationis measured from this stripe's first I/O, so the field is written once and read-only for the rest of the stripe's life. Say that instead: publish it with
This portion needs editing too. My brain tripped over "Say that instead" while reading.
Quoted 3 lines
try_cmpxchg64()and let the first I/O win, and read it withREAD_ONCE(). Only the first I/O to a stripe pays the read-modify-write; every one after it short circuits on theREAD_ONCE().
This sentence is just rewording the previous sentence.
Quoted 35 lines
ffl->last_report_timetwo lines away is already published this way, so the pattern is not new to this file.This also stops
start_timedepending on there being exactly one lock per stripe, which a later patch changes.Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 21 ++++++++++++++++++--- fs/nfs/flexfilelayout/flexfilelayout.h | 3 ++- 2 files changed, 20 insertions(+), 4 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +2
-0@@ -729,6 +729,22 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now)return ktime_sub (now, start); } +/* +* ffl_duration is measured from this stripe's first I/O, so
start_timeis
+ * written once and read-only for the rest of the stripe's life. Publish it
+ * with a cmpxchg and let the first I/O win, rather than depending on a lock
+ * that every reader would then have to take.
+ */fs/nfs/flexfilelayout/flexfilelayout.c +2
-0+static void +nfs4_ff_layoutstat_set_start_time(struct nfs4_ff_layout_ds_stripe
*dss_info,
fs/nfs/flexfilelayout/flexfilelayout.c nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) · line 745
Quoted 6 lines
fs/nfs/flexfilelayout/flexfilelayout.c +6
-0+ ktime_t now) +{ +ktime_t unset =0 ; + +if (!READ_ONCE (dss_info->start_time)) +try_cmpxchg64 (&dss_info->start_time, &unset, now);
I have the same atomic64_try_cmpxchg() question as in patch 1 here.
Thanks,
Anna
Quoted 51 lines
fs/nfs/flexfilelayout/flexfilelayout.c +3
-2+} + static bool nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, u32 dss_id,@@ -740,8 +756,7 @@ nfs4_ff_layoutstat_start_io(structnfs4_ff_layout_mirror *mirror, ktime_t last_report;nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); -if (!mirror->dss[dss_id].start_time) - mirror->dss[dss_id].start_time = now; +nfs4_ff_layoutstat_set_start_time (&mirror->dss[dss_id], now);if (mirror->report_interval !=0 ) report_interval = (s64 )mirror->report_interval *1000LL ;fs/nfs/flexfilelayout/flexfilelayout.c +1
-1else if (layoutstats_timer !=0 )@@ -3087,7 +3102,7 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr,/* nfstime4 */ ff_layout_encode_nfstime (xdr,ktime_sub (ktime_get (), - dss_info->start_time)); +READ_ONCE (dss_info->start_time)));/* bool */ p =xdr_reserve_space (xdr,4 );fs/nfs/flexfilelayout/flexfilelayout.c +0
-0*p =cpu_to_be32 (false);fs/nfs/flexfilelayout/flexfilelayout.h +2
-1@@ -88,7 +88,7 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *rw_cred;struct nfs_file_localio nfl;/* -* Protects read_stat, write_stat and start_time below. +* Protects read_stat and write_stat below. * * Cacheline aligned so that the layoutstats accounting done on * every I/O to this stripe does not share cachelines with the @@ -100,6 +100,7 @@ struct nfs4_ff_layout_ds_stripe {spinlock_t lock____cacheline_aligned_in_smp ;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat; +/* Published once by the first I/O; see
nfs4_ff_layoutstat_set_start_time()*/
ktime_tstart_time;
};Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 05/11] NFSv4/flexfiles: drop the redundant atomic from the busy timer
In reply to Mike SnitzerHi Mike,
Quoted 4 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
nfs4_ff_busy_timer::n_opsis anatomic_t, but every access to it is already serialised by the stripe lock:
Honestly, the commit message doesn't need to say much more than this sentence. The rest of the message basically repeats the concept using more words.
Quoted 10 lines
nfs4_ff_start_busy_timer()has one caller,nfs4_ff_layoutstat_start_io(), reached only from ff_layout_{read,write}_record_layoutstats_start() with dss[dss_id].lock held.
nfs4_ff_end_busy_timer()has one caller,nfs4_ff_layout_stat_io_update_completed(), reached only from ff_layout_{read,write}_record_layoutstats_done() with the same lock held.
Quoted 6 lines
The two reporting paths,
ff_layout_mirror_prepare_stats()andff_layout_encode_ff_layoutupdate(), take the stripe lock too and do not touchn_opsat all. There is no lockless reader.So the atomic_inc_return()/atomic_dec_return() are locked read-modify- writes on a line the CPU already owns exclusively -- pure serialisation
^^^^^^^^^^^^^
Interesting that there's some UK spellings sneaking into your text. I noticed "neighbour" in an earlier patch, too.
Thanks,
Anna
Quoted 62 lines
overhead, paid twice per DS RPC on the hottest path in this file, at the 275k-450k RPC/s the per-stripe lock work was measured at.
Make
n_opsa plain int and document the locking on the struct. Its size is unchanged, so the stripe layout is unaffected. The underflow check is preserved.Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 6 ++---- fs/nfs/flexfilelayout/flexfilelayout.h | 6 +++++- 2 files changed, 7 insertions(+), 5 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +2
-4@@ -711,9 +711,8 @@ static void nfs4_ff_start_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {/* first IO request? */ -if (atomic_inc_return (&timer->n_ops) ==1 ) { +if (++timer->n_ops ==1 ) timer->start_time = now; - } } static ktime_t@@ -721,8 +720,7 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {ktime_t start; -if (atomic_dec_return (&timer->n_ops) <0 ) -WARN_ON_ONCE (1 ); +WARN_ON_ONCE (--timer->n_ops <0 ); start = timer->start_time;fs/nfs/flexfilelayout/flexfilelayout.c +0
-0timer->start_time = now;fs/nfs/flexfilelayout/flexfilelayout.h +5
-1@@ -64,9 +64,13 @@ struct nfs4_ff_io_stat {ktime_t aggregate_completion_time; }; +/* +* Both members are serialised by the enclosing stripe's +* nfs4_ff_layout_ds_stripe::lock; see nfs4_ff_{start,end}_busy_timer(). +*/ struct nfs4_ff_busy_timer {ktime_t start_time; -atomic_t n_ops; +int n_ops; };struct nfs4_ff_layoutstat {Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 06/11] NFSv4/flexfiles: pair the in-flight count with the layoutstats updates
In reply to Mike SnitzerHi Mike,
Quoted 143 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
The busy timer's in-flight op count and the requested-side statistics are two halves of the same accounting, but they lived in two helpers that the callers invoked back to back. Nothing stopped a new requested-side path from updating one and missing the other.
Fold
nfs4_ff_layout_stat_io_update_requested()intonfs4_ff_layoutstat_start_io()so both happen in one place, and renamenfs4_ff_layout_stat_io_update_completed()tonfs4_ff_layoutstat_end_io()for symmetry -- it already dropped the in-flight count alongside the completed-side statistics.Rename
nfs4_ff_busy_timer::n_opstoops_in_flightwhile here. It is what the field holds, and it makes its relationship to the op counters legible at the point of use.Both helpers now take the stripe rather than mirror +
dss_id, which lets them assert the lock they run under. Since the previous patch madeops_in_flighta plain int, that lock is the only thing keeping it consistent, so assert it rather than document it;lockdep_assert_held()compiles out withoutCONFIG_PROVE_LOCKING. The callers get adss_infolocal, which is what they were open coding.No functional change.
Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 104 ++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 +- 2 files changed, 58 insertions(+), 48 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +2
-2@@ -711,7 +711,7 @@ static void nfs4_ff_start_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {/* first IO request? */ -if (++timer->n_ops ==1 ) +if (++timer->ops_in_flight ==1 ) timer->start_time = now; }@@ -720,7 +720,7 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) {ktime_t start; -WARN_ON_ONCE (--timer->n_ops <0 ); +WARN_ON_ONCE (--timer->ops_in_flight <0 ); start = timer->start_time;fs/nfs/flexfilelayout/flexfilelayout.c +18
-5timer->start_time = now;@@ -743,18 +743,31 @@ nfs4_ff_layoutstat_set_start_time(struct nfs4_ff_layout_ds_stripe *dss_info,try_cmpxchg64 (&dss_info->start_time, &unset, now); } +/* +* Account for an I/O about to be sent to this stripe, and return true if a +* LAYOUTSTATS report is due. +* +* The in-flight count and the requested-side counters are bumped here +* together, so a new requested-side path cannot update one and miss the +* other. +*/ static bool -nfs4_ff_layoutstat_start_io (struct nfs4_ff_layout_mirror *mirror, -u32 dss_id, +nfs4_ff_layoutstat_start_io (struct nfs4_ff_layout_ds_stripe *dss_info,struct nfs4_ff_layoutstat *layoutstat, -ktime_t now) +__u64 requested,ktime_t now) { -s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ; +struct nfs4_ff_layout_mirror *mirror = dss_info->mirror;struct nfs4_flexfile_layout *ffl =FF_LAYOUT_FROM_HDR (mirror->layout); +s64 report_interval =FF_LAYOUTSTATS_REPORT_INTERVAL ;ktime_t last_report; +lockdep_assert_held (&dss_info->lock); +nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); -nfs4_ff_layoutstat_set_start_time (&mirror->dss[dss_id], now); + layoutstat->io_stat.ops_requested++; + layoutstat->io_stat.bytes_requested += requested; + +nfs4_ff_layoutstat_set_start_time (dss_info, now);if (mirror->report_interval !=0 ) report_interval = (s64 )mirror->report_interval *1000LL ;fs/nfs/flexfilelayout/flexfilelayout.c +13
-15else if (layoutstats_timer !=0 )@@ -772,27 +785,25 @@ nfs4_ff_layoutstat_start_io(structnfs4_ff_layout_mirror *mirror, return try_cmpxchg64(&ffl->last_report_time, &last_report, now); } +/* +* Account for an I/O that has completed on this stripe. The in-flight count +* is dropped here, alongside the completed-side counters, for the same reason +* it is bumped in nfs4_ff_layoutstat_start_io(). +*/ static void -nfs4_ff_layout_stat_io_update_requested (struct nfs4_ff_layoutstat *layoutstat, -__u64 requested) -{ -struct nfs4_ff_io_stat *iostat = &layoutstat->io_stat; - - iostat->ops_requested++; - iostat->bytes_requested += requested; -} - -static void -nfs4_ff_layout_stat_io_update_completed (struct nfs4_ff_layoutstat *layoutstat, -__u64 requested, -__u64 completed, -ktime_t time_completed, -ktime_t time_started) +nfs4_ff_layoutstat_end_io (struct nfs4_ff_layout_ds_stripe *dss_info, +struct nfs4_ff_layoutstat *layoutstat, +__u64 requested, +__u64 completed, +ktime_t time_completed, +ktime_t time_started) {struct nfs4_ff_io_stat *iostat = &layoutstat->io_stat;ktime_t completion_time =ktime_sub (time_completed, time_started);ktime_t timer; +lockdep_assert_held (&dss_info->lock); + iostat->ops_completed++; iostat->bytes_completed += completed;fs/nfs/flexfilelayout/flexfilelayout.c +1
-0iostat->bytes_not_delivered += requested - completed;@@ -811,14 +822,13 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,u32 dss_id, __u64 requested, ktime_t now) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report;
fs/nfs/flexfilelayout/flexfilelayout.c nfs4_ff_layout_stat_io_start_read(struct inode *inode, · line 831
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.c +4
-6-spin_lock (&mirror->dss[dss_id].lock); - report =nfs4_ff_layoutstat_start_io ( - mirror, dss_id, &mirror->dss[dss_id].read_stat, now); -nfs4_ff_layout_stat_io_update_requested ( - &mirror->dss[dss_id].read_stat, requested); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); + report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, + requested, now); +spin_unlock (&dss_info->lock);
I noticed that this function and nfs4_ff_layout_stat_io_start_write() are almost identical (the only difference being read_stat vs write_stat). I was wondering if it might be possible to combine them in the future to reduce code duplication?
Thanks,
Anna
Quoted 83 lines
fs/nfs/flexfilelayout/flexfilelayout.c +19
-19set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)@@ -832,11 +842,13 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, __u64 requested, __u64 completed) { -spin_lock (&mirror->dss[dss_id].lock); -nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].read_stat, - requested, completed, -ktime_get (), task->tk_start); -spin_unlock (&mirror->dss[dss_id].lock); +struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; + +spin_lock (&dss_info->lock); +nfs4_ff_layoutstat_end_io (dss_info, &dss_info->read_stat, + requested, completed, +ktime_get (), task->tk_start); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }@@ -846,18 +858,13 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,u32 dss_id, __u64 requested, ktime_t now) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&mirror->dss[dss_id].lock); - report =nfs4_ff_layoutstat_start_io ( - mirror, - dss_id, - &mirror->dss[dss_id].write_stat, - now); -nfs4_ff_layout_stat_io_update_requested ( - &mirror->dss[dss_id].write_stat, - requested); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); + report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, + requested, now); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)@@ -872,13 +879,16 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,__u64 completed, enum nfs3_stable_how committed) { + struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; +if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&mirror->dss[dss_id].lock); -nfs4_ff_layout_stat_io_update_completed (&mirror->dss[dss_id].write_stat, - requested, completed,ktime_get (), task->tk_start); -spin_unlock (&mirror->dss[dss_id].lock); +spin_lock (&dss_info->lock); +nfs4_ff_layoutstat_end_io (dss_info, &dss_info->write_stat, + requested, completed, +ktime_get (), task->tk_start); +spin_unlock (&dss_info->lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }fs/nfs/flexfilelayout/flexfilelayout.h +1
-1@@ -70,7 +70,7 @@ struct nfs4_ff_io_stat {*/ struct nfs4_ff_busy_timer {ktime_t start_time; -int n_ops; +int ops_in_flight; };struct nfs4_ff_layoutstat {Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 07/11] NFSv4/flexfiles: derive ffil_ops_requested rather than storing it
In reply to Mike SnitzerHi Mike,
Quoted 11 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
nfs4_ff_io_stat::ops_requestedcarries no information that the stripe does not already hold. An op is in flight from the moment it is counted as requested until it is counted as completed, and the previous patch put both halves of that accounting in one place each, soops_requested == ops_completed + ops_in_flightholds at all times under the lock.
Quoted 7 lines
Drop the field and compute the value at encode time. It had exactly one producer and one consumer, the LAYOUTSTATS encoder, so nothing else needs to change and the wire format is untouched --
ffil_ops_requestedis still a uint64, andnfs4_ff_ops_requested()returns__u64.ff_layout_encode_io_latency()now takes the wholenfs4_ff_layoutstat, since the in-flight count lives beside the statistics rather than in them.
Let's cut the above paragraph.
Quoted 72 lines
This takes 8 bytes off each direction's statistics, which is what lets each direction fit in a cacheline of its own; see the next patch.
Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 12 +++++------- fs/nfs/flexfilelayout/flexfilelayout.h | 19 ++++++++++++++++++- 2 files changed, 23 insertions(+), 8 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +0
-1@@ -764,7 +764,6 @@ nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_ds_stripe *dss_info,lockdep_assert_held (&dss_info->lock);nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); - layoutstat->io_stat.ops_requested++; layoutstat->io_stat.bytes_requested += requested;fs/nfs/flexfilelayout/flexfilelayout.c +5
-6nfs4_ff_layoutstat_set_start_time (dss_info, now);@@ -3066,12 +3065,13 @@ ff_layout_encode_nfstime(struct xdr_stream *xdr,static void ff_layout_encode_io_latency (struct xdr_stream *xdr, -struct nfs4_ff_io_stat *stat) +const struct nfs4_ff_layoutstat *layoutstat) { +const struct nfs4_ff_io_stat *stat = &layoutstat->io_stat;__be32 *p; p =xdr_reserve_space (xdr,5 *8 ); - p =xdr_encode_hyper (p, stat->ops_requested); + p =xdr_encode_hyper (p,nfs4_ff_ops_requested (layoutstat)); p =xdr_encode_hyper (p, stat->bytes_requested); p =xdr_encode_hyper (p, stat->ops_completed); p =xdr_encode_hyper (p, stat->bytes_completed);@@ -3101,11 +3101,9 @@ ff_layout_encode_ff_layoutupdate(struct xdr_stream *xdr,xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ spin_lock (&dss_info->lock); -ff_layout_encode_io_latency (xdr, - &dss_info->read_stat.io_stat); +ff_layout_encode_io_latency (xdr, &dss_info->read_stat);/* ff_io_latency4 write */ -ff_layout_encode_io_latency (xdr, - &dss_info->write_stat.io_stat); +ff_layout_encode_io_latency (xdr, &dss_info->write_stat);spin_unlock (&dss_info->lock);/* nfstime4 */ fs/nfs/flexfilelayout/flexfilelayout.c +0
-0ff_layout_encode_nfstime(xdr,fs/nfs/flexfilelayout/flexfilelayout.h +0
-1@@ -55,7 +55,6 @@ struct nfs4_ff_layout_ds_err { };struct nfs4_ff_io_stat { -__u64 ops_requested;__u64 bytes_requested;__u64 ops_completed;__u64 bytes_completed;@@ -78,6 +77,24 @@ struct nfs4_ff_layoutstat {struct nfs4_ff_busy_timer busy_timer; };
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layoutstat { · line 89
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.h +10
-0+/* +* ffil_ops_requested is not stored. An op is in flight from the moment it is +* counted as requested until it is counted as completed, so +* +* ops_requested == ops_completed + ops_in_flight +* +* holds by construction; see nfs4_ff_layoutstat_start_io() and +* nfs4_ff_layoutstat_end_io(), which maintain both halves together. +* +* Caller must hold the lock guarding this nfs4_ff_layoutstat.
This comment could be cut down too, and leave the one note about holding the lock.
Otherwise, this is a nice cleanup!
Thanks,
Anna
Quoted 13 lines
fs/nfs/flexfilelayout/flexfilelayout.h +8
-0+*/ +static inline __u64 +nfs4_ff_ops_requested (const struct nfs4_ff_layoutstat *layoutstat) +{ +return layoutstat->io_stat.ops_completed + + layoutstat->busy_timer.ops_in_flight; +} +struct nfs4_ff_layout_mirror ;struct nfs4_ff_layout_ds_stripe {Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 08/11] NFSv4/flexfiles: give each direction its own lock and cacheline
In reply to Mike SnitzerHi Mike,
Quoted 257 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
struct
nfs4_ff_layout_ds_stripeshrinks from 384 bytes to 320, and an I/O to a stripe now dirties one cacheline instead of two. The 384 was what "NFSv4/flexfiles: cacheline align the per-stripe layoutstats" had to pay to get the array stride line aligned; this gets that back and takes another 64 bytes off, on top of removing the last shared lock between the two I/O directions.Cacheline aligning the stripe's lock stopped stripes sharing lines with each other, but within a stripe an I/O still dirtied more than one line, and reads and writes still serialised on one spinlock.
The set one RPC touches is not the whole guarded region. A read takes the lock, bumps the in-flight count and updates every field of
read_stat--bytes_requestedand the busy timer on the way out,ops_completed,bytes_completed,bytes_not_deliveredand the twoktime_tsums on completion -- whilewrite_statstays untouched, and a write is the mirror image. The two directions share nothing but the lock.So stop sharing the lock. Move it into struct
nfs4_ff_layoutstat, alongside the in-flight count and the statistics it guards, and foldnfs4_ff_busy_timerin as a barektime_t. Withops_requestedderived rather than stored, that comes tospinlock_t lock 4 int ops_in_flight 4struct
nfs4_ff_io_statio_stat48
ktime_tbusy_start_time8which is 64 bytes with no padding. Align the structure to a cacheline and each direction occupies one, exactly.
An I/O now dirties a single line, and a read and a write to the same stripe touch neither the same lock nor the same line. The two reporting paths,
ff_layout_mirror_prepare_stats()andff_layout_encode_ff_layoutupdate(), take each lock in turn; they need no atomicity across the pair, as the statistics are advisory and were never sampled atomically with respect to in-flight I/O anyway.pahole on flexfilelayout.o built with DWARF5, identical on
x86_64, arm64 4K pages and arm64 64K pages:struct nfs4_ff_layoutstat { spinlock_t lock; /* 0 4 */ int ops_in_flight; /* 4 4 */ struct nfs4_ff_io_stat io_stat; /* 8 48 */ ktime_t busy_start_time; /* 56 8 */ /* size: 64, cachelines: 1, members: 4 */ } __attribute__((__aligned__(64)));
ktime_tstart_time; /* 136 8 */
/* XXX 48 bytes hole */
/* --- cacheline 3 boundary (192 bytes) --- */
structnfs4_ff_layoutstatread_stat; /* 192 64 */
/* --- cacheline 4 boundary (256 bytes) --- */
structnfs4_ff_layoutstatwrite_stat; /* 256 64 */
/* size: 320, cachelines: 5 */
start_timestays out of both lines, since it neither fits nor needs to. "NFSv4/flexfiles: makedss_info->start_timewrite-once" removed its dependence on any lock, and nothing writes it after the stripe's first I/O, so its line never bounces; the same I/O already readsfh_versions, stateid and the creds from the lines around it. In the hole ahead ofread_statit is free.The array stride stays a multiple of 64 and of 128, so stripes remain separated on implementations with a 128 byte coherency granule.
Note that
CONFIG_DEBUG_SPINLOCKorCONFIG_PROVE_LOCKINGfattenspinlock_tenough to pushnfs4_ff_layoutstatto 128 bytes and the stripe to 448. Correctness is unaffected; such kernels simply do not get the packing.Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 53 ++++++++++++++------------ fs/nfs/flexfilelayout/flexfilelayout.h | 46 +++++++++++----------- 2 files changed, 52 insertions(+), 47 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +11
-10@@ -293,7 +293,8 @@ static struct nfs4_ff_layout_mirror *ff_layout_alloc_mirror(u32 dss_count, } for (u32 dss_id = 0; dss_id < mirror->dss_count; dss_id++) { -spin_lock_init (&mirror->dss[dss_id].lock); +spin_lock_init (&mirror->dss[dss_id].read_stat.lock); +spin_lock_init (&mirror->dss[dss_id].write_stat.lock);nfs_localio_file_init (&mirror->dss[dss_id].nfl); }@@ -708,22 +709,22 @@ static u32 calc_dss_id_from_commit(struct pnfs_layout_segment *lseg, }static void -nfs4_ff_start_busy_timer (struct nfs4_ff_busy_timer *timer,ktime_t now) +nfs4_ff_start_busy_timer (struct nfs4_ff_layoutstat *layoutstat,ktime_t now) {/* first IO request? */ -if (++timer->ops_in_flight ==1 ) - timer->start_time = now; +if (++layoutstat->ops_in_flight ==1 ) + layoutstat->busy_start_time = now; }static ktime_t -nfs4_ff_end_busy_timer (struct nfs4_ff_busy_timer *timer,ktime_t now) +nfs4_ff_end_busy_timer (struct nfs4_ff_layoutstat *layoutstat,ktime_t now) {ktime_t start; -WARN_ON_ONCE (--timer->ops_in_flight <0 ); +WARN_ON_ONCE (--layoutstat->ops_in_flight <0 ); - start = timer->start_time; - timer->start_time = now; + start = layoutstat->busy_start_time; + layoutstat->busy_start_time = now;return ktime_sub (now, start); }@@ -761,9 +762,9 @@ nfs4_ff_layoutstat_start_io(structnfs4_ff_layout_ds_stripe *dss_info, s64 report_interval = FF_LAYOUTSTATS_REPORT_INTERVAL;ktime_t last_report; -lockdep_assert_held (&dss_info->lock); +lockdep_assert_held (&layoutstat->lock); -nfs4_ff_start_busy_timer (&layoutstat->busy_timer, now); +nfs4_ff_start_busy_timer (layoutstat, now); layoutstat->io_stat.bytes_requested += requested;fs/nfs/flexfilelayout/flexfilelayout.c +2
-2nfs4_ff_layoutstat_set_start_time (dss_info, now);@@ -801,13 +802,13 @@ nfs4_ff_layoutstat_end_io(structnfs4_ff_layout_ds_stripe *dss_info, ktime_t completion_time = ktime_sub(time_completed, time_started);ktime_t timer; -lockdep_assert_held (&dss_info->lock); +lockdep_assert_held (&layoutstat->lock); iostat->ops_completed++; iostat->bytes_completed += completed; iostat->bytes_not_delivered += requested - completed; - timer =nfs4_ff_end_busy_timer (&layoutstat->busy_timer, time_completed); + timer =nfs4_ff_end_busy_timer (layoutstat, time_completed); iostat->total_busy_time =ktime_add (iostat->total_busy_time, timer);fs/nfs/flexfilelayout/flexfilelayout.c +12
-10iostat->aggregate_completion_time =@@ -824,10 +825,10 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock); report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, requested, now); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->read_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)@@ -843,11 +844,11 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, {struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id]; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock);nfs4_ff_layoutstat_end_io (dss_info, &dss_info->read_stat, requested, completed,ktime_get (), task->tk_start); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->read_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }@@ -860,10 +861,10 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,struct nfs4_ff_layout_ds_stripe *dss_info = &mirror->dss[dss_id];bool report; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->write_stat.lock); report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, requested, now); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)@@ -883,11 +884,11 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,if (committed ==NFS_UNSTABLE ) requested = completed =0 ; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->write_stat.lock);nfs4_ff_layoutstat_end_io (dss_info, &dss_info->write_stat, requested, completed,ktime_get (), task->tk_start); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); }@@ -3100,11 +3101,13 @@ ff_layout_encode_ff_layoutupdate(structxdr_stream *xdr, p =xdr_reserve_space (xdr,4 + fh->size);xdr_encode_opaque (p, fh->data, fh->size);/* ff_io_latency4 read */ -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock);ff_layout_encode_io_latency (xdr, &dss_info->read_stat); +spin_unlock (&dss_info->read_stat.lock);/* ff_io_latency4 write */ +spin_lock (&dss_info->write_stat.lock);ff_layout_encode_io_latency (xdr, &dss_info->write_stat); -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock);/* nfstime4 */ ff_layout_encode_nfstime(xdr,fs/nfs/flexfilelayout/flexfilelayout.c +4
-2ktime_sub(ktime_get(),@@ -3180,16 +3183,18 @@ ff_layout_mirror_prepare_stats(structpnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&dss_info->lock); +spin_lock (&dss_info->read_stat.lock); devinfo->read_count = dss_info->read_stat.io_stat.ops_completed; devinfo->read_bytes = dss_info->read_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->read_stat.lock); +spin_lock (&dss_info->write_stat.lock); devinfo->write_count = dss_info->write_stat.io_stat.ops_completed; devinfo->write_bytes = dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->lock); +spin_unlock (&dss_info->write_stat.lock); devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops;fs/nfs/flexfilelayout/flexfilelayout.c +0
-0priv->dss_info = dss_info;fs/nfs/flexfilelayout/flexfilelayout.h +0
-0@@ -64,18 +64,22 @@ struct nfs4_ff_io_stat { };
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_io_stat { · line 73
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.h +7
-2/* -* Both members are serialised by the enclosing stripe's -* nfs4_ff_layout_ds_stripe::lock; see nfs4_ff_{start,end}_busy_timer(). +* One direction's statistics for one stripe, with the lock that guards them. +* +* Exactly one cacheline, and aligned to one, so that an I/O dirties a single +* line: the lock, the in-flight count and every counter it touches are all in +* here, and the other direction's copy is in a line of its own. Adding to +* this structure, or letting CONFIG_DEBUG_SPINLOCK or CONFIG_PROVE_LOCKING +* fatten spinlock_t, spills it into a second line.
Let's leave the cacheline talk to the commit message and not in the code.
Quoted 41 lines
fs/nfs/flexfilelayout/flexfilelayout.h +10
-14*/ -struct nfs4_ff_busy_timer { -ktime_t start_time; -int ops_in_flight; -}; -struct nfs4_ff_layoutstat { -struct nfs4_ff_io_stat io_stat; -struct nfs4_ff_busy_timer busy_timer; -}; +/* Protects every other member of this structure */ +spinlock_t lock; +int ops_in_flight; +struct nfs4_ff_io_stat io_stat; +/* Start of the interval io_stat.total_busy_time accrues */ +ktime_t busy_start_time; +}____cacheline_aligned_in_smp ;/* * ffil_ops_requested is not stored. An op is in flight from the moment it is @@ -91,8 +95,7 @@ struct nfs4_ff_layoutstat {static inline __u64 nfs4_ff_ops_requested (const struct nfs4_ff_layoutstat *layoutstat) { -return layoutstat->io_stat.ops_completed + - layoutstat->busy_timer.ops_in_flight; +return layoutstat->io_stat.ops_completed + layoutstat->ops_in_flight; }struct nfs4_ff_layout_mirror ;@@ -108,21 +111,18 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; +/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */ +ktime_t start_time;/* -* Protects read_stat and write_stat below. -* -* Cacheline aligned so that the layoutstats accounting done on -* every I/O to this stripe does not share cachelines with the
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 122
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.h +6
-4-* read-mostly members above, nor with the neighbouring stripes -* in mirror->dss[]: aligning the first member of the guarded -* set also rounds sizeof() up to a multiple of the cacheline, -* which is what makes the array stride line-aligned. +* A line each, and each carrying its own lock, so that a read and a +* write to this stripe neither serialise against each other nor +* share a line. Their alignment also rounds this structure's +* sizeof() up to a multiple of the cacheline, which is what keeps +* the mirror->dss[] stride line-aligned and so stops stripe i +* sharing a line with stripe i+1.
Same here.
Anna
Quoted 11 lines
fs/nfs/flexfilelayout/flexfilelayout.h +0
-3*/ -spinlock_t lock____cacheline_aligned_in_smp ;struct nfs4_ff_layoutstat read_stat;struct nfs4_ff_layoutstat write_stat; -/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */ -ktime_t start_time; };struct nfs4_ff_layout_mirror {Signature · 1 line
2.52.0
Anna Schumaker
maintainerRe: [PATCH 09/11] NFSv4/flexfiles: drop NFS4_FF_MIRROR_STAT_AVAIL
In reply to Mike SnitzerHi Mike,
Quoted 6 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
The "this needs reporting" hint was a bit in
nfs4_ff_layout_mirror::flags, set on every DS RPC and consumed once per report interval. It was wrong in both directions.
Quoted 9 lines
It cost too much.
set_bit()ran twice per RPC -- once from nfs4_ff_layout_stat_io_start_{read,write}() and again from the matching _end_ -- on a mirror-wide cacheline that every stripe of the mirror shares. That is the same false sharing the per-stripe lock was introduced to remove, and after the preceding patches gave each direction a line of its own it was the last write to the mirror's line left on the hot path. Two unconditional atomic read-modify-writes per RPC, at the 275k-450k RPC/s this series was measured at, to set a hint read once every 60 seconds.
I have read this sentence 3 times and can't quite figure out what it's trying to say.
Quoted 7 lines
And it said too little. The bit is per mirror, but the statistics it gates are per stripe, and
ff_layout_mirror_prepare_stats()tests it inside the per-stripe loop. The first stripe with a usable deviceid cleared it, so every other stripe of that mirror hit thetest_and_clearand skipped.
Quoted 9 lines
What the server sees today: for a mirror with N stripes, a LAYOUTSTATS report carries exactly one
ff_layoutupdate4, for the lowest usabledss_id, no matter how many stripes are busy or how the I/O is spread across them. Stripes 1..N-1 are reported only when the layout is returned. For the 32-stripe RAID0 layout this series targets, the server is picking data servers on 1/32 of the picture, and always the same 1/32 -- a stripe that is idle while the other 31 are saturated reports zeros, and a stripe that is saturated while stripe 0 is idle never reports at all.
I'm not sure we need the point-of-view change in the commit message. It would be nice if the entire message could be rephrased to be more concise and easier to follow.
Quoted 153 lines
So drop the bit rather than fix it. The reporting path already reads
ops_completedfor both directions under their locks; remember what it reported and skip a stripe whose count has not moved since. That is strictly more accurate than the flag was -- it is per stripe by construction, and it cannot be left set by I/O that has already been reported -- and it costs the I/O paths nothing.What the server sees after this patch: up to
dev_limitff_layoutupdate4entries, one per stripe that has completed I/O since its own last report, indss_idorder. The numbers themselves are unchanged -- same cumulativeops_completedandbytes_completed, same deviceid, same encoding -- there are just more of them, and they describe the stripes that did the work. A stripe with no completions since its last report is now omitted rather than reported as an unchanged repeat; a server differencing successive reports sees the same thing either way. LAYOUTRETURN still reports every stripe it can fit, as before.
nfs4_ff_layout_ds_stripe::last_reported_opsis deliberately out in the read-mostly region rather than in eithernfs4_ff_layoutstat: it is touched only byff_layout_mirror_prepare_stats(), about once per report interval, so it must not consume one of the 64 bytes an I/O dirties. It lands in the existing hole ahead ofread_stat, so sizeof is still 320. Both callers offf_layout_mirror_prepare_stats()hold the inode'si_lock, which serialises it; no new lock is needed.
NFS4_FF_MIRROR_STAT_AVAILwas the only bit ever defined onnfs4_ff_layout_mirror::flags, so the field goes too.Note that
dev_limitis 4 in both callers, so N > 4 stripes still cannot all be reported at once; the next patch rotates which ones are.Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 43 +++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 12 +++++-- 2 files changed, 34 insertions(+), 21 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +0
-1@@ -829,7 +829,6 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode, report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->read_stat, requested, now);spin_unlock (&dss_info->read_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)fs/nfs/flexfilelayout/flexfilelayout.c +21
-5pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());@@ -849,7 +848,6 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task, requested, completed, ktime_get(), task->tk_start);spin_unlock (&dss_info->read_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); } static void@@ -865,7 +863,6 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode, report =nfs4_ff_layoutstat_start_io (dss_info, &dss_info->write_stat, requested, now);spin_unlock (&dss_info->write_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags);if (report)pnfs_report_layoutstat (inode,nfs_io_gfp_mask ());@@ -889,7 +886,6 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task, requested, completed, ktime_get(), task->tk_start);spin_unlock (&dss_info->write_stat.lock); -set_bit (NFS4_FF_MIRROR_STAT_AVAIL , &mirror->flags); } static struct nfs4_ff_layout_ds *@@ -3156,6 +3152,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,struct nfs4_ff_layout_mirror *mirror;struct nfs4_ff_layout_ds_stripe *dss_info;struct nfs4_ff_layout_ds *mirror_ds; +__u64 read_count, read_bytes, write_count, write_bytes, ops;int i =0 , dss_id;rcu_read_lock ();@@ -3167,13 +3164,31 @@ ff_layout_mirror_prepare_stats(structpnfs_layout_hdr *lo, mirror_ds =rcu_dereference (dss_info->mirror_ds);if (IS_ERR_OR_NULL (mirror_ds))continue ; -if (!test_and_clear_bit (NFS4_FF_MIRROR_STAT_AVAIL , - &mirror->flags) && + +spin_lock (&dss_info->read_stat.lock); + read_count = dss_info->read_stat.io_stat.ops_completed; + read_bytes = dss_info->read_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->read_stat.lock); +spin_lock (&dss_info->write_stat.lock); + write_count = dss_info->write_stat.io_stat.ops_completed; + write_bytes = dss_info->write_stat.io_stat.bytes_completed; +spin_unlock (&dss_info->write_stat.lock); + +/* +* Nothing has completed on this stripe since it last +* reported, so do not spend one of the few devinfo +* slots on repeating those numbers. A LAYOUTRETURN is +* the last chance to report and says everything. +*/ + ops = read_count + write_count; +if (ops == dss_info->last_reported_ops && type !=NFS4_FF_OP_LAYOUTRETURN )continue ; +/* mirror refcount put in cleanup_layoutstats */ if (!refcount_inc_not_zero (&mirror->ref))continue ; + dss_info->last_reported_ops = ops;/* The pin holds a reference; it is exchanged out only * under i_lock. Put in ff_layout_free_layoutstats(). fs/nfs/flexfilelayout/flexfilelayout.c +4
-12*/ @@ -3183,18 +3198,10 @@ ff_layout_mirror_prepare_stats(structpnfs_layout_hdr *lo, NFS4_DEVICEID4_SIZE); devinfo->offset =0 ; devinfo->length =NFS4_MAX_UINT64 ; -spin_lock (&dss_info->read_stat.lock); - devinfo->read_count = - dss_info->read_stat.io_stat.ops_completed; - devinfo->read_bytes = - dss_info->read_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->read_stat.lock); -spin_lock (&dss_info->write_stat.lock); - devinfo->write_count = - dss_info->write_stat.io_stat.ops_completed; - devinfo->write_bytes = - dss_info->write_stat.io_stat.bytes_completed; -spin_unlock (&dss_info->write_stat.lock); + devinfo->read_count = read_count; + devinfo->read_bytes = read_bytes; + devinfo->write_count = write_count; + devinfo->write_bytes = write_bytes; devinfo->layout_type =LAYOUT_FLEX_FILES ; devinfo->ld_private.ops = &layoutstat_ops;fs/nfs/flexfilelayout/flexfilelayout.c +0
-0priv->dss_info = dss_info;fs/nfs/flexfilelayout/flexfilelayout.h +0
-0@@ -113,6 +113,15 @@ struct nfs4_ff_layout_ds_stripe {struct nfs_file_localio nfl;/* Published once by the first I/O; see
nfs4_ff_layoutstat_set_start_time()*/
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 123
Quoted 9 lines
fs/nfs/flexfilelayout/flexfilelayout.h +8
-0ktime_t start_time; +/* +* Completed ops, both directions, as of this stripe's last +* LAYOUTSTATS report. Touched only by +* ff_layout_mirror_prepare_stats(), so roughly once per report +* interval, and serialised by the inode's i_lock, which both of its +* callers hold. Deliberately out here rather than in either +* nfs4_ff_layoutstat: it must not cost the I/O paths a cacheline. +*/
Cut all but the first sentence of this comment, please.
Otherwise, it looks okay.
Thanks,
Anna
Quoted 19 lines
fs/nfs/flexfilelayout/flexfilelayout.h +1
-3+__u64 last_reported_ops;/* * A line each, and each carrying its own lock, so that a read and a * write to this stripe neither serialise against each other nor @@ -131,12 +140,9 @@ struct nfs4_ff_layout_mirror {u32 dss_count;struct nfs4_ff_layout_ds_stripe *dss;refcount_t ref; -unsigned long flags;u32 report_interval; }; -#define NFS4_FF_MIRROR_STAT_AVAIL (0) -struct nfs4_ff_layout_segment {struct pnfs_layout_segment generic_hdr;u64 stripe_unit;Signature · 1 line
2.52.0
Re: [PATCH 01/11] NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock
In reply to Anna SchumakerQuoted 63 lines · Anna Schumaker
Hi Mike,
Quoted 60 lines · Mike Snitzer
From: Jonathan Flynn <[email protected]>
Since striped layouts were introduced, a mirror owns an array of struct
nfs4_ff_layout_ds_stripeand each stripe carries its ownread_statandwrite_stat. The lock protecting those statistics stayed behind in the mirror, so every DS RPC in the layout serialises on the same spinlock twice: once in ff_layout_{read,write}_prepare_common() and again on completion in ff_layout_{read,write}_count_stats().For a RAID0-style layout (one mirror, many stripes) that lock is effectively per-file, and striping then multiplies RPC parallelism without multiplying lock parallelism.
On a 64-core client reading a single file from a 32-stripe layout with a 128KiB stripe unit (2MB
O_DIRECTreads, ~36.7 GB/s, ~275k NFSv3 READ RPCs/s, i.e. ~550k acquisitions/s of one lock), perf reports 45% of all CPU innative_queued_spin_lock_slowpath, and lock:contention_begin attributes 73% of the contention events to the two layoutstats paths:36.4%
rpc_exit_task->ff_layout_read_count_stats->ff_layout_read_record_layoutstats_done -> _raw_spin_lock 36.3% ff_layout_read_prepare_v3 -> ff_layout_read_record_layoutstats_start -> _raw_spin_lockMove the lock into struct
nfs4_ff_layout_ds_stripe, next to the statistics it protects, so that I/O to different stripes no longer contends. The reporting paths take the same per-stripe lock.
NFS4_FF_MIRROR_STAT_AVAILis a mirror-wide bit operation and does not need the lock; set it outside the critical section so the mirror's cacheline is not written under it.
ffl->last_report_timeis shared by every mirror and stripe of a layout and was already updated under different mirror locks, so it was never serialised. Make that explicit with acmpxchg(): whichever racer wins sends the LAYOUTSTATS report.Signed-off-by Jonathan Flynn <[email protected]>Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 45 +++++++++++++++----------- fs/nfs/flexfilelayout/flexfilelayout.h | 2 ++ 2 files changed, 28 insertions(+), 19 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +0
-5@@ -744,13 +747,17 @@ nfs4_ff_layoutstat_start_io(structnfs4_ff_layout_mirror *mirror, report_interval = (s64 )mirror->report_interval *1000LL ; else if (layoutstats_timer != 0) report_interval = (s64 )layoutstats_timer *1000LL ; -if (ktime_to_ms (ktime_sub (now, ffl->last_report_time)) >= - report_interval) { - ffl->last_report_time = now; -return true; - }
Quoted 16 lines
fs/nfs/flexfilelayout/flexfilelayout.c nfs4_ff_layoutstat_start_io(struct nfs4_ff_layout_mirror *mirror, · line 760
Quoted 11 lines
fs/nfs/flexfilelayout/flexfilelayout.c +10
-1-return false; +/* +* last_report_time is shared by every mirror and stripe of this +* layout, so it was never serialised by the mirror lock either. +* Make that explicit: whoever wins the cmpxchg() reports. +*/ + last_report =READ_ONCE (ffl->last_report_time); +if (ktime_to_ms (ktime_sub (now, last_report)) < report_interval) +return false; + +return try_cmpxchg64 (&ffl->last_report_time, &last_report, now);As far as I can tell, this is the first use of
try_cmpxchg64in any filesystem code. This has me worried it's the wrong function to be calling here. Is there any reason we can't useatomic64_try_cmpxchg(), which is what I see being used by other filesystem code?
Not seeing any other caller of atomic64_try_cmpxchg() that isn't using atomic64_t.
nfs4_flexfile_layout's last_report_time datatype is s64 (ktime_t), not atomic64_t.
Later in this series nfs4_ff_layoutstat_set_start_time() also uses it with nfs4_ff_layout_mirror's start_time.
Re: [PATCH 03/11] NFSv4/flexfiles: cacheline align the per-stripe layoutstats
In reply to Anna SchumakerQuoted 61 lines · Anna Schumaker
Hi Mike,
Quoted 58 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
Moving the layoutstats lock into struct
nfs4_ff_layout_ds_stripestopped stripes serialising on one spinlock, but it left them sharing cachelines with each other.The lock and the statistics it guards sat at the tail of a struct whose
sizeof()was 296 and whosealignof()was only 8, so in mirror->dss[] stripe i began at i * 296. 296 mod 64 = 40, so every stripe started 40 bytes further into a line than the last and no stripe's lock was line aligned. Acquiring stripe i's lock therefore invalidated the line holding stripe i+1's lock and counters. A RAID0-style layout -- one mirror, many stripes -- is exactly the case the per-stripe lock was introduced for, so this false sharing ate into the win.Align the first member of the guarded set. That does two things: the lock now starts on a line boundary, so the guarded set occupies lines of its own and shares none with the read-mostly members above; and it raises the struct's alignment to 64, which forces
sizeof()up to a multiple of 64 and makes the array stride line aligned. The stride is what actually separates stripe i from stripe i+1.Verified with pahole on flexfilelayout.o built with DWARF5. The layout is identical on
x86_64, arm64 4K pages and arm64 64K pages -- cachelines are 64 bytes on both architectures, so page size does not enter into it -- and was taken with neitherCONFIG_DEBUG_SPINLOCKnorCONFIG_PROVE_LOCKING, both of which fattenspinlock_tand change it:/* --- cacheline 3 boundary (192 bytes) --- */ spinlock_t lock __aligned(64); /* 192 4 */ /* XXX 4 bytes hole */ struct nfs4_ff_layoutstat read_stat; /* 200 72 */ struct nfs4_ff_layoutstat write_stat; /* 272 72 */ ktime_t start_time; /* 344 8 */ /* size: 384, cachelines: 6 */ } __attribute__((__aligned__(64)));A read RPC now dirties two lines of its own stripe and a write three, and neither disturbs a neighbouring stripe. Later patches give each direction a line of its own.
Cost is 296 -> 384 bytes per stripe, i.e. 2816 bytes for a 32 stripe layout, which is negligible beside the rest of the per-file layout state. 384 is also a multiple of 128, so the stride holds for implementations with a 128 byte coherency granule.
Note that the alignment attribute only orders the stripes relative to the start of mirror->dss[], not that array's own base. The base is aligned too, under either allocator
kvmalloc()may pick for it:- From the slab.
kmalloc()documents a size based alignment guarantee for power-of-two sizes only, but the smallest possible request is now 384 bytes, and every kmalloc bucket at or above that is a power of two whose objects SLUB lays out at multiples of the object size from a page aligned slab, so the base is at least 512 byte aligned. The exceptions, kmalloc-96 and kmalloc-192, are both smaller than one stripe and cannot be selected.
Quoted 7 lines
Quoted 5 lines
- From vmalloc. A stripe array wide enough to need a high order allocation is served by the
vmalloc()fallback instead, which returns page aligned memory.Either is comfortably more than the 64 bytes this needs.
This commit message is very wordy, and can probably be edited down.
OK, will fix. (I'll go over the entire series on this point.)
Quoted 30 lines
Quoted 30 lines
Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.h +6
-2@@ -14,6 +14,7 @@#define FF_FLAGS_NO_IO_THRU_MDS 2#define FF_FLAGS_NO_READ_IO 4 +#include <linux/cache.h> #include <linux/refcount.h> #include "../pnfs.h" @@ -86,8 +87,17 @@ struct nfs4_ff_layout_ds_stripe {const struct cred __rcu *ro_cred;const struct cred __rcu *rw_cred;struct nfs_file_localio nfl; -/* Protects read_stat, write_stat and start_time below */ -spinlock_t lock; +/* +* Protects read_stat, write_stat and start_time below. +* +* Cacheline aligned so that the layoutstats accounting done on +* every I/O to this stripe does not share cachelines with the
Quoted 8 lines
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 98
Quoted 4 lines
fs/nfs/flexfilelayout/flexfilelayout.h +4
-0+* read-mostly members above, nor with the neighbouring stripes +* in mirror->dss[]: aligning the first member of the guarded +* set also rounds sizeof() up to a multiple of the cacheline, +* which is what makes the array stride line-aligned. Let's leave the original comment alone, since most of this information is duplicated in the commit message and can be easily found with
git blame.
Sure.
Quoted 4 lines
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_ds_stripe { · line 100
Quoted 2 lines
fs/nfs/flexfilelayout/flexfilelayout.h +2
-0+*/ +spinlock_t lock____cacheline_aligned_in_smp ;Making this cacheline aligned seems like a good idea!
Yes, thanks.
Mike
Re: [PATCH 04/11] NFSv4/flexfiles: make dss_info->start_time write-once
In reply to Anna SchumakerQuoted 18 lines · Anna Schumaker
Hi MIke,
Quoted 7 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
start_timewas written under the stripe lock innfs4_ff_layoutstat_start_io()and read outside it inff_layout_encode_ff_layoutupdate(). Pre-existing --mirror->lockhad the same shape -- and benign, since it is a single naturally aligned 8 byte value, but it is an unannotated race either way.We can probably drop this last sentence.
Quoted 4 lines
The lock is the wrong tool for it.
ffl_durationis measured from this stripe's first I/O, so the field is written once and read-only for the rest of the stripe's life. Say that instead: publish it with
Quoted 8 lines
This portion needs editing too. My brain tripped over "Say that instead" while reading.
Quoted 3 lines
try_cmpxchg64()and let the first I/O win, and read it withREAD_ONCE(). Only the first I/O to a stripe pays the read-modify-write; every one after it short circuits on theREAD_ONCE().This sentence is just rewording the previous sentence.
Sorry, I will review all commit headers and code comments.
Quoted 35 lines
Quoted 35 lines
ffl->last_report_timetwo lines away is already published this way, so the pattern is not new to this file.This also stops
start_timedepending on there being exactly one lock per stripe, which a later patch changes.Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 21 ++++++++++++++++++--- fs/nfs/flexfilelayout/flexfilelayout.h | 3 ++- 2 files changed, 20 insertions(+), 4 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +2
-0@@ -729,6 +729,22 @@ nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now)return ktime_sub (now, start); } +/* +* ffl_duration is measured from this stripe's first I/O, so
start_timeis
+ * written once and read-only for the rest of the stripe's life. Publish it
+ * with a cmpxchg and let the first I/O win, rather than depending on a lock
+ * that every reader would then have to take.
+ */fs/nfs/flexfilelayout/flexfilelayout.c +2
-0+static void +nfs4_ff_layoutstat_set_start_time(struct nfs4_ff_layout_ds_stripe
*dss_info,
Quoted 8 lines
fs/nfs/flexfilelayout/flexfilelayout.c nfs4_ff_end_busy_timer(struct nfs4_ff_busy_timer *timer, ktime_t now) · line 745
Quoted 6 lines
fs/nfs/flexfilelayout/flexfilelayout.c +6
-0+ ktime_t now) +{ +ktime_t unset =0 ; + +if (!READ_ONCE (dss_info->start_time)) +try_cmpxchg64 (&dss_info->start_time, &unset, now);I have the same
atomic64_try_cmpxchg()question as in patch 1 here.
Right, I noted that in my reply to patch 1. I don't think its something to be overly concerned about. Given the use of ktime_t (s64) it really is the more appropriate interface AFAICT.
Happy to take instruction from others on this though!
Thanks,
Mike
Anna Schumaker
maintainerRe: [PATCH 10/11] NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes
In reply to Mike SnitzerHi Mike,
Quoted 7 lines · Mike Snitzer
From: Mike Snitzer <[email protected]>
Both callers of
ff_layout_mirror_prepare_stats()cap it at fourff_layoutupdate4entries --PNFS_LAYOUTSTATS_MAXDEVfor LAYOUTSTATS andARRAY_SIZE(ff_args->devinfo), which isFF_LAYOUTSTATS_MAXDEV, for LAYOUTRETURN. The scan walks mirror->dss[] from index 0 every time and stops once the budget is spent.
Why does the commit message need to specify starting at index 0? That's just standard looping over an array behavior.
Quoted 9 lines
What the server sees without this patch: for a mirror with more than four busy stripes, the same low
dss_idsevery report, forever. The previous patch widened that from one stripe to four -- it did not stop it being the same four. On the 32-stripe RAID0 layout this series targets, stripes 4..31 are reported only when the layout is returned, so a data server that is only ever backing a high stripe index stays invisible to the server's placement decisions no matter how much I/O it does.
The "what the server sees" interludes seem randomly placed in this patch and break the flow of what I'm reading. I don't think it's adding much.
Quoted 55 lines
Remember where each mirror's scan stopped and resume there. The cursor advances past each stripe that actually fills a slot, so a report picks up where its predecessor left off and idle stripes, which are skipped without consuming a slot, do not stall the rotation.
What the server sees with this patch: still up to four entries per report, but rotating, so every busy stripe of a mirror is reported within
ceil(busy_stripes / 4)report intervals -- eight intervals, or about eight minutes at the default 60 second interval, for 32 busy stripes. A given stripe's numbers therefore arrive less often than before but, unlike before, they arrive. The values are unchanged:
cumulativeops_completedandbytes_completedsince the layout was granted, so a server differencing successive reports for a stripe still gets a correct delta, just over a longer window. Nothing about the encoding, the deviceid or the ordering within a report changes.The mirror loop gets the same
dev_limittest as the stripe loop, so a full report stops scanning instead of walking the remaining mirrors to break immediately on each.[jonathan.flynn: read
dss_report_startonce per mirror, before the stripe
loop. The loop advances it after each stripe it reports, so re-reading it moved the index twice: stripes were skipped, and on a mirror with stripes not yet written (and so skipped) the index wrapped back onto a stripe already reported. That put the same deviceid in oneff_iostats4array twice with identical counters.]Signed-off-by Jonathan Flynn <[email protected]>Signed-off-by Mike Snitzer <[email protected]>--- fs/nfs/flexfilelayout/flexfilelayout.c | 22 +++++++++++++++++++--- fs/nfs/flexfilelayout/flexfilelayout.h | 7 +++++++ 2 files changed, 26 insertions(+), 3 deletions(-)fs/nfs/flexfilelayout/flexfilelayout.c +5
-3@@ -3153,14 +3153,28 @@ ff_layout_mirror_prepare_stats(structpnfs_layout_hdr *lo, struct nfs4_ff_layout_ds_stripe *dss_info;struct nfs4_ff_layout_ds *mirror_ds;__u64 read_count, read_bytes, write_count, write_bytes, ops; -int i =0 , dss_id; +u32 dss_id, scanned, base; +int i =0 ;rcu_read_lock ();list_for_each_entry (mirror, &ff_layout->mirrors, mirrors) { -for (dss_id =0 ; dss_id < mirror->dss_count; ++dss_id) { - dss_info = &mirror->dss[dss_id]; +if (i >= dev_limit) +break ; + base = mirror->dss_report_start;
fs/nfs/flexfilelayout/flexfilelayout.c ff_layout_mirror_prepare_stats(struct · line 3174
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.c +8
-0+for (scanned =0 ; scanned < mirror->dss_count; ++scanned) {if (i >= dev_limit)break ; +/* Resume where the last report of this mirror left +* off; one subtraction suffices to wrap, as both +* terms are below dss_count. Use the start as it +* was on entry: this loop advances it for the next +* report, and re-reading it would skip stripes and +* wrap back onto one already reported. +*/
The code is self-documenting enough to not need this comment.
Quoted 23 lines
fs/nfs/flexfilelayout/flexfilelayout.c +4
-0+ dss_id = base + scanned; +if (dss_id >= mirror->dss_count) + dss_id -= mirror->dss_count; + dss_info = &mirror->dss[dss_id]; mirror_ds =rcu_dereference (dss_info->mirror_ds);if (IS_ERR_OR_NULL (mirror_ds))fs/nfs/flexfilelayout/flexfilelayout.c +2
-0continue ;@@ -3189,6 +3203,8 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo, if (!refcount_inc_not_zero(&mirror->ref)) continue; dss_info->last_reported_ops = ops; + mirror->dss_report_start = + dss_id +1 < mirror->dss_count ? dss_id +1 :0 ;/* The pin holds a reference; it is exchanged out only * under i_lock. Put in ff_layout_free_layoutstats(). fs/nfs/flexfilelayout/flexfilelayout.c +0
-0*/ fs/nfs/flexfilelayout/flexfilelayout.h +0
-0
fs/nfs/flexfilelayout/flexfilelayout.h struct nfs4_ff_layout_mirror { · line 150
Quoted 10 lines
fs/nfs/flexfilelayout/flexfilelayout.h +7
-0@@ -141,6 +141,13 @@ struct nfs4_ff_layout_mirror {struct nfs4_ff_layout_ds_stripe *dss;refcount_t ref;u32 report_interval; +/* +* dss[] index to begin the next LAYOUTSTATS scan of this mirror at, +* so that a mirror with more stripes than a report has room for +* rotates through them rather than always reporting the lowest. +* Serialised by the inode's i_lock, like last_reported_ops. +*/ +u32 dss_report_start;
Can this be renamed to either dss_report_base or dss_report_index? Then we can drop the wordy comment entirely.
Thanks,
Anna
Quoted 5 lines
fs/nfs/flexfilelayout/flexfilelayout.h +0
-0};struct nfs4_ff_layout_segment {Signature · 1 line
2.52.0