block: add bio split counters for iostat
Subsystems (2)
cover block: add bio split counters for iostat
There's a mechanism which splits IO's if their size is greater than queue/max_sectors_kb setting. But there's no counter for split requests. Having splits counter helps to understand workload profile and necessity
of settings tuning.
bi_flags converted to int from short.
Add BIO_SPLITTED flag and set it in bio_split() for new bio. Add splits[] counters.
In blk_account_io_start() check the BIO_SPLITTED flag and if it's set increment appropriate splits counter.
Aleksei Zakharov (2):
block: extend bi_flags to int and introduce BIO_SPLITTED
block: add iostat counters for requests splits
Documentation/ABI/testing/procfs-diskstats | 3 +++ Documentation/ABI/testing/sysfs-block | 5 ++++- Documentation/admin-guide/iostats.rst | 10 ++++++++++ block/bio.c | 2 ++ block/blk-core.c | 2 ++ block/genhd.c | 8 ++++++-- block/partition-generic.c | 8 ++++++-- include/linux/blk_types.h | 5 +++-- include/linux/genhd.h | 1 + 9 files changed, 37 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 block: extend bi_flags to int and introduce BIO_SPLITTED
BIO_SPLITTED flag will be used to gather statistics for splitted bio's.
--- include/linux/blk_types.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
include/linux/blk_types.h +3 -2
@@ -148,7 +148,7 @@ struct bio {* top bits REQ_OP. Use * accessors. */ -unsigned short bi_flags;/* status, etc and bvec pool number */ +unsigned int bi_flags;/* status, etc and bvec pool number */ unsigned short bi_ioprio;unsigned short bi_write_hint;blk_status_t bi_status;
@@ -207,6 +207,7 @@ struct bio {* bio flags */ enum { +BIO_SPLITTED ,/* set if bio was split in bio_split */ BIO_NO_PAGE_REF ,/* don't put release vec pages */ BIO_CLONED ,/* doesn't own data */ BIO_BOUNCED ,/* bio is a bounce bio */
@@ -240,7 +241,7 @@ enum {* freed. */ #define BVEC_POOL_BITS (3) -#define BVEC_POOL_OFFSET (16 - BVEC_POOL_BITS) +#define BVEC_POOL_OFFSET (32 - BVEC_POOL_BITS)#define BVEC_POOL_IDX (bio) ((bio)->bi_flags >> BVEC_POOL_OFFSET)#if (1 <<BVEC_POOL_BITS ) < (BVEC_POOL_NR +1 )# error "BVEC_POOL_BITS is too small"
Signature · 1 line
2.17.1
PATCH 2/2 block: add iostat counters for requests splits
In reply to Aleksei ZakharovI/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.
Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.
This patch adds three counters to /sys/class/block/$dev/stat and /proc/diskstats: number of reads, writes and discards splitted.
There's also counter for flush requests, but it is ignored, because flush cannot be splitted.
--- Documentation/ABI/testing/procfs-diskstats | 3 +++ Documentation/ABI/testing/sysfs-block | 5 ++++- Documentation/admin-guide/iostats.rst | 10 ++++++++++ block/bio.c | 2 ++ block/blk-core.c | 2 ++ block/genhd.c | 8 ++++++-- block/partition-generic.c | 8 ++++++-- include/linux/genhd.h | 1 + 8 files changed, 34 insertions(+), 5 deletions(-)
Documentation/ABI/testing/procfs-diskstats +3 -0
@@ -33,5 +33,8 @@ Description: 19 - flush requests completed successfully 20 - time spent flushing + 21 - reads splitted + 22 - writes splitted + 23 - discards splitted For more details refer to Documentation/admin-guide/iostats.rst
Documentation/ABI/testing/sysfs-block +4 -1
@@ -3,7 +3,7 @@ Date: February 2008 Contact: Jerome Marchand <[email protected]> Description: The /sys/block/<disk>/stat files displays the I/O - statistics of disk <disk>. They contain 11 fields: + statistics of disk <disk>. They contain 20 fields: 1 - reads completed successfully 2 - reads merged 3 - sectors read
@@ -21,6 +21,9 @@ Description: 15 - time spent discarding (ms) 16 - flush requests completed 17 - time spent flushing (ms) + 18 - reads splitted + 19 - writes splitted + 20 - discrads splitted For more details refer Documentation/admin-guide/iostats.rst
Documentation/admin-guide/iostats.rst +10 -0
@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed Field 17 -- # of milliseconds spent flushing This is the total number of milliseconds spent by all flush requests. +Field 18 -- # of reads splitted, field 19 -- # of writes splitted + This is the total number of requests splitted before queue. + + The maximum size of I/O is limited by queue/max_sectors_kb. + If size of I/O is greater than this limit, it will be splitted + as many times as needed to keep I/O size withing the limit. + +Field 20 -- # of discrads Splitted + See description of fileds 18 and 19. + To avoid introducing performance bottlenecks, no locks are held while modifying these counters. This implies that minor inaccuracies may be introduced when changes collide, so (for instance) adding up all the
block/bio.c +2 -0
@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,if (bio_flagged (bio,BIO_TRACE_COMPLETION ))bio_set_flag (split,BIO_TRACE_COMPLETION ); +bio_set_flag (split,BIO_SPLITTED ); +return split; }EXPORT_SYMBOL (bio_split);
block/blk-core.c +2 -0
@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io) }part_inc_in_flight (rq->q, part, rw); rq->part = part; +if (bio_flagged (rq->bio,BIO_SPLITTED )) +part_stat_inc (part, splits[rw]); }update_io_ticks (part, jiffies);
block/genhd.c +6 -2
@@ -1386,7 +1386,8 @@ static int diskstats_show(struct seq_file *seqf, void *v)"%lu %lu %lu %u " "%u %u %u " "%lu %lu %lu %u " -"%lu %u" +"%lu %u " +"%lu %lu %lu" "\n" ,MAJOR (part_devt (hd)),MINOR (part_devt (hd)), disk_name(gp, hd->partno, buf),
@@ -1406,7 +1407,10 @@ static int diskstats_show(struct seq_file *seqf, void *v)part_stat_read (hd, sectors[STAT_DISCARD ]), (unsigned int )part_stat_read_msecs (hd,STAT_DISCARD ),part_stat_read (hd, ios[STAT_FLUSH ]), - (unsigned int )part_stat_read_msecs (hd,STAT_FLUSH ) + (unsigned int )part_stat_read_msecs (hd,STAT_FLUSH ), +part_stat_read (hd, splits[STAT_READ ]), +part_stat_read (hd, splits[STAT_WRITE ]), +part_stat_read (hd, splits[STAT_DISCARD ]) ); }disk_part_iter_exit (&piter);
block/partition-generic.c +6 -2
@@ -128,7 +128,8 @@ ssize_t part_stat_show(struct device *dev,"%8lu %8lu %8llu %8u " "%8u %8u %8u " "%8lu %8lu %8llu %8u " -"%8lu %8u" +"%8lu %8u " +"%8lu %8lu %8lu" "\n" ,part_stat_read (p, ios[STAT_READ ]), part_stat_read(p, merges[STAT_READ]),
@@ -146,7 +147,10 @@ ssize_t part_stat_show(struct device *dev, (unsigned long long )part_stat_read (p, sectors[STAT_DISCARD ]), (unsigned int )part_stat_read_msecs (p,STAT_DISCARD ),part_stat_read (p, ios[STAT_FLUSH ]), - (unsigned int)part_stat_read_msecs(p, STAT_FLUSH)); + (unsigned int )part_stat_read_msecs (p,STAT_FLUSH ), +part_stat_read (p, splits[STAT_READ ]), +part_stat_read (p, splits[STAT_WRITE ]), + part_stat_read(p, splits[STAT_DISCARD])); } ssize_t part_inflight_show(struct device *dev, struct device_attribute *attr,
include/linux/genhd.h +1 -0
@@ -88,6 +88,7 @@ struct disk_stats {unsigned long sectors[NR_STAT_GROUPS ];unsigned long ios[NR_STAT_GROUPS ];unsigned long merges[NR_STAT_GROUPS ]; +unsigned long splits[NR_STAT_GROUPS ];unsigned long io_ticks;unsigned long time_in_queue;local_t in_flight[2 ];
Signature · 1 line
2.17.1
Re: [PATCH 2/2] block: add iostat counters for requests splits
Quoted 7 lines · Aleksei Zakharov
I/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.
Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.
IO split should be one static behavior, so if one bio's sector/size is provided, it is easy to figure out the split pattern, not sure if it is worth of adding statistics
in iostat.
That said the IO split analysis can be done simply as post-process of IO trace, or even it can be done runtime via bcc/bpftrace script without much difficulty.
Quoted 93 lines
This patch adds three counters to
/sys/class/block/$dev/statand/proc/diskstats: number of reads, writes and discards splitted.There's also counter for flush requests, but it is ignored, because flush cannot be splitted.
--- Documentation/ABI/testing/procfs-diskstats | 3 +++ Documentation/ABI/testing/sysfs-block | 5 ++++- Documentation/admin-guide/iostats.rst | 10 ++++++++++ block/bio.c | 2 ++ block/blk-core.c | 2 ++ block/genhd.c | 8 ++++++-- block/partition-generic.c | 8 ++++++-- include/linux/genhd.h | 1 + 8 files changed, 34 insertions(+), 5 deletions(-)Documentation/ABI/testing/procfs-diskstats +3
-0@@ -33,5 +33,8 @@ Description: 19 - flush requests completed successfully 20 - time spent flushing + 21 - reads splitted + 22 - writes splitted + 23 - discards splitted For more details refer to Documentation/admin-guide/iostats.rstDocumentation/ABI/testing/sysfs-block +4
-1@@ -3,7 +3,7 @@ Date: February 2008 Contact: Jerome Marchand <[email protected]> Description: The /sys/block/<disk>/stat files displays the I/O - statistics of disk <disk>. They contain 11 fields: + statistics of disk <disk>. They contain 20 fields: 1 - reads completed successfully 2 - reads merged 3 - sectors read@@ -21,6 +21,9 @@ Description: 15 - time spent discarding (ms) 16 - flush requests completed 17 - time spent flushing (ms) + 18 - reads splitted + 19 - writes splitted + 20 - discrads splitted For more details refer Documentation/admin-guide/iostats.rstDocumentation/admin-guide/iostats.rst +10
-0@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed Field 17 -- # of milliseconds spent flushing This is the total number of milliseconds spent by all flush requests. +Field 18 -- # of reads splitted, field 19 -- # of writes splitted + This is the total number of requests splitted before queue. + + The maximum size of I/O is limited by queue/max_sectors_kb. + If size of I/O is greater than this limit, it will be splitted + as many times as needed to keep I/O size withing the limit. + +Field 20 -- # of discrads Splitted + See description of fileds 18 and 19. + To avoid introducing performance bottlenecks, no locks are held while modifying these counters. This implies that minor inaccuracies may be introduced when changes collide, so (for instance) adding up all theblock/bio.c +2
-0@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,if (bio_flagged (bio,BIO_TRACE_COMPLETION ))bio_set_flag (split,BIO_TRACE_COMPLETION ); +bio_set_flag (split,BIO_SPLITTED ); +return split; }EXPORT_SYMBOL (bio_split);block/blk-core.c +0
-0
block/blk-core.c void blk_account_io_start(struct request *rq, bool new_io) · line 1383
Quoted 5 lines
block/blk-core.c +2
-0@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io) }part_inc_in_flight (rq->q, part, rw); rq->part = part; +if (bio_flagged (rq->bio,BIO_SPLITTED )) +part_stat_inc (part, splits[rw]);
It is the only use of BIO_SPLITTED, and it should have been done as one rq flag, then extra change in bio struct can be avoided. If the bio is splitted can be easily
figured out in blk_mq_make_request(), then you may pass that info to blk_mq_bio_to_request().
Thanks,
Ming
Re: [PATCH 2/2] block: add iostat counters for requests splits
вт, 10 дек. 2019 г. в 06:09, Ming Lei <[email protected]>:
Quoted 10 lines · Aleksei Zakharov
Quoted 7 lines
I/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.
Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.
Quoted 7 lines
IO split should be one static behavior, so if one bio's sector/size is provided, it is easy to figure out the split pattern, not sure if it is worth of adding statistics
in iostat.That said the IO split analysis can be done simply as post-process of IO trace, or even it can be done runtime via bcc/bpftrace script without much difficulty.
Yes, you're right, it can be done via bpftrace. But iostat is just well-known and overall used tool. It's better to have most of stats with default tool, than have a lot of tools for different parts of one subsystem layer, IMO.
Quoted 97 lines
Quoted 97 lines
This patch adds three counters to
/sys/class/block/$dev/statand/proc/diskstats: number of reads, writes and discards splitted.There's also counter for flush requests, but it is ignored, because flush cannot be splitted.
--- Documentation/ABI/testing/procfs-diskstats | 3 +++ Documentation/ABI/testing/sysfs-block | 5 ++++- Documentation/admin-guide/iostats.rst | 10 ++++++++++ block/bio.c | 2 ++ block/blk-core.c | 2 ++ block/genhd.c | 8 ++++++-- block/partition-generic.c | 8 ++++++-- include/linux/genhd.h | 1 + 8 files changed, 34 insertions(+), 5 deletions(-)Documentation/ABI/testing/procfs-diskstats +3
-0@@ -33,5 +33,8 @@ Description: 19 - flush requests completed successfully 20 - time spent flushing + 21 - reads splitted + 22 - writes splitted + 23 - discards splitted For more details refer to Documentation/admin-guide/iostats.rstDocumentation/ABI/testing/sysfs-block +4
-1@@ -3,7 +3,7 @@ Date: February 2008 Contact: Jerome Marchand <[email protected]> Description: The /sys/block/<disk>/stat files displays the I/O - statistics of disk <disk>. They contain 11 fields: + statistics of disk <disk>. They contain 20 fields: 1 - reads completed successfully 2 - reads merged 3 - sectors read@@ -21,6 +21,9 @@ Description: 15 - time spent discarding (ms) 16 - flush requests completed 17 - time spent flushing (ms) + 18 - reads splitted + 19 - writes splitted + 20 - discrads splitted For more details refer Documentation/admin-guide/iostats.rstDocumentation/admin-guide/iostats.rst +10
-0@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed Field 17 -- # of milliseconds spent flushing This is the total number of milliseconds spent by all flush requests. +Field 18 -- # of reads splitted, field 19 -- # of writes splitted + This is the total number of requests splitted before queue. + + The maximum size of I/O is limited by queue/max_sectors_kb. + If size of I/O is greater than this limit, it will be splitted + as many times as needed to keep I/O size withing the limit. + +Field 20 -- # of discrads Splitted + See description of fileds 18 and 19. + To avoid introducing performance bottlenecks, no locks are held while modifying these counters. This implies that minor inaccuracies may be introduced when changes collide, so (for instance) adding up all theblock/bio.c +2
-0@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,if (bio_flagged (bio,BIO_TRACE_COMPLETION ))bio_set_flag (split,BIO_TRACE_COMPLETION ); +bio_set_flag (split,BIO_SPLITTED ); +return split; }EXPORT_SYMBOL (bio_split);block/blk-core.c +0
-0@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io) }part_inc_in_flight (rq->q, part, rw); rq->part = part;
Quoted 8 lines
block/blk-core.c void blk_account_io_start(struct request *rq, bool new_io) · line 1383
Quoted 2 lines
block/blk-core.c +2
-0+if (bio_flagged (rq->bio,BIO_SPLITTED )) +part_stat_inc (part, splits[rw]);It is the only use of
BIO_SPLITTED, and it should have been done as one rq flag, then extra change in bio struct can be avoided. If the bio is splitted can be easily
figured out inblk_mq_make_request(), then you may pass that info toblk_mq_bio_to_request().
Thanks for pointing that out!
I'll rethink this patch.
Quoted 3 lines
Thanks,
Ming
Signature · 5 lines
Best Regards, Aleksei Zakharov System administrator www.selectel.com