Skip to contentSkip to conversation

block: add bio split counters for iostat

PATCH In review Aleksei Zakharov 2 participants 5 messages Dec 9, 2019 – Dec 10, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover block: add bio split counters for iostat

There's a mechanism which splits IO's if their size is greater than queue/max_sectors_kb setting. But there's no counter for split requests. Having splits counter helps to understand workload profile and necessity
of settings tuning.

bi_flags converted to int from short.
Add BIO_SPLITTED flag and set it in bio_split() for new bio. Add splits[] counters.
In blk_account_io_start() check the BIO_SPLITTED flag and if it's set increment appropriate splits counter.

Aleksei Zakharov (2):
block: extend bi_flags to int and introduce BIO_SPLITTED
block: add iostat counters for requests splits

 Documentation/ABI/testing/procfs-diskstats |  3 +++
 Documentation/ABI/testing/sysfs-block      |  5 ++++-
 Documentation/admin-guide/iostats.rst      | 10 ++++++++++
 block/bio.c                                |  2 ++
 block/blk-core.c                           |  2 ++
 block/genhd.c                              |  8 ++++++--
 block/partition-generic.c                  |  8 ++++++--
 include/linux/blk_types.h                  |  5 +++--
 include/linux/genhd.h                      |  1 +
 9 files changed, 37 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 block: extend bi_flags to int and introduce BIO_SPLITTED

BIO_SPLITTED flag will be used to gather statistics for splitted bio's.

---
 include/linux/blk_types.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
include/linux/blk_types.h +3 -2
index 70254ae11769..7ff32e7c48fb 100644
@@ -148,7 +148,7 @@ struct bio {
 						 * top bits REQ_OP. Use
 						 * accessors.
 						 */
-	unsigned short		bi_flags;	/* status, etc and bvec pool number */
+	unsigned int		bi_flags;	/* status, etc and bvec pool number */
 	unsigned short		bi_ioprio;
 	unsigned short		bi_write_hint;
 	blk_status_t		bi_status;
@@ -207,6 +207,7 @@ struct bio {
  * bio flags
  */
 enum {
+	BIO_SPLITTED,		/* set if bio was split in bio_split */
 	BIO_NO_PAGE_REF,	/* don't put release vec pages */
 	BIO_CLONED,		/* doesn't own data */
 	BIO_BOUNCED,		/* bio is a bounce bio */
@@ -240,7 +241,7 @@ enum {
  * freed.
  */
 #define BVEC_POOL_BITS		(3)
-#define BVEC_POOL_OFFSET	(16 - BVEC_POOL_BITS)
+#define BVEC_POOL_OFFSET	(32 - BVEC_POOL_BITS)
 #define BVEC_POOL_IDX(bio)	((bio)->bi_flags >> BVEC_POOL_OFFSET)
 #if (1<< BVEC_POOL_BITS) < (BVEC_POOL_NR+1)
 # error "BVEC_POOL_BITS is too small"
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/2 block: add iostat counters for requests splits

In reply to Aleksei Zakharov

I/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.

Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.

This patch adds three counters to /sys/class/block/$dev/stat and /proc/diskstats: number of reads, writes and discards splitted.

There's also counter for flush requests, but it is ignored, because flush cannot be splitted.

---
 Documentation/ABI/testing/procfs-diskstats |  3 +++
 Documentation/ABI/testing/sysfs-block      |  5 ++++-
 Documentation/admin-guide/iostats.rst      | 10 ++++++++++
 block/bio.c                                |  2 ++
 block/blk-core.c                           |  2 ++
 block/genhd.c                              |  8 ++++++--
 block/partition-generic.c                  |  8 ++++++--
 include/linux/genhd.h                      |  1 +
 8 files changed, 34 insertions(+), 5 deletions(-)
Documentation/ABI/testing/procfs-diskstats +3 -0
index 70dcaf2481f4..e1473e93d901 100644
@@ -33,5 +33,8 @@ Description:
 
 		19 - flush requests completed successfully
 		20 - time spent flushing
+		21 - reads splitted
+		22 - writes splitted
+		23 - discards splitted
 
 		For more details refer to Documentation/admin-guide/iostats.rst
Documentation/ABI/testing/sysfs-block +4 -1
index ed8c14f161ee..ffbac0e72508 100644
@@ -3,7 +3,7 @@ Date:		February 2008
 Contact:	Jerome Marchand <[email protected]>
 Description:
 		The /sys/block/<disk>/stat files displays the I/O
-		statistics of disk <disk>. They contain 11 fields:
+		statistics of disk <disk>. They contain 20 fields:
 		 1 - reads completed successfully
 		 2 - reads merged
 		 3 - sectors read
@@ -21,6 +21,9 @@ Description:
 		15 - time spent discarding (ms)
 		16 - flush requests completed
 		17 - time spent flushing (ms)
+		18 - reads splitted
+		19 - writes splitted
+		20 - discrads splitted
 		For more details refer Documentation/admin-guide/iostats.rst
 
 
Documentation/admin-guide/iostats.rst +10 -0
index 4f0462af3ca7..7f3f374b82b7 100644
@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed
 Field 17 -- # of milliseconds spent flushing
     This is the total number of milliseconds spent by all flush requests.
 
+Field 18 -- # of reads splitted, field 19 -- # of writes splitted
+    This is the total number of requests splitted before queue.
+
+    The maximum size of I/O is limited by queue/max_sectors_kb.
+    If size of I/O is greater than this limit, it will be splitted
+    as many times as needed to keep I/O size withing the limit.
+
+Field 20 -- # of discrads Splitted
+    See description of fileds 18 and 19.
+
 To avoid introducing performance bottlenecks, no locks are held while
 modifying these counters.  This implies that minor inaccuracies may be
 introduced when changes collide, so (for instance) adding up all the
block/bio.c +2 -0
index 8f0ed6228fc5..c8a051e128f8 100644
@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,
 	if (bio_flagged(bio, BIO_TRACE_COMPLETION))
 		bio_set_flag(split, BIO_TRACE_COMPLETION);
 
+	bio_set_flag(split, BIO_SPLITTED);
+
 	return split;
 }
 EXPORT_SYMBOL(bio_split);
block/blk-core.c +2 -0
index f0d82227a2fc..776d28b9a5bf 100644
@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io)
 		}
 		part_inc_in_flight(rq->q, part, rw);
 		rq->part = part;
+		if (bio_flagged(rq->bio, BIO_SPLITTED))
+			part_stat_inc(part, splits[rw]);
 	}
 
 	update_io_ticks(part, jiffies);
block/genhd.c +6 -2
index ff6268970ddc..adb38baa24dc 100644
@@ -1386,7 +1386,8 @@ static int diskstats_show(struct seq_file *seqf, void *v)
 			   "%lu %lu %lu %u "
 			   "%u %u %u "
 			   "%lu %lu %lu %u "
-			   "%lu %u"
+			   "%lu %u "
+			   "%lu %lu %lu"
 			   "\n",
 			   MAJOR(part_devt(hd)), MINOR(part_devt(hd)),
 			   disk_name(gp, hd->partno, buf),
@@ -1406,7 +1407,10 @@ static int diskstats_show(struct seq_file *seqf, void *v)
 			   part_stat_read(hd, sectors[STAT_DISCARD]),
 			   (unsigned int)part_stat_read_msecs(hd, STAT_DISCARD),
 			   part_stat_read(hd, ios[STAT_FLUSH]),
-			   (unsigned int)part_stat_read_msecs(hd, STAT_FLUSH)
+			   (unsigned int)part_stat_read_msecs(hd, STAT_FLUSH),
+			   part_stat_read(hd, splits[STAT_READ]),
+			   part_stat_read(hd, splits[STAT_WRITE]),
+			   part_stat_read(hd, splits[STAT_DISCARD])
 			);
 	}
 	disk_part_iter_exit(&piter);
block/partition-generic.c +6 -2
index 3db8b73a96b1..61e26ec21256 100644
@@ -128,7 +128,8 @@ ssize_t part_stat_show(struct device *dev,
 		"%8lu %8lu %8llu %8u "
 		"%8u %8u %8u "
 		"%8lu %8lu %8llu %8u "
-		"%8lu %8u"
+		"%8lu %8u "
+		"%8lu %8lu %8lu"
 		"\n",
 		part_stat_read(p, ios[STAT_READ]),
 		part_stat_read(p, merges[STAT_READ]),
@@ -146,7 +147,10 @@ ssize_t part_stat_show(struct device *dev,
 		(unsigned long long)part_stat_read(p, sectors[STAT_DISCARD]),
 		(unsigned int)part_stat_read_msecs(p, STAT_DISCARD),
 		part_stat_read(p, ios[STAT_FLUSH]),
-		(unsigned int)part_stat_read_msecs(p, STAT_FLUSH));
+		(unsigned int)part_stat_read_msecs(p, STAT_FLUSH),
+		part_stat_read(p, splits[STAT_READ]),
+		part_stat_read(p, splits[STAT_WRITE]),
+		part_stat_read(p, splits[STAT_DISCARD]));
 }
 
 ssize_t part_inflight_show(struct device *dev, struct device_attribute *attr,
include/linux/genhd.h +1 -0
index 8b5330dd5ac0..ea502e7b12b6 100644
@@ -88,6 +88,7 @@ struct disk_stats {
 	unsigned long sectors[NR_STAT_GROUPS];
 	unsigned long ios[NR_STAT_GROUPS];
 	unsigned long merges[NR_STAT_GROUPS];
+	unsigned long splits[NR_STAT_GROUPS];
 	unsigned long io_ticks;
 	unsigned long time_in_queue;
 	local_t in_flight[2];
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH 2/2] block: add iostat counters for requests splits

Quoted 7 lines · Aleksei Zakharov

I/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.

Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.

IO split should be one static behavior, so if one bio's sector/size is provided, it is easy to figure out the split pattern, not sure if it is worth of adding statistics
in iostat.

That said the IO split analysis can be done simply as post-process of IO trace, or even it can be done runtime via bcc/bpftrace script without much difficulty.

Quoted 93 lines

This patch adds three counters to /sys/class/block/$dev/stat and /proc/diskstats: number of reads, writes and discards splitted.

There's also counter for flush requests, but it is ignored, because flush cannot be splitted.

---
 Documentation/ABI/testing/procfs-diskstats |  3 +++
 Documentation/ABI/testing/sysfs-block      |  5 ++++-
 Documentation/admin-guide/iostats.rst      | 10 ++++++++++
 block/bio.c                                |  2 ++
 block/blk-core.c                           |  2 ++
 block/genhd.c                              |  8 ++++++--
 block/partition-generic.c                  |  8 ++++++--
 include/linux/genhd.h                      |  1 +
 8 files changed, 34 insertions(+), 5 deletions(-)
Documentation/ABI/testing/procfs-diskstats +3 -0
index 70dcaf2481f4..e1473e93d901 100644
@@ -33,5 +33,8 @@ Description:
 
                19 - flush requests completed successfully
                20 - time spent flushing
+               21 - reads splitted
+               22 - writes splitted
+               23 - discards splitted
 
                For more details refer to Documentation/admin-guide/iostats.rst
Documentation/ABI/testing/sysfs-block +4 -1
index ed8c14f161ee..ffbac0e72508 100644
@@ -3,7 +3,7 @@ Date:           February 2008
 Contact:       Jerome Marchand <[email protected]>
 Description:
                The /sys/block/<disk>/stat files displays the I/O
-               statistics of disk <disk>. They contain 11 fields:
+               statistics of disk <disk>. They contain 20 fields:
                 1 - reads completed successfully
                 2 - reads merged
                 3 - sectors read
@@ -21,6 +21,9 @@ Description:
                15 - time spent discarding (ms)
                16 - flush requests completed
                17 - time spent flushing (ms)
+               18 - reads splitted
+               19 - writes splitted
+               20 - discrads splitted
                For more details refer Documentation/admin-guide/iostats.rst
 
 
Documentation/admin-guide/iostats.rst +10 -0
index 4f0462af3ca7..7f3f374b82b7 100644
@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed
 Field 17 -- # of milliseconds spent flushing
     This is the total number of milliseconds spent by all flush requests.
 
+Field 18 -- # of reads splitted, field 19 -- # of writes splitted
+    This is the total number of requests splitted before queue.
+
+    The maximum size of I/O is limited by queue/max_sectors_kb.
+    If size of I/O is greater than this limit, it will be splitted
+    as many times as needed to keep I/O size withing the limit.
+
+Field 20 -- # of discrads Splitted
+    See description of fileds 18 and 19.
+
 To avoid introducing performance bottlenecks, no locks are held while
 modifying these counters.  This implies that minor inaccuracies may be
 introduced when changes collide, so (for instance) adding up all the
block/bio.c +2 -0
index 8f0ed6228fc5..c8a051e128f8 100644
@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,
        if (bio_flagged(bio, BIO_TRACE_COMPLETION))
                bio_set_flag(split, BIO_TRACE_COMPLETION);
 
+       bio_set_flag(split, BIO_SPLITTED);
+
        return split;
 }
 EXPORT_SYMBOL(bio_split);
block/blk-core.c +0 -0
index f0d82227a2fc..776d28b9a5bf 100644

block/blk-core.c void blk_account_io_start(struct request *rq, bool new_io) · line 1383

Quoted 5 lines
block/blk-core.c +2 -0
@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io)
                }
                part_inc_in_flight(rq->q, part, rw);
                rq->part = part;
+               if (bio_flagged(rq->bio, BIO_SPLITTED))
+                       part_stat_inc(part, splits[rw]);

It is the only use of BIO_SPLITTED, and it should have been done as one rq flag, then extra change in bio struct can be avoided. If the bio is splitted can be easily
figured out in blk_mq_make_request(), then you may pass that info to blk_mq_bio_to_request().

Thanks,
Ming

#4 lore raw

Re: [PATCH 2/2] block: add iostat counters for requests splits

вт, 10 дек. 2019 г. в 06:09, Ming Lei <[email protected]>:

Quoted 10 lines · Aleksei Zakharov
Quoted 7 lines

I/O request can be splitted for two or more requests, if it's size is greater than queue/max_sectors_kb setting.

Knowledge of splits frequency helps to understand workload profile better and to make decision to tune queue/max_sectors_kb.

Quoted 7 lines

IO split should be one static behavior, so if one bio's sector/size is provided, it is easy to figure out the split pattern, not sure if it is worth of adding statistics
in iostat.

That said the IO split analysis can be done simply as post-process of IO trace, or even it can be done runtime via bcc/bpftrace script without much difficulty.

Yes, you're right, it can be done via bpftrace. But iostat is just well-known and overall used tool. It's better to have most of stats with default tool, than have a lot of tools for different parts of one subsystem layer, IMO.

Quoted 97 lines
Quoted 97 lines

This patch adds three counters to /sys/class/block/$dev/stat and /proc/diskstats: number of reads, writes and discards splitted.

There's also counter for flush requests, but it is ignored, because flush cannot be splitted.

---
 Documentation/ABI/testing/procfs-diskstats |  3 +++
 Documentation/ABI/testing/sysfs-block      |  5 ++++-
 Documentation/admin-guide/iostats.rst      | 10 ++++++++++
 block/bio.c                                |  2 ++
 block/blk-core.c                           |  2 ++
 block/genhd.c                              |  8 ++++++--
 block/partition-generic.c                  |  8 ++++++--
 include/linux/genhd.h                      |  1 +
 8 files changed, 34 insertions(+), 5 deletions(-)
Documentation/ABI/testing/procfs-diskstats +3 -0
index 70dcaf2481f4..e1473e93d901 100644
@@ -33,5 +33,8 @@ Description:
 
                19 - flush requests completed successfully
                20 - time spent flushing
+               21 - reads splitted
+               22 - writes splitted
+               23 - discards splitted
 
                For more details refer to Documentation/admin-guide/iostats.rst
Documentation/ABI/testing/sysfs-block +4 -1
index ed8c14f161ee..ffbac0e72508 100644
@@ -3,7 +3,7 @@ Date:           February 2008
 Contact:       Jerome Marchand <[email protected]>
 Description:
                The /sys/block/<disk>/stat files displays the I/O
-               statistics of disk <disk>. They contain 11 fields:
+               statistics of disk <disk>. They contain 20 fields:
                 1 - reads completed successfully
                 2 - reads merged
                 3 - sectors read
@@ -21,6 +21,9 @@ Description:
                15 - time spent discarding (ms)
                16 - flush requests completed
                17 - time spent flushing (ms)
+               18 - reads splitted
+               19 - writes splitted
+               20 - discrads splitted
                For more details refer Documentation/admin-guide/iostats.rst
 
 
Documentation/admin-guide/iostats.rst +10 -0
index 4f0462af3ca7..7f3f374b82b7 100644
@@ -130,6 +130,16 @@ Field 16 -- # of flush requests completed
 Field 17 -- # of milliseconds spent flushing
     This is the total number of milliseconds spent by all flush requests.
 
+Field 18 -- # of reads splitted, field 19 -- # of writes splitted
+    This is the total number of requests splitted before queue.
+
+    The maximum size of I/O is limited by queue/max_sectors_kb.
+    If size of I/O is greater than this limit, it will be splitted
+    as many times as needed to keep I/O size withing the limit.
+
+Field 20 -- # of discrads Splitted
+    See description of fileds 18 and 19.
+
 To avoid introducing performance bottlenecks, no locks are held while
 modifying these counters.  This implies that minor inaccuracies may be
 introduced when changes collide, so (for instance) adding up all the
block/bio.c +2 -0
index 8f0ed6228fc5..c8a051e128f8 100644
@@ -1855,6 +1855,8 @@ struct bio *bio_split(struct bio *bio, int sectors,
        if (bio_flagged(bio, BIO_TRACE_COMPLETION))
                bio_set_flag(split, BIO_TRACE_COMPLETION);
 
+       bio_set_flag(split, BIO_SPLITTED);
+
        return split;
 }
 EXPORT_SYMBOL(bio_split);
block/blk-core.c +0 -0
index f0d82227a2fc..776d28b9a5bf 100644
@@ -1379,6 +1379,8 @@ void blk_account_io_start(struct request *rq, bool new_io)
                }
                part_inc_in_flight(rq->q, part, rw);
                rq->part = part;
Quoted 8 lines

block/blk-core.c void blk_account_io_start(struct request *rq, bool new_io) · line 1383

Quoted 2 lines
block/blk-core.c +2 -0
+               if (bio_flagged(rq->bio, BIO_SPLITTED))
+                       part_stat_inc(part, splits[rw]);

It is the only use of BIO_SPLITTED, and it should have been done as one rq flag, then extra change in bio struct can be avoided. If the bio is splitted can be easily
figured out in blk_mq_make_request(), then you may pass that info to blk_mq_bio_to_request().

Thanks for pointing that out!
I'll rethink this patch.

Quoted 3 lines

Thanks,
Ming

Signature · 5 lines
Best Regards,
Aleksei Zakharov
System administrator

www.selectel.com