block/diskstats: more accurate io_ticks and optimization
Subsystems (2)
v4 cover block/diskstats: more accurate io_ticks and optimization
Simplified estimation for io_ticks introduced in patch
https://lore.kernel.org/linux-block/[email protected]/
could be very inaccurate for request longer than jiffy (i.e. any HDD)
There is at least one another report about this:
https://lore.kernel.org/linux-block/[email protected]/
See detail in comment for first patch.
v1: https://lore.kernel.org/lkml/155413438394.3201.15211440151043943989.stgit@buzz/
v2: https://lore.kernel.org/lkml/158314549775.1788.6529015932237292177.stgit@buzz/
v3: https://lore.kernel.org/lkml/158503038812.1955.7827988255138056389.stgit@buzz/
* update documentation
* rebase to current linux-next
* fix compilation for CONFIG_SMP=n
v4:
* rebase to for-5.7/block
* make part_stat_read_all static in block/genhd.c
---
Konstantin Khlebnikov (3):
block/diskstats: more accurate approximation of io_ticks for slow disks block/diskstats: accumulate all per-cpu counters in one pass block/diskstats: replace time_in_queue with sum of request times
Documentation/admin-guide/iostats.rst | 5 +- block/bio.c | 9 +-- block/blk-core.c | 5 +- block/genhd.c | 109 +++++++++++++++++++++++---------- include/linux/genhd.h | 6 -- 5 files changed, 88 insertions(+), 46 deletions(-)
--
Signature
PATCH v4 1/3 block/diskstats: more accurate approximation of io_ticks for slow disks
Currently io_ticks is approximated by adding one at each start and end of requests if jiffies counter has changed. This works perfectly for requests shorter than a jiffy or if one of requests starts/ends at each jiffy.
If disk executes just one request at a time and they are longer than two jiffies then only first and last jiffies will be accounted.
Fix is simple: at the end of request add up into io_ticks jiffies passed since last update rather than just one jiffy.
Example: common HDD executes random read 4k requests around 12ms.
fio --name=test --filename=/dev/sdb --rw=randread --direct=1 --runtime=30 & iostat -x 10 sdb
Note changes of iostat's "%util" 8,43% -> 99,99% before/after patch:
Before:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0,00 0,00 82,60 0,00 330,40 0,00 8,00 0,96 12,09 12,09 0,00 1,02 8,43
After:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0,00 0,00 82,50 0,00 330,00 0,00 8,00 1,00 12,10 12,10 0,00 12,12 99,99
Now io_ticks does not loose time between start and end of requests, but for queue-depth > 1 some I/O time between adjacent starts might be lost.
For load estimation "%util" is not as useful as average queue length, but it clearly shows how often disk queue is completely empty.
--- Documentation/admin-guide/iostats.rst | 5 ++++- block/bio.c | 8 ++++---- block/blk-core.c | 4 ++-- include/linux/genhd.h | 2 +- 4 files changed, 11 insertions(+), 8 deletions(-)
Documentation/admin-guide/iostats.rst +4 -1
@@ -100,7 +100,7 @@ Field 10 -- # of milliseconds spent doing I/Os (unsigned int) Since 5.0 this field counts jiffies when at least one request was started or completed. If request runs more than 2 jiffies then some - I/O time will not be accounted unless there are other requests. + I/O time might be not accounted in case of concurrent requests. Field 11 -- weighted # of milliseconds spent doing I/Os (unsigned int) This field is incremented at each I/O start, I/O completion, I/O
@@ -143,6 +143,9 @@ are summed (possibly overflowing the unsigned long variable they are summed to) and the result given to the user. There is no convenient user interface for accessing the per-CPU counters themselves. +Since 4.19 request times are measured with nanoseconds precision and +truncated to milliseconds before showing in this interface. + Disks vs Partitions -------------------
block/bio.c +4 -4
@@ -1768,14 +1768,14 @@ void bio_check_pages_dirty(struct bio *bio)schedule_work (&bio_dirty_work); } -void update_io_ticks (struct hd_struct *part,unsigned long now) +void update_io_ticks (struct hd_struct *part,unsigned long now,bool end) {unsigned long stamp; again: stamp =READ_ONCE (part->stamp);if (unlikely (stamp != now)) {if (likely (cmpxchg (&part->stamp, stamp, now) == stamp)) { -__part_stat_add (part, io_ticks,1 ); +__part_stat_add (part, io_ticks, end ? now - stamp :1 ); } }if (part->partno) {
@@ -1791,7 +1791,7 @@ void generic_start_io_acct(struct request_queue *q, int op,part_stat_lock (); -update_io_ticks (part, jiffies); +update_io_ticks (part, jiffies, false);part_stat_inc (part, ios[sgrp]);part_stat_add (part, sectors[sgrp], sectors);part_inc_in_flight (q, part,op_is_write (op));
@@ -1809,7 +1809,7 @@ void generic_end_io_acct(struct request_queue *q, int req_op,part_stat_lock (); -update_io_ticks (part, now); +update_io_ticks (part, now, true);part_stat_add (part, nsecs[sgrp],jiffies_to_nsecs (duration));part_stat_add (part, time_in_queue, duration);part_dec_in_flight (q, part,op_is_write (req_op));
block/blk-core.c +2 -2
@@ -1337,7 +1337,7 @@ void blk_account_io_done(struct request *req, u64 now)part_stat_lock (); part = req->part; -update_io_ticks (part, jiffies); +update_io_ticks (part, jiffies, true);part_stat_inc (part, ios[sgrp]);part_stat_add (part, nsecs[sgrp], now - req->start_time_ns);part_stat_add (part, time_in_queue,nsecs_to_jiffies64 (now - req->start_time_ns));
@@ -1379,7 +1379,7 @@ void blk_account_io_start(struct request *rq, bool new_io) rq->part = part; } -update_io_ticks (part, jiffies); +update_io_ticks (part, jiffies, false);part_stat_unlock (); }
include/linux/genhd.h +1 -1
@@ -422,7 +422,7 @@ void part_dec_in_flight(struct request_queue *q, struct hd_struct *part,void part_inc_in_flight (struct request_queue *q,struct hd_struct *part,int rw); -void update_io_ticks (struct hd_struct *part,unsigned long now); +void update_io_ticks (struct hd_struct *part,unsigned long now,bool end);/* block/genhd.c */ extern void device_add_disk(struct device *parent, struct gendisk *disk,
PATCH v4 2/3 block/diskstats: accumulate all per-cpu counters in one pass
In reply to Konstantin KhlebnikovReading /proc/diskstats iterates over all cpus for summing each field. It's faster to sum all fields in one pass.
Hammering /proc/diskstats with fio shows 2x performance improvement:
fio --name=test --numjobs=$JOBS --filename=/proc/diskstats \
--size=1k --bs=1k --fallocate=none --create_on_open=1 \
--time_based=1 --runtime=10 --invalidate=0 --group_report
JOBS=1 JOBS=10Before: 7k iops 64k iops
After: 18k iops 120k iops
Also this way code is more compact:
add/remove: 1/0 grow/shrink: 0/2 up/down: 194/-1540 (-1346)
Function old new delta part_stat_read_all - 194 +194 diskstats_show 1344 631 -713 part_stat_show 1219 392 -827
Total: Before=14966947, After=14965601, chg -0.01%
--- block/genhd.c | 102 ++++++++++++++++++++++++++++++++++--------------- include/linux/genhd.h | 3 - 2 files changed, 70 insertions(+), 35 deletions(-)
block/genhd.c +70 -32
@@ -92,6 +92,34 @@ const char *bdevname(struct block_device *bdev, char *buf) }EXPORT_SYMBOL (bdevname); +#ifdef CONFIG_SMP +static void part_stat_read_all (struct hd_struct *part,struct disk_stats *stat) +{ +int cpu; + +memset (stat,0 ,sizeof (struct disk_stats )); +for_each_possible_cpu (cpu) { +struct disk_stats *ptr =per_cpu_ptr (part->dkstats, cpu); +int group; + +for (group =0 ; group <NR_STAT_GROUPS ; group++) { + stat->nsecs[group] += ptr->nsecs[group]; + stat->sectors[group] += ptr->sectors[group]; + stat->ios[group] += ptr->ios[group]; + stat->merges[group] += ptr->merges[group]; + } + + stat->io_ticks += ptr->io_ticks; + stat->time_in_queue += ptr->time_in_queue; + } +} +#else /* CONFIG_SMP */ +static void part_stat_read_all (struct hd_struct *part,struct disk_stats *stat) +{ +memcpy (stat, &part->dkstats,sizeof (struct disk_stats )); +} +#endif /* CONFIG_SMP */ +void part_inc_in_flight (struct request_queue *q,struct hd_struct *part,int rw) {if (queue_is_mq (q))
@@ -1214,9 +1242,12 @@ ssize_t part_stat_show(struct device *dev, {struct hd_struct *p =dev_to_part (dev);struct request_queue *q =part_to_disk (p)->queue; +struct disk_stats stat;unsigned int inflight; +part_stat_read_all (p, &stat); inflight =part_in_flight (q, p); +return sprintf (buf,"%8lu %8lu %8llu %8u " "%8lu %8lu %8llu %8u "
@@ -1224,23 +1255,23 @@ ssize_t part_stat_show(struct device *dev,"%8lu %8lu %8llu %8u " "%8lu %8u" "\n" , -part_stat_read (p, ios[STAT_READ ]), -part_stat_read (p, merges[STAT_READ ]), - (unsigned long long )part_stat_read (p, sectors[STAT_READ ]), - (unsigned int )part_stat_read_msecs (p,STAT_READ ), -part_stat_read (p, ios[STAT_WRITE ]), -part_stat_read (p, merges[STAT_WRITE ]), - (unsigned long long )part_stat_read (p, sectors[STAT_WRITE ]), - (unsigned int )part_stat_read_msecs (p,STAT_WRITE ), + stat.ios[STAT_READ ], + stat.merges[STAT_READ ], + (unsigned long long )stat.sectors[STAT_READ ], + (unsigned int )div_u64 (stat.nsecs[STAT_READ ],NSEC_PER_MSEC ), + stat.ios[STAT_WRITE ], + stat.merges[STAT_WRITE ], + (unsigned long long )stat.sectors[STAT_WRITE ], + (unsigned int )div_u64 (stat.nsecs[STAT_WRITE ],NSEC_PER_MSEC ), inflight, -jiffies_to_msecs (part_stat_read (p, io_ticks)), -jiffies_to_msecs (part_stat_read (p, time_in_queue)), -part_stat_read (p, ios[STAT_DISCARD ]), -part_stat_read (p, merges[STAT_DISCARD ]), - (unsigned long long )part_stat_read (p, sectors[STAT_DISCARD ]), - (unsigned int )part_stat_read_msecs (p,STAT_DISCARD ), -part_stat_read (p, ios[STAT_FLUSH ]), - (unsigned int)part_stat_read_msecs(p, STAT_FLUSH)); +jiffies_to_msecs (stat.io_ticks), +jiffies_to_msecs (stat.time_in_queue), + stat.ios[STAT_DISCARD ], + stat.merges[STAT_DISCARD ], + (unsigned long long )stat.sectors[STAT_DISCARD ], + (unsigned int )div_u64 (stat.nsecs[STAT_DISCARD ],NSEC_PER_MSEC ), + stat.ios[STAT_FLUSH ], + (unsigned int)div_u64(stat.nsecs[STAT_FLUSH], NSEC_PER_MSEC)); } ssize_t part_inflight_show(struct device *dev, struct device_attribute *attr,
@@ -1492,6 +1523,7 @@ static int diskstats_show(struct seq_file *seqf, void *v)struct hd_struct *hd;char buf[BDEVNAME_SIZE ];unsigned int inflight; +struct disk_stats stat;/* if (&disk_to_dev(gp)->kobj.entry == block_class.devices.next)
@@ -1503,7 +1535,9 @@ static int diskstats_show(struct seq_file *seqf, void *v)disk_part_iter_init (&piter, gp,DISK_PITER_INCL_EMPTY_PART0 );while ((hd =disk_part_iter_next (&piter))) { +part_stat_read_all (hd, &stat); inflight =part_in_flight (gp->queue, hd); +seq_printf (seqf,"%4d %7d %s " "%lu %lu %lu %u " "%lu %lu %lu %u "
@@ -1513,23 +1547,27 @@ static int diskstats_show(struct seq_file *seqf, void *v)"\n" ,MAJOR (part_devt (hd)),MINOR (part_devt (hd)),disk_name (gp, hd->partno, buf), -part_stat_read (hd, ios[STAT_READ ]), -part_stat_read (hd, merges[STAT_READ ]), -part_stat_read (hd, sectors[STAT_READ ]), - (unsigned int )part_stat_read_msecs (hd,STAT_READ ), -part_stat_read (hd, ios[STAT_WRITE ]), -part_stat_read (hd, merges[STAT_WRITE ]), -part_stat_read (hd, sectors[STAT_WRITE ]), - (unsigned int )part_stat_read_msecs (hd,STAT_WRITE ), + stat.ios[STAT_READ ], + stat.merges[STAT_READ ], + stat.sectors[STAT_READ ], + (unsigned int )div_u64 (stat.nsecs[STAT_READ ], +NSEC_PER_MSEC ), + stat.ios[STAT_WRITE ], + stat.merges[STAT_WRITE ], + stat.sectors[STAT_WRITE ], + (unsigned int )div_u64 (stat.nsecs[STAT_WRITE ], +NSEC_PER_MSEC ), inflight, -jiffies_to_msecs (part_stat_read (hd, io_ticks)), -jiffies_to_msecs (part_stat_read (hd, time_in_queue)), -part_stat_read (hd, ios[STAT_DISCARD ]), -part_stat_read (hd, merges[STAT_DISCARD ]), -part_stat_read (hd, sectors[STAT_DISCARD ]), - (unsigned int )part_stat_read_msecs (hd,STAT_DISCARD ), -part_stat_read (hd, ios[STAT_FLUSH ]), - (unsigned int )part_stat_read_msecs (hd,STAT_FLUSH ) +jiffies_to_msecs (stat.io_ticks), +jiffies_to_msecs (stat.time_in_queue), + stat.ios[STAT_DISCARD ], + stat.merges[STAT_DISCARD ], + stat.sectors[STAT_DISCARD ], + (unsigned int )div_u64 (stat.nsecs[STAT_DISCARD ], +NSEC_PER_MSEC ), + stat.ios[STAT_FLUSH ], + (unsigned int )div_u64 (stat.nsecs[STAT_FLUSH ], +NSEC_PER_MSEC ) ); }disk_part_iter_exit (&piter);
include/linux/genhd.h +0 -3
@@ -380,9 +380,6 @@ static inline void free_part_stats(struct hd_struct *part)#endif /* CONFIG_SMP */ -#define part_stat_read_msecs (part, which) \ - div_u64(part_stat_read(part, nsecs[which]), NSEC_PER_MSEC) -#define part_stat_read_accum (part, field) \ (part_stat_read(part, field[STAT_READ]) + \ part_stat_read(part, field[STAT_WRITE]) + \
PATCH v4 3/3 block/diskstats: replace time_in_queue with sum of request times
In reply to Konstantin KhlebnikovColumn "time_in_queue" in diskstats is supposed to show total waiting time of all requests. I.e. value should be equal to the sum of times from other columns. But this is not true, because column "time_in_queue" is counted separately in jiffies rather than in nanoseconds as other times.
This patch removes redundant counter for "time_in_queue" and shows total time of read, write, discard and flush requests.
--- block/bio.c | 1 - block/blk-core.c | 1 - block/genhd.c | 13 ++++++++++--- include/linux/genhd.h | 1 - 4 files changed, 10 insertions(+), 6 deletions(-)
block/bio.c +0 -1
@@ -1811,7 +1811,6 @@ void generic_end_io_acct(struct request_queue *q, int req_op,update_io_ticks (part, now, true);part_stat_add (part, nsecs[sgrp],jiffies_to_nsecs (duration)); -part_stat_add (part, time_in_queue, duration);part_dec_in_flight (q, part,op_is_write (req_op));part_stat_unlock ();
block/blk-core.c +0 -1
@@ -1340,7 +1340,6 @@ void blk_account_io_done(struct request *req, u64 now)update_io_ticks (part, jiffies, true);part_stat_inc (part, ios[sgrp]);part_stat_add (part, nsecs[sgrp], now - req->start_time_ns); -part_stat_add (part, time_in_queue,nsecs_to_jiffies64 (now - req->start_time_ns));part_dec_in_flight (req->q, part,rq_data_dir (req));hd_struct_put (part);
block/genhd.c +10 -3
@@ -110,7 +110,6 @@ static void part_stat_read_all(struct hd_struct *part, struct disk_stats *stat) } stat->io_ticks += ptr->io_ticks; - stat->time_in_queue += ptr->time_in_queue; } }#else /* CONFIG_SMP */
@@ -1265,7 +1264,11 @@ ssize_t part_stat_show(struct device *dev, (unsigned int )div_u64 (stat.nsecs[STAT_WRITE ],NSEC_PER_MSEC ), inflight,jiffies_to_msecs (stat.io_ticks), -jiffies_to_msecs (stat.time_in_queue), + (unsigned int )div_u64 (stat.nsecs[STAT_READ ] + + stat.nsecs[STAT_WRITE ] + + stat.nsecs[STAT_DISCARD ] + + stat.nsecs[STAT_FLUSH ], +NSEC_PER_MSEC ), stat.ios[STAT_DISCARD ], stat.merges[STAT_DISCARD ], (unsigned long long)stat.sectors[STAT_DISCARD],
@@ -1559,7 +1562,11 @@ static int diskstats_show(struct seq_file *seqf, void *v) NSEC_PER_MSEC), inflight,jiffies_to_msecs (stat.io_ticks), -jiffies_to_msecs (stat.time_in_queue), + (unsigned int )div_u64 (stat.nsecs[STAT_READ ] + + stat.nsecs[STAT_WRITE ] + + stat.nsecs[STAT_DISCARD ] + + stat.nsecs[STAT_FLUSH ], +NSEC_PER_MSEC ), stat.ios[STAT_DISCARD ], stat.merges[STAT_DISCARD ], stat.sectors[STAT_DISCARD],
include/linux/genhd.h +0 -1
@@ -46,7 +46,6 @@ struct disk_stats {unsigned long ios[NR_STAT_GROUPS ];unsigned long merges[NR_STAT_GROUPS ];unsigned long io_ticks; -unsigned long time_in_queue;local_t in_flight[2 ]; };
Jens Axboe
maintainerQuoted 8 lines · Konstantin Khlebnikov
Simplified estimation for
io_ticksintroduced in patch
https://lore.kernel.org/linux-block/[email protected]/
could be very inaccurate for request longer than jiffy (i.e. any HDD)There is at least one another report about this:
https://lore.kernel.org/linux-block/[email protected]/
See detail in comment for first patch.
Quoted 9 lines
v1: https://lore.kernel.org/lkml/155413438394.3201.15211440151043943989.stgit@buzz/
v2: https://lore.kernel.org/lkml/158314549775.1788.6529015932237292177.stgit@buzz/
v3: https://lore.kernel.org/lkml/158503038812.1955.7827988255138056389.stgit@buzz/
* update documentation
* rebase to current linux-next
* fix compilation forCONFIG_SMP=n
v4:
* rebase to for-5.7/block
* makepart_stat_read_allstatic inblock/genhd.c
Applied for 5.7, thanks.
Signature · 1 line
Jens Axboe