block,nvme: latency-based I/O scheduler
Subsystems (2)
cover block,nvme: latency-based I/O scheduler
Hi all,
there had been several attempts to implement a latency-based I/O scheduler for native nvme multipath, all of which had its issues.
So time to start afresh, this time using the QoS framework already present in the block layer.
It consists of two parts:
- a new 'blk-nodelat' QoS module, which is just a simple per-node latency tracker
- a 'latency' nvme I/O policy
Using the 'tiobench' fio script I'm getting:
WRITE: bw=531MiB/s (556MB/s), 33.2MiB/s-52.4MiB/s (34.8MB/s-54.9MB/s), io=4096MiB (4295MB), run=4888-7718msec
WRITE: bw=539MiB/s (566MB/s), 33.7MiB/s-50.9MiB/s
(35.3MB/s-53.3MB/s), io=4096MiB (4295MB), run=5033-7594msec
READ: bw=898MiB/s (942MB/s), 56.1MiB/s-75.4MiB/s
(58.9MB/s-79.0MB/s), io=4096MiB (4295MB), run=3397-4560msec
READ: bw=1023MiB/s (1072MB/s), 63.9MiB/s-75.1MiB/s(67.0MB/s-78.8MB/s), io=4096MiB (4295MB), run=3408-4005msec
for 'round-robin' and
WRITE: bw=574MiB/s (601MB/s), 35.8MiB/s-45.5MiB/s (37.6MB/s-47.7MB/s), io=4096MiB (4295MB), run=5629-7142msec
WRITE: bw=639MiB/s (670MB/s), 39.9MiB/s-47.5MiB/s
(41.9MB/s-49.8MB/s), io=4096MiB (4295MB), run=5388-6408msec
READ: bw=1024MiB/s (1074MB/s), 64.0MiB/s-73.7MiB/s
(67.1MB/s-77.2MB/s), io=4096MiB (4295MB), run=3475-4000msec
READ: bw=1013MiB/s (1063MB/s), 63.3MiB/s-72.6MiB/s(66.4MB/s-76.2MB/s), io=4096MiB (4295MB), run=3524-4042msec
for 'latency' with 'decay' set to 10.
That's on a 32G FC testbed running against a brd target, fio running with 16 thread.
As usual, comments and reviews are welcome.
Hannes Reinecke (2):
block: track per-node I/O latency
nvme: add 'latency' iopolicy
block/Kconfig | 7 + block/Makefile | 1 + block/blk-mq-debugfs.c | 2 + block/blk-nodelat.c | 368 ++++++++++++++++++++++++++++++++++ block/blk-rq-qos.h | 6 + drivers/nvme/host/multipath.c | 46 ++++- drivers/nvme/host/nvme.h | 2 + include/linux/blk-mq.h | 11 + 8 files changed, 439 insertions(+), 4 deletions(-) create mode 100644 block/blk-nodelat.c
Signature · 1 line
2.35.3
PATCH 1/2 block: track per-node I/O latency
Add a new option 'BLK_NODE_LATENCY' to track per-node I/O latency. This can be used by I/O scheduler to determine the 'best' queue to send I/O to.
--- block/Kconfig | 7 + block/Makefile | 1 + block/blk-mq-debugfs.c | 2 + block/blk-nodelat.c | 368 +++++++++++++++++++++++++++++++++++++++++ block/blk-rq-qos.h | 6 + include/linux/blk-mq.h | 11 ++ 6 files changed, 395 insertions(+) create mode 100644 block/blk-nodelat.c
block/Kconfig +7 -0
@@ -186,6 +186,13 @@ config BLK_CGROUP_IOPRIO scheduler and block devices process requests. Only some I/O schedulers and some block devices support I/O priorities. +config BLK_NODE_LATENCY + bool "Track per-node I/O latency" + help + Enable the .nlat interface for tracking per-node I/O latency. + This can be used by I/O schedulers to determine the queue with the + least latency. + config BLK_DEBUG_FS bool "Block layer debugging information in debugfs" default y
block/Makefile +1 -0
@@ -21,6 +21,7 @@ obj-$(CONFIG_BLK_DEV_THROTTLING) += blk-throttle.o obj-$(CONFIG_BLK_CGROUP_IOPRIO) += blk-ioprio.o obj-$(CONFIG_BLK_CGROUP_IOLATENCY) += blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOCOST) += blk-iocost.o +obj-$(CONFIG_BLK_NODE_LATENCY) += blk-nodelat.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o
block/blk-mq-debugfs.c +2 -0
@@ -762,6 +762,8 @@ static const char *rq_qos_id_to_name(enum rq_qos_id id)return "latency" ;case RQ_QOS_COST :return "cost" ; +case RQ_QOS_NLAT : +return "node-latency" ; }return "unknown" ; }
block/blk-nodelat.c new file +368 -0
368 lines not on this page · raw message
block/blk-rq-qos.h +6 -0
@@ -17,6 +17,7 @@ enum rq_qos_id {RQ_QOS_WBT ,RQ_QOS_LATENCY ,RQ_QOS_COST , + RQ_QOS_NLAT, }; struct rq_wait {
@@ -79,6 +80,11 @@ static inline struct rq_qos *iolat_rq_qos(struct request_queue *q)return rq_qos_id (q,RQ_QOS_LATENCY ); } +static inline struct rq_qos *nlat_rq_qos (struct request_queue *q) +{ +return rq_qos_id (q,RQ_QOS_NLAT ); +} +static inline void rq_wait_init (struct rq_wait *rq_wait) {atomic_set (&rq_wait->inflight,0 );
include/linux/blk-mq.h +11 -0
@@ -1229,4 +1229,15 @@ static inline bool blk_req_can_dispatch_to_zone(struct request *rq) }#endif /* CONFIG_BLK_DEV_ZONED */ +#ifdef CONFIG_BLK_NODE_LATENCY +int blk_nodelat_enable (struct request_queue *q); +void blk_nodelat_disable (struct request_queue *q); +u64 blk_nodelat_latency (struct request_queue *q,int node); +int blk_nodelat_init (struct gendisk *disk); +#else +static inline in blk_nodelat_enable (struct request_queue *q) {return 0 ; } +static inline void blk_nodelat_disable (struct request_queue *q) {} +u64 blk_nodelat_latency (struct request_queue *q,int node) {return 0 ; } +static inline in blk_nodelat_init (struct gendisk *disk) {return -ENOTSUPP ; } +#endif #endif /* BLK_MQ_H */
Signature · 1 line
2.35.3
kernel test robot
build botRe: [PATCH 1/2] block: track per-node I/O latency
kernel test robot noticed the following build errors:
Hi Hannes,
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe-block/for-next]
[also build test ERROR on linus/master v6.9-rc1 next-20240327] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Hannes-Reinecke/block-track-per-node-I-O-latency/20240326-234521 base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next patch link: https://lore.kernel.org/r/20240326153529.75989-2-hare%40kernel.org patch subject: [PATCH 1/2] block: track per-node I/O latency config: openrisc-allnoconfig (https://download.01.org/0day-ci/archive/20240328/[email protected]/config) compiler: or1k-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240328/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
In file included from include/linux/blk-integrity.h:5,
from block/bdev.c:15:
>> include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^~
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^~~~~~~~~~~~~~~~~~~
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^~vim +/in +1240 include/linux/blk-mq.h
1233
1234 #ifdef CONFIG_BLK_NODE_LATENCY
1235 int blk_nodelat_enable(struct request_queue *q);
1236 void blk_nodelat_disable(struct request_queue *q);
1237 u64 blk_nodelat_latency(struct request_queue *q, int node);
1238 int blk_nodelat_init(struct gendisk *disk);
1239 #else
> 1240 static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
1241 static inline void blk_nodelat_disable(struct request_queue *q) {}
> 1242 u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 1/2] block: track per-node I/O latency
In reply to Hannes Reineckekernel test robot noticed the following build warnings:
Hi Hannes,
kernel test robot noticed the following build warnings:
[auto build test WARNING on axboe-block/for-next]
[also build test WARNING on linus/master v6.9-rc1 next-20240327] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Hannes-Reinecke/block-track-per-node-I-O-latency/20240326-234521 base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next patch link: https://lore.kernel.org/r/20240326153529.75989-2-hare%40kernel.org patch subject: [PATCH 1/2] block: track per-node I/O latency config: arm-randconfig-001-20240327 (https://download.01.org/0day-ci/archive/20240328/[email protected]/config) compiler: clang version 19.0.0git (https://github.com/llvm/llvm-project 23de3862dce582ce91c1aa914467d982cb1a73b4) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240328/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from drivers/scsi/aic7xxx/aic79xx_pci.c:44:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:46:
In file included from include/linux/blkdev.h:9:
In file included from include/linux/blk_types.h:10:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/arm/include/asm/cacheflush.h:10:
In file included from include/linux/mm.h:2208:
include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
522 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from drivers/scsi/aic7xxx/aic79xx_pci.c:44:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:57:
In file included from include/scsi/scsi_cmnd.h:7:
In file included from include/linux/t10-pi.h:6:
include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for function 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
include/linux/blk-mq.h:1242:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
| static
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^ 2 warnings and 2 errors generated.
--
In file included from drivers/scsi/aic7xxx/aic79xx_core.c:43:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:46:
In file included from include/linux/blkdev.h:9:
In file included from include/linux/blk_types.h:10:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/arm/include/asm/cacheflush.h:10:
In file included from include/linux/mm.h:2208:
include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
522 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from drivers/scsi/aic7xxx/aic79xx_core.c:43:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:57:
In file included from include/scsi/scsi_cmnd.h:7:
In file included from include/linux/t10-pi.h:6:
include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for function 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
include/linux/blk-mq.h:1242:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
| static
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^
drivers/scsi/aic7xxx/aic79xx_core.c:5694:13: warning: variable 'data_addr' set but not used [-Wunused-but-set-variable]
5694 | uint64_t data_addr;
| ^ 3 warnings and 2 errors generated.
--
In file included from drivers/scsi/aic7xxx/aic7xxx_core.c:43:
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.h:63:
In file included from include/linux/blkdev.h:9:
In file included from include/linux/blk_types.h:10:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/arm/include/asm/cacheflush.h:10:
In file included from include/linux/mm.h:2208:
include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
522 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from drivers/scsi/aic7xxx/aic7xxx_core.c:43:
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.h:74:
In file included from include/scsi/scsi_cmnd.h:7:
In file included from include/linux/t10-pi.h:6:
include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for function 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
include/linux/blk-mq.h:1242:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
| static
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^
drivers/scsi/aic7xxx/aic7xxx_core.c:4171:13: warning: variable 'data_addr' set but not used [-Wunused-but-set-variable]
4171 | uint32_t data_addr;
| ^ 3 warnings and 2 errors generated.
--
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.c:123:
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.h:63:
In file included from include/linux/blkdev.h:9:
In file included from include/linux/blk_types.h:10:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/arm/include/asm/cacheflush.h:10:
In file included from include/linux/mm.h:2208:
include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
522 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.c:123:
In file included from drivers/scsi/aic7xxx/aic7xxx_osm.h:74:
In file included from include/scsi/scsi_cmnd.h:7:
In file included from include/linux/t10-pi.h:6:
include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for function 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
include/linux/blk-mq.h:1242:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
| static
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^
drivers/scsi/aic7xxx/aic7xxx_osm.c:1435:24: warning: bitwise operation between different enumeration types ('ahc_feature' and 'ahc_flag') [-Wenum-enum-conversion]
1435 | && (ahc->features & AHC_SCB_BTT) == 0) {
| ~~~~~~~~~~~~~ ^ ~~~~~~~~~~~ 3 warnings and 2 errors generated.
--
In file included from drivers/scsi/aic7xxx/aic79xx_osm_pci.c:42:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:46:
In file included from include/linux/blkdev.h:9:
In file included from include/linux/blk_types.h:10:
In file included from include/linux/bvec.h:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/arm/include/asm/cacheflush.h:10:
In file included from include/linux/mm.h:2208:
include/linux/vmstat.h:522:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
522 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from drivers/scsi/aic7xxx/aic79xx_osm_pci.c:42:
In file included from drivers/scsi/aic7xxx/aic79xx_osm.h:57:
In file included from include/scsi/scsi_cmnd.h:7:
In file included from include/linux/t10-pi.h:6:
include/linux/blk-mq.h:1240:15: error: unknown type name 'in'
1240 | static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
| ^
>> include/linux/blk-mq.h:1242:5: warning: no previous prototype for function 'blk_nodelat_latency' [-Wmissing-prototypes]
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
include/linux/blk-mq.h:1242:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
1242 | u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }
| ^
| static
include/linux/blk-mq.h:1243:15: error: unknown type name 'in'
1243 | static inline in blk_nodelat_init(struct gendisk *disk) { return -ENOTSUPP; }
| ^
drivers/scsi/aic7xxx/aic79xx_osm_pci.c:177:25: warning: shift count >= width of type [-Wshift-count-overflow]
177 | dma_set_mask(dev, DMA_BIT_MASK(64)) == 0)
| ^~~~~~~~~~~~~~~~ include/linux/dma-mapping.h:77:54: note: expanded from macro 'DMA_BIT_MASK'
77 | #define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1))
| ^ ~~~
3 warnings and 2 errors generated.
vim +/blk_nodelat_latency +1242 include/linux/blk-mq.h
1233
1234 #ifdef CONFIG_BLK_NODE_LATENCY
1235 int blk_nodelat_enable(struct request_queue *q);
1236 void blk_nodelat_disable(struct request_queue *q);
1237 u64 blk_nodelat_latency(struct request_queue *q, int node);
1238 int blk_nodelat_init(struct gendisk *disk);
1239 #else
> 1240 static inline in blk_nodelat_enable(struct request_queue *q) { return 0; }
1241 static inline void blk_nodelat_disable(struct request_queue *q) {}
> 1242 u64 blk_nodelat_latency(struct request_queue *q, int node) { return 0; }Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 2/2 nvme: add 'latency' iopolicy
In reply to Hannes ReineckeFrom: Hannes Reinecke <[email protected]>
Add a latency-based I/O policy for multipathing. It uses the blk-nodelat latency tracker to provide latencies for each node, and schedules I/O on the path with the least latency for the submitting node.
--- drivers/nvme/host/multipath.c | 46 ++++++++++++++++++++++++++++++++--- drivers/nvme/host/nvme.h | 2 ++ 2 files changed, 44 insertions(+), 4 deletions(-)
drivers/nvme/host/multipath.c +42 -4
@@ -17,6 +17,7 @@ MODULE_PARM_DESC(multipath,static const char *nvme_iopolicy_names[] = { [NVME_IOPOLICY_NUMA ] ="numa" , [NVME_IOPOLICY_RR ] ="round-robin" , + [NVME_IOPOLICY_LAT ] ="latency" , };static int iopolicy =NVME_IOPOLICY_NUMA ;
@@ -29,6 +30,10 @@ static int nvme_set_iopolicy(const char *val, const struct kernel_param *kp) iopolicy =NVME_IOPOLICY_NUMA ; else if (!strncmp(val, "round-robin", 11)) iopolicy =NVME_IOPOLICY_RR ; +#ifdef CONFIG_BLK_NODE_LATENCY + else if (!strncmp(val, "latency", 7)) + iopolicy =NVME_IOPOLICY_LAT ; +#endif else return -EINVAL;
@@ -40,6 +45,26 @@ static int nvme_get_iopolicy(char *buf, const struct kernel_param *kp)return sprintf (buf,"%s\n" , nvme_iopolicy_names[iopolicy]); } +static void nvme_activate_iopolicy (struct nvme_subsystem *subsys,int iopolicy) +{ +struct nvme_ns_head *h; +struct nvme_ns *ns; +bool enable = iopolicy ==NVME_IOPOLICY_LAT ; + +mutex_lock (&subsys->lock); +list_for_each_entry (h, &subsys->nsheads, entry) { +list_for_each_entry_rcu (ns, &h->list, siblings) { +if (!test_bit (NVME_NS_NLAT , &ns->flags)) +continue ; +if (enable) +blk_nodelat_enable (ns->queue); +else +blk_nodelat_disable (ns->queue); + } + } +mutex_unlock (&subsys->lock); +} +module_param_call (iopolicy, nvme_set_iopolicy, nvme_get_iopolicy, &iopolicy,0644 ); MODULE_PARM_DESC(iopolicy,
@@ -242,13 +267,16 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node) {int found_distance =INT_MAX , fallback_distance =INT_MAX , distance;struct nvme_ns *found =NULL , *fallback =NULL , *ns; +int iopolicy =READ_ONCE (head->subsys->iopolicy);list_for_each_entry_rcu (ns, &head->list, siblings) {if (nvme_path_is_disabled (ns))continue ; -if (READ_ONCE (head->subsys->iopolicy) ==NVME_IOPOLICY_NUMA ) +if (iopolicy ==NVME_IOPOLICY_NUMA ) distance =node_distance (node, ns->ctrl->numa_node); +else if (iopolicy ==NVME_IOPOLICY_LAT ) + distance =blk_nodelat_latency (ns->queue, node);else distance =LOCAL_DISTANCE ;
@@ -339,15 +367,17 @@ static inline bool nvme_path_is_optimized(struct nvme_ns *ns)inline struct nvme_ns *nvme_find_path (struct nvme_ns_head *head) {int node =numa_node_id (); +int iopolicy =READ_ONCE (head->subsys->iopolicy);struct nvme_ns *ns; ns =srcu_dereference (head->current_path[node], &head->srcu);if (unlikely (!ns))return __nvme_find_path (head, node); -if (READ_ONCE (head->subsys->iopolicy) ==NVME_IOPOLICY_RR ) +if (iopolicy ==NVME_IOPOLICY_RR )return nvme_round_robin_path (head, node, ns); -if (unlikely (!nvme_path_is_optimized (ns))) +if (iopolicy ==NVME_IOPOLICY_LAT || +unlikely (!nvme_path_is_optimized (ns)))return __nvme_find_path (head, node);return ns; }
@@ -808,10 +838,10 @@ static ssize_t nvme_subsys_iopolicy_store(struct device *dev,for (i =0 ; i <ARRAY_SIZE (nvme_iopolicy_names); i++) {if (sysfs_streq (buf, nvme_iopolicy_names[i])) {WRITE_ONCE (subsys->iopolicy, i); +nvme_activate_iopolicy (subsys, i);return count; } } -return -EINVAL ; } SUBSYS_ATTR_RW(iopolicy, S_IRUGO | S_IWUSR,
@@ -847,6 +877,14 @@ static int nvme_lookup_ana_group_desc(struct nvme_ctrl *ctrl,void nvme_mpath_add_disk (struct nvme_ns *ns,__le32 anagrpid) { +if (!blk_nodelat_init (ns->disk)) { +int iopolicy =READ_ONCE (ns->head->subsys->iopolicy); + +set_bit (NVME_NS_NLAT , &ns->flags); +if (iopolicy ==NVME_IOPOLICY_LAT ) +blk_nodelat_enable (ns->queue); + } +if (nvme_ctrl_use_ana (ns->ctrl)) {struct nvme_ana_group_desc desc = { .grpid = anagrpid,
drivers/nvme/host/nvme.h +2 -0
@@ -402,6 +402,7 @@ static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)enum nvme_iopolicy {NVME_IOPOLICY_NUMA ,NVME_IOPOLICY_RR , +NVME_IOPOLICY_LAT , };struct nvme_subsystem {
@@ -519,6 +520,7 @@ struct nvme_ns {#define NVME_NS_ANA_PENDING 2#define NVME_NS_FORCE_RO 3#define NVME_NS_READY 4 +#define NVME_NS_NLAT 5struct cdev cdev;struct device cdev_device;
Signature · 1 line
2.35.3
Sagi Grimberg
maintainerQuoted 33 lines · Hannes Reinecke
Hi all,
there had been several attempts to implement a latency-based I/O scheduler for native nvme multipath, all of which had its issues.
So time to start afresh, this time using the QoS framework already present in the block layer.
It consists of two parts:
- a new 'blk-nodelat' QoS module, which is just a simple per-node latency tracker
- a 'latency' nvme I/O policyUsing the 'tiobench' fio script I'm getting:
WRITE: bw=531MiB/s (556MB/s), 33.2MiB/s-52.4MiB/s (34.8MB/s-54.9MB/s), io=4096MiB (4295MB), run=4888-7718msec WRITE: bw=539MiB/s (566MB/s), 33.7MiB/s-50.9MiB/s (35.3MB/s-53.3MB/s), io=4096MiB (4295MB), run=5033-7594msec READ: bw=898MiB/s (942MB/s), 56.1MiB/s-75.4MiB/s (58.9MB/s-79.0MB/s), io=4096MiB (4295MB), run=3397-4560msec READ: bw=1023MiB/s (1072MB/s), 63.9MiB/s-75.1MiB/s (67.0MB/s-78.8MB/s), io=4096MiB (4295MB), run=3408-4005msecfor 'round-robin' and
WRITE: bw=574MiB/s (601MB/s), 35.8MiB/s-45.5MiB/s (37.6MB/s-47.7MB/s), io=4096MiB (4295MB), run=5629-7142msec WRITE: bw=639MiB/s (670MB/s), 39.9MiB/s-47.5MiB/s (41.9MB/s-49.8MB/s), io=4096MiB (4295MB), run=5388-6408msec READ: bw=1024MiB/s (1074MB/s), 64.0MiB/s-73.7MiB/s (67.1MB/s-77.2MB/s), io=4096MiB (4295MB), run=3475-4000msec READ: bw=1013MiB/s (1063MB/s), 63.3MiB/s-72.6MiB/s (66.4MB/s-76.2MB/s), io=4096MiB (4295MB), run=3524-4042msec
Quoted 3 lines
for 'latency' with 'decay' set to 10.
That's on a 32G FC testbed running against a brd target, fio running with 16 thread.
Can you quantify the improvement? Also, the name latency suggest that latency should be improved no?
Quoted 18 lines
As usual, comments and reviews are welcome.
Hannes Reinecke (2):
block: track per-node I/O latency
nvme: add 'latency' iopolicyblock/Kconfig | 7 + block/Makefile | 1 + block/blk-mq-debugfs.c | 2 + block/blk-nodelat.c | 368 ++++++++++++++++++++++++++++++++++ block/blk-rq-qos.h | 6 + drivers/nvme/host/multipath.c | 46 ++++- drivers/nvme/host/nvme.h | 2 + include/linux/blk-mq.h | 11 + 8 files changed, 439 insertions(+), 4 deletions(-)create mode 100644
block/blk-nodelat.c
Quoted 39 lines · Sagi Grimberg
Quoted 24 lines · Hannes Reinecke
Hi all,
there had been several attempts to implement a latency-based I/O scheduler for native nvme multipath, all of which had its issues.
So time to start afresh, this time using the QoS framework already present in the block layer.
It consists of two parts:
- a new 'blk-nodelat' QoS module, which is just a simple per-node latency tracker
- a 'latency' nvme I/O policyUsing the 'tiobench' fio script I'm getting:
WRITE: bw=531MiB/s (556MB/s), 33.2MiB/s-52.4MiB/s (34.8MB/s-54.9MB/s), io=4096MiB (4295MB), run=4888-7718msec
WRITE: bw=539MiB/s (566MB/s), 33.7MiB/s-50.9MiB/s (35.3MB/s-53.3MB/s), io=4096MiB (4295MB), run=5033-7594msec
READ: bw=898MiB/s (942MB/s), 56.1MiB/s-75.4MiB/s (58.9MB/s-79.0MB/s), io=4096MiB (4295MB), run=3397-4560msec
READ: bw=1023MiB/s (1072MB/s), 63.9MiB/s-75.1MiB/s (67.0MB/s-78.8MB/s), io=4096MiB (4295MB), run=3408-4005msecfor 'round-robin' and
Quoted 11 lines
WRITE: bw=574MiB/s (601MB/s), 35.8MiB/s-45.5MiB/s (37.6MB/s-47.7MB/s), io=4096MiB (4295MB), run=5629-7142msec
WRITE: bw=639MiB/s (670MB/s), 39.9MiB/s-47.5MiB/s (41.9MB/s-49.8MB/s), io=4096MiB (4295MB), run=5388-6408msec
READ: bw=1024MiB/s (1074MB/s), 64.0MiB/s-73.7MiB/s (67.1MB/s-77.2MB/s), io=4096MiB (4295MB), run=3475-4000msec
READ: bw=1013MiB/s (1063MB/s), 63.3MiB/s-72.6MiB/s (66.4MB/s-76.2MB/s), io=4096MiB (4295MB), run=3524-4042msec for 'latency' with 'decay' set to 10.
That's on a 32G FC testbed running against a brd target, fio running with 16 thread.
Quoted 3 lines
Can you quantify the improvement? Also, the name latency suggest that latency should be improved no?
'latency' refers to 'latency-based' I/O scheduler, ie it selects the path with the least latency. It does not necessarily _improve_ the latency. Eg for truly symmetric fabrics it doesn't. It _does_ improve matters when running on asymmetric fabrics (eg on a two socket system with two PCI HBAs, each connected to one socket, or like the example above with one path via 'loop', and the other via 'tcp' and address '127.0.0.1').
And, of course, if you have congested fabrics, where it should be able to direct I/O to the least congested path.
But I'll see to extract the latency numbers, too.
What I really wanted to show is that we _can_ track latency without harming performance.
Cheers,
Hannes