null_blk: documentation, refactor and warning
Subsystems (2)
cover null_blk: documentation, refactor and warning
Hi Jens,
I took the liberty to write up some documentation for the blk_null driver.
Additionally, factored the init code paths to use the standard error return codes. Shuffled some code to aid sharing between the block-layer instantiations. And at last added a warning when submit_queues parameter is ignored.
Thanks,
Matias
Matias Bjorling (3):
null_blk: documentation
null_blk: refactor init and init errors code paths
null_blk: warning on ignored submit_queues param
Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++ drivers/block/null_blk.c | 66 +++++++++++++++++++++++-------------- 2 files changed, 112 insertions(+), 25 deletions(-) create mode 100644 Documentation/block/null_blk.txt
Signature · 1 line
1.8.3.2
PATCH 1/3 null_blk: documentation
Add description of module and its parameters.
--- Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 Documentation/block/null_blk.txt
Documentation/block/null_blk.txt new file +71 -0
@@ -0,0 +1,71 @@ +Null block device driver +================================================================================ + +I. Overview + +The null block device (/dev/nullb*) is used for benchmarking the various +block-layer implementations. It emulates a block device of X gigabytes in size. +The following instances are possible: + + Single-queue block-layer + - Request-based. + - Single submission queue per device. + - Implements IO scheduling algorithms (CFQ, Deadline, noop). + Multi-queue block-layer + - Request-based. + - Configurable submission queues per device. + No block-layer (Known as bio-based) + - Bio-based. IO requests are submitted directly to the device driver. + - Directly accepts bio data structure and returns them. + +All of them has a completion queue for each core in the system. + +II. Module parameters applicable for all instances: + +queue_mode=[0-2]: Default: 2-Multi-queue + Selects which block-layer the module should instantiate with. + + 0: Bio-based. + 1: Single-queue. + 2: Multi-queue. + +home_node=[0--nr_nodes]: Default: NUMA_NO_NODE + Selects what socket the data structures is allocated from. + +gb=[Size in GB]: Default: 250GB + The size of the device reported to the system. + +bs=[Block size (in bytes)]: Default: 512 bytes + The block size reported to the system. + +nr_devices=[Num. devices]: Default: 2 + Number of block devices instantiated. They are instantiated as /dev/nullb0, + etc. + +irq_mode=[0-2]: Default: Soft-irq + The completion mode used for completing IOs to the block-layer. + + 0: None. + 1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead + when IOs are issued from another socket than the home the device is + connected to. + 2: Timer: Waits a specific period (completion_nsec) for each IO before + completion. + +completion_nsec=[Num. ns]: Default: 10.000ns + Combined with irq_mode=2 (timer). The time each completion event must wait. + +submit_queues=[0..nr_cpus]: + The number of submission queues attached to the device driver. If unset, it + defaults to 1 on single-queue and bio-based instances. For multi-queue, + its ignored when use_per_node_hctx module parameter is 1. + +hw_queue_depth=[0..qdepth]: Defaults: 64 + The hardware queue depth of the device. + +III: Multi-queue specific parameters + +use_per_node_hctx=[0/1]: Defaults: 1 + If 1, the multi-queue block layer is instantiated with a hardware dispatch + queue for each CPU node in the system. If 0, it is instantiated with the + number of queues defined in the submit_queues parameter.
Signature · 1 line
1.8.3.2
Randy Dunlap
reviewerRe: [PATCH 1/3] null_blk: documentation
Quoted 25 lines · Matias Bjorling
Add description of module and its parameters.
Signed-off-by Matias Bjorling <[email protected]>--- Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 Documentation/block/null_blk.txtDocumentation/block/null_blk.txt new file +11
-0@@ -0,0 +1,71 @@ +Null block device driver +================================================================================ + +I. Overview + +The null block device (/dev/nullb*) is used for benchmarking the various +block-layer implementations. It emulates a block device of X gigabytes in size. +The following instances are possible: + + Single-queue block-layer + - Request-based.
Documentation/block/null_blk.txt line 21
Quoted 10 lines
Documentation/block/null_blk.txt new file +10
-0+ - Single submission queue per device. + - Implements IO scheduling algorithms (CFQ, Deadline, noop). + Multi-queue block-layer + - Request-based. + - Configurable submission queues per device. + No block-layer (Known as bio-based) + - Bio-based. IO requests are submitted directly to the device driver. + - Directly accepts bio data structure and returns them. + +All of them has a completion queue for each core in the system.
have
The text sometimes uses "core" vs. "socket" vs. "cpu(s)". Did you double-check all of these uses? Thanks.
Documentation/block/null_blk.txt line 33
Quoted 12 lines
Documentation/block/null_blk.txt new file +12
-0+ +II. Module parameters applicable for all instances: + +queue_mode=[0-2]: Default: 2-Multi-queue + Selects which block-layer the module should instantiate with. + + 0: Bio-based. + 1: Single-queue. + 2: Multi-queue. + +home_node=[0--nr_nodes]: Default: NUMA_NO_NODE + Selects what socket the data structures is allocated from.
are
Documentation/block/null_blk.txt line 41
Quoted 8 lines
Documentation/block/null_blk.txt new file +8
-0+ +gb=[Size in GB]: Default: 250GB + The size of the device reported to the system. + +bs=[Block size (in bytes)]: Default: 512 bytes + The block size reported to the system. + +nr_devices=[Num. devices]: Default: 2
Number
Documentation/block/null_blk.txt line 45
Quoted 4 lines
Documentation/block/null_blk.txt new file +4
-0+ Number of block devices instantiated. They are instantiated as /dev/nullb0, + etc. + +irq_mode=[0-2]: Default: Soft-irq
Default: 1-Soft-irq
Documentation/block/null_blk.txt line 49
Quoted 4 lines
Documentation/block/null_blk.txt new file +4
-0+ The completion mode used for completing IOs to the block-layer. + + 0: None. + 1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead
IPI
Documentation/block/null_blk.txt line 55
Quoted 6 lines
Documentation/block/null_blk.txt new file +6
-0+ when IOs are issued from another socket than the home the device is + connected to. + 2: Timer: Waits a specific period (completion_nsec) for each IO before + completion. + +completion_nsec=[Num. ns]: Default: 10.000ns
Number
Documentation/block/null_blk.txt line 61
Quoted 6 lines
Documentation/block/null_blk.txt new file +6
-0+ Combined with irq_mode=2 (timer). The time each completion event must wait. + +submit_queues=[0..nr_cpus]: + The number of submission queues attached to the device driver. If unset, it + defaults to 1 on single-queue and bio-based instances. For multi-queue, + its ignored when use_per_node_hctx module parameter is 1.
it is
Documentation/block/null_blk.txt line 63
Quoted 2 lines
Documentation/block/null_blk.txt new file +2
-0+ +hw_queue_depth=[0..qdepth]: Defaults: 64
Default:
Documentation/block/null_blk.txt line 68
Quoted 5 lines
Documentation/block/null_blk.txt new file +5
-0+ The hardware queue depth of the device. + +III: Multi-queue specific parameters + +use_per_node_hctx=[0/1]: Defaults: 1
Default:
Quoted 4 lines
Documentation/block/null_blk.txt new file +3
-0+ If 1, the multi-queue block layer is instantiated with a hardware dispatch + queue for each CPU node in the system. If 0, it is instantiated with the + number of queues defined in the submit_queues parameter.
Signature · 1 line
~Randy
PATCH 2/3 null_blk: refactor init and init errors code paths
In reply to Matias BjorlingSimplify the initialization logic of the three block-layers.
- The queue initialization is split into two parts. This allows reuse of code when initializing the sq-, bio- and mq-based layers.
- Set submit_queues default value to 0 and always set it at init time.
- Simplify the init error code paths.
--- drivers/block/null_blk.c | 63 +++++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 25 deletions(-)
drivers/block/null_blk.c +38 -25
@@ -65,7 +65,7 @@ enum { NULL_Q_MQ = 2, }; -static int submit_queues =1 ; +static int submit_queues;module_param (submit_queues, int,S_IRUGO );MODULE_PARM_DESC (submit_queues,"Number of submission queues" );
@@ -355,16 +355,24 @@ static void null_free_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_index)kfree (hctx); } +static void null_init_queue (struct nullb *nullb,struct nullb_queue *nq) +{ +BUG_ON (!nullb); +BUG_ON (!nq); + +init_waitqueue_head (&nq->wait); + nq->queue_depth = nullb->queue_depth; +} +static int null_init_hctx (struct blk_mq_hw_ctx *hctx,void *data,unsigned int index) {struct nullb *nullb = data;struct nullb_queue *nq = &nullb->queues[index]; -init_waitqueue_head (&nq->wait); - nq->queue_depth = nullb->queue_depth; - nullb->nr_queues++; hctx->driver_data = nq; +null_init_queue (nullb, nq); + nullb->nr_queues++;return 0 ; }
@@ -417,13 +425,13 @@ static int setup_commands(struct nullb_queue *nq) nq->cmds =kzalloc (nq->queue_depth *sizeof (*cmd),GFP_KERNEL );if (!nq->cmds) -return 1 ; +return -ENOMEM ; tag_size =ALIGN (nq->queue_depth,BITS_PER_LONG ) /BITS_PER_LONG ; nq->tag_map =kzalloc (tag_size *sizeof (unsigned long ),GFP_KERNEL );if (!nq->tag_map) {kfree (nq->cmds); -return 1 ; +return -ENOMEM ; }for (i =0 ; i < nq->queue_depth; i++) {
@@ -454,33 +462,37 @@ static void cleanup_queues(struct nullb *nullb)static int setup_queues (struct nullb *nullb) { -struct nullb_queue *nq; -int i; - - nullb->queues =kzalloc (submit_queues *sizeof (*nq),GFP_KERNEL ); + nullb->queues =kzalloc (submit_queues *sizeof (struct nullb_queue ), +GFP_KERNEL );if (!nullb->queues) -return 1 ; +return -ENOMEM ; nullb->nr_queues =0 ; nullb->queue_depth = hw_queue_depth; -if (queue_mode ==NULL_Q_MQ ) -return 0 ; +return 0 ; +} + +static int init_driver_queues (struct nullb *nullb) +{ +struct nullb_queue *nq; +int i, ret =0 ;for (i =0 ; i < submit_queues; i++) { nq = &nullb->queues[i]; -init_waitqueue_head (&nq->wait); - nq->queue_depth = hw_queue_depth; -if (setup_commands (nq)) -break ; + +null_init_queue (nullb, nq); + + ret =setup_commands (nq); +if (ret) + goto err_queue; nullb->nr_queues++; } -if (i == submit_queues) -return 0 ; - +return 0 ; +err_queue:cleanup_queues (nullb); -return 1 ; +return ret; }static int null_add_dev (void )
@@ -495,9 +507,6 @@ static int null_add_dev(void)spin_lock_init (&nullb->lock); -if (queue_mode ==NULL_Q_MQ && use_per_node_hctx) - submit_queues = nr_online_nodes; -if (setup_queues (nullb)) goto err;
@@ -518,11 +527,13 @@ static int null_add_dev(void) }else if (queue_mode ==NULL_Q_BIO ) { nullb->q =blk_alloc_queue_node (GFP_KERNEL , home_node);blk_queue_make_request (nullb->q, null_queue_bio); +init_driver_queues (nullb); }else { nullb->q =blk_init_queue_node (null_request_fn, &nullb->lock, home_node);blk_queue_prep_rq (nullb->q, null_rq_prep_fn);if (nullb->q)blk_queue_softirq_done (nullb->q, null_softirq_done_fn); +init_driver_queues (nullb); }if (!nullb->q)
@@ -579,7 +590,9 @@ static int __init null_init(void) }#endif -if (submit_queues > nr_cpu_ids) +if (queue_mode ==NULL_Q_MQ && use_per_node_hctx) + submit_queues = nr_online_nodes; +else if (submit_queues > nr_cpu_ids) submit_queues = nr_cpu_ids;else if (!submit_queues) submit_queues =1 ;
Signature · 1 line
1.8.3.2
PATCH 3/3 null_blk: warning on ignored submit_queues param
In reply to Matias BjorlingLet the user know when the number of submission queues are being ignored.
--- drivers/block/null_blk.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/block/null_blk.c +5 -2
@@ -590,9 +590,12 @@ static int __init null_init(void) }#endif -if (queue_mode ==NULL_Q_MQ && use_per_node_hctx) +if (queue_mode ==NULL_Q_MQ && use_per_node_hctx) { +if (submit_queues >0 ) +pr_warn ("null_blk: submit_queues param is set to %u." , + nr_online_nodes); submit_queues = nr_online_nodes; -else if (submit_queues > nr_cpu_ids) + }else if (submit_queues > nr_cpu_ids) submit_queues = nr_cpu_ids;else if (!submit_queues) submit_queues =1 ;
Signature · 1 line
1.8.3.2
Jens Axboe
maintainerQuoted 7 lines · Matias Bjorling
Hi Jens,
I took the liberty to write up some documentation for the
blk_nulldriver.Additionally, factored the init code paths to use the standard error return codes. Shuffled some code to aid sharing between the block-layer instantiations. And at last added a warning when
submit_queuesparameter is ignored.
Thanks Matias, applied! We still need a patch to not default to per-node hctx, though. And to be more clever when submit_queues is given and per-node is true.
Signature · 1 line
Jens Axboe