Skip to contentSkip to conversation

null_blk: documentation, refactor and warning

PATCH In review Matias Bjorling 3 participants 6 messages Dec 18, 2013 – Dec 19, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover null_blk: documentation, refactor and warning

Hi Jens,

I took the liberty to write up some documentation for the blk_null driver.

Additionally, factored the init code paths to use the standard error return codes. Shuffled some code to aid sharing between the block-layer instantiations. And at last added a warning when submit_queues parameter is ignored.

Thanks,
Matias

Matias Bjorling (3):
null_blk: documentation
null_blk: refactor init and init errors code paths
null_blk: warning on ignored submit_queues param

 Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
 drivers/block/null_blk.c         | 66 +++++++++++++++++++++++--------------
 2 files changed, 112 insertions(+), 25 deletions(-)
 create mode 100644 Documentation/block/null_blk.txt
Signature · 1 line
1.8.3.2
#1 lore raw

PATCH 1/3 null_blk: documentation

Add description of module and its parameters.

---
 Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 Documentation/block/null_blk.txt
Documentation/block/null_blk.txt new file +71 -0
new file mode 100644
index 0000000..9e1b047
@@ -0,0 +1,71 @@
+Null block device driver
+================================================================================
+
+I. Overview
+
+The null block device (/dev/nullb*) is used for benchmarking the various
+block-layer implementations. It emulates a block device of X gigabytes in size.
+The following instances are possible:
+
+  Single-queue block-layer
+    - Request-based.
+    - Single submission queue per device.
+    - Implements IO scheduling algorithms (CFQ, Deadline, noop).
+  Multi-queue block-layer
+    - Request-based.
+    - Configurable submission queues per device.
+  No block-layer (Known as bio-based)
+    - Bio-based. IO requests are submitted directly to the device driver.
+    - Directly accepts bio data structure and returns them.
+
+All of them has a completion queue for each core in the system.
+
+II. Module parameters applicable for all instances:
+
+queue_mode=[0-2]: Default: 2-Multi-queue
+  Selects which block-layer the module should instantiate with.
+
+  0: Bio-based.
+  1: Single-queue.
+  2: Multi-queue.
+
+home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
+  Selects what socket the data structures is allocated from.
+
+gb=[Size in GB]: Default: 250GB
+  The size of the device reported to the system.
+
+bs=[Block size (in bytes)]: Default: 512 bytes
+  The block size reported to the system.
+
+nr_devices=[Num. devices]: Default: 2
+  Number of block devices instantiated. They are instantiated as /dev/nullb0,
+  etc.
+
+irq_mode=[0-2]: Default: Soft-irq
+  The completion mode used for completing IOs to the block-layer.
+
+  0: None.
+  1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead
+     when IOs are issued from another socket than the home the device is
+     connected to.
+  2: Timer: Waits a specific period (completion_nsec) for each IO before
+     completion.
+
+completion_nsec=[Num. ns]: Default: 10.000ns
+  Combined with irq_mode=2 (timer). The time each completion event must wait.
+
+submit_queues=[0..nr_cpus]:
+  The number of submission queues attached to the device driver. If unset, it
+  defaults to 1 on single-queue and bio-based instances. For multi-queue,
+  its ignored when use_per_node_hctx module parameter is 1.
+
+hw_queue_depth=[0..qdepth]: Defaults: 64
+  The hardware queue depth of the device.
+
+III: Multi-queue specific parameters
+
+use_per_node_hctx=[0/1]: Defaults: 1
+  If 1, the multi-queue block layer is instantiated with a hardware dispatch
+  queue for each CPU node in the system. If 0, it is instantiated with the
+  number of queues defined in the submit_queues parameter.
Signature · 1 line
1.8.3.2
reviewer
#2 lore raw

Re: [PATCH 1/3] null_blk: documentation

Quoted 25 lines · Matias Bjorling

Add description of module and its parameters.

---
 Documentation/block/null_blk.txt | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 Documentation/block/null_blk.txt
Documentation/block/null_blk.txt new file +11 -0
new file mode 100644
index 0000000..9e1b047
@@ -0,0 +1,71 @@
+Null block device driver
+================================================================================
+
+I. Overview
+
+The null block device (/dev/nullb*) is used for benchmarking the various
+block-layer implementations. It emulates a block device of X gigabytes in size.
+The following instances are possible:
+
+  Single-queue block-layer
+    - Request-based.

Documentation/block/null_blk.txt line 21

Quoted 10 lines
Documentation/block/null_blk.txt new file +10 -0
+    - Single submission queue per device.
+    - Implements IO scheduling algorithms (CFQ, Deadline, noop).
+  Multi-queue block-layer
+    - Request-based.
+    - Configurable submission queues per device.
+  No block-layer (Known as bio-based)
+    - Bio-based. IO requests are submitted directly to the device driver.
+    - Directly accepts bio data structure and returns them.
+
+All of them has a completion queue for each core in the system.

have

The text sometimes uses "core" vs. "socket" vs. "cpu(s)". Did you double-check all of these uses? Thanks.

Documentation/block/null_blk.txt line 33

Quoted 12 lines
Documentation/block/null_blk.txt new file +12 -0
+
+II. Module parameters applicable for all instances:
+
+queue_mode=[0-2]: Default: 2-Multi-queue
+  Selects which block-layer the module should instantiate with.
+
+  0: Bio-based.
+  1: Single-queue.
+  2: Multi-queue.
+
+home_node=[0--nr_nodes]: Default: NUMA_NO_NODE
+  Selects what socket the data structures is allocated from.

are

Documentation/block/null_blk.txt line 41

Quoted 8 lines
Documentation/block/null_blk.txt new file +8 -0
+
+gb=[Size in GB]: Default: 250GB
+  The size of the device reported to the system.
+
+bs=[Block size (in bytes)]: Default: 512 bytes
+  The block size reported to the system.
+
+nr_devices=[Num. devices]: Default: 2

Number

Documentation/block/null_blk.txt line 45

Quoted 4 lines
Documentation/block/null_blk.txt new file +4 -0
+  Number of block devices instantiated. They are instantiated as /dev/nullb0,
+  etc.
+
+irq_mode=[0-2]: Default: Soft-irq

Default: 1-Soft-irq

Documentation/block/null_blk.txt line 49

Quoted 4 lines
Documentation/block/null_blk.txt new file +4 -0
+  The completion mode used for completing IOs to the block-layer.
+
+  0: None.
+  1: Soft-irq. Uses ipi to complete IOs across sockets. Simulates the overhead

IPI

Documentation/block/null_blk.txt line 55

Quoted 6 lines
Documentation/block/null_blk.txt new file +6 -0
+     when IOs are issued from another socket than the home the device is
+     connected to.
+  2: Timer: Waits a specific period (completion_nsec) for each IO before
+     completion.
+
+completion_nsec=[Num. ns]: Default: 10.000ns

Number

Documentation/block/null_blk.txt line 61

Quoted 6 lines
Documentation/block/null_blk.txt new file +6 -0
+  Combined with irq_mode=2 (timer). The time each completion event must wait.
+
+submit_queues=[0..nr_cpus]:
+  The number of submission queues attached to the device driver. If unset, it
+  defaults to 1 on single-queue and bio-based instances. For multi-queue,
+  its ignored when use_per_node_hctx module parameter is 1.

it is

Documentation/block/null_blk.txt line 63

Quoted 2 lines
Documentation/block/null_blk.txt new file +2 -0
+
+hw_queue_depth=[0..qdepth]: Defaults: 64

Default:

Documentation/block/null_blk.txt line 68

Quoted 5 lines
Documentation/block/null_blk.txt new file +5 -0
+  The hardware queue depth of the device.
+
+III: Multi-queue specific parameters
+
+use_per_node_hctx=[0/1]: Defaults: 1

Default:

Quoted 4 lines
Documentation/block/null_blk.txt new file +3 -0
+  If 1, the multi-queue block layer is instantiated with a hardware dispatch
+  queue for each CPU node in the system. If 0, it is instantiated with the
+  number of queues defined in the submit_queues parameter.
Signature · 1 line
~Randy
#3 lore raw

PATCH 2/3 null_blk: refactor init and init errors code paths

In reply to Matias Bjorling

Simplify the initialization logic of the three block-layers.

- The queue initialization is split into two parts. This allows reuse of code when initializing the sq-, bio- and mq-based layers.
- Set submit_queues default value to 0 and always set it at init time.
- Simplify the init error code paths.

---
 drivers/block/null_blk.c | 63 +++++++++++++++++++++++++++++-------------------
 1 file changed, 38 insertions(+), 25 deletions(-)
drivers/block/null_blk.c +38 -25
index f370fc1..f0aeb2a 100644
@@ -65,7 +65,7 @@ enum {
 	NULL_Q_MQ		= 2,
 };
 
-static int submit_queues = 1;
+static int submit_queues;
 module_param(submit_queues, int, S_IRUGO);
 MODULE_PARM_DESC(submit_queues, "Number of submission queues");
 
@@ -355,16 +355,24 @@ static void null_free_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_index)
 	kfree(hctx);
 }
 
+static void null_init_queue(struct nullb *nullb, struct nullb_queue *nq)
+{
+	BUG_ON(!nullb);
+	BUG_ON(!nq);
+
+	init_waitqueue_head(&nq->wait);
+	nq->queue_depth = nullb->queue_depth;
+}
+
 static int null_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 			  unsigned int index)
 {
 	struct nullb *nullb = data;
 	struct nullb_queue *nq = &nullb->queues[index];
 
-	init_waitqueue_head(&nq->wait);
-	nq->queue_depth = nullb->queue_depth;
-	nullb->nr_queues++;
 	hctx->driver_data = nq;
+	null_init_queue(nullb, nq);
+	nullb->nr_queues++;
 
 	return 0;
 }
@@ -417,13 +425,13 @@ static int setup_commands(struct nullb_queue *nq)
 
 	nq->cmds = kzalloc(nq->queue_depth * sizeof(*cmd), GFP_KERNEL);
 	if (!nq->cmds)
-		return 1;
+		return -ENOMEM;
 
 	tag_size = ALIGN(nq->queue_depth, BITS_PER_LONG) / BITS_PER_LONG;
 	nq->tag_map = kzalloc(tag_size * sizeof(unsigned long), GFP_KERNEL);
 	if (!nq->tag_map) {
 		kfree(nq->cmds);
-		return 1;
+		return -ENOMEM;
 	}
 
 	for (i = 0; i < nq->queue_depth; i++) {
@@ -454,33 +462,37 @@ static void cleanup_queues(struct nullb *nullb)
 
 static int setup_queues(struct nullb *nullb)
 {
-	struct nullb_queue *nq;
-	int i;
-
-	nullb->queues = kzalloc(submit_queues * sizeof(*nq), GFP_KERNEL);
+	nullb->queues = kzalloc(submit_queues * sizeof(struct nullb_queue),
+								GFP_KERNEL);
 	if (!nullb->queues)
-		return 1;
+		return -ENOMEM;
 
 	nullb->nr_queues = 0;
 	nullb->queue_depth = hw_queue_depth;
 
-	if (queue_mode == NULL_Q_MQ)
-		return 0;
+	return 0;
+}
+
+static int init_driver_queues(struct nullb *nullb)
+{
+	struct nullb_queue *nq;
+	int i, ret = 0;
 
 	for (i = 0; i < submit_queues; i++) {
 		nq = &nullb->queues[i];
-		init_waitqueue_head(&nq->wait);
-		nq->queue_depth = hw_queue_depth;
-		if (setup_commands(nq))
-			break;
+
+		null_init_queue(nullb, nq);
+
+		ret = setup_commands(nq);
+		if (ret)
+			goto err_queue;
 		nullb->nr_queues++;
 	}
 
-	if (i == submit_queues)
-		return 0;
-
+	return 0;
+err_queue:
 	cleanup_queues(nullb);
-	return 1;
+	return ret;
 }
 
 static int null_add_dev(void)
@@ -495,9 +507,6 @@ static int null_add_dev(void)
 
 	spin_lock_init(&nullb->lock);
 
-	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
-		submit_queues = nr_online_nodes;
-
 	if (setup_queues(nullb))
 		goto err;
 
@@ -518,11 +527,13 @@ static int null_add_dev(void)
 	} else if (queue_mode == NULL_Q_BIO) {
 		nullb->q = blk_alloc_queue_node(GFP_KERNEL, home_node);
 		blk_queue_make_request(nullb->q, null_queue_bio);
+		init_driver_queues(nullb);
 	} else {
 		nullb->q = blk_init_queue_node(null_request_fn, &nullb->lock, home_node);
 		blk_queue_prep_rq(nullb->q, null_rq_prep_fn);
 		if (nullb->q)
 			blk_queue_softirq_done(nullb->q, null_softirq_done_fn);
+		init_driver_queues(nullb);
 	}
 
 	if (!nullb->q)
@@ -579,7 +590,9 @@ static int __init null_init(void)
 	}
 #endif
 
-	if (submit_queues > nr_cpu_ids)
+	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
+		submit_queues = nr_online_nodes;
+	else if (submit_queues > nr_cpu_ids)
 		submit_queues = nr_cpu_ids;
 	else if (!submit_queues)
 		submit_queues = 1;
Signature · 1 line
1.8.3.2
#4 lore raw

PATCH 3/3 null_blk: warning on ignored submit_queues param

In reply to Matias Bjorling

Let the user know when the number of submission queues are being ignored.

---
 drivers/block/null_blk.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/block/null_blk.c +5 -2
index f0aeb2a..8f2e7c3 100644
@@ -590,9 +590,12 @@ static int __init null_init(void)
 	}
 #endif
 
-	if (queue_mode == NULL_Q_MQ && use_per_node_hctx)
+	if (queue_mode == NULL_Q_MQ && use_per_node_hctx) {
+		if (submit_queues > 0)
+			pr_warn("null_blk: submit_queues param is set to %u.",
+							nr_online_nodes);
 		submit_queues = nr_online_nodes;
-	else if (submit_queues > nr_cpu_ids)
+	} else if (submit_queues > nr_cpu_ids)
 		submit_queues = nr_cpu_ids;
 	else if (!submit_queues)
 		submit_queues = 1;
Signature · 1 line
1.8.3.2
maintainer
#5 lore raw
In reply to Matias Bjorling
Quoted 7 lines · Matias Bjorling

Hi Jens,

I took the liberty to write up some documentation for the blk_null driver.

Additionally, factored the init code paths to use the standard error return codes. Shuffled some code to aid sharing between the block-layer instantiations. And at last added a warning when submit_queues parameter is ignored.

Thanks Matias, applied! We still need a patch to not default to per-node hctx, though. And to be more clever when submit_queues is given and per-node is true.

Signature · 1 line
Jens Axboe