Skip to contentSkip to conversation

NVMe: increase depth of admin queue

PATCH New Jens Axboe 3 participants 3 messages Mar 6, 2015 – Mar 16, 2015

Subsystems (2)
Block layer NVMe
lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

PATCH 1/1 NVMe: increase depth of admin queue

Hi,

Usually the admin queue depth of 64 is plenty, but for some use cases we really need it larger. Examples are use cases like MAT, where you have to touch all of NAND for init/format like purposes. In those cases, we see a good 2x increase with an increased queue depth.

Signed-off-by Jens Axboe <axboe at fb.com>
drivers/block/nvme-core.c +1 -1
index ceb32dd52a6c..b67b520e489d 100644
@@ -44,7 +44,7 @@
 
 #define NVME_MINORS		(1U << MINORBITS)
 #define NVME_Q_DEPTH		1024
-#define NVME_AQ_DEPTH		64
+#define NVME_AQ_DEPTH		256
 #define SQ_SIZE(depth)		(depth * sizeof(struct nvme_command))
 #define CQ_SIZE(depth)		(depth * sizeof(struct nvme_completion))
 #define ADMIN_TIMEOUT		(admin_timeout * HZ)
Signature · 1 line
Jens Axboe
#1 lore raw

PATCH 1/1 NVMe: increase depth of admin queue

Quoted 14 lines · Jens Axboe

Hi,

Usually the admin queue depth of 64 is plenty, but for some use cases we really need it larger. Examples are use cases like MAT, where you have to touch all of NAND for init/format like purposes. In those cases, we see a good 2x increase with an increased queue depth.

Signed-off-by Jens Axboe <axboe at fb.com>
drivers/block/nvme-core.c +0 -0
index ceb32dd52a6c..b67b520e489d 100644

drivers/block/nvme-core.c line 47

Quoted 5 lines
drivers/block/nvme-core.c +1 -1
@@ -44,7 +44,7 @@
 
 #define NVME_MINORS		(1U << MINORBITS)
 #define NVME_Q_DEPTH		1024
-#define NVME_AQ_DEPTH		64
+#define NVME_AQ_DEPTH		256

Would this be better served to have it be a module parameter so someone can change it to/from 64 to whatever depth they think they need for their targeted system and workload?

Quoted 4 lines
drivers/block/nvme-core.c +0 -0
 #define SQ_SIZE(depth)		(depth * sizeof(struct nvme_command))
 #define CQ_SIZE(depth)		(depth * sizeof(struct nvme_completion))
 #define ADMIN_TIMEOUT		(admin_timeout * HZ)
#2 lore raw

PATCH 1/1 NVMe: increase depth of admin queue

In reply to Jens Axboe
Quoted 6 lines · Jens Axboe

Usually the admin queue depth of 64 is plenty, but for some use cases we really need it larger. Examples are use cases like MAT, where you have to touch all of NAND for init/format like purposes. In those cases, we see a good 2x increase with an increased queue depth.

Signed-off-by Jens Axboe <axboe at fb.com>

Sounds good to me. If we encounter a case where changing this again is desirable, we might want to take Jay's suggestion to make it a module param.

Acked-by Keith Busch <keith.busch at intel.com>