nvme: Add weighted-round-robin arbitration support
A message that is not in the archive (sent off-list or not yet received): CGME20180104150334epcas1p14579a6ab7bcf1b6ce326d0bce89c91e1@epcas1p1.samsung.com
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
This patch enables support for Weighted-Round-Robin (WRR) arbitration, so that applications can make use of the prioritization capabilities natively present in NVMe controller.
- It links existing io-nice classes (real-time, best-effort, none, low) to NVMe priorities (urgent, high, medium, low). This is done through 'request->ioprio' field inside 'queue_rq' function.
- Current driver has 1:1 mapping (1 SQ, 1 CQ) per cpu, encapsulated in 'nvmeq' structure. This patch refactors the code so that N:1 mapping per cpu can be created; 'nvmeq' has been changed to contain variable number of SQ related fields. For WRR, 4 submission-queues (corresponding to each queue priorites) need to be created on each cpu.
- When 'enable_wrr' module param is passed, it creates 4:1 mapping and enables controller in WRR mode. Otherwise, it cotinues to retain 1:1 mapping and controller remains in RR mode.
- NVMe device may have less number of queues than required for 4:1 mapping per cpu. For example, when num_possible_cpus is 64, 256 submission-queues are required for 4:1 mapping while device may support, say, 128.
This case is handled by creating 32 queue-pairs which are shared among 64 cpus. Another way to handle this could have been reducing to 3:1 or 2:1 mapping (and remapping 4 ionice classes as well).
-Admin queue, contains 1:1 mapping irrespective of the mode (RR or WRR) used.
Earlier I had collected results on 4.10
kernel, which indicate distribution happening as per weights applied. Please refer to (section 5) in this paper -
http://www.usenix.org/system/files/conference/hotstorage17/hotstorage17-paper-joshi.pdf
I see similar results in current kernel as well.
--- drivers/nvme/host/core.c | 4 +- drivers/nvme/host/pci.c | 310 +++++++++++++++++++++++++++++++---------------- include/linux/nvme.h | 1 + 3 files changed, 210 insertions(+), 105 deletions(-)
drivers/nvme/host/core.c +2 -2
@@ -1649,9 +1649,9 @@ int nvme_enable_ctrl(struct nvme_ctrl *ctrl, u64 cap) ctrl->page_size =1 << page_shift; - ctrl->ctrl_config =NVME_CC_CSS_NVM ; + ctrl->ctrl_config |=NVME_CC_CSS_NVM ; ctrl->ctrl_config |= (page_shift -12 ) <<NVME_CC_MPS_SHIFT ; - ctrl->ctrl_config |=NVME_CC_AMS_RR |NVME_CC_SHN_NONE ; + ctrl->ctrl_config |=NVME_CC_SHN_NONE ; ctrl->ctrl_config |=NVME_CC_IOSQES |NVME_CC_IOCQES ; ctrl->ctrl_config |=NVME_CC_ENABLE ;
drivers/nvme/host/pci.c +207 -103
680 lines not on this page · raw message
include/linux/nvme.h +1 -0
@@ -116,6 +116,7 @@ enum { };#define NVME_CAP_MQES (cap) ((cap) & 0xffff) +#define NVME_CAP_WRR (cap) (((cap) >> 17) & 0x1)#define NVME_CAP_TIMEOUT (cap) (((cap) >> 24) & 0xff)#define NVME_CAP_STRIDE (cap) (((cap) >> 32) & 0xf)#define NVME_CAP_NSSRC (cap) (((cap) >> 36) & 0x1)
Signature · 1 line
2.7.4
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
Quoted 8 lines · Kanchan Joshi
This patch enables support for Weighted-Round-Robin (WRR) arbitration, so that applications can make use of the prioritization capabilities natively present in NVMe controller.
- It links existing io-nice classes (real-time, best-effort, none, low) to NVMe priorities (urgent, high, medium, low). This is done through '
request->ioprio' field inside 'queue_rq' function.
Quoted 5 lines
- Current driver has 1:1 mapping (1 SQ, 1 CQ) per cpu, encapsulated in 'nvmeq' structure. This patch refactors the code so that N:1 mapping per cpu can be created; 'nvmeq' has been changed to contain variable number of SQ related fields. For WRR, 4 submission-queues (corresponding to each queue priorites) need to be created on each cpu.
You have a single tagset per CQ for up to N oustandanding commands, but allocate enough submission entries for 4 * N. And since they're sharing tags, a lower pri task can limit a high-pri one from getting a tag. I think we could use a little more help from the block layer for WRR.
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
Hi Keith,
It seems to me that some sort of differentiation (among tasks of various classes) during tag allocation is required to handle this. Perhaps, static division of available tags into 4 priority classes needs to be done in block layer.
Or does there exist any other facility in block layer which you hinted at? I would appreciate your suggestions/concerns.
Thanks
Kanchan
Quoted 22 lines · Keith Busch
Quoted 8 lines · Kanchan Joshi
This patch enables support for Weighted-Round-Robin (WRR) arbitration, so that applications can make use of the prioritization capabilities natively present in NVMe controller.
- It links existing io-nice classes (real-time, best-effort, none, low) to NVMe priorities (urgent, high, medium, low). This is done through '
request->ioprio' field inside 'queue_rq' function.Quoted 5 lines
- Current driver has 1:1 mapping (1 SQ, 1 CQ) per cpu, encapsulated in 'nvmeq' structure. This patch refactors the code so that N:1 mapping per cpu can be created; 'nvmeq' has been changed to contain variable number of SQ related fields. For WRR, 4 submission-queues (corresponding to each queue priorites) need to be created on each cpu.
You have a single tagset per CQ for up to N oustandanding commands, but allocate enough submission entries for 4 * N. And since they're sharing tags, a lower pri task can limit a high-pri one from getting a tag. I think we could use a little more help from the block layer for WRR.
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
Quoted 4 lines · Kanchan Joshi
It seems to me that some sort of differentiation (among tasks of various classes) during tag allocation is required to handle this. Perhaps, static division of available tags into 4 priority classes needs to be done in block layer.
Yes, I was thinking either some kind priorty based tag reservation, or possibly entirely different hctx's for each priority. The block layer currently doesn't provide this kind of separation, but it was mentioned the latter might be appropriate for hipri/polled IO, so maybe that idea can extend to WRR.
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
Different hctx sounds good, and can be used for both polled IO and WRR (somewhat more resource-heavy for WRR though). May I know if that work is in progress already.
It seems NVMe driver does not have to do much in that scheme-of-things, while block layer takes most of the load.
Quoted 14 lines · Keith Busch
Quoted 4 lines · Kanchan Joshi
It seems to me that some sort of differentiation (among tasks of various classes) during tag allocation is required to handle this. Perhaps, static division of available tags into 4 priority classes needs to be done in block layer.
Yes, I was thinking either some kind priorty based tag reservation, or possibly entirely different hctx's for each priority. The block layer currently doesn't provide this kind of separation, but it was mentioned the latter might be appropriate for hipri/polled IO, so maybe that idea can extend to WRR.
PATCH 1/1 nvme: Add weighted-round-robin arbitration support
Quoted 3 lines · Kanchan Joshi
Different hctx sounds good, and can be used for both polled IO and WRR (somewhat more resource-heavy for WRR though). May I know if that work is in progress already.
As far as I know, no one has anything developed in this area yet, or at least not to the point of sharing.
Quoted 2 lines
It seems NVMe driver does not have to do much in that scheme-of-things, while block layer takes most of the load.
Yep, that's the idea. Just need to see if anyone comes up with something. ;)
Sagi Grimberg
maintainerPATCH 1/1 nvme: Add weighted-round-robin arbitration support
In reply to Kanchan JoshiHi Joshi,
Quoted 18 lines
This patch enables support for Weighted-Round-Robin (WRR) arbitration, so that applications can make use of the prioritization capabilities natively present in NVMe controller.
- It links existing io-nice classes (real-time, best-effort, none, low) to NVMe priorities (urgent, high, medium, low). This is done through '
request->ioprio' field inside 'queue_rq' function.- Current driver has 1:1 mapping (1 SQ, 1 CQ) per cpu, encapsulated in 'nvmeq' structure. This patch refactors the code so that N:1 mapping per cpu can be created; 'nvmeq' has been changed to contain variable number of SQ related fields. For WRR, 4 submission-queues (corresponding to each queue priorites) need to be created on each cpu.
- When '
enable_wrr' module param is passed, it creates 4:1 mapping and enables controller in WRR mode. Otherwise, it cotinues to retain 1:1 mapping and controller remains in RR mode.
Quoted 8 lines
- NVMe device may have less number of queues than required for 4:1 mapping per cpu. For example, when
num_possible_cpusis 64, 256 submission-queues are required for 4:1 mapping while device may support, say, 128.
This case is handled by creating 32 queue-pairs which are shared among 64 cpus. Another way to handle this could have been reducing to 3:1 or 2:1 mapping (and remapping 4 ionice classes as well).-Admin queue, contains 1:1 mapping irrespective of the mode (RR or WRR) used.
Regardless the dicussion with Keihth, this patch should be divided into three or four preparatory patches and the wrr patch.
1. keeping nvmeq->cq_db
2. changing nvme_enable_ctrl not to set ctrl_config (needs to verify doesn't break anything)
3. keeping multiple sqs per nvmeq and plumbing the sq_index
4. wire up wrr
This is true also if this is moving to blk-mq