Delete the get_vector_affinity leftovers
Subsystems (3)
cover Delete the get_vector_affinity leftovers
From: Leon Romanovsky <[email protected]>
There are no drivers that implement .get_vector_affinity(), so delete the RDMA function and simplify block code.
Thanks
P.S. Probably it should go through block tree.
Leon Romanovsky (2):
blk-mq-rdma: Delete not-used multi-queue RDMA map queue code RDMA/core: Delete not-implemented get_vector_affinity
block/Kconfig | 5 ---- block/Makefile | 1 - block/blk-mq-rdma.c | 44 -------------------------------- drivers/infiniband/core/device.c | 1 - drivers/nvme/host/rdma.c | 7 ++--- include/linux/blk-mq-rdma.h | 11 -------- include/rdma/ib_verbs.h | 23 ----------------- 7 files changed, 2 insertions(+), 90 deletions(-) delete mode 100644 block/blk-mq-rdma.c delete mode 100644 include/linux/blk-mq-rdma.h
--
2.26.2
PATCH 1/2 blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
From: Leon Romanovsky <[email protected]>
The RDMA vector affinity code is not backed up by any driver and always returns NULL to every ib_get_vector_affinity() call.
This means that blk_mq_rdma_map_queues() always takes fallback path.
--- block/Kconfig | 5 ----- block/Makefile | 1 - block/blk-mq-rdma.c | 44 ------------------------------------- drivers/nvme/host/rdma.c | 7 ++---- include/linux/blk-mq-rdma.h | 11 ---------- 5 files changed, 2 insertions(+), 66 deletions(-) delete mode 100644 block/blk-mq-rdma.c delete mode 100644 include/linux/blk-mq-rdma.h
block/Kconfig +0 -5
@@ -227,11 +227,6 @@ config BLK_MQ_VIRTIO depends on BLOCK && VIRTIO default y -config BLK_MQ_RDMA - bool - depends on BLOCK && INFINIBAND - default y - config BLK_PM def_bool BLOCK && PM
block/Makefile +0 -1
@@ -29,7 +29,6 @@ obj-$(CONFIG_BLK_DEV_INTEGRITY) += bio-integrity.o blk-integrity.o obj-$(CONFIG_BLK_DEV_INTEGRITY_T10) += t10-pi.o obj-$(CONFIG_BLK_MQ_PCI) += blk-mq-pci.o obj-$(CONFIG_BLK_MQ_VIRTIO) += blk-mq-virtio.o -obj-$(CONFIG_BLK_MQ_RDMA) += blk-mq-rdma.o obj-$(CONFIG_BLK_DEV_ZONED) += blk-zoned.o obj-$(CONFIG_BLK_WBT) += blk-wbt.o obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o
block/blk-mq-rdma.c deleted +0 -44
44 lines not on this page · raw message
drivers/nvme/host/rdma.c +2 -5
@@ -12,7 +12,6 @@#include <linux/string.h> #include <linux/atomic.h> #include <linux/blk-mq.h> -#include <linux/blk-mq-rdma.h> #include <linux/types.h> #include <linux/list.h> #include <linux/mutex.h>
@@ -2171,10 +2170,8 @@ static int nvme_rdma_map_queues(struct blk_mq_tag_set *set) ctrl->io_queues[HCTX_TYPE_DEFAULT ]; set->map[HCTX_TYPE_READ ].queue_offset =0 ; } -blk_mq_rdma_map_queues (&set->map[HCTX_TYPE_DEFAULT ], - ctrl->device->dev,0 ); -blk_mq_rdma_map_queues (&set->map[HCTX_TYPE_READ ], - ctrl->device->dev,0 ); +blk_mq_map_queues (&set->map[HCTX_TYPE_DEFAULT ]); +blk_mq_map_queues (&set->map[HCTX_TYPE_READ ]);if (opts->nr_poll_queues && ctrl->io_queues[HCTX_TYPE_POLL ]) {/* map dedicated poll queues only if we have queues left */
include/linux/blk-mq-rdma.h deleted +0 -11
@@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_BLK_MQ_RDMA_H -#define _LINUX_BLK_MQ_RDMA_H - -struct blk_mq_tag_set ; -struct ib_device ; - -int blk_mq_rdma_map_queues (struct blk_mq_queue_map *map, -struct ib_device *dev,int first_vec); - -#endif /* _LINUX_BLK_MQ_RDMA_H */
--
2.26.2
Christoph Hellwig
maintainerRe: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 9 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The RDMA vector affinity code is not backed up by any driver and always returns NULL to every
ib_get_vector_affinity()call.This means that
blk_mq_rdma_map_queues()always takes fallback path.Fixes 9afc97c29b03 ("mlx5: remove support for ib_get_vector_affinity")Signed-off-by Leon Romanovsky <[email protected]>
So you guys totally broken the nvme queue assignment without even telling anyone? Great job!
Re: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The RDMA vector affinity code is not backed up by any driver and always returns NULL to every
ib_get_vector_affinity()call.
Quoted 7 lines
Quoted 4 lines
This means that
blk_mq_rdma_map_queues()always takes fallback path.Fixes 9afc97c29b03 ("mlx5: remove support for ib_get_vector_affinity")Signed-off-by Leon Romanovsky <[email protected]>So you guys totally broken the nvme queue assignment without even telling anyone? Great job!
Who is "you guys" and it wasn't silent either? I'm sure that Sagi knows the craft.
https://lore.kernel.org/linux-rdma/[email protected]/
commit 759ace7832802eaefbca821b2b43a44ab896b449
Author: Sagi Grimberg <[email protected]> Date: Thu Nov 1 13:08:07 2018 -0700
i40iw: remove support for ib_get_vector_affinity
....
commit 9afc97c29b032af9a4112c2f4a02d5313b4dc71f
Author: Sagi Grimberg <[email protected]> Date: Thu Nov 1 09:13:12 2018 -0700
mlx5: remove support for ib_get_vector_affinity
Thanks
Sagi Grimberg
maintainerRe: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
From: Leon Romanovsky <[email protected]>
The RDMA vector affinity code is not backed up by any driver and always returns NULL to every
ib_get_vector_affinity()call.Quoted 7 lines
Quoted 4 lines
This means that
blk_mq_rdma_map_queues()always takes fallback path.Fixes 9afc97c29b03 ("mlx5: remove support for ib_get_vector_affinity")Signed-off-by Leon Romanovsky <[email protected]>So you guys totally broken the nvme queue assignment without even telling anyone? Great job!
Who is "you guys" and it wasn't silent either? I'm sure that Sagi knows the craft.
https://lore.kernel.org/linux-rdma/[email protected]/commit
759ace7832802eaefbca821b2b43a44ab896b449Author: Sagi Grimberg <[email protected]> Date: Thu Nov 1 13:08:07 2018 -0700i40iw: remove support for
ib_get_vector_affinity....
Quoted 7 lines
commit
9afc97c29b032af9a4112c2f4a02d5313b4dc71fAuthor: Sagi Grimberg <[email protected]> Date: Thu Nov 1 09:13:12 2018 -0700mlx5: remove support for
ib_get_vector_affinityThanks
Yes, basically usage of managed affinity caused people to report regressions not being able to change irq affinity from procfs.
Back then I started a discussion with Thomas to make managed affinity to still allow userspace to modify this, but this was dropped at some point. So currently rdma cannot do automatic irq affinitization out of the box.
Christoph Hellwig
maintainerRe: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 2 lines · Sagi Grimberg
Yes, basically usage of managed affinity caused people to report regressions not being able to change irq affinity from procfs.
Well, why would they change it? The whole point of the infrastructure is that there is a single sane affinity setting for a given setup. Now that setting needed some refinement from the original series (e.g. the current series about only using housekeeping cpus if cpu isolation is in use). But allowing random users to modify affinity is just a receipe for a trainwreck.
So I think we need to bring this back ASAP, as doing affinity right out of the box is an absolute requirement for sane performance without all the benchmarketing deep magic.
Sagi Grimberg
maintainerRe: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 10 lines
Quoted 3 lines
Yes, basically usage of managed affinity caused people to report regressions not being able to change irq affinity from procfs.
Well, why would they change it? The whole point of the infrastructure is that there is a single sane affinity setting for a given setup. Now that setting needed some refinement from the original series (e.g. the current series about only using housekeeping cpus if cpu isolation is in use). But allowing random users to modify affinity is just a receipe for a trainwreck.
Well allowing people to mangle irq affinity settings seem to be a hard requirement from the discussions in the past.
Quoted 3 lines
So I think we need to bring this back ASAP, as doing affinity right out of the box is an absolute requirement for sane performance without all the benchmarketing deep magic.
Well, it's hard to say that setting custom irq affinity settings is deemed non-useful to anyone and hence should be prevented. I'd expect that irq settings have a sane default that works and if someone wants to change it, it can but there should be no guarantees on optimal performance. But IIRC this had some dependencies on drivers and some more infrastructure to handle dynamic changes...
Christoph Hellwig
maintainerRe: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 14 lines · Sagi Grimberg
Quoted 6 lines
Well, why would they change it? The whole point of the infrastructure is that there is a single sane affinity setting for a given setup. Now that setting needed some refinement from the original series (e.g. the current series about only using housekeeping cpus if cpu isolation is in use). But allowing random users to modify affinity is just a receipe for a trainwreck.
Well allowing people to mangle irq affinity settings seem to be a hard requirement from the discussions in the past.
Quoted 3 lines
So I think we need to bring this back ASAP, as doing affinity right out of the box is an absolute requirement for sane performance without all the benchmarketing deep magic.
Quoted 6 lines
Well, it's hard to say that setting custom irq affinity settings is deemed non-useful to anyone and hence should be prevented. I'd expect that irq settings have a sane default that works and if someone wants to change it, it can but there should be no guarantees on optimal performance. But IIRC this had some dependencies on drivers and some more infrastructure to handle dynamic changes...
The problem is that people change random settings. We need to generalize it into a sane API (e.g. the housekeeping CPUs thing which totally makes sense).
Re: [PATCH blk-next 1/2] blk-mq-rdma: Delete not-used multi-queue RDMA map queue code
Quoted 22 lines · Christoph Hellwig
Quoted 14 lines · Sagi Grimberg
Quoted 6 lines
Well, why would they change it? The whole point of the infrastructure is that there is a single sane affinity setting for a given setup. Now that setting needed some refinement from the original series (e.g. the current series about only using housekeeping cpus if cpu isolation is in use). But allowing random users to modify affinity is just a receipe for a trainwreck.
Well allowing people to mangle irq affinity settings seem to be a hard requirement from the discussions in the past.
Quoted 3 lines
So I think we need to bring this back ASAP, as doing affinity right out of the box is an absolute requirement for sane performance without all the benchmarketing deep magic.
Quoted 6 lines
Well, it's hard to say that setting custom irq affinity settings is deemed non-useful to anyone and hence should be prevented. I'd expect that irq settings have a sane default that works and if someone wants to change it, it can but there should be no guarantees on optimal performance. But IIRC this had some dependencies on drivers and some more infrastructure to handle dynamic changes...
Quoted 3 lines
The problem is that people change random settings. We need to generalize it into a sane API (e.g. the housekeeping CPUs thing which totally makes sense).
I don't see many people jump on the bandwagon, someone should do it, but who will? I personally have no knowledge in that area to do anything meaningful.
Thanks
PATCH 2/2 RDMA/core: Delete not-implemented get_vector_affinity
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
There are no drivers that support .get_vector_affinity(), so delete it.
--- drivers/infiniband/core/device.c | 1 - include/rdma/ib_verbs.h | 23 ----------------------- 2 files changed, 24 deletions(-)
drivers/infiniband/core/device.c +0 -1
@@ -2619,7 +2619,6 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)SET_DEVICE_OP (dev_ops, get_link_layer);SET_DEVICE_OP (dev_ops, get_netdev);SET_DEVICE_OP (dev_ops, get_port_immutable); -SET_DEVICE_OP (dev_ops, get_vector_affinity);SET_DEVICE_OP (dev_ops, get_vf_config);SET_DEVICE_OP (dev_ops, get_vf_guid);SET_DEVICE_OP (dev_ops, get_vf_stats);
include/rdma/ib_verbs.h +0 -23
@@ -2319,8 +2319,6 @@ struct ib_device_ops {int (*modify_device)(struct ib_device *device,int device_modify_mask,struct ib_device_modify *device_modify);void (*get_dev_fw_str)(struct ib_device *device,char *str); -const struct cpumask *(*get_vector_affinity)(struct ib_device *ibdev, -int comp_vector);int (*query_port)(struct ib_device *device,u8 port_num,struct ib_port_attr *port_attr); int (*modify_port)(struct ib_device *device, u8 port_num,
@@ -4545,27 +4543,6 @@ static inline __be16 ib_lid_be16(u32 lid)return cpu_to_be16 ((u16 )lid); } -/** -* ib_get_vector_affinity - Get the affinity mappings of a given completion -* vector -* @device: the rdma device -* @comp_vector: index of completion vector -* -* Returns NULL on failure, otherwise a corresponding cpu map of the -* completion vector (returns all-cpus map if the device driver doesn't -* implement get_vector_affinity). -*/ -static inline const struct cpumask * -ib_get_vector_affinity (struct ib_device *device,int comp_vector) -{ -if (comp_vector <0 || comp_vector >= device->num_comp_vectors || - !device->ops.get_vector_affinity) -return NULL ; - -return device->ops.get_vector_affinity (device, comp_vector); - -} -/** * rdma_roce_rescan_device - Rescan all of the network devices in the system * and add their gids, as needed, to the relevant RoCE devices.
--
2.26.2
Leon Romanovsky
maintainerQuoted 5 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
There are no drivers that implement .get_vector_affinity(), so delete the RDMA function and simplify block code.
Quoted 7 lines
Thanks
P.S. Probably it should go through block tree.Leon Romanovsky (2):
blk-mq-rdma: Delete not-used multi-queue RDMA map queue code RDMA/core: Delete not-implementedget_vector_affinity
Jens, Keith
How can we progress here?
Thanks
Quoted 15 lines
block/Kconfig | 5 ---- block/Makefile | 1 - block/blk-mq-rdma.c | 44 -------------------------------- drivers/infiniband/core/device.c | 1 - drivers/nvme/host/rdma.c | 7 ++--- include/linux/blk-mq-rdma.h | 11 -------- include/rdma/ib_verbs.h | 23 ----------------- 7 files changed, 2 insertions(+), 90 deletions(-) delete mode 100644 block/blk-mq-rdma.c delete mode 100644 include/linux/blk-mq-rdma.h--
2.26.2
Jens Axboe
maintainerQuoted 10 lines · Leon Romanovsky
Quoted 9 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
There are no drivers that implement .get_vector_affinity(), so delete the RDMA function and simplify block code.
Thanks
P.S. Probably it should go through block tree.
Quoted 7 lines
Quoted 3 lines
Leon Romanovsky (2):
blk-mq-rdma: Delete not-used multi-queue RDMA map queue code RDMA/core: Delete not-implementedget_vector_affinityJens, Keith
How can we progress here?
I'd really like for the nvme side to sign off on this first.
Signature · 1 line
Jens Axboe