Increase number of char devices in RDMA
cover Increase number of char devices in RDMA
Hi,
The following two patches from Huy with a nice diffstat (more removed than added) increases upto 1024 the maximum number of available in RDMA stack char devices, so it will be possible to use in large SR-IOV instances.
Thanks
Huy Nguyen (2):
IB/core: Remove the locking for character device bitmap IB/core: Refactor allocation code to increase number of char device
drivers/infiniband/core/core_priv.h | 3 + drivers/infiniband/core/ucm.c | 73 +++++++++-------------- drivers/infiniband/core/user_mad.c | 108 +++++++++++++++------------------- drivers/infiniband/core/uverbs_main.c | 95 +++++++++++------------------- 4 files changed, 113 insertions(+), 166 deletions(-)
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20180108101538.16451-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH 1/2 IB/core: Remove the locking for character device bitmap
From: Huy Nguyen <huyn-VPRAkNaXOzVWk0Htik3J/[email protected]>
Remove the locks that protect character device bitmap of uverbs, umad and issm.
The character device bit maps are accessed in "client->add" and "client->remove" calls from ib_register_device and ib_unregister_device respectively. These calls are already protected by the "device_mutex" mutex. Thus, the spinlocks are not needed.
--- drivers/infiniband/core/user_mad.c | 5 ----- drivers/infiniband/core/uverbs_main.c | 5 ----- 2 files changed, 10 deletions(-)
drivers/infiniband/core/user_mad.c +0 -5
@@ -129,7 +129,6 @@ static struct class *umad_class;static const dev_t base_dev =MKDEV (IB_UMAD_MAJOR ,IB_UMAD_MINOR_BASE ); -static DEFINE_SPINLOCK (port_lock); static DECLARE_BITMAP(dev_map, IB_UMAD_MAX_PORTS);static void ib_umad_add_one (struct ib_device *device);
@@ -1172,15 +1171,12 @@ static int ib_umad_init_port(struct ib_device *device, int port_num,int devnum;dev_t base; -spin_lock (&port_lock); devnum =find_first_zero_bit (dev_map,IB_UMAD_MAX_PORTS );if (devnum >=IB_UMAD_MAX_PORTS ) { -spin_unlock (&port_lock); devnum =find_overflow_devnum (device);if (devnum <0 )return -1 ; -spin_lock (&port_lock); port->dev_num = devnum +IB_UMAD_MAX_PORTS ; base = devnum + overflow_maj;set_bit (devnum, overflow_map);
@@ -1189,7 +1185,6 @@ static int ib_umad_init_port(struct ib_device *device, int port_num, base = devnum + base_dev;set_bit (devnum, dev_map); } -spin_unlock (&port_lock); port->ib_dev = device; port->port_num = port_num;
drivers/infiniband/core/uverbs_main.c +0 -5
@@ -69,7 +69,6 @@ enum {static struct class *uverbs_class; -static DEFINE_SPINLOCK (map_lock); static DECLARE_BITMAP(dev_map, IB_UVERBS_MAX_DEVICES); static ssize_t (*uverbs_cmd_table[])(struct ib_uverbs_file *file,
@@ -1062,15 +1061,12 @@ static void ib_uverbs_add_one(struct ib_device *device)INIT_LIST_HEAD (&uverbs_dev->uverbs_file_list);INIT_LIST_HEAD (&uverbs_dev->uverbs_events_file_list); -spin_lock (&map_lock); devnum =find_first_zero_bit (dev_map,IB_UVERBS_MAX_DEVICES );if (devnum >=IB_UVERBS_MAX_DEVICES ) { -spin_unlock (&map_lock); devnum =find_overflow_devnum ();if (devnum <0 ) goto err; -spin_lock (&map_lock); uverbs_dev->devnum = devnum +IB_UVERBS_MAX_DEVICES ; base = devnum + overflow_maj;set_bit (devnum, overflow_map);
@@ -1079,7 +1075,6 @@ static void ib_uverbs_add_one(struct ib_device *device) base = devnum +IB_UVERBS_BASE_DEV ;set_bit (devnum, dev_map); } -spin_unlock (&map_lock);rcu_assign_pointer (uverbs_dev->ib_dev, device); uverbs_dev->num_comp_vectors = device->num_comp_vectors;
Signature · 6 lines
2.15.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 2/2 IB/core: Refactor allocation code to increase number of char device
In reply to a message not in the archiveFrom: Huy Nguyen <huyn-VPRAkNaXOzVWk0Htik3J/[email protected]>
There is a need to increase number of possible char devices to support large number of SR-IOV instances. The current limit is in range of 64-128 devices/ports and this refactoring patch increases it to support upto 1024.
The patch performs the following steps to refactor the code:
1. Removes the split bitmap for fixed and overflow dev numbers.
2. Pre-allocates the non-legacy major number range during driver initialization for the simplicity.
3. Add new define (RDMA_MAX_PORTS) that is shared between all drivers for the future expansion of non-legacy minor number range. This is the maximum total number of ports on all struct ib_devices.
4. Set RDMA_MAX_PORTS to 1024.
--- drivers/infiniband/core/core_priv.h | 3 + drivers/infiniband/core/ucm.c | 73 +++++++++--------------- drivers/infiniband/core/user_mad.c | 103 ++++++++++++++++------------------ drivers/infiniband/core/uverbs_main.c | 90 +++++++++++------------------ 4 files changed, 113 insertions(+), 156 deletions(-)
drivers/infiniband/core/core_priv.h +3 -0
@@ -43,6 +43,9 @@#include <rdma/restrack.h> #include "mad_priv.h" +/* Total number of ports combined across all struct ib_devices's */ +#define RDMA_MAX_PORTS 1024 +struct pkey_index_qp_list {struct list_head pkey_index_list;u16 pkey_index;
drivers/infiniband/core/ucm.c +28 -45
136 lines not on this page · raw message
drivers/infiniband/core/user_mad.c +48 -55
189 lines not on this page · raw message
drivers/infiniband/core/uverbs_main.c +34 -56
148 lines not on this page · raw message
Signature · 6 lines
2.15.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe
AppliedQuoted 11 lines · Leon Romanovsky
Hi,
The following two patches from Huy with a nice diffstat (more removed than added) increases upto 1024 the maximum number of available in RDMA stack char devices, so it will be possible to use in large SR-IOV instances.
Thanks
Huy Nguyen (2):
IB/core: Remove the locking for character device bitmap IB/core: Refactor allocation code to increase number of char device
Applied to for-next
Thanks,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html