infiniband limit of 32 cards per system?
Hello,
I'm pretty unfamiliar with Infiniband, so apologies for the stupid question.
Is there a limit on how many IB devices a system might support?
My colleague points out the following enum in uverbs_main.c:
enum {
IB_UVERBS_MAJOR = 231,
IB_UVERBS_BASE_MINOR = 192,
IB_UVERBS_MAX_DEVICES = 32
};Experimentally, we've determined that on a system where we plugged in 40 IB cards, OFED only reports 32 cards are present.
If that enum is indeed the limiting factor, would someone mind explaining (or pointing me at TFM ;) why it's limited to 32 devices?
Thanks,
/ac
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines
My colleague points out the following enum in
uverbs_main.c:enum { IB_UVERBS_MAJOR = 231, IB_UVERBS_BASE_MINOR = 192, IB_UVERBS_MAX_DEVICES = 32 };Experimentally, we've determined that on a system where we plugged in 40 IB cards, OFED only reports 32 cards are present.
wow, 40 HCAs in one system !
Quoted 3 lines
If that enum is indeed the limiting factor, would someone mind explaining (or pointing me at TFM ;) why it's limited to 32 devices?
That dates back to when device #s had 8 bits for major and 8 bits for minor. We got one major assigned for IB, and had to split up the 256 minors that gave us among userspace verbs, management access, etc. And 32 seemed like a pretty reasonable limit for most uses.
Nowadays I guess we should look into expanding that to dynamic device numbers on overflow, assuming you do have a realistic situation where someone would want to use that many adapters per system.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
* Roland Dreier <[email protected]>:
Quoted 9 lines
Quoted 9 lines
My colleague points out the following enum in
uverbs_main.c:enum { IB_UVERBS_MAJOR = 231, IB_UVERBS_BASE_MINOR = 192, IB_UVERBS_MAX_DEVICES = 32 };
Quoted 4 lines
Quoted 2 lines
Experimentally, we've determined that on a system where we plugged in 40 IB cards, OFED only reports 32 cards are present.
wow, 40 HCAs in one system !
HP sell some pretty big systems. :)
Quoted 8 lines
Quoted 3 lines
If that enum is indeed the limiting factor, would someone mind explaining (or pointing me at TFM ;) why it's limited to 32 devices?
That dates back to when device #s had 8 bits for major and 8 bits for minor. We got one major assigned for IB, and had to split up the 256 minors that gave us among userspace verbs, management access, etc. And 32 seemed like a pretty reasonable limit for most uses.
Thanks for the explanation.
Quoted 3 lines
Nowadays I guess we should look into expanding that to dynamic device numbers on overflow, assuming you do have a realistic situation where someone would want to use that many adapters per system.
Think of a large scale-up ia64 box, possibly running some virtualization stack.
I'm guessing that it's not just a simple kernel fix though since OFED has to change too, right?
/ac
Quoted 3 lines
I'm guessing that it's not just a simple kernel fix though since OFED has to change too, right?
Dunno about OFED. Nothing sane is hard-coding major/minor numbers though -- so I think OFED should be OK, asuming there are no crazy scripts that bypass udev creating device nodes etc.
I don't think that it's _totally_ trivial in the kernel -- we do need to add some code in several places to allocate dynamic device numbers when we run out of the static allocation (probably best to keep the legacy device numbers for "small" < 32 adapter systems, since there may be really small systems with static hard-coded /dev etc).
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
* Roland Dreier <[email protected]>:
Quoted 7 lines
Quoted 3 lines
I'm guessing that it's not just a simple kernel fix though since OFED has to change too, right?
Dunno about OFED. Nothing sane is hard-coding major/minor numbers though -- so I think OFED should be OK, asuming there are no crazy scripts that bypass udev creating device nodes etc.
Ok.
Quoted 6 lines
I don't think that it's _totally_ trivial in the kernel -- we do need to add some code in several places to allocate dynamic device numbers when we run out of the static allocation (probably best to keep the legacy device numbers for "small" < 32 adapter systems, since there may be really small systems with static hard-coded /dev etc).
I take it this concern is what prevents us from simply increasing IB_UVERBS_MAX_DEVICES to 64 or something?
thanks,
/ac
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines
I take it this concern is what prevents us from simply increasing
IB_UVERBS_MAX_DEVICESto 64 or something?
Yes, I think then the device numbers will overlap.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
* Roland Dreier <[email protected]>:
Quoted 10 lines
Quoted 4 lines
If that enum is indeed the limiting factor, would someone mind explaining (or pointing me at TFM ;) why it's limited to 32 devices?
That dates back to when device #s had 8 bits for major and 8 bits for minor. We got one major assigned for IB, and had to split up the 256 minors that gave us among userspace verbs, management access, etc. And 32 seemed like a pretty reasonable limit for most uses.
Hm...
IB_UMAD_MAX_PORTS = 64,
IB_UMAD_MAJOR = 231,
IB_UMAD_MINOR_BASE = 0
register_chrdev_region(base_dev, IB_UMAD_MAX_PORTS * 2, "infiniband_mad");
IB_UVERBS_MAJOR = 231,
IB_UVERBS_BASE_MINOR = 192,
IB_UVERBS_MAX_DEVICES = 32
register_chrdev_region(IB_UVERBS_BASE_DEV, IB_UVERBS_MAX_DEVICES, "infiniband_verbs");
IB_UCM_MAJOR = 231,
IB_UCM_BASE_MINOR = 224,
IB_UCM_MAX_DEVICES = 32
register_chrdev_region(IB_UCM_BASE_DEV, IB_UCM_MAX_DEVICES, "infiniband_cm");It looks like we have a hole from [128, 192).
Would it be something as simple as this?
drivers/infiniband/core/uverbs_main.c +2 -2
@@ -57,8 +57,8 @@ MODULE_LICENSE("Dual BSD/GPL");enum {IB_UVERBS_MAJOR =231 , -IB_UVERBS_BASE_MINOR =192 , -IB_UVERBS_MAX_DEVICES =32 +IB_UVERBS_BASE_MINOR =128 , +IB_UVERBS_MAX_DEVICES =64 };#define IB_UVERBS_BASE_DEV MKDEV(IB_UVERBS_MAJOR, IB_UVERBS_BASE_MINOR)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
drivers/infiniband/core/uverbs_main.c MODULE_LICENSE("Dual BSD/GPL"); · line 61
Quoted 9 lines
It looks like we have a hole from [128, 192).
Would it be something as simple as this?
drivers/infiniband/core/uverbs_main.c +2
-2-IB_UVERBS_BASE_MINOR =192 , -IB_UVERBS_MAX_DEVICES =32 +IB_UVERBS_BASE_MINOR =128 , +IB_UVERBS_MAX_DEVICES =64
I don't think this is a good idea for two reasons:
- It doesn't take into account the fact that the infiniband_mad and infiniband_cm drivers will take up more minors if more devices appear (in the best case, you would only be able to run opensm on the first 32 HCAs or something like that).
- It changes the minor of the first uverbs device, so something like a system with hardcoded static /dev would break in a mysterious way.
I think unfortunately we have to extend the device # assignment so the first 32 HCAs get the same minors they would have and then overflow into some dynamic region.
- R.