IB/core: Add option to limit user mad receive list
PATCH 1/1 IB/core: Add option to limit user mad receive list
From: Michael Guralnik <[email protected]>
ib_umad is keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.
As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.
Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.
--- .../ABI/stable/sysfs-class-infiniband | 12 ++++ drivers/infiniband/core/user_mad.c | 63 ++++++++++++++++++- 2 files changed, 72 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-class-infiniband +12 -0
@@ -275,6 +275,18 @@ Description: =============== =========================================== +What: /sys/class/infiniband_mad/umad<N>/max_recv_list_size +Date: January, 2024 +KernelVersion: v6.9 +Contact: [email protected] +Description: + (RW) Limit the size of the list of MAD packets waiting to be + read by the user-space agent. + The default value is 0, which means unlimited list size. + Packets received when the list is full will be silently + dropped. + + What: /sys/class/infiniband_verbs/abi_version Date: Sep, 2005 KernelVersion: v2.6.14
drivers/infiniband/core/user_mad.c +60 -3
123 lines not on this page · raw message
Signature · 1 line
2.44.0
Jason Gunthorpe
maintainerQuoted 30 lines · Leon Romanovsky
From: Michael Guralnik <[email protected]>
ib_umadis keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.
Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.
Signed-off-by Michael Guralnik <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- .../ABI/stable/sysfs-class-infiniband | 12 ++++ drivers/infiniband/core/user_mad.c | 63 ++++++++++++++++++- 2 files changed, 72 insertions(+), 3 deletions(-)Documentation/ABI/stable/sysfs-class-infiniband +0
-0@@ -275,6 +275,18 @@ Description: =============== ===========================================
Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287
Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10
-0+What: /sys/class/infiniband_mad/umad<N>/max_recv_list_size +Date: January, 2024 +KernelVersion: v6.9 +Contact: [email protected] +Description: + (RW) Limit the size of the list of MAD packets waiting to be + read by the user-space agent. + The default value is 0, which means unlimited list size. + Packets received when the list is full will be silently + dropped.
I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.
Why do we need a tunable? Can we just set it to something large and be done with it?
Jason
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Leon Romanovsky
From: Michael Guralnik <[email protected]>
ib_umadis keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.
Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.
Signed-off-by Michael Guralnik <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- .../ABI/stable/sysfs-class-infiniband | 12 ++++ drivers/infiniband/core/user_mad.c | 63 ++++++++++++++++++- 2 files changed, 72 insertions(+), 3 deletions(-)Documentation/ABI/stable/sysfs-class-infiniband +0
-0@@ -275,6 +275,18 @@ Description: =============== ===========================================Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287
Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10
-0+What: /sys/class/infiniband_mad/umad<N>/max_recv_list_size +Date: January, 2024 +KernelVersion: v6.9 +Contact: [email protected] +Description: + (RW) Limit the size of the list of MAD packets waiting to be + read by the user-space agent. + The default value is 0, which means unlimited list size. + Packets received when the list is full will be silently + dropped.I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.
Quoted 2 lines
Why do we need a tunable? Can we just set it to something large and be done with it?
I don't know which value to set to be large enough from one side and small enough to do not cause to OOM while host gets MAD packets.
Thanks
Quoted 3 lines
Jason
Jason Gunthorpe
maintainerQuoted 46 lines · Leon Romanovsky
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Leon Romanovsky
From: Michael Guralnik <[email protected]>
ib_umadis keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.
Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.
Signed-off-by Michael Guralnik <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>--- .../ABI/stable/sysfs-class-infiniband | 12 ++++ drivers/infiniband/core/user_mad.c | 63 ++++++++++++++++++- 2 files changed, 72 insertions(+), 3 deletions(-)Documentation/ABI/stable/sysfs-class-infiniband +0
-0@@ -275,6 +275,18 @@ Description: =============== ===========================================Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287
Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10
-0+What: /sys/class/infiniband_mad/umad<N>/max_recv_list_size +Date: January, 2024 +KernelVersion: v6.9 +Contact: [email protected] +Description: + (RW) Limit the size of the list of MAD packets waiting to be + read by the user-space agent. + The default value is 0, which means unlimited list size. + Packets received when the list is full will be silently + dropped.I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.
Quoted 5 lines
Quoted 2 lines
Why do we need a tunable? Can we just set it to something large and be done with it?
I don't know which value to set to be large enough from one side and small enough to do not cause to OOM while host gets MAD packets.
I'd say something like 32M worth of packets is probably good from both directions? Or 1% of system memory, or something like that.
Jason