Skip to contentSkip to conversation

IB/core: Add option to limit user mad receive list

PATCH In review Leon Romanovsky 2 participants 4 messages Apr 2, 2024 – Apr 5, 2024 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 IB/core: Add option to limit user mad receive list

From: Michael Guralnik <[email protected]>

ib_umad is keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.

As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.

Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.

---
 .../ABI/stable/sysfs-class-infiniband         | 12 ++++
 drivers/infiniband/core/user_mad.c            | 63 ++++++++++++++++++-
 2 files changed, 72 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-class-infiniband +12 -0
index 694f23a03a28..0ea9d590ab0e 100644
@@ -275,6 +275,18 @@ Description:
 		=============== ===========================================
 
 
+What:		/sys/class/infiniband_mad/umad<N>/max_recv_list_size
+Date:		January, 2024
+KernelVersion:	v6.9
+Contact:	[email protected]
+Description:
+		(RW) Limit the size of the list of MAD packets waiting to be
+		     read by the user-space agent.
+		     The default value is 0, which means unlimited list size.
+		     Packets received when the list is full will be silently
+		     dropped.
+
+
 What:		/sys/class/infiniband_verbs/abi_version
 Date:		Sep, 2005
 KernelVersion:	v2.6.14
drivers/infiniband/core/user_mad.c +60 -3

123 lines not on this page · raw message

Signature · 1 line
2.44.0
maintainer
#1 lore raw
Quoted 30 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

ib_umad is keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.

As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.

Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.

---
 .../ABI/stable/sysfs-class-infiniband         | 12 ++++
 drivers/infiniband/core/user_mad.c            | 63 ++++++++++++++++++-
 2 files changed, 72 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-class-infiniband +0 -0
index 694f23a03a28..0ea9d590ab0e 100644
@@ -275,6 +275,18 @@ Description:
 		=============== ===========================================
 
 

Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287

Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10 -0
+What:		/sys/class/infiniband_mad/umad<N>/max_recv_list_size
+Date:		January, 2024
+KernelVersion:	v6.9
+Contact:	[email protected]
+Description:
+		(RW) Limit the size of the list of MAD packets waiting to be
+		     read by the user-space agent.
+		     The default value is 0, which means unlimited list size.
+		     Packets received when the list is full will be silently
+		     dropped.

I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.

Why do we need a tunable? Can we just set it to something large and be done with it?

Jason

#2 lore raw
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

ib_umad is keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.

As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.

Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.

---
 .../ABI/stable/sysfs-class-infiniband         | 12 ++++
 drivers/infiniband/core/user_mad.c            | 63 ++++++++++++++++++-
 2 files changed, 72 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-class-infiniband +0 -0
index 694f23a03a28..0ea9d590ab0e 100644
@@ -275,6 +275,18 @@ Description:
 		=============== ===========================================
 
 

Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287

Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10 -0
+What:		/sys/class/infiniband_mad/umad<N>/max_recv_list_size
+Date:		January, 2024
+KernelVersion:	v6.9
+Contact:	[email protected]
+Description:
+		(RW) Limit the size of the list of MAD packets waiting to be
+		     read by the user-space agent.
+		     The default value is 0, which means unlimited list size.
+		     Packets received when the list is full will be silently
+		     dropped.

I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.

Quoted 2 lines

Why do we need a tunable? Can we just set it to something large and be done with it?

I don't know which value to set to be large enough from one side and small enough to do not cause to OOM while host gets MAD packets.

Thanks

Quoted 3 lines

Jason

maintainer
#3 lore raw
Quoted 46 lines · Leon Romanovsky
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

ib_umad is keeping the received MAD packets in a list that is not limited in size. As the extraction of packets from this list is done from user-space application, there is no way to guarantee the extraction rate to be faster than the rate of incoming packets. This can cause to the list to grow uncontrollably.

As a solution, let's add new ysfs control knob for the users to limit the number of received MAD packets in the list.

Packets received when the list is full would be dropped. Sent packets that are queued on the receive list for whatever reason, like timed out sends, are not dropped even when the list is full.

---
 .../ABI/stable/sysfs-class-infiniband         | 12 ++++
 drivers/infiniband/core/user_mad.c            | 63 ++++++++++++++++++-
 2 files changed, 72 insertions(+), 3 deletions(-)
Documentation/ABI/stable/sysfs-class-infiniband +0 -0
index 694f23a03a28..0ea9d590ab0e 100644
@@ -275,6 +275,18 @@ Description:
 		=============== ===========================================
 
 

Documentation/ABI/stable/sysfs-class-infiniband Description: · line 287

Quoted 10 lines
Documentation/ABI/stable/sysfs-class-infiniband +10 -0
+What:		/sys/class/infiniband_mad/umad<N>/max_recv_list_size
+Date:		January, 2024
+KernelVersion:	v6.9
+Contact:	[email protected]
+Description:
+		(RW) Limit the size of the list of MAD packets waiting to be
+		     read by the user-space agent.
+		     The default value is 0, which means unlimited list size.
+		     Packets received when the list is full will be silently
+		     dropped.

I'm really not keen on this as a tunable, when we get to future designs it may be hard to retain this specific behavior.

Quoted 5 lines
Quoted 2 lines

Why do we need a tunable? Can we just set it to something large and be done with it?

I don't know which value to set to be large enough from one side and small enough to do not cause to OOM while host gets MAD packets.

I'd say something like 32M worth of packets is probably good from both directions? Or 1% of system memory, or something like that.

Jason