Skip to contentSkip to conversation

sctp: fully support memory accounting

PATCH Superseded Xin Long 5 participants 17 messages Mar 31, 2019 – Apr 5, 2019

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Xin Long

author maintainer
#0 lore raw

cover sctp: fully support memory accounting

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Signature · 1 line
2.1.0
author maintainer
#1 lore raw

PATCH 1/2 sctp: implement memory accounting on tx path

Now when sending packets, sk_mem_charge() and sk_mem_uncharge() have been used to set sk_forward_alloc. We just need to call sk_wmem_schedule() to check if the allocated should be raised, and call sk_mem_reclaim() to check if the allocated should be reduced when it's under memory pressure.

If sk_wmem_schedule() returns false, which means no memory is allowed to allocate, it will block and wait for memory to become available.

Note different from tcp, sctp wait_for_buf happens before allocating any skb, so memory accounting check is done with the whole msg_len before it too.

---
 net/sctp/socket.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
net/sctp/socket.c +8 -2
index 6140471..06c6f4a 100644
@@ -1913,7 +1913,10 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
 	if (sctp_wspace(asoc) < (int)msg_len)
 		sctp_prsctp_prune(asoc, sinfo, msg_len - sctp_wspace(asoc));
 
-	if (sctp_wspace(asoc) <= 0) {
+	if (sk_under_memory_pressure(sk))
+		sk_mem_reclaim(sk);
+
+	if (sctp_wspace(asoc) <= 0 || !sk_wmem_schedule(sk, msg_len)) {
 		timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
 		err = sctp_wait_for_sndbuf(asoc, &timeo, msg_len);
 		if (err)
@@ -8891,7 +8894,10 @@ static int sctp_wait_for_sndbuf(struct sctp_association *asoc, long *timeo_p,
 			goto do_error;
 		if (signal_pending(current))
 			goto do_interrupted;
-		if ((int)msg_len <= sctp_wspace(asoc))
+		if (sk_under_memory_pressure(sk))
+			sk_mem_reclaim(sk);
+		if ((int)msg_len <= sctp_wspace(asoc) &&
+		    sk_wmem_schedule(sk, msg_len))
 			break;
 
 		/* Let another process have a go.  Since we are going
Signature · 1 line
2.1.0
author maintainer
#2 lore raw

PATCH 2/2 sctp: implement memory accounting on rx path

sk_forward_alloc's updating is also done on rx path, but to be consistent we change to use sk_mem_charge() in sctp_skb_set_owner_r().

In sctp_eat_data(), it's not enough to check sctp_memory_pressure only, which doesn't work for mem_cgroup_sockets_enabled, so we change to use sk_under_memory_pressure().

When it's under memory pressure, sk_mem_reclaim() and sk_rmem_schedule() should be called on both RENEGE or CHUNK DELIVERY path exit the memory pressure status as soon as possible.

Note that sk_rmem_schedule() is using datalen to make things easy there.

---
 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 4 files changed, 15 insertions(+), 15 deletions(-)
include/net/sctp/sctp.h +1 -1
index 1d13ec3..eefdfa5 100644
@@ -421,7 +421,7 @@ static inline void sctp_skb_set_owner_r(struct sk_buff *skb, struct sock *sk)
 	/*
 	 * This mimics the behavior of skb_set_owner_r
 	 */
-	sk->sk_forward_alloc -= event->rmem_len;
+	sk_mem_charge(sk, event->rmem_len);
 }
 
 /* Tests if the list has one and only one entry. */
net/sctp/sm_statefuns.c +4 -2
index c9ae340..7dfc34b 100644
@@ -6412,13 +6412,15 @@ static int sctp_eat_data(const struct sctp_association *asoc,
 	 * in sctp_ulpevent_make_rcvmsg will drop the frame if we grow our
 	 * memory usage too much
 	 */
-	if (*sk->sk_prot_creator->memory_pressure) {
+	if (sk_under_memory_pressure(sk)) {
 		if (sctp_tsnmap_has_gap(map) &&
 		    (sctp_tsnmap_get_ctsn(map) + 1) = tsn) {
 			pr_debug("%s: under pressure, reneging for tsn:%u\n",
 				 __func__, tsn);
 			deliver = SCTP_CMD_RENEGE;
-		 }
+		} else {
+			sk_mem_reclaim(sk);
+		}
 	}
 
 	/*
net/sctp/ulpevent.c +8 -11
index 8cb7d98..c2a7478 100644
@@ -634,8 +634,9 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 						gfp_t gfp)
 {
 	struct sctp_ulpevent *event = NULL;
-	struct sk_buff *skb;
-	size_t padding, len;
+	struct sk_buff *skb = chunk->skb;
+	struct sock *sk = asoc->base.sk;
+	size_t padding, datalen;
 	int rx_count;
 
 	/*
@@ -646,15 +647,12 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 	if (asoc->ep->rcvbuf_policy)
 		rx_count = atomic_read(&asoc->rmem_alloc);
 	else
-		rx_count = atomic_read(&asoc->base.sk->sk_rmem_alloc);
+		rx_count = atomic_read(&sk->sk_rmem_alloc);
 
-	if (rx_count >= asoc->base.sk->sk_rcvbuf) {
+	datalen = ntohs(chunk->chunk_hdr->length);
 
-		if ((asoc->base.sk->sk_userlocks & SOCK_RCVBUF_LOCK) ||
-		    (!sk_rmem_schedule(asoc->base.sk, chunk->skb,
-				       chunk->skb->truesize)))
-			goto fail;
-	}
+	if (rx_count >= sk->sk_rcvbuf || !sk_rmem_schedule(sk, skb, datalen))
+		goto fail;
 
 	/* Clone the original skb, sharing the data.  */
 	skb = skb_clone(chunk->skb, gfp);
@@ -681,8 +679,7 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 	 * The sender should never pad with more than 3 bytes.  The receiver
 	 * MUST ignore the padding bytes.
 	 */
-	len = ntohs(chunk->chunk_hdr->length);
-	padding = SCTP_PAD4(len) - len;
+	padding = SCTP_PAD4(datalen) - datalen;
 
 	/* Fixup cloned skb with just this chunks data.  */
 	skb_trim(skb, chunk->chunk_end - padding - skb->data);
net/sctp/ulpqueue.c +2 -1
index 5dde921..770ff1f 100644
@@ -1106,7 +1106,8 @@ void sctp_ulpq_renege(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk,
 			freed += sctp_ulpq_renege_frags(ulpq, needed - freed);
 	}
 	/* If able to free enough room, accept this chunk. */
-	if (freed >= needed) {
+	if (sk_rmem_schedule(asoc->base.sk, chunk->skb, needed) &&
+	    freed >= needed) {
 		int retval = sctp_ulpq_tail_data(ulpq, chunk, gfp);
 		/*
 		 * Enter partial delivery if chunk has not been
Signature · 1 line
2.1.0
author maintainer
#5 lore raw
In reply to Xin Long

Simple tests are as attached.

Quoted 42 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Quoted 3 lines

--
2.1.0

#!/bin/sh

# TOPO: local (10.73.131.158) <--> peer (10.73.131.202) [hosts not netns]

peer_run()

{
	ssh [email protected] "$@"
	# echo " - peer run: $@"
	# echo " - print any key to continue after it's done"
	# read
}

modprobe sctp

echo "sctp sysctl memory control rmem:"
echo "--------------------------------"

echo "10 20 50 rmem limit ->"
pkill sctp_test 2> /dev/null
sysctl -w net.sctp.sctp_mem="10 20 50"
sctp_test -H 10.73.131.158 -P 1234 -l > /dev/null 2>&1 &
sleep 3
peer_run "time sctp_test -H 10.73.131.202 -P 8000 -h 10.73.131.158 -p 1234 -s -c 5 > /dev/null 2>&1" echo ""

echo "20 50 100 rmem limit ->"
pkill sctp_test 2> /dev/null
sysctl -w net.sctp.sctp_mem="20 50 100"
sctp_test -H 10.73.131.158 -P 1234 -l > /dev/null 2>&1 &
sleep 3
peer_run "time sctp_test -H 10.73.131.202 -P 8000 -h 10.73.131.158 -p 1234 -s -c 5 > /dev/null 2>&1" echo ""

echo "sctp sysctl memory control wmem:"
echo "--------------------------------"

peer_run "sctp_test -H 10.73.131.202 -P 8000 -l > /dev/null 2>&1 &"
tc qdisc add dev eth0 root netem delay 50ms
echo "10 20 50 wmem limit ->"
sysctl -w net.sctp.sctp_mem="5 10 20"
time sctp_test -h 10.73.131.202 -p 8000 -H 10.73.131.158 -P 1234 -s -c 5 > /dev/null 2>&1 echo ""

echo "10 50 100 wmem limit ->"
sysctl -w net.sctp.sctp_mem="10 50 100 ->"
time sctp_test -h 10.73.131.202 -p 8000 -H 10.73.131.158 -P 1234 -s -c 5 > /dev/null 2>&1 echo ""
tc qdisc delete dev eth0 root
peer_run "pkill sctp_test 2>/dev/null"

#!/bin/sh

# TOPO: 127.0.0.1 lo

modprobe sctp
pkill sctp_test 2> /dev/null
sctp_test -H 127.0.0.1 -P 1234 -l > /dev/null 2>&1 & sleep 3
mkdir /sys/fs/cgroup/memory/sctp_mem > /dev/null 2>&1;

echo "sctp_wmem testing:"
echo "------------------"

echo "$((8<<14)) bytes for wmem limit ->"
echo $((8<<14)) >/sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

{
	echo $BASHPID > /sys/fs/cgroup/memory/sctp_mem/cgroup.procs
	time sctp_test -H 127.0.0.1 -P 8000 -h 127.0.0.1 -p 1234 -s -c 5  > /dev/null 2>&1
	echo ""
} &

wait $!

echo "$((8<<16)) bytes for wmem limit ->"
echo $((8<<16)) >/sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

{
	echo $BASHPID > /sys/fs/cgroup/memory/sctp_mem/cgroup.procs
	time sctp_test -H 127.0.0.1 -P 8000 -h 127.0.0.1 -p 1234 -s -c 5  > /dev/null 2>&1
	echo ""
} &

wait $!

echo "sctp_rmem testing:"
echo "------------------"

echo "$((8<<13)) bytes for rmem limit ->"
pkill sctp_test 2> /dev/null
echo $((8<<13)) >/sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

{
	echo $BASHPID > /sys/fs/cgroup/memory/sctp_mem/cgroup.procs
	sctp_test -H 127.0.0.1 -P 1234 -l > /dev/null 2>&1
} &

sleep 3
time sctp_test -H 127.0.0.1 -P 8000 -h 127.0.0.1 -p 1234 -s -c 5 > /dev/null 2>&1 echo ""

echo "$((8<<14)) bytes for rmem limit ->"
pkill sctp_test 2> /dev/null
echo $((8<<14)) >/sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

{
	echo $BASHPID > /sys/fs/cgroup/memory/sctp_mem/cgroup.procs
	sctp_test -H 127.0.0.1 -P 1234 -l > /dev/null 2>&1
} &

sleep 3
time sctp_test -H 127.0.0.1 -P 8000 -h 127.0.0.1 -p 1234 -s -c 5 > /dev/null 2>&1 echo ""

echo "sctp_wmem_and_rmem testing:"
echo "---------------------------"

echo "$((8<<14)) bytes for wmem AND rmem limit ->"
pkill sctp_test 2> /dev/null
echo $((8<<14)) >/sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

{
	echo $BASHPID > /sys/fs/cgroup/memory/sctp_mem/cgroup.procs
	sctp_test -H 127.0.0.1 -P 1234 -l > /dev/null 2>&1 &
	sleep 3
	time sctp_test -H 127.0.0.1 -P 8000 -h 127.0.0.1 -p 1234 -s -c 5  > /dev/null 2>&1
	echo ""
} &

wait

maintainer
#6 lore raw

From: Xin Long <[email protected]>
Date: Sun, 31 Mar 2019 16:56:40 +0800

Quoted 1 line

Simple tests are as attached.

Are these integratable into selftests somehow?

Tested-by on the v1 series
#10 lore raw
In reply to Xin Long
Quoted 23 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"
Quoted 8 lines

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

I have tested this series with a tool which creates lot of SCTP sockets and writes into them, without never reading. Before the patch I was able to escape the cgroup limit and fill the system memory, and the OOM killer killed random processes:

[  317.348911] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/system.slice/[email protected],task=dhclient,pid8,uid=0
[  317.349084] Out of memory: Killed process 188 (dhclient)
total-vm:9484kB, anon-rss:1280kB, file-rss:1424kB, shmem-rss:0kB
[  317.743943] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/system.slice/systemd-journald.service,task=systemd-journal,pid…,uid=0
[  317.744093] Out of memory: Killed process 85 (systemd-journal)
total-vm:24592kB, anon-rss:1024kB, file-rss:1112kB, shmem-rss:652kB
[  317.921049] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/system.slice/cron.service,task=cron,pid"2,uid=0
[  317.921209] Out of memory: Killed process 222 (cron)

total-vm:8692kB, anon-rss:276kB, file-rss:1540kB, shmem-rss:0kB

Now the OOM killer behaves correctly and always kills the processes in the right cgroup:

[  512.100054] Tasks state (memory values in pages):
[  512.100122] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes
swapents oom_score_adj name
[  512.100256] [    838]     0   838      550      184    36864
0             0 sctprank
[  512.100452] [    841]     0   841      550       18    36864
0             0 sctprank
[  512.100573] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),oom_memcg=/sctp,task_memcg=/sctp,task=sctprank,pidƒ8,uid=0
[  512.100700] Memory cgroup out of memory: Killed process 838
(sctprank) total-vm:2200kB, anon-rss:64kB, file-rss:672kB,
shmem-rss:0kB
[  512.100899] oom_reaper: reaped process 838 (sctprank), now

anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Series ACK.

--
Matteo Croce
per aspera ad upstream

#11 lore raw
In reply to Xin Long
Quoted 2 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

Please give me till tomorrow to review this one.

Thanks,
Marcelo

maintainer
#12 lore raw

From: Marcelo Ricardo Leitner <[email protected]>
Date: Sun, 31 Mar 2019 17:10:48 -0300

Quoted 5 lines · Xin Long
Quoted 2 lines

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

Please give me till tomorrow to review this one.

No problem.

author maintainer
#13 lore raw
In reply to Xin Long
Quoted 42 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Quoted 3 lines

--
2.1.0

#14 lore raw
In reply to Xin Long
Quoted 34 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

Quoted 11 lines
 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Signature · 1 line
2.1.0

I don't have a problem with either of these patches in terms of altering memory accounting, but SCTP has the notion of accounting buffers based on either sockets space or association space (based on the sndbuf_policy and rcvbuf_policy sysctls). This patch eliminates them. I don't see this patch addressing either the removal of that functionality (as the proposed accounting scheme renders those sysctls useless and ignored, which may cause regressions in some environments), nor does it address the possibiliy of one association starving others on the same socket when they share the same socket level accounting. I think you need to look how to address that (either by re-adding the ability to account in either case based on the sysctls, or deprecating eliminating the sysctls and addressing the starvation issue.

Best
Neil

#15 lore raw
Quoted 46 lines · Neil Horman
Quoted 34 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

Quoted 11 lines
 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Signature · 1 line
2.1.0
Quoted 11 lines

I don't have a problem with either of these patches in terms of altering memory accounting, but SCTP has the notion of accounting buffers based on either sockets space or association space (based on the sndbuf_policy and rcvbuf_policy sysctls). This patch eliminates them. I don't see this patch addressing either the removal of that functionality (as the proposed accounting scheme renders those sysctls useless and ignored, which may cause regressions in some environments), nor does it address the possibiliy of one association starving others on the same socket when they share the same socket level accounting. I think you need to look how to address that (either by re-adding the ability to account in either case based on the sysctls, or deprecating eliminating the sysctls and addressing the starvation issue.

That's not how I'm reading these. All original conditions are still there while these patches are adding a couple of restrictions more. What that means is that they are adding a ceiling to it, even if the limits are per socket or per assoc. Considering the idea of the cgroup limit being added here, it makes sense to me. If the cgroup is configured to allow at most X MB, it doesn't matter how that is allocated. That's a sysadmin task then, to adjust the other sysctls (net.sctp.sctp_mem & cia) and balance the usage, be it per socket or per asoc.

Cheers,
Marcelo

Acked-by
#16 lore raw
Quoted 59 lines · Marcelo Ricardo Leitner
Quoted 46 lines · Neil Horman
Quoted 34 lines · Xin Long

sctp memory accounting is added in this patchset by using these kernel APIs on send side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_wmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

and these on receive side:

- sk_mem_charge()
- sk_mem_uncharge()
- sk_rmem_schedule()
- sk_under_memory_pressure()
- sk_mem_reclaim()

With sctp memory accounting, we can limit the memory allocation by either sysctl:

  # sysctl -w net.sctp.sctp_mem="10 20 50"

or cgroup:

  # echo $((8<<14)) > \
    /sys/fs/cgroup/memory/sctp_mem/memory.kmem.tcp.limit_in_bytes

When the socket is under memory pressure, the send side will block and wait, while the receive side will renege or drop.

Xin Long (2):
sctp: implement memory accounting on tx path
sctp: implement memory accounting on rx path

Quoted 11 lines
 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/socket.c       | 10 ++++++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 5 files changed, 23 insertions(+), 17 deletions(-)
Signature · 1 line
2.1.0
Quoted 11 lines

I don't have a problem with either of these patches in terms of altering memory accounting, but SCTP has the notion of accounting buffers based on either sockets space or association space (based on the sndbuf_policy and rcvbuf_policy sysctls). This patch eliminates them. I don't see this patch addressing either the removal of that functionality (as the proposed accounting scheme renders those sysctls useless and ignored, which may cause regressions in some environments), nor does it address the possibiliy of one association starving others on the same socket when they share the same socket level accounting. I think you need to look how to address that (either by re-adding the ability to account in either case based on the sysctls, or deprecating eliminating the sysctls and addressing the starvation issue.

Quoted 10 lines

That's not how I'm reading these. All original conditions are still there while these patches are adding a couple of restrictions more. What that means is that they are adding a ceiling to it, even if the limits are per socket or per assoc. Considering the idea of the cgroup limit being added here, it makes sense to me. If the cgroup is configured to allow at most X MB, it doesn't matter how that is allocated. That's a sysadmin task then, to adjust the other sysctls (net.sctp.sctp_mem & cia) and balance the usage, be it per socket or per asoc.

You're right, I had the sense on the conditional backwards in my head. My bad

Series

maintainer Acked-by on v1 1/2
#4 lore raw

Re: [PATCH net-next 1/2] sctp: implement memory accounting on tx path

In reply to Xin Long
Quoted 5 lines · Xin Long

Now when sending packets, sk_mem_charge() and sk_mem_uncharge() have been used to set sk_forward_alloc. We just need to call sk_wmem_schedule() to check if the allocated should be raised, and call sk_mem_reclaim() to check if the allocated should be reduced when it's under memory pressure.

Quoted 8 lines

If sk_wmem_schedule() returns false, which means no memory is allowed to allocate, it will block and wait for memory to become available.

Note different from tcp, sctp wait_for_buf happens before allocating any skb, so memory accounting check is done with the whole msg_len before it too.

Quoted 35 lines
---
 net/sctp/socket.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
net/sctp/socket.c +8 -2
index 6140471..06c6f4a 100644
@@ -1913,7 +1913,10 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
 	if (sctp_wspace(asoc) < (int)msg_len)
 		sctp_prsctp_prune(asoc, sinfo, msg_len - sctp_wspace(asoc));
 
-	if (sctp_wspace(asoc) <= 0) {
+	if (sk_under_memory_pressure(sk))
+		sk_mem_reclaim(sk);
+
+	if (sctp_wspace(asoc) <= 0 || !sk_wmem_schedule(sk, msg_len)) {
 		timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
 		err = sctp_wait_for_sndbuf(asoc, &timeo, msg_len);
 		if (err)
@@ -8891,7 +8894,10 @@ static int sctp_wait_for_sndbuf(struct sctp_association *asoc, long *timeo_p,
 			goto do_error;
 		if (signal_pending(current))
 			goto do_interrupted;
-		if ((int)msg_len <= sctp_wspace(asoc))
+		if (sk_under_memory_pressure(sk))
+			sk_mem_reclaim(sk);
+		if ((int)msg_len <= sctp_wspace(asoc) &&
+		    sk_wmem_schedule(sk, msg_len))
 			break;
 
 		/* Let another process have a go.  Since we are going
Signature · 1 line
2.1.0
maintainer Acked-by on v1 2/2
#3 lore raw

Re: [PATCH net-next 2/2] sctp: implement memory accounting on rx path

In reply to Xin Long
Quoted 7 lines · Xin Long

sk_forward_alloc's updating is also done on rx path, but to be consistent we change to use sk_mem_charge() in sctp_skb_set_owner_r().

In sctp_eat_data(), it's not enough to check sctp_memory_pressure only, which doesn't work for mem_cgroup_sockets_enabled, so we change to use sk_under_memory_pressure().

Quoted 7 lines

When it's under memory pressure, sk_mem_reclaim() and sk_rmem_schedule() should be called on both RENEGE or CHUNK DELIVERY path exit the memory pressure status as soon as possible.

Note that sk_rmem_schedule() is using datalen to make things easy there.

Quoted 105 lines
---
 include/net/sctp/sctp.h |  2 +-
 net/sctp/sm_statefuns.c |  6 ++++--
 net/sctp/ulpevent.c     | 19 ++++++++-----------
 net/sctp/ulpqueue.c     |  3 ++-
 4 files changed, 15 insertions(+), 15 deletions(-)
include/net/sctp/sctp.h +1 -1
index 1d13ec3..eefdfa5 100644
@@ -421,7 +421,7 @@ static inline void sctp_skb_set_owner_r(struct sk_buff *skb, struct sock *sk)
 	/*
 	 * This mimics the behavior of skb_set_owner_r
 	 */
-	sk->sk_forward_alloc -= event->rmem_len;
+	sk_mem_charge(sk, event->rmem_len);
 }
 
 /* Tests if the list has one and only one entry. */
net/sctp/sm_statefuns.c +4 -2
index c9ae340..7dfc34b 100644
@@ -6412,13 +6412,15 @@ static int sctp_eat_data(const struct sctp_association *asoc,
 	 * in sctp_ulpevent_make_rcvmsg will drop the frame if we grow our
 	 * memory usage too much
 	 */
-	if (*sk->sk_prot_creator->memory_pressure) {
+	if (sk_under_memory_pressure(sk)) {
 		if (sctp_tsnmap_has_gap(map) &&
 		    (sctp_tsnmap_get_ctsn(map) + 1) = tsn) {
 			pr_debug("%s: under pressure, reneging for tsn:%u\n",
 				 __func__, tsn);
 			deliver = SCTP_CMD_RENEGE;
-		 }
+		} else {
+			sk_mem_reclaim(sk);
+		}
 	}
 
 	/*
net/sctp/ulpevent.c +8 -11
index 8cb7d98..c2a7478 100644
@@ -634,8 +634,9 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 						gfp_t gfp)
 {
 	struct sctp_ulpevent *event = NULL;
-	struct sk_buff *skb;
-	size_t padding, len;
+	struct sk_buff *skb = chunk->skb;
+	struct sock *sk = asoc->base.sk;
+	size_t padding, datalen;
 	int rx_count;
 
 	/*
@@ -646,15 +647,12 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 	if (asoc->ep->rcvbuf_policy)
 		rx_count = atomic_read(&asoc->rmem_alloc);
 	else
-		rx_count = atomic_read(&asoc->base.sk->sk_rmem_alloc);
+		rx_count = atomic_read(&sk->sk_rmem_alloc);
 
-	if (rx_count >= asoc->base.sk->sk_rcvbuf) {
+	datalen = ntohs(chunk->chunk_hdr->length);
 
-		if ((asoc->base.sk->sk_userlocks & SOCK_RCVBUF_LOCK) ||
-		    (!sk_rmem_schedule(asoc->base.sk, chunk->skb,
-				       chunk->skb->truesize)))
-			goto fail;
-	}
+	if (rx_count >= sk->sk_rcvbuf || !sk_rmem_schedule(sk, skb, datalen))
+		goto fail;
 
 	/* Clone the original skb, sharing the data.  */
 	skb = skb_clone(chunk->skb, gfp);
@@ -681,8 +679,7 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
 	 * The sender should never pad with more than 3 bytes.  The receiver
 	 * MUST ignore the padding bytes.
 	 */
-	len = ntohs(chunk->chunk_hdr->length);
-	padding = SCTP_PAD4(len) - len;
+	padding = SCTP_PAD4(datalen) - datalen;
 
 	/* Fixup cloned skb with just this chunks data.  */
 	skb_trim(skb, chunk->chunk_end - padding - skb->data);
net/sctp/ulpqueue.c +2 -1
index 5dde921..770ff1f 100644
@@ -1106,7 +1106,8 @@ void sctp_ulpq_renege(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk,
 			freed += sctp_ulpq_renege_frags(ulpq, needed - freed);
 	}
 	/* If able to free enough room, accept this chunk. */
-	if (freed >= needed) {
+	if (sk_rmem_schedule(asoc->base.sk, chunk->skb, needed) &&
+	    freed >= needed) {
 		int retval = sctp_ulpq_tail_data(ulpq, chunk, gfp);
 		/*
 		 * Enter partial delivery if chunk has not been
Signature · 1 line
2.1.0
author maintainer
#7 lore raw
In reply to David Miller
Quoted 7 lines · David Miller

From: Xin Long <[email protected]>
Date: Sun, 31 Mar 2019 16:56:40 +0800

Quoted 1 line

Simple tests are as attached.

Are these integratable into selftests somehow?

We're thinking about adding some sctp tests into selftests,
but these ones are not good to be the first one, I think. sysctl_sockets_memctl.sh is a multi-host test, even netns can't work for it.

Each subcomponent seems to have its own test case in other git repo, can I ask what kind of tests should I put into kernel selftests in the future?

maintainer
#8 lore raw

From: Xin Long <[email protected]>
Date: Thu, 4 Apr 2019 17:45:07 +0800

Quoted 3 lines

We're thinking about adding some sctp tests into selftests,
but these ones are not good to be the first one, I think. sysctl_sockets_memctl.sh is a multi-host test, even netns can't work for it.

You can't set per-netns memory controls appropriately to make the tests work?

Quoted 3 lines

Each subcomponent seems to have its own test case in other git repo, can I ask what kind of tests should I put into kernel selftests in the future?

It would be ideal that we have a decent base set of SCTP tests and then when bug fixes are added, we get a unit test with the fix.

I guess a good baseline would be testing basic comminucation between SCTP sockets in different netns, and then adding tests for setting the various socket options and making sure the socket option had the desired effect.

author maintainer
#9 lore raw
Quoted 10 lines · David Miller

From: Xin Long <[email protected]>
Date: Thu, 4 Apr 2019 17:45:07 +0800

Quoted 3 lines

We're thinking about adding some sctp tests into selftests,
but these ones are not good to be the first one, I think. sysctl_sockets_memctl.sh is a multi-host test, even netns can't work for it.

You can't set per-netns memory controls appropriately to make the tests work?

static atomic_long_t sctp_memory_allocated;

struct proto sctp_prot = {
.memory_allocated = &sctp_memory_allocated,

sysctl limitation is per-netns, but the counter 'sctp_memory_allocated' is not. It means other netns buffer allocation will affect the current netns.

Quoted 5 lines
Quoted 4 lines

Each subcomponent seems to have its own test case in other git repo, can I ask what kind of tests should I put into kernel selftests in the future?

Quoted 7 lines

It would be ideal that we have a decent base set of SCTP tests and then when bug fixes are added, we get a unit test with the fix.

I guess a good baseline would be testing basic comminucation between SCTP sockets in different netns, and then adding tests for setting the various socket options and making sure the socket option had the desired effect.

got your point. thanks