Skip to contentSkip to conversation

shmget with SHM_HUGETLB flag: Operation not permitted

DISCUSSION Jochen Roemling 5 participants 13 messages Feb 27, 2004 – Feb 29, 2004 4 messages not in the archive HugeTLB

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 4 lines · Chris Wright
Quoted 2 lines · * William Lee Irwin III ([email protected]) wrote:

On Thu, Feb 26, 2004 at 11:36:03PM +0100, Jochen Roemling wrote:

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

How can I grant the permission to use HUGETLB to ordinary users?

(a) use the fs which uses fs permissions to grant users permission to fiddle with hugetlb
(b) man 2 capset

In case that part wasn't clear, it would be CAP_IPC_LOCK capability.

Thanks. Capset was the keyword I couldn't remember.

_Background:_
I would like to install Oracle 10g Database on Linux with HUGETLB support. The oracle binary exits with -EPERM because it is not allowed to create a shared memory segment with the SHM_HUGETLB flag set.

I installed the libcap2 package (from debian testing) and now have the tool "setcap" available. I wanted to test this on my example pgm mentioned in the original post using:

roesrv01~ # setcap CAP_IPC_LOCK a.out
fatal error: Invalid argument
usage: setcap [-q] (-|<caps>) <filename> [ ... (-|<capsN>) <filenameN> ]

using the number "14" instead of the name "CAP_IPC_LOCK" doesn't work either. I don't have any glue. Do have a simple example for me?

By the way: CAP_IPC_LOCK is only checked in line 508 of ipc/shm.c:

        case SHM_LOCK:
        case SHM_UNLOCK:
        {
/* Allow superuser to lock segment in memory */
/* Should the pages be faulted in here or leave it to user? */
/* need to determine interaction with current->swappable */
                if (!capable(CAP_IPC_LOCK)) {
                        err = -EPERM;
                        goto out;
                }

There is nothing around that says: "Allow this only without HUGETLB". Are you sure that this capability is my problem?

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 9 lines · * Jochen Roemling ([email protected]) wrote:
Quoted 2 lines · Chris Wright

In case that part wasn't clear, it would be CAP_IPC_LOCK capability.

Thanks. Capset was the keyword I couldn't remember.

_Background:_
I would like to install Oracle 10g Database on Linux with HUGETLB support. The oracle binary exits with -EPERM because it is not allowed to create a shared memory segment with the SHM_HUGETLB flag set.

OK, as expected.

Quoted 10 lines

I installed the libcap2 package (from debian testing) and now have the tool "setcap" available. I wanted to test this on my example pgm mentioned in the original post using:

roesrv01~ # setcap CAP_IPC_LOCK a.out
fatal error: Invalid argument
usage: setcap [-q] (-|<caps>) <filename> [ ... (-|<capsN>) <filenameN> ]

using the number "14" instead of the name "CAP_IPC_LOCK" doesn't work either. I don't have any glue. Do have a simple example for me?

did you try setpcaps? smth like setpcaps cap_ipc_lock+e <pid>

Quoted 1 line

By the way: CAP_IPC_LOCK is only checked in line 508 of ipc/shm.c:

<snip>

Quoted 7 lines
                 if (!capable(CAP_IPC_LOCK)) {
                         err = -EPERM;
                         goto out;
                 }

There is nothing around that says: "Allow this only without HUGETLB". Are you sure that this capability is my problem?

Yes, take a look at fs/hugetlbfs/inode.c::hugetlb_zero_setup()

thanks,
-chris

Signature · 1 line
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net
#6 lore raw
Quoted 1 line · * Chris Wright ([email protected]) wrote:

did you try setpcaps? smth like setpcaps cap_ipc_lock+e <pid>

bah, sorry, i should point out, that isn't going to work w/out CAP_SETPCAP which is disabled. you'll want to start with full privs (i.e. root) and drop all but CAP_IPC_LOCK. SuSE used to have a tool called compartment that helped with this, might google for it.

thanks,
-chris

Signature · 1 line
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net
#7 lore raw
Quoted 4 lines · Chris Wright

SuSE used to have a tool called compartment
that helped with this, might google for it.

sounds good, but does not work either :-(

roesrv01~ # compartment --cap CAP_IPC_LOCK bash
bash-2.05b# /sbin/getpcaps 3226
Capabilities for `3226': =ep cap_ipc_lock+i cap_setfcap-p cap_setpcap-ep bash-2.05b# su - jochen
jochen@roesrv01:~> /sbin/getpcaps 3233
Capabilities for `3233': = cap_ipc_lock+i
jochen@roesrv01:~> ./a.out
Failure:: Operation not permitted
jochen@roesrv01:~> ps ax
[...]

3226 pts/0    S      0:00 bash
3233 pts/0    S      0:00 -su
#8 lore raw
Quoted 11 lines · Jochen Roemling

sounds good, but does not work either :-(
roesrv01~ # compartment --cap CAP_IPC_LOCK bash
bash-2.05b# /sbin/getpcaps 3226
Capabilities for `3226': =ep cap_ipc_lock+i cap_setfcap-p cap_setpcap-ep bash-2.05b# su - jochen
jochen@roesrv01:~> /sbin/getpcaps 3233
Capabilities for `3233': = cap_ipc_lock+i
jochen@roesrv01:~> ./a.out
Failure:: Operation not permitted
jochen@roesrv01:~> ps ax
[...]

Quoted 2 lines
3226 pts/0    S      0:00 bash
3233 pts/0    S      0:00 -su

Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.

-- wli

#9 lore raw
Quoted 3 lines · William Lee Irwin III

Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.

cat /proc/sys/vm/nr_hugepages
64

cat /proc/sys/kernel/shmmax
33554432

cat /proc/meminfo | grep Huge

HugePages_Total:    64
HugePages_Free:     62
Hugepagesize:     4096 kB

but again: root can, users cannot, so sizes won't matter, would they?

#10 lore raw
Quoted 1 line · William Lee Irwin III
Quoted 1 line

Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.

Quoted 2 lines · Jochen Roemling

cat /proc/sys/vm/nr_hugepages
64

256MB limit there.

Quoted 2 lines · Jochen Roemling

cat /proc/sys/kernel/shmmax
33554432

32MB limit there.

Quoted 5 lines · Jochen Roemling

cat /proc/meminfo | grep Huge

HugePages_Total:    64
HugePages_Free:     62
Hugepagesize:     4096 kB
but again: root can, users cannot, so sizes won't matter, would they?

It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0. It will barf if you ask for more than either one of those limits. It will also barf if you ask for an amount not a multiple of the hugepage size. Please show the test program's code and strace the test program to determine what response it's getting.

-- wli

#11 lore raw
Quoted 8 lines · William Lee Irwin III

It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0. It will barf if you ask for more than either one of those limits. It will also barf if you ask for an amount not a multiple of the hugepage size. Please show the test program's code and strace the test program to determine what response it's getting.

I attached the test pgm. It is nearly the same as shown in Documentation/vm/hugetlbpage.txt

If you run it as root, it allocates 1 Hugepage, if run as user, it fails.

roesrv01:~ # ./a.out
shmid: 0x220004
shmaddr: 0x40167000
Starting the writes:
....
Starting the Check...Done.
roesrv01:~ # su - jochen
jochen@roesrv01:~> ./a.out
Failure:: Operation not permitted

I guess, a strace is not necessary.
The pgm has only the main function and only one position where it says "Failure"

What do I have to do to make this pgm run as an ordinary user with a stock kernel?

Curious...
Jochen

/* Example of using hugepage in user application using Sys V shared memory
* system calls. In this example, app is requesting memory of size 256MB that
* is backed by huge pages. Application uses the flag SHM_HUGETLB in shmget
* system call to informt the kernel that it is requesting hugepages. For
* IA-64 architecture, Linux kernel reserves Region number 4 for hugepages.
* That means the addresses starting with 0x800000....will need to be
* specified.
*/
#include <sys/types.h>
#include <sys/shm.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <errno.h>

extern int errno;
#define SHM_HUGETLB 04000
#define LPAGE_SIZE (4UL*1024UL*1024UL)
#define dprintf(x) printf(x)
#define ADDR (0x8000000000000000UL)
main()

{
        int shmid;
        int     i, j, k;
        volatile        char    *shmaddr;

	if ((shmid =shmget(IPC_PRIVATE, LPAGE_SIZE, SHM_HUGETLB|IPC_CREAT|SHM_R|SHM_W )) < 0) {
                perror("Failure:");
                exit(1);
        }
        printf("shmid: 0x%x\n", shmid);
        shmaddr = shmat(shmid, (void *)ADDR, SHM_RND) ;
        if (errno != 0) {
                perror("Shared Memory Attach Failure:");
                exit(2);
        }
        printf("shmaddr: %p\n", shmaddr);

        dprintf("Starting the writes:\n");
        for (i=0;i<LPAGE_SIZE;i++) {
                shmaddr[i] = (char) (i);
                if (!(i%(1024*1024))) dprintf(".");
        }
        dprintf("\n");
        dprintf("Starting the Check...");
        for (i=0; i<LPAGE_SIZE;i++)
                if (shmaddr[i] != (char)i)
                        printf("\nIndex %d mismatched.");
        dprintf("Done.\n");
        if (shmdt((const void *)shmaddr) != 0) {
                perror("Detached Failure:");
                exit (3);
        }
}
#12 lore raw
Quoted 5 lines · William Lee Irwin III
Quoted 5 lines

It's capable(CAP_IPC_LOCK) || in_group_p(0), not current->uid == 0. It will barf if you ask for more than either one of those limits. It will also barf if you ask for an amount not a multiple of the hugepage size. Please show the test program's code and strace the test program to determine what response it's getting.

Quoted 2 lines · Jochen Roemling

What do I have to do to make this pgm run as an ordinary user with a stock kernel?

Locked memory is a privileged resource, so you do have to do something to authenticate lest any user be able to consume all memory on your system with no possibility of paging it. Examples of what to do to acquire locked memory specifically for hugetlb shm segments in mainline:

(a) give the user gid 0 as a primary or supplementary group
(b) grant the capability -- yes, it can be done (and is being done in practice elsewhere), something is going wrong on your end I haven't been able to diagnose.
(c) make requests from a shmget() proxy daemon where you make requests over a socket and it hands back shm segment ID's that have had their uid's/perms set to the end user. Once shmget() is done, shmat() uses normal shm permissions checks.
(d) use a setuid root shmget() helper app.
(e) launch as root, then retain capabilities
(f) launch as root and shmget before dropping privs

(e) and (f) are probably not options in your case. I can't predict what's going to be desirable on your end in general. You will have to jump through a hoop of some kind, though, and be glad you do, since otherwise unbounded amounts of locked memory requested by arbitrary users could cripple the system's performance or worse.

-- wli

#13 lore raw
In reply to Jochen Roemling
Quoted 11 lines · Jochen Roemling
Quoted 3 lines · William Lee Irwin III

Check /proc/sys/vm/nr_hugepages and /proc/sys/kernel/shmmax also.

cat /proc/sys/vm/nr_hugepages
64

cat /proc/sys/kernel/shmmax
33554432

Quoted 6 lines

cat /proc/meminfo | grep Huge

HugePages_Total:    64
HugePages_Free:     62
Hugepagesize:     4096 kB

but again: root can, users cannot, so sizes won't matter, would they?

Of course! Appended simple patch is what i did (ugly, I know) and that helped me install Oracle10g on Debian unstable (with two other adaptations). I don't know how in the hell I forgot to put that important patch on my page where I explain how to install Oracle10g on Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
later today or tomorrow, after I check some other problems people have reported to me (and you Jochen, too :)).

Index: 3.3/fs/hugetlbfs/inode.c

fs/hugetlbfs/inode.c Mon, 23 Feb 2004 09:33:52 +0100 zcalusic (linux26/D/6_inode.c 1.1.1.3 644) renamed from fs/hugetlbfs/inode.c Thu, 19 Feb 2004 19:05:15 +0100 zcalusic (linux26/D/6_inode.c 1.1.1.2 644) +0 -3
@@ -694,9 +694,6 @@
 	struct qstr quick_string;
 	char buf[16];
 
-	if (!capable(CAP_IPC_LOCK))
-		return ERR_PTR(-EPERM);
-
 	if (!is_hugepage_mem_enough(size))
 		return ERR_PTR(-ENOMEM);
 	n = atomic_read(&hugetlbfs_counter);

Regards,

Signature · 3 lines
Zlatko

P.S. Please Cc: me, I'm not subscribed.
#14 lore raw
Quoted 7 lines · Zlatko Calusic

Of course! Appended simple patch is what i did (ugly, I know) and that helped me install Oracle10g on Debian unstable (with two other adaptations). I don't know how in the hell I forgot to put that important patch on my page where I explain how to install Oracle10g on Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
later today or tomorrow, after I check some other problems people have reported to me (and you Jochen, too :)).

You have to be a bit more careful than this; this gives any user the ability to consume locked memory via shmget().

-- wli

#15 lore raw
Quoted 11 lines · William Lee Irwin III
Quoted 7 lines · Zlatko Calusic

Of course! Appended simple patch is what i did (ugly, I know) and that helped me install Oracle10g on Debian unstable (with two other adaptations). I don't know how in the hell I forgot to put that important patch on my page where I explain how to install Oracle10g on Debian?! Sorry, it'll be on http://linux.inet.hr/oracle10g_on_debian.html
later today or tomorrow, after I check some other problems people have reported to me (and you Jochen, too :)).

You have to be a bit more careful than this; this gives any user the ability to consume locked memory via shmget().

Yes, I know! But hopefully this security implication is not so important for people who just want to test new database on their workstations (like me), or even install it on the production database server where you mostly don't see any other shell user beside the administrator of the machine.

But yes, you're right, we need to warn people.

DON'T use the patch if you have untrusty shell users on your machine!!!

Signature · 6 lines
Zlatko

P.S. Although, if superuser properly limits the number of hugepages
     that can be allocated (echo "valid nr of pages" > nr_hugepages)
     what does attacker do to consume more pages than that? Just
     curious...
#16 lore raw
In reply to Chris Wright
Quoted 2 lines · Chris Wright
Quoted 1 line · * Jochen Roemling ([email protected]) wrote:

Chris Wright wrote:

<snip>

Quoted 9 lines
Quoted 7 lines
                 if (!capable(CAP_IPC_LOCK)) {
                         err = -EPERM;
                         goto out;
                 }

There is nothing around that says: "Allow this only without HUGETLB". Are you sure that this capability is my problem?

Yes, take a look at fs/hugetlbfs/inode.c::hugetlb_zero_setup()

Rik had a patch in rhel3 for nonroot mlock() which made this all work, I will post a patch for 2.6. from what I understand it's very useful for the gpg folks to have, eg wide audience for this patch. just need to do a bit more testing and having Rik review it then ll send it out (or maybe he feels like getting shot and he will ;)

Wim