2.4.8 Resource leaks + limits

Can anyone please explain something to me?
I saw Alan's comments on Ivan Kalvatchev's malloc "bomb".
So i thought maybe i should test to impose some limits, and try some changes with Ivan's code.

Box:

Linux UltraSparc 2.4.8 #3 Tue Aug 14 01:03:31 CEST 2001 sparc64 unknown (a Sun Ultra10)

Code:

--------8<-------8<-------8<-------8<-------8<----------- struct entry { struct entry *next; char *p;} * head=0;

int main() {
int i;
struct entry *e;

   for(i=0;;i+=2*4) {
      if (!(e=malloc(4096)) || !(e->p = malloc(4096)))
        break;
      e->next = head;
      head = e;
      if (!(i%1024)) printf("malloc @ %dkb\n",i);

}

   printf("Stopped at %dkb \nTouching-in-a-loop.\n",i);

   for (e=head ;; e=e->next?e->next:head )
     memset(e->p,++i % 256,4096);

return 0;
} --------8<-------8<-------8<-------8<-------8<-----------

Memory is (after run + swapoff + swapon):

[root@UltraSparc src]# free -m
total used free shared buffers cached
Mem: 122 38 83 0 1 17
-/+ buffers/cache: 20 101
Swap: 1057 0 1057

And the limits are:

*               -       rss             64000
*               -       memlock         32000
*               -       as              128000
*               -       data            64000
*               hard    nproc           50
*               soft    nproc           40
*               -       stack           666
*               soft    priority        16
*               hard    priority        12

I ran maybe 20-30 instances at once.
All process allocated the 64MB that it was allowed by the "data" limit. The system went down to really crawling (but never locked up completly). After a while the oom killer kicked in as well, when i ran out of swap+ram.

Question: why isn't there a limit for global memory usage per user? No way could i stop these processes without decreasing the nproc stuff. That's not really what one want's some time, you know :)

The more "real" issue here is that after this run i couldn't log on as another user than root.
Even now (after swapoff -a and so) i can't log on as another user:

[root@UltraSparc src]# su -l mag2
can not fork user shell: Resource temporarily unavailable

How can i get my fargin resources back? :)

Nice that the system survived (kinda, took 15 min to get logged on via ssh ;) ), but it's not cool if i have to reboot anyways because my resources are farged.

Can i do anything, or what?

Magnus

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

In article <3ce801c12548$b7971750$020a0a0a@totalmef> you write:

Answer: because traditionally Linux (or UNIX) hasn't had a good and efficient way to have per-user statistics.

This is why you mainly find per-process stuff in all the limits.

Linux has had (for a while now) a "struct user" that is actually quickly accessible through a direct pointer off every process that is associated with that user, and we could (and _will_) start adding these kinds of limits. However, part of the problem is that because the limits haven't historically existed, there is also no accepted and nice way of setting the limits.

Oh, well.

Linus

This should be the least of the problems. Simply add new RLIMIT_* values[1] (and possibly [gs]etrlimit64 syscalls). The shell's ulimit command can easily pick those up. Non-standard, but every other solution will be, too.

---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

"Linus Torvalds" <[email protected]> wrote in message news:[email protected]...

Well this problem is not so hard, i'll just set "nproc = maxglobal / data" or something like that for now.
Too bad if any of my users want to run many diet processes, but that's life.

The more serious part of my little alloc adventure is much more dangerous:

Whattaheck happened to my resources?
I _still_ can't log in to that box as a luser (root works).

[snip]

Magnus

May be memory fragmentation. You need an order 1 allocation for fork(), just to allocate task_struct...

From: "Alexander Viro" <[email protected]>

[snap[crackle[pop]]]

just

1) Does that mean i'm screwed (then why, i got about 80 MB free here + 1gb swap, howto defrag?)

2) I can ssh in as root, why does that still work?

:-)

Magnus

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

No, 2.4.8 seems to like to soft lockup in cases after it used up all swap. I also run some trivial memory stressing tests on a UP, 128 MB, 256 swap, 7 MB/sec UDMA disk subsytem box and after a couple of successful recovery [couple = max 1 in my case] the system soft locks. swap space was 0, no disk activity, CPU apparently spins in kswapd, all relevant zones, inactive_* had plenty free pages and no memory fragmentation. After it soft locked none of the VM stat value changed at all. Rik also called for help in another thread but the problem seems to be not out_of_memory() tuning (when to jump in) however either accounting bug or other (kswapd related?) thing - kernel stacks were a bit strange [using Right_ALT+Scroll_Lock when soft locked], like page_launder
do_try_to_free_pages
kswapd
kswapd
kswapd

Szaka

From: "Szabolcs Szakacsits" <[email protected]>
[snip]

Well as i said, my system _never_ locks up completly ( but it might look that way because it's crawlin'-like-a-dog ).
The problem is that i can shh in as root, but not as any other user ( not via login or su or either ).
Root always works, and mind you, this is _after_ the "attack", and the system resources is back to normal as far as i can tell.

It leads me to think something is wrong with the nproc rlimit accounting or something like that, maybe in the oom kill code?

Magnus

Are you using < 0.73 PAM without the change_uid pam_limit option? You set in /etc/security/limits.conf:
* soft nproc 40

the '*' valid for users but not for root, the relevant parts of a default login/pam works like:

<running as root>
setrlimit()
fork()
setuid(user_uid)

So if you have at least 40 root processes running already for whatever reason then the result is what you see.

The livelocks what I mentioned is indeed different and fixed in 2.4.9 (I guess the 'kswapd thought shortage for highmem zone when its size is actually 0' issue what Linus said in the 'kswapd using all cpu for long periods' thread). Sorry for the confusion, hope the above fixes your problem.

Szaka

Uhm, this wanted to be "... < 0.73 OR >= 0.73 without the ...."

Szaka

From: "Szabolcs Szakacsits" <[email protected]>

not

Well root is unlimited, and i have _no_ processes running as the user i'm trying to log on as.
This must be a bug then.
I even tried adduser <newuser>, but i couldn't log in as that either.

I never had any lockups.

Magnus

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
PGP Key: http://www.genline.nu/mag_pgp.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

This may be related to another thread. See the mail "Re: kswapd using all cpu for long periods in 2.4.9-pre4" posted by Linus a couple of hours ago.

/Tobias

On Tue, Aug 14, 2001, Ulrich Drepper wrote about "Re: 2.4.8 Resource leaks + limits":

I don't see how this would work without confusing users. ulimit currently fits the model where limits are enforced per process, because each process can have its own different limits.
If add per-user limits, what do you do if the user has several processes with different per-user limits? You'll need to have "ulimit" and the likes set a per-user limit shared by all processes of this user, and the last one set by any process "wins" and takes effect. But this will not be expected by the users who expect ulimits to effect only children processes (e.g., now it's common to lower a limit and fork/exec a program which you want to limit).

So it's doable this way, but the manuals will have to be very clear as to which limits are inherited how.

Nadav Har'El                        |       Wednesday, Aug 15 2001, 26 Av 5761
[email protected]             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |I used to work in a pickle factory, until
http://nadav.harel.org.il           |I got canned.

So when you do impose this, where will it be setable, will there be a flat file in /etc like solaris, or compile time for the kernel?

Eh?

Why wouldn't it be like most parameters in Linux, e.g. dynamically adjustable via a sysctl or /proc?

IMHO, of course...

cu

jjs

i would think to put global limits in /proc or in a flat text /etc and per user limits in something like /etc/passwd or /etc/shadow? Is it against any standard to have extra fields in those files?

-Tony
.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-.
Anthony J. Biacco Network Administrator/Engineer
[email protected] Intergrafix Internet Services

"Dream as if you'll live forever, live as if you'll die today"
http://www.asteroid-b612.org http://www.intergrafix.net
.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-._.-.

Take a look at the pam modules, they already handle limit configuration per user, and I think all the major Linux (and also stuff like Solaris) distros run PAM based auth

Alan, would you accept the trivial version of per-user CPU scheduling for 2.4 if there is a lot of demand ?

The code has been running in Conectiva's kernel RPM for over a year now, is implemented entirely in the recalculate part of the scheduler, leaves the scheduler fast path all alone and only needs to be properly ported to newer 2.4 kernels now...

regards,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

I'd rather it was a patch outside the main tree

Ok, I'll just keep it the way it was, then ;)

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

I was thinking along the lines of a /etc/system file where all the memconfig can be set can be set in one fail swoop..

Probably both -- he's thinking more about the userspace side of it, though.

Since you want these settings to vary by user and persist across reboots, the default levels will need to be stored in some table/database and /etc seems like the appropriate place. Of course, that's more PAM's domain than a kernel issue, but the userspace connection is still important.

Will there be a utility for viewing/changing per-user limits? If so, what should it be called? Will we just pile more values into ulimit? We're running a little short on intuitive flags.

/proc entries are okay for tweaking kernel parameters, but it seems a little weak as a primary interface. You might as well have ps say 'Go grep it yourself!'

-- Brian

Something along the lines of the nice little program sysctlconfig-gtk, to set the values with a gui interface, stores them in e.g. /etc/sysctl.conf, and activtates the new parameters with a mouse click....

Real men could edit /etc/sysctl.conf themselves, while newbies could use the GUI program -

cu

jjs