Skip to contentSkip to conversation

linux swap freeze STILL in 2.4.x

DISCUSSION Marc Lehmann 2 participants 9 messages Feb 25, 2001 – Feb 26, 2001

lore
Share & tools
lore
#0 lore raw

It seems linux-2.4 still freezes on out-of-memory situations:

I was using 2.4.2-ac3 SMP and had a fairly large background job that takes hundreds of megabytes of memory, much more than I have:

Mem:        255296      81836     173460          0      10324      30608
Swap:        99992          0      99992

Usually I swapon ./swap some 512MB swapfile, but today I forgot it. When the machine started to get sluggish I sent the process a -STOP signal.

Swap: 99992 99992 0

O.k, (I had about 12MB of main memory free (in the +/- buffers line of free) and the machine was sluggish but workable for about five minutes. At the instant I did a swapon ./swap the machine froze hard (no sysrq, no ping etc...)

I thought these complete freezes on OOM-situations had been fixed in 2.4.x? Do I have to watch out for andrea's fix-2.4-oom patches?

;)

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#1 lore raw
Quoted 1 line · Marc Lehmann

It seems linux-2.4 still freezes on out-of-memory situations:

<snip>

Quoted 2 lines

Usually I swapon ./swap some 512MB swapfile, but today I forgot it. When the machine started to get sluggish I sent the process a -STOP signal.

Signal delivery during oomest does not work (last time I tested). Andrea fixed this once.. long time ~problem.

-Mike

#2 lore raw
In reply to Marc Lehmann
Quoted 5 lines · Mike Galbraith
Quoted 2 lines

Usually I swapon ./swap some 512MB swapfile, but today I forgot it. When the machine started to get sluggish I sent the process a -STOP signal.

Signal delivery during oomest does not work (last time I tested). Andrea fixed this once.. long time ~problem.

Well, the signal delivery seemed to have worked fine - the machine was quite usable (it swapped a lot, but the system was never unusable for longer than a second or so). The problem started when I did the swapon. Well, it didn't start, the system just froze.

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#3 lore raw
Quoted 11 lines · Marc Lehmann
Quoted 5 lines · Mike Galbraith
Quoted 2 lines

Usually I swapon ./swap some 512MB swapfile, but today I forgot it. When the machine started to get sluggish I sent the process a -STOP signal.

Signal delivery during oomest does not work (last time I tested). Andrea fixed this once.. long time ~problem.

Well, the signal delivery seemed to have worked fine - the machine was quite usable (it swapped a lot, but the system was never unusable for longer than a second or so). The problem started when I did the swapon. Well, it didn't start, the system just froze.

Ok.. I guess that got fixed. Used to be it was all over as soon as the last of swap was consumed.

-Mike

#4 lore raw
In reply to Marc Lehmann
Quoted 2 lines · Mike Galbraith

Signal delivery during oomest does not work (last time I tested). Andrea fixed this once.. long time ~problem.

Hmm, here is soemthing that is new: Just now, the machine gets VERY very sluggish and swaps:

             total       used       free     shared    buffers     cached
Mem:        255296     253708       1588          0      29808     183020

-/+ buffers/cache: 40880 214416
Swap: 99992 99992 0

now, there is plenty of free memory (200megs!) but no spwapsace and the kernel keeps swapping. The only interesting processes here are:

  PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
  112 ?        S      0:00    742  1366 38921 3460  1.3 /opt/mysql//libexec/mysqld --basedir=/opt/mysql/ --datadir=/var/mysql --user=root --pid-
  205 ?        S      2:28  12335  1444 27167 4294966180 6728.9 /usr/bin/X11/X :0 -audit 1 -auth /etc/cfg/Xauthority -a 2 -once -t 5 vt02 -defer
  421 pts/13   TN     1:00    804   707 31552 17444  6.8 /usr/bin/perl ./summarize
  376 pts/10   R      7:07    269   129 22614 1852  0.7 rsync -av . doom cerebro-root/. --delete

when I SIGSTOP the summarize script (which uses mysql very intensively) the system starts to work again but the memory situation does not improve. The RSS size of X puzzles me a bit, but this was always the case under 2.4.2 and 2.4.2ac3 (and maybe before) and didn't cause a problem before.

Another bug I found is that initializing md on the kernel commandline in the wrong order (first md1 then md0) keeps the kernel from mounting md0 as root-device. Another problem is that, when I "startraid /dev/md1" (a two-partition, striped raid without persistent superblock) I get strange errors in /var/log/kernel (if anybody asks I'll provide them) but it works fine when I sue md=x on the kernel commandline. It's not a configuration problem sicne I got the same strange probkems with the mdstart I used successfully under 2.1 and 2.2.

Another nitpick is kernel-pcmcia: For some unexplainable reason, the kernel SWITCHES OFF POWER to the pcmcia slots BEFORE notifying apmd, which then tries to save important data and locks (not the machine, just the script) since the network is suddenly dead although interface etc.. all still exist. Under the pcmcia-cs package one could work around this bug by specifying do_apm=0 for the pcmcia_core module, which has no effect under 2.4.

So I do keep asking me: does anybody actually use 2.4 on production machines? ;-> (Historically, it seesm that my machines tend to freeze easily because of sudden OOM and/or reiserfs ;)

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#5 lore raw
In reply to Marc Lehmann

Oh, and one last thing I forgot: loop devices. Since 2.4.1 (the first version I used) through 2.4.2 and 2.4.2ac3 I only get:

cerebro:~# strace -f -o x losetup -e rc6 /dev/loop0 /dev/hdd
Memory Fault

And then no access to the loop device works anymore (clearly this is after the 2.4.0.something crypto-patch applied, so this is probably not a 2.4.2 issue anyway since there is no 2.4.2 crypto patch).

Happy Hacking ;)

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#6 lore raw
Quoted 9 lines · Marc Lehmann

Oh, and one last thing I forgot: loop devices. Since 2.4.1 (the first version I used) through 2.4.2 and 2.4.2ac3 I only get:

cerebro:~# strace -f -o x losetup -e rc6 /dev/loop0 /dev/hdd
Memory Fault

And then no access to the loop device works anymore (clearly this is after the 2.4.0.something crypto-patch applied, so this is probably not a 2.4.2 issue anyway since there is no 2.4.2 crypto patch).

Hmm.. I remember having this problem and it was a problem with strace. I fiddled with it and got it to work, but damned if I remember what I did (had to be something trivial since I know jack spit about it:).

Anyway, it works fine here with virgin 2.4.2, so it seems unlikely it's a kernel problem.

259   execve("/sbin/losetup", ["losetup", "/dev/loop0", "/dev/hda5"], [/* 47 vars */]) = 0
259   brk(0)                            = 134525772
259   old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
259   open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
259   open("/etc/ld.so.cache", O_RDONLY) = 4
259   fstat64(4, {st_mode=S_IFREG|0644, st_size=78316, ...}) = 0
259   old_mmap(NULL, 78316, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40019000
259   close(4)                          = 0
259   open("/lib/libc.so.6", O_RDONLY)  = 4
259   read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\324"..., 1024) = 1024
259   fstat64(4, {st_mode=S_IFREG|0755, st_size=1401805, ...}) = 0
259   old_mmap(NULL, 1134564, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x4002d000
259   mprotect(0x40138000, 40932, PROT_NONE) = 0
259   old_mmap(0x40138000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x10a000) = 0x40138000
259   old_mmap(0x4013f000, 12260, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4013f000
259   close(4)                          = 0
259   munmap(0x40019000, 78316)         = 0
259   getpid()                          = 259
259   brk(0)                            = 134525772
259   brk(0x804b374)                    = 134525812
259   brk(0x804c000)                    = 134529024
259   open("/dev/hda5", O_RDWR)         = 4
259   open("/dev/loop0", O_RDWR)        = -1 ENOSYS (Function not implemented)
259   open("/dev/loop0", O_RDWR)        = -1 ENOSYS (Function not implemented)
259   open("/dev/loop0", O_RDWR)        = 5
259   mlockall(0x3, 0xbffff9d3)         = 0
259   ioctl(5, LOOP_SET_FD, 0x4)        = 0
259   ioctl(5, LOOP_SET_STATUS, 0xbffff700) = 0
259   close(5)                          = 0
259   close(4)                          = 0
259   _exit(0)                          = ?
#7 lore raw
In reply to Marc Lehmann
Quoted 1 line · Mike Galbraith

Hmm.. I remember having this problem and it was a problem with strace.

Well, I obviously strace'd it to find out why I get a memory fault without one (I would be happy if it worked without strace ;->)

Quoted 4 lines

Anyway, it works fine here with virgin 2.4.2, so it seems unlikely it's a kernel problem.

259 execve("/sbin/losetup", ["losetup", "/dev/loop0", "/dev/hda5"], [/* 47 vars */]) = 0

The -e switch is causing the memory fault and subsequent breakage:

743   open("/dev/hdd", O_RDWR)          = 4
743   open("/dev/loop0", O_RDWR)        = 5
743   mlockall(0x3, 0x804c272)          = 0
743   ioctl(5, LOOP_SET_FD, 0x4)        = -1 ENOSYS (Function not implemented)
743   ioctl(5, LOOP_SET_FD, 0x4)        = 0
743   ioctl(5, LOOP_SET_STATUS, 0xbffff5d8) = -1 ENOSYS (Function not implemented)
743   ioctl(5, LOOP_SET_STATUS, 0xbffff5d8) = -1 ENOSYS (Function not implemented)
743   ioctl(5, LOOP_SET_STATUS, 0xbffff5d8) = -1 ENOSYS (Function not implemented)
743   ioctl(5, LOOP_SET_STATUS, 0xbffff5d8) = -1 ENOSYS (Function not implemented)
743   ioctl(5, LOOP_SET_STATUS <unfinished ...>
743   +++ killed by SIGSEGV +++

(which is a strange strace anyway...)

However, I just need to wait until there is a new crypto patch (and, if not, I'll eventually have to hack it myself to gte my data. After all it's source... ...)

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#8 lore raw
Quoted 8 lines · Marc Lehmann
Quoted 4 lines · Mike Galbraith

Anyway, it works fine here with virgin 2.4.2, so it seems unlikely it's a kernel problem.

259 execve("/sbin/losetup", ["losetup", "/dev/loop0", "/dev/hda5"], [/* 47 vars */]) = 0

The -e switch is causing the memory fault and subsequent breakage:

No problem here using -e xor. (have no real crypto to try)

Quoted 3 lines

However, I just need to wait until there is a new crypto patch (and, if not, I'll eventually have to hack it myself to gte my data. After all it's source... ...)

Probably.

-Mike