2.4.20-aa and LARGE Squid process -> SIGSEGV
Hi!
Right now we're trying to implement a large scale Squid proxy on Debian/testing. We're using 2.4.20-aa and Squid-2.4.7-1.
We're encountering sporadic crashes of the squid children (SIGSEGV, signal 11). We were investigating in several directions:
* the Kernel has highmem support enabled (we have 2GB RAM and 4 GB swap)
* we tried different kernels (with or without highmem support)
* we tried another squid version (Squid-2.4.6 from the "stable" distribution of Debian)
* We recompiled Squid using gcc-3.2, since -- according to the FAQ -- squid may crash with signal 11 with optimization enabled when using gcc-2.95.4 (Debian uses gcc-2.95.4, but still build squid using -O!)
* we closely observed dmesg, messages and syslog. No oddities were found. Squid simply crashes.
* we tried both ufs and aufs as cache filesystems, since the FAQ tells us the async I/O may have bugs.
* We use two identical machines -- the crash happens on both machines. Same CPU, disks, RAM, manufacturer, heck -- even the same series!
To no avail -- Squid simply crashes from time to time. It's impossible to predict when.
Then we wrote a program which allocates large amounts of memory:
--- snip ---
#include <stdlib.h>
#include <stdio.h>
main(){
char *buf;
long c;
FILE *fp;
fp = fopen("/dev/null","a");
while(1){
buf = (char *)malloc(100000000);
c = random();
if (c > 100000000)
continue;
fprintf(fp,"%c",buf[c]);
printf("hier\n"); }
}
--- snip ---
And we found that this program will be killed with a SIGSEGV as well.
So, what are we doing wrong? Since we have 2GB RAM and 4GB swap, shouldn't the machine start to swap and execute the program above successfully?
# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 2119065600 2111709184 7356416 0 299008 1438240768
Swap: 4293509120 0 4293509120
MemTotal: 2069400 kB
MemFree: 7184 kB
MemShared: 0 kB
Buffers: 292 kB
Cached: 1404532 kB
SwapCached: 0 kB
Active: 387796 kB
Inactive: 1317064 kB
HighTotal: 1703872 kB
HighFree: 2960 kB
LowTotal: 365528 kB
LowFree: 4224 kB
SwapTotal: 4192880 kB
SwapFree: 4192880 kB
BigFree: 0 kB# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) CPU 2.80GHz stepping : 7 cpu MHz : 2785.629 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm bogomips : 5557.45
# cat /proc/swaps Filename Type Size Used Priority /dev/sda6 partition 2096440 0 -1 /dev/sda7 partition 2096440 0 -2
Signature · 8 lines
Ralf Hildebrandt (Im Auftrag des Referat V a) [email protected] Charite Campus Mitte Tel. +49 (0)30-450 570-155 Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916 I've got the perfect system. I never need to do maintenance on it, or software upgrades, patches, or anything. It's great. It never wakes me up, or gets hacked into. It's completely perfect. That was the first step in my plan to build the perfect Postfix system. The second step is to plug it in.
Quoted 1 line · On 2002.12.20 Ralf Hildebrandt wrote:
Hi!
[real problem snipped]
Quoted 20 lines
Then we wrote a program which allocates large amounts of memory:
--- snip ---
#include <stdlib.h>
#include <stdio.h>main(){
char *buf;
long c;
FILE *fp;fp =
fopen("/dev/null","a");while(1){ buf = (char *)malloc(100000000); c = random(); if (c > 100000000) continue; fprintf(fp,"%c",buf[c]); printf("hier\n");
Quoted 6 lines
}
}
--- snip ---And we found that this program will be killed with a SIGSEGV as well.
Normal. You are running OOM. Look at what you do:
while (1)
{
malloc(much mem)
// do not free the mem !!!!
}So in a couple steps you are OOM.
I suppose what you want to do is
buf = malloc(...)
while (1)
touch random pageBut...you 'touch' is read-only (the printf), so the page will never really be allocated. Try with this:
#include <stdlib.h>
// 4Gb
#define SZ 4*1024*1024*1024
main(){
char *buf;
buf = malloc(SZ);
if (!buf)
{
perror("bad try");
exit(1);
}
while(1){
buf[random()%SZ] = 0;
}
}Ah, with 2Gb of ram you will need to compile with 3Gb userspace, to let a one only process allocate a chunk of mem that does not fit into core memory. Or, easier, run several instances...
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.20-jam2 (gcc 3.2 (Mandrake Linux 9.1 3.2-4mdk))
* J.A. Magallon <[email protected]>:
Quoted 1 line
Normal. You are running OOM. Look at what you do:
But that manifests itself in a segfault? The kernel never logs an OOM in the logs.
Quoted 3 lines
Ah, with 2Gb of ram you will need to compile with 3Gb userspace, to let a one only process allocate a chunk of mem that does not fit into core memory. Or, easier, run several instances...
Right now we're running a Xeon with 2GB - from the menuconfig:
(4GB) High Memory Support
This was based on the help in the kernel config:
"If the machine has between 1 and 4 Gigabytes physical RAM, then answer "4GB" here."
(3.5GB) User address space size
[*] HIGHMEM I/O support
dmesg says:
Linux version 2.4.20aa1 (root@spidergirl) (gcc version 2.95.4 20011002 (Debian prerelease)) #1 Thu Dec 19 14:35:35 CET 2002 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
BIOS-e820: 000000007fff0000 - 000000007fffec00 (ACPI data)
BIOS-e820: 000000007fffec00 - 000000007ffff000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
1663MB HIGHMEM available.
384MB LOWMEM available.
...
Calibrating delay loop... 5557.45 BogoMIPS
Memory: 2069264k/2097088k available (1659k kernel code, 27440k reserved, 651k data, 136k init, 1703872k highmem)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes) Inode cache hash table entries: 131072 (order: 8, 1048576 bytes) Mount-cache hash table entries: 32768 (order: 6, 262144 bytes) Buffer-cache hash table entries: 131072 (order: 7, 524288 bytes) Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
CPU: L1 I cache: 0K, L1 D cache: 8K
CPU: L2 cache: 512K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After generic, caps: bfebfbff 00000000 00000000 00000000 CPU: Common caps: bfebfbff 00000000 00000000 00000000 CPU: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
...
Signature · 4 lines
Ralf Hildebrandt (Im Auftrag des Referat V a) [email protected] Charite Campus Mitte Tel. +49 (0)30-450 570-155 Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916 "All programmers are playwrights and all computers are lousy actors." -Anon.
Quoted 7 lines · On 2002.12.20 Ralf Hildebrandt wrote:
* J.A. Magallon <[email protected]>:
Quoted 1 line
Normal. You are running OOM. Look at what you do:
But that manifests itself in a segfault? The kernel never logs an OOM in the logs.
Well, I used the OOM term, but I suspect it 'technically' only applies to the situation when a kernel thread needs memory to do its work, and it can not allocate it. The system can not continue without that memory, you are OOM, and the OOM killer will begin to kill other things.
For user space memory, there is no real OOM state. The system (glibc) just does not give you the memory, returns NULL in the malloc, and it is your responsibility to check malloc's return value. If you do not check it, you try to access a null pointer and _bang_. So in your case, after enough iterations on malloc() without free(), it returns NULL and you fall into a null pointer dereference.
AFAIK.
Signature · 4 lines
J.A. Magallon <[email protected]> \ Software is like sex: werewolf.able.es \ It's better when it's free Mandrake Linux release 9.1 (Cooker) for i586 Linux 2.4.20-jam2 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-2mdk))
* J.A. Magallon <[email protected]>:
Quoted 6 lines
For user space memory, there is no real OOM state. The system (glibc) just does not give you the memory, returns NULL in the malloc, and it is your responsibility to check malloc's return value. If you do not check it, you try to access a null pointer and _bang_. So in your case, after enough iterations on
malloc()withoutfree(), it returns NULL and you fall into a null pointer dereference.
Ergo: Squid is br0ken.
Signature · 5 lines
Ralf Hildebrandt (Im Auftrag des Referat V a) [email protected] Charite Campus Mitte Tel. +49 (0)30-450 570-155 Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916 Windows is the answer, but only if the question was 'what is the intellectual equivalent of being a galley slave?'
No, squid is not br0ken in this fashion. If squid cannot be allocated enough memory by the system, it logs a message and _dies_. Relevant files to look at in your squid source are squid/lib/util.c for xcalloc() and xmalloc().
Aside from this, if squid ever does get to the point of swapping, it is misconfigured and your performance has just gone to hell anyway... (see the FAQ at www.squid-cache.org)
Questions relating to squid performance and stability should really be discussed on the squid users mailing list, you can find details on this list at http://www.squid-cache.org/mailing-lists.html
Reuben
Quoted 23 lines · Ralf Hildebrandt
* J.A. Magallon <[email protected]>:
Quoted 6 lines
For user space memory, there is no real OOM state. The system (glibc) just does not give you the memory, returns NULL in the malloc, and it is your responsibility to check malloc's return value. If you do not check it, you try to access a null pointer and _bang_. So in your case, after enough iterations on
malloc()withoutfree(), it returns NULL and you fall into a null pointer dereference.Ergo: Squid is br0ken.
--
Ralf Hildebrandt (Im Auftrag des Referat V a) [email protected] Charite Campus Mitte Tel. +49 (0)30-450 570-155 Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916Windows is the answer, but only if the question was 'what is the intellectual equivalent of being a galley slave?'
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
* Reuben Farrelly <[email protected]>:
Quoted 4 lines
No, squid is not br0ken in this fashion. If squid cannot be allocated enough memory by the system, it logs a message and _dies_. Relevant files to look at in your squid source are
squid/lib/util.cforxcalloc()andxmalloc().
Why can't squid allocate more than 1GB on a system with 2GB RAM?
Quoted 3 lines
Aside from this, if squid ever does get to the point of swapping, it is misconfigured and your performance has just gone to hell anyway... (see the FAQ at www.squid-cache.org)
It's not swapping. That's the whole point. We have 2GB and can use at most 1GB for Squid.
Signature · 6 lines
Ralf Hildebrandt (Im Auftrag des Referat V a) [email protected] Charite Campus Mitte Tel. +49 (0)30-450 570-155 Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916 Microsoft: "Where do you want to go today?" Linux: "Where do you want to be tomorrow?" BSD: "Are you guys coming, or what?"
[Forwarded for the purposes of continuity, Robert is not on lkml]
Quoted 30 lines
Subject: Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV
From: Robert Collins <[email protected]>
To: Reuben Farrelly <[email protected]>
Cc: Ralf Hildebrandt <[email protected]>
X-Mailer: Ximian Evolution 1.0.8
Date: 21 Dec 2002 20:12:21 +1100Quoted 4 lines · Ralf Hildebrandt
* Reuben Farrelly <[email protected]>:
Quoted 2 lines
No, squid is not br0ken in this fashion. If squid cannot be allocated enough memory by the system, it logs a message and _dies_. Relevant
files
Quoted 4 lines
Quoted 2 lines
to look at in your squid source are
squid/lib/util.cforxcalloc()andxmalloc().Why can't squid allocate more than 1GB on a system with 2GB RAM?
In general, it can.
Folk run it with up to 2Gb with no special options on linux, all the time.Have you tried asking on the *right* list?
Or running it under gdb?
Or getting a stacktrace?SEGSIGV does *not* mean out of memory, you *may* have hit a genuine bug, but it *not* due to squid handling a failed malloc wrongly.
Rob