Skip to contentSkip to conversation

2.4.4 kernel freeze for unknown reason

DISCUSSION Jacky Liu 5 participants 11 messages May 10, 2001 – May 14, 2001

lore
Share & tools
lore

Jacky Liu

author
#0 lore raw

Dear All,

I would like to post a question regarding to a problem of unknown freeze of my linux firewall/gateway.

Here is the hardware configuration of my machine:

AMD K-6 233 MHz
2theMax P-55 VP3 mobo
64Mb RAM in a single module (PC-100)
Maxtor 6G UDMA-33 harddisk
Matrox MG-II display card w/8Mb RAM
3Com 3C905B-TX NIC
RealTek 8129 10/100 NIC

It's running 2.4.4 kernel (RedHat 7.1) and acting as a firewall using Netfilter (gShield and Snort), DNS (Cache-Only DNS) and NAT gateway (ip-masq.) for my home network. I used 3C905B-TX NIC as my internal NIC and RealTek 8129 as my external NIC. Here is the problem:

The machine has been randomly lockup (totally freeze) for number of times without any traceable clue or error message. Usually the time frame between each lockup is between 24 to 72 hours. The screen just freeze when it's lockup (either in Console or X) and no "kernel panic" type or any error message prompt up. All services (SSH, DNS, etc..) are dead when it's lockup and I cannot find any useful information in /var/log/messages. I cannot reproduce the lockup since it's totally randomly. The lockup happened either when I was playing online game (A LOT, like getting thousands of server status in counter-strike in a very short time frame, of NAT traffic), surfing the web (normal traffic) or the machine was totally in idle (lockup when I was sleeping). It was lockup this morning when I was playing online game (when my game machine was trying to establish connection to a game server).

If there is any information you would like to obtain, please let me know. I would like to receive a copy of your reply, thank you very much for your kindly attention.

Best Regards,
Jacky Liu

"Genius or Wacko? Majority or Minority� "

------------------------------------------------------------
--== Sent via Deja.com ==--
http://www.deja.com/

#1 lore raw
Quoted 34 lines · Jacky Liu

Dear All,

I would like to post a question regarding to a problem of unknown freeze of my linux firewall/gateway.

Here is the hardware configuration of my machine:

AMD K-6 233 MHz
2theMax P-55 VP3 mobo
64Mb RAM in a single module (PC-100)
Maxtor 6G UDMA-33 harddisk
Matrox MG-II display card w/8Mb RAM
3Com 3C905B-TX NIC
RealTek 8129 10/100 NIC

It's running 2.4.4 kernel (RedHat 7.1) and acting as a firewall using Netfilter (gShield and Snort), DNS (Cache-Only DNS) and NAT gateway (ip-masq.) for my home network. I used 3C905B-TX NIC as my internal NIC and RealTek 8129 as my external NIC. Here is the problem:

The machine has been randomly lockup (totally freeze) for number of times without any traceable clue or error message. Usually the time frame between each lockup is between 24 to 72 hours. The screen just freeze when it's lockup (either in Console or X) and no "kernel panic" type or any error message prompt up. All services (SSH, DNS, etc..) are dead when it's lockup and I cannot find any useful information in /var/log/messages. I cannot reproduce the lockup since it's totally randomly. The lockup happened either when I was playing online game (A LOT, like getting thousands of server status in counter-strike in a very short time frame, of NAT traffic), surfing the web (normal traffic) or the machine was totally in idle (lockup when I was sleeping). It was lockup this morning when I was playing online game (when my game machine was trying to establish connection to a game server).

Quoted 4 lines

If there is any information you would like to obtain, please let me know. I would like to receive a copy of your reply, thank you very much for your kindly attention.

I have been experiencing these same problems since version 2.4.0. Although, I think it has improved a little in 2.4.4, it still locks up. The problem seems to be related to memory management and/or swap, and is seems to do it primarily on machines with over 128Mb of RAM. Although, I have not tested systematically enough to confirm this.

I have been monitoring the memory usage constantly with the gnome memory usage meter and noticed that as swap grows it is never freed back up. I can kill off most of the large applications, such as netscape, xemacs, etc, and little or no memory and swap will be freed. Once swap is full after a few days, my machine will lock up.

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

I am running on an AMD K6-400 with 256 Mb RAM but we have experienced these problems with various other machines as well.

I hope this information is helpfull in tracking down the problem.

The features of the 2.4.x kernels are great and I believe all the kernel developers have done a fantastic job! However, I am disappointed that we are now on the forth 2.4.x kernel version and such as serious problem that has been there since 2.4.0 still exists. This is pretty much a show stopper for having a production machine. In the past when I was running on 2.0.3x kernels, I boasted to everybody about how rock solid Linux was and I was able to run for 5 months without a single reboot or problem, but for the last year or more I have not had much better uptime than certain other commercial operating systems. As important as the new features are, stability should be top priority for production systems. I apologize if this sounds a bit like a rant but I feel that ever since the 2.2.x kernel series, the Linux kernel community has veered away from it's original philosophy of only releasing stable kernels with even numbered versions. In fact, I have seen several even numbered kernels released with far less stability than most of the odd numbered development kernels.

- Vincent

#2 lore raw
Quoted 3 lines

I have been monitoring the memory usage constantly with the gnome memory usage meter and noticed that as swap grows it is never freed back up. I can kill off most of the large applications, such as

The swap handling in 2.4 is somewhat hosed at the moment.

Quoted 3 lines

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

That sounds right. I can give you a tiny patch that should fix the lockups and instead it will kill processes out of memory but thats obviously not the actual fix 8)

#3 lore raw
Quoted 3 lines · Alan Cox
Quoted 3 lines

I have been monitoring the memory usage constantly with the gnome memory usage meter and noticed that as swap grows it is never freed back up. I can kill off most of the large applications, such as

I've seen this mentioned a few times now and am curious enough to ask.

Quoted 5 lines

The swap handling in 2.4 is somewhat hosed at the moment.

Quoted 3 lines

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

Why do I not see this behavior with a heavy swap throughput test load? It seems decidedly odd to me that swapspace should remain allocated on other folks lightly loaded boxen given that my heavily loaded box does release swapspace quite regularly. What am I missing?

-Mike

#4 lore raw
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

Why do I not see this behavior with a heavy swap throughput test load? It seems decidedly odd to me that swapspace should remain allocated on other folks lightly loaded boxen given that my heavily loaded box does release swapspace quite regularly. What am I missing?

If you swap really hard it seems much happier. If you vaguely swap stuff out over time then I too see the description above only I have Rik's dont deadlock on oom tweak so I see apps die.

Alan

#5 lore raw
Quoted 4 lines · Alan Cox
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

Quoted 8 lines
Quoted 4 lines

Why do I not see this behavior with a heavy swap throughput test load? It seems decidedly odd to me that swapspace should remain allocated on other folks lightly loaded boxen given that my heavily loaded box does release swapspace quite regularly. What am I missing?

If you swap really hard it seems much happier. If you vaguely swap stuff out over time then I too see the description above only I have Rik's dont deadlock on oom tweak so I see apps die.

Does any swap write/release if you hit such a box with heavy duty IO? (pages on dirty list, swapspace allocated but writeout defered?)

If not, I'd be interested in seeing sysrq-m of a box in such a state.. particularly so if the total pages on active, dirty and clean lists is only a small fraction of total pages. (information leak?)

-Mike

#6 lore raw
Quoted 2 lines

Does any swap write/release if you hit such a box with heavy duty IO? (pages on dirty list, swapspace allocated but writeout defered?)

Hard to tell. I switched my desktop box back to 2.2 a while back until the VM works.

Alan

#7 lore raw
Quoted 5 lines · Alan Cox
Quoted 2 lines

Does any swap write/release if you hit such a box with heavy duty IO? (pages on dirty list, swapspace allocated but writeout defered?)

Hard to tell. I switched my desktop box back to 2.2 a while back until the VM works.

I should have reversed to/cc.. my question was intended for the list.

-Mike

#8 lore raw
In reply to Mike Galbraith

Hi Mike,

Quoted 5 lines · Mike Galbraith

Why do I not see this behavior with a heavy swap throughput test load? It seems decidedly odd to me that swapspace should remain allocated on other folks lightly loaded boxen given that my heavily loaded box does release swapspace quite regularly. What am I missing?

Are you using a database or something other which mostly uses shared mem/tmpfs? This does reclaim swap space on swap in.

Greetings
Christoph

#9 lore raw
Quoted 11 lines · Christoph Rohland

Hi Mike,

Quoted 5 lines · Mike Galbraith

Why do I not see this behavior with a heavy swap throughput test load? It seems decidedly odd to me that swapspace should remain allocated on other folks lightly loaded boxen given that my heavily loaded box does release swapspace quite regularly. What am I missing?

Are you using a database or something other which mostly uses shared mem/tmpfs? This does reclaim swap space on swap in.

No, just doing parallel kernel builds.

-Mike

#10 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 3 lines

I have been monitoring the memory usage constantly with the gnome memory usage meter and noticed that as swap grows it is never freed back up. I can kill off most of the large applications, such as

The swap handling in 2.4 is somewhat hosed at the moment.

Quoted 7 lines
Quoted 3 lines

If I turn swap off all together or turn it off and back on periodically to clear the swap before it gets full, I do not seem to experience the lockups.

That sounds right. I can give you a tiny patch that should fix the lockups and instead it will kill processes out of memory but thats obviously not the actual fix 8)

I am not sure if you got my reply to this Alan, but I would like to have you send me the patch. Thanks.

Also, I got to thinking back and it dawned on me that there was another significant kernel change when we started experiencing the freezes from the vm problems. We changed the compiler from gcc-2.7.2.3 to egcs-1.1.2 starting with kernel 2.4.0-test10 (That is the version in which gcc-2.7.2.3 stopped being supported for the kernel) and it seems like that was just about the same time frame that we started experiencing the vm/swap problems. I would have to go back and run on the test10 kernel for a while to confirm if it started with it or 2.4.0.

I am assuming the source of the vm problem is still unknown or it would have been fixed by now. Is it possible that it could be related to the compiler change? If nobody has considered that, I just thought I would bring the possibility to your attention.

- Vincent Stemen