2.4.20, .text.lock.swap cpu usage? (ibm x440)
Quoted 15 lines
Quoted 5 lines
Quoted 4 lines
At some point in the past, I wrote:
Either pollwait tables (invisible in 2.4 and 2.5), kernel stacks of threads (which don't get pae_pgd's and are hence invisible in 2.4 and 2.5), or pagecache, with a much higher likelihood of pagecache.Quoted 9 lines
On Thu, Jan 09, 2003 at 06:44:10PM -0600, Brian Tinsley wrote:
The "kernel stacks of threads" may have some bearing on my incarnation of this problem. We have several heavily threaded Java applications running at the time the live-locks occur. At our most problematic site, one application has a bug that can cause hundreds of timer threads (I mean like 800 or so!) to be "accidentally" created. This site is scheduled for an upgrade either tonight or tomorrow, so I will leave the system as it is and see if I can still cause the live-lock to manifest itself after the upgrade.
Quoted 1 line
There is no extant implementation of paged stacks yet.
For the most part, this is probably a boundary condition, right? Anyone that intentionally has 800+ threads in a single application probably needs to reevaluate their design :)
Quoted 3 lines
I'm working on a different problem (
mem_mapon 64GB on 2.5.x). I probably won't have time to implement it in the near future, I probably won't be doing it vs. 2.4.x, and I won't have to if someone else does it first.
Is that a hint to someone in particular?
Signature · 5 lines
-[========================]- -[ Brian Tinsley ]- -[ Chief Systems Engineer ]- -[ Emageon ]- -[========================]-
Quoted 1 line · At some point in the past, I wrote:
Quoted 1 line
There is no extant implementation of paged stacks yet.
Quoted 3 lines · Brian Tinsley
For the most part, this is probably a boundary condition, right? Anyone that intentionally has 800+ threads in a single application probably needs to reevaluate their design :)
IMHO multiprogramming is as valid a use for memory as any other. Or even otherwise, it's not something I care to get in design debates about, it's just how the things are used.
The only trouble is support for what you're doing is unimplemented.
Quoted 4 lines · At some point in the past, I wrote:
Quoted 4 lines
I'm working on a different problem (
mem_mapon 64GB on 2.5.x). I probably won't have time to implement it in the near future, I probably won't be doing it vs. 2.4.x, and I won't have to if someone else does it first.
Quoted 1 line · Brian Tinsley
Is that a hint to someone in particular?
Only you, if anyone. My intentions and patchwriting efforts on the 64GB and highmem multiprogramming fronts are long since public, and publicly stated to be targeted at 2.7. Since there isn't a 2.7 yet, 2.5-CURRENT must suffice until there is.
Bill
Quoted 10 lines · William Lee Irwin III
Quoted 3 lines · At some point in the past, I wrote:
Quoted 3 lines
There is no extant implementation of paged stacks yet.
On Thu, Jan 09, 2003 at 09:17:56PM -0600, Brian Tinsley wrote:
Quoted 8 lines
Quoted 3 lines
For the most part, this is probably a boundary condition, right? Anyone that intentionally has 800+ threads in a single application probably needs to reevaluate their design :)
IMHO multiprogramming is as valid a use for memory as any other. Or even otherwise, it's not something I care to get in design debates about, it's just how the things are used.
I agree with the philosophy in general, but if I sit down to write a threaded application for Linux on IA-32 and wind up with a design that uses 800+ threads in any instance (other than a bug, which was our case), it's time to give up the day job and start riding on the back of the garbage truck ;)
Quoted 2 lines
The only trouble is support for what you're doing is unimplemented.
You mean the 800+ threads or Java on Linux?
Quoted 13 lines
Quoted 6 lines · At some point in the past, I wrote:
Quoted 6 lines
I'm working on a different problem (
mem_mapon 64GB on 2.5.x). I probably won't have time to implement it in the near future, I probably won't be doing it vs. 2.4.x, and I won't have to if someone else does it first.On Thu, Jan 09, 2003 at 09:17:56PM -0600, Brian Tinsley wrote:
Quoted 9 lines
Quoted 3 lines
Is that a hint to someone in particular?
Only you, if anyone. My intentions and patchwriting efforts on the 64GB and highmem multiprogramming fronts are long since public, and publicly stated to be targeted at 2.7. Since there isn't a 2.7 yet, 2.5-CURRENT must suffice until there is.
In all honesty, I would enjoy nothing more than contributing to kernel development. Unfortunately it's a bit out of my scope right now (but not forever). If I only believed aliens seeded our gene pool with clones, I could hook up with those folks that claim to have cloned a human and get one of me made! ;)
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
IMHO multiprogramming is as valid a use for memory as any other. Or even otherwise, it's not something I care to get in design debates about, it's just how the things are used.
Quoted 5 lines · Brian Tinsley
I agree with the philosophy in general, but if I sit down to write a threaded application for Linux on IA-32 and wind up with a design that uses 800+ threads in any instance (other than a bug, which was our case), it's time to give up the day job and start riding on the back of the garbage truck ;)
I could care less what userspace does: mechanism, not policy. Userspace wants, and I give if I can, just as the kernel does with system calls.
800 threads isn't even a high thread count anyway, the 2.5.x testing was with a peak thread count of 100,000. 800 threads, even with an 8KB stack, is no more than 6.4MB of lowmem for stacks and so shouldn't stress the system unless many instances of it are run. I suspect your issue is elsewhere. I'll submit accounting patches for Marcelo's and/or Andrea's trees so you can find out what's actually going on.
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
Only you, if anyone. My intentions and patchwriting efforts on the 64GB and highmem multiprogramming fronts are long since public, and publicly stated to be targeted at 2.7. Since there isn't a 2.7 yet, 2.5-CURRENT must suffice until there is.
Quoted 5 lines · Brian Tinsley
In all honesty, I would enjoy nothing more than contributing to kernel development. Unfortunately it's a bit out of my scope right now (but not forever). If I only believed aliens seeded our gene pool with clones, I could hook up with those folks that claim to have cloned a human and get one of me made! ;)
I don't know what to tell you here. I'm lucky that this is my day job and that I can contribute so much. However, there are plenty who contribute major changes (many even more important than my own) without any such sponsorship. Perhaps emulating them would satisfy your wish.
Bill
Quoted 16 lines · William Lee Irwin III
Quoted 5 lines · William Lee Irwin III
Quoted 5 lines
IMHO multiprogramming is as valid a use for memory as any other. Or even otherwise, it's not something I care to get in design debates about, it's just how the things are used.
Quoted 3 lines · Brian Tinsley
I agree with the philosophy in general, but if I sit down to write a threaded application for Linux on IA-32 and wind up with a design that uses 800+ threads in any instance (other than a bug, which was our case), it's time to give up the day job and start riding on the back of the garbage truck ;)
Quoted 8 lines
I could care less what userspace does: mechanism, not policy. Userspace wants, and I give if I can, just as the kernel does with system calls.
800 threads isn't even a high thread count anyway, the 2.5.x testing was with a peak thread count of 100,000. 800 threads, even with an 8KB stack, is no more than 6.4MB of lowmem for stacks and so shouldn't stress the system unless many instances of it are run.
I understand your perspective here. I won't get into application design issues as it is far out of context from this list.
Quoted 2 lines
I suspect your issue is elsewhere. I'll submit accounting patches for Marcelo's and/or Andrea's trees so you can find out what's actually going on.
Much appreciated! I look forward to it.
Quoted 12 lines · Brian Tinsley
Quoted 5 lines
In all honesty, I would enjoy nothing more than contributing to kernel development. Unfortunately it's a bit out of my scope right now (but not forever). If I only believed aliens seeded our gene pool with clones, I could hook up with those folks that claim to have cloned a human and get one of me made! ;)
I don't know what to tell you here. I'm lucky that this is my day job and that I can contribute so much. However, there are plenty who contribute major changes (many even more important than my own) without any such sponsorship. Perhaps emulating them would satisfy your wish.
It would!
I cannot say thanks enough for the efforts of you and everyone else out there. Frankly, I would not have my day job and would not have been able to make Emageon what it is today were it not for you all!
Oh, please excuse the stupid humor tonight. I'm in a giddy mood for some reason. Must be the excitement from the prospect of getting resolution to this problem!
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines
I don't know what to tell you here. I'm lucky that this is my day job and that I can contribute so much. However, there are plenty who contribute major changes (many even more important than my own) without any such sponsorship. Perhaps emulating them would satisfy your wish.
Quoted 7 lines · Brian Tinsley
It would!
I cannot say thanks enough for the efforts of you and everyone else out there. Frankly, I would not have my day job and would not have been able to make Emageon what it is today were it not for you all! Oh, please excuse the stupid humor tonight. I'm in a giddy mood for some reason. Must be the excitement from the prospect of getting resolution to this problem!
We're straying from the subject here. Please describe your machine, in terms of how many cpus it has and how much highmem it has, and your workload, so I can better determine the issue. Perhaps we can cooperatively devise something that works well for you.
Or perhaps the kernel version is not up-to-date. Please also provide the precise kernel version (and included patches). And workload too.
Thanks,
Bill
Quoted 4 lines
We're straying from the subject here.
Sorry
Quoted 5 lines
Please describe your machine,
in terms of how many cpus it has and how much highmem it has, and your workload, so I can better determine the issue. Perhaps we can cooperatively devise something that works well for you.
IBM x360
Pentium 4 Xeon MP processors
2 processor system has 4GB RAM 4 processor system has 8GB RAM
1 IBM ServeRAID controller 2 Intel PRO/1000MT NICs
2 QLogic 2340 Fibre Channel HBAs
Quoted 3 lines
Or perhaps the kernel version is not up-to-date. Please also provide the precise kernel version (and included patches). And workload too.
The kernel version is stock 2.4.20 with Chris Mason's data logging and journal relocation patches for ReiserFS (neither of which are actually in use for any mounted filesystems). It is compiled for 64GB highmem support. And just to refresh, I have seen this exact behavior on stock 2.4.19 and stock 2.4.17 (no patches on either of these) also compiled with 64GB highmem support.
Workload:
When the live-lock occurs, the system is performing intensive network I/O and intensive disk reads from the fibre channel storage (i.e., the backup program is reading files from disk and transferring them to the backup server). I posted a snapshot of sar data collection earlier today showing selected stats leading up to and just after the live-lock occurs (which is noted by a ~2 minute gap in sar logging). After the live-lock is released, the only thing that stands out is an unusual increase in runtime for kswapd (as reported by ps).
The various Java programs mentioned in prior postings are *mostly* idle at this point in time as it is after hours for our clients.
Quoted 13 lines
IBM x360
Pentium 4 Xeon MP processors2 processor system has 4GB RAM 4 processor system has 8GB RAM
1 IBM ServeRAID controller 2 Intel PRO/1000MT NICs
2 QLogic 2340 Fibre Channel HBAsQuoted 3 lines
Or perhaps the kernel version is not up-to-date. Please also provide the precise kernel version (and included patches). And workload too.
Quoted 6 lines
The kernel version is stock 2.4.20 with Chris Mason's data logging and journal relocation patches for ReiserFS (neither of which are actually in use for any mounted filesystems). It is compiled for 64GB highmem support. And just to refresh, I have seen this exact behavior on stock 2.4.19 and stock 2.4.17 (no patches on either of these) also compiled with 64GB highmem support.
Workload:
When the live-lock occurs, the system is performing intensive network I/O and intensive disk reads from the fibre channel storage (i.e., the backup program is reading files from disk and transferring them to the backup server). I posted a snapshot of sar data collection earlier today showing selected stats leading up to and just after the live-lock occurs (which is noted by a ~2 minute gap in sar logging). After the live-lock is released, the only thing that stands out is an unusual increase in runtime for kswapd (as reported by ps).The various Java programs mentioned in prior postings are *mostly* idle at this point in time as it is after hours for our clients.
If you don't have any individual processes that need to be particularly large (eg > 1Gb of data), I suggest you just cheat^Wfinesse the problem and move PAGE_OFFSET from C0000000 to 80000000 - will give you more than twice as much lowmem to play with. I think this might even be a config option in RedHat kernels.
Martin.
Quoted 2 lines · At some point in the past, I wrote:
Quoted 2 lines
Or perhaps the kernel version is not up-to-date. Please also provide the precise kernel version (and included patches). And workload too.
Quoted 6 lines · Brian Tinsley
The kernel version is stock 2.4.20 with Chris Mason's data logging and journal relocation patches for ReiserFS (neither of which are actually in use for any mounted filesystems). It is compiled for 64GB highmem support. And just to refresh, I have seen this exact behavior on stock 2.4.19 and stock 2.4.17 (no patches on either of these) also compiled with 64GB highmem support.
Okay, can you try with either 2.4.x-aa or 2.5.x-CURRENT?
I'm suspecting either bh problems or lowpte problems.
Also, could you monitor your load with the scripts I posted?
Thanks,
Bill
Quoted 4 lines
Okay, can you try with either 2.4.x-aa or 2.5.x-CURRENT?
Yes, I *just* booted a machine with 2.4.20-aa1 in our lab. I was having problems compiling the Linux Virtual Server code, but it's fixed now.
Quoted 5 lines
I'm suspecting either bh problems or lowpte problems.
Also, could you monitor your load with the scripts I posted?
Yes, they are already uploaded to a customer site and ready to go. I need to flex the -aa1 kernel a bit before I load it there as well.
Thanks!