Stress testing 2.4.14-pre6
Hi Linus,
We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.
Unfortunately, this is as detailed a bug report as I can submit. It looks like Magic SysRq is broken in this kernel. <alt><SysRq>T will print the column headings but nothing else.
Signature · 2 lines
Bob Matthews Red Hat, Inc.
Quoted 3 lines
Unfortunately, this is as detailed a bug report as I can submit. It looks like Magic SysRq is broken in this kernel. <alt><SysRq>T will print the column headings but nothing else.
Thats consistent with memory corruption trashig the task list
In article <[email protected]>,
Quoted 6 lines · Bob Matthews
Hi Linus,
We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.
There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).
Working on it, in other words.
Linus
Quoted 9 lines · Linus Torvalds
In article <[email protected]>,
Quoted 6 lines · Bob Matthews
Hi Linus,
We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.
Quoted 8 lines
There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).
Working on it, in other words.
Would you mind to describe this race?
Thanks
Quoted 11 lines · Marcelo Tosatti
Quoted 9 lines
There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).
Working on it, in other words.
Would you mind to describe this race?
Both 2.2.x and the new VM (which, through Andrea, has a lot of the same things) have this notion of a per-process "free pages list" that it replenished by any freeing that the process does itself when it gets into the "try_to_free_memory()" path.
The trigger for refilling this list is "current->flags & PF_FREE_PAGES".
The bug is that ytou can be in the middle of adding such a recently free'd page to the per-process list of free pages, and an interrupt comes in.
The interrupt (or bottom half), in turn, might do something like
page = get_free_page(GFP_ATOMIC); ... free_page(page);
and now the free_page() inside the interrupt context will _also_ trigger the PF_FREE_PAGES test, and _also_ add the page to the list. Except, of course, the list is totally unprotected by any locks, so it may not be valid at this point.
Fix is to only care about the PF_FREE_PAGES bit when not in an interrupt context.
Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.
Linus
Quoted 4 lines · Linus Torvalds
Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.
Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno
Quoted 9 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds
Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.
Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.
Cerberus doesn't use networking in the tested setup iirc
Quoted 8 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds
Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.
Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.
Which IDE interrupts allocate memory?!
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
Quoted 6 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds
Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.
Quoted 4 lines
Quoted 2 lines
Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.
Which IDE interrupts allocate memory?!
Sorry, I meant as in, IDE interrupts occur more frequently than Rx interrupts.
English is my first language... really.
Signature · 3 lines
Jeff Garzik | Only so many songs can be sung Building 1024 | with two lips, two lungs, and one tongue. MandrakeSoft | - nomeansno