Skip to contentSkip to conversation

Stress testing 2.4.14-pre6

DISCUSSION Bob Matthews 7 participants 9 messages Oct 31, 2001 – Nov 1, 2001

lore
Share & tools
lore
#0 lore raw

Hi Linus,

We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.

Unfortunately, this is as detailed a bug report as I can submit. It looks like Magic SysRq is broken in this kernel. <alt><SysRq>T will print the column headings but nothing else.

Signature · 2 lines
Bob Matthews
Red Hat, Inc.
#1 lore raw
Quoted 3 lines

Unfortunately, this is as detailed a bug report as I can submit. It looks like Magic SysRq is broken in this kernel. <alt><SysRq>T will print the column headings but nothing else.

Thats consistent with memory corruption trashig the task list

#2 lore raw
In reply to Bob Matthews

In article <[email protected]>,

Quoted 6 lines · Bob Matthews

Hi Linus,

We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.

There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).

Working on it, in other words.

Linus

#3 lore raw
Quoted 9 lines · Linus Torvalds

In article <[email protected]>,

Quoted 6 lines · Bob Matthews

Hi Linus,

We've been doing some stress-testing on 2.4.14-pre6 and have encountered a couple of problems. The platform is an 8xPIII with 8G RAM and 32G swap. After running Cerberus for about 3 hours, the machine hung completely. I was not able to switch VC's.

Quoted 8 lines

There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).

Working on it, in other words.

Would you mind to describe this race?

Thanks

#4 lore raw
Quoted 11 lines · Marcelo Tosatti
Quoted 9 lines

There is some race somewhere - I've found one interrupt race (that actually seems to exist in the 2.2.x VM too, but is probably _much_ harder to trigger where an interrupt at _just_ the right time will corrupt the per-process local page list. That looks so unlikely that I doubt that is it, but I'm looking for others (the irq one wasn't even a SMP race - it's on UP too, surprise surprise).

Working on it, in other words.

Would you mind to describe this race?

Both 2.2.x and the new VM (which, through Andrea, has a lot of the same things) have this notion of a per-process "free pages list" that it replenished by any freeing that the process does itself when it gets into the "try_to_free_memory()" path.

The trigger for refilling this list is "current->flags & PF_FREE_PAGES".

The bug is that ytou can be in the middle of adding such a recently free'd page to the per-process list of free pages, and an interrupt comes in.

The interrupt (or bottom half), in turn, might do something like

	page = get_free_page(GFP_ATOMIC);
	...
	free_page(page);

and now the free_page() inside the interrupt context will _also_ trigger the PF_FREE_PAGES test, and _also_ add the page to the list. Except, of course, the list is totally unprotected by any locks, so it may not be valid at this point.

Fix is to only care about the PF_FREE_PAGES bit when not in an interrupt context.

Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.

Linus

#5 lore raw
Quoted 4 lines · Linus Torvalds

Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.

Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#6 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds

Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.

Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.

Cerberus doesn't use networking in the tested setup iirc

#7 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds

Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.

Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.

Which IDE interrupts allocate memory?!

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Jens Axboe
Quoted 6 lines · Jeff Garzik
Quoted 4 lines · Linus Torvalds

Anyway, I seriously doubt this explains any real-world bad behaviour: the window for the interrupt hitting a half-way updated list is something like two instructions long out of the whole memory freeing path. AND most interrupts don't actually do any allocation.

Quoted 4 lines
Quoted 2 lines

Network Rx interrupts do.... definitely not as frequent as IDE interrupts, but not infrequent.

Which IDE interrupts allocate memory?!

Sorry, I meant as in, IDE interrupts occur more frequently than Rx interrupts.

English is my first language... really.

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno