Linux-2.4.5
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Andrea Arcangeli
I don't see where you fixed the deadlock in
create_buffers, if you did please show me which line of code is supposed to do that, I show you below which lines of code in my patch should fix thewait_eventdeadlock increate_buffers:
Andrea, look at the page_alloc() path, and the "don't loop forever if __GFP_IO isn't set and we're not making progress". That looks entirely sane.
It's the other limiting that I don't think really addresses the problem (and I like your patch that removes some more magic limits - I suspect the proper fix is the 5 lines from Rik's patch in page_alloc.c, and your patch together - amybody mind testing that out?)
Oh, and I still _do_ think that we should rename the silly "async" flag as "can_do_io", and then use that to determine whether to do SLAB_KERNEL or SLAB_BUFFER. That would make more things able to do IO, which in turn should help balance things out.
Linus "leaving for the airport" Torvalds
Quoted 4 lines · Linus Torvalds
Oh, and I still _do_ think that we should rename the silly "async" flag as "
can_do_io", and then use that to determine whether to doSLAB_KERNELorSLAB_BUFFER. That would make more things able to do IO, which in turn should help balance things out.
Agreed, this simplifies things a lot.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
Quoted 4 lines · Andrea Arcangeli
(quoted diff) +0
-0@@ -1416,11 +1416,9 @@ */ run_task_queue(&tq_disk);
Review line 1423
Quoted 10 lines
(quoted diff) +3
-5- /* - * Set our state for sleeping, then check again for buffer heads. - * This ensures we won't miss a wake_up from an interrupt. - */ - wait_event(buffer_wait, nr_unused_buffer_heads >= MAX_BUF_PER_PAGE); + current->policy |= SCHED_YIELD; + __set_current_state(TASK_RUNNING); + schedule(); goto try_again; }
This cannot possibly fix the problem because this code is never reached.
What was observed in the backtraces by arjan, ben, marcelo and people at IBM was:
create_buffers -> get_unused_buffer_head -> __alloc_pages
with the system looping infinitely in __alloc_pages. The code you are changing above ONLY gets reached in case the __alloc_pages (and thus, get_unused_buffer_head) returns failure.
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
Quoted 23 lines · Rik van Riel
Quoted 4 lines · Andrea Arcangeli
(quoted diff) +0
-0@@ -1416,11 +1416,9 @@ */ run_task_queue(&tq_disk);Review line 1423
Quoted 10 lines
(quoted diff) +3
-5- /* - * Set our state for sleeping, then check again for buffer heads. - * This ensures we won't miss a wake_up from an interrupt. - */ - wait_event(buffer_wait, nr_unused_buffer_heads >= MAX_BUF_PER_PAGE); + current->policy |= SCHED_YIELD; + __set_current_state(TASK_RUNNING); + schedule(); goto try_again; }This cannot possibly fix the problem because this code is never reached.
What was observed in the backtraces by arjan, ben, marcelo and people at IBM was:
Quoted 6 lines
create_buffers->get_unused_buffer_head->__alloc_pageswith the system looping infinitely in
__alloc_pages. The code you are changing above ONLY gets reached in case the__alloc_pages(and thus,get_unused_buffer_head) returns failure.
Fine, then post the strict __alloc_pages patch, after that you will run into the above code. Those are different issues, like I'm claiming since the first place, your patch didn't addressed the above.
I definitely agree that if __alloc_pages itself deadlocks the above cannot make differences.
Andrea
Quoted 10 lines · Andrea Arcangeli
Quoted 10 lines
Review
Quoted 10 lines
(quoted diff) +3
-5- /* - * Set our state for sleeping, then check again for buffer heads. - * This ensures we won't miss a wake_up from an interrupt. - */ - wait_event(buffer_wait, nr_unused_buffer_heads >= MAX_BUF_PER_PAGE); + current->policy |= SCHED_YIELD; + __set_current_state(TASK_RUNNING); + schedule(); goto try_again; }
I'm still curious ... is it _really_ needed to busy-spin here?
I've seen some big problems with processes eating CPU time while not getting any work done and am slowly trying to eliminate those places in the VM by waiting on things.
Is it really needed to introduce a new busy-wait place in the kernel?
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
Quoted 14 lines · Rik van Riel
Quoted 10 lines · Andrea Arcangeli
Quoted 10 lines
Review
Quoted 10 lines
(quoted diff) +3
-5- /* - * Set our state for sleeping, then check again for buffer heads. - * This ensures we won't miss a wake_up from an interrupt. - */ - wait_event(buffer_wait, nr_unused_buffer_heads >= MAX_BUF_PER_PAGE); + current->policy |= SCHED_YIELD; + __set_current_state(TASK_RUNNING); + schedule(); goto try_again; }I'm still curious ... is it _really_ needed to busy-spin here?
As said we cannot wait_event, because those reserved bh will be released only by the VM if there is memory pressure. If we sleep in wait_event while I/O is in progress on the reserved bh and a big chunk of memory is released under us, then those reserved bh may never get released and we will hang in wait_event until there's memory pressure again, so unless we implement another logic we need to somehow poll there and to try to allocate again later. We should enter that path very seldom so I'm not very concerned about the performance during the polling loop.
Andrea
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Andrea Arcangeli
I don't see where you fixed the deadlock in
create_buffers, if you did please show me which line of code is supposed to do that, I show you below which lines of code in my patch should fix thewait_eventdeadlock increate_buffers:Andrea, look at the
page_alloc()path, and the "don't loop forever if__GFP_IOisn't set and we're not making progress". That looks entirely sane.
yes, I was only talking about create_buffers, not __alloc_pages. That patch can certainly address problems in alloc_pages.
Quoted 3 lines
(and I like your patch that removes some more magic limits - I suspect the proper fix is the 5 lines from Rik's patch in
page_alloc.c, and your patch together - amybody mind testing that out?)
Sounds the same to me.
Quoted 4 lines
Oh, and I still _do_ think that we should rename the silly "async" flag as "
can_do_io", and then use that to determine whether to doSLAB_KERNELorSLAB_BUFFER. That would make more things able to do IO, which in turn should help balance things out.
getblk still needs to use SLAB_BUFFER, not sure how many callers will be allowed to use SLAB_KERNEL, but certainly the "async" name was not very appropriate to indicate if the bh allocation can fail or not.
Andrea
Quoted 4 lines · Andrea Arcangeli
getblk still needs to use
SLAB_BUFFER, not sure how many callers will be allowed to useSLAB_KERNEL, but certainly the "async" name was not very appropriate to indicate if the bh allocation can fail or not.
Note that these days, on reasonable filesystems, getblk() and friends are only used by meta-data. So on a normal setup that uses the page cache for data (pretty much everything), you'd probably go from "100% SLAB_BUFFER" to "less than 10% SLAB_BUFFER".
Which should cut down on the "this can happen under real load" stuff.
Assuming, of course, that there aren't any other causes. I can imagine schenarios where the buffer heads are the major cause of problems, and the fact that Rik special-cased GFP_BUFFER makes me suspect that that is _his_ schenario, but there may be other, less obvious, ways to get into similar trouble.
MM is hard. No question about it.
Linus
Quoted 1 line · Ben LaHaise
No. It does not fix the deadlock. Neither does the patch you posted.
can you give a try if you can deadlock 2.4.5aa1 just in case, and post a SYSRQ+T + system.map if it still deadlocks?
Quoted 2 lines
But, if you're going to add a reserve pool for buffer heads and bounce pages, please do it with generic, not yet another special case hack.
The reserved pool for buffer heads is there since the first time I used linux I think. I won't certainly object to convert all reserved pool to an unified interface, as I'd like if all hashtables would be also sized with an unified interface, but that's just a stylistic issue, at least for 2.4 that's a very secondary object compared to people who can get dealdocks during production.
Andrea
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line · Ben LaHaise
No. It does not fix the deadlock. Neither does the patch you posted.
can you give a try if you can deadlock 2.4.5aa1 just in case, and post a SYSRQ+T + system.map if it still deadlocks?
Andrea, can you rather start running the Cerberus testsuite instead? All these deadlocks happen pretty early during the test, and we've been fixing tons of these deadlocks, and no, it's not easy.
Ingo
Quoted 1 line · Ingo Molnar
Andrea, can you rather start running the Cerberus testsuite instead? All
I run cerberus all the time but I don't have locally x86 machines with
Quoted 1 line
1G of ram so it will take some time for me to try on a real highmem, I
am pretty sure I just tested cerberus with highmem emulation and it didn't deadlocked for me, I can try again with an higher highmem/normal ratio later. The ratio I am using right now is half of the ram in highmem (that is almost the same ratio of a 2G machine):
arch/i386/config.in +3 -0
@@ -369,4 +369,7 @@ #bool 'Debug kmalloc/kfree' CONFIG_DEBUG_MALLOC bool 'Magic SysRq key' CONFIG_MAGIC_SYSRQ +if [ "$CONFIG_HIGHMEM" = "y" ]; then + bool 'Debug HIGHMEM on lowmem machines' CONFIG_HIGHMEM_DEBUG +fi endmenu
arch/i386/kernel/setup.c +12 -0
@@ -649,7 +649,19 @@*/ #define VMALLOC_RESERVE (unsigned long)(128 << 20)#define MAXMEM (unsigned long)(-PAGE_OFFSET-VMALLOC_RESERVE) +#ifdef CONFIG_HIGHMEM_DEBUG +#define MAXMEM_PFN \ +({ \ + int __max_pfn; \ + if (max_pfn > PFN_DOWN(MAXMEM)) \ + __max_pfn = PFN_DOWN(MAXMEM); \ + else \ + __max_pfn = max_pfn / 2; \ + __max_pfn; \ +}) +#else #define MAXMEM_PFN PFN_DOWN(MAXMEM) +#endif #define MAX_NONPAE_PFN (1 << 20)/*
Andrea
Quoted 3 lines · Andrea Arcangeli
not being able to reproduce has nothing to do with a deadlock fixed or not. Also Ben's patch I think was claiming to fix the deadlock and it didn't even addressed the
create_bouncesthat it is fixed in 2.4.5.
No. It does not fix the deadlock. Neither does the patch you posted. But, if you're going to add a reserve pool for buffer heads and bounce pages, please do it with generic, not yet another special case hack.
-ben