2.6.5-rc3-aa1
The xfs warning during truncate will be fixed with a later update (Nathan is currently working on it).
next thing to do is to fixup the merging in mprotect.
In the meantime this is already mergeable in -mm if Andrew is interested.
I'm not touching mremap at this time (i'll reistantiate the merging in mprotect first, then Hugh testcase will work as well as in mainline). I wait comments on the truncate/mremap race fixes (they clearly partly collides with prio-tree but part of them seems shareable). That race isn't trivial to fix but at the same time it doesn't worry me much right now. I could add the semaphore, but mremap is an extreme fast path for some workload like apache, so I'm not going to risk a scalability-slowdown in there with an obviously safe patch of taking the i_shared_sem semaphore, because even if that race triggers the kernel shouldn't crash anymore.
URL:
http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.6/2.6.5-rc3-aa1.gz http://www.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.6/2.6.5-rc3-aa1/
Changelog diff between 2.6.5-rc2-aa6 and 2.6.5-rc3-aa1:
Files 2.6.5-rc2-aa6/anon-vma.gz and 2.6.5-rc3-aa1/anon-vma.gz differ
Track anonymous pages generated by cows on dma regions (bug spotted by Hugh Dickins).
Files 2.6.5-rc2-aa6/extraversion and 2.6.5-rc3-aa1/extraversion differ
Rediffed.
Only in 2.6.5-rc3-aa1: initramfs-search-for-init
Initramfs handy patch from -mm.
Files 2.6.5-rc2-aa6/prio-tree.gz and 2.6.5-rc3-aa1/prio-tree.gz differ
Fix xfs to check nonlinear mappings too. Fix x86-64 compilation.
Only in 2.6.5-rc3-aa1: tag-writeback-pages-fix.patch.gz
Avoid crashes in rw_swap_page_sync due -mm writeback changes, final (more highlevel) fix from Andrew.
Quoted 3 lines · Andrea Arcangeli
In the meantime this is already mergeable in -mm if Andrew is
interested.
It's a bit early for that, I feel. I'd like to see thing settle down a little more at your end first, then see that Rajesh, Hugh and if possible Ingo have had a good go through everything.
And then there are the mechanics of swallowing a largely-undocumented 4,600-line patch which touches 60 files and tosses 30-odd rejects across 16 files.
We have a way to go yet. Once I've dumped a lot of the current pending stuff into 2.6.6-early we'll be in better shape.
Quoted 7 lines · Andrew Morton
Quoted 3 lines · Andrea Arcangeli
In the meantime this is already mergeable in -mm if Andrew is
interested.It's a bit early for that, I feel. I'd like to see thing settle down a little more at your end first, then see that Rajesh, Hugh and if possible
no problem in settling it down more on my end.
Quoted 1 line
Ingo have had a good go through everything.
the thing isn't changing much anymore, there are only a few lines fixes across the last updates.
the mprotect merging won't be as small, but it should not be more than an hundred lines rewrite and it's orthogonal with the rest of the changes. I plan to complete it before the end of the week.
Keep in mind this whole thing is going in production in a matter of a week, so please test and review now. I'm uncertain if to add the mprotect merging to the production tree after I completed it, I don't see it as a requirement, infact I believe mprotect has always been more important for file mappings than anonymous memory (I know this is certainly the case for some big app where merging file mappings in mprotect actually would help a bit too). At this point in time unless I get a complaint for a real app I'll probably avoid any further change to avoid invalidating the current testing. So I may keep the mprotect merging in a separate patch.
I'm _very_ confortable that whatever might go wrong with this new code that I cannot imagine right now, it'll always be a lot better than the stuff that we know goes wrong with rmap or/and 4:4.
Quoted 2 lines
We have a way to go yet. Once I've dumped a lot of the current pending stuff into 2.6.6-early we'll be in better shape.
I'd like to see the -mm writeback changes merged ASAP since they're the most invasive from my point of view ;)
thanks.
Quoted 5 lines · On Wed, 31 Mar 2004 05:09:21 +0200 Andrea Arcangeli <[email protected]> wrote:
The xfs warning during truncate will be fixed with a later update (Nathan is currently working on it).
next thing to do is to fixup the merging in mprotect.
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
SysRq : Show Memory
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 28, high 84, batch 14
cpu 0 cold: low 0, high 28, batch 14
HighMem per-cpu: empty
Mar 31 06:48:12 bongani kernel:
Free pages: 2752kB (0kB HighMem)
Active:27927 inactive:1585 dirty:4 writeback:0 unstable:0 free:688
DMA free:1008kB min:28kB low:56kB high:84kB active:308kB inactive:256kB present:16384kB Normal free:1744kB min:476kB low:952kB high:1428kB active:111400kB inactive:6084kB present:245696kB HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1008kB
Normal: 162*4kB 17*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1744kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap: 257000kB
65520 pages of RAM
0 pages of HIGHMEM
1579 reserved pages
19651 pages shared
Quoted 6 lines · Bongani Hlope
Quoted 3 lines · On Wed, 31 Mar 2004 05:09:21 +0200 Andrea Arcangeli <[email protected]> wrote:
The xfs warning during truncate will be fixed with a later update (Nathan is currently working on it).
Quoted 7 lines
Quoted 2 lines
next thing to do is to fixup the merging in mprotect.
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
weird, it really loks like it doesn't swap anything. At least it's not a race condition. Which fs are you using?
can you try to actively push it into swap with a script like this?
#!/usr/bin/env python
while 1:
try: a = 'a' while 1: a += a except MemoryError: pass
this should push something into swap.
Can you also upgrade to 2.6.5-rc3-aa1? There were a few bugs in previous releases, though nothing that would prevent it to swap.
thanks for the feedback!
Quoted 25 lines
SysRq : Show Memory
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 28, high 84, batch 14
cpu 0 cold: low 0, high 28, batch 14
HighMem per-cpu: emptyMar 31 06:48:12 bongani kernel:Free pages: 2752kB (0kB HighMem)
Active:27927 inactive:1585 dirty:4 writeback:0 unstable:0 free:688
DMA free:1008kB min:28kB low:56kB high:84kB active:308kB inactive:256kB present:16384kB Normal free:1744kB min:476kB low:952kB high:1428kB active:111400kB inactive:6084kB present:245696kB HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1008kB
Normal: 162*4kB 17*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1744kB
HighMem: empty
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap: 257000kB
65520 pages of RAM
0 pages of HIGHMEM
1579 reserved pages
19651 pages shared
Quoted 22 lines · On Wed, 31 Mar 2004 23:22:42 +0200 Andrea Arcangeli <[email protected]> wrote:
Quoted 6 lines · Bongani Hlope
Quoted 3 lines · On Wed, 31 Mar 2004 05:09:21 +0200 Andrea Arcangeli <[email protected]> wrote:
The xfs warning during truncate will be fixed with a later update (Nathan is currently working on it).
Quoted 7 lines
Quoted 2 lines
next thing to do is to fixup the merging in mprotect.
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
weird, it really loks like it doesn't swap anything. At least it's not a race condition. Which fs are you using?
can you try to actively push it into swap with a script like this?
#!/usr/bin/env python
Quoted 8 lines
while 1:
try: a = 'a' while 1: a += a except MemoryError: pass
I'm using ext3. and your script did push things to swap. I'm busy compiling 2.6.5-rc3-aa1 now.
Thanx Andrea
On Wednesday 31 March 2004 21:16, Bongani Hlope wrote:
Hi Bongani,
Quoted 4 lines
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
hmm, I am running 2.6.5-rc3-aa1 stuff ontop of 2.6.5-rc3-mm3. It works very well. What is the value of /proc/sys/vm/swappiness?
ciao, Marc
On Thursday 01 April 2004 11:57, Marc-Christian Petersen wrote:
Hi again,
Quoted 7 lines
Quoted 4 lines
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
hmm, I am running 2.6.5-rc3-aa1 stuff ontop of 2.6.5-rc3-mm3. It works very well. What is the value of
/proc/sys/vm/swappiness?
I really manage it to forget something again and again ;(
SysRq : Show Memory
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 14, high 42, batch 7
cpu 0 cold: low 0, high 14, batch 7
Free pages: 14380kB (252kB HighMem)
Active:149756 inactive:45587 dirty:47 writeback:0 unstable:0 free:3595 DMA free:2280kB min:16kB low:32kB high:48kB active:1316kB inactive:5876kB present:16384kB
protections[]: 8 476 540
Normal free:11848kB min:936kB low:1872kB high:2808kB active:494812kB inactive:155796kB present:901120kB
protections[]: 0 468 532
HighMem free:252kB min:128kB low:256kB high:384kB active:102896kB inactive:20676kB present:130992kB
protections[]: 0 0 64
DMA: 28*4kB 1*8kB 15*16kB 2*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2280kB
Normal: 2222*4kB 110*8kB 0*16kB 3*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 11848kB
HighMem: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 252kB
Swap cache: add 281, delete 281, find 28/28, race 0+0
Free swap: 986944kB
262124 pages of RAM
32748 pages of HIGHMEM
2864 reserved pages
159610 pages shared
0 pages swap cached
cat /proc/meminfo
MemTotal: 1037272 kB MemFree: 12096 kB Buffers: 29640 kB Cached: 512264 kB SwapCached: 0 kB Active: 600180 kB Inactive: 183568 kB HighTotal: 130992 kB HighFree: 616 kB LowTotal: 906280 kB LowFree: 11480 kB SwapTotal: 987956 kB SwapFree: 986944 kB Dirty: 740 kB Writeback: 0 kB Mapped: 328296 kB Slab: 226804 kB Committed_AS: 446724 kB PageTables: 4104 kB VmallocTotal: 106488 kB VmallocUsed: 23068 kB VmallocChunk: 83260 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 4096 kB
uptime
12:12:41 up 16:24, 6 users, load average: 1.03, 1.07, 1.36
cat /proc/sys/vm/swappiness
48
ciao, Marc
Quoted 9 lines · On Thu, 01 Apr 2004 12:21:21 +0200 Marc-Christian Petersen…
On Thursday 01 April 2004 11:57, Marc-Christian Petersen wrote:
Hi again,
Quoted 4 lines
Quoted 4 lines
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
Quoted 4 lines
Quoted 2 lines
hmm, I am running 2.6.5-rc3-aa1 stuff ontop of 2.6.5-rc3-mm3. It works very well. What is the value of
/proc/sys/vm/swappiness?I really manage it to forget something again and again ;(
8<
Quoted 4 lines
cat
/proc/sys/vm/swappiness
48ciao, Marc
My swappiness is 60, and 2.6.5-rc2-aa4 was running fine for a while until that occured. Andrea sent me a python script that showed that it does swap. I'll try to reproduce it still.
Quoted 11 lines · Marc-Christian Petersen
On Wednesday 31 March 2004 21:16, Bongani Hlope wrote:
Hi Bongani,
Quoted 4 lines
I'm running 2.6.5-rc2-aa4, when I woke-up in the morning almost all of my memory was gone, but my swap was never touched. I managed to get only the output of SysRq-M before it hard-locked. For some reason it doesn't swap. I'll try to reproduce.
hmm, I am running 2.6.5-rc3-aa1 stuff ontop of 2.6.5-rc3-mm3. It works very well. What is the value of
/proc/sys/vm/swappiness?
my script is swapping well for him too, so maybe this was more a genuine deadlock somewhere than something else, it would been interesting to see SYSRQ+P. It's not necessairly related to my changes.