experiences beyond 4 GB RAM with 2.4.22
Subsystems (4)
Hello,
lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:
1) nfs clients see timeouts again, like
Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OK
Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
echo 2097152 >/proc/sys/net/core/rmem_max
echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.
3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.
I am really astonished about this. Can some kind soul give me hints or maybe patches to try?
Regards,
Stephan
Quoted 32 lines · Stephan von Krawczynski
Hello,
lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:1) nfs clients see timeouts again, like
Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OKBoth are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
echo 2097152 >/proc/sys/net/core/rmem_max echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.
Quoted 6 lines
3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.
I am really astonished about this. Can some kind soul give me hints or maybe patches to try?
for the vm issues my suggestion is to try again with 2.4.22aa1.
Andrea
/*
* If you refuse to depend on closed software for a critical
* part of your business, these links may be useful:
*
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5/
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.4/
* http://www.cobite.com/cvsps/
*
* svn://svn.kernel.org/linux-2.6/trunk
* svn://svn.kernel.org/linux-2.4/trunk
*/
Quoted 20 lines · On Tuesday September 9, [email protected] wrote:
Hello,
lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:1) nfs clients see timeouts again, like
Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still trying Sep 9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OK
Quoted 7 lines
Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
echo 2097152 >/proc/sys/net/core/rmem_max echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.
Quoted 8 lines
2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.
3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.
My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory. Have you tried 2.6 ??
How about CONFIG_HIGHMEM4G ?
It won't use all the RAM, but it would be interesting if it were faster.
NeilBrown
Quoted 11 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 7 lines
Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
echo 2097152 >/proc/sys/net/core/rmem_max echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.
Oh. That's interesting. Then everything should be the same if I deleted those...
Quoted 10 lines
Quoted 7 lines
2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.
3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.
My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.
I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"
Quoted 1 line
Have you tried 2.6 ??
No, not yet. I have not dared :-)
Quoted 3 lines
How about
CONFIG_HIGHMEM4G?
It won't use all the RAM, but it would be interesting if it were faster.
I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.
Regards,
Stephan
Quoted 10 lines · Stephan von Krawczynski
Quoted 2 lines
My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.
I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"
That is not enough, the 3ware driver only sets a 32-bit IO capability mask. So you will still be bouncing to/from the upper 2G, Neils diagnosis is absolutely right. If you want to verify this fact, boot with profile=2 and run readprofile -r; updatedb; readprofile | sort -nr and you should see the bounce copy functions near the top.
As a paying customer, you should ask 3ware about their hardware. They might be able to support > 32-bit dma just noone has asked about this feature yet. A quick peek at their driver shows they define their sg address element as a 32-bit quantity, so maybe not...
Signature · 1 line
Jens Axboe
Quoted 6 lines
Quoted 3 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 3 lines
Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
echo 2097152 >/proc/sys/net/core/rmem_max echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.
So then what do I need to to get those error messages off of my nfs clients? I have seen this with for a long time through 2.4 and 2.5 (didn't use nfs with 2.2...).
Quoted 45 lines · Stephan von Krawczynski
Quoted 11 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 7 lines
Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and
echo 2097152 >/proc/sys/net/core/rmem_max echo 2097152 >/proc/sys/net/core/wmem_maxAre these values too small for 6 GB?
No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.
Oh. That's interesting. Then everything should be the same if I deleted those...
Quoted 10 lines
Quoted 7 lines
2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.
3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.
My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.
I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"Quoted 1 line
Have you tried 2.6 ??
No, not yet. I have not dared :-)
Quoted 3 lines
How about
CONFIG_HIGHMEM4G?
It won't use all the RAM, but it would be interesting if it were faster.
Quoted 6 lines
I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.
Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.
Have you tried 2.4.22-aa?
Thanks
Quoted 9 lines · On Mon, 15 Sep 2003 19:01:42 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines
I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.
Hello Marcelo,
it looks like -pre4 performs not well even in 4 GB environment. After few days of running I find hanging 2.4.22 nfs-clients on a 2.4.23-pre4 server.
On the client I get a bunch of those:
Sep 16 03:02:00 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:32 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:32 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:37 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:37 brenda last message repeated 3 times Sep 16 03:02:37 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:37 brenda last message repeated 3 times Sep 16 03:02:38 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:38 brenda last message repeated 6 times Sep 16 03:02:38 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:38 brenda last message repeated 6 times Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:42 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:42 brenda last message repeated 2 times Sep 16 03:02:42 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:02:42 brenda last message repeated 2 times Sep 16 03:02:43 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:02:43 brenda last message repeated 8 times Sep 16 03:02:43 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:03:08 brenda last message repeated 7 times Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 not responding, still trying Sep 16 03:03:09 brenda last message repeated 2 times Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:03:10 brenda last message repeated 2 times
And then the nfs-action is dead. Process hangs. This has not happened with 2.4.22 as a server. It showed up after a day of creating 4,7 GB dvd iso images. Creating theses isos was ok, no error or so during the action.
Is it possible that pre4 does not recover all that well from former memory pressure?
Quoted 1 line
Have you tried 2.4.22-aa?
Sorry, not yet.
I will go back to 2.4.22 and stress it to see if these effects show up.
Regards,
Stephan
Quoted 15 lines · On Tue, 16 Sep 2003 10:21:13 +0200 Stephan von Krawczynski <[email protected]>…
Quoted 9 lines · On Mon, 15 Sep 2003 19:01:42 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines
I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.
Hello Marcelo,
Quoted 7 lines
it looks like -pre4 performs not well even in 4 GB environment. After few days of running I find hanging 2.4.22 nfs-clients on a 2.4.23-pre4 server.
On the client I get a bunch of those:
Sep 16 03:02:00 brenda kernel: nfs: server 192.168.1.1 OK[...]
Hello again,
you will love to hear that you can drop the above statement completely. After digging deeper into the case I found out that it was caused by a dead switch. So this is no -pre4 problem, but a hardware issue. The switch corrupted about every 20th packet.
So I can tell nothing negative so far about -pre4 with 4 GB.
I'll try with 6 GB now.
Regards,
Stephan
Quoted 10 lines · Stephan von Krawczynski
Sep 16 03:03:09 brenda last message repeated 2 times Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 OK Sep 16 03:03:10 brenda last message repeated 2 timesAnd then the nfs-action is dead. Process hangs. This has not happened with 2.4.22 as a server. It showed up after a day of creating 4,7 GB dvd iso images. Creating theses isos was ok, no error or so during the action.
Is it possible that pre4 does not recover all that well from former memory pressure?
Quoted 4 lines
Quoted 1 line
Have you tried 2.4.22-aa?
Sorry, not yet.
I will go back to 2.4.22 and stress it to see if these effects show up.
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Maybe you have a SCSI card+disks to test ? 8)
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Maybe you have a SCSI card+disks to test ? 8)
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...
Regards,
Stephan
Quoted 15 lines · Stephan von Krawczynski
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Maybe you have a SCSI card+disks to test ? 8)
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...
Quoted 2 lines
Regards,
Stephan
Can you explain what you mean by "network connections go bogus". Whether or not you have more that 4GB or RAM and therefore have to page it into lower virtual addresses has nothing at all to do with networking unless you have a network device driver that did not allocate memory properly. If so, that should be checked out.
Since there is only 32 bits of address space in the Intel machines, the virtual memory seen by a process can't exceed that, including the kernel itself. However, the physical memory can come from anywhere and that's what the extended-memory specification attempts to provide. If something is hurting the network, it shouldn't be so.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 1 line · On Tue, 16 Sep 2003 09:55:36 -0400 (EDT) "Richard B. Johnson"…
Can you explain what you mean by "network connections go bogus".
Sure. Do this:
Use a controller (like 3ware) that cannot DMA beyond a 4 GB range. Now put some GBs of data onto that and start to tar the data around on the same disk. While doing this you can watch network go crazy and drop packets at will. You can of course force the whole setup further by (trying) additional nfs-action, but this is really not needed.
Just about the same thing can be experienced if you do a simple find all over the disk. CPU load explodes and network is _dead_.
To try this you need some GBs of data, a box with more than 4 GB ram, 3ware controller and a usual SuSE 8.2. Wait until after midnight for "updatedb" to run and try to login during that time ;-)
"Works" with out-of-the-box equipment and distro :-)
Regards,
Stephan
Quoted 5 lines · Stephan von Krawczynski
Quoted 2 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Quoted 7 lines
Quoted 1 line
Maybe you have a SCSI card+disks to test ? 8)
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable.
All is fine with 4GB?
Quoted 3 lines
Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...
No way to omit the bounce buffers.
Quoted 7 lines · On Tue, 16 Sep 2003 11:33:27 -0300 (BRT) Marcelo Tosatti…
Quoted 5 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable.
All is fine with 4GB?
Absolutely perfect.
Regards,
Stephan
Quoted 6 lines · Stephan von Krawczynski
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.
I don't know anyone who explored that however
Quoted 10 lines · On Tue, 16 Sep 2003 15:36:14 +0100 Alan Cox <[email protected]> wrote:
Quoted 6 lines · Stephan von Krawczynski
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally.
Hm, "not ideally" is a real friendly word for describing the mess ;-) Isn't there a possibility to flag this part of the memory as nonDMA-able, kind of "do whatever you want with it, but don't expect any dma-driven i/o"... I know this gets a problem when swap jumps in, though.
But really it is far better for the box to flag it more or less unusable compared to a DoS done by user-space "find" ...
I know this is a real corner case of life. It looks more like taking a different decision than current to improve the situation and not so much a real development topic. Probably a note in kernel docs reading "DON'T DO THIS" is either sufficient...
Regards,
Stephan
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Maybe not what you WILL DO, but what you HAVE DONE. Those tasks which have done the most DMA-requiring I/O could get preference for low memory. No?
Yeah, I know.. show you the code. :)
Quoted 11 lines · On Tue, 16 Sep 2003 11:49:10 -0400 Timothy Miller <[email protected]> wrote:
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Maybe not what you WILL DO, but what you HAVE DONE. Those tasks which have done the most DMA-requiring I/O could get preference for low memory. No?
Yeah, I know.. show you the code. :)
That really sounds complex and therefore I would not try that in first place. And it does not sound like a solution to my special problem of few tasks operating on a lot of data (a lot more than fits to physical mem). For this type of situation it would be really intelligent not to give away non-dma-able memory. How can the kernel do that?
At least it knows that there are I/O devices that cannot cope with the mem it is presenting. This sounds in fact simple.
The problem is: what _can_ be done with this type of mem at all? It does not even help a lot if there are other devices that can handle it, because you still must face the fact of a simple file-copy from such a capable device to one that is not.
No wonder that the gurus did not have any good idea yet ;-)
Regards,
Stephan
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
OG.
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Quoted 6 lines · Alan Cox
Quoted 5 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Quoted 6 lines
Quoted 1 line
Is there a way to specifically turn that ram into a tmpfs though?
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Quoted 6 lines
Quoted 3 lines
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Aditionally, you need
GFP_DMA32or similar. Would also alleviate the nasty pressure onZONE_NORMALwhich is often quite stressed.
IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?
What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?
Thanks
Quoted 21 lines · On Wed, 17 Sep 2003 16:30:45 -0300 (BRT) Marcelo Tosatti…
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Quoted 6 lines
Quoted 3 lines
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Aditionally, you need
GFP_DMA32or similar. Would also alleviate the nasty pressure onZONE_NORMALwhich is often quite stressed.
Quoted 6 lines
IMO such
GFP_DMA32flag is a bit intrusive for 2.4, isnt it?What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?
Thanks
Before running too far in this direction I would suggest to solve Oliviers' problem with the aic driver. I really would like to know if he sees the same positive effects in pre4 like me. It seems Andreas' vm patches have a very positive influence on the issue. At least I cannot see the "crawling effect" up to now with 6GB and pre4 compared to 2.4.22. It would surely be of interest if Oliviers' 8 GB variant improves, too.
May well be that the bouncing is not that bad compared to other corner effects of the vm in this special situation.
Interactivity during load and especially network seems far better in pre4. For sure it is not as speedy as a 4GB setup, but it works pretty well (up to now).
Regards,
Stephan
Quoted 14 lines · Marcelo Tosatti
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Aditionally, you need
GFP_DMA32or similar. Would also alleviate the nasty pressure onZONE_NORMALwhich is often quite stressed.IMO such
GFP_DMA32flag is a bit intrusive for 2.4, isnt it?
Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...
Quoted 2 lines
What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?
Nothing, afaic. 2.6 isn't even completely deadlock free when it comes to bounce buffering.
Signature · 1 line
Jens Axboe
Quoted 19 lines · Jens Axboe
Quoted 18 lines · Marcelo Tosatti
Quoted 14 lines · Jens Axboe
Quoted 12 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Aditionally, you need
GFP_DMA32or similar. Would also alleviate the nasty pressure onZONE_NORMALwhich is often quite stressed.IMO such
GFP_DMA32flag is a bit intrusive for 2.4, isnt it?Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...
This is the latest I had, for 2.4.5. Pretty simple and nonintrusive at that time.
arch/i386/mm/init.c renamed from opt/kernel/linux-2.4.5/arch/i386/mm/init.c +9 -5
@@ -25,6 +25,7 @@#include <linux/highmem.h> #include <linux/pagemap.h> #include <linux/bootmem.h> +#include <linux/pci.h> #include <asm/processor.h> #include <asm/system.h>
@@ -348,12 +349,15 @@kmap_init ();#endif { -unsigned long zones_size[MAX_NR_ZONES ] = {0 ,0 ,0 }; -unsigned int max_dma, high, low; +unsigned long zones_size[MAX_NR_ZONES ] = {0 ,0 ,0 ,0 }; +unsigned int max_dma, max_dma32, high, low, high32; max_dma =virt_to_phys ((char *)MAX_DMA_ADDRESS ) >>PAGE_SHIFT ; + max_dma32 =PCI_MAX_DMA32 >>PAGE_SHIFT ; low = max_low_pfn; - high = highend_pfn; + high32 = high = highend_pfn; +if (high32 > max_dma32) + high32 = max_dma32 +1 ;/* first map in HIGHMEM */ if (low < max_dma) zones_size[ZONE_DMA ] = low;
@@ -361,12 +365,12 @@ zones_size[ZONE_DMA ] = max_dma; zones_size[ZONE_NORMAL ] = low - max_dma;#ifdef CONFIG_HIGHMEM - zones_size[ZONE_HIGHMEM ] = high - low; + zones_size[ZONE_DMA32 ] = high32 - low; + zones_size[ZONE_HIGHMEM ] = high - high32;#endif }free_area_init (zones_size); } -return ; }/*
include/linux/mm.h renamed from opt/kernel/linux-2.4.5/include/linux/mm.h +3 -1
@@ -476,8 +476,10 @@#define __GFP_IO 0x04#define __GFP_DMA 0x08#ifdef CONFIG_HIGHMEM -#define __GFP_HIGHMEM 0x10 +#define __GFP_DMA32 0x10 +#define __GFP_HIGHMEM 0x20#else +#define __GFP_DMA32 0x0/* noop */ #define __GFP_HIGHMEM 0x0/* noop */ #endif
include/linux/mmzone.h renamed from opt/kernel/linux-2.4.5/include/linux/mmzone.h +6 -4
@@ -27,7 +27,8 @@* * ZONE_DMA < 16 MB ISA DMA capable memory * ZONE_NORMAL 16-896 MB direct mapped by the kernel -* ZONE_HIGHMEM > 896 MB only page cache and user processes +* ZONE_DMA32 > 892MB < 4GB For 32-bit DMA +* ZONE_HIGHMEM > 4GB only page cache and user processes */ typedef struct zone_struct { /*
@@ -62,8 +63,9 @@#define ZONE_DMA 0#define ZONE_NORMAL 1 -#define ZONE_HIGHMEM 2 -#define MAX_NR_ZONES 3 +#define ZONE_DMA32 2 +#define ZONE_HIGHMEM 3 +#define MAX_NR_ZONES 4/* * One allocation request operates on a zonelist. A zonelist
@@ -81,7 +83,7 @@int gfp_mask; } zonelist_t; -#define NR_GFPINDEX 0x20 +#define NR_GFPINDEX 0x40/* * The pg_data_t structure is used in machines with CONFIG_DISCONTIGMEM
mm/page_alloc.c renamed from opt/kernel/linux-2.4.5/mm/page_alloc.c +15 -1
@@ -598,6 +598,7 @@while (pgdat) { pages += pgdat->node_zones[ZONE_HIGHMEM ].free_pages; + pages += pgdat->node_zones[ZONE_DMA32 ].free_pages; pgdat = pgdat->node_next; }return pages;
@@ -683,6 +684,8 @@ k =ZONE_NORMAL ;if (i & __GFP_HIGHMEM) k =ZONE_HIGHMEM ; +if (i & __GFP_DMA32) + k =ZONE_DMA32 ;if (i & __GFP_DMA) k =ZONE_DMA ;
@@ -700,6 +703,14 @@#endif zonelist->zones[j++] = zone; } +case ZONE_DMA32 : + zone = pgdat->node_zones +ZONE_DMA32 ; +if (zone->size) { +#ifndef CONFIG_HIGHMEM +BUG (); +#endif + zonelist->zones[j++] = zone; + }case ZONE_NORMAL : zone = pgdat->node_zones +ZONE_NORMAL ;if (zone->size)
@@ -833,8 +844,11 @@for (i =0 ; i < size; i++) {struct page *page = mem_map + offset + i; page->zone = zone; -if (j !=ZONE_HIGHMEM ) +if (j !=ZONE_HIGHMEM && j !=ZONE_DMA32 ) { page->virtual =__va (zone_start_paddr); + }else + page->virtual =NULL ; + zone_start_paddr +=PAGE_SIZE ; }
Signature · 1 line
Jens Axboe
Quoted 25 lines · On Thu, 18 Sep 2003 09:12:49 +0200 Jens Axboe <[email protected]> wrote:
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Marcelo Tosatti
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox
The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.
Is there a way to specifically turn that ram into a tmpfs though?
Quoted 6 lines
Quoted 3 lines
Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here
Aditionally, you need
GFP_DMA32or similar. Would also alleviate the nasty pressure onZONE_NORMALwhich is often quite stressed.IMO such
GFP_DMA32flag is a bit intrusive for 2.4, isnt it?
Quoted 7 lines
Quoted 2 lines
Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...
This is the latest I had, for 2.4.5. Pretty simple and nonintrusive at that time.
From a design point of view I would pretty much agree with your idea. In fact
there is a ram attribute (dma32) which currently is not reflected in the data structures in a selectable way. I can see no good reason for this lack.
Regards,
Stephan
Quoted 1 line · Marcelo Tosatti
Quoted 1 line
IMO such
GFP_DMA32flag is a bit intrusive for 2.4, isnt it?
Quoted 2 lines · Jens Axboe
Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...
Quoted 2 lines · Marcelo Tosatti
Quoted 2 lines
What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?
Quoted 2 lines · Jens Axboe
Nothing, afaic. 2.6 isn't even completely deadlock free when it comes to bounce buffering.
It'd be great to have ZONE_DMA32 around for 2.6.
-- wli
Hi!
Quoted 11 lines
Quoted 6 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.
If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.
Pavel
Signature · 2 lines
Pavel Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
Quoted 9 lines · Pavel Machek
Hi!
Quoted 7 lines
Quoted 6 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
Quoted 8 lines
Quoted 4 lines
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.
If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.
The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.
We're quite eager to fix the problem too, if you want us to test some things.
OG.
Hi!
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.
Quoted 7 lines
Quoted 3 lines
If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.
The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.
That seems like bug ;-). Can you do some kind of memstat to see if it is not something like atomic pages shortage? Also try to run vanilla kernel. And try running it UP.
Quoted 2 lines
We're quite eager to fix the problem too, if you want us to test some things.
I'm afraid I do not have big-enough box close-enough to fix that.
Does it happen with another disk driver, too? What about interrupts, are not they disabled for too long? Can you enable PREEMPT to see 'scheduling in atomic' warnings?
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 19 lines · Olivier Galibert
Quoted 9 lines · Pavel Machek
Hi!
Quoted 7 lines
Quoted 6 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
Quoted 8 lines
Quoted 4 lines
The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.
If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.
Quoted 6 lines
The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.
We're quite eager to fix the problem too, if you want us to test some things.
Which card and driver are you using for IO? 3ware?
How much RAM do you have?
I remember I tested heavy IO loads (heavy swapping and dbench) on 8GB machine and all worked fine (interactive terminal, etc) but that was a looong time ago back in 2.4.
Quoted 1 line · Marcelo Tosatti
Which card and driver are you using for IO? 3ware?
Bus 6, device 2, function 0:
SCSI storage controller: Adaptec AIC-7902 U320 (rev 3).
Bus 6, device 2, function 1:
SCSI storage controller: Adaptec AIC-7902 U320 (#2) (rev 3).
0:
Adaptec AIC79xx driver version: 1.3.10
Adaptec AIC7902 Ultra320 SCSI adapter
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs Allocated SCBs: 32, SG List Length: 85
Target 0 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Goal: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Curr: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit) Transmission Errors 0
Channel A Target 0 Lun 0 Settings
Commands Queued 6712736
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
1:
Adaptec AIC79xx driver version: 1.3.10
Adaptec AIC7902 Ultra320 SCSI adapter
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs Allocated SCBs: 64, SG List Length: 85
Serial EEPROM:0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x09f4 0x0146 0x2807 0x0010 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0x0410 0xb3d7
Target 0 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Goal: 160.000MB/s transfers (80.000MHz DT, 16bit)
Curr: 160.000MB/s transfers (80.000MHz DT, 16bit) Transmission Errors 0
Channel A Target 0 Lun 0 Settings
Commands Queued 5523328
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Channel A Target 0 Lun 1 Settings
Commands Queued 14646734
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Target 1 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Goal: 160.000MB/s transfers (80.000MHz DT, 16bit)
Curr: 160.000MB/s transfers (80.000MHz DT, 16bit) Transmission Errors 0
Channel A Target 1 Lun 0 Settings
Commands Queued 4
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Channel A Target 1 Lun 1 Settings
Commands Queued 4
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
dmesg:
SCSI subsystem driver Revision: 1.00
scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
blk: queue f7965018, I/O limit 524287Mb (mask 0x7fffffffff) (scsi0:A:0): 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
scsi1:A:0:0: DV failed to configure device. Please file a bug report against this driver. scsi1:A:1:0: DV failed to configure device. Please file a bug report against this driver. Vendor: SEAGATE Model: ST373307LC Rev: 0004 Type: Direct-Access ANSI SCSI revision: 03
blk: queue f7913e18, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: SUPER Model: GEM318 Rev: 0
Type: Processor ANSI SCSI revision: 02
blk: queue f7913018, I/O limit 524287Mb (mask 0x7fffffffff) scsi0:A:0:0: Tagged Queuing enabled. Depth 32 (scsi1:A:0): 160.000MB/s transfers (80.000MHz DT, 16bit)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cbc18, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cba18, I/O limit 524287Mb (mask 0x7fffffffff) (scsi1:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cb618, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cb418, I/O limit 524287Mb (mask 0x7fffffffff)
scsi1:A:0:0: Tagged Queuing enabled. Depth 32 scsi1:A:0:1: Tagged Queuing enabled. Depth 32 scsi1:A:1:0: Tagged Queuing enabled. Depth 32 scsi1:A:1:1: Tagged Queuing enabled. Depth 32
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0 Attached scsi disk sdc at scsi1, channel 0, id 0, lun 1 Attached scsi disk sdd at scsi1, channel 0, id 1, lun 0 Attached scsi disk sde at scsi1, channel 0, id 1, lun 1 SCSI device sda: 143374744 512-byte hdwr sectors (73408 MB) Partition check:
sda: sda1 sda2 sda3 sda4
SCSI device sdb: 2788016128 512-byte hdwr sectors (1427464 MB)
sdb: sdb1
SCSI device sdc: 2788016128 512-byte hdwr sectors (1427464 MB)
sdc: sdc1
SCSI device sdd: 4101521408 512-byte hdwr sectors (2099979 MB)
sdd: sdd1
SCSI device sde: 4101521408 512-byte hdwr sectors (2099979 MB)
sde: sde1
Attached scsi generic sg1 at scsi0, channel 0, id 6, lun 0, type 3
Quoted 1 line
How much RAM do you have?
8Gb, currently down to 4Gb using mem=4G so that things actually work.
OG.
Quoted 8 lines · On Tue, 16 Sep 2003 23:23:02 +0200 Olivier Galibert…
Quoted 1 line · Marcelo Tosatti
Which card and driver are you using for IO? 3ware?
Bus 6, device 2, function 0:
SCSI storage controller: Adaptec AIC-7902 U320 (rev 3).
Bus 6, device 2, function 1:SCSI storage controller: Adaptec AIC-7902 U320 (#2) (rev 3).
Hello Olivier,
Pretty interesting.
Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...
Regards,
Stephan
Quoted 2 lines · Stephan von Krawczynski
Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...
Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.
OG.
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Stephan von Krawczynski
Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...
Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.
Actually I had booted the wrong kernel (2.6.0t4) by mistake. SCSI works, and there is indeed a remarkable difference. The system holds perfectly under filled-ram, high i/o usage now. Excellent.
Now if only the CPU enumeration worked and both CPUs were detected...
OG.
Quoted 11 lines · On Thu, 18 Sep 2003 11:58:45 +0200 Olivier Galibert <[email protected]> wrote:
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Stephan von Krawczynski
Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...
Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.
Actually I had booted the wrong kernel (2.6.0t4) by mistake. SCSI works, and there is indeed a remarkable difference. The system holds perfectly under filled-ram, high i/o usage now. Excellent.
Fine. So we seem to agree 2.4.23 will be another big hit in the 2.4 line :-)
Quoted 1 line
Now if only the CPU enumeration worked and both CPUs were detected...
Hm, I have not yet seen any configuration where multiple CPUs are not detected. Are you sure you have compiled in SMP support? What does dmesg look like?
Regards,
Stephan
Quoted 1 line · Stephan von Krawczynski
Fine. So we seem to agree 2.4.23 will be another big hit in the 2.4 line :-)
Indeed, it's working beautifully.
Quoted 4 lines
Quoted 1 line
Now if only the CPU enumeration worked and both CPUs were detected...
Hm, I have not yet seen any configuration where multiple CPUs are not detected. Are you sure you have compiled in SMP support? What does dmesg look like?
I found the problem. The meaning of the option "number of supported CPUs" is not what is expected. It is not fixing the maximum number of CPUs, but the number of the last CPU checked for. Specifically, in our system the CPUs are numbered 0 and 6. Setting the MNCPU to 2 prevents the second CPU to be taken into account.
OG.
Quoted 3 lines · Alan Cox
I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.
4G? Above 4G? The limit should be configurable a lot earlier.
I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.
Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).
Roger.
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 11 lines · Rogier Wolff
Quoted 3 lines · Alan Cox
I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.
4G? Above 4G? The limit should be configurable a lot earlier.
I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.
Quoted 6 lines
Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).
You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.
To answer one of the other questions regarding slowdown - it can be nastier than 2x, remember that for reads the copy back happens inside the interrupt handler... It would also be interesting to note (with vmstat 1) whether it's all system time, or if you see something like kswapd going crazy too. If the attached patch makes a difference, then it could be a vm issue as well.
Still doesn't change that fact that if you build a machine with 6GB of RAM and expect it to perform, then you don't add io controllers that cannot DMA to all of your RAM.
===== mm/highmem.c 1.15 vs edited =====
mm/highmem.c +1 -1
@@ -335,7 +335,7 @@struct list_head *tmp;struct page *page; - page =alloc_page (GFP_NOHIGHIO ); + page =alloc_page (GFP_ATOMIC );if (page)return page;/*
Signature · 1 line
Jens Axboe
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Rogier Wolff
Quoted 3 lines · Alan Cox
I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.
4G? Above 4G? The limit should be configurable a lot earlier.
Quoted 8 lines
Quoted 8 lines
I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.
Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap,
^^^^^^^^^^^^^^^^^
Quoted 8 lines
Quoted 2 lines
you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).
You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.
As I understand things (But this is from following discussions on linux-kernel from afar, not from personal poking at the code!) there is also a performance penalty for the kernel not having direct physically mapped access to RAM. We map up to 3G of virtual memory of userspace, and up to 1Gb of physical RAM into the kernel memory map for performance reasons. So if I have 2G RAM and want to keep 3G userspace, I have to use some "highmem" stuff right?
This will not directly require the use of bounce buffers, but it will require the kernel to remap regions when it needs to access them.
If this doesn't have a performance impact, why do I have the option of directly mapping 1G, 2G, or 3G?
Roger.
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 35 lines · Rogier Wolff
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Rogier Wolff
Quoted 3 lines · Alan Cox
I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.
4G? Above 4G? The limit should be configurable a lot earlier.
Quoted 8 lines
Quoted 8 lines
I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.
Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap,
^^^^^^^^^^^^^^^^^Quoted 8 lines
Quoted 2 lines
you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).
You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.
As I understand things (But this is from following discussions on linux-kernel from afar, not from personal poking at the code!) there is also a performance penalty for the kernel not having direct physically mapped access to RAM. We map up to 3G of virtual memory of userspace, and up to 1Gb of physical RAM into the kernel memory map for performance reasons. So if I have 2G RAM and want to keep 3G userspace, I have to use some "highmem" stuff right?
Quoted 2 lines
This will not directly require the use of bounce buffers, but it will require the kernel to remap regions when it needs to access them.
That is completely correct. Your original post just didn't make this distinction, and there is an order of magnitude performance difference between kmap() and bouncing! :)
Quoted 2 lines
If this doesn't have a performance impact, why do I have the option of directly mapping 1G, 2G, or 3G?
It does cost something of course, but not nearly as expensive as bounce buffering. You cannot compare the two.
Signature · 1 line
Jens Axboe
Hi!
Quoted 10 lines
Quoted 3 lines
and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).
You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.
Quoted 3 lines
To answer one of the other questions regarding slowdown - it can be nastier than 2x, remember that for reads the copy back happens inside the interrupt handler... It would also be interesting to note (with
Ouch, I guess I see. If big part of time is spent in interrupt copying data, network is going to loose packets and perform awfully. Could he run some interrupt latency tester?
Perhaps we are copying way too much in one chunk, therefore starving network?
Heh, old ide disk in PIO mode should allow that 6GB machine to perform better...
At least you don't loose packets during PIO reads...
Pavel
Signature · 2 lines
Pavel Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
Quoted 3 lines · Pavel Machek
Heh, old ide disk in PIO mode should allow that 6GB machine to perform better...
At least you don't loose packets during PIO reads...
As long as you tune it
hdparm -i1 /dev/hdX
Roger.
Quoted 1 line
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 2 lines · Rogier Wolff
As long as you tune it
hdparm -i1/dev/hdX
ehmm make that
hdparm -u1 /dev/hdX
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 ** *-- BitWizard writes Linux device drivers for any device you may have! --* **** "Linux is like a wigwam - no windows, no gates, apache inside!" ****
Quoted 9 lines · Stephan von Krawczynski
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Maybe you have a SCSI card+disks to test ? 8)
Quoted 7 lines
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...
If there were some kind of tracking to determine which processes are doing I/O which requires the process to be in low memory. Then, processes can be migrated around in physical memory so as to optimize for that.
Or is that already being done?
Quoted 11 lines
Quoted 3 lines
Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.
Maybe you have a SCSI card+disks to test ? 8)
Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...
It won't need to bounce buffer if you have a decent driver & hardware.
M.