Skip to contentSkip to conversation

experiences beyond 4 GB RAM with 2.4.22

DISCUSSION Stephan von Krawczynski 17 participants 48 messages Sep 9, 2003 – Sep 18, 2003

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hello,

lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:

1) nfs clients see timeouts again, like

Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OK

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.

3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.

I am really astonished about this. Can some kind soul give me hints or maybe patches to try?

Regards,
Stephan

#1 lore raw
Quoted 32 lines · Stephan von Krawczynski

Hello,

lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:

1) nfs clients see timeouts again, like

Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OK

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.

Quoted 6 lines

3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.

I am really astonished about this. Can some kind soul give me hints or maybe patches to try?

for the vm issues my suggestion is to try again with 2.4.22aa1.

Andrea

/*
* If you refuse to depend on closed software for a critical
* part of your business, these links may be useful:
*
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.5/
* rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.4/
* http://www.cobite.com/cvsps/
*
* svn://svn.kernel.org/linux-2.6/trunk
* svn://svn.kernel.org/linux-2.4/trunk
*/

#2 lore raw
In reply to Stephan von Krawczynski
Quoted 20 lines · On Tuesday September 9, [email protected] wrote:

Hello,

lately I upgraded my testbox from 2 to 6 GB ram and found out some oddities I would like to hear your opinions.
The box ran flawlessly and performant with 2 GB - was in fact a real joy. After upgrading the ram and recompiling kernel 2.4.22 with support for 64 GB I noticed:

1) nfs clients see timeouts again, like

Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:37:35 clienta kernel: nfs: server 192.168.1.1 OK
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep  9 03:41:13 clienta kernel: nfs: server 192.168.1.1 OK
Quoted 7 lines

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.

Quoted 8 lines

2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.

3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.

My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory. Have you tried 2.6 ??
How about CONFIG_HIGHMEM4G ?
It won't use all the RAM, but it would be interesting if it were faster.

NeilBrown

#3 lore raw
Quoted 11 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 7 lines

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.

Oh. That's interesting. Then everything should be the same if I deleted those...

Quoted 10 lines
Quoted 7 lines

2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.

3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.

My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.

I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"

Quoted 1 line

Have you tried 2.6 ??

No, not yet. I have not dared :-)

Quoted 3 lines

How about CONFIG_HIGHMEM4G ?
It won't use all the RAM, but it would be interesting if it were faster.

I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.

Regards,
Stephan

#4 lore raw
Quoted 10 lines · Stephan von Krawczynski
Quoted 2 lines

My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.

I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"

That is not enough, the 3ware driver only sets a 32-bit IO capability mask. So you will still be bouncing to/from the upper 2G, Neils diagnosis is absolutely right. If you want to verify this fact, boot with profile=2 and run readprofile -r; updatedb; readprofile | sort -nr and you should see the bounce copy functions near the top.

As a paying customer, you should ask 3ware about their hardware. They might be able to support > 32-bit dma just noone has asked about this feature yet. A quick peek at their driver shows they define their sg address element as a 32-bit quantity, so maybe not...

Signature · 1 line
Jens Axboe
#5 lore raw
In reply to Stephan von Krawczynski
Quoted 6 lines
Quoted 3 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 3 lines

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.

So then what do I need to to get those error messages off of my nfs clients? I have seen this with for a long time through 2.4 and 2.5 (didn't use nfs with 2.2...).

#6 lore raw
In reply to Stephan von Krawczynski
Quoted 45 lines · Stephan von Krawczynski
Quoted 11 lines · On Fri, 12 Sep 2003 12:46:46 +1000 Neil Brown <[email protected]> wrote:
Quoted 7 lines

Both are 2.4.22. 192.168.1.1 is the testbox. I saw those with 2GB, but could fix it through more nfs-daemons and

        echo 2097152 >/proc/sys/net/core/rmem_max
        echo 2097152 >/proc/sys/net/core/wmem_max

Are these values too small for 6 GB?

No. The values are proportional to the number of server threads, not the amount of RAM... and they should be un-necessary after 2.4.20 anyway as nfsd in the kernel makes the appropriate settings.

Oh. That's interesting. Then everything should be the same if I deleted those...

Quoted 10 lines
Quoted 7 lines

2) Box is very slow, kswapd looks very active during tar of a local harddisk. Interactivity is really bad. Seems vm has a high time looking for free or usable pages. Compared to 2 GB the behaviour is unbelievably bad.

3) Network performance has a remarkable dropdown during above tar. In fact doing simple pings every few minutes shows that quite a lot of them are simply dropped, never make it over the ethernet.

My only guess is that it is doing a lot of copying into low memory because your devices can only DMA into/outof low memory.

I forgot to mention: Both network card and controller are 64 bit cards. Network card is (vendor 3com):
Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15) (tg3-driver)
Controller is:
RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01) I have "CONFIG_HIGHIO=y"

Quoted 1 line

Have you tried 2.6 ??

No, not yet. I have not dared :-)

Quoted 3 lines

How about CONFIG_HIGHMEM4G ?
It won't use all the RAM, but it would be interesting if it were faster.

Quoted 6 lines

I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.

Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.

Have you tried 2.4.22-aa?

Thanks

#7 lore raw
Quoted 9 lines · On Mon, 15 Sep 2003 19:01:42 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines

I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.

Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.

Hello Marcelo,

it looks like -pre4 performs not well even in 4 GB environment. After few days of running I find hanging 2.4.22 nfs-clients on a 2.4.23-pre4 server.

On the client I get a bunch of those:

Sep 16 03:02:00 brenda kernel: nfs: server 192.168.1.1 OK
Sep 16 03:02:32 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:32 brenda kernel: nfs: server 192.168.1.1 OK
Sep 16 03:02:37 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:37 brenda last message repeated 3 times
Sep 16 03:02:37 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:02:37 brenda last message repeated 3 times
Sep 16 03:02:38 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:38 brenda last message repeated 6 times
Sep 16 03:02:38 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:02:38 brenda last message repeated 6 times
Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 OK
Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:41 brenda kernel: nfs: server 192.168.1.1 OK
Sep 16 03:02:42 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:42 brenda last message repeated 2 times
Sep 16 03:02:42 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:02:42 brenda last message repeated 2 times
Sep 16 03:02:43 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:02:43 brenda last message repeated 8 times
Sep 16 03:02:43 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:03:08 brenda last message repeated 7 times
Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 not responding, still
trying
Sep 16 03:03:09 brenda last message repeated 2 times
Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:03:10 brenda last message repeated 2 times

And then the nfs-action is dead. Process hangs. This has not happened with 2.4.22 as a server. It showed up after a day of creating 4,7 GB dvd iso images. Creating theses isos was ok, no error or so during the action.
Is it possible that pre4 does not recover all that well from former memory pressure?

Quoted 1 line

Have you tried 2.4.22-aa?

Sorry, not yet.
I will go back to 2.4.22 and stress it to see if these effects show up.

Regards,
Stephan

#8 lore raw
Quoted 15 lines · On Tue, 16 Sep 2003 10:21:13 +0200 Stephan von Krawczynski <[email protected]>…
Quoted 9 lines · On Mon, 15 Sep 2003 19:01:42 -0300 (BRT) Marcelo Tosatti…
Quoted 6 lines

I already thought about that and tried. In fact it is as fast and fine as 2 GB setup. It runs really smooth.
The really simple test for the problem is running "updatedb" (find over the whole filesystem). The box comes to a crawl while this is running, network is absolutely bad, interactivity is rather dead, very often not even a ssh login works.

Does -pre4 (with the VM changes from Andrea) show any difference? There are significant changes in the per-zone decisions which might help.

Hello Marcelo,

Quoted 7 lines

it looks like -pre4 performs not well even in 4 GB environment. After few days of running I find hanging 2.4.22 nfs-clients on a 2.4.23-pre4 server.

On the client I get a bunch of those:

Sep 16 03:02:00 brenda kernel: nfs: server 192.168.1.1 OK

[...]

Hello again,

you will love to hear that you can drop the above statement completely. After digging deeper into the case I found out that it was caused by a dead switch. So this is no -pre4 problem, but a hardware issue. The switch corrupted about every 20th packet.
So I can tell nothing negative so far about -pre4 with 4 GB.
I'll try with 6 GB now.

Regards,
Stephan

#9 lore raw
In reply to Stephan von Krawczynski
Quoted 10 lines · Stephan von Krawczynski
Sep 16 03:03:09 brenda last message repeated 2 times
Sep 16 03:03:09 brenda kernel: nfs: server 192.168.1.1 OK 
Sep 16 03:03:10 brenda last message repeated 2 times

And then the nfs-action is dead. Process hangs. This has not happened with 2.4.22 as a server. It showed up after a day of creating 4,7 GB dvd iso images. Creating theses isos was ok, no error or so during the action.
Is it possible that pre4 does not recover all that well from former memory pressure?

Quoted 4 lines
Quoted 1 line

Have you tried 2.4.22-aa?

Sorry, not yet.
I will go back to 2.4.22 and stress it to see if these effects show up.

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Maybe you have a SCSI card+disks to test ? 8)

#10 lore raw
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Maybe you have a SCSI card+disks to test ? 8)

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...

Regards,
Stephan

#11 lore raw
Quoted 15 lines · Stephan von Krawczynski
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Maybe you have a SCSI card+disks to test ? 8)

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...

Quoted 2 lines

Regards,
Stephan

Can you explain what you mean by "network connections go bogus". Whether or not you have more that 4GB or RAM and therefore have to page it into lower virtual addresses has nothing at all to do with networking unless you have a network device driver that did not allocate memory properly. If so, that should be checked out.

Since there is only 32 bits of address space in the Intel machines, the virtual memory seen by a process can't exceed that, including the kernel itself. However, the physical memory can come from anywhere and that's what the extended-memory specification attempts to provide. If something is hurting the network, it shouldn't be so.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (794.73 BogoMips).
Note 96.31% of all statistics are fiction.

#12 lore raw
Quoted 1 line · On Tue, 16 Sep 2003 09:55:36 -0400 (EDT) "Richard B. Johnson"…

Can you explain what you mean by "network connections go bogus".

Sure. Do this:

Use a controller (like 3ware) that cannot DMA beyond a 4 GB range. Now put some GBs of data onto that and start to tar the data around on the same disk. While doing this you can watch network go crazy and drop packets at will. You can of course force the whole setup further by (trying) additional nfs-action, but this is really not needed.
Just about the same thing can be experienced if you do a simple find all over the disk. CPU load explodes and network is _dead_.
To try this you need some GBs of data, a box with more than 4 GB ram, 3ware controller and a usual SuSE 8.2. Wait until after midnight for "updatedb" to run and try to login during that time ;-)
"Works" with out-of-the-box equipment and distro :-)

Regards,
Stephan

#13 lore raw
In reply to Stephan von Krawczynski
Quoted 5 lines · Stephan von Krawczynski
Quoted 2 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Quoted 7 lines
Quoted 1 line

Maybe you have a SCSI card+disks to test ? 8)

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable.

All is fine with 4GB?

Quoted 3 lines

Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...

No way to omit the bounce buffers.

#14 lore raw
Quoted 7 lines · On Tue, 16 Sep 2003 11:33:27 -0300 (BRT) Marcelo Tosatti…
Quoted 5 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable.

All is fine with 4GB?

Absolutely perfect.

Regards,
Stephan

#15 lore raw
In reply to Stephan von Krawczynski
Quoted 6 lines · Stephan von Krawczynski

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.

I don't know anyone who explored that however

#16 lore raw
Quoted 10 lines · On Tue, 16 Sep 2003 15:36:14 +0100 Alan Cox <[email protected]> wrote:
Quoted 6 lines · Stephan von Krawczynski

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally.

Hm, "not ideally" is a real friendly word for describing the mess ;-) Isn't there a possibility to flag this part of the memory as nonDMA-able, kind of "do whatever you want with it, but don't expect any dma-driven i/o"... I know this gets a problem when swap jumps in, though.
But really it is far better for the box to flag it more or less unusable compared to a DoS done by user-space "find" ...
I know this is a real corner case of life. It looks more like taking a different decision than current to improve the situation and not so much a real development topic. Probably a note in kernel docs reading "DON'T DO THIS" is either sufficient...

Regards,
Stephan

#46 lore raw
In reply to Stephan von Krawczynski
Quoted 9 lines · Stephan von Krawczynski
Quoted 3 lines · On Tue, 16 Sep 2003 10:11:49 -0300 (BRT) Marcelo Tosatti…

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Maybe you have a SCSI card+disks to test ? 8)

Quoted 7 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...

If there were some kind of tracking to determine which processes are doing I/O which requires the process to be in low memory. Then, processes can be migrated around in physical memory so as to optimize for that.

Or is that already being done?

#47 lore raw
In reply to Stephan von Krawczynski
Quoted 11 lines
Quoted 3 lines

Oh... Jens just pointed bounce buffering is needed for the upper 2Gs.

Maybe you have a SCSI card+disks to test ? 8)

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?
We should not leave the current bad situation as is...

It won't need to bounce buffer if you have a decent driver & hardware.

M.

#18 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Maybe not what you WILL DO, but what you HAVE DONE. Those tasks which have done the most DMA-requiring I/O could get preference for low memory. No?

Yeah, I know.. show you the code. :)

#19 lore raw
Quoted 11 lines · On Tue, 16 Sep 2003 11:49:10 -0400 Timothy Miller <[email protected]> wrote:
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Maybe not what you WILL DO, but what you HAVE DONE. Those tasks which have done the most DMA-requiring I/O could get preference for low memory. No?

Yeah, I know.. show you the code. :)

That really sounds complex and therefore I would not try that in first place. And it does not sound like a solution to my special problem of few tasks operating on a lot of data (a lot more than fits to physical mem). For this type of situation it would be really intelligent not to give away non-dma-able memory. How can the kernel do that?
At least it knows that there are I/O devices that cannot cope with the mem it is presenting. This sounds in fact simple.
The problem is: what _can_ be done with this type of mem at all? It does not even help a lot if there are other devices that can handle it, because you still must face the fact of a simple file-copy from such a capable device to one that is not.
No wonder that the gurus did not have any good idea yet ;-)

Regards,
Stephan

#29 lore raw
In reply to Alan Cox

Hi!

Quoted 11 lines
Quoted 6 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.

If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.
Pavel

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#30 lore raw
Quoted 9 lines · Pavel Machek

Hi!

Quoted 7 lines
Quoted 6 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

Quoted 8 lines
Quoted 4 lines

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.

If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.

The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.

We're quite eager to fix the problem too, if you want us to test some things.

OG.

#20 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

OG.

#31 lore raw
In reply to Olivier Galibert

Hi!

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.

Quoted 7 lines
Quoted 3 lines

If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.

The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.

That seems like bug ;-). Can you do some kind of memstat to see if it is not something like atomic pages shortage? Also try to run vanilla kernel. And try running it UP.

Quoted 2 lines

We're quite eager to fix the problem too, if you want us to test some things.

I'm afraid I do not have big-enough box close-enough to fix that.

Does it happen with another disk driver, too? What about interrupts, are not they disabled for too long? Can you enable PREEMPT to see 'scheduling in atomic' warnings?
Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#32 lore raw
In reply to Olivier Galibert
Quoted 19 lines · Olivier Galibert
Quoted 9 lines · Pavel Machek

Hi!

Quoted 7 lines
Quoted 6 lines

Well, I do understand the bounce buffer problem, but honestly the current way of handling the situation seems questionable at least. If you ever tried such a system you notice it is a lot worse than just dumping the additional ram above 4GB. You can really watch your network connections go bogus which is just unacceptable. Is there any thinkable way to ommit the bounce buffers and still do something useful with the beyond-4GB ram parts?

Quoted 8 lines
Quoted 4 lines

The 2.6 tree is somewhat better about this but at the end of the day if your I/O subsystem can't do the job your box will not perform ideally. For some workloads its a huge win to have the extra RAM, for others the I/O is a real pain.

If he has trouble logging in, then there's a bug somewhere. Bounce buffers should not slow machine down more than 2x, and from his description it looks like way worse slowdown.

Quoted 6 lines

The box does not just slowdown, the box crawls on the floor wimpering. Nothing works except ping until the i/os are finished (and they seem to crawl too), then everything works perfectly again.

We're quite eager to fix the problem too, if you want us to test some things.

Which card and driver are you using for IO? 3ware?

How much RAM do you have?

I remember I tested heavy IO loads (heavy swapping and dbench) on 8GB machine and all worked fine (interactive terminal, etc) but that was a looong time ago back in 2.4.

#33 lore raw
Quoted 1 line · Marcelo Tosatti

Which card and driver are you using for IO? 3ware?

Bus 6, device 2, function 0:
SCSI storage controller: Adaptec AIC-7902 U320 (rev 3).
Bus 6, device 2, function 1:

    SCSI storage controller: Adaptec AIC-7902 U320 (#2) (rev 3).

0:
Adaptec AIC79xx driver version: 1.3.10
Adaptec AIC7902 Ultra320 SCSI adapter
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs Allocated SCBs: 32, SG List Length: 85

Target 0 Negotiation Settings

        User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
        Goal: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
        Curr: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)

Transmission Errors 0
Channel A Target 0 Lun 0 Settings
Commands Queued 6712736
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0

1:
Adaptec AIC79xx driver version: 1.3.10
Adaptec AIC7902 Ultra320 SCSI adapter
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs Allocated SCBs: 64, SG List Length: 85

Serial EEPROM:
0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x17c8 0x09f4 0x0146 0x2807 0x0010 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0x0410 0xb3d7

Target 0 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Goal: 160.000MB/s transfers (80.000MHz DT, 16bit)
Curr: 160.000MB/s transfers (80.000MHz DT, 16bit) Transmission Errors 0
Channel A Target 0 Lun 0 Settings
Commands Queued 5523328
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Channel A Target 0 Lun 1 Settings
Commands Queued 14646734
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Target 1 Negotiation Settings
User: 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)
Goal: 160.000MB/s transfers (80.000MHz DT, 16bit)
Curr: 160.000MB/s transfers (80.000MHz DT, 16bit) Transmission Errors 0
Channel A Target 1 Lun 0 Settings
Commands Queued 4
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0
Channel A Target 1 Lun 1 Settings
Commands Queued 4
Commands Active 0
Command Openings 32
Max Tagged Openings 32
Device Queue Frozen Count 0

dmesg:
SCSI subsystem driver Revision: 1.00
scsi0 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs

scsi1 : Adaptec AIC79XX PCI-X SCSI HBA DRIVER, Rev 1.3.10
<Adaptec AIC7902 Ultra320 SCSI adapter>
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs

blk: queue f7965018, I/O limit 524287Mb (mask 0x7fffffffff) (scsi0:A:0): 320.000MB/s transfers (160.000MHz DT|IU|QAS, 16bit)

scsi1:A:0:0: DV failed to configure device.  Please file a bug report against this driver.
scsi1:A:1:0: DV failed to configure device.  Please file a bug report against this driver.
  Vendor: SEAGATE   Model: ST373307LC        Rev: 0004
  Type:   Direct-Access                      ANSI SCSI revision: 03

blk: queue f7913e18, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: SUPER Model: GEM318 Rev: 0
Type: Processor ANSI SCSI revision: 02
blk: queue f7913018, I/O limit 524287Mb (mask 0x7fffffffff) scsi0:A:0:0: Tagged Queuing enabled. Depth 32 (scsi1:A:0): 160.000MB/s transfers (80.000MHz DT, 16bit)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cbc18, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cba18, I/O limit 524287Mb (mask 0x7fffffffff) (scsi1:A:1): 160.000MB/s transfers (80.000MHz DT, 16bit)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cb618, I/O limit 524287Mb (mask 0x7fffffffff)
Vendor: transtec Model: Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f78cb418, I/O limit 524287Mb (mask 0x7fffffffff)

scsi1:A:0:0: Tagged Queuing enabled.  Depth 32
scsi1:A:0:1: Tagged Queuing enabled.  Depth 32
scsi1:A:1:0: Tagged Queuing enabled.  Depth 32
scsi1:A:1:1: Tagged Queuing enabled.  Depth 32

Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0 Attached scsi disk sdc at scsi1, channel 0, id 0, lun 1 Attached scsi disk sdd at scsi1, channel 0, id 1, lun 0 Attached scsi disk sde at scsi1, channel 0, id 1, lun 1 SCSI device sda: 143374744 512-byte hdwr sectors (73408 MB) Partition check:
sda: sda1 sda2 sda3 sda4
SCSI device sdb: 2788016128 512-byte hdwr sectors (1427464 MB)
sdb: sdb1
SCSI device sdc: 2788016128 512-byte hdwr sectors (1427464 MB)
sdc: sdc1
SCSI device sdd: 4101521408 512-byte hdwr sectors (2099979 MB)
sdd: sdd1
SCSI device sde: 4101521408 512-byte hdwr sectors (2099979 MB)
sde: sde1
Attached scsi generic sg1 at scsi0, channel 0, id 6, lun 0, type 3

Quoted 1 line

How much RAM do you have?

8Gb, currently down to 4Gb using mem=4G so that things actually work.

OG.

#39 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.

4G? Above 4G? The limit should be configurable a lot earlier.

I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.

Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).

Roger.

Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****
#40 lore raw
Quoted 11 lines · Rogier Wolff
Quoted 3 lines · Alan Cox

I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.

4G? Above 4G? The limit should be configurable a lot earlier.

I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.

Quoted 6 lines

Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).

You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.

To answer one of the other questions regarding slowdown - it can be nastier than 2x, remember that for reads the copy back happens inside the interrupt handler... It would also be interesting to note (with vmstat 1) whether it's all system time, or if you see something like kswapd going crazy too. If the attached patch makes a difference, then it could be a vm issue as well.

Still doesn't change that fact that if you build a machine with 6GB of RAM and expect it to perform, then you don't add io controllers that cannot DMA to all of your RAM.

===== mm/highmem.c 1.15 vs edited =====

mm/highmem.c +1 -1
@@ -335,7 +335,7 @@
 	struct list_head *tmp;
 	struct page *page;
 
-	page = alloc_page(GFP_NOHIGHIO);
+	page = alloc_page(GFP_ATOMIC);
 	if (page)
 		return page;
 	/*
Signature · 1 line
Jens Axboe
#41 lore raw
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Rogier Wolff
Quoted 3 lines · Alan Cox

I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.

4G? Above 4G? The limit should be configurable a lot earlier.

Quoted 8 lines
Quoted 8 lines

I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.

Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap,

        ^^^^^^^^^^^^^^^^^
Quoted 8 lines
Quoted 2 lines

you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).

You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.

As I understand things (But this is from following discussions on linux-kernel from afar, not from personal poking at the code!) there is also a performance penalty for the kernel not having direct physically mapped access to RAM. We map up to 3G of virtual memory of userspace, and up to 1Gb of physical RAM into the kernel memory map for performance reasons. So if I have 2G RAM and want to keep 3G userspace, I have to use some "highmem" stuff right?

This will not directly require the use of bounce buffers, but it will require the kernel to remap regions when it needs to access them.

If this doesn't have a performance impact, why do I have the option of directly mapping 1G, 2G, or 3G?

Roger.

Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****
#42 lore raw
Quoted 35 lines · Rogier Wolff
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Rogier Wolff
Quoted 3 lines · Alan Cox

I/O is a real pain. Also in some cases it might be interesting to try using the extra RAM above the 4G boundary as a giant ram disk and using it as first swap device.

4G? Above 4G? The limit should be configurable a lot earlier.

Quoted 8 lines
Quoted 8 lines

I'd want to configure that on the machines I'm installing tomorrow. 4G RAM, but I'd rather not use the highmem stuff. I think the workload that this machine is likely to get will work very well with this setup.

Why does this have the opportunity to work better than just using the 2 or 4G of RAM? Because after you've used the bottom 1G, that might just remain there, requiring lots of IO to go through bounce buffers and memory remappings. By considering the top part of RAM as swap,

        ^^^^^^^^^^^^^^^^^
Quoted 8 lines
Quoted 2 lines

you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).

You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.

As I understand things (But this is from following discussions on linux-kernel from afar, not from personal poking at the code!) there is also a performance penalty for the kernel not having direct physically mapped access to RAM. We map up to 3G of virtual memory of userspace, and up to 1Gb of physical RAM into the kernel memory map for performance reasons. So if I have 2G RAM and want to keep 3G userspace, I have to use some "highmem" stuff right?

Quoted 2 lines

This will not directly require the use of bounce buffers, but it will require the kernel to remap regions when it needs to access them.

That is completely correct. Your original post just didn't make this distinction, and there is an order of magnitude performance difference between kmap() and bouncing! :)

Quoted 2 lines

If this doesn't have a performance impact, why do I have the option of directly mapping 1G, 2G, or 3G?

It does cost something of course, but not nearly as expensive as bounce buffering. You cannot compare the two.

Signature · 1 line
Jens Axboe
#34 lore raw
In reply to Olivier Galibert
Quoted 8 lines · On Tue, 16 Sep 2003 23:23:02 +0200 Olivier Galibert…
Quoted 1 line · Marcelo Tosatti

Which card and driver are you using for IO? 3ware?

Bus 6, device 2, function 0:
SCSI storage controller: Adaptec AIC-7902 U320 (rev 3).
Bus 6, device 2, function 1:

    SCSI storage controller: Adaptec AIC-7902 U320 (#2) (rev 3).

Hello Olivier,

Pretty interesting.
Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...

Regards,
Stephan

#35 lore raw
Quoted 2 lines · Stephan von Krawczynski

Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...

Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.

OG.

#21 lore raw
In reply to Olivier Galibert
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

#22 lore raw
Quoted 6 lines · Alan Cox
Quoted 5 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Quoted 6 lines
Quoted 1 line

Is there a way to specifically turn that ram into a tmpfs though?

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

Signature · 1 line
Jens Axboe
#23 lore raw
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Quoted 6 lines
Quoted 3 lines

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?

Thanks

#24 lore raw
Quoted 21 lines · On Wed, 17 Sep 2003 16:30:45 -0300 (BRT) Marcelo Tosatti…
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Quoted 6 lines
Quoted 3 lines

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

Quoted 6 lines

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?

Thanks

Before running too far in this direction I would suggest to solve Oliviers' problem with the aic driver. I really would like to know if he sees the same positive effects in pre4 like me. It seems Andreas' vm patches have a very positive influence on the issue. At least I cannot see the "crawling effect" up to now with 6GB and pre4 compared to 2.4.22. It would surely be of interest if Oliviers' 8 GB variant improves, too.
May well be that the bouncing is not that bad compared to other corner effects of the vm in this special situation.
Interactivity during load and especially network seems far better in pre4. For sure it is not as speedy as a 4GB setup, but it works pretty well (up to now).

Regards,
Stephan

#25 lore raw
In reply to Marcelo Tosatti
Quoted 14 lines · Marcelo Tosatti
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...

Quoted 2 lines

What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?

Nothing, afaic. 2.6 isn't even completely deadlock free when it comes to bounce buffering.

Signature · 1 line
Jens Axboe
#26 lore raw
Quoted 19 lines · Jens Axboe
Quoted 18 lines · Marcelo Tosatti
Quoted 14 lines · Jens Axboe
Quoted 12 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...

This is the latest I had, for 2.4.5. Pretty simple and nonintrusive at that time.

arch/i386/mm/init.c renamed from opt/kernel/linux-2.4.5/arch/i386/mm/init.c +9 -5
diff -ur --exclude-from /home/axboe/exclude /opt/kernel/linux-2.4.5/arch/i386/mm/init.c linux/arch/i386/mm/init.c
@@ -25,6 +25,7 @@
 #include <linux/highmem.h>
 #include <linux/pagemap.h>
 #include <linux/bootmem.h>
+#include <linux/pci.h>
 
 #include <asm/processor.h>
 #include <asm/system.h>
@@ -348,12 +349,15 @@
 	kmap_init();
 #endif
 	{
-		unsigned long zones_size[MAX_NR_ZONES] = {0, 0, 0};
-		unsigned int max_dma, high, low;
+		unsigned long zones_size[MAX_NR_ZONES] = {0, 0, 0, 0};
+		unsigned int max_dma, max_dma32, high, low, high32;
 
 		max_dma = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;
+		max_dma32 = PCI_MAX_DMA32 >> PAGE_SHIFT;
 		low = max_low_pfn;
-		high = highend_pfn;
+		high32 = high = highend_pfn;
+		if (high32 > max_dma32)
+			high32 = max_dma32 + 1;	/* first map in HIGHMEM */
 
 		if (low < max_dma)
 			zones_size[ZONE_DMA] = low;
@@ -361,12 +365,12 @@
 			zones_size[ZONE_DMA] = max_dma;
 			zones_size[ZONE_NORMAL] = low - max_dma;
 #ifdef CONFIG_HIGHMEM
-			zones_size[ZONE_HIGHMEM] = high - low;
+			zones_size[ZONE_DMA32] = high32 - low;
+			zones_size[ZONE_HIGHMEM] = high - high32;
 #endif
 		}
 		free_area_init(zones_size);
 	}
-	return;
 }
 
 /*
include/linux/mm.h renamed from opt/kernel/linux-2.4.5/include/linux/mm.h +3 -1
diff -ur --exclude-from /home/axboe/exclude /opt/kernel/linux-2.4.5/include/linux/mm.h linux/include/linux/mm.h
@@ -476,8 +476,10 @@
 #define __GFP_IO	0x04
 #define __GFP_DMA	0x08
 #ifdef CONFIG_HIGHMEM
-#define __GFP_HIGHMEM	0x10
+#define __GFP_DMA32	0x10
+#define __GFP_HIGHMEM	0x20
 #else
+#define __GFP_DMA32	0x0 /* noop */
 #define __GFP_HIGHMEM	0x0 /* noop */
 #endif
 
include/linux/mmzone.h renamed from opt/kernel/linux-2.4.5/include/linux/mmzone.h +6 -4
diff -ur --exclude-from /home/axboe/exclude /opt/kernel/linux-2.4.5/include/linux/mmzone.h linux/include/linux/mmzone.h
@@ -27,7 +27,8 @@
  *
  * ZONE_DMA	  < 16 MB	ISA DMA capable memory
  * ZONE_NORMAL	16-896 MB	direct mapped by the kernel
- * ZONE_HIGHMEM	 > 896 MB	only page cache and user processes
+ * ZONE_DMA32	> 892MB < 4GB	For 32-bit DMA
+ * ZONE_HIGHMEM	> 4GB		only page cache and user processes
  */
 typedef struct zone_struct {
 	/*
@@ -62,8 +63,9 @@
 
 #define ZONE_DMA		0
 #define ZONE_NORMAL		1
-#define ZONE_HIGHMEM		2
-#define MAX_NR_ZONES		3
+#define ZONE_DMA32		2
+#define ZONE_HIGHMEM		3
+#define MAX_NR_ZONES		4
 
 /*
  * One allocation request operates on a zonelist. A zonelist
@@ -81,7 +83,7 @@
 	int gfp_mask;
 } zonelist_t;
 
-#define NR_GFPINDEX		0x20
+#define NR_GFPINDEX		0x40
 
 /*
  * The pg_data_t structure is used in machines with CONFIG_DISCONTIGMEM
mm/page_alloc.c renamed from opt/kernel/linux-2.4.5/mm/page_alloc.c +15 -1
diff -ur --exclude-from /home/axboe/exclude /opt/kernel/linux-2.4.5/mm/page_alloc.c linux/mm/page_alloc.c
@@ -598,6 +598,7 @@
 
 	while (pgdat) {
 		pages += pgdat->node_zones[ZONE_HIGHMEM].free_pages;
+		pages += pgdat->node_zones[ZONE_DMA32].free_pages;
 		pgdat = pgdat->node_next;
 	}
 	return pages;
@@ -683,6 +684,8 @@
 		k = ZONE_NORMAL;
 		if (i & __GFP_HIGHMEM)
 			k = ZONE_HIGHMEM;
+		if (i & __GFP_DMA32)
+			k = ZONE_DMA32;
 		if (i & __GFP_DMA)
 			k = ZONE_DMA;
 
@@ -700,6 +703,14 @@
 #endif
 					zonelist->zones[j++] = zone;
 				}
+			case ZONE_DMA32:
+				zone = pgdat->node_zones + ZONE_DMA32;
+				if (zone->size) {
+#ifndef CONFIG_HIGHMEM
+					BUG();
+#endif
+					zonelist->zones[j++] = zone;
+				}
 			case ZONE_NORMAL:
 				zone = pgdat->node_zones + ZONE_NORMAL;
 				if (zone->size)
@@ -833,8 +844,11 @@
 		for (i = 0; i < size; i++) {
 			struct page *page = mem_map + offset + i;
 			page->zone = zone;
-			if (j != ZONE_HIGHMEM)
+			if (j != ZONE_HIGHMEM && j != ZONE_DMA32) {
 				page->virtual = __va(zone_start_paddr);
+			} else
+				page->virtual = NULL;
+
 			zone_start_paddr += PAGE_SIZE;
 		}
 
Signature · 1 line
Jens Axboe
#36 lore raw
In reply to Olivier Galibert
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Stephan von Krawczynski

Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...

Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.

Actually I had booted the wrong kernel (2.6.0t4) by mistake. SCSI works, and there is indeed a remarkable difference. The system holds perfectly under filled-ram, high i/o usage now. Excellent.

Now if only the CPU enumeration worked and both CPUs were detected...

OG.

#37 lore raw
Quoted 11 lines · On Thu, 18 Sep 2003 11:58:45 +0200 Olivier Galibert <[email protected]> wrote:
Quoted 6 lines · Olivier Galibert
Quoted 2 lines · Stephan von Krawczynski

Can you please give 2.4.23-pre4 a short test. I think I can see a remarkable difference tp 2.4.22 and would like to find confirmation ...

Well, I tried but the aic7xxx does not work for me, see other mail. I'll try again once the LUN enumeration is fixed.

Actually I had booted the wrong kernel (2.6.0t4) by mistake. SCSI works, and there is indeed a remarkable difference. The system holds perfectly under filled-ram, high i/o usage now. Excellent.

Fine. So we seem to agree 2.4.23 will be another big hit in the 2.4 line :-)

Quoted 1 line

Now if only the CPU enumeration worked and both CPUs were detected...

Hm, I have not yet seen any configuration where multiple CPUs are not detected. Are you sure you have compiled in SMP support? What does dmesg look like?

Regards,
Stephan

#27 lore raw
In reply to Jens Axboe
Quoted 25 lines · On Thu, 18 Sep 2003 09:12:49 +0200 Jens Axboe <[email protected]> wrote:
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Marcelo Tosatti
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Alan Cox
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Alan Cox

The kernel has no idea what you will do with given ram. It does try to make some guesses but you are basically trying to paper over hardware limits.

Is there a way to specifically turn that ram into a tmpfs though?

Quoted 6 lines
Quoted 3 lines

Something like z2ram copied and hacked a little to kmap the blocks it wants would give you a block device you could use for swap or for /tmp. Im not sure tmpfs would work here

Aditionally, you need GFP_DMA32 or similar. Would also alleviate the nasty pressure on ZONE_NORMAL which is often quite stressed.

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

Quoted 7 lines
Quoted 2 lines

Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...

This is the latest I had, for 2.4.5. Pretty simple and nonintrusive at that time.

From a design point of view I would pretty much agree with your idea. In fact

there is a ram attribute (dma32) which currently is not reflected in the data structures in a selectable way. I can see no good reason for this lack.

Regards,
Stephan

#38 lore raw
In reply to Stephan von Krawczynski
Quoted 1 line · Stephan von Krawczynski

Fine. So we seem to agree 2.4.23 will be another big hit in the 2.4 line :-)

Indeed, it's working beautifully.

Quoted 4 lines
Quoted 1 line

Now if only the CPU enumeration worked and both CPUs were detected...

Hm, I have not yet seen any configuration where multiple CPUs are not detected. Are you sure you have compiled in SMP support? What does dmesg look like?

I found the problem. The meaning of the option "number of supported CPUs" is not what is expected. It is not fixing the maximum number of CPUs, but the number of the last CPU checked for. Specifically, in our system the CPUs are numbered 0 and 6. Setting the MNCPU to 2 prevents the second CPU to be taken into account.

OG.

#43 lore raw
In reply to Jens Axboe

Hi!

Quoted 10 lines
Quoted 3 lines

and memory remappings. By considering the top part of RAM as swap, you'll force the important stuff into the more easily accessable RAM (Compare to fastram as it was called on the Amiga!).

You are misunderstanding the problem. You don't use bounce buffers just because the page happens to reside in high memory, it is only used if the hardware cannot DMA to it. And that is exactly the problem here with the 3ware adapter, it cannot dma to > 4GB. So in a 6GB setup (with potentially 5G of highmem), only the last 2G requires bouncing.

Quoted 3 lines

To answer one of the other questions regarding slowdown - it can be nastier than 2x, remember that for reads the copy back happens inside the interrupt handler... It would also be interesting to note (with

Ouch, I guess I see. If big part of time is spent in interrupt copying data, network is going to loose packets and perform awfully. Could he run some interrupt latency tester?

Perhaps we are copying way too much in one chunk, therefore starving network?
Heh, old ide disk in PIO mode should allow that 6GB machine to perform better...
At least you don't loose packets during PIO reads...
Pavel

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#44 lore raw
Quoted 3 lines · Pavel Machek

Heh, old ide disk in PIO mode should allow that 6GB machine to perform better...
At least you don't loose packets during PIO reads...

As long as you tune it
hdparm -i1 /dev/hdX

Roger.

Quoted 1 line
Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****
#45 lore raw
Quoted 2 lines · Rogier Wolff

As long as you tune it
hdparm -i1 /dev/hdX

ehmm make that

hdparm -u1 /dev/hdX

Signature · 3 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
**** "Linux is like a wigwam -  no windows, no gates, apache inside!" ****
#28 lore raw
In reply to Jens Axboe
Quoted 1 line · Marcelo Tosatti
Quoted 1 line

IMO such GFP_DMA32 flag is a bit intrusive for 2.4, isnt it?

Quoted 2 lines · Jens Axboe

Not really, it's just an extra zone. Maybe I can dig such a patch up, I had one for 2.4.2-pre something...

Quoted 2 lines · Marcelo Tosatti
Quoted 2 lines

What has been done in 2.6 in respect to the excessive normal zone pressure and bounce buffering problems?

Quoted 2 lines · Jens Axboe

Nothing, afaic. 2.6 isn't even completely deadlock free when it comes to bounce buffering.

It'd be great to have ZONE_DMA32 around for 2.6.

-- wli