Skip to contentSkip to conversation

Highmem on PPC?

DISCUSSION Val Henson 5 participants 18 messages Feb 5, 2002 – Feb 8, 2002

lore
Share & tools
lore

Val Henson

author
#0 lore raw

What's the current state of highmem on the G4? Does it work, not work, kinda work, only work on UP... ?

I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which point I'll be able to work on highmem seriously.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#1 lore raw
Quoted 6 lines

What's the current state of highmem on the G4? Does it work, not work, kinda work, only work on UP... ?

I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which point I'll be able to work on highmem seriously.

It's supposed to work. However, I'm having reports of dual 7450 with 1G or 1.5G of RAM locking up, I haven't been able to track those down.

Is this an Apple machine ? The 7450 has a bug that prevents use on SMP with HW hash table walk. Apple is supposed to implement a HW workaround for this on their machines, but if it's not an Apple machine, then you may have to implement software TLB load.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#2 lore raw
Quoted 7 lines · On Tue, Feb 05, 2002 at 01:52:56PM +0100, [email protected] wrote:
Quoted 6 lines

What's the current state of highmem on the G4? Does it work, not work, kinda work, only work on UP... ?

I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which point I'll be able to work on highmem seriously.

Quoted 7 lines

It's supposed to work. However, I'm having reports of dual 7450 with 1G or 1.5G of RAM locking up, I haven't been able to track those down.

Is this an Apple machine ? The 7450 has a bug that prevents use on SMP with HW hash table walk. Apple is supposed to implement a HW workaround for this on their machines, but if it's not an Apple machine, then you may have to implement software TLB load.

The dual 7450 is an apple machine. I also have one of our beta boards, a dual 7410 with 1 GB RAM, and we're planning on having a dual 7450 with 1 GB RAM just as soon as we get the chips.

What's your philosophy about highmem? We map all lowmem with BATs on SMP in order to avoid trashing SRR0/SRR1 according to your new comment in entry.S, but it also mentions "other cpus" without saying which cpu you're talking about. :) How are you avoiding recursive TLB faults on the highmem pages themselves? What cpus did you have in mind when you wrote the SMP stuff?

I'm currently getting hard crashes on the dual 7410 with highmem. Not even the SMI starts xmon. I'm not sure that it's a software bug, which is why I'm getting the dual 7450.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#3 lore raw
Quoted 6 lines

The dual 7450 is an apple machine. I also have one of our beta boards, a dual 7410 with 1 GB RAM, and we're planning on having a dual 7450 with 1 GB RAM just as soon as we get the chips.

Quoted 6 lines

What's your philosophy about highmem? We map all lowmem with BATs on SMP in order to avoid trashing SRR0/SRR1 according to your new comment in entry.S, but it also mentions "other cpus" without saying which cpu you're talking about. :) How are you avoiding recursive TLB faults on the highmem pages themselves? What cpus did you have in mind when you wrote the SMP stuff?

By "other CPUs", I'm thinking about CPUs that have no BATs. However, currently, and except for the Power4 which is handled differently, those are the embedded (4xx, 8xx) which don't do SMP, at least not now.

I don't think kernel stacks & code can ever be in highmem pages, so there should be no risk taking a fault on highmem page in the return from exception path.

Quoted 3 lines

I'm currently getting hard crashes on the dual 7410 with highmem. Not even the SMI starts xmon. I'm not sure that it's a software bug, which is why I'm getting the dual 7450.

Interesting. I have similar reports about dual 800s with 1.5G of RAM. I don't personally have a machine with that much RAM, so it makes things a bit difficult to debug. All I noticed so far is that enabling interrupt distribution will cause occasional lockups on my dual 7400 at work (not related to highmem). So if you plan to chase this highmem bug, start by disabling that option to avoid mixing problems.

Just in case it's a spinlock bug (who knows...) you may want to hack the spinlock debug output to use the btext engine (basically #define printf to xmon_printf in the spinlock code and hack xmon/start.c to force use_screen).

And let me know if you find something ;)

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#4 lore raw
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 4 lines

I'm currently getting hard crashes on the dual 7410 with highmem. Not even the SMI starts xmon. I'm not sure that it's a software bug, which is why I'm getting the dual 7450.

Quoted 7 lines

Interesting. I have similar reports about dual 800s with 1.5G of RAM. I don't personally have a machine with that much RAM, so it makes things a bit difficult to debug. All I noticed so far is that enabling interrupt distribution will cause occasional lockups on my dual 7400 at work (not related to highmem). So if you plan to chase this highmem bug, start by disabling that option to avoid mixing problems.

I don't use interrupt distribution.

I triggered the absolute hard hang with my usual tests: 5 or 6 of the following running in the background:

find /mnt/cdrom -exec cat {} \; > /dev/null 2>&1 & find /usr -exec cat {} \; > /dev/null 2>&1 &

And a make -j8 zImage in the foreground. I didn't even start the flood ping before it hung. :)

Quoted 4 lines

Just in case it's a spinlock bug (who knows...) you may want to hack the spinlock debug output to use the btext engine (basically #define printf to xmon_printf in the spinlock code and hack xmon/start.c to force use_screen).

This is actually over serial console, but I'll keep that in mind when I get the apple.

Quoted 1 line

And let me know if you find something ;)

:) You bet. You're sure you're not seeing these hangs? What are you using, dual 7400 with 1 GB RAM?

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#5 lore raw
Quoted 5 lines
Quoted 2 lines

And let me know if you find something ;)

:) You bet. You're sure you're not seeing these hangs? What are you using, dual 7400 with 1 GB RAM?

I don't have a machine with enough RAM to do highmem. Actually, I could probably test by hacking max_lowmem to 256Mb on my dual 7400, but it's my work prod. machine so it's a bit tricky to do such tests on it. It's not even supposed to run linux most of the time ;)

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#6 lore raw
In reply to Benjamin Herrenschmidt
Quoted 3 lines · Benjamin Herrenschmidt

Interesting. I have similar reports about dual 800s with 1.5G of RAM. I don't personally have a machine with that much RAM, so it makes things a bit difficult to debug.

Hi Ben,

When I get back from holidays I'm going to try those fixes you
recommended (using 2.4.18pre7 or later and with SUNGEM). I'm going to try it on my Dual450 first because my Dual800 is currently in production and I don't have another machine to replace it with at the moment. FWIW, the Dual800 is very stable running 2.4.18pre1 (1GB) in single CPU mode, only when I enable SMP does it lock up.

If there is any particular thing you want me to do to test this thing
(run particular progs, etc.) let me know.

Quoted 1 line

And let me know if you find something ;)

Will do.

Cheers,

Chris

--

Christopher Murtagh
Webmaster / Sysadmin
Web Communications Group
McGill University
Montreal, Quebec
Canada

Tel.: (514) 398-3122
Fax:  (514) 398-2017

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#7 lore raw
In reply to Benjamin Herrenschmidt
Quoted 2 lines · Benjamin Herrenschmidt

And let me know if you find something ;)

I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran an overnight test with:

NFS mounted root filesystem
while (true) do make clean; make -j8 zImage; done
crashme +2000 666 100 8:00:00 2
Flood ping in and out

On a dual 7410 Gemini with 1 GB RAM. Used all but 70MB of memory. No hangs.

The weird thing is that crashme was supposed to run for 8 hours but it's been running for over 14 hours now. Any ideas?

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#8 lore raw
Quoted 12 lines

I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran an overnight test with:

NFS mounted root filesystem
while (true) do make clean; make -j8 zImage; done
crashme +2000 666 100 8:00:00 2
Flood ping in and out

On a dual 7410 Gemini with 1 GB RAM. Used all but 70MB of memory. No hangs.

Quoted 8 lines

The weird thing is that crashme was supposed to run for 8 hours but it's been running for over 14 hours now. Any ideas?

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

Well, I don't know what your previous kernel had, but recent _devel has my fix to limit lowmem to what can be mapped by BATs on SMP. That may help with the 7410. If the machine ends up beeing really stable, then I'll have to look into CPU bugs for my 7450 problems.

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#11 lore raw
In reply to Val Henson
Quoted 5 lines · Val Henson

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

You could try lowering max_low_mem. You can do this by setting "Code maturity level option"/"Prompt for advanced kernel configuration options" and then going under where you selected HIGH_MEM and setting max_low_mem size to something small.

It will likely take some trial and error to get it stressed enough. You'll have to start a bunch or processes and look at where they're getting their memory

I'd be interested in hearing how it goes.

Mark

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#9 lore raw
In reply to Benjamin Herrenschmidt
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 2 lines

as a result. Any ideas for stressing the system harder?

Well, I don't know what your previous kernel had, but recent _devel has my fix to limit lowmem to what can be mapped by BATs on SMP. That may help with the 7410. If the machine ends up beeing really stable, then I'll have to look into CPU bugs for my 7450 problems.

The previous kernel was 2.4.12 based, with a hack from me to limit lowmem to 512MB. And I was using SCSI, which has hardware bugs on my board, in addition to exercising bounce buffers much more strongly. In summary, I don't have the faintest idea what was causing the hangs. I'll try booting my old 2.4.12 kernel NFS root and see if it's just as stable.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#12 lore raw
In reply to Mark A. Greer
Quoted 9 lines · Mark A. Greer
Quoted 5 lines · Val Henson

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

Quoted 8 lines

You could try lowering max_low_mem. You can do this by setting "Code maturity level option"/"Prompt for advanced kernel configuration options" and then going under where you selected HIGH_MEM and setting max_low_mem size to something small.

It will likely take some trial and error to get it stressed enough. You'll have to start a bunch or processes and look at where they're getting their memory

See, I don't think that user processes using highmem pages is testing highmem much. So we have user pages mapped by PTE's - what's changed? Not a whole lot. It's when we use bounce buffers in the kernel or ask the kernel to map user pages or otherwise trigger the kmap/kunmap code that I'm interested in.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#10 lore raw
In reply to Val Henson
Quoted 7 lines

The previous kernel was 2.4.12 based, with a hack from me to limit lowmem to 512MB. And I was using SCSI, which has hardware bugs on my board, in addition to exercising bounce buffers much more strongly. In summary, I don't have the faintest idea what was causing the hangs. I'll try booting my old 2.4.12 kernel NFS root and see if it's just as stable.

Could well be a generic kernel bug fixed in the meantime. 2.4.12 wasn't bug free...

Ben.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#13 lore raw
In reply to Val Henson
Quoted 19 lines · Val Henson
Quoted 9 lines · Mark A. Greer
Quoted 5 lines · Val Henson

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

Quoted 8 lines

You could try lowering max_low_mem. You can do this by setting "Code maturity level option"/"Prompt for advanced kernel configuration options" and then going under where you selected HIGH_MEM and setting max_low_mem size to something small.

It will likely take some trial and error to get it stressed enough. You'll have to start a bunch or processes and look at where they're getting their memory

Quoted 5 lines

See, I don't think that user processes using highmem pages is testing highmem much. So we have user pages mapped by PTE's - what's changed? Not a whole lot. It's when we use bounce buffers in the kernel or ask the kernel to map user pages or otherwise trigger the kmap/kunmap code that I'm interested in.

I agree. What I had in mind was causing the kernel to kmap pages for some reason like to copy user data into the buffer cache of a block device (see generic_file_write). I would make a pgm to allocate some mem (like the size of your lowmem), mlock it down, then fork some processes that do short writes to a block device or something like that. I think that would do it.

Did you mentioned that your scsi ctlr or drive was bad? If so, you'll have to find some other block device, /dev/ram? maybe??

Mark

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#14 lore raw
In reply to Val Henson
Quoted 13 lines · Val Henson
Quoted 2 lines · Benjamin Herrenschmidt

And let me know if you find something ;)

I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran an overnight test with:

NFS mounted root filesystem
while (true) do make clean; make -j8 zImage; done
crashme +2000 666 100 8:00:00 2
Flood ping in and out

Quoted 5 lines

On a dual 7410 Gemini with 1 GB RAM. Used all but 70MB of memory. No hangs.

The weird thing is that crashme was supposed to run for 8 hours but it's been running for over 14 hours now. Any ideas?

Hmm. Have you tried timing some 'simple' things and seeing if thats correct?

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

#15 lore raw
In reply to Val Henson
Quoted 5 lines · Val Henson

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

My Dual800 ran fine for well over a week in my office with 1GB of RAM and
with SMP. It also ran with a load average of around 10 for a couple of hours when I tried to stress test it. It was only when it started getting a bit of network traffic after moving it into our machine room did it crash, and then it crashed often. (It was also compiled with GMAC and BMAC network drivers).

So, don't know if this info helps or not, but I'd say try giving it a
network stress test as well as a heavy CPU load. Once I turned off SMP, it seems to be very stable (still with 1 GB of RAM).

Cheers,

Chris

--

Christopher Murtagh
Webmaster / Sysadmin
Web Communications Group
McGill University
Montreal, Quebec
Canada

Tel.: (514) 398-3122
Fax:  (514) 398-2017

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#16 lore raw
Quoted 15 lines · Christopher Murtagh
Quoted 5 lines · Val Henson

Also, I don't think this was a particularly good test of highmem since I don't think many bounce buffers were used, or that the kernel had much reason to map/unmap many highmem pages. Unfortunately, my SCSI controller isn't working quite right and I can't test with a hard disk as a result. Any ideas for stressing the system harder?

My Dual800 ran fine for well over a week in my office with 1GB of RAM and
with SMP. It also ran with a load average of around 10 for a couple of hours when I tried to stress test it. It was only when it started getting a bit of network traffic after moving it into our machine room did it crash, and then it crashed often. (It was also compiled with GMAC and BMAC network drivers).

Quoted 3 lines

So, don't know if this info helps or not, but I'd say try giving it a
network stress test as well as a heavy CPU load. Once I turned off SMP, it seems to be very stable (still with 1 GB of RAM).

Thanks for the suggestion, but I was doing all that work on an NFS mounted root in addition to the two flood pings. :) I think the network was being stressed quite well.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

author
#17 lore raw
In reply to Benjamin Herrenschmidt
Quoted 2 lines · Benjamin Herrenschmidt

And let me know if you find something ;)

In my quest for a non-SCSI block device, I tried using loopback over NFS. :) Okay, loopback doesn't normally work, and it's not surprising that it doesn't work over NFS with highmem enabled. If anyone is interested, this script immediately crashes on the latest 2_4_devel, only with NFS mounted root and with some memory in highmem:

#!/bin/bash
dd if=/dev/zero of=file bs=1024 count=10000
losetup /dev/loop0 file
mke2fs /dev/loop0

Exception info and backtrace from xmon (I have sysmap included on Gemini):

1:mon> e
cpu 1: vector: 300 at pc = c001239c (T memcpy+0x24), lr = c00d5b9c (t transfer_none+0x40) msr = 9032, sp = df9fdf60 [df9fdeb0]
dar = 0, dsisr = 42000000
current = df9fc000, pid = 426, comm = loop0
1:mon> t
backtrace:
c00899e4 (t nfs_prepare_write+0x10)
c00d5e5c (t lo_send+0x13c)
c00d6278 (t do_bh_filebacked+0x98)
c00d6ba0 (t loop_thread+0x194)
c00076b8 (T kernel_thread+0x2c)

I suspect that nfs_prepare_write is not properly kmapping/kunmapping something.

If anyone else feels like fixing this, feel free... It's not high on my list of priorities. :)

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/