Skip to contentSkip to conversation

Multiple Memory Chunks

DISCUSSION Geert Uytterhoeven 3 participants 6 messages Aug 10, 2013 – Aug 16, 2013 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 2ff3a4b6bf1251e63cd24d4bda680c5b@localhost

#1 lore raw
Quoted 2 lines · Ingo Jürgensmann

Apparently the linux kernel only utilizes the largest memory chunk with highest priority on Amigas:

That's not entirely correct: it uses all (up to the configured limit) chunks, as long as the first chunk is the chunk with the lowest physical address.

Quoted 8 lines
[    0.000000] Ignoring memory chunk at 0x7400000:0xc00000 before the first

chunk

That's the 12 MB FastMem on the mainboard of Spice (A3000), running 3.2.0-4-amiga.

I load the kernel with a memory file:

Quoted 7 lines

amiboot-5.6 -m sys:work/debian/amiga/memfile -k vmlinux-3.2.0-4-amiga "root=/dev/sda5 video=cirrusfb:mode=800x600-60 devtmpfs.mount=1"

Memfile:
2097152
0x08000000 67108864
0x07400000 12582912

As 0x07400000 < 0x08000000, the kernel ignores the second chunk.

Quoted 5 lines

So, apparently the second FastMem chunk is ignored by the kernel itself. Is there a way to make use of the additional 12 MB RAM? Although it's not as fast as the 32bit memory on the accelerator card (64MB), it is still faster than swapping to/from disk.

If you reverse the order of the chunks, it will use both. Unfortunately the kernel will be in the slowest chunk.

Quoted 4 lines

Especially this becomes important when we want to put BigRamPlus RAM extensions into Amigas. Those are 256MB RAM extensions via ZorroIII bus and the port would highly benefit from supporting multiple memory chunks. Is this possible?

I'd expect a Zorro III RAM expansion to end up at either 0xff000000 or 0x40000000, which is a higher address than your other RAM, so it should be OK.

Now why do you have a memfile like this: a long time ago, the m68k kernel used its own mapping code for system RAM, where virtual and physical addresses differed. This was needed as unlike on PC, (a) memory doesn't always start at address zero, and (b) memory isn't always contiguous. Hence we used the MMU to map all memory chunks next to each other, starting at virtual address zero.

With the unification of memory mapping across the different architectures, ouw own mapping code (and this feature) was lost.
Due to lack of manpower, we (IIRC Roman Zippel) switched to the simplest memory model, where virtual == physical, but where memory doesn't have to start at zero.

However, it should be possible to revive the virtual mapping using generic code, as some of the big NUMA systems also needed it.

See mm/Kconfig, which offers DISCONTIGMEM (what we use), SPARSEMEM (what you need), and FLATMEM.

Note that SPARSEMEM may give better performance with BigRamPlus, as there's probably a huge gap between its physical address and the addresses of your other memory chunks.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#2 lore raw

Am 10.08.2013 um 13:46 schrieb Geert Uytterhoeven <[email protected]>:

Quoted 5 lines
Quoted 4 lines

Memfile:
2097152
0x08000000 67108864
0x07400000 12582912

As 0x07400000 < 0x08000000, the kernel ignores the second chunk.

True, but as you state yourself below, everyone would usually want to get the kernel loaded at 0x08000000, because it's way faster.

Quoted 6 lines
Quoted 4 lines

So, apparently the second FastMem chunk is ignored by the kernel itself. Is there a way to make use of the additional 12 MB RAM? Although it's not as fast as the 32bit memory on the accelerator card (64MB), it is still faster than swapping to/from disk.

If you reverse the order of the chunks, it will use both. Unfortunately the kernel will be in the slowest chunk.

Yes, and I would like to avoid this, of course. ;-) It really slows down the machine and you can even feel it... ;)

Quoted 7 lines
Quoted 4 lines

Especially this becomes important when we want to put BigRamPlus RAM extensions into Amigas. Those are 256MB RAM extensions via ZorroIII bus and the port would highly benefit from supporting multiple memory chunks. Is this possible?

I'd expect a Zorro III RAM expansion to end up at either 0xff000000 or 0x40000000, which is a higher address than your other RAM, so it should be OK.

Except that the ZorroIII RAM will be 4-6x slower than the RAM on the accelerator cards.

Quoted 2 lines

Now why do you have a memfile like this: a long time ago, the m68k kernel used its own mapping code for system RAM, where virtual and physical

Remember that there were problems with loading 3.2.0-4-amiga without the memfile.

Quoted 5 lines

addresses differed. This was needed as unlike on PC, (a) memory doesn't always start at address zero, and (b) memory isn't always contiguous. Hence we used the MMU to map all memory chunks next to each other, starting at virtual address zero.

Quoted 8 lines

With the unification of memory mapping across the different architectures, ouw own mapping code (and this feature) was lost.
Due to lack of manpower, we (IIRC Roman Zippel) switched to the simplest memory model, where virtual == physical, but where memory doesn't have to start at zero.

However, it should be possible to revive the virtual mapping using generic code, as some of the big NUMA systems also needed it.

Yeah, that would be great...

But we would still have the problem with the loading the kernel to the proper memory chunk, i.e. the fastest.
AmigaOS has the concept of priorities for memory chunks. The memory on Cyberstorm MK2 accel cards have a priority of 40 whereas normal FastRAM on the mainboard has a priority of 10 (IIRC). The Amiga memory manager automatically selects the priority based on actually memory speeds on each boot, as stated at http://amigadev.elowar.com/read/ADCD_2.1/Hardware_Manual_guide/node00D2.html and http://amigadev.elowar.com/read/ADCD_2.1/Hardware_Manual_guide/node00D4.html outlines that the fastest memory will be located between the mainboard memory and the ZorroIII memory, as you said above. So the trick is get the kernel loaded in the fastest memory chunk and then reorder the other chunks accordingly. I think it's easier to have the kernel do the right thing[TM] than to rely on a fixed version of amiboot. That way we could use the memfile to determine where the kernel will get loaded (fastest chunk) by amiboot.

Quoted 6 lines

See mm/Kconfig, which offers DISCONTIGMEM (what we use), SPARSEMEM (what you need), and FLATMEM.

Note that SPARSEMEM may give better performance with BigRamPlus, as there's probably a huge gap between its physical address and the addresses of your other memory chunks.

So, to test this, I could edit that file, make the change and recompile the kernel and see whether amiboot loads the kernel to 0x08000000 and the kernel still uses the 12 MB memory on the mainboard?

Signature · 5 lines
Ciao...            //      Fon: 0381-2744150
      Ingo       \X/       http://blog.windfluechter.net


gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc
#3 lore raw
Quoted 9 lines · Ingo Jürgensmann

Am 10.08.2013 um 13:46 schrieb Geert Uytterhoeven <[email protected]>:

Quoted 6 lines
Quoted 4 lines

So, apparently the second FastMem chunk is ignored by the kernel itself. Is there a way to make use of the additional 12 MB RAM? Although it's not as fast as the 32bit memory on the accelerator card (64MB), it is still faster than swapping to/from disk.

If you reverse the order of the chunks, it will use both. Unfortunately the kernel will be in the slowest chunk.

Yes, and I would like to avoid this, of course. ;-) It really slows down the machine and you can even feel it... ;)

We may also use the slow mainboard RAM (I get ca. 12 MiB/s) as a fast swap device.
For Z2 RAM we have our good old z2ram driver, but in the mean time there's a more generic solution in drivers/mtd, IIRC.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

Especially this becomes important when we want to put BigRamPlus RAM extensions into Amigas. Those are 256MB RAM extensions via ZorroIII bus and the port would highly benefit from supporting multiple memory chunks. Is this possible?

I'd expect a Zorro III RAM expansion to end up at either 0xff000000 or 0x40000000, which is a higher address than your other RAM, so it should be OK.

Except that the ZorroIII RAM will be 4-6x slower than the RAM on the accelerator cards.

So this is also a candidate for swap? Does anyone have benchmark results? I found that ZoRAM does 12 MiB/s.

BTW, http://www.bigbookofamigahardware.com/bboah/product.aspx?id=1936 claims it's only guaranteed to work with the original daughter board, and thus may fail with '060 accelerators.

Quoted 4 lines
Quoted 2 lines

Now why do you have a memfile like this: a long time ago, the m68k kernel used its own mapping code for system RAM, where virtual and physical

Remember that there were problems with loading 3.2.0-4-amiga without the memfile.

Yes, going out of memory when allocating the page table arrays? That won't improve when adding 256 MiB of Z3 RAM in the far end of the physical address space...

Quoted 11 lines
Quoted 11 lines

addresses differed. This was needed as unlike on PC, (a) memory doesn't always start at address zero, and (b) memory isn't always contiguous. Hence we used the MMU to map all memory chunks next to each other, starting at virtual address zero.

With the unification of memory mapping across the different architectures, ouw own mapping code (and this feature) was lost.
Due to lack of manpower, we (IIRC Roman Zippel) switched to the simplest memory model, where virtual == physical, but where memory doesn't have to start at zero.

Quoted 8 lines
Quoted 2 lines

However, it should be possible to revive the virtual mapping using generic code, as some of the big NUMA systems also needed it.

Yeah, that would be great...

But we would still have the problem with the loading the kernel to the proper memory chunk, i.e. the fastest.
AmigaOS has the concept of priorities for memory chunks. The memory on Cyberstorm MK2 accel cards have a priority of 40 whereas normal FastRAM on the mainboard has a priority of 10 (IIRC). The Amiga memory manager automatically selects the priority based on actually memory speeds on each boot, as stated at http://amigadev.elowar.com/read/ADCD_2.1/Hardware_Manual_guide/node00D2.html and http://amigadev.elowar.com/read/ADCD_2.1/Hardware_Manual_guide/node00D4.html outlines that the fastest memory will be located between the mainboard memory and the ZorroIII memory, as you said above. So the trick is get the kernel loaded in the fastest memory chunk and then reorder the other chunks accordingly. I think it's easier to have the kernel do the right thing[TM] than to rely on a fixed version of amiboot. That way we could use the memfile to determine where the kernel will get loaded (fastest chunk) by amiboot.

The NUMA code should also take into account priorities, at least for CPU affinity (although we have only one CPU).

Quoted 8 lines
Quoted 6 lines

See mm/Kconfig, which offers DISCONTIGMEM (what we use), SPARSEMEM (what you need), and FLATMEM.

Note that SPARSEMEM may give better performance with BigRamPlus, as there's probably a huge gap between its physical address and the addresses of your other memory chunks.

So, to test this, I could edit that file, make the change and recompile the kernel and see whether amiboot loads the kernel to 0x08000000 and the kernel still uses the 12 MB memory on the mainboard?

Unfortunately it's not that simple. I haven't checked the details, but I think we'll
have to change other parts in arch/m68k/mm, too. Definitely the check to ignore blocks with lower addresses has to go ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#4 lore raw

Geert Uytterhoeven dixit:

Quoted 4 lines

We may also use the slow mainboard RAM (I get ca. 12 MiB/s) as a fast swap device.
For Z2 RAM we have our good old z2ram driver, but in the mean time there's a more generic solution in drivers/mtd, IIRC.

bwh mentioned transcendent memory in
http://womble.decadent.org.uk/talks/debconf13-kernel-news/

But I think with total memory sizes at the magnitude that real m68k machines have, we’re probably better off using them for main memory as much as possible. (Compressed swap comes to mind, but even so, main memory is likely to be under pressure; this might help with machines with _very_ slow disc, or root on nfs/nbd with slow NIC, though.)

Quoted 5 lines
Quoted 1 line

Remember that there were problems with loading 3.2.0-4-amiga without the memfile.

Yes, going out of memory when allocating the page table arrays? That won't improve when adding 256 MiB of Z3 RAM in the far end of the physical address space...

Right, that’s another reason I made the 3.10 kernel much more modular and the best one to swallow the bitter pill of having to construct an initrd.

You can always build your own custom kernel, of course, but I think that Debian buildds should, as much as possible, use the proper Debian kernels of course ;-)

bye,
//mirabilos

Signature · 4 lines
<gcc> ncal.c: In function 'parsemonth': warning: comparison between pointer
and integer  • <mirabilos> ↑ hab da „in function parselmouth“ gelesen
<Natureshadow> ICH AUCH! • <Natureshadow> Ich hab gerade gedacht "Häh? Wie,
hab da parselmouth gelesen ... steht da doch auch :o?"	-- too much fanfic…
#5 lore raw
In reply to Geert Uytterhoeven

Am 13.08.2013 um 11:14 schrieb Geert Uytterhoeven <[email protected]>:

Quoted 3 lines
Quoted 1 line

Yes, and I would like to avoid this, of course. ;-) It really slows down the machine and you can even feel it... ;)

We may also use the slow mainboard RAM (I get ca. 12 MiB/s) as a fast swap device.

I would like to avoid using those kind of RAM as swap. Reasons:
- it would add an additional layer of paging in/out. Even if it is comparable fast, pages of data and code would need to swapped in and out.
- to make sense it would need an higher swap priority to get used first, but then again it will most likey result in being filled with something that accidently gets swapped out first, whereas stuff that gets frequently used and swapped in/out will still be swapped to disk.
- to avoid this we would need some kind of write through cache like it's often used for SSD hybrid disks where stuff that is often needed will reside within SSD side of that hybrid disks while other stuff will be stored on the slower spinning disk.

Quoted 3 lines
Quoted 1 line

Except that the ZorroIII RAM will be 4-6x slower than the RAM on the accelerator cards.

So this is also a candidate for swap? Does anyone have benchmark results? I found that ZoRAM does 12 MiB/s.

12 MB/s is about the maximum of the ZIII bus, so that's ok. As already said above, I'm no big fan of the swap idea, but would favor the usage as "real" memory. ;)

Quoted 3 lines

BTW, http://www.bigbookofamigahardware.com/bboah/product.aspx?id=1936 claims it's only guaranteed to work with the original daughter board, and thus may fail with '060 accelerators.

The accelerator cards reside in the CPU slot. The daughter board is the riser card where the Zorro slots are. So, this claim only counts for third party tower cases were the Desktop mainboards riser card (vertical to mainboard) gets replaced by a custom made new riser card (parallel to mainboard):

Original riser card

   --| <- riser card w/ attached cards
   --|  
   --|
   --|
     |

--------------------
mainboard

Tower mod:

riser card w/ attached cards
| mainboard
\/ \/

      | 
      |
      | 
--|   | 
--|   | 
--|---|
--|   | 
--|   | 
--|   | 
      | 
      | 
      | 
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Now why do you have a memfile like this: a long time ago, the m68k kernel used its own mapping code for system RAM, where virtual and physical

Remember that there were problems with loading 3.2.0-4-amiga without the memfile.

Yes, going out of memory when allocating the page table arrays? That won't improve when adding 256 MiB of Z3 RAM in the far end of the physical address space...

No, but we would have more memory to waste for this. Excluding 8 MB for this from 128 MB does hurt more than 8 MB out of 384 MB. ;)

But this problem was solved by using initrds now.

Quoted 2 lines

The NUMA code should also take into account priorities, at least for CPU affinity (although we have only one CPU).

Actually any Amiga with a accel card has two CPUs. Unfortunately the second CPU on the mainboard will get electrically deactivated when there is an accel card. Otherwise we could use the 68030/25 for I/O or other things. I don't know if the Copper can be used for anything as it only knows 3 commands. But I guess the effort to make it usuable would be too big for the benefit. ;)

But when the NUMA code could use the priorities for the memory and we could then use any memory that is in the machine as real main memory we would greatly benefit from that. Remember that we could maybe even plug two BigRamPlus cards into one host, resulting in 2x 256 MB Z3RAM + 128 MB AccelRAM + 16 MB mainboard + 2 MB chip instead of 128 MB AccelRAM maximum now. (the other 2 Zorro slots would be occupied by NIC and graphic card in a standard A3000).

Quoted 5 lines
Quoted 1 line

So, to test this, I could edit that file, make the change and recompile the kernel and see whether amiboot loads the kernel to 0x08000000 and the kernel still uses the 12 MB memory on the mainboard?

Unfortunately it's not that simple. I haven't checked the details, but I think we'll
have to change other parts in arch/m68k/mm, too. Definitely the check to ignore blocks with lower addresses has to go ;-)

If you think we could made this work as desired, I would order a BigRamPlus and put it into Spice so we could test it. :)

Signature · 5 lines
Ciao...            //      Fon: 0381-2744150
      Ingo       \X/       http://blog.windfluechter.net


gpg pubkey:  http://www.juergensmann.de/ij_public_key.asc
#6 lore raw
Quoted 9 lines · Ingo Jürgensmann

Am 13.08.2013 um 11:14 schrieb Geert Uytterhoeven <[email protected]>:

Quoted 3 lines
Quoted 1 line

Yes, and I would like to avoid this, of course. ;-) It really slows down the machine and you can even feel it... ;)

We may also use the slow mainboard RAM (I get ca. 12 MiB/s) as a fast swap device.

I would like to avoid using those kind of RAM as swap. Reasons:
- it would add an additional layer of paging in/out. Even if it is comparable fast, pages of data and code would need to swapped in and out.
- to make sense it would need an higher swap priority to get used first, but then again it will most likey result in being filled with something that accidently gets swapped out first, whereas stuff that gets frequently used and swapped in/out will still be swapped to disk.
- to avoid this we would need some kind of write through cache like it's often used for SSD hybrid disks where stuff that is often needed will reside within SSD side of that hybrid disks while other stuff will be stored on the slower spinning disk.

IIRC, Linux itself now also has a software variant of the SSD hybrid disk approach.
So we could use that on a RAM block device instead of an SSD.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Now why do you have a memfile like this: a long time ago, the m68k kernel used its own mapping code for system RAM, where virtual and physical

Remember that there were problems with loading 3.2.0-4-amiga without the memfile.

Yes, going out of memory when allocating the page table arrays? That won't improve when adding 256 MiB of Z3 RAM in the far end of the physical address space...

No, but we would have more memory to waste for this. Excluding 8 MB for this from 128 MB does hurt more than 8 MB out of 384 MB. ;)

I mean that we would waste more memory too, due to the Z3 RAM being far from the rest of RAM in the address space.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds