2GB address space limit on 32-bit PowerPC Macintosh

The 2GB limit on TASK_SIZE for user address space on 32-bit PowerPC Apple Macintosh is getting in the way of database, filesystem maintenance, and scientific applications. In user mode, most i686 desktops can address 3GB. While 64-bit systems are becoming available, the vast majority of existing systems are still 32-bit. What can be done to get a larger address space for existing Macs?

According to arch/ppc/syslib/prom_init.c, the first problem is that CONFIG_BOOTX_TEXT restricts CONFIG_TASK_SIZE to only 0x80000000. Is this an initialization-only restriction that just happens to persist beyond initialization? What would it take to remove this collision from affecting life after boot? How much of booting does CONFIG_BOOTX_TEXT affect?

The limitation related to CONFIG_BOOTX_TEXT is only a problem on G5, which isn't supported for 32 bits kernel anymore anyway. "Normal" 32 bits kernel should work fine with TASK_SIZE set to 3Gb. However, there is another reason why it defaults to 2GB currently, and it is non-32 bits machines.

We "inherited" from some historic junk in the prep and chrp support, that a lot of embedded platforms blindly copied, where archs use io_block_mapping() early during boot to hard-wire various IO stuffs in various places in the address space, including just after 2Gb. It's totally bogus, but nobody really cared to fix it so far. The 2Gb TASK_SIZE limit doesn't seem to have ever been an issue for ppc32 users so far I must say, at least you are the first one to complain ;)

Anyway, nothing prevents you, on a Mac, to force it to 3Gb in the .config, it will work fine hopefully. If you are using a G5, just use a 64 bits kernel.

Ben.

I believe that prep and chrp are also now OK with a 3GB TASK_SIZE limit, it's just various embedded board ports that will blow up with 3GB. We should change the default to 3GB to encourage the embedded guys to fix their ports properly (or else to put in the appropriate Kconfig stuff to force it back to 2GB for their port).

Paul.

static void __init
prep_map_io(void)

{
	io_block_mapping(0x80000000, PREP_ISA_IO_BASE, 0x10000000, _PAGE_IO);
	io_block_mapping(0xf0000000, PREP_ISA_MEM_BASE, 0x08000000, _PAGE_IO);
}

We need to fix that too :) Though I suppose we can just switch that to page tables, I don't really see the point of using a BAT here...

Ben.

with

embedded

0x10000000,=20

0x08000000,=20

Are the embedded board ports broken because of similar=20 io_block_mapping() calls or for some other reason?

I'm in agreement that we should bump TASK_SIZE to 3GB and fix things,=20 how about after 2.6.12 is out?

- kumar=

Mostly because of the above, though some embedded ports may do even more horrible things for what i know :) The problem with io_block_mapping() is that 1) it lets you put those BARs or mappings where you want, and 2) you end up with plenty of code hard-coding this virtual address here or there that need to be fixed.

Agreed. I'll fix PReP/CHRP/pmac & defconfig. Every embedded board vendor/maintainer will be responsible for fixing his/her boards support.

Ben.

here...

io_block_mapping()

I'm interested to see if Paul is aware of any other embedded port=20 issues, since he mentioned the problem there.

Agreed, the other problem that comes up from time to time due to=20 io_block_mapping() that I've see is problem with modules not working=20 because the kernel doesn't think it has any vmalloc() space left.

things,

Agreed. We should probably send an email to linuxppc-embedded with a=20 more proper subject line to let people know.

- kumar

Well, we need to dig out from the part of our memory containing things we don't want to have to deal with anymore things details about the booke, 4xx, 8xx, ... memory management :)

How so ? Because we ioremap too much ? Well, that's an ooooold problem. ppc32 PCI code for example goes at length trying to minimize the amount of IO space mapped by PCI bridges (since IO space, unlike memory space, on our current PCI implementation is permanently mapped in). With a standard setup, we can only ioremap/vmalloc what is left between top of memory and 1Gb ... Unless you force your lowmem limit to something lower than the current limit which I think is 768Mb.

There isn't any reason to break everything and force updates. If someone wants
a 3G task size, then just configure that with the advanced options. All of that
grew out of the embedded board support, so just make PMac support it too. In particular, I don't want to force the 8xx to a 3G task space, because it adds more instructions to the TLB handlers, on a system that has never required such application space. We are 12 releases in 2.6, and it still seems like a development kernel :-) I think if someone wants this feature on a board port that doesn't support it, then just fix that one board port instead of breaking everything.

Thanks.

	-- Dan

I suppose it would add about ... 1 instruction :) Which is probably barely measurable compared to the cost of the cache misses of getting to the page table entires...

But I agree that we don't abolutely _need_ to break everybody. Maybe a better compromise would be to have the default beeing per CPU family ? 6xx/7xx/7xxx could default to 3G and 8xx stay at 2G...

Ben.

It's a little more than that :-)

That's still going to break nearly every embedded board. My point is that is all configurable for each individual board, so if you want that to be the default for a board, then just update the defconfig for that board. The PMac is only one "board" out of many we support, and I don't think it should be considered the default or generic configuration any more than any other board. The PMac is the easiest to update because it's a single configuration file. If you want that to have a 3G default task space, then update that one configuration file. As we have time we will update all of the others, and when we get to the point where most boards are of that configuration, we'll make it the default and the minority of boards become the special cases.

Thanks.

	-- Dan

Single ? Hrm... not so sure :) It's also the biggest (in terms of line of code) subarch of the ppc32 architecture :) Besides, pmac is also chrp & prep as I'm really talking about CONFIG_PPC_MULTIPLATFORM (too bad no embedded vendor ever tried to be part of the common kernel... heh)

Hrm... you mean the defconfig then ? Ok, right, well, I suppose we could update pmac, prep and chrp defconfigs. It's still not the default as per Kconfig which I find a little bit annoying.

On the other hands, how many embedded boards care about getting the latest "linus" tree ? I mean, I do have to update pmac support regulary as new developpement occurs, and you know as well as I do that 2.6.x series are by no mean stable. Embedded code has been rather frozen in the rock, I don't think it's much to ask from the appropriate maintainer to have a quick look at possibly upgrading their board support as well. And it isn't a difficult change for most 6xx/7xx/7xxx based boards anyway. What I'm worried about is that without some "pressure" (like breaking them), it will simply never be fixed...

The whole io_block_mapping() was a bad idea in the first place. We introduced that API to replace an even worse one which was to set BATs directly in the early days of the kernel, but I for one think we should just have killed the whole thing in the first place.

Ben.

Why are you PMac guys always treating the embedded boards as second class citizens? Don't any of you remember that all of initial PowerPC work was done on an embedded board and the PMac was a beneficiary of that work? :-)

Of course we care the "linus" tree is up to date. We use that all of the time for new product development. If it wasn't for so few of us having permission to write updates, we would probably register many more than you do.

It's not frozen, it's just more tedious because we are trying to find solutions that are commonly good for everyone, not demanding changes and forcing everyone else to adapt.

Who cares? We have the ability to support the memory map flexibility for everyone. If you need it, use it, and fix it if necessary. There are too many other things that need attention. Again, you are taking one of the features we added for embedded boards a long time ago, and
demanding everyone support it. That's not necessary. If you want
to use this feature on a PMac, then use it, but please don't demand everyone else do so. We didn't do that when the feature was added.

All of the ideas in 2.6 are going to look like bad ideas in the future :-) It's what worked and was necessary given the lack of any other APIs at the time. We are working on removing these in the embedded boards, and it has to be done on the PMac as well if you want to take advantage of something other than a 2G task space.

As time permits, I will make whatever changes to the board ports I'm aware of so they will accommodate a 3G task space, or fix up the configuration files so the default is 2G. At some point, we can make it the configuration default.

Thanks.

	-- Dan

Gack ? None of this was 'added for embedded boards'. The initial 2G/2G split was Cort's decision afaik for the PReP port, and it was PReP and pmac that introduced the use of BATs for IO mappings, which then got turned into io_block_mapping() for clarity. Those platorms predate by far any embedded involvement in the ppc kernel.

It's more than a "feature", it's the way it should have been done in the first place on 6xx/7xx/7xxx CPUs and wasn't done because of a cheap hack for IO mapping. I have no problem with other CPU family maintainers deciding to restrict TASK_SIZE for the sake of saving a couple of instructions (on 4xx, it's really replacing andis.;beq with rlwinm;cmpl;bne or something like that, so one instruction, to support any TASK_SIZE).

Hrm... If you say so ...

ioremap and variants was available at this time and has always been.

Well, you know what ? I've removed them from pmac a long time ago :) And you know what also ? what you just stated above is _exactly_ what this thread is all about ... so what are you complaining about ? :)

Good, I'm not asking for anything else here.

Ben.

Seems reasonable to me.

From a quick grep, it looks like apus, gemini, k2, lopec, pplus, prep, and prpmc800 are the offenders in the 6xx/7xx/74xx world. Does anyone even care about the lopec and k2 anymore?

Probably true...

Mark

Oops...
/me unintentionally stepped into the middle of a flame war :)

To be clear, "offender" == board that has an io_block_mapping below the 3 GB line. There are still io_block_mapping calls necessary to map regs that are used to determine memory size & whatnot but many are above the 3 GB line and even >= 0xf0000000.

Mark

Just pee on it :-)

That is one of the obvious changes. You are likely to find some other assumptions that may need attention. The problem with io_block_mapping is you just can't remove it, you have to fix up all of the code that is based on the assumption these spaces are mapped. You are likely to find yourself in a situation where you need access to some board control registers before VM is set up and you can call ioremap(). So, you will find yourself doing some hack in head.S to map BAT registers to get access to this, which is exactly what io_block_mapping() does, only in a way that is obvious :-)

Thanks.

	-- Dan

Heh!

I think I understand all of that. I wasn't clear in my email. I was talking about "fixing" such that 3GB works (i.e., io_block_mapping are above 3GB virt, if possible) not "fixing" such that we can get rid of io_block_mapping altogether. We still need io_block_mapping for exactly the reasons you state.

I need to learn how to write better emails... :(

Mark

I think I've seen hacks from you that temporarily set up BATs for serial console access until the BAT is forcefully re-used later :-) I just wanted to document the issue for everyone, that breaking this is more than just a five minute fix and test exercise (which may only further obfuscate a hack than actually change anything) :-)

Thanks.

	-- Dan

Well, I agree that way we currently do it has this issue, but I think this is because we did it wrong in the first place. We really shouldn't need to do anything before ioremap can be used. And if we need an early ioremap, we could implement bolted hash entries like we do on ppc64 :) However, don't get wrong here, I'm not 100% opposed to the use of BATs for early mappings, I just think that the whole hard-coding of the virtual address is a bad idea, and that we shouldn't have allowed them to be below KERNELBASE. The BAT could still have been "dynamically" positioned using the ioremap_bot mecanism with appropriate alignment, or at worse, we could have kept the current mecanism, but just not let it work below KERNELBASE.

Ben.

Note that there is only a problem if those io_block_mapping calls are done to map things in the virtual region between 2G and 3G. If they are done to map things "high" (like 0xf*) they are still fine.

Ben.

That would be just another "just 1 instruction" modifircation to a time critical path in a kernel which already now is so slow compared against an old 2.4 kernel that I'm really surprised to nobody seems to care.

How many systems actually need this? Why make the <1% of the systems that need it the default?

Best regards,

Wolfgang Denk

Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Q: Why do PCs have a reset button on the front?
A: Because they are expected to run Microsoft operating systems.

Well, this just one instruction is needed to make code correct anyway, because currently head_8xx.S, head_4xx.S, head_44x.s just hardcode 0x80000000 TASK_SIZE assumption, and you cannot change TASK_SIZE on all this hardware.

Eugene