PCI memory allocation bug with CONFIG_HIGHMEM

In arch/i386/kernel/setup.c we have:

	/* Tell the PCI layer not to allocate too close to the RAM area.. */
	low_mem_size = ((max_low_pfn << PAGE_SHIFT) + 0xfffff) & ~0xfffff;
	if (low_mem_size > pci_mem_start)
		pci_mem_start = low_mem_size;

which is meant to round up pci_mem_start to the nearest 1 MB boundary past the top of physical RAM. However this does not consider highmem. Should this just be using max_pfn rather than max_low_pfn?

(I have a report of this failing on a laptop with a highmem kernel, causing a PCI memory resource to be allocated on top of a RAM area)

-- Dave

Beware - people sometimes use mem= to tell the kernel how much RAM is available for its use. Unfortunately, this overrides the E820 map, and causes the kernel to believe that all memory above the end of RAM is available for use.

This is not the case, especially on ACPI systems.

I have come to the conclusion that the use of mem= is a _very_ bad idea unless someone has an extremely good reason to override the E820 map. And even then, it must be used with extreme care, and also in combination with the reserve= parameter to ensure that reserved memory areas remain marked as such. (Reserved regions as in the ACPI data tables.)

Failure to follow this will result in non-functional PCMCIA/Cardbus because of memory resource collisions between system RAM and PCI memory space.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Yes and that was the original reason for this snippet of code. It is just a quick fix and shouldn't be needed if the E820 map is correct or if the user has specified a correct mem= parameter.

-- Dave

Yes and no. That doesn't really work either, for any machine with more than 4GB of RAM.

We want to find the memory hole (in the low 4GB region), and usually the e820 memory map should make that all happen properly. What does that report on this laptop?

This is why we put the memory resources in /proc/iomem, and mark them
busy: so that the PCI subsystem won't try to allocate PCI memory in the RAM (or ACPI reserved) area. The "pci_mem_start" thing is just a point to _start_ the allocation, the PCI subsystem still should honor the fact that we have memory above it. That's the whole point of doing proper resource allocation, after all.

Does this not work, or have you disabled e820 for some reason?

Linus

Ugh.

The original problem was actually that grub was passing a bogus mem= parameter to the kernel that was 4K too small, I guess because it was intending to indicate the amount of "available" memory (the top 4K is reserved for ACPI). If highmem had not been enabled, the above code would have corrected the problem; but with highmem, the computed low_mem_size was incorrect. I would say that grub is just broken and is misusing the mem= parameter, but this has been a problem for years and they don't seem interested in fixing it.

-- Dave

Hmm.. I suspect that it might be ok to check "max_pfn" for being less than 4GB, and use that if so. Add something like

	if (max_pfn < 0x100000)
		if (pci_mem_start < (max_pfn << PAGE_SHIFT))
			pci_mem_start  = max_pfn << PAGE_SHIFT;

to that sequence too.. I dunno. Ugly as hell. The basic issue is that if the kernel doesn't know the RAM layout, there's no way it will get things right all the time, so e820 or another other "good" memory layout should really always be used.

"mem=xxx" really doesn't work too well on modern machines. The issue is just too complex, with RAM that is reserved etc..

Linus

Actually, that would suck.

I think the proper fix would be to make the "mem=" stuff do the right thing to the iomem_resource handling, and add the "round up" code there too (and mark it as being reserved).

Basically, it shouldn't be impossible to get a "reasonably good" map from "mem=xxxx" that would work more of the time. It wouldn't necessarily be perfect, but it would be better than what we have now.

You can always use much more complicated "exactmap" stuff to really generate a full e820 map, but I suspect nobody has ever done that in real life. Something like

	mem=exactmap mem=0x9f000@0 mem=0xfe00000@0x100000 mem=0x100000$0xff00000

can be used to give you 255MB of RAM with the last 1MB marked as being "reserved".

Or it _should_ work that way. I've never used it myself ;)

Anyway, we could change what the "simple" form of "mem=xxx" means to something that is more likely to have success. Anybody willing to look at this?

Linus