Running out of vmalloc space
Hello. My name is David Pinedo and I subscribed to this list a few days ago. I work for Hewlett-Packard, and am in the process of porting the drivers for the FX10 graphics boards from Red Hat 6.2 to Red Hat 7.1. Our customers are begging us to support the FX10 on RH7.1, primarily because of the large memory capabilities in the 2.4.2 kernel.
I only have minimal knowledge of the Linux kernel, enough to be able to create the kernel module driver for the FX10. My apologies for jumping into this email list and the inevitable newbie mistakes I may make. I have a strong business need to get these graphics boards working correctly in the 2.4.2 kernel, and I think this email list may be the only place I can get some help. If I should be using some other forum for my questions, I would appreciate if someone would point me to it.
Porting the driver to 2.4.2 was not terribly difficult. Most of the changes were in code that translates to and from physical addresses and virtual addresses. There were a few changes I had to make due to difference in the gcc compiler on RH7.1 vs RH6.2.
The FX10 has a very large frame buffer and control space (the control space is where the registers to control the device reside). The frame buffer is 16Mbytes and the control space is 32Mbytes. The address space for the frame buffer and control space is allocated from the kernel vm address space, using get_vm_area() in mm/vmalloc.c.
On Linux, HP supports up to two FX10 boards in the system. In order to use two FX10 boards, the kernel driver needs to map the frame buffer and control space for both of the boards. That's a lot of address space, 2*(16M+32M)=96M to be exact. Using this much virtual address space on a stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to be a problem. However, a colleague reported a problem to me on his system with 1.0G of memory -- the X server was exiting with an error message indicating that it couldn't map both devices.
On investigating the problem, I found that a call to get_vm_area was failing because the kernel was running out of vmalloc space. It seems that the vmalloc space is smaller when more memory was installed on the system:
.5G RAM 1.0G RAM
---------- ---------
VMALLOC_END 0xfdffe000 0xfdffe000
VMALLOC_START 0xe0800000 0xf8800000
---------- ---------
space avail 0x1d7fe000(471M) 0x057FE000(87M)I found that if I reconfigure the kernel with Maximum Virtual Memory set to 2G (sets CONFIG_2GB), the vmalloc space is larger and the problem goes away. I couldn't quite figure out what the implications of changing Maximum Virtual Memory really are. The Help button when using "make xconfig" says there is no help available. Could someone enlighten me? Will this fix also work when I add more memory to the system?
Another method of fixing this problem that seems to work is to change the constant VMALLOC_RESERVE in arch/i386/kernel/setup.c. I changed the line that defines it from:
#define VMALLOC_RESERVE (unsigned long)(128 << 20)
to:
#define VMALLOC_RESERVE (unsigned long)(256 << 20)
What are the implications of making such a change? Will it work when there is less or more memory in the system? Should this be a configurable kernel parameter?
Thanks for any information anyone can provide.
David Pinedo
Hewlett-Packard Company
Fort Collins, Colorado
[email protected]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 8 lines · David Pinedo
On Linux, HP supports up to two FX10 boards in the system. In order to use two FX10 boards, the kernel driver needs to map the frame buffer and control space for both of the boards. That's a lot of address space, 2*(16M+32M)=96M to be exact. Using this much virtual address space on a stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to be a problem. However, a colleague reported a problem to me on his system with 1.0G of memory -- the X server was exiting with an error message indicating that it couldn't map both devices.
You obviously want to be able to map this memory into the X server's virtual address space, but do you really need to map it into the kernel's VA too?
Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 14 lines · Stephen C. Tweedie
Hi,
Quoted 8 lines · David Pinedo
On Linux, HP supports up to two FX10 boards in the system. In order to use two FX10 boards, the kernel driver needs to map the frame buffer and control space for both of the boards. That's a lot of address space, 2*(16M+32M)=96M to be exact. Using this much virtual address space on a stock RH7.1 smp kernel on a system with 0.5G of memory didn't seem to be a problem. However, a colleague reported a problem to me on his system with 1.0G of memory -- the X server was exiting with an error message indicating that it couldn't map both devices.
Quoted 6 lines
You obviously want to be able to map this memory into the X server's virtual address space, but do you really need to map it into the kernel's VA too?
Cheers,
Stephen
Unfortunately, yes. It has to be in the kernel's virtual address space, because the kernel graphics driver initiates DMAs to and from the graphics board, which can only be done from the kernel using locked down physical memory.
Thanks for all the replies to the questions I raised.
David Pinedo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 4 lines · David Pinedo
Unfortunately, yes. It has to be in the kernel's virtual address space, because the kernel graphics driver initiates DMAs to and from the graphics board, which can only be done from the kernel using locked down physical memory.
If it doesn't do any direct CPU accesses to the graphics board you could do the DMA even without having the board mapped in the MMU. You just have to work with physical addresses.
-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 4 lines · David Pinedo
Unfortunately, yes. It has to be in the kernel's virtual address space, because the kernel graphics driver initiates DMAs to and from the graphics board, which can only be done from the kernel using locked down physical memory.
Why does it have to be virtually contiguous? If you are using vmalloc then you are necessarily using physically-discontiguous space. If you are doing DMA on that space then the kernel isn't accessing it virtually at all, except perhaps to populate it, which can be done trivially page by page without having the space virtually contiguous.
It is often a lot easier on the kernel programmer if the addresses are virtually contiguous, but it is very rarely necessary. It is trivial to create an "offset_to_virt" helper function which translates an offset within one of your pci regions to a virtual kernel address by indexing a physical page location array which contains the list of allocated pages. The *only* thing which is measurably more difficult without vmalloc is crossing page boundaries.
Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 4 lines · David Pinedo
Unfortunately, yes. It has to be in the kernel's virtual address space, because the kernel graphics driver initiates DMAs to and from the graphics board, which can only be done from the kernel using locked down physical memory.
Take a look at drivers/char/raw.c on how to lock down userpages.
Christoph
Signature · 5 lines
Of course it doesn't work. We've performed a software upgrade. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/
I followed up on the suggestion of several folks to not map the graphics board into kernel vm space. While investigating how to do that, I discovered that the frame buffer space did not need to be mapped -- it was already being mapped with the control space. So instead of needing (32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change seemed easier than figuring out how not to map the board into kernel vm space, so...
Given that VMALLOC_RESERVE is 128M, and there was about 92M left when my graphics driver was being initialized, I thought I should have plenty of room to map the two graphics boards. It worked -- the first time. If I exitted the X server and restarted it, I would get the same errors from the X server as when it was not able to map one of the devices.
I verified that the kernel vm space was being freed when the X server is shut down, so that wasn't the problem. On further investigation, I found that after the X server initialized the graphics boards, somebody else was allocating a 24k chunk of vm space, immiediately after the two chunks allocated by the graphics driver. When the graphics driver tried to re-allocate those two chunks the next time the X server was started, it was able to get the first, but not the second chunk. I looked at the get_vm_area code in vmalloc.c, I found that it will not allow a vm space to be allocated from a free space that is exactly the size of the space being allocated. I think the statement:
if (size + addr < (unsigned long) tmp->addr)
break; should be:
if (size + addr <= (unsigned long) tmp->addr)
break;Making this change seems to fix my problem. :-)
David Pinedo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 7 lines · David Pinedo
I followed up on the suggestion of several folks to not map the graphics board into kernel vm space. While investigating how to do that, I discovered that the frame buffer space did not need to be mapped -- it was already being mapped with the control space. So instead of needing (32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change seemed easier than figuring out how not to map the board into kernel vm space, so...
...so you'll end up with a driver which will work fine as long as nobody tries to load it in parallel with another driver which tries to pull the same stunt. It's an easy way out which doesn't work if everybody takes the same easy way out.
I *really* think you need to be avoiding the mapping in the first place if at all possible.
Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 17 lines · Stephen C. Tweedie
Hi,
Quoted 7 lines · David Pinedo
I followed up on the suggestion of several folks to not map the graphics board into kernel vm space. While investigating how to do that, I discovered that the frame buffer space did not need to be mapped -- it was already being mapped with the control space. So instead of needing (32M+16M)*2=96M of vmalloc space, I only need 32M*2=64M. That change seemed easier than figuring out how not to map the board into kernel vm space, so...
...so you'll end up with a driver which will work fine as long as nobody tries to load it in parallel with another driver which tries to pull the same stunt. It's an easy way out which doesn't work if everybody takes the same easy way out.
Quoted 5 lines
I *really* think you need to be avoiding the mapping in the first place if at all possible.
Cheers,
Stephen
The graphics kernel driver needs to access the device, so it needs to be mapped into kernel vm space. I might be able to get away with not mapping all of the device. For example, the driver only accesses a subset of the registers on the graphics board, and doesn't access the framebuffer. I have a customer waiting for other bug fixes, so such a change would have to take a lower priority.
I could easily imagine a scenario on a future graphics device where the kernel driver accesses a large percentage of the address space of the device, so the demand on kernel vm memory space would be high. As framebuffers get larger in the future, the need for kernel vm space will also increase.
David P
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 6 lines · David Pinedo
The graphics kernel driver needs to access the device, so it needs to be mapped into kernel vm space. I might be able to get away with not mapping all of the device. For example, the driver only accesses a subset of the registers on the graphics board, and doesn't access the framebuffer. I have a customer waiting for other bug fixes, so such a change would have to take a lower priority.
That's the core of the issue. I can't see why you need direct access to all 64MB of device address space. Doing an ioremap() of the relevant register space is fine. It's what PCI device drivers are expected to do, and it will cooperate with other drivers. Mapping the entire framebuffer is excessive.
Quoted 5 lines
I could easily imagine a scenario on a future graphics device where the kernel driver accesses a large percentage of the address space of the device, so the demand on kernel vm memory space would be high. As framebuffers get larger in the future, the need for kernel vm space will also increase.
I remain to be convinced. WHY does the kernel need to access the framebuffer at all? The X server can map the entire frame buffer into its own VA without it being in the kernel. The kernel can set up DMA into the framebuffer without having that memory mapped *anywhere*. It can also talk to the acceleration engine without having the framebuffer mapped. I can't see a justification for having the whole framebuffer kernel-mapped here.
Indeed, the X server is sufficiently performance-critical that you usually want to avoid the kernel accessing the framebuffer at all: far better to do it in the X server and avoid any protection switches into the kernel domain.
Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 10 lines · David Pinedo
if (size + addr < (unsigned long) tmp->addr) break;should be:
if (size + addr <= (unsigned long) tmp->addr) break;Making this change seems to fix my problem. :-)
For the record - I sent this change on to Linus and he applied it, incorrectly attributed to myself :( --
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
[ Why vmalloc() space is so small ? ]
Hua Ji summarized quite well what the kernel does, and where.
There are 32bit machines which *can* access whole 4G kernel space separate from simultaneous 4G user space, however i386 is not one of those.
PAE36 doesn't help either -- aside of the PHYSICAL memory addressability increase, the problem is in 4G choke point in address calculations which causes 32-bit segment register value be added on 32-bits address, but only for the low 32 bits, loosing "up-shifted" top 4 bits. The mapping tables will then expand that 32-bit result to 36 bits of PAE.
If you can come up with some magic instruction which does data move from/to alternate memory mapping context than what is currently running (e.g. userspace or kernel), preferrably privileged instruction, a LOT of people would be very glad -- and very nearly overnight we could supply 4G space for both user and kernel spaces.
In Motorola 68k series there is such a thing, called 'movec'.
Quoted 6 lines
Thanks for any information anyone can provide.
David Pinedo
Hewlett-Packard Company
Fort Collins, Colorado
[email protected]
/Matti Aarnio -- who much prefers clean 64-bit pointers... --
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Quoted 8 lines · Matti Aarnio
On Thu, May 17, 2001 at 11:13:00AM -0600, David Pinedo wrote:
[ Whyvmalloc()space is so small ? ]Hua Ji summarized quite well what the kernel does, and where.
There are 32bit machines which *can* access whole 4G kernel space separate from simultaneous 4G user space, however i386 is not one of those.
Kanoj Sarcar has written a patch for Linux 2.2 to allow exactly this. Take a look at http://oss.sgi.com/projects/bigmem/, the page also contains a nice explanation of what the changes actually do.
Christoph
Signature · 5 lines
Whip me. Beat me. Make me maintain AIX. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/
Quoted 5 lines · Christoph Hellwig
Quoted 5 lines · David Pinedo
[ Why
vmalloc()space is so small ? ]Hua Ji summarized quite well what the kernel does, and where.
Quoted 7 lines
Quoted 3 lines
There are 32bit machines which *can* access whole 4G kernel space separate from simultaneous 4G user space, however i386 is not one of those.
Kanoj Sarcar has written a patch for Linux 2.2 to allow exactly this. Take a look at http://oss.sgi.com/projects/bigmem/, the page also contains a nice explanation of what the changes actually do.
It doesn't supply separate SIMULTANEOUS address spaces. It does pageing table juggling by having some 0.2 GB always mapped, and then switching things back and forth at tables.
Kanoj's approach gives larger address spaces for both spaces (user, and kernel), but it can't circumvent i386 hardware limitations.
And like Kanoj notes, there are performance penalities. If a way can be found which shows none of those penalties, Linus would accept the code, I believe.
Quoted 1 line
Christoph
/Matti Aarnio
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/