How much to malloc(), without running into swap...?

Hi all, (please cc me)

I'm hacking GNU Parted, which can (amongst other things) resize file systems. It's performance is GREATLY improved if large disk buffers are used, provided it doesn't need to swap to access the buffers ;-)

So, how can I maximize the buffer sizes, without running into swap? Note: I don't want to disable swap, because a certain (large) minimum is required for storing metadata, etc., so low-memory machines might want to use a swap (despite it being slow).

So, I want to know:
(a) how much I can malloc() without swapping
(b) how much I can malloc() with swapping

Also, I presume all IO is going to have to go through the buffer cache, etc., so having a larger buffers means more consumption on the kernel side of things. OTOH, it can probably kick out old cached data fairly quickly (i.e. data used by other programs) Any ideas on how to do the calc's?

BTW, is /proc/meminfo a good idea?

Thanks!
Andrew Clausen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

mlock() is your friend.

That's probably your best bet if your app is the only thing running in the system. Note that you'll get some feedback about things at mlock time -- if there just isn't enough memory free, mlock() will fail, so you can use that to scale back your memory requirements if other parts of the system are eating memory. With swapping, things get trickier -- you'll probably want to minimise the mlock()'d data. It would probably be better to avoid swapping altogether for your app since it would cause a ridiculous growth in the running time (and probably can be avoided).

You might want to look into using the raw devices when possible since you can probably make a better guess at the best way to order your io than the buffer cache currently will. Doing that will also get the added benefit of avoiding any excess swapping that might be generated by competition for memory by the buffer cache. Hope this helps,

-ben

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Well, the idea is to maximize the mlock() data, ensuring that you don't swap... OTOH, if it really must swap, then swap (parted) metadata, not buffers.

mlock() has given me some ideas... thanks!

Except on very low-memory systems, yes.

Interesting. But it isn't in standard 2.2.x, but it's something to look to in the future...

There's also another side effect. If it does run out of memory, it'll split the IO into short chunks (i.e., do short read()s or write()s), which obviously slows it down.

Thanks!
Andrew Clausen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/