computing drop-able caches

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

The point of the whole exercise is to get a better idea of free memory for our employer. Does it make sense to do this for computing free memory?

Quoted 8 lines · Daniel Walker

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

[...]

Quoted 2 lines

The point of the whole exercise is to get a better idea of free memory for our employer. Does it make sense to do this for computing free memory?

/proc/meminfo::MemAvailable was added for this purpose. See the doc text in Documentation/filesystem/proc.txt.

It's an approximation, however, because this question is not easy to answer. Pages might be in various states and uses that can make them unreclaimable.

Quoted 16 lines · Johannes Weiner

Quoted 8 lines · Daniel Walker

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

[...]

Quoted 2 lines

The point of the whole exercise is to get a better idea of free memory for our employer. Does it make sense to do this for computing free memory?

/proc/meminfo::MemAvailable was added for this purpose. See the doc text in Documentation/filesystem/proc.txt.

Quoted 3 lines

It's an approximation, however, because this question is not easy to answer. Pages might be in various states and uses that can make them unreclaimable.

Khalid was telling me that our internal sources rejected MemAvailable because it was not accurate enough. It says in the description,
"The estimate takes into account that the system needs some page cache to function well". I suspect that's part of the inaccuracy. I asked Khalid to respond with more details on this.

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But, its again based on heuristic, it takes 1/2 of page cache as reclaimable..

Quoted 5 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But, its again based on heuristic, it takes 1/2 of page cache as reclaimable..

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

We don't age cache without memory pressure, you don't know how much is used until you start taking some away. Heuristics is all we can offer.

Quoted 11 lines · Johannes Weiner

Quoted 5 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But, its again based on heuristic, it takes 1/2 of page cache as reclaimable..

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

We don't age cache without memory pressure, you don't know how much is used until you start taking some away. Heuristics is all we can offer.

With a simple busybox root system I get this,

MemTotal:          16273996 kB
MemFree:          16137920 kB
MemAvailable:   16046132 kB

shouldn't MemAvailable be at least the same as MemFree ? I changed the code somewhat so it subtracted the wmark_low only, or the pagecache/2 only, both are still under MemFree. This system has very little drop-able caches.

Quoted 5 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But, its again based on heuristic, it takes 1/2 of page cache as reclaimable..

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

Quoted 10 lines · Johannes Weiner

Quoted 7 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But,
its again based on heuristic, it takes 1/2 of page cache as reclaimable..

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

No, not quite. The page cache calculation spans two lines:

       pagecache = pages[LRU_ACTIVE_FILE] + pages[LRU_INACTIVE_FILE];
       pagecache -= min(pagecache / 2, wmark_low);

The assumption is that ALL of active & inactive file LRUs are freeable, except for the minimum of the low watermark, or half the page cache.

--
All rights reversed

Quoted 15 lines · Daniel Walker

Quoted 8 lines · Johannes Weiner

Quoted 7 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But,
its again based on heuristic, it takes 1/2 of page cache as reclaimable..

Quoted 5 lines

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

We don't age cache without memory pressure, you don't know how much is used until you start taking some away. Heuristics is all we can offer.

With a simple busybox root system I get this,

Quoted 8 lines

MemTotal:          16273996 kB
MemFree:          16137920 kB
MemAvailable:   16046132 kB

shouldn't MemAvailable be at least the same as MemFree ? I changed the code
somewhat so it subtracted the wmark_low only, or the pagecache/2 only, both
are still under MemFree. This system has very little drop-able caches.

No, a portion of memory is reserved for the kernel and not available to userland. If the kernel doesn't use it it will remain free. Hence the lower MemAvailable.

Quoted 8 lines · Johannes Weiner

Quoted 7 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]

This new metric purportedly helps usrespace assess available memory. But,
its again based on heuristic, it takes 1/2 of page cache as reclaimable..

Quoted 5 lines

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

We don't age cache without memory pressure, you don't know how much is used until you start taking some away. Heuristics is all we can offer.

With a simple busybox root system I get this,

MemTotal:          16273996 kB
MemFree:          16137920 kB
MemAvailable:   16046132 kB

shouldn't MemAvailable be at least the same as MemFree ? I changed the code somewhat so it subtracted the wmark_low only, or the pagecache/2 only, both are still under MemFree. This system has very little drop-able caches.

Daniel

Quoted 4 lines · Daniel Walker

On 01/28/2016 05:03 PM, Daniel Walker wrote:
[regarding MemAvaiable]
This new metric purportedly helps usrespace assess available memory. But, its again based on heuristic, it takes 1/2 of page cache as reclaimable..

No, it takes the smaller value of cache/2 and the low watermark, which is a fraction of memory. Actually, that does look a little weird. Rik?

We don't age cache without memory pressure, you don't know how much is used until you start taking some away. Heuristics is all we can offer.

Quoted 8 lines · Daniel Walker

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

[...]

Quoted 3 lines

The point of the whole exercise is to get a better idea of free memory for
our employer. Does it make sense to do this for computing free memory?

/proc/meminfo::MemAvailable was added for this purpose. See the doc text in Documentation/filesystem/proc.txt.

It's an approximation, however, because this question is not easy to answer. Pages might be in various states and uses that can make them unreclaimable.

Quoted 8 lines · Daniel Walker

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

[...]

Quoted 3 lines

The point of the whole exercise is to get a better idea of free memory for
our employer. Does it make sense to do this for computing free memory?

/proc/meminfo::MemAvailable was added for this purpose. See the doc text in Documentation/filesystem/proc.txt.

It's an approximation, however, because this question is not easy to answer. Pages might be in various states and uses that can make them unreclaimable.

Hi,

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Here's a quote Khalid,

"Currently there is no way to figure out the droppable pagecache size from the meminfo output. The MemFree size can shrink during normal system operation, when some of the memory pages get cached and is reflected in "Cached" field. Similarly for file operations some of the buffer memory gets cached and it is reflected in "Buffers" field. The kernel automatically reclaims all this cached & buffered memory, when it is needed elsewhere on the system. The only way to manually reclaim this memory is by writing 1 to /proc/sys/vm/drop_caches. "

So my impression is that the drop-able cache is spread over two fields in meminfo.

Alright, the question is does this info live someplace else that we don't know about? Or someplace in the kernel where it could be added to meminfo trivially ?

The point of the whole exercise is to get a better idea of free memory for our employer. Does it make sense to do this for computing free memory?

Any comments welcome..

Daniel

How do we explain the discrepancy between MemAvaiable and MemFree count after we drop cache? In following output, which one represents correct data?

[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:          747832 kB
MemAvailable:    1441736 kB
Buffers:          123976 kB
Cached:          1210272 kB
Active:          2496932 kB
Inactive:         585364 kB
Active(anon):    2243932 kB
Inactive(anon):   142676 kB
Active(file):     253000 kB
Inactive(file):   442688 kB
Dirty:                44 kB
AnonPages:       1748088 kB
Mapped:           406512 kB
Shmem:            638564 kB
Slab:              65656 kB
SReclaimable:      30120 kB
SUnreclaim:        35536 kB
KernelStack:        5920 kB
PageTables:        19040 kB
CommitLimit:     1988916 kB
Committed_AS:    3765252 kB

[Linux_0:/]$ echo 3 > /proc/sys/vm/drop_caches
[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:         1095012 kB
MemAvailable:    1434148 kB

Quoted 5 lines

[Linux_0:/]$ echo 3 > /proc/sys/vm/drop_caches
[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:         1095012 kB
MemAvailable:    1434148 kB

I suspect MemAvailable takes into account more than just the droppable caches. For instance, reclaimable slab is included, but I don't think drop_caches drops that part.

Quoted 5 lines · Daniel Walker

Quoted 5 lines

[Linux_0:/]$ echo 3 > /proc/sys/vm/drop_caches
[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:         1095012 kB
MemAvailable:    1434148 kB

Quoted 3 lines

I suspect MemAvailable takes into account more than just the droppable caches. For instance, reclaimable slab is included, but I don't think drop_caches drops that part.

There's a bit for page cache and a bit for slab, see:

https://kernel.org/doc/Documentation/sysctl/vm.txt

Quoted 6 lines · Dave Hansen

Quoted 5 lines · Daniel Walker

Quoted 5 lines

[Linux_0:/]$ echo 3 > /proc/sys/vm/drop_caches
[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:         1095012 kB
MemAvailable:    1434148 kB

Quoted 10 lines

Quoted 5 lines

I suspect MemAvailable takes into account more than just the droppable
caches. For instance, reclaimable slab is included, but I don't think
drop_caches drops that part.

There's a bit for page cache and a bit for slab, see:

https://kernel.org/doc/Documentation/sysctl/vm.txt

Ok, then this looks like a defect then. I would think MemAvailable would
always be smaller then MemFree (after echo 3 > /proc/sys/vm/drop_caches).. Unless there is something else be accounted
for that we aren't aware of.

Quoted 18 lines · Daniel Walker

Quoted 6 lines · Dave Hansen

Quoted 5 lines · Daniel Walker

Quoted 5 lines

[Linux_0:/]$ echo 3 > /proc/sys/vm/drop_caches
[Linux_0:/]$ cat /proc/meminfo
MemTotal:        3977836 kB
MemFree:         1095012 kB
MemAvailable:    1434148 kB

Quoted 10 lines

Quoted 5 lines

I suspect MemAvailable takes into account more than just the droppable
caches. For instance, reclaimable slab is included, but I don't think
drop_caches drops that part.

There's a bit for page cache and a bit for slab, see:

https://kernel.org/doc/Documentation/sysctl/vm.txt

Quoted 6 lines

Ok, then this looks like a defect then. I would think MemAvailable would
always be smaller then MemFree (after echo 3 > /proc/sys/vm/drop_caches).. Unless there is something else be accounted
for that we aren't aware of.

echo 3 > /proc/sys/vm/drop_caches will only drop unmapped page cache, IIRC

The system may still have a number of page cache pages left that are mapped in processes, but will be reclaimable if the VM needs the memory for something else.

Signature · 1 line

All rights reversed

If you look at the attached pdf, you will notice that OOM messages start to appear when memAvailable is showing 253MB (259228 KB) Free, memFree is 13.5MB (14008 KB) Free, and dropcache based calculation �Available memory� is showing 21MB (21720 KB) Free.

So, it appears that memAvailable is not as accurate, especially if data is used to warn user about system running low on memory.

Quoted 5 lines · Khalid Mughal (khalidm)

If you look at the attached pdf, you will notice that OOM messages start to appear when memAvailable is showing 253MB (259228 KB) Free, memFree is 13.5MB (14008 KB) Free, and dropcache based calculation �Available memory� is showing 21MB (21720 KB) Free.

Quoted 2 lines

So, it appears that memAvailable is not as accurate, especially if data is used to warn user about system running low on memory.

Yep, that's true.

But, MemAvailable is calculated from some very cheap counters. The "dropcache-based-calculation" requires iterating over every 4k page cache page in the system.

We track dirty and writebackw with counters, so we should theoretically be able to pull those out of MemAvailable fairly cheaply.

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Quoted 5 lines · Daniel Walker

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Code, please.

Quoted 7 lines · Dave Hansen

Quoted 5 lines · Daniel Walker

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Code, please.

We have a process for release code which doesn't allow us to send it immediately. B

Quoted 7 lines · Daniel Walker

Quoted 6 lines · Dave Hansen

Quoted 5 lines · Daniel Walker

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Quoted 3 lines

Quoted 1 line

Code, please.

We have a process for release code which doesn't allow us to send it immediately. B

OK, how about we continue this discussion once you can release it?

Quoted 7 lines · Daniel Walker

Quoted 6 lines · Dave Hansen

Quoted 5 lines · Daniel Walker

My colleague Khalid and I are working on a patch which will provide a /proc file to output the size of the drop-able page cache. One way to implement this is to use the current drop_caches /proc routine, but instead of actually droping the caches just add up the amount.

Quoted 3 lines

Quoted 1 line

Code, please.

We have a process for release code which doesn't allow us to send it immediately. B

OK, how about we continue this discussion once you can release it?