mincore() & fincore()

DISCUSSION Cédric Villemain 3 participants 6 messages Jul 25, 2013 – Jul 27, 2013 Memory management

lore

Share & tools

lore

Cédric Villemain

author

2013-07-25 14:58 UTC #0 lore raw

Hello

First, the proposed changes in this email are to be used at least for PostgreSQL extensions, maybe for core.

Purpose is to offer better monitoring/tracking of the hot/cold areas (and read/write paterns) in the tables and indexes, in PostgreSQL those are by default written in segments of 1GB.

There are some possible usecase already:

* planning of hardware upgrade
* easier configuration setup (both PostgreSQL and linux)
* provide more informations to the planner/executor of PostgreSQL

My ideas so far are to

* improve mincore() in linux and add it information like in freeBSD (at least adding 'mincore_modified' to track clean vs dirty pages).
* adding fincore() to make the information easier to grab from PostgreSQL (no mmap)
* maybe some access to those stats in /proc/

It makes years that libprefetch, mincore() and fincore() are discussed on linux mailling lists. And they got a good feedback... So I hope it is ok to keep on those and provide updated patches.

Johannes, I add you in CC because you're the last one who proposed something. Should I update your patch ?

Signature · 3 lines

Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Cédric Villemain

author

2013-07-25 15:07 UTC #1 lore raw

[sorry, previous mail was sent earlier than expected]

Quoted 21 lines

First, the proposed changes in this email are to be used at least for PostgreSQL extensions, maybe for core.
Purpose is to offer better monitoring/tracking of the hot/cold areas (and read/write paterns) in the tables and indexes, in PostgreSQL those are by default written in segments of 1GB.
There are some possible usecase already:
* planning of hardware upgrade
* easier configuration setup (both PostgreSQL and linux)
* provide more informations to the planner/executor of PostgreSQL
My ideas so far are to
* improve mincore() in linux and add it information like in freeBSD (at least adding 'mincore_modified' to track clean vs dirty pages).
* adding fincore() to make the information easier to grab from PostgreSQL (no mmap)
* maybe some access to those stats in /proc/

Quoted 3 lines

It makes years that libprefetch, mincore() and fincore() are discussed on linux mailling lists. And they got a good feedback... So I hope it is ok to keep on those and provide updated patches.

Johannes, I add you in CC because you're the last one who proposed something. Can I update your patch with previous suggestions from reviewers ?

I'm also asking for feedback in this area, others ideas are very welcome.

Signature · 3 lines

Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Johannes Weiner

2013-07-25 15:32 UTC #2 lore raw

Quoted 23 lines · Cedric Villemain

[sorry, previous mail was sent earlier than expected]
Quoted 21 lines
First, the proposed changes in this email are to be used at least for PostgreSQL extensions, maybe for core.
Purpose is to offer better monitoring/tracking of the hot/cold areas (and read/write paterns) in the tables and indexes, in PostgreSQL those are by default written in segments of 1GB.
There are some possible usecase already:
* planning of hardware upgrade
* easier configuration setup (both PostgreSQL and linux)
* provide more informations to the planner/executor of PostgreSQL
My ideas so far are to
* improve mincore() in linux and add it information like in freeBSD (at least adding 'mincore_modified' to track clean vs dirty pages).
* adding fincore() to make the information easier to grab from PostgreSQL (no mmap)
* maybe some access to those stats in /proc/

Quoted 6 lines

Quoted 3 lines
It makes years that libprefetch, mincore() and fincore() are discussed on linux mailling lists. And they got a good feedback... So I hope it is ok to keep on those and provide updated patches.
Johannes, I add you in CC because you're the last one who proposed something. Can I update your patch with previous suggestions from reviewers ?

Absolutely!

Quoted 1 line

I'm also asking for feedback in this area, others ideas are very welcome.

Andrew didn't like the idea of the one byte per covered page representation but all proposals to express continuous ranges in a more compact fashion had worse worst cases and a much more involved interface.

I do wonder if we should model fincore() after mincore() and add a separate syscall to query page cache coverage with statistical output (x present [y dirty, z active, whatever] in specified area) rather than describing individual pages or continuous chunks of pages in address order. That might leave us with better interfaces than trying to integrate all of this into one arcane syscall.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-07-26 01:56 UTC #3 lore raw

Quoted 31 lines · Johannes Weiner

Quoted 23 lines · Cedric Villemain
[sorry, previous mail was sent earlier than expected]
Quoted 21 lines
First, the proposed changes in this email are to be used at least for PostgreSQL extensions, maybe for core.
Purpose is to offer better monitoring/tracking of the hot/cold areas (and read/write paterns) in the tables and indexes, in PostgreSQL those are by default written in segments of 1GB.
There are some possible usecase already:
* planning of hardware upgrade
* easier configuration setup (both PostgreSQL and linux)
* provide more informations to the planner/executor of PostgreSQL
My ideas so far are to
* improve mincore() in linux and add it information like in freeBSD (at least adding 'mincore_modified' to track clean vs dirty pages).
* adding fincore() to make the information easier to grab from PostgreSQL (no mmap)
* maybe some access to those stats in /proc/
Quoted 6 lines
Quoted 3 lines
It makes years that libprefetch, mincore() and fincore() are discussed on linux mailling lists. And they got a good feedback... So I hope it is ok to keep on those and provide updated patches.
Johannes, I add you in CC because you're the last one who proposed something. Can I update your patch with previous suggestions from reviewers ?

Quoted 6 lines

Absolutely!
Quoted 1 line
I'm also asking for feedback in this area, others ideas are very welcome.
Andrew didn't like the idea of the one byte per covered page representation but all proposals to express continuous ranges in a

mincore utilize byte array and the least significant bit is used to check if the corresponding page is currently resident in memory, I don't know the history, what's the reason for not using bitmap?

Quoted 10 lines

more compact fashion had worse worst cases and a much more involved interface.
I do wonder if we should model fincore() after mincore() and add a separate syscall to query page cache coverage with statistical output (x present [y dirty, z active, whatever] in specified area) rather than describing individual pages or continuous chunks of pages in address order. That might leave us with better interfaces than trying to integrate all of this into one arcane syscall.

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Cédric Villemain

author

2013-07-27 20:08 UTC #4 lore raw

Quoted 5 lines

Quoted 5 lines
Quoted 3 lines
Johannes, I add you in CC because you're the last one who proposed something. Can I update your patch with previous suggestions from reviewers ?
Absolutely!

OK.

Quoted 19 lines

Quoted 5 lines
Quoted 2 lines
I'm also asking for feedback in this area, others ideas are very welcome.
Andrew didn't like the idea of the one byte per covered page representation but all proposals to express continuous ranges in a
mincore utilize byte array and the least significant bit is used to check if the corresponding page is currently resident in memory, I don't know the history, what's the reason for not using bitmap?
Quoted 9 lines
more compact fashion had worse worst cases and a much more involved interface.
I do wonder if we should model fincore() after mincore() and add a separate syscall to query page cache coverage with statistical output (x present [y dirty, z active, whatever] in specified area) rather than describing individual pages or continuous chunks of pages in address order. That might leave us with better interfaces than trying to integrate all of this into one arcane syscall.

It should works too. My tool pgfincore (for postgresql) also outputs the number of group of contiguous in-memory page, it is to get a quick idea of the access pattern: from large number of groups (random) to few groups (sequential). So for this usage, I don't really need the full vector and page level information, but some stats are needed to make those sums useful.

However another usage is to snapshot/restore in-memory pages, it is useful in at least 2 scenarios. One for simple server restart, PostgreSQL is back to full speed faster when you're able to restore the previous cache content. The other one is similar, switchover to a previously 'cold' server or prepare a server to get traffic.
For those use-cases, it is interesting to have the details.

Signature · 3 lines

Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Wanpeng Li

2013-07-26 01:56 UTC #5 lore raw

In reply to Johannes Weiner

Quoted 31 lines · Johannes Weiner

Quoted 23 lines · Cédric Villemain
[sorry, previous mail was sent earlier than expected]
Quoted 21 lines
First, the proposed changes in this email are to be used at least for PostgreSQL extensions, maybe for core.
Purpose is to offer better monitoring/tracking of the hot/cold areas (and read/write paterns) in the tables and indexes, in PostgreSQL those are by default written in segments of 1GB.
There are some possible usecase already:
* planning of hardware upgrade
* easier configuration setup (both PostgreSQL and linux)
* provide more informations to the planner/executor of PostgreSQL
My ideas so far are to
* improve mincore() in linux and add it information like in freeBSD (at least adding 'mincore_modified' to track clean vs dirty pages).
* adding fincore() to make the information easier to grab from PostgreSQL (no mmap)
* maybe some access to those stats in /proc/
Quoted 6 lines
Quoted 3 lines
It makes years that libprefetch, mincore() and fincore() are discussed on linux mailling lists. And they got a good feedback... So I hope it is ok to keep on those and provide updated patches.
Johannes, I add you in CC because you're the last one who proposed something. Can I update your patch with previous suggestions from reviewers ?

Quoted 6 lines

Absolutely!
Quoted 1 line
I'm also asking for feedback in this area, others ideas are very welcome.
Andrew didn't like the idea of the one byte per covered page representation but all proposals to express continuous ranges in a

mincore utilize byte array and the least significant bit is used to check if the corresponding page is currently resident in memory, I don't know the history, what's the reason for not using bitmap?

Quoted 10 lines

more compact fashion had worse worst cases and a much more involved interface.
I do wonder if we should model fincore() after mincore() and add a separate syscall to query page cache coverage with statistical output (x present [y dirty, z active, whatever] in specified area) rather than describing individual pages or continuous chunks of pages in address order. That might leave us with better interfaces than trying to integrate all of this into one arcane syscall.

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>