Skip to contentSkip to conversation

mm documentation

LSF/MM/BPF TOPIC Mike Rapoport 4 participants 9 messages Feb 1, 2022 – May 23, 2023 Memory management

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm suggesting this topic for a while now, maybe if we finally get to talk about it in person something will improve :)

The mm documentation is, well, not entirely up to date. We can opt for dropping the outdated parts, which would generate a nice negative diffstat, but identifying the outdated documentation requires nearly as much effort as updating it, so I think that making and keeping the docs up to date would be a better option.

I'd like to discuss what can be done process-wise to improve the situation.

Some points I had in mind:

* Pay more attention to docs during review
* Set an expectation level for docs accompanying a changeset
* Spend some cycles to review and update the existing docs
* Spend some more cycles to add new documentation
* Participate in programs like Google Season of Docs

I'd appreciate a discussion about how we can improve the existing memory management documentation so that a reader can get a coherent view of it, what are the gaps (although they are too many), and what would be the best way to close these gaps.

Signature · 2 lines
Sincerely yours,
Mike.
#1 lore raw
Quoted 23 lines · Mike Rapoport

Hi all,

I'm suggesting this topic for a while now, maybe if we finally get to talk about it in person something will improve :)

The mm documentation is, well, not entirely up to date. We can opt for dropping the outdated parts, which would generate a nice negative diffstat, but identifying the outdated documentation requires nearly as much effort as updating it, so I think that making and keeping the docs up to date would be a better option.

I'd like to discuss what can be done process-wise to improve the situation.

Some points I had in mind:

* Pay more attention to docs during review
* Set an expectation level for docs accompanying a changeset
* Spend some cycles to review and update the existing docs
* Spend some more cycles to add new documentation
* Participate in programs like Google Season of Docs

Quoted 5 lines

I'd appreciate a discussion about how we can improve the existing memory management documentation so that a reader can get a coherent view of it, what are the gaps (although they are too many), and what would be the best way to close these gaps.

I've been thinking about this since the lsf/mm discussion and wonder whether there might be some way I can contribute portions of the book that _do_ overlap the aims of the mm documentation? Specifically those parts which are descriptive, rather than the parts that are code commentary (which I still consider to be inappropriate for, and orthogonal to, the docs).

An example, as Mike pointed out on the relevant thread, is the diagram I made for the vma merge cases [1]. I feel this is quite handy for looking at this code and have used it a lot for my work in this area. There are a number of parts of the book that seem relevant like this.

HOWEVER, there are some issues here:-

1. The book is pure LaTeX. Not sure how easy it would be to port any part of it to the mm codebase.
2. I explicitly target v6 out of necessity. Therefore some explanations are simply incorrect for $curr kernel, and others which are accurate right now will be inaccurate as soon as Willy decides to change them :)
3. I don't have the time to put in the effort to port changes to $curr kernel, nor can I stand too much nitpicky review because that'll just hold up the remaining book work.

So I wonder whether it would be helpful to provide parts of this work 'as is'? I am sure there are some diagrams at the very least I can provide. I am happy to do so (and accept a GPL/whatever license for those bits).

Also, as always, I am happy to send the current WIP book to anybody in MAINTAINERS if they would like to take a look!

Relatedly, I am shortly going to be working on the page cache chapter, Willy - I wondered if you would like to take a look when I am done with that?

Cheers, Lorenzo

[1]:https://ljs.io/merge_cases.png

Quoted 4 lines

--
Sincerely yours,
Mike.

#2 lore raw
Quoted 54 lines · Lorenzo Stoakes
Quoted 23 lines · Mike Rapoport

Hi all,

I'm suggesting this topic for a while now, maybe if we finally get to talk about it in person something will improve :)

The mm documentation is, well, not entirely up to date. We can opt for dropping the outdated parts, which would generate a nice negative diffstat, but identifying the outdated documentation requires nearly as much effort as updating it, so I think that making and keeping the docs up to date would be a better option.

I'd like to discuss what can be done process-wise to improve the situation.

Some points I had in mind:

* Pay more attention to docs during review
* Set an expectation level for docs accompanying a changeset
* Spend some cycles to review and update the existing docs
* Spend some more cycles to add new documentation
* Participate in programs like Google Season of Docs

Quoted 5 lines

I'd appreciate a discussion about how we can improve the existing memory management documentation so that a reader can get a coherent view of it, what are the gaps (although they are too many), and what would be the best way to close these gaps.

I've been thinking about this since the lsf/mm discussion and wonder whether there might be some way I can contribute portions of the book that _do_ overlap the aims of the mm documentation? Specifically those parts which are descriptive, rather than the parts that are code commentary (which I still consider to be inappropriate for, and orthogonal to, the docs).

An example, as Mike pointed out on the relevant thread, is the diagram I made for the vma merge cases [1]. I feel this is quite handy for looking at this code and have used it a lot for my work in this area. There are a number of parts of the book that seem relevant like this.

HOWEVER, there are some issues here:-

1. The book is pure LaTeX. Not sure how easy it would be to port any part of it to the mm codebase.
2. I explicitly target v6 out of necessity. Therefore some explanations are simply incorrect for $curr kernel, and others which are accurate right now will be inaccurate as soon as Willy decides to change them :)
3. I don't have the time to put in the effort to port changes to $curr kernel, nor can I stand too much nitpicky review because that'll just hold up the remaining book work.

Quoted 4 lines

So I wonder whether it would be helpful to provide parts of this work 'as is'? I am sure there are some diagrams at the very least I can provide. I am happy to do so (and accept a GPL/whatever license for those bits).

If you are comfortable with random people taking your work and adjusting it to fit into Documentation/mm, I think there are a few ways to move this forward without requiring you to do the actual work.

First, I'm ready to give it a try and see how much effort is required for LaTeX to rst conversion and what are the changes that need to be made to the content. Maybe others in the community would be able to contribute too.

Next, if people ask about entry level MM projects that's something that we can suggest as well.

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Quoted 15 lines

Also, as always, I am happy to send the current WIP book to anybody in MAINTAINERS if they would like to take a look!

Relatedly, I am shortly going to be working on the page cache chapter, Willy - I wondered if you would like to take a look when I am done with that?

Cheers, Lorenzo

[1]:https://ljs.io/merge_cases.png

Quoted 4 lines

--
Sincerely yours,
Mike.

Signature · 2 lines
Sincerely yours,
Mike.
#3 lore raw

On Fri, May 12, 2023 at 10:37:32AM -0700, Mike Rapoport wrote:
[snip]

Quoted 4 lines

If you are comfortable with random people taking your work and adjusting it to fit into Documentation/mm, I think there are a few ways to move this forward without requiring you to do the actual work.

Yeah I'm not against that for snippets of the work, I do think that a large chunk of the book isn't quite so useful (the parts that are more code commentary), but there are definitely descriptive elements that are more suited.

Quoted 4 lines

First, I'm ready to give it a try and see how much effort is required for LaTeX to rst conversion and what are the changes that need to be made to the content. Maybe others in the community would be able to contribute too.

Thanks, I will try to pull out some bits that I think might be useful. I could even try converting some myself as a starting point. I wonder if a 'staging' area in the docs section could be useful? ;)

Quoted 7 lines

Next, if people ask about entry level MM projects that's something that we can suggest as well.

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

Quoted 20 lines
Quoted 15 lines

Also, as always, I am happy to send the current WIP book to anybody in MAINTAINERS if they would like to take a look!

Relatedly, I am shortly going to be working on the page cache chapter, Willy - I wondered if you would like to take a look when I am done with that?

Cheers, Lorenzo

[1]:https://ljs.io/merge_cases.png

Quoted 4 lines

--
Sincerely yours,
Mike.

--
Sincerely yours,
Mike.

#4 lore raw
Quoted 8 lines · Lorenzo Stoakes
Quoted 3 lines · Mike Rapoport

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

I'm going to suggest "MM docs" for the next Outreachy round and I'll ping you then about the "Convert Lorenzo's book to kernel documentation" project.

As for participation in the Google Season of Docs, maybe this should be more broad than only mm docs.
Jon, what do you think?

Signature · 2 lines
Sincerely yours,
Mike.
#5 lore raw
Quoted 10 lines · Mike Rapoport
Quoted 8 lines · Lorenzo Stoakes
Quoted 3 lines · Mike Rapoport

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

Quoted 7 lines

I'm going to suggest "MM docs" for the next Outreachy round and I'll ping you then about the "Convert Lorenzo's book to kernel documentation" project.

As for participation in the Google Season of Docs, maybe this should be more broad than only mm docs.
Jon, what do you think?

Sorry, I'm still catching up from travel and have a lot of digging out to do...

Converting relevant bits of the book to RST seems like a great project. In theory pandoc can do that, but I've never tried it and can imagine that a fair amount of tweaking is required.

Season of docs definitely seems worth looking into, but we've missed the bus for this year. Something to keep in mind next January.

Thanks,

jon

#6 lore raw
Quoted 18 lines · Jonathan Corbet
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

Quoted 7 lines

I'm going to suggest "MM docs" for the next Outreachy round and I'll ping you then about the "Convert Lorenzo's book to kernel documentation" project.

As for participation in the Google Season of Docs, maybe this should be more broad than only mm docs.
Jon, what do you think?

Sorry, I'm still catching up from travel and have a lot of digging out to do...

Quoted 7 lines

Converting relevant bits of the book to RST seems like a great project. In theory pandoc can do that, but I've never tried it and can imagine that a fair amount of tweaking is required.

Season of docs definitely seems worth looking into, but we've missed the bus for this year. Something to keep in mind next January.

I think both of these programs are useful for importing mm documentation that is up-to-date at the time that it's imported.

I'm concerned that the documentation will quickly become out of date, however. If mm documentation were imported before the introduction of folios or per-vma locking, for example, it would likely need large scale changes.

Maybe the idea is that any documentation, no matter how outdated, is better than no documentation. This may not always be the case, however:
how often have people become frustrated that the documentation that does exist doesn't actually reflect the current state of the kernel anymore?

Mike, I think one of the goals you were trying to achieve at LSF/MM/BPF was how this documentation would not only get originally created/imported, but also how it would stay relatively up-to-date. To keep the documentation current, I think the burden would likely need to fall on the kernel hakcers who are actually developing the code?

#7 lore raw
Quoted 28 lines · David Rientjes
Quoted 18 lines · Jonathan Corbet
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

Quoted 7 lines

I'm going to suggest "MM docs" for the next Outreachy round and I'll ping you then about the "Convert Lorenzo's book to kernel documentation" project.

As for participation in the Google Season of Docs, maybe this should be more broad than only mm docs.
Jon, what do you think?

Sorry, I'm still catching up from travel and have a lot of digging out to do...

Quoted 7 lines

Converting relevant bits of the book to RST seems like a great project. In theory pandoc can do that, but I've never tried it and can imagine that a fair amount of tweaking is required.

Season of docs definitely seems worth looking into, but we've missed the bus for this year. Something to keep in mind next January.

Quoted 8 lines

I think both of these programs are useful for importing mm documentation that is up-to-date at the time that it's imported.

I'm concerned that the documentation will quickly become out of date, however. If mm documentation were imported before the introduction of folios or per-vma locking, for example, it would likely need large scale changes.

Yeah, this is one of the ways in which the book differs quite a bit from the documentation process - I get to 'cheat' in that I have frozen the target on v6, skip all arches but x86-64 and also don't cover quite a few things for the sake of brevity/feasibility.

Key purpose is to a. describe core algorithms and concepts that should (probably) not change too much [relevant to mm docs] and b. describe code that might become/already is obsolete in considerable detail, intending to give a smaller delta to book/code + provide basis for reader to go to code [not relevant to mm docs].

Obviously kernel documentation cannot take this approach.

Quoted 5 lines

Maybe the idea is that any documentation, no matter how outdated, is better than no documentation. This may not always be the case, however:
how often have people become frustrated that the documentation that does exist doesn't actually reflect the current state of the kernel anymore?

My thinking on book/doc overlap was precisely the area for which docs can be written without too much concern about obsolescence - core concepts/algorithms that are highly unlikely to change any time soon.

This kind of area seems like a good early target for docs, and Mike has already gone down this road with the (excellent) node/zone documentation.

I do want to reiterate what I raised at LSF/MM, which is that documentation is the ultimate bikeshed bait and we do have to try to reach a bar of 'good enough' to prevent endless nitpicks slowing documentation patches to a glacial pace.

Perhaps providing broad strokes descriptions of e.g. page cache/reclaim can set the groundwork and momentum for future changes. Adjusting a document to tweak a description of how reclaim works in scenario XYZ is a lot less daunting than having to write something from scratch.

Quoted 6 lines

Mike, I think one of the goals you were trying to achieve at LSF/MM/BPF was how this documentation would not only get originally created/imported, but also how it would stay relatively up-to-date. To keep the documentation current, I think the burden would likely need to fall on the kernel hakcers who are actually developing the code?

I will let Mike comment on this obviously but from my perspective and as discussed at LSF/MM, process seems to me to be absolutely key to this - new features such as per-VMA locks or maple trees should proactively _require_ documentation in my view.

Equally changes to logic that would render documentation out of date should require changes there also.

As a hobbyist my constraints differ quite a bit from you guys working full time on this so I realise there might be difficulty justifying time on such things, but maintainers have the ability to insist + establish new requirements :)

Just my 2 English pence however, as the process side is something that maintainers need to figure out.

I feel like this is a game of two halves - the core algos are the low-hanging fruit, as they can be written at our leisure. The detailed stuff that is in constant flux really requires a change in process to be viable.

#8 lore raw
In reply to David Rientjes
Quoted 28 lines · David Rientjes
Quoted 18 lines · Jonathan Corbet
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And if we are going to participate in projects like Outreachy and Google Season of Docs, we can suggest a project "Convert Lorenzo's book to kernel documentation" alongside "Write more MM documentation" project.

Yeah, this could be a good starting point actually, as rather than starting from zero, people would have some material that they can cross-check.

Quoted 7 lines

I'm going to suggest "MM docs" for the next Outreachy round and I'll ping you then about the "Convert Lorenzo's book to kernel documentation" project.

As for participation in the Google Season of Docs, maybe this should be more broad than only mm docs.
Jon, what do you think?

Sorry, I'm still catching up from travel and have a lot of digging out to do...

Quoted 7 lines

Converting relevant bits of the book to RST seems like a great project. In theory pandoc can do that, but I've never tried it and can imagine that a fair amount of tweaking is required.

Season of docs definitely seems worth looking into, but we've missed the bus for this year. Something to keep in mind next January.

Quoted 7 lines

I think both of these programs are useful for importing mm documentation that is up-to-date at the time that it's imported.

I'm concerned that the documentation will quickly become out of date, however. If mm documentation were imported before the introduction of folios or per-vma locking, for example, it would likely need large scale changes.

Well, yes and no. Willy did a decent job of updating the docs along with the folio conversion and out of hundreds of commits only about 50 touch Documentation/. I believe there are still some gaps, but I'd imaging they are not that big.

With per-vma lock I believe there are some updates pending and if we had more elaborate documentation it would have required more work, but still I don't think it would have been something large scale, but rather semi-mechanical replacement of mmap_lock mentions with vma lock and updating a few paragraphs that actually described the locking.

But this is all theory of course :)

Quoted 4 lines

Maybe the idea is that any documentation, no matter how outdated, is better than no documentation. This may not always be the case, however:
how often have people become frustrated that the documentation that does exist doesn't actually reflect the current state of the kernel anymore?

What we have now is Mel's book and some assorted pieces here and there, so having more documentation seems really useful to me, even if it won't always be up to date.

I agree with Lorenzo that the most important bits, like, for example, general description of allocation, reclaim, compaction, won't require frequent changes and I think it'll be manageable.

Quoted 5 lines

Mike, I think one of the goals you were trying to achieve at LSF/MM/BPF was how this documentation would not only get originally created/imported, but also how it would stay relatively up-to-date. To keep the documentation current, I think the burden would likely need to fall on the kernel hakcers who are actually developing the code?

The hackers and maintainers, yes. I believe that as a part of our process we should pay more attention to documentation and ask contributors to update the documentation along with the code.
I personally do it for some time now, but unfortunately I can't always keep up with the pace.

Signature · 2 lines
Sincerely yours,
Mike.