Tools for explaining memory mappings/usage/pressure
Subsystems (2)
Hi all,
I'm trying to crowdsource information on open source tools that can be used directly by customers to explain memory mappings, usage, pressure, etc.
We encounter both internal and external users that are looking for this insight and it often requires significant engineering time to collect data to make any conclusions.
A recent example is an external customer that recently upgraded their userspace and started to run into memcg constrained memory pressure that wasn't previously observed. After handing off a hacky script to run in the background, it was immediately obvious that the source of the direct reclaim was all of the MADV_FREE memory that was sitting around. Converting to MADV_DONTNEED solved their issue.
A month ago, a different external customer was concerned about increased memory access latency in their guest on some instances although there were no issues observable on the host. After handing off a hacky script to run in the background, it was immediately obvious that memory fragmentation was resulting in a large disparity in the number of hugepages that were available on some instances.
Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file."
Are thre any open source tools that do a good job of this today that I can latch onto? If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well.
Thanks!
SeongJae Park
maintainerHi David,
On Sat, 6 Jul 2024 13:55:11 -0700 (PDT) David Rientjes <[email protected]> wrote:
[...]
Quoted 10 lines
Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file."
Are thre any open source tools that do a good job of this today that I can latch onto?
DAMON user-space tool, damo[1], provides background recording and reporting of memory access information including size of memory showing specific access pattern (e.g., working set size). Nowadays we're extending the tool to capture and provide more information for holistic and intuitive system investigations. Currently basic memory footprints and CPU usage of functions are provided.
The current status of the tool would be far from what you're looking for, though. I'm also not sure if current future plan of the tool would perfectly meet your requirements. We're open to any contributions to damo, though.
Hopefully others may know better tools for this. I'm looking forward to a chance to learn from those.
Quoted 2 lines
If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well.
We're open at not only receiving contributions for damo, but also providing contributions to other projects (and using it). So, yes, I'm interested in this :)
[1] https://github.com/awslabs/damo
Thanks,
SJ
Quoted 2 lines
Thanks!
Quoted 8 lines · David Rientjes
Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file."
Quoted 6 lines
Are thre any open source tools that do a good job of this today that I can latch onto? If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well.
Thanks!
Hi David,
At meta we have built and deployed Below[1] for this purpose. It's a tool similar to top or others, but can record system state periodically and allow for replaying. We run this on our production fleet, periodically recording system state to the local disk. When we need to debug a machine at a point in the past, we can log in and replay the state. This uses a TUI (see the link for a demo) to make navigating the data more natural.
I'm aware of a few other organizations who have also deployed Below, but tend to run it more in the manner you suggest - have it record data but then use the snapshot command to export the state (e.g. as if it was a log file) that can then be viewed off-host. Some organizations eschew the TUI altogether and export the data to Prometheus/Grafana.
I'll caution though that having the data is one thing, being able to interpret it is entirely different. While we try and put the most useful and easily-understood metrics front-and-center in the TUI, debugging an issue like you describe would probably require some domain-expertise.
Quoted 33 lines · Dan Schatzberg
Quoted 8 lines · David Rientjes
Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file."
Quoted 6 lines
Are thre any open source tools that do a good job of this today that I can latch onto? If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well.
Thanks!
Hi David,
At meta we have built and deployed Below[1] for this purpose. It's a tool similar to
topor others, but can record system state periodically and allow for replaying. We run this on our production fleet, periodically recording system state to the local disk. When we need to debug a machine at a point in the past, we can log in and replay the state. This uses a TUI (see the link for a demo) to make navigating the data more natural.I'm aware of a few other organizations who have also deployed Below, but tend to run it more in the manner you suggest - have it record data but then use the snapshot command to export the state (e.g. as if it was a log file) that can then be viewed off-host. Some organizations eschew the TUI altogether and export the data to Prometheus/Grafana.
Quoted 8 lines
I'll caution though that having the data is one thing, being able to interpret it is entirely different. While we try and put the most useful and easily-understood metrics front-and-center in the TUI, debugging an issue like you describe would probably require some domain-expertise.
Thanks Dan, this is fantastic! I've been playing with it locally.
This does indeed appear to meet the exact needs of what I was referring to above, I'm excited that this already exists.
Few questions for you:
- Do you know of anybody who has deployed this in their guest when running on a public cloud?
- Is there a motivation to add this to well known distros so it is "just there" and can run out of the box? There's some configuration and setup that it requires
- How receptive are the maintainers to adding new data points, things like additional fields from vmstat, adding in /proc/pagetypeinfo, etc?
- Any plans to support cgroup v1? :) Would that be nacked outright? Some customers still run this in their guest
- For the "/usr/bin/below record --retain-for-s 604800 --compress" support, is there an appetite for separating this out into its own non-systemd managed process? IOW, the ability to tell the customer "go run 'mini-below' and send over the data" that *just* does the record operation and doesn't require installing/configuring anything?
This could be potentially very exciting.
Happy to take this discussion off-list as well: if anybody else from this thread (or not yet on this thread) is interested, please let me know so I include you.
Thanks!
Quoted 6 lines · David Rientjes
Thanks Dan, this is fantastic! I've been playing with it locally.
This does indeed appear to meet the exact needs of what I was referring to above, I'm excited that this already exists.
Few questions for you:
Just a brief preface to my answers: Below is maintained by just a couple engineers and our primary focus is internal debugging use-cases. We welcome contributions as expanding Below's user base leads to benefits for our internal use-cases. I'll try and speak to what we would and wouldn't welcome, but before embarking on some more specific work it may be worth circling back with us to avoid misalignment.
Quoted 3 lines
- Do you know of anybody who has deployed this in their guest when running on a public cloud?
I believe so, engineers from Aviatrix have been contributing to Below recently as they have their customers use below to collect data for off-host debugging. I've heard anecdotally that Netflix has been using Below, but not entirely confident in that still being true.
Quoted 4 lines
- Is there a motivation to add this to well known distros so it is "just there" and can run out of the box? There's some configuration and setup that it requires
https://github.com/facebookincubator/below?tab=readme-ov-file#installing
It's packaged for Below, Alpine Linux and Gentoo already. We'd welcome any additional contributions to package Below for other distros so long as the maintenance burden is not too high.
Quoted 3 lines
- How receptive are the maintainers to adding new data points, things like additional fields from vmstat, adding in
/proc/pagetypeinfo, etc?
In general, we welcome contributions adding additional data collection, so long as it is sufficiently performant (e.g. collecting data for each thread in the system may require more rigour to ensure it doesn't blow up storage costs or cpu overhead of running Below) or at least made optional. Of course we expect this to be done in a fashion that doesn't overly burden the maintenance of the codebase as well.
We're a bit more scrutinizing about adding data to the TUI (more specifically, we scrutinize where the data gets added) just because adding everyone's personal favorite metric in the most prominent spot leads to UI clutter and devalues the tool as a visual guide to debugging.
Quoted 3 lines
- Any plans to support cgroup v1? :) Would that be nacked outright? Some customers still run this in their guest
No plans, but we're not opposed to contributions. I don't think it would be too challenging, just need to make sure there's some (github) testing setup for it since we are not running cgroup v1 in our internal CI.
Quoted 6 lines
- For the "
/usr/bin/belowrecord --retain-for-s 604800 --compress" support, is there an appetite for separating this out into its own non-systemd managed process? IOW, the ability to tell the customer "go run 'mini-below' and send over the data" that *just* does the record operation and doesn't require installing/configuring anything?
I think I follow what you're suggesting here - basically something fully self-contained (relies on no external configuration) to run below record followed by below snapshot or some way to record directly to a snapshot so data can be analyzed off-host. That seems perfectly reasonable. I believe Aviatrix would benefit from making this easier for their customers as well.
Quoted 10 lines · David Rientjes
Hi all,
I'm trying to crowdsource information on open source tools that can be used directly by customers to explain memory mappings, usage, pressure, etc.
We encounter both internal and external users that are looking for this insight and it often requires significant engineering time to collect data to make any conclusions.
Quoted 6 lines
A recent example is an external customer that recently upgraded their userspace and started to run into memcg constrained memory pressure that wasn't previously observed. After handing off a hacky script to run in the background, it was immediately obvious that the source of the direct reclaim was all of the
MADV_FREEmemory that was sitting around. Converting toMADV_DONTNEEDsolved their issue.
BTW, was this reported/fixed upstream? Sounds like a bug to me that would better be fixed than suggesting the MADV_DONTNEED workaround to everyone from now on.
Quoted 21 lines
A month ago, a different external customer was concerned about increased memory access latency in their guest on some instances although there were no issues observable on the host. After handing off a hacky script to run in the background, it was immediately obvious that memory fragmentation was resulting in a large disparity in the number of hugepages that were available on some instances.
Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file."
Are thre any open source tools that do a good job of this today that I can latch onto? If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well.
Thanks!
Quoted 18 lines · Vlastimil Babka (SUSE)
Quoted 10 lines · David Rientjes
Hi all,
I'm trying to crowdsource information on open source tools that can be used directly by customers to explain memory mappings, usage, pressure, etc.
We encounter both internal and external users that are looking for this insight and it often requires significant engineering time to collect data to make any conclusions.
Quoted 6 lines
A recent example is an external customer that recently upgraded their userspace and started to run into memcg constrained memory pressure that wasn't previously observed. After handing off a hacky script to run in the background, it was immediately obvious that the source of the direct reclaim was all of the
MADV_FREEmemory that was sitting around. Converting toMADV_DONTNEEDsolved their issue.
Quoted 4 lines
BTW, was this reported/fixed upstream? Sounds like a bug to me that would better be fixed than suggesting the
MADV_DONTNEEDworkaround to everyone from now on.
From the kernel perspective, it was working as intended: nearly half of the customer's memory (32GB VM instance) was lazily freeable under memory pressure and they were not expecting any reclaim stalls for future page faults. A recent upgrade of their userspace had switched heap freeing in a library from MADV_DONTNEED -> MADV_FREE implicitly without the user's direct knowledge :/
Quoted 9 lines · David Rientjes
Hi all,
I'm trying to crowdsource information on open source tools that can be used directly by customers to explain memory mappings, usage, pressure, etc.
We encounter both internal and external users that are looking for this insight and it often requires significant engineering time to collect data to make any conclusions.
Hello David,
sorry for replying late (as we took few months to make the tool "opensource").
Not sure whether it exactly fits the requirement but the tools has been very helpful for us (developers) to "visualize" the system behavior (that is exported via procfs /sysfs interface) and initial analysis.
Deploying is easy since it is provided as a python package.
Typical usage:
1. collect the behavior of workload using
$syswit collect
e.g.,
$syswit collect -c "<PWD>/syswit/collector_configs/numa_cxl.yaml" -C -T -m 1 -w "<workload>"
collected information is stored in Json format and it can be compared/ visualized further with
$syswit analyze (behavior of single workload)
OR
$syswit compare (comparing the result with multiple workload run (e.g., before / after the patch).
Still a Long way to go.. But hope it is useful.
[Idea about the tool was seeded by Bharata, Bharata/Me helped in some of the design and optimization and Ayush is the sole developer and many inside AMD helped in reviewing/improving (Shivank,....)].
USP:
1. we can tune the information to collect.
2. It can run for longer time and store the data that can be analyzed/ visualized offline for anomaly.
(Thinking whether I should also post as a main thread in linux-mm for greater awareness)
Thanks and Regards
- Raghu