mm: Add reclaim type to memory.reclaim
Subsystems (8)
RFC PATCH 1/1 mm: Add reclaim type to memory.reclaim
In our container environment, we've observed that certain containers may accumulate more than 40GB of slabs, predominantly negative dentries. These negative dentries remain unreclaimed unless there is memory pressure. Even after the containers exit, these negative dentries persist. To manage disk storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.
However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance. To mitigate this issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.
We propose adding a "type=" parameter to memory.reclaim to allow reclamation of pagecache pages only, slabs only, or both:
- type=1: Reclaim pagecache pages only
- type=2: Reclaim slabs only
- type=3: Reclaim both pagecache pages and slabs
For instance:
echo "1M type=1" > /sys/fs/cgroup/test/memory.reclaim
will perform the reclaim on the 'test' memcg to reclaim pagecache pages only. Please note that due to the derferred freeing of slabs, the amount of reclaimed slabs may higher than 1M during this process.
--- Documentation/admin-guide/cgroup-v2.rst | 12 ++++++++++++ include/linux/swap.h | 2 ++ mm/memcontrol.c | 22 +++++++++++++++++++++- mm/vmscan.c | 13 ++++++++++--- 4 files changed, 45 insertions(+), 4 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +12 -0
@@ -1322,6 +1322,18 @@ The following nested keys are defined. same semantics as vm.swappiness applied to memcg reclaim with all the existing limitations and potential future extensions. + ==== ============================== + type Type of memory to reclaim with + ==== ============================== + + Specifying a memory type value instructs the kernel to perform + the reclaim with that memory type. The current supported + values are: + + 1 - Reclaim pagecache pages only + 2 - Reclaim slabs only + 3 - Reclaim both pagecache pages and slabs + memory.peak A read-only single value file which exists on non-root cgroups.
include/linux/swap.h +2 -0
@@ -404,6 +404,8 @@ extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,#define MEMCG_RECLAIM_MAY_SWAP (1 << 1)#define MEMCG_RECLAIM_PROACTIVE (1 << 2) +#define MEMCG_RECLAIM_PAGECACHE_ONLY (1 << 3) +#define MEMCG_RECLAIM_SLAB_ONLY (1 << 4)#define MIN_SWAPPINESS 0#define MAX_SWAPPINESS 200 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
mm/memcontrol.c +21 -1
@@ -54,6 +54,7 @@#include <linux/seq_file.h> #include <linux/parser.h> #include <linux/vmpressure.h> +#include <linux/parser.h> #include <linux/memremap.h> #include <linux/mm_inline.h> #include <linux/swap_cgroup.h>
@@ -6930,11 +6931,13 @@ static ssize_t memory_oom_group_write(struct kernfs_open_file *of,enum {MEMORY_RECLAIM_SWAPPINESS =0 , +MEMORY_RECLAIM_TYPE =1 ,MEMORY_RECLAIM_NULL , };static const match_table_t tokens = { {MEMORY_RECLAIM_SWAPPINESS ,"swappiness=%d" }, + {MEMORY_RECLAIM_TYPE ,"type=%d" }, {MEMORY_RECLAIM_NULL ,NULL }, };
@@ -6944,7 +6947,7 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf,struct mem_cgroup *memcg =mem_cgroup_from_css (of_css (of));unsigned int nr_retries =MAX_RECLAIM_RETRIES ;unsigned long nr_to_reclaim, nr_reclaimed =0 ; -int swappiness =-1 ; +int swappiness =-1 , type =0 ;unsigned int reclaim_options;char *old_buf, *start;substring_t args[MAX_OPT_ARGS ];
@@ -6968,12 +6971,29 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf,if (swappiness <MIN_SWAPPINESS || swappiness >MAX_SWAPPINESS )return -EINVAL ;break ; +case MEMORY_RECLAIM_TYPE : +if (match_int (&args[0 ], &type)) +return -EINVAL ; +if (type >3 || type <=0 ) +return -EINVAL ; +break ;default :return -EINVAL ; } } reclaim_options =MEMCG_RECLAIM_MAY_SWAP |MEMCG_RECLAIM_PROACTIVE ; +switch (type) { +case 1 : + reclaim_options |=MEMCG_RECLAIM_PAGECACHE_ONLY ; +break ; +case 2 : + reclaim_options |=MEMCG_RECLAIM_SLAB_ONLY ; +break ; +default : +break ; + } +while (nr_reclaimed < nr_to_reclaim) {/* Will converge on zero, but reclaim enforces a minimum */ unsigned long batch_size = (nr_to_reclaim - nr_reclaimed) /4 ;
mm/vmscan.c +10 -3
@@ -141,6 +141,9 @@ struct scan_control {/* Always discard instead of demoting to lower tier memory */ unsigned int no_demotion:1; + unsigned int pagecache_only:1; + unsigned int slab_only:1; +/* Allocation order */ s8 order;
@@ -5881,10 +5884,12 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) reclaimed = sc->nr_reclaimed; scanned = sc->nr_scanned; -shrink_lruvec (lruvec, sc); +if (!sc->slab_only) +shrink_lruvec (lruvec, sc); -shrink_slab (sc->gfp_mask, pgdat->node_id, memcg, - sc->priority); +if (!sc->pagecache_only) +shrink_slab (sc->gfp_mask, pgdat->node_id, memcg, + sc->priority);/* Record the group's reclaim efficiency */ if (!sc->proactive)
@@ -6522,6 +6527,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_unmap = 1, .may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP), .proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE), + .pagecache_only = !!(reclaim_options & MEMCG_RECLAIM_PAGECACHE_ONLY), + .slab_only = !!(reclaim_options & MEMCG_RECLAIM_SLAB_ONLY), };/* * Traverse the ZONELIST_FALLBACK zonelist of the current node to put
Signature · 1 line
2.39.1
Quoted 6 lines · Yafang Shao
In our container environment, we've observed that certain containers may accumulate more than 40GB of slabs, predominantly negative dentries. These negative dentries remain unreclaimed unless there is memory pressure. Even after the containers exit, these negative dentries persist. To manage disk storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.
I understand why you've written this patch, but we really do need to fix this for non-container workloads. See also:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
I'm sure theer have been many other threads on this over the years.
Quoted 12 lines · Matthew Wilcox
Quoted 6 lines · Yafang Shao
In our container environment, we've observed that certain containers may accumulate more than 40GB of slabs, predominantly negative dentries. These negative dentries remain unreclaimed unless there is memory pressure. Even after the containers exit, these negative dentries persist. To manage disk storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.
I understand why you've written this patch, but we really do need to fix this for non-container workloads. See also:
Quoted 7 lines
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
I'm sure theer have been many other threads on this over the years.
Thank you for sharing your insights. I've reviewed the proposals and related discussions. It appears that a consensus has not yet been reached on how to tackle the issue. While I may not fully comprehend all aspects of the discussions, it seems that the challenges stemming from slab shrinking can be distilled into four key questions:
- When should the shrinker be triggered?
- Which task is responsible for performing the shrinking?
- Which slab should be reclaimed?
- How many slabs should be reclaimed?
Addressing all these questions within the kernel might introduce unnecessary complexity. Instead, one potential approach could be to extend the functionality of memory.reclaim or introduce a new interface, such as memory.shrinker, and delegate decision-making to userspace based on the workload. Since memory.reclaim is also supported in the root memcg, it can effectively address issues outside of container environments. Here's a rough idea, which needs validation:
1. Expose detailed shrinker information via debugfs
We've already exposed details of the slab through /sys/kernel/debug/slab, so extending this to include shrinker details shouldn't be too challenging. For example, for the dentry shrinker, we could expose /sys/kernel/debug/shrinker/super_cache_scan/{shrinker_id, kmem_cache, ...}.
2. Shrink specific slabs with a specific count
This could be implemented by extending memory.reclaim with parameters like "shrinker_id=" and "scan_count=". Currently, memory.reclaim is byte-based, which isn't ideal for shrinkers due to the deferred freeing of slabs. Using scan_count to specify the number of slabs to reclaim could be more effective.
These are preliminary ideas, and I welcome any feedback.
Additionally, since this patch offers a straightforward solution to address the issue in container environments, would it be feasible to apply this patch initially?
Signature · 2 lines
Regards Yafang
Quoted 3 lines · Yafang Shao
Additionally, since this patch offers a straightforward solution to address the issue in container environments, would it be feasible to apply this patch initially?
There are lot sof other problems with this patch, but since I disagree with the idea of this patch, I didn't enumerate them.
Quoted 8 lines · Matthew Wilcox
Quoted 3 lines · Yafang Shao
Additionally, since this patch offers a straightforward solution to address the issue in container environments, would it be feasible to apply this patch initially?
There are lot sof other problems with this patch, but since I disagree with the idea of this patch, I didn't enumerate them.
Could you please provide an enumeration of the issues you've identified? One issue is that the reclaimed bytes may not be optimal for slab shrinking, particularly because many dentries are freed deferredly.
Signature · 2 lines
Regards Yafang
Quoted 4 lines · Yafang Shao
In our container environment, we've observed that certain containers may accumulate more than 40GB of slabs, predominantly negative dentries. These negative dentries remain unreclaimed unless there is memory pressure. Even after the containers exit, these negative dentries persist.
Have you considered using memory.high limit to trigger that memory reclaim? That is surely not focused on the neg. dentry cache but it should keep the overal memory consumption under control.
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 4 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.
Matthew has already pointed out that this has been proposed several times already and rejected. Dedicated slab shrinking interface is especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Michal Hocko
Quoted 4 lines · Yafang Shao
In our container environment, we've observed that certain containers may accumulate more than 40GB of slabs, predominantly negative dentries. These negative dentries remain unreclaimed unless there is memory pressure. Even after the containers exit, these negative dentries persist.
Have you considered using memory.high limit to trigger that memory reclaim? That is surely not focused on the neg. dentry cache but it should keep the overal memory consumption under control.
Upon reaching the high watermark, both pagecache and slabs are reclaimed. However, it tends to prioritize reclaiming more pagecache pages over slab pages, leading to persistent memory consumption by slabs. Setting memory.high too low may negatively impact the working set.
Quoted 10 lines
Quoted 7 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.
The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
--
Regards
Yafang
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Quoted 6 lines
Quoted 2 lines
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Quoted 15 lines
Quoted 7 lines
Quoted 4 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.
Quoted 5 lines
The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through force_empty can also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues in force_empty and rmdir are not identical, as they involve different locks.
Quoted 22 lines
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.Quoted 5 lines
The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.
Signature · 2 lines
Regards Yafang
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.
There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
Quoted 62 lines · Yosry Ahmed
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.
Quoted 5 lines
There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
What if we introduce a new file, like memory.shrink? This could serve as a foundation for potential future extensions, allowing us to shrink specific slabs with specific counts.
Signature · 2 lines
Regards Yafang
Quoted 70 lines · Yafang Shao
Quoted 62 lines · Yosry Ahmed
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.Quoted 5 lines
There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
Quoted 3 lines
What if we introduce a new file, like memory.shrink? This could serve as a foundation for potential future extensions, allowing us to shrink specific slabs with specific counts.
Shrinking specific slabs is something that shouldn't be exposed as an interface, as this is a kernel implementation detail. Also, memory.reclaim and memory.shrink would still have overlapping functionalities.
Quoted 72 lines · Yosry Ahmed
Quoted 70 lines · Yafang Shao
Quoted 67 lines · Yosry Ahmed
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
Quoted 6 lines
Quoted 3 lines
What if we introduce a new file, like memory.shrink? This could serve as a foundation for potential future extensions, allowing us to shrink specific slabs with specific counts.
Shrinking specific slabs is something that shouldn't be exposed as an interface, as this is a kernel implementation detail. Also,
If that's the case, why was slabs info initially exposed through /proc/slabinfo? Isn't that level of detail considered a kernel implementation detail? Currently, users can identify which slab is consuming the most memory but lack the ability to take action based on that information. This suggests a flaw in the kernel implementation.
Quoted 2 lines
memory.reclaim and memory.shrink would still have overlapping functionalities.
Signature · 2 lines
Regards Yafang
Quoted 78 lines · Yafang Shao
Quoted 76 lines · Yosry Ahmed
Quoted 73 lines · Yafang Shao
Quoted 62 lines · Yosry Ahmed
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.Quoted 5 lines
There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
What if we introduce a new file, like memory.shrink? This could serve as a foundation for potential future extensions, allowing us to shrink specific slabs with specific counts.
Quoted 8 lines
Quoted 2 lines
Shrinking specific slabs is something that shouldn't be exposed as an interface, as this is a kernel implementation detail. Also,
If that's the case, why was slabs info initially exposed through
/proc/slabinfo? Isn't that level of detail considered a kernel implementation detail? Currently, users can identify which slab is consuming the most memory but lack the ability to take action based on that information. This suggests a flaw in the kernel implementation.
BTW, we even expose more detailed kernel implementation details through /sys/kernel/slab.
That is really confusing...
Signature · 2 lines
Regards Yafang
Quoted 89 lines · Yafang Shao
Quoted 78 lines · Yafang Shao
Quoted 76 lines · Yosry Ahmed
Quoted 73 lines · Yafang Shao
Quoted 62 lines · Yosry Ahmed
Quoted 51 lines · Yafang Shao
Quoted 14 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Quoted 6 lines
Quoted 3 lines
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 8 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
As observed, numerous memcg interfaces have been deprecated in recent years.Quoted 5 lines
There has been recent work to add a swapiness= argument to memory.reclaim to balance between anon and file pages. Adding a type= argument together with that is a recipe for eternal confusion. *If* we want to support this, we need to have a way to combine these two into something more user-friendly.
What if we introduce a new file, like memory.shrink? This could serve as a foundation for potential future extensions, allowing us to shrink specific slabs with specific counts.
Quoted 8 lines
Quoted 2 lines
Shrinking specific slabs is something that shouldn't be exposed as an interface, as this is a kernel implementation detail. Also,
If that's the case, why was slabs info initially exposed through
/proc/slabinfo? Isn't that level of detail considered a kernel implementation detail? Currently, users can identify which slab is consuming the most memory but lack the ability to take action based on that information. This suggests a flaw in the kernel implementation.
Quoted 3 lines
BTW, we even expose more detailed kernel implementation details through
/sys/kernel/slab.
That is really confusing...
There is even a /sys/kernel/slab/dentry/shrink .... oh please...
Signature · 2 lines
Regards Yafang
Quoted 1 line · Yafang Shao
On Tue, Feb 27, 2024 at 8:09 PM Yafang Shao <[email protected]> wrote:
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If that's the case, why was slabs info initially exposed through
/proc/slabinfo?
because that helps to better understand the memory consumption by slab consumers.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Isn't that level of detail considered a kernel implementation detail? Currently, users can identify which slab is consuming the most memory but lack the ability to take action based on that information. This suggests a flaw in the kernel implementation.
I disgree!
Quoted 6 lines
Quoted 3 lines
BTW, we even expose more detailed kernel implementation details through
/sys/kernel/slab.
That is really confusing...There is even a
/sys/kernel/slab/dentry/shrink.... oh please...
We also have /proc/sys/vm/drop_caches and we have learned those are really terrible interfaces and we have good reasons to not replicate those into memcg interfaces. Using bad interfaces as an example is not the way argue for new ones.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Tue, Feb 27, 2024 at 8:09 PM Yafang Shao <[email protected]> wrote:
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If that's the case, why was slabs info initially exposed through
/proc/slabinfo?because that helps to better understand the memory consumption by slab consumers.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Isn't that level of detail considered a kernel implementation detail? Currently, users can identify which slab is consuming the most memory but lack the ability to take action based on that information. This suggests a flaw in the kernel implementation.
I disgree!
Quoted 4 lines
Quoted 3 lines
BTW, we even expose more detailed kernel implementation details through
/sys/kernel/slab.
That is really confusing...
Quoted 7 lines
Quoted 2 lines
There is even a
/sys/kernel/slab/dentry/shrink.... oh please...We also have
/proc/sys/vm/drop_cachesand we have learned those are really terrible interfaces and we have good reasons to not replicate those into memcg interfaces. Using bad interfaces as an example is not the way argue for new ones.
And then we introduce a similar memory.reclaim......
Signature · 2 lines
Regards Yafang
Quoted 19 lines · Yafang Shao
Quoted 18 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Quoted 8 lines
Quoted 2 lines
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.
Please be more specific about those issues.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 7 lines
Quoted 4 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 7 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
Right, I do not believe we would deter anybody from doing that. This is just not an easy problem to tackle. So either you find solid arguments that previous conclusions do not hold anymore or you need to look into options which haven't been discussed so far. I do realize that chasing previous discussions in email archives is not fun but maybe a good (re)start would be documenting those problems somewhere under Documentation/.
Quoted 1 line
As observed, numerous memcg interfaces have been deprecated in recent years.
yes, for very good reasons
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 24 lines · Michal Hocko
Quoted 22 lines · Yafang Shao
Quoted 20 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Mon, Feb 26, 2024 at 10:05 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
To manage disk
storage efficiently, we employ an agent that identifies container images eligible for destruction once all instances of that image exit.However, during destruction, dealing with directories containing numerous negative dentries can significantly impact performance.
Quoted 6 lines
Quoted 2 lines
Performance of what. I have to say I am kind of lost here. We are talking about memory or a disk storage?
Removing an empty directory with numerous dentries can significantly prolong the process of freeing associated dentries, leading to high system CPU usage that adversely affects overall system performance.
Is there anything that prevents you from reclaiming the memcg you are about to remove? We do have interfaces for that.
Quoted 7 lines
Quoted 5 lines
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Please be more specific about those issues.
Both of these issues stem from lock contention:
- rmdir
When executing rmdir, the lock of the inode of the empty directory is held. If this directory contains numerous negative dentries, this lock is held for an extended duration. Consequently, if other processes attempt to acquire this lock, they are blocked.
A simple reproducer involves:
1. Generating numerous negative dentries in an empty directory.
2. Running rmdir ~/test and ls ~/ concurrently.
This setup demonstrates that ls takes a significant amount of time to complete due to lock contention.
- force_emptyForce_empty holds the lock of super_block->dentry_list. However, I haven't yet had the opportunity to produce a specific example to illustrate this issue.
Quoted 40 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 8 lines
Quoted 5 lines
To mitigate this
issue, we aim to proactively reclaim these dentries using a user agent. Extending the memory.reclaim functionality to specifically target slabs aligns with our requirements.Matthew has already pointed out that this has been proposed several times already and rejected.
With that being said, we haven't come up with any superior solutions compared to the proposals mentioned.
Quoted 3 lines
Dedicated slab shrinking interface is
especially tricky because you would need a way to tell which shrinkers to invoke and that would be very kernel version specific.The persistence of this issue over several years without any discernible progress suggests that we might be heading in the wrong direction. Perhaps we could consider providing a kernel interface to users, allowing them to tailor the reclamation process based on their workload requirements.
Quoted 7 lines
Quoted 4 lines
There are clear problems identified with type specific reclaim and those might easily strike back with future changes. Once we put an interface in place we won't be able remove it and that could lead to problems with future changes in the memory reclaim.
That shouldn't deter us from actively seeking a resolution to an issue that has persisted for tens of years.
Right, I do not believe we would deter anybody from doing that. This is just not an easy problem to tackle. So either you find solid arguments that previous conclusions do not hold anymore or you need to look into options which haven't been discussed so far. I do realize that chasing previous discussions in email archives is not fun but maybe a good (re)start would be documenting those problems somewhere under
Documentation/.Quoted 1 line
As observed, numerous memcg interfaces have been deprecated in recent years.
yes, for very good reasons
Signature · 2 lines
Regards Yafang
Quoted 1 line · Yafang Shao
On Tue, Feb 27, 2024 at 5:45 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Please be more specific about those issues.
Both of these issues stem from lock contention:
- rmdir
When executing rmdir, the lock of the inode of the empty directory is held. If this directory contains numerous negative dentries, this lock is held for an extended duration. Consequently, if other processes attempt to acquire this lock, they are blocked.
A simple reproducer involves:
Quoted 2 lines
1. Generating numerous negative dentries in an empty directory.
2. Runningrmdir ~/testandls ~/concurrently.
I fail to see how is this relevant to memcg reclaim
Quoted 7 lines
This setup demonstrates that ls takes a significant amount of time to complete due to lock contention.
-
force_emptyForce_emptyholds the lock ofsuper_block->dentry_list. However, I haven't yet had the opportunity to produce a specific example to illustrate this issue.
OK, get back to us once you can identify an actual problem. We might be talking about different things here though. My question is directed at existing memcg interfaces to reclaim the memory. That would be legacy (and effectively deprecated) force_empty and memory.reclaim that we have.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Tue, Feb 27, 2024 at 5:45 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Please be more specific about those issues.
Both of these issues stem from lock contention:
- rmdir
When executing rmdir, the lock of the inode of the empty directory is held. If this directory contains numerous negative dentries, this lock is held for an extended duration. Consequently, if other processes attempt to acquire this lock, they are blocked.
A simple reproducer involves:
Quoted 4 lines
Quoted 2 lines
1. Generating numerous negative dentries in an empty directory.
2. Runningrmdir ~/testandls ~/concurrently.I fail to see how is this relevant to memcg reclaim
It appears there might still be some misunderstanding regarding the issue at hand.
The numerous negative dentries are generated within a memcg. I simplified the explanation by omitting the memcg context.
Quoted 10 lines
Quoted 8 lines
This setup demonstrates that ls takes a significant amount of time to complete due to lock contention.
-
force_emptyForce_emptyholds the lock ofsuper_block->dentry_list. However, I haven't yet had the opportunity to produce a specific example to illustrate this issue.OK, get back to us once you can identify an actual problem. We might be
Pls. take a look at the force_empty->prune_dcache_sb->shrink_dentry_list.
Quoted 4 lines
talking about different things here though. My question is directed at existing memcg interfaces to reclaim the memory. That would be legacy (and effectively deprecated)
force_emptyand memory.reclaim that we have.
Signature · 2 lines
Regards Yafang
Quoted 39 lines · Yafang Shao
Quoted 21 lines · Michal Hocko
Quoted 1 line · Yafang Shao
On Tue, Feb 27, 2024 at 5:45 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 17 lines
Quoted 7 lines
Quoted 5 lines
Reclaiming numerous dentries through
force_emptycan also lead to potential issues, which is why we attempt to shrink the slab gradually to mitigate them. However, it's important to note that the underlying causes of the issues inforce_emptyand rmdir are not identical, as they involve different locks.Please be more specific about those issues.
Both of these issues stem from lock contention:
- rmdir
When executing rmdir, the lock of the inode of the empty directory is held. If this directory contains numerous negative dentries, this lock is held for an extended duration. Consequently, if other processes attempt to acquire this lock, they are blocked.
A simple reproducer involves:Quoted 4 lines
Quoted 2 lines
1. Generating numerous negative dentries in an empty directory.
2. Runningrmdir ~/testandls ~/concurrently.I fail to see how is this relevant to memcg reclaim
It appears there might still be some misunderstanding regarding the issue at hand.
The numerous negative dentries are generated within a memcg. I simplified the explanation by omitting the memcg context.
Quoted 4 lines
Quoted 4 lines
This setup demonstrates that ls takes a significant amount of time to complete due to lock contention.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
-
force_emptyForce_emptyholds the lock ofsuper_block->dentry_list. However, I haven't yet had the opportunity to produce a specific example to illustrate this issue.OK, get back to us once you can identify an actual problem. We might be
Pls. take a look at the
force_empty->prune_dcache_sb->shrink_dentry_list.
Please note that the spinlock is located within list_lru_walk_one. Additionally, the issue persists even with the newly introduced memory.reclaim, as it functions similarly to force_empty.
Quoted 11 lines
Quoted 5 lines
talking about different things here though. My question is directed at existing memcg interfaces to reclaim the memory. That would be legacy (and effectively deprecated)
force_emptyand memory.reclaim that we have.--
Regards
Yafang
Signature · 2 lines
Regards Yafang