MM global locks as core counts quadruple
Hi all,
As core counts are rapidly expanding over the next four years, Namhyung and I were looking at global locks that we're already seeing high contention on even today.
Some of these are not MM specific:
- cgroup_mutex
- cgroup_threadgroup_rwsem
- tasklist_lock
- kernfs_mutex (although should now be substantially better with the kernfs_locks array)
Others *are* MM specific:
- list_lrus_mutex
- pcpu_drain_mutex
- shrinker_mutex (formerly shrinker_rwsem)
- vmap_purge_lock
- slab_mutex
This is only looking at fleet data for global static locks, not locks like zone->lock that get dynamically allocated.
(mmap_lock was substantially improved by per-vma locking, although does show up for very large vmas.)
Couple questions:
(1) How are people quantifying these pain points, if at all, in synthetic testing? Any workloads or benchmarks that are really good at doing this in the lab beyond the traditional will-it-scale? (The above is from production data.)
(2) Is anybody working on any of the above global locks? Trying to surface gaps for locks that will likely become even more painful in the coming years.
Thanks!
Quoted 1 line · David Rientjes
-
slab_mutex
This is weird. You must have very strange workloads running ;-)
It protects the list of slabs. So it's taken for
- Slab creation / destruction
- CPU online/offline
- Memory hotplug
- Reading /proc/slabinfo
All of these should be rare events, but don't seem to be for you?
Quoted 9 lines · David Rientjes
Hi all,
As core counts are rapidly expanding over the next four years, Namhyung and I were looking at global locks that we're already seeing high contention on even today.
Some of these are not MM specific:
-cgroup_mutex
We previously discussed replacing cgroup_mutex with RCU in some critical paths [0], but I haven't had the time to implement it yet.
[0]. https://lore.kernel.org/bpf/CALOAHbAgWoFGSc=uF5gFWXmsALECUaGGScQuXpRcwjgzv+TPGQ@mail.gmail.com/
Quoted 12 lines
-
cgroup_threadgroup_rwsem
-tasklist_lock
-kernfs_mutex(although should now be substantially better with thekernfs_locksarray)Others *are* MM specific:
-list_lrus_mutex
-pcpu_drain_mutex
-shrinker_mutex(formerlyshrinker_rwsem)
-vmap_purge_lock
-slab_mutex
Quoted 2 lines
This is only looking at fleet data for global static locks, not locks like
zone->lockthat get dynamically allocated.
We have encountered latency spikes caused by the zone->lock. Do we have any plans to eliminate this lock, such as implementing a lockless buddy list? I believe this could be a viable solution.
Quoted 17 lines
(
mmap_lockwas substantially improved by per-vma locking, although does show up for very large vmas.)Couple questions:
(1) How are people quantifying these pain points, if at all, in synthetic testing? Any workloads or benchmarks that are really good at doing this in the lab beyond the traditional will-it-scale? (The above is from production data.)
(2) Is anybody working on any of the above global locks? Trying to surface gaps for locks that will likely become even more painful in the coming years.
Thanks!
Signature · 2 lines
Regards Yafang
Quoted 6 lines · Yafang Shao
Quoted 2 lines
This is only looking at fleet data for global static locks, not locks like
zone->lockthat get dynamically allocated.We have encountered latency spikes caused by the
zone->lock. Do we have any plans to eliminate this lock, such as implementing a lockless buddy list? I believe this could be a viable solution.
Lockless how? I see three operations being performed on the buddy list:
- Add to end (freeing)
- Remove from end (allocation)
- Remove from middle (buddy was freed, needs to be coalesced)
I don't see how we can handle this locklessly.
I think a more productive solution to contention on the LRU lock is to increase the number of zones. I don't think it's helpful to have a 1TB zone of memory. Maybe we should limit each zone to 16GB or so. That means we'd need to increase the number of zones we support, but I think that's doable.
Quoted 5 lines · Matthew Wilcox
I think a more productive solution to contention on the LRU lock is to increase the number of zones. I don't think it's helpful to have a 1TB zone of memory. Maybe we should limit each zone to 16GB or so. That means we'd need to increase the number of zones we support, but I think that's doable.
What do you mean by zones? The usual ZONE_{DMA|DMA32|NORMAL|HIGHMEM}?
But that's historically existed to deal with physical addressing limitations (e.g. DMA for devices that can't deal with addresses larger than 16MiB, or HIGHMEM on 32-bits to access physical memory beyond what could be directly mapped by the kernel).
Maybe you mean turning ZONE_NORMAL into an array with each entry pointing to a smaller ZONE_NORMAL region of, let's say, 64GiB or smthng. Or it could be divided by the number of CPUs within the NUMA node and each CPU will be given one ZONE_NORMAL segment with a fallback list to other CPUs segments in case it runs out of memory. Does that make sense?
Cheers
Karim
PhD Student
Edinburgh University
Quoted 5 lines · Karim Manaouil
Maybe you mean turning
ZONE_NORMALinto an array with each entry pointing to a smallerZONE_NORMALregion of, let's say, 64GiB or smthng. Or it could be divided by the number of CPUs within the NUMA node and each CPU will be given oneZONE_NORMALsegment with a fallback list to other CPUs segments in case it runs out of memory. Does that make sense?
More zones means longer zonelists for the page allocator to walk during memory allocation. VM statistics are also per zone so that would decrease counter cacheline contention. Would be good for scaling VM things in general.
Hello,
Quoted 6 lines · David Rientjes
As core counts are rapidly expanding over the next four years, Namhyung and I were looking at global locks that we're already seeing high contention on even today.
Some of these are not MM specific:
-cgroup_mutex
Our machines are getting bigger but we aren't creating and destroying cgroups frequently enough for this to matter. But yeah, I can see how this can be a problem.
Quoted 1 line
-
cgroup_threadgroup_rwsem
This one shouldn't matter at all in setups where new cgroups are populated with CLONE_INTO_CGROUP and not migrated further. The lock isn't grabbed in such usage pattern, which should be the vast majority already, I think. Are you guys migrating tasks a lot or not using CLONE_INTO_CGROUP?
Thanks.
Signature · 1 line
tejun
Hello,
Quoted 15 lines · Tejun Heo
Hello,
Quoted 6 lines · David Rientjes
As core counts are rapidly expanding over the next four years, Namhyung and I were looking at global locks that we're already seeing high contention on even today.
Some of these are not MM specific:
-cgroup_mutexOur machines are getting bigger but we aren't creating and destroying cgroups frequently enough for this to matter. But yeah, I can see how this can be a problem.
Quoted 6 lines
Quoted 1 line
-
cgroup_threadgroup_rwsemThis one shouldn't matter at all in setups where new cgroups are populated with
CLONE_INTO_CGROUPand not migrated further. The lock isn't grabbed in such usage pattern, which should be the vast majority already, I think. Are you guys migrating tasks a lot or not usingCLONE_INTO_CGROUP?
I'm afraid there are still some use cases in Google that migrate processes and/or threads between cgroups. :(
Thanks,
Namhyung
Hello,
Quoted 9 lines · Namhyung Kim
Quoted 6 lines
Quoted 1 line
-
cgroup_threadgroup_rwsemThis one shouldn't matter at all in setups where new cgroups are populated with
CLONE_INTO_CGROUPand not migrated further. The lock isn't grabbed in such usage pattern, which should be the vast majority already, I think. Are you guys migrating tasks a lot or not usingCLONE_INTO_CGROUP?I'm afraid there are still some use cases in Google that migrate processes and/or threads between cgroups. :(
I see. I wonder whether we can turn this into a cgroup lock. It's not straightforward tho. It's protecting migration against forking and exiting and the only way to turn it into per-cgroup lock would be tying it to the source cgroup as that's the only thing identifiable from the fork and exit paths. The problem is that a single atomic migration operation can pull tasks from multiple cgroups into one destination cgroup, even on cgroup2 due to the threaded cgroups. This would be pretty rare on cgroup2 but still need to be handled which means grabbing multiple locks from the migration path. Not the end of the world but a bit nasty.
But, as long as it's well encapsulated and out of line, I don't see problems with such approach.
As for cgroup_mutex, it's more complicated as the usage is more spread, but yeah, the only solution there too would be going for finer grained locking whether that's hierarchical or hashed.
Thanks.
Signature · 1 line
tejun
Quoted 23 lines · Tejun Heo
Hello,
Quoted 9 lines · Namhyung Kim
Quoted 6 lines
Quoted 1 line
-
cgroup_threadgroup_rwsemThis one shouldn't matter at all in setups where new cgroups are populated with
CLONE_INTO_CGROUPand not migrated further. The lock isn't grabbed in such usage pattern, which should be the vast majority already, I think. Are you guys migrating tasks a lot or not usingCLONE_INTO_CGROUP?I'm afraid there are still some use cases in Google that migrate processes and/or threads between cgroups. :(
I see. I wonder whether we can turn this into a cgroup lock. It's not straightforward tho. It's protecting migration against forking and exiting and the only way to turn it into per-cgroup lock would be tying it to the source cgroup as that's the only thing identifiable from the fork and exit paths. The problem is that a single atomic migration operation can pull tasks from multiple cgroups into one destination cgroup, even on cgroup2 due to the threaded cgroups. This would be pretty rare on cgroup2 but still need to be handled which means grabbing multiple locks from the migration path. Not the end of the world but a bit nasty.
Quoted 7 lines
But, as long as it's well encapsulated and out of line, I don't see problems with such approach.
As for
cgroup_mutex, it's more complicated as the usage is more spread, but yeah, the only solution there too would be going for finer grained locking whether that's hierarchical or hashed.
Thanks all for the great discussion in the thread so far!
Beyond the discussion of cgroup mutexes above, we also discussed increasing the number of zones within a NUMA node. I'm thinking that this would actually be an implementation detail, i.e. we wouldn't need to change any user visible interfaces like /proc/zoneinfo. IOW, we could have 64 16GB ZONE_NORMALs spanning 1TB of memory, and we could sum up the memory resident across all of those when describing the memory to userspace.
Anybody else working on any of the following or have thoughts/ideas for how they could be improved as core counts increase?
- list_lrus_mutex
- pcpu_drain_mutex
- shrinker_mutex (formerly shrinker_rwsem)
- vmap_purge_lock
Also, any favorite benchmarks that people use with high core counts to measure the improvement when generic MM locks become more sharded? I can imagine running will-it-scale on platforms with >= 256 cores per socket but if there are specific stress tests that can help quantify the impact, that would be great to know about.