Skip to contentSkip to conversation

LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

DISCUSSION Uladzislau Rezki 4 participants 9 messages Feb 28, 2023 – May 11, 2023 Memory management

lore
Share & tools
lore
#0 lore raw

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

Requirements:
* It should be a per-cpu approach;
* Search of freed ptrs should not interfere with other freeing(as much as we can);
* - offload allocated areas(buzy ones) per-cpu;
* Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
* Lazily-freed areas either drained per-cpu individually or by one CPU for all;
* Prefetch a fixed size in front and allocate per-cpu

Goals:
* Implement a per-cpu way of allocation to eliminate a contention.

Thanks!

--
Uladzislau Rezki

#1 lore raw
Quoted 22 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

Requirements:
* It should be a per-cpu approach;
* Search of freed ptrs should not interfere with other freeing(as much as we can);
* - offload allocated areas(buzy ones) per-cpu;
* Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
* Lazily-freed areas either drained per-cpu individually or by one CPU for all;
* Prefetch a fixed size in front and allocate per-cpu

Quoted 8 lines

Goals:
* Implement a per-cpu way of allocation to eliminate a contention.

Thanks!

--
Uladzislau Rezki

This is a really interesting topic and I'd really like this to go ahead as there's certainly been a lot of discussion regarding vmalloc locking of late. I for one would certainly attend it.

#2 lore raw
In reply to Uladzislau Rezki
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

Quoted 19 lines

Requirements:
* It should be a per-cpu approach;
* Search of freed ptrs should not interfere with other freeing(as much as we can);
* - offload allocated areas(buzy ones) per-cpu;
* Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
* Lazily-freed areas either drained per-cpu individually or by one CPU for all;
* Prefetch a fixed size in front and allocate per-cpu

Goals:
* Implement a per-cpu way of allocation to eliminate a contention.

Thanks!

--
Uladzislau Rezki
_______________________________________________
Lsf-pc mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/lsf-pc

Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 9 lines
Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

--
Uladzislau Rezki

#4 lore raw
Quoted 20 lines · Uladzislau Rezki
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 9 lines
Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

Quoted 6 lines

The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

Thanks for the clarification. The usual LSFMM format is strongly discussion focused. Long presentations are usually discouraged and they should only introduce people to the underlying problem to kick of a discussion.

That being said, IMO it would be helpful to have some material on the mailing list before any discussion could be productive.

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
Quoted 28 lines · Michal Hocko
Quoted 20 lines · Uladzislau Rezki
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 9 lines
Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

Quoted 6 lines

The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

Quoted 5 lines

Thanks for the clarification. The usual LSFMM format is strongly discussion focused. Long presentations are usually discouraged and they should only introduce people to the underlying problem to kick of a discussion.

I have not posted yet any RFC and have not kicked it yet. Though people are aware the problem.

Quoted 4 lines

That being said, IMO it would be helpful to have some material on the mailing list before any discussion could be productive.

This is what i have so far:

wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf

I can, of course, move it forward over lkml only. If you are fully booked or there other reason then please just withdraw my proposal from your conference.

--
Uladzislau Rezki

#6 lore raw
Quoted 41 lines · Uladzislau Rezki
Quoted 28 lines · Michal Hocko
Quoted 20 lines · Uladzislau Rezki
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 9 lines
Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

Quoted 6 lines

The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

Quoted 5 lines

Thanks for the clarification. The usual LSFMM format is strongly discussion focused. Long presentations are usually discouraged and they should only introduce people to the underlying problem to kick of a discussion.

I have not posted yet any RFC and have not kicked it yet. Though people are aware the problem.

Quoted 4 lines

That being said, IMO it would be helpful to have some material on the mailing list before any discussion could be productive.

Quoted 7 lines

This is what i have so far:

wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf

I can, of course, move it forward over lkml only. If you are fully booked or there other reason then please just withdraw my proposal from your conference.

For what it's worth I'm definitely interested in attending this session if it goes ahead. I am sure Baoquan, if he's attending LSF/MM, would be too! (cc'd).

#7 lore raw
Quoted 44 lines · Lorenzo Stoakes
Quoted 43 lines · Uladzislau Rezki
Quoted 28 lines · Michal Hocko
Quoted 20 lines · Uladzislau Rezki
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Uladzislau Rezki

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

Quoted 9 lines
Quoted 4 lines

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

It's been some time since you brough this up. Has there been any progress on the topic? Do you still find it important to discuss it at LSFMM?

Quoted 6 lines

The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

Quoted 5 lines

Thanks for the clarification. The usual LSFMM format is strongly discussion focused. Long presentations are usually discouraged and they should only introduce people to the underlying problem to kick of a discussion.

I have not posted yet any RFC and have not kicked it yet. Though people are aware the problem.

Quoted 4 lines

That being said, IMO it would be helpful to have some material on the mailing list before any discussion could be productive.

This is what i have so far:

Quoted 8 lines
Quoted 5 lines

wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf

I can, of course, move it forward over lkml only. If you are fully booked or there other reason then please just withdraw my proposal from your conference.

For what it's worth I'm definitely interested in attending this session if it goes ahead. I am sure Baoquan, if he's attending LSF/MM, would be too! (cc'd).

Thanks for CC.

Yes, I am very interested in this topic, have read it from the beginning. Unfortunately, I can't attend LSF/MM this year because of some reasons. I look forward to learning the decision or conclusion of the sessin, and reviewing Uladzislau's RFC or formal patchset later. Wish the session a great success, and you guys a pleasant meeting and discussion.

Thanks
Baoquan

#8 lore raw
In reply to Uladzislau Rezki
Quoted 23 lines

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
Currently the vmap code is not scaled to number of CPU cores in a system because a global vmap space is protected by a single spinlock. Such approach has a clear bottleneck if many CPUs simultaneously access to one resource.

In this talk i would like to describe a drawback, show some data related to contentions and places where those occur in a code. Apart of that i would like to share ideas how to eliminate it providing a few approaches and compare them.

Requirements:
* It should be a per-cpu approach;
* Search of freed ptrs should not interfere with other freeing(as much as we can);
* - offload allocated areas(buzy ones) per-cpu;
* Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
* Lazily-freed areas either drained per-cpu individually or by one CPU for all;
* Prefetch a fixed size in front and allocate per-cpu

Quoted 3 lines

Goals:
* Implement a per-cpu way of allocation to eliminate a contention.

Thank you for giving a time to talk about this topic on the LSFMMBPF. The slides can
be accessed here:

wget ftp://vps418301.ovh.net/incoming/Mitigate_a_vmalloc_lock_contention_in_SMP_env_v1.pdf

There were some issues with a camera, but i was told in the end you could see me.2

Thank you again!

Signature · 1 line
Uladzislau Rezki