Skip to contentSkip to conversation

tmpfs: Improve tmpfs scalability

PATCH Superseded tim 3 participants 4 messages May 18, 2010 – May 21, 2010

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

tim

author
#0 lore raw

cover tmpfs: Improve tmpfs scalability

We created a token jar library implementing
per cpu cache of tokens to avoid lock contentions whenever we retrieve or return a token to a token jar. Using this library with tmpfs, we find Aim7 fserver throughput improved 270% on a 4 socket, 32 cores NHM-EX system.

In current implementation of tmpfs, whenever we
get a new page, stat_lock in shmem_sb_info needs to be acquired. This causes a lot of lock contentions when multiple threads are using tmpfs simultaneously, which makes system with large number of cpus scale poorly.
Almost 75% of cpu time was spent contending on stat_lock when we ran Aim7 fserver load with 128 threads on a 4 socket, 32 cores NHM-EX system.

The first patch in the series implements the quick token jar. The second patch update the shmem code of tmpfs to use this library to improve tmpfs performance.

Regards,
Tim Chen

Reviewed-by on the v1 series
#1 lore raw
Quoted 16 lines · tim

We created a token jar library implementing
per cpu cache of tokens to avoid lock contentions whenever we retrieve or return a token to a token jar. Using this library with tmpfs, we find Aim7 fserver throughput improved 270% on a 4 socket, 32 cores NHM-EX system.

In current implementation of tmpfs, whenever we
get a new page, stat_lock in shmem_sb_info needs to be acquired. This causes a lot of lock contentions when multiple threads are using tmpfs simultaneously, which makes system with large number of cpus scale poorly.
Almost 75% of cpu time was spent contending on stat_lock when we ran Aim7 fserver load with 128 threads on a 4 socket, 32 cores NHM-EX system.

Quoted 3 lines

The first patch in the series implements the quick token jar. The second patch update the shmem code of tmpfs to use this library to improve tmpfs performance.

I reviewed both patches and they look good to me. Especially the token jar library should be useful in other places too.

-Andi

#2 lore raw
In reply to tim
Quoted 16 lines · tim

We created a token jar library implementing
per cpu cache of tokens to avoid lock contentions whenever we retrieve or return a token to a token jar. Using this library with tmpfs, we find Aim7 fserver throughput improved 270% on a 4 socket, 32 cores NHM-EX system.

In current implementation of tmpfs, whenever we
get a new page, stat_lock in shmem_sb_info needs to be acquired. This causes a lot of lock contentions when multiple threads are using tmpfs simultaneously, which makes system with large number of cpus scale poorly.
Almost 75% of cpu time was spent contending on stat_lock when we ran Aim7 fserver load with 128 threads on a 4 socket, 32 cores NHM-EX system.

Quoted 3 lines

The first patch in the series implements the quick token jar. The second patch update the shmem code of tmpfs to use this library to improve tmpfs performance.

Interesting, thank you - I'll take a look, but not this week.

I do hope you're using Aim7 just as an example: you know that mounting tmpfs with nr_blocks=0,nr_inodes=0 skips those shmem_sb_info updates altogether? Mounting in such a way should be fine for getting better numbers out of Aim7; but yes, there are reallife uses for tmpfs which are safer with the nr_blocks,nr_inodes limits.

Hugh

author
#3 lore raw
Quoted 9 lines · Hugh Dickins

Interesting, thank you - I'll take a look, but not this week.

I do hope you're using Aim7 just as an example: you know that mounting tmpfs with nr_blocks=0,nr_inodes=0 skips those shmem_sb_info updates altogether? Mounting in such a way should be fine for getting better numbers out of Aim7; but yes, there are reallife uses for tmpfs which are safer with the nr_blocks,nr_inodes limits.

Yes, Aim7 was provided as an example to illustrate the locking bottleneck when blocks limit are imposed.

Tim