Skip to contentSkip to conversation

devpts mounts too slowly

DISCUSSION Alexey Vlasov 5 participants 16 messages Jun 3, 2009 – Jul 7, 2009 VFS

lore
Share & tools
lore
#0 lore raw

Hi.

In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.

# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts
real    0m0.295s
user    0m0.000s
sys     0m0.288s
# time mount -i -n --bind /var/bin /home/staff/.server-1/bin
real    0m0.004s
user    0m0.000s
sys     0m0.004s
Signature · 1 line
BRGDS. Alexey Vlasov.
#1 lore raw
Quoted 7 lines · Alexey Vlasov

Hi.

In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.

Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts
real    0m0.295s
user    0m0.000s
sys     0m0.288s

A profile would be handy here. There is no sensible reason for this to that 0.3 seconds. This might be RCU-epoch related?

-hpa

#2 lore raw
Quoted 8 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 7 lines · Alexey Vlasov

Hi.

In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.

Quoted 9 lines
Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts
real    0m0.295s
user    0m0.000s
sys     0m0.288s

A profile would be handy here. There is no sensible reason for this to that 0.3 seconds. This might be RCU-epoch related?

It seems OK here.

akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts
mount -n -i -t devpts none /tmp/pts  0.00s user 0.00s system 0% cpu 0.002 total
#3 lore raw
Quoted 17 lines · Andrew Morton
Quoted 14 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 12 lines · Alexey Vlasov

Hi.

In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.

# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts
real    0m0.295s
user    0m0.000s
sys     0m0.288s
Quoted 8 lines
Quoted 3 lines

A profile would be handy here.  There is no sensible reason for this to that 0.3 seconds.  This might be RCU-epoch related?

It seems OK here.

akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts
mount -n -i -t devpts none /tmp/pts  0.00s user 0.00s system 0% cpu 0.002 total

I read the Original Post as that test being performed with 5k /dev/pts already mounted.

#4 lore raw
Quoted 22 lines · On Tue, 9 Jun 2009 16:30:31 -0700 Ray Lee <[email protected]> wrote:
Quoted 21 lines · Andrew Morton
Quoted 17 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 7 lines · Alexey Vlasov

Hi.

In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.

Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts
real __ __0m0.295s
user __ __0m0.000s
sys __ __ 0m0.288s

A profile would be handy here. __There is no sensible reason for this to that 0.3 seconds. __This might be RCU-epoch related?

Quoted 7 lines
Quoted 4 lines

It seems OK here.

akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts
mount -n -i -t devpts none /tmp/pts __0.00s user 0.00s system 0% cpu 0.002 total

I read the Original Post as that test being performed with 5k /dev/pts already mounted.

hm, OK.

I've now mounted 15000 devpts's and still no slowdown is evident.

#5 lore raw
Quoted 3 lines · Andrew Morton

hm, OK.

I've now mounted 15000 devpts's and still no slowdown is evident.

This is the test script I used; graphing the results is quite interesting.

-hpa

#6 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

hm, OK.

I've now mounted 15000 devpts's and still no slowdown is evident.

I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.

Graphing the delays seem to indicate O(n^2) behavior.

umounts do not appear affected; each umount still take negible time.

-hpa

maintainer
#7 lore raw
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines · Andrew Morton

hm, OK.

I've now mounted 15000 devpts's and still no slowdown is evident.

Quoted 7 lines

I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.

Graphing the delays seem to indicate O(n^2) behavior.

umounts do not appear affected; each umount still take negible time.

I think I know what's going on. /sbin/mount is linked against libselinux /sbin/umount is not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.

What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire /proc/mounts in search of selinuxfs mounts.

If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained. I'd expect similar slowdown from ls on an empty directory, BTW - /bin/ls is linked against the same thing, so it gets hit as well. Before it even gets to main().

#8 lore raw
Quoted 20 lines · On Wed, 24 Jun 2009 21:01:08 +0100 Al Viro <[email protected]> wrote:
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines · Andrew Morton

hm, OK.

I've now mounted 15000 devpts's and still no slowdown is evident.

Quoted 7 lines

I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.

Graphing the delays seem to indicate O(n^2) behavior.

umounts do not appear affected; each umount still take negible time.

I think I know what's going on. /sbin/mount is linked against libselinux /sbin/umount is not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.

Quoted 6 lines

What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire /proc/mounts in search of selinuxfs mounts.

If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained.

CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1 CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULT is not set # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set

Quoted 3 lines

I'd expect similar slowdown from
ls on an empty directory, BTW - /bin/ls is linked against the same thing, so it gets hit as well. Before it even gets to main().

#9 lore raw
Quoted 14 lines · Andrew Morton
Quoted 7 lines
Quoted 2 lines

umounts do not appear affected; each umount still take negible time.

I think I know what's going on. /sbin/mount is linked against libselinux /sbin/umount is not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.

Quoted 6 lines

What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire /proc/mounts in search of selinuxfs mounts.

If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained.

Quoted 10 lines

CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1 CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULT is not set # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set

Wow. It doesn't even check for selinux actually being enabled before embarking on this stupidity.

FAIL.

-hpa

#10 lore raw
In reply to Ray Lee
Quoted 6 lines · Ray Lee
Quoted 3 lines
akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts
mount -n -i -t devpts none /tmp/pts  0.00s user 0.00s system 0% cpu 0.002 total

I read the Original Post as that test being performed with 5k /dev/pts already mounted.

A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.

The same thing happens with ramfs, so the problem is either in libfs or in the VFS; I suspect the latter as I don't really think we have had a huge reason to optimize a very large number of mounts in the past.

-hpa

#11 lore raw
Quoted 5 lines · H. Peter Anvin

A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.

I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:

# wc -l /proc/mounts
153481 /proc/mounts
# fgrep -c devpts /proc/mounts
10231

On my other servers there're slowdown already by this number of:

# wc -l /proc/mounts
49889 /proc/mounts
# fgrep -c devpts /proc/mounts
3323

But the first box is totally empty, others are in use, may be this matters?

Signature · 1 line
BRGDS. Alexey Vlasov.
#12 lore raw
Quoted 16 lines · Alexey Vlasov
Quoted 5 lines · H. Peter Anvin

A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.

I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:

# wc -l /proc/mounts
153481 /proc/mounts
# fgrep -c devpts /proc/mounts
10231
Quoted 5 lines

On my other servers there're slowdown already by this number of:

# wc -l /proc/mounts
49889 /proc/mounts
# fgrep -c devpts /proc/mounts
3323

One more notice, box with slowdowns:
...
geteuid() = 0
getcwd("/home/staff/r_renton"..., 4095) = 21 readlink("/home/staff/r_renton/none", 0x7ffffa32aa00, 4096) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 mount("none", "/home/www/server_7e014b11/dev/pts",
"devpts", MS_MGC_VAL, NULL) = 0 getcwd("/home/staff/r_renton"..., 4095) = 21
...

And this on the new one, about 100k mounting points and no slowdowns at all.
...
getuid() = 0
geteuid() = 0
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 mount("none", "/home/www/server_56d48c7d/dev/pts", "devpts", MS_MGC_VAL, NULL) = 0
readlink("/home", 0x7fff79a82e10, 4096) = -1 EINVAL (Invalid argument)
...

Signature · 1 line
BRGDS. Alexey Vlasov.
#13 lore raw
In reply to Alexey Vlasov
Quoted 16 lines · Alexey Vlasov
Quoted 5 lines · H. Peter Anvin

A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.

I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:

# wc -l /proc/mounts
153481 /proc/mounts
# fgrep -c devpts /proc/mounts
10231
Quoted 9 lines

On my other servers there're slowdown already by this number of:

# wc -l /proc/mounts
49889 /proc/mounts
# fgrep -c devpts /proc/mounts
3323

But the first box is totally empty, others are in use, may be this matters?

Do you have /selinux mounted on one of them?

-hpa

#14 lore raw
Quoted 2 lines · H. Peter Anvin

Do you have /selinux mounted on one of them?

I don't use SELinux anywhere.

Signature · 1 line
BRGDS. Alexey Vlasov.
#15 lore raw
Quoted 5 lines · Alexey Vlasov
Quoted 1 line · H. Peter Anvin

Do you have /selinux mounted on one of them?

I don't use SELinux anywhere.

That's, apparently, the problem: some version of the mount binary will search all of /proc/mounts for selinux if it isn't mounted on /selinux.
This is a serious performance bug, obviously.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.