devpts mounts too slowly
Hi.
In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts real 0m0.295s user 0m0.000s sys 0m0.288s
# time mount -i -n --bind /var/bin /home/staff/.server-1/bin real 0m0.004s user 0m0.000s sys 0m0.004s
Signature · 1 line
BRGDS. Alexey Vlasov.
Quoted 7 lines · Alexey Vlasov
Hi.
In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.
Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts real 0m0.295s user 0m0.000s sys 0m0.288s
A profile would be handy here. There is no sensible reason for this to that 0.3 seconds. This might be RCU-epoch related?
-hpa
Quoted 8 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 7 lines · Alexey Vlasov
Hi.
In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.
Quoted 9 lines
Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts real 0m0.295s user 0m0.000s sys 0m0.288sA profile would be handy here. There is no sensible reason for this to that 0.3 seconds. This might be RCU-epoch related?
It seems OK here.
akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts mount -n -i -t devpts none /tmp/pts 0.00s user 0.00s system 0% cpu 0.002 total
Quoted 17 lines · Andrew Morton
Quoted 14 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 12 lines · Alexey Vlasov
Hi.
In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts real 0m0.295s user 0m0.000s sys 0m0.288s
Quoted 8 lines
Quoted 3 lines
A profile would be handy here. There is no sensible reason for this to that 0.3 seconds. This might be RCU-epoch related?
It seems OK here.
akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts mount -n -i -t devpts none /tmp/pts 0.00s user 0.00s system 0% cpu 0.002 total
I read the Original Post as that test being performed with 5k /dev/pts already mounted.
Quoted 22 lines · On Tue, 9 Jun 2009 16:30:31 -0700 Ray Lee <[email protected]> wrote:
Quoted 21 lines · Andrew Morton
Quoted 17 lines · On Wed, 03 Jun 2009 14:28:56 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 7 lines · Alexey Vlasov
Hi.
In comparison with bind/proc mounting it takes a lot more time mounting devpts. Is it possible to somehow accelerate the process? Now there are about 5k devpts mounting, it already took 20 minutes, further will take even more time.
Quoted 5 lines
# time mount -n -i -t devpts none /home/staff/.server-1/dev/pts real __ __0m0.295s user __ __0m0.000s sys __ __ 0m0.288sA profile would be handy here.
__Thereis no sensible reason for this to that 0.3 seconds.__Thismight be RCU-epoch related?
Quoted 7 lines
Quoted 4 lines
It seems OK here.
akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts mount -n -i -t devpts none /tmp/pts __0.00s user 0.00s system 0% cpu 0.002 totalI read the Original Post as that test being performed with 5k
/dev/ptsalready mounted.
hm, OK.
I've now mounted 15000 devpts's and still no slowdown is evident.
Quoted 3 lines · Andrew Morton
hm, OK.
I've now mounted 15000 devpts's and still no slowdown is evident.
This is the test script I used; graphing the results is quite interesting.
-hpa
Quoted 4 lines · Andrew Morton
hm, OK.
I've now mounted 15000 devpts's and still no slowdown is evident.
I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.
Graphing the delays seem to indicate O(n^2) behavior.
umounts do not appear affected; each umount still take negible time.
-hpa
Al Viro
maintainerQuoted 6 lines · H. Peter Anvin
Quoted 4 lines · Andrew Morton
hm, OK.
I've now mounted 15000 devpts's and still no slowdown is evident.
Quoted 7 lines
I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.
Graphing the delays seem to indicate O(n^2) behavior.
umounts do not appear affected; each umount still take negible time.
I think I know what's going on. /sbin/mount is linked against libselinux /sbin/umount is not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.
What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire /proc/mounts in search of selinuxfs mounts.
If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained. I'd expect similar slowdown from ls on an empty directory, BTW - /bin/ls is linked against the same thing, so it gets hit as well. Before it even gets to main().
Quoted 20 lines · On Wed, 24 Jun 2009 21:01:08 +0100 Al Viro <[email protected]> wrote:
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines · Andrew Morton
hm, OK.
I've now mounted 15000 devpts's and still no slowdown is evident.
Quoted 7 lines
I ran my test script, mounting ramfs, with n=100000, and well, gave up since it hadn't gotten any further than 57000 or so overnight. At that time each individual mount was taking several seconds.
Graphing the delays seem to indicate O(n^2) behavior.
umounts do not appear affected; each umount still take negible time.
I think I know what's going on.
/sbin/mountis linked against libselinux/sbin/umountis not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.
Quoted 6 lines
What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire
/proc/mountsin search of selinuxfs mounts.If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained.
CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1 CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULT is not set # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
Quoted 3 lines
I'd expect similar slowdown from
ls on an empty directory, BTW -/bin/lsis linked against the same thing, so it gets hit as well. Before it even gets tomain().
Quoted 14 lines · Andrew Morton
Quoted 7 lines
Quoted 2 lines
umounts do not appear affected; each umount still take negible time.
I think I know what's going on.
/sbin/mountis linked against libselinux/sbin/umountis not. And FPOS in question blows if you
* do not have selinuxfs mounted (e.g. because selinux is not enabled)
* have a lot of mounts.Quoted 6 lines
What happens is that this piece of crap checks for presence of selinuxfs on /selinux; then, if the thing isn't there, we go and scan the entire
/proc/mountsin search of selinuxfs mounts.If akpm has selinux enabled on his testbox and you don't have it on yours, we have all observations explained.
Quoted 10 lines
CONFIG_SECURITY_SELINUX=yCONFIG_SECURITY_SELINUX_BOOTPARAM=yCONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1CONFIG_SECURITY_SELINUX_DISABLE=yCONFIG_SECURITY_SELINUX_DEVELOP=yCONFIG_SECURITY_SELINUX_AVC_STATS=yCONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
#CONFIG_SECURITY_SELINUX_ENABLE_SECMARK_DEFAULTis not set #CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAXis not set
Wow. It doesn't even check for selinux actually being enabled before embarking on this stupidity.
FAIL.
-hpa
Quoted 6 lines · Ray Lee
Quoted 3 lines
akpm2:/home/akpm/xx# time mount -n -i -t devpts none /tmp/pts mount -n -i -t devpts none /tmp/pts 0.00s user 0.00s system 0% cpu 0.002 totalI read the Original Post as that test being performed with 5k
/dev/ptsalready mounted.
A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.
The same thing happens with ramfs, so the problem is either in libfs or in the VFS; I suspect the latter as I don't really think we have had a huge reason to optimize a very large number of mounts in the past.
-hpa
Quoted 5 lines · H. Peter Anvin
A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.
I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:
# wc -l /proc/mounts 153481 /proc/mounts # fgrep -c devpts /proc/mounts 10231
On my other servers there're slowdown already by this number of:
# wc -l /proc/mounts 49889 /proc/mounts # fgrep -c devpts /proc/mounts 3323
But the first box is totally empty, others are in use, may be this matters?
Signature · 1 line
BRGDS. Alexey Vlasov.
Quoted 16 lines · Alexey Vlasov
Quoted 5 lines · H. Peter Anvin
A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.
I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:# wc -l /proc/mounts 153481 /proc/mounts # fgrep -c devpts /proc/mounts 10231
Quoted 5 lines
On my other servers there're slowdown already by this number of:
# wc -l /proc/mounts 49889 /proc/mounts # fgrep -c devpts /proc/mounts 3323
One more notice, box with slowdowns:
...geteuid() = 0getcwd("/home/staff/r_renton"..., 4095) = 21 readlink("/home/staff/r_renton/none", 0x7ffffa32aa00, 4096) = -1 ENOENT (No such file or directory)rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 mount("none", "/home/www/server_7e014b11/dev/pts",
"devpts", MS_MGC_VAL, NULL) = 0 getcwd("/home/staff/r_renton"..., 4095) = 21
...
And this on the new one, about 100k mounting points and no slowdowns at all.
...getuid() = 0geteuid() = 0rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0 mount("none", "/home/www/server_56d48c7d/dev/pts", "devpts", MS_MGC_VAL, NULL) = 0readlink("/home", 0x7fff79a82e10, 4096) = -1 EINVAL (Invalid argument)
...
Signature · 1 line
BRGDS. Alexey Vlasov.
Quoted 16 lines · Alexey Vlasov
Quoted 5 lines · H. Peter Anvin
A quick script seems to confirm that the time starts creeping up as the number of mount points increase; on my Core i7 system the 10000th mount takes around 50 ms whereas the early instances are submillisecond, and the increase *seems* to be O(n log n) or thereabouts -- slightly superlinear but not a good fit for a power law.
I have prepared a new machine for testing, and I don't see any slowdown by mounting devpts.
Total mounts on it:# wc -l /proc/mounts 153481 /proc/mounts # fgrep -c devpts /proc/mounts 10231
Quoted 9 lines
On my other servers there're slowdown already by this number of:
# wc -l /proc/mounts 49889 /proc/mounts # fgrep -c devpts /proc/mounts 3323But the first box is totally empty, others are in use, may be this matters?
Do you have /selinux mounted on one of them?
-hpa
Quoted 2 lines · H. Peter Anvin
Do you have /selinux mounted on one of them?
I don't use SELinux anywhere.
Signature · 1 line
BRGDS. Alexey Vlasov.
Quoted 5 lines · Alexey Vlasov
Quoted 1 line · H. Peter Anvin
Do you have /selinux mounted on one of them?
I don't use SELinux anywhere.
That's, apparently, the problem: some version of the mount binary will search all of /proc/mounts for selinux if it isn't mounted on /selinux.
This is a serious performance bug, obviously.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.