rewrite very slow
Hello,
I need some help to tunning my storage.
Very simple test at begin (perl script test.pl):
#!/usr/bin/perl
$time=time();
for ($i=1;$i<100;$i++)
{
open (SUBOR,">$i.txt");
print SUBOR "aaaaaaaaaaaaaaaaaaa\n";
close (SUBOR);
print "WRITE $i. FILE\n";
}
$time2=time();
$rozdiel_casov=$time2-$time;
print "TIME ".$rozdiel_casov." sekund\n";
First run:
file1:/mnt/hosting/test# ./test.pl WRITE 1. FILE WRITE 2. FILE ... WRITE 98. FILE WRITE 99. FILE TIME 0 sekund
Every next run (rewrite existing files) is very very very slow:
file1:/mnt/hosting/test# ./test.pl WRITE 1. FILE WRITE 2. FILE ...
WRITE 98. FILE
WRITE 99. FILE
TIME 43 sekund
I try it on ext3. Every run is only 1-3seconds.
I tried tunning my storage.
My last mount specification:
(rw,noatime,nodiratime,nobarrier,osyncisosync,logbsize=256k,logbufs=8,quota) I tried all possible configuration (barrier, nobarrier, osyncisosync, logbssize=XX,logbufs=2-8) nothing help me.
file1:/mnt/hosting/test# xfs_info /mnt/hosting/
meta-data=/dev/mapper/vghosting-hosting isize=256 agcount=32,
agsize=7626752 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=244056064,
imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0I test it on 4 other xfs storages and results was similar.
Any ideas?
Thanks.
Dusan
Quoted 3 lines · DS
Hello,
I need some help to tunning my storage.
...
Quoted 1 line
TIME 43 sekund
What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.
but granted, on ext3 I get 0 seconds for every run.
Also the difference appears to be O_TRUNC (which the perl script does); if I code it in c:
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
void main(void)
{
int i;
int fd;
char file[4];
for (i = 0; i < 100; i++) {
sprintf(file, "%d.txt", i);
fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
write(fd, "aaaaaaaaaaaaaaaaaaa\n");
close(fd);
}
}rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)
-Eric
Quoted 35 lines · Eric Sandeen
Quoted 3 lines · DS
Hello,
I need some help to tunning my storage.
...
Quoted 1 line
TIME 43 sekund
What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.
but granted, on ext3 I get 0 seconds for every run.
Also the difference appears to be
O_TRUNC(which the perl script does); if I code it in c:#include <
sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>void main(void)
{ int i; int fd; char file[4]; for (i = 0; i < 100; i++) { sprintf(file, "%d.txt", i); fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644); write(fd, "aaaaaaaaaaaaaaaaaaa\n"); close(fd); } }
Quoted 3 lines
rewrite is a bit slower w/
O_TRUNCin place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)
Yup - after a truncate we use flush-on-close semantics if the file is closed before pdflush does writeback. yes, it has a measurable impact on silly microbenchmarks like this, but nobody even noticed it when we introduced this code 2-3 years ago...
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Hmm, and what cat I do to get it work now?
DS
Quoted 52 lines · David Chinner
Quoted 35 lines · Eric Sandeen
Quoted 3 lines · DS
Hello,
I need some help to tunning my storage.
...
Quoted 1 line
TIME 43 sekund
What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.
but granted, on ext3 I get 0 seconds for every run.
Also the difference appears to be
O_TRUNC(which the perl script does); if I code it in c:#include <
sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>void main(void)
{ int i; int fd; char file[4]; for (i = 0; i < 100; i++) { sprintf(file, "%d.txt", i); fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644); write(fd, "aaaaaaaaaaaaaaaaaaa\n"); close(fd); } }Quoted 3 lines
rewrite is a bit slower w/
O_TRUNCin place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)Yup - after a truncate we use flush-on-close semantics if the file is closed before pdflush does writeback. yes, it has a measurable impact on silly microbenchmarks like this, but nobody even noticed it when we introduced this code 2-3 years ago...
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Test configuration:
Linux kernel 2.6.23.1 #1 SMP
2x Intel(R) Xeon(TM) CPU 2.40GHz with HT
iSCSI storage (1TB - 7 sata disks in RAID6, 2GB cache controler)
Yes, your "test" works fine:
file1:/mnt/hosting/test# time ./test
real 0m0.334s user 0m0.000s sys 0m0.000s
Is there any way to get it work for perl/php/other scripts/programs?
DS
Quoted 42 lines · Eric Sandeen
Quoted 3 lines · DS
Hello,
I need some help to tunning my storage.
...
Quoted 1 line
TIME 43 sekund
What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.
but granted, on ext3 I get 0 seconds for every run.
Also the difference appears to be
O_TRUNC(which the perl script does); if I code it in c:#include <
sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>void main(void)
{ int i; int fd; char file[4]; for (i = 0; i < 100; i++) { sprintf(file, "%d.txt", i); fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644); write(fd, "aaaaaaaaaaaaaaaaaaa\n"); close(fd); } }rewrite is a bit slower w/
O_TRUNCin place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)-Eric