Skip to contentSkip to conversation

rewrite very slow

DISCUSSION DS 3 participants 5 messages Feb 15, 2008 – Feb 16, 2008

lore
Share & tools
lore

DS

author
#0 lore raw

Hello,

I need some help to tunning my storage.

Very simple test at begin (perl script test.pl):

#!/usr/bin/perl

$time=time();
for ($i=1;$i<100;$i++)
{
open (SUBOR,">$i.txt");
print SUBOR "aaaaaaaaaaaaaaaaaaa\n";
close (SUBOR);
print "WRITE $i. FILE\n";
}
$time2=time();
$rozdiel_casov=$time2-$time;

print "TIME ".$rozdiel_casov." sekund\n";

First run:

file1:/mnt/hosting/test# ./test.pl
WRITE 1. FILE
WRITE 2. FILE
...
WRITE 98. FILE
WRITE 99. FILE
TIME 0 sekund

Every next run (rewrite existing files) is very very very slow:

file1:/mnt/hosting/test# ./test.pl
WRITE 1. FILE
WRITE 2. FILE
...

WRITE 98. FILE
WRITE 99. FILE
TIME 43 sekund

I try it on ext3. Every run is only 1-3seconds.

I tried tunning my storage.
My last mount specification:

(rw,noatime,nodiratime,nobarrier,osyncisosync,logbsize=256k,logbufs=8,quota) I tried all possible configuration (barrier, nobarrier, osyncisosync, logbssize=XX,logbufs=2-8) nothing help me.

file1:/mnt/hosting/test# xfs_info /mnt/hosting/
meta-data=/dev/mapper/vghosting-hosting isize=256    agcount=32,
agsize=7626752 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=244056064,
imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

I test it on 4 other xfs storages and results was similar.

Any ideas?

Thanks.

Dusan

#1 lore raw
Quoted 3 lines · DS

Hello,

I need some help to tunning my storage.

...

Quoted 1 line

TIME 43 sekund

What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.

but granted, on ext3 I get 0 seconds for every run.

Also the difference appears to be O_TRUNC (which the perl script does); if I code it in c:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)

{
        int i;
        int fd;
        char file[4];

        for (i = 0; i < 100; i++) {
                sprintf(file, "%d.txt", i);
                fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
                write(fd, "aaaaaaaaaaaaaaaaaaa\n");
                close(fd);
        }
}

rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)

-Eric

#2 lore raw
Quoted 35 lines · Eric Sandeen
Quoted 3 lines · DS

Hello,

I need some help to tunning my storage.

...

Quoted 1 line

TIME 43 sekund

What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.

but granted, on ext3 I get 0 seconds for every run.

Also the difference appears to be O_TRUNC (which the perl script does); if I code it in c:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)

{
        int i;
        int fd;
        char file[4];

        for (i = 0; i < 100; i++) {
                sprintf(file, "%d.txt", i);
                fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
                write(fd, "aaaaaaaaaaaaaaaaaaa\n");
                close(fd);
        }
}
Quoted 3 lines

rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)

Yup - after a truncate we use flush-on-close semantics if the file is closed before pdflush does writeback. yes, it has a measurable impact on silly microbenchmarks like this, but nobody even noticed it when we introduced this code 2-3 years ago...

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
author
#3 lore raw

Hmm, and what cat I do to get it work now?

DS

Quoted 52 lines · David Chinner
Quoted 35 lines · Eric Sandeen
Quoted 3 lines · DS

Hello,

I need some help to tunning my storage.

...

Quoted 1 line

TIME 43 sekund

What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.

but granted, on ext3 I get 0 seconds for every run.

Also the difference appears to be O_TRUNC (which the perl script does); if I code it in c:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)

{
        int i;
        int fd;
        char file[4];

        for (i = 0; i < 100; i++) {
                sprintf(file, "%d.txt", i);
                fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
                write(fd, "aaaaaaaaaaaaaaaaaaa\n");
                close(fd);
        }
}
Quoted 3 lines

rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)

Yup - after a truncate we use flush-on-close semantics if the file is closed before pdflush does writeback. yes, it has a measurable impact on silly microbenchmarks like this, but nobody even noticed it when we introduced this code 2-3 years ago...

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
author
#4 lore raw
In reply to Eric Sandeen

Test configuration:
Linux kernel 2.6.23.1 #1 SMP
2x Intel(R) Xeon(TM) CPU 2.40GHz with HT

iSCSI storage (1TB - 7 sata disks in RAID6, 2GB cache controler)

Yes, your "test" works fine:

file1:/mnt/hosting/test# time ./test
real    0m0.334s
user    0m0.000s
sys     0m0.000s

Is there any way to get it work for perl/php/other scripts/programs?

DS

Quoted 42 lines · Eric Sandeen
Quoted 3 lines · DS

Hello,

I need some help to tunning my storage.

...

Quoted 1 line

TIME 43 sekund

What kernel? when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I see 2 and 7 seconds for rewrite, respectively.

but granted, on ext3 I get 0 seconds for every run.

Also the difference appears to be O_TRUNC (which the perl script does); if I code it in c:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)

{
        int i;
        int fd;
        char file[4];

        for (i = 0; i < 100; i++) {
                sprintf(file, "%d.txt", i);
                fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
                write(fd, "aaaaaaaaaaaaaaaaaaa\n");
                close(fd);
        }
}

rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it. Not sure about the xfs/ext3 difference... this is probably a side-effect of flushes xfs put into place on truncate (IIRC?)

-Eric