Skip to contentSkip to conversation

A problem about DIRECT IO on ext3

DISCUSSION colin 6 participants 14 messages Aug 29, 2005 – Oct 17, 2005

lore
Share & tools
lore

colin

author
#0 lore raw

Hi all,
I wrote a simple program to test direct io, and found that there are some strange behaviors of it on "ext3".
My simple program is below. Assume that the executable file name is "directio". If I do the following:
1. cp directio aaa
2. ./directio directio aaa

The size of aaa is about the same with directio. This is wrong. It should be 3 times the size of directio because there are 2 write operations and one lseek to the file end.

If the second file is not opened with "O_DIRECT", the result is correct.

What's the problem of direct io? I found that if I remove the instruction of lseek, the result is correct.
Is there any problem of lseek when doing direct io on ext3?
My platform is 2.6.11.

Regards,
Colin

#define _GNU_SOURCE

#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdlib.h>

int main(int argc, char **argv) {

int fd1, fd2, count;
char *ptr1, *ptr2;

if(argc == 3) {
fd1 = open(argv[1], O_RDONLY | O_DIRECT, S_IRWXU); fd2 = open(argv[2], O_RDWR | O_CREAT | O_DIRECT);
} else {
printf("Error syntax\n");
exit(1);
}
printf("%d\n", lseek(fd2, 0, SEEK_END));

ptr1 = malloc(4096 + 4096-1);
ptr2 = (void*)((int)ptr1 - (int)ptr1 % 4096 + 4096);

do {
count = read(fd1, ptr2, 4096);
if(!count)

    break;

write(fd2, ptr2, 4096);
write(fd2, ptr2, 4096);
} while(count > 0);

free(ptr1);
close(fd1);
close(fd2);
}

#1 lore raw
Quoted 8 lines · colin

I wrote a simple program to test direct io, and found that there are some strange behaviors of it on "ext3".
My simple program is below. Assume that the executable file name is "directio". If I do the following:
1. cp directio aaa
2. ./directio directio aaa

The size of aaa is about the same with directio. This is wrong.

No, it's right, but it's not what you expected.

Quoted 2 lines

It should be 3 times the size of directio because there are 2 write operations and one lseek to the file end.

I suggest to strace() your program to see what happens.

Quoted 4 lines

If the second file is not opened with "O_DIRECT", the result is correct.

What's the problem of direct io? I found that if I remove the instruction of lseek, the result is correct.

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use getpagesize() instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size

Quoted 2 lines

Is there any problem of lseek when doing direct io on ext3? My platform is 2.6.11.

There is no problem.

Quoted 10 lines

Regards,
Colin

#define _GNU_SOURCE

#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdlib.h>

Compile your program with -Wall, you're missing quite some include files over here.

Quoted 5 lines

int main(int argc, char **argv) {

int fd1, fd2, count;
char *ptr1, *ptr2;

Quoted 8 lines

if(argc == 3) {
fd1 = open(argv[1], O_RDONLY | O_DIRECT, S_IRWXU); fd2 = open(argv[2], O_RDWR | O_CREAT | O_DIRECT);
} else {
printf("Error syntax\n");
exit(1);
}
printf("%d\n", lseek(fd2, 0, SEEK_END));

Make that lseek(fd2, 4 * 4096, SEEK_SET);

Quoted 2 lines

ptr1 = malloc(4096 + 4096-1);
ptr2 = (void*)((int)ptr1 - (int)ptr1 % 4096 + 4096);

Use memalign() or posix_memalign().

Quoted 4 lines

do {
count = read(fd1, ptr2, 4096);
if(!count)

    break;

And what happens when count < 0 ?

Quoted 2 lines
  write(fd2, ptr2, 4096);
  write(fd2, ptr2, 4096);

Check return values.

Quoted 5 lines

} while(count > 0);

free(ptr1);
close(fd1);
close(fd2);

return 0;

Quoted 1 line

}

With the changes, the result is:

erik@arthur:/tmp > ls -l directio aaa -rwxr-xr-x 1 erik erik 49152 2005-08-29 15:26 aaa* -rwxr-xr-x 1 erik erik 12628 2005-08-29 15:26 directio*

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#2 lore raw
Quoted 6 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use getpagesize() instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size

Actually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use getpagesize() instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size

Actually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.

OK, so that's different from 2.4 where reads/writes needed to be soft block aligned and buffers page aligned.

Erik

Signature · 3 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
| Data lost? Stay calm and contact Harddisk-recovery.com
#4 lore raw
Quoted 9 lines · Erik Mouw
Quoted 8 lines · Jens Axboe
Quoted 6 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use getpagesize() instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size

Quoted 5 lines
Quoted 2 lines

Actually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.

OK, so that's different from 2.4 where reads/writes needed to be soft block aligned and buffers page aligned.

Yes, 2.6 has relaxed the restrictions there somewhat.

Signature · 1 line
Jens Axboe
#5 lore raw
In reply to Jens Axboe
Quoted 7 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use

getpagesize() instead

Quoted 2 lines

of assuming that a page is 4k
- reads and writes need to happen *in* multiples

Quoted 11 lines

of the soft block size

Quoted 1 line

- reads and writes need to happen *at* multiples

of the soft block size

Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.

Signature · 1 line
Jens Axboe

On 2.4 the open call succeeds with O_DIRECT but read returns -EINVAL for any block size (512, 1024 ..16384)

open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270) = 4 read(3, 0xbfffdc40, 16384) = -1 EINVAL (Invalid argument)

how to correct this problem ?

__________________________________
Start your day with Yahoo! - Make it your home page!
http://www.yahoo.com/r/hs

#6 lore raw
Quoted 15 lines · --- li nux
Quoted 11 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use

getpagesize() instead

Quoted 2 lines

of assuming that a page is 4k
- reads and writes need to happen *in* multiples

of the soft block size

Quoted 1 line

- reads and writes need to happen *at* multiples

of the soft block size

Quoted 7 lines
Quoted 7 lines

Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.

Signature · 1 line
Jens Axboe

On 2.4 the open call succeeds with O_DIRECT
but read returns -EINVAL for any block size (512,
1024 ..16384)

open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270) = 3
read(3, 0xbfffdc40, 16384) = -1 EINVAL (Invalid argument)

how to correct this problem ?

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

#7 lore raw
In reply to li nux
Quoted 26 lines · li nux
Quoted 7 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw

There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use

getpagesize() instead

Quoted 2 lines

of assuming that a page is 4k
- reads and writes need to happen *in* multiples

Quoted 11 lines

of the soft block size

Quoted 1 line

- reads and writes need to happen *at* multiples

of the soft block size

Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.

Signature · 1 line
Jens Axboe

On 2.4 the open call succeeds with O_DIRECT but read returns -EINVAL for any block size (512, 1024 ..16384)

Quoted 6 lines

open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270) = 4 read(3, 0xbfffdc40, 16384) = -1 EINVAL (Invalid argument)

how to correct this problem ?

See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.

So in short, just align your read buffer to the same as your block size and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Signature · 1 line
Jens Axboe
#8 lore raw
Quoted 10 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.

So in short, just align your read buffer to the same as your block size and you will be fine. Example:

Quoted 8 lines

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Shouldn't one use posix_memalign(3) for that?

Grzegorz Kulewski

#9 lore raw
Quoted 15 lines · Grzegorz Kulewski
Quoted 14 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.

So in short, just align your read buffer to the same as your block size and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))

Quoted 6 lines
Quoted 4 lines

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Shouldn't one use posix_memalign(3) for that?

Dunno if one 'should', one 'can' if one wants to. I prefer to do it manually so I don't have to jump through #define hoops to get at it (which, btw, still doesn't expose it on this machine).

Signature · 1 line
Jens Axboe
#10 lore raw
Quoted 27 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You

need to align that as

Quoted 1 line
Quoted 1 line

well. This is needed because the hardware will

dma directly to the user

Quoted 1 line
Quoted 1 line

buffer, and to be on the safe side we require the

same alignment as the

Quoted 1 line
Quoted 1 line

block layer will normally generate for file

system io.

Quoted 2 lines
Quoted 2 lines

So in short, just align your read buffer to the

same as your block size

Quoted 5 lines
Quoted 5 lines

and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +

MASK) & ~(MASK))

Quoted 6 lines
Quoted 5 lines

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Quoted 11 lines
Quoted 1 line

Shouldn't one use posix_memalign(3) for that?

Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).

Signature · 1 line
Jens Axboe

Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.

Can somebody please throw some light on how to find your system's hard/soft block size ?

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

#11 lore raw
Quoted 40 lines · li nux
Quoted 36 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You

need to align that as

Quoted 1 line
Quoted 1 line

well. This is needed because the hardware will

dma directly to the user

Quoted 1 line
Quoted 1 line

buffer, and to be on the safe side we require the

same alignment as the

Quoted 1 line
Quoted 1 line

block layer will normally generate for file

system io.

Quoted 2 lines
Quoted 2 lines

So in short, just align your read buffer to the

same as your block size

Quoted 5 lines
Quoted 5 lines

and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +

MASK) & ~(MASK))

Quoted 7 lines
Quoted 5 lines

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Shouldn't one use posix_memalign(3) for that?

Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).

Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.

2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?

Quoted 2 lines

Can somebody please throw some light on how to find your system's hard/soft block size ?

It's a per-device (or even per-partition, in case of mounted partitions) setting, you can use the BLKBSZGET and BLKSSZGET ioctls to query for soft/hard sector sizes.

Signature · 1 line
Jens Axboe
#12 lore raw
Quoted 44 lines · Jens Axboe
Quoted 43 lines · li nux
Quoted 38 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You

need to align that as

Quoted 1 line
Quoted 1 line

well. This is needed because the hardware will

dma directly to the user

Quoted 1 line
Quoted 1 line

buffer, and to be on the safe side we require the

same alignment as the

Quoted 1 line
Quoted 1 line

block layer will normally generate for file

system io.

Quoted 2 lines
Quoted 2 lines

So in short, just align your read buffer to the

same as your block size

Quoted 5 lines
Quoted 5 lines

and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +

MASK) & ~(MASK))

Quoted 7 lines
Quoted 5 lines

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Shouldn't one use posix_memalign(3) for that?

Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).

Signature · 1 line
Jens Axboe
Quoted 8 lines
Quoted 4 lines

Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.

2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?

2.6 also has the same restriction. But, if the "filesystem blocksize alignment" (soft block size) fails, we try to see if its aligned with hard sector size (512). If so, we can do the IO.

2.4 fails if the offset or buffer is NOT filesystem blocksize aligned. Period.

So, its possible that your buffer is atleast 512byte aligned, there by succeeding on 2.6

BTW, posix_memalign() or valloc() should be safe.

Quoted 8 lines
Quoted 3 lines

Can somebody please throw some light on how to find your system's hard/soft block size ?

It's a per-device (or even per-partition, in case of mounted partitions) setting, you can use the BLKBSZGET and BLKSSZGET ioctls to query for soft/hard sector sizes.

Thanks,
Badari

#13 lore raw
Quoted 54 lines · Badari Pulavarty
Quoted 44 lines · Jens Axboe
Quoted 43 lines · li nux
Quoted 38 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line

how to correct this problem ?

See your buffer address, it's not aligned. You

need to align that as

Quoted 1 line
Quoted 1 line

well. This is needed because the hardware will

dma directly to the user

Quoted 1 line
Quoted 1 line

buffer, and to be on the safe side we require the

same alignment as the

Quoted 1 line
Quoted 1 line

block layer will normally generate for file

system io.

Quoted 2 lines
Quoted 2 lines

So in short, just align your read buffer to the

same as your block size

Quoted 5 lines
Quoted 5 lines

and you will be fine. Example:

#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +

MASK) & ~(MASK))

Quoted 7 lines
Quoted 5 lines

char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);

read(fd, buf, BS);

Shouldn't one use posix_memalign(3) for that?

Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).

Signature · 1 line
Jens Axboe
Quoted 8 lines
Quoted 4 lines

Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.

2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?

Quoted 6 lines

2.6 also has the same restriction. But, if the "filesystem blocksize alignment" (soft block size) fails, we try to see if its aligned with hard sector size (512). If so, we can do the IO.

2.4 fails if the offset or buffer is NOT filesystem blocksize aligned. Period.

I'm aware of that, however this particular case was about the buffer alignment (which was 32-bytes in the strace). And that should not work for 2.6 either.

The block-size alignment is really a separate property of correctness.

Quoted 1 line

BTW, posix_memalign() or valloc() should be safe.

Certainly.

Signature · 1 line
Jens Axboe