A problem about DIRECT IO on ext3
Hi all,
I wrote a simple program to test direct io, and found that there are some strange behaviors of it on "ext3".
My simple program is below. Assume that the executable file name is "directio". If I do the following:
1. cp directio aaa
2. ./directio directio aaa
The size of aaa is about the same with directio. This is wrong. It should be 3 times the size of directio because there are 2 write operations and one lseek to the file end.
If the second file is not opened with "O_DIRECT", the result is correct.
What's the problem of direct io? I found that if I remove the instruction of lseek, the result is correct.
Is there any problem of lseek when doing direct io on ext3?
My platform is 2.6.11.
Regards,
Colin
#define _GNU_SOURCE
#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdlib.h>
int main(int argc, char **argv) {
int fd1, fd2, count;
char *ptr1, *ptr2;
if(argc == 3) {
fd1 = open(argv[1], O_RDONLY | O_DIRECT, S_IRWXU); fd2 = open(argv[2], O_RDWR | O_CREAT | O_DIRECT);
} else {
printf("Error syntax\n");
exit(1);
}printf("%d\n", lseek(fd2, 0, SEEK_END));
ptr1 = malloc(4096 + 4096-1);
ptr2 = (void*)((int)ptr1 - (int)ptr1 % 4096 + 4096);
do {
count = read(fd1, ptr2, 4096);
if(!count)
break;
write(fd2, ptr2, 4096);
write(fd2, ptr2, 4096);
} while(count > 0);
free(ptr1);
close(fd1);
close(fd2);
}
Quoted 8 lines · colin
I wrote a simple program to test direct io, and found that there are some strange behaviors of it on "ext3".
My simple program is below. Assume that the executable file name is "directio". If I do the following:
1. cp directio aaa
2. ./directio directio aaaThe size of aaa is about the same with directio. This is wrong.
No, it's right, but it's not what you expected.
Quoted 2 lines
It should be 3 times the size of directio because there are 2 write operations and one lseek to the file end.
I suggest to strace() your program to see what happens.
Quoted 4 lines
If the second file is not opened with "
O_DIRECT", the result is correct.What's the problem of direct io? I found that if I remove the instruction of lseek, the result is correct.
There are four prerequisites for direct IO:
- the file needs to be opened with O_DIRECT
- the buffer needs to be page aligned (hint: use getpagesize() instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size
Quoted 2 lines
Is there any problem of lseek when doing direct io on ext3? My platform is 2.6.11.
There is no problem.
Quoted 10 lines
Regards,
Colin#define
_GNU_SOURCE#include <
stdio.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdlib.h>
Compile your program with -Wall, you're missing quite some include files over here.
Quoted 5 lines
int
main(int argc, char **argv){int fd1, fd2, count;
char *ptr1, *ptr2;
Quoted 8 lines
if(argc == 3) {
fd1 =open(argv[1], O_RDONLY | O_DIRECT, S_IRWXU); fd2 =open(argv[2], O_RDWR | O_CREAT | O_DIRECT);
} else {
printf("Error syntax\n");
exit(1);
}printf("%d\n", lseek(fd2, 0, SEEK_END));
Make that lseek(fd2, 4 * 4096, SEEK_SET);
Quoted 2 lines
ptr1 = malloc(4096 + 4096-1);
ptr2 = (void*)((int)ptr1 - (int)ptr1 % 4096 + 4096);
Use memalign() or posix_memalign().
Quoted 4 lines
do {
count =read(fd1, ptr2, 4096);
if(!count)break;
And what happens when count < 0 ?
Quoted 2 lines
write(fd2, ptr2, 4096); write(fd2, ptr2, 4096);
Check return values.
Quoted 5 lines
} while(count > 0);
free(ptr1);
close(fd1);
close(fd2);
return 0;
Quoted 1 line
}
With the changes, the result is:
erik@arthur:/tmp > ls -l directio aaa -rwxr-xr-x 1 erik erik 49152 2005-08-29 15:26 aaa* -rwxr-xr-x 1 erik erik 12628 2005-08-29 15:26 directio*
Erik
Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
Quoted 6 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: usegetpagesize()instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size
Actually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: usegetpagesize()instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block sizeActually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.
OK, so that's different from 2.4 where reads/writes needed to be soft block aligned and buffers page aligned.
Erik
Signature · 3 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands | Data lost? Stay calm and contact Harddisk-recovery.com
Quoted 9 lines · Erik Mouw
Quoted 8 lines · Jens Axboe
Quoted 6 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: usegetpagesize()instead of assuming that a page is 4k
- reads and writes need to happen *in* multiples of the soft block size
- reads and writes need to happen *at* multiples of the soft block size
Quoted 5 lines
Quoted 2 lines
Actually, the buffer only needs to be hard block size aligned, same goes for the chunk size used for reads/writes.
OK, so that's different from 2.4 where reads/writes needed to be soft block aligned and buffers page aligned.
Yes, 2.6 has relaxed the restrictions there somewhat.
Signature · 1 line
Jens Axboe
Quoted 7 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: use
getpagesize()insteadQuoted 2 lines
of assuming that a page is 4k
- reads and writes need to happen *in* multiples
Quoted 11 lines
of the soft block size
Quoted 1 line
- reads and writes need to happen *at* multiples
of the soft block size
Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.Signature · 1 line
Jens Axboe
On 2.4 the open call succeeds with O_DIRECT but read returns -EINVAL for any block size (512, 1024 ..16384)
open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270) = 4 read(3, 0xbfffdc40, 16384) = -1 EINVAL (Invalid argument)
how to correct this problem ?
__________________________________
Start your day with Yahoo! - Make it your home page!
http://www.yahoo.com/r/hs
Quoted 15 lines · --- li nux
Quoted 11 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: use
getpagesize()insteadQuoted 2 lines
of assuming that a page is 4k
- reads and writes need to happen *in* multiplesof the soft block size
Quoted 1 line
- reads and writes need to happen *at* multiples
of the soft block size
Quoted 7 lines
Quoted 7 lines
Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.Signature · 1 line
Jens Axboe
On 2.4 the open call succeeds with O_DIRECT
but read returns -EINVAL for any block size (512,
1024 ..16384)
open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270) = 3
read(3, 0xbfffdc40, 16384) = -1 EINVAL (Invalid argument)
how to correct this problem ?
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
Quoted 26 lines · li nux
Quoted 7 lines · --- Jens Axboe
Quoted 3 lines · Erik Mouw
There are four prerequisites for direct IO:
- the file needs to be opened withO_DIRECT
- the buffer needs to be page aligned (hint: use
getpagesize()insteadQuoted 2 lines
of assuming that a page is 4k
- reads and writes need to happen *in* multiplesQuoted 11 lines
of the soft block size
Quoted 1 line
- reads and writes need to happen *at* multiples
of the soft block size
Actually, the buffer only needs to be hard block
size aligned, same goes
for the chunk size used for reads/writes.Signature · 1 line
Jens AxboeOn 2.4 the open call succeeds with
O_DIRECTbut read returns-EINVALfor any block size (512, 1024 ..16384)
Quoted 6 lines
open("/tmp/midstress_idx10", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 01001101270)= 4read(3, 0xbfffdc40, 16384)= -1 EINVAL (Invalid argument)how to correct this problem ?
See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.
So in short, just align your read buffer to the same as your block size and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);
Signature · 1 line
Jens Axboe
Quoted 10 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.
So in short, just align your read buffer to the same as your block size and you will be fine. Example:
Quoted 8 lines
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);
Shouldn't one use posix_memalign(3) for that?
Grzegorz Kulewski
Quoted 15 lines · Grzegorz Kulewski
Quoted 14 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You need to align that as well. This is needed because the hardware will dma directly to the user buffer, and to be on the safe side we require the same alignment as the block layer will normally generate for file system io.
So in short, just align your read buffer to the same as your block size and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) + MASK) & ~(MASK))
Quoted 6 lines
Quoted 4 lines
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);Shouldn't one use
posix_memalign(3)for that?
Dunno if one 'should', one 'can' if one wants to. I prefer to do it manually so I don't have to jump through #define hoops to get at it (which, btw, still doesn't expose it on this machine).
Signature · 1 line
Jens Axboe
Quoted 27 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You
need to align that as
Quoted 1 line
Quoted 1 line
well. This is needed because the hardware will
dma directly to the user
Quoted 1 line
Quoted 1 line
buffer, and to be on the safe side we require the
same alignment as the
Quoted 1 line
Quoted 1 line
block layer will normally generate for file
system io.
Quoted 2 lines
Quoted 2 lines
So in short, just align your read buffer to the
same as your block size
Quoted 5 lines
Quoted 5 lines
and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +MASK) & ~(MASK))
Quoted 6 lines
Quoted 5 lines
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);
Quoted 11 lines
Quoted 1 line
Shouldn't one use
posix_memalign(3)for that?Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).Signature · 1 line
Jens Axboe
Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.
Can somebody please throw some light on how to find your system's hard/soft block size ?
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
Quoted 40 lines · li nux
Quoted 36 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You
need to align that as
Quoted 1 line
Quoted 1 line
well. This is needed because the hardware will
dma directly to the user
Quoted 1 line
Quoted 1 line
buffer, and to be on the safe side we require the
same alignment as the
Quoted 1 line
Quoted 1 line
block layer will normally generate for file
system io.
Quoted 2 lines
Quoted 2 lines
So in short, just align your read buffer to the
same as your block size
Quoted 5 lines
Quoted 5 lines
and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +MASK) & ~(MASK))
Quoted 7 lines
Quoted 5 lines
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);Shouldn't one use
posix_memalign(3)for that?Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).
Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeThanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.
2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?
Quoted 2 lines
Can somebody please throw some light on how to find your system's hard/soft block size ?
It's a per-device (or even per-partition, in case of mounted partitions) setting, you can use the BLKBSZGET and BLKSSZGET ioctls to query for soft/hard sector sizes.
Signature · 1 line
Jens Axboe
Quoted 44 lines · Jens Axboe
Quoted 43 lines · li nux
Quoted 38 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You
need to align that as
Quoted 1 line
Quoted 1 line
well. This is needed because the hardware will
dma directly to the user
Quoted 1 line
Quoted 1 line
buffer, and to be on the safe side we require the
same alignment as the
Quoted 1 line
Quoted 1 line
block layer will normally generate for file
system io.
Quoted 2 lines
Quoted 2 lines
So in short, just align your read buffer to the
same as your block size
Quoted 5 lines
Quoted 5 lines
and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +MASK) & ~(MASK))
Quoted 7 lines
Quoted 5 lines
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);Shouldn't one use
posix_memalign(3)for that?Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).Signature · 1 line
Jens Axboe
Quoted 8 lines
Quoted 4 lines
Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?
2.6 also has the same restriction. But, if the "filesystem blocksize alignment" (soft block size) fails, we try to see if its aligned with hard sector size (512). If so, we can do the IO.
2.4 fails if the offset or buffer is NOT filesystem blocksize aligned. Period.
So, its possible that your buffer is atleast 512byte aligned, there by succeeding on 2.6
BTW, posix_memalign() or valloc() should be safe.
Quoted 8 lines
Quoted 3 lines
Can somebody please throw some light on how to find your system's hard/soft block size ?
It's a per-device (or even per-partition, in case of mounted partitions) setting, you can use the BLKBSZGET and BLKSSZGET ioctls to query for soft/hard sector sizes.
Thanks,
Badari
Quoted 54 lines · Badari Pulavarty
Quoted 44 lines · Jens Axboe
Quoted 43 lines · li nux
Quoted 38 lines · --- Jens Axboe
Quoted 4 lines · Grzegorz Kulewski
Quoted 3 lines · Jens Axboe
Quoted 1 line
how to correct this problem ?
See your buffer address, it's not aligned. You
need to align that as
Quoted 1 line
Quoted 1 line
well. This is needed because the hardware will
dma directly to the user
Quoted 1 line
Quoted 1 line
buffer, and to be on the safe side we require the
same alignment as the
Quoted 1 line
Quoted 1 line
block layer will normally generate for file
system io.
Quoted 2 lines
Quoted 2 lines
So in short, just align your read buffer to the
same as your block size
Quoted 5 lines
Quoted 5 lines
and you will be fine. Example:
#define BS (4096)
#define MASK (BS - 1)
#define ALIGN(buf) (((unsigned long) (buf) +MASK) & ~(MASK))
Quoted 7 lines
Quoted 5 lines
char *ptr = malloc(BS + MASK);
char *buf = (char *) ALIGN(ptr);
read(fd, buf, BS);Shouldn't one use
posix_memalign(3)for that?Dunno if one 'should', one 'can' if one wants to. I prefer to do it
manually so I don't have to jump through #define hoops to get at it
(which, btw, still doesn't expose it on this machine).Signature · 1 line
Jens AxboeQuoted 8 lines
Quoted 4 lines
Thanx a lot Jens :-)
Its working now.
I did not have to make these adjustments on 2.6
Is looks to be having more relaxation.2.6 does have the option of checking the hardware dma requirement seperately, but for this path you should run into the same restrictions. Perhaps you just got lucky when testing 2.6?
Quoted 6 lines
2.6 also has the same restriction. But, if the "filesystem blocksize alignment" (soft block size) fails, we try to see if its aligned with hard sector size (512). If so, we can do the IO.
2.4 fails if the offset or buffer is NOT filesystem blocksize aligned. Period.
I'm aware of that, however this particular case was about the buffer alignment (which was 32-bytes in the strace). And that should not work for 2.6 either.
The block-size alignment is really a separate property of correctness.
Quoted 1 line
BTW,
posix_memalign()orvalloc()should be safe.
Certainly.
Signature · 1 line
Jens Axboe