O_DIRECT reads appear to be cached on block device partition file?
Running a 2.6.31 kernel on a blade chassis system with multiple blades sharing common JBOD storage. The application intelligently divides the drives up among the blades, but one blade in particular is charged with monitoring. As part of this, this one monitoring blade can perform reads of a certain 512B sector of all disks in the system. This sector is often written by other blades, these writes are sync'd to disk. To work around the lack of cache coherency between the distinct blades, I'm using O_DIRECT on the monitoring blade such that it always reads from the media to get the latest copy of this sector. The basic steps are:
# grab a 512B aligned buffer (use 4KB to be safe) posix_memalign(&ptr, getpagesize(), 512B) open(/dev/sdX3, O_RDONLY|O_DIRECT) lseek(fd, offset, SEEK_SET) read(fd, ptr, 512B)
If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the /dev/sdX3 partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the /dev/sdX device file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for /dev/sdX3.
I've searched google for too long, coming up empty. I've tried changing the open flags to O_RDWR|O_DIRECT in case the kernel was doing something special b/c it believed there to be no writing to this fd. Perhaps there is an issue with reading only 512B rather than the 1KB Linux block size or 4KB page size?
Am I missing something about O_DIRECT, especially as it pertains to a device (and a partition on that device) file? I thought NO caching would be involved.
Any clues appreciated,
thanks,
Brett
Quoted 17 lines · Brett Russ
Running a 2.6.31 kernel on a blade chassis system with multiple blades sharing common JBOD storage. The application intelligently divides the drives up among the blades, but one blade in particular is charged with monitoring. As part of this, this one monitoring blade can perform reads of a certain 512B sector of all disks in the system. This sector is often written by other blades, these writes are sync'd to disk. To work around the lack of cache coherency between the distinct blades, I'm using
O_DIRECTon the monitoring blade such that it always reads from the media to get the latest copy of this sector. The basic steps are:# grab a 512B aligned buffer (use 4KB to be safe) posix_memalign(&ptr, getpagesize(), 512B) open(/dev/sdX3, O_RDONLY|O_DIRECT) lseek(fd, offset, SEEK_SET) read(fd, ptr, 512B)
Quoted 9 lines
If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the
/dev/sdX3partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the/dev/sdXdevice file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for/dev/sdX3.
What does blktrace tell you?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 11 lines · Dave Chinner
Quoted 9 lines · Brett Russ
If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the
/dev/sdX3partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the/dev/sdXdevice file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for/dev/sdX3.
Quoted 1 line
What does blktrace tell you?
Thanks Dave for the pointer to blktrace. I'd not used this before.
The short answer is that I now trust O_DIRECT. The cause for me going down this path to begin with was caused by a stale cache in our application.
The longer answer of how my dd double-check could have gone wrong follows:
I've discovered that the start-of-partition LBA does not *always* agree between the kernel (reported by blktrace and sysfs) and utilities such as {fdisk|sfdisk}. This means that my experiment of accessing the sector within the partition via the parent device may have been invalid, since I was trusting fdisk to determine the correct sector offset of the partition.
Quoted 1 line
spu0103# fdisk -l -u
/dev/sdbk
...
Quoted 3 lines
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
...
Quoted 3 lines
/dev/sdbk31197742140 1944780704 373519282+ 83 Linuxspu0103# sfdisk -uS -l
/dev/sdbk
...
Quoted 3 lines
Units = sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
...
Quoted 4 lines
/dev/sdbk31197742140 1944780704 747038565 83 Linuxspu0103# cat
/sys/block/sdbk/sdbk3/start
1197934920
The above discrepancy was also shown with blktrace:
Quoted 5 lines
spu0103# blkparse -q 1
Input file 1.blktrace.5 added
Input file 1.blktrace.6 added
Input file 1.blktrace.7 added
This command:
Quoted 1 line
spu0103# dd-7.1 if=/dev/sdbk3 bs=512 count=1 iflag=direct |hexdump -C
Produced this trace:
Quoted 1 line
67,224 5 1 0.000000000 29726 A R 1197934920 + 1 <- (67,227) 0
Note the kernel remapped the access to sdbk3 (offset 0) to sdbk (offset 1197934920) (see the major:minor numbers listed after the trace), which is quite different from the partition start shown in fdisk of 1197742140.
Quoted 7 lines
67,224 5 2 0.000000564 29726 Q R 1197934920 + 1 [dd-7.1] 67,224 5 3 0.000004032 29726 G R 1197934920 + 1 [dd-7.1] 67,224 5 4 0.000006223 29726 P N [dd-7.1] 67,224 5 5 0.000008152 29726 I R 1197934920 + 1 [dd-7.1] 67,224 5 6 0.000009916 29726 U N [dd-7.1] 1 67,224 5 7 0.000012286 29726 D R 1197934920 + 1 [dd-7.1] 67,224 7 1 0.006802504 0 C R 1197934920 + 1 [0]
And this command (accessing the start of partition using fdisk sector offset):
Quoted 1 line
spu0103# dd-7.1 if=/dev/sdbk skip=1197742140 bs=512 count=1 iflag=direct |hexdump -C
Produced this trace (as expected):
Quoted 7 lines
67,224 7 2 75.330506824 29924 Q R 1197742140 + 1 [dd-7.1] 67,224 7 3 75.330509804 29924 G R 1197742140 + 1 [dd-7.1] 67,224 7 4 75.330511985 29924 P N [dd-7.1] 67,224 7 5 75.330513836 29924 I R 1197742140 + 1 [dd-7.1] 67,224 7 6 75.330515495 29924 U N [dd-7.1] 1 67,224 7 7 75.330517901 29924 D R 1197742140 + 1 [dd-7.1] 67,224 6 1 75.340722638 0 C R 1197742140 + 1 [0]
The aforementioned major/minor numbers:
Quoted 3 lines
spu0103# ls -l /dev/|grep 67|grep '22[47]'
brw-rw-rw- 1 root root 67, 224 Sep 15 11:59 sdbk brw-rw-rw- 1 root root 67, 227 Sep 15 11:59 sdbk3
*All* other drives in my system that I tested do show a match between the 3 methods above (fdisk, sfdisk, sysfs).
I don't know how this discrepancy with the partition start could have been introduced, but it is most likely a byproduct of my testing.
Thanks,
Brett