Skip to contentSkip to conversation

O_DIRECT reads appear to be cached on block device partition file?

DISCUSSION Brett Russ 2 participants 3 messages Sep 14, 2010 – Sep 17, 2010

lore
Share & tools
lore

Brett Russ

author
#0 lore raw

Running a 2.6.31 kernel on a blade chassis system with multiple blades sharing common JBOD storage. The application intelligently divides the drives up among the blades, but one blade in particular is charged with monitoring. As part of this, this one monitoring blade can perform reads of a certain 512B sector of all disks in the system. This sector is often written by other blades, these writes are sync'd to disk. To work around the lack of cache coherency between the distinct blades, I'm using O_DIRECT on the monitoring blade such that it always reads from the media to get the latest copy of this sector. The basic steps are:

# grab a 512B aligned buffer (use 4KB to be safe)
posix_memalign(&ptr, getpagesize(), 512B)
open(/dev/sdX3, O_RDONLY|O_DIRECT)
lseek(fd, offset, SEEK_SET)
read(fd, ptr, 512B)

If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the /dev/sdX3 partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the /dev/sdX device file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for /dev/sdX3.

I've searched google for too long, coming up empty. I've tried changing the open flags to O_RDWR|O_DIRECT in case the kernel was doing something special b/c it believed there to be no writing to this fd. Perhaps there is an issue with reading only 512B rather than the 1KB Linux block size or 4KB page size?

Am I missing something about O_DIRECT, especially as it pertains to a device (and a partition on that device) file? I thought NO caching would be involved.

Any clues appreciated,
thanks,
Brett

#1 lore raw
Quoted 17 lines · Brett Russ

Running a 2.6.31 kernel on a blade chassis system with multiple blades sharing common JBOD storage. The application intelligently divides the drives up among the blades, but one blade in particular is charged with monitoring. As part of this, this one monitoring blade can perform reads of a certain 512B sector of all disks in the system. This sector is often written by other blades, these writes are sync'd to disk. To work around the lack of cache coherency between the distinct blades, I'm using O_DIRECT on the monitoring blade such that it always reads from the media to get the latest copy of this sector. The basic steps are:

# grab a 512B aligned buffer (use 4KB to be safe)
posix_memalign(&ptr, getpagesize(), 512B)
open(/dev/sdX3, O_RDONLY|O_DIRECT)
lseek(fd, offset, SEEK_SET)
read(fd, ptr, 512B)
Quoted 9 lines

If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the /dev/sdX3 partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the /dev/sdX device file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for /dev/sdX3.

What does blktrace tell you?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author
#2 lore raw
Quoted 11 lines · Dave Chinner
Quoted 9 lines · Brett Russ

If I run the above on the monitoring blade, then sync an update to the sector in question from another blade, then re-reun the above code on the monitoring blade, believe it or not I appear to be reading stale data. If I use dd with iflag=direct, reading the same sector offset at the /dev/sdX3 partition file, I see the same stale data as seen from the code above. If, however, I instead access this sector offset from the /dev/sdX device file using the (offset of partition 3 + offset of the sector) I see the intended data, which makes me believe some caching occurred locally for /dev/sdX3.

Quoted 1 line

What does blktrace tell you?

Thanks Dave for the pointer to blktrace. I'd not used this before.

The short answer is that I now trust O_DIRECT. The cause for me going down this path to begin with was caused by a stale cache in our application.

The longer answer of how my dd double-check could have gone wrong follows:

I've discovered that the start-of-partition LBA does not *always* agree between the kernel (reported by blktrace and sysfs) and utilities such as {fdisk|sfdisk}. This means that my experiment of accessing the sector within the partition via the parent device may have been invalid, since I was trusting fdisk to determine the correct sector offset of the partition.

Quoted 1 line

spu0103# fdisk -l -u /dev/sdbk

...

Quoted 3 lines

Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System

...

Quoted 3 lines

/dev/sdbk3 1197742140 1944780704 373519282+ 83 Linux

spu0103# sfdisk -uS -l /dev/sdbk

...

Quoted 3 lines

Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System

...

Quoted 4 lines

/dev/sdbk3 1197742140 1944780704 747038565 83 Linux

spu0103# cat /sys/block/sdbk/sdbk3/start
1197934920

The above discrepancy was also shown with blktrace:

Quoted 5 lines

spu0103# blkparse -q 1
Input file 1.blktrace.5 added
Input file 1.blktrace.6 added
Input file 1.blktrace.7 added

This command:

Quoted 1 line

spu0103# dd-7.1 if=/dev/sdbk3 bs=512 count=1 iflag=direct |hexdump -C

Produced this trace:

Quoted 1 line

67,224 5 1 0.000000000 29726 A R 1197934920 + 1 <- (67,227) 0

Note the kernel remapped the access to sdbk3 (offset 0) to sdbk (offset 1197934920) (see the major:minor numbers listed after the trace), which is quite different from the partition start shown in fdisk of 1197742140.

Quoted 7 lines
 67,224  5        2     0.000000564 29726  Q   R 1197934920 + 1 [dd-7.1]
 67,224  5        3     0.000004032 29726  G   R 1197934920 + 1 [dd-7.1]
 67,224  5        4     0.000006223 29726  P   N [dd-7.1]
 67,224  5        5     0.000008152 29726  I   R 1197934920 + 1 [dd-7.1]
 67,224  5        6     0.000009916 29726  U   N [dd-7.1] 1
 67,224  5        7     0.000012286 29726  D   R 1197934920 + 1 [dd-7.1]
 67,224  7        1     0.006802504     0  C   R 1197934920 + 1 [0]

And this command (accessing the start of partition using fdisk sector offset):

Quoted 1 line

spu0103# dd-7.1 if=/dev/sdbk skip=1197742140 bs=512 count=1 iflag=direct |hexdump -C

Produced this trace (as expected):

Quoted 7 lines
 67,224  7        2    75.330506824 29924  Q   R 1197742140 + 1 [dd-7.1]
 67,224  7        3    75.330509804 29924  G   R 1197742140 + 1 [dd-7.1]
 67,224  7        4    75.330511985 29924  P   N [dd-7.1]
 67,224  7        5    75.330513836 29924  I   R 1197742140 + 1 [dd-7.1]
 67,224  7        6    75.330515495 29924  U   N [dd-7.1] 1
 67,224  7        7    75.330517901 29924  D   R 1197742140 + 1 [dd-7.1]
 67,224  6        1    75.340722638     0  C   R 1197742140 + 1 [0]

The aforementioned major/minor numbers:

Quoted 3 lines

spu0103# ls -l /dev/|grep 67|grep '22[47]'

brw-rw-rw-    1 root     root      67, 224 Sep 15 11:59 sdbk
brw-rw-rw-    1 root     root      67, 227 Sep 15 11:59 sdbk3

*All* other drives in my system that I tested do show a match between the 3 methods above (fdisk, sfdisk, sysfs).

I don't know how this discrepancy with the partition start could have been introduced, but it is most likely a byproduct of my testing.

Thanks,
Brett