Unaligned AIO/DIO
Hello,
Running fio 1.55 on RHEL 6.1 x86_64 with a multipath FC storage backend on an EXT4 file system.
Some minutes after starting fio, I see the following in syslog:
kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio; performance will be poor.
Should I be worried? Is there something wrong with my storage layout? Or am I using fio in strange way?
My job file:
[changed-from-iometer-file-access-server] bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256write_bw_logwrite_lat_log
numjobs=6
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/
Quoted 12 lines · Troels Arvin
Hello,
Running fio 1.55 on RHEL 6.1
x86_64with a multipath FC storage backend on an EXT4 file system.Some minutes after starting fio, I see the following in syslog:
kernel: EXT4-fs (dm-28): Unaligned AIO/DIO on inode 4063238 by fio; performance will be poor.Should I be worried? Is there something wrong with my storage layout? Or am I using fio in strange way?
My job file:
Quoted 11 lines
[changed-from-iometer-file-access-server] bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256write_bw_logwrite_lat_log
numjobs=6
Do you have the RHEL6.1 kernel sources handy? That message is not in the upstream kernel. Unaligned could be memory, or it could be offset/size. In general O_DIRECT does not work for unaligned IO. Perhaps it's referring to writes that aren't stripe aligned, but that seems fairly expected and something dm should handle.
So to render an opinion on this, I'd need to see a copy of the file that prints that message :-)
Signature · 1 line
Jens Axboe
Hello,
Quoted 1 line · Jens Axboe
Do you have the RHEL6.1 kernel sources handy?
I just grabbed the particular kernel package'es source-package, installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file which should result in all RHs patches to be applied.
From the resulting code in ~/rpmbuild/BUILD/..., I grabbed file.c and placed it here:
http://troels.arvin.dk/misc/fio/unaligned/
Lines close to the error message (probably wrapped by my mailer):
/* Unaligned direct AIO must be serialized; see comment above */
if (unaligned_aio) {
static unsigned long unaligned_warn_time;
/* Warn about this once per day */
if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
ext4_msg(inode->i_sb, KERN_WARNING,
"Unaligned AIO/DIO on inode %ld by %s; "
"performance will be poor.",
inode->i_ino, current->comm);
mutex_lock(&EXT4_I(inode)->i_aio_mutex);
ext4_aiodio_wait(inode);}
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/
Quoted 12 lines · Troels Arvin
Hello,
Quoted 1 line · Jens Axboe
Do you have the RHEL6.1 kernel sources handy?
I just grabbed the particular kernel package'es source-package, installed it into ~/rpmbuild and ran "rpmbuild -bp" on its spec-file which should result in all RHs patches to be applied.
From the resulting code in ~/rpmbuild/BUILD/..., I grabbed
file.cand placed it here:
http://troels.arvin.dk/misc/fio/unaligned/Lines close to the error message (probably wrapped by my mailer):
Quoted 14 lines
/* Unaligned direct AIO must be serialized; see comment above */
if (unaligned_aio) {
static unsigned long unaligned_warn_time;
/* Warn about this once per day */
if (printk_timed_ratelimit(&unaligned_warn_time, 60*60*24*HZ))
ext4_msg(inode->i_sb, KERN_WARNING,
"Unaligned AIO/DIO on inode %ld by %s; "
"performance will be poor.",
inode->i_ino, current->comm);
mutex_lock(&EXT4_I(inode)->i_aio_mutex);
ext4_aiodio_wait(inode);
}Ah, so this is indeed the < 4KB O_DIRECT writes. The iometer like test case you are using is meant to be used on a raw block device, not on a file system. For a file system, doing less than the block size of IO will incur RMW cycles and thus be very slow compared to full block size (or more) O_DIRECT writes.
If you want to run it on the fs, either change the block size mappings to avoid anything less than 4K. Or change the fs to be formatted with 512b blocks.
Or just run the test on the raw device :-)
Signature · 1 line
Jens Axboe
Hello,
Quoted 3 lines · Jens Axboe
The iometer like test
case you are using is meant to be used on a raw block device, not on a file system.
OK, but how does one use fio with a raw device? Can you give an example of a job specification file for working with a raw block device?
And can that be done in a way which simulates parallel I/O (several processes/threads working simultaneously)?
Signature · 3 lines
Regards, Troels Arvin <[email protected]> http://troels.arvin.dk/
Quoted 8 lines · Troels Arvin
Hello,
Quoted 3 lines · Jens Axboe
The iometer like test
case you are using is meant to be used on a raw block device, not on a file system.OK, but how does one use fio with a raw device? Can you give an example of a job specification file for working with a raw block device?
To take your example:
[global] bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 rw=randrw
rwmixread=70
direct=1
size=10g
ioengine=libaio
iodepth=256write_bw_logwrite_lat_log
numjobs=6
[sda]
filename=/dev/sda
would run the workload in question directly on the raw sda device.
Quoted 2 lines
And can that be done in a way which simulates parallel I/O (several processes/threads working simultaneously)?
For the above example, you have 6 processes doing random read/write patterns with the 70/30 read/write mix and with the block sizes given.
Signature · 1 line
Jens Axboe