Reason for md raid1 max_sectors_kb limited to 127?

Hi list,

I'm wondering why max_sectors_kb is set to 127 with md raid1. This value is at 512 on normal HDDs.

If I do a file copy on ext4 on a normal HDD then I can see with blktrace/blkparse that 128 KiB chunks are read and 512 KiB chunks are written.

With md raid it looks like this: 124 KiB, 4 KiB, 124 KiB, 4 KiB, ... . This looks very inefficient.

So, is there a particular reason that max_sectors_kb is set to 127?

Cheers,
Sebastian

You didn't say which kernel you are running.

However md/raid1 bases all those settings on the minimum or maximum (as appropriate) of the setting of the underlying devices, using blk_stack_limits (in block/blk-settings.c).

So the likely answer is that one of your HDDs has a smaller max_sectors_kb?

NeilBrown

Quoted 9 lines · NeilBrown

You didn't say which kernel you are running.

However md/raid1 bases all those settings on the minimum or maximum (as appropriate) of the setting of the underlying devices, using blk_stack_limits (in block/blk-settings.c).

So the likely answer is that one of your HDDs has a smaller max_sectors_kb?

NeilBrown

Thanks for your answer! Kernel version is vanilla 3.2, but I've also tested 2.6.32. There is no difference. Distribution: Debian Squeeze.

I can even reproduce this behaviour with RAM disks:

# modprobe brd rd_nr=2 rd_size=1048576
# cat /sys/block/ram0/queue/max_sectors_kb
512
# cat /sys/block/ram1/queue/max_sectors_kb
512
# mdadm -C /dev/md200 --force --assume-clean -n 2 -l raid1 -a md
/dev/ram0 /dev/ram1
# cat /sys/block/md200/queue/max_sectors_kb
127

I'll have a look at that blk_stack_limits() function.

I think you need something like this. I thought there already went something in in recent kernel, will check it later on today.

Index: 2.6.32.13/drivers/md/raid0.c
===================================================================

drivers/md/raid0.c +10 -2

@@ -96,6 +96,7 @@ static int create_strip_zones(mddev_t *m
 	int cnt;
 	char b[BDEVNAME_SIZE];
 	raid0_conf_t *conf = kzalloc(sizeof(*conf), GFP_KERNEL);
+	unsigned int opt_io_size;
 
 	if (!conf)
 		return -ENOMEM;

@@ -256,9 +257,16 @@ static int create_strip_zones(mddev_t *m
 		goto abort;
 	}
 
+	/*
+	 * To send large IOs to the drives we need sufficient segments
+	 * for our own queue first.
+	 */
+	opt_io_size = (mddev->chunk_sectors << 9) * mddev->raid_disks;
+	blk_queue_max_phys_segments(mddev->queue, opt_io_size >> PAGE_SHIFT);
+	blk_queue_max_hw_segments(mddev->queue, opt_io_size >> PAGE_SHIFT);
+
 	blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
-	blk_queue_io_opt(mddev->queue,
-			 (mddev->chunk_sectors << 9) * mddev->raid_disks);
+	blk_queue_io_opt(mddev->queue, opt_io_size);
 
 	printk(KERN_INFO "raid0: done.\n");
 	mddev->private = conf;

Cheers,
Bernd

could check if it have any performace increase? maybe i consider upgrading my kernel to get more performace inn raid1

Quoted 3 lines · Roberto Spadim

could check if it have any performace increase? maybe i consider upgrading my kernel to get more performace inn raid1

Depends on your hardware. Hardware that can handle small IO sizes, such as common hard disks usually don't have problems with 512KB IOs. But if you should use software on top of hardware raid large IOs might be very important (again depends on the hw-raid vendor then).

Cheers,
Bernd

Quoted 5 lines · Bernd Schubert

Quoted 3 lines · Roberto Spadim

could check if it have any performace increase? maybe i consider upgrading my kernel to get more performace inn raid1

Quoted 8 lines

Depends on your hardware. Hardware that can handle small IO sizes, such as common hard disks usually don't have problems with 512KB IOs. But if you should use software on top of hardware raid large IOs might be very important (again depends on the hw-raid vendor then).

Cheers,
Bernd

O.K., I've also tested 3.2.16 and there the problem still exists. Bernd pinpointed me to commit b1bd055d397e09f99dcef9b138ed104ff1812fcb (block: Introduce blk_set_stacking_limits function).

After cherry-picking it on 3.2.16 it worked. Tomorrow I'll test the performance impact and verify it by block tracing.