Skip to contentSkip to conversation

Question about make_request_fn based block drivers

DISCUSSION [email protected] 2 participants 4 messages Mar 20, 2013

lore
Share & tools
lore
#0 lore raw

When running mke2fs against the make_request_fn based block driver I'm working on, I'm seeing only single-block bios. Other such drivers (e.g. nvme) are getting, for example, 4k bios coming in from the same mke2fs command.

mke2fs /dev/sop0

This is on a 3.9-rc1 kernel.

I've tried setting:

	blk_queue_max_hw_sectors(rq, 2048);
	blk_queue_max_segments(rq, 32);
	blk_queue_io_opt(rq, 4096);
	blk_queue_io_min(rq, 4096);
	blk_queue_physical_block_size(rq, 4096);
	blk_queue_logical_block_size(h->rq, 512);
	blk_queue_physical_block_size(h->rq, 4096);

all to no avail.

If I do this:

dd if=/dev/sop0 of=/dev/null bs=4k iflag=direct

with that, I can get 4k bios coming in to the make_request_fn.

Driver source is here: https://github.com/HPSmartStorage/scsi-over-pcie
(still a work in progress -- that source doesn't do all the blk_queue_* settings mentioned above, those are just the things I've tried.)

In /sys/block/sop0/queue...

[scameron@localhost queue]$ for x in *

Quoted 4 lines

do
echo ===== $x ======
cat $x
done

===== add_random ======
1
===== discard_granularity ======
0
===== discard_max_bytes ======
0
===== discard_zeroes_data ======
0
===== hw_sector_size ======
512
===== iostats ======
1
===== logical_block_size ======
512
===== max_hw_sectors_kb ======
1024
===== max_integrity_segments ======
0
===== max_sectors_kb ======
512
===== max_segments ======
32
===== max_segment_size ======
65536
===== minimum_io_size ======
4096
===== nomerges ======
2
===== nr_requests ======
128
===== optimal_io_size ======
4096
===== physical_block_size ======
4096
===== read_ahead_kb ======
128
===== rotational ======
0
===== rq_affinity ======
1
===== scheduler ======
none
===== write_same_max_bytes ======
0
[scameron@localhost queue]$

Any ideas what I'm missing to get I/O's bigger than 1 block to come in?

Thanks,

-- steve

#1 lore raw

Got it! It is fixed now!!

Just one line fix, like last time. The drive capacity is 1 more than the value returned by READ_CAPACITY. Since the value we were setting was not a multiple of 4K, all the I/O were coming in the lowest accessible size of 512 bytes.

Sumant gave me the idea to explore this angle and see what capacity we are setting. Earlier, I created a dummy block driver (ramdrive - sbull) of 1024 blocks and found it always gets 4K IO. Changing the size to 1023 made us get 512 byte I/O.

I am going to push the changes to github.

Thanks,
Chayan

Quoted 94 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Wednesday, March 20, 2013 2:59 PM
To: [email protected]
Cc: [email protected]; [email protected]; Chayan Biswas; [email protected]
Subject: Question about make_request_fn based block drivers

When running mke2fs against the make_request_fn based block driver I'm working on, I'm seeing only single-block bios. Other such drivers (e.g. nvme) are getting, for example, 4k bios coming in from the same mke2fs command.

mke2fs /dev/sop0

This is on a 3.9-rc1 kernel.

I've tried setting:

      blk_queue_max_hw_sectors(rq, 2048);
      blk_queue_max_segments(rq, 32);
      blk_queue_io_opt(rq, 4096);
      blk_queue_io_min(rq, 4096);
      blk_queue_physical_block_size(rq, 4096);
      blk_queue_logical_block_size(h->rq, 512);
      blk_queue_physical_block_size(h->rq, 4096);

all to no avail.

If I do this:

dd if=/dev/sop0 of=/dev/null bs=4k iflag=direct

with that, I can get 4k bios coming in to the make_request_fn.

Driver source is here: https://github.com/HPSmartStorage/scsi-over-pcie
(still a work in progress -- that source doesn't do all the blk_queue_* settings mentioned above, those are just the things I've tried.)

In /sys/block/sop0/queue...

[scameron@localhost queue]$ for x in *

Quoted 4 lines

do
echo ===== $x ======
cat $x
done

===== add_random ======
1
===== discard_granularity ======
0
===== discard_max_bytes ======
0
===== discard_zeroes_data ======
0
===== hw_sector_size ======
512
===== iostats ======
1
===== logical_block_size ======
512
===== max_hw_sectors_kb ======
1024
===== max_integrity_segments ======
0
===== max_sectors_kb ======
512
===== max_segments ======
32
===== max_segment_size ======
65536
===== minimum_io_size ======
4096
===== nomerges ======
2
===== nr_requests ======
128
===== optimal_io_size ======
4096
===== physical_block_size ======
4096
===== read_ahead_kb ======
128
===== rotational ======
0
===== rq_affinity ======
1
===== scheduler ======
none
===== write_same_max_bytes ======
0
[scameron@localhost queue]$

Quoted 5 lines

Any ideas what I'm missing to get I/O's bigger than 1 block to come in?

Thanks,

-- steve

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

#2 lore raw
Quoted 7 lines · Chayan Biswas

Got it! It is fixed now!!

Just one line fix, like last time. The drive capacity is 1 more than the value returned by READ_CAPACITY. Since the value we were setting was not a multiple of 4K, all the I/O were coming in the lowest accessible size of 512 bytes.

Quoted 6 lines

Sumant gave me the idea to explore this angle and see what capacity we are setting. Earlier, I created a dummy block driver (ramdrive - sbull) of 1024 blocks and found it always gets 4K IO. Changing the size to 1023 made us get 512 byte I/O.

I am going to push the changes to github.

Great!

Didn't I ask you before if the nvme drive and the sop drive were the same size? ;-)

-- steve

Quoted 109 lines

Thanks,
Chayan

Quoted 94 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Wednesday, March 20, 2013 2:59 PM
To: [email protected]
Cc: [email protected]; [email protected]; Chayan Biswas; [email protected]
Subject: Question about make_request_fn based block drivers

When running mke2fs against the make_request_fn based block driver I'm working on, I'm seeing only single-block bios. Other such drivers (e.g. nvme) are getting, for example, 4k bios coming in from the same mke2fs command.

mke2fs /dev/sop0

This is on a 3.9-rc1 kernel.

I've tried setting:

      blk_queue_max_hw_sectors(rq, 2048);
      blk_queue_max_segments(rq, 32);
      blk_queue_io_opt(rq, 4096);
      blk_queue_io_min(rq, 4096);
      blk_queue_physical_block_size(rq, 4096);
      blk_queue_logical_block_size(h->rq, 512);
      blk_queue_physical_block_size(h->rq, 4096);

all to no avail.

If I do this:

dd if=/dev/sop0 of=/dev/null bs=4k iflag=direct

with that, I can get 4k bios coming in to the make_request_fn.

Driver source is here: https://github.com/HPSmartStorage/scsi-over-pcie
(still a work in progress -- that source doesn't do all the blk_queue_* settings mentioned above, those are just the things I've tried.)

In /sys/block/sop0/queue...

[scameron@localhost queue]$ for x in *

Quoted 4 lines

do
echo ===== $x ======
cat $x
done

===== add_random ======
1
===== discard_granularity ======
0
===== discard_max_bytes ======
0
===== discard_zeroes_data ======
0
===== hw_sector_size ======
512
===== iostats ======
1
===== logical_block_size ======
512
===== max_hw_sectors_kb ======
1024
===== max_integrity_segments ======
0
===== max_sectors_kb ======
512
===== max_segments ======
32
===== max_segment_size ======
65536
===== minimum_io_size ======
4096
===== nomerges ======
2
===== nr_requests ======
128
===== optimal_io_size ======
4096
===== physical_block_size ======
4096
===== read_ahead_kb ======
128
===== rotational ======
0
===== rq_affinity ======
1
===== scheduler ======
none
===== write_same_max_bytes ======
0
[scameron@localhost queue]$

Quoted 5 lines

Any ideas what I'm missing to get I/O's bigger than 1 block to come in?

Thanks,

-- steve

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

#3 lore raw

Hi Steve,

Quoted 6 lines
Quoted 1 line

I am going to push the changes to github.

Great!

Didn't I ask you before if the nvme drive and the sop drive were the same size? ;-)

[CB] Right! But the value returned by the FW was same, and the utilities round that off to 400GB in both cases. So to me it was the same size.... The devil is in the details :-(

Thanks,
Chayan

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).