Performance issue with port multiplier
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
I recently picked up a Thermaltake BlacX Duet dual drive esata dock and I am having some performance problems trying to use both drives concurrently. The link is running at 3 Gbps both to the PMP and from there to each drive. An individual drive pulls ~175 MB/s, but trying to use both drops to ~60 MB/s each. This is with a simple sequential dd to /dev/null with a 1 MiB block size. Oddly, disabling NCQ results in an increase to ~80 MB/s per drive. I wouldn't think that NCQ would really help for a sequential read, but it certainly shouldn't hurt.
Now it does seem that my intel ahci chipset does not support fis based switching, but it doesn't seem like that should make a whole lot of difference for sequential IO since while you are busy sending commands to one drive, the other should still be performing its own readahead into its cache and be ready to satisfy the request quickly once the first drive finishes.
How can this behavior make sense? Any ideas on other things I could try?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBCgAGBQJTWHoaAAoJEI5FoCIzSKrwgY4H/jIJw7EC8Oh6eXqNma67tVfn eNYgjKOKv1qsn4QDAva1F6TyUdmTzeKVoym4yxOgu51hPzcnPCoeOyOg0ZNNbOSa QSUxWHfgeGVHQvn5j30AZDqbfyIXd062mAjwDndFl4QSYRmAsZmijRavR33bZ8LP S81myeHoRSSIJNJDkRAXtiJckUxjU8HdDkLzvowGCcvgn6MNi6uMtaNepdCOGD5V 9u91uUB7HcsrfBCPutSW3Yv1GoSn4wEPL0/lIzqfP0q81qErsWjXkjqi8VEYXPys FcNT4lSGH0isjSCBtatDYMaWolLsOdkeZW4gwx07WyvsOP4H5fKFpVNrkFoAjoE= =X8IN
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 9 lines · Phillip Susi
I recently picked up a Thermaltake BlacX Duet dual drive esata dock and I am having some performance problems trying to use both drives concurrently. The link is running at 3 Gbps both to the PMP and from there to each drive. An individual drive pulls ~175 MB/s, but trying to use both drops to ~60 MB/s each. This is with a simple sequential dd to
/dev/nullwith a 1 MiB block size. Oddly, disabling NCQ results in an increase to ~80 MB/s per drive. I wouldn't think that NCQ would really help for a sequential read, but it certainly shouldn't hurt.
Quoted 8 lines
Now it does seem that my intel ahci chipset does not support fis based switching, but it doesn't seem like that should make a whole lot of difference for sequential IO since while you are busy sending commands to one drive, the other should still be performing its own readahead into its cache and be ready to satisfy the request quickly once the first drive finishes.
How can this behavior make sense? Any ideas on other things I could try?
So I would have thought that if you asked a disk drive with 64 mb of cache to read a few mb, then waited a second before issuing more commands, it would continue to read the next several mb into cache while waiting for new commands. I wrote a simple benchmark to test this by reading 16 256k blocks ( with O_DIRECT from a random starting offset to start with cold cache ), sleeping for 1 second, then reading the next 16, and the results seem to show that the drive stops reading ahead just after 1 MB. Do drives really have such terrible readahead algorithms in general? If so, this would seem to explain the terrible port multiplier performance in the absence of FBS. If this is really the norm, then perhaps libata needs to do a better job of balancing requests between the drives?
My test results:
read took 6598 us
read took 1157 us
read took 1207 us
read took 1178 us
read took 1187 us
read took 2373 us << odd delay
read took 1172 us
read took 1180 us
read took 1163 us
read took 1171 us
read took 1172 us
read took 1136 us
read took 2373 us
read took 1129 us
read took 1170 us
read took 1171 us
- ---
read took 636 us
read took 628 us
read took 625 us
read took 627 us
read took 626 us
read took 626 us
read took 580 us
read took 563 us
read took 6733 us << why is this even longer than the first seek read took 1237 us << cache is now empty, waiting on actual reads read took 2326 us << odd delay
read took 1165 us
read took 1169 us
read took 1171 us
read took 1173 us
read took 1169 us
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBCgAGBQJTcuWsAAoJEI5FoCIzSKrwMEEH/03Qne3kWo3UdW62OZVTcD0Z DgCtOSIx//feHDgXOIu46f8QO6BnvAFG+RyrplVjMvNTKio4M4cgCHJU7EOUaka0 tN2eturZFnzCo5ZQLgaQ8ZWWihbxVdT9BVjt0ixn+lUNjbhLfLf+qvagzOeLenOa RtMV2RbywK/G0DTEurNocQgJq/bVtoAtHGwv24WADL4ITd+5oRQHqBB5JupnUCD7 Nr8/uwLKkJZ7aVAX0zYAvFIUu3O0AucX/fPl8W63rm3wuB+E8BlLpHkMoF9hV8g0 qJiobIvUnbJ3y8luRlZjO5uH39osGfhFWH4r+cBDnI8eFoxdfcigyk3KEMeI48w= =ROo0
-----END PGP SIGNATURE-----
Re: understanding non fis based switching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 5 lines · Phillip Susi
So I would have thought that if you asked a disk drive with 64 mb of cache to read a few mb, then waited a second before issuing more commands, it would continue to read the next several mb into cache while waiting for new commands. I wrote a simple benchmark to test this by reading 16 256k blocks ( with
O_DIRECTfrom a
Quoted 8 lines
random starting offset to start with cold cache ), sleeping for 1 second, then reading the next 16, and the results seem to show that the drive stops reading ahead just after 1 MB. Do drives really have such terrible readahead algorithms in general? If so, this would seem to explain the terrible port multiplier performance in the absence of FBS. If this is really the norm, then perhaps libata needs to do a better job of balancing requests between the drives?
I have been trying to follow the code to see how it switches back and forth between the two drives, and what, if anything, is done to keep that switching smooth and fair. It looks like the libata layer just returns an error to the scsi layer telling it to try the request again later whenever the other disk has outstanding commands and holds the port exclusively. I can't find where in the scsi code it decides it is time to retry that command, or if it does anything to stop any more commands being issued to the first drive so it can free up the port so the second drive can get a crack at it.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTc8svAAoJEI5FoCIzSKrw9/EIAJSV6wvL/BFeOpuv7M8Q6dQz MExvhLe4dO/Nlc6bHr7I9bK6Pa1UL9vOmqpIbTDpmL2o8fQVri/0SVPrZ2fUw3BS rmkq7md/eM875/NQAdExnZ8EQYzpevAOTobUnJSRkdtNvLFSmqK1HhZAipHmI1Bu TW2Xy7pp7qj8qs/6Cc1Z+Ny60GY9yB6fVR6hlltL5M56wrBCT+q6vTeJatBvf8Ra +E3Sxa/7X8Tv8/MQJ0oE1q5sMA3Vwz3Mrg+8cqr064SGD0JhDn7WwCOkxj3kQBIO +K4wnQdkB3wUzQsm5CAnTztJTH2hrx6eR1RTac+9HncPqo0eMuXubupkhtb6KVU= =kQSu
-----END PGP SIGNATURE-----