interest in blk-mq, scsi-mq, dm-cache, dm-thinp, dm-*
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
Thanks,
Mike
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1
I've been thinking on (re-) implementing multipathing on top of blk-mq, and would like to discuss the probability of which. There are some design decisions in blk-mq (eg statically allocating the number of queues) which do not play well with that.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
Quoted 7 lines
+1
I've been thinking on (re-) implementing multipathing on top of blk-mq, and would like to discuss the probability of which. There are some design decisions in blk-mq (eg statically allocating the number of queues) which do not play well with that.
I think I have been thinking about going a completely different direction.
The thing about dm-multipath is that request based adds the extra queue locking and that of course is bad. In our testing it is a major perf issue. We got things like ioscheduling though.
If we went back to bio based multipathing then it turns out that when scsi also supports multiqueue then it all works pretty nicely. There is room for improvement in general like with some dm allocations being numa/cpu aware, but the request_queue locking issues we have go away and it is very simple code wise.
We could go the route of making request based dm-multipath:
1. aware of underlying multiqueue devices. So just basically keep what we have more or less but then have dm-multipath make a request that can be sent to a multiqueue device then call blk_mq_insert_request. This would all be hidden by nice interfaces that hide if it is multiqueue underlying device or not.
2. make dm-multipath do multiqueue (so implement map_queue, queue_rq, etc) and also making it aware of underlying multiqueue devices.
#1 just keeps the existing request spin_lock problem so there is not much point other than just getting things working.
#2 is a good deal of work and what does it end up buying us over just making multipath bio based. We lose iosched support. If we are going to make advanced multiqueue ioschedulers that rely on request structs then #2 could be useful.
Quoted 15 lines · Mike Christie
Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
Quoted 7 lines
+1
I've been thinking on (re-) implementing multipathing on top of blk-mq, and would like to discuss the probability of which. There are some design decisions in blk-mq (eg statically allocating the number of queues) which do not play well with that.
Quoted 6 lines
I think I have been thinking about going a completely different direction.
The thing about dm-multipath is that request based adds the extra queue locking and that of course is bad. In our testing it is a major perf issue. We got things like ioscheduling though.
Indeed. And without that we cannot do true load-balancing.
Quoted 6 lines
If we went back to bio based multipathing then it turns out that when scsi also supports multiqueue then it all works pretty nicely. There is room for improvement in general like with some dm allocations being numa/cpu aware, but the
request_queuelocking issues we have go away and it is very simple code wise.
If and when.
The main issue I see with that is that it might take some time (if ever) for SCSI LLDDs to go fully multiqueue.
In fact, I strongly suspect that only newer LLDDs will ever support multiqueue; for the older cards the HW interface it too much tied to single queue operations.
Quoted 11 lines
We could go the route of making request based dm-multipath:
1. aware of underlying multiqueue devices. So just basically keep what we have more or less but then have dm-multipath make a request that can be sent to a multiqueue device then call
blk_mq_insert_request. This would all be hidden by nice interfaces that hide if it is multiqueue underlying device or not.2. make dm-multipath do multiqueue (so implement
map_queue,queue_rq, etc) and also making it aware of underlying multiqueue devices.
Quoted 8 lines
#1 just keeps the existing request
spin_lockproblem so there is not much point other than just getting things working.#2 is a good deal of work and what does it end up buying us over just making multipath bio based. We lose iosched support. If we are going to make advanced multiqueue ioschedulers that rely on request structs then #2 could be useful.
Obviously we need iosched support when going multiqueue. I wouldn't dream of dropping them.
So my overall idea here is to move multipath over to block-mq, making each path identical to one queue.
(As mentioned above, currently every single FC HBA exposes a single HW queue anyway)
The ioschedulers would be moved to the map_queue function.
This approach has several issues which I would like to discuss:
- block-mq ctx allocation currently is static. This doesn't play well with multipathing, were paths (=queues) might get configured on-the-fly.
- Queues might be coming from different HBAs; one would need to audit the block-mq stuff if that's possible.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Mike Christie
Quoted 13 lines · Hannes Reinecke
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1
I've been thinking on (re-) implementing multipathing on top of blk-mq, and would like to discuss the probability of which. There are some design decisions in blk-mq (eg statically allocating the number of queues) which do not play well with that.
Quoted 7 lines
Quoted 6 lines
I think I have been thinking about going a completely different direction.
The thing about dm-multipath is that request based adds the extra queue locking and that of course is bad. In our testing it is a major perf issue. We got things like ioscheduling though.
Indeed. And without that we cannot do true load-balancing.
I think I might be using some terms differently. You do not need ioscheduling like deadline and cfq for path load balancing in multipath do you? Do you mean IO merging?
With noop load balancing works as expected.
Quoted 7 lines
Quoted 7 lines
If we went back to bio based multipathing then it turns out that when scsi also supports multiqueue then it all works pretty nicely. There is room for improvement in general like with some dm allocations being numa/cpu aware, but the
request_queuelocking issues we have go away and it is very simple code wise.
Quoted 7 lines
If and when.
The main issue I see with that is that it might take some time (if ever) for SCSI LLDDs to go fully multiqueue.
In fact, I strongly suspect that only newer LLDDs will ever support multiqueue; for the older cards the HW interface it too much tied to single queue operations.
Maybe we are talking about different things. We will do software/partial blk/scsi mq support soon and dm-multipath is not compatible with underlying devices that do multiqueue as it is now. I am just saying that dm cannot do its current bio/request cloning and then call blk_insert_cloned_request on a request when the underlying device does even the partial/software multiqueue that people are making patches for scsi for.
We can hack in support like I mentioned or we can do a bigger change like yours or we can do bio based or something else. The point is that something will need to be done and it is not dependent on hw multiqueue support.
However, on that subject of hw multiqueue support, Bart is experimenting with some tricks for SRP so it is not that far off. That is why he keeps asking about hw mutiqueue support for scsi.
Quoted 15 lines
Quoted 15 lines
We could go the route of making request based dm-multipath:
1. aware of underlying multiqueue devices. So just basically keep what we have more or less but then have dm-multipath make a request that can be sent to a multiqueue device then call
blk_mq_insert_request. This would all be hidden by nice interfaces that hide if it is multiqueue underlying device or not.2. make dm-multipath do multiqueue (so implement
map_queue,queue_rq, etc) and also making it aware of underlying multiqueue devices.#1 just keeps the existing request
spin_lockproblem so there is not much point other than just getting things working.
Quoted 8 lines
Quoted 5 lines
#2 is a good deal of work and what does it end up buying us over just making multipath bio based. We lose iosched support. If we are going to make advanced multiqueue ioschedulers that rely on request structs then #2 could be useful.
Obviously we need iosched support when going multiqueue. I wouldn't dream of dropping them.
I was just saying that blk-mq already dropped support for advanced ioscheds like deadline and cfq. It basically only does noop where there is a list/fifo.
Quoted 3 lines
So my overall idea here is to move multipath over to block-mq, making each path identical to one queue.
There are too many queues now :) Do you mean request_queue or blk_mq_hw_ctx or blk_mq_ctx?
Quoted 4 lines
(As mentioned above, currently every single FC HBA exposes a single HW queue anyway)
The ioschedulers would be moved to themap_queuefunction.
Are you maybe talking about path selectors here? map_queue would be too early for ioscheduling like deadline/cfq or you would need more callouts. Depending on how you are implementing blk multipath multiqueue I think map_queue could be good for path selection.
Quoted 1 line
This approach has several issues which I would like to discuss:
Hey, yeah, maybe best to discuss at LSF.
Quoted 4 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1 on the scsi-mq bit.
--nab
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1 for scsi-mq.
Quoted 8 lines · Sagi Grimberg
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1 for scsi-mq.
Sparing a few words on why I voted on this topic,
As part of me being involved in fast RDMA based initiators (iSER, SRP) I can say that SCSI layer became a real performance bottleneck. Implementations bypassing SCSI has started to pop-up lately. scsi-mq can and should boost performance and it is important that we work also on LLDs to provide a total solution.
Sagi.
Quoted 4 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1 for scsi-mq.
-- Giri
Quoted 7 lines
Thanks,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Giridhar Malavali
Quoted 4 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months following LSF.
+1 for scsi-mq.
(removed lsf-pc from CC-list)
Hello Giridhar,
I definitely appreciate QLogic's continuous efforts to improve performance of their initiator and target drivers. Via your e-mail I learned that QLogic is looking at scsi-mq, which is great news. However, the following issue might have to be addressed before the qla2xxx driver can fully take advantage of the scsi-mq work:
https://bugzilla.kernel.org/show_bug.cgi?id=69201. It would be appreciated if someone could have a look at the measurements described in that bugzilla entry.
Thanks,
Bart.
Quoted 14 lines · Bart Van Assche
Quoted 8 lines · Giridhar Malavali
Quoted 5 lines · Mike Snitzer
I would like to attend to participate in discussions related to topics listed in the subject. As a maintainer of DM I'd be interested to learn/discuss areas that should become a development focus in the months
following LSF.+1 for scsi-mq.
(removed lsf-pc from CC-list)
Hello Giridhar,
Quoted 8 lines
I definitely appreciate QLogic's continuous efforts to improve performance of their initiator and target drivers. Via your e-mail I learned that QLogic is looking at scsi-mq, which is great news. However, the following issue might have to be addressed before the qla2xxx driver can fully take advantage of the scsi-mq work:
https://bugzilla.kernel.org/show_bug.cgi?id=69201. It would be appreciated if someone could have a look at the measurements described in that bugzilla entry.
One of our internal runs have indicated lock contention issues and we are working on this actively.
We will be posting the patches once we are done with it.
Thanks for bringing this to our attention.
-- Giri
Quoted 4 lines
Thanks,
Bart.