SCSI Hardware Handler and slow failover with large number of LUNS
Subsystems (2)
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
Thanks Chandra for bringing this up.
Yes this will be a major drawback for the users who use rdac handler for their storage with large configurations.
Thanks
Babu Moger
-----Original Message-----
From: Chandra Seetharaman [mailto:[email protected]]
Sent: Saturday, April 04, 2009 4:31 PM
To: dm-devel; Linux SCSI Mailing list
Cc: Moger, Babu
Subject: SCSI Hardware Handler and slow failover with large number of LUNS
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
Hi Chandra.
We're currently testing 1024 LUNs across four paths with the
older handler but will be doing the same with the new handler later this month. It sounds like I should move this up in the queue.
Regards,
Wayne.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chandra Seetharaman
Sent: Saturday, April 04, 2009 5:31 PM
To: dm-devel; Linux SCSI Mailing list
Cc: Moger, Babu
Subject: SCSI Hardware Handler and slow failover with large number of LUNS
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
Quoted 4 lines · On Mon, 2009-04-06 at 11:38 -0400, [email protected] wrote:
Hi Chandra.
We're currently testing 1024 LUNs across four paths with the
older handler but will be doing the same with the new handler later this
Hi Wayne,
What do you mean by "older" handler ? dm_emc ?
Quoted 47 lines
month. It sounds like I should move this up in the queue.
Regards,
Wayne.-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chandra Seetharaman
Sent: Saturday, April 04, 2009 5:31 PM
To: dm-devel; Linux SCSI Mailing list
Cc: Moger, Babu
Subject: SCSI Hardware Handler and slow failover with large number of LUNSHello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: SCSI Hardware Handler and slow failover with large number ofLUNS
Hi Chandra.
Yes, it is the dm_emc.
Regards,
Wayne.
-----Original Message-----
From: Chandra Seetharaman [mailto:[email protected]]
Sent: Monday, April 06, 2009 1:47 PM
To: berthiaume, wayne
Cc: [email protected]; [email protected]; [email protected]
Subject: RE: SCSI Hardware Handler and slow failover with large number ofLUNS
Quoted 4 lines · On Mon, 2009-04-06 at 11:38 -0400, [email protected] wrote:
Hi Chandra.
We're currently testing 1024 LUNs across four paths with the
older handler but will be doing the same with the new handler later
this
Hi Wayne,
What do you mean by "older" handler ? dm_emc ?
Quoted 15 lines
month. It sounds like I should move this up in the queue.
Regards,
Wayne.-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Chandra Seetharaman
Sent: Saturday, April 04, 2009 5:31 PM
To: dm-devel; Linux SCSI Mailing list
Cc: Moger, Babu
Subject: SCSI Hardware Handler and slow failover with large number of LUNS
Quoted 7 lines
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize
activate
Quoted 12 lines
in dm on the work queue.
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
Quoted 6 lines
regards,
chandra
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
in
Quoted 5 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Chandra Seetharaman
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
I thought we talked about this during the review?
Quoted 3 lines
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.
I think most of the ugliness in the original async mode was due to trying to use the REQ_BLOCK* path. With the scsi_dh_activate path, it should now be easier because in the send path we do not have to worry about queue locks being held and context.
I think we could just use blk_execute_rq_nowait to send the IO. Then we would have a workqueue/thread per something (maybe per dh module I thought), that would be queued/notified when the IO completed. The thread could then process the IO and handle the next stage if needed.
Why use the thread you might wonder? I think it fixes another issue with the original async mode, and makes it easier if the scsi_dh module has to send more IO. When using the thread it would not have to worry about the queue_lock being held in the IO completion path and does not have to worry about being run from more restrictive contexts.
Quoted 3 lines
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
They should all have the same problem.
Quoted 12 lines
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Thanks for the response Mike.
Quoted 7 lines · Mike Christie
Quoted 6 lines · Chandra Seetharaman
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
Quoted 5 lines
Quoted 3 lines
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
I thought we talked about this during the review?
Yes, we did and the results were compared to the virgin code (w.r.t rdac handler) and the results were good (also I used only 49 luns) :
http://marc.info/?l=dm-devel&m=120889858019762&w=2
Quoted 10 lines
Quoted 4 lines
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.I think most of the ugliness in the original async mode was due to trying to use the REQ_BLOCK* path. With the
scsi_dh_activatepath, it should now be easier because in the send path we do not have to worry about queue locks being held and context.
little confused... we still are using REQ_TYPE_BLOCK_PC
Quoted 7 lines
I think we could just use
blk_execute_rq_nowaitto send the IO. Then we would have a workqueue/thread per something (maybe per dh module I thought), that would be queued/notified when the IO completed. The thread could then process the IO and handle the next stage if needed.Why use the thread you might wonder? I think it fixes another issue with the original async mode, and makes it easier if the
scsi_dhmodule has
can you elaborate the issue ?
Quoted 3 lines
to send more IO. When using the thread it would not have to worry about the
queue_lockbeing held in the IO completion path and does not have to worry about being run from more restrictive contexts.
You think queue_lock contention is an issue ?
I agree with the restrictive context issue though.
So, your suggestion is to move everything to async ?
Quoted 22 lines
Quoted 5 lines
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
They should all have the same problem.
Quoted 12 lines
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Quoted 20 lines · Chandra Seetharaman
Thanks for the response Mike.
Quoted 11 lines · Mike Christie
Quoted 9 lines · Chandra Seetharaman
Hello All,
During testing with the latest SCSI DH Handler on a rdac storage, Babu found that the failover time with 100+ luns takes about 15 minutes, which is not good.
We found that the problem is due to the fact that we serialize activate in dm on the work queue.
I thought we talked about this during the review?
Yes, we did and the results were compared to the virgin code (w.r.t rdac handler) and the results were good (also I used only 49 luns) :
http://marc.info/?l=dm-devel&m=120889858019762&w=2
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.I think most of the ugliness in the original async mode was due to trying to use the REQ_BLOCK* path. With the
scsi_dh_activatepath, it should now be easier because in the send path we do not have to worry about queue locks being held and context.little confused... we still are using
REQ_TYPE_BLOCK_PC
But we only have one level of requests. I am talking about when we tried to send a request with REQ_BLOCK_LINUX_BLOCK to the module to tell it to send another request/s with REQ_TYPE_BLOCK_PC. Now we just have the callout and then like you said we can fire REQ_TYPE_BLOCK_PC reuqests from there.
I think when I wrote easier above, I meant to write a cleaner implementation.
Quoted 5 lines
Quoted 3 lines
I think we could just use
blk_execute_rq_nowaitto send the IO. Then we would have a workqueue/thread per something (maybe per dh module I thought), that would be queued/notified when the IO completed. The thread could then process the IO and handle the next stage if needed.Why use the thread you might wonder? I think it fixes another issue with the original async mode, and makes it easier if the
scsi_dhmodule hascan you elaborate the issue ?
I think people did not like the complexity of trying to send IO with soft irq context with spin locks held, then also having the extra REQ_BLOCK_LINUX_BLOCK layering.
Quoted 9 lines
Quoted 2 lines
to send more IO. When using the thread it would not have to worry about the
queue_lockbeing held in the IO completion path and does not have to worry about being run from more restrictive contexts.You think
queue_lockcontention is an issue ?I agree with the restrictive context issue though.
So, your suggestion is to move everything to async ?
Do mean vs #1 or would you want to seperate and send some stuff async and synchronously?
Quoted 22 lines
Quoted 17 lines
Quoted 3 lines
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
They should all have the same problem.
Quoted 11 lines
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Mike Christie
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
We can solve the problem in rdac handler in 2 ways
1. batch up the activates (mode_selects) and send few of them.
2. Do mode selects in async mode.Quoted 8 lines
Quoted 5 lines
I think most of the ugliness in the original async mode was due to trying to use the REQ_BLOCK* path. With the
scsi_dh_activatepath, it should now be easier because in the send path we do not have to worry about queue locks being held and context.little confused... we still are using
REQ_TYPE_BLOCK_PC
Quoted 8 lines
But we only have one level of requests. I am talking about when we tried to send a request with
REQ_BLOCK_LINUX_BLOCKto the module to tell it to send another request/s withREQ_TYPE_BLOCK_PC. Now we just have the callout and then like you said we can fireREQ_TYPE_BLOCK_PCreuqests from there.I think when I wrote easier above, I meant to write a cleaner implementation.
Now I understand.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
I think we could just use
blk_execute_rq_nowaitto send the IO. Then we would have a workqueue/thread per something (maybe per dh module I thought), that would be queued/notified when the IO completed. The thread could then process the IO and handle the next stage if needed.Why use the thread you might wonder? I think it fixes another issue with the original async mode, and makes it easier if the
scsi_dhmodule has
Quoted 6 lines
Quoted 1 line
can you elaborate the issue ?
I think people did not like the complexity of trying to send IO with soft irq context with spin locks held, then also having the extra
REQ_BLOCK_LINUX_BLOCKlayering.
clear now :)
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
to send more IO. When using the thread it would not have to worry about the
queue_lockbeing held in the IO completion path and does not have to worry about being run from more restrictive contexts.You think
queue_lockcontention is an issue ?
Quoted 7 lines
Quoted 4 lines
I agree with the restrictive context issue though.
So, your suggestion is to move everything to async ?
Do mean vs #1 or would you want to seperate and send some stuff async and synchronously?
The other option I was thinking about was to utilize the capability of the underlying device. For example, for lsi rdac, we can batch up mode selects and send fewer commands down the wire, which would speed things up.
I understand that not all devices would have a feature like rdac. Nevertheless, it doesn't matter as this will be _inside_ the hardware handler and they _don't_ have to be the same solution.
If async is the best option for all other handler, we can as well do async for rdac also, which will keep the handlers looking the same :)
Quoted 28 lines
Quoted 19 lines
Quoted 18 lines
Quoted 4 lines
Just wondering if anybody had seen the same problem in other storages (EMC, HP and Alua).
They should all have the same problem.
Quoted 11 lines
Please share your experiences, so we can come up with a solution that works for all hardware handlers.
regards,
chandra
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-develQuoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html