iSCSI MQ adoption via MCS discussion

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Quoted 62 lines · Mike Christie

Quoted 43 lines · Hannes Reinecke

Quoted 38 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is
implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the
scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S.

RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER...

(we might want to leave something for LSF ;)).

Quoted 3 lines · Sagi Grimberg

Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER…

Hello Sagi,

Which command ordering requirement are you referring to ? The Linux storage stack does not guarantee that block layer or SCSI commands will be processed in the same order as these commands have been submitted.

Quoted 3 lines · Sagi Grimberg

Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER...

Hello Sagi,

Which command ordering requirement are you referring to ? The Linux storage stack does not guarantee that block layer or SCSI commands will be processed in the same order as these commands have been submitted.

Quoted 8 lines · Bart Van Assche

Quoted 3 lines · Sagi Grimberg

Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER...

Which command ordering requirement are you referring to ? The Linux storage stack does not guarantee that block layer or SCSI commands will be processed in the same order as these commands have been submitted.

I was referring to the iSCSI session requirement. I initially thought of an approach to maintain multiple iSER connections under a single session but pretty soon I realized that preserving commands ordering this way is not feasible. So independent iSER connections means independent iSCSI sessions (each with a single connection). This is indeed another choice, which we are clearly debating on...

I'm just wandering if we are not trying to force-fit this model. How would this model look like? We will need to define another entity to track and maintain the sessions and to allocate the scsi_host. Will that be communicated to user-space? How will error recovery look like?

Quoted 17 lines · Sagi Grimberg

Quoted 8 lines · Bart Van Assche

Quoted 3 lines · Sagi Grimberg

Actually I started with that approach, but the independent connections under a single session (I-T-Nexus) violates the command ordering requirement. Plus, such a solution is specific to iSER...

Which command ordering requirement are you referring to ? The Linux storage stack does not guarantee that block layer or SCSI commands will be processed in the same order as these commands have been submitted.

I was referring to the iSCSI session requirement. I initially thought of an approach to maintain multiple iSER connections under a single session but pretty soon I realized that preserving commands ordering this way is not feasible. So independent iSER connections means independent iSCSI sessions (each with a single connection). This is indeed another choice, which we are clearly debating on...

Quoted 4 lines

I'm just wandering if we are not trying to force-fit this model. How would this model look like? We will need to define another entity to track and maintain the sessions and to allocate the scsi_host. Will that be communicated to user-space? How will error recovery look like?

Hello Sagi,

As you probably remember scsi-mq support was added in the SRP initiator by changing the 1:1 relationship between scsi_host and RDMA connection into a 1:n relationship.

I don't know how much work it would take to
implement a similar transformation in the SCSI initiator. Maybe we should wait until Mike's workday starts such that Mike has a chance to comment on this.

Quoted 7 lines

RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

So this comment is spot on the pros/cons of the discussion (we might want to leave something for LSF ;)).
MCS would not allow a completely lockless data-path due to command ordering. On the other hand implementing some kind of multiple sessions solution feels somewhat like a mis-fit (at least in my view).

One of my thoughts about how to overcome the contention on commands sequence numbering was to suggest some kind of negotiable "relaxed ordering" mode but of course I don't have anything figured out yet.

Quoted 60 lines · Mike Christie

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

Quoted 75 lines · Bart Van Assche

Quoted 60 lines · Mike Christie

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

Quoted 10 lines

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

The idea that a simple session wide counter for command sequence number assignment adds such a degree of contention that it renders MC/S at a performance disadvantage vs. multi-session configurations with all of the extra multipath logic overhead on top is at best, a naive proposition.

On the initiator side for MC/S, literally the only thing that needs to be serialized is the assignment of the command sequence number to individual non-immediate PDUs. The sending of the outgoing PDUs + immediate data by the initiator can happen out-of-order, and it's up to the target to ensure that the submission of the commands to the device server is in command sequence number order.

All of the actual immediate data + R2T -> data-out processing by the target can also be done out-of-order as well.

Quoted 100 lines · Nicholas A. Bellinger

Quoted 75 lines · Bart Van Assche

Quoted 60 lines · Mike Christie

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

Quoted 10 lines

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

The idea that a simple session wide counter for command sequence number assignment adds such a degree of contention that it renders MC/S at a performance disadvantage vs. multi-session configurations with all of the extra multipath logic overhead on top is at best, a naive proposition.

On the initiator side for MC/S, literally the only thing that needs to be serialized is the assignment of the command sequence number to individual non-immediate PDUs. The sending of the outgoing PDUs + immediate data by the initiator can happen out-of-order, and it's up to the target to ensure that the submission of the commands to the device server is in command sequence number order.

Quoted 2 lines

All of the actual immediate data + R2T -> data-out processing by the target can also be done out-of-order as well.

Right, but what he's saying is that we've taken great pains in the MQ situation to free our issue queues of all entanglements and cross queue locking so they can fly as fast as possible. If we have to assign an in-order sequence number across all the queues, this becomes both a cross CPU bus lock point to ensure atomicity and a sync point to ensure sequencing. Naïvely that does look to be a bottleneck which wouldn't necessarily be mitigated simply by allowing everything to proceed out of order after this point.

Quoted 104 lines · James Bottomley

Quoted 100 lines · Nicholas A. Bellinger

Quoted 75 lines · Bart Van Assche

Quoted 60 lines · Mike Christie

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

Quoted 10 lines

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

The idea that a simple session wide counter for command sequence number assignment adds such a degree of contention that it renders MC/S at a performance disadvantage vs. multi-session configurations with all of the extra multipath logic overhead on top is at best, a naive proposition.

On the initiator side for MC/S, literally the only thing that needs to be serialized is the assignment of the command sequence number to individual non-immediate PDUs. The sending of the outgoing PDUs + immediate data by the initiator can happen out-of-order, and it's up to the target to ensure that the submission of the commands to the device server is in command sequence number order.

Quoted 2 lines

All of the actual immediate data + R2T -> data-out processing by the target can also be done out-of-order as well.

Quoted 9 lines

Right, but what he's saying is that we've taken great pains in the MQ situation to free our issue queues of all entanglements and cross queue locking so they can fly as fast as possible. If we have to assign an in-order sequence number across all the queues, this becomes both a cross CPU bus lock point to ensure atomicity and a sync point to ensure sequencing. Naïvely that does look to be a bottleneck which wouldn't necessarily be mitigated simply by allowing everything to proceed out of order after this point.

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 25 lines

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

Quoted 3 lines

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 20 lines

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 8 lines

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

Quoted 45 lines · Nicholas A. Bellinger

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 20 lines

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 8 lines

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Quoted 9 lines

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Wouldn't that need to be coordinated with the networking layer? Doesn't it do the same thing, matching TX/RX queues to CPUs? If so, wouldn't we decrease bandwidth by restricting things to one CPU?

Quoted 56 lines · Michael Christie

Quoted 45 lines · Nicholas A. Bellinger

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 20 lines

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 8 lines

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

Quoted 7 lines

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Wouldn't that need to be coordinated with the networking layer?

Doesn't it do the same thing, matching TX/RX queues to CPUs?

If so, wouldn't we decrease bandwidth by restricting things to one CPU?

Quoted 64 lines · Hannes Reinecke

Quoted 63 lines · Michael Christie

Quoted 45 lines · Nicholas A. Bellinger

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 20 lines

Quoted 4 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

I don't see how that's relevant to issue speed, which was the measure we were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the multipath hopper is efficient enough to keep the queues loaded there's no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it can cause a stall that reduces the issue speed. so the counter sequence point causes a degraded issue speed over the multipath hopper approach above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but it's only in the flat out CPU case.

Quoted 8 lines

Quoted 4 lines

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Quoted 1 line

Wouldn't that need to be coordinated with the networking layer?

Yes.

Quoted 1 line

Doesn't it do the same thing, matching TX/RX queues to CPUs?

Yes.

Quoted 1 line

If so, wouldn't we decrease bandwidth by restricting things to one CPU?

We have a session or connection per CPU though, so we end up hitting the same problem you talked about last year where one hctx (iscsi session or connection's socket or nic hw queue) could get overloaded. This is what I meant in my original mail where iscsi would rely on whatever blk/mq load balancers we end up implementing at that layer to balance requests across hctxs.

I would say there is no need for specific coordination from iSCSI PoV. This is exactly what flow steering is designed for. As I see it, in order to get the TX/RX to match rings, the user can attach 5-tuple rules (using standard ethtool) to steer packets to the right rings.

Quoted 4 lines · Sagi Grimberg

I would say there is no need for specific coordination from iSCSI PoV. This is exactly what flow steering is designed for. As I see it, in order to get the TX/RX to match rings, the user can attach 5-tuple rules (using standard ethtool) to steer packets to the right rings.

Hello Sagi,

Can the 5-tuple rules be chosen such that it is guaranteed that the sockets used to implement per-CPU queues are spread evenly over MSI-X completion vectors ? If not, would it help to add a socket option to the Linux network stack that allows to select the TX ring explicitly, just like ib_create_cq() in the Linux RDMA stack allows to select a completion vector explicitly ? My concerns are as follows:
- If the number of queues exceeds the number of MSI-X vectors then I expect that it will be much easier to guarantee even spreading by selecting tx queues explicitly instead of relying on a hashing scheme.
- On multi-socket systems it is important to process completion interrupts on the CPU socket from where the I/O was initiated. I'm not sure it is possible to guarantee this when using a hashing algorithm to select the TX ring.

Quoted 81 lines · Mike Christie

Quoted 80 lines · Hannes Reinecke

Quoted 79 lines · Michael Christie

Quoted 59 lines · Nicholas A. Bellinger

Quoted 3 lines · James Bottomley

Quoted 2 lines · Nicholas A. Bellinger

Quoted 1 line · James Bottomley

On Thu, 2015-01-08 at 14:16 -0800, Nicholas A. Bellinger wrote:

<SNIP>

Quoted 36 lines

Quoted 5 lines

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple
sessions.

I don't see how that's relevant to issue speed, which was the measure we
were using: The layers above are just a hopper. As long as they're loaded, the MQ lower layer can issue at full speed. So as long as the
multipath hopper is efficient enough to keep the queues loaded there's
no speed degradation.

The problem with a sequence point inside the MQ issue layer is that it
can cause a stall that reduces the issue speed. so the counter sequence
point causes a degraded issue speed over the multipath hopper approach
above even if the multipath approach has a higher CPU overhead.

Now, if the system is close to 100% cpu already, *then* the multipath overhead will try to take CPU power we don't have and cause a stall, but
it's only in the flat out CPU case.

Quoted 7 lines

Not to mention that our iSCSI/iSER initiator is already taking a session
wide lock when sending outgoing PDUs, so adding a session wide counter
isn't adding any additional synchronization overhead vs. what's already
in place.

Quoted 3 lines

I'll leave it up to the iSER people to decide whether they're redoing this as part of the MQ work.

Session wide command sequence number synchronization isn't something to
be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of
every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may
be sent to the target.

Quoted 6 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's
a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Quoted 8 lines

Quoted 1 line

Wouldn't that need to be coordinated with the networking layer?

Yes.

Quoted 1 line

Doesn't it do the same thing, matching TX/RX queues to CPUs?

Yes.

Hey Hannes, Mike,

I would say there is no need for specific coordination from iSCSI PoV. This is exactly what flow steering is designed for. As I see it, in order to get the TX/RX to match rings, the user can attach 5-tuple rules (using standard ethtool) to steer packets to the right rings.

Sagi.

Quoted 9 lines

Quoted 1 line

If so, wouldn't we decrease bandwidth by restricting things to one CPU?

We have a session or connection per CPU though, so we end up hitting the same problem you talked about last year where one hctx (iscsi session or connection's socket or nic hw queue) could get overloaded. This is what I meant in my original mail where iscsi would rely on whatever blk/mq load balancers we end up implementing at that layer to balance requests across hctxs.

I'm not sure I understand,

The submission flow is CPU bound. In the current single queue model both CPU X and CPU Y will end up using a single socket. In the multi-queue solution, CPU X will go to socket X and CPU Y will go to socket Y. This is equal to what we have today (if only CPU X is active) or better (if more CPUs are active).

Am I missing something?

Quoted 11 lines

Quoted 1 line

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

If we just tried to do this with the old code where the session could be accessed on multiple CPUs then you are right, we need locks/atomics like how we do in the MCS case.

On 1/9/2015 8:00 PM, Michael Christie wrote:
<SNIP>

Quoted 26 lines

Quoted 11 lines

Quoted 1 line

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

Quoted 4 lines

If we just tried to do this with the old code where the session could be accessed on multiple CPUs then you are right, we need locks/atomics like how we do in the MCS case.

I don't think we will want to restrict session per CPU. There is a tradeoff question of system resources. We might want to allow a user to configure multiple HW queues but still not to use too much of the system resources. So the session locks would still be used but definitely less congested...

Quoted 33 lines · Sagi Grimberg

On 1/9/2015 8:00 PM, Michael Christie wrote:
<SNIP>

Quoted 30 lines

Quoted 11 lines

Quoted 1 line

Session wide command sequence number synchronization isn't something to be removed as part of the MQ work. It's a iSCSI/iSER protocol requirement.

That is, the expected + maximum sequence numbers are returned as part of every response PDU, which the initiator uses to determine when the command sequence number window is open so new non-immediate commands may be sent to the target.

Quoted 5 lines

So, given some manner of session wide synchronization is required between different contexts for the existing single connection case to update the command sequence number and check when the window opens, it's a fallacy to claim MC/S adds some type of new initiator specific synchronization overhead vs. single connection code.

I think you are assuming we are leaving the iscsi code as it is today.

For the non-MCS mq session per CPU design, we would be allocating and binding the session and its resources to specific CPUs. They would only be accessed by the threads on that one CPU, so we get our serialization/synchronization from that. That is why we are saying we do not need something like atomic_t/spin_locks for the sequence number handling for this type of implementation.

If we just tried to do this with the old code where the session could be accessed on multiple CPUs then you are right, we need locks/atomics like how we do in the MCS case.

Quoted 5 lines

I don't think we will want to restrict session per CPU. There is a tradeoff question of system resources. We might want to allow a user to configure multiple HW queues but still not to use too much of the system resources. So the session locks would still be used but definitely less congested...

Are you talking about specifically the session per CPU or also MCS and doing a connection per CPU?

Based on the srp work, how bad do you think it will be to do a session/connection per CPU? What are you thinking will be more common? Session per 4 CPU? 2 CPUs? 8?

There is also multipath to take into account here. We could do a mq/MCS session/connection per CPU (or group of CPS) then also one of those per transport path. We could also do a mq/MCS session/connection per transport path, then bind those to specific CPUs. Or something in between.

Quoted 104 lines · James Bottomley

Quoted 100 lines · Nicholas A. Bellinger

Quoted 75 lines · Bart Van Assche

Quoted 60 lines · Mike Christie

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

Quoted 10 lines

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

The idea that a simple session wide counter for command sequence number assignment adds such a degree of contention that it renders MC/S at a performance disadvantage vs. multi-session configurations with all of the extra multipath logic overhead on top is at best, a naive proposition.

On the initiator side for MC/S, literally the only thing that needs to be serialized is the assignment of the command sequence number to individual non-immediate PDUs. The sending of the outgoing PDUs + immediate data by the initiator can happen out-of-order, and it's up to the target to ensure that the submission of the commands to the device server is in command sequence number order.

Quoted 2 lines

All of the actual immediate data + R2T -> data-out processing by the target can also be done out-of-order as well.

Quoted 9 lines

Right, but what he's saying is that we've taken great pains in the MQ situation to free our issue queues of all entanglements and cross queue locking so they can fly as fast as possible. If we have to assign an in-order sequence number across all the queues, this becomes both a cross CPU bus lock point to ensure atomicity and a sync point to ensure sequencing. Naïvely that does look to be a bottleneck which wouldn't necessarily be mitigated simply by allowing everything to proceed out of order after this point.

The point is that a simple session wide counter for command sequence number assignment is significantly less overhead than all of the overhead associated with running a full multipath stack atop multiple sessions.

Not to mention that our iSCSI/iSER initiator is already taking a session wide lock when sending outgoing PDUs, so adding a session wide counter isn't adding any additional synchronization overhead vs. what's already in place.

I am not sure if we want this to be a deciding factor. I think the session wide lock is something that can be removed in the main IO paths.

A lot of what it is used for now is cmd/task related handling like list accesses. When we have the scsi layer alloc/free/manage that, then we can simplify that a lot for iser/bnx2i/cxgb*i since there send path is less complicated than software iscsi.

It is also used for the state check but I think that is overkill.

Quoted 62 lines · Mike Christie

Quoted 43 lines · Hannes Reinecke

Quoted 38 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is
implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the
scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

Quoted 5 lines

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Hello Sagi and Mike,

I agree with Sagi that adding scsi-mq support in the iSER initiator would help iSER users because that would allow these users to configure a single iSER target and use the multiqueue feature instead of having to configure multiple iSER targets to spread the workload over multiple cpus at the target side.

And I agree with Mike that implementing scsi-mq support in the iSER initiator as multiple independent connections probably is a better choice than MC/S. RFC 3720 namely requires that iSCSI numbering is session-wide. This means maintaining a single counter for all MC/S sessions. Such a counter would be a contention point. I'm afraid that because of that counter performance on a multi-socket initiator system with a scsi-mq implementation based on MC/S could be worse than with the approach with multiple iSER targets. Hence my preference for an approach based on multiple independent iSER connections instead of MC/S.

Quoted 41 lines · Hannes Reinecke

Quoted 36 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

Quoted 4 lines

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 7 lines

Yes, that's a really good topic.

I've pondered implementing MC/S for iscsi/TCP but then I've figured my network implementation knowledge doesn't spread that far. So yeah, a discussion here would be good.

Mike? Any comments?

I have been working under the assumption that people would be ok with MCS upstream if we are only using it to handle the issue where we want to do something like have a tcp/iscsi connection per CPU then map the connection to a blk_mq_hw_ctx. In this more limited MCS implementation there would be no iscsi layer code to do something like load balance across ports or transport paths like how dm-multipath does, so there would be no feature/code duplication. For balancing across hctxs, then the iscsi layer would also leave that up to whatever we end up with in upper layers, so again no feature/code duplication with upper layers.

So pretty non controversial I hope :)

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Quoted 3 lines

If people want to add something like round robin connection selection in the iscsi layer, then I think we want to leave that for after the initial merge, so people can argue about that separately.

Well, you're right, we can argue about it later, but if it's just round robin, why would it be better done in the initator rather than dm?

So pretty non controversial I hope

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Cheers,
Sagi.

Quoted 40 lines · Sagi Grimberg

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.

Quoted 2 lines

Cheers,
Sagi.

I started looking at this last year (and Hannes' suggestion), and would love to join the discussion.

Please add me to the list of those that wish to attend.

Hi everyone,

Now that scsi-mq is fully included, we need an iSCSI initiator that would use it to achieve scalable performance. The need is even greater for iSCSI offload devices and transports that support multiple HW queues. As iSER maintainer I'd like to discuss the way we would choose to implement that in iSCSI.

My measurements show that iSER initiator can scale up to ~2.1M IOPs with multiple sessions but only ~630K IOPs with a single session where the most significant bottleneck the (single) core processing completions.

In the existing single connection per session model, given that command ordering must be preserved session-wide, we end up in a serial command execution over a single connection which is basically a single queue model. The best fit seems to be plugging iSCSI MCS as a multi-queued scsi LLDD. In this model, a hardware context will have a 1x1 mapping with an iSCSI connection (TCP socket or a HW queue).

iSCSI MCS and it's role in the presence of dm-multipath layer was discussed several times in the past decade(s). The basic need for MCS is implementing a multi-queue data path, so perhaps we may want to avoid doing any type link aggregation or load balancing to not overlap dm-multipath. For example we can implement ERL=0 (which is basically the scsi-mq ERL) and/or restrict a session to a single portal.

As I see it, the todo's are:
1. Getting MCS to work (kernel + user-space) with ERL=0 and a round-robin connection selection (per scsi command execution).
2. Plug into scsi-mq - exposing num_connections as nr_hw_queues and using blk-mq based queue (conn) selection.
3. Rework iSCSI core locking scheme to avoid session-wide locking as much as possible.
4. Use blk-mq pre-allocation and tagging facilities.

I've recently started looking into this. I would like the community to agree (or debate) on this scheme and also talk about implementation with anyone who is also interested in this.