QoS for iSCSI target?

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

Quoted 5 lines · Chris Friesen

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

Quoted 5 lines

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with blk cgroups.

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Chris Friesen

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

Quoted 8 lines

Quoted 5 lines

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with blk cgroups.

For iscsi-target application QoS, the per session command sequence number window depth (CmdSN) exists to enforce per InitiatorName limits via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs attributes.

Note these values can be changed on the fly for iscsi-target using explicit se_node_acl->acl_group, but currently require a se_session reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

Quoted 16 lines · Nicholas A. Bellinger

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Chris Friesen

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

Quoted 8 lines

Quoted 5 lines

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with blk cgroups.

Quoted 8 lines

For iscsi-target application QoS, the per session command sequence number window depth (CmdSN) exists to enforce per InitiatorName limits via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs attributes.

Note these values can be changed on the fly for iscsi-target using explicit se_node_acl->acl_group, but currently require a se_session reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

Quoted 26 lines · Chris Friesen

Quoted 16 lines · Nicholas A. Bellinger

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Chris Friesen

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

Quoted 8 lines

Quoted 5 lines

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with blk cgroups.

Quoted 8 lines

For iscsi-target application QoS, the per session command sequence number window depth (CmdSN) exists to enforce per InitiatorName limits via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs attributes.

Note these values can be changed on the fly for iscsi-target using explicit se_node_acl->acl_group, but currently require a se_session reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

Quoted 39 lines · Nicholas A. Bellinger

Quoted 26 lines · Chris Friesen

Quoted 16 lines · Nicholas A. Bellinger

Quoted 6 lines · Christoph Hellwig

Quoted 5 lines · Chris Friesen

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any way to do QoS between traffic driven by different initiators.

Quoted 8 lines

Quoted 5 lines

I'm trying to make sure that one initiator can't do a denial-of-service attack against others.

Does the kernel target have this sort of thing built-in, or do I need to look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with blk cgroups.

Quoted 8 lines

For iscsi-target application QoS, the per session command sequence number window depth (CmdSN) exists to enforce per InitiatorName limits via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs attributes.

Note these values can be changed on the fly for iscsi-target using explicit se_node_acl->acl_group, but currently require a se_session reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Quoted 3 lines

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

How would I use cgroups with the kernel iSCSI target? Is there a set of kernel threads dedicated to the iSCSI target that I can assign to a particular group?

Quoted 12 lines · Chris Friesen

Quoted 12 lines

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Quoted 8 lines

Quoted 3 lines

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

How would I use cgroups with the kernel iSCSI target? Is there a set of kernel threads dedicated to the iSCSI target that I can assign to a particular group?

block cgroups can set I/O throttling (bandwidth + IOPs) limits for any normal struct block_device.

These values are configured via blkio.throttle.* resource class, and the limits are imposed independently of the block_device's association with target_core_mod backend driver export. Namely:

blkio.throttle.write_iops_device
blkio.throttle.read_iops_device
blkio.throttle.write_bps_device
blkio.throttle.read_bdp_device

Some examples using these values, plus more blkio.* info is here:
http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.

Quoted 35 lines · Nicholas A. Bellinger

Quoted 12 lines · Chris Friesen

Quoted 12 lines

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Quoted 8 lines

Quoted 3 lines

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

How would I use cgroups with the kernel iSCSI target? Is there a set of kernel threads dedicated to the iSCSI target that I can assign to a particular group?

block cgroups can set I/O throttling (bandwidth + IOPs) limits for any normal struct block_device.

These values are configured via blkio.throttle.* resource class, and the limits are imposed independently of the block_device's association with target_core_mod backend driver export. Namely:

blkio.throttle.write_iops_device
blkio.throttle.read_iops_device
blkio.throttle.write_bps_device
blkio.throttle.read_bdp_device

Quoted 3 lines

Some examples using these values, plus more blkio.* info is here:
http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.

I understand how to set up the cgroups, but don't I need to ensure that the IO operations happen in the context of a particular cgroup? If so, how do I ensure that the in-kernel iSCSI target traffic happens in the context of the specified group? Are there a set of kernel threads dedicated to processing iSCSI requests?

Quoted 36 lines · Chris Friesen

Quoted 35 lines · Nicholas A. Bellinger

Quoted 12 lines · Chris Friesen

Quoted 12 lines

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Quoted 8 lines

Quoted 3 lines

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

How would I use cgroups with the kernel iSCSI target? Is there a set of kernel threads dedicated to the iSCSI target that I can assign to a particular group?

block cgroups can set I/O throttling (bandwidth + IOPs) limits for any normal struct block_device.

These values are configured via blkio.throttle.* resource class, and the limits are imposed independently of the block_device's association with target_core_mod backend driver export. Namely:

blkio.throttle.write_iops_device
blkio.throttle.read_iops_device
blkio.throttle.write_bps_device
blkio.throttle.read_bdp_device

Quoted 9 lines

Quoted 3 lines

Some examples using these values, plus more blkio.* info is here:
http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.

I understand how to set up the cgroups, but don't I need to ensure that the IO operations happen in the context of a particular cgroup? If so, how do I ensure that the in-kernel iSCSI target traffic happens in the context of the specified group? Are there a set of kernel threads dedicated to processing iSCSI requests?

block cgroups does I/O ratelimiting at struct block_device level.

Eg: The process cgroup (which AFAICT is what your thinking about) is separate from block cgroups, and doesn't need to be explicitly enabled in order for block cgroup to function.

Quoted 47 lines · Nicholas A. Bellinger

Quoted 36 lines · Chris Friesen

Quoted 35 lines · Nicholas A. Bellinger

Quoted 12 lines · Chris Friesen

Quoted 12 lines

Quoted 8 lines

On a slightly different note, is there any way to throttle or limit the overall bandwidth consumed by the iSCSI target in the kernel? I'd like to ensure that the iSCSI traffic doesn't completely swamp the host accesses to the same block device.

I suppose I could do networking-based traffic shaping, but are there any controls in the block IO subsystem?

On a individual block_device backend basis, block cgroups is the preferred method for doing this.

Quoted 8 lines

Quoted 3 lines

Note that any rate limiting imposed by block cgroups is subject to the current se_node_acl->queue_depth enforced across all LUNs within a given iscsi session.

How would I use cgroups with the kernel iSCSI target? Is there a set of kernel threads dedicated to the iSCSI target that I can assign to a particular group?

block cgroups can set I/O throttling (bandwidth + IOPs) limits for any normal struct block_device.

These values are configured via blkio.throttle.* resource class, and the limits are imposed independently of the block_device's association with target_core_mod backend driver export. Namely:

blkio.throttle.write_iops_device
blkio.throttle.read_iops_device
blkio.throttle.write_bps_device
blkio.throttle.read_bdp_device

Quoted 9 lines

Quoted 3 lines

Some examples using these values, plus more blkio.* info is here:
http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.

I understand how to set up the cgroups, but don't I need to ensure that the IO operations happen in the context of a particular cgroup? If so, how do I ensure that the in-kernel iSCSI target traffic happens in the context of the specified group? Are there a set of kernel threads dedicated to processing iSCSI requests?

Quoted 6 lines

block cgroups does I/O ratelimiting at struct block_device level.

Eg: The process cgroup (which AFAICT is what your thinking about) is separate from block cgroups, and doesn't need to be explicitly enabled in order for block cgroup to function.

I'm still confused.

I'm not trying to globally throttle IO on a particular block device. I'm trying to control how much IO the iSCSI target in the kernel is allowed to drive on a particular block device.

The goal is to ensure that the iSCSI target does not consume all of the available bandwidth for a particular block device. I want to ensure that some of the bandwidth for that device is available to other processes on the host (for management purposes) rather than being consumed by a greedy iSCSI initiator.

In an ideal world I would like a set of rules that say things like:
1) if there is contention, ensure that the host is guaranteed X percent of the available /dev/sdb IOPS
2) if there is contention, do not allow the iSCSI target traffic to consume more than Y percent of /dev/sdb's write traffic

Quoted 9 lines · Chris Friesen

I'm not trying to globally throttle IO on a particular block device. I'm trying to control how much IO the iSCSI target in the kernel is allowed to drive on a particular block device.

The goal is to ensure that the iSCSI target does not consume all of the available bandwidth for a particular block device. I want to ensure that some of the bandwidth for that device is available to other processes on the host (for management purposes) rather than being consumed by a greedy iSCSI initiator.

AFAIK, different traffic classes for a single block device is not supported by block cgroups.

Also given target_core_iblock claims a given block_device for exclusive access, you can't actually use the same block device for a mounted filesystem, LVM, MD, etc, once it's been registered for target usage.

Quoted 6 lines

In an ideal world I would like a set of rules that say things like:
1) if there is contention, ensure that the host is guaranteed X percent of the available /dev/sdb IOPS
2) if there is contention, do not allow the iSCSI target traffic to consume more than Y percent of /dev/sdb's write traffic

Yeah, that doesn't exist in standlone block groups.

You could probably use network cgroups to limit bandwidth at the socket level for iscsi_target_mod, but that's going to be across all LUNs in a session, and not at individual LUN level.