IB_CQ_VECTOR_LEAST_ATTACHED

What's the history of this patch?

	http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 8 lines · Shirley Ma

What's the history of this patch?

    http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo git://git.infradead.org/users/hch/scsi-queue.git.

Quoted 10 lines · Bart Van Assche

Quoted 8 lines · Shirley Ma

What's the history of this patch?

     http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 1 line

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo git://git.infradead.org/users/hch/scsi-queue.git.

What I did is that I manually controlled IRQ and working thread on the same socket. The CQ is created when mounting the file system in NFS/RDMA, but the workload thread might start from different socket, so per-cpu based implementation might not apply. I will look at SRP implementation.

Quoted 21 lines · Shirley Ma

Quoted 13 lines · Bart Van Assche

Quoted 11 lines · Shirley Ma

What's the history of this patch?

http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O
performance. I think this CQ_VECTOR loading balance should be maintained
in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 6 lines

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo
git://git.infradead.org/users/hch/scsi-queue.git.

Quoted 6 lines

What I did is that I manually controlled IRQ and working thread on the same socket. The CQ is created when mounting the file system in NFS/RDMA, but the workload thread might start from different socket, so per-cpu based implementation might not apply. I will look at SRP implementation.

Hey Shirley,

Bart is correct, in general the LEAST_ATTACHED approach might not be optimal in the NUMA case. The thread <-> QP/CQ/CPU assignment is addressed by the multi-channel approach which to my understanding won't be implemented in NFSoRDMA in the near future (right Chuck?)

However, the LEAST_ATTACH vector hint will revive again in the future as there is a need to spread applications on different interrupt vectors (especially for user-space).

CC'ing Matan who is working on this, perhaps he can comment on this as well.

Sagi.

Applications might want to create a CQ on n different cores

Quoted 1 line · Matan Barak

Applications might want to create a CQ on n different cores

You mean like an IRQ can flush on a mask potentially made of multiple CPUs?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 2 lines · Or Gerlitz

Quoted 1 line · Matan Barak

Applications might want to create a CQ on n different cores

Quoted 2 lines

You mean like an IRQ can flush on a mask potentially made of multiple CPUs?

Sort of. In both cases you try to spread the resources such that you'll get best performance (that should be done by the device driver itself). The user needs to somehow get n different least-used resources. Hopefully, if the device driver does a decent job - the user would get his resources potentially on multiple cpus.

Quoted 11 lines · Shirley Ma

Quoted 10 lines · Bart Van Assche

Quoted 8 lines · Shirley Ma

What's the history of this patch?

    http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 4 lines

Quoted 1 line

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo git://git.infradead.org/users/hch/scsi-queue.git.

What I did is that I manually controlled IRQ and working thread on the same socket. The CQ is created when mounting the file system in NFS/RDMA, but the workload thread might start from different socket, so per-cpu based implementation might not apply. I will look at SRP implementation.

Hey Shirley,

Bart is correct, in general the LEAST_ATTACHED approach might not be optimal in the NUMA case. The thread <-> QP/CQ/CPU assignment is addressed by the multi-channel approach which to my understanding won't be implemented in NFSoRDMA in the near future (right Chuck?)

However, the LEAST_ATTACH vector hint will revive again in the future as there is a need to spread applications on different interrupt vectors (especially for user-space).

CC'ing Matan who is working on this, perhaps he can comment on this as well.

Quoted 17 lines · Sagi Grimberg

Quoted 11 lines · Shirley Ma

Quoted 10 lines · Bart Van Assche

Quoted 8 lines · Shirley Ma

What's the history of this patch?

    http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 4 lines

Quoted 1 line

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo git://git.infradead.org/users/hch/scsi-queue.git.

What I did is that I manually controlled IRQ and working thread on the same socket. The CQ is created when mounting the file system in NFS/RDMA, but the workload thread might start from different socket, so per-cpu based implementation might not apply. I will look at SRP implementation.

Quoted 6 lines

Hey Shirley,

Bart is correct, in general the LEAST_ATTACHED approach might not be optimal in the NUMA case. The thread <-> QP/CQ/CPU assignment is addressed by the multi-channel approach which to my understanding won't be implemented in NFSoRDMA in the near future (right Chuck?)

As I understand it, the preference of the Linux NFS community is that any multi-pathing solution should be transparent to the ULP (NFS and RPC, in this case). mp-tcp is ideal in that the ULP is presented with a single virtual transport instance, but under the covers, that instance can be backed by multiple active paths.

Alternately, pNFS can be deployed. This allows a dataset to be striped across multiple servers (and networks). There is a rather high bar to entering this arena however.

Speculating aloud, multiple QPs per transport instance may require implementation changes on the server as well as the client. Any interoperability dependencies should be documented via a standards process.

And note that an RPC transport (at least in kernel) is shared across many user applications and mount points. I find it difficult to visualize an intuitive and comprehensive administrative interface where enough guidance is provided to place a set of NFS applications and an RPC transport in the same resource domain (maybe cgroups?).

So for the time being I prefer staying with a single QP per client- server pair.

A large NFS client can actively use many NFS servers, however. Each client-server pair would benefit from finding "least-used" resources when QP and CQs are created. That is something we can leverage today.

Quoted 6 lines

However, the LEAST_ATTACH vector hint will revive again in the future as there is a need to spread applications on different interrupt vectors (especially for user-space).

CC'ing Matan who is working on this, perhaps he can comment on this as well.

Quoted 19 lines · Sagi Grimberg

Quoted 11 lines · Shirley Ma

Quoted 10 lines · Bart Van Assche

Quoted 8 lines · Shirley Ma

What's the history of this patch?

     http://lists.openfabrics.org/pipermail/general/2008-May/050813.html

I am working on multiple QPs workload. And I created a similar approach with IB_CQ_VECTOR_LEAST_ATTACHED, which can bring up about 17% small I/O performance. I think this CQ_VECTOR loading balance should be maintained in provider not the caller. I didn't see this patch was submitted to mainline kernel, wonder any reason behind?

Quoted 4 lines

Quoted 1 line

My interpretation is that an approach similar to IB_CQ_VECTOR_LEAST_ATTACHED is useful on single-socket systems but suboptimal on multi-socket systems. Hence the code for associating CQ sets with CPU sockets in the SRP initiator. These changes have been queued for kernel 3.19. See also branch drivers-for-3.19 in git repo git://git.infradead.org/users/hch/scsi-queue.git.

What I did is that I manually controlled IRQ and working thread on the same socket. The CQ is created when mounting the file system in NFS/RDMA, but the workload thread might start from different socket, so per-cpu based implementation might not apply. I will look at SRP implementation.

Hey Shirley,

Quoted 4 lines

Bart is correct, in general the LEAST_ATTACHED approach might not be optimal in the NUMA case. The thread <-> QP/CQ/CPU assignment is addressed by the multi-channel approach which to my understanding won't be implemented in NFSoRDMA in the near future (right Chuck?)

As I understand it, the preference of the Linux NFS community is that any multi-pathing solution should be transparent to the ULP (NFS and RPC, in this case).

mp-tcp is ideal in that the ULP is presented with
a single virtual transport instance, but under the covers, that instance can be backed by multiple active paths.

Alternately, pNFS can be deployed. This allows a dataset to be striped across multiple servers (and networks). There is a rather high bar to entering this arena however.

Speculating aloud, multiple QPs per transport instance may require implementation changes on the server as well as the client. Any interoperability dependencies should be documented via a standards process.

And note that an RPC transport (at least in kernel) is shared across many user applications and mount points. I find it difficult to visualize an intuitive and comprehensive administrative interface where enough guidance is provided to place a set of NFS applications and an RPC transport in the same resource domain (maybe cgroups?).

So for the time being I prefer staying with a single QP per client- server pair.

A large NFS client can actively use many NFS servers, however. Each client-server pair would benefit from finding "least-used" resources when QP and CQs are created. That is something we can leverage today.