Add support for enhanced atomic operations
v3 cover Add support for enhanced atomic operations
Hi Roland,
This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and add
These operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.
Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operations
changes from V2:
- patch #1:
Updated description
Renamed:
IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP
IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADD In the ib_send_wr struct the new fields added before the rkey field
- patch #2:
Set IB_DEVICE_MASKED_ATOMIC flag with other flags that get set for all devices
Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Vlad,
Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?
Thanks, Håkon
Quoted 31 lines · Vladimir Sokolovsky
Hi Roland,
This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and addThese operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operationschanges from V2:
- patch #1:
Updated description
Renamed:IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADDIn the
ib_send_wrstruct the new fields added before the rkey field- patch #2:
SetIB_DEVICE_MASKED_ATOMICflag with other flags that get set for all devices
Quoted 6 lines
Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Håkon Bugge
[email protected]
+47 924 84 514
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): A27E2E88-63A0-45EE-9A70-F95B2A0C25AD-U0mLk4xYmo8@public.gmane.org
Quoted 1 line
Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?
I think we can worry about that if/when an HCA comes along that supports global atomics for ordinary atomics but not enhanced atomics. Although perhaps it would be cleaner to change the atomic_cap enum to:
/*
* IB_ATOMIC_NONE: no atomic capability
* IB_ATOMIC_HCA: all ops are atomic within HCA
* IB_ATOMIC_GLOB: standard ops atomic with respect to all
memory ops; masked ops atomic within HCA
* IB_ATOMIC_GLOB_MASKED: all ops atomic with respect to all
* memory ops
*/
enum ib_atomic_cap {
IB_ATOMIC_NONE,
IB_ATOMIC_HCA,
IB_ATOMIC_GLOB,
IB_ATOMIC_GLOB_MASKED
};(with better wording for the comments)
Thoughts?
- R.
Signature · 7 lines
Roland Dreier <[email protected]> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adaaaueelby.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 2 lines · Roland Dreier
I think we can worry about that if/when an HCA comes along that supports global atomics for ordinary atomics but not enhanced atomics.
With the proposed patches in place, how do you know if masked atomics are implemented or not? Guess apps need to know this information already on todays HCAs.
Quoted 5 lines
Although perhaps it would be cleaner to change the
atomic_capenum to:/*
*IB_ATOMIC_NONE: no atomic capability
*IB_ATOMIC_HCA: all ops are atomic within HCA
But IB_ATOMIC_HCA does not tell you if the masked ones are supported or not.
Quoted 2 lines
*
IB_ATOMIC_GLOB: standard ops atomic with respect to all
memory ops; masked ops atomic within HCA
What if an HCA supports standards ops with respect to all memory ops, but the HCA does not support masked atomics?
Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).
Thanks, Håkon
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): BA082DF0-B3C6-4405-B480-F69B0193BF17-UdXhSnd/[email protected]
Quoted 14 lines
With the proposed patches in place, how do you know if masked atomics are implemented or not? Guess apps need to know this information already on todays HCAs.
From the patch:
(quoted diff) +1
-0@@ -105,6 +105,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_MEM_MGT_EXTENSIONS = (1<<21), IB_DEVICE_BLOCK_MULTICAST_LOOPBACK = (1<<22), + IB_DEVICE_MASKED_ATOMIC = (1<<23), }; Hence, I think it would be cleaner if a new capability,
Quoted 2 lines
masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).
Actually that does seem more orthogonal to me.
Signature · 7 lines
Roland Dreier <[email protected]> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adavdd1a64g.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 3 lines
Hence, I think it would be cleaner if a new capability,
masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).
Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.
Signature · 7 lines
Roland Dreier <[email protected]> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adaeijj96a6.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 7 lines · Roland Dreier
Quoted 3 lines
Hence, I think it would be cleaner if a new capability,
masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.
Hi Roland,
Do you propose to use IB_ATOMIC_GLOB instead of IB_ATOMIC_HCA while setting atomic capability in the code below?
props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ?
IB_ATOMIC_HCA : IB_ATOMIC_NONE;Or add IB_MASKED_ATOMIC to ib_atomic_cap enum and use this one instead of IB_ATOMIC_HCA?
All this, of course, comes to replace setting IB_DEVICE_MASKED_ATOMIC for device capability.
Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4BA0F623.1050606-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
Quoted 17 lines · Vladimir Sokolovsky
Quoted 7 lines · Roland Dreier
Quoted 3 lines
Hence, I think it would be cleaner if a new capability,
masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.
Hi Roland,
Do you propose to useIB_ATOMIC_GLOBinstead ofIB_ATOMIC_HCAwhile setting atomic capability in the code below?props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ? IB_ATOMIC_HCA : IB_ATOMIC_NONE;
Quoted 8 lines
Or add
IB_MASKED_ATOMICtoib_atomic_capenum and use this one instead ofIB_ATOMIC_HCA?All this, of course, comes to replace setting
IB_DEVICE_MASKED_ATOMICfor device capability.Thanks,
Vladimir
Hi Roland,
Can you comment?
Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines
Do you propose to use
IB_ATOMIC_GLOBinstead ofIB_ATOMIC_HCAwhile setting atomic capability in the code below?props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ? IB_ATOMIC_HCA : IB_ATOMIC_NONE;
Quoted 5 lines
Or add
IB_MASKED_ATOMICtoib_atomic_capenum and use this one instead ofIB_ATOMIC_HCA?All this, of course, comes to replace setting
IB_DEVICE_MASKED_ATOMICfor device capability.
Sorry, missed this reply first time around I guess.
Anyway, no I don't think we should use ATOMIC_GLOB unless an HCA truly provides global atomic ops; not possible without PCIe 3.0 atomic support I think. But as I said, I think the original suggestion:
Quoted 3 lines
Quoted 3 lines
Hence, I think it would be cleaner if a new capability,
masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).
makes sense, ie create a new props->masked_atomic_cap and set it to NONE, HCA or GLOB as appropriate (ie NONE for everyone except mlx4, and HCA for mlx4).
Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Håkon Bugge
Hi Vlad,
Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?
Thanks, Håkon
Hi Håkon,
ConnectX devices do not support, currently, PCIe atomic operations.
Regards,
Vladimir
Quoted 40 lines · Vladimir Sokolovsky
Quoted 32 lines
Hi Roland,
This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and addThese operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operationschanges from V2:
- patch #1:
Updated description
Renamed:IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADDIn the
ib_send_wrstruct the new fields added before the rkey field- patch #2:
SetIB_DEVICE_MASKED_ATOMICflag with other flags that get set for all devicesQuoted 6 lines
Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
Håkon Bugge
[email protected]
+47 924 84 514
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html