Skip to contentSkip to conversation

Add support for enhanced atomic operations

PATCH Superseded Vladimir Sokolovsky 3 participants 10 messages Mar 10, 2010 – Apr 13, 2010 7 messages not in the archive InfiniBand

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add support for enhanced atomic operations

Hi Roland,

This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and add

These operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.

Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operations

changes from V2:
- patch #1:
Updated description
Renamed:

        IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP
        IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADD

In the ib_send_wr struct the new fields added before the rkey field

- patch #2:
Set IB_DEVICE_MASKED_ATOMIC flag with other flags that get set for all devices

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw

Hi Vlad,

Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?

Thanks, Håkon

Quoted 31 lines · Vladimir Sokolovsky

Hi Roland,

This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and add

These operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.

Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operations

changes from V2:
- patch #1:
Updated description
Renamed:

       IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP
       IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADD

In the ib_send_wr struct the new fields added before the rkey field

- patch #2:
Set IB_DEVICE_MASKED_ATOMIC flag with other flags that get set for all devices

Quoted 6 lines

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Håkon Bugge
[email protected]
+47 924 84 514

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): A27E2E88-63A0-45EE-9A70-F95B2A0C25AD-U0mLk4xYmo8@public.gmane.org

#3 lore raw
Quoted 1 line

Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?

I think we can worry about that if/when an HCA comes along that supports global atomics for ordinary atomics but not enhanced atomics. Although perhaps it would be cleaner to change the atomic_cap enum to:

	/*
	 * IB_ATOMIC_NONE:	no atomic capability
	 * IB_ATOMIC_HCA:	all ops are atomic within HCA
	 * IB_ATOMIC_GLOB:	standard ops atomic with respect to all
				memory ops; masked ops atomic within HCA
	 * IB_ATOMIC_GLOB_MASKED: all ops atomic with respect to all
	 *			memory ops
	 */
	enum ib_atomic_cap {
		IB_ATOMIC_NONE,
		IB_ATOMIC_HCA,
		IB_ATOMIC_GLOB,
		IB_ATOMIC_GLOB_MASKED
	};

(with better wording for the comments)
Thoughts?

- R.

Signature · 7 lines
Roland Dreier  <[email protected]>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaaaueelby.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#5 lore raw
Quoted 2 lines · Roland Dreier

I think we can worry about that if/when an HCA comes along that supports global atomics for ordinary atomics but not enhanced atomics.

With the proposed patches in place, how do you know if masked atomics are implemented or not? Guess apps need to know this information already on todays HCAs.

Quoted 5 lines

Although perhaps it would be cleaner to change the atomic_cap enum to:

/*
* IB_ATOMIC_NONE: no atomic capability
* IB_ATOMIC_HCA: all ops are atomic within HCA

But IB_ATOMIC_HCA does not tell you if the masked ones are supported or not.

Quoted 2 lines

* IB_ATOMIC_GLOB: standard ops atomic with respect to all
memory ops; masked ops atomic within HCA

What if an HCA supports standards ops with respect to all memory ops, but the HCA does not support masked atomics?

Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

Thanks, Håkon

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): BA082DF0-B3C6-4405-B480-F69B0193BF17-UdXhSnd/[email protected]

#7 lore raw
Quoted 14 lines

With the proposed patches in place, how do you know if masked atomics are implemented or not? Guess apps need to know this information already on todays HCAs.

From the patch:

(quoted diff) +1 -0
@@ -105,6 +105,7 @@ enum ib_device_cap_flags {
 	IB_DEVICE_UD_TSO		= (1<<19),
 	IB_DEVICE_MEM_MGT_EXTENSIONS	= (1<<21),
 	IB_DEVICE_BLOCK_MULTICAST_LOOPBACK = (1<<22),
+	IB_DEVICE_MASKED_ATOMIC		= (1<<23),
 };
 
 Hence, I think it would be cleaner if a new capability,
Quoted 2 lines

masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

Actually that does seem more orthogonal to me.

Signature · 7 lines
Roland Dreier  <[email protected]>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adavdd1a64g.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#9 lore raw
Quoted 3 lines

Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.

Signature · 7 lines
Roland Dreier  <[email protected]>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaeijj96a6.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#11 lore raw
Quoted 7 lines · Roland Dreier
Quoted 3 lines

Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.

Hi Roland,
Do you propose to use IB_ATOMIC_GLOB instead of IB_ATOMIC_HCA while setting atomic capability in the code below?

        props->atomic_cap          = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ?
                IB_ATOMIC_HCA : IB_ATOMIC_NONE;

Or add IB_MASKED_ATOMIC to ib_atomic_cap enum and use this one instead of IB_ATOMIC_HCA?

All this, of course, comes to replace setting IB_DEVICE_MASKED_ATOMIC for device capability.

Thanks,
Vladimir

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BA0F623.1050606-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

#13 lore raw
Quoted 17 lines · Vladimir Sokolovsky
Quoted 7 lines · Roland Dreier
Quoted 3 lines

Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

Vlad, what do you think about that? The more I think about it, the cleaner this seems to me. And it doesn't even consume a device capability flag bit, which is a nice bonus.

Hi Roland,
Do you propose to use IB_ATOMIC_GLOB instead of IB_ATOMIC_HCA while setting atomic capability in the code below?

        props->atomic_cap          = dev->dev->caps.flags & 
MLX4_DEV_CAP_FLAG_ATOMIC ?
                IB_ATOMIC_HCA : IB_ATOMIC_NONE;
Quoted 8 lines

Or add IB_MASKED_ATOMIC to ib_atomic_cap enum and use this one instead of IB_ATOMIC_HCA?

All this, of course, comes to replace setting IB_DEVICE_MASKED_ATOMIC for device capability.

Thanks,
Vladimir

Hi Roland,
Can you comment?

Thanks,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw
Quoted 7 lines

Do you propose to use IB_ATOMIC_GLOB instead of IB_ATOMIC_HCA while setting atomic capability in the code below?

        props->atomic_cap          = dev->dev->caps.flags &
MLX4_DEV_CAP_FLAG_ATOMIC ?
                IB_ATOMIC_HCA : IB_ATOMIC_NONE;
Quoted 5 lines

Or add IB_MASKED_ATOMIC to ib_atomic_cap enum and use this one instead of IB_ATOMIC_HCA?

All this, of course, comes to replace setting
IB_DEVICE_MASKED_ATOMIC for device capability.

Sorry, missed this reply first time around I guess.

Anyway, no I don't think we should use ATOMIC_GLOB unless an HCA truly provides global atomic ops; not possible without PCIe 3.0 atomic support I think. But as I said, I think the original suggestion:

Quoted 3 lines
Quoted 3 lines

Hence, I think it would be cleaner if a new capability, masked_atomic_cap, were introduced, using the original definitions (NONE, HCA, GLOB).

makes sense, ie create a new props->masked_atomic_cap and set it to NONE, HCA or GLOB as appropriate (ie NONE for everyone except mlx4, and HCA for mlx4).

Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#16 lore raw
In reply to a message not in the archive
Quoted 8 lines · Håkon Bugge

Hi Vlad,

Did you consider my input in http://www.mail-archive.com/[email protected]/msg02803.html wrt. to these enhancements?

Thanks, Håkon

Hi Håkon,
ConnectX devices do not support, currently, PCIe atomic operations.

Regards,
Vladimir

Quoted 40 lines · Vladimir Sokolovsky
Quoted 32 lines

Hi Roland,

This patchset adds support for the following enhanced atomic operations:
- Masked atomic compare and swap
- Masked atomic fetch and add

These operations enable using a smaller amount of memory when using multiple locks by using portions of a 64 bit value in an atomic operation.
For some applications the memory savings are very significant. One example is fine grain lock implementations for huge data sets. In other cases, the benefit is the ability to update multiple fields with a single io operation.

Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations mlx4/IB: Add support for enhanced atomic operations

changes from V2:
- patch #1:
Updated description
Renamed:

       IB_WR_ATOMIC_MASKED_CMP_AND_SWP -> IB_WR_MASKED_ATOMIC_CMP_AND_SWP
       IB_WR_ATOMIC_MASKED_FETCH_AND_ADD -> IB_WR_MASKED_ATOMIC_FETCH_AND_ADD

In the ib_send_wr struct the new fields added before the rkey field

- patch #2:
Set IB_DEVICE_MASKED_ATOMIC flag with other flags that get set for all devices

Quoted 6 lines

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines

Håkon Bugge
[email protected]
+47 924 84 514

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html