Skip to contentSkip to conversation

InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what's in infiniband.git)

DISCUSSION Roland Dreier 6 participants 22 messages Apr 1, 2008 – Apr 4, 2008

lore
Share & tools
lore
#0 lore raw

The 2.6.26 will open soon, so it's time to review what my plans are for the merge window opens.

As usual, patch review by non-me people is always welcome.

Anyway, here are all the pending things that I'm aware of. As usual, if something isn't already in my tree and isn't listed below, I probably missed it or dropped it by mistake. Please remind me again in that case.

Core:

- I did a bunch of cleanups all over drivers/infiniband and the gcc and sparse warning noise is down to a pretty reasonable level. Further cleanups welcome of course.

ULPs:

- I merged Eli's IPoIB stateless offload changes for checksum offload and LSO changes. The interrupt moderation changes are next, and should not be a problem to merge. Please test IPoIB on all sorts of hardware!

- Shirley's IPoIB 4 KB MTU changes. I expect these to make it in, although I would certainly appreciate review from Eli or anyone else.

HW specific:

- Vlad's mlx4 resize CQ support. Looks basically OK, so I think we should be able to get it in.

- ipath support for 7220 HCAs. I don't expect any issues here once the patches appear.

Here are a few topics that I believe will not be ready in time for the 2.6.26 window and will need to wait for 2.6.27 at least:

- XRC. I still don't have a good feeling that we have settled on all the nuances of the ABI we want to expose to userspace for this, and ideally I would like to understand how ehca LL QPs fit into the picture as well.

- Remove LLTX from IPoIB. I haven't had time to finish this yet, so I guess it will probably wait for 2.6.27 now...

- Multiple CQ event vector support. I still haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.

Here all the patches I already have in my for-2.6.26 branch:

Arthur Jones (4):
IB/ipath: Fix sparse warning about pointer signedness IB/ipath: Misc sparse warning cleanup IB/ipath: Provide I/O bus speeds for diagnostic purposes IB/ipath: Fix link up LED display

Dave Olson (4):
IB/ipath: Make some constants chip-specific, related cleanup IB/ipath: Shared context code needs to be sure device is usable IB/ipath: Enable 4KB MTU
IB/ipath: HW workaround for case where chip can send but not receive

David Dillow (1):
IB/srp: Enforce protocol limit on srp_sg_tablesize

Eli Cohen (7):
IPoIB: Use checksum offload support if available IB/mlx4: Add IPoIB checksum offload support IB/mthca: Add IPoIB checksum offload support IB/core: Add creation flags to struct ib_qp_init_attr IB/core: Add IPoIB UD LSO support
IPoIB: Add LSO support
IB/mlx4: Add IPoIB LSO support

Harvey Harrison (1):
IB: Replace remaining __FUNCTION__ occurrences with __func__

Hoang-Nam Nguyen (1):
IB/ehca: Remove tgid checking

John Gregor (1):
IB/ipath: Head of Line blocking vs forward progress of user apps

Julia Lawall (1):
RDMA/iwcm: Test rdma_create_id() for IS_ERR rather than 0

Michael Albaugh (2):
IB/ipath: Prevent link-recovery code from negating admin disable IB/ipath: EEPROM support for 7220 devices, robustness improvements, cleanup

Ralph Campbell (11):
IB/ipath: Fix byte order of pioavail in handle_errors() IB/ipath: Fix error recovery for send buffer status after chip freeze mode IB/ipath: Don't try to handle freeze mode HW errors if diagnostic mode IB/ipath: Make debug error message match the constraint that is checked for IB/ipath: Add code to support multiple link speeds and widths IB/ipath: Remove useless comments
IB/ipath: Fix sanity checks on QP number of WRs and SGEs IB/ipath: Change the module author
IB/ipath: Remove some useless (void) casts
IB/ipath: Make send buffers available for kernel if not allocated to user IB/ipath: Use PIO buffer for RC ACKs

Robert P. J. Day (2):
IB: Use shorter list_splice_init() for brevity RDMA/nes: Use more concise list_for_each_entry()

Roland Dreier (28):
IB/mthca: Formatting cleanups
IB/mlx4: Convert "if(foo)" to "if (foo)"
mlx4_core: Move opening brace of function onto a new line RDMA/amso1100: Don't use 0UL as a NULL pointer RDMA/cxgb3: IDR IDs are signed
IB: Make struct ib_uobject.id a signed int
IB/ipath: Fix sparse warning about shadowed symbol IB/mlx4: Endianness annotations
IB/cm: Endianness annotations
RDMA/ucma: Endian annotation
RDMA/nes: Trivial endianness annotations
RDMA/nes: Delete unused variables
RDMA/amso1100: Start of endianness annotation RDMA/amso1100: Endian annotate mqsq allocator
mlx4_core: Fix confusion between mlx4_event and mlx4_dev_event enums IB/uverbs: Don't store struct file * for event files IB/uverbs: Use alloc_file() instead of get_empty_filp() RDMA/nes: Remove redundant NULL check in nes_unregister_ofa_device() RDMA/nes: Remove unused nes_netdev_exit() function RDMA/nes: Use proper format and cast to print dma_addr_t RDMA/nes: Make symbols used only in a single source file static IB/ehca: Make symbols used only in a single source file static IB/core: Add support for "send with invalidate" work requests RDMA/amso1100: Add support for "send with invalidate" work requests IB/mthca: Avoid integer overflow when dealing with profile size IB/mthca: Avoid integer overflow when allocating huge ICM table IB/ipath: Fix PCI config write size used to clear linkctrl error bits RDMA/nes: Remove session_id from nes_cm stuff

#1 lore raw
Quoted 4 lines · Roland Dreier

- Multiple CQ event vector support. I still haven't seen any discussions about how ULPs or userspace apps should decide which vector to use, and hence no progress has been made since we deferred this during the 2.6.23 merge window.

I did some prototype for IPoIB to enable multiple CQ event support. I did see the approach improved multiple links aggregation performance. I also see some customers' requirements in userspace. I will start the discussion as soon as possible. But it would most likely miss 2.6.26 window.

Thanks
Shirley

#2 lore raw
In reply to Roland Dreier
Quoted 6 lines · Roland Dreier

Core:

- I did a bunch of cleanups all over drivers/infiniband and the gcc and sparse warning noise is down to a pretty reasonable level. Further cleanups welcome of course.

We want to add send with invalidate & mask compare and swap.
Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

Quoted 7 lines

ULPs:

- I merged Eli's IPoIB stateless offload changes for checksum offload and LSO changes. The interrupt moderation changes are next, and should not be a problem to merge. Please test IPoIB on all sorts of hardware!

What about the split CQ for UD mode? It's improved the IPoIB performance for small messages significantly.

Quoted 4 lines

HW specific:

mlx4- we plan to send patches for the low level driver only to enable mlx4_en. These only affect our low level driver. Should be ready next week. I hope these can get in too.

Quoted 8 lines

Here are a few topics that I believe will not be ready in time for the 2.6.26 window and will need to wait for 2.6.27 at least:

- XRC. I still don't have a good feeling that we have settled on all the nuances of the ABI we want to expose to userspace for this, and ideally I would like to understand how ehca LL QPs fit into the picture as well.

I think we should try to push for XEC in 2.6.26 since there are already MPI implementation that use it and this ties them to use OFED only. Also this feature is stable and now being defined in IBTA
Not taking it causing changes between OFED and the kernel and your libibverbs and we wish to avoid such gaps.
Is there any thing we can do to help and make it into 2.6.26?

#10 lore raw
Quoted 1 line

What's the status of RDS?

I've never seen any patches. I guess ask the RDS guys if/when they want to start working on getting RDS merged.

- R.

#12 lore raw
In reply to Richard Frank
Quoted 1 line

What is the work we need to do here - I was thinking RDS should just work ?

Stuff doesn't get merged into the kernel on its own. If you want RDS upstream then the first step is to post patches in a form suitable for reviewing. Then respond to the review comments.

The files Documentation/SubmittingPatches and to some extent Documentation/SubmittingDrivers in the kernel source have more info.

- R.

#13 lore raw

Yes, I see this is for pushing RDS upstream - but what about running RDS as is over IWARP NICs - that should just work right ?

Quoted 11 lines · Roland Dreier
Quoted 1 line

What is the work we need to do here - I was thinking RDS should just work ?

Stuff doesn't get merged into the kernel on its own. If you want RDS upstream then the first step is to post patches in a form suitable for reviewing. Then respond to the review comments.

The files Documentation/SubmittingPatches and to some extent Documentation/SubmittingDrivers in the kernel source have more info.

- R.

#3 lore raw
In reply to Tziporet Koren
Quoted 3 lines

We want to add send with invalidate & mask compare and swap. Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

Send with invalidate should be OK. Let's see about the masked atomics stuff -- we have a ton of new verbs and I think we might want to slow down and make sure it all makes sense.

Quoted 2 lines

What about the split CQ for UD mode? It's improved the IPoIB performance for small messages significantly.

Oh yeah... I'll try to get that in too.

Quoted 2 lines

mlx4- we plan to send patches for the low level driver only to enable mlx4_en. These only affect our low level driver.

No problem in principle, let's see the actual patches.

Quoted 7 lines

I think we should try to push for XEC in 2.6.26 since there are already MPI implementation that use it and this ties them to use OFED only.
Also this feature is stable and now being defined in IBTA Not taking it causing changes between OFED and the kernel and your libibverbs and we wish to avoid such gaps.
Is there any thing we can do to help and make it into 2.6.26?

I don't have a good feeling that the user-kernel interface is well thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs and make sure we have something that makes sense for the future.

- R.

#11 lore raw
In reply to Roland Dreier

What is the work we need to do here - I was thinking RDS should just work ?

Quoted 13 lines · Roland Dreier
Quoted 1 line

What's the status of RDS?

I've never seen any patches. I guess ask the RDS guys if/when they want to start working on getting RDS merged.

- R.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

#14 lore raw
In reply to Richard Frank
Quoted 2 lines

Yes, I see this is for pushing RDS upstream - but what about running RDS as is over IWARP NICs - that should just work right ?

No idea. It depends on whether you took into account the differences between IB and iWARP. Anyway that's not really what this thread was about.

#15 lore raw

got it...

Quoted 6 lines · Roland Dreier
Quoted 2 lines

Yes, I see this is for pushing RDS upstream - but what about running RDS as is over IWARP NICs - that should just work right ?

No idea. It depends on whether you took into account the differences between IB and iWARP. Anyway that's not really what this thread was about.

#17 lore raw

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

In reply to Richard Frank
Quoted 3 lines

WRT to merging RDS into the kernel - our current plans are to wait to see RDS adopted by more than Oracle - before approaching the kernel community about inclusion of RDS.

I've seen statements before from someone from Oracle that RDS was only for Oracle's use, for example, that person did not want netperf changed to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

#16 lore raw
In reply to Roland Dreier

WRT to merging RDS into the kernel - our current plans are to wait to see RDS adopted by more than Oracle - before approaching the kernel community about inclusion of RDS.

Quoted 11 lines · Roland Dreier
Quoted 1 line

What is the work we need to do here - I was thinking RDS should just work ?

Stuff doesn't get merged into the kernel on its own. If you want RDS upstream then the first step is to post patches in a form suitable for reviewing. Then respond to the review comments.

The files Documentation/SubmittingPatches and to some extent Documentation/SubmittingDrivers in the kernel source have more info.

- R.

#18 lore raw

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

In reply to Scott Weitzenkamp (sweitzen)

I believe there is a patch for NetPerf which supports RDS - although it may need to be updated - and submitted.

The only prior discussion I can think of - was whether or not NetPerf exercises RDS as Oracle would.

I'm not proposing that we should enhance NetPerf to do that (but that's OK with me).

We created a tool rds-stress which does that.

Quoted 12 lines · Scott Weitzenkamp (sweitzen)
Quoted 2 lines

WRT to merging RDS into the kernel - our current plans are to wait to see RDS adopted by more than Oracle - before approaching the kernel community about inclusion of RDS.

I've seen statements before from someone from Oracle that RDS was only for Oracle's use, for example, that person did not want netperf changed to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

#19 lore raw

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

Rich,

On Nov 1, 2007, you wrote this to rds-devel:

"Netperf is too simplistic in that all it seems to do is stream data
in a
simple loop. This is not how Oracle uses the IPC and again does not reflect what it would take to make UDP reliable.

For this reason we are not interested in having Netperf support RDS
and
or seeing Netperf data."

I would like to see RDS supported by existing common tools like netperf, iperf, etc. so we can easily compare how RDS performs to UDP for IPC models other than Oracle.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

Quoted 42 lines

-----Original Message-----
From: Richard Frank [mailto:[email protected]]
Sent: Wednesday, April 02, 2008 10:38 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Roland Dreier (rdreier); [email protected]; [email protected]; [email protected]
Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

I believe there is a patch for NetPerf which supports RDS - although it
may need to be updated - and submitted.

The only prior discussion I can think of - was whether or not NetPerf exercises RDS as Oracle would.

I'm not proposing that we should enhance NetPerf to do that (but that's
OK with me).

We created a tool rds-stress which does that.

Quoted 1 line · Scott Weitzenkamp (sweitzen)
Quoted 1 line

WRT to merging RDS into the kernel - our current plans are

to wait to

Quoted 1 line
Quoted 1 line

see RDS adopted by more than Oracle - before approaching

the kernel

Quoted 4 lines
Quoted 2 lines

community about inclusion of RDS.

I've seen statements before from someone from Oracle that

RDS was only

Quoted 1 line

for Oracle's use, for example, that person did not want

netperf changed

Quoted 7 lines

to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

#20 lore raw

Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

OK - and the conversation was about using NetPerf to compare performance of RDS to UDP relative to suitability for Oracle use ... so I think those statements still illustrate my points...

1) NetPerf does not do what Oracle does - and hence is not useful from Oracle's perspective in comparing ULPs.
2) For some metrics - it's not valid to compare a non-reliable IPC to a reliable IPC - it's not an apples to apples comparison. Especially when the app is considered and what the app must do to use UDP vs RDS.

I did not say that NetPerf should not be extended to support RDS - just that using it to do a comparison of ULPs to determine how well Oracle would run - is not what we (Oracle) would want - at least that was my intention..

Quoted 69 lines · Scott Weitzenkamp (sweitzen)

Rich,

On Nov 1, 2007, you wrote this to rds-devel:

"Netperf is too simplistic in that all it seems to do is stream data
in a
simple loop. This is not how Oracle uses the IPC and again does not reflect what it would take to make UDP reliable.

For this reason we are not interested in having Netperf support RDS
and
or seeing Netperf data."

I would like to see RDS supported by existing common tools like netperf, iperf, etc. so we can easily compare how RDS performs to UDP for IPC models other than Oracle.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

Quoted 43 lines

-----Original Message-----
From: Richard Frank [mailto:[email protected]] Sent: Wednesday, April 02, 2008 10:38 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Roland Dreier (rdreier); [email protected]; [email protected]; [email protected]
Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

I believe there is a patch for NetPerf which supports RDS - although it may need to be updated - and submitted.

The only prior discussion I can think of - was whether or not NetPerf exercises RDS as Oracle would.

I'm not proposing that we should enhance NetPerf to do that (but that's OK with me).

We created a tool rds-stress which does that.

Quoted 2 lines · Scott Weitzenkamp (sweitzen)
Quoted 2 lines

WRT to merging RDS into the kernel - our current plans are

to wait to

Quoted 2 lines
Quoted 2 lines

see RDS adopted by more than Oracle - before approaching

the kernel

Quoted 5 lines
Quoted 3 lines

community about inclusion of RDS.

I've seen statements before from someone from Oracle that

RDS was only

Quoted 2 lines

for Oracle's use, for example, that person did not want

netperf changed

Quoted 8 lines

to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

#21 lore raw

RE: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

I'd like to see netperf comparisions of UDP_STREAM/UDP_RR vs RDS_STREAM/RDS_RR, does anyone have a patch that will apply cleanly to a recent netperf?

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

Quoted 113 lines

-----Original Message-----
From: Richard Frank [mailto:[email protected]]
Sent: Wednesday, April 02, 2008 11:00 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Roland Dreier (rdreier); [email protected]; [email protected]; [email protected]
Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

OK - and the conversation was about using NetPerf to compare performance
of RDS to UDP relative to suitability for Oracle use ... so I think those statements still illustrate my points...

1) NetPerf does not do what Oracle does - and hence is not useful from
Oracle's perspective in comparing ULPs.
2) For some metrics - it's not valid to compare a non-reliable IPC to a
reliable IPC - it's not an apples to apples comparison. Especially when
the app is considered and what the app must do to use UDP vs RDS.

I did not say that NetPerf should not be extended to support RDS - just
that using it to do a comparison of ULPs to determine how well Oracle would run - is not what we (Oracle) would want - at least that was my intention..

Quoted 5 lines · Scott Weitzenkamp (sweitzen)

Rich,

On Nov 1, 2007, you wrote this to rds-devel:

"Netperf is too simplistic in that all it seems to do is

stream data

Quoted 2 lines

in a
simple loop. This is not how Oracle uses the IPC and

again does not

Quoted 3 lines

reflect what it would take to make UDP reliable.

For this reason we are not interested in having Netperf

support RDS

Quoted 4 lines

and
or seeing Netperf data."

I would like to see RDS supported by existing common tools

like netperf,

Quoted 26 lines

iperf, etc. so we can easily compare how RDS performs to UDP for IPC models other than Oracle.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

Quoted 14 lines

-----Original Message-----
From: Richard Frank [mailto:[email protected]]
Sent: Wednesday, April 02, 2008 10:38 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Roland Dreier (rdreier); [email protected]; [email protected]; [email protected]
Subject: Re: [ofa-general] InfiniBand/iWARP/RDMA merge plans for 2.6.26 (what'sin infiniband.git)

I believe there is a patch for NetPerf which supports RDS - although it
may need to be updated - and submitted.

The only prior discussion I can think of - was whether or

not NetPerf

Quoted 37 lines
Quoted 37 lines

exercises RDS as Oracle would.

I'm not proposing that we should enhance NetPerf to do that (but that's
OK with me).

We created a tool rds-stress which does that.

Quoted 2 lines · Scott Weitzenkamp (sweitzen)
Quoted 2 lines

WRT to merging RDS into the kernel - our current plans are

to wait to

Quoted 2 lines
Quoted 2 lines

see RDS adopted by more than Oracle - before approaching

the kernel

Quoted 5 lines
Quoted 3 lines

community about inclusion of RDS.

I've seen statements before from someone from Oracle that

RDS was only

Quoted 2 lines

for Oracle's use, for example, that person did not want

netperf changed

Quoted 8 lines

to support RDS.

Scott Weitzenkamp
SQA and Release Manager
Data Center Access Engineering
Cisco Systems

#4 lore raw
In reply to Roland Dreier
Quoted 4 lines · Roland Dreier

Send with invalidate should be OK. Let's see about the masked atomics stuff -- we have a ton of new verbs and I think we might want to slow down and make sure it all makes sense.

OK - will send and then we will see what will come out.

Quoted 5 lines
Quoted 2 lines

What about the split CQ for UD mode? It's improved the IPoIB performance for small messages significantly.

Oh yeah... I'll try to get that in too.

thanks

Quoted 5 lines
Quoted 2 lines

mlx4- we plan to send patches for the low level driver only to enable mlx4_en. These only affect our low level driver.

No problem in principle, let's see the actual patches.

Sure

Quoted 8 lines
Quoted 7 lines

I think we should try to push for XEC in 2.6.26 since there are already MPI implementation that use it and this ties them to use OFED only.
Also this feature is stable and now being defined in IBTA Not taking it causing changes between OFED and the kernel and your libibverbs and we wish to avoid such gaps.
Is there any thing we can do to help and make it into 2.6.26?

Quoted 5 lines

I don't have a good feeling that the user-kernel interface is well thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs and make sure we have something that makes sense for the future.

I see - but can't we figure this all for the 2.6.26 window?

Tziporet

#8 lore raw
In reply to Tziporet Koren
Quoted 2 lines · Tziporet Koren

We want to add send with invalidate
Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

Does send with invalidate applies to rkeys generated through the proprietary FMR API?
if not, what usage you envision to the new verb under nowadays IB devices?

Or.

#5 lore raw
In reply to Roland Dreier
Quoted 3 lines

We want to add send with invalidate & mask compare and swap. Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

We are very interested in these new operations and are moving in the direction of tightly integrating RDMA along with atomics (if available) into Oracle. We plan on testing some early prototypes of the these in the few months.

Send with invalidate is an exact match for our current RDS V3 rdma driver - and should be more efficient than the current background syncing of the tpt to ensure keys are invalidated.

We intend on exposing the atomics via the RDS driver along with simple low level rdma operations to Oracle's internal clients. If Oracle is running over a transport which exports atomics and rdma - Oracle will see a dramatic performance boost for several database operations.

Quoted 38 lines · Roland Dreier
Quoted 3 lines

We want to add send with invalidate & mask compare and swap. Eli will be able to send the patches next week and since they are small I think they can be in for 2.6.26

Send with invalidate should be OK. Let's see about the masked atomics stuff -- we have a ton of new verbs and I think we might want to slow down and make sure it all makes sense.

Quoted 2 lines

What about the split CQ for UD mode? It's improved the IPoIB performance for small messages significantly.

Oh yeah... I'll try to get that in too.

Quoted 2 lines

mlx4- we plan to send patches for the low level driver only to enable mlx4_en. These only affect our low level driver.

No problem in principle, let's see the actual patches.

Quoted 7 lines

I think we should try to push for XEC in 2.6.26 since there are already MPI implementation that use it and this ties them to use OFED only.
Also this feature is stable and now being defined in IBTA Not taking it causing changes between OFED and the kernel and your libibverbs and we wish to avoid such gaps.
Is there any thing we can do to help and make it into 2.6.26?

I don't have a good feeling that the user-kernel interface is well thought out, so I want to consider XRC + ehca LL stuff + new iWARP verbs and make sure we have something that makes sense for the future.

- R.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

#6 lore raw
Quoted 4 lines

We are very interested in these new operations and are moving in the direction of tightly integrating RDMA along with atomics (if available) into Oracle. We plan on testing some early prototypes of the these in the few months.

And you need the ConnectX-only masked atomics? Or do the standard IB atomic operations work for you? Of course using atomics at all means that things don't work on iWARP.

Quoted 3 lines

Send with invalidate is an exact match for our current RDS V3 rdma driver - and should be more efficient than the current background syncing of the tpt to ensure keys are invalidated.

How does send with invalidate interact with the current IB FMR stuff? Seems that you would run into trouble keeping the state of the FMR straight if the remote side is invalidating them.

Also I would think that send-with-invalidate would be much more expensive than the current FMR method of batching up the invalidates, since you don't get to amortize the cost of syncing up all the internal HCA state.

- R.

#7 lore raw
Quoted 10 lines · Roland Dreier
Quoted 4 lines

We are very interested in these new operations and are moving in the direction of tightly integrating RDMA along with atomics (if available) into Oracle. We plan on testing some early prototypes of the these in the few months.

And you need the ConnectX-only masked atomics? Or do the standard IB atomic operations work for you? Of course using atomics at all means that things don't work on iWARP.

We specifically asked for the masked operations.

Yes, this means Oracle will not get the performance boost of atomics on IWARP - but we still get rdma - and that's a real win / benefit for Oracle today - and more so over the next few months.

Quoted 9 lines
Quoted 3 lines

Send with invalidate is an exact match for our current RDS V3 rdma driver - and should be more efficient than the current background syncing of the tpt to ensure keys are invalidated.

How does send with invalidate interact with the current IB FMR stuff? Seems that you would run into trouble keeping the state of the FMR straight if the remote side is invalidating them.

The model we implement is based on "use once" keys - we issue the key to the rdma server and want to toss it as soon as the rdma is complete. Today, we explicitly free the key after the rdma completes and we get a message from the rdma server - saying rdma is complete. If the key is auto invalidated by the recv'ing HCA then we do not need to do it in the driver... which also meanswe do not need to issue the sync tpts to force the HCA to be update its cache.

At least this is how I think it works - Olaf is the divine source here.

Quoted 6 lines

Also I would think that send-with-invalidate would be much more expensive than the current FMR method of batching up the invalidates, since you don't get to amortize the cost of syncing up all the internal HCA state.

This is the one piece we do not know - our plans are to test this and see where the trade offs are. We will keep the current design / implementation to run over NICs that do not support send-with-invalidate.

Quoted 2 lines

- R.