dapltest question

Hey Arlin,

I have a dumb question: Is dapltest designed to support multiple dapltest client processes running on multiple physical systems to connect to a single dapltest server process/system?

I'm running tests where we have 4 client nodes running dapltest transaction clients connecting to the same dapltest server on a 5th node and I'm debugging a dapltest server hang. I have more details if you're interested, but I just wanna make sure dapltest is designed to support this mode.

Thanks,

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Steve,
Arlin is out for at least a week.
'dapltest' server is designed to service [1...n] dapltest clients and is routinely tested in this manner. Send me your dapltest client/server invocations and I'll give them a try.
Which DAPL provider are you using?

Stan.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Ok thanks. That's wanted to confirm.

This is with OFED-1.5.3.1 dapl-2.0.32.

Here are the /etc/dat.conf entries:

chelsio1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth10 0" "" chelsio2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth11 0" "" chelsio3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth12 0" "" chelsio4 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth13 0" ""

The test is running 4 dapltest servers, each bound to a specific chelsio port. Each port/interface is on its own IP network. There are 4 client nodes, each with 4 port cards connected. Each client has 4 scripts looping running this dapltest client, one to each server via the appropriate dapl device:

# cat scripts/dapl_run.sh
host=$1
device=$2
while [ 1 ]
do
dapltest -T T -s ${host} -D ${device} -i 100000 -t 2 -w 2 \
        client SR 256                                      \
        server RW 4096                                     \
        server SR 256                                      \
        client SR 256                                      \
        server RW 4096                                     \
        server SR 256                                      \
        client SR 4096                                     \
        server SR 256                                      \
        client RW 4096 1 \
        server RW 2048 4 \
        client SR 1024 4   server  SR  4096  2 \
        client SR 1024 3 -f server SR 2048 1 -f server RR 8192 1  client RR
8192 1
sleep 5
done

The first issue we ran into was the dapltest server connection request EVD was overflowing. Its hard coded to a depth of 8, so I bumped it up to 256. That alleviated over failures on the server. With this patch, the clients ran longer, but eventually they all start failing and cannot connect to the servers anymore. I got core dumps for the servers, and the master server thread, DT_cs_Server(), is stuck waiting on the current test threads to wake it up indicating they're ready to accept connections. So its stuck at line 560 in an infinite wait. After analysis of the core dump, I see the test threads that were created, have all exited, but never woke up the master server thread. Most likely they bailed out due to some other error before waking up the server. I'm continuing to debug.

Most likely there is some error in the RDMA provider that is causing these downstream issues.

So at this point, I think 2 changes might make dapltest more robust:

1) make the DT_cs_Server connection request EVD bigger.

2) Any test threads that start up should always wake up the synch_wait_object waiter no matter what. -or- DT_cs_Server should have a timeout on the wait.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html