Skip to contentSkip to conversation

tgtd: send/recv iSCSI PDUs in worker threads

RFC PATCH Superseded Hitoshi Mitake 2 participants 7 messages Nov 10, 2013 – Nov 24, 2013 Kbuild

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/1 tgtd: send/recv iSCSI PDUs in worker threads

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.

---

v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed

 usr/iscsi/conn.c      |  12 ++-
 usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++---
 usr/iscsi/iscsid.c    |  61 +++++++----
 usr/iscsi/iscsid.h    |   8 +-
 4 files changed, 334 insertions(+), 40 deletions(-)
usr/iscsi/conn.c +8 -4
index 17aa3e3..21b552e 100644
@@ -83,7 +83,7 @@ void conn_exit(struct iscsi_connection *conn)
 		session_put(session);
 }
 
-void conn_close(struct iscsi_connection *conn)
+int conn_close(struct iscsi_connection *conn)
 {
 	struct iscsi_task *task, *tmp;
 	int ret;
@@ -180,14 +180,18 @@ void conn_close(struct iscsi_connection *conn)
 		iscsi_free_task(task);
 	}
 done:
-	conn_put(conn);
+	return conn_put(conn);
 }
 
-void conn_put(struct iscsi_connection *conn)
+int conn_put(struct iscsi_connection *conn)
 {
 	conn->refcount--;
-	if (!conn->refcount)
+	if (!conn->refcount) {
 		conn->tp->ep_release(conn);
+		return 0;
+	}
+
+	return conn->refcount;
 }
 
 int conn_get(struct iscsi_connection *conn)
usr/iscsi/iscsi_tcp.c +281 -12

361 lines not on this page · raw message

usr/iscsi/iscsid.c +39 -22

103 lines not on this page · raw message

usr/iscsi/iscsid.h +6 -2
index c7f6801..aabbeca 100644
@@ -309,8 +309,8 @@ extern int cmnd_exec_auth_chap(struct iscsi_connection *conn);
 /* conn.c */
 extern int conn_init(struct iscsi_connection *conn);
 extern void conn_exit(struct iscsi_connection *conn);
-extern void conn_close(struct iscsi_connection *conn);
-extern void conn_put(struct iscsi_connection *conn);
+extern int conn_close(struct iscsi_connection *conn);
+extern int conn_put(struct iscsi_connection *conn);
 extern int conn_get(struct iscsi_connection *conn);
 extern struct iscsi_connection * conn_find(struct iscsi_session *session, uint32_t cid);
 extern int conn_take_fd(struct iscsi_connection *conn);
@@ -336,6 +336,10 @@ extern int iscsi_param_parse_portals(char *p, int do_add, int do_delete);
 extern void iscsi_update_conn_stats_rx(struct iscsi_connection *conn, int size, int opcode);
 extern void iscsi_update_conn_stats_tx(struct iscsi_connection *conn, int size, int opcode);
 extern void iscsi_rsp_set_residual(struct iscsi_cmd_rsp *rsp, struct scsi_cmd *scmd);
+extern void iscsi_tcp_work_rx_done_fn(struct iscsi_connection *conn);
+extern void iscsi_tcp_work_tx_done_fn(struct iscsi_connection *conn);
+extern int is_conn_rx_end(struct iscsi_connection *conn);
+extern int is_conn_tx_end(struct iscsi_connection *conn);
 
 /* iscsid.c iscsi_task */
 extern void iscsi_free_task(struct iscsi_task *task);
Signature · 1 line
1.8.1.2
#1 lore raw

At Sun, 10 Nov 2013 17:49:07 +0900,

Quoted 28 lines · Hitoshi Mitake

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.

Quoted 7 lines

---

v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed

Ping?
Could someone review this patch? > all

Thanks,
Hitoshi

#2 lore raw
In reply to Hitoshi Mitake
Quoted 16 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Quoted 7 lines

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

Can you check how this scales with tcp connections on a reasonably-high-performance server?

#3 lore raw

At Sun, 17 Nov 2013 22:50:58 +0900 (JST),

Quoted 24 lines · FUJITA Tomonori
Quoted 20 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

Quoted 6 lines
Quoted 3 lines

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

Can you check how this scales with tcp connections on a reasonably-high-performance server?

Sure. I'd like to evaluate and post a result in the near future.

Thanks,
Hitoshi

#4 lore raw
In reply to Hitoshi Mitake
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.

---

v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed

Quoted 5 lines
 usr/iscsi/conn.c      |  12 ++-
 usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++---
 usr/iscsi/iscsid.c    |  61 +++++++----
 usr/iscsi/iscsid.h    |   8 +-
 4 files changed, 334 insertions(+), 40 deletions(-)

This can handle a single iscsi session having multiple tcp connections? For example, if iscsi_scsi_cmd_rx_start() is called by multiple threads, conn->session->cmd_list is accessed by them without a lock?

#5 lore raw

At Fri, 22 Nov 2013 09:29:49 +0900,

Quoted 45 lines · FUJITA Tomonori
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.

---

v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed

Quoted 5 lines
 usr/iscsi/conn.c      |  12 ++-
 usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++---
 usr/iscsi/iscsid.c    |  61 +++++++----
 usr/iscsi/iscsid.h    |   8 +-
 4 files changed, 334 insertions(+), 40 deletions(-)
Quoted 4 lines

This can handle a single iscsi session having multiple tcp connections? For example, if iscsi_scsi_cmd_rx_start() is called by multiple threads, conn->session->cmd_list is accessed by them without a lock?

As you say, some functions which manipulate data structures related to session can be called by multiple threads in an unsafe manner, sorry.

I'll post v3 later. In the next version, iscsi_task_rx_start() and iscsi_task_tx_start() are executed in a serialized manner. AFAIK, other part of multithreaded code don't touch the data structures.

Thanks for your review,
Hitoshi

#6 lore raw

At Sun, 24 Nov 2013 23:06:02 +0900,

Quoted 53 lines · Hitoshi Mitake

At Fri, 22 Nov 2013 09:29:49 +0900,

Quoted 45 lines · FUJITA Tomonori
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…

From: Hitoshi Mitake <[email protected]>

Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.

This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.

The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.

Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec

tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec

This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.

---

v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed

Quoted 5 lines
 usr/iscsi/conn.c      |  12 ++-
 usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++---
 usr/iscsi/iscsid.c    |  61 +++++++----
 usr/iscsi/iscsid.h    |   8 +-
 4 files changed, 334 insertions(+), 40 deletions(-)
Quoted 4 lines

This can handle a single iscsi session having multiple tcp connections? For example, if iscsi_scsi_cmd_rx_start() is called by multiple threads, conn->session->cmd_list is accessed by them without a lock?

Quoted 6 lines

As you say, some functions which manipulate data structures related to session can be called by multiple threads in an unsafe manner, sorry.

I'll post v3 later. In the next version, iscsi_task_rx_start() and iscsi_task_tx_start() are executed in a serialized manner. AFAIK, other part of multithreaded code don't touch the data structures.

BTW, I'll be able to execute a benchmark on 10 GbE environment next week. I'd like to post the result soon.

Thanks,
Hitoshi