tgtd: send/recv iSCSI PDUs in worker threads
RFC PATCH v2 1/1 tgtd: send/recv iSCSI PDUs in worker threads
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec
tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec
This change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.
---
v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug in iscsi_tcp_init() introduced in the previous patch is removed
usr/iscsi/conn.c | 12 ++- usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++--- usr/iscsi/iscsid.c | 61 +++++++---- usr/iscsi/iscsid.h | 8 +- 4 files changed, 334 insertions(+), 40 deletions(-)
usr/iscsi/conn.c +8 -4
@@ -83,7 +83,7 @@ void conn_exit(struct iscsi_connection *conn)session_put (session); } -void conn_close (struct iscsi_connection *conn) +int conn_close (struct iscsi_connection *conn) {struct iscsi_task *task, *tmp;int ret;
@@ -180,14 +180,18 @@ void conn_close(struct iscsi_connection *conn)iscsi_free_task (task); } done: -conn_put (conn); +return conn_put (conn); } -void conn_put (struct iscsi_connection *conn) +int conn_put (struct iscsi_connection *conn) { conn->refcount--; -if (!conn->refcount) +if (!conn->refcount) { conn->tp->ep_release (conn); +return 0 ; + } + +return conn->refcount; }int conn_get (struct iscsi_connection *conn)
usr/iscsi/iscsi_tcp.c +281 -12
361 lines not on this page · raw message
usr/iscsi/iscsid.c +39 -22
103 lines not on this page · raw message
usr/iscsi/iscsid.h +6 -2
@@ -309,8 +309,8 @@ extern int cmnd_exec_auth_chap(struct iscsi_connection *conn);/* conn.c */ extern int conn_init (struct iscsi_connection *conn);extern void conn_exit (struct iscsi_connection *conn); -extern void conn_close (struct iscsi_connection *conn); -extern void conn_put (struct iscsi_connection *conn); +extern int conn_close (struct iscsi_connection *conn); +extern int conn_put (struct iscsi_connection *conn);extern int conn_get (struct iscsi_connection *conn);extern struct iscsi_connection *conn_find (struct iscsi_session *session,uint32_t cid);extern int conn_take_fd (struct iscsi_connection *conn);
@@ -336,6 +336,10 @@ extern int iscsi_param_parse_portals(char *p, int do_add, int do_delete);extern void iscsi_update_conn_stats_rx (struct iscsi_connection *conn,int size,int opcode);extern void iscsi_update_conn_stats_tx (struct iscsi_connection *conn,int size,int opcode);extern void iscsi_rsp_set_residual (struct iscsi_cmd_rsp *rsp,struct scsi_cmd *scmd); +extern void iscsi_tcp_work_rx_done_fn (struct iscsi_connection *conn); +extern void iscsi_tcp_work_tx_done_fn (struct iscsi_connection *conn); +extern int is_conn_rx_end (struct iscsi_connection *conn); +extern int is_conn_tx_end (struct iscsi_connection *conn);/* iscsid.c iscsi_task */ extern void iscsi_free_task (struct iscsi_task *task);
Signature · 1 line
1.8.1.2
At Sun, 10 Nov 2013 17:49:07 +0900,
Quoted 28 lines · Hitoshi Mitake
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msectgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msecThis change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.
Quoted 7 lines
Signed-off-by Hitoshi Mitake <[email protected]>---
v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug iniscsi_tcp_init()introduced in the previous patch is removed
Ping?
Could someone review this patch? > all
Thanks,
Hitoshi
Quoted 16 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Quoted 7 lines
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msectgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msec
Can you check how this scales with tcp connections on a reasonably-high-performance server?
At Sun, 17 Nov 2013 22:50:58 +0900 (JST),
Quoted 24 lines · FUJITA Tomonori
Quoted 20 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msec
Quoted 6 lines
Quoted 3 lines
tgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msecCan you check how this scales with tcp connections on a reasonably-high-performance server?
Sure. I'd like to evaluate and post a result in the near future.
Thanks,
Hitoshi
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msectgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msecThis change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.
Signed-off-by Hitoshi Mitake <[email protected]>---
v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug iniscsi_tcp_init()introduced in the previous patch is removed
Quoted 5 lines
usr/iscsi/conn.c | 12 ++- usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++--- usr/iscsi/iscsid.c | 61 +++++++---- usr/iscsi/iscsid.h | 8 +- 4 files changed, 334 insertions(+), 40 deletions(-)
This can handle a single iscsi session having multiple tcp connections? For example, if iscsi_scsi_cmd_rx_start() is called by multiple threads, conn->session->cmd_list is accessed by them without a lock?
At Fri, 22 Nov 2013 09:29:49 +0900,
Quoted 45 lines · FUJITA Tomonori
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msectgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msecThis change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.
Signed-off-by Hitoshi Mitake <[email protected]>---
v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug iniscsi_tcp_init()introduced in the previous patch is removedQuoted 5 lines
usr/iscsi/conn.c | 12 ++- usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++--- usr/iscsi/iscsid.c | 61 +++++++---- usr/iscsi/iscsid.h | 8 +- 4 files changed, 334 insertions(+), 40 deletions(-)
Quoted 4 lines
This can handle a single iscsi session having multiple tcp connections? For example, if
iscsi_scsi_cmd_rx_start()is called by multiple threads,conn->session->cmd_listis accessed by them without a lock?
As you say, some functions which manipulate data structures related to session can be called by multiple threads in an unsafe manner, sorry.
I'll post v3 later. In the next version, iscsi_task_rx_start() and iscsi_task_tx_start() are executed in a serialized manner. AFAIK, other part of multithreaded code don't touch the data structures.
Thanks for your review,
Hitoshi
At Sun, 24 Nov 2013 23:06:02 +0900,
Quoted 53 lines · Hitoshi Mitake
At Fri, 22 Nov 2013 09:29:49 +0900,
Quoted 45 lines · FUJITA Tomonori
Quoted 35 lines · On Sun, 10 Nov 2013 17:49:07 +0900 Hitoshi Mitake <[email protected]>…
From: Hitoshi Mitake <[email protected]>
Current tgtd sends and receives iSCSI PDUs in its main event loop. This design can cause bottleneck when many iSCSI clients connect to single tgtd process. For example, we need multiple tgtd processes for utilizing fast network like 10 GbE because typical single processor core isn't fast enough for processing bunch of requests.
This patch lets tgtd send/receive iSCSI PDUs and check digests in its worker threads. After applying this patch, the bottleneck in the main event loop is removed and the performance is improved.
The improvement can be seen even if tgtd and iSCSI initiator are running on a single host. Below is a snippet of fio result on my laptop. The workload is 128MB random RW. Backingstore is sheepdog.
Original tgtd:
read : io=65392KB, bw=4445.2KB/s, iops=1111, runt= 14711msec
write: io=65680KB, bw=4464.8KB/s, iops=1116, runt= 14711msectgtd with this patch:
read : io=65392KB, bw=5098.9KB/s, iops=1274, runt= 12825msec
write: io=65680KB, bw=5121.3KB/s, iops=1280, runt= 12825msecThis change will be more effective when a number of iSCSI clients increases. I'd like to hear your comments on this change.
Signed-off-by Hitoshi Mitake <[email protected]>---
v2:
- correct handling of connection closing based on a reference count of an iSCSI connection
- a silly bug iniscsi_tcp_init()introduced in the previous patch is removedQuoted 5 lines
usr/iscsi/conn.c | 12 ++- usr/iscsi/iscsi_tcp.c | 293 +++++++++++++++++++++++++++++++++++++++++++++++--- usr/iscsi/iscsid.c | 61 +++++++---- usr/iscsi/iscsid.h | 8 +- 4 files changed, 334 insertions(+), 40 deletions(-)Quoted 4 lines
This can handle a single iscsi session having multiple tcp connections? For example, if
iscsi_scsi_cmd_rx_start()is called by multiple threads,conn->session->cmd_listis accessed by them without a lock?
Quoted 6 lines
As you say, some functions which manipulate data structures related to session can be called by multiple threads in an unsafe manner, sorry.
I'll post v3 later. In the next version,
iscsi_task_rx_start()andiscsi_task_tx_start()are executed in a serialized manner. AFAIK, other part of multithreaded code don't touch the data structures.
BTW, I'll be able to execute a benchmark on 10 GbE environment next week. I'd like to post the result soon.
Thanks,
Hitoshi