Skip to contentSkip to conversation

target: fix a race condition between login_work and the login thread

PATCH Superseded Maurizio Lombardi 2 participants 4 messages Nov 4, 2022 – Nov 7, 2022 SCSI target

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 target: fix a race condition between login_work and the login thread

In case a malicious initiator sends some random data immediately after a login PDU; the iscsi_target_sk_data_ready() callback will schedule the login_work and, at the same time,
the negotiation may end without clearing the LOGIN_FLAGS_INITIAL_PDU flag (because no additional PDU exchanges are required to complete the login).

The login has been completed but the login_work function will find the LOGIN_FLAGS_INITIAL_PDU flag set and will never stop from rescheduling itself;
at this point, if the initiator drops the connection, the iscsit_conn structure will be freed, login_work will dereference a released socket structure and the kernel crashes.

BUG: kernel NULL pointer dereference, address: 0000000000000230
PF: supervisor write access in kernel mode
PF: error_code(0x0002) - not-present page
Workqueue: events iscsi_target_do_login_rx [iscsi_target_mod]
RIP: 0010:_raw_read_lock_bh+0x15/0x30
Call trace:
 iscsi_target_do_login_rx+0x75/0x3f0 [iscsi_target_mod]
 process_one_work+0x1e8/0x3c0

Fix this bug by forcing login_work to stop after the login has been completed and the socket callbacks have been restored.

---
 drivers/target/iscsi/iscsi_target_nego.c | 1 +
 1 file changed, 1 insertion(+)
drivers/target/iscsi/iscsi_target_nego.c +1 -0
index f2919319ad38..e58bdffe2931 100644
@@ -1058,6 +1058,7 @@ static int iscsi_target_do_login(struct iscsit_conn *conn, struct iscsi_login *l
 				login->tsih = conn->sess->tsih;
 				login->login_complete = 1;
 				iscsi_target_restore_sock_callbacks(conn);
+				cancel_delayed_work_sync(&conn->login_work);
 				if (iscsi_target_do_tx_login_io(conn,
 						login) < 0)
 					return -1;
Signature · 1 line
2.31.1
#1 lore raw

pá 4. 11. 2022 v 10:53 odesílatel Maurizio Lombardi <[email protected]> napsal:

Quoted 8 lines

The login has been completed but the login_work function will find the LOGIN_FLAGS_INITIAL_PDU flag set and will never stop from rescheduling itself;
at this point, if the initiator drops the connection, the iscsit_conn structure will be freed, login_work will dereference a released socket structure and the kernel crashes.

A reproducer is available here:

http://bsdbackstore.eu/misc/target_kill/iscsitgtkiller.c

Maurizio

#2 lore raw
In reply to Maurizio Lombardi
Quoted 35 lines · Maurizio Lombardi

In case a malicious initiator sends some random data immediately after a login PDU; the iscsi_target_sk_data_ready() callback will schedule the login_work and, at the same time,
the negotiation may end without clearing the LOGIN_FLAGS_INITIAL_PDU flag (because no additional PDU exchanges are required to complete the login).

The login has been completed but the login_work function will find the LOGIN_FLAGS_INITIAL_PDU flag set and will never stop from rescheduling itself;
at this point, if the initiator drops the connection, the iscsit_conn structure will be freed, login_work will dereference a released socket structure and the kernel crashes.

BUG: kernel NULL pointer dereference, address: 0000000000000230
PF: supervisor write access in kernel mode
PF: error_code(0x0002) - not-present page
Workqueue: events iscsi_target_do_login_rx [iscsi_target_mod]
RIP: 0010:_raw_read_lock_bh+0x15/0x30
Call trace:
 iscsi_target_do_login_rx+0x75/0x3f0 [iscsi_target_mod]
 process_one_work+0x1e8/0x3c0

Fix this bug by forcing login_work to stop after the login has been completed and the socket callbacks have been restored.

---
 drivers/target/iscsi/iscsi_target_nego.c | 1 +
 1 file changed, 1 insertion(+)
drivers/target/iscsi/iscsi_target_nego.c +0 -0
index f2919319ad38..e58bdffe2931 100644

drivers/target/iscsi/iscsi_target_nego.c static int iscsi_target_do_login(struct iscsit_conn *conn, struct iscsi_login *l · line 1061

Quoted 4 lines
drivers/target/iscsi/iscsi_target_nego.c +1 -0
@@ -1058,6 +1058,7 @@ static int iscsi_target_do_login(struct iscsit_conn *conn, struct iscsi_login *l
 				login->tsih = conn->sess->tsih;
 				login->login_complete = 1;
 				iscsi_target_restore_sock_callbacks(conn);
+				cancel_delayed_work_sync(&conn->login_work);

You can remove the cancel_delayed_work in iscsi_target_do_login_rx in the chunk that checks for 1 being returned since you do it here now.

For the error path, I think you could also move the cancel_delayed_work_sync from them and put it in here.

If we leave it to the callers, in iscsi_target_do_login_rx in the "goto err" handling should this be reversed and do you want to use the _sync call like above?

        iscsi_target_restore_sock_callbacks(conn);
        cancel_delayed_work(&conn->login_work);
Quoted 3 lines
drivers/target/iscsi/iscsi_target_nego.c +0 -0
 				if (iscsi_target_do_tx_login_io(conn,
 						login) < 0)
 					return -1;
#3 lore raw

ne 6. 11. 2022 v 1:13 odesílatel Mike Christie
<[email protected]> napsal:

Quoted 4 lines

You can remove the cancel_delayed_work in iscsi_target_do_login_rx in the chunk that checks for 1 being returned since you do it here now.

Correct, I missed this one

Quoted 9 lines

For the error path, I think you could also move the cancel_delayed_work_sync from them and put it in here.

If we leave it to the callers, in iscsi_target_do_login_rx in the "goto err" handling should this be reversed and do you want to use the _sync call like above?

        iscsi_target_restore_sock_callbacks(conn);
        cancel_delayed_work(&conn->login_work);

Yes, I noticed it too and I was already preparing a follow-up patch to fix this. There is also another issue in iscsi_target_start_negotiation():

        if (ret < 0) {
                cancel_delayed_work_sync(&conn->login_work);
                iscsi_target_restore_sock_callbacks(conn);

This is obviously wrong because it leaves a small window open to race conditions I will submit a V2.

Maurizio