Skip to contentSkip to conversation

scsi: target: iscsi: set memalloc_noio with loopback network connections

PATCH In review David Jeffery 4 participants 6 messages Feb 8, 2023 – Feb 13, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scsi: target: iscsi: set memalloc_noio with loopback network connections

If an admin connects an iscsi initiator to an iscsi target on the same system, the iscsi connection is vulnerable to deadlocks during memory allocations. Memory allocations in the target task accepting the I/O from the initiator can wait on the initiator's I/O when the system is under memory pressure, causing a deadlock situation between the iscsi target and initiator.

When in this configuration, the deadlock scenario can be avoided by use of GFP_NOIO allocations. Rather than force all configurations to use NOIO, memalloc_noio_save/restore can be used to force GFP_NOIO allocations only when in this loopback configuration.

---
 drivers/target/iscsi/iscsi_target.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/target/iscsi/iscsi_target.c +21 -0
index baf4da7bb3b4..a68e47e2cdf9 100644
@@ -16,6 +16,7 @@
 #include <linux/vmalloc.h>
 #include <linux/idr.h>
 #include <linux/delay.h>
+#include <linux/sched/mm.h>
 #include <linux/sched/signal.h>
 #include <asm/unaligned.h>
 #include <linux/inet.h>
@@ -4168,7 +4169,10 @@ int iscsi_target_rx_thread(void *arg)
 {
 	int rc;
 	struct iscsit_conn *conn = arg;
+	struct dst_entry *dst;
 	bool conn_freed = false;
+	bool loopback = false;
+	unsigned int flags;
 
 	/*
 	 * Allow ourselves to be interrupted by SIGINT so that a
@@ -4186,8 +4190,25 @@ int iscsi_target_rx_thread(void *arg)
 	if (!conn->conn_transport->iscsit_get_rx_pdu)
 		return 0;
 
+	/*
+	 * If the iscsi connection is over a loopback device from using
+	 * iscsi and iscsit on the same system, we need to set memalloc_noio to
+	 * prevent memory allocation deadlocks between target and initiator.
+	 */
+	rcu_read_lock();
+	dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+	if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+		loopback = true;
+	rcu_read_unlock();
+
+	if (loopback)
+		flags = memalloc_noio_save();
+
 	conn->conn_transport->iscsit_get_rx_pdu(conn);
 
+	if (loopback)
+		memalloc_noio_restore(flags);
+
 	if (!signal_pending(current))
 		atomic_set(&conn->transport_failed, 1);
 	iscsit_take_action_for_connection_exit(conn, &conn_freed);
Signature · 1 line
2.39.1
#1 lore raw
Quoted 66 lines · David Jeffery

If an admin connects an iscsi initiator to an iscsi target on the same
system, the iscsi connection is vulnerable to deadlocks during memory allocations. Memory allocations in the target task accepting the I/O from
the initiator can wait on the initiator's I/O when the system is under
memory pressure, causing a deadlock situation between the iscsi target and
initiator.

When in this configuration, the deadlock scenario can be avoided by use of
GFP_NOIO allocations. Rather than force all configurations to use NOIO,
memalloc_noio_save/restore can be used to force GFP_NOIO allocations only
when in this loopback configuration.

---
 drivers/target/iscsi/iscsi_target.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/target/iscsi/iscsi_target.c +7 -0
index baf4da7bb3b4..a68e47e2cdf9 100644
@@ -16,6 +16,7 @@
 #include <linux/vmalloc.h>
 #include <linux/idr.h>
 #include <linux/delay.h>
+#include <linux/sched/mm.h>
 #include <linux/sched/signal.h>
 #include <asm/unaligned.h>
 #include <linux/inet.h>
@@ -4168,7 +4169,10 @@ int iscsi_target_rx_thread(void *arg)
 {
 	int rc;
 	struct iscsit_conn *conn = arg;
+	struct dst_entry *dst;
 	bool conn_freed = false;
+	bool loopback = false;
+	unsigned int flags;
 
 	/*
 	 * Allow ourselves to be interrupted by SIGINT so that a
@@ -4186,8 +4190,25 @@ int iscsi_target_rx_thread(void *arg)
 	if (!conn->conn_transport->iscsit_get_rx_pdu)
 		return 0;
 
+	/*
+	 * If the iscsi connection is over a loopback device from using
+	 * iscsi and iscsit on the same system, we need to set

memalloc_noio to

drivers/target/iscsi/iscsi_target.c +1 -0
+	 * prevent memory allocation deadlocks between target and

initiator.

drivers/target/iscsi/iscsi_target.c +8 -0
+	 */
+	rcu_read_lock();
+	dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+	if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+		loopback = true;
+	rcu_read_unlock();
+
+	if (loopback)

drivers/target/iscsi/iscsi_target.c int iscsi_target_rx_thread(void *arg) · line 4214

Quoted 10 lines
drivers/target/iscsi/iscsi_target.c +5 -0
+		flags = memalloc_noio_save();
+
 	conn->conn_transport->iscsit_get_rx_pdu(conn);
 
+	if (loopback)
+		memalloc_noio_restore(flags);
+
 	if (!signal_pending(current))
 		atomic_set(&conn->transport_failed, 1);
 	iscsit_take_action_for_connection_exit(conn, &conn_freed);

I had mentioned to Mike that this was already tested at a large customer and in our labs and resolved the deadlocks .

Regards
Laurence Oberman

Tested-by Reviewed-by
#2 lore raw
Quoted 84 lines · Laurence Oberman
Quoted 69 lines · David Jeffery

If an admin connects an iscsi initiator to an iscsi target on the same
system, the iscsi connection is vulnerable to deadlocks during memory
allocations. Memory allocations in the target task accepting the I/O
from
the initiator can wait on the initiator's I/O when the system is under
memory pressure, causing a deadlock situation between the iscsi target and
initiator.

When in this configuration, the deadlock scenario can be avoided by use of
GFP_NOIO allocations. Rather than force all configurations to use NOIO,
memalloc_noio_save/restore can be used to force GFP_NOIO allocations
only
when in this loopback configuration.

---
 drivers/target/iscsi/iscsi_target.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
drivers/target/iscsi/iscsi_target.c +7 -0
index baf4da7bb3b4..a68e47e2cdf9 100644
@@ -16,6 +16,7 @@
 #include <linux/vmalloc.h>
 #include <linux/idr.h>
 #include <linux/delay.h>
+#include <linux/sched/mm.h>
 #include <linux/sched/signal.h>
 #include <asm/unaligned.h>
 #include <linux/inet.h>
@@ -4168,7 +4169,10 @@ int iscsi_target_rx_thread(void *arg)
 {
 	int rc;
 	struct iscsit_conn *conn = arg;
+	struct dst_entry *dst;
 	bool conn_freed = false;
+	bool loopback = false;
+	unsigned int flags;
 
 	/*
 	 * Allow ourselves to be interrupted by SIGINT so that a
@@ -4186,8 +4190,25 @@ int iscsi_target_rx_thread(void *arg)
 	if (!conn->conn_transport->iscsit_get_rx_pdu)
 		return 0;
 
+	/*
+	 * If the iscsi connection is over a loopback device from using
+	 * iscsi and iscsit on the same system, we need to set

memalloc_noio to

drivers/target/iscsi/iscsi_target.c +1 -0
+	 * prevent memory allocation deadlocks between target and

initiator.

drivers/target/iscsi/iscsi_target.c +8 -0
+	 */
+	rcu_read_lock();
+	dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+	if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+		loopback = true;
+	rcu_read_unlock();
+
+	if (loopback)

drivers/target/iscsi/iscsi_target.c int iscsi_target_rx_thread(void *arg) · line 4214

Quoted 10 lines
drivers/target/iscsi/iscsi_target.c +5 -0
+		flags = memalloc_noio_save();
+
 	conn->conn_transport->iscsit_get_rx_pdu(conn);
 
+	if (loopback)
+		memalloc_noio_restore(flags);
+
 	if (!signal_pending(current))
 		atomic_set(&conn->transport_failed, 1);
 	iscsit_take_action_for_connection_exit(conn, &conn_freed);

I had mentioned to Mike that this was already tested at a large customer and in our labs and resolved the deadlocks .

Quoted 3 lines

Regards
Laurence Oberman

I hate to nag here but we have a pressing customer issue and are keen to get others to weigh in here.

Regards
Laurence

Thanks
Laurence

#3 lore raw
In reply to David Jeffery

st 8. 2. 2023 v 21:10 odesílatel David Jeffery <[email protected]> napsal:

drivers/target/iscsi/iscsi_target.c int iscsi_target_rx_thread(void *arg) · line 4202

Quoted 12 lines
drivers/target/iscsi/iscsi_target.c +10 -0
+       /*
+        * If the iscsi connection is over a loopback device from using
+        * iscsi and iscsit on the same system, we need to set memalloc_noio to
+        * prevent memory allocation deadlocks between target and initiator.
+        */
+       rcu_read_lock();
+       dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+       if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+               loopback = true;
+       rcu_read_unlock();

Hi Mike,
I tested it, it works. The customer also confirmed that it fixes the deadlock on his setup.

Maurizio

#4 lore raw
Quoted 14 lines · Maurizio Lombardi

st 8. 2. 2023 v 21:10 odesílatel David Jeffery <[email protected]> napsal:

drivers/target/iscsi/iscsi_target.c int iscsi_target_rx_thread(void *arg) · line 4202

Quoted 12 lines
drivers/target/iscsi/iscsi_target.c +10 -0
+       /*
+        * If the iscsi connection is over a loopback device from using
+        * iscsi and iscsit on the same system, we need to set memalloc_noio to
+        * prevent memory allocation deadlocks between target and initiator.
+        */
+       rcu_read_lock();
+       dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+       if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+               loopback = true;
+       rcu_read_unlock();
Quoted 3 lines

Hi Mike,
I tested it, it works. The customer also confirmed that it fixes the deadlock on his setup.

You never responded about why/how it's used in production. Is it some sort of clustering or container or what?

The login related code can still swing back on you if it's run for a relogin. It would happen if we overqueue and a nop timesout because the iscsi recv thread is waiting for backend resources like a request/queue slot, or if management tools disable/enable the tpgt for reconfigs, etc.

#5 lore raw
Quoted 27 lines · Mike Christie
Quoted 17 lines · Maurizio Lombardi

st 8. 2. 2023 v 21:10 odesílatel David Jeffery <[email protected]

Quoted 9 lines

napsal:

(quoted diff) +2 -0
+       /*
+        * If the iscsi connection is over a loopback device from

using
+ * iscsi and iscsit on the same system, we need to set
memalloc_noio to
+ * prevent memory allocation deadlocks between target and
initiator.

drivers/target/iscsi/iscsi_target.c int iscsi_target_rx_thread(void *arg) · line 4202

Quoted 6 lines
drivers/target/iscsi/iscsi_target.c +6 -0
+        */
+       rcu_read_lock();
+       dst = rcu_dereference(conn->sock->sk->sk_dst_cache);
+       if (dst && dst->dev && dst->dev->flags & IFF_LOOPBACK)
+               loopback = true;
+       rcu_read_unlock();
Quoted 4 lines

Hi Mike,
I tested it, it works. The customer also confirmed that it fixes the
deadlock on his setup.

You never responded about why/how it's used in production. Is it some sort
of clustering or container or what?

Quoted 8 lines

The login related code can still swing back on you if it's run for a relogin.
It would happen if we overqueue and a nop timesout because the iscsi recv thread
is waiting for backend resources like a request/queue slot, or if management tools
disable/enable the tpgt for reconfigs, etc.

Hi Mike,
The use case described is as follows:

"This customer moved their on-premise system to the cloud. Their on-premise system runs with two servers and one external storage and uses data mirroring software to mirror data.

When moving to the cloud, customer wanted to implement a data mirror using data mirror software with two instances to reduce the cost of using the cloud infrastructure.
To build a system with two instances, we use iSCSI to mirror data between a local disk on one instance and a local disk on the other instance.
We coexist iSCSI initiator and target so that data mirroring software can access each disk through a unified interface."

Thanks
Laurence