Skip to contentSkip to conversation

net: ipv4: remove erroneous advancement of list pointer

PATCH In review Florian Westphal 4 participants 10 messages Jun 17, 2019 – Jun 26, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover net: ipv4: remove erroneous advancement of list pointer

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list").

While reviewing above patch I found a regression when addresses have a lifetime specified.

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

#1 lore raw

PATCH 1/2 net: ipv4: remove erroneous advancement of list pointer

Causes crash when lifetime expires on an adress as garbage is dereferenced soon after.

This used to look like this:

 for (ifap = &ifa->ifa_dev->ifa_list;
      *ifap != NULL; ifap = &(*ifap)->ifa_next) {
          if (*ifap == ifa) ...

but this was changed to:

struct in_ifaddr *tmp;

ifap = &ifa->ifa_dev->ifa_list;
tmp = rtnl_dereference(*ifap);
while (tmp) {
tmp = rtnl_dereference(tmp->ifa_next); // Bogus

   if (rtnl_dereference(*ifap) == ifa) {
     ...

ifap = &tmp->ifa_next; // Can be NULL
tmp = rtnl_dereference(*ifap); // Dereference
}
}

Remove the bogus assigment/list entry skip.

Fixes 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list")
---
 net/ipv4/devinet.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
net/ipv4/devinet.c +1 -2
index 925dffa915cb..914ccc7f192a 100644
@@ -745,8 +745,7 @@ static void check_lifetime(struct work_struct *work)
 				ifap = &ifa->ifa_dev->ifa_list;
 				tmp = rtnl_dereference(*ifap);
 				while (tmp) {
-					tmp = rtnl_dereference(tmp->ifa_next);
-					if (rtnl_dereference(*ifap) == ifa) {
+					if (tmp == ifa) {
 						inet_del_ifa(ifa->ifa_dev,
 							     ifap, 1);
 						break;
Signature · 1 line
2.21.0
#2 lore raw

PATCH 2/2 selftests: rtnetlink: add addresses with fixed life time

In reply to Florian Westphal

This exercises kernel code path that deal with addresses that have a limited lifetime.

Without previous fix, this triggers following crash on net-next:

 BUG: KASAN: null-ptr-deref in check_lifetime+0x403/0x670
 Read of size 8 at addr 0000000000000010 by task kworker [..]
---
 tools/testing/selftests/net/rtnetlink.sh | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
tools/testing/selftests/net/rtnetlink.sh +21 -0
index b25c9fe019d2..ed606a2e3865 100755
@@ -249,6 +249,26 @@ kci_test_route_get()
 	echo "PASS: route get"
 }
 
+kci_test_addrlft()
+{
+	for i in $(seq 10 100) ;do
+		lft=$(((RANDOM%3) + 1))
+		ip addr add 10.23.11.$i/32 dev "$devdummy" preferred_lft $lft valid_lft $((lft+1))
+		check_err $?
+	done
+
+	sleep 5
+
+	ip addr show dev "$devdummy" | grep "10.23.11."
+	if [ $? -eq 0 ]; then
+		echo "FAIL: preferred_lft addresses remaining"
+		check_err 1
+		return
+	fi
+
+	echo "PASS: preferred_lft addresses have expired"
+}
+
 kci_test_addrlabel()
 {
 	ret=0
@@ -1140,6 +1160,7 @@ kci_test_rtnl()
 
 	kci_test_polrouting
 	kci_test_route_get
+	kci_test_addrlft
 	kci_test_tc
 	kci_test_gre
 	kci_test_gretap
Signature · 1 line
2.21.0
#3 lore raw
In reply to Florian Westphal
Quoted 9 lines · Florian Westphal

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list").

While reviewing above patch I found a regression when addresses have a lifetime specified.

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

Thanks Florian.

Ran, can you please test?

#4 lore raw
Quoted 20 lines

-----Original Message-----
From: Tariq Toukan
Sent: Monday, June 17, 2019 19:16
To: Florian Westphal <[email protected]>; [email protected]
Cc: Ran Rozenstein <[email protected]>; Maor Gottlieb <[email protected]>; [email protected]
Subject: Re: [PATCH net-next 0/2] net: ipv4: remove erroneous advancement of list pointer

Quoted 6 lines · Florian Westphal

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list").

While reviewing above patch I found a regression when addresses have a lifetime specified.

Quoted 7 lines
Quoted 3 lines

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

Thanks Florian.

Ran, can you please test?

Tested, still reproduce.

#5 lore raw
Quoted 11 lines · Ran Rozenstein
Quoted 11 lines · Florian Westphal
Quoted 9 lines

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list").

While reviewing above patch I found a regression when addresses have a lifetime specified.

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

Quoted 5 lines
Quoted 3 lines

Thanks Florian.

Ran, can you please test?

Tested, still reproduce.

Can you be a little more specific? Is there any reproducer?

#6 lore raw
Quoted 25 lines

-----Original Message-----
From: Florian Westphal [mailto:[email protected]]
Sent: Tuesday, June 25, 2019 12:19
To: Ran Rozenstein <[email protected]>
Cc: Tariq Toukan <[email protected]>; Florian Westphal <[email protected]>; [email protected]; Maor Gottlieb <[email protected]>; [email protected]
Subject: Re: [PATCH net-next 0/2] net: ipv4: remove erroneous advancement of list pointer

Quoted 3 lines · Ran Rozenstein
Quoted 3 lines · Florian Westphal
Quoted 2 lines

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for

ifa_list").

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

While reviewing above patch I found a regression when addresses have a lifetime specified.

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Thanks Florian.

Ran, can you please test?

Tested, still reproduce.

Can you be a little more specific? Is there any reproducer?

The test dose stress on the interface by running this 2 commands in loop:

command is: /sbin/ip -f inet addr add $IP/16 brd + dev ens8f1 command is: ifconfig ens8f1 $IP netmask 255.255.0.0

when $IP change every iteration.

It execute every second when we see the reproduce somewhere between 40 to 200 seconds of execution.

Thanks,
Ran

#7 lore raw
Quoted 8 lines · Ran Rozenstein

The test dose stress on the interface by running this 2 commands in loop:

command is: /sbin/ip -f inet addr add $IP/16 brd + dev ens8f1 command is: ifconfig ens8f1 $IP netmask 255.255.0.0

when $IP change every iteration.

It execute every second when we see the reproduce somewhere between 40 to 200 seconds of execution.

I tried this without success:

DEV=dummy0
for j in $(seq 2 254);do
for i in $(seq 2 254);do

    IP="10.$((RANDOM%254)).$((RANDOM%254)).$i"
    ip -f inet addr add $IP/16 brd + dev $DEV
    ifconfig $DEV $IP netmask 255.255.0.0

done
done

I'll let this loop overnight, but so far nothing turned up in dmesg (lockdep/kasan kernel).

#8 lore raw
In reply to Ran Rozenstein
Quoted 8 lines · Ran Rozenstein

The test dose stress on the interface by running this 2 commands in loop:

command is: /sbin/ip -f inet addr add $IP/16 brd + dev ens8f1 command is: ifconfig ens8f1 $IP netmask 255.255.0.0

when $IP change every iteration.

It execute every second when we see the reproduce somewhere between 40 to 200 seconds of execution.

I can reproduce this now, I'll submit a fix tomorrow.

maintainer
#9 lore raw
In reply to Florian Westphal

From: Florian Westphal <[email protected]>
Date: Mon, 17 Jun 2019 16:02:26 +0200

Quoted 8 lines

Tariq reported a soft lockup on net-next that Mellanox was able to bisect to 2638eb8b50cf ("net: ipv4: provide __rcu annotation for ifa_list").

While reviewing above patch I found a regression when addresses have a lifetime specified.

Second patch extends rtnetlink.sh to trigger crash (without first patch applied).

Series applied, thanks Florian.