Skip to contentSkip to conversation

drivers/net/netfront: Fix NULL sring after live migration

PATCH In review Lin Liu 2 participants 3 messages Nov 29, 2022 – Dec 1, 2022

Subsystems (3)
lore
Share & tools
lore

Lin Liu

author
#0 lore raw

PATCH 1/1 drivers/net/netfront: Fix NULL sring after live migration

A NAPI is setup for each network sring to poll data to kernel The sring with source host is destroyed before live migration and new sring with target host is setup after live migration. The NAPI for the old sring is not deleted until setup new sring with target host after migration. With busy_poll/busy_read enabled, the NAPI can be polled before got deleted when resume VM.

[50116.602938] BUG: unable to handle kernel NULL pointer dereference at
0000000000000008
[50116.603047] IP: xennet_poll+0xae/0xd20
[50116.603090] PGD 0 P4D 0
[50116.603118] Oops: 0000 [#1] SMP PTI
[50116.604624] Call Trace:
[50116.604674]  ? finish_task_switch+0x71/0x230
[50116.604745]  ? timerqueue_del+0x1d/0x40
[50116.604807]  ? hrtimer_try_to_cancel+0xb5/0x110
[50116.604882]  ? xennet_alloc_rx_buffers+0x2a0/0x2a0
[50116.604958]  napi_busy_loop+0xdb/0x270
[50116.605017]  sock_poll+0x87/0x90
[50116.605066]  do_sys_poll+0x26f/0x580
[50116.605125]  ? tracing_map_insert+0x1d4/0x2f0
[50116.605196]  ? event_hist_trigger+0x14a/0x260
...
[50116.613598]  ? finish_task_switch+0x71/0x230
[50116.614131]  ? __schedule+0x256/0x890
[50116.614640]  ? recalc_sigpending+0x1b/0x50
[50116.615144]  ? xen_sched_clock+0x15/0x20
[50116.615643]  ? __rb_reserve_next+0x12d/0x140
[50116.616138]  ? ring_buffer_lock_reserve+0x123/0x3d0
[50116.616634]  ? event_triggers_call+0x87/0xb0
[50116.617138]  ? trace_event_buffer_commit+0x1c4/0x210
[50116.617625]  ? xen_clocksource_get_cycles+0x15/0x20
[50116.618112]  ? ktime_get_ts64+0x51/0xf0
[50116.618578]  SyS_ppoll+0x160/0x1a0
[50116.619029]  ? SyS_ppoll+0x160/0x1a0
[50116.619475]  do_syscall_64+0x73/0x130
[50116.619901]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
...
[50116.806230] RIP: xennet_poll+0xae/0xd20 RSP: ffffb4f041933900
[50116.806772] CR2: 0000000000000008
[50116.807337] ---[ end trace f8601785b354351c ]---

xen frontend should remove the NAPIs for the old srings before live migration as the bond srings are destroyed

There is a tiny window between the srings are set to NULL and the NAPIs are disabled, It is safe as the NAPI threads are still frozen at that time

---
 drivers/net/xen-netfront.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/xen-netfront.c +6 -0
index 9af2b027c19c..dc404e05970c 100644
@@ -1862,6 +1862,12 @@ static int netfront_resume(struct xenbus_device *dev)
 	netif_tx_unlock_bh(info->netdev);
 
 	xennet_disconnect_backend(info);
+
+	rtnl_lock();
+	if (info->queues)
+		xennet_destroy_queues(info);
+	rtnl_unlock();
+
 	return 0;
 }
 
Signature · 1 line
2.17.1
maintainer
#1 lore raw
Quoted 7 lines · On Tue, 29 Nov 2022 06:17:02 +0000 Lin Liu wrote:

A NAPI is setup for each network sring to poll data to kernel The sring with source host is destroyed before live migration and new sring with target host is setup after live migration. The NAPI for the old sring is not deleted until setup new sring with target host after migration. With busy_poll/busy_read enabled, the NAPI can be polled before got deleted when resume VM.

Quoted 15 lines
[50116.602938] BUG: unable to handle kernel NULL pointer dereference at
0000000000000008
[50116.603047] IP: xennet_poll+0xae/0xd20
[50116.603090] PGD 0 P4D 0
[50116.603118] Oops: 0000 [#1] SMP PTI
[50116.604624] Call Trace:
[50116.604674]  ? finish_task_switch+0x71/0x230
[50116.604745]  ? timerqueue_del+0x1d/0x40
[50116.604807]  ? hrtimer_try_to_cancel+0xb5/0x110
[50116.604882]  ? xennet_alloc_rx_buffers+0x2a0/0x2a0
[50116.604958]  napi_busy_loop+0xdb/0x270
[50116.605017]  sock_poll+0x87/0x90
[50116.605066]  do_sys_poll+0x26f/0x580
[50116.605125]  ? tracing_map_insert+0x1d4/0x2f0
[50116.605196]  ? event_hist_trigger+0x14a/0x260

You can trim all the ' ? ' entries from the stack trace, and the time stamps, FWIW. Makes it easier to read.

Quoted 19 lines
[50116.613598]  ? finish_task_switch+0x71/0x230
[50116.614131]  ? __schedule+0x256/0x890
[50116.614640]  ? recalc_sigpending+0x1b/0x50
[50116.615144]  ? xen_sched_clock+0x15/0x20
[50116.615643]  ? __rb_reserve_next+0x12d/0x140
[50116.616138]  ? ring_buffer_lock_reserve+0x123/0x3d0
[50116.616634]  ? event_triggers_call+0x87/0xb0
[50116.617138]  ? trace_event_buffer_commit+0x1c4/0x210
[50116.617625]  ? xen_clocksource_get_cycles+0x15/0x20
[50116.618112]  ? ktime_get_ts64+0x51/0xf0
[50116.618578]  SyS_ppoll+0x160/0x1a0
[50116.619029]  ? SyS_ppoll+0x160/0x1a0
[50116.619475]  do_syscall_64+0x73/0x130
[50116.619901]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
...
[50116.806230] RIP: xennet_poll+0xae/0xd20 RSP: ffffb4f041933900
[50116.806772] CR2: 0000000000000008
[50116.807337] ---[ end trace f8601785b354351c ]---
Quoted 7 lines

xen frontend should remove the NAPIs for the old srings before live migration as the bond srings are destroyed

There is a tiny window between the srings are set to NULL and the NAPIs are disabled, It is safe as the NAPI threads are still frozen at that time

Since this is a fix please add a Fixes tag, and add [PATCH net] to the subject.

drivers/net/xen-netfront.c static int netfront_resume(struct xenbus_device *dev) · line 1869

Quoted 13 lines
drivers/net/xen-netfront.c +5 -0
index 9af2b027c19c..dc404e05970c 100644
@@ -1862,6 +1862,12 @@ static int netfront_resume(struct xenbus_device *dev)
 	netif_tx_unlock_bh(info->netdev);
 
 	xennet_disconnect_backend(info);
+
+	rtnl_lock();
+	if (info->queues)
+		xennet_destroy_queues(info);
+	rtnl_unlock();

Now all callers of xennet_disconnect_backend() destroy queues soon after, can we just move the destroy queues into disconnect ?

Quoted 3 lines
drivers/net/xen-netfront.c +0 -0
 	return 0;
 }
 
#2 lore raw
Quoted 26 lines

[CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.

Quoted 7 lines · On Tue, 29 Nov 2022 06:17:02 +0000 Lin Liu wrote:

A NAPI is setup for each network sring to poll data to kernel The sring with source host is destroyed before live migration and new sring with target host is setup after live migration. The NAPI for the old sring is not deleted until setup new sring with target host after migration. With busy_poll/busy_read enabled, the NAPI can be polled before got deleted when resume VM.

Quoted 15 lines
[50116.602938] BUG: unable to handle kernel NULL pointer dereference at
0000000000000008
[50116.603047] IP: xennet_poll+0xae/0xd20
[50116.603090] PGD 0 P4D 0
[50116.603118] Oops: 0000 [#1] SMP PTI
[50116.604624] Call Trace:
[50116.604674]  ? finish_task_switch+0x71/0x230
[50116.604745]  ? timerqueue_del+0x1d/0x40
[50116.604807]  ? hrtimer_try_to_cancel+0xb5/0x110
[50116.604882]  ? xennet_alloc_rx_buffers+0x2a0/0x2a0
[50116.604958]  napi_busy_loop+0xdb/0x270
[50116.605017]  sock_poll+0x87/0x90
[50116.605066]  do_sys_poll+0x26f/0x580
[50116.605125]  ? tracing_map_insert+0x1d4/0x2f0
[50116.605196]  ? event_hist_trigger+0x14a/0x260
Quoted 2 lines

You can trim all the ' ? ' entries from the stack trace, and the time stamps, FWIW. Makes it easier to read.

Sure, will do in next version

Quoted 22 lines
Quoted 22 lines
[50116.613598]  ? finish_task_switch+0x71/0x230
[50116.614131]  ? __schedule+0x256/0x890
[50116.614640]  ? recalc_sigpending+0x1b/0x50
[50116.615144]  ? xen_sched_clock+0x15/0x20
[50116.615643]  ? __rb_reserve_next+0x12d/0x140
[50116.616138]  ? ring_buffer_lock_reserve+0x123/0x3d0
[50116.616634]  ? event_triggers_call+0x87/0xb0
[50116.617138]  ? trace_event_buffer_commit+0x1c4/0x210
[50116.617625]  ? xen_clocksource_get_cycles+0x15/0x20
[50116.618112]  ? ktime_get_ts64+0x51/0xf0
[50116.618578]  SyS_ppoll+0x160/0x1a0
[50116.619029]  ? SyS_ppoll+0x160/0x1a0
[50116.619475]  do_syscall_64+0x73/0x130
[50116.619901]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
...
[50116.806230] RIP: xennet_poll+0xae/0xd20 RSP: ffffb4f041933900
[50116.806772] CR2: 0000000000000008
[50116.807337] ---[ end trace f8601785b354351c ]---

xen frontend should remove the NAPIs for the old srings before live migration as the bond srings are destroyed

Quoted 8 lines
Quoted 4 lines

There is a tiny window between the srings are set to NULL and the NAPIs are disabled, It is safe as the NAPI threads are still frozen at that time

Since this is a fix please add a Fixes tag, and add [PATCH net] to the subject.

Will do in next version

Quoted 7 lines
Quoted 7 lines
drivers/net/xen-netfront.c +0 -0
index 9af2b027c19c..dc404e05970c 100644
@@ -1862,6 +1862,12 @@ static int netfront_resume(struct xenbus_device *dev)
       netif_tx_unlock_bh(info->netdev);
 
Quoted 9 lines

drivers/net/xen-netfront.c static int netfront_resume(struct xenbus_device *dev) · line 1869

Quoted 6 lines
drivers/net/xen-netfront.c +5 -0
       xennet_disconnect_backend(info);
+
+     rtnl_lock();
+     if (info->queues)
+             xennet_destroy_queues(info);
+     rtnl_unlock();

Now all callers of xennet_disconnect_backend() destroy queues soon after, can we just move the destroy queues into disconnect ?

After the sring is destroyed, the queue and the bond NAPI should also be destroyed, so Yes, destroy queues can be part of xennet_disconnect_backend, However, some caller of xennet_disconnect_backend hold rtnl_lock while some others not, I think it is simpler to keep seperate.

Quoted 4 lines
Quoted 4 lines
drivers/net/xen-netfront.c +0 -0
       return 0;
 }