net/virtio: fix crash when device reconnecting
PATCH 1/1 net/virtio: fix crash when device reconnecting
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. This will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by splitting device initial process and device reinit process. Virt queues won't be freed or realloc in reinit process. Also moved virtio device initialization from configuration to start stage, which can reduce number of reinitialization times.
drivers/net/virtio/virtio_ethdev.c +17 -13
@@ -1670,7 +1670,9 @@ virtio_configure_intr(struct rte_eth_dev *dev)/* reset device and renegotiate features if needed */ static int -virtio_init_device (struct rte_eth_dev *eth_dev,uint64_t req_features) +virtio_init_device (struct rte_eth_dev *eth_dev, +uint64_t req_features, +bool reinit) {struct virtio_hw *hw = eth_dev->data->dev_private;struct virtio_net_config *config;
@@ -1681,7 +1683,7 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)/* Reset the device although not necessary at startup */ vtpci_reset (hw); -if (hw->vqs) { +if (hw->vqs && !reinit) {virtio_dev_free_mbufs (eth_dev);virtio_free_queues (hw); }
@@ -1794,9 +1796,11 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)VLAN_TAG_LEN - hw->vtnet_hdr_size; } - ret =virtio_alloc_queues (eth_dev); -if (ret <0 ) -return ret; +if (!reinit) { + ret =virtio_alloc_queues (eth_dev); +if (ret <0 ) +return ret; + }if (eth_dev->data->dev_conf.intr_conf.rxq) {if (virtio_configure_intr (eth_dev) <0 ) {
@@ -1925,7 +1929,8 @@ eth_virtio_dev_init(struct rte_eth_dev *eth_dev)rte_spinlock_init (&hw->state_lock);/* reset device and negotiate default features */ - ret =virtio_init_device (eth_dev,VIRTIO_PMD_DEFAULT_GUEST_FEATURES ); + ret =virtio_init_device (eth_dev,VIRTIO_PMD_DEFAULT_GUEST_FEATURES , + false);if (ret <0 ) goto err_virtio_init;
@@ -2091,12 +2096,6 @@ virtio_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } -if (dev->data->dev_conf.intr_conf.rxq) { - ret =virtio_init_device (dev, hw->req_guest_features); -if (ret <0 ) -return ret; - } -if (rxmode->max_rx_pkt_len > hw->max_mtu + ether_hdr_len) req_features &= ~(1ULL <<VIRTIO_NET_F_MTU );
@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);if (ret <0 )return ret; }
@@ -2235,6 +2234,11 @@ virtio_dev_start(struct rte_eth_dev *dev)struct virtio_hw *hw = dev->data->dev_private;int ret; +/* reinit the device */ + ret =virtio_init_device (dev, hw->req_guest_features, true); +if (ret <0 ) +return ret; +/* Finish the initialization of the queues */ for (i =0 ; i < dev->data->nb_rx_queues; i++) { ret =virtio_dev_rx_queue_setup_finish (dev, i);
Signature · 1 line
2.17.1
Quoted 110 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Marvin Liu
Sent: 2020年4月14日 20:56
To: [email protected]; Wang, Zhihong <[email protected]>; Ye, Xiaolong <[email protected]>
Cc: [email protected]; Ding, Xuan <[email protected]>; Liu, Yong <[email protected]>
Subject: [dpdk-dev] [PATCH] net/virtio: fix crash when device reconnectingWhen doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. This will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by splitting device initial process and device reinit process. Virt queues won't be freed or realloc in reinit process. Also moved virtio device initialization from configuration to start stage, which can reduce number of reinitialization times.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")Signed-off-by Marvin Liu <[email protected]>drivers/net/virtio/virtio_ethdev.c +4
-1@@ -1670,7 +1670,9 @@ virtio_configure_intr(struct rte_eth_dev *dev) /* reset device and renegotiate features if needed */ static int -virtio_init_device (struct rte_eth_dev *eth_dev,uint64_t req_features) +virtio_init_device(struct rte_eth_dev *eth_dev, + uint64_t req_features, + bool reinit) {struct virtio_hw *hw = eth_dev->data->dev_private;struct virtio_net_config *config;@@ -1681,7 +1683,7 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)/* Reset the device although not necessary at startup */ vtpci_reset (hw ); -if (hw->vqs) { +if (hw->vqs && !reinit) {virtio_dev_free_mbufs (eth_dev);virtio_free_queues (hw);drivers/net/virtio/virtio_ethdev.c +5
-3}@@ -1794,9 +1796,11 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features) VLAN_TAG_LEN - hw->vtnet_hdr_size; } - ret =virtio_alloc_queues (eth_dev); -if (ret <0 ) -return ret; +if (!reinit) { + ret =virtio_alloc_queues (eth_dev); +if (ret <0 ) +return ret; + }if (eth_dev->data->dev_conf.intr_conf.rxq) {if (
virtio_configure_intr(eth_dev)< 0) { @@ -1925,7 +1929,8
@@eth_virtio_dev_init(struct rte_eth_dev *eth_dev)drivers/net/virtio/virtio_ethdev.c +0
-1rte_spinlock_init (&hw->state_lock);/* reset device and negotiate default features */ - ret = virtio_init_device(eth_dev,
VIRTIO_PMD_DEFAULT_GUEST_FEATURES);drivers/net/virtio/virtio_ethdev.c +1
-0+ ret = virtio_init_device(eth_dev,
VIRTIO_PMD_DEFAULT_GUEST_FEATURES,drivers/net/virtio/virtio_ethdev.c +7
-8+ false);if (ret <0 ) goto err_virtio_init;@@ -2091,12 +2096,6 @@ virtio_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } -if (dev->data->dev_conf.intr_conf.rxq) { - ret =virtio_init_device (dev, hw->req_guest_features); -if (ret <0 ) -return ret; - } -if (rxmode->max_rx_pkt_len > hw->max_mtu + ether_hdr_len) req_features &= ~(1ULL <<VIRTIO_NET_F_MTU );@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);if (ret <0 )return ret; }@@ -2235,6 +2234,11 @@ virtio_dev_start(struct rte_eth_dev *dev)struct virtio_hw *hw = dev->data->dev_private;int ret; +/* reinit the device */ + ret =virtio_init_device (dev, hw->req_guest_features, true); +if (ret <0 ) +return ret; +/* Finish the initialization of the queues */ for (i =0 ; i < dev->data->nb_rx_queues; i++) { ret =virtio_dev_rx_queue_setup_finish (dev, i); --2.17.1
Hi, Marvin
Quoted 10 lines · Marvin Liu
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. This will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by splitting device initial process and device reinit process. Virt queues won't be freed or realloc in reinit process. Also moved virtio device initialization from configuration to start stage, which can reduce number of reinitialization times.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")
I think it also needs to cc stable.
Quoted 66 lines
Signed-off-by Marvin Liu <[email protected]>drivers/net/virtio/virtio_ethdev.c +0
-0@@ -1670,7 +1670,9 @@ virtio_configure_intr(struct rte_eth_dev *dev)/* reset device and renegotiate features if needed */ static int
drivers/net/virtio/virtio_ethdev.c +3
-1-virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features) +virtio_init_device(struct rte_eth_dev *eth_dev, + uint64_t req_features, + bool reinit){ struct virtio_hw *hw = eth_dev->data->dev_private; struct virtio_net_config *config;drivers/net/virtio/virtio_ethdev.c +8
-11@@ -1681,7 +1683,7 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)/* Reset the device although not necessary at startup */ vtpci_reset (hw); -if (hw->vqs) { +if (hw->vqs && !reinit) {virtio_dev_free_mbufs (eth_dev);virtio_free_queues (hw); }@@ -1794,9 +1796,11 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)VLAN_TAG_LEN - hw->vtnet_hdr_size; } - ret =virtio_alloc_queues (eth_dev); -if (ret <0 ) -return ret; +if (!reinit) { + ret =virtio_alloc_queues (eth_dev); +if (ret <0 ) +return ret; + }if (eth_dev->data->dev_conf.intr_conf.rxq) {if (virtio_configure_intr (eth_dev) <0 ) {@@ -1925,7 +1929,8 @@ eth_virtio_dev_init(struct rte_eth_dev *eth_dev)rte_spinlock_init (&hw->state_lock);/* reset device and negotiate default features */ - ret =virtio_init_device (eth_dev,VIRTIO_PMD_DEFAULT_GUEST_FEATURES ); + ret =virtio_init_device (eth_dev,VIRTIO_PMD_DEFAULT_GUEST_FEATURES , + false);if (ret <0 ) goto err_virtio_init;@@ -2091,12 +2096,6 @@ virtio_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } -if (dev->data->dev_conf.intr_conf.rxq) { - ret =virtio_init_device (dev, hw->req_guest_features); -if (ret <0 ) -return ret; - } -if (rxmode->max_rx_pkt_len > hw->max_mtu + ether_hdr_len) req_features &= ~(1ULL <<VIRTIO_NET_F_MTU );
drivers/net/virtio/virtio_ethdev.c virtio_dev_configure(struct rte_eth_dev *dev) · line 2122
Quoted 5 lines
drivers/net/virtio/virtio_ethdev.c +1
-1@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);
Why do we need to change virtio_init_device to virtio_negotiate_features here?
Thanks,
Xiaolong
Quoted 18 lines
drivers/net/virtio/virtio_ethdev.c +5
-0if (ret <0 )return ret; }@@ -2235,6 +2234,11 @@ virtio_dev_start(struct rte_eth_dev *dev)struct virtio_hw *hw = dev->data->dev_private;int ret; +/* reinit the device */ + ret =virtio_init_device (dev, hw->req_guest_features, true); +if (ret <0 ) +return ret; +/* Finish the initialization of the queues */ for (i =0 ; i < dev->data->nb_rx_queues; i++) { ret =virtio_dev_rx_queue_setup_finish (dev, i);Signature · 1 line
2.17.1
Quoted 18 lines
-----Original Message-----
From: Ye, Xiaolong <[email protected]>
Sent: Wednesday, April 15, 2020 3:24 PM
To: Liu, Yong <[email protected]>
Cc: [email protected]; Wang, Zhihong <[email protected]>; [email protected]; Ding, Xuan <[email protected]>
Subject: Re: [PATCH] net/virtio: fix crash when device reconnectingHi, Marvin
Quoted 4 lines · Marvin Liu
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. This will cause issue because queues have been freed in the beginning of device initialization.
Quoted 9 lines
Quoted 6 lines
Fix this issue by splitting device initial process and device reinit process. Virt queues won't be freed or realloc in reinit process. Also moved virtio device initialization from configuration to start stage, which can reduce number of reinitialization times.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")I think it also needs to cc stable.
Hi Xiaolong,
Packed ring server mode fix was merged in 20.02 release. So there's no stable branch for it.
Thanks,
Marvin
Quoted 101 lines
Review
Quoted 4 lines
Signed-off-by Marvin Liu <[email protected]>(quoted diff) +0
-0
b/drivers/net/virtio/virtio_ethdev.cdrivers/net/virtio/virtio_ethdev.c virtio_init_device(struct rte_eth_dev *eth_dev,
Quoted 15 lines
index
21570e5cf..8c84bfe91100644drivers/net/virtio/virtio_ethdev.c +0
-0@@ -1670,7 +1670,9 @@ virtio_configure_intr(struct rte_eth_dev *dev)/* reset device and renegotiate features if needed */ static int
drivers/net/virtio/virtio_ethdev.c +3
-1-virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features) +virtio_init_device(struct rte_eth_dev *eth_dev, + uint64_t req_features, + bool reinit){ struct virtio_hw *hw = eth_dev->data->dev_private; struct virtio_net_config *config;drivers/net/virtio/virtio_ethdev.c +0
-0@@ -1681,7 +1683,7 @@ virtio_init_device(struct rte_eth_dev *eth_dev,
uint64_treq_features)drivers/net/virtio/virtio_ethdev.c virtio_init_device(struct rte_eth_dev *eth_dev,
Quoted 9 lines
drivers/net/virtio/virtio_ethdev.c +1
-1/* Reset the device although not necessary at startup */ vtpci_reset (hw); -if (hw->vqs) { +if (hw->vqs && !reinit) {virtio_dev_free_mbufs (eth_dev);virtio_free_queues (hw); }@@ -1794,9 +1796,11 @@ virtio_init_device(struct rte_eth_dev *eth_dev,
uint64_treq_features)Quoted 15 lines
drivers/net/virtio/virtio_ethdev.c +5
-3VLAN_TAG_LEN - hw->vtnet_hdr_size; } - ret =virtio_alloc_queues (eth_dev); -if (ret <0 ) -return ret; +if (!reinit) { + ret =virtio_alloc_queues (eth_dev); +if (ret <0 ) +return ret; + }if (eth_dev->data->dev_conf.intr_conf.rxq) {if (virtio_configure_intr (eth_dev) <0 ) {drivers/net/virtio/virtio_ethdev.c eth_virtio_dev_init(struct rte_eth_dev *eth_dev) · line 1928
Quoted 4 lines
drivers/net/virtio/virtio_ethdev.c +0
-1@@ -1925,7 +1929,8 @@ eth_virtio_dev_init(struct rte_eth_dev *eth_dev)rte_spinlock_init (&hw->state_lock);/* reset device and negotiate default features */ - ret = virtio_init_device(eth_dev,
VIRTIO_PMD_DEFAULT_GUEST_FEATURES);drivers/net/virtio/virtio_ethdev.c eth_virtio_dev_init(struct rte_eth_dev *eth_dev) · line 1932
Quoted 1 line
drivers/net/virtio/virtio_ethdev.c +1
-0+ ret = virtio_init_device(eth_dev,
VIRTIO_PMD_DEFAULT_GUEST_FEATURES,Quoted 18 lines
drivers/net/virtio/virtio_ethdev.c +1
-6+ false);if (ret <0 ) goto err_virtio_init;@@ -2091,12 +2096,6 @@ virtio_dev_configure(struct rte_eth_dev *dev)return -EINVAL ; } -if (dev->data->dev_conf.intr_conf.rxq) { - ret =virtio_init_device (dev, hw->req_guest_features); -if (ret <0 ) -return ret; - } -if (rxmode->max_rx_pkt_len > hw->max_mtu + ether_hdr_len) req_features &= ~(1ULL <<VIRTIO_NET_F_MTU );drivers/net/virtio/virtio_ethdev.c virtio_dev_configure(struct rte_eth_dev *dev) · line 2122
Quoted 5 lines
drivers/net/virtio/virtio_ethdev.c +1
-1@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);Why do we need to change
virtio_init_devicetovirtio_negotiate_featureshere?Thanks,
XiaolongQuoted 18 lines
drivers/net/virtio/virtio_ethdev.c +5
-1if (ret <0 )return ret; }@@ -2235,6 +2234,11 @@ virtio_dev_start(struct rte_eth_dev *dev)struct virtio_hw *hw = dev->data->dev_private;int ret; +/* reinit the device */ + ret =virtio_init_device (dev, hw->req_guest_features, true); +if (ret <0 ) +return ret; +/* Finish the initialization of the queues */ for (i =0 ; i < dev->data->nb_rx_queues; i++) { ret =virtio_dev_rx_queue_setup_finish (dev, i); --2.17.1
Hi Marvin,
Quoted 8 lines · Yong
drivers/net/virtio/virtio_ethdev.c virtio_dev_configure(struct rte_eth_dev *dev) · line 2122
Quoted 6 lines
drivers/net/virtio/virtio_ethdev.c +1
-1@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);Why do we need to change
virtio_init_devicetovirtio_negotiate_featureshere?
You missed to reply to that question from Xiaolong.
Regards,
Maxime
Sorry for missed this question. The purpose of change function is to skip device initialization which is not needed in configuration stage.
When features not matched, can just do feature negotiation in configuration stage and do related actions when virtio device start.
Regards,
Marvin
Quoted 25 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, April 17, 2020 11:18 PM
To: Liu, Yong <[email protected]>; Ye, Xiaolong <[email protected]>
Cc: Wang, Zhihong <[email protected]>; [email protected]; Ding, Xuan <[email protected]>
Subject: Re: [PATCH] net/virtio: fix crash when device reconnectingHi Marvin,
Quoted 8 lines · Yong
drivers/net/virtio/virtio_ethdev.c virtio_dev_configure(struct rte_eth_dev *dev) · line 2122
Quoted 6 lines
drivers/net/virtio/virtio_ethdev.c +1
-1@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);Why do we need to change
virtio_init_devicetovirtio_negotiate_featureshere?You missed to reply to that question from Xiaolong.
Regards,
Maxime
Quoted 2 lines · Yong
Sorry for missed this question. The purpose of change function is to skip device initialization which is not needed in configuration stage.
When features not matched, can just do feature negotiation in configuration stage and do related actions when virtio device start.
Thanks for the explanation, then seems we needs to adjust the comment above accordingly.
Thanks,
Xiaolong
Quoted 30 lines
Regards,
MarvinQuoted 25 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, April 17, 2020 11:18 PM
To: Liu, Yong <[email protected]>; Ye, Xiaolong <[email protected]>
Cc: Wang, Zhihong <[email protected]>; [email protected]; Ding, Xuan <[email protected]>
Subject: Re: [PATCH] net/virtio: fix crash when device reconnectingHi Marvin,
Quoted 8 lines · Yong
drivers/net/virtio/virtio_ethdev.c virtio_dev_configure(struct rte_eth_dev *dev) · line 2122
Quoted 6 lines
drivers/net/virtio/virtio_ethdev.c +1
-1@@ -2120,7 +2119,7 @@ virtio_dev_configure(struct rte_eth_dev *dev)/* if request features changed, reinit the device */ if (req_features != hw->req_guest_features) { - ret =virtio_init_device (dev, req_features); + ret =virtio_negotiate_features (hw, req_features);Why do we need to change
virtio_init_devicetovirtio_negotiate_featureshere?You missed to reply to that question from Xiaolong.
Regards,
Maxime
PATCH v2 1/1 net/virtio: fix crash when device reconnecting
In reply to Marvin LiuWhen doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. It will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by checking whether device has been initialized before reset. If device hasn't been initialized, there's no need to reset queues.
drivers/net/virtio/virtio_user_ethdev.c +2 -1
@@ -89,7 +89,8 @@ virtio_user_server_reconnect(struct virtio_user_dev *dev) dev->features &= dev->device_features;/* For packed ring, resetting queues is required in reconnection. */ -if (vtpci_packed_queue (hw)) { +if (vtpci_packed_queue (hw) && + (vtpci_get_status (hw) &VIRTIO_CONFIG_STATUS_DRIVER_OK )) {PMD_INIT_LOG (NOTICE ,"Packets on the fly will be dropped" " when packed ring reconnecting." );virtio_user_reset_queues_packed (eth_dev);
Signature · 1 line
2.17.1
Quoted 11 lines · Marvin Liu
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. It will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by checking whether device has been initialized before reset. If device hasn't been initialized, there's no need to reset queues.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")Signed-off-by Marvin Liu <[email protected]>
Thanks,
Maxime
Maxime Coquelin
AppliedQuoted 11 lines · Marvin Liu
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. It will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by checking whether device has been initialized before reset. If device hasn't been initialized, there's no need to reset queues.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")Signed-off-by Marvin Liu <[email protected]>
Applied to dpdk-next-virtio/master
Thanks,
Maxime
Wang, Yinan
Tested-byQuoted 28 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Maxime Coquelin
Sent: 2020年5月7日 22:18
To: Liu, Yong <[email protected]>; Ye, Xiaolong <[email protected]>; Wang, Zhihong <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: fix crash when device reconnectingQuoted 11 lines · Marvin Liu
When doing virtio device initialization, virtqueues will be reset in server mode if ring type is packed. It will cause issue because queues have been freed in the beginning of device initialization.
Fix this issue by checking whether device has been initialized before reset. If device hasn't been initialized, there's no need to reset queues.
Fixes 6ebbf4109f35 ("net/virtio-user: fix packed ring server mode")Signed-off-by Marvin Liu <[email protected]>Applied to dpdk-next-virtio/master
Thanks,
Maxime