bad pages when up/down network cable

DISCUSSION Xishi Qiu 2 participants 5 messages Jul 14, 2015 – Jul 15, 2015 1 message not in the archive Memory management

lore

Share & tools

lore

Xishi Qiu

author

2015-07-14 08:32 UTC #0 lore raw

1、the host directly link to the storage device，by intel ixgbe NIC; between them, no switch or router.
2、the nic of the storage device suddenly become unused and then OK after a little time, this happened frequency.
3、the host printk a lot of message like these:

The kernel is SUSE 3.0.13, use slab, and the following log shows the page still have PG_slab when free_pages(). Does anyone have seen the problem?

Jul  9 11:31:36 root kernel: [1042291.977565] BUG: Bad page state in process swapper  pfn:00bf2
Jul  9 11:31:36 root kernel: [1042291.977568] page:ffffea0000029cf0 count:0 mapcount:0 mapping:          (null) index:0x7f6d4f500
Jul  9 11:31:36 root kernel: [1042291.977571] page flags: 0x40000000000100(slab)  // here is the reason
Jul  9 11:31:36 root kernel: [1042291.977574] Pid: 0, comm: swapper Tainted: G    B       X 3.0.13-0.27-default #1
Jul  9 11:31:36 root kernel: [1042291.977577] Call Trace:
Jul  9 11:31:36 root kernel: [1042291.977583]  [<ffffffff810048b5>] dump_trace+0x75/0x300
Jul  9 11:31:36 root kernel: [1042291.977639]  [<ffffffff8143ea0f>] dump_stack+0x69/0x6f
Jul  9 11:31:36 root kernel: [1042291.977644]  [<ffffffff810f53a1>] bad_page+0xb1/0x120
Jul  9 11:31:37 root kernel: [1042291.977649]  [<ffffffff810f5926>] free_pages_prepare+0xe6/0x110
Jul  9 11:31:37 root kernel: [1042291.977654]  [<ffffffff810f9259>] free_hot_cold_page+0x49/0x1f0
Jul  9 11:31:37 root kernel: [1042291.977660]  [<ffffffff8137a3b4>] skb_release_data+0xb4/0xe0
Jul  9 11:31:37 root kernel: [1042291.977665]  [<ffffffff81379e79>] __kfree_skb+0x9/0x90
Jul  9 11:31:37 root kernel: [1042291.977676]  [<ffffffffa02784a9>] ixgbe_clean_tx_irq+0xa9/0x480 [ixgbe]
Jul  9 11:31:37 root kernel: [1042291.977693]  [<ffffffffa02788cb>] ixgbe_poll+0x4b/0x1a0 [ixgbe]
Jul  9 11:31:37 root kernel: [1042291.977705]  [<ffffffff81389c3a>] net_rx_action+0x10a/0x2c0
Jul  9 11:31:37 root kernel: [1042291.977711]  [<ffffffff81060a1f>] __do_softirq+0xef/0x220
Jul  9 11:31:37 root kernel: [1042291.977716]  [<ffffffff8144a8bc>] call_softirq+0x1c/0x30
Jul  9 11:31:37 root kernel: [1042291.978974] DWARF2 unwinder stuck at call_softirq+0x1c/0x30

Thanks,
Xishi Qiu

A message that is not in the archive (sent off-list or not yet received): CAD=hENdLy_K6LFE3Cm2nfxxVOhWRZWiJmKX5-EKdoBtnWP3MJQ@mail.gmail.com

Xishi Qiu

author

2015-07-14 09:28 UTC #2 lore raw

Quoted 2 lines · zhuyj

Do you use the default ixgbe driver? or the ixgbe driver is modified by you?

Yesi 1/4 ?no modify.

Thanks,
Xishi Qiu

Quoted 38 lines · On Tue, Jul 14, 2015 at 4:31 PM, Xishi Qiu <[email protected]…

    1a??the host directly link to the storage devicei 1/4 ?by intel ixgbe NIC;
    between them, no switch or router.
    2a??the nic of the storage device suddenly become unused and then OK
    after a little time, this happened frequency.
    3a??the host printk a lot of message like these:

    The kernel is SUSE 3.0.13, use slab, and the following log shows the
    page still have PG_slab when free_pages(). Does anyone have seen the
    problem?

    Jul  9 11:31:36 root kernel: [1042291.977565] BUG: Bad page state in process swapper  pfn:00bf2
    Jul  9 11:31:36 root kernel: [1042291.977568] page:ffffea0000029cf0 count:0 mapcount:0 mapping:          (null) index:0x7f6d4f500
    Jul  9 11:31:36 root kernel: [1042291.977571] page flags: 0x40000000000100(slab)  // here is the reason
    Jul  9 11:31:36 root kernel: [1042291.977574] Pid: 0, comm: swapper Tainted: G    B       X 3.0.13-0.27-default #1
    Jul  9 11:31:36 root kernel: [1042291.977577] Call Trace:
    Jul  9 11:31:36 root kernel: [1042291.977583]  [<ffffffff810048b5>] dump_trace+0x75/0x300
    Jul  9 11:31:36 root kernel: [1042291.977639]  [<ffffffff8143ea0f>] dump_stack+0x69/0x6f
    Jul  9 11:31:36 root kernel: [1042291.977644]  [<ffffffff810f53a1>] bad_page+0xb1/0x120
    Jul  9 11:31:37 root kernel: [1042291.977649]  [<ffffffff810f5926>] free_pages_prepare+0xe6/0x110
    Jul  9 11:31:37 root kernel: [1042291.977654]  [<ffffffff810f9259>] free_hot_cold_page+0x49/0x1f0
    Jul  9 11:31:37 root kernel: [1042291.977660]  [<ffffffff8137a3b4>] skb_release_data+0xb4/0xe0
    Jul  9 11:31:37 root kernel: [1042291.977665]  [<ffffffff81379e79>] __kfree_skb+0x9/0x90
    Jul  9 11:31:37 root kernel: [1042291.977676]  [<ffffffffa02784a9>] ixgbe_clean_tx_irq+0xa9/0x480 [ixgbe]
    Jul  9 11:31:37 root kernel: [1042291.977693]  [<ffffffffa02788cb>] ixgbe_poll+0x4b/0x1a0 [ixgbe]
    Jul  9 11:31:37 root kernel: [1042291.977705]  [<ffffffff81389c3a>] net_rx_action+0x10a/0x2c0
    Jul  9 11:31:37 root kernel: [1042291.977711]  [<ffffffff81060a1f>] __do_softirq+0xef/0x220
    Jul  9 11:31:37 root kernel: [1042291.977716]  [<ffffffff8144a8bc>] call_softirq+0x1c/0x30
    Jul  9 11:31:37 root kernel: [1042291.978974] DWARF2 unwinder stuck at call_softirq+0x1c/0x30

    Thanks,
    Xishi Qiu


    ------------------------------------------------------------------------------
    Don't Limit Your Business. Reach for the Cloud.
    GigeNET's Cloud Solutions provide you with the tools and support that

Quoted 10 lines

    you need to offload your IT needs and focus on growing your business.
    Configured For All Businesses. Start Your Cloud Today.
    https://www.gigenetcloud.com/
    _______________________________________________
    E1000-devel mailing list
    [email protected] <mailto:[email protected]>
    https://lists.sourceforge.net/lists/listinfo/e1000-devel
    To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Xishi Qiu

author

2015-07-14 09:37 UTC #3 lore raw

Quoted 7 lines · Xishi Qiu

Quoted 2 lines · zhuyj
Do you use the default ixgbe driver? or the ixgbe driver is modified by you?
Yesi 1/4 ?no modify.

Sorry, it is modified by us...

the driver come from inteli 1/4 ?the infoi 1/4 ?
root:~ # ethtool -i p2p2
driver: ixgbe
version: 3.9.16-NAPI
firmware-version: 0x18f10001
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes

Thanks,
Xishi Qiu

Quoted 36 lines

Quoted 36 lines · On Tue, Jul 14, 2015 at 4:31 PM, Xishi Qiu <[email protected]…

    1a??the host directly link to the storage devicei 1/4 ?by intel ixgbe NIC;
    between them, no switch or router.
    2a??the nic of the storage device suddenly become unused and then OK
    after a little time, this happened frequency.
    3a??the host printk a lot of message like these:

    The kernel is SUSE 3.0.13, use slab, and the following log shows the
    page still have PG_slab when free_pages(). Does anyone have seen the
    problem?

    Jul  9 11:31:36 root kernel: [1042291.977565] BUG: Bad page state in process swapper  pfn:00bf2
    Jul  9 11:31:36 root kernel: [1042291.977568] page:ffffea0000029cf0 count:0 mapcount:0 mapping:          (null) index:0x7f6d4f500
    Jul  9 11:31:36 root kernel: [1042291.977571] page flags: 0x40000000000100(slab)  // here is the reason
    Jul  9 11:31:36 root kernel: [1042291.977574] Pid: 0, comm: swapper Tainted: G    B       X 3.0.13-0.27-default #1
    Jul  9 11:31:36 root kernel: [1042291.977577] Call Trace:
    Jul  9 11:31:36 root kernel: [1042291.977583]  [<ffffffff810048b5>] dump_trace+0x75/0x300
    Jul  9 11:31:36 root kernel: [1042291.977639]  [<ffffffff8143ea0f>] dump_stack+0x69/0x6f
    Jul  9 11:31:36 root kernel: [1042291.977644]  [<ffffffff810f53a1>] bad_page+0xb1/0x120
    Jul  9 11:31:37 root kernel: [1042291.977649]  [<ffffffff810f5926>] free_pages_prepare+0xe6/0x110
    Jul  9 11:31:37 root kernel: [1042291.977654]  [<ffffffff810f9259>] free_hot_cold_page+0x49/0x1f0
    Jul  9 11:31:37 root kernel: [1042291.977660]  [<ffffffff8137a3b4>] skb_release_data+0xb4/0xe0
    Jul  9 11:31:37 root kernel: [1042291.977665]  [<ffffffff81379e79>] __kfree_skb+0x9/0x90
    Jul  9 11:31:37 root kernel: [1042291.977676]  [<ffffffffa02784a9>] ixgbe_clean_tx_irq+0xa9/0x480 [ixgbe]
    Jul  9 11:31:37 root kernel: [1042291.977693]  [<ffffffffa02788cb>] ixgbe_poll+0x4b/0x1a0 [ixgbe]
    Jul  9 11:31:37 root kernel: [1042291.977705]  [<ffffffff81389c3a>] net_rx_action+0x10a/0x2c0
    Jul  9 11:31:37 root kernel: [1042291.977711]  [<ffffffff81060a1f>] __do_softirq+0xef/0x220
    Jul  9 11:31:37 root kernel: [1042291.977716]  [<ffffffff8144a8bc>] call_softirq+0x1c/0x30
    Jul  9 11:31:37 root kernel: [1042291.978974] DWARF2 unwinder stuck at call_softirq+0x1c/0x30

    Thanks,
    Xishi Qiu

Quoted 15 lines

Quoted 13 lines
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
E1000-devel mailing list
[email protected] <mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired

Skidmore, Donald C

2015-07-14 14:40 UTC #4 lore raw

Hey Xishi,

You are using a really old driver, our current is 4.1.1 and 3.9 dates to over 3 years ago. We have makes changes to support surprise removal that very well may address this issue. Have you attempted to recreate this failure with the latest out of tree driver?

Thanks,
-Don <[email protected]>

Quoted 111 lines

-----Original Message-----
From: [email protected] [mailto:linux-nics- [email protected]] On Behalf Of Xishi Qiu
Sent: Tuesday, July 14, 2015 2:37 AM
To: zhuyj
Cc: guozhibin 00179312; Linux MM; Linux NICS; LKML; e1000- [email protected]
Subject: Re: [linux-nics] [E1000-devel] bad pages when up/down network cable
Quoted 3 lines · Xishi Qiu
Quoted 1 line · zhuyj
Do you use the default ixgbe driver? or the ixgbe driver is modified by
you?
Quoted 4 lines
Quoted 1 line
Yes，no modify.
Sorry, it is modified by us...
the driver come from intel，the info：
root:~ # ethtool -i p2p2
driver: ixgbe
version: 3.9.16-NAPI
firmware-version: 0x18f10001
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
Thanks,
Xishi Qiu
Quoted 2 lines
Quoted 2 lines
On Tue, Jul 14, 2015 at 4:31 PM, Xishi Qiu <[email protected]
Quoted 7 lines · <mailto:[email protected]>> wrote:
Quoted 7 lines
1、the host directly link to the storage device，by intel ixgbe NIC; between them, no switch or router.
2、the nic of the storage device suddenly become unused and then OK after a little time, this happened frequency.
3、the host printk a lot of message like these:
Quoted 5 lines
Quoted 5 lines
The kernel is SUSE 3.0.13, use slab, and the following log shows the page still have PG_slab when free_pages(). Does anyone have seen the problem?
Jul 9 11:31:36 root kernel: [1042291.977565] BUG: Bad page state in
process swapper pfn:00bf2
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977568] page:ffffea0000029cf0
count:0 mapcount:0 mapping: (null) index:0x7f6d4f500
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977571] page flags:
0x40000000000100(slab) // here is the reason
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977574] Pid: 0, comm: swapper
Tainted: G    B       X 3.0.13-0.27-default #1
Quoted 2 lines
Quoted 2 lines
    Jul  9 11:31:36 root kernel: [1042291.977577] Call Trace:
    Jul  9 11:31:36 root kernel: [1042291.977583]  [<ffffffff810048b5>]
dump_trace+0x75/0x300
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977639]  [<ffffffff8143ea0f>]
dump_stack+0x69/0x6f
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977644]  [<ffffffff810f53a1>]
bad_page+0xb1/0x120
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977649]  [<ffffffff810f5926>]
free_pages_prepare+0xe6/0x110
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977654]  [<ffffffff810f9259>]
free_hot_cold_page+0x49/0x1f0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977660]  [<ffffffff8137a3b4>]
skb_release_data+0xb4/0xe0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977665]  [<ffffffff81379e79>]
__kfree_skb+0x9/0x90
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977676]  [<ffffffffa02784a9>]
ixgbe_clean_tx_irq+0xa9/0x480 [ixgbe]
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977693]  [<ffffffffa02788cb>]
ixgbe_poll+0x4b/0x1a0 [ixgbe]
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977705]  [<ffffffff81389c3a>]
net_rx_action+0x10a/0x2c0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977711]  [<ffffffff81060a1f>]
__do_softirq+0xef/0x220
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977716]  [<ffffffff8144a8bc>]
call_softirq+0x1c/0x30
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.978974] DWARF2 unwinder stuck at
call_softirq+0x1c/0x30
Quoted 5 lines
Quoted 5 lines
Thanks,
Xishi Qiu
Quoted 9 lines
Quoted 9 lines
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
E1000-devel mailing list [email protected] <mailto:E1000-
[email protected]>
Quoted 2 lines
Quoted 2 lines
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
Quoted 4 lines
Quoted 2 lines

Quoted 3 lines

_______________________________________________
Linux-nics mailing list
[email protected]

��{.n�+��+%��ݶ��w��{.n�+��{��G��{ay�ʇڙ�,j��f��h��z_�(�階�ݢj"��m��G��?��&��~��iO��z��v�^�m��?�I�

Xishi Qiu

author

2015-07-15 01:31 UTC #5 lore raw

Quoted 7 lines · Donald C

Hey Xishi,
You are using a really old driver, our current is 4.1.1 and 3.9 dates to over 3 years ago. We have makes changes to support surprise removal that very well may address this issue. Have you attempted to recreate this failure with the latest out of tree driver?
Thanks,
-Don <[email protected]>

Thank you very much, we'll try the latest driver.

Thanks,
Xishi Qiu

Quoted 113 lines

Quoted 113 lines
-----Original Message-----
From: [email protected] [mailto:linux-nics- [email protected]] On Behalf Of Xishi Qiu
Sent: Tuesday, July 14, 2015 2:37 AM
To: zhuyj
Cc: guozhibin 00179312; Linux MM; Linux NICS; LKML; e1000- [email protected]
Subject: Re: [linux-nics] [E1000-devel] bad pages when up/down network cable
Quoted 3 lines · Xishi Qiu
Quoted 1 line · zhuyj
Do you use the default ixgbe driver? or the ixgbe driver is modified by
you?
Quoted 4 lines
Quoted 1 line
Yesi 1/4 ?no modify.
Sorry, it is modified by us...
the driver come from inteli 1/4 ?the infoi 1/4 ?
root:~ # ethtool -i p2p2
driver: ixgbe
version: 3.9.16-NAPI
firmware-version: 0x18f10001
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
Thanks,
Xishi Qiu
Quoted 2 lines
Quoted 2 lines
On Tue, Jul 14, 2015 at 4:31 PM, Xishi Qiu <[email protected]
Quoted 7 lines · <mailto:[email protected]>> wrote:
Quoted 7 lines
1a??the host directly link to the storage devicei 1/4 ?by intel ixgbe NIC; between them, no switch or router.
2a??the nic of the storage device suddenly become unused and then OK after a little time, this happened frequency.
3a??the host printk a lot of message like these:
Quoted 5 lines
Quoted 5 lines
The kernel is SUSE 3.0.13, use slab, and the following log shows the page still have PG_slab when free_pages(). Does anyone have seen the problem?
Jul 9 11:31:36 root kernel: [1042291.977565] BUG: Bad page state in
process swapper pfn:00bf2
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977568] page:ffffea0000029cf0
count:0 mapcount:0 mapping: (null) index:0x7f6d4f500
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977571] page flags:
0x40000000000100(slab) // here is the reason
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977574] Pid: 0, comm: swapper
Tainted: G    B       X 3.0.13-0.27-default #1
Quoted 2 lines
Quoted 2 lines
    Jul  9 11:31:36 root kernel: [1042291.977577] Call Trace:
    Jul  9 11:31:36 root kernel: [1042291.977583]  [<ffffffff810048b5>]
dump_trace+0x75/0x300
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977639]  [<ffffffff8143ea0f>]
dump_stack+0x69/0x6f
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:36 root kernel: [1042291.977644]  [<ffffffff810f53a1>]
bad_page+0xb1/0x120
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977649]  [<ffffffff810f5926>]
free_pages_prepare+0xe6/0x110
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977654]  [<ffffffff810f9259>]
free_hot_cold_page+0x49/0x1f0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977660]  [<ffffffff8137a3b4>]
skb_release_data+0xb4/0xe0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977665]  [<ffffffff81379e79>]
__kfree_skb+0x9/0x90
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977676]  [<ffffffffa02784a9>]
ixgbe_clean_tx_irq+0xa9/0x480 [ixgbe]
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977693]  [<ffffffffa02788cb>]
ixgbe_poll+0x4b/0x1a0 [ixgbe]
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977705]  [<ffffffff81389c3a>]
net_rx_action+0x10a/0x2c0
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977711]  [<ffffffff81060a1f>]
__do_softirq+0xef/0x220
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.977716]  [<ffffffff8144a8bc>]
call_softirq+0x1c/0x30
Quoted 1 line
Quoted 1 line
    Jul  9 11:31:37 root kernel: [1042291.978974] DWARF2 unwinder stuck at
call_softirq+0x1c/0x30
Quoted 5 lines
Quoted 5 lines
Thanks,
Xishi Qiu
Quoted 9 lines
Quoted 9 lines
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
E1000-devel mailing list [email protected] <mailto:E1000-
[email protected]>
Quoted 2 lines
Quoted 2 lines
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
Quoted 4 lines
Quoted 2 lines

Quoted 3 lines

Quoted 3 lines
_______________________________________________
Linux-nics mailing list
[email protected]