Skip to contentSkip to conversation

Face some error after applying commit 7dfa4b414d4(net/mlx4_en: Code cleanups in tx path)

DISCUSSION Wei Yang 4 participants 11 messages Nov 10, 2014 – Nov 11, 2014 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Wei Yang

author
#2 lore raw
In reply to a message not in the archive
Quoted 26 lines · Eric Dumazet
Quoted 7 lines · Wei Yang

Eric and Amir

I am testing the VF on PowerNV platform with 3.18-rc2. After applying this patch I face some errors.

First is the compiling error.

Quoted 16 lines
    drivers/net/ethernet/mellanox/mlx4//en_tx.c: In function ‘mlx4_en_xmit’:
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: error: ‘shinfo’ undeclared (first use in this function)
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: note: each undeclared identifier is reported only once for each function it appears in
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    make[1]: *** [drivers/net/ethernet/mellanox/mlx4//en_tx.o] Error 1
    make: *** [_module_drivers/net/ethernet/mellanox/mlx4/] Error 2
Quoted 3 lines

This compilation error seems strange.

Are you sure your tree is pristine, not corrupted in any way ?

I believe I did the revert one by one with git revert.

Quoted 20 lines
Quoted 20 lines

I tried to fix this with following change:

    [root@tian-lp1 3.18]# git diff
    diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/m
    index eaf23eb..d2f06a7 100644
    --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    @@ -799,8 +799,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_dev
             * set flag for further reference
             */
            if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
    -                    shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
    -               shinfo->tx_flags |= SKBTX_IN_PROGRESS;
    +                    skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
    +               skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
                    tx_info->ts_requested = 1;
            }
Quoted 5 lines
Quoted 2 lines

But seems to face another error.

I suspect your tree is not the official tree, I do not see how you got this compilation error.

I checked the upstream git tree again, and find this commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7dfa4b414d4eec8da56e44fb2b4aea3e549b092f

And I want to say the shinfo local variable is introduced in commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b9d8839a44092cb4268ef2813c34d5dbf3363603

And in my log tree, also checked the upstream, this one is applyed after the first one. And the compiling error will disappear untill I apply this one.

So this compiling issue can't reproduced at your side? You have reset --hard to the "Code cleanup" one, and can't see the error? That is strange.

Signature · 2 lines
Richard Yang
Help you, Help me
author
#3 lore raw

This is the git output in on my machine.

[ywywyang@tian-lp1 3.18]$ git status
# On branch p8-sriov-3.18-rc2-mlx4
# Your branch is behind 'origin/p8-sriov-3.18-rc2-mlx4' by 14 commits, and can be fast-forwarded.
# (use "git pull" to update your local branch)
#
nothing to commit, working directory clean
[ywywyang@tian-lp1 3.18]$ git oneline -40
40c4198 Revert "net/mlx4_en: Align tx path structures to cache lines"
7d071a0 Revert "net/mlx4_en: Avoid calling bswap in tx fast path"
5aa717e Revert "net/mlx4_en: tx_info allocated with kmalloc() instead of vmalloc 77ab7f7 Revert "net/mlx4_en: Avoid a cache line miss in TX completion for single 6fee4f6 Revert "net/mlx4_en: Use prefetch in tx path"
6509cd2 Revert "net/mlx4_en: Avoid false sharing in mlx4_en_en_process_tx_cq()" 50b7df1 Revert "net/mlx4_en: mlx4_en_xmit() reads ring->cons once, and ahead of a0cc8e8 Revert "net/mlx4_en: Use local var in tx flow for skb_shinfo(skb)" 8cc9b1d Revert "net/mlx4_en: Use local var for skb_headlen(skb)"
2894ad1 Revert "net/mlx4_en: tx_info->ts_requested was not cleared"
10f191f Revert "net/mlx4_en: Enable the compiler to make is_inline() inlined" 5b6e300 Revert "net/mlx4_en: Use the new tx_copybreak to set inline threshold" 277c194 Revert "net/mlx4_en: remove NETDEV_TX_BUSY"
3f626de Revert "net: add netdev_txq_bql_{enqueue, complete}_prefetchw() helpers" c1d0c02 Revert "mlx4: fix race accessing page->_count"
cac7f24 Linux 3.18-rc2

You could see current tree is clean and based on v3.18-rc2.

Quoted 78 lines · Wei Yang
Quoted 26 lines · Eric Dumazet
Quoted 7 lines · Wei Yang

Eric and Amir

I am testing the VF on PowerNV platform with 3.18-rc2. After applying this patch I face some errors.

First is the compiling error.

Quoted 16 lines
    drivers/net/ethernet/mellanox/mlx4//en_tx.c: In function ‘mlx4_en_xmit’:
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: error: ‘shinfo’ undeclared (first use in this function)
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: note: each undeclared identifier is reported only once for each function it appears in
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    make[1]: *** [drivers/net/ethernet/mellanox/mlx4//en_tx.o] Error 1
    make: *** [_module_drivers/net/ethernet/mellanox/mlx4/] Error 2
Quoted 3 lines

This compilation error seems strange.

Are you sure your tree is pristine, not corrupted in any way ?

I believe I did the revert one by one with git revert.

Quoted 20 lines
Quoted 20 lines

I tried to fix this with following change:

    [root@tian-lp1 3.18]# git diff
    diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/m
    index eaf23eb..d2f06a7 100644
    --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    @@ -799,8 +799,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_dev
             * set flag for further reference
             */
            if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
    -                    shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
    -               shinfo->tx_flags |= SKBTX_IN_PROGRESS;
    +                    skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
    +               skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
                    tx_info->ts_requested = 1;
            }
Quoted 5 lines
Quoted 2 lines

But seems to face another error.

I suspect your tree is not the official tree, I do not see how you got this compilation error.

I checked the upstream git tree again, and find this commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7dfa4b414d4eec8da56e44fb2b4aea3e549b092f

And I want to say the shinfo local variable is introduced in commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b9d8839a44092cb4268ef2813c34d5dbf3363603

And in my log tree, also checked the upstream, this one is applyed after the first one. And the compiling error will disappear untill I apply this one.

So this compiling issue can't reproduced at your side? You have reset --hard to the "Code cleanup" one, and can't see the error? That is strange.

Signature · 2 lines
Richard Yang
Help you, Help me
Signature · 2 lines
Richard Yang
Help you, Help me
#4 lore raw
In reply to Wei Yang
Quoted 67 lines · Wei Yang
Quoted 26 lines · Eric Dumazet
Quoted 7 lines · Wei Yang

Eric and Amir

I am testing the VF on PowerNV platform with 3.18-rc2. After applying this patch I face some errors.

First is the compiling error.

Quoted 16 lines
    drivers/net/ethernet/mellanox/mlx4//en_tx.c: In function ‘mlx4_en_xmit’:
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: error: ‘shinfo’ undeclared (first use in this function)
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: note: each undeclared identifier is reported only once for each function it appears in
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    make[1]: *** [drivers/net/ethernet/mellanox/mlx4//en_tx.o] Error 1
    make: *** [_module_drivers/net/ethernet/mellanox/mlx4/] Error 2
Quoted 3 lines

This compilation error seems strange.

Are you sure your tree is pristine, not corrupted in any way ?

I believe I did the revert one by one with git revert.

Quoted 20 lines
Quoted 20 lines

I tried to fix this with following change:

    [root@tian-lp1 3.18]# git diff
    diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/m
    index eaf23eb..d2f06a7 100644
    --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    @@ -799,8 +799,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_dev
             * set flag for further reference
             */
            if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
    -                    shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
    -               shinfo->tx_flags |= SKBTX_IN_PROGRESS;
    +                    skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
    +               skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
                    tx_info->ts_requested = 1;
            }
Quoted 5 lines
Quoted 2 lines

But seems to face another error.

I suspect your tree is not the official tree, I do not see how you got this compilation error.

I checked the upstream git tree again, and find this commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7dfa4b414d4eec8da56e44fb2b4aea3e549b092f

And I want to say the shinfo local variable is introduced in commit:

Quoted 8 lines

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b9d8839a44092cb4268ef2813c34d5dbf3363603

And in my log tree, also checked the upstream, this one is applyed after the first one. And the compiling error will disappear untill I apply this one.

So this compiling issue can't reproduced at your side? You have reset --hard to the "Code cleanup" one, and can't see the error? That is strange.

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Have you tried to disable Blue Frame ?

author
#5 lore raw
Quoted 85 lines · Eric Dumazet
Quoted 67 lines · Wei Yang
Quoted 26 lines · Eric Dumazet
Quoted 7 lines · Wei Yang

Eric and Amir

I am testing the VF on PowerNV platform with 3.18-rc2. After applying this patch I face some errors.

First is the compiling error.

Quoted 16 lines
    drivers/net/ethernet/mellanox/mlx4//en_tx.c: In function ‘mlx4_en_xmit’:
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: error: ‘shinfo’ undeclared (first use in this function)
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    drivers/net/ethernet/mellanox/mlx4//en_tx.c:802:8: note: each undeclared identifier is reported only once for each function it appears in
            shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
            ^
    include/linux/compiler.h:160:42: note: in definition of macro ‘unlikely’
     # define unlikely(x) __builtin_expect(!!(x), 0)
                                              ^
    make[1]: *** [drivers/net/ethernet/mellanox/mlx4//en_tx.o] Error 1
    make: *** [_module_drivers/net/ethernet/mellanox/mlx4/] Error 2
Quoted 3 lines

This compilation error seems strange.

Are you sure your tree is pristine, not corrupted in any way ?

I believe I did the revert one by one with git revert.

Quoted 20 lines
Quoted 20 lines

I tried to fix this with following change:

    [root@tian-lp1 3.18]# git diff
    diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/m
    index eaf23eb..d2f06a7 100644
    --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
    @@ -799,8 +799,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_dev
             * set flag for further reference
             */
            if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
    -                    shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
    -               shinfo->tx_flags |= SKBTX_IN_PROGRESS;
    +                    skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) {
    +               skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
                    tx_info->ts_requested = 1;
            }
Quoted 5 lines
Quoted 2 lines

But seems to face another error.

I suspect your tree is not the official tree, I do not see how you got this compilation error.

I checked the upstream git tree again, and find this commit:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=7dfa4b414d4eec8da56e44fb2b4aea3e549b092f

And I want to say the shinfo local variable is introduced in commit:

Quoted 8 lines

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=b9d8839a44092cb4268ef2813c34d5dbf3363603

And in my log tree, also checked the upstream, this one is applyed after the first one. And the compiling error will disappear untill I apply this one.

So this compiling issue can't reproduced at your side? You have reset --hard to the "Code cleanup" one, and can't see the error? That is strange.

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

Quoted 9 lines

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Signature · 2 lines
Richard Yang
Help you, Help me
#6 lore raw
Quoted 6 lines · Wei Yang
Quoted 5 lines · Eric Dumazet
Quoted 4 lines · Wei Yang
Quoted 3 lines · Eric Dumazet
Quoted 2 lines · Wei Yang

Eric and Amir

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 1 line

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Quoted 7 lines
Quoted 2 lines

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.
3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

Amir.

author
#7 lore raw
Quoted 42 lines · Amir Vadai
Quoted 6 lines · Wei Yang
Quoted 5 lines · Eric Dumazet
Quoted 4 lines · Wei Yang
Quoted 3 lines · Eric Dumazet
Quoted 2 lines · Wei Yang

Eric and Amir

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 1 line

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Quoted 7 lines
Quoted 2 lines

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.

Quoted 5 lines

From the log, seems yes.

3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

I tried to use this after mlx4_en is loaded, still see the CQE error.

Quoted 3 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

And this error is reported from VF always. After the error, the other network interface seems can't function.

Quoted 2 lines

Amir.

Signature · 2 lines
Richard Yang
Help you, Help me
#8 lore raw
Quoted 6 lines · Wei Yang
Quoted 2 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

Quoted 6 lines

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

And this error is reported from VF always. After the error, the other network interface seems can't function.

Guys, do we have here something which isn't bisectable (i.e the code doesn't work if you force it to an arbitrary commit within in the chain) - something which doesn't work as a whole? in case the former, we will work to do better next times.. in case the latter, Wei, as Amir asked, please send us full logs when you load/run the driver with debug level set, for mlx4_core it would be loading with debug_level=1 for mlx4_en you should be able to raise the debug level using ethtool

Or.

#9 lore raw
In reply to Wei Yang
Quoted 55 lines · Wei Yang
Quoted 34 lines · Amir Vadai
Quoted 6 lines · Wei Yang
Quoted 5 lines · Eric Dumazet
Quoted 4 lines · Wei Yang
Quoted 3 lines · Eric Dumazet
Quoted 2 lines · Wei Yang

Eric and Amir

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 1 line

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Quoted 7 lines
Quoted 2 lines

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Quoted 7 lines

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.

From the log, seems yes.

Quoted 3 lines

3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

I tried to use this after mlx4_en is loaded, still see the CQE error.

Quoted 3 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

Quoted 6 lines

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

The log in the first mail has only mlx4_en logs. I would like to see the full log, that has mlx4_core messages too. And as Or suggested, debug_level=1 could be useful here too.

Quoted 15 lines

And this error is reported from VF always. After the error, the other network interface seems can't function.

Quoted 2 lines

Amir.

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw
Quoted 59 lines · Amir Vadai
Quoted 58 lines · Wei Yang
Quoted 34 lines · Amir Vadai
Quoted 6 lines · Wei Yang
Quoted 5 lines · Eric Dumazet
Quoted 4 lines · Wei Yang
Quoted 3 lines · Eric Dumazet
Quoted 2 lines · Wei Yang

Eric and Amir

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 1 line

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Quoted 7 lines
Quoted 2 lines

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Quoted 7 lines

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.

From the log, seems yes.

Quoted 3 lines

3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

I tried to use this after mlx4_en is loaded, still see the CQE error.

Quoted 3 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

Quoted 8 lines
Quoted 3 lines

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

The log in the first mail has only mlx4_en logs. I would like to see the full log, that has mlx4_core messages too. And as Or suggested, debug_level=1 could be useful here too.

Ah, you need the log from mlx4_core too. Ok, I will do it again.

BTW, how to add the debug_level=1 in the command line? Like this?

modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2 debug_level=1

But for mlx4_en, I am not sure I could raise the debug level with ethtool, since the ethernet driver may not work properly. Actually I am not sure how to raise the level with ethtool. Could you give me an example?

Quoted 15 lines
Quoted 15 lines

And this error is reported from VF always. After the error, the other network interface seems can't function.

Quoted 2 lines

Amir.

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Richard Yang
Help you, Help me
#11 lore raw
Quoted 69 lines · Wei Yang
Quoted 59 lines · Amir Vadai
Quoted 58 lines · Wei Yang
Quoted 34 lines · Amir Vadai
Quoted 6 lines · Wei Yang
Quoted 5 lines · Eric Dumazet
Quoted 4 lines · Wei Yang
Quoted 3 lines · Eric Dumazet
Quoted 2 lines · Wei Yang

Eric and Amir

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 1 line

Okay, your message was not clear : I thought you had a compilation error on current tree.

The true story of these patches is that Mellanox split an initial big chunk [1] I gave into multiple patches.

Maybe they missed that one patch did not actually compile.

[1] https://patchwork.ozlabs.org/patch/394256/

Now, it is done, there is nothing we can do.

I'll let Mellanox comment, but it looks like your hardware does not like something.

Quoted 7 lines
Quoted 2 lines

Have you tried to disable Blue Frame ?

Yep, looks the PF works fine. But the current FW I can't just enable the PF.

How to disable Blue Frame? I am not clear about this.

Quoted 7 lines

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.

From the log, seems yes.

Quoted 3 lines

3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

I tried to use this after mlx4_en is loaded, still see the CQE error.

Quoted 3 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

Quoted 8 lines
Quoted 3 lines

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

The log in the first mail has only mlx4_en logs. I would like to see the full log, that has mlx4_core messages too. And as Or suggested, debug_level=1 could be useful here too.

Quoted 5 lines

Ah, you need the log from mlx4_core too. Ok, I will do it again.

BTW, how to add the debug_level=1 in the command line? Like this?

modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2 debug_level=1

yes

Quoted 4 lines

But for mlx4_en, I am not sure I could raise the debug level with ethtool, since the ethernet driver may not work properly. Actually I am not sure how to raise the level with ethtool. Could you give me an example?

# ethtool -s ens5f1d1 msglvl 0xffff
Quoted 25 lines
Quoted 8 lines
Quoted 8 lines

And this error is reported from VF always. After the error, the other network interface seems can't function.

Quoted 2 lines

Amir.

Quoted 8 lines
Quoted 8 lines

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#12 lore raw
Quoted 33 lines · Amir Vadai
Quoted 32 lines · Wei Yang
Quoted 31 lines
Quoted 20 lines
Quoted 7 lines

Hi,

Lets see that we're on the same page here:
1. There was a compilation problem that you fixed (Yes, it was my fault
- I just discovered it a minute after the code was applied).
2. When you're using SR-IOV, during initialization, you get a CQE error with syndrome 0x2 on one of the probed VF's.

From the log, seems yes.

Quoted 3 lines

3. Regarding the BlueFlame - I don't see how it is related to the issue that you see. But it is a very easy experiment. Issue: "ethtool --set-priv-flags eth1 blueflame off"

I tried to use this after mlx4_en is loaded, still see the CQE error.

Quoted 3 lines

Please send me the module parameters you used when loading mlx4_core, a full dmesg with both mlx4_core and mlx4_en loading.

Quoted 6 lines

The command line I use is:
modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2

The log I sent in the first mail is the full log, including the CQE error, one warning in watchdog, and then print the CQE error periodicly. What else message you would like me to capture?

The log in the first mail has only mlx4_en logs. I would like to see the full log, that has mlx4_core messages too. And as Or suggested, debug_level=1 could be useful here too.

Quoted 7 lines
Quoted 5 lines

Ah, you need the log from mlx4_core too. Ok, I will do it again.

BTW, how to add the debug_level=1 in the command line? Like this?

modprobe mlx4_core num_vfs=1 probe_vf=1 port_type_array=2,2 debug_level=1

yes

I past the full log here, but I don't see the log level affact.

[ 5723.612882] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[ 5723.612979] mlx4_core: Initializing 0003:05:00.0
[ 5723.613023] pci 0003:05: 0.1: [PE# 222] VF 0003:05:00.1 associated with PE#222
[ 5723.613299] pci 0003:05: 0.1: [PE# 222] Setting up 32-bit TCE table at 0..80000000
[ 5723.684140] pci 0003:05: 0.1: [PE# 222] Enabling 64-bit DMA bypass
[ 5723.684215] pci 0003:05: 0.2: [PE# 223] VF 0003:05:00.2 associated with PE#223
[ 5723.684409] pci 0003:05: 0.2: [PE# 223] Setting up 32-bit TCE table at 0..80000000
[ 5723.754871] pci 0003:05: 0.2: [PE# 223] Enabling 64-bit DMA bypass
[ 5723.754944] pci 0003:05: 0.3: [PE# 224] VF 0003:05:00.3 associated with PE#224
[ 5723.755150] pci 0003:05: 0.3: [PE# 224] Setting up 32-bit TCE table at 0..80000000
[ 5723.825646] pci 0003:05: 0.3: [PE# 224] Enabling 64-bit DMA bypass
[ 5723.825718] pci 0003:05: 0.4: [PE# 225] VF 0003:05:00.4 associated with PE#225
[ 5723.825911] pci 0003:05: 0.4: [PE# 225] Setting up 32-bit TCE table at 0..80000000
[ 5723.896342] pci 0003:05: 0.4: [PE# 225] Enabling 64-bit DMA bypass
[ 5723.896416] pci 0003:05: 0.5: [PE# 226] VF 0003:05:00.5 associated with PE#226
[ 5723.896610] pci 0003:05: 0.5: [PE# 226] Setting up 32-bit TCE table at 0..80000000
[ 5723.967052] pci 0003:05: 0.5: [PE# 226] Enabling 64-bit DMA bypass
[ 5723.967135] pci 0003:05: 0.6: [PE# 227] VF 0003:05:00.6 associated with PE#227
[ 5723.967328] pci 0003:05: 0.6: [PE# 227] Setting up 32-bit TCE table at 0..80000000
[ 5724.037755] pci 0003:05: 0.6: [PE# 227] Enabling 64-bit DMA bypass
[ 5724.037829] pci 0003:05: 0.7: [PE# 228] VF 0003:05:00.7 associated with PE#228
[ 5724.038022] pci 0003:05: 0.7: [PE# 228] Setting up 32-bit TCE table at 0..80000000
[ 5724.108452] pci 0003:05: 0.7: [PE# 228] Enabling 64-bit DMA bypass
[ 5724.108525] pci 0003:05: 1.0: [PE# 229] VF 0003:05:01.0 associated with PE#229
[ 5724.108718] pci 0003:05: 1.0: [PE# 229] Setting up 32-bit TCE table at 0..80000000
[ 5724.179209] pci 0003:05: 1.0: [PE# 229] Enabling 64-bit DMA bypass
[ 5724.179293] pci 0003:05: 1.1: [PE# 230] VF 0003:05:01.1 associated with PE#230
[ 5724.179486] pci 0003:05: 1.1: [PE# 230] Setting up 32-bit TCE table at 0..80000000
[ 5724.249946] pci 0003:05: 1.1: [PE# 230] Enabling 64-bit DMA bypass
[ 5724.250020] pci 0003:05: 1.2: [PE# 231] VF 0003:05:01.2 associated with PE#231
[ 5724.250214] pci 0003:05: 1.2: [PE# 231] Setting up 32-bit TCE table at 0..80000000
[ 5724.320703] pci 0003:05: 1.2: [PE# 231] Enabling 64-bit DMA bypass
[ 5724.320787] pci 0003:05: 1.3: [PE# 232] VF 0003:05:01.3 associated with PE#232
[ 5724.320982] pci 0003:05: 1.3: [PE# 232] Setting up 32-bit TCE table at 0..80000000
[ 5724.391424] pci 0003:05: 1.3: [PE# 232] Enabling 64-bit DMA bypass
[ 5724.391497] pci 0003:05: 1.4: [PE# 233] VF 0003:05:01.4 associated with PE#233
[ 5724.391690] pci 0003:05: 1.4: [PE# 233] Setting up 32-bit TCE table at 0..80000000
[ 5724.462129] pci 0003:05: 1.4: [PE# 233] Enabling 64-bit DMA bypass
[ 5724.462215] pci 0003:05: 1.5: [PE# 234] VF 0003:05:01.5 associated with PE#234
[ 5724.462408] pci 0003:05: 1.5: [PE# 234] Setting up 32-bit TCE table at 0..80000000
[ 5724.532834] pci 0003:05: 1.5: [PE# 234] Enabling 64-bit DMA bypass
[ 5724.532907] pci 0003:05: 1.6: [PE# 235] VF 0003:05:01.6 associated with PE#235
[ 5724.533100] pci 0003:05: 1.6: [PE# 235] Setting up 32-bit TCE table at 0..80000000
[ 5724.603556] pci 0003:05: 1.6: [PE# 235] Enabling 64-bit DMA bypass
[ 5724.603628] pci 0003:05: 1.7: [PE# 236] VF 0003:05:01.7 associated with PE#236
[ 5724.603822] pci 0003:05: 1.7: [PE# 236] Setting up 32-bit TCE table at 0..80000000
[ 5724.674261] pci 0003:05: 1.7: [PE# 236] Enabling 64-bit DMA bypass
[ 5724.674334] pci 0003:05: 2.0: [PE# 237] VF 0003:05:02.0 associated with PE#237
[ 5724.674528] pci 0003:05: 2.0: [PE# 237] Setting up 32-bit TCE table at 0..80000000
[ 5724.745012] pci 0003:05: 2.0: [PE# 237] Enabling 64-bit DMA bypass
[ 5724.745088] pci 0003:05: 2.1: [PE# 238] VF 0003:05:02.1 associated with PE#238
[ 5724.745281] pci 0003:05: 2.1: [PE# 238] Setting up 32-bit TCE table at 0..80000000
[ 5724.815714] pci 0003:05: 2.1: [PE# 238] Enabling 64-bit DMA bypass
[ 5724.815786] pci 0003:05: 2.2: [PE# 239] VF 0003:05:02.2 associated with PE#239
[ 5724.815979] pci 0003:05: 2.2: [PE# 239] Setting up 32-bit TCE table at 0..80000000
[ 5724.886413] pci 0003:05: 2.2: [PE# 239] Enabling 64-bit DMA bypass
[ 5724.886487] pci 0003:05: 2.3: [PE# 240] VF 0003:05:02.3 associated with PE#240
[ 5724.886680] pci 0003:05: 2.3: [PE# 240] Setting up 32-bit TCE table at 0..80000000
[ 5724.957126] pci 0003:05: 2.3: [PE# 240] Enabling 64-bit DMA bypass
[ 5724.957210] pci 0003:05: 2.4: [PE# 241] VF 0003:05:02.4 associated with PE#241
[ 5724.957403] pci 0003:05: 2.4: [PE# 241] Setting up 32-bit TCE table at 0..80000000
[ 5725.027842] pci 0003:05: 2.4: [PE# 241] Enabling 64-bit DMA bypass
[ 5725.027917] pci 0003:05: 2.5: [PE# 242] VF 0003:05:02.5 associated with PE#242
[ 5725.028109] pci 0003:05: 2.5: [PE# 242] Setting up 32-bit TCE table at 0..80000000
[ 5725.098551] pci 0003:05: 2.5: [PE# 242] Enabling 64-bit DMA bypass
[ 5725.098624] pci 0003:05: 2.6: [PE# 243] VF 0003:05:02.6 associated with PE#243
[ 5725.098818] pci 0003:05: 2.6: [PE# 243] Setting up 32-bit TCE table at 0..80000000
[ 5725.169255] pci 0003:05: 2.6: [PE# 243] Enabling 64-bit DMA bypass
[ 5725.169338] pci 0003:05: 2.7: [PE# 244] VF 0003:05:02.7 associated with PE#244
[ 5725.169532] pci 0003:05: 2.7: [PE# 244] Setting up 32-bit TCE table at 0..80000000
[ 5725.239962] pci 0003:05: 2.7: [PE# 244] Enabling 64-bit DMA bypass
[ 5725.240036] pci 0003:05: 3.0: [PE# 245] VF 0003:05:03.0 associated with PE#245
[ 5725.240229] pci 0003:05: 3.0: [PE# 245] Setting up 32-bit TCE table at 0..80000000
[ 5725.310678] pci 0003:05: 3.0: [PE# 245] Enabling 64-bit DMA bypass
[ 5725.310751] pci 0003:05: 3.1: [PE# 246] VF 0003:05:03.1 associated with PE#246
[ 5725.310944] pci 0003:05: 3.1: [PE# 246] Setting up 32-bit TCE table at 0..80000000
[ 5725.381375] pci 0003:05: 3.1: [PE# 246] Enabling 64-bit DMA bypass
[ 5725.381459] pci 0003:05: 3.2: [PE# 247] VF 0003:05:03.2 associated with PE#247
[ 5725.381653] pci 0003:05: 3.2: [PE# 247] Setting up 32-bit TCE table at 0..80000000
[ 5725.452087] pci 0003:05: 3.2: [PE# 247] Enabling 64-bit DMA bypass
[ 5725.452171] pci 0003:05: 3.3: [PE# 248] VF 0003:05:03.3 associated with PE#248
[ 5725.452365] pci 0003:05: 3.3: [PE# 248] Setting up 32-bit TCE table at 0..80000000
[ 5725.522815] pci 0003:05: 3.3: [PE# 248] Enabling 64-bit DMA bypass
[ 5725.522889] pci 0003:05: 3.4: [PE# 249] VF 0003:05:03.4 associated with PE#249
[ 5725.523082] pci 0003:05: 3.4: [PE# 249] Setting up 32-bit TCE table at 0..80000000
[ 5725.593516] pci 0003:05: 3.4: [PE# 249] Enabling 64-bit DMA bypass
[ 5725.593590] pci 0003:05: 3.5: [PE# 250] VF 0003:05:03.5 associated with PE#250
[ 5725.593783] pci 0003:05: 3.5: [PE# 250] Setting up 32-bit TCE table at 0..80000000
[ 5725.664330] pci 0003:05: 3.5: [PE# 250] Enabling 64-bit DMA bypass
[ 5725.664403] pci 0003:05: 3.6: [PE# 251] VF 0003:05:03.6 associated with PE#251
[ 5725.664597] pci 0003:05: 3.6: [PE# 251] Setting up 32-bit TCE table at 0..80000000
[ 5725.735081] pci 0003:05: 3.6: [PE# 251] Enabling 64-bit DMA bypass
[ 5725.735154] pci 0003:05: 3.7: [PE# 252] VF 0003:05:03.7 associated with PE#252
[ 5725.735348] pci 0003:05: 3.7: [PE# 252] Setting up 32-bit TCE table at 0..80000000
[ 5725.805792] pci 0003:05: 3.7: [PE# 252] Enabling 64-bit DMA bypass
[ 5725.805865] pci 0003:05: 4.0: [PE# 253] VF 0003:05:04.0 associated with PE#253
[ 5725.806058] pci 0003:05: 4.0: [PE# 253] Setting up 32-bit TCE table at 0..80000000
[ 5725.876496] pci 0003:05: 4.0: [PE# 253] Enabling 64-bit DMA bypass
[ 5725.876570] pci 0003:05: 4.1: [PE# 254] VF 0003:05:04.1 associated with PE#254
[ 5725.876763] pci 0003:05: 4.1: [PE# 254] Setting up 32-bit TCE table at 0..80000000
[ 5725.945133] pci 0003:05: 4.1: [PE# 254] Enabling 64-bit DMA bypass
[ 5725.945212] pci 0003:05: 4.2: [PE# 255] VF 0003:05:04.2 associated with PE#255
[ 5725.945316] pci 0003:05: 4.2: [PE# 255] Setting up 32-bit TCE table at 0..80000000
[ 5726.006975] pci 0003:05: 4.2: [PE# 255] Enabling 64-bit DMA bypass
[ 5726.007163] mlx4_core 0003:05:00.0: Using 64-bit DMA iommu bypass
[ 5726.007227] mlx4_core 0003:05:00.0: Enabling SR-IOV with 1 VFs
[ 5726.116742] mlx4_core: Initializing 0003:05:00.1
[ 5726.116825] mlx4_core 0003:05:00.1: enabling device (0000 -> 0002)
[ 5726.116914] mlx4_core 0003:05:00.1: Using 64-bit DMA iommu bypass
[ 5726.116976] mlx4_core 0003:05:00.1: Detected virtual function - running in slave mode
[ 5726.117191] mlx4_core 0003:05:00.1: PF is not ready - Deferring probe
[ 5726.117276] pci 0003:05:00.1: Driver mlx4_core requests probe deferral
[ 5726.117363] mlx4_core 0003:05:00.0: Running in master mode
[ 5731.123148] mlx4_core 0003:05:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
[ 5731.123227] mlx4_core 0003:05:00.0: PCIe link width is x8, device supports x8
[ 5731.151307] mlx4_core: Initializing 0003:05:00.1
[ 5731.151389] mlx4_core 0003:05:00.1: enabling device (0000 -> 0002)
[ 5731.151501] mlx4_core 0003:05:00.1: Using 64-bit DMA iommu bypass
[ 5731.151559] mlx4_core 0003:05:00.1: Detected virtual function - running in slave mode
[ 5731.151636] mlx4_core 0003:05:00.1: Sending reset
[ 5731.151770] mlx4_core 0003:05:00.0: Received reset from slave:1
[ 5731.152189] mlx4_core 0003:05:00.1: Sending vhcr0
[ 5731.152991] mlx4_core 0003:05:00.1: HCA minimum page size:512
[ 5731.153444] mlx4_core 0003:05:00.1: Timestamping is not supported in slave mode
[root@tian-lp1 ywywyang]# [ 5731.229958] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
[ 5731.230572] mlx4_en 0003:05:00.0: registered PHC clock
[ 5731.230817] mlx4_en 0003:05:00.0: Activating port:1
[ 5731.262377] mlx4_en: eth12: Using 256 TX rings
[ 5731.262494] mlx4_en: eth12: Using 4 RX rings
[ 5731.262546] mlx4_en: eth12:   frag:0 - size:1518 prefix:0 align:0 stride:1536
[ 5731.262761] mlx4_en: eth12: Initializing port
[ 5731.263357] mlx4_en 0003:05:00.1: Activating port:1
[ 5731.263467] mlx4_en: 0003:05:00.1: Port 1: Assigned random MAC address d6:0f:0b:57:36:24
[ 5731.442926] mlx4_en: eth13: Using 256 TX rings
[ 5731.443010] mlx4_en: eth13: Using 4 RX rings
[ 5731.443018] mlx4_en: eth13:   frag:0 - size:1518 prefix:0 align:0 stride:1536
[ 5731.443304] mlx4_en: eth13: Initializing port
[ 5732.096548] mlx4_en: eth13:   frag:0 - size:1518 prefix:0 align:0 stride:1536
[ 5732.452283] IPv6: ADDRCONF(NETDEV_UP): eth13: link is not ready
[ 5732.951420] mlx4_en: eth12: Link Up
[ 5732.951477] mlx4_en: eth13: Link Up
[ 5732.951627] IPv6: ADDRCONF(NETDEV_CHANGE): eth13: link becomes ready
[ 5733.171020] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5733.315756] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5734.165387] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5748.045084] NETDEV WATCHDOG: eth13 (mlx4_core): transmit queue 1 timed out
[ 5748.045245] ------------[ cut here ]------------
[ 5748.045301] WARNING: at net/sched/sch_generic.c:303
[ 5748.045355] Modules linked in: mlx4_en(O) mlx4_core nf_conntrack_ipv6 nf_defrag_ipv6 bnep nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack bluetooth rfkill ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_mangle iptable_security iptable_raw be2net tg3 ptp pps_core nfsd nfs_acl ses enclosure kvm_hv kvm_pr kvm lockd grace binfmt_misc shpchp sunrpc uinput lpfc ipr scsi_transport_fc [last unloaded: mlx4_core]
[ 5748.046069] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P           O   3.18.0-rc2yw+ #161
[ 5748.046142] task: c0000027ef4f92d0 ti: c0000027ef574000 task.ti: c0000027ef574000
[ 5748.046215] NIP: c00000000079a9d4 LR: c00000000079a9d0 CTR: 0000000030044878
[ 5748.046286] REGS: c0000027ef5772d0 TRAP: 0700   Tainted: P           O    (3.18.0-rc2yw+)
[ 5748.046357] MSR: 9000000000029032 <SF,HV,EE,ME,IR,DR,RI>  CR: 28004024  XER: 00000000
[ 5748.046524] CFAR: c0000000008be150 SOFTE: 1 

GPR00: c00000000079a9d0 c0000027ef577550 c00000000147acf0 000000000000003e
GPR04: c000002004145888 c000002004156240 9000000000009032 0000000000000029
GPR08: c000000000d1acf0 0000000000000000 0000002003430000 0000000030044954
GPR12: 0000000028004022 c00000000fdc0900 0000000000000001 c0000000008f10a8
GPR16: c0000027ef795438 c0000027ef795838 c0000027ef795c38 0000000000000000
GPR20: c0000027ef795038 c0000000014b2200 0000000000000000 0000000000000000
GPR24: 0000000000000000 ffffffffffffffff 0000000000000000 0000000000000001
GPR28: 0000000000000004 c0000000014b2200 c000002781340000 0000000000000001

[ 5748.047435] NIP [c00000000079a9d4] .dev_watchdog+0x354/0x370
[ 5748.047496] LR [c00000000079a9d0] .dev_watchdog+0x350/0x370
[ 5748.047544] Call Trace:
[ 5748.047570] [c0000027ef577550] [c00000000079a9d0] .dev_watchdog+0x350/0x370 (unreliable)
[ 5748.047655] [c0000027ef577600] [c0000000001275a4] .call_timer_fn+0x64/0x190
[ 5748.047727] [c0000027ef5776b0] [c00000000012814c] .run_timer_softirq+0x2bc/0x3d0
[ 5748.047812] [c0000027ef5777b0] [c0000000000ad738] .__do_softirq+0x168/0x390
[ 5748.047885] [c0000027ef5778b0] [c0000000000adcc8] .irq_exit+0xc8/0x110
[ 5748.047958] [c0000027ef577930] [c00000000001d15c] .timer_interrupt+0x9c/0xd0
[ 5748.048030] [c0000027ef5779b0] [c000000000002658] decrementer_common+0x158/0x180
[ 5748.048116] --- interrupt: 901 at .arch_local_irq_restore+0x74/0x90
[ 5748.048116]     LR = .arch_local_irq_restore+0x74/0x90
[ 5748.048224] [c0000027ef577ca0] [c0000027ef577d30] 0xc0000027ef577d30 (unreliable)
[ 5748.048309] [c0000027ef577d10] [c00000000070887c] .cpuidle_enter_state+0xac/0x260
[ 5748.048393] [c0000027ef577dd0] [c0000000000f6c90] .cpu_startup_entry+0x410/0x460
[ 5748.048478] [c0000027ef577ed0] [c000000000040b40] .start_secondary+0x310/0x340
[ 5748.048562] [c0000027ef577f90] [c000000000008b6c] start_secondary_prolog+0x10/0x14
[ 5748.048643] Instruction dump:
[ 5748.048680] 994d02d4 4bffff10 7fc3f378 4bfd5ff1 60000000 7fc4f378 7fe6fb78 7c651b78 
[ 5748.048799] 3c62ff75 3863afb8 48123721 60000000 <0fe00000> 39200001 3d02fff0 99282b88 
[ 5748.048920] ---[ end trace a04deb5eef8eba41 ]---
[ 5748.385369] mlx4_en: eth13:   frag:0 - size:1518 prefix:0 align:0 stride:1536
[ 5760.841519] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5781.165457] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5781.315631] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
[ 5782.165413] mlx4_en: eth13: CQE error - vendor syndrome: 0x6f syndrome: 0x2
Quoted 32 lines
Quoted 4 lines

But for mlx4_en, I am not sure I could raise the debug level with ethtool, since the ethernet driver may not work properly. Actually I am not sure how to raise the level with ethtool. Could you give me an example?

# ethtool -s ens5f1d1 msglvl 0xffff
Quoted 25 lines
Quoted 8 lines
Quoted 8 lines

And this error is reported from VF always. After the error, the other network interface seems can't function.

Quoted 2 lines

Amir.

Quoted 8 lines
Quoted 8 lines

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Richard Yang
Help you, Help me

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Richard Yang
Help you, Help me