Skip to contentSkip to conversation

netconsole: Initialize after all core networking drivers

PATCH In review Calvin Owens 6 participants 11 messages Dec 17, 2015 – Dec 29, 2015

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Calvin Owens

author
#0 lore raw

PATCH 1/1 netconsole: Initialize after all core networking drivers

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/Makefile +1 -1
index 900b0c5..31557d0 100644
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/

@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
 
 #
 # Networking Drivers
Signature · 1 line
2.5.0
#1 lore raw
Quoted 48 lines · Calvin Owens

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/Makefile +0 -1
index 900b0c5..31557d0 100644
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/

drivers/net/Makefile obj-$(CONFIG_VXLAN) += vxlan.o · line 31

Quoted 7 lines
drivers/net/Makefile +1 -0
@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
 
 #
 # Networking Drivers

Looks odd to rely on link order, but we might already rely on this...

Have you considered using device_initcall() instead of late_initcall() in vxlan ?

In any case, a comment would really be good to avoid future mistakes.

author
#2 lore raw
Quoted 54 lines · Eric Dumazet
Quoted 53 lines · Calvin Owens

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/Makefile +1 -1
index 900b0c5..31557d0 100644
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/

@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
Quoted 8 lines

drivers/net/Makefile obj-$(CONFIG_VXLAN) += vxlan.o · line 31

Quoted 2 lines
drivers/net/Makefile +0 -0
 #
 # Networking Drivers

Looks odd to rely on link order, but we might already rely on this...

Have you considered using device_initcall() instead of late_initcall() in vxlan ?

I'll look. As-is though, I think a similar problem would happen if you tried to use a virtio_net device with netconsole= cmdline (although that is admittedly a bizarre use case). The Makefile patch seemed like the best way to ensure this can't recur elsewhere.

Quoted 1 line

In any case, a comment would really be good to avoid future mistakes.

Good point, I'll add something.

author
#3 lore raw
Quoted 61 lines · Calvin Owens
Quoted 60 lines · Eric Dumazet
Quoted 48 lines · Calvin Owens

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/Makefile +0 -1
index 900b0c5..31557d0 100644
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/

drivers/net/Makefile obj-$(CONFIG_VXLAN) += vxlan.o · line 31

Quoted 7 lines
drivers/net/Makefile +1 -0
@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
 
 #
 # Networking Drivers

Looks odd to rely on link order, but we might already rely on this...

Quoted 4 lines
Quoted 2 lines

Have you considered using device_initcall() instead of late_initcall() in vxlan ?

I'll look.

So this does work, but commit 7332a13b038be05c explicitly changed it to late_initcall() because of dependencies on IPv6:

When vxlan is compiled as builtin, its init code runs before IPv6 init, this could cause problems if we create IPv6 socket in the latter patch.

So I guess something like the following patch is needed to go that route? It's ugly, IMHO the Makefile patch is cleaner...

Stephen / Cong, what do you think?

Quoted 4 lines

As-is though, I think a similar problem would happen if you tried to use a virtio_net device with netconsole= cmdline (although that is admittedly a bizarre use case). The Makefile patch seemed like the best way to ensure this can't recur elsewhere.

I misunderstood this, it works fine as is.

---8<---
From: Calvin Owens <[email protected]>
Subject: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()

Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible") changed vxlan to use late_initcall(), because vxlan relies on ipv6 being loaded when a new device is opened.

This causes netconsole to panic at boot when configured via the kernel cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has already been initialized:

netpoll: netconsole: device eth0 not up yet, forcing it ixgbe 0000:03:00.0: registered PHC device on eth0

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
  <snip>
  Call Trace:
  [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
  [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
  [<ffffffff8168045c>] __dev_open+0xac/0x120
  [<ffffffff81680506>] dev_open+0x36/0x70
  [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
  [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff81d79882>] init_netconsole+0xda/0x262
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
  [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
  [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
  [<ffffffff81778610>] ? rest_init+0x80/0x80
  [<ffffffff8177861e>] kernel_init+0xe/0xe0
  [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
  [<ffffffff81778610>] ? rest_init+0x80/0x80

This patch addresses the issue cited in 7332a13b038be05c by making vxlan actually check if ipv6 is loaded, and reverts it to module_init() so that it becomes device_initcall() when built-in, eliminating the netconsole issue.

The ipv6 module is permanent, so there's no need to actually do the usual module_get/module_put dance: once we find it loaded, we can just assume that it always will be.

AFAICS, nothing actually ends up calling vxlan_open() during initcalls, so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has initialized.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/vxlan.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)
drivers/net/vxlan.c +34 -1
index ba363ce..e3061b7 100644
@@ -158,6 +158,36 @@ static int vxlan_nla_put_addr(struct sk_buff *skb, int attr,
 		return nla_put_in_addr(skb, attr, ip->sin.sin_addr.s_addr);
 }
 
+#if IS_MODULE(CONFIG_IPV6)
+MODULE_SOFTDEP("pre: ipv6");
+
+/*
+ * IPv6 is permanent, so once we find it loaded we can just assume it always
+ * will be, and avoid walking the module_list every time we open a new vxlan
+ * device.
+ */
+static bool vxlan_ipv6_is_loaded(void)
+{
+	static bool ipv6_loaded = false;
+
+	if (!ipv6_loaded) {
+		mutex_lock(&module_mutex);
+		ipv6_loaded = !!find_module("ipv6");
+		mutex_unlock(&module_mutex);
+	}
+
+	return ipv6_loaded;
+}
+
+#elif IS_BUILTIN(CONFIG_IPV6)
+
+static bool vxlan_ipv6_is_loaded(void)
+{
+	return true;
+}
+
+#endif /* IS_{MODULE,BUILTIN}(CONFIG_IPV6) */
+
 #else /* !CONFIG_IPV6 */
 
 static inline

@@ -2628,6 +2658,9 @@ static struct socket *vxlan_create_sock(struct net *net, bool ipv6,
 	memset(&udp_conf, 0, sizeof(udp_conf));
 
 	if (ipv6) {
+		if (!vxlan_ipv6_is_loaded())
+			return ERR_PTR(-EAFNOSUPPORT);
+
 		udp_conf.family = AF_INET6;
 		udp_conf.use_udp6_rx_checksums =
 		    !(flags & VXLAN_F_UDP_ZERO_CSUM6_RX);

@@ -3259,7 +3292,7 @@ out1:
 	destroy_workqueue(vxlan_wq);
 	return rc;
 }
-late_initcall(vxlan_init_module);
+module_init(vxlan_init_module);
 
 static void __exit vxlan_cleanup_module(void)
 {
Signature · 1 line
2.5.0
build bot
#4 lore raw

Re: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()

reproduce:

Hi Calvin,

[auto build test ERROR on net-next/master]
[also build test ERROR on v4.4-rc6 next-20151223]

url:    https://github.com/0day-ci/linux/commits/Calvin-Owens/vxlan-Properly-depend-on-ipv6-and-revert-to-module_init/20151224-070557
config: i386-randconfig-b0-12241357 (attached as .config)
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386

All error/warnings (new ones prefixed by >>):

In file included from include/linux/linkage.h:4:0,
from include/linux/kernel.h:6, from drivers/net/vxlan.c:13:
drivers/net/vxlan.c: In function 'vxlan_create_sock':

>> drivers/net/vxlan.c:2661:8: error: implicit declaration of function 'vxlan_ipv6_is_loaded' [-Werror=implicit-function-declaration]
      if (!vxlan_ipv6_is_loaded())
           ^
   include/linux/compiler.h:147:28: note: in definition of macro '__trace_if'
     if (__builtin_constant_p((cond)) ? !!(cond) :   \
                               ^
>> drivers/net/vxlan.c:2661:3: note: in expansion of macro 'if'
      if (!vxlan_ipv6_is_loaded())
      ^

cc1: some warnings being treated as errors

vim +/vxlan_ipv6_is_loaded +2661 drivers/net/vxlan.c

2655 struct udp_port_cfg udp_conf;
2656 int err;
2657
2658 memset(&udp_conf, 0, sizeof(udp_conf));
2659
2660 if (ipv6) {

> 2661			if (!vxlan_ipv6_is_loaded())

2662 return ERR_PTR(-EAFNOSUPPORT);
2663
2664 udp_conf.family = AF_INET6;

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#5 lore raw
In reply to Calvin Owens

Hi,

Quoted 130 lines · Calvin Owens
Quoted 61 lines · Calvin Owens
Quoted 60 lines · Eric Dumazet
Quoted 48 lines · Calvin Owens

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>
---
 drivers/net/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/Makefile +0 -1
index 900b0c5..31557d0 100644
@@ -15,7 +15,6 @@ obj-$(CONFIG_MACVTAP) += macvtap.o
 obj-$(CONFIG_MII) += mii.o
 obj-$(CONFIG_MDIO) += mdio.o
 obj-$(CONFIG_NET) += Space.o loopback.o
-obj-$(CONFIG_NETCONSOLE) += netconsole.o
 obj-$(CONFIG_PHYLIB) += phy/
 obj-$(CONFIG_RIONET) += rionet.o
 obj-$(CONFIG_NET_TEAM) += team/

drivers/net/Makefile obj-$(CONFIG_VXLAN) += vxlan.o · line 31

Quoted 7 lines
drivers/net/Makefile +1 -0
@@ -26,6 +25,7 @@ obj-$(CONFIG_VXLAN) += vxlan.o
 obj-$(CONFIG_GENEVE) += geneve.o
 obj-$(CONFIG_NLMON) += nlmon.o
 obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_NETCONSOLE) += netconsole.o
 
 #
 # Networking Drivers

Looks odd to rely on link order, but we might already rely on this...

Quoted 4 lines
Quoted 2 lines

Have you considered using device_initcall() instead of late_initcall() in vxlan ?

I'll look.

So this does work, but commit 7332a13b038be05c explicitly changed it to late_initcall() because of dependencies on IPv6:

When vxlan is compiled as builtin, its init code runs before IPv6 init, this could cause problems if we create IPv6 socket in the latter patch.

So I guess something like the following patch is needed to go that route? It's ugly, IMHO the Makefile patch is cleaner...

Stephen / Cong, what do you think?

Quoted 4 lines

As-is though, I think a similar problem would happen if you tried to use a virtio_net device with netconsole= cmdline (although that is admittedly a bizarre use case). The Makefile patch seemed like the best way to ensure this can't recur elsewhere.

I misunderstood this, it works fine as is.

---8<---
From: Calvin Owens <[email protected]>
Subject: [PATCH] vxlan: Properly depend on ipv6 and revert to module_init()

Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible") changed vxlan to use late_initcall(), because vxlan relies on ipv6 being loaded when a new device is opened.

This causes netconsole to panic at boot when configured via the kernel cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has already been initialized:

netpoll: netconsole: device eth0 not up yet, forcing it ixgbe 0000:03:00.0: registered PHC device on eth0

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
  <snip>
  Call Trace:
  [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
  [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
  [<ffffffff8168045c>] __dev_open+0xac/0x120
  [<ffffffff81680506>] dev_open+0x36/0x70
  [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
  [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff81d79882>] init_netconsole+0xda/0x262
  [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
  [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
  [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
  [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
  [<ffffffff81778610>] ? rest_init+0x80/0x80
  [<ffffffff8177861e>] kernel_init+0xe/0xe0
  [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
  [<ffffffff81778610>] ? rest_init+0x80/0x80

This patch addresses the issue cited in 7332a13b038be05c by making vxlan actually check if ipv6 is loaded, and reverts it to module_init() so that it becomes device_initcall() when built-in, eliminating the netconsole issue.

The ipv6 module is permanent, so there's no need to actually do the usual module_get/module_put dance: once we find it loaded, we can just assume that it always will be.

Quoted 5 lines

AFAICS, nothing actually ends up calling vxlan_open() during initcalls, so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has initialized.

Signed-off-by Calvin Owens <[email protected]>

This architecture just sucks. :(

ixgbe should not have to call into vxlan but vxlan has to call to ixgbe. Thus the vxlan_get_rx_port is absolutely unnecessary and should be removed. This also lets ixgbe depend on vxlan which is absurd.

Simply let vxlan_get_rx_port be called from vxlan_notifier_block on NETDEV_REGISTER or NETDEV_UP events, which is already available.

For the second vxlan_get_rx_port case, which is a IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also push that over to the vxlan_notifier_block, maybe with a new event type for the notifiers.

After this change ixgbe would not depend on vxlan module any more.

Thanks,
Hannes

#6 lore raw
Quoted 28 lines · Hannes Frederic Sowa

Hi,

Quoted 9 lines · Calvin Owens

This patch addresses the issue cited in 7332a13b038be05c by making vxlan actually check if ipv6 is loaded, and reverts it to module_init() so that it becomes device_initcall() when built-in, eliminating the netconsole issue.

The ipv6 module is permanent, so there's no need to actually do the usual module_get/module_put dance: once we find it loaded, we can just assume that it always will be.

Quoted 5 lines

AFAICS, nothing actually ends up calling vxlan_open() during initcalls, so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has initialized.

Signed-off-by Calvin Owens <[email protected]>

This architecture just sucks. :(

ixgbe should not have to call into vxlan but vxlan has to call to ixgbe. Thus the vxlan_get_rx_port is absolutely unnecessary and should be removed. This also lets ixgbe depend on vxlan which is absurd.

Simply let vxlan_get_rx_port be called from vxlan_notifier_block on NETDEV_REGISTER or NETDEV_UP events, which is already available.

Quoted 7 lines

For the second vxlan_get_rx_port case, which is a IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also push that over to the vxlan_notifier_block, maybe with a new event type for the notifiers.

After this change ixgbe would not depend on vxlan module any more.

Agreed with this direction, but not yet convinced with this approach.

My feeling is that we should move this piece of code into core networking, so that both vxlan and ixgbe driver can call it without worrying about the dependencies. It looks like all we need here is a per-netns structure about family and ports for all vxlans. But I could overlook something here.

Thanks.

#7 lore raw
Quoted 38 lines · Cong Wang
Quoted 28 lines · Hannes Frederic Sowa

Hi,

Quoted 9 lines · Calvin Owens

This patch addresses the issue cited in 7332a13b038be05c by making vxlan actually check if ipv6 is loaded, and reverts it to module_init() so that it becomes device_initcall() when built-in, eliminating the netconsole issue.

The ipv6 module is permanent, so there's no need to actually do the usual module_get/module_put dance: once we find it loaded, we can just assume that it always will be.

Quoted 5 lines

AFAICS, nothing actually ends up calling vxlan_open() during initcalls, so in the (IPV6=y && VXLAN=y) case we can't end up there before ipv6 has initialized.

Signed-off-by Calvin Owens <[email protected]>

This architecture just sucks. :(

ixgbe should not have to call into vxlan but vxlan has to call to ixgbe. Thus the vxlan_get_rx_port is absolutely unnecessary and should be removed. This also lets ixgbe depend on vxlan which is absurd.

Simply let vxlan_get_rx_port be called from vxlan_notifier_block on NETDEV_REGISTER or NETDEV_UP events, which is already available.

Quoted 7 lines

For the second vxlan_get_rx_port case, which is a IXGBE_FLAG2_VXLAN_REREG_NEEDED needed event, I would suggest we also push that over to the vxlan_notifier_block, maybe with a new event type for the notifiers.

After this change ixgbe would not depend on vxlan module any more.

Quoted 7 lines

Agreed with this direction, but not yet convinced with this approach.

My feeling is that we should move this piece of code into core networking, so that both vxlan and ixgbe driver can call it without worrying about the dependencies. It looks like all we need here is a per-netns structure about family and ports for all vxlans. But I could overlook something here.

We actually have that already in terms of net_generic(vxlan_net_id). We will also need that for geneve and other offloads supported by hardware.

Your approach seems to be basically the same like mine, just in your proposal the driver calls to all the helper functions, while in my proposal the vxlan/geneve and other layers can update the offload settings on the hardware.

As the notifier_blocks are already available and registered, why not just use them?

Thanks,
Hannes

#8 lore raw
In reply to Calvin Owens
Quoted 7 lines · Calvin Owens

Commit 7332a13b038be05c ("vxlan: defer vxlan init as late as possible") changed vxlan to use late_initcall(), because vxlan relies on ipv6 being loaded when a new device is opened.

This causes netconsole to panic at boot when configured via the kernel cmdline on an IXGBE NIC, because ixgbe_open() assumes that vxlan has already been initialized:

BTW, it is not fair to blame my commit above, since it was merged long before ixgbe or any hardware driver has vxlan support. Not a big deal, just FYI. ;)

#9 lore raw
In reply to Calvin Owens
Quoted 26 lines · On Thu, 17 Dec 2015 15:52:39 -0800 Calvin Owens <[email protected]> wrote:

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80
Quoted 5 lines

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Signed-off-by Calvin Owens <[email protected]>

Fixing this by changing Makefile order is too fragile.
You are depending on the fact that Makefile order determines link order and that determines initialization order. Down that path demons lie.

author
#10 lore raw
Quoted 33 lines · Stephen Hemminger
Quoted 30 lines · On Thu, 17 Dec 2015 15:52:39 -0800 Calvin Owens <[email protected]> wrote:

With built-in netconsole and IXGBE, configuring netconsole via the kernel cmdline results in the following panic at boot:

    netpoll: netconsole: device eth0 not up yet, forcing it
    usb 2-1: new high-speed USB device number 2 using ehci-pci
    ixgbe 0000:03:00.0: registered PHC device on eth0
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000810
    <snip>
    Call Trace:
     [<ffffffff81578661>] ? vxlan_get_rx_port+0x41/0xa0
     [<ffffffff81586828>] ixgbe_open+0x4e8/0x540
     [<ffffffff8168045c>] __dev_open+0xac/0x120
     [<ffffffff81680506>] dev_open+0x36/0x70
     [<ffffffff8169abec>] netpoll_setup+0x23c/0x300
     [<ffffffff8169a66a>] ? netpoll_parse_options+0x19a/0x200
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff81d79882>] init_netconsole+0xda/0x262
     [<ffffffff81d797a8>] ? option_setup+0x1f/0x1f
     [<ffffffff810003a8>] do_one_initcall+0x88/0x1b0
     [<ffffffff81d31144>] kernel_init_freeable+0x14a/0x1e3
     [<ffffffff81d308f1>] ? do_early_param+0x8c/0x8c
     [<ffffffff81778610>] ? rest_init+0x80/0x80
     [<ffffffff8177861e>] kernel_init+0xe/0xe0
     [<ffffffff8177dc5f>] ret_from_fork+0x3f/0x70
     [<ffffffff81778610>] ? rest_init+0x80/0x80

This happens because IXGBE assumes that vxlan has already been initialized. The cleanest way to fix this is to just initialize netconsole after all the other core networking stuff has completed.

Quoted 5 lines
Quoted 1 line
Signed-off-by Calvin Owens <[email protected]>

Fixing this by changing Makefile order is too fragile.
You are depending on the fact that Makefile order determines link order and that determines initialization order. Down that path demons lie.

Hmm, include/linux/init.h explicitly states that "Ordering inside the subsections is determined by link order", and has since before the beginning of the history in Git.

I agree it seems magic/scary, but as Eric says I would imagine this behavior is relied upon in other places.